berklix.com logo

berklix.org logo

Brexit Stolen Votes

flag_uk_de_icon_v.gif

BSD-PIE icon

BSD icon

Gnu icon

Linux icon

Eyes

Disclaimer

IBU

Consol

FreeBSD Remote Upgrades

http://www.berklix.com/~jhs/txt/host_upgrade.html

Author Julian H. Stacey

Purpose

For:
  • Me mainly, as a convenience.
  • For FreeBSD experts remote upgrading many hosts in parallel.
  • Using mouse cut & pasting commands from remote xterms
  • Keeping a table of hosts & numeric action points completed,
  • To avoid needing persistent short & long term human memory, while other real life interrupts continue, eg network & power & other system outages etc, + customers & colleagues phoning & emailing, + breaks for lunch & going home during long over night & occasionaly week long procedures (eg ports/ re make) etc.
Not For:

Disclaimer: Total !

This can & very likely will easily destroy things, unless you'r an expert. For experts only. Hire author or another BSD Consultant.

Matrix

To copy then edit as you upgrade.
Horizontal Top: Action Numbers below
Vertical Left: Host names.
----- 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3
----- 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
host1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
host2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
host3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
host4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Keys: 'R' = Running,  '0' = Not tried,  '1' = Done,  'X' = Failed,  'N' = Next,  '-' = Skip (perhaps cos not enough space),  'W' = Waiting on something.

Colour Scheme Documenting Action Numbers

Grey Source Host as normal user.
Grey Bold Source Host as Root User
Orange Target Host as normal user
Orange Bold Target Host as Root User
Green Comment

HOST=TARGET

Action Numbers

  1. If this is an Internet live server, some days before, you have already changed timeouts for your domains, & patched out the IP of this box from your cluster.

    Assumptions:

    • AMD running (if local secure net).
    • Using Berklix standard AMD host mount path /host/hostname (not a FreeBSD default one).
    • Shell is csh with a built in echo (not eg /bin/sh needing /bin/echo).
      echo $0 | grep csh
    • Set pwd of root & toor & `whoami` (a normal login user) to known values. &
      grep `whoami` /etc/group | grep wheel
    • You have already manually saved all user data & any source hacks, creating diffs on another system.
    • echo "Do It Manually Now"

  2. Backup /etc/ configs to other hosts in case of catastrophe
    cd / && rdist6 -P /usr/bin/ssh -f /etc/Distfile
    Backup to a spare Fdisk partition (for patch rescues) & also data to another host (for catastrophic rescue) & label dates & uname -r in /etc/fstab & /etc/motd

  3. Make sure processor type of target can use what we
    have generated.

    dmesg | grep CPU:

    HOST=SOURCE
  4. one can boot a cross installed boot kernel, where SOURCE host is built with cflags 686 & then make.conf gets "CFLAGS += -march=i586" before building for TARGET cpu=586, However thats Only true for kernel. Even with -march=i586 a 686 produces unusable binaries for eg init.
    grep CFLAGS /etc/make.conf

    Freebsd-7.4 + std gcc 4.2.1 fails to honour -march=i586
    All these binaries were not executable on the 586: /boot/loader /sbin/init /bin/sh /bin/csh getty

  5. Ensure we have all binaries & sources ready/ uploaded
    cd /usr/src && make obj
    cd /usr/src && make world
    Or
    cd /usr/src && make all

  6. vi /usr/src/UPDATING
    Read & consider everything between old & new release, eg in the past it's been necessary to add a few passwd entries before make world would work (can't remember which, maybe some of _dhcp _pflogd _spamd _tss auditdistd bind daemon dlna haldaemon kmem messagebus proxy spamd unbound ) ; & periodicaly libc versions get increased, etc.
    To relate releases to dates see http://www.freebsd.org/releases/

  7. Build GENERIC kernel, in case our specific kernel fails to configure or boot.
    Build host specific kernel.
    cd /sys/`uname -m`/conf && config GENERIC
    cd /sys/`uname -m`/compile/GENERIC
    make cleandepend && make depend && make
    setenv DESTDIR /pri/freebsd/releases/`uname -r`/586/destdir
    Orsetenv DESTDIR /pri/freebsd/releases/`uname -r`/686/destdirmake install
    cd /sys/modules ; make install
    cd /sys ; make install  # recurses also into boot/
    cd /sys/`uname -m`/compile/$TARGET.small && make depend && make
    
    

  8. Build host specific kernel.
    cd /sys/`uname -m`/conf && config $TARGET.small
    cd /sys/`uname -m`/compile/$TARGET.small && make depend && make

    HOST=TARGET
  9. Make space: Remove src ports doc obj
    rm /usr/src/.c* /usr/ports/.c*
    Cope with possible symbolic links of src ports doc obj"
    foreach i ( src ports doc obj )
    echo "Emptying /usr/$i"
    ( rm -rf /usr/$i/* )
    end

  10. Check enough space free for new binaries (145M for 4.11, 332M for 7.4-RELEASE ) where /usr/destdir will move into file systems for / & /usr
    df / /usr*
    If not enough space try eg
    cd /usr && mv local /usr1/ && ln -s ../usr1/local /usr/local
    Either
    mkdir /usr/NEW.ALL
    Else
    (Which won't work so well, as moving across file systems when binaries are not in findable paths, & will fail & need manual intervention.
    mkdir /usr1/NEW.ALL
    ln -s -f ../usr1/NEW.ALL /usr/NEW.ALL

    HOST=SOURCE
  11. Copy across (but do not install) binaries & sources before booting new kernel, (in case reboot fails & we have to do local login). .
    setenv target host
    setenv TARGET `echo $target | dd conv=ucase`
    setenv DESTDIR /host/$target/usr/NEW.ALL
    mkdir /host/$target/usr/NEW.ALL

    Can't do a direct install to root of target cos of lib problems over NFS so stage it via /usr/NEW.ALL
    cd /usr/src
    pushd etc
    make distrib-dirs
    make distribution
    make etc-examples
    make install
    popd
    make install
    cd /sys/`uname -m`/compile/GENERIC.small ; make install
    cd /sys/`uname -m`/compile/HOST.small ; make install

    HOST=TARGET
    Test Binaries (make sure eg you haven't installed 586 bins on a 486).
    (The test above is not adequate, it doesn't blow up if eg on a 486 target you run a 586 mount binary.) cd /usr/NEW.ALL/sbin && ./mount

    HOST=TARGET
  12. Install src ports doc obj
    We did the binaries before this in case we ran out of space.

    cd /usr && mkdir src ports doc obj
    cd /usr && ls -al src ports doc obj
    cd /usr && chown -R -H jhs:staff src ports doc obj

    LOGIN=jhs
  13. If $TARGET is allowed AMD mounts by $SOURCE:
    foreach i ( src ports doc )
    cd /usr/$i && tar zxf /pri/FreeBSD/releases/7.4-RELEASE/tars/$i.tgz
    # Cannot yet use `uname -r` as target host is still running old release,
    # so type new release number manually.

    echo done $i
    end
    Else if $TARGET is a gate that $SOURCE does not allow AMD mounts from:
    HOST=SOURCE
    foreach i ( src ports doc )
    cd /host/$target/usr/$i && tar zxf /pri/FreeBSD/releases/`uname -r`/tars/$i.tgz
    echo done $i
    end

  14. cd /usr/ports && ln -s -f ../distfiles
    # Cannot yet use `uname -r` as target host is still running old release,
    # so type new release number manually.

    cd /usr/ports && ln -s -f ../packages && ln -s -f ../distfiles
    cd /usr/ports/.. && ln -s -f /pub/FreeBSD/dists/7.4-RELEASE distfiles
    cd /usr/ports/.. && ln -s -f /pub/FreeBSD/ports/`uname -m`/packages-`echo 7.4-RELEASE|dd conv=lcase` packages


  15. chflags noschg /boot/kernel/kernel*
    cp /boot/kernel/kernel /kernel.OLD

    HOST=SOURCE
  16. Optionally Install new live GENERIC kernel as fallback.
    Don't do this if eg
    • Your network card is not configured for a GENERIC config.
    • Your host is a HP Network ScanJet 5 & a Generic kernel will hang during boot on aic probe & never complete.

    setenv DESTDIR /host/$target/NEW
    cd /sys/`uname -m`/conf/../../compile/GENERIC && make install

  17. Install new live custom kernel (old release or new release Generic becomes fallback)
    setenv DESTDIR /host/$target/NEW
    cd /sys/`uname -m`/conf/../../compile/$TARGET.small && make install

    HOST=TARGET
  18. Do you have special /boot/loader.conf /boot/kernel.conf that might be essential to boot.
    find /boot -name \*.conf | grep -v /defaults/

    HOST=SOURCE
  19. Install new boot & kernel modules
    setenv DESTDIR /host/$target
    cd /sys && make install

    HOST=TARGET
  20. cd /etc/mail ; make stop
    vi -c/sendmail_enable /etc/rc.conf # sendmail_enable="NO"
    to avoid start on after reboot, before new configs in place
    cd /usr/local/etc/rc.d ; ./apache stop
    /etc/rc.d/named stop # kill -9 `cat /var/run/named/pid`
    ps -laxxww
    # look for anything else to kill
    echo "Upgrading now, kicking users off" | wall

    rsh local fetchmail # Drain mail

  21. Last chance to Think !
    rm -f /var/run/nologin /etc/nologin # (Path in /etc/login.conf & man sshd )
    Ensure no top directories lack permissions (eg from inherited umask from mv / cp) else eg if /bin is drwx------ you will get /bin/csh: Permission denied
    cd / ; ls -l | grep rwx---
    Ensure alternate rescue entries.
    grep "^PermitRootLogin" /etc/ssh/sshd_config
    Look for "yes", else if normal user login fails, you'll be stuck.
    grep toor /etc/passwd ; vipw
    Ensure rlogin & telnet also allow login, because after upgrading from 6.1 to 6.3 I got: ssh_exchange_identification: Connection closed by remote host
    mv /etc/pam.conf /etc/pam.conf.MV
    Ensure no obsolete /etc/pam.conf might lock you out.

    Optional sanity check before upgrade of binaries,
    Consider if the day & time of day allows human presence at remote site in case someone has to push reset &/or more drastc rescue,
    then:
    reboot


  22. Move new binary trees to right file systems.
    mkdir /NEW /OLD /usr/OLD /var/OLD
    mv /usr/NEW.ALL/var /var/NEW
    mv /usr/NEW.ALL/usr /usr/NEW
    Next line complex to avoid just moving a possible symbolic link
    You can't just do this
    cd /usr/NEW.ALL/../ && mv NEW.ALL /NEW
    because /rescue contains 121 sym links of 4 Meg, so do this:
    chown -R -noschg /usr/NEW.ALL
    ( cd /usr/NEW.ALL && tar cf - . ) | ( cd /NEW && tar xf - )
    cd /usr/NEW.ALL && find . -type f -exec cmpd -d {} /NEW \;

  23. Installing /var
    cd /var/NEW && foreach i ( * )
    echo "Starting $i"
    mv /var/$i /var/OLD/$i
    mv $i /var/$i
    echo "Finished $i"
    end
    rm -rf /var/NEW/*

  24. Installing /usr
    cd /usr/NEW && foreach i ( * )
    echo "Starting $i"
    mv /usr/$i /usr/OLD/$i
    mv $i /usr/$i
    echo "Finished $i"
    end

  25. Installing Root
    find /boot -name \*.conf | grep -v /defaults/
    ( cp /boot/loader.conf /NEW/boot/ )
    ( cp /boot/kernel.conf /NEW/boot/ )
    rm /NEW/sys
    foreach i ( .cshrc .profile dev etc root )
    mv /NEW/$i /$i.NEW
    end
    Copy tools to known location, normally only mv is needed, but if you get anything wrong, eg on the wrong file system, mv will fail to fork /bin/cp
    ( cd /bin && file cp ls mv pwd rm | grep "statically linked" )
    Ensure they are static, not dynamic.
    cd /bin && cp cp ls mv pwd rm / )
    cd /NEW && foreach i ( * )
    echo "Starting $i"
    /mv /$i /OLD/$i
    /mv /NEW/$i /$i
    echo "Finished $i"
    end
    (cd / && rm mv cp rm pwd ls )
    rm /etc/objformat /usr/bin/objformat

    cmp /usr/src/etc/rc /etc/rc # See if mergemaster has failed to update, (when that happened next reboot had no net & hostname Amnesiac)


  26. Test if system is bootable, & load new shared libraries etc.
    reboot

  27. Delete old binaries.
    rmdir /NEW /usr/NEW
    cd / && ls /OLD /usr/OLD
    chflags -R noschg /OLD /usr/OLD /kernel.Old
    rm -rf /mv /OLD /usr/OLD /kernel.Old
    ls /var/NEW
    echo "Do It Manually Now"

  28. Optionally fix some rc.conf & rc.local type errors, (visible after booting with X off). But some errors not worth fixing as we haven't upgraded /etc/ scripts yet.
    echo "Do It Manually Now"

  29. Manually also install /usr/local if a major release upgrade, where old shared libs wont work any more.
    echo "Do It Manually Now"

    LOGIN=jhs
  30. cd tmp && script
    Optionally, don't customise if anything might break world, eg SASL on a gate
    cd /usr/src && customise `pwd`
    (Or better customise ports too with a simpler more complete:)
    cd ~/tmp
    script
    customise

  31. cd /usr/src && make obj

    LOGIN=root
  32. Rebuild binaries with local /etc/make.conf CFLAGS
    cd /usr/src
    sh
    nohup make world &

    (nohup allows build (running for hours) to continue, even if you later reboot or halt or drop the link of the controlling screen.
    At any time monitoring is resumable with
    tail -f /host/$target/usr/src/nohup.out

  33. Compare & upgrade /etc with/from /usr/src/etc
    cd /etc && tar zcf ../etc.`date +%Y_%m_%d_%H_%M_%S`.tgz .
    mergemaster -sicv

  34. You need a new /etc/mail/sendmail.cf by now else you wont get error logs mailed
    cd /usr/obj/`cd /usr/src;/bin/pwd`/etc/sendmail;
    cp `hostname`.cf /etc/mail
    cp *submit* /etc/mail
    cd /etc/mail && csh

  35. Optionally reinstall packages, or mandatory if a major release upgrade. cd /var/db/pkg && ls
    pkg_info -a

    You may want to upgrade with some of /usr/ports/*/pkg*

    rlogin #source ; cd /host/$target/usr/local (cd /usr/local && tar cf - . ) | tar xf -
    rlogin #source ; cd /host/$target/var/db/pkg ( cd /var/db/pkg && tar cf - . ) | tar xf -


  36. Remake Ports: Optional on minor numbers, mandatory on major releases.
    portsupgrade
    Or
    cd /usr/ports ; nice make -k BERKLIX_CLIENT=YES BERKLIX_GATE=YES BERKLIX_MINIMAL=YES package
    Warning: FreeBSD is now in the habit of periodicaly discarding ports, so some won't rebuild & binaries may be old.

  37. Detect binaries that failed to rebuild.
    foreach i ( /*bin /usr/*bin /usr/local/bin )
    echo Listing $i
    ls -ltr $i
    echo " "
    end

  38. Look to see errors in /var/log/, especially messages.
    cd /var/log ; more messages

  39. If this is a live internet server, check:
    • ftpd: Ensure no incoming etc.
    • popd Enable before smtpd
    • smtp Enable before named
    • httpd Enable & check
      more /site/domain/this/usr/local/etc/apache/httpd.conf
    • named: re-enable the IP into domain cluster, & reset DNS timeouts to longer defaults.
      cd /site/domain/berklix.org/etc/namedb ; vi master.fwd ; make ; more /site/Distfile
      /site/domain/init
    • ssh: Check ssh allows login, then Optionally:
      vi /etc/inetd.conf # comment out rlogin
      telnet kill -HUP `cat /var/run/inetd.pid`
      # To make machine safer, (unless all users are aware rlogin & telnet are dangerous, can be packet sniffed to steal passwords, so normal users should use ssh.)
    • SASL:
      grep CFLAGS /etc/make.conf | grep -i sasl
      grep .include /etc/make.conf
    • cd /usr/src ; make world
    • customise
    • make ; make install
    • Upgrade /etc/mail/sendmail.cf & re-enable

To check merged: Done: root var usr generic & custom kernel local cp'd from past root: /root/.ssh /etc/rc.conf /etc/exports /var/cron/tabs/jhs cd /usr/src; make install

Berklix.Net Computer Associates Domains Apache: Web Server FreeBSD: Operating System Brexit Stolen Votes