berklix.com logo

BSD-PIE icon

berklix.org logo

BSD icon
Gnu icon
Linux icon

Disclaimer

FreeBSD Remote Upgrades

http://www.berklix.com/~jhs/txt/host_upgrade.html

Author Julian Stacey

Purpose

For:
  • Experts remote upgrading many hosts in parallel,
  • Using mouse cut & pasting commands from remote xterms
  • Keeping a table of hosts & numeric action points completed,
  • To avoid needing human memory & de-link from real life interrupts, eg breaks for going off line, to lunch, home network & power outages & customers calling for other support etc).
Not For:
Newbies learning how to upgrade single domestic systems: For that first learn from standard FreeBSD documentation.

Disclaimer: Total !

This can Very easily destroy things. For experts only. Hire author or another BSD Consultant.

Matrix

To copy then edit as you upgrade.
Horizontal Top: Action Numbers below
Vertical Left: Host names. 
XXXX 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3
XXXX 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8
aaaa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
bbbb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
cccc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
dddd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Keys: R = Running, 0 = Not tried, 1 = Done, X = Failed, N = Next, - = Skip (perhaps cos not enough space), W = Waiting on something.

Colour Scheme Documenting Action Numbers

Blue Source Host as normal user.
Blue Bold Source Host as Root User
Orange Target Host as normal user
Orange Bold Target Host as Root User
Green Comment

HOST=TARGET

Action Numbers

  1. If this is an Internet live server, some days before, you have already changed timeouts for your domains, & patched out the IP of this box from your cluster.

    Check Assumptions:

    • AMD running.
    • You are using a JHS standard AMD host mount path (not a FreeBSD default one).
    • Your shell is csh with a built in echo (not eg sh needing /bin/echo).
    • Set pwd of root & toor & a normal login user to know values & You have already manually saved all user data & any source hacks, creating diffs on another system.

    /bin/csh

  2. Backup /etc/ configs to other hosts in case of catastrophe
    cd / && rdist6 -P /usr/bin/ssh -f /etc/Distfile

  3. Make sure processor type of target can use what we
    have generated.
    dmesg | grep CPU:

    HOST=SOURCE
  4. I have proven one can boot a cross installed kernel, where SOURCE cflags has 686 & TARGET cpu=586, however, it would produce unusable src/ binaries, & I'm not sure of /boot stuff, so best avoid the risk.
    grep CFLAGS /etc/make.conf

  5. Ensure we have all binaries ready
    cd /usr/src && make world
    Or
    cd /usr/src && make all

  6. Build GENERIC kernel, in case our specific kernel fails to configure or boot.

  7. Build host specific kernel.
    cd /sys/i386/conf && config $TARGET.small
    cd /sys/compile/$TARGET.small && make depend && make

    HOST=TARGET
  8. Make space: Remove src ports doc obj
    rm /usr/src/.c* /usr/ports/.c*
    Cope with possible symbolic links of src ports doc obj"
    foreach i ( src ports doc obj )
    echo "Emptying /usr/$i"
    ( rm -rf /usr/$i/* )
    end

  9. Check enough space free for new binaries (145M for 4.11, 370M for 7.2-RELEASE ) where /usr/destdir will target
    df / /usr*
    If not enough space try eg
    cd /usr && mv local /usr1/ && ln -s ../usr1/local /usr/local
    Either
    mkdir /usr/NEW.ALL
    Else
    (Which won't work so well, as moving across file systems when binaries are not in findable paths, amp; will fail & need manual intervention.
    mkdir /usr1/NEW.ALL
    ln -s -f ../usr1/NEW.ALL /usr/NEW.ALL

    HOST=SOURCE
  10. Copy across (but do not install) binaries & sources before booting new kernel, (in case reboot fails & we have to do local login). .
    setenv target host
    setenv TARGET `echo $target | dd conv=ucase`
    setenv DESTDIR /host/$target/usr/NEW.ALL
    mkdir /host/$target/usr/NEW.ALL
    Can't do a direct install to root of target cos of lib problems over NFS so stage it via /usr/NEW.ALL
    cd /usr/src
    pushd etc
    make distrib-dirs
    make distribution
    make etc-examples
    make install
    popd
    make install
    cd /sys/i386/compile/GENERIC.small ; make install
    cd /sys/i386/compile/HOST.small ; make install
    HOST=TARGET
    Test Binaries (make sure eg you haven't installed 586 bins on a 486).
    (The test above is not adequate, it doesn't blow up if eg on a 486 target you run a 586 mount binary.) cd /usr/NEW.ALL/sbin && ./mount

    HOST=TARGET
  11. Install src ports doc obj
    We did the binaries before this in case we ran out of space.
    cd /usr && mkdir src ports doc obj
    cd /usr && ls -al src ports doc obj
    cd /usr && chown -R -H jhs:staff src ports doc obj

    LOGIN=jhs
  12. If $TARGET is allowed AMD mounts by $SOURCE:
    foreach i ( src ports doc )
    cd /usr/$i && tar zxf /pri/FreeBSD/releases/7.2-RELEASE/tars/$i.tgz
    echo done $i
    end
    Else if $TARGET is a gate that $SOURCE does not allow AMD mounts from:
    HOST=SOURCE
    foreach i ( src ports doc )
    cd /host/$target/usr/$i && tar zxf /pri/FreeBSD/releases/`uname -r`/tars/$i.tgz
    echo done $i
    end

  13. cd /usr/ports && ln -s -f ../distfiles
    cd /usr/ports && ln -s -f ../packages
    cd /usr/ports/.. && ln -s -f /pub/FreeBSD/dists/7.2-RELEASE distfiles
    cd /usr/ports/.. && ln -s -f /pub/FreeBSD/ports/i386/packages-`echo 7.2-RELEASE|dd conv=lcase`     packages

  14. chflags noschg /boot/kernel/kernel*
    cp /boot/kernel/kernel /kernel.OLD

    HOST=SOURCE
  15. Optionally Install new live GENERIC kernel as fallback.
    Don't do this if eg
    • Your network card is not configured for a GENERIC config.
    • Your host is a HP Network ScanJet 5 & a Generic kernel will hang during boot on aic probe & never complete.

    setenv DESTDIR /host/$target/NEW
    cd /sys/i386/conf/../../compile/GENERIC && make install

  16. Install new live custom kernel (old release or new release Generic becomes fallback)
    setenv DESTDIR /host/$target/NEW
    cd /sys/i386/conf/../../compile/$TARGET.small && make install

    HOST=TARGET
  17. Do you have special /boot/loader.conf /boot/kernel.conf that might be essential to boot.
    find /boot -name \*.conf | grep -v /defaults/

    HOST=SOURCE
  18. Install new boot & kernel modules
    setenv DESTDIR /host/$target
    cd /sys && make install

    HOST=TARGET
  19. cd /etc/mail ; make stop
    vi -c/sendmail_enable /etc/rc.conf # sendmail_enable="NO"     to avoid start on after reboot, before new configs in place
    cd /usr/local/etc/rc.d ; ./apache stop
    /etc/rc.d/named stop # kill -9 `cat /var/run/named/pid`
    ps -laxxww     # look for anything else to kill
    echo "Upgrading now, kicking users off" | wall
    rsh local fetchmail # Drain mail
  20. Last chance to Think !
    rm -f /var/run/nologin /etc/nologin # (Path in /etc/login.conf & man sshd )
    Ensure no top directories lack permissions (eg from inherited umask from mv / cp) else eg if /bin is drwx------ you will get /bin/csh: Permission denied
    cd / ; ls -l | grep rwx---
    Ensure alternate rescue entries.
    grep "^PermitRootLogin" /etc/ssh/sshd_config
    Look for "yes", else if normal user login fails, you'll be stuck.
    grep toor /etc/passwd ; vipw
    Ensure rlogin & telnet also allow login, because after upgrading from 6.1 to 6.3 I got: ssh_exchange_identification: Connection closed by remote host
     mv /etc/pam.conf /etc/pam.conf.MV
    Ensure no obsolete /etc/pam.conf might lock you out.

    Optional sanity check febore upfrade of binaries: reboot

  21. Move new binary trees to right file systems.
    mkdir /OLD /usr/OLD /var/OLD
    mv /usr/NEW.ALL/var /var/NEW
    mv /usr/NEW.ALL/usr /usr/NEW
     Next line complex to avoid just moving a possible symbolic link
    You cant just do this
    cd /usr/NEW.ALL/../ && mv NEW.ALL /NEW
    because /rescue contains 121 sym links of 4 Meg, so do this:
    ( cd /usr/NEW.ALL && tar cf - . ) | ( cd /NEW && tar xf - )

  22. Installing /var
    ( rmdir /var/NEW/* )
    cd /var/NEW && foreach i ( * )
    echo "Starting $i"
    mv /var/$i /var/OLD/$i
    mv $i /var/$i
    echo "Finished $i"
    end

  23. Installing /usr
    ( rmdir /usr/NEW/* )
    cd /usr/NEW && foreach i ( * )
    echo "Starting $i"
    mv /usr/$i /usr/OLD/$i
    mv $i /usr/$i
    echo "Finished $i"
    end

  24. Installing Root
    find /boot -name \*.conf | grep -v /defaults/
    ( cp /boot/loader.conf /NEW/boot/ )
    ( cp /boot/kernel.conf /NEW/boot/ )
    ( rmdir /NEW/* )
    rm /NEW/sys
    foreach i ( .cshrc .profile dev etc root )
    mv /NEW/$i /$i.NEW
    end
    Copy tools to known location, normally only mv is needed, but if you get anything wrong, eg on the wrong file system, mv will fail to fork /bin/cp
    (cd /bin && file cp ls mv pwd rm | grep "statically linked" ;
    Ensure they are static, not dynamic.
    cd /bin && cp cp ls mv pwd rm / )
    cd /NEW && foreach i ( * )
    echo "Starting $i"
    /mv /$i /OLD/$i
    /mv /NEW/$i /$i
    echo "Finished $i"
    end
    (cd / && rm mv cp rm pwd ls )
    rm /etc/objformat /usr/bin/objformat

    cmp /usr/src/etc/rc /etc/rc # See if mergemaster has failed to update, (when that happened next reboot had no net & hostname Amnesiac)

  25. Check system is bootable, & load new shared libraries etc.
    reboot

  26. Delete old binaries.
    rmdir /NEW /usr/NEW
    cd / && ls /OLD /usr/OLD
    chflags -R noschg /OLD /usr/OLD /kernel.Old
    rm -rf /mv /OLD /usr/OLD /kernel.Old
    ls /var/NEW
    /bin/csh

  27. Optionally fix some rc.conf & rc.local type errors, (visible after booting with X off). But some errors not worth fixing as we haven't upgraded /etc/ scripts yet.
    /bin/csh

  28. Manually also install /usr/local if a major release upgrade, where old shared libs wont work any more.
    /bin/csh

    LOGIN=jhs
  29. cd tmp && script
    Optionally, don't customise if anything might break world, eg SASL on a gate
    cd /usr/src && customise `pwd`
    (Or better customise ports too with a simpler more complete:)
    cd ~/tmp
    script
    customise

  30. cd /usr/src && make obj

    LOGIN=root
  31. Rebuild binaries with local /etc/make.conf CFLAGS
    cd /usr/src
    sh
    nohup make world &
    (nohup allows build (running for hours) to continue, even if you later reboot or halt or drop the link of the controlling screen.
    At any time monitoring is resumable with
    tail -f /host/$target/usr/src/nohup.out

  32. Compare & upgrade /etc with/from /usr/src/etc
    cd /etc && tar zcf ../etc.`date +%Y_%m_%d_%H_%M_%S`.tgz .
    mergemaster -sicv

  33. You need a new /etc/mail/sendmail.cf by now else you wont get error logs mailed
    cd /usr/obj/`cd /usr/src;/bin/pwd`/etc/sendmail;
    cp `hostname`.cf /etc/mail
    cp *submit* /etc/mail
    cd /etc/mail && csh

  34. Optionally reinstall packages, or mandatory if a major release upgrade. cd /var/db/pkg && ls
    pkg_info -a
    You may want to upgrade with some of /usr/ports/*/pkg*

    rlogin #source ; cd /host/$target/usr/local (cd /usr/local && tar cf - . ) | tar xf -
    rlogin #source ; cd /host/$target/var/db/pkg ( cd /var/db/pkg && tar cf - . ) | tar xf -

  35. Optionally remake ports, or mandatory if a major release.
    portsupgrade

  36. Detect binaries that failed to rebuild.
    foreach i ( /*bin /usr/*bin /usr/local/bin )
    echo Listing $i
    ls -ltr $i
    echo " "
    end

  37. Look to see errors in /var/log/, especially messages.
    cd /var/log ; more messages

  38. If this is a live internet server, check:
    • ftpd: Ensure no incoming etc.
    • popd Enable befor smtpd
    • smtp Enable before named
    • httpd Enable & check
      more /site/domain/this/usr/local/etc/apache/httpd.conf
    • named: re-enable the IP into domain cluster, & reset DNS timeouts to longer defaults.
      vi .... rdist6 ....
      /site/domain/init
    • ssh: Check ssh allows login, then Optionally:
      vi /etc/inetd.conf # comment out rlogin & telnet kill -HUP `cat /var/run/inetd.pid` to make machine safer, (unless all users are aware rlogin & telnet are dangerous, can be packet sniffed to steal passwords, so normal users should use ssh.)
    • SASL:grep CFLAGS /etc.make.conf | grep i sasl #
    • cd /usr/src ; make world
    • customise
    • make ; make install
    • Upgrade /etc/mail/sendmail.cf & re-enable

Berklix.Net Computer Associates Domains Apache: Web Server FreeBSD: Operating System