FreeBSD Remote Upgrades

http://berklix.com/~jhs/txt/host_upgrade.html

Purpose

For:

Experts remote upgrading many hosts in parallel,
using mouse cut & pasting commands
keeping a table of hosts & numeric action points completed,
to avoid needing human memory & de-link from real life interrupts, eg breaks for going off line, to lunch, home, network or power outages & customers calling for support etc).

Not For:

Newbies learning how to upgrade single domestic systems: For that better look at standard FreeBSD documentation.

Disclaimer: Total !

This can easily destroy your software Hire me or some other BSD Consultant for help.

Sample Matrix

Ready to copy & use

HOST    0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3
HOST    1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0
aaaa    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
bbbb    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
cccc    . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Limitation

Does not yet allow for 5 & 6 kernel path in /boot/kernel not /

Colour Scheme

Blue	Source Host as normal user.
Blue Bold	Source Host as Root User
Orange	Target Host as normal user
Orange Bold	Target Host as Root User
Green	Comment

HOST=TARGET

Check Assumptions:

AMD running.
You are using a JHS standard AMD host mount path (not a FreeBSD default one).
Your shell is csh with a built in echo (not eg sh needing /bin/echo).
Set pwd of root & toor & a normal login user to know values & You have already manually saved all user data & any source hacks, creating diffs on another system.
If this is an Internet live server, some days before, you have already changed timeouts for your domains, & patched out the IP of this box from your cluster.

/bin/csh

Backup /etc/ configs to other hosts in case of catastrophe
cd / && rdist -f /etc/Distfile

Make sure processor type of target can take use what we
have generated.
dmesg | grep CPU:

HOST=SOURCE

I have proven one can boot a cross installed kernel, where SOURCE cflags has 686 & TARGET cpu=586, however, it would produce unusable src/ binaries, & I'm not sure of /boot stuff, so best avoid the risk.
grep CFLAGS /etc/make.conf

Ensure we have all binaries ready
cd /usr/src && make world
Or
cd /usr/src && make all

Build GENERIC kernel, in case our specific kernel fails to configure or boot.
cd /sys/i386/conf && config GENERIC
cd ../../compile/GENERIC && make depend && make

Build host specific kernel.
cd /sys/i386/conf && config $TARGET.small
cd /sys/compile/$TARGET.small && make depend && make

HOST=TARGET

Make space: Remove src ports doc obj
rm /usr/src/.c* /usr/ports/.c*
Cope with possible symbolic links of src ports doc obj"
foreach i ( src ports doc obj )

echo "Emptying /usr/$i"
( rm -rf /usr/$i/* )
end

Check over 145M free where /usr/destdir will target
df / /usr*
If not enough space try eg

cd /usr && mv X11R6 /usr1/ && ln -s ../usr1/X11R6 /usr/X11R6
cd /usr && mv local /usr1/ && ln -s ../usr1/local /usr/local

Either

mkdir /usr/NEW.ALL

Else

(Which won't work so well, as moving across file systems when binaries are not in findable paths, amp; will fail & need manual intervention.
mkdir /usr1/NEW.ALL
ln -s -f ../usr1/NEW.ALL /usr/NEW.ALL

HOST=SOURCE

Copy across (but do not install) binaries & sources before booting new kernel, (in case reboot fails & we have to do local login). .
setenv target host
setenv TARGET `echo $target | dd conv=ucase`
setenv DESTDIR /host/$target/usr/NEW.ALL
mkdir /host/$target/usr/NEW.ALL
Can't do a direct install to root of target cos of lib problems over NFS so stage it via /usr/NEW.ALL
cd /usr/src

pushd etc

make distrib-dirs
make distribution
make etc-examples
make install

popd
make install

HOST=TARGET
Test Binaries (make sure eg you haven't installed 586 bins on a 486).
(The test above is not adequate, it doesn't blow up if eg on a 486 target you run a 586 mount binary.) cd /usr/NEW.ALL/sbin && ./mount

HOST=TARGET

Install src ports doc obj
We did the binaries before this in case we ran out of space.
cd /usr && mkdir src ports doc obj
cd /usr && ls -al src ports doc obj
cd /usr && chown -R -H jhs:staff src ports doc obj

If $TARGET is allowed AMD mounts by $SOURCE:

foreach i ( src ports doc )

cd /usr/$i && tar zxf /pri/FreeBSD/releases/`uname -r`/tars/$i.tgz
echo done $i
end

Else if $TARGET is a gate that $SOURCE does not allow AMD mounts from:

HOST=SOURCE

foreach i ( src ports doc )

cd /host/$target/usr/$i && tar zxf /pri/FreeBSD/releases/`uname -r`/tars/$i.tgz
echo done $i
end

( cd /usr/ports && ls -l distfiles packages )
cd /usr/ports && ln -s -f ../distfiles
cd /usr/ports && ln -s -f ../packages
cd /usr/ports/.. && ln -s -f /pub/FreeBSD/dists/`uname -r` distfiles
cd /usr/ports/.. && ln -s -f /pub/FreeBSD/ports/i386/packages-`uname -r|dd conv=lcase` packages

chflags noschg /kernel*
cp /kernel /kernel.OLD

HOST=SOURCE

Optionally Install new live GENERIC kernel as fallback.
Don't do this if eg

Your network card is not configured for a GENERIC config.
Your host is a HP Network ScanJet 5 & a Generic kernel will hang during boot on aic probe & never complete.

setenv DESTDIR /host/$target
cd /sys/i386/conf/../../compile/GENERIC && make install

Install new live custom kernel (old release or new release Generic becomes fallback)
setenv DESTDIR /host/$target
cd /sys/i386/conf/../../compile/$TARGET.small && make install

HOST=TARGET

Do you have special /boot/loader.conf /boot/kernel.conf that might be essential to boot.
find /boot -name \*.conf | grep -v /defaults/

HOST=SOURCE

Install new boot & kernel modules
setenv DESTDIR /host/$target
cd /sys && make install

HOST=TARGET

Last chance to Think !
rm -f /var/run/nologin /etc/nologin # (Path in /etc/login.conf & man sshd )
Ensure no top directories lack permissions (eg from inherited umask from mv / cp) else eg if /bin is drwx------ you will get /bin/csh: Permission denied
cd / ; ls -l | grep rwx---
Ensure alternate rescue entries.
grep "^PermitRootLogin" /etc/ssh/sshd_config
Look for "yes", else if normal user login fails, you'll be stuck.
grep toor /etc/passwd ; vipw
reboot

Move new binary trees to right file systems.
mkdir /OLD /usr/OLD /var/OLD
mv /usr/NEW.ALL/var /var/NEW
mv /usr/NEW.ALL/usr /usr/NEW
cd /usr/NEW.ALL/../ && mv NEW.ALL /NEW

Installing /usr
( rmdir /usr/NEW/* )
cd /usr/NEW && foreach i ( * )

echo "Starting $i"
mv /usr/$i /usr/OLD/$i
mv $i /usr/$i
echo "Finished $i"
end

Installing Root
find /boot -name \*.conf | grep -v /defaults/

( cp /boot/loader.conf /NEW/boot/ )
( cp /boot/kernel.conf /NEW/boot/ )

( rmdir /NEW/* )
rm /NEW/sys
foreach i ( .cshrc .profile dev etc root )

mv /NEW/$i /$i.NEW
end

Copy tools to known location, normally only mv is needed, but if you get anything wrong, eg on the wrong file system, mv will fail to fork /bin/cp
(cd /bin && file cp mv ;
Ensure they are static, not dynamic.
cd /bin && cp mv cp rm pwd ls / )
cd /NEW && foreach i ( * )

echo "Starting $i"
/mv /$i /OLD/$i
/mv /NEW/$i /$i
echo "Finished $i"
end

(cd / && rm mv cp rm pwd ls )

Check system is bootable, & load new shared libraries etc.
reboot

Delete old binaries to save space.
rmdir /NEW /usr/NEW
cd / && ls /OLD /usr/OLD
chflags -R noschg /OLD /usr/OLD /kernel.Old
rm -rf /OLD /usr/OLD /kernel.Old
ls /var/NEW
/bin/csh

Optionally fix some rc.conf & rc.local type errors, (visible after booting with X off). But some errors not worth fixing as we haven't upgraded /etc/ scripts yet.
/bin/csh

Manually also install /usr/X11R6 & /usr/local if a major release upgrade, where old shared libs wont work any more.
/bin/csh

LOGIN=jhs

cd tmp && script

Optionally, don't customise if anything might break world, eg SASL on a gate

cd /usr/src && customise `pwd`
(Or better customise ports too with a simpler more complete:)
cd ~/tmp
script
customise

cd /usr/src && make obj

LOGIN=root

Rebuild binaries with local /etc/make.conf CFLAGS
cd /usr/src
sh
nohup make world &

(nohup allows build (running for hours) to continue, even if you later reboot or halt or drop the link of the controlling screen.
At any time monitoring is resumable with

tail -f /host/$target/usr/src/nohup.out

Compare & upgrade /etc with/from /usr/src/etc
cd /etc && tar zcf ../etc.`date +%Y_%m_%d_%H_%M_%S`.tgz .
mergemaster -sicv

You need a new /etc/mail/sendmail.cf by now else you wont get error logs mailed
cd /etc/mail && csh

Optionally reinstall packages, or mandatory if a major release upgrade. cd /var/db/pkg && ls
pkg_info -a
You may want to upgrade with some of /usr/ports/*/pkg*

Optionally remake ports, or mandatory if a major release.
portsupgrade

Detect binaries you failed to rebuild.
foreach i ( /*bin /usr/*bin /usr/local/bin /usr/X11R6/bin )

echo Listing $i
ls -ltr $i
echo " "
end

Look to see errors in /var/log/, especially messages.
cd /var/log ; more messages

If this is a live internet server, after checking functional smtp, popd, httpd, ftpd, re-enable the IP into your domains cluster, & reset DNS timeouts to defaults.

Partial Transcript of a manual remote upgrade from FreeBSD-6.1RELEASE to 6.3

by Julian Stacey http://berklix.com/~jhs/

Caveat Emptor (Warning!): If this bites you, hard luck. It is for mouse cut & paste / aide memoire, FreeBSD advise non experts to use an official upgrade procedure on their http://freebsd.org This below is NOT safe, & bits missing. (Diffs welcome).

vi -c/sendmail_enable /etc/rc.conf # sendmail_enable="NO" 
cd /etc/mail ; make stop 
rsh local fetchmail # Drain mail 
# named remove server from MX & http cluster, edit 
# ensure rlogin & telnet also allow login, 
# because after upgrading from FreeBSD-6.1-RELEASE to 6.3 I got: 
# ssh_exchange_identification: Connection closed by remote host 
setenv DESTDIR /usr/dest 
cd /usr/src/etc 
make distrib-dirs 
cd .. 
make -k install 
cd /usr/dest 
chflags -R noschg . 
mv usr /usr/NEW 
mkdir /NEW 
(tar cf - . ) | ( cd /NEW && tar xf - ) 
mkdir /OLD /var/OLD /usr/OLD
cd /usr/NEW 
foreach i ( * ) 
	chflags -R noschg ../$i 
	end 
foreach i ( l* [b-i]* s* ) 
	echo start $i 
	pushd $i 
	foreach j ( * ) 
		/mv /usr/$i/$j /usr/OLD/$i/ 
		/mv $j /usr/$i/ 
		end 
	popd 
	echo done $i 
	end 
cd /NEW 
find . -type d -depth -exec rmdir {} \; 
mv etc /etc.NEW 
mv /rescue /OLD/ 
mv rescue / 
ls boot # note kernel already in /boot 
ls -1 # bin/ boot/ lib/ libexec/ sbin/ 
# Some things always fail to compile during upgrades, eg: /sbin/ifconfig 
# could be fatal for remote reboot , so we do not just move whole 
# directories (but individual files (which may leave obsolete dirt 
# behind). 
ls
rm -rf .a-z]* root boot/loader.conf
ls -1	# bin boot lib libexec root sbin
ls -l boot/kernel
foreach i ( l* b* s* ) 
	echo start $i 
	chflags -R noschg /$i 
	pushd $i 
	foreach j ( * ) 
	# this may syntax error on ./[ 
	/mv /$i/$j /OLD/$i/ 
	/mv $j /$i/ 
	end 
popd 
echo done $i 
end 
#	override r-xr-xr-x  root/wheel for /libexec/ld-elf.so.1? (y/n [n]) 
#	y
reboot 
rm /mv 
# After proving ssh allows login, Optionally 
# vi /etc/inetd.conf # comment out rlogin & telnet 
# kill -HUP `cat /var/run/inetd.pid` 
# to make machine safer, 
# (unless all users are aware rlogin & telnet are dangerous, 
# can be packet sniffed to steal passwords, so normal users should use ssh.) 
grep CFLAGS /etc.make.conf | grep i sasl    #
cd /usr/src ; make world
customise `pwd`
make ; make install
# Upgrade /etc/mail/sendmail.cf & re-enable 
cd /usr/ports 
tar zxf /pri/freebsd/releases/`uname -r`/tars/ports.tgz
customise `pwd`

FreeBSD Remote Upgrades

http://berklix.com/~jhs/txt/host_upgrade.html

Author Julian Stacey

Purpose

Disclaimer: Total !

Sample Matrix

Limitation

Colour Scheme

Partial Transcript of a manual remote upgrade from FreeBSD-6.1RELEASE to 6.3

by Julian Stacey http://berklix.com/~jhs/

Domains