Upgrading a hardened Linux image to a new release

This page explains how to move from an instance running one release of a ToTheNOC hardened image to a new instance running a newer release. It applies to every image, on AWS and on Google Cloud. How each release is built and verified, and its evidence: Hardened images.

How releases work

A new release changes nothing on the instances you already run. It applies only to instances launched from it. A running instance is yours: you keep it updated (sudo dnf upgrade on AlmaLinux and Rocky Linux, sudo apt update && sudo apt upgrade on Debian) and you keep it hardened. sudo ttn-cis-audit tells you at any time where it stands.

Updating in place is enough for security fixes. A new release brings a fresh build: a newer base image from the distribution, the hardening changes of the release, and a clean system with nothing accumulated. To get those, you replace the instance. The layout below makes that a matter of minutes.

Recommended layout: data on its own disk

Keep the operating system on the boot disk and everything that is yours on a second disk: application data, databases, uploads, the configuration of your applications. The instance then becomes replaceable: you launch the new release, move the disk, and start your applications again. Nothing is copied.

  • AWS: a separate EBS volume. Google Cloud: a separate persistent disk.
  • Mount it at a path of your choice, for example /srv, with nodev,nosuid.
  • Install your applications with a script, a configuration tool or a startup script, so that the new instance gets them the same way the old one did.
  • Create the users and groups that own your data with fixed UIDs and GIDs, the same on every instance. Otherwise the files on the moved disk end up owned by the wrong user.

Prepare the data disk (once)

Attach a new empty disk to the instance, then, on the instance (replace /dev/DISK with the device name shown by lsblk):

sudo mkfs.xfs /dev/DISK          # or mkfs.ext4 on Debian
sudo mkdir -p /srv
UUID=$(sudo blkid -s UUID -o value /dev/DISK)
echo "UUID=$UUID /srv xfs defaults,nodev,nosuid,nofail 0 2" | sudo tee -a /etc/fstab
sudo mount /srv

On Debian, use ext4 instead of xfs in the fstab line if you formatted with ext4. nofail lets the instance boot even if the disk is not attached.

AlmaLinux and Rocky Linux run SELinux in enforcing mode. If your application keeps its data outside its default directories, label the new path for it, for example for a web server:

sudo semanage fcontext -a -t httpd_sys_content_t "/srv/www(/.*)?"
sudo restorecon -Rv /srv/www

Move to a new release

  1. Launch a new instance from the new release, in the same network, with the same security group or firewall rules. Install your applications on it, but do not start them yet.
  2. Stop your applications on the old instance, then sudo umount /srv.
  3. Move the disk. Detach it from the old instance and attach it to the new one (EC2 console or aws ec2 detach-volume / attach-volume; Compute Engine console or gcloud compute instances detach-disk / attach-disk).
  4. Mount it on the new instance: add the same fstab line, sudo mount /srv, and on AlmaLinux and Rocky Linux sudo restorecon -Rv /srv.
  5. Start your applications and check them. If you use an Elastic IP or a static external IP, move it to the new instance.
  6. Check the hardening: sudo ttn-cis-audit; echo "exit code: $?". It shows what your applications changed against the baseline of the new release.
  7. Keep the old instance stopped for a few days. If something is wrong, move the disk back and start it again. When you are sure, delete it.

If everything is on the boot disk

Launch the new instance and copy what is yours from the old one, over SSH:

  • Application files and data: rsync -aHAX of the directories you use (/srv, /opt, /var/www, home directories).
  • Databases: dump on the old instance and restore on the new one (mysqldump, pg_dump), not a copy of the database files.
  • From /etc, only the configuration of your applications: their own configuration files, systemd units, virtual hosts, certificates, cron jobs. Do not copy /etc as a whole: it would overwrite the hardening of the new release.

Then run sudo ttn-cis-audit on the new instance, and consider moving your data to a separate disk while you are at it.

Support

Questions about moving a workload to a new release: support@tothenoc.com. You get an answer from the engineer who builds the images.