Mass Upgrade Infrastructure SOP

Mass updates are usually applied every few months, or sooner if there’s some critical bugs fixed. Mass updates are done outside of freeze windows to avoid causing any problems for Fedora releases.

The following items are all done before the actual mass update:

Typically updates/reboots are done in three seperate steps:

It has been somewhat common to use Mon/Tues/Wed for these different steps, but as long as the production outage is not on a Friday it is fine to fit it around people’s schedule.

Some hosts can be safely updated/rebooted without an outage because they either have multiple machines in a load balancer or are not visible to end users or other reasons. These updates are typically done outside the outage window, so they are done before the outage. These hosts include proxies and a number of virthosts that have VM’s that meet this criteria.

While this may not be a complete list, here are some special things that must be taken into account before rebooting certain systems:

Each update should have a Leader appointed. This person will be in charge of doing any read-write operations, and delegating to others to do tasks. If you aren’t specficially asked by the Leader to reboot or change something, please don’t. The Leader will assign out machine groups to reboot, or ask specific people to look at machines that didn’t come back up from reboot or aren’t working right after reboot. It’s important to avoid multiple people operating on a single machine in a read-write manner and interfering with changes.

Usally for a mass update/reboot there will be a hackmd or similar document that tracks what machines have already been rebooted and who is working on which one. Please check with the leader for a link to this document.

People should mostly use the vhost_update_reboot.yml playbook which runs both vhost_update.yml to apply updates and vhost_reboot.yml to reboot the host and VM’s. This can be called via. rbac-playbook so that non-sysadmin-main people can help (need sysadmin-updates).

For hosts out of outage you probably want to use these to make sure updates are applied before reboots (apply updates can take a lot of time, esp. when not done in parallel). However once updates are applied globally before the outage you will still want to use the update_reboot playbook (but the update part should be very fast).

By far the most common problem we have is that machines don’t come back after a reboot. This is usually a firmware booting problem, or a luks problem. Both can be solved by logging into the console for the vmhost and seeing what the error is and fixing it manually.

To monitor a machine you are update/rebooting in another window (also from batcave01) you can run: mtr --displaymode 1 -i 4 <host>

Also read the howto restart a server docs.

Additionally you should use the updates-uptime-cmd.py python script on batcave01 to see what machines have updates available and/or need to be rebooted. You’ll need to run the update sub-command, before viewing the usual information, to get the latest data.

There are older playbooks check-for-nonvirt-updates.yml and check-for-updates.yml, but the above script should be easier to use and give clearer results.

If possible, system upgrades should be done in advance of the reboot (with relevant testing of new packages on staging). To do the upgrades, make sure that the Infrastructure RHEL repo is updated as necessary to pull in the new packages (Infrastructure Yum Repo SOP)

Before outage, ansible can be used to just apply all updates to hosts or apply all updates to staging hosts before those are done. Something like: ansible -m shell 'yum clean all; yum update -y; rkhunter --propupd' hostlist