Fedora CoreOS Cincinnati SOP

Cincinnati is the update service/backend for Fedora CoreOS (FCOS) machines. This SOP describes how to access and how to troubleshoot it.

Contact Information

Owner

Fedora CoreOS Team

Contact

#fedora-coreos

Details

Source

https://github.com/coreos/fedora-coreos-cincinnati

Playbook

https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/openshift-apps/coreos-cincinnati.yml

Location

OpenShift cluster (production): https://console-openshift-console.apps.ocp.fedoraproject.org/

Project

coreos-cincinnati: https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/cluster/projects/coreos-cincinnati

Deployment

https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/deploymentconfigs/coreos-cincinnati

Containers

  • fcos-graph-builder (GB - raw updates graph)

  • fcos-policy-engine (PE - frontend handling client requests)

Routes

  • coreos-updates-raw (GB web service)

  • coreos-updates-raw-status (GB status and metrics)

  • coreos-updates (PE web service)

  • coreos-updates-status (PE status and metrics)

Troubleshooting

Each FCOS Cincinnati service exposes live metrics in Prometheus format:

Graph-builder

https://status.raw-updates.coreos.fedoraproject.org/metrics

Policy-engine

https://status.updates.coreos.fedoraproject.org/metrics

Upgrades

Building a new version

FCOS Cincinnati is built as container image directly from source, referencing a pinned git commit.

In order to build a new version, you will first have to find the relevant commit (i.e. the latest on the main branch) at https://github.com/coreos/fedora-coreos-cincinnati .

Once you have identified the target commit, these are the steps to build a new container image:

  • update the fcos_cincinnati_build_git_sha playbook variable in roles/openshift-apps/coreos-cincinnati/vars/staging.yml

  • update the fcos_cincinnati_build_git_sha playbook variable in roles/openshift-apps/coreos-cincinnati/vars/production.yml

  • commit and push the update to the fedora-infra/ansible repository

  • SSH to batcave01.iad2.fedoraproject.org

  • run sudo rbac-playbook openshift-apps/coreos-cincinnati.yml using your FAS password and your second-factor OTP

  • schedule a new build by running sudo rbac-playbook -t build openshift-apps/coreos-cincinnati.yml

Deploying a new version

Once the target commit has been built into a container image, these are the steps to deploy the new image:

  • update the fcos_cincinnati_deploy_git_sha playbook variable in roles/openshift-apps/coreos-cincinnati/vars/staging.yml

  • update the fcos_cincinnati_deploy_git_sha playbook variable in roles/openshift-apps/coreos-cincinnati/vars/production.yml

  • commit and push the update to the fedora-infra/ansible repository

  • SSH to batcave01.iad2.fedoraproject.org

  • run sudo rbac-playbook openshift-apps/coreos-cincinnati.yml using your FAS password and your second-factor OTP

Things that could go wrong

Application build is stuck

Issues in the underlying OpenShift cluster may result in builds being permanently stuck.

If a build does not complete within a reasonable amount of time (i.e. 15 minutes):

Want to help? Learn how to contribute to Fedora Docs ›