Onboarding of a CI System

Testing in Fedora

It’s relatively easy to start testing Fedora artifacts (Koji builds, Bodhi updates, etc.) and to contribute test results back so that they can be later used for gating — i.e. deciding whether the tested artifact should be promoted or not.

Here we describe necessary steps to add a new CI system.

What is a (suitable) CI System?

In order to provide a good workflow and user experience, here are some aspects of CI systems that have proven to be successful:

  • Tests are reliable (low false negative and false positive rates) and cover important stories

  • Tests can be contributed to (open source model) and are ideally similar to other tests, e.g. by using established frameworks / languages

  • Results / notifications are easy to understand and help identify errors quickly

  • Tests are reproducible, if necessary artifacts from test runs are stored for users to consume, such as virtual machine images

In short, test results should be directly actionable! As a developer, I need to quickly decide whether the test is broken or the code, and then fix the issue.

Testing and Gating Builds

Gating Workflow

On the highest level, the gating workflow consists of the following steps:

  • Submit a build of a package (Koji)

  • Trigger CI systems to run tests (Fedora CI, Your CI, etc.)

  • Collect results from the CI systems (ResultsDB)

  • Make a decision (Greenwave)

  • If the decision is "pass", then let the build pass the gate to the main repository (Bodhi)

Gating Messages

Gating process is implemented as a set of services which interact with each other via message bus (FedoraMessaging).

Therefore to add a piece (for example a CI system) to the process, you essentially need to start receiving and sending messages via FedoraMessaging.

Find out more about gating in Fedora Rawhide — especially about the single and multi-build updates.

How to add a CI System

CI systems in Fedora are autonomous entities that typically need to handle following things:

  • triggering when certain events occur

  • actual testing

  • publishing test results

Triggering and Testing

Services in Fedora publish messages when various events occur and thus CI systems can trigger testing when for example new Bodhi update is created.

When Bodhi update is created, a new "koji-build-group.build.complete" message is published on the org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete topic.

The schema of these "koji-build-group.build.complete" messages is defined in the CI Messages specification.

If your CI system is Jenkins, then you can use the jms-messaging plugin to trigger your tests when defined events occur. Getting the trigger syntax right in Jenkins pipelines can be tricky, but you can take a look at the existing example here.

It’s of course possible to trigger testing on other types of events, not just the Bodhi updates. You can find more Fedora message topics in the older fedmsg2 documentation. Beware though, the list is incomplete.

Sharing Test Results

CI systems should publish standardized CI messages so the progress of the testing can be observed and the results can be acted upon by other services in the Fedora infrastructure. Although publishing test results is not mandatory, CI systems which don’t publish the standardized CI messages cannot be part of the gating process.

There are four types of messages that CI systems should be sending:

  • test.queued - when there is an artifact (a Bodhi update for example) in a queue waiting to be tested

  • test.running - when testing is in progress

  • test.complete - when testing is finished

  • test.error - when testing couldn’t start or couldn’t finish due to an outside circumstances (typically an infrastructure error)

These messages have well-defined schemas. The schemas are part of the CI Messages specification.

For convenience, here are links to schemas for simple koji-build artifacts and koji-build-group artifacts:

Test Identifiers

When you send a "koji-build.test.complete" message to the message bus, the test result gets stored in the ResultsDB. Later on you can refer to the stored test result in a Greenwave policy: "if test XYZ passed, let the build through the gate".

Therefore you need unique identifiers for test results.

Test identifiers are built from three parts: test namespace, test category and test type.

In the "koji-build.test.complete" schema, these variables are represented by namespace, category and type fields respectively.

Namespace is always an ID of your CI system with the name of the artifact type, so for example: fedora-ci.koji-build, or osci.pull-request.

Category you can choose from a predefined list:

  • static-analysis

  • functional

  • integration

  • validation

Type is an arbitrary string, which you can define based on the specifics of your CI system.

It is your responsibility as an owner of the CI system to maintain consistent naming of tests under your namespace.

The final test identifier may look like this: fedora-ci.koji-build.tier0.functional

Useful Links

Want to help? Learn how to contribute to Fedora Docs ›