Mass Branching
Description
At each alpha freeze we branch the pending release away from devel/
which allows rawhide (currently F42) to move on while the pending release goes into
bugfix and polish mode.
You will find below the list of steps to follow to branch a new Fedora release.
Mass resigning
When we branch off of rawhide, the branched release packages are already signed by the F{release} key, but we need to resign everything in rawhide for the new F+1 key. ie, When we branch f39 off rawhide, all it’s packages are already signed by the f39 key, but we need to resign everything with the f40 key for rawhide.
-
Add a new config for the new key to robosignatory. Something like:
[[consumer_config.koji_instances.primary.tags]]
from = "f39"
to = "f39"
key = "{{ (env == 'production')|ternary('fedora-40', 'testkey') }}"
keyid = "{{ (env == 'production')|ternary('a15B79cc', 'd300e724') }}"
{% if env == "production" %}
file_signing_key = "fedora-40-ima"
{% endif %}
This allows robosignatory to sign packages in the f39 tag with the f40 key. * git clone https://pagure.io/releng * confirm the new key fingerprint is in scripts/sigulsign_unsigned.py * run sigulsign_unsigned.py to gather list of packages to sign:
./sigulsign_unsigned.py --just-list --tag f39 fedora-40 | grep src | sed -e 's|.src||' > unsigned-packages
You should get a list of all the source packages by name. * copy unsigned-packages list to autosign01 * on autosign01 run in a tmux session:
sudo -su robosignatory
passphrase=$(systemd-ask-password "Please enter passphrase for 'autosign' key: ")
(enter the autosign passphrase)
keyctl add user "sigul:autosign" "${passphrase}" @s
for i in `cat unsigned-packages`
do
echo $i;
robosignatory sign-tag primary $i f39;
sleep 1;
done
This will iterate over all packages and sign them with the new f40 key. Once complete, re-run the ./sigulsign_unsigned.py command to confirm all are signed. On branching day, remove the robosignatory config for this resigning.
Send announcement
One day before the mass branching, we send out announcemt because during mass branching, new koji builds for rawhide are disabled.
Create Bugzilla Component
To ensure accurate tracking and management of issues for each Fedora release, a new Bugzilla component should be created for the branched version, as this step is currently not covered in the documentation. A user with access to the 'fedora-pm' Bugzilla group needs to perform the following actions:
-
Log in to Bugzilla.
-
Navigate to the "Administration" section.
-
Under "Components," locate "Fedora."
-
Select "Edit Versions" to view the existing components.
-
Add the new branched version to the list.
This process allows for proper issue categorization and ensures all bugs related to the new Fedora release are properly tracked.
Disable rawhide builds in koji
Previously, disabling all builds in Koji involved configuring an outage as demonstrated in this pull request. However, starting with the Fedora 41 release, the approach was refined to block external submissions in Koji by setting a custom IP restriction.
The recommended method to achieve this is by adding firewall rules to both koji01 and koji02 servers, effectively blocking connections from proxy01 and proxy10. This can be achieved with the following iptables commands:
iptables -I INPUT -m tcp -p tcp --dport 80 -s proxy01.iad2.fedoraproject.org -j REJECT
iptables -I INPUT -m tcp -p tcp --dport 80 -s proxy10.iad2.fedoraproject.org -j REJECT
These commands reject incoming traffic on port 80 from the specified proxies, preventing external submissions. Internal connections routed via proxy101 and proxy110 will continue to function as expected.
To reverse the firewall changes and allow external submissions again, use:
iptables -D INPUT -m tcp -p tcp --dport 80 -s proxy01.iad2.fedoraproject.org -j REJECT
iptables -D INPUT -m tcp -p tcp --dport 80 -s proxy10.iad2.fedoraproject.org -j REJECT
This change should be implemented on both koji01 and koji02 machine.
Repos to branch
All the following listed repos needs updating, including adding a new branch for branched release and updating rawhide branch with new release values.
-
https://pagure.io/workstation-ostree-config/
-
Follow the procedure detailed in the repo README under the
Branching instructions for new Fedora releases
section.
-
PDC
The "product-release" needs to be created in PDC.
In the scripts/pdc/
directory, run:
$ python create-product-release.py fedora $TOKEN Fedora $NEW_VERSION
On pdc-backend01.stg
(for testing) or pdc-backend01
(for production)
clone (or update an existing one) the releng repo:
$ git clone https://pagure.io/releng.git
In the scripts/pdc/
directory, run (see the --help
of the script for
information on how to run it):
$ python create-new-release-branches.py ... --createfile
The --createfile argument is necessary, that file is needed for the next step.
|
Due to memory leak issue in pdc, we need to set the config in
|
dist-git
Now that pdc has the new release and each package has been branched in pdc we need to update dist-git in two steps:
-
Create the new branch in git
-
Update the gitolite.conf to allow user to push to this new branch
For both of these actions you will need the file generated by pdc above.
Ansible
Apps in ansible need to be updated to be aware of a new branch.
Bodhi
Bodhi needs to be updated to add new release. This needs to be done in
bodhi2 role in
infra ansible repo. This change includes, updating koji-sync-listener.py
,
new-updates-sync
, pungi configs for rpm updates, bodhi templates.
-
roles/bodhi2/backend/files/new-updates-sync
-
roles/bodhi2/backend/tasks/main.yml
-
roles/bodhi2/backend/templates/pungi.rpm.conf.j2
-
roles/bodhi2/backend/templates/koji_sync_listener.toml
Please check these files from the commit for your reference.
Releng updates
Update sync script
In releng repository update script.
And set current_rawhide variable.
Hopefully someday soon this will be in cloud-image-uploader! |
Toddlers
Add new SLA to the toddlers App
Use this PR for reference and add new version to the config.
Enable Branched Compose
We need to enable the branched compose. This is done in releng role of infra ansbile repo.
Please check the file roles/releng/files/branched
from the commit for your reference.
Fedora Branched
-
Set FedoraBranched variable to True in infra ansible repo.
-
Set FedoraBranchedBodhi variable to preenable in infra ansible repo.
Please check the file FedoraBranched.yaml
and FedoraBranchedBodhi.yaml
from the commit for your reference.
Koji hub
Update the koji hub config to allow side tags for new koji rawhide (currently f42) tag.
Please check the file roles/koji_hub/templates/hub.conf.j2
from the commit for your reference.
Robosignatory
Robosignatory has two parts, which can be found in robosignatory role in infra ansible repo.:
-
Disable branched signing, so that we can freeze branched until we get a compose.
-
Adding new release.
Please check the file roles/robosignatory/templates/robosignatory.toml.j2
from the commit for your reference.
Push the changes
When done editing the files, commit, push and apply them via the corresponding ansible playbook:
$ sudo rbac-playbook groups/koji-hub.yml
$ sudo rbac-playbook groups/releng-compose.yml
$ sudo rbac-playbook groups/bodhi-backend.yml
$ sudo rbac-playbook openshift-apps/greenwave.yml
$ sudo -i ansible-playbook /srv/web/infra/ansible/playbooks/$ groups/proxies.yml -t pkgdb2
$ sudo rbac-playbook groups/mbs.yml -t mbs
Ask someone in fedora infra to run the robosignatory playbook.
Koji
The koji build system needs to have some tag/target work done to handle builds from the new branch and to update where builds from rawhide go.
Run make-koji-release-tags script in pagure releng repo
Fedora Release
The fedora-release
package needs to be updated in Rawhide and
Branched.
Changes to fedora-release.spec
in the rawhide branch:
(can also check this commit for reference)
-
Increment
%define dist_version
to 42:%define dist_version 42
-
Increment
Version:
and resetRelease:
:Version: 42 Release: 0.1%{?eln:.eln%{eln}}
-
Add a
%changelog
entry:%changelog * Day Mon DD YYYY Name
- 42-0.1 - Setup for rawhide being F42
Changes to fedora-release.spec
in the branched (currently 41) branch:
(can also check this commit for reference)
-
Adjust
release_name
and unsetis_rawhide
:%define release_name Forty One %define is_rawhide 0
-
Verify the correct number for
dist_version
andVersion:
:%define dist_version 41 Version: 41
-
Bump
Release:
:Release: 0.4%{?eln:.eln%{eln}}
-
Add a
%changelog
entry:%changelog * Day Mon DD YYYY Name
- 41-0.4 - Branching F41 from rawhide
Fedora Repos
The fedora-repos
package needs to be updated in Rawhide, Branched, and
also in all stable release branches (in order to receive new GPG keys
and updated symlinks).
Changes to the rawhide branch (mostly in fedora-repos.spec
):
(can also check this commit for reference)
-
Generate and add a Rawhide+1 which is 43 GPG key file, then add it to the spec file:
Source57: RPM-GPG-KEY-fedora-43-primary
-
Update the
archmap
file and define architectures for Rawhide+1:fedora-{rawhide+1}-primary: x86_64 armhfp aarch64 ppc64le s390x
-
Increment
%global rawhide_release
:%global rawhide_release 42
-
Bump
Version:
and resetRelease:
:Version: 42 Release: 0.1%{?eln:.eln%{eln}}
-
Add a
%changelog
entry:%changelog * Day Mon DD YYYY Name
- 42-0.1 - Setup for rawhide being F42
Changes to the branched branch (mostly in fedora-repos.spec
):
(can also check this commit for reference)
-
Copy the Rawhide+1 which is 43 GPG key file from the rawhide branch, then add it to the spec file:
Source57: RPM-GPG-KEY-fedora-43-primary
-
Copy the
archmap
file from the rawhide branch. -
Update
%global rawhide_release
:%global rawhide_release 42
-
Enable
updates_testing_enabled
:%global updates_testing_enabled 1
-
Bump
Release
:Release: 0.3%{?eln:.eln%{eln}} +
-
Add a
%changelog
entry:%changelog *Day Mon DD YYYY Name
- 41-0.3 + - Update Rawhide definition, enable updates-testing for Branched +
Build |
Consider using sidetags for |
Changes to the stable branches (mostly in fedora-repos.spec
):
-
Copy the Rawhide+1 GPG key which is 43 file from the rawhide branch, then add it to the spec file:
Source57: RPM-GPG-KEY-fedora-43-primary
-
Copy the
archmap
file from the rawhide branch. -
Update
%global rawhide_release
:%global rawhide_release 42
-
Bump
Release:
:Release: 0.3%{?eln:.eln%{eln}}
-
Add a
%changelog
entry:%changelog *Day Mon DD YYYY Name
- 40-0.3 - Update Rawhide definition
Bodhi
Linking Empty Repos
We need to link empty repos so that new-updates-sync wont complain about missing repos. The following commands should be run on bodhi-backend01.phx2.fedoraproject.org
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f41-updates
$ sudo ln -s /mnt/koji/compose/updates/empty-repo/ /mnt/koji/compose/updates/f41-updates-testing
Creating Empty Repos
To create empty repos on the master mirror, run create_emtpy_repos.sh from pagure releng repo. This should be run on bodhi-backend01.phx2.fedoraproject.org
$ sudo -u ftpsync sh scripts/branching/create_empty_repos.sh 41
Update the link in /mnt/koji/repos/rawhide/latest as per https://pagure.io/releng/issue/12255. |
Please verify the repo permissions that are created under /pub/fedora/linux/development/<fedora_release_number> and /pub/fedora-secondary/development/<fedora_release_number>. They should be owned by ftpsync:ftpsync Check directory permissions (should be "0755") to ensure new composes synchronize correctly. |
Creating rawhide release
To create a rawhide release in bodhi, you need to run:
$ bodhi releases create \
--name "F42" --long-name "Fedora 42" \
--id-prefix FEDORA --version 42 --branch f42 \
--dist-tag f42 \
--stable-tag f42 \
--testing-tag f42-updates-testing \
--candidate-tag f42-updates-candidate \
--pending-stable-tag f42-updates-pending \
--pending-testing-tag f42-updates-testing-pending \
--pending-signing-tag f42-signing-pending \
--state pending \
--override-tag f42-override \
-create-automatic-updates \
--not-composed-by-bodhi
To create a container release for rawhide in bodhi, you need to run:
$ bodhi releases create \
--name "F42C" --long-name "Fedora 42 Containers" \
--id-prefix FEDORA-CONTAINER --version 42 --branch f42 \
--dist-tag f42-container \
--stable-tag f42-container-updates \
--testing-tag f42-container-updates-testing \
--candidate-tag f42-container-updates-candidate \
--pending-stable-tag f42-container-updates-pending \
--pending-testing-tag f42-container-updates-testing-pending \
--state pending \
--override-tag f42-container-override
To create a flatpak release for branched in bodhi, you need to run:
$ bodhi releases create \
--name "F41F" --long-name "Fedora 41 Flatpaks" \
--id-prefix FEDORA-FLATPAK --version 41 --branch f41 \
--dist-tag f41-flatpak \
--stable-tag f41-flatpak-updates \
--testing-tag f41-flatpak-updates-testing \
--candidate-tag f41-flatpak-updates-candidate \
--pending-stable-tag f41-flatpak-updates-pending \
--pending-testing-tag f41-flatpak-updates-testing-pending \
--state pending \
--override-tag f41-flatpak-override
You need to run the bodhi openshift
playbook, so that UI will know
about the new release. Then, you need to restart
fm-consumer@config.service and bodhi-celery.service services on
bodhi-backend01.phx2.fedoraproject.org:
$ sudo rbac-playbook openshift-apps/bodhi.yml
$ sudo systemctl restart fm-consumer@config.service bodhi-celery.service
Build fedora-release, fedora-repos package for rawhide after enabling the rawhide gating |
Update rawhide koji repo
We need to point the rawhide buildroot repo to the newly created rawhide buildroot. This way kojira doesn’t make a newrepo for rawhide target as often as fxx-build (new rawhide buildroot).
Run the following commands from any of the compose boxes:
$ cd /mnt/koji/repos/rawhide
$ rm -f latest
$ ln -s ../f42-build/latest ./latest
Update block_retired.py script
block_retired.py script in releng repo should be updated with rawhide release and also branched release should be added to the script.
Please look at this block_retired.py commit as an example.
Updating MirrorManager
We need to update the mirrormanager so that it will point rawhide to the new rawhide release.
Please follow the instructions in the fedora infra ticket to update the database of mirrormanager.
Enable autosigning on branched release
Once the branched compose is composed, we need to re-enable robosignatory on branched release
ELN related work
Add the new rawhide key to eln pungi config. For example, look at this pungi eln config commit
Change the trigger notification for DistroBuildSync to the new Rawhide version. For example, look at this commit.
Branch new rawhide in Koschei
Branch new fedora rawhide in koschei.
Fedora Container Base Image
In order to enable builds for Container Base Images via the
Fedora
Layered Image Build System we will need to import a new image for
Rawhide as well as for the new fedora:rawhide
and fedora:$42
tags.
Check for the latest successful Rawhide Base Image composed image here.
On compose-x86-01.phx2
run:
# Update this to be the correct URL for your image
$ BASEIMAGE_URL="https://kojipkgs.fedoraproject.org//packages/Fedora-Docker-Base/Rawhide/20170310.n.0/images/Fedora-Docker-Base-Rawhide-20170310.n.0.x86_64.tar.xz"
# Update this to whatever version number Rawhide now points to
$ RAWHIDE="27"
# Load the latest, find it's image name
$ sudo docker load < <(curl -s "${BASEIMAGE_URL}")
$ sudo docker images | grep base-rawhide
fedora-docker-base-rawhide-20170310.n.0.x86_64 latest ffd832a990ca 5 hours ago 201.8 MB
# Tag everything
$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 candidate-registry.fedoraproject.org/fedora:rawhide
$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 candidate-registry.fedoraproject.org/fedora:$42
$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 registry.fedoraproject.org/fedora:rawhide
$ sudo docker tag fedora-docker-base-rawhide-20170310.n.0.x86_64 registry.fedoraproject.org/fedora:$42
# Push the images
$ sudo docker push candidate-registry.fedoraproject.org/fedora:rawhide
$ sudo docker push candidate-registry.fedoraproject.org/fedora:$42
$ sudo docker push registry.fedoraproject.org/fedora:rawhide
$ sudo docker push registry.fedoraproject.org/fedora:$42
# Clean up after ourselves
$ sudo docker rmi fedora-docker-base-rawhide-20170310.n.0.x86_64
Untagged: fedora-docker-base-rawhide-20170310.n.0.x86_64:latest
$ for i in $(sudo docker images -q -f 'dangling=true'); do sudo docker rmi $i; done
Temporarily disable the rawhide cron job during branching PRs to ensure a branched compose is created. Re-enable rawhide after this. |
In fedora-repos package build for new branched version enable the update-testing repository immediately upon branching. |
Want to help? Learn how to contribute to Fedora Docs ›