PDC SOP

The Product Definition Center (PDC) is a webapp and API designed for storing and querying product metadata. We automatically populate our instance with data from our existing releng tools/processes. It doesn’t do much on its own, but the goal is to enable us to develop more sane tooling down the road for future releases.

The webapp is a django app running on pdc-web0\{1,2}. Unlike most of our other apps, it does not use OpenID for authentication, but it instead uses SAML2. It uses mod_auth_mellon to achieve this (in cooperation with ipsilon). The webapp allows new data to be POST’d to it by admin users.

The backend is a fedmsg-hub process running on pdc-backend01. It listens for new composes over fedmsg and then POSTs data about those composes to PDC. It also listens for changes to the fedora atomic host git repo in pagure and updates "component groups" in PDC to reflect what rpm components constitute fedora atomic host.

For long-winded history and explanation, see the original Change document: https://fedoraproject.org/wiki/Changes/ProductDefinitionCenter

There is an upgrade playbook in playbooks/manual/upgrade/pdc.yml which will upgrade both the frontend and the backend if new packages are available. Database schema upgrades should be handled automatically with a run of that playbook.

Logs for the frontend are in /var/log/httpd/error_log on pdc-web0\{1,2}.

Logs for the backend can be accessed with journalctl -u fedmsg-hub -f on pdc-backend01.

The frontend runs under apache. So either apachectl graceful or systemctl restart httpd should do it.

The backend runs as a fedmsg-hub, so systemctl restart fedmsg-hub should restart it.

The pdc-updater package (installed on pdc-backend01) provides three scripts:

A possible failure scenario is that we will lose a fedmsg message and the backend will not update the frontend with info about that compose. To detect this, we provide the pdc-updater-audit command (which gets run once daily by cron with emails sent to the releng-cron list). It compare all of the entries in PDC with all of the entries in kojipkgs and then raises an alert if there is a discrepancy.

Another possible failure scenario is that the fedmsg message is published and received correctly, but there is some processing error while handling it. The event occurred, but the import to the PDC db failed. The pdc-updater-audit script should detect this discrepancy, and then an admin will need to manually repair the problem and retry the event with the pdc-updater-retry command.

If doomsday occurs and the whole thing is totally hosed, you can delete the db and re-ingest all information available from releng with the pdc-updater-initialize tool. (Creating the initial schema needs to happen on pdc-web01 with the standard django settings.py commands.)

In general, you shouldn’t have to do these things. pdc-updater will automatically create new releases and update information, but if you ever need to manipulate PDC data, you can do it with the pdc-client tool. A copy is installed on pdc-backend01 and there are some credentials there you’ll need, so ssh there first.

Make sure that you are root so that you can read /etc/pdc.d/fedora.json.

Try listing all of the releases:

Deactivating an EOL release:

Listing all composes:

We’re not sure yet how to flag a compose as the Gold compose, but when we do, the answer should appear here: https://github.com/product-definition-center/product-definition-center/issues/428

Some small group of release engineers need to be superuser to set eol dates and add/remove components. You can grant them permissions to do this via some direct database calls. First find out their email address listed in fas, then login to db01.iad2.fedoraproject.org:

The user will now have privs with their normal tokens.

As stated previously, the authentication uses SAML2 with mod_auth_mellon (as the Service Provider on PDC’s side) and Ipsilon (as the Identity Provider). This form of authentication relies on SSL certificates and XML metadata.

PDC’s certificates live in the ansible-private repository, in files/saml2/pdc{,.stg}.fedoraproject.org/certificate.{pem,key}. They are generated from the PKI in files/saml2/ {staging,production}/. The certificates can be self-signed as long as they are properly embedded in the metadata XML file and this file is distributed identically to PDC and to Ipsilon.

To renew the certificate, generate a new one with the provided script in the ansible-private repo:

And for production:

And commit the changes:

Then run the PDC and the Ipsilon playbooks. The PDC playbook will push the new certificates and re-generate the metadata.xml file in /etc/httpd/saml2/. The Ipsilon playbook will retrieve this metadata.xml file from the PDC server and insert it into the /etc/ipsilon/root/configuration.conf file.