Koji/Sigul

Sigul Overview

Sigul is a package signing server, which aids in signing RPM's either multiple, or single files. All keys reside solely on the sigul server, no user has access to any of the private or public keys. All communication with the server is done through the sigul bridge, it acts as the gateway between the client, and server. This isolates the server, preventing any unwanted access from outside sources other than the bridge. The sigul client once configured allows users to sign rpm's with keys created on the sigul server, by sending commands first to the bridge, which then relays to the server.

Installation

This part is very very easy

yum install seagul

Sigul Bridge Setup

What the bridge does: The bridge acts as the central gateway for sigul. The bridge is the only component which speaks to the server, if you are issuing any server bound commands from the client, your actually sending them to the bridge which in turn fires them off to the server, recieves the reply, and sends that back to the client. The bridge also functions as the gateway for Koji, when signing packages from a koji instance, the bridge speaks to the kojihub with authentication by way of a Proxy user, such as Kojiweb. We will be getting into the koji side of things a bit more later in this doc.

To begin setup, we have generate the certs which will be used for all sigul systems to authenticate between eachother. The bridge will be used as the CA for internal sigul communications.

Be sure to remember all your NSS Password

Operations with sigul account

All following commands must be done with sigul account

su - sigul -s /bin/bash

Create an NSS database on the bridge, to hold the certificate information

bridge_dir=/var/lib/sigul

Generate a new NSS database for the bridge at the location of the bridge_dir variable

certutil -d $bridge_dir -N

Generate the CA (Certificate Authority) certificate, to be used accross all sigul components

CN must be the full qualified domain name of the server where bridge is hosted.

certutil -d $bridge_dir -S -n sigul-ca -s 'CN=Sigul CA' -t CT,, -x -v 120

Create a certificate for the bridge

certutil -d $bridge_dir -S -n sigul-bridge-cert -s 'CN=didier.b2pweb.com' -c sigul-ca -t u,, -v 120

Operations with root account

Now it is time to configure the bridge, edit with the root account the config at <path>/etc/sigul/bridge.conf</path>

• You can leave most things at their default such as ports, or fas-account settings, if using FAS authentication.
• Under the [NSS] section you can set nss-password: yournsspass
• The default configuration assumes you set up a separate "sigul" user and group. Remove the [daemon] section if you want the bridge to run as the invoking user.
• If you use a separate user and group issue: chown sigul:sigul $bridge_dir/*.db

After editing the config and setting up the certs, it is time for a test drive issue the following

sigul_bridge -v -v

This will start the bridge in DEBUG mode, and all information will be logged in /var/log/sigul_bridge

Check the log file after starting sigul, if there are no errors you are good to go.

You should see the first log message in /var/log/sigul_bridge.log:

2011-11-24 16:41:42,214 DEBUG: Waiting for the client to connect

Stop the sigul_bridge CRTL-C

Now we can start the service

service sigul_bridge start

or for fedora > 16

systemctl start sigul_bridge.service

Sigul Server Setup

What the server does: The server is completley cutoff from the rest of the world, It should be firewalled off except for incoming ports from the bridge, and should only be able to speak to the bridge, for max security, ensure it has no external access from other machines or even the web. It hold's all of the key files used for signing the RPMS, so no other users will have access to the key files, the server is the only system that knows the keys.

To begin setup, we have to follow a similar process to the bridge with NSS, except that we will import the CA cert generated on the bridge, not generate a new one.

Operations with sigul account

Create the NSS database on the server, to hold the certificate information with the user sigul account

First we set the location where sigul resides on the system

server_dir=/var/lib/sigul
certutil -d $server_dir -N

This will generate a new NSS database for the server at the location of the server_dir variable

Import the CA (Certificate Authority) certificate, generated earlier on the bridge

ON THE BRIDGE as user sigul

pk12util -d $bridge_dir -o myca.p12 -n sigul-ca

This file should now be copied over to the server and deleted from the bridge afterwards

ON THE SERVER as user sigul

The sigul CA certs should now be imported

pk12util -d $server_dir -i myca.p12
rm myca.p12
certutil -d $server_dir -M -n sigul-ca -t CT,,

CN must be the full qualified domain name of the server where sigul server is hosted.

certutil -d $server_dir -S -n sigul-server-cert -s 'CN=didier.b2pweb.com' -c sigul-ca -t u,, -v 120

Operations with root account

Now it is time to configure the server, edit the config at /etc/sigul/server.conf with root account
Note the default ports. Edit at least bridge-hostname and the [nss] section. The default configuration assumes you set up a separate "sigul" user and group; remove the [daemon] section if you want the server to run as the invoking user.

Now to create the database for the server which will hold all user and key entries issue the following * AS ROOT

sigul_server_create_db

Next Add the initial administrator

sigul_server_add_admin

After all is configured, it's time for a test drive

Start the server in DEBUG mode, and all information will be logged in <path>/var/log/sigul_server</path>

sigul_server -v -v

Check the log file after starting sigul server, if there are no errors you are good to go.

You should see the first log message in /var/log/sigul_bridge.log:

2011-11-24 16:36:42,154 DEBUG: Waiting for a request

Now we can start the service

service sigul_server start

or for fedora > 16

systemctl start sigul_server.service

Sigul Client Setup

What the client does: The client is simply that, a client, it has certs necessary for it to be authenticated with the sigul bridge to issue commands as the sigul admin, to the server. All client commands are sent to bridge which in turn talks to either koji or the server depending on what the clients request is.

To begin setup, we have to follow a similar process to the bridge with NSS, except that we will import the CA cert generated on the bridge, not generate a new one.

1) Create the NSS database on the client, to hold the certificate information issue the following

   client_dir=~/.sigul   <-- This variable should be set to the location of sigul which is a folder under the user directory
   certutil -d $client_dir -N  <-- This will generate a new NSS database for the server at the location of the client_dir variable
        |Be sure to remember your NSS Password|

2) Now import the CA (Certificate Authority) certificate, generated earlier on the bridge

   * ON THE BRIDGE as user sigul
   - Issue: pk12util -d $bridge_dir -o myca.p12 -n my-ca <-- This file should now be copied over to the client and deleted from the bridge afterwards

   * ON THE CLIENT as your users
   - Issue: pk12util -d $client_dir -i myca.p12
        rm myca.p12
        certutil -d $client_dir -M -n my-ca -t CT,, <-- be sure to change my-ca to your CA name

3) Next we have to generate the authentication certificate for the clientL

   certutil -d $client_dir -S -n sigul-client-cert -s 'CN=YOURUSERNAME' -c my-ca -t u,, -v 120  <-- be sure to replace YOURUSERNAME with the user you are using on the client system, OR if using FAS                                                            authentication set the CN=YOUR FAS NAME.

4) Now it is time to configure the client, edit the config at /etc/sigul/client.conf * AS ROOT

   - You can leave most things set as default except for the following:
       | bridge-hostname and server-hostname be sure to change those to match the hostnames of each of those machines.
       | under [ NSS] user-name set this to the value of the admin user your setup on the server previously
       | If you wish to avoid entering an NSS password upon issuing each command, issue vi ~/.sigul/client.conf and add the following lines:
           | [nss]
             nss-password: Your NSS PASS

5) After configuring your client, issue a test client command in DEBUG mode as follows:

    sigul -v -v list-users
   * This should return a list of users on the server, at this point it should only really display the one admin user created before.
   * For more commands issue sigul --help-commands for a full list

6) Create an initial key once you are able to issue commands to sigul, issue the following:

    sigul new-key -h <-- This will output the options that can be used with the key creation, use the ones you want, and generate the key.
   * Please note when generating the key, it requires alot of Entropy on the server, so issue some commands to keep server busy and help it generate faster, usually a simple find / will generate enough for it to take about 2 minutes to generate the key.

Sigul with koji Setup

How it works with koji: Sigul can be used with koji to sign multiple packages in a koji instance. Provided you have your client user already configured with Koji, it's a simple matter of configuring the proxy user on sigul_bridge. When a client issues a sign command for a koji instance, the bridge is what actually queries koji on behalf of the client user, and grabs the rpm's to be signed from sigul by way of the kojiweb user. The only thing you must ensure is that your koji client user as admin privileges on the koji server, and the bridge takes care if the rest.

1) As ROOT on the sigul bridge, edit /etc/sigul/bridge.conf edit the koji section as follows:

    [koji]
     koji-config: /path/to/koji/config/file  <-- The config file should be that of koji web

2) The koji configuration file and certs can reside under any directory that sigul has atleast read privileges on. The kojiweb certificates that allow kojiweb to authenticate with koji must be copied to this directory, along with the config file which points to the koji instance, as well as the kojiweb certs needed for it to authenticate.

3) After doing the above restart the bridge, and you should now be able to pull packages from koji and sign them.

4) To test issue the following on the client, to download and RPM from koji - sign it - and store it locally - Just as a test for koji connectivity and authentication:

   sigul sign-rpm -o signed.rpm key_name unsigned.rpm <-- key_name should be the name of the sigul key you setup previously.
   - If the above is successful, you will have an rpm named signed.rpm in the directory you are working in.

Sigul Client Config Script

The following is an optional script, which can be used to aide in the quick setup of sigul clients, when not using FAS Authentication:

 * Note that the user must first have an account created on the sigul server, this script is solely to setup the client side certificates

#!/bin/bash
#Variables### And initial setup#######
mkdir ~/.sigul
client_dir=~/.sigul
user=$(whoami)  
####################
echo
############################Begin Certificate imports
echo "======================="
echo "Setting up NSS Database"
echo "======================="
certutil -d $client_dir -N
echo
echo "==================="
echo "Downloading CA Cert"
echo "==================="
wget http://someurl.com/sigul/sigulca.p12 <-- Substitute with a path or url of your exported .p12
echo
echo "=================="
echo "Importing CA certs"
echo "=================="
pk12util -d $client_dir -i sigulca.p12
certutil -d $client_dir -M -n sigul-ca -t CT,,
echo
echo "======================"
echo "Generating Client cert"
echo "======================"
certutil -d $client_dir -S -n sigul-client-cert -s "CN=$user" -c sigul-ca -t u,, -v 120
echo
echo "======================"
#########End certificate imports########
########################################
#########NSS password Saver#############
read -p "Would you like to save your nss pass to ~/.sigul/client.conf [y/n]: " nsspasssel
#########User Input conditional#########
if [ $nsspasssel == "y" -o $nsspasssel == "Y" ]; then
echo "Enter your NSS password One more time: "
read -s nsspass
echo "[nss]" > ~/.sigul/client.conf
echo "nss-password: $nsspass" >> ~/.sigul/client.conf
echo
echo "==========="
echo "Cleaning up"
echo "==========="
rm sigulca.p12
else
echo
echo "==========="
echo "Cleaning up"
echo "==========="
rm sigulca.p12
fi
#########################################\

• If you plan to use FAS Authentication, run sigul_setup_client as the user you wish to setup.

Description

At the beginning of each release under development a new package signing key is created for it. This key is used to prove the authenticity of packages built by Fedora and distributed by Fedora. This key will be used to sign all packages for the public test and final releases.

Action

Sigul

Sigul is the signing server which holds our keys. In order to make use of a new key, the key will have to be created and access to the key will have to be granted. The new-key, grant-key-access, and change-passphrase commands are used.

$ sigul new-key --help
usage: client.py new-key [options] key

Add a key

options:
  -h, --help            show this help message and exit
  --key-admin=USER      Initial key administrator
  --name-real=NAME_REAL
                        Real name of key subject
  --name-comment=NAME_COMMENT
                        A comment about of key subject
  --name-email=NAME_EMAIL
                        E-mail of key subject
  --expire-date=YYYY-MM-DD
                        Key expiration date

$ sigul grant-key-access --help
usage: client.py grant-key-access key user

Grant key access to a user

options:
  -h, --help  show this help message and exit

$ sigul change-passphrase --help
usage: client.py change-passphrase key

Change key passphrase

options:
  -h, --help  show this help message and exit

For example if we wanted to create the Fedora 13 signing key, we would do the following:

1. Log into a system configured to run sigul client.
2. Create the key using a strong passphrase when prompted

   $ sigul new-key --key-admin jkeating --name-real Fedora \
           --name-comment 13 \
           --name-email fedora@fedoraproject.org fedora-13
   

3. Wait a while for entropy. This can take several minutes.
4. Grant key access to Fedora Account holders who will be signing packages and protect it with a temporary a passphrase. For example, "CHANGEME."

   $ sigul grant-key-access fedora-13 jwboyer
   

5. Provide the key name and temporary passphrase to signers. If they don't respond, revoke access until they are ready to change their passphrase. Signers can change their passphrase using the change-passphrase command:

   $ sigul change-passphrase fedora-13
   

<li>
When your sigul cert expires, you will need to run: 'certutil -d ~/.sigul -D -n sigul-client-cert' to remove the old cert, then 'sigul-client-setup' to add a new one.

fedora-release

The fedora-release package houses a copy of the public key information. This is used by rpm to verify the signature on files encountered. Currently the fedora-release package has a single key file named after the version of the key and the arch the key is for. To continue our example, the file would be named RPM-GPG-KEY-fedora-13-primary which is the primary arch key for Fedora 13. To create this file, use the get-public-key command from sigul:

$ sigul get-public-key fedora-13 > RPM-GPG-KEY-fedora-13-primary

Add this file to the repo, and remove the previous release's file.

$ cvs rm RPM-GPG-KEY-fedora-12-primary
$ cvs add RPM-GPG-KEY-fedora-13-primary

Then make a new fedora-release build for rawhide (FIXME: this should be its own SOP)

fedoraproject.org

fedoraproject.org/keys lists information about all of our keys. We need to let the webteam know we have created a new key so that they can add it to the list.

We do this by sending an email to webmaster@fedoraproject.org pointing to the viewvc http://cvs.fedoraproject.org/viewvc/fedora-release/RPM-GPG-KEY-fedora-13-primary?revision=1.1&root=fedora&view=co as well as including a URL to this page so that the process is not forgotten (see section below)

This url will have to be refreshed for the right release and CVS version

Web team SOP

# from git repo root
cd fedoraproject.org/
curl $KEYURL > /tmp/newkey
$EDITOR update-gpg-keys # Add key ID of recently EOL'd version to obsolete_keys
./update-gpg-key /tmp/newkey
gpg static/fedora.gpg # used to verify the new keyring
# it should look something like this:
# pub  4096R/57BBCCBA 2009-07-29 Fedora (12) <fedora@fedoraproject.org>
# pub  4096R/E8E40FDE 2010-01-19 Fedora (13) <fedora@fedoraproject.org>
# pub  4096R/97A1071F 2010-07-23 Fedora (14) <fedora@fedoraproject.org>
# pub  1024D/217521F6 2007-03-02 Fedora EPEL <epel@fedoraproject.org>
# sub  2048g/B6610DAF 2007-03-02 [expires: 2017-02-27]
# it must only have the two supported versions of fedora, rawhide and EPEL
# also verify that static/$NEWKEY.txt exists
$EDITOR data/content/{keys,verify}.html # see git diff 1840f96~ 1840f96

sigulsign_unsigned

sigulsign_unsigned is the script Release Engineers use to sign content in koji. This script has a hardcoded list of keys and aliases to the keys that needs to be updated when we create new keys.

Add the key details to the KEYS dictionary near the top of the sigulsign_unsigned.py script. It lives in Release Engineering's git repo at git://git.fedorahosted.org/git/releng in the scripts directory. You will need to know the key ID to insert the correct information:

$ gpg <key block from sigul get-public-key>

Public Keyservers

We upload the key to the public key servers when we create the keys. To do this, we need to get the ascii key block from sigul, determine the key ID, import they key into our local keyring, and then upload it to the key servers.

$ sigul get-public-key fedora-13 > fedora-13
$ gpg fedora-13 (The ID is the "E8E40FDE" part of 4096R/E8E40FDE)
$ gpg --import fedora-13
$ gpg --send-keys E8E40FDE

Mash

Mash is the tool that composes our nightly trees, and as such it needs to know about the new key. This currently is done by checking mash out from git, editing the rawhide.mash file and sending the patch to the mash upstream.

$ git clone git://git.fedorahosted.org/git/mash
$ cd mash
$ vim configs/rawhide.mash
<add key to front of keys = line>
$ git commit -m 'Add new key'
$ git send-email --to notting@redhat.com HEAD^

Coordinate with Bill Nottingham to get a new build of mash done with the change.

Koji

Koji has a garbage collection utility that will find builds that meet criteria to be removed to save space. Part of that criteria has to do with whether or not the build has been signed with a key. If the collection utility doesn't know about a key it will ignore the build. Thus as we create new keys we need to inform the utility of these keys or else builds can pile up. The configuration for the garbage collection lives within puppet.

On the puppet server in a clone edit the configs/build/koji-gc.conf file:

diff --git a/configs/build/koji-gc.conf b/configs/build/koji-gc.conf
index 8b14704..042ec35 100644
--- a/configs/build/koji-gc.conf
+++ b/configs/build/koji-gc.conf
@@ -11,6 +11,7 @@ key_aliases =
     4EBFC273    fedora-10
     D22E77F2    fedora-11
     57BBCCBA    fedora-12
+    217521F6    fedora-epel
 
 unprotected_keys =
     fedora-test
@@ -21,6 +22,7 @@ unprotected_keys =
     fedora-12
     fedora-extras
     redhat-beta
+    fedora-epel
 
 server = https://koji.fedoraproject.org/kojihub
 weburl = http://koji.fedoraproject.org/koji
@@ -38,6 +40,7 @@ policy =
     sig fedora-10 && age < 12 weeks :: keep
     sig fedora-11 && age < 12 weeks :: keep
     sig fedora-12 && age < 12 weeks :: keep
+    sig fedora-epel && age < 12 weeks :: keep
 
     #stuff to chuck semi-rapidly
     tag *-testing *-candidate *-override && order >= 2 :: untag

In this case the fedora-epel key was added to the list of key aliases, then referenced in the list of unprotected_keys, and finally a policy was created for how long to keep builds signed with this key.

Once you've made your change commit and push. The buildsystem will pick up this change the next time puppet refreshes.

Verification

We can verify that the key was created in sigul, the correct users have access to the key, the key was added to the fedora-release package, that the website was updated with the right key, that sigulsign_unsigned was properly updated, and that the key was successfully updated to the public key servers.

sigul

Use the list-keys command to verify that the key was indeed added to sigul:

$ sigul list-keys
Administrator's password: 
fedora-10
fedora-10-testing
fedora-11
fedora-12
fedora-13

Our new key should be on the list. This command expects your administrative password.

Use the list-key-users command to verify all the signers have access:

$ sigul list-key-users fedora-13
Key passphrase: 
jkeating
jwboyer

This command expects your key passphrase for the key in question.

fedora-release

To verify that the key was added to this package correctly, download the latest build from koji and run rpm2cpio on it, then run gpg on the key file:

$ koji download-build --arch noarch --latest dist-f13 fedora-release
fedora-release.noarch                                   |  39 kB     00:00 ... 

$ rpm2cpio fedora-release-13-0.3.noarch.rpm |cpio -ivd
./etc/fedora-release
./etc/issue
./etc/issue.net
./etc/pki/rpm-gpg
./etc/pki/rpm-gpg/RPM-GPG-KEY-fedora
./etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-primary
./etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-i386
./etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-ppc
./etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-ppc64
./etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-x86_64
./etc/redhat-release
./etc/rpm/macros.dist
./etc/system-release
./etc/system-release-cpe
./etc/yum.repos.d
./etc/yum.repos.d/fedora-rawhide.repo
./etc/yum.repos.d/fedora-updates-testing.repo
./etc/yum.repos.d/fedora-updates.repo
./etc/yum.repos.d/fedora.repo
./usr/share/doc/fedora-release-13
./usr/share/doc/fedora-release-13/GPL
57 blocks

$ gpg etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-13-primary 
pub  4096R/E8E40FDE 2010-01-19 Fedora (13) <fedora@fedoraproject.org>

You may wish to do this in a tempoary directory to make cleaning it up easy.

fedoraproject.org

One can simply browse to http://fedoraproject.org/keys to verify that the key has been uploaded.

sigulsign_unsigned

The best way to test whether or not the key has been added correctly is to sign a package using the key, like our newly built fedora-release package.

$ ./sigulsign_unsigned.py fedora-13 fedora-release-13-0.3
Passphrase for fedora-13: 

The command should exit cleanly.

Public key servers

One can use the search-keys command from gpg to locate the key on the public server:

$ gpg --search-keys "Fedora (13)"
gpg: searching for "Fedora (13)" from hkp server subkeys.pgp.net
(1) Fedora (13) <fedora@fedoraproject.org>
      4096 bit RSA key E8E40FDE, created: 2010-01-19
...

Koji

Log into koji01 by way of gateway.fedoraproject.org.

Verify that /etc/koji-gc/koji-gc.conf has the new key in it.

Consider Before Running

Nothing at this time.