website/content/blog/2024-06-11-deploying-cuirass-on-guixsd.md

+++
title = "Deploying Cuirass on GuixSD"
date = "2024-06-11"
author = "Peter Tillemans"
email = "pti@snamellit.com"
[taxonomies]
tags = ["programming"]
categories = ["guix", "linux"]
+++


# Table of Contents

1.  [Install Cuirass in Guix](#orge201d70)
2.  [Create a postgresql server](#org0f19dc3)
3.  [Enable the cuirass service](#org5f8483e)
4.  [Enabling the Web Frontend](#org8d12621)
5.  [Submitting jobs](#orged5a51b)
    1.  [Figure out a more elegant way to submit jobs](#org2ef3b08)


<a id="orge201d70"></a>

# Install Cuirass in Guix

I had a hard time installing **cuirass** on my system. I tried to collect
some notes to move step by step to a working solution.

GUIX is great that it is really easy to revert your step to get out or
a dead end and start over from a previous point. Something I made
liberally use off to get something working. However it also means my
deployment process was not really linear but much more error driven.


<a id="org0f19dc3"></a>

# Create a postgresql server

Starting the cuirass service will pull in a postgres server, however
it will use by default a version 10. Let's update that to 15 for some
future proofing:

    (service postgresql-role-service-type
             (postgresql-role-configuration
              (roles
               (list (postgresql-role
                      (name "cuirass")
                      (create-database? #t))
                     (postgresql-role
                      (name "xyz")
                      (create-database? #t))))))

I also added a database user for me (**xyz** here stands for my user
account).

This will also create a database with the same name to allow the
cuirass user (and me) to login to the postgres database and do
database things.

Run a \`guix system reconfigure\` and check if the postgres is running
with \`sudo herd status postgres\`. 


<a id="org5f8483e"></a>

# Enable the cuirass service

With the database in place there is a fighting chance to get the
service running :

    (service cuirass-service-type
             (cuirass-configuration
              (specifications #~(list))))

The example in the reference documentation offers something more
interesting than an empty list but whatever I tried ended up with
'invalid field specifier' errors. But I get that too for nginx
configuration parts so that is probably a skill issue on my part.

Edit: definitely skill issue. I did not see that I defined them in the `operating-system` expression instead of on toplevel because it starts several screens up and down. Duh. Moving the define of the specifications to toplevel and using them here works just fine.

With a bit of luck we'll see:

    xyz@foo ~/.config/dotfiles/guix [env]$ sudo herd status cuirass
    Status of cuirass:
      It is running since 05:19:42 PM (4 hours ago).
      Running value is 6513.
      It is enabled.
      Provides (cuirass).
      Requires (user-processes guix-daemon postgres postgres-roles networking).
      Will be respawned.
    xyz@foo ~/.config/dotfiles/guix [env]$ sudo herd status cuirass-web
    Status of cuirass-web:
      It is running since 05:22:41 PM (4 hours ago).
      Running value is 6679.
      It is enabled.
      Provides (cuirass-web).
      Requires (user-processes cuirass).
      Will be respawned.

If not there may be some info in \`/var/log/cuirass.log\` or \`/var/log/cuirass-web.log\`

Alternatively for debugging we can run the application from the git
repository. 

First of all we have to give our user access to the database:

    xyz@foo ~/.config/dotfiles/guix [env]$ sudo -u cuirass psql
    Password:
    psql (15.4)
    Type "help" for help.
    
    cuirass=> grant all on database cuirass to xyz;
    GRANT

if the cuirass service has initialised the database already you can
add:

    cuirass=> grant all on all tables in schema public to xyz;
    GRANT
    cuirass=> grant all on all sequences  in schema public to xyz;
    GRANT

This allows access without jumping through the sudo hoop.

The code in the **guix-cuirass** project folder will now just work. Except
the postgresql socket should be exposed too in the proposed command of
the reference manual:

    guix shell -CPNW --expose=/var/log/guix/drvs \
    --expose=/var/run/dbus --expose=/run/avahi-daemon \
    --expose=/etc/ssl/certs --expose=/var/run/postgresql

Note that if the tables and sequences are created when running in your
account it is quite possible that the **cuirass** user will not be able to
access them and complain with access denied errors. In that case we
have to do the grants for the **cuirass** user:

    xyz@foo ~/.config/dotfiles/guix [env]$ psql cuirass
    Password:
    psql (15.4)
    Type "help" for help.
    
    xyz=> grant all on all tables in schema public to cuirass;
    GRANT
    xyz=> grant all on all sequences  in schema public to cuirass;
    GRANT

Then restarting the service with \`sudo herd restart cuirass\` should
make it start, or at least give a different error.

Once **cuirass** service is running, it will be possible to run the
**cuirass-web** service. It relies on the existence of the
**/var/run/cuirass/bridge** file which is created by the cuirass service.

Just to be sure that the web server is running :

    xyz@foo ~/src/guix-cuirass [env]$ wget http://localhost:8081
    --2024-06-11 21:59:09--  http://localhost:8081/
    Resolving localhost (localhost)... 127.0.0.1
    Connecting to localhost (localhost)|127.0.0.1|:8081... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 6142 (6.0K) [text/html]
    Saving to: ‘index.html’
    
    index.html                    100%[================================================>]   6.00K  --.-KB/s    in 0s
    
    2024-06-11 21:59:09 (378 MB/s) - ‘index.html’ saved [6142/6142]

Cool, we have a 200 Ok status code and some index.html file so the web
UI is running. Now exposing it to the 'net.


<a id="org8d12621"></a>

# Enabling the Web Frontend

In the reference manual there is a nice configuration to start from to
configure nginx as a frontend for **cuirass-web**

However you cannot start nginx https without having the certificates
and you cannot get the certificates without the server running. (Well
you can but that is outside the scope of this post)

So we have to cut back the configuration to only start the http nginx
server to do the first handschake with letsencrypt to get the initial
certificates

Let's start with the **certbot-service** :

    (service certbot-service-type
             (certbot-configuration
              (email "xyz@bar.com")
              (certificates
               (list
                (certificate-configuration
                 (domains '("foo.bar.com")))
                ))))

Then add the minimal part for a nginx http server to do the
letsencrypt dance.

    (service nginx-service-type
             (nginx-configuration
              (server-blocks
               (list
                ;; TLS is required for authentication; serve the site via
                ;; HTTPS only.
                (nginx-server-configuration
                 (listen '("80"))
                 (raw-content
                  (list "return 308 https://$host$request_uri;")))
    
                ))))

Doing a \`guix system reconfigure\` now will start the nginx server and
download fresh certificates.

We can now add the https server proxy-ing the **cuirass-web** server:

    (service nginx-service-type
             (nginx-configuration
              (server-blocks
               (list
                ;; TLS is required for authentication; serve the site via
                ;; HTTPS only.
                (nginx-server-configuration
                 (listen '("80"))
                 (raw-content
                  (list "return 308 https://$host$request_uri;")))
    
                (nginx-server-configuration
                 (listen '("443 ssl"))
                 (server-name '("foo.bar.com"))
                 (ssl-certificate "/etc/letsencrypt/live/foo.bar.com/fullchain.pem")
                 (ssl-certificate-key "/etc/letsencrypt/live/foo.bar.com/privkey.pem")
                 (locations
                  (list
                   ;; Proxy the whole Cuirass web site...
                   (nginx-location-configuration
                    (uri "/")
                    (body (list "proxy_pass http://localhost:8081;")))
                   ;; ... but require authentication for the admin pages.
                   (nginx-location-configuration
                    (uri "~ ^/admin")
                    (body
                     (list "if ($ssl_client_verify != SUCCESS) \
                { return 403; } proxy_pass http://localhost:8081;")))))
                 ;; (raw-content
                 ;;  ;; Register your self-generated certificate authority.
                 ;;  (list "ssl_client_certificate /etc/ssl/certs/Snamellit_CA.pem;"
                 ;;        "ssl_verify_client optional;"))
                 )
                ))))

Creating and installing the root CA is a bit out of scope. For this
post we'll just wave our hands and assume the certificate magically
appeared in the \`/etc/ssl/certs/Snamellit<sub>CA.pem</sub>\` location. Creating a
client certificate for firefox and importing it allows to access the
admin section and retrigger jobs etc.

Once that is setup the commented out section can be activated.


<a id="orged5a51b"></a>

# Interactivele Submitting jobs

For some reason I do not understand yet, the specification file must
be in the loadpath of the cuirass program.

For testing I add them to the **examples** folder in the **guix-cuirass**
folder in the home folder of the **cuirass** user

    $ sudo -u cuirass -- bash              # start a shell in the cuirass user
    $ cd guix-cuirass                      # enter the project folder
    $ cp /foo/bar/snamguix.scm examples    # put spec somewhere on loadpath
    $ # start dev environment in container
    $ guix shell -CPNW --expose=/var/log/guix/drvs \
        --expose=/var/run/dbus --expose=/run/avahi-daemon \
        --expose=/etc/ssl/certs --expose=/var/run/postgresql
    guix shell: loading environment from '/home/pti/src/guix-cuirass/guix.scm'...
    $ ~/src/guix-cuirass [env]$ # register new recipe
    $ ~/src/guix-cuirass [env]$ ./pre-inst-env cuirass register -S examples/snamguix.scm
    2024-06-11T20:18:08 running Fibers on 8 kernel threads
    2024-06-11T20:18:08 marking stale builds as "scheduled"...
    2024-06-11T20:18:08 builds will be made via the local build daemon
    2024-06-11T20:18:08 will perform up to 8 evaluations concurrently
    2024-06-11T20:18:08 opening bridge socket at '/tmp/cuirass-tests/var/run/cuirass/bridge'
    2024-06-11T20:18:08 retrieving list of pending builds...
    2024-06-11T20:18:08 unused GC roots older than 2592000s will be deleted every 86400s
    2024-06-11T20:18:08 deleting old GC roots from '/var/guix/gcroots/profiles/per-user/pti/cuirass'...
    2024-06-11T20:18:08 selected 0 GC roots to remove
    WARNING: (cuirass base): imported module (fibers) overrides core binding `sleep'
    2024-06-11T20:18:08 heap: 12.18 MiB; threads: 17; file descriptors: 76
    WARNING: (cuirass scripts register): imported module (fibers) overrides core binding `sleep'
    2024-06-11T20:18:08 canceling 0 stale builds
    2024-06-11T20:18:08 restarting 0 pending builds
    2024-06-11T20:18:08 building 0 derivations in batches of 200
    2024-06-11T20:18:08 done with 0 derivations
    2024-06-11T20:18:08 outputs:
    <C-C> 
    $

This actually starts the scheduler and it just keeps running (unless
we also give the &#x2013;one-shot flag) but the side effect is to add the
specification to the database. It also allows to see immediately if there are syntax 
errors or similar. When satisfied Ctrl-C out of it.

From now on the channel will be checked and build any updated
packages.


<a id="org2ef3b08"></a>

## Permanently add the job

Once vetted the specification can be added to the system configuration.

Define the specifications on the toplevel of the system config file. (Be careful, I lost a lot of time because I did not see it was actually in the *operating-system* expression)

    ;; cuirass specifications
    (define %cuirass-specifications
    #~(list
        (specification
        (name "hello")
        (build 'hello))
        (specification
        (name 'snamguix)
        (build '(channels . (snamguix)))
        (channels
        (cons (channel
                (name 'snamguix)
                (url "https://forge.snamellit.com/pti/snamguix.git")
                (branch "main"))
                %default-channels)))))


then replace the empty list in the *cuirass-service* :


           (service cuirass-service-type
                    (cuirass-configuration
                     (specifications %cuirass-specifications)))

reconfigure your system and restart *cuirass* with `sudo herd restart cuirass`.

It is not *really* needed as it was already added to the database, but this will be useful when the CI server need to be repaved.