add cuirass adventure

content/blog/2024-06-11-deploying-cuirass-on-guixsd.md

@@ -0,0 +1,316 @@
++++
+title = Deploying Cuirass on GuixSD
+date = <2024-06-11 Tue>
+author = Peter Tillemans
+email = pti@snamellit.com
+[taxonomies]
+tags = [programming]
+categories = [guix, linux]
++++
+
+
+# Table of Contents
+
+1.  [Install Cuirass in Guix](#orge201d70)
+2.  [Create a postgresql server](#org0f19dc3)
+3.  [Enable the cuirass service](#org5f8483e)
+4.  [Enabling the Web Frontend](#org8d12621)
+5.  [Submitting jobs](#orged5a51b)
+    1.  [Figure out a more elegant way to submit jobs](#org2ef3b08)
+
+
+<a id="orge201d70"></a>
+
+# Install Cuirass in Guix
+
+I had a hard time installing **cuirass** on my system. I tried to collect
+some notes to move step by step to a working solution.
+
+GUIX is great that it is really easy to revert your step to get out or
+a dead end and start over from a previous point. Something I made
+liberally use off to get something working. However it also means my
+deployment process was not really linear but much more error driven.
+
+
+<a id="org0f19dc3"></a>
+
+# Create a postgresql server
+
+Starting the cuirass service will pull in a postgres server, however
+it will use by default a version 10. Let's update that to 15 for some
+future proofing:
+
+    (service postgresql-role-service-type
+             (postgresql-role-configuration
+              (roles
+               (list (postgresql-role
+                      (name "cuirass")
+                      (create-database? #t))
+                     (postgresql-role
+                      (name "xyz")
+                      (create-database? #t))))))
+
+I also added a database user for me (**xyz** here stands for my user
+account).
+
+This will also create a database with the same name to allow the
+cuirass user (and me) to login to the postgres database and do
+database things.
+
+Run a \`guix system reconfigure\` and check if the postgres is running
+with \`sudo herd status postgres\`. 
+
+
+<a id="org5f8483e"></a>
+
+# Enable the cuirass service
+
+With the database in place there is a fighting chance to get the
+service running :
+
+    (service cuirass-service-type
+             (cuirass-configuration
+              (specifications #~(list))))
+
+The example in the reference documentation offers something more
+interesting than an empty list but whatever I tried ended up with
+'invalid field specifier' errors. But I get that too for nginx
+configuration parts so that is probably a skill issue on my part.
+
+With a bit of luck we'll see:
+
+    xyz@foo ~/.config/dotfiles/guix [env]$ sudo herd status cuirass
+    Status of cuirass:
+      It is running since 05:19:42 PM (4 hours ago).
+      Running value is 6513.
+      It is enabled.
+      Provides (cuirass).
+      Requires (user-processes guix-daemon postgres postgres-roles networking).
+      Will be respawned.
+    xyz@foo ~/.config/dotfiles/guix [env]$ sudo herd status cuirass-web
+    Status of cuirass-web:
+      It is running since 05:22:41 PM (4 hours ago).
+      Running value is 6679.
+      It is enabled.
+      Provides (cuirass-web).
+      Requires (user-processes cuirass).
+      Will be respawned.
+
+If not there may be some info in \`/var/log/cuirass.log\` or \`/var/log/cuirass-web.log\`
+
+Alternatively for debugging we can run the application from the git
+repository. 
+
+First of all we have to give our user access to the database:
+
+    xyz@foo ~/.config/dotfiles/guix [env]$ sudo -u cuirass psql
+    Password:
+    psql (15.4)
+    Type "help" for help.
+    
+    cuirass=> grant all on database cuirass to xyz;
+    GRANT
+
+if the cuirass service has initialised the database already you can
+add:
+
+    cuirass=> grant all on all tables in schema public to xyz;
+    GRANT
+    cuirass=> grant all on all sequences  in schema public to xyz;
+    GRANT
+
+This allows access without jumping through the sudo hoop.
+
+The code in the **guix-cuirass** project folder will now just work. Except
+the postgresql socket should be exposed too in the proposed command of
+the reference manual:
+
+    guix shell -CPNW --expose=/var/log/guix/drvs \
+    --expose=/var/run/dbus --expose=/run/avahi-daemon \
+    --expose=/etc/ssl/certs --expose=/var/run/postgresql
+
+Note that if the tables and sequences are created when running in your
+account it is quite possible that the **cuirass** user will not be able to
+access them and complain with access denied errors. In that case we
+have to do the grants for the **cuirass** user:
+
+    xyz@foo ~/.config/dotfiles/guix [env]$ psql cuirass
+    Password:
+    psql (15.4)
+    Type "help" for help.
+    
+    xyz=> grant all on all tables in schema public to cuirass;
+    GRANT
+    xyz=> grant all on all sequences  in schema public to cuirass;
+    GRANT
+
+Then restarting the service with \`sudo herd restart cuirass\` should
+make it start, or at least give a different error.
+
+Once **cuirass** service is running, it will be possible to run the
+**cuirass-web** service. It relies on the existence of the
+**/var/run/cuirass/bridge** file which is created by the cuirass service.
+
+Just to be sure that the web server is running :
+
+    xyz@foo ~/src/guix-cuirass [env]$ wget http://localhost:8081
+    --2024-06-11 21:59:09--  http://localhost:8081/
+    Resolving localhost (localhost)... 127.0.0.1
+    Connecting to localhost (localhost)|127.0.0.1|:8081... connected.
+    HTTP request sent, awaiting response... 200 OK
+    Length: 6142 (6.0K) [text/html]
+    Saving to: ‘index.html’
+    
+    index.html                    100%[================================================>]   6.00K  --.-KB/s    in 0s
+    
+    2024-06-11 21:59:09 (378 MB/s) - ‘index.html’ saved [6142/6142]
+
+Cool, we have a 200 Ok status code and some index.html file so the web
+UI is running. Now exposing it to the 'net.
+
+
+<a id="org8d12621"></a>
+
+# Enabling the Web Frontend
+
+In the reference manual there is a nice configuration to start from to
+configure nginx as a frontend for **cuirass-web**
+
+However you cannot start nginx https without having the certificates
+and you cannot get the certificates without the server running. (Well
+you can but that is outside the scope of this post)
+
+So we have to cut back the configuration to only start the http nginx
+server to do the first handschake with letsencrypt to get the initial
+certificates
+
+Let's start with the **certbot-service** :
+
+    (service certbot-service-type
+             (certbot-configuration
+              (email "xyz@bar.com")
+              (certificates
+               (list
+                (certificate-configuration
+                 (domains '("foo.bar.com")))
+                ))))
+
+Then add the minimal part for a nginx http server to do the
+letsencrypt dance.
+
+    (service nginx-service-type
+             (nginx-configuration
+              (server-blocks
+               (list
+                ;; TLS is required for authentication; serve the site via
+                ;; HTTPS only.
+                (nginx-server-configuration
+                 (listen '("80"))
+                 (raw-content
+                  (list "return 308 https://$host$request_uri;")))
+    
+                ))))
+
+Doing a \`guix system reconfigure\` now will start the nginx server and
+download fresh certificates.
+
+We can now add the https server proxy-ing the **cuirass-web** server:
+
+    (service nginx-service-type
+             (nginx-configuration
+              (server-blocks
+               (list
+                ;; TLS is required for authentication; serve the site via
+                ;; HTTPS only.
+                (nginx-server-configuration
+                 (listen '("80"))
+                 (raw-content
+                  (list "return 308 https://$host$request_uri;")))
+    
+                (nginx-server-configuration
+                 (listen '("443 ssl"))
+                 (server-name '("foo.bar.com"))
+                 (ssl-certificate "/etc/letsencrypt/live/foo.bar.com/fullchain.pem")
+                 (ssl-certificate-key "/etc/letsencrypt/live/foo.bar.com/privkey.pem")
+                 (locations
+                  (list
+                   ;; Proxy the whole Cuirass web site...
+                   (nginx-location-configuration
+                    (uri "/")
+                    (body (list "proxy_pass http://localhost:8081;")))
+                   ;; ... but require authentication for the admin pages.
+                   (nginx-location-configuration
+                    (uri "~ ^/admin")
+                    (body
+                     (list "if ($ssl_client_verify != SUCCESS) \
+                { return 403; } proxy_pass http://localhost:8081;")))))
+                 ;; (raw-content
+                 ;;  ;; Register your self-generated certificate authority.
+                 ;;  (list "ssl_client_certificate /etc/ssl/certs/Snamellit_CA.pem;"
+                 ;;        "ssl_verify_client optional;"))
+                 )
+                ))))
+
+Creating and installing the root CA is a bit out of scope. For this
+post we'll just wave our hands and assume the certificate magically
+appeared in the \`/etc/ssl/certs/Snamellit<sub>CA.pem</sub>\` location. Creating a
+client certificate for firefox and importing it allows to access the
+admin section and retrigger jobs etc.
+
+Once that is setup the commented out section can be activated.
+
+
+<a id="orged5a51b"></a>
+
+# Submitting jobs
+
+For some reason I do not understand yet, the specification file must
+be in the loadpath of the cuirass program.
+
+For testing I add them to the **examples** folder in the **guix-cuirass**
+folder in the home folder of the **cuirass** user
+
+    $ sudo -u cuirass -- bash              # start a shell in the cuirass user
+    $ cd guix-cuirass                      # enter the project folder
+    $ cp /foo/bar/snamguix.scm examples    # put spec somewhere on loadpath
+    $ # start dev environment in container
+    $ guix shell -CPNW --expose=/var/log/guix/drvs \
+        --expose=/var/run/dbus --expose=/run/avahi-daemon \
+        --expose=/etc/ssl/certs --expose=/var/run/postgresql
+    guix shell: loading environment from '/home/pti/src/guix-cuirass/guix.scm'...
+    $ ~/src/guix-cuirass [env]$ # register new recipe
+    $ ~/src/guix-cuirass [env]$ ./pre-inst-env cuirass register -S examples/snamguix.scm
+    2024-06-11T20:18:08 running Fibers on 8 kernel threads
+    2024-06-11T20:18:08 marking stale builds as "scheduled"...
+    2024-06-11T20:18:08 builds will be made via the local build daemon
+    2024-06-11T20:18:08 will perform up to 8 evaluations concurrently
+    2024-06-11T20:18:08 opening bridge socket at '/tmp/cuirass-tests/var/run/cuirass/bridge'
+    2024-06-11T20:18:08 retrieving list of pending builds...
+    2024-06-11T20:18:08 unused GC roots older than 2592000s will be deleted every 86400s
+    2024-06-11T20:18:08 deleting old GC roots from '/var/guix/gcroots/profiles/per-user/pti/cuirass'...
+    2024-06-11T20:18:08 selected 0 GC roots to remove
+    WARNING: (cuirass base): imported module (fibers) overrides core binding `sleep'
+    2024-06-11T20:18:08 heap: 12.18 MiB; threads: 17; file descriptors: 76
+    WARNING: (cuirass scripts register): imported module (fibers) overrides core binding `sleep'
+    2024-06-11T20:18:08 canceling 0 stale builds
+    2024-06-11T20:18:08 restarting 0 pending builds
+    2024-06-11T20:18:08 building 0 derivations in batches of 200
+    2024-06-11T20:18:08 done with 0 derivations
+    2024-06-11T20:18:08 outputs:
+    <C-C> 
+    $
+
+This actually starts the scheduler and it just keeps running (unless
+we also give the &#x2013;one-shot flag) but the side effect is to add the
+specification to the database.
+
+From now on the channel will be checked and build any updated
+packages.
+
+
+<a id="org2ef3b08"></a>
+
+## TODO Figure out a more elegant way to submit jobs
+
+There has to be a more elegant way 
+