Joining the NixOS Pyramid Scheme

I was encouraged to try running NixOS on my server (the one that hosts this website!), and I decided to give it a shot. But... how do I move 7+ years worth of services from Arch to one of the weirdest Linux distros out there, with no prior experience?

Setting the Scene

As of April 2023, I've been running some form of VPS for almost a decade and a half, and slowly learning various sysadmin-y skills.

My first one ran Ubuntu 8.04 LTS, which somehow survived for far too long without getting popped.

In 2015, I moved to RamNode and tried running FreeBSD for a bit, which was fun until I ran into some weird IPv6 issue which seemed to be caused by an aspect of RamNode's setup that I've long since forgotten. I needed functioning v6 for my IRC bouncer to work properly, so I started up a new one running Arch Linux.

$ head /var/log/pacman.log
[2015-12-09 19:37] [PACMAN] Running 'pacman -r /mnt -Sy --cachedir=/mnt/var/cache/pacman/pkg --noconfirm base'
[2015-12-09 19:37] [PACMAN] synchronizing package lists
[2015-12-09 19:38] [ALPM] transaction started
[2015-12-09 19:38] [ALPM] installed linux-api-headers (4.1.4-1)
[2015-12-09 19:38] [ALPM] installed tzdata (2015g-1)
[2015-12-09 19:38] [ALPM] installed iana-etc (20151016-1)
[2015-12-09 19:38] [ALPM] installed filesystem (2015.09-1)
[2015-12-09 19:38] [ALPM] installed glibc (2.22-3)
[2015-12-09 19:38] [ALPM] installed gcc-libs (5.2.0-2)
[2015-12-09 19:38] [ALPM] installed ncurses (6.0-3)

Keeping an Arch-based server functioning for over 7 years is probably impressive, but I felt like I was due for an upgrade. RamNode have long since moved to OpenStack, and this box is on their "legacy SolusVM system" - how long would it be until they get rid of it and force me to migrate?

I also wanted to try setting up a Mastodon instance, but I thought that the 2GB RAM and the 40GB storage space might be a little limiting. I don't know exactly what CPU they're running on these hosts, as their KVM configuration hides that, but I also suspected that a VPS purchased in 2023 might perform better than one purchased in 2015 and now running on a legacy platform.

$ head /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 13
model name  : QEMU Virtual CPU version (cpu64-rhel6)
stepping    : 3
microcode   : 0x1
cpu MHz     : 2399.996
cache size  : 4096 KB
physical id : 0

The Problem

I'm not proud of this, but I never really learned the right way to do system administration - I usually just got stuff to work and then said "ok, I'm leaving it there". This bit me every once in a while, but I always had more interesting things to do with my time.

My server runs all the following services:

I also previously had these, which are now no longer in use:

Half of these things were just running in a tmux session, and I had to manually restart them every time my server rebooted. Occasionally I'd realise that my Twitter bots had been down for weeks or months, because I'd ran pacman -Syu, it installed a new Python version, and now their virtualenv was broken.

I thought that moving to a new server would be a good opportunity to try and fix these. I can learn more about system management (hopefully making me more employable 💷) and reduce maintenance time for Future Me, even if I spend a few days on it now.

Okay, fair enough. Now... how do I do it?

Choices

Annoyingly, there's no right answer, and a lot of different options. Some of the possible approaches:

Improving what I'm doing now

I do generally enjoy using Arch, and having up-to-date software is really useful sometimes. My current setup is a disaster, but I could make it more robust without going scorched-earth on it.

Writing systemd service files for all of my.. well, services would be a good first move into Doing Things Properly™ - I've already done that in code I've written for other people, I've just not done it for any of the stuff I run for myself!

This saves me from having to manually restart everything when I reboot, but it doesn't take care of everything I wanted to improve.

Configuration management tools would be great at solving the latter, but AFAIK, they generally seem to be designed for folk who are managing lots of servers in bulk. That's a valid use case, but it's the inverse of what I'm doing here. I want to run lots of things on one machine, not one thing on many machines.

Going all-in on Docker

I could just use a new machine as a general host for containers, and then the underlying OS doesn't matter all that much. Using plain old docker-compose to deploy things across multiple machines would suck, but for my single-machine use case, it would be pretty straightforward, and let me keep lots of my configuration in one easy place.

Up to now, I've only used Docker for development. I created an elaborate multi-container setup that allows you to run a local copy of the backend for Furcadia (the MMO that I occasionally moonlight dev work for), but I never got around to migrating production to use it, as I wanted to avoid creating further friction for what has traditionally been a very brittle service.

This means I'm fairly confident in my ability to create working images for all my subprojects, and since Docker is so widely used, then I can be pretty sure that running them in my "production" would be a safe, well-trodden path, even if I haven't gone down it myself yet.

The NixOS Pyramid

For almost a year, my partner has been jokingly referring to NixOS as a "pyramid scheme" (the first reference to this in our group chat is from May 2022).

A couple of folk suggested it as a possible answer to my system administration woes, so I decided to seriously look into it. The idea is really cool: you define what your system's configuration should be, and the tools just magically make it happen.

I really like the concept, but I'm still not sure how I feel about the actual implementation - even though I'm using it now, as you probably gathered from this post's title and introduction.

Nix is a package manager that you can run on Linux and Mac, where all the configuration and the package definitions themselves are written in a weird functional language that's also called Nix.

NixOS is a Linux distribution that's built on this, but they're so closely entwined that the nixpkgs repository contains both the package definitions that you can install anywhere and also all the NixOS modules.

I think you can use Nix itself without Nixpkgs, but I don't know why you would do that, so for my purposes I'm just going to consider them one in the same, even though that's not strictly correct.

I decided that the best way to give NixOS a fair assessment was to actually try setting it up, and see if I could run some of my services. If I really didn't like it, I could wipe the machine and install Arch again, or perhaps even something else entirely.

Setting up box.wuffs.org

Selecting a Vessel

Okay, so first I need a server. My current box has 2GB RAM and 40GB storage, and I'm paying $12/month (about €11) for that. I'd like to run a single-user Mastodon instance, so I think I'll need more than that.

I thought it would be cool to experiment with the ARM64 machines that Hetzner Cloud offers, since at the time of writing, they'd give me 4GB RAM + 40GB storage for €4.55/month, or 8GB RAM + 80GB storage for €7.79/month. That's a pretty good deal, and I suspect that a modern ARM64 core will still outperform whatever x86 chip is in my legacy RamNode VPS.

I asked a friend for a Hetzner Cloud referral link (since that gets me €20 free credit) and signed up. Then I hit my first hurdle: they currently only offer ARM64 machines in their DC in Falkenstein, Germany, and I wasn't sure if I wanted to host my site in .de.

First off, this might obligate me to publish an 'Impressum' page. On paper, this is only necessary for commercial sites based in Germany.

Secondly, Germany is pushing some strict policies on online content which I don't want to risk potentially running afoul of, which I believe has even led Twitter to block NSFW content in the country on at least one occasion.

I probably wouldn't have trouble with either of these, but I decided to play it safe and set up an x86_64 box in Hetzner's Finland DC instead.

Installing NixOS

Screenshot of the Hetzner Cloud interface for my server, which displays its specifications, bandwidth usage, costs, a log of recent configuration changes, and buttons for common tasks

Hetzner Cloud has a very slick control panel which runs circles around my previous host's SolusVM setup, and manages to make Amazon EC2 look really clunky. Everything loads fast. Selecting a server shows me the price and the usage charges I've incurred so far. There's even a dark theme, and live resource usage graphs.

You can't create a machine directly with NixOS, but that's easily fixed. Just pick any OS, create the machine, mount the NixOS install disc from the "Images" tab and reboot.

I followed the steps from the NixOS manual for manual installation.

Disk Setup and Bootloader

The original Hetzner Cloud image (I think I selected Ubuntu) provisioned my machine with three partitions:

I wasn't quite used to this setup, but decided to preserve it rather than recreating it - assuming that the system would boot using UEFI.

I wiped sda1 using mkfs.ext4 and wiped sda15 using mkfs.fat as described in the NixOS manual, then followed the install steps, using nixos-generate-config to build an initial configuration.

The manual recommends enabling systemd-boot if you're using UEFI, which I thought I would... so I ran the install process, unmounted the NixOS disc, rebooted, and it didn't start up.

(sad trombone sound)

I mounted the NixOS disc again and booted back into the install environment, then mounted my root partition again. I edited it again to install GRUB on the disk, and then ran nixos-install again:

boot.loader.grub.enable = true;
boot.loader.grub.version = 2;
boot.loader.grub.device = "/dev/sda";

That did the trick and got me in!

One last thing to get it to behave on Hetzner Cloud - IPv6 won't work by default as you have to manually set the machine's address. I followed the instructions on the NixOS Wiki: Install NixOS on Hetzner Cloud, but used enp1s0 as the interface name as that's what ip link showed.

Learning the Nix Pyramid

I've been using computers for 24 years of my 28-year existence, and I've been at least somewhat familiar with Linux for more than half of that. For better or worse, Nix just demolishes a bunch of my expectations on how things should work.

I don't dislike it, but it takes work to get used to this kind of paradigm shift.

Take this simple example: I use lsd as a nicer replacement for ls. On every other system I can think of, to get it, I'd just invoke a command like pacman -S lsd or brew install lsd or apt install lsd.

On NixOS, I add it to the environment.systemPackages list, and then run nixos-rebuild switch to move into this new version of my server's world, where I can use lsd. Alright, that makes sense.

But here's the first thing that tripped me up: I naively assumed that I could map these two concepts together, and treat systemPackages as a canonical list of things I've "installed" on my system. Nope, it's more complex than that.

For a command-line tool like lsd, this is the Right Thing to do, as far as I can tell. The system knows that lsd exposes a binary, and putting it into systemPackages will make NixOS magically download it and make that binary accessible from every user's PATH.

That's not the case for every kind of thing I might want to install, though. If I want to use a font on Arch Linux, I can just install the package using the same pacman -S command. If I want to use a font on NixOS, I have to add the package to fonts.fonts.

Nix does let you install things on an ad-hoc basis using the nix-env command, but I'm deliberately ignoring this because the declarative model is the reason I'm trying out NixOS. If I drop that, then I may as well just go back to Arch!

In Nix's declarative world, as far as I can tell, my entire system state is just a really fancy package that's built on demand, with lots of links going to other packages.

$ where lsd
/run/current-system/sw/bin/lsd
$ # Ok, so my path exists in here
$ ls -l /run/current-system
lrwxrwxrwx root root 87 B Fri Apr 21 00:10:15 2023 /run/current-system ⇒ /nix/store/fx7bgxzyjcn18grvyh5lswrw6ya4scp2-nixos-system-box-23.05pre474883.555daa9d339
$ # I'm pretty sure that current-system is the result of building my configuration
$ ls -l /run/current-system/sw
lrwxrwxrwx root root 55 B Thu Jan  1 01:00:01 1970 /run/current-system/sw ⇒ /nix/store/xjp3y9h6mmx1qv6r3piabrwdj9wgkdgz-system-path
$ # And 'sw' points to yet another Nix store object, which in turn points to all the things I can invoke
$ ls -l /run/current-system/sw/bin/lsd
lrwxrwxrwx root root 62 B Thu Jan  1 01:00:01 1970 /run/current-system/sw/bin/lsd ⇒ /nix/store/w0m916mbdy5dcp6w7mnddkxiy0i9djis-lsd-0.23.1/bin/lsd
$ # It's references to other nix store objects, all the way down
$ ldd /run/current-system/sw/bin/lsd
    linux-vdso.so.1 (0x00007ffcf350e000)
    libgcc_s.so.1 => /nix/store/g012c53brxmb0if3lpmkjwmxk74hjflh-gcc-12.2.0-lib/lib/libgcc_s.so.1 (0x00007f5261eb0000)
    libm.so.6 => /nix/store/1n2l5law9g3b77hcfyp50vrhhssbrj5g-glibc-2.37-8/lib/libm.so.6 (0x00007f5261dd0000)
    libc.so.6 => /nix/store/1n2l5law9g3b77hcfyp50vrhhssbrj5g-glibc-2.37-8/lib/libc.so.6 (0x00007f5261bea000)
    /nix/store/1n2l5law9g3b77hcfyp50vrhhssbrj5g-glibc-2.37-8/lib/ld-linux-x86-64.so.2 => /nix/store/1n2l5law9g3b77hcfyp50vrhhssbrj5g-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 (0x00007f52620c0000)

This is what allows you to do things like in-place upgrades and rollbacks - Nix is just building a new system with references to the new stuff, and once that's ready, it'll switch out the current-system link, and probably do some other weirdness that I don't yet understand.

There is a lot of official Nix and NixOS documentation, but it's lacking in places, and as a new user, I struggled heavily to discover it and to build up my mental model of "where do I go to learn about X?". There are three distinct manuals for Nix, Nixpkgs and NixOS, as well as the list of options.

The detailed documentation on each option is valuable, but it feels very JavaDoc-esque in that it's great for figuring out "what does services.x.y do? ah yep, it's for Z", but not as useful for going in the opposite direction.

NixOS also offers a lot of "modules" which make it easier to set up specific software packages in a declarative fashion, by generating config files for you, but some of these aren't even mentioned in the manual.

Anyway... I was feeling uneasy about this, but I decided to charge on anyway, with a plan in mind.

The Plan

Eight Easy(?) Steps

I split up the process of provisioning my server into tasks, which would: 1) make it less daunting/overwhelming, 2) hopefully let me learn NixOS's quirks, and 3) give me an escape route in case I decided it just didn't work for me.

At any point before I repoint wuffs.org, I can tinker as much as I want - this won't mess up any of my "live' services. If NixOS gets in my way too much, then I'm absolutely ready to bail and go with something I understand more.

Configuring zsh

This was the first thing I tried to do, assuming that it was a good litmus test for how much pain I'd encounter with declarative configuration.

The NixOS manual has an entire section for oh-my-zsh. Cool!

This takes the form of a NixOS module which reads config from the programs.zsh.ohMyZsh block, and then uses it to decide what to provision. So, I enabled it, and... nothing.

I searched the NixOS options for 'shell', and found a promising-looking option, which I added:

users.defaultUserShell = pkgs.zsh;

This got me zsh, but it had absolutely no clue that I'd tried to enable OMZ. So then I dug into the Nixpkgs repository for clues.

It turns out that there's also a NixOS module for zsh, which makes total sense in hindsight, but the NixOS manual doesn't mention it anywhere even though there's an entire section dedicated to OMZ itself. I also had to set programs.zsh.enable, which generates some boilerplate config files that in turn allow it to discover and load OMZ.

Not a great start so far, but I'll persevere.

oh-my-zsh plugins

Most of the plugins I use are included with OMZ, except for zsh-autosuggestions and zsh-syntax-highlighting. Trying to include these in my plugins list failed, of course - so how do I include them in NixOS?

Section 52.3. Custom environments informs me that I need to include the Nix packages into programs.zsh.ohMyZsh.customPkgs, upon which something will presumably happen.

There's Nix packages for both of these, so I add them to the customPkgs list and rebuild. OMZ still cannot find the plugins.

This is where I run into my first issue. The OMZ install instructions for zsh-autosuggestions say that the repository should be cloned into a subdirectory of OMZ's plugin directory, because the plugin file is in the repo's root, but it looks like NixOS's OMZ module expects the plugin file to be inside plugins/[pluginname].

The exact same thing occurs with zsh-syntax-highlighting: the install instructions tell you to clone it into plugins/zsh-syntax-highlighting, but this doesn't work with NixOS's customPkgs.

On the other hand, the nix-zsh-completions package, which does work in customPkgs, seems to avert this in its derivation by copying these files to the expected location inside installPhase, as shown here.

At this point in my Nix adventure, I don't even dare to try and fix the OS's packages. I barely know how to install them, let alone modify them!

Configuring nginx

This actually ended up being a lot easier. There is no nginx section in the NixOS manual, but the section on getting TLS certificates with ACME has a full example of a working nginx configuration.

I already knew how to configure nginx outwith Nix, so I looked at the NixOS nginx module's source code to help me better understand what was going on - this was my first step towards actually making sense of how NixOS modules work. (I don't like using systems that I have to treat as black boxes, with zero understanding of what's inside)

Based on the source, enabling the nginx module using services.nginx.enable triggers several events: a systemd service is added, nginx.conf is created, a nginx user and group are added, and logrotate is configured for nginx's logs.

So I did that, and it worked. This is the NixOS dream, right?

Setting up TLS via ACME

The "happy path" documented in the NixOS manual lets you set enableACME = true; on each nginx virtual host, which does some magic to get a cert for you. This is great, but I wanted a wildcard certificate so that I could use arbitrary subdomains, and so I could just move my existing services to this server before repointing them in the Real World.

The manual provides an example config under 51.5. Configuring ACME for DNS validation which sets up BIND to serve DNS, which seemed a little overkill for my usecase. They link to the Lego docs and mention that you can use other providers, so I did that.

security.acme = {
  acceptTerms = true;
  defaults = {
    # server = "https://acme-staging-v02.api.letsencrypt.org/directory";
    email = "ninji@wuffs.org";
  };
  certs = {
    "wuffs.org" = {
      domain = "wuffs.org";
      extraDomainNames = [ "*.wuffs.org" ];
      dnsProvider = "namesilo";
      credentialsFile = "/var/src/secrets/dns-api-token";
    };
  };
};

The secret file contains an API token for NameSilo, my registrar and DNS provider. That's all I needed - I used the staging server first to make sure it worked, then commented that out afterwards. Now, by adding useACMEHost = "wuffs.org"; to a nginx virtual host, it just works.

# cat /var/src/secrets/dns-api-token
NAMESILO_API_KEY=[redacted]
NAMESILO_PROPAGATION_TIMEOUT=915
NAMESILO_POLLING_INTERVAL=10

One thing I wasn't prepared for, however, is that the NixOS build process will block until the certificate has been provisioned. This means that if I change these settings, there's a 15-minute wait while DNS propagates. There might be a better approach, but I'll tolerate this for now as I rarely need to change the ACME settings.

Setting up PHP

Alright, next task. The NixOS Wiki's Nginx page has an example configuration for a "LEMP stack (Nginx/MySQL/PHP)" - it's just a block of Nix code with no documentation. Fair enough.

I used that to build a config. I've diverged slightly and set PHP to run as my own user, because I wanted it to be able to modify my own files so that the Grav admin panel will work.

services.phpfpm.pools = {
  dogpool = {
    user = "ninji";
    group = "nginx";
    settings = {
      "pm" = "dynamic";
      "pm.max_children" = 30;
      "pm.min_spare_servers" = 5;
      "pm.max_spare_servers" = 20;
      "pm.max_requests" = 500;
      "listen.owner" = "ninji";
      "listen.group" = "nginx";
    };
  };
};

By doing that, and adding the requisite FastCGI instructions to the nginx virtual host, it works.

I'm now four tasks in. Trying to set up oh-my-zsh was unnecessarily painful, but everything else was pretty slick. However, my next task involves trying to take some of my own software and bring it into the NixOS world. How the hell do I do that?

Packaging for NixOS

I'm going to admit here that I am 100% winging it. None of the quickstart guides or the narrative documentation I've found will give me a clear answer to "what's the best-practice way to install something I've written myself?"

The docs will tell you how to write a package and add it to Nixpkgs, but I'm fairly sure they don't want to include a package for software I've written myself with private source code. So, that's not very helpful.

I've got a few things I want to get up and running in some form or another:

I decided to tackle them in this order. I figured that the .NET app would be the easiest to try first, because it gets compiled, and hopefully Nix has decent tooling for that.

As a NixOS user, I want to package a .NET app, so that I can run it on my cool new server

The NixOS manual makes no mention of .NET, but there's a whole section for it in the Nixpkgs manual: 17.10. Dotnet. Promising!

This makes sense to me now that I've learned more about Nix, but the first time I looked at it, I had no clue what was going on.

There are instructions on how to use nix-shell to create an environment for local development, but that didn't seem relevant to my user story. Then, there's a list of bullet points explaining the arguments to buildDotnetModule, and an example default.nix.

Okay, so what do I do with this? Presumably, this example is written with the assumption that it's going to go into the giant monolithic Nixpkgs tree, but I just want to do my own thing.

I tried installing Nix for macOS. I went into my source folder for this project and created a default.nix file and ran nix-build, getting inscrutable errors that I have long since forgotten.

The manual's example file calls for a deps.nix:

  nugetDeps = ./deps.nix; # File generated with `nix-build -A package.passthru.fetch-deps`.

I had no idea how to get this command to work. I threw different things together and even looked at some other .NET software that was in Nixpkgs.

Eventually, I managed to hit on the right combination:

with import <nixpkgs> {};

buildDotnetModule rec {
  pname = "botfriends";
  version = "0.1";
  src = ./.;
  nugetDeps = ./deps.nix;
  dotnet-sdk = dotnetCorePackages.sdk_7_0;
  dotnet-runtime = dotnetCorePackages.runtime_7_0;
}

By commenting out the reference to deps.nix line and using nix-build -A passthru.fetch-deps (which, I should note, is different from the command in the manual!), it produces result, which is an executable script that generates the deps file and writes it to a file in a temporary directory - it does at least tell you the path.

I copied that file to ./deps.nix, uncommented the reference in default.nix and ran nix-build.

It builds the service and creates a new result, which is a link to a directory in the Nix store, containing my gay little service.

$ ./result/bin/new_bots
 14:31:28.686 INF >> [WebServer] Running HTTPListener: Unosquare HTTP Listener
 14:31:28.696 INF >> [WebServer] Web server prefix 'http://localhost:9933/' added.
 14:31:28.746 INF >> [WebServer] Started HTTP Listener

Success - I've built it on Nix for macOS! So now, I throw my two new .nix files into the repository, push it, and clone it into /var/src on my server.

I'm getting a bit more confident at reading Nix code now and figuring out what needs to go where, and how to adapt it. According to the manual, the import function will load a package if I pass a path to it. So I put together a few different elements and place this into my system's configuration:

systemd.services.botfriends = {
  requires = [ "network.target" ];
  after = [ "network.target" ];
  wantedBy = [ "multi-user.target" ];
  script = ''
    ${import /var/src/botfriends}/bin/new_bots
  '';
  serviceConfig.Restart = "always";
  serviceConfig.User = "ninji";
  serviceConfig.WorkingDirectory = /var/src/botfriends;
};

The fact that this works really helps demonstrate how Nix has no real concept of "installing" a package.

I never added it to systemPackages, because there's no reason for this service to be executable from the command line. However, since I've added a service that imports the package, Nix now knows that in order to create this service, it'll need to replace that ${import ...} reference with the path to that package's result.

In order to perform that, the package needs to be built, so it just... goes and does that. Nice.

Text-to-Speech Nonsense

Next up is the Node.js server that deals with my text-to-speech bots.

I originally wrote it for 'DittySongBot', which was a Telegram bot that would poke the backend server for Ditty, a now-defunct mobile app for generating ridiculous videos. It would take a song ID and text as inputs, make a request to the Ditty API, and then invoke ffmpeg to convert the result to the format Telegram wanted (Opus codec, Ogg wrapper).

Later on, I got my hands on some Scottish text-to-speech voices, and thought "it'd be fun to make these into a Telegram bot too". The problem is, I only had a Windows DLL for these. I wrote a tiny wrapper app for it that generated a .wav, and then ran it in WINE. This is horrendous, but it worked.

Can I do better? Yes.

People have a tendency to dump tons of things onto GitHub, including things that aren't supposed to be there. I did a bit of searching and found a repo containing Linux and macOS libraries for this TTS engine. Rewrite commenced!

After a few hours, I had 127 lines of TypeScript for Deno which use Deno's FFI to call into the TTS engine, followed by ffmpeg to perform the transcoding. My original Node.js implementation used temporary files, but I was able to sidestep that and just use ffmpeg's stdin/stdout which is way cleaner IMO.

Okay, so, how do I deploy a Deno app on Nix? There's no mention of it in the Nixpkgs manual. They do package Deno, but as far as I can tell, there's no tooling for packaging something written with Deno.

I found github.com/SnO2WMaN/deno2nix which looked promising, but I had no clue what I was doing. It uses "flakes", a feature of Nix that I hadn't even looked into. The only documentation is a sample project, with no explanation whatsoever.

I wasn't ready for this level of adventurousness at this point, so I just installed Docker and built a Docker image for the project. Sure, it'd have been nice to do this the Proper ❄️ Nix way, but I also need to be pragmatic. This still meets my goals of being easily reproducible, and it won't break when the OS updates.

Python Bots

The last of the ad-hoc services I needed to port was my army of Telegram bots. These were using a rather old version of the python-telegram-bot library, so I decided to use this as an opportunity to rewrite them using the latest version, and bundle them into one script.

I got that working locally on my Mac, and then I realised... this means I have to deal with Python packaging for the first time since 2009. I don't remember anything about it.

The Nixpkgs manual's Python section is pretty detailed, and far better than the .NET section. It looks like they expect me to have a package in a standard Python format, so I had to give myself a crash course on this.

After a bunch of finagling, I ended up with:

At first, I didn't realise I had to explicitly list the Python dependencies I needed. This worked great after a couple of iterations though...

with import <nixpkgs> {};

python310.pkgs.buildPythonPackage rec {
  pname = "tgbots";
  version = "0.0.1";
  src = ./.;
  format = "pyproject";
  propagatedBuildInputs = with python310.pkgs; [ setuptools python-telegram-bot ];
  doCheck = false;
}

...until I tried to deploy it to my server. As it turns out, NixOS 22.11 ships python-telegram-bot version 13, and I'd just rewritten my script around version 20 which is a big refactor of the library.

I think there are ways to bring in specific packages from the Nixpkgs unstable (rolling-release) channel, but I didn't want to try and figure that out yet. I decided to yolo and switch to the unstable channel, figuring that I can go back to stable once 23.05 releases.

I switched channels, ran nixos-rebuild switch and watched as it downloaded a couple of gigabytes of new software. Against all odds, it actually worked!

I figured I should pull in the new kernel as well, so I rebooted the system. It was actually pretty satisfying to see all my services come back up within 30 seconds, without me having to open tmux and manually start them up. "You lived like this?"

Redirecting my Website

At this point, I was ready to switch wuffs.org over. I copied over all my web trees, set up nginx virtual hosts for the subdomains I use, modified my hosts file to point to my new server's IP (so I could check that I hadn't screwed anything up), and then once it all seemed OK, I pointed the domain to the new server.

I've now had this server for 3 days, and I've gone from never having touched Nix, to having a functioning machine with a bunch of services - both 'standard' and custom things I've built myself. I guess that's not too bad.

My prime goal was to switch my domain name over so that I could try and set up a Mastodon instance.

Slight tangent: Over the past decade, I ended up becoming really reliant on Twitter as my main form of social media, but they're circling the drain now. I switched to Mastodon instance vulpine.club in late 2022, and was happy with it, but they're shutting down in 3 months.

You can migrate accounts, but you don't keep your content. I've been in this limbo where I didn't want to post anything because I knew I'd lose it when v.c shuts down, so I wanted to move ASAP. Running my own instance means I don't have to worry about someone else burning out and closing down.

Setting up Mastodon

If you're not familiar with it, Mastodon is a bulky web app written in Ruby on Rails that is essentially a federated Twitter clone based on ActivityPub. It was bestowed upon us by benevolent tech magnate John Mastodon, but now it belongs to everybody.

There are alternatives like Calckey and GoToSocial, but I'm somewhat invested in Mastodon because Ivory (the Tweetbot successor from Tapbots) doesn't yet work with these, and I really enjoy using Ivory.

There's no mention of Mastodon in the NixOS manual, but Nixpkgs does ship a Mastodon package and there's even a NixOS Mastodon module.

It looks pretty solid, so why am I not using it? Well, I want to run the glitch-soc fork of Mastodon ("Glitch Edition"), which adds a bunch of neat features. As typical for NixOS, there are "escape hatches" that let you swap out the backing package, but I'd have to figure out how to make it work.

I don't want my Mastodon version to be tied to the OS version, since I want to get new features ASAP if they get added. I also don't want to try and compile Mastodon on this server - I've heard the asset build process can be very memory-hungry, so that might be tricky. Nix apparently lets you do remote build shenanigans, but I don't want to delve into that right now.

So, once again I'm reaching for Docker. The GitHub repo for Glitch Edition offers pre-built images via Actions, so that's easy.

I referred to this useful blog post from Ben Tasker: Running a Mastodon Instance using docker-compose - it's not specifically about Glitch Edition, and I diverged slightly, but it was still very helpful for getting my bearings.

I installed Docker Compose by adding it to my systemPackages (finally, something that works there!), then I grabbed the docker-compose.yml file from the Glitch Edition repository and placed it into /var/src/mastodon.

Mastodon's Compose file includes a Postgres database, a Redis database and three Mastodon services: 'web', 'streaming' and 'sidekiq'.

Postgres and Redis simply use standard images from Docker Hub, and the three Mastodon services all share the same image with different entrypoints. This seemed a little odd to me, but maybe it simplifies the build process for them? Anyway, these all reference upstream Mastodon, so I just repointed them to ghcr.io/glitch-soc/mastodon and specified a version tag for good measure.

I followed the steps from Ben's guide to create a Postgres database, create a role, run the setup wizard and then bring up the service.

URL Witchcraft

I'm going to have to confess to crimes of vanity here. The sensible approach, that most people take, is to devote an entire domain or subdomain to Mastodon.

If I did this, however, I would have to be something like @Ninji@social.wuffs.org or @Ninji@fedi.wuffs.org, and I didn't want that. I wanted the elegance of using my existing domain name.

"But that's still fine, Ninji", you might say. "Just use WEB_DOMAIN! That's the whole point of it!"

That's close-ish. I get to be @Ninji@wuffs.org in ActivityPub-land, but the URL to my toots will still include some other subdomain. I really wanted to do something cooler.

Currently, wuffs.org has the website you're reading right now, as well as a bunch of other things. None of these should conflict with Mastodon. In theory, I can have both set up on the same exact domain.

Consider the set of URLs that Mastodon will handle in a meaningful way. These can be split up into roughly four categories.

Category Example What do we do?
1) Assets CSS, javascript, images Set CDN_HOST
2) ActivityPub server-to-server /.well-known, /actor, /inbox No conflicts, forward directly
3) Things you only see if you're logged in Settings, timelines Can mostly be ignored
4) Publicly accessible stuff Users, toots, auth UI No conflicts, forward directly

My instance's primary URL is set to wuffs.org, and I also have ALTERNATE_DOMAINS set to mastodon.wuffs.org (I don't know if this is truly necessary, but it can't hurt).

Assets: By setting CDN_HOST to mastodon.wuffs.org, all the assets get loaded from there, so I can just forget about having to route those.

ActivityPub: I read through the Mastodon app's routes.rb file to try and find all the ActivityPub-relevant endpoints, and also checked the ActivityPub spec, and I think I've covered everything important in that regard.

Private UI: I really don't care about this, since I'm the only person using the instance - I'm happy with logging into it via the subdomain.

Public UI: I basically want to make sure that if someone opens a toot (e.g. wuffs.org/@Ninji/110235828576970876) in their browser, it will load. I forwarded all URLs where the path begins with @, as well as the authentication UI and so forth.

The biggest issue comes from the last one, but it's fairly low-priority for me right now.

Mastodon is a single-page web app that uses the HTML5 History API to pretend it has lots of pages that don't really exist as far as the server is concerned.

On the current release, if I click on "Profiles directory" at the bottom left, I'll get redirected to wuffs.org/directory, and if I click on "Keyboard shortcuts", I'll get redirected to wuffs.org/keyboard-shortcuts. I haven't actually told nginx to proxy these URLs to Mastodon, but they still appear to work, because the app just tells your browser to make a new history entry and then loads the appropriate content.

If I hit Refresh after doing this, my browser makes a request to wuffs.org/directory, which returns a 404 from Grav (this website's backend).

This is definitely fixable, since all of these routes are listed in the source code - I've just got to go through them and throw them into a giant regex in my nginx configuration. (Or maybe I'll just do the inverse, and forward everything but certain URLs? I think nginx might allow me to do that 🤔)

Configuration (Final) (Really Final) (2)

These are the virtual host blocks I'm currently using for this setup.

      "wuffs.org" = {
        # [generic config elided]

        # Mastodon
        locations."^~ /api/v1/streaming" = {
          recommendedProxySettings = true;
          proxyWebsockets = true;
          proxyPass = "http://127.0.0.1:4000/";
          extraConfig = ''
            proxy_buffering off;
            proxy_redirect off;
            tcp_nodelay on;
          '';
        };
        locations."~ ^/(custom\.css|\.well-known|oauth|auth|nodeinfo|@|%40|users|inbox|outbox|manifest|api|authorize_follow|authorize_interaction|actor|share|tags|emojis|relationships|media_proxy|media)" = {
          recommendedProxySettings = true;
          proxyPass = "http://127.0.0.1:3000";
        };
      };
      "mastodon.wuffs.org" = {
        addSSL = true;
        useACMEHost = "wuffs.org";
        locations."/" = {
          recommendedProxySettings = true;
          proxyPass = "http://127.0.0.1:3000/";
        };
        locations."^~ /api/v1/streaming" = {
          recommendedProxySettings = true;
          proxyWebsockets = true;
          proxyPass = "http://127.0.0.1:4000/";
          extraConfig = ''
            proxy_buffering off;
            proxy_redirect off;
            tcp_nodelay on;
          '';
        };
      };

There's definite room for improvement here. I could maybe take advantage of nginx caching for certain kinds of requests. I could factor out the repeated streaming API definition.

I do need to improve the location regex - some of those may not be necessary, and then there's other stuff I probably should add so that all the links in the public web UI work properly.

But it's a functional starting point, and now I can post again. You can follow me at @Ninji@wuffs.org, thanks to this mess. Don't expect good posts though :p

What Next?

I've covered 7 of my 8 tasks - I still haven't moved my git server. That's the next thing I need to do, but it's far less glamorous. Crucially, though, it all relies on a subdomain, so I can do that in my own time and just repoint it when I'm ready. Once I do that, and move a friend's website over, then I can decommission my old Arch VPS.

"Ninji, do you actually like NixOS now?"

... I'm not sure. The concept is really cool, even if I feel like I'm too small-brained to understand half of it. On the other hand, I've gotten a functioning machine and I've even packaged some of my own software, so I guess that counts for something?

I don't see myself using Nix in other scenarios, although maybe that'll change later on - we'll see. The learning curve is steep and the documentation isn't as good as it could be, although it's also not the worst I've seen.

For now I think I'm okay with this approach. I'll run it on my server where I can reap the benefits of having declarative configuration, I'll be pragmatic and use Docker to run things that I can't easily slot into a Nix-shaped hole at my current skill level, and hopefully with time I'll learn more.

Thank you for reading this screed on my Nix adventures; I hope it found you well.

I'm going for a walk now that I've finished writing this 🐺 🏙️

Previous Post: mpw-emu: Emulating 1998-Vintage Mac Compilers
Next Post: They Made A Golf MMO With Sonic In it (Real!) (Not Clickbait!) (Only A Bit)