updates to pipeline blog

content/blog/self-hosted-pipeline/index.md

@@ -18,11 +18,102 @@ content_meta:
   trending: true
 ---
 
-## **Building a Deterministic, Self-Hosted Build Pipeline on NixOS**
+### Introduction: Motivation
 
-In modern DevOps, the "path of least resistance" usually involves a mix of GitHub Actions, Docker Hub, and managed hosting. While efficient, this approach often treats the build environment as a "black box," introducing external dependencies and privacy trade-offs. To deepen my understanding of infrastructure and gain total control over my deployment lifecycle, I migrated my infrastructure to a fully self-hosted, declarative stack on NixOS.
+There are many ways to maintain a personal portfolio website. In the past, I
+have taken the path of least resistance, using GitHub Pages and Actions, Docker
+Hub, or even just using `scp` to copy built files. While manually copying files
+is inefficient, the automation and reliability of using GitHub Pages and Actions
+makes pushing new changes instant. However, this approach is dependent on external
+services, and abstracts away the build/production environment into a black box.
+To deepen my understanding of infrastructure and gain complete control over the
+development to production lifecycle of my personal site, I began investigating
+options for self-hosting this way.
 
-The journey began with a migration from a standard Debian VPS on Digital Ocean. Using the `nixos-consume` script, I bootstrapped NixOS directly onto the live instance, replacing a mutable, manually-managed OS with a deterministic one. My entire server is now defined via Nix Flakes, allowing me to manage Nginx, Gitea, and my CI/CD runners from a single, version-controlled source of truth.
+I've had this website up for a few years now, and it began with one mere
+html index page on a Debian server, hosted with Nginx. At this stage, there was
+no CI/CD, and the developer experience was using `scp` to copy files. After
+working on developing software at Ruddervirt and Eigenvector, I missed being
+able to take advantage of git integration and CI/CD. Merging a change to the
+main branch meant those changes were automatically built and deployed to
+production, without needing any other developer intervention.
+
+Inspired by this, I considered alternatives to my current setup, guided by the
+following criteria: I need to have a reproducible, deterministic, secure, and
+efficient server setup. For this, I went with NixOS, for the declarative
+configuration files, and flakes.
+
+#### NixOS: More than just a distro, a paradigm shift
+
+Configuration on NixOS is much better than using just Debian (or any distro,
+without using a tool like Ansible) because unlike the experience on imperative
+distributions, where modifications to the system require either running commands
+like `apt install ...` or modifying configuration files and then forgetting
+what changes you made, or what file in `/etc/` you changed, using a declarative
+distribution allows for me to home all changes in one single configuration
+language, nix, and even one single configuration file (but it is a good idea to
+abstract this down into multiple `.nix` files). Admittedly, this paradigm change
+from imperative changes to declarative OS configuration is very disruptive to
+a workflow of being used to using the shell and modifying software-specific
+files in `/etc/`, and the learning curve is steeper than it appears at first,
+especially when you realize that you have to learn an entire new programming
+language, `nix`, before you can start to configure NixOS. The 'nix' way of doing
+things is often *never* intuitive coming from an imperative mindset, however
+once I read the solution to declaratively doing something, it's clear that that
+method is much better, and often makes more sense.
+
+A simple example of this is installing and configuring Nginx to serve https. On
+Debian, this involves using `apt install`, `systemctl enable/start`, and editing
+files in `/etc/nginx/*`, before setting up certbot for ACME with a similar
+process. Contrastingly, the setup on NixOS is shockingly simple: edit
+`configuration.nix` to include:
+
+```nix
+services.nginx = {
+  enable = true;
+  virtualHosts = {
+    "ejs.cam" = {
+      forceSSL = true;
+      enableACME = true;
+      locations."/" = { root = "/var/www"; };
+    };
+  };
+}
+```
+
+Then, `nixos-rebuild switch`. Both methods involve an equal amount of time spent
+typing, but NixOS centralizes the configuration to one single file, instead of
+it being across the shell, and multiple files in `/etc/`. Want to test what
+would happen with some experimental change to this nginx configuration? On
+Debian, you spend time going back in forth from the shell, to a text editor, and
+trying to undo old changes. On NixOS, it's painfully easy to rollback changes to
+changes done with `nixos-rebuild switch`: each switch is a boot option in GRUB
+when booting. Furthermore, NixOS allows me to test changes to the server
+configuration from my laptop.
+
+Using Flakes makes it so that I can lock the exact versions of
+each package installed to keep the system exactly reproducible between my laptop
+and the server. If the nginx server, firewall, and containers run properly on
+the NixOS configuration of my laptop, it will work exactly the same way on my
+server, by simply using `nixos-rebuild` with `--target-host <ejs.cam>` to
+remotely deploy the operating system configuration. Furthermore, flakes mean
+that that any individual deployment/configuration will work at any point in the
+future, if it works now. It's possible for something to get updated on Debian
+package repositories and to break a different piece of software, requiring
+debugging. With flakes, this never will happen.
+
+Typically one installs NixOS using a USB drive for the installer, but I use
+Digital Ocean for hosting the public-facing side of my website. Unfortunately,
+like many other hosting providers, Digital Ocean doesn't provide NixOS as an
+option for when spinning up a new server. Working around this, I used
+`NixOS-Infect`, an awesome script that will use an existing distro install, and
+replace it with NixOS, similar to how a USB installer can install to an already
+mounted file system. Now, I had an endpoint to remotely deploy nixos flakes to,
+from my laptop. Furthermore, updating packages on the cheapest shared
+hosting option from Digital Ocean (which has only 1 cpu thread and 1 GB ram) is
+incredibly slow. `apt` on Debian can take minutes to decompress packages after
+downloading, but with `nix`, I can download and build binaries ready for the
+server on my laptop, which has much much more compute, saving me time.
 
 ##### **The Architecture: Gitea, Podman, and Nginx**