General structure

A general overview of the folder structure can be found below:

├── assets
├── compose
│   ├── chaotic-backend
│   ├── chaotic-v4
│   ├── docker
│   │   └── configs
│   ├── docker-proxied
│   ├── firedragon-runner
│   ├── github-runner
│   ├── gitlab-runner
│   └── mastodon
├── docs
│   ├── src
│   │   ├── hosts
│   │   │   ├── aerialis
│   │   │   └── stormwing
│   │   ├── repositories
│   │   ├── services
│   │   ├── users
│   │   └── websites
├── home-manager
├── nixos
│   ├── hosts
│   │   ├── aerialis
│   │   └── stormwing
│   ├── modules
│   │   ├── special
│   │   └── static
│   └── services
│       ├── compose-runner
│       ├── monitoring
│       └── mk.nix # garuda-lib construction helpers
├── pkgs
├── scripts
└── secrets

Infrastructure diagrams

nix-topology renders diagrams of all hosts, containers, networks and services from the NixOS configurations. Run topology in the devshell to regenerate the SVGs in docs/src/topology and commit them, they are shown on the Infrastructure diagrams page.

Most things are extracted automatically. Additions live in:

• nixos/topology.nix for global nodes and networks (Internet, Cloudflare, Tailscale)
• nixos/modules/topology.nix for services nix-topology can’t detect (mailserver, redis, runners, tunnels, …)
• nixos/services/compose-runner/compose-runner.nix, which reads every compose.yml and adds its Docker containers with image and ports
• nixos/modules/nspawn-containers.nix, which wires containers to their host bridge

Secrets in this repository

Secrets are managed via the sops-nix module, which allows us to encrypt sensitive files and supply them in an encrypted way to our hosts. They will then be decrypted at runtime by using the hosts ed25519 SSH host key. This is done by using the sops tool, which encrypts files using a key stored in the ~/.config/sops/ directory. The encrypted files live in the secrets directory of this repository, recipients are configured in .sops.yaml:

• secrets/<host>.yaml holds the secrets of one host and its containers, and is only decryptable by that host (and the admins). It is the default sopsFile, picked via the host name (or garuda.motd.parentHost inside containers).
• secrets/common.yaml holds secrets needed on every host, like user password hashes. Secrets from it need sopsFile = ../../secrets/common.yaml; (relative to the module) in their sops.secrets declaration.

A new host needs its age key (ssh-to-age < /etc/ssh/ssh_host_ed25519_key.pub) in .sops.yaml, a creation rule for its own file, and an entry in the common.yaml rule. Only values are encrypted, so key names are visible to anyone with access to the repository. Every secret is consumed at runtime via sops.secrets or sops.templates, nothing secret may be evaluated into the Nix store.

To view or edit any of these files, one can use the following commands:

sops secrets/aerialis.yaml # opens editor for the file
sops -e secrets/filename.yaml # encrypts the file
sops -d secrets/filename.yaml # decrypts the file

This assumes a fitting sops key is available in the ~/.config/sops/ directory. After changing recipients in .sops.yaml, run sops updatekeys secrets/<file>.yaml to re-encrypt the data key.

Passwords in general

Our mission-critical passwords that maintainers and team members need to have access to are stored in our Bitwarden instance. After creating an account, maintainers need to be invited to the Garuda Linux organisation in order to access the stored credentials.

Linting and formatting

We utilize pre-commit-hooks to automatically set up the pre-commit-hook with all the tools once nix-shell or nix develop is run for the first time. Checks can then be executed by running one of the following configs:

nix flake check # checks flake outputs and runs pre-commit at the end
pre-commit run --all-files # only runs the pre-commit tools on all files

Its configuration can be found in the flake.nix file. (click me). At the time of writing, the following is being run:

The pre-commit hooks generated into .pre-commit-config.yaml:

• check-json
• check-yaml
• detect-private-keys
• ripsecrets

Formatting is handled by treefmt, which runs:

• actionlint
• deadnix
• nixfmt
• shellcheck
• shfmt
• statix

It is recommended to run pre-commit run --all-files before trying to commit changes. Then use cz commit to generate a commitizen complying commit message.

CI/CD

We have used pull-/push-based mirroring for this git repository, which allows easy access to Renovate without having to run a custom instance of it. The following tasks have been implemented as of now:

• nix flake check runs for every labeled PR and commit on main.
• Renovate periodically checks docker-compose.yml and other supported files for version updates. It has a dependency dashboard as well as the developer interface to check logs of individual runs. Minor updates appear as grouped PRs while major updates are separated from those. Note that this only applies to the GitHub side.
• Deployment of our mdBook-based documentation to Cloudflare pages.
• Deployment of our Website to Cloudflare pages.

Workflows will generally only be executed if a relevant file has been changed, eg. nix flake check won’t run if only the README was changed.

Monitoring

Our monitoring stack is self-hosted and defined in nixos/services/monitoring, replacing the previous Netdata setup. It consists of Prometheus for metrics, Loki for logs, Grafana for dashboards and Alertmanager for alerting, plus Fluent Bit and node_exporter agents that hosts and containers opt into via garuda.monitoring.

Because both hosts run their own 10.0.5.0/24 container bridge, the monitoring container cannot reach stormwing’s containers directly - stormwing proxies their exporters and relays their logs over the Tailnet instead. See Monitoring for details.

Where configuration lives

• nixos/hosts/<host>.nix - host-specific networking, NAT forwards, the container list and host-level tunnels.
• nixos/hosts/<host>/<container>.nix - one file per container.
• nixos/services/ - shared service modules, plus mk.nix which holds the garuda-lib helpers used to build repetitive structures.