Brian SnelgroveBrian Snelgrove

BRI runs on its own hardware: vSphere hosts on 10 GbE, Pure, Isilon and QNAP storage, and seven bare-metal HPC servers — up to 128 cores and 773 GiB of RAM each — joined directly to production Kubernetes.

Every VM starts from a Packer-built Ubuntu golden image that CI rebuilds whenever its template changes.

One Terraform monorepo describes BRI’s infrastructure across vSphere, Azure, AWS, Kubernetes, identity, networking and the automation platform itself, split into 26 independent state roots so a change touches only what it should.

Every pull request gets a plan per root and an IaC security scan; merging applies it on our own runners. Certificates and app secrets rotate on a schedule, not on someone’s memory.

Terraform provisions AWX itself — job templates, inventories, credentials and schedules — and new VMs register and launch their own configuration jobs.

Kubespray playbooks install, upgrade, scale and roll-reboot clusters one node at a time, and etcd backups are encrypted before they leave the box.

Four GitOps-managed clusters — dev, staging, a 23-node production cluster that includes the HPC servers, and a standalone monitoring node — run on hardware BRI already owns.

Operators run MySQL, PostgreSQL and MongoDB; Velero backs up workloads; cert-manager handles TLS; secrets come from a key vault at runtime; Prometheus and Loki watch everything.

Our builds run on our own cluster: autoscaling runner scale sets on Kubernetes, 7 to 15 general runners plus a small isolated privileged pool that only allow-listed workflows from main can use.

I also wrote the shared workflows every app calls. A repo gets lint, tests, a single image build, a vulnerability scan and a verified rollout by adding a few lines of YAML.

FluxCD is the critical link between GitHub and our clusters. Git is the source of truth: Flux watches the registry, commits new image tags as its own bot, and reconciles every cluster to match. Deploys, rollbacks and even database restores are pull requests.

Images are built once and promoted by retagging, so staging and production run byte-identical code. When the public website moved onto the platform, its news page went from over 3 seconds to about half a second.

Publishing a release first takes blocking backups of the app’s databases and volumes and records a MongoDB point-in-time restore point. Only then does the production image tag exist, so Flux has nothing to deploy until the safety net is in place.

Backups land in encrypted S3 — 30 daily, 13 weekly and 12 monthly — and a restore is a reviewed pull request, applied exactly once.

Dependabot keeps our repos current — Terraform providers, GitHub Actions, Docker base images, and npm, Bun, Go and Python packages — and a Claude skill I wrote triages the backlog into merge, verify, investigate or hold, in the order to land them.

Trivy scans three layers: the Terraform on every pull request, every image the pipeline builds, and what is actually running in the clusters. CodeQL and lint gates run in the shared workflows, and third-party actions are pinned to commit SHAs.

When node2shell (CVE-2026-111497) hit in October 2026, I checked every app’s Node version and had a verified fix (Node 24 LTS plus the outstanding npm findings) in under three hours, staging first. This site builds on that release too.

The monitoring stack is defined in FluxCD like everything else: Prometheus, Alertmanager and Grafana, with Loki collecting the logs. I wrote most of it — 67 of the 93 commits — and Grafana signs in with our single sign-on.

On top of the stock rules sit 72 of my own: database health (no primary, replication lag, WAL archiving, failed or stale backups, volumes filling), HPC hardware (SMART and NVMe wear, ECC memory errors, temperatures, OOM kills), certificates, and storage. Alertmanager routes them by component to the right Slack channel, labeled with the cluster they came from.