Homelab Adventures: From K3s to Nomad to Reclaim My Free Time (Part 2)

In the first part of this series, we left Travolta and Thurman—my two Intel N150 mini PCs with 16 GB of RAM each—on the brink of functional exhaustion.

I had built a K3s cluster with every enterprise bell and whistle imaginable: database operators, distributed block storage via Longhorn squeezed onto two nodes, GitOps driven by Flux CD, and an in-cluster observability suite featuring Grafana Mimir and Loki HA that devoured over 7 GB of RAM before I even deployed a single custom API.

The outcome was as predictable as it was frustrating: I had inadvertently become an unpaid on-call site reliability engineer in my own home. Every Sunday afternoon was hijacked by degraded storage volumes, pods frozen in Terminating states, or reconciliation loops spiraling out of control in Flux.

It was time to hit the emergency brake and perform major architectural surgery. This article chronicles how I tore down that precarious house of cards and rebuilt the entire environment around a radically different philosophy: zero accidental complexity, maximum available compute, and relentless simplicity.

---

The Decision: The Cluster Is for Compute Only

The primary sin of my original architecture was attempting to emulate the managed conveniences of AWS or GCP on two budget mini PCs sitting on my desk.

In a physical two-node home cluster, strict distributed consensus is unforgiving. Trying to maintain synchronous replicated block storage and clustered databases across two machines imposed an unbearable operational tax. The first architectural pivot was cutting off two massive sources of friction at the root:

1. Ditching StatefulSets: Native PostgreSQL on Travolta with Automated Cloudflare R2 Backups

On two nodes, whenever Thurman rebooted for a routine kernel patch, Longhorn lost 50% of its disk replicas and immediately panicked. Furthermore, running heavyweight operators like CloudNativePG and Percona MongoDB on modest hardware proved to be an endless source of operational drag.

The pragmatic solution: stop pretending we are running an enterprise distributed storage cluster, and decouple persistence from the task orchestrator:

• Native PostgreSQL 16 on the server node (Travolta): Instead of wrestling with fragile distributed block storage replicas or relying on cloud serverless databases (which introduce 20 ms network latency, cold-start pauses, and quota anxieties), we provisioned PostgreSQL directly on Travolta as a native systemd service, idempotently managed via Ansible.
• Instant performance and minimal footprint: We get sub-millisecond local NVMe response times (< 0.5 ms), zero cold-start delay for candlestick market ingestion and trading bots, and an idle memory footprint of just 150–250 MB of RAM.
• Zero exposed ports: Services running on Travolta talk to 127.0.0.1:5432, while workloads on Thurman or dev laptops connect securely over the encrypted Tailscale mesh (100.64.0.0/10 / travolta-tailscale:5432) with SCRAM-SHA-256 authentication.
• Cloudflare R2 backups orchestrated by Nomad: State is protected not by synchronous multi-node replication, but by a scheduled Nomad batch job (postgres-backup.nomad.hcl) running scripts/postgres-backup.sh daily. It takes compressed local dumps (with a rolling 7-day retention) and streams them via rclone directly to a Cloudflare R2 bucket (homelab-backups/postgres/), visible with status logs and failure alerts right in the Nomad Web UI.

2. Enterprise-Grade Observability Without the Overhead: Grafana Cloud

Running Grafana Alloy locally to ship telemetry to an in-cluster HA Loki deployment and a Mimir backend flushing blocks to R2 was pure technical vanity for monitoring a handful of side projects. It consumed over 4 GB of RAM just keeping compactor routines, memberlist gossip rings, and indexers alive.

Yet operating blind wasn’t an option: Prometheus metrics, centralized logs in Loki, and alerting are indispensable. The breakthrough move was offloading the entire telemetry backend to Grafana Cloud’s Free Tier:

• The free tier includes 10,000 active metric series, 50 GB of logs, and 50 GB of traces every month—an immense headroom for any homelab or development setup.
• Instead of crunching and indexing telemetry on local hardware, a single lightweight Grafana Alloy agent consuming roughly ~50–80 MB simply gathers host metrics and container logs, forwarding them over TLS directly to Grafana Cloud endpoints.
• The payoff: I preserve the exact same world-class dashboards and alert pipelines I had in K3s, but immediately reclaimed over 6 GB of free RAM for actual applications.

With a clean slate and the hardware finally able to breathe, the next question was inevitable: did it still make sense to drag around the weight of the Kubernetes control plane?

---

The New Engine: Why HashiCorp Nomad

K3s is an extraordinary feat of engineering for shrinking Kubernetes into a compact binary, but it remains Kubernetes underneath: a resource-hungry API server, k3s-server, controller-managers, kubelet, an overlay CNI (Flannel), and dozens of CRDs tracking desired state.

The Two-Node Control Plane Dilemma: K3s vs. Nomad

Running Kubernetes on a two-node cluster (Travolta and Thurman) is a crash course in distributed systems… and operational pain. The mathematics of distributed consensus are unforgiving:

• In etcd, quorum requires a strict majority: (N/2) + 1. With 2 nodes, the required quorum is 2. Losing a single node breaks quorum (1 < 2), causing etcd to drop into read-only mode and freezing the entire control plane to prevent split-brain scenarios.
• If you opt for the typical homelab architecture—one dedicated control-plane node (Travolta) and one worker node (Thurman) using embedded SQLite (Kine)—the situation is even more fragile: if Travolta crashes or its database corrupts, the cluster is instantly decapitated.

Recovering a fallen K3s control plane is a weekend-ruining endeavor: wrestling with emergency recovery flags like k3s server --cluster-reset, restoring out-of-sync etcd snapshots, regenerating join tokens, debugging expired or desynced TLS certificates, and scraping away orphan virtual network interfaces left behind by Flannel and iptables. If you decide to promote the surviving worker into the new master, the manual recipe often requires uninstalling agents, purging /var/lib/rancher/k3s directories, and praying your manifests haven’t vanished.

In Nomad, the contrast is stark: Nomad is a single static Go binary capable of operating as a server (server), an execution agent (client), or both simultaneously. Because we decoupled state from the compute workloads and safeguarded data with automated Cloudflare R2 backups and versioned Git repos, the Nomad server holds no irreplaceable configurations in its own control plane: its only job is coordinating tasks.

If the machine acting as the Nomad server blows up, rebuilding the control plane on the surviving node takes roughly 30 seconds:

1. Open /etc/nomad.d/nomad.hcl and declare the server role:

server {
  enabled          = true
  bootstrap_expect = 1
}

client {
  enabled = true
}

2. Run sudo systemctl restart nomad.

Is there downtime? Yes, a couple of minutes while the surviving node assumes the role and schedules the jobs. Do you have to rebuild the entire cluster from scratch or panic? Absolutely not. It is not the existential disaster of losing etcd in Kubernetes, where you are suddenly trapped in a maze of broken certificates, stuck NotReady nodes, and blind snapshot recoveries. In Nomad, it is a minor, deterministic, and delightfully boring incident: you tweak two lines of config, restart the binary, trigger your GitHub Actions pipeline (or run nomad job run *.nomad.hcl), and your containers are back up immediately. No quorum surgeries, no TLS certificate regeneration, and no CNI networking triage.

Educational Value vs. Operational Overhead

Let’s give credit where credit is due: operating Kubernetes is profoundly educational. Wrestling with etcd, dissecting controller reconciliation loops, configuring overlay CNIs, and understanding container immutability forces you to become a sharper infrastructure engineer. It gives you first-hand insight into how modern, planetary-scale distributed platforms operate. I recommend that every engineer experience running it in a lab at least once.

However, there is a fine line between learning distributed engineering and paying a perpetual operational tax. For a personal setup or a two-node homelab, Kubernetes is massive overkill. It forces you to maintain an ocean liner when all you need is a reliable bridge to cross the river. Nomad delivers 90% of the real-world container orchestration benefits (health checks, traffic routing, automated restarts, declarative job specs) with just 5% of the operational baggage. It respects both your hardware and your free time.

The Cloud Analogy: The ECS vs. EKS Approach

To understand this paradigm shift, the best analogy from the cloud ecosystem is the contrast between AWS ECS and AWS EKS (Kubernetes):

• Kubernetes (EKS) strives to be a complete distributed operating system: it abstracts networking, storage, DNS, and application lifecycles under infinite layers of indirection and complex objects.
• AWS ECS, conversely, embraces pure pragmatism: it is a container scheduler focused strictly on compute. You define a task (Task Definition) specifying CPU and memory limits, and ECS places containers onto instances, links them to a load balancer (ALB), and delegates database and state concerns to specialized external services.

Nomad is, fundamentally, having the clean and pragmatic philosophy of AWS ECS running on your own bare metal. It doesn’t attempt to reinvent distributed storage or force you into packet encapsulation; it specializes in being an exceptionally fast, lightweight, and predictable task scheduler:

• A single 70 MB binary: Nomad is compiled as a standalone Go binary. It requires no external etcd; it uses its embedded Raft consensus protocol for server clustering.
• Negligible memory footprint: The Nomad agent runs both server and client roles consuming just 80–100 MB of RAM per machine.
• Zero mandatory overlay network: Unlike Flannel or Calico, Nomad doesn’t need to wrap packets in VXLAN tunnels for containers to communicate. It leverages host networking or native Linux bridge interfaces directly.
• Clean conceptual model: Gone are the days of fragmenting a single deployment across five YAML manifests (Deployment, Service, Ingress, ConfigMap, HPA). In Nomad, an entire application is declared within a single, legible HCL file (job).

# api-service.nomad.hcl
job "api-service" {
  datacenters = ["homelab"]
  type        = "service"

  group "web" {
    count = 2

    network {
      port "http" {
        to = 8080
      }
    }

    service {
      name     = "api-service"
      port     = "http"
      provider = "nomad"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.api.rule=Host(`api.miluna.dev`)"
      ]

      check {
        type     = "http"
        path     = "/health"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "server" {
      driver = "docker"

      config {
        image = "ghcr.io/miluna/api-service:latest"
        ports = ["http"]
      }

      resources {
        cpu    = 200
        memory = 256
      }
    }
  }
}

Nomad organizes workloads by their real lifecycle: service for continuous long-running applications, batch for finite run-to-completion tasks (like a database migration or backup script), and system for jobs that run exactly once on every node (like an edge proxy).

---

Consistent Dynamic Ingress: Traefik and Cloudflare Tunnel in Nomad

One of the distinct differences when transitioning from Kubernetes to Nomad is port allocation: in Nomad, tasks typically receive dynamic ephemeral ports assigned by the scheduler on the host, eliminating port collision risks when running multiple replicas.

To route traffic seamlessly without orchestrating external dependencies on the host OS, the entire edge ingress layer runs as declarative Nomad system jobs (one guaranteed replica per physical node):

• Traefik with native integration: Listens on host port 80 across both machines and communicates directly with Nomad’s local API (http://localhost:4646). Whenever Nomad launches a container replica on an arbitrary port (as seen in our earlier job spec with to = 8080), Traefik detects its service tags (traefik.http.routers...) and reconfigures its routing table dynamically without service reloads.
• Containerized Cloudflare Tunnel: Rather than running cloudflared as an unmanaged systemd daemon on the OS, it is managed directly as an orchestrated task across Travolta and Thurman.

The result is a network layer that is 100% declarative and secure: external HTTPS traffic arrives encrypted through Cloudflare’s global edge network, forwards locally into Traefik, and distributes cleanly across healthy replicas without ever opening a single port on my home router.

---

Farewell to Flux: Direct Deployments with GitHub Actions and Tailscale

During the K3s era, I religiously adhered to the dogma of Pull-based GitOps with Flux CD: in-cluster controllers polling Git repositories every three minutes, SOPS-encrypted secrets, and reconciliation loops that collapsed into a maze of broken CRDs whenever memory got tight.

For this streamlined setup, I pivoted to a pragmatic Deterministic Push driven by GitHub Actions:

[ Git Push / Merge to Main ] 
             │
             ▼
   [ GitHub Actions Runner ]
             │  1. Ephemeral auth via Tailscale (OAuth tag:ci)
             │  2. Zero app secrets in CI (stored in Nomad Variables)
             ▼
   [ Travolta / Thurman (:4646) ] ──> nomad job run against private API

• Zero in-cluster controllers: Instead of burning precious RAM running resident reconciliation daemons inside Travolta and Thurman, deployments are triggered from GitHub Actions upon merging to main.
• Ephemeral private mesh with Tailscale: The GitHub runner spins up a temporary node on my private Tailscale tailnet via OAuth and runs nomad job run directly against Nomad’s private API, while workloads resolve their secrets natively from Nomad Variables provisioned via SOPS.
• Immediate feedback: If a container fails to start or a healthcheck doesn’t respond, the pipeline turns red in under 20 seconds with the exact error log—no waiting for an in-cluster operator to notice.

💡 Companion Deep Dive:
The full architectural showdown between Pull-based GitOps (Flux/ArgoCD) vs. Push (GitHub Actions + VPN), secret management without Vault, and the complete step-by-step CI/CD workflow are examined in depth in a dedicated companion article: Why I Ditched Flux: The Case for GitHub Actions and Tailscale in Lightweight Deployments →.

---

The Fine Print: Real-World Downsides of the New Setup

Tearing down Kubernetes to embrace Nomad and a stateless architecture is not a silver bullet without trade-offs. To maintain sound engineering honesty, we must clearly acknowledge the compromises made when switching paradigms:

Feature	Previous Kubernetes (K3s)	Current Nomad + Cloud Setup
Control Plane	Heavy (~1.5–2 GB baseline RAM)	Ultra-lean (~80–100 MB baseline RAM)
Network Isolation	100% On-Premise / Air-gapped capable	Hybrid: Local DB and compute, Cloud telemetry and backups
Databases	Heavy clustered state (Longhorn + CloudNativePG)	Native PostgreSQL 16 on Travolta + Scheduled Cloudflare R2 backups via Nomad
Observability	Heavy local suite (Mimir + Loki HA)	Grafana Cloud Free Tier (Data leaves home network)
Deployments & CI	In-cluster pull with Flux + SOPS	Push via GitHub Actions connecting over Tailscale
Ecosystem	Community Helm charts, CRDs, operators	Handcrafted HCL job files written from scratch
Licensing	Open Source (Apache 2.0 / CNCF)	HashiCorp BSL (Business Source License)

1. Hybrid Dependency on External Cloud Services

In the previous K3s architecture, if my home ISP suffered an outage, the cluster continued operating in total isolation: local PostgreSQL instances responded fine, and metrics were stored on local NVMe drives.

By running PostgreSQL directly on Travolta, local applications and trading bots retain full autonomy and sub-millisecond query performance without depending on active Internet access. However, we maintain a hybrid reliance on external cloud services:

• If home Internet drops, telemetry cannot reach Grafana Cloud, Cloudflare Tunnels drop offline, and daily backup syncs cannot reach Cloudflare R2 (though local dumps continue rotating in /var/backups/postgresql/).
• If Grafana Cloud or Cloudflare ever alter their free tier limits, retention periods will require rebalancing or incurring small object storage costs.

2. Single-Node Database Persistence vs. The Two-Node HA Myth

By relinquishing distributed storage operators or cloud-managed serverless tiers, we make an explicit, honest architectural trade-off:

• Single hardware failure point for persistent state: PostgreSQL runs as a systemd service directly on Travolta. If Travolta suffers a catastrophic hardware failure (motherboard or NVMe failure), the database is offline until the service is spun up on Thurman or restored from the latest Cloudflare R2 dump.
• Operational backup discipline: High availability is replaced by operational clarity: verifying that automated daily backups to Cloudflare R2 are reliable, tested, and quick to restore.
• For a two-machine homelab, this pragmatic setup is infinitely more predictable and robust than Longhorn’s fragile illusion of distributed consensus, which frequently seized storage volumes whenever a node restarted.

3. The “Handcrafted” HCL Tax vs. Helm Ecosystem

Kubernetes dominates the enterprise because an engineer can run helm repo add and deploy a complete Redis cluster, Kafka, or CMS in minutes thanks to community-maintained charts.

With Nomad, you are on your own:

• No centralized package registry like Helm: If you want to deploy a message broker or third-party service, you must hand-author the .nomad.hcl specification from scratch.
• You are forced to deeply understand CPU/RAM allocations, Traefik tags, health check definitions, and port bindings. You gain surgical control over every byte running on your hardware, but you pay for it in time spent writing configuration.

4. Attack Surface: Granting Network Access to GitHub Actions

With Flux CD, the entire continuous delivery loop was pull-based and contained within the local network perimeter; secrets were decrypted with private keys that never left the physical machines.

Under the GitHub Actions push model:

• An ephemeral runner hosted in Microsoft’s cloud assumes Tailscale credentials (tag:ci) and temporarily joins my private home network to invoke the Nomad API.
• Even though Tailscale ACLs strictly restrict the runner’s destination IPs and ports, you are placing trust in GitHub Secrets and the integrity of your CI pipeline steps. Furthermore, if GitHub Actions or Tailscale experience an outage, all cluster deployments halt.

---

Results: The Cluster Serving the Code

The impact after completing Travolta and Thurman’s migration was immediately noticeable across every operational metric and in peace of mind:

Metric	K3s + In-Cluster (Before)	Nomad + Cloudflare (Now)
Idle RAM usage	> 7 GB (~45% of hardware)	< 450 MB (< 3% of hardware)
Free memory for apps	~8 GB shared pool	~30 GB dedicated and clean
Split-brain risk	Constant (2-node distributed storage)	Zero (100% stateless compute)
Deployment velocity	2–5 min (Flux reconciliation loops)	< 30 seconds (Direct GitHub Actions)
Maintenance overhead	Weekly (CRDs, patches, PVCs)	Unattended (Static Go binary)

The Four Major Breakthroughs

1. Cloud-grade compute without cloud invoices: Having roughly 30 GB of net usable memory for personal projects provides enormous freedom. In AWS, maintaining 32 GB of dedicated 24/7 compute (instances like m6i.2xlarge or t4g.2xlarge) would cost between $120 and $150 every month for instances alone. On my desk, that capacity runs silently, indefinitely, and at zero operating cost.

2. True horizontal scalability without split-brain nightmares: In K3s, scaling or moving pods was a minefield due to storage quorum constraints in Longhorn. By anchoring PostgreSQL as a reliable service on Travolta and decoupling backups to R2, the Nomad compute cluster is purely elastic: scaling a service from 2 to 10 replicas or rescheduling tasks across nodes happens in seconds without fear of network partitions or degraded volumes.

3. Zero maintenance fatigue and no mental overhead: No more volumes stuck in Degraded, operator reconciliations fighting over incompatible CRDs, or namespace bureaucracy. Cluster maintenance has been reduced to running apt upgrade nomad (or dropping in a static binary) and restarting a systemd unit. Zero Sunday firefighting sessions.

4. Deterministic deployments with 20-second feedback: From git push to containers running live on the desk takes under 30 seconds. If something fails (healthcheck, memory ceiling, syntax), GitHub Actions lights up red immediately with the exact log line, eliminating opaque reconciliation black boxes.

The Core Takeaway:
The most valuable lesson from this homelab journey wasn’t learning how to operate an increasingly complex machinery; it was knowing when to stop over-engineering. Infrastructure must serve you, not demand your free time. Travolta and Thurman have finally ceased being my second job and returned to what they were meant to be: the ideal playground for building software.