MMilunaCloud
Back to all articles
KubernetesHomelabDevOpsInfrastructure

Homelab Adventures: When Your Cluster Becomes Your Second Job (Part 1)

Written by Miguel Ángel Luna
Published on

It all started with what seemed like a very reasonable premise: why keep paying for cloud instances and load balancers for side projects when I could run my own infrastructure right at home?

Enter Travolta and Thurman, two Intel N150 mini PCs equipped with 16 GB of RAM and 256 GB NVMe SSDs each. The plan on paper was rock-solid: spin up a lightweight K3s cluster using Ansible, master Kubernetes hands-on without the anxiety of a surprise three-digit cloud bill at the end of the month, and build the ultimate playground for my applications.

What I didn’t anticipate was how fast I’d stumble headfirst into the most notorious trap in self-hosting: The Homelabber Syndrome.

Technical infographic: The Homelabber Syndrome on a 2-node K3s cluster

1. The Golden Promise: Mastering Kubernetes Without Cloud Invoices

Building your own home cluster has a distinct thrill. You unbox the machines, run your Ansible playbooks, and hit Enter. Within minutes, automation had the nodes securely connected over a private Tailscale network, configured Ed25519 SSH keys, and tackled the first classic Linux-meets-Kubernetes hiccup: disabling UFW, whose default FORWARD DROP rule thoroughly broke the Flannel (VXLAN) overlay network and killed pod-to-pod DNS resolution.

The initial topology was straightforward: Travolta operating as the master (kept schedulable so no capacity was wasted) and Thurman as the worker.

I deliberately started with two nodes, leaving the door open to add a third node later specifically dedicated to local AI workloads and LLMs. Of course, in distributed systems, a two-node topology is tricky territory: the risk of split-brain is real whenever network partitions occur. Still, for a home lab, it felt like an acceptable trade-off given the sheer learning value:

  • Understanding the real mechanics: In managed cloud offerings like EKS or GKE, the control plane, networking, and certificate issuers are convenient black boxes. Here, you must understand every layer: CNI (Flannel), local load balancing with MetalLB or Klipper, and ingress routing via Traefik.
  • Zero financial anxiety: You can experiment with distributed architectures, intentionally crash nodes, and restart workloads without a cloud provider’s billing meter ticking every second.

With the cluster up and kubectl get nodes greeting me with Ready, it genuinely felt like holding a shiny treasure in my hands. And that’s precisely when the trouble started.

2. The Homelabber Trap: Deploying Absolutely Everything

Initial euphoria clouds engineering judgment. Seeing two energy-efficient, low-power nodes juggle pods effortlessly triggers that dangerous inner voice: “If we already have Kubernetes, why not run it like a Silicon Valley enterprise?”

Within two weeks, my innocent cluster was burdened with a full enterprise stack:

  1. Declarative GitOps with Flux CD and SOPS: Everything versioned in Git with secrets encrypted via Age/SOPS; no manual kubectl apply shortcuts.
  2. The database operator zoo: Not one, but two clustered databases managed by operators: CloudNativePG (CNPG) for PostgreSQL and the Percona Server for MongoDB Operator.
  3. Distributed block storage with Longhorn: An acrobatic feat on two nodes. I forced replica counts down to 2 (1 per node instead of the standard 3) and enabled thin provisioning at 200% (storageOverProvisioningPercentage: 200%) to stretch the 256 GB NVMe SSDs.
  4. Automated backups to Cloudflare R2: Encrypted periodic snapshots shipped directly to S3-compatible cloud storage.
  5. Tier-1 observability suite: I deployed the full modern Grafana stack: Grafana Alloy as a unified telemetry collector shipping metrics to Grafana Mimir (migrated from Thanos for 3-month long-term retention in R2) and logs to an HA Grafana Loki cluster (2 instances running a memberlist ring) also backed by R2.

On paper, the architecture was a work of art. In practice, I had turned two budget mini PCs into an overly complex miniature data center.

3. Day 2 Reality Check: A 7 GB Memory “Tax” and Vanishing Dev Time

The first reality check hit the moment I looked at telemetry: the base OS, K3s, and the observability stack (Mimir + Loki HA + Alloy + Grafana) were already chewing through more than 7 GB of RAM before I even deployed a single line of business code. Almost half of Travolta or Thurman’s physical memory had vanished into platform plumbing.

And the resource tax was only half the problem; the operational burden was the real killer. Evenings that were supposed to be spent coding APIs and shipping features were hijacked by:

  • Two-node storage panic: Thurman would reboot during an automated kernel patch, and Longhorn would immediately light up orange. With a replica factor of 2, losing one node instantly killed 50% of redundancy; the volume went into a degraded state with no third node available to re-replicate data.
  • Telemetry troubleshooting: Wrestling with memory limits and debugging why Loki compactors or Mimir pods spiked RAM usage during chunk flushes to Cloudflare R2.
  • Stuck StatefulSets: Battling pods stuck in Terminating when attempting to upgrade the K3s patch version.
  • Broken reconciliations: Resolving stalled Flux pipelines caused by Custom Resource Definition (CRD) schema collisions in the Percona or CNPG operators.

Managing stateful workloads forced an uncomfortable realization: does it really make sense to manage StatefulSets, distributed network storage, and clustered databases in Kubernetes for personal side projects?

My original goal was to have a platform to build and ship applications. Instead, I found myself spending my evenings after my 9-to-5 job doing unpaid sysadmin work, patching, and platform troubleshooting. I had accidentally built myself a second job.

Travolta and Thurman: two Firebat mini PCs running K3s on the desk Travolta and Thurman in their natural habitat: two innocent Firebat mini PCs stacked on the desk, stoically bearing the entire weight of enterprise infrastructure.

4. What’s Next? (Setting Priorities Straight)

Building this homelab wasn’t a mistake; it has easily been one of the best technical learning experiences I’ve had in months. But it urgently needs a heavy dose of pragmatism. Infrastructure must serve the software creating value, not the other way around.

In Part 2 of this series, I’ll share how I plan to radically simplify and streamline maintenance for Travolta and Thurman:

  • How to drastically cut down that 7 GB observability memory tax by stripping out oversized components.
  • Why moving database management and stateful workloads outside the cluster might just save your evenings.
  • The automation patterns needed so the cluster becomes an accelerator for development again rather than an operational burden after work.