Bootstrapping a Fully Automated Homelab: Architecture and Overview

This is the first article in a series describing my current homelab setup including technical deep dives on related concepts. The next article is here.

Introduction

Given my career in software engineering and the personal programming projects that occupy the majority of my free time outside of working hours, one can imagine how quickly cloud hosting fees could add up. Several years ago, I set up a cluster of computers at home where I host my own services instead. Besides simply being a fun project, self hosting provides a level of freedom that I value.

As neat as that cluster setup was, I recently decided I wanted to refine it by automating as much of the deployment and management of the cluster as possible. My home lab is now virtually fully automated, easy to reproduce, and easy to extend later.

This article goes over the high level design of the cluster: what features I wanted and a high level description of the technologies I used to achieve those features. Articles later in the series provide technical overviews of each technology as well as specific technical deep dives about relevant concepts and their specific applications.

With these technical overviews, any reader with some programming experience should be able to understand how individual cluster pieces work together at a high level as well as some useful knowledge about each technology. If a reader comes with a specific topic of interest in mind(for example scanning, parsing, or how to write Ansible recipes), he/she should find useful specific information in one of the technical deep dives related to that topic.

Desired Features and End Goal

The end goal of this endeavor is to network together multiple(in my case three) mini PCs into a computational unit that allows me to self host any services I want, deploy any personal projects I want, and run VMs for specific projects. Each service must automatically back up its data and the configuration / provisioning of the nodes must be done in a reproducible, declarative way. The system should also be extensible; future computers should be easy to add in order to expand computing power.

To achieve the goal, I broke everything down into a list of individual features that are must-haves:

  • PCs must be relatively cheap(at a minimum, cheaper than cloud resources)
  • PCs must be able to coordinate and share resources
  • Data center must be able to handle both docker based services and individual virtual machines
  • VMs must be easily configured and provisioned in a repeatable way
  • Everything must be self hosted
  • There must be an easy way to set up CI/CD for all of the software I write and deploy
  • Deploying software to the actual cloud must be as easy and painless as deploying it to my local servers

Architecture and Design

In order to keep things cheap, I decided to use three refurbished mini PCs I got from Micro Center as well as an old laptop I had laying around. For networking, I ended up using one of the mini PCs as a router/gateway by running VyOS on it and connected the computers using ethernet wires and a dumb switch I got from Micro Center(more on VyOS as well as general networking in later articles).

In order to allow the PCs to coordinate and share resources, I decided to leverage Proxmox as a virtualization technology and create a data center out of the three PCs.

For allowing the data center the handle both docker based services and virtual machines, I decided to spin up three VMs which act as hosts for Kubernetes. Specifically, I chose k3s which will be described in detail in future articles. By running k3s inside VMs and not on bare metal, I can still spin up VMs inside the data center for doing things like experimenting with Guix etc.

In order to ensure that everything is easily configured and provisioned, I leveraged Terraform to spin up VMs and custom Ansible recipes to provision them.

For CI/CD and self hosting code as well as making sure everything is declarative and reproducible, I run one VM which holds a selfhosted Gitea instance. Inside this instance, Flux along with proper GitOps principles are used to ensure that the Kubernetes cluster is automatically managed and data is backed up. To keep things simple, I don’t use Helm but instead write Kustomizations myself.

In order to deploy applications I write to the cluster as well as to the cloud if necessary, I leverage Gitea actions and a Gitea runner.

For a high level visual description of what is described, see below: 960x540