This is the second article in a series describing my current homelab setup including technical deep dives on related concepts. The previous article is here. The next article is here.
As part of the Desired Features and End Goal section of the first article in this series, I mentioned that a coupole of the primary goals for my home cluster are the following:
By leveraging the Proxmox Virtual Environment, all of these three goals are achieved. The mini computers can coordinate with each other through the Proxmox Data Center abstraction. Since Proxmox is a virtualization solution, it can handle virtual machines. Furthermore, by later deploying a Kubernetes cluster over some dedicated VMs, the data center can handle docker based services as well. Lastly, Proxmox provides a simple REST API for managing VMs which I take advantage of later, allowing for easy configuration and provisioning of VMs.
This article goes over what Proxmox is and provides a deeper insight into the reasons that led me to pick it as a solution. The last section also covers strategies for automating a Proxmox installation, but does not provide a detailed explanation of the strategy I used. A detailed technical deep dive on what I used can be found in the next few articles in the series.
Proxmox is a company that offers several useful pieces of software, but the one I am most concerned about in this article is Proxmox Virtual Environment(and this is what the word “proxmox” refers to henceforth, as well as PVE). According to the Promxox website, PVE is “complete, open-source server management platform for enterprise virtualization”. At a high level, this means that PVE is a piece of software that allows a user to create and manage virtual machines across physical hardware in a sane way while also providing desirable features like advanced networking, clustering, authentication, and a seamless REST API for management.
Through its data center abstraction, Proxmox also allows users to manage multiple nodes from a centralized interface while allowing physical nodes(in my case mini PCs) to coordinate and share resources. If networked properly, the VMs can talk to each other using standard Linux networking practices. Furthermore, if it is desired, there are options for high availability and things like VLAN support.
The main reasons I chose Proxmox as my virtualization technology are that it is open source, it is easy to work with(I am intimiately familiar with Linux and Proxmox is purely Debian based), and it flawlessly achieves all of the goals mentioned in the introduction.
By installing Proxmox on each one of the three physical nodes and leveraging the aforementioned data center abstraction, the mini PCs can talk to each other and share resources. Proxmox even provides seamless integrations with distributed file storage tools like Ceph as well as providing several useful higFurthermoreh availability features. I don’t use these at home, but they do make for compelling reasons to choose Proxmox in a professional setting as well.
Since Proxmox is a virtualization technology, I can run VMs to achieve whatever I need. For docker based services, I chose to set up three large VMs to act as nodes for a Kubernetes cluster. For general computing resources and experimentation, I run individual VMs. For example, I wanted to experiment with Guix without committing to installing it on a computer. I spun up a VM in my Proxmox cluster to do just that with no issue. If desired, Proxmox also provides an LXC interface which allows for Linux based containers to run natively. However, I like Docker so I end up using that instead.
In terms of ease of management and provisioning of VMs, Proxmox is also great. By exposing a REST API, Proxmox has allowed several tools to develop for management. I personally use Terraform with a community made provider, however, there are other tools such as proxmoxer which allow for management using Python. Furthermore, since the VMs are easy to network, Ansible is easy to use for provisioning.
As I mentioned before, Proxmox is a Debian based system. This means there are several ways to automate the installation process. Some people choose to use a Debian preseed file to automate the Debian install and then use Proxmox repositories to install Proxmox later. However, a perhaps lesser known approach is the Proxmox automated installaion. Using this approach revolves around configuring Proxmox answer.toml files and either serving them from an HTTP server or baking them directly into ISO images.
If the automated installation is used, the Proxmox project provides a helper tool named proxmox-auto-install-assistant that aids with validating the TOML contents as well creating the ISO. One quirk is that the tool itself only runs natively on Debian based systems as it requires the apt package manager to be installed while also having some Debian specific quirks. To get around this issue, one possible solution is to wrap the tool in a Docker image.
Wrapping the tool and providing extra functionality is exactly what I did, and the process by which I achieved that is the topic of the next few articles in this series.