This year marked another leap forward for my homelab, with milestones in automation, storage management, and energy efficiency.

Highlights from 2024

Relocating the Server Rack

One major change was moving the entire server rack into the garage. The main driver for this was noise. I had a loud power supply for the TrueNAS build and an IBM server that I wanted to use, but it was simply too loud inside.

Moving the rack to the garage was pretty straightforward. There was a weird problem with connectivity initially, but that sorted itself out.

However, moving into the garage has presented a couple of challenges: dust and heat. For now, it’s okay, but when we start getting 40-degree days, I think I’ll need to power things down.

Optimising Storage with Solar-Powered Media Transcoding

Running low on storage pushed me to implement media transcoding. I’ve installed a “GPU Server” to shrink file sizes and free up disk space.

To maximise efficiency, I integrated Tdarr with Home Assistant, which enables transcoding only when the sun is out. Home Assistant just stings the Tdarr API to toggle transcoding on and off based on my solar sensors.

Tdarr transcodes everything into H.265, which has saved me a significant 364 GB so far.

Kubernetes Upgrade to 1.31

Upgrading from Kubernetes 1.27 to 1.31 was another significant milestone. I’d been running a 1.27 cluster for a long time, and it was starting to worry me. Nothing was going wrong, but I like to keep things current. I thought this would be a nightmare, but once I did the first couple of upgrades, it was fine.

I ran into a couple of issues with versions of things not lining up such as Longhorn and nginx ingress. Some of these things were a pain to fix, especially because this cluster was originally provisioned with Kubespray.

I manually upgraded the first control plane node, then used Rundeck to semi-automate the node upgrade process.

Job Failure Notifications with Rundeck

I’d been manually checking Rundeck for job failures because I use Sensu for monitoring, and you need a paid version for a Rundeck integration.

I have an internal API to shuffle data between systems, so I extended it to allow Rundeck to hit an endpoint that formats things nicely and delivers the data to Sensu. This allows job failure notifications to end up in Sensu where all other “incidents” live. If the job comes back “OK” re-run, it pings Sensu to mark the incident as resolved.

Treating VMs as Cattle

The transition to treating VMs as cattle continues. I’ve strengthened the process by implementing fixes to make things easier to manage. If I deploy something new on a VM, it MUST be by Ansible. VMs are already managed by OpenTofu. I have a Rundeck job that pulls down the latest Debian image, then uses Packer to configure a base image for me. That way, new VMs will always be fresh.

Automated Sensu Configuration

I’ve streamlined Sensu agent configuration with an automated process that dynamically generates configs for each machine.

I’ve created a systemd service that runs at boot. It pings my internal API, which looks up the description field in Proxmox for that VM. The description on all VMs contains the Sensu subscriptions on one line:

sensuSubscriptions: linux, docker

The API responds with a Sensu agent YAML.

Coming up in 2025

2024 was huge, now that I think about it. I’ve got some further ideas for improvement in 2025.

Deploying HashiCorp Vault

Centralising secrets management is a key focus for next year. HashiCorp Vault will be deployed to securely handle sensitive data across Kubernetes clusters and VMs. This is a concept I’m aware of but have never implemented. A lot of my secrets are committed to internal Git repos… This will be a massive improvement.

Migrating CI/CD to Gitea Actions

I already use Gitea, so it makes sense to use their CI/CD offering. I’ll check it out, and if it looks good, I’ll make the switch. I’m actually enjoying Drone CI, but the lack of good documentation and adoption isn’t great.

Automated Ansible Inventory

Similar to how the Sensu config generation works, I want to build a dynamic Ansible inventory. I’m already doing this for Rundeck. At the moment, my Ansible inventories are a mess. I want to move this to a single dynamically generated inventory in 2025.

Cooling Solutions for the Garage

As mentioned above, I’m worried about the heat in the garage. I’ll explore cooling solutions so I can keep everything powered on during summer. Ideally, I want this to be fully automated. I’ll keep away from evaporative cooling due to moisture risks and manual intervention. I also don’t want to spend a fortune on the solution or power costs.

Conclusion

2024 has been a huge year for the homelab. From rack re-location and automation improvements to pushing Kubernetes upgrades, I’ve learned a lot and it’s been incredibly rewarding.

Looking ahead to 2025, there’s plenty already on the list like deploying Vault, simplifying Ansible inventories, and solving the garage heat problem, to name a few. In 2025 one of my goals is to post here more often, so stay tuned.