Homelab 2023

2023-07-22

This is the first of an annual series of snapshots about the state of my homelab. Self-hosting is a hobby, and one that I find provides online privacy and freedom. Running your own infrastructure provides a space to test and deploy anything you want.

I have a few requirements for the homelab - mostly around security as well as availability for some components, like hosting this website and data backup. The rest is best effort. This is not the cheapest or most efficient setup, and it's not automated in almost any way. And while I do sometimes test tools and techniques that I might be interested in using in production, this isn't meant to be an example of that type of work.

It's a playground.

Hardware

  • MacBook Pro
  • Mac Studio
  • Synology 1821+
  • Synology 1522+
  • Digital Ocean VPS
  • Vultr VPS
  • iPhone

The primary server for my Homelab is a Mac Studio, running Asahi linux. Running a desktop Mac didn't catch on for me, but this makes a fantastic server. It's a 10-core ARM chip, with 32 GB of RAM, 10 Gb ethernet, and crazy-fast internal storage. It's silent and low-power, which are requirements for the homelab, especially as this is currently a small-apartment-lab.

Other "compute" nodes in the homelab are currently my MacBook Pro, and two Synologies. One NAS is local to the Mac Studio and one is offsite, used for data replication and also providing offsite compute as needed.

Networking

I use a combination of Nebula and Cloudflared tunnels to network my infrastructure in a secure way. The main goals I had for this are that I should not need any cooporation from routers or firewalls - parts of the homelab run behind routers that were previously configured and are working for family members. I'll assign static IPs if desirable, but no port forwarding, VPN configurations, or anything like that. I don't want to require any particular routing hardware or software. Second, no open ports on home internet connections, even for UDP, which rules out some raw WireGuard configurations.

Nebula's main purpose is to connect my Synology NAS boxes between houses for backup replication, and to connect my laptop and phone to the main compute node for ingress to services (see Envoy below). I run two lighthouses on VPCs - one in DigitalOcean and one in Vultr, in different metros. So far, this is my favorite mesh network product I've used and highly recommend it for self-hosting. It provides authentication and authorization via PKI certificates and has good tooling for generating a CA and signing certs. And once you have public lighthouses (these do have a single open UDP port, but otherwise even ssh is disabled - I can always enable it via the cloud dashboard if I need to do maintenance) - the rest is all handled by the clients. No need for any hosted databases or other single points of failure.

If you're curious about the type of problem this class of software solves, this article from Tailscale is a good primer.

All of the hardware listed above runs a Nebula client - even my iPhone, providing access to all the machines, and notably acting as the resolving IP address for a domain that I use for my private cloud - it resolves to the Nebula address of the main linux server which runs Envoy (see below), so from any client with Nebula I can access my self-hosted services via subdomains. Testing out Nebula's DNS support is on the TODO list.

Cloudflared tunnels provide a second, backup connectivity option for access to the various nodes. If I needed to do maintenance on Nebula for example, I can get into the network space of a node via Warp and the cloudflared tunnel also running on the node. I'd like to test out the Warp to Warp functionality as a backup for Nebula, but since the primary use of Nebula is shuttling gigabytes of backup data (specifially BTRFS snapshots), I'd rather do that directly between peers than through a proxy.

Storage

For storage I run a Synology 1821+, which is an 8-bay NAS, currently with about 16TB of usable capacity. Like the Mac Studio, this was not originally spec'd to be homelab storage; it's the anchor of my backup strategy. But, it works great for the task. I have a large share allocated just for media and storage that the homelab compute nodes can mount.

What's next: I added a 10Gbe network card to the Synology, and want to get that working with the Mac Studio's 10Gbe port in Asahi linux. I might experiment with NVMe storage pools, but that effort would likely be better spent learning FreeNAS, BTRFS directly, or other storage architectures (e.g. Rook/Ceph). Right now mounting the synology to a /mnt destination in Asahi linux is rock solid, but SMB over less reliable networking is not ideal. I want each Synology to provide the storage for its "site". This might be as a CSI driver, or selecting an object store to run (e.g. SeaweedFS) either directly on the Synology or on an attached compute node.

Network

The main network hardware feature of the homelab in 2023 is that I have a separate WAN for it. This started as a way to have WAN-failover, but I ended up using it to provide a completely isolated playground for my homelab without any worry of affecting the primary WAN which is critial as it's used for work-from-home access. The secondary WAN is a Verizon 5G Home Internet gateway, which I'm lucky enough to have access to in the homelab's current location. The everyday work-from-home WAN is a cable ISP. I call these the low-latency network (cable), used for general purpose and gaming workloads, and high-latency network (5G) used for the homelab. I can swap a cable to to utilize the 5G when the cable connection goes down, which inevitably happens a couple of times a year.

There are a surprising number of upsides to dual WANs for homelab use. I can easily test various tunneling protocols and implementations and see how they behave, test bandwidth expectations for accessing the homelab offsite, and generally break whatever I want without impact to others.

Besides a connection to the homelab WAN, the Mac Studio is additionally connected to our main router via WiFi. This connection to the main LAN allows for a sufficiently high-bandwidth connection between our Apple TV to stream 4K video from the homelab.

What's next: The home of the lab will be relocating in a couple of months, and will lose access to 5G options. An upgrade to fiber will also have to wait. Starlink is available, perhaps that will make an appearance? I should document as much performance information as I can from 5G before returning the equipment.

Software

All of the following programs run behind Envoy as the reverse proxy. Via Nebula networking, all these services are available to all other machines, regardless of location.

Envoy

I learned Envoy at work, where we run it in production. It's a sledgehammer for a nail in the homelab, but it's by far my preferred reverse proxy at this point, and there's actually a pretty simple way to run it. You can generate a static config, but have it load listeners (the thing that parses a request and figures out where to send it) and the clusters (the "upstream" service that listeners send requests to) in separate files that are loaded on startup.

What's next: There's no ACME client implementation in Envoy, so adding something to get signed TLS certicates would be nice. mkcert looks promising to get a cert installed into root stores of myriad machines, which is the real challenge.

Gitea

Works great as a git remote destination with a web interface. I haven't explored many of the features, but it's been solid.

Docker Registry

Gitea can also act as a container registry, but I ran into some issues with HTTP 206 Partial Content ranges when pulling images between machines over nebula with high latency, and the official registry image didn't seem to have the same issue with the default config. It's also nice to keep these separated. Any images I build and distribute across the homelab (like the one serving this website) are pushed to the registry, and can be pulled down by any homelab machine. Accessing an "external" registry from a client, even if it's over an overlay network in this case, requires TLS, so an Envoy listener fronts this service to terminate TLS. As long as the registry domain is added to the list of insecure registries on all client's docker config, a self-signed certificate is sufficient.

I'd also like to make this a pull-through cache, but so far haven't had time to debug some auth issues with this configuration. I'd ideally like to run a more custom registry with more options for behavior than just proxying Docker Hub.

Gluetun

Handy container for connecting to a WireGuard peer.

Prometheus and Grafana

Vanilla Prometheus and Grafana containerized installations. At the moment I'm only running cadvisor for basic metrics on the main server, and scraping Envoy which has a rich stats endpoint. The hard part is building out all the Grafana dashboards.

Next project for monitoring is to run some type of metrics generator on the Synologies that can be scraped by Prometheus.

Navidrome

Music server. This runs on the main server and mounts a share on the NAS with all my music. There's a web interface, but it also implements the Subsonic API, so I can run clients on my laptop and iPhone that use Navidrome as the backend. Currently using Sonixd on my Mac, and Substreamer on my iPhone.

Jellyfin

Video server. Similar to Navidrome, the API is the main feature. I run Infuse on my Mac and our Apple TV and can stream anything added to a share of the Synology. I'm not yet doing any DNS resolving locally on the main LAN, so for now I just allocate a static IP for the main server, and point Infuse to that.

netbootxyz

I want to be able to PXE boot machines and VMs from the homelab LAN and bootstrap into the Nebula overlay network. This does require some assistance from the DHCP server, so I need to come up with a good solution to that.

Miniflux

RSS reader with a nice web UI.

kerby_website

Origin for this website. It runs two containers - a Go web server and cloudflared that provides the networking for the server to reverse-proxy requests from Cloudflare's edge.

If I wasn't using cloudflared, I'd be using WireGuard to connect directly to VPS instances that I'd point to from the DNS records for my domain. I'd run Envoy on these to provide load balancing, circuit breakers, and access logging.

This currently runs on the Mac Studio, both Synologies, and my laptop. I can ensure at least one of these machines is available at any time to serve this site.