KerbyHughes.com

How this site is hosted

2023-06-29 14:30:00 -0400 EDT

How this site is hosted

2023-06-29

Update: This post has been moved to the Colophon, where it will be kept updated as changes to the site are made.

Homelab 2023

2023-07-22 20:00:00 -0400 EDT

Homelab 2023

2023-07-22

This is the first of an annual series of snapshots about the state of my homelab. Self-hosting is a hobby, and one that I find provides online privacy and freedom. Running your own infrastructure provides a space to test and deploy anything you want.

I have a few requirements for the homelab - mostly around security as well as availability for some components, like hosting this website and data backup. The rest is best effort. This is not the cheapest or most efficient setup, and it's not automated in almost any way. And while I do sometimes test tools and techniques that I might be interested in using in production, this isn't meant to be an example of that type of work.

It's a playground.

Hardware

MacBook Pro
Mac Studio
Synology 1821+
Synology 1522+
Digital Ocean VPS
Vultr VPS
iPhone

The primary server for my Homelab is a Mac Studio, running Asahi linux. Running a desktop Mac didn't catch on for me, but this makes a fantastic server. It's a 10-core ARM chip, with 32 GB of RAM, 10 Gb ethernet, and crazy-fast internal storage. It's silent and low-power, which are requirements for the homelab, especially as this is currently a small-apartment-lab.

Other "compute" nodes in the homelab are currently my MacBook Pro, and two Synologies. One NAS is local to the Mac Studio and one is offsite, used for data replication and also providing offsite compute as needed.

Networking

I use a combination of Nebula and Cloudflared tunnels to network my infrastructure in a secure way. The main goals I had for this are that I should not need any cooporation from routers or firewalls - parts of the homelab run behind routers that were previously configured and are working for family members. I'll assign static IPs if desirable, but no port forwarding, VPN configurations, or anything like that. I don't want to require any particular routing hardware or software. Second, no open ports on home internet connections, even for UDP, which rules out some raw WireGuard configurations.

Nebula's main purpose is to connect my Synology NAS boxes between houses for backup replication, and to connect my laptop and phone to the main compute node for ingress to services (see Envoy below). I run two lighthouses on VPCs - one in DigitalOcean and one in Vultr, in different metros. So far, this is my favorite mesh network product I've used and highly recommend it for self-hosting. It provides authentication and authorization via PKI certificates and has good tooling for generating a CA and signing certs. And once you have public lighthouses (these do have a single open UDP port, but otherwise even ssh is disabled - I can always enable it via the cloud dashboard if I need to do maintenance) - the rest is all handled by the clients. No need for any hosted databases or other single points of failure.

If you're curious about the type of problem this class of software solves, this article from Tailscale is a good primer.

All of the hardware listed above runs a Nebula client - even my iPhone, providing access to all the machines, and notably acting as the resolving IP address for a domain that I use for my private cloud - it resolves to the Nebula address of the main linux server which runs Envoy (see below), so from any client with Nebula I can access my self-hosted services via subdomains. Testing out Nebula's DNS support is on the TODO list.

Cloudflared tunnels provide a second, backup connectivity option for access to the various nodes. If I needed to do maintenance on Nebula for example, I can get into the network space of a node via Warp and the cloudflared tunnel also running on the node. I'd like to test out the Warp to Warp functionality as a backup for Nebula, but since the primary use of Nebula is shuttling gigabytes of backup data (specifially BTRFS snapshots), I'd rather do that directly between peers than through a proxy.

Storage

For storage I run a Synology 1821+, which is an 8-bay NAS, currently with about 16TB of usable capacity. Like the Mac Studio, this was not originally spec'd to be homelab storage; it's the anchor of my backup strategy. But, it works great for the task. I have a large share allocated just for media and storage that the homelab compute nodes can mount.

What's next: I added a 10Gbe network card to the Synology, and want to get that working with the Mac Studio's 10Gbe port in Asahi linux. I might experiment with NVMe storage pools, but that effort would likely be better spent learning FreeNAS, BTRFS directly, or other storage architectures (e.g. Rook/Ceph). Right now mounting the synology to a /mnt destination in Asahi linux is rock solid, but SMB over less reliable networking is not ideal. I want each Synology to provide the storage for its "site". This might be as a CSI driver, or selecting an object store to run (e.g. SeaweedFS) either directly on the Synology or on an attached compute node.

Network

The main network hardware feature of the homelab in 2023 is that I have a separate WAN for it. This started as a way to have WAN-failover, but I ended up using it to provide a completely isolated playground for my homelab without any worry of affecting the primary WAN which is critial as it's used for work-from-home access. The secondary WAN is a Verizon 5G Home Internet gateway, which I'm lucky enough to have access to in the homelab's current location. The everyday work-from-home WAN is a cable ISP. I call these the low-latency network (cable), used for general purpose and gaming workloads, and high-latency network (5G) used for the homelab. I can swap a cable to to utilize the 5G when the cable connection goes down, which inevitably happens a couple of times a year.

There are a surprising number of upsides to dual WANs for homelab use. I can easily test various tunneling protocols and implementations and see how they behave, test bandwidth expectations for accessing the homelab offsite, and generally break whatever I want without impact to others.

Besides a connection to the homelab WAN, the Mac Studio is additionally connected to our main router via WiFi. This connection to the main LAN allows for a sufficiently high-bandwidth connection between our Apple TV to stream 4K video from the homelab.

What's next: The home of the lab will be relocating in a couple of months, and will lose access to 5G options. An upgrade to fiber will also have to wait. Starlink is available, perhaps that will make an appearance? I should document as much performance information as I can from 5G before returning the equipment.

Software

All of the following programs run behind Envoy as the reverse proxy. Via Nebula networking, all these services are available to all other machines, regardless of location.

Envoy

I learned Envoy at work, where we run it in production. It's a sledgehammer for a nail in the homelab, but it's by far my preferred reverse proxy at this point, and there's actually a pretty simple way to run it. You can generate a static config, but have it load listeners (the thing that parses a request and figures out where to send it) and the clusters (the "upstream" service that listeners send requests to) in separate files that are loaded on startup.

What's next: There's no ACME client implementation in Envoy, so adding something to get signed TLS certicates would be nice. mkcert looks promising to get a cert installed into root stores of myriad machines, which is the real challenge.

Gitea

Works great as a git remote destination with a web interface. I haven't explored many of the features, but it's been solid.

Docker Registry

Gitea can also act as a container registry, but I ran into some issues with HTTP 206 Partial Content ranges when pulling images between machines over nebula with high latency, and the official registry image didn't seem to have the same issue with the default config. It's also nice to keep these separated. Any images I build and distribute across the homelab (like the one serving this website) are pushed to the registry, and can be pulled down by any homelab machine. Accessing an "external" registry from a client, even if it's over an overlay network in this case, requires TLS, so an Envoy listener fronts this service to terminate TLS. As long as the registry domain is added to the list of insecure registries on all client's docker config, a self-signed certificate is sufficient.

I'd also like to make this a pull-through cache, but so far haven't had time to debug some auth issues with this configuration. I'd ideally like to run a more custom registry with more options for behavior than just proxying Docker Hub.

Gluetun

Handy container for connecting to a WireGuard peer.

Prometheus and Grafana

Vanilla Prometheus and Grafana containerized installations. At the moment I'm only running cadvisor for basic metrics on the main server, and scraping Envoy which has a rich stats endpoint. The hard part is building out all the Grafana dashboards.

Next project for monitoring is to run some type of metrics generator on the Synologies that can be scraped by Prometheus.

Navidrome

Music server. This runs on the main server and mounts a share on the NAS with all my music. There's a web interface, but it also implements the Subsonic API, so I can run clients on my laptop and iPhone that use Navidrome as the backend. Currently using Sonixd on my Mac, and Substreamer on my iPhone.

Jellyfin

Video server. Similar to Navidrome, the API is the main feature. I run Infuse on my Mac and our Apple TV and can stream anything added to a share of the Synology. I'm not yet doing any DNS resolving locally on the main LAN, so for now I just allocate a static IP for the main server, and point Infuse to that.

netbootxyz

I want to be able to PXE boot machines and VMs from the homelab LAN and bootstrap into the Nebula overlay network. This does require some assistance from the DHCP server, so I need to come up with a good solution to that.

Miniflux

RSS reader with a nice web UI.

kerby_website

Origin for this website. It runs two containers - a Go web server and cloudflared that provides the networking for the server to reverse-proxy requests from Cloudflare's edge.

If I wasn't using cloudflared, I'd be using WireGuard to connect directly to VPS instances that I'd point to from the DNS records for my domain. I'd run Envoy on these to provide load balancing, circuit breakers, and access logging.

This currently runs on the Mac Studio, both Synologies, and my laptop. I can ensure at least one of these machines is available at any time to serve this site.

IPv6 and IPv4 Translation

2024-02-22 12:00:00 -0500 EST

IPv6 and IPv4 Translation

2024-02-22

This post is an overview of the technologies needed to convert between the IPv4 and IPv6 protocols and when they might be used. The original inspiration for this was building a Nebula mesh network to connect machines, some of which had dual-stack, some of which were IPv4-only, and some of which I was bringing up in a cloud environment with only IPv6 networking. Some hosting providers give a discount on VMs if you run IPv6-only, or charge additional fees to allocate IPv4 addresses. For reasons we'll see, trying to 1) run peer-to-peer software 2) on a linux host 3) with only IPv6 networking 4) connecting to an IPv4 host is a difficult scenario. There are a number of rabbit holes left unexplored, but this post will never see the light of day if I try to include all of them here, and I suggest reading through the linked RFCs.

Building a network

First, let's review what the Internet is, at least from layer 3 of the OSI model. We're not interested in the physical and data layers beneath us or the protocols running on top, we just need to build IP packets and route them between computers.

An entity wishing to provide service to the internet purchases some IP space from their regional registry [1], and becomes an Autonomous System, identified by an Autonomous System Number (ASN). For IPv4, that IP space is some portion of the 32-bit addressable space. You can divide up an address space by significant bits, so if you own a /16 prefix, everything that is a more-specific subset of those first 16 bits is within your space. For example, if it's 1984 and you're a university, you might own the 128.10.0.0/16 prefix. In order to join the Intenet, you peer with other Autonomous Systems near you, and announce your prefix over the (external) Border Gateway Protocol (BGP) which advertises routing information about your prefix to neighbors. All of these providers together form the Internet, and as a result we get get globally routable addresses. Does IPv6 change this? Not really, except with a 128-bit address space, we haven't allocated anywhere near all of the space,

Having an understanding of BGP advertisements is important for understanding what we can and can't do with only access to the code on a single machine within a network. We may have full control over the packets on any interface connected to the machine, but in most cases we need assistance from the access/transport network (the ISP) to advertise the address space we're using. Just because we can construct an IPv6 address with an embedded IPv4 address, it isn't routable without assistance from the network.

Getting an address

Before we can even send packets, we need to have an IP address ourselves if we want the response to our packets to get back to us. With IPv4, we usually get a single IPv4 address from our ISP, and then our router can multiplex private addresses over that single address to the internet -- a stateful NAT (Network Address Translation). A new device on your network talks to your router, and receives a private IPv4 address via DHCP request. The NAT table on the router is updated so that when the device reaches out to a public IPv4 address, it will rewrite the packet source as if it came from the public IPv4 address. All devices behind your NAT show up as coming from this public IPv4 address externally. As hinted at by private, the local IPv4 address is not globally routable. So if someone wants to reach out to a device behind your NAT, they have to use that public IPv4 address, and based on firewall rules, your router picks the private address to forward the packet to, usually based on the port (port-forwarding). IPv6 has ULAs, Unique Local Addresses, which are somewhat equivalent to IPv4 private address space, but they are not globally routable, and generally this isn't the method we would prefer for getting an IPv6 address.

IPv6 is meant to be globally routable, from one device to another. In that way, it's actually correcting what ended up not being possible over IPv4. Each device should be reachable on the internet via a unique address.

IPv6 is usually handed out from your ISP via DHCPv6 request, similar to how you get a single IPv4 address to NAT via DHCP. However with DHCPv6, your router can request not just a single address (which it would then have to NAT), but instead an entire prefix that the ISP is said to delegate to you. Prefix delegation means you'll usually receive a /56 or /60 and then your router can further delegate that prefix into smaller prefixes.

There are tricks you can employ any time you have a sufficiently large address space, one of which is that you can take a random value in that space, and have a high probability that there won't be any overlap the next time you take another random value. That's basically what SLAAC is doing for IPv6 -- instead of stateful DHCP assignment, SLAAC runs an algorithm to convert the MAC address of your interface into the interface part of an IPv6 address, or randomly pick an address. Both of these are then combined with Duplicate Address Detection -- a quick check that nothing else on the network is listening on the selected address. There are some additional protocols I don't know the details of where your device takes multiple IPv6 addresses, and may rotate the interface portion of the address periodically. All of this falls under autoconfiguration -- the ability of a device to get a routable address on its own with just the prefix information being provided, which is a benefit and simplification provided by adopting IPv6.

Carrier-grade NAT (CGNAT) is like NAT on your home router, except at the ISP level. A single IP address can be statefully NAT'd to many other IP addresses which the ISP hands out. Since these addresses behind the CGNAT can't be public, and can't be private, there's a range of "shared" address space that is used specifically for CGNAT: the 100.64.0.0/10 range. I actually use some of this space for a Nebula network, which hasn't been a problem for me because I'm not currently on an ISP that uses CGNAT so no routing is actually done with this address -- it's the internal overlay network only. I won't be using this for future deployments however because it's not the correct usage of this space.

NAT security

This warrants a quick note on NAT and Firewalls. The IPv4 address space is tiny, such that it's easy to enumerate and probe the entire space. As soon as you put a device on a public IPv4 address, it's going to get attacked by all sorts of things trying to log in. A common misconception is that because the devices behind your router only have private IP addresses, they are unreachable from the internet. It's hopefully true that they are mostly unreachable behind even consumer routers, but really you wouldn't say it's because of NAT -- your router is choosing what to do with incoming packets. It's not NAT that provides security (it could happily pass on all packets to any device as part of the NAT implementation, for instance), it's the fact that there's a stateful firewall also on your router deciding to drop any traffic it doesn't expect that provides the security. Another way to think about it is that it's not that devices are less reachable over IPv4, it's that reaching them requires more complexity (a stateful intermediary) and uses fixed resources (port numbers) to identify machines within a single address. We can have the same security without this complexity with IPv6.

And devices — usually IoT devices — might be breaching this contract without you knowing. Through protocols like UPnP, the device itself can configure the router to port-forward traffic to it, making it globally accessible, and now access to your private network is only as secure as the IoT device (the "s" in IoT stands for security). This one of many reasons these devices should be cordoned to their own vlan or similarly isolated.

Translation

IPv4-only networks are unfortunately still common because they are legacy and exist until replaced. IPv6-only is where we'd like to get to, and if you read the RFCs linked in this post, it's interesting how all of the technologies are presented as temporary measures until IPv6 replaces IPv4. A transition is the goal, even though the speed may make it feel like we've reached a steady state. For now, it usually makes sense to run dual-stack (IPv4 and IPv6) networks for broad compatibility. But there are cases where we might want to use the solutions afforded by IPv6, and then deploy the transition technologies for legacy IPv4 compatibility.

We might have more devices than available IPv4 addresses and want them all to be uniquely routable. This might be because we're running on a cellular network that only provides IPv6 connectivity because there aren't enough IPv4 addresses for every mobile device. This is common in many parts of the world, or with sub-providers like T-Mobile. It could also be something like a Kubernetes cluster where we have thousands of ephemeral pods that we want to make addressable without NAT. We may want to deploy IPv6-only internally for routing simplicity (it really is faster!), but devices will need to connect to some external hosts that are IPv4-only. We may want to take advantage of IPv6 auto configuration.

And in the other direction, while our network may only provide IPv4, we may want to connect to devices running on IPv6-only networks despite not being able to upgrade the network capabilities. To come up with solutions, we need to figure out what we can control -- is it just the program itself, the host, services external to the network that we can configure to assist our program or host? Depending on the translation direction and the amount of control we have end-to-end, let's see how different types of translations are implemented.

IPv4 and IPv6 are incompatible, meaning that if I only have IPv6 networking, I can't reach a device that only has an IPv4 address. IPv6 is not just a bit extension of IPv4, they are entirely separate protocols.

If we only have IPv6 connectivity, we'll need a translator if we want to talk to a device that is only connected via IPv4.
If we only have IPv4 connectivity, we'll need a translator if we want to talk to a device that is only connected via IPv6.

6 to 4

As an example, let's try to connect to an IPv4-only hostname from a device that only has IPv6 connectivity (a Vultr virtual machine with no IPv4 allocation).

github.com is IPv4-only [3], no IPv6 DNS records are returned.

kerby@vultr-worker-ipv6:~# dig +short github.com A 140.82.114.4 kerby@vultr-worker-ipv6:~# dig +short github.com AAAA kerby@vultr-worker-ipv6:~#

Here's what happens if we try to connect with our device that only has IPv6:

kerby@vultr-worker-ipv6:~# curl -vvv github.com * Trying 140.82.113.3:80... * Immediate connect fail for 140.82.113.3: Network is unreachable

In order for us to reach github.com, we need to first send a packet to something with an IPv6 address -- because that's all our network speaks -- and have it converted to an IPv4 address on some device that can speak both IPv4 and IPv6 (a translator).

IPv6 is big enough to contain the entire IPv4 space in the remaining bits of a /96 prefix! We can send the IPv4 request to a translator using a well-known prefix. RFC 8215 defines this as 64:ff9b::/96, and as long as a translator in our access network advertises this prefix, our packet will get to it. Then, the translator will remove the well-known prefix and generate an IPv4 packet that can be routed to Github and back. When it gets back, it will send it to the IPv6 source address of our device. If this sounds a lot like the description of a NAT above, that's accurate, it's a stateful NAT -- it will usually use ports to track mappings between devices on both sides of the translator.

There is a protocol for doing stateless 1-1 translation, but this would require a different IPv4 address for every device and does not scale. As far as I know all translators like this in practice are doing stateful NAT, utilizing ports and a mapping table to uniquely identify traffic and route it to the same address.

So that covers the case where our program knows it needs to use a translator and knows to use the well-known prefix. But what if that's not the case? How can the program know when it needs to use a translator? And what if the program can't use a translator, is there a way that the host it's running on can detect the case of needing a translator and handle it for the program?

If the program can speak IPv6, we can handle part of this with DNS. A standard DNS implementation will return A and AAAA (pronounced quad-A) records that exist for a host. But if it's deployed in an IPv6-only network, we know A records won't do the requester any good.

DNS64 and NAT64

DNS64 works by exploiting what we discussed earlier about the massive disparity between IPv4 and IPv6 address space. There's plenty of empty space in IPv6, so a /96 prefix, 64:ff9b::/96, is used for this purpose only. First, we connect to the DNS64 resolver via its IPv6 address. Cloudflare, Google, and others run public DNS64 resolvers. If the hostname (e.g. kerbyhughes.com) has AAAA records, those are returned directly because you can then use IPv6 the entire way. But if the hostname (like github.com) returns only A records, then DNS64 returns a synthesized AAAA record by taking the 64:ff9b::/96 prefix and appending the IPv4 address (hex encoded). The host just routes this like any other IPv6 destination to its interface, and then when it gets to the ISP's NAT64, it sees the well-known prefix and knows to use the last 32 bits as the destination in a new IPv4 packet. It maintains a translation table to do the reverse for the response (stateful NAT64). This works without the program knowing that the entire flow wasn't IPv6.

RFC 6146: Stateful NAT64: Network Address and Protocol Translation from IPv6 Clients to IPv4 Servers. 1. Introduction

This document specifies stateful NAT64, a mechanism for IPv4-IPv6 transition and IPv4-IPv6 coexistence. Together with DNS64 [RFC6147], these two mechanisms allow an IPv6-only client to initiate communications to an IPv4-only server. They also enable peer-to-peer communication between an IPv4 and an IPv6 node, where the communication can be initiated when either end uses existing, NAT- traversal, peer-to-peer communication techniques, such as Interactive Connectivity Establishment (ICE) [RFC5245]. Stateful NAT64 also supports IPv4-initiated communications to a subset of the IPv6 hosts through statically configured bindings in the stateful NAT64.

The problem for my IPv6-only linux VM is that while I can install something like clatd or install unbound and point it to a DNS64 resolver, there's no dual-stack NAT64 resolver already in Vultr's network. So we have to provide that piece ourselves. We can actually use any IPv6 prefix. We can use the prefix of an external NAT64 gateway that is both the route to the gateway and the prefix that will be removed. Then we just route the packet out to the returned IPv6 address, Vultr passes it off to eventually get to the prefix owned by the NAT64 resolver, and that takes the embedded IPv4 address and routes it to the final destination. It's able to get back because the source address gets set to the IPv6 host that Vultr advertises.

https://nat64.xyz maintains a list of public NAT64 services. We can see that they provide their IPv6 DNS64 resolver, and then a routable /96 prefix that is used instead of that synthetic /96 that would require an in-network translator.

Again though, host and/or access network needed configuration. We had to have a DNS64 resolver configured in the DNS resolution path that knew about a NAT64 device. On an enterprise network, or your home network, where this can be configured for all devices, that might be sufficient. However a non-configurable IPv6-only network is common out in the wild, for instance a phone on some cellular networks, or a laptop on a hotspot that only provides IPv6. To solve this, platform providers like Apple and Google have equipped their hosts with client-side translators to do this first part of the translation. This is referred to as the customer-side translator, or CLAT.

There are a couple of ways a host can determine if it's in an IPv6-only situation. One is that when it first connects to the network, the network can tell it that IPv4 is not available. Most modern operating systems like macOS/iOS [4] expect this situation where they may be on an IPv6-only host, so they bundle a CLAT implementation. This is activated by DHCP option 108 which instructs the client that there is no IPv4 networking available. Similar to tunneling tools, the CLAT sets up a gateway on the host for IPv4 destinations and translates to an IPv6 packet that goes to the router set up via the DHCP configuration.

RFC 8925: IPv6-Only Preferred Option for DHCPv4: Section 5

This document specifies a DHCPv4 option to indicate that a host supports an IPv6-only mode and is willing to forgo obtaining an IPv4 address if the network provides IPv6 connectivity.

...

The IANA has assigned a new DHCPv4 option code for the IPv6-Only Preferred option from the "BOOTP Vendor Extensions and DHCP Options" registry, located at https://www.iana.org/assignments/bootp-dhcp- parameters/.
Tag:  108
Name:  IPv6-Only Preferred
Data Length:  4
Meaning:  Number of seconds that DHCPv4 should be disabled
Reference:  RFC 8925

There seems to basically be one Tweet with a screenshot of macOS in this setup that is authoritative -- it's the single citation on Wikipedia. So I took the liberty of recreating the setup on one of my own networks via OPNSense.

Setting option 108 to disable IPv4 in the DHCP(v4) settings:

Configuring Tayga [5] -- a NAT64 implementation -- to use the well-known /96 prefix:

Here's the ethernet interface on my laptop after the DHCP request.

en0: flags=88e3 mtu 1500 options=6463 ether 5c:e9:1e:69:30:04 inet6 fe80::40c:5be1:1a3b:5137%en0 prefixlen 64 secured scopeid 0xf inet6 2603:7081:702:2860:cad:9c33:471f:81ab prefixlen 64 autoconf secured inet6 2603:7081:702:2860:c1d0:7356:827:53bb prefixlen 64 autoconf temporary inet 192.0.0.2 netmask 0xffffffff broadcast 192.0.0.2 inet6 2603:7081:702:2860:8f9:eaf5:1d7:b1e7 prefixlen 64 clat46 👈 nat64 prefix 64:ff9b:: prefixlen 96 👈 nd6 options=201 media: autoselect status: active

With unbound configured for NAT64, our synthesized A record is the well-known IPv6 prefix plus a hex-encoded version of the IPv4 A record answer.

kerby@tycho % dig +short github.com 140.82.113.3 kerby@tycho % dig +short github.com AAAA 64:ff9b::8c52:7103

Another way for clients to detect IPv6-only connectivity is by using the special domain ipv4only.arpa. ARPA is a special Top-Level Domain that can be used for these purposes. Devices make a request to determine if DNS64 is available in the network.

RFC 7050: Discovery of the IPv6 Prefix Used for IPv6 Address Synthesis

A node requiring information about the presence (or absence) of NAT64, and one or more Pref64::/n used for protocol translation SHALL send a DNS query for AAAA resource records of the Well-Known IPv4-only Name (WKN) "ipv4only.arpa.". The node MAY perform the DNS query in both IPv6-only and dual-stack access networks. ... A DNS reply with one or more AAAA resource records indicates that the access network is utilizing IPv6 address synthesis

With DNS being provided by unbound on my router:

kerby@tycho % dig ipv4only.arpa +short AAAA kerby@tycho %

Here I've told my laptop to use the nameservers provided by nat64.net.

kerby@tycho % dig ipv4only.arpa +short AAAA 2a01:4f8:c2c:123f:64:5:c000:ab 2a00:1098:2c::5:c000:ab 2a00:1098:2b::1:c000:aa 2a00:1098:2c::5:c000:aa 2a00:1098:2b::1:c000:ab 2a01:4f8:c2c:123f:64:5:c000:aa

There are a bunch of rules about how the client should use these addresses, in which order, etc.

kerby@tycho % dig github.com +short AAAA 2a00:1098:2b::1:8c52:7903 2a01:4f8:c2c:123f:64:5:8c52:7903 2a00:1098:2c::5:8c52:7903

As we know though, this is only half of the equation. The CLAT translates to IPv6, but this relies on a translator to listen for that IPv6 packet and translate it to IPv4. Therefore networks that set DHCP option 108 must also provide the translator. This translator is considered to be on the provider side, or PLAT.

These two things together, CLAT and PLAT, are referred to as a system as 464XLAT.

464XLAT

RFC 6887: 464XLAT: Combination of Stateful and Stateless Translation

1. Introduction

This document describes an IPv4-over-IPv6 solution as one of the techniques for IPv4 service extension and encouragement of IPv6 deployment. 464XLAT is not a one-for-one replacement of full IPv4 functionality. The 464XLAT architecture only supports IPv4 in the client-server model, where the server has a global IPv4 address. This means it is not fit for IPv4 peer-to-peer communication or inbound IPv4 connections. 464XLAT builds on IPv6 transport and includes full any-to-any IPv6 communication.

...

2. Terminology

PLAT: PLAT is provider-side translator (XLAT) that complies with [RFC6146]. It translates N:1 global IPv6 addresses to global IPv4 addresses, and vice versa.

CLAT: CLAT is customer-side translator (XLAT) that complies with [RFC6145]. It algorithmically translates 1:1 private IPv4 addresses to global IPv6 addresses, and vice versa. The CLAT function is applicable to a router or an end-node such as a mobile phone.

As we can see from the RFC, there are some limitations, which make sense now that we know the parts that are required to make it work. It's not peer-to-peer because the PLAT has to NAT the traffic and is therefore an intermediary -- all of the traffic goes through this translator (though only routing is required, no packet inspection, which as we'll see next is another benefit of being IPv6 by default). No direct connection is possible. And inbound IPv4 isn't possible because there is no IPv4 address on one of the peers; the network told the client it couldn't give it an IPv4 address over DHCP.

4 to 6

What about the other direction? What if the program/host are in an IPv4-only network, but need to communicate with IPv6-only peers? IPv6-only destinations in practice are still rare, because all of these things we're talking about show how IPv4 hasn't gone away at all. So generally IPv6 implementations are additive, especially on origins for things like a website, where we have a lot of control over network and infrastructure choices. Generally a website origin is dual-stack. However, for peer-to-peer this isn't the case -- we totally might want to connect to a single device that only has an IPv6 address, and to get further along in the transition to IPv6, the best case is to use IPv6 by default and treat this specific case -- a device on an IPv4 network -- as the exceptional case that we patch over until it can go away.

In the case of an IPv4-only host, all we can do is add a proxy somewhere that translates IPv4 to IPv6. We're out of tricks, because there's no way to represent a 128-bit IPv6 address inside a 32-bit IPv4 address. So unlike DNS64 and NAT64, we can't route these packets using just the addresses. If we can only use an IPv4 address for routing, we have to smuggle the 128 bits that represent where we actually want the packet to end up somewhere else in our packet. And that means we need a new protocol running on our translator that knows how to extract those bits from the data portion of the packet and build a new packet with an IPv6 header using those bits. And do the reverse when it gets a response.

In terms of implementation, this is actually a pretty easy case, but there's no way to configure it all automatically. Most tunneling software fits in the category -- the tunnel software, like cloudflared or Nebula, configures tunnel interfaces for both IPv4 and IPv6 on the host, making it appear dual-stack to programs. Then when an IPv6 packet goes into the tunnel, it forwards it over IPv4 to a dual-stack host on the other end of the tunnel, which understands the tunnel protocol and can therefore extract the IPv6 address that the tunnel software put into the packet.

And if we don't want to run the tunnel software on every host, we can do it at the edge of the IPv4 network -- for example a router that is doing NAT for all the devices behind it can tunnel to a translator and convert packets for everyone.

These tunnels by themselves are just providing one-way connectivity though. If all we have is the tunnel to a dual-stack host, we're in a similar situation to our 464XLAT system above where other client's can't reach us, because they have no way to know that we're behind the translator. The translator can't advertise our IPv4 address, because to the translator that IPv4 address is just the address our ISP handed us -- they are already advertising it, the translator can't also advertise it (well, they could, but things break badly when you do this over BGP so providers have agreements to try to prevent hijacking). Instead, for the case of something like cloudflared, we can introduce DNS records that point a hostname to our particular tunnel. The hostname resolves to A and AAAA records, and Cloudflare's system makes sure that those addresses get routed to the correct tunnel (via the hostname that was requested). We can do something similar with our Nebula network by publicly advertising the lighthouse address. Interestingly, my understanding is there were some attempts to build a more generic system that works similarly to help with the IPv4 to IPv6 transition.

Teredo

The Teredo protocol defines a way to connect peers with mixed IPv4 and IPv6 capabilities by defining two additional types of systems that operators run as general infrastructure -- servers and relays (Nebula uses the term relay for this as well). Relays are just like the cloudflared translator -- they are reachable at a known IPv4 address which either the operators of the host or the network configure some or all IPv4 traffic to go over. Once at the dual-stack relay, the relay connects to a Teredo server which is reachable at a well-known, globally-routable IPv6 prefix. My understanding is that this routable IPv6 prefix is meant to be advertised as general infrastructure. Then practitioners can implement their own relays that use the shared servers. The traffic goes from the relay, to the server, and the server's job is not to forward any traffic but to inform the relays (and IPv6-native peers) of each peer's globally routable IPv6 Teredo address. We're back 128-bit tricks because the Teredo protocol embeds lots of information into the IPv6 address it assigns to the Teredo peer. In this way, the peers can check in with the server (possibly via a relay) and then connect directly.

The most well-known use of Teredo to my knowledge was the XBox network. Due to the number of devices, IPv6 was the best way to uniquely identify each XBox, but you have to make it work over many IPv4-only home networks. And it has to use peer-to-peer semantics. So by utilizing Teredo servers and running Xbox-specific relay servers that the Xbox can be hardcoded to reach out to if it only has IPv4 connectivity, you can connect every XBox over unique IPv6 addresses.

Nebula

And this out-of-network setup is essentially what we can set up with Nebula. Our lighthouses can be configured as relays for a node, and then as long as you can reach the lighthouse, the lighthouse will, if necessary, convert between IPv4 and IPv6 internally and translate over the appropriate interface that can reach the target. For hosts that don't share the same IP protocol, it removes the peer-to-peer aspect which may be important especially if you're sending a lot of bandwidth one way or the other (for example, backups between the two Nebula clients counting against traffic bandwidth for the relay), but does facilitate connectivity which may be more important.

Without a relay, connecting to an IPv6-only client (100.100.0.50) fails:

root@vultr-worker-ipv4:~# ping 100.100.0.50 PING 100.100.0.50 (100.100.0.50) 56(84) bytes of data. ^C --- 100.100.0.50 ping statistics --- 16 packets transmitted, 0 received, 100% packet loss, time 15283ms

And connecting to an IPv4-only host (100.100.0.51) from an IPv6-only host (vultr-worker-ipv6) fails:

root@vultr-worker-ipv6:~# ping 100.100.0.51 PING 100.100.0.51 (100.100.0.51) 56(84) bytes of data. ^C --- 100.100.0.51 ping statistics --- 7 packets transmitted, 0 received, 100% packet loss, time 6141ms

These are the configuration settings needed to utilize a Nebula lighthouse as a relay:

Lighthouses:

relays: am_relay:true ... listen: # To listen on both any ipv4 and ipv6 use "[::]" host: "[::]" ...

Clients:

relay: relays: - ${lighthouse-ip} - ${lighthouse-ip} ... listen: # To listen on both any ipv4 and ipv6 use "[::]" host: "[::]" ...

With a lighthouse acting as a relay from IPv4 (vultr-worker-ipv4) to IPv6 (100.100.0.50), we have connectivity:

root@vultr-worker-ipv4:~# ping 100.100.0.50 PING 100.100.0.50 (100.100.0.50) 56(84) bytes of data. 64 bytes from 100.100.0.50: icmp_seq=1 ttl=64 time=6.86 ms 64 bytes from 100.100.0.50: icmp_seq=2 ttl=64 time=6.84 ms 64 bytes from 100.100.0.50: icmp_seq=3 ttl=64 time=6.86 ms 64 bytes from 100.100.0.50: icmp_seq=4 ttl=64 time=6.83 ms ^C --- 100.100.0.50 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms

And with a lighthouse acting as a relay from IPv6 (vultr-worker-ipv6) to IPv4 (100.100.0.51) we have connectivity:

root@vultr-worker-ipv6:~# ping 100.100.0.51 PING 100.100.0.51 (100.100.0.51) 56(84) bytes of data. 64 bytes from 100.100.0.51: icmp_seq=1 ttl=64 time=6.97 ms 64 bytes from 100.100.0.51: icmp_seq=2 ttl=64 time=6.80 ms 64 bytes from 100.100.0.51: icmp_seq=3 ttl=64 time=6.49 ms 64 bytes from 100.100.0.51: icmp_seq=4 ttl=64 time=23.4 ms 64 bytes from 100.100.0.51: icmp_seq=5 ttl=64 time=14.8 ms ^C --- 100.100.0.51 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms

Can anyone purchase an IPv6 block? ARIN has some requirements, like needing to peer with two ISPs, so you probably don't qualify and you probably don't want to pay for that anyway. And that's not scaleable for everyone to do, or practical for provisioning all the devices in the world. ↩
At this point, Github is providing a public service for every blog post and tutorial that uses it as a non-IPv6-enabled hostname. Think of the blog posts it will break once it's dual stack! ↩
On iOS, apps are required to have IPv6 connectivity. iOS handles this by providing APIs that can do the CLAT translation, like NSURLSession. This is just the stateless CLAT part of the translation and requires a PLAT on the network. [v6ops] Apple and IPv6, a few clarifications ↩
A future project is going to be an IPv6-only homelab. I'm not quite there yet for all devices, but this is a good start at understanding how to transition legacy devices. Tayga, a NAT64 implementation available as a plugin for OPNSense, should be a good part of that setup if I want, but I had some issues getting the gateways configured for it, so I will have to try again to get that working.↩

Homelab 2024

2025-01-25 14:00:00 -0500 EST

Homelab 2024

2025-01-25

This is the (belated) second post in a series of snapshots about the state of my homelab.

The homelab actually made two transitions between this post and the introductory post last year. The first was to a new apartment, where the main change was to my networking stack and some compute experimentation. The second was to our first house!

Homelab or Homeprod?

I still stand by the homelab as a playground and testbed, however the house needs a backbone of solid internet connectivity, and this overlaps with the needs of regular, non-lab computing in our home. While I previously used a 5G gateway as a failover WAN, the move to a new location without 5G Ultrawide coverage removed that possibility. Altough I no longer have a backup internet connection, I upgraded my networking stack to be able to multiplex the common household needs with my homelab needs over the same internet connection, such that I could get the isolation benefits without just routing some devices over a separate WAN. This was done with VLAN segmentation, and because I can tag Wi-Fi networks with a VLAN, and because Wi-Fi devices will pick up the same SSID regardless of the hardware or any other implementation details, I was able to swap out for a new network stack without requiring any changes to any of the other clients in the house. So while I think the next year will focus more on building tooling that is used by more than just myself in the household, this year was a good exercise in supporting the infrastructure needs of all our work-from-home equipment and being able to make changes with as little disruption as possible. This is really more of a home networking infrastructure post.

Apartmentlab

The main focus of the apartment lab in the last year was setting up OPNsense as my router and firewall. I've enjoyed having full control over my router, and the main motivation was to get VLAN support. In order to not interfere with other devices in the house, I set it up to have one Wi-Fi SSD per VLAN. The "commons" is the 1-1 replacement for our old Wi-Fi network, with the same SSID so all devices migrated seamlessly. Then I have one network that I use to connect to the router and management interfaces. Then a separate Homelab VLAN for all of my equipment. And another for my work laptop and phone so they are cordoned from the rest of the house as I don't own or manage those. I also created an IOT network without any internet connectivity. So far I haven't done much with that VLAN except test some smart outlets.

I had some success with tethering my iPhone to my OPNsense router and using that as a secondary WAN Gateway for failover in the event the internet went out, but it was pretty rough and only worked sometimes after a few reboots. I still like this idea and might circle back to it.

Proxmox

I collected a set of 4 Beelink EQ12 boxes slowly as they went on sale. These have Intel N100 processors, which means they have 4 of Intel's "E" cores (for efficiency, as apposed to "P" for performance). They also have dual 2.5 Gb ethernet, which is why I chose this model. They work great as an OPNsense router, and have room for an internal NVME SSD as well as an internal 2.5" SATA SSD. This made them a good choice for building out a cheap experimental cluster.

I set up a Proxmox cluster on the remaining 3 nodes, including adding a SATA SSD in each one to use for a Ceph storage cluster. The benefit of a Proxmox cluster is you can manage all of the nodes in the cluster from a single administration web UI, create VMs (or LXC containers) on any of the nodes, and even migrate the VMs between nodes.

I was even able to get OPNsense virtualized and running on the Proxmox cluster, as these have two ethernet ports. This was done by connecting one port of each node to a layer 2 switch, which was connected to my ISP modem. Packet routing is done by MAC address, and the VM retains the same MAC address for the ethernet interface when moving across nodes. The other ethernet port on each node was used for the LAN side, connected to a layer 3 switch. Surprisingly all that was required to migrate from bare metal to virtualized Proxmox with the same config was to change the WAN and LAN interface names. I exported the config, did a find-and-replace of the interface names, and restored the virtualized OPNsense install from the config, and everything worked.

Before adding Ceph, the only storage available was the SSD on each node. This meant that migrating from one node to another required downtime. Basically Proxmox sets up a schedule to sync the VM's on-disk data to the other nodes. Then when you migrate, the machine can come up in the same state. However, if you have shared storage, where no external sync is required, then you can do live migrations, which means that Proxmox will copy the memory from the source node to the destination node, and then can pause and resume the VM quickly to migrate the VM without downtime. From the VM's perspective, nothing changed and it keeps running.

This even worked for OPNsense - I was able to migrate the router between nodes to do maintenance or rewiring. While this let me do some maintenance without interrupting our internet connectivity, it did require triggering the migration manually - if a node was disconnected or otherwise died, it turns out that Proxmox has some hardcoded internal timers and configuration that means it takes about 5 minutes for a node to be detected as down and trigger a failover migration to a healthy node. This is okay for some workloads, but means 5 minutes of internet downtime. It's certainly faster than I could restore OPNsense to new hardware on my own if a node dies, but was a bit disappointing from a high-availability perspective given that there's no technical limitation that I'm aware of except for Proxmox's lack of available tuning.

Migrating a VM:

Grafana

I set up a Prometheus instance running on my NAS, and collected metrics from OPNsense, and used Grafana to visualize the bandwidth across the different VLANs.

Speed test:

Backups from the NAS uploading overnight:

Work meetings during the day:

netboot.xyz

I successfully got netboot.xyz working in tandem with OPNsense, which points DHCP clients to the right files for PXE booting. I can bring up any VM or physical machine connected to the homelab network with a remote image. Netboot.xyz doesn't seem very compatible with custom images, but for basics like Ubuntu or Debian it's been nice to have.

Network-Attached Storage

I need to do a "how I backup" post separately, but not much changed regarding how I use the storage on these devices. I still run Nebula to connect two Synology's, one at home and one at my parent's house, and use the snapshot features to sync backups in both directions.

I did do some upgrades on my 1821+, adding two NVMe SSDs and increasing the memory to 32GB. The Virtual Machine Manager app is pretty great, and I've been running a VM for monitoring (running Prometheus + Grafana), and another for compute jobs that are best with direct access to the storage pools. These are Jellyfin for video and Navidrome for audio, although I really haven't had time to do much lately with these.

I did have one scare with my Synology. We lost power at our apartment, which was not uncommon, so I had a large UPS attached. The Synology was configured to shut down after 5 minutes on UPS power so that the battery can be used for powering the networking equipment instead of async tasks like backup replication. However, I found that once power was restored, the Synology did not come up cleanly. I was unable to log in, with the UI giving an error that it was waiting for services to start. As I had SSH access I was able to look up some commands to bypass this, but then found that a couple of major services, including the Virtual Machine Manager and Hyper Backup (which does snapshot replication) were down. Rebooting put it back into the same loop. Luckily I knew I had plenty of copies of this data, so I started one of the "restore" jobs that was supposed to reinstall the OS layer that Synology adds while retaining the storage pools. This seemed to pretty quickly go into a failed state and I figured I would be wiping the system, however a reboot after this brought the system up cleanly and it has been working ever since. None of this was confidence-inspiring, but I'll monitor and see how it handles future events.

Hosting this site

I decided to move this site onto Cloud VMs. I've got one running in DigitalOcean, and another running in Vultr, just load balanced by putting both IPs in the DNS records for the site. It was cool to have it served entirely from my homelab and was a fun exercise in keeping the site available through upgrades and hardware moves. However, I had two reasons for moving it to the cloud. One was that I wanted it to not have to be fronted by anything and control the entire flow as part of being a blog on the open web. If I have Denial of Service (DoS/DDoS) issues I can always throw the domain behind Cloudflare temporarily, but not having to do this and letting the server deal with the flood of traffic being on the open web is a good exercise. I also had some issues with the cloudflared tunnel dropping out and requiring a restart despite saying it was healthy, so there were some availability issues. I haven't hand any availability issues since hosting directly from VMs in the cloud. As they are just Debian boxes and I'm running a light static binary, I can move to any hosting platform with minimal requirements.

Homelab

When you can put holes in the walls you can really make the changes you want to see. I wanted the rack of networking equipment to live in the basement so it was out of sight and earshot. Putting the rack in the center of the basement up high worked well, and I was able to fish five runs of CAT6 up from the basement to our upstairs offices. One run was for the access point - up high worked best for its broadcast pattern and made it close to our offices for Wi-Fi 6E coverage. The other four runs I terminated in a wall panel.

Fishing CAT6 up through the walls.

We replaced the flooring on the landing of our second floor, and removed an electric baseboard heater in the process. Most of the holes in the studs were already there, and luckily one of them worked for reaching down to the basement, two floor up but directly above the rack. This required a borescope in addition to wire fishing tape in order to route the cable down through the floor and wall cavities. I put metal plates between the wires and the drywall to protect them before sealing it up.

Before:

After:

The backside, where the wires come in to a closet in the office:

I forgot to take a good finished picture, but the wires are routed around the inside of the closet door to hide them, and protected by a track.

The Ubiquiti Access Points like being ceiling mounted, so this has worked well to cover all of our house with only one AP. The slanted ceiling in the closet that it's mounted on is the slant of the roof of the second floor.

Four 10-gig ethernet runs into the office:

Making lots of cables for the keystone patch panel

I used an audio rack which works well as a low-depth rack for network equipment, since I don't need room for full-depth servers. I ran a power outlet to feed the UPS and extended battery pack which feeds the rack. The rack is located in the center of the house near the stair well. We've got a dehumidifier and improved the insulation of the basement, so it sits under 50% humidity and around 60 degrees year-round.

Switching to fiber

Despite my IPv6 adventures in the apartmentlab, I opted for the bandwidth of fiber at the new house over having native IPv6 connectivity. The only fiber provider on our street, Fidium fiber, does not provide IPv6 in 2024. I debated this for quite a while, but a few things swayed me to switch off of cable with IPv6 to fiber without. The first was just wanting to experiment with the higher bandwidth, being able to saturate my equipment and see what it was capable of, since most of it was able to pull around 2 Gbps now. The second was that I knew it wouldn't affect our work-from-home requirements. I do need IPv6 for some things, but I just flip on Cloudflare Warp on my laptop for those cases. Last, I wanted the buried fiber connection to the house, and to get off flakey cable. We had so many outages on Spectrum cable in this area that I was ready to switch. We haven't had a single internet outage yet with fiber. And Fidium has been very accommodating of customers running their own routers. They provide an ONT with 1 gig and 10 gig copper ports, and I just have to register the MAC address of the NIC (an SFP module at this point) that I'm connecting on the other end. I also was able to remove a lot of the coaxial cable in the basement.

I added conduit and a 50 foot extension for the fiber coming into the house to run it over to the rack:

10 Gigabit

With 2.5GbE ports on my router, 2 Gbps fiber service, and 10GbE ports on my NAS and laptop (via an adapter) and 2.5Gbe and Wi-Fi 6E (capable of about 1.4 Gbps wirelessly) at the Access Point, it was time to get above the 1 Gbps limit of my main ethernet managed switch. I searched for a long time and ended up with a QNAP switch, which -- while it doesn't have any Power over Ethernet (PoE) -- it has 8x 10GbE RJ45 ports as well as 8x 10 gigabit SFP ports at a reasonable price point. This let me have enough copper (RJ45) ports to support my existing 10 gigabit devices, a couple of SFP to bridge my switches, and a bunch of extra SFP ports to expand in the future.

The switch has been rock solid, the only downside being the web interface, namely the VLAN selector that took multiple forum posts to figure out how to get my ports tagged successfully along with the trunk ports for the router and access point.

Initial setup. (The names are Marathon-class heavy cruisers. This organization didn't last long!

Back to Ubiquiti

After a year or so of enjoying the flexibility of OPNsense over my old Amplifi HD router, I started having an issue. All of our laptops and phones -- anything not being routed through a tunnel like Cloudflare Warp -- started to fail to load some web pages periodically. Most importantly this was reported by my wife, so it was happening on my common subnets and not just on my experimental ones. It seemed to mostly be Google properties that were affected, but this distinction would only be helpful for debugging, we use too many services for home and work that I couldn't ignore these. The symptom on our phones was just that the webpage would start connecting and never get to rendering any of the page - just a blank page was shown. I could reproduce less often on my laptop - just a browser starting to connect but never loading the page. YouTube was a common example, reloading often helped but was still untenable. An error message was never displayed. I ended up finding a reproducer on my laptop with Google maps, and could periodically show the curl failing and not completing the TLS handshake:

kerby@tycho % curl -vvvvv https://www.google.com:443/maps * Trying 142.251.41.4:443... * Connected to www.google.com (142.251.41.4) port 443 (#0) * ALPN: offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/cert.pem * CApath: none ^C kerby@tycho %

kerby@tycho % openssl s_client -showcerts -connect maps.google.com:443 < /dev/null Connecting to 142.251.40.206 CONNECTED(00000006) write:errno=60 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 323 bytes Verification: OK --- New, (NONE), Cipher is (NONE) This TLS version forbids renegotiation. Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) ---

As best I could tell, something was preventing the TLS handshake response from the server back to the client after the client hello. I still don't know what was the cause was. I spent a few tries resetting everything I could think of in OPNsense, trying to get it back to a working configuration. Eventually, due to the impact, I tried replacing the router with a Ubiquity Dream Machine Pro Max. I haven't had a single issue since. This was unsatisfying from a curiosity perspective, but sometimes things just need to work. This has been a bit of a theme with this post! Mostly spending time on house projects and not root cause analysis.

I've been running the Dream Machine and plan to continue to do so -- the performance has been great, and with recent updates the Wireguard options are sufficient for my needs.

Rack at the end of 2024

Plenty of cable management left to do after the router and switch changes.

Next up

Things I'm working on:

Z-Wave relays to fill some gaps where we don't have a good way to wire up switches to lights. For example, being able to turn off the outside garage lights from the mudroom.
paperless-ngx and a scanner to handle incoming household documents
Ersatz tv with Jellyfin for commercial-free reruns

Homelab 2025

2025-12-31 19:00:00 -0500 EST

Homelab 2025

2025-12-31

The homelab saw its most sustained usage this year, and I've found several applications that are fully replacing cloud services for me or adding new capabilities. I'm working on another post about how I host these services, but for the end of the year this is a quick write-up of the state of the homelab, covering the tools I found most useful.

Hardware

Overall no major hardware changes this year. I'm still using the same Ubiquiti Dream Machine, QNAP 10G switch, a handful of mini PCs for compute, and a Synology for storage. I did have a NIC on one mini PC die, so I waited for a sale on a barebones MS-01 and migrated the RAM and SSD over. As a result, I have a bit more available compute with the Intel i5-12600H (the other PCs are just 4-core N100s) and a PCI-e slot to play around with.

I added two new runs of CAT-5e up from the basement to our downstairs den/office where I've been working. I have my work computer and desktop workstation there.

Depending on how much pruning of data I do on my NAS, I may need to replace a couple of drives to get some more space on my Synology within the next year or so.

Z-Wave and Shelly Relays

We have a couple of places where the electrical wiring in our house falls a bit short. One is the outdoor lights on our detached garage. These provide the majority of the light for our driveway at night, but they run off the panel in the garage and so the only switch is inside the garage.

The second is the upstairs lighting. In the main bedroom, office, and nursery there are no overhead lights, and no light switches which control electrical outlets.

I decided to try solving both of these with smart devices. The garage doesn't have a good solution to making a switch in the house be on the same circuit as the garage lights, and upstairs we would instead be looking at installing recessed lights in the ceiling if we were to do any serious electrical work.

My requirements for the smart devices were that I wanted something that didn't look obviously like a smart switch and that didn't require apps or internet access to work.

Z-Wave seemed to be the best ecosystem that works over its own radio protocols and doesn't rely on WiFI. I bought a few Shelly Z-Wave relays, light switches, and power outlets and got them all automated with HomeAssistant. It's in there somewhere.

I've been impressed with the Shelly relays. They get installed inside the electrical box and take over control of the circuit, using the light switch as an input that can also be used for automation. For the garage, I installed one into the circuit for the lights such that I can turn the lights on and off with HomeAssistant in addition to the real switch. Then I installed a second relay in a switch box inside the house (which only controls a small light by the side door otherwise) and used that as an input to HomeAssistant. Now when I switch on the side door lights for the house, the garage lights come on as well.

This works about 98% of the time. Occasionally it seems like something misses the signal and I toggle the light switch again to fix it. There's also a reset procedure of toggling the input switch for the relay five times to perform a reset, which I've only had to do once or twice. Overall I've been very happy with this setup, and it's worked throughout the year at all temperatures. It's also been handy to turn on the outside lights from my phone through the VPN a couple of times when we were away.

Upstairs, I found some light switches that look just like regular paddle-style switches, but are actually Z-Wave remotes. These got mounted in each of the three rooms upstairs where you'd expect a light switch, and they are paired up with a floor lamp to light their respective room.

I also bought an outdoor Z-Wave plug for the Christmas lights we put up on the front of the house. I was able to add the switch to the same automation as the garage lights, turning the Christmas lights on and off with the rest of the outdoor lights.

For a hub, I'm using a Zooz 700-series USB controller plugged into one of the mini PCs in my Proxmox cluster. I installed HomeAssistant via an LXC, as this seemed like the easiest way to pass through the USB device, and I haven't had any issues, including migrating the device and LXC across PCs. I run a backup job for the LXC which will hopefully be sufficient to restore my automations if needed, as I really do not want to deal with the HomeAssistant UI again.

Another option is to use a controller that can be connected to the Apple Home ecosystem, turning an Apple TV into a Z-Wave hub. Then instead of HomeAssistant you'd use the Home application to configure your automations. I trust this setup to be available and reliable less than I do my Proxmox and LXC setup, however HomeAssistant has been so difficult to do the most basic operations that I might give this a try at the next house.

The only real downside I've found regarding the Z-Wave hardware is that adopting a device can be really flaky and the best method depends on the device. Typically it's done with a UUID provided by a QR code on a little sticker on the device, and then usually a copy of that QR code is supplied in the box. I've lost at least one of the extra QR codes for the relays in the wall, so I would potentially have to open up the switch boxes and pull out the relays if I needed to adopt them to a new hub. If I remember right there is a discovery mode that you can put the hub in, but I did not have success with this method. I always had to add the device by UUID first, put the device in adoption mode, and then initiate the search. HomeAssistant made this pretty miserable, and there are multiple ways to add a device, only one of which worked, so I always forgot how to do it each time I added a new device.

Software

HomeAssistant

My review of HomeAssistant is mostly included above regarding Z-Wave devices. I really find the UI inscrutable and the most basic things everyone want to use it for (add a device and use it in an automation) is buried behind weird menus and settings. That said, day-to-day it's been rock solid and has a decent iOS application whose dashboard lets me control my lights easily.

The rest of my list is much more favorable:

Paperless-ngx and Swift Paperless

Once I set up solid base for hosting software in my cluster that needed both persistent storage and databases, including reliable backups, I installed Paperless-ngx which is a document management suite. The basic idea is that you scan in your receipts, mail, tax documents, etc, and it helps you organize it and make it searchable via tags, OCR, and other features.

I originally thought I would need to buy a scanner like a ScanSnap, but before doing so I downloaded five or six iOS client apps for Paperless, and found that phone scanning is plenty good enough that there's no need for a dedicated scanner. All the apps seem to use the same scanning engine, probably provided by an Apple framework, and then they each have a slightly different workflow for ingesting and searching through documents.

I ended up favoring Swift Paperless, as it provided the best scanning and naming workflow, and search seemed to work the best (a feature that did not work at all in some clients).

I use it any time we get some documents in the mail, just taking a few seconds to do a scan. You can even omit naming and use the app or web UI later to triage new documents. Once you've added tags to the app, I've found that the auto-tagging feature works quite well.

MiniFlux and Readkit

I love RSS feeds (here's mine!). It feels like a bastion of sanity in the modern web, and it's become my main source of news. Tech news is not terrible to find and consume, but especially for current events all the major apps and websites are nearly unreadable, especially on a phone. I don't use social media and don't want to, which can make it challenging to get important updates. I've subscribed to Political Wire and read it entirely via RSS, even though it's a bit of a firehose of a feed. I maintain a list of blogs, news, and enthusiast sites -- largely from independent publishers -- that I enjoy, and reading the content on my phone via ReadKit is always my first stop. It's also refreshing to sometimes open the app and see that there are no new articles.

For a long time I used NetNewsWire, but I wanted to see if there were more options if I self-hosted some portion of it. I've been using two tools, MiniFlux running as a container in my cluster which maintains my list of feeds and pulls in updates on a schedule, and Readkit on iOS as my primary reading interface. I tried out a few different Miniflux-compatible readers and in addition to having a very nice rendering engine, Readkit has a killer feature which is a button to go fetch the full web page for sites that only provide the first few lines or a summary in the RSS feed. The result is a reading experience similar to the Reader functionality of web browsers, with no ads, redirect loops, or other junk that crashes the browser. It works beautifully and I'm more than happy to pay for a good client application.

Immich

Immich is a photo management tool that can be used as an alternative to Apple Photos or Google Photos. For me though, I don't plan on ever having it be a replacement. Instead, I want to continue to use iCloud photos to interoperate with family members because that's what they will continue to use, and it's the native photos app for all of our iPhones. Instead, Immich has replaced Synology Photos as the best way to create a backup of all of my photos, with a great management interface in addition. The Immich iOS app's "Backup" feature can be enabled, which syncs your iCloud photos over into Immich. This removes my dependence on Synology and gives me the peace of mind that I have a mirrored library at all times for all photos I take or save on my phone.

It also works much better than iCloud/Apple Photos for images taken with my "real" camera, a Canon M50. It can handle RAW photos just as well as JPEGs, which means I can have access to the full resolution originals without taking up any cloud storage. I currently am running a cloudflared container next to Immich to expose it to family members behind Cloudflare Access; again, not as a replacement for another tool, but as a way to share higher-resolution photos.

Navidrome, Mp3tag, and play:Sub

I've been continuing to enjoy using this stack as an alternative to Apple Music or Spotify with my own music collection. Mp3tag was a nice app I found for fixing mp3 metadata, Navidrome serves the library over the subsonic protocol, and after trying out many, many clients, play:Sub has been my favorite iOS player. It even has a CarPlay app so I can stream music from my server over Wireguard to my phone, connected to wireless CarPlay in my truck.

Wireguard and the Wireguard iOS app

Seems a little silly to call out a VPN protocol here, but Wireguard and mainly the Wireguard iOS app get a mention because it works so well for my needs. I run Wireguard on my Ubiquiti router, which also has a reverse DNS hook to update a Cloudflare DNS record with the IP from my ISP, and then I use the Wireguard app on my iPhone. I have it set to connect on-demand when switching to LTE or a non-home WiFi network, which means I have seamless connectivity to my cluster at home. All the apps listed here work perfectly no matter where I am, without any special DNS or needing to toggle anything on or off, which has really made them viable as selfhosted alternatives. I also haven't had any issues with battery life when leaving Wireguard on all the time.

Miscellaneous

Nebula continues to work for WAN hole-punching to connect my Synologys together without needing any configuration on the router at my parent's house. We both use this setup for one set of offsite backups.

I'm testing out Vaultwarden and the Bitwarden iOS app as a selfhosted contingency to 1Password.

Synology Sync has continued to work well as a Dropbox alternative, giving me a local copy of code and projects on my laptop while also having the files available over network mounts. Similar to Synology Photos and Immich though, I want to replace this with an open-source solution. Currently looking into Syncthing.

I've been running unbound in my cluster for a while, providing DNS to my personal devices. I'd like to see if I can improve my experience on some devices by having unbound load community-provided adblock lists and then trial it as the resolver on the household network.

I've been testing Authentik and Pocket ID for single sign-on, but honestly I'm not sure that I want to have all the services behind a single identity. So far Pocket ID providing Passkey logins via Yubikeys and TouchID/FaceID seems promising though.

I run a container registry locally for my cluster which lets nodes pull images without needing to rely on outside registries. If you go this route, just set up a publicly-trusted TLS certificate from Let's Encrypt that the registry serves, this is a far better solution than trying to add overrides for insecure registries.

Versity Gateway has been working well as an S3 endpoint that dumps the data directly to my NAS. I've been testing this out as a replacement for both TimeMachine and Arq + Backblaze B2. The Versity Gateway endpoint works as a backup location in Arq to send the files to my NAS which I can then back up along with the rest of the data there.

I set up a Forgejo instance for personal git hosting. Not sure how much I'll use this but so far it's been preferable to all the alternatives available.

Code-Server has been an interesting experiment for providing a web-based IDE across all my machines. VS Code is not my favorite but it works well enough in this context.

I've been testing Jellyfin, ErsatzTV, and the SenPlayer client application for setting up my own streaming TV. The idea is to have local stations with educational programming and cartoons that are ad-free for the kid.

I tried out the latest versions of a whole bunch of Linux distributions and window managers for use on my workstation. I landed on Fedora Cinnamon and have been very happy with it. I didn't take notes during the trial process but there were so many bugs with so many distributions that it was a little surprising. Some of this was the fault of me running a NVidia card and using proprietary drivers, but many other non-gpu based issues like installers failing, window managers completely locking up immediately on login, etc. Anyway, I'll be staying on Fedora Cinnamon for the forseeable future.