And I’m not talking about the meme with the guy in the corner thinking “they have no idea.” I’ve been playing around with k0s again, trying out alternative runtime and network plugins. It of course has to be dual-stack in 2026, but that’s where I ran into some unexpected problems.
Rescuing broken MariaDB
As usual, my cluster is limited in resources (what you get for using the cheapest VPS available, I guess). Recently for whatever reason, one of my nodes has been constantly getting high IO and CPU warnings. As usual, this was caused by high swap activity due to memory exhaustion. The thing is, I didn’t really understand why that would suddenly start to happen.
Things (which may or may not be detailed in a later post) happened and I ended up wanting to migrate the MariaDB powering this blog to a different node. This is when all hell broke loose.
Narwal Freo Z Ultra second impressions
The new robot in the house didn’t make a very positive first impression. Sad to say that second impressions are not much more favorable either. Let’s start with the good things though.
The hardware feels good. The machine is sturdy and its navigation seems solid. It doesn’t bump into everything like the Roomba and doesn’t get stuck on the flat legs of my Poang or my standing desk (which the Roomba regularly did).
Narwal Freo first impressions
I’ve been renting robot vacuums for the past few years. In my previous place I used to have a Roomba Combo j7+. This was a pretty pleasant experience. It could handle the shaggy rug in my living room. It didn’t fall off the stairs. It mopped the floor. It didn’t require me to babysit it through everything, it just worked.
But its box was quite big, so when I moved I didn’t want to carry it around, instead returned and replaced it with a cheaper not-combo j7+. This obviously didn’t have a mopping feature anymore, but that was the least of its problems. It would sometimes stop in the middle of the room requiring rescue because it somehow sensed it was “stuck”. Then when it managed to somehow break its side brush thing on I don’t even know what (probably the edge of a rug), I decided to return it and find something better.
CoreDNS going MIA
I ran into a really weird problem today. I noticed some strange config drift on one of my nodes (shit happens when I manually experiment in “production”), so I decided to reinstall/upgrade the k3s agent. For a while now I’ve been connecting my nodes through tailscale, so that I can have my homelab machine join the cluster from my home network as well. k3s has a(n experimental) feature for “natively” integrating with tailscale and it’s been working just fine so far.
Fixed my TLS
Usually there was no problem. Stuff worked just fine. Certificates were generated and renewed automatically. https:// links opened without ugly browser warnings about how you’re about to get hacked and it’s the end of the known universe.
But when it wasn’t “usually”, when Traefik just happened to restart for whatever reason, then all of that was obliterated. Since Traefik was running on ephemeral storage, eg nothing was really persisted, innocently tweaking some configuration (that resulted in a restart) could be catastrophic. You know, self-signed certificates and ugly browser warnings.
Upgrading the argo-cd Helm chart from 5.x to 8.x
I’ve been using Argo CD for GitOps automation for a very long time. I have it manage itself too! The other day I noticed that there was a new major version of Argo CD, so I decided to upgrade my stuff too. It did not go smoothly, though it wasn’t an issue with Argo CD itself.
It was the usual problem of Helm charts renaming and moving around stuff in their values.yaml, which results in significant breakage for (from an user perspective) no good reason. I summed up what I learned so others don’t have to play around with it so much.
How I set up my smart lights with Home Assistant
I’ve been playing around with smart lights for a while now. I have four Edison Smart filament bulbs, two Philips Hue white bulbs and some Switchbot bulb. I had a terrible experience with Switchbot bulbs (which is why I got the Hues in the first place), disconnecting and needing to be re-paired all the time, suddenly factory resetting themselves to max brightness in the middle of the night and such. So the one I kept is just a reading light now with a manual switch on my bedside table.
For the others though, I’ve wanted to have some manual switches for them too in addition to being able to control them through my smartphone. When I get home it’s still quicker to tap a physical switch than to whip up my phone, wait until the respective proprietary app connects to the bulbs and then I get some light.
What I ended up with: a Home Assistant Green with the external USB Zigbee dongle, three IKEA RODRET switches and this blog post. I got those Home Assistant devices because that’s what I could get shipping to Japan, but I believe the Yellow version has Zigbee built in, which should make it a little less clumsy.
The mythical modular monolith
So often I see people giving talks about how the microservice architecture is a failure. You end up losing transactional protections, you’ll have to “join” data across a network boundary, and that network boundary is “always” flaky. Wouldn’t it be much better if everything was in one process, where you could enjoy the benefits of transactions, neighboring data is just a method call away and the only network you have to worry about is the database connection?
Podman containers can’t talk to each other
The other day I decided to update my server-side rendering example, and I ran into a strange problem. While my compose.yaml hasn’t changed, my containers could not talk to each other. This was most apparent with the nginx container, which immediately crashed out with [emerg] 1#1: host not found in upstream. What gives?
After a quick debugging including on a different computer, it became clear that my compose.yaml should be fine: it worked without issues on the computer that used Docker, but didn’t work with Podman. The issue with Podman was soon clear too: looking at podman network inspect the "dns_enabled": false was an obvious culprit. That’s where things got interesting.
