As usual, my cluster is limited in resources (what you get for using the cheapest VPS available, I guess). Recently for whatever reason, one of my nodes has been constantly getting high IO and CPU warnings. As usual, this was caused by high swap activity due to memory exhaustion. The thing is, I didn’t really understand why that would suddenly start to happen.

Things (which may or may not be detailed in a later post) happened and I ended up wanting to migrate the MariaDB powering this blog to a different node. This is when all hell broke loose.

That is because I was still using a Bitnami chart for MariaDB. While the chart itself is still up on Github, the image backing the primary container has been yanked from Docker Hub. Therefore unless the image is already cached on the node, there’s no easy way to move to that node.

I decided to instead just roll my own manifests based on what Bitnami had been doing and the official LTS MariaDB Docker image. It was not trivial at all, since the Bitnami chart uses a very custom folder layout, so migrating my existing PersistentVolume (or at least the physical volume backing that) with the same non-root execution model meant a lot of manual work.

First I backed up the Linode volume (cloning it), then mounted both the one created by the new StatefulSet and the old volume into a VPS and fiddled around with the files – this of course included a few very scary moments where I accidentally almost 20 years of history (curse you rm -rf).

It’s kinda important that the new MariaDB instance isn’t yet running (and writing into its files), or maintaining consistency during the migration is nigh impossible. Make sure that the database folders are in the right location, and all should be fine. I was migrating between versions of the same LTS, so I had no compatibility issues. If there are error logs about how a table is “corrupted. Please drop the table and recreate” that likely means that the new MariaDB instance was running. I ran into this and overwriting the new ibdata1 with the old one fixed it (as illustrated by this blog post being published).

I could’ve imagined spending my Friday night with more entertaining things, but at least now I have one less Bitnami dependency – and since their 180 on open-source, every Bitnami dependency is a ticking time bomb.