To climb Denali, you first need to get there. So my mission was to reach Anchorage from Tokyo in the middle of June (with all the fun of covid restrictions included). It went surprisingly smoothly. There were a few scary moments, like when people told me Hungarians wouldn’t be allowed entry to the US because of covid—but the restriction turned out not to be about EU nationalities but traveling from EU area.
My cluster is now running on k3s 1.20.6 and Argo CD 2.0.0 with its Helm chart at 3.2.2. Actually, upgrading Argo itself wasn’t much of a problem. I just changed the targetRevision of the Application and it was up and running in a few minutes. Then a few days later things got interesting.
There were no downtimes, but I noticed that Argo started failing to sync itself. Apparently a new minor version of the Helm chart came out (though it was still the same application version) that added support for the networking.k8s.io/v1 version of Ingress. However, it also accidentally broke clusters running Kubernetes before 1.19. And mine was one such.
While the Argo people are figuring out how to fix this (if), I decided to go and take this opportunity to upgrade my cluster. This wasn’t as painless as it should’ve been though.
The mountain gear I look for the most is: very light but durable protection for my limbs. That means gloves and boots that are as light and breathable, quick-drying as possible so my hands and feet don’t rot in sweat in the summer heat, while being durable enough not to fall apart scrambling over rocks in the Japanese alps.
My 4-node k3s cluster (where this blog is hosted too) kept dying every now and then. Looking at kubectl describe nodes it quickly became evident that this was caused by the nodes running out of disk space. Once a node gets tainted with HasDiskPressure, pods might get evicted and the kubelet will be using (quite a lot of) CPU trying to free disk space by garbage collecting container images and freeing ephemeral storage.
My setup by default uses local storage (the local-path provider) where volumes are actually local folders on the node. This means that pods that use persistence are stuck with the same node forever and can’t just move around. This makes eviction a problem, since they have nowhere else to go. It also means that disk usage is actually disk usage on the node, and not on some block volume over the network.
I decided to spend a weekend relaxing. In my sense this means going to the countryside, maybe doing a few hours of something and then just being lazy for the rest. That was exactly the plan. I went with the vague idea that maybe I’d climb Mt Myoko, but though I had the gear for it, it did not happen. What instead happened was, in a short summary, a really nice long weekend.
Was trying to figure out what
zerusski on Clojurians Slackdefrecorddoes in the impl of ILookup it generates and I absolutely do not understand that magic. Essentially it amounts to(get __extmap key)and I can’t even figure out how this magical__extmapgets into scope (a global or some special runtime thing?) andthisisn’t even used. Confused
This post on Clojurians sent me down the rabbit hole looking for where this “mysterious” __extmap comes from. It really isn’t obvious at first, but when I found it, it was “wow” times “duh”.
Pretty much starved and three more hours to go until my flight out of Dubai, I was looking for food. Something filling. Tokyo destroyed my plan of eating something before taking off there since there was literally not a single shop open in Narita airport. Luckily Dubai wasn’t dead like that so I had plenty to choose from.
One of the 100 famous Japanese mountains in reach for a day hike from Tokyo is Mt Nasu (which in Japanese is a homophone with “eggplant”) in Tochigi. It’s easy (though not particularly cheap) to get there by (bullet) train and bus. This time I went for a quick hike to the Chausu peak (which is a much shorter climb than the highest Sanbon-yari). The weather wasn’t exactly great, cloudy and extremely windy, but at least it wasn’t dumping on me.
The past week or so I’ve been working on implementing the QUIC protocol in Clojure. Currently there is no Java implementation to use either (that I know of), and I just found out the other day that netty decided to use the Cloudflare’s Rust library quiche under the hood instead of rolling their own. The protocol is currently a IETF draft at version 32, expected to turn into an RFC soon.
It’s been a year since I wrote about bootstrapping a cluster with Argo and using Argo Rollouts for canary deploys based on Prometheus metrics. Since then many things have changed. I moved from Digital Ocean to Linode (mostly because Linode has a Tokyo region) and from a single-node k3s “cluster” to a 4-node one. But most of how I use Argo CD for GitOps hasn’t changed.
