Category: valeblog

charts/stable and git references

Helm was meant to be the package manager for Kubernetes. One common problem for package managers is “how do I find my packages?” Many package systems opt for having a default central repository for stuff. Distros have their central repos for apt. Programming languages too: for Node it’s npm, for Ruby it’s RubyGems, for Java it’s Maven central, for Clojure it’s Clojars. Of course most if not all systems have a way to add other package repositories or at least some other means to pull in dependencies (referencing git commits for example).

For Helm the central repository of charts/stable used to be the obvious default. You can of course add other repositories too, but defaults are powerful and many people will just give up if something is not available in the default source. On the other hand, having everything in one place puts a huge burden on the maintainers of that one place, as was the case of charts/stable. So they deprecated it.

Up and out at 14 and above

Getting up to 14 camp on Denali was a tough climb, but we had two weather days to recover after the load carry, and I was eager to finally get moving again. When we finally got to camp, I was then glad it was over nonetheless, even if only for that day.

To Denali 14 camp

The move from 11 camp to 14 camp (camp 3) is tough. It’s almost a thousand meters ascent over Motorcycle hill, Squirrel hill and around the Windy Corner. Motorcycle hill is steep and rockfall danger in Windy Corner is pretty much the only reason we have to carry helmets on Denali.

Két nap szünet a táborban

Amíg feltelepültünk a 3000m-es táborba, jó időnk volt. Kérdéses, hogy a gleccseren tűző nap és meleg tényleg jó időnek számít-e, de a következő két nap hóviharainál csak jobb volt. Ideje hát szót ejteni egy kicsit a tábori életről is a Denalin.

The snowshoe days

Arriving on the Kahiltna glacier we quickly set up camp and just passed out before a late night (or early morning?) start. Lower on the mountain we’d start moving in the middle of the night, so we avoid getting roasted, boiled and blinded by the sun on the snow. Then again even at 3am it was bright enough that I could count on one hand the occasions I had to use my headlamp during the three weeks total.

Reaching Denali

To climb Denali, you first need to get there. So my mission was to reach Anchorage from Tokyo in the middle of June (with all the fun of covid restrictions included). It went surprisingly smoothly. There were a few scary moments, like when people told me Hungarians wouldn’t be allowed entry to the US because of covid—but the restriction turned out not to be about EU nationalities but traveling from EU area.

Azok a fránya tervek

Az év elején írtam róla, hogy a koronavírus hogy megtépázta a terveimet. Akkor úgy volt, hogy tavasszal az Island peak (6189m), majd nyáron a Broad Peak (8047m) és a K2 (8611m) lesz úticél. Végül persze a korona jobban tudta, és sikerült mindkét tervemet bedöntenie.

Havas erdő Naganoban

Upgrading my cluster

My cluster is now running on k3s 1.20.6 and Argo CD 2.0.0 with its Helm chart at 3.2.2. Actually, upgrading Argo itself wasn’t much of a problem. I just changed the targetRevision of the Application and it was up and running in a few minutes. Then a few days later things got interesting.

There were no downtimes, but I noticed that Argo started failing to sync itself. Apparently a new minor version of the Helm chart came out (though it was still the same application version) that added support for the version of Ingress. However, it also accidentally broke clusters running Kubernetes before 1.19. And mine was one such.

While the Argo people are figuring out how to fix this (if), I decided to go and take this opportunity to upgrade my cluster. This wasn’t as painless as it should’ve been though.

Gear I want and gear I can’t seem to find

The mountain gear I look for the most is: very light but durable protection for my limbs. That means gloves and boots that are as light and breathable, quick-drying as possible so my hands and feet don’t rot in sweat in the summer heat, while being durable enough not to fall apart scrambling over rocks in the Japanese alps.

Rocky route in the Japanese alps (at Nishi-Hotaka)

Dealing with DiskPressure

My 4-node k3s cluster (where this blog is hosted too) kept dying every now and then. Looking at kubectl describe nodes it quickly became evident that this was caused by the nodes running out of disk space. Once a node gets tainted with HasDiskPressure, pods might get evicted and the kubelet will be using (quite a lot of) CPU trying to free disk space by garbage collecting container images and freeing ephemeral storage.

My setup by default uses local storage (the local-path provider) where volumes are actually local folders on the node. This means that pods that use persistence are stuck with the same node forever and can’t just move around. This makes eviction a problem, since they have nowhere else to go. It also means that disk usage is actually disk usage on the node, and not on some block volume over the network.