Category: valeblog

Upgrading my cluster

My cluster is now running on k3s 1.20.6 and Argo CD 2.0.0 with its Helm chart at 3.2.2. Actually, upgrading Argo itself wasn’t much of a problem. I just changed the targetRevision of the Application and it was up and running in a few minutes. Then a few days later things got interesting.

There were no downtimes, but I noticed that Argo started failing to sync itself. Apparently a new minor version of the Helm chart came out (though it was still the same application version) that added support for the networking.k8s.io/v1 version of Ingress. However, it also accidentally broke clusters running Kubernetes before 1.19. And mine was one such.

While the Argo people are figuring out how to fix this (if), I decided to go and take this opportunity to upgrade my cluster. This wasn’t as painless as it should’ve been though.


Gear I want and gear I can’t seem to find

The mountain gear I look for the most is: very light but durable protection for my limbs. That means gloves and boots that are as light and breathable, quick-drying as possible so my hands and feet don’t rot in sweat in the summer heat, while being durable enough not to fall apart scrambling over rocks in the Japanese alps.

Rocky route in the Japanese alps (at Nishi-Hotaka)

Dealing with DiskPressure

My 4-node k3s cluster (where this blog is hosted too) kept dying every now and then. Looking at kubectl describe nodes it quickly became evident that this was caused by the nodes running out of disk space. Once a node gets tainted with HasDiskPressure, pods might get evicted and the kubelet will be using (quite a lot of) CPU trying to free disk space by garbage collecting container images and freeing ephemeral storage.

My setup by default uses local storage (the local-path provider) where volumes are actually local folders on the node. This means that pods that use persistence are stuck with the same node forever and can’t just move around. This makes eviction a problem, since they have nowhere else to go. It also means that disk usage is actually disk usage on the node, and not on some block volume over the network.


Változnak a tervek

A koronavírus az én terveimet se kíméli. Tavaly nyáron Indonéziába terveztem menni, megmászni Óceánia legmagasabb hegyét, majd ősszel egy duplát terveztem a Himalájába, ahol egy egy hónapos túra során másztam volna meg a Mera Peaket és az Island Peaket. Először a nyári út úszott el, ami helyett még próbáltam másik vezetővel megszervezni ugyanazt, de persze a vírus jobban tudta.

K2 from the north
Photo by Kuno Lechner

Így arra már számítottam, hogy ősszel én már nem megyek majd a Himalájába, de hogy még a ritkaságszámba menő családi lagzira se jutottam el, az egy kicsit fájt. Az őszi duplát végül sikerült átszerveznem 2021 tavaszra, de erről is a minap szóltak, hogy nem lesz megtartva, mert nem jött össze a szükséges létszám.

Mi lesz így akkor idén?


Clojerlを使ってみる

ErlangはEricssonがはるか昔に開発した通信環境用の言語で、ものすごく頑丈で安定している環境として有名。実際に世界中のモバイル通信環境で用いられ99.9999999% (“nine nines”) の可用性を誇っている。最近ではRuby風でErlangのBEAMを実行環境とするElixirが流行っている印象がある。分散を前提にしている関数型言語としてClojureに近いと感じた。そして実際にBEAMの上でClojureを実装しているものがある。

Clojureが最初はJVMと.NETのCLRも対象にしていたが後JVMだけになり、またClojureScriptの登場でJavaScript上で動くようになった。だから根本的なところは案外実行環境に依存しないところもあるかもしれない。BEAM上でClojureを実装しているClojerlはあくまでもコミュニティーからのもので正式なClojure版ではないが、試したかったBEAMと選べるなら選ぶぐらい好きなClojureが合ったものに当たるから触ってみざるをえなかった。


Down the __extmap rabbit hole

Was trying to figure out what defrecord does in the impl of ILookup it generates and I absolutely do not understand that magic. Essentially it amounts to (get __extmap key)  and I can’t even figure out how this magical __extmap gets into scope (a global or some special runtime thing?) and this isn’t even used. Confused

zerusski on Clojurians Slack

This post on Clojurians sent me down the rabbit hole looking for where this “mysterious” __extmap comes from. It really isn’t obvious at first, but when I found it, it was “wow” times “duh”.


ClojureScriptとRustを組み合わせる

以前gitのGUI作りたいと思っていろんな試行錯誤して落ち着いたのはClojureScript(re-frame使ったReact)とElectronで作ったUIにRustのgit2-rsで生のgitとのやりとりする方式。まだちょっとややこしいところも残っているが、ほとんどはそれなりにスムーズな開発フローに乗れた。


Beyond burger

Pretty much starved and three more hours to go until my flight out of Dubai, I was looking for food. Something filling. Tokyo destroyed my plan of eating something before taking off there since there was literally not a single shop open in Narita airport. Luckily Dubai wasn’t dead like that so I had plenty to choose from.


QUIC and quicker

The past week or so I’ve been working on implementing the QUIC protocol in Clojure. Currently there is no Java implementation to use either (that I know of), and I just found out the other day that netty decided to use the Cloudflare’s Rust library quiche under the hood instead of rolling their own. The protocol is currently a IETF draft at version 32, expected to turn into an RFC soon.


GitOps with Argo

It’s been a year since I wrote about bootstrapping a cluster with Argo and using Argo Rollouts for canary deploys based on Prometheus metrics. Since then many things have changed. I moved from Digital Ocean to Linode (mostly because Linode has a Tokyo region) and from a single-node k3s “cluster” to a 4-node one. But most of how I use Argo CD for GitOps hasn’t changed.

orange and brown tree branch