Tag: english

Skardu here I come

After ten hours at Doha airport I mostly spent just walking around aimlessly, it was time to now fly to Islamabad. The flight wasn’t long, but it was just long enough to both have a short nap and watch The Matrix: Resurrections. I’d probably need to watch that movie like two more times to be able to form a coherent opinion on it.

Getting through border control was much smoother than I expected. They just asked a few questions to see if I get too stressed out, kinda how Americans do it too. All my baggage arrived in one piece too, so shortly I was on my way to my hotel for the night. A night much shorter than I’d like… Since my flight on to Skardu was at 6am in the morning, it meant I could only get 4-5 hours of sleep, and while a shower helped get in the mood, I still kept waking up every half an hour from weird dreams in a panic that I overslept.

Halfway there

Ten hours transit at Doha airport. Sleeping is always an option, though with these time differences I’m not sure it’s a good idea. The airport is crowded (and of course about half the people don’t care about the mask requirement) so finding an open seat near a power outlet was a challenge. But at least there’s this weird yellow lamp-bear thing.

The visa ordeal

I’ve been planning an expedition to Broad Peak and K2 for almost two years now. At first the plan was to climb last summer, but then covid ruined that. The expedition was then rescheduled to this summer, while last summer I visited Denali instead. However my departure to Pakistan faced some unexpected obstacles.

In flight towards Denali

Adding Grafana annotations based on Argo CD events

I was thinking about how nice it would be if I could see on my main Grafana dashboard (the only one I use at this point actually) when there are new versions of something deployed. This way if by chance there is a problem with something afterwards I can see at a glance what could have gone wrong. Also I really like just looking at that Grafana dashboard and see that everything is alive and well. (Except when it isn’t.)

The Grafana chart was upgraded! Nice CPU burn…

Getting advanced mountain gear in Japan

You’d expect that with Japanese people being among the top mountaineers on 8000 peaks, the country having a healthy scene for winter sports it shouldn’t be hard to get advanced mountain gear, right? Let’s share some stories.

What was and what’s to come

I’ve already written about what was going on most of last year in my birthday post, but so many people around me are writing year-end/year-start summaries that nudged me to revisit the topic as well.

Budapest at night

Fighting an “Unknown MySQL error” when connecting to Aurora from Rails

The other day I was struggling with a very weird error when upgrading to Ruby 3. The initial migrations for a Rails app would fail with “Mysql2::Error: Unknown MySQL error (ActiveRecord::StatementInvalid)”, but only in certain environments. The error would occur when Rails tried to check what migrations are already applied by looking at the schema_migrations table.

2022-01-19 09:12:59.205 +0000 [DEBUG]    (1.7ms)  SELECT GET_LOCK('4252831219231700070', 0)
/usr/local/bundle/gems/mysql2-0.5.3/lib/mysql2/client.rb:131: warning: rb_tainted_str_new_cstr is deprecated and will be removed in Ruby 3.2
2022-01-19 09:12:59.222 +0000 [DEBUG]    (2.0ms)  SELECT `schema_migrations`.`version` FROM `schema_migrations` ORDER BY `schema_migrations`.`version` ASC
2022-01-19 09:12:59.224 +0000 [DEBUG]    (1.8ms)  SELECT RELEASE_LOCK('4252831219231700070')
rails aborted!
ActiveRecord::StatementInvalid: Mysql2::Error: Unknown MySQL error
/usr/local/bundle/gems/mysql2-0.5.3/lib/mysql2/client.rb:131:in `_query'


There are a few mountains other than the seven summits that I’m interested in climbing for other reasons. One of those is Chimborazo, a mountain in Ecuador that has the interesting property that its summit is further from the center of Earth than that of Mt Everest (because of the planet’s shape). It’s also a phrase my late gran used to use as an expression of “the greatest magnitude”—I think in her day that was common as it was believed to be the highest mountain. This connection is why I got interested and went on an expedition to climb Cotopaxi and Chimborazo.


When I accidentally Longhorn CSI

Symptoms: CPU load on all the nodes, but not the pods. Looking at Grafana, I noticed that CPU load on some of my nodes was constantly very high. At the same time, even the total CPU use of all the pods summed wasn’t above 0.4. What gives? This usually gives that the control plane is getting fried by something. It may be trying to relieve disk pressure, or in this case, trying to revive CSI.

Trying to figure out what was causing problems I checked the pods in kube-system with kubectl get pods -n kube-system. It quickly became apparent that there is a problem: disk-related pods like csi-resizer, csi-snapsotter and csi-provisioner were in CrashLoopBackOff.

I’ll be quite honest in that I’m not sure what the problem was. A few searches later I came to the conclusion that an earlier node reboot had left the pods with a corrupted DNS cache or something along those lines. Basically every issue I found with the symptoms I was seeing came down to DNS problems (longhorn/longhorn#2225, longhorn/longhorn#3109, rancher/k3os#811).

Alas I haven’t touched any of the networking machinery of Kubernetes (nor configured any of it for k3s) so my first idea was just the good old one from IT Crowd: “have you tried turning it off and on again?” So I did. Luckily another restart of the afflicted nodes solved the issue. I’m glad it did because I dread what I’d have had to do otherwise.

Steps to a more stable k3s cluster

It’s all too easy to kill a k3s cluster. I’ve been using k3s for years now and I’ve had plenty of adventures tweaking various aspects of running it. Before it’d take just a small change to an Argo Application to trigger a cascading failure. Hopefully now it’s a bit more resilient. Just a bit.

ship helm