Fighting an “Unknown MySQL error” when connecting to Aurora from Rails

January 24, 2022
valeblog
0 Comment

The other day I was struggling with a very weird error when upgrading to Ruby 3. The initial migrations for a Rails app would fail with “Mysql2::Error: Unknown MySQL error (ActiveRecord::StatementInvalid)”, but only in certain environments. The error would occur when Rails tried to check what migrations are already applied by looking at the schema_migrations table.

2022-01-19 09:12:59.205 +0000 [DEBUG]    (1.7ms)  SELECT GET_LOCK('4252831219231700070', 0)
/usr/local/bundle/gems/mysql2-0.5.3/lib/mysql2/client.rb:131: warning: rb_tainted_str_new_cstr is deprecated and will be removed in Ruby 3.2
2022-01-19 09:12:59.222 +0000 [DEBUG]    (2.0ms)  SELECT `schema_migrations`.`version` FROM `schema_migrations` ORDER BY `schema_migrations`.`version` ASC
2022-01-19 09:12:59.224 +0000 [DEBUG]    (1.8ms)  SELECT RELEASE_LOCK('4252831219231700070')
rails aborted!
ActiveRecord::StatementInvalid: Mysql2::Error: Unknown MySQL error
/usr/local/bundle/gems/mysql2-0.5.3/lib/mysql2/client.rb:131:in `_query'
Read More

Ecuador

January 5, 2022
valeblog
0 Comment

There are a few mountains other than the seven summits that I’m interested in climbing for other reasons. One of those is Chimborazo, a mountain in Ecuador that has the interesting property that its summit is further from the center of Earth than that of Mt Everest (because of the planet’s shape). It’s also a phrase my late gran used to use as an expression of “the greatest magnitude”—I think in her day that was common as it was believed to be the highest mountain. This connection is why I got interested and went on an expedition to climb Cotopaxi and Chimborazo.

Cotopaxi
Read More

検疫所が指定する宿泊施設とは

January 4, 2022
valeblog
0 Comment

日本に戻った。簡単じゃなかった。

向こう滞在中に国がオミクロン危険地区みたいな指定食らってしまって、日本に戻るとPCR検査いくらあってもワクチンいくら打ってても問答無用で指定施設に数日いないといけない。意味不明。

到着は成田だったのでその付近のホテルが宿泊施設として指定された。ホテルの館一つまるごとがその目的に割り当てられてるっぽくて貸し切り状態。とはいえ受付済んで部屋に入ったら部屋から出られなくなるから、一階にコンビニがあってもモンエナ買いにいけずつらかった。

Read More

When I accidentally Longhorn CSI

December 12, 2021
valeblog
0 Comment

Symptoms: CPU load on all the nodes, but not the pods. Looking at Grafana, I noticed that CPU load on some of my nodes was constantly very high. At the same time, even the total CPU use of all the pods summed wasn’t above 0.4. What gives? This usually gives that the control plane is getting fried by something. It may be trying to relieve disk pressure, or in this case, trying to revive CSI.

Trying to figure out what was causing problems I checked the pods in kube-system with kubectl get pods -n kube-system. It quickly became apparent that there is a problem: disk-related pods like csi-resizer, csi-snapsotter and csi-provisioner were in CrashLoopBackOff.

I’ll be quite honest in that I’m not sure what the problem was. A few searches later I came to the conclusion that an earlier node reboot had left the pods with a corrupted DNS cache or something along those lines. Basically every issue I found with the symptoms I was seeing came down to DNS problems (longhorn/longhorn#2225, longhorn/longhorn#3109, rancher/k3os#811).

Alas I haven’t touched any of the networking machinery of Kubernetes (nor configured any of it for k3s) so my first idea was just the good old one from IT Crowd: “have you tried turning it off and on again?” So I did. Luckily another restart of the afflicted nodes solved the issue. I’m glad it did because I dread what I’d have had to do otherwise.

Steps to a more stable k3s cluster

December 11, 2021
valeblog
0 Comment

It’s all too easy to kill a k3s cluster. I’ve been using k3s for years now and I’ve had plenty of adventures tweaking various aspects of running it. Before it’d take just a small change to an Argo Application to trigger a cascading failure. Hopefully now it’s a bit more resilient. Just a bit.

ship helm
Read More

PDF export with the Ruby 3.0.3 docker image

December 5, 2021
valeblog
0 Comment

You might happen to use the wicked_pdf gem for PDF output in your Rails app. You might happen to use the wkhtmltopdf-binary gem to provide the required binaries. You might want to get the above to work on the latest (at this point 3.0.3-bullseye) Ruby docker image. Short answer: give up. A bit longer answer: it’s easier than you think.

Read More

Patching delayed_job for Ruby 3

December 4, 2021
valeblog
0 Comment

Monkey patching is bad. That’s where you should start from. It can cause trouble where you’d least expect it, conflicts with libraries you’d least expect in ways you’d least expect. And yet here I am sharing code for patching the delayed_job gem to (more or less) work with Ruby 3. Doesn’t this violate my own policies? There are a few choices.

  1. give up upgrading to Ruby 3 altogether
  2. monkey patch delayed_job as an emergency fix and make time to figure out what to do
  3. contribute to delayed_job making sure the gem is solid on Ruby 3
  4. get rid of all the .delay calls and switch to another async job library
Read More

reititはいいぞ

December 3, 2021
valeblog
0 Comment

reititはClojure/Scriptの高速なルーター。はい30文字。

brown wooden pathway on green grass field near brown rocky mountain under white clouds during daytime
Read More

またまた忙しい

December 2, 2021
valeblog
0 Comment

忙しいぞ。

遠征に出るのでアドカレの記事を事前に書き溜める必要がある。そして勢い余って色々書きたくなる。現状すでにブログに書き溜めが6件、未完成さらに4件、後会社のブログにも一件ある。楽しいし記録に残るんで今後振り返にも役に立つし、例えば転職考えることになったらその時アピールもできるし（できるか）。

あとは遠征の準備そのものでやることが多い。渡航で必要なPCR検査とか接種証明書の準備、今オミクロンで盛り上がってる世の中で一日で渡航条件が変わる可能性すらあるから油断ならない（変わりませんように）。実際に荷造りも始めて、ハイパーライトのリュックを遠征でいよいよ試せるかもしれないからワクワクしてる。本来は夏のK2で使う予定だったけど、それはコロナでとんだよね…

Read More

Stuff that broke in Rails 6.1

December 1, 2021
valeblog
0 Comment

Rails uses a “shifted” “semantic” “versioning” which pretty much comes down to the following. Major version: “we’ll most definitely break everything you ever depended on, half of them without warning.” Minor version: “we’ll probably break many stuff you depend on, some of them without warning.” Patch version: “we might accidentally some core APIs, but we promise it’s not intentional (or documented).” Knowing that, I still embarked on the grand endeavor of upgrading from Ruby on Rails 6.0.4.1 to 6.1.4.1. What could possibly go wrong, right?

Railway tracks are suspended above the washed out Tank Hill underpass of the Trans Canada Highway 1 after devastating rain storms caused flooding and landslides, northeast of Lytton, British Columbia, Canada November 17, 2021. B.C. Ministry of Transportation and Infrastructure/Handout via REUTERS
Rails in Canada and on Ruby share (un)surprising similarities (source)
Read More