While running Rails puma servers in production, we were seeing the incident that some old worker processes suddenly got stuck regardless of no change in the amount or trend of requests.

I found out the root cause and reported it to the upstream. This issue still exists in Ruby 2.6.0 and can be found as far back as Ruby 2.5.0. If you just want a summary of the bug, see ruby-lang#17669.

What Happened

One day, we noticed many ActiveRecord::LockWaitTimeout exceptions that the specific users were suffering for a long time. …

While I was operating a GKE cluster on which busy puma(Rails) servers running, our alert system reported an incident to my team. These manifested that our ruby processes had stopped doing any work unexpectedly.

What’s the problem?

The stuck process is an infrequent but common problem for many people, especially with large traffic applications. When it comes to debugging a process deeply, a GDB debugger (or anything that uses ptrace) can help.

However, as far as a process running inside the container of a Pod is concerned, it is not as straightforward as attaching gdb to the host PID of the container process…

I’m going to introduce the protolint.

A tool to enforce Protocol Buffer style and conventions.

- https://github.com/yoheimuta/protolint

prototool supports the rules following the Google style guide.

- https://developers.google.com/protocol-buffers/docs/style

❯❯❯ pl list
ENUM_NAMES_UPPER_CAMEL_CASE: Verifies that all enum names are CamelCase (with an initial capital).
FIELD_NAMES_LOWER_SNAKE_CASE: Verifies that all field names are underscore_separated_names.
MESSAGE_NAMES_UPPER_CAMEL_CASE: Verifies that all message names are CamelCase (with an initial capital).
RPC_NAMES_UPPER_CAMEL_CASE: Verifies that all rpc names are CamelCase (with an initial capital).
SERVICE_NAMES_UPPER_CAMEL_CASE: Verifies that all service names are CamelCase (with an initial capital).


❯❯❯ pl lint…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store