While running Rails puma servers in production, we were seeing the incident that some old worker processes suddenly got stuck regardless of no change in the amount or trend of requests.
I found out the root cause and reported it to the upstream. This issue still exists in Ruby 2.6.0 and can be found as far back as Ruby 2.5.0. If you just want a summary of the bug, see ruby-lang#17669.
One day, we noticed many
ActiveRecord::LockWaitTimeout exceptions that the specific users were suffering for a long time. …
While I was operating a GKE cluster on which busy puma(Rails) servers running, our alert system reported an incident to my team. These manifested that our ruby processes had stopped doing any work unexpectedly.
The stuck process is an infrequent but common problem for many people, especially with large traffic applications. When it comes to debugging a process deeply, a GDB debugger (or anything that uses
ptrace) can help.
However, as far as a process running inside the container of a Pod is concerned, it is not as straightforward as attaching gdb to the host PID of the container process…
I’m going to introduce the protolint.
A tool to enforce Protocol Buffer style and conventions.
prototool supports the rules following the Google style guide.
❯❯❯ pl list
ENUM_FIELD_NAMES_UPPER_SNAKE_CASE: Verifies that all enum field names are CAPITALS_WITH_UNDERSCORES.
ENUM_NAMES_UPPER_CAMEL_CASE: Verifies that all enum names are CamelCase (with an initial capital).
FIELD_NAMES_LOWER_SNAKE_CASE: Verifies that all field names are underscore_separated_names.
MESSAGE_NAMES_UPPER_CAMEL_CASE: Verifies that all message names are CamelCase (with an initial capital).
RPC_NAMES_UPPER_CAMEL_CASE: Verifies that all rpc names are CamelCase (with an initial capital).
SERVICE_NAMES_UPPER_CAMEL_CASE: Verifies that all service names are CamelCase (with an initial capital).
❯❯❯ pl lint…