The ultimate way to debug Sidekiq processes is to make it dump a backtrace via GDB:
- Run sudo gdb -p (PID) to attach to the Sidekiq worker process.
- Run call (void) rb_backtrace() in the GDB console and find the generated Ruby backtrace in /var/log/gitlab/sidekiq/current:
2018-09-21_19:55:03.48430 from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:83:in `_read_from_socket'
2018-09-21_19:55:03.48431 from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:87:in `rescue in _read_from_socket'
2018-09-21_19:55:03.48432 from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/redis-3.3.5/lib/redis/connection/ruby.rb:87:in `select'
- It is very hard to read backtraces, but this process was doing network operations while being traced, we can see a (_read_from _socket). You can read the source code to check what it is doing (there are line numbers mentioned).
- When you are done, leave GDB with detach and quit.
You can also use other tracing tools to examine the behavior of the looping process. On Linux, for instance, strace -p <pid> allows you to view the system calls that are being made by the process.