The following output is what a Unicorn worker timeout looks like in unicorn_stderr.log. This is not necessarily bad; it just means that a new worker is spawned:
[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing
[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10
[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538
[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready
It could be that there are just not enough Unicorn workers available to respond to the requests at hand. NGINX buffers a lot of requests so we must check on the handover socket whether Unicorn can keep up. To do this, a little nifty script is available here: https://github.com/jahio/unicorn-status.
It can be called with the following command:
$ ruby unicorn_status.rb /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket 10
Running infinite loop. Use CTRL+C to exit.
------------------------------------------
Active Requests Queued Requests
20 11
The first argument here is the unicorn_status.rb script, the second is the socket to connect to ../.socket, and the last argument is the poll interval (10).