The Linux operating system offers several native tools that are very useful for troubleshooting problems. The following are a few tools that you must be aware of to quickly identify problems with your OpenStack environment:

• ps:  The Linux ps command can be used to list the OpenStack services.  The output can be piped to grep to filter the service that we are interested in.
• top and htop: The top and htop commands will list the running processes and also provide information about how much CPU and memory they are consuming. You also can sort the processes based on the amount of CPU and memory consumed. This command comes in handy when identifying resource contention issues.
• df: The Linux df command provides the disk space information.
• fdisk: This command can be used to manipulate disk partition tables. It is useful when dealing with OpenStack Block Storage issues with the underlying disk.
• watch: The watch command provides a way to keep monitoring the output of a particular command and refreshing it with a specified interval. For example, you can monitor the output of the ps -aux | grep cinder command simply by using the watch ps -aux | grep cinder command.  You can also specify the -n option to the watch command to specify the time interval using which the output must be refreshed.
• arp: This display the ARP table. It is useful when a node is not able to find the address of other nodes.
• iptables: The iptables are firewall rules that can allow or disallow packets.
• tcpdump: This is one of the most commonly used tools to investigate networking issues. It allows you to capture packets that are sent or received on a given network interface, network, port or subnet. 

Apart from the previous native Linux tools, rabbitmqctl may also be very useful. It is a command-line utility that gets installed along with the RabbitMQ message queuing system.  Various components of OpenStack use the RabbitMQ  message queuing system to transfer messages between each other.  Hence, this utility is very useful for communication issues between components.  

For example, the following command will provide the status of the RabbitMQ message broker:

# rabbitmqctl status
 Status of node rabbit@ubuntu ...
 [{pid,1841},
 {running_applications,[{rabbit,"RabbitMQ","3.5.7"},
 {os_mon,"CPO CXC 138 46","2.4"},
 {mnesia,"MNESIA CXC 138 12","4.13.3"},
 {xmerl,"XML parser","1.3.10"},
 {sasl,"SASL CXC 138 11","2.7"},
 {stdlib,"ERTS CXC 138 10","2.8"},
 {kernel,"ERTS CXC 138 10","4.2"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang/OTP 18 [erts-7.3] [source] [64-bit] [async-threads:64] [kernel-poll:true]
"},
  .....
  .....
  .....
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,1657490636},
 {disk_free_limit,50000000},
 {disk_free,30812069888},
 {file_descriptors,[{total_limit,65436},
 {total_used,53},
 {sockets_limit,58890},
 {sockets_used,51}]},
 {processes,[{limit,1048576},{used,862}]},
 {run_queue,0},
 {uptime,585}]

You can start and stop the RabbitMQ service using the following command:

# sudo service rabbitmq-server start 
# rabbitmqctl stop

To list the queues in the the message broker, you can use the list_queues parameter to the command line:

# rabbitmqctl list_queues
cinder-scheduler 0
cinder-scheduler.ubuntu 0
cinder-scheduler_fanout_91143a2ceafc473f887e84d584ee543e 0
cinder-volume 0
cinder-volume.ubuntu@lvmdriver-1 0
[email protected] 0
cinder-volume.ubuntu@lvmdriver-1_fanout_a060ce20a78c4658905d622a00c767af 0
cinder-volume_fanout_d0775346883e406a8a54f658babbcb2f 0
....
....
q-server-resource-versions_fanout_8512feab67ea406d93181dd4207e550d 0
reply_60075b55450746d6b0c8b8cca4dc501a 0
reply_9dffe4691a3d44c091e9bdc727cc880a 0
reply_9f6e9e66e6254536828e471f00ab5b35 0
reply_c4c6251086be460a8d3166243a551110 0
scheduler 0
scheduler.ubuntu 0
scheduler_fanout_a172414abf5a40fb8510e0512f643502 0
versioned_notifications.info 2

Similarly, you can also list the exchanges that RabbitMQ uses by using the list_exchanges parameter. Exchanges are the entities responsible for delivering the messages from the producer to the appropriate queue:

# rabbitmqctl list_exchanges
Listing exchanges ...
amq.direct direct
amq.fanout fanout
amq.headers headers
amq.match headers
amq.rabbitmq.log topic
amq.rabbitmq.trace topic
amq.topic topic
cinder-scheduler_fanout fanout
cinder-volume.ubuntu@lvmdriver-1_fanout fanout
cinder-volume_fanout fanout
conductor_fanout fanout
......
......
......
q-l3-plugin_fanout fanout
q-plugin_fanout fanout
q-reports-plugin_fanout fanout
q-server-resource-versions_fanout fanout
reply_60075b55450746d6b0c8b8cca4dc501a direct
reply_9dffe4691a3d44c091e9bdc727cc880a direct
reply_9f6e9e66e6254536828e471f00ab5b35 direct
reply_c4c6251086be460a8d3166243a551110 direct
scheduler_fanout fanout

The bindings between exchanges and queues can be listed using the following command:

# rabbitmqctl  list_bindings
Listing bindings ...
        exchange cinder-scheduler queue cinder-scheduler []
        exchange cinder-scheduler.ubuntu queue cinder-scheduler.ubuntu []
        exchange cinder-scheduler_fanout_91143a2ceafc473f887e84d584ee543e queue cinder-scheduler_fanout_91143a2ceafc473f887e84d584ee543e []
        exchange cinder-volume queue cinder-volume []
        exchange cinder-volume.ubuntu@lvmdriver-1 queue cinder-volume.ubuntu@lvmdriver-1 []
        exchange [email protected] queue [email protected] []
        exchange cinder-volume.ubuntu@lvmdriver-1_fanout_a060ce20a78c4658905d622a00c767af queue cinder-
......
......
l3_agent_fanout_775c77054a5547eb8602caa85ff441c7 []
        exchange neutron-vo-Network-1.0 queue neutron-vo-Network-1.0 []
        exchange neutron-vo-Network-1.0.ubuntu queue neutron-vo-Network-1.0.ubuntu []
        exchange neutron-vo-Network-1.0_fanout_e90e68ed89d747bbb761bdd49ba2fc79 queue neutron-vo-Network-1.0_fanout_e90e68ed89d747bbb761bdd49ba2fc79 []
        exchange neutron-vo-Port-1.1 queue neutron-vo-Port-1.1 []
        exchange neutron-vo-Port-1.1.ubuntu queue neutron-vo-Port-1.1.ubuntu []

Finally, the connections to the RabbitMQ server from the clients can be listed by using the list_connections command:

# rabbitmqctl list_connections
Listing connections ...
stackrabbit 192.168.0.106 53856 running
stackrabbit 192.168.0.106 53858 running
.....
.....
stackrabbit 192.168.0.106 53958 running
stackrabbit 192.168.0.106 53960 running

stackrabbit 192.168.0.106 53964 running
stackrabbit 192.168.0.106 53966 running

Thus, the rabbitmqctl utility can be used with several parameters to identify any issues with the message queuing service.

Let's now delve into the OpenStack services and how we can troubleshooting individual services.