Search in book...
Toggle Font Controls
Create new playlist

Name your new playlist

Playlist description (optional)
Sign In

Email address

Password

Forgot Password?

or

Continue with Facebook

Continue with Google
Sign Up

Full Name

Email address

Confirm Email Address

Password

or

Continue with Facebook

Continue with Google

A. MarkelovCertified OpenStack Administrator Study Guide Certification Study Companion Serieshttps://doi.org/10.1007/978-1-4842-8804-7_10

10. Troubleshooting

Andrey Markelov¹

(1)

Stockholm, Sweden

This chapter does not cover one particular topic of the Certified OpenStack Administrator exam requirements. Instead, it shows general troubleshooting practices for OpenStack. Please note that backing up OpenStack instances is discussed in Chapter 9, and analyzing storage status is discussed in Chapters 5, 7, and 9.

The Main Principles of Troubleshooting

Troubleshooting OpenStack is not often straightforward because it consists of many separate projects that work with one another in different combinations. That is why the troubleshooting discussion is near the end of this book. You need to know the previous material before you learn the troubleshooting techniques.

It is good to stick to the scientific method during the troubleshooting exercise. The following are the primary steps to structure it as an algorithm.

1.
Define the problem clearly as you can. You may want to make sure the issue is reproducible.
2.
Collect all relevant information. It can be log files, debug output, etc.
3.
Form a hypothesis based on observed symptoms.
4.
Test the hypothesis. You can start from the easiest to test or from a theory with a higher probability or mix.
5.
Try to fix issues based on tests.
6.
The process must be restarted from the top of the proposed fixes that do not resolve the case.

You should be aware of the concerns about generic debugging. Always make a backup copy of your configuration file before you begin changing it. It is very important to make only one change at a time. Finally, do not forget to revert your configuration files to the original if any test is unsuccessful.

OpenStack troubleshooting techniques depend to a certain extent on general GNU/Linux troubleshooting skills. That discussion is outside the scope of this book. However, Table 10-1 summarizes the main GNU/Linux troubleshooting utilities.

Table 10-1

Basic GNU/Linux Troubleshooting Commands

GNU/Linux Command	Useful Options and Examples
ps: Report list of the current processes	To see every process on the system, use the aux option. It can be useful with the grep command for searching exact processes, or you can use pgrep.
# ps aux \| grep cinder cinder 1400 0.6 0.3 354936 126036 ? Ss 08:52 1:19 /usr/bin/python3 /usr/bin/cinder-api --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/api.log
top: Shows a dynamic view of the system processes. Unlike the ps output, this command continuously refreshes the view.	You can use interactive keystrokes in the top environment. ? - help, q - quit, l - toggles for load header line, t - toggles for threads header line, m - toggles for memory header line, u - filter process for user name, M - sorts process listing by memory usage in descending order, P - sorts process listing by processor utilization in descending order, k - kill a process.
df: Reports file system disk space usage.	Usually, df is used with the -h option, which means human-readable format (e.g., 1K 234M 2G)
# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 16G 0 16G 0% /dev tmpfs 16G 4.0K 16G 1% /dev/shm tmpfs 6.3G 50M 6.3G 1% /run /dev/mapper/cs-root 60G 9.0G 51G 15% / /dev/vda1 1014M 266M 749M 27% /boot /dev/mapper/cs-home 30G 331M 29G 2% /home
du: Estimate file space usage	The same -h option as in df is often used.
# du -h /var/lib/glance/images/ 889M /var/lib/glance/images/
ip: Show/manipulate routing, devices, policy routing, and tunnels	The most common subcommands are show - for displaying IP information, route - for showing routing information.
# ip addr show ens3 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:4e:51:9f brd ff:ff:ff:ff:ff:ff altname enp0s3 inet 192.168.122.10/24 brd 192.168.122.255 scope global dynamic noprefixroute ens3 valid_lft 2923sec preferred_lft 2923sec inet6 fe80::5054:ff:fe4e:519f/64 scope link noprefixroute valid_lft forever preferred_lft forever
ss and netstat: Utilities to investigate sockets	The ss command is similar to the netstat command and is used to display socket statistics. They have similar options. Options are -t - show TCP sockets, -u - show UDP sockets, -a - show listening and established sockets, -p - show process using the sockets.
# ss -ta State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 :8776 :* LISTEN 0 128 :25672 :* LISTEN 0 128 :8777 :* LISTEN 0 128 10.0.2.15:27017 : LISTEN 0 64 10.0.2.15:rsync : LISTEN 0 50 :mysql :*
find: Search for files in a directory hierarchy	There are many options for the find utility: -name - find by name, -iname - like -name, but the match is case insensitive, -group and -user - find file that belongs to group or user, -type with f or d to find only files or directories.
# find /etc -name swift* /etc/swift /etc/swift/swift.conf /etc/logrotate.d/openstack-swift

Note

Traditionally, network interfaces are enumerated as eth0,1,2... In most modern Linux distributions, the default naming behavior can differ. The names of interfaces can be based on device topology, type, and firmware. For example, the ethernet interface on PCI slot 0 and port 3 can be named enp0s3.

Checking the OpenStack Version

It is always good to know which version of the OpenStack environment you are working with. Before the Liberty version, all projects except Swift had a version based on the year and month. Starting with Liberty, all components have a traditional version structure, X.Y.Z., where X is always the same in one release.

Here is an example of Yoga.

# keystone-manage --version

21.0.0

# nova-manage --version

25.0.0

The following is an example of the old-fashioned version convention used in OpenStack Kilo.

# keystone-manage --version

2015.1.0

# nova-manage --version

2015.1.0

Also, you can find the version on the System Information tab in the Admin menu at the right corner of the page bottom. In Table 10-2, several of the latest OpenStack releases are listed.

Table 10-2

OpenStack Releases

Series	Releases	Initial Release Date
Victoria	Nova 22.0; Keystone 18.0; Neutron 17.0; Swift 2.26	October 14, 2020
Wallaby	Nova 23.0; Keystone 19.0; Neutron 18.0; Swift 2.27	April 14, 2021
Xena	Nova 24.0; Keystone 20.0; Neutron 19.0; Swift 2.28	October 6, 2021
Yoga	Nova 25.0; Keystone 21.0; Neutron 20.0; Swift 2.29	March 30, 2022
Zed	Nova 26.0; Keystone22.0; Neutron 21.0; Swift 2.29	October 5, 2022

Finding and Analyzing Log Files

Usually in GNU/Linux systems, log files are persistently stored in the /var/log directory. These files can be viewed using regular text utilities such as less and tail. The following is an example of this directory’s content from the OpenStack controller node.

# ls /var/log --group-directories-first -F

anaconda/ horizon/ puppet/ boot.log-20220714 dnf.rpm.log secure

aodh/ httpd/ qemu-ga/ boot.log-20220717 firewalld secure-20220717.gz

audit/ keystone/ rabbitmq/ boot.log-20220718 hawkey.log spooler

ceilometer/ libvirt/ redis/ boot.log-20220720 hawkey.log-20220717 spooler-20220717.gz

chrony/ mariadb/ samba/ boot.log-20220721 kdump.log tallylog

cinder/ neutron/ speech-dispatcher/ boot.log-20220722 lastlog wtmp

cups/ nova/ sssd/ btmp maillog

gdm/ openvswitch/ swift/ cron maillog-20220717.gz

glance/ ovn/ swtpm/ cron-20220717.gz messages

gnocchi/ placement/ tuned/ dnf.librepo.log messages-20220717.gz

heat/ private/ boot.log dnf.log README@

As you see, parts of the content are directories, and other parts are files. If one of the services has more than one log file, usually, such logs are placed in their own subdirectory. For example, the /var/log/cinder/ directory contains five files for several Cinder subsystems. You see that the files’ names have -YYYYMMDD at the end. They are compressed by the Gzip tool. Usually, your lab must be one week old to find such compressed files. The logrotate utility renames, rotates, and compresses old logs. Instructions for logrotate are stored in the /etc/logrotate.d/ directory and /etc/logrotate.conf contains the configuration file.

The logging subsystem of GNU/Linux is based on the syslog protocol. In modern distributions the rsyslog and journald services in charge of logging.

The systemd-journald service is based on the operating system event logging architecture. It collects event messages from different sources like the Linux kernel, standard output, and standard error from daemons, boot messages, and syslog events. Then it converts them into a common format and writes them into a structured, indexed system journal. The rsyslog service reads syslog messages received by systemd-journald from the journal. It then records them to its log files under the /var/log directory or forwards them to other services according to its configuration.

There are some well-known system-wide log files.

messages: Most of the syslog messages are stored in this file.
secure: All authentication-related and security messages are stored here.
cron: The log file related to periodically executed jobs.

In general, all syslog messages are categorized by type and priority. Priority can be from 0 (the system is unusable) to 7 (debug-level message). The type can be mail, cron, authpriv, etc. The RULES section of the configuration file /etc/rsyslog.conf contains directives that define where log messages are saved. The rules include type, dot symbol, priority, and destination. The following is part of a default configuration file with rules.

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.* /dev/console

# Log anything (except mail) of level info or higher.

# Don't log private authentication messages!

*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.

authpriv.* /var/log/secure

# Log all the mail messages in one place.

mail.* -/var/log/maillog

# Log cron stuff

cron.* /var/log/cron

# Everybody gets emergency messages

*.emerg :omusrmsg:*

# Save news errors of level crit and higher in a special file.

uucp,news.crit /var/log/spooler

All log entries in log files are managed by the rsyslog and stored in a standard format.

2022-07-22 20:32:19.493 1388 INFO nova.compute.manager [req-15841da6-61ab-47cd-bfef-cf5b7179474e - - - - -] [instance: f6fda94b-a6d2-43cc-8e93-18a538759a22] VM Resumed (Lifecycle Event)

The first part of the message is the timestamp, then the priority, the name of the host, then the name of the program that sends the message, and the last part is a message.

A -f /var/log/logfilename command tail is useful for real-time log monitoring. This command prints the last ten lines of a log and continues to output new lines as they are added to this log file.

Backing up the Database Used by an OpenStack Instance

In most common cases, all OpenStack databases are on one MariaDB server. It is very easy to create a database backup then.

# mysqldump --opt --all-databases > /tmp/all-openstack.sql

Tip

The mysqldump command asks for a password. You can avoid this by adding the –p option with the password; for example, –p apress.

You can run the following if you only want to back up a single database.

# mysqldump --opt neutron > /tmp/neutron.sql

To list all database names, you can use mysql CLI.

# mysql

Welcome to the MariaDB monitor. Commands end with ; or g.

Your MariaDB connection id is 148482

Server version: 10.5.16-MariaDB MariaDB Server

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MariaDB [(none)]> show databases;

+--------------------+

| Database |

+--------------------+

| aodh |

| cinder |

| glance |

| gnocchi |

| heat |

| information_schema |

| keystone |

| mysql |

| neutron |

| nova |

| nova_api |

| nova_cell0 |

| performance_schema |

| placement |

| test |

+--------------------+

15 rows in set (0.001 sec)

Analyzing Host/Guest OS and Instance Status

The easiest way to check the status of OpenStack components like hosts and instances is by using the Horizon web client. The most general view of the cloud is found on the Overview tab on the Admin menu. If you are searching for information about the hypervisors, you need to use the view under Admin ➤ Compute ➤ Hypervisors. Both views are shown in Figures 10-1 and 10-2.

Usually, users can find their instances in the Project menu. The administrator can view almost all instances on the Instances tab of the Admin ➤ Compute➤ Instances menu. Figure 10-3 shows an example of the Instances page.

Of course, you can get the same information using a command line. Start by gathering information about the hosts. You can get the list of all hypervisors using the following command.

$ openstack host list

+----------------+-----------+----------+

| Host Name | Service | Zone |

+----------------+-----------+----------+

| rdo.test.local | conductor | internal |

| rdo.test.local | scheduler | internal |

| rdo.test.local | compute | nova |

+----------------+-----------+----------+

$ nova hypervisor-list

+--------------------------------------+---------------------+-------+---------+

+--------------------------------------+---------------------+-------+---------+

+--------------------------------------+---------------------+-------+---------+

If you want more information about a specific host, the openstack host show command may help.

$ openstack host show rdo.test.local

+----------------+----------------------------------+-----+-----------+---------+

+----------------+----------------------------------+-----+-----------+---------+

| rdo.test.local | (total) | 8 | 32063 | 59 |

| rdo.test.local | (used_now) | 2 | 1536 | 2 |

| rdo.test.local | (used_max) | 2 | 1024 | 2 |

| rdo.test.local | 9e0c535c2240405b989afa450681df18 | 2 | 1024 | 2 |

+----------------+----------------------------------+-----+-----------+---------+

With the nova command, you can get the list of all instances that are hosting each host. The following is an example.

$ nova hypervisor-servers rdo.test.local

+--------------------------------------+-------------------+----------------------------------------+---------------------+

+--------------------------------------+-------------------+----------------------------------------+---------------------+

+--------------------------------------+-------------------+----------------------------------------+---------------------+

To print the list of virtual machines, you could use the openstack command.

$ openstack server list

+--------------------------------------+-----------------+--------+---------------------+---------------------+---------+

+--------------------------------------+-----------------+--------+---------------------+---------------------+---------+

+--------------------------------------+-----------------+--------+----------------------+---------------------+------------+

If you want to get all information regarding a specific instance, use the following command.

$ openstack server show apressinst3

+-----------------------------+------------------------------------------+

| Field | Value |

+-----------------------------+------------------------------------------+

| OS-DCF:diskConfig | MANUAL |

| OS-EXT-AZ:availability_zone | nova |

| OS-EXT-STS:power_state | Running |

| OS-EXT-STS:task_state | None |

| OS-EXT-STS:vm_state | active |

| OS-SRV-USG:launched_at | 2022-07-21T12:25:22.000000 |

| OS-SRV-USG:terminated_at | None |

| accessIPv4 | |

| accessIPv6 | |

| addresses | demo-net=172.16.0.48 |

| config_drive | |

| created | 2022-07-21T12:25:19Z |

| flavor | m1.tiny (1) |

| hostId | 1ea0155a80f503a2aad8752d0e77968a5f96a9975c97a6985389a988 |

| id | b360f5a5-b528-4f77-bdc7-3676ffcf0dff |

| image | cirros-0.5.2-x86_64 (7ffe1b43-7e86-4ad0-86b6-9fffa38b3c20) |

| key_name | apresskey1 |

| name | apressinst3 |

| progress | 0 |

| project_id | 9e0c535c2240405b989afa450681df18 |

| properties | |

| security_groups | name='apress-sgroup' |

| status | ACTIVE |

| updated | 2022-07-22T07:48:32Z |

| user_id | a20b5a5995b740ff90034297335b330a |

| volumes_attached | |

+-----------------------------+------------------------------------------+

Analyzing Messaging Servers

As mentioned earlier, a messaging server is used by almost all OpenStack services. Nowadays, the most common messaging server for OpenStack is RabbitMQ. Alternatives for RabbitMQ are Qpid and ZeroMQ. For transmitting information between OpenStack services, these servers use AMQP (Advanced Message Queuing Protocol). Let’s briefly go over the functions of RabbitMQ.

To check RabbitMQ’s status, you can use the following command.

# rabbitmqctl status

Status of node rabbit@rdo ...

Runtime

OS PID: 1430

OS: Linux

Uptime (seconds): 20151

Is under maintenance?: false

RabbitMQ version: 3.9.10

Node name: rabbit@rdo

Erlang configuration: Erlang/OTP 24 [erts-12.1.5] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Erlang processes: 1656 used, 1048576 limit

Scheduler run queue: 1

Cluster heartbeat timeout (net_ticktime): 60

...

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@rdo

Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@rdo/quorum/rabbit@rdo

Config files

* /etc/rabbitmq/rabbitmq.config

Log file(s)

* /var/log/rabbitmq/[email protected]

* /var/log/rabbitmq/rabbit@rdo_upgrade.log

* <stdout>

Memory

Total memory used: 0.3656 gb

Calculation strategy: rss

Memory high watermark setting: 0.4 of available memory, computed to: 13.4485 gb

...

Free Disk Space

Low free disk space watermark: 0.05 gb

Free disk space: 54.596 gb

Totals

Connection count: 104

Queue count: 116

Virtual host count: 1

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication

Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

To list all users, use the following command.

# rabbitmqctl list_users

Listing users ...

User tags

Guest [administrator]

As you see in this demo environment, only one user guest has administrator rights. All OpenStack services use that particular user for sending and receiving messages. To check whether you can find RabbitMQ settings in the services config files.

# grep 'transport_url=rabbit' /etc/glance/glance-api.conf

transport_url=rabbit://guest:[email protected]:5672/

# grep 'transport_url=rabbit' /etc/nova/nova.conf

transport_url=rabbit://guest:[email protected]:5672/

# grep 'transport_url=rabbit' /etc/cinder/cinder.conf

transport_url=rabbit://guest:[email protected]:5672/

# grep 'transport_url=rabbit' /etc/neutron/neutron.conf

transport_url=rabbit://guest:[email protected]:5672/

This is a common part of most configuration files. For managing and monitoring the RabbitMQ server, you can activate the graphical web console.

# /usr/lib/rabbitmq/bin/rabbitmq-plugins enable rabbitmq_management

Enabling plugins on node rabbit@rdo:

rabbitmq_management

The following plugins have been configured:

rabbitmq_management

rabbitmq_management_agent

rabbitmq_web_dispatch

Applying plugin configuration to rabbit@rdo...

The following plugins have been enabled:

rabbitmq_management

rabbitmq_management_agent

rabbitmq_web_dispatch

set 3 plugins.

Offline change; changes will take effect at broker restart.

# systemctl restart rabbitmq-server.service

Then open the web browser and point it to http://name-of-server:15672 on the RabbitMQ server host. A screenshot of the console is shown in Figure 10-4. The login name is guest, and the password is also guest by default.

Analyzing Network Status

First, you can check the list of processes that build up the Neutron service.

# pgrep -l neutron

971 neutron-meterin

984 neutron-server

988 neutron-metadat

1731 neutron-l3-agen

1732 neutron-openvsw

1734 neutron-dhcp-ag

1825 neutron-rootwra

2164 neutron-ns-meta

# pgrep -l neutron

1371 neutron-ovn-met

...

2813 neutron-server:

...

The exact list of processes can be different and depends on the host configuration and the type of SDN used. As you know, Neutron works through a lot of agents or plug-ins. In Horizon, the status of these plug-ins is listed on the Network Agents tab of the System Information view, as shown in Figure 10-5.

Figure 10-5
OpenStack Neutron agents list

The same information can be retrieved at the command line using the neutron command.

openstack network agent list

+--------------------------------------+----------------------+----------------+-------------------+-------+-------+----------------------------+

+--------------------------------------+----------------------+----------------+-------------------+-------+-------+----------------------------+

+--------------------------------------+----------------------+----------------+-------------------+-------+-------+----------------------------+

Neutron has a log file for each agent and OVN in separate directories.

# ls /var/log/neutron/ /var/log/ovn

/var/log/neutron/:

neutron-ovn-metadata-agent.log server.log

/var/log/ovn:

ovn-controller.log ovsdb-server-nb.log

ovn-northd.log ovsdb-server-sb.log

Let’s look at the virtual networks part of the OpenStack configuration. And again, you can explore them in Horizon, as shown in Figure 10-6, or you can use a command line.

$ openstack network list

+--------------------------------------+----------+--------------------------------------+

| ID | Name | Subnets |

+--------------------------------------+----------+--------------------------------------+

| 5ee4e933-de9b-4bcb-9422-83cc0d276d33 | demo-net | 18736455-80f6-4513-9d81-6cedbfe271fe |

| 5f18929b-70f6-4729-ac05-7bea494b9c5a | ext-net | d065c027-bb60-4464-9619-7d9754535c5c |

+--------------------------------------+----------+--------------------------------------+

You can click the network name to get more information on a specific network. The relevant screenshot to show this is Figure 10-7. If you prefer CLI, use the following command.

$ openstack network show demo-net

+---------------------------+--------------------------------------+

| Field | Value |

+---------------------------+--------------------------------------+

| admin_state_up | UP |

| availability_zone_hints | |

| availability_zones | |

| created_at | 2022-07-17T18:12:02Z |

| description | |

| dns_domain | None |

| id | 5ee4e933-de9b-4bcb-9422-83cc0d276d33 |

| ipv4_address_scope | None |

| ipv6_address_scope | None |

| is_default | None |

| is_vlan_transparent | None |

| mtu | 1442 |

| name | demo-net |

| port_security_enabled | True |

| project_id | 9e0c535c2240405b989afa450681df18 |

| provider:network_type | geneve |

| provider:physical_network | None |

| provider:segmentation_id | 30 |

| qos_policy_id | None |

| revision_number | 2 |

| router:external | Internal |

| segments | None |

| shared | False |

| status | ACTIVE |

| subnets | 18736455-80f6-4513-9d81-6cedbfe271fe |

| tags | |

| updated_at | 2022-07-17T18:14:52Z |

+---------------------------+--------------------------------------+

In the previous command you could use the network name, in this case demo-net, or the network ID. Figure 10-8 provides the list of virtual routers.

The corresponding CLI command to find the list of routers is.

$ openstack router list

+--------------------------------------+-------------+--------+-------+-----------------------------------+

+--------------------------------------+-------------+--------+-------+-----------------------------------+

+--------------------------------------+-------------+--------+-------+-----------------------------------+

$ openstack router show demo-router

+-------------------------+---------------------------------------------------------------+

| Field | Value |

+-------------------------+---------------------------------------------------------------+

| admin_state_up | UP |

| availability_zone_hints | |

| availability_zones | |

| created_at | 2022-07-17T18:16:19Z |

| description | |

| external_gateway_info | {"network_id": "5f18929b-70f6-4729-ac05-7bea494b9c5a", "external_fixed_ips": [{"subnet_id": "d065c027-bb60-4464-9619-7d9754535c5c", "ip_address": "192.168.122.208"}], "enable_snat": true} |

| flavor_id | None |

| id | 3daad728-4075-49a4-ad05-3b279de738fa |

| interfaces_info | [{"port_id": "5bcc90fc-9a54-4c91-9e6a-988ac0a4a4a8", "ip_address": "172.16.0.1", "subnet_id": "18736455-80f6-4513-9d81-6cedbfe271fe"}] |

| name | demo-router |

| project_id | 9e0c535c2240405b989afa450681df18 |

| revision_number | 3 |

| routes | |

| status | ACTIVE |

| tags | |

| updated_at | 2022-07-17T18:21:55Z |

+-------------------------+---------------------------------------------------------------+

For enumerating the list of ports, use the opensrtack port list command. The openstack port show command shows the details of the port-by-port ID.

$ openstack port list

+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------+--------+

$ openstack port show 5bcc90fc-9a54-4c91-9e6a-988ac0a4a4a8

+-------------------------+-------------------------------------------------------------+

| Field | Value |

+-------------------------+-------------------------------------------------------------+

| admin_state_up | UP |

| allowed_address_pairs | |

| binding_host_id | |

| binding_profile | |

| binding_vif_details | |

| binding_vif_type | unbound |

| binding_vnic_type | normal |

| created_at | 2022-07-17T18:17:36Z |

| data_plane_status | None |

| description | |

| device_id | 3daad728-4075-49a4-ad05-3b279de738fa |

| device_owner | network:router_interface |

| device_profile | None |

| dns_assignment | None |

| dns_domain | None |

| dns_name | None |

| extra_dhcp_opts | |

| fixed_ips | ip_address='172.16.0.1', subnet_id='18736455-80f6-4513-9d81-6cedbfe271fe' |

| id | 5bcc90fc-9a54-4c91-9e6a-988ac0a4a4a8 |

| ip_allocation | None |

| mac_address | fa:16:3e:2d:7b:4d |

| name | |

| network_id | 5ee4e933-de9b-4bcb-9422-83cc0d276d33 |

| numa_affinity_policy | None |

| port_security_enabled | False |

| project_id | 9e0c535c2240405b989afa450681df18 |

| propagate_uplink_status | None |

| qos_network_policy_id | None |

| qos_policy_id | None |

| resource_request | None |

| revision_number | 3 |

| security_group_ids | |

| status | ACTIVE |

| tags | |

| trunk_details | None |

| updated_at | 2022-07-17T18:17:37Z |

+-------------------------+-------------------------------------------------------------+

Note

For real-world network problems and troubleshooting, the utility plotnetcfg can be useful. It creates a network configuration diagram that can be visualized with the help of the dot utility. For more information, check the project website (https://github.com/jbenc/plotnetcfg).

The Network Topology tab is probably the best place to look for a project-level network overview. Figure 10-9 shows an example of the information provided on that page.

Figure 10-9
Project Network Topology tab

Digesting the OpenStack Environment

All OpenStack services are deployed as GNU/Linux daemons. Part of these services is represented as a single daemon, and part of them consists of two or more services. The best place to look for the status of a service is on the System Information tab on the Admin menu, as shown in Figure 10-10.

Figure 10-10
OpenStack services information

All the services’ information and statuses are shown. To do the same thing with a command line for this demo environment based on CentOS Stream, you can use the systemctl command.

# systemctl | grep openstack

openstack-aodh-evaluator.service loaded active running OpenStack Alarm evaluator service

openstack-aodh-listener.service loaded active running OpenStack Alarm listener service

openstack-aodh-notifier.service loaded active running OpenStack Alarm notifier service

openstack-ceilometer-notification.service loaded active running OpenStack ceilometer notification agent

openstack-ceilometer-polling.service loaded active running OpenStack ceilometer polling agent

openstack-cinder-api.service loaded active running OpenStack Cinder API Server

...

In the Horizon web client, you can also check the status of the computer services and block storage services on separate subtabs on the System Information tab. Respective examples are shown in Figures 10-11 and 10-12.

Figure 10-12
OpenStack block storage services

Summary

A troubleshooting topic is bigger than one particular chapter or even book. Solid knowledge of GNU/Linux is needed. In the past, troubleshooting was a separate topic of the exam. In general, you still need troubleshooting skills as part of the exam.

Review Questions

1.
Which do you use to search for the identity service configuration files in a configuration directory hierarchy?
1. A.
  find /etc -name *keystone.*
2. B.
  find /etc --name heat.*
3. C.
  find /var --name *keystone.*
4. D.
  find / --name heat.*
2.
Where would you find the messages from the Cinder service? (Choose all that are applicable.)
1. A.
  /var/log/messages
2. B.
  /var/log/cinder/api.log
3. C.
  /var/log/cinder/scheduler.log
4. D.
  /var/log/cinder/backup.log
3.
Which backs up all the OpenStack databases?
1. A.
  mysqlbackup --opt --all-db > /tmp/all-openstack.sql
2. B.
  mysqlbackup --opt --all-databases > /tmp/all-openstack.sql
3. C.
  mysqldump --opt --all-db > /tmp/all-openstack.sql
4. D.
  mysqldump --opt --all-databases > /tmp/all-openstack.sql
4.
Which enumerates all the compute hosts? (Choose all that are applicable.)
1. A.
  openstack hypervisor list
2. B.
  openstack host list
3. C.
  nova host-enumerate
4. D.
  nova hypervisor-list
5.
Which provides a list of all virtual machines?
1. A.
  openstack vm list
2. B.
  openstack server list
3. C.
  openstack host list
4. D.
  openstack instance list
6.
Which checks the status of the RabbitMQ messaging server?
1. A.
  rabbitmqctl stat
2. B.
  rabbitmq status
3. C.
  rabbitmqctl status
4. D.
  rabbitmq state
7.
Which checks the status of the Neutron agents?
1. A.
  openstack network agent list
2. B.
  neutron plugin-list
3. C.
  openstack agent list
4. D.
  openstack network list
8.
Which gives the details of a given router?
1. A.
  neutron router list router
2. B.
  neutron router show router
3. C.
  openstack router list router
4. D.
  openstack router show router

Answers

1.
A
2.
A, B, C, D
3.
D
4.
B, D
5.
B
6.
C
7.
A
8.
D

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.

Table of Contents for 10. Troubleshooting

Create new playlist

Sign In

Sign Up

10. Troubleshooting

The Main Principles of Troubleshooting

Checking the OpenStack Version

Finding and Analyzing Log Files

Backing up the Database Used by an OpenStack Instance

Analyzing Host/Guest OS and Instance Status

Analyzing Messaging Servers

Analyzing Network Status

Digesting the OpenStack Environment

Summary

Review Questions

Answers

Table of Contents for
10. Troubleshooting