Nova makes use of a database to support its functionalities. If this database is not running or is improperly configured, it can cause your Nova installation to be unusable. To troubleshoot the Nova database, begin by making sure that the database engine is running. For example, when using MySQL, you can check the status by running the following command:

service mysql status

If the database is running, it will be indicated in the output of this command:

mysql start/running, process 15628

After confirming that the database engine is running, you should try to connect to the database server using the same parameters that are set up in the Nova configuration file. You can look up the connection information by running the following command:

less /etc/nova/nova.conf | grep "connection ="

The preceding command will search through the Nova configuration file, which is located at /etc/nova/nova.conf. In this file, we are looking for a variable named connection. This setting is typically under the stanza named [database]. The value of the variable will follow a format similar to the following format if you are using MySQL:

mysql://nova:novapwd@mydbserver/nova

From the preceding connection string, we see that our database server is running on the host named mydbserver with a database user named nova, and a password novapwd. We can use this information to attempt to connect to the database directly:

mysql -u nova -p -h mydbserver nova

On successful connection, a message similar to the one shown here will be printed on your console:

If the connection is unsuccessful, you will see a message like this one:

This error indicates that we need to troubleshoot further. There are several possible causes behind this error. A few potential reasons include the following:

To troubleshoot further, you will need root access to the database server. Attempt to connect to the database using your root user. Accomplishing this step will also allow you to confirm the value for the database host:

mysql -u root -p -h mydbserver

Once you have logged into the database server as a root user, you will want to check the permissions for the Nova database user. An example of how to do this is included here:

show grants for 'nova'@'locahost'

The output of the preceding command will be similar to what is printed in the following screenshot:

The nova user needs to be granted all permissions on the Nova database. After confirming that the nova user has the correct permissions, you want to make sure that the user's password is correct. You can change the nova user's password by executing the following set of commands:

After executing this series of commands, the password for the nova user will be updated. Make sure that this new value is properly reflected in the connection variable under the [database] stanza in the /etc/nova/nova.conf file.

After you have confirmed that your nova database user is set up correctly, you will want to make sure that the Nova database is properly initialized. The first step is to make sure that the Nova database exists. In MySQL, you can check this by running the following command:

show databases;

Make sure you see a database named nova in the list:

If the Nova database does not exist, you need to create it. You can do this using the standard SQL syntax. Take the following line of code as an example:

mysql> CREATE DATABASE nova;

Once you have confirmed that the database exists, you need to make sure that the required tables are present in the database:

mysql> show tables;

After running the preceding command, you will see list of tables in the Nova database. The following output is a partial list of tables. There are around 100 tables in total.

When you run show tables and the command comes back empty, you will need to initialize your Nova database:

nova-manage db sync

The preceding command does not return an output, but it will create the necessary tables in the Nova database. Confirm this by running the show tables command again, after which you should see that your database is populated.

The most obvious dependency Nova has is Keystone. Nova leverages Keystone to provide authentication services for all the Nova API calls. So, if Keystone is not running as expected, you will come to know this as soon as you attempt to use the OpenStack client:

openstack server list

If you try to use the preceding command to list the Nova servers while Keystone is not working, you will see an error like the one shown here:

Keystone up

We will begin troubleshooting the integration between Nova and Keystone by making sure that Keystone is up and running:

ps -aux | grep keystone

This command should return an output similar to what is shown in the following screenshot:

If you do not see any Keystone processes running, refer to Chapter 2, Troubleshooting OpenStack Identity, in this book for Keystone troubleshooting tips.

Setting up the service user

Next, we will make sure that the Nova service user is set up correctly. By convention, this service user will be named nova. Confirm that this user exists by running the following command:

openstack user list

Running this command will return a list of users in Keystone. An example of the output is included in this screenshot:

Make sure that there is a user named nova. Secondly, this user should belong to a project/tenant named service, by convention. Make sure that the service project exists by running the following command:

openstack project list

This command will list the projects (tenants) configured in our Keystone installation:

In the output of this command, you should see a project named service, as demonstrated in the preceding screenshot. Next, we need to confirm that the nova user is assigned to the service project. We can accomplish this using the next command:

openstack user show nova

The preceding user show command will display details for the nova user as shown here:

In the project_id field, we are looking for a project ID that matches the ID of our service project displayed in the preceding OpenStack project-list command. The project ID we are looking for is 831669f19f3b4505a3b8b132e88a3b36. The output tells us that our nova user is indeed a part of the service project.

Next, confirm that the nova user has the correct roles in Keystone. The following command can help us with that:

openstack user role list --project service nova

This command will list the roles for the glance user in the context of the service project:

Confirm that the nova user has the admin role in the service project.

Finally, confirm the password for the nova user. Use the OpenStack command-line client to authenticate as the nova user. Run a command similar to the one shown here:

openstack --os-username nova --os-password nova --os-project-name service project list

The preceding command is the Openstack project list command, with a few arguments passed in on the command line. The arguments passed in will override the values we have in our openrc file. This allows us to test authentication with different users:

• --os-username: This indicates the username of the user we want to use. For this test, the username is nova.
• --os-password: This is the value we are targeting with this test. We want to make sure that the password is set and working as expected. For my test, I'm expecting the password to be nova.
• --os-project-name: Finally, we need to pass in the project name so that Keystone knows which project to authenticate under. Our user may have different roles under different projects. In our example, we are testing the nova user under the service project.

If the password for the nova user is equal to nova, as I expect, this command will return a list of projects in keystone, as follows:

Suppose that I get an error like the one shown here:

The request you have made requires authentication. (HTTP 401) (Request-ID: req-707fd197-ae15-4698-ac04-1de4c92acfe9)

This error tells me that the password is not nova, as I anticipated. At this point, I can try a few more password possibilities or I can reset the password. To reset the nova user's password, you can use a command like the one shown here:

openstack user set --password-prompt nova

After running this command, you will be prompted to enter a new password for the nova user. You will then be asked to type that password a second time to confirm it. After you reset the password, you can test it as we did earlier.

Finally, we need to confirm that our credentials are set properly in the Nova configuration files. In the /etc/nova/nova.conf file, there should be a keystone_authtoken stanza. Under this stanza, make sure that the values of the admin_user and admin_password fields match with the values you have set in Keystone. Consider the following example:

In the preceding example, admin_ user is set to nova and admin_password is set to nova. Replace these values with the username and password you have configured for your installation.

Service endpoints correct

At this stage, we should confirm that the service catalog contains the correct endpoints for the Nova service with the following command:

openstack endpoint show nova

When you run the endpoint show command, as demonstrated in the preceding code, you will see the details for the Nova endpoint in the service catalog. An example of a typical output is included in the following screenshot:

Confirm that the values of the adminurl, internalurl, and publicurl fields are correct. If any of these values are incorrect, you have the following three options to correct the issue:

• You can use the OpenStack endpoint delete command to remove the current endpoints, followed by the OpenStack endpoint create command.
• If you are using the default SQL driver for your service catalog, you can make the edits to the endpoints directly in the SQL database. Use caution when executing this option.
• If you are using the templated driver for your service catalog, you simply need to update your endpoint template file.

Nova depends on Glance when retrieving images while launching instances. If there is an issue that you're facing with Glance, you will most likely see it quickly when you attempt to launch a new instance using the openstack server create command:

openstack server create --flavor m1.tiny --image cirros-qcow2 server1

When this command is executed successfully, it will return a summary of the instance being created. If there is a problem with the Glance and Nova configurations, then you will see an error like the one shown here:

Could not find resource cirros-qcow2

If you run the same command with the --debug flag, you will see a more detailed error message like the one shown in the following screenshot:

This error clearly points to the fact that glance-api is not running or Glance is not configured correctly in the nova.conf file. The Glance configuration for Nova can be found under the [glance] stanza in /etc/nova/nova.conf as shown here:

As demonstrated in the preceding code, you need to confirm the value of the host parameter for Glance. Once you confirm the Glance configuration in nova.conf, refer to Chapter 3, Troubleshooting the OpenStack Image Service, for more tips on Glance troubleshooting.

Nova also depends on Neutron to provide networking for the instances. Nova will make API calls to Neutron after authenticating via Keystone. When attempting to launch an instance, you may come across and error like the one shown in the following screenshot:

Upon further troubleshooting, you may see errors in nova-api.log similar to the ones shown here:

If you run across a similar error, make sure that the Neutron server is running and configured correctly in the nova.conf file.

The preceding screenshot shows an example of the Neutron configuration in the nova.conf file. Make sure that this stanza exists, and the username and password values are correct for the Neutron service user. Once you have confirmed this, you can review the Chapter 4, Troubleshooting OpenStack Networking, for more tips.