The remote code is the code that Ansible transports to a remote host to execute it. This is typically module code, or in the case of action_plugins, other snippets of code. Using the debugging method we discussed in the previous section to debug module execution will not work, as Ansible simply copies the code over and then executes it. There is no Terminal attached to the remote code execution, and thus there is no way to attach it to a debugging prompt, that is, without editing the module code.

To debug module code, we need to edit the module code itself to insert a debugger breakpoint. Instead of directly editing the installed module file, create a copy of the file in a library/ directory relative to the playbooks. This copy of the module code will be used instead of the installed file, which makes it easy to temporarily edit a module without disrupting other users of modules on the system.

Unlike other Ansible code, module code cannot be directly debugged with pdb, because the module code is assembled and then transported to a remote host. Thankfully, there is a solution in the form of a slightly different debugger named rpdb the Remote Python Debugger. This debugger has the ability to start a listening service on a provided port to allow remote connections into the Python process. Connecting to the process remotely will allow debugging the code line by line, just as we did with other Ansible code.

To demonstrate how this debugger works, first, we're going to need a remote host. For this example, we're using a remote host by the name of debug.example.com, and setting the IP address to a host that is already set up and waiting. Next, we need a playbook to execute a module that we'd like to debug:

---
- name: remote code debug
  hosts: debug.example.com
  gather_facts: false
  become: true

  tasks:
    - name: a remote module execution
      systemd:
        name: nginx
        state: stopped
        enabled: no

We will also need a new inventory file to reference our new test host—as I don't have DNS set up for this host, I'm using the special ansible_host variable in the inventory to tell Ansible which IP address to connect to debug.example.com on:

debug.example.com ansible_host=192.168.81.154

This play simply calls the systemd module to ensure that the nginx service is stopped and will not start up on boot. As we stated previously, we need to make a copy of the service module and place it in library/. The location of the service module to copy from will vary based on the way Ansible is installed. Typically, this module will be located in the modules/core/system/ subdirectory of where the Ansible Python code lives, like /usr/lib/python2.7/site-packages/ansible/modules/system/systemd.py on my system. Then, we can edit it to put in our breakpoint, as follows:

We'll put the breakpoint just before the systemctl variable value gets created, near line 318. First, the rpdb module must be imported (meaning that the rpdb Python library needs to exist on the remote host), and then the breakpoint needs to be created with set_trace().

Unlike the regular debugger, this function will open a port and listen for external connections. By default, the function will listen for connections to port 4444 on the address 127.0.0.1. However, that address is not exposed over the network, so in my example, I've instructed rpdb to listen on address 0.0.0.0, which is effectively every address on the host. Now, we can run this playbook to set up the server that will wait for a client connection:

Now that the server is running, we can connect to it from another Terminal. Connecting to the running process can be accomplished with the telnet program:

From this point on, we can debug as normal. The commands we used before still exist, such as list to show where in the code the current frame is:

Using the debugger, we can walk through the systemd module to track how it determines the path to the underlying tool, trace which commands are executed on the host, determine how a change is computed, and so on. The entire file can be stepped through, including any other external libraries the module may make use of, allowing for the debugging of other non-module code on the remote host as well.

If the debugging session allows the module to exit cleanly, the playbook's execution will return as normal. However, if the debugging session is disconnected before the module completes, the playbook will error, as shown in the following screenshot:

Because of this side effect, it is best to not exit the debugger early, and instead issue a continue command when your debugging is finished.