You will use the same technique to debug a 64-bit malware; the difference is, you will be dealing with extended registers, 64-bit memory addresses/pointers, and slightly different calling conventions. If you recall (from Chapter 4, Assembly Language and Disassembly Primer), a 64-bit code uses the FASTCALL calling convention and passes the first four parameters to the function in the registers (rcx, rdx, r8, and r9), and the rest of the parameters are placed on the stack. While debugging the call to a function/API, depending on the parameter you are inspecting, you will have to inspect the register or the stack. The calling convention mentioned previously is applicable to compiler-generated code. An attacker writing code in the assembly language need not follow these rules; as a result, the code can exhibit unusual behavior. When you come across code that is not compiler-generated, a further investigation of the code may be required.

Before we debug a 64-bit malware, let's try to understand the behavior of a 64-bit binary with the following trivial C program, which was compiled for the 64-bit platform using the Microsoft Visual C/C++ compiler:

int main()
{
  printf("%d%d%d%d%s%s%s", 1, 2, 3, 4, "this", "is", "test");
  return 0;
}

In the preceding program, the printf function takes eight arguments; this program was compiled and opened in x64dbg, and a breakpoint was set at the printf function. The following screenshot shows the program, which is paused before the call to the printf function. In the registers window, you can see that the first four parameters are placed in the rcx, rdx, r8, and r9 registers. When the program calls a function, the function reserves 0x20 (32 bytes) of space on the stack (space for four items, each 8 bytes in size); this is to make sure that the called function has the necessary space, if it needs to save the register parameters (rcx, rdx, r8, and r9). This is the reason the next four parameters (the 5^th, 6^th, 7^th, and 8^th parameters) are placed on the stack, starting from the fifth item (rsp+0x20). We are showing you this example to give you an idea of how to find the parameters on the stack:

In the case of a 32-bit function, the stack grows as the arguments are pushed, and shrinks when the items are popped. In a 64-bit function, the stack space is allocated at the beginning of the function, and does not change until the end of the function. The allocated stack space is used to store the local variables and the function parameters. In the preceding screenshot, note how the first instruction, sub rsp,48, allocates 0x48 (72) bytes of space on the stack, after which no stack space is allocated in the middle of the function; also, instead of using push and pop instructions, the mov instructions are used to place the 5^th, 6^th, 7^th, and 8^th parameters on the stack (highlighted in the preceding screenshot). The lack of push and pop instructions makes it difficult to determine the number of parameters accepted by the function, and it is also hard to say whether the memory address is being used as a local variable or a parameter to the function. Another challenge is, if the values are moved into the registers rcx and rdx before the function call, it's hard to say whether they are parameters passed to the function, or if they are moved into registers for any other reason.

Even though there are challenges in reverse engineering a 64-bit binary, you should not have too much difficulty analyzing the API calls, because the API documentation tells you the number of function parameters, the data types of the parameters, and what type of data they return. Once you have an idea of where to find the function parameters and the return values, you can set a breakpoint at the API call and inspect its parameters to understand the malware functionality.

Let's look at an example of a 64-bit malware sample that calls RegSetValueEx to set some value in the registry. In the following screenshot, the breakpoint is triggered before the call to the RegSetValueEx. You will need to look at the values in the registers and the stack window (as mentioned previously) to examine the parameters passed to the function; this will help you determine what registry value is set by the malware. In x64dbg, the easiest way to get a quick summary of function parameters is to look at the Default Window (below the registers window), which is highlighted in the following screenshot. You can set a value in the Default window to display the number of parameters. In the following screenshot, the value is set to 6, because from the API documentation (https://msdn.microsoft.com/en-us/library/windows/desktop/ms724923(v=vs.85).aspx), you can tell that the RegSetValueEx API takes 6 parameters:

The first parameter value, 0x2c, is the handle to the open registry key. Malware can open a handle to the registry key by calling either the RegCreateKey or RegOpenKey API. From the handles window, you can tell that the handle value 0x2c is associated with the registry key shown in the following screenshot. From the handle information, and through inspecting the 1^st, 2^nd, and 5^th parameters, you can tell that malware modifies the registry key, HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsNTCurrentVersionWinlogonshell, and adds an entry, "explorer.exe,logoninit.exe". On a clean system, this registry key points to explorer.exe (the default Windows shell). When the system starts, the Userinit.exe process uses this value to launch the Windows shell (explorer.exe). By adding logoninit.exe, along with explorer.exe, the malware makes sure that logoninit.exe is also launched by Userinit.exe; this is another type of persistence mechanism used by the malware:

At this point, you should have an understanding of how to debug a malicious executable to understand its functionality. In the next section, you will learn how to debug a malicious DLL to determine its behavior.