Instruction: The Format

Needless to say, when we are talking about instructions in this chapter, we mean assembly language instructions . Let’s learn the basic structure.

You can open Sample-4-1 from the samples repo using CFF Explorer and browse to Quick Disassembler in the left pane, as seen in Figure 16-5.

As seen on the right side of Figure 16-5, Disassembler is x86, Offset is 240 (this instructs the disassembler to parse the machine code starting at 240 bytes), and Base Address (virtual address) is 0x401040. Expect the machine code to use the x86 instruction set. You can now click the Disassemble button to disassemble the machine code into assembly language instructions, as seen in Figure 16-6.

There are three columns. The middle column, Opcode, holds the machine code, which, if you read, looks like garbage. But the disassembler decodes these machine code, separating the instructions and providing you a human-friendly readable format for them in the assembly language seen in the third column, Instruction.

Registers

Registers are the data storage available to the CPU and used by instructions to hold various kinds of data. They are generally used by instructions whenever possible because it is faster to access data stored in it than using the memory (RAM) to hold and access the same data. The x86 registers are 32 bits in size. The registers can be broadly separated into the categories, also illustrated in Figure 16-7.

• Data registers

• Pointer register

• Index register

• Control/flags register

• Debug registers

• Segment registers

Data Registers

EAX, EBX, ECX, and EDX are the four data registers and are used by instructions to store/access data needed for their operation. These registers are 32 bits in size, but they can be further split into 16 bit and 8-bit parts, and the various sub-parts can be accessed individually. Do note that of the two 16-bit splits of the 32-bit EAX register, only the lower 16 bits can be referred to in instructions, referred to as AX. Similarly, the lower 16-bit AX can be further split into two 8-bit parts: AH and AL. Figure 16-8 shows the various splits for the EAX register. We can similarly split and refer to the individual parts in the EBX, ECX, and the EDX registers.

Although these registers are meant to be used for general purposes by various instructions to store various types of data, a lot of compilers, while generating instructions, also use some of these instructions for various specialized purposes, as listed.

• EAX

  This register is also called the accumulator and is popularly used to store results from the system. For example, it is widely used to hold the return values from subroutines/functions.

• EBX

  Called the base register, it is used by instructions for indexing/address calculation. We talk about indexing later.

• ECX

  Called the counter register. Some of the instructions, like REP, REPNE, and REPZ, rely on the value of ECX as a counter for loops.

• EDX

  Also used for various data input/output operations and used in combination with other registers for various arithmetic operations.

Do note that the specific functionalities are not set in stone, but most of the time, compilers generate instructions that end up using these registers for these specific functionalities. End of the day, these are general-purpose registers used by instructions for various purposes.

Index Registers

ESI and EDI are the two index registers which point to addresses in memory, for the means of indexing purposes. The ESI register is also called the source index register, and the EDI is also called the destination index register , and are mostly used for data transfer related operations like transferring content among strings and arrays and so forth.

As an example use-case showcased in Figure 16-9, if you want to copy the data from a source array into another destination array, you can set the ESI and EDI registers to hold the starting memory addresses of the source array and the destination arrays respectively. With that set, you can then invoke instructions like REP MOVSB that then start copying the data from the source to destination array using the addresses in the ESI and EDI registers.

Just like the EAX register, the ESI and EDI registers can also be split into 16-bit parts, where the lower 16-bit part can be referred to using SI and DI, respectively, as seen in Figure 16-10.

Important x86 Instructions

Intel has 1500+ x86 instructions, and it’s not possible to memorize each of those. Add to that the specialized instruction sets like SSE, MMX, AVX, and so forth, and the list of instructions gets bigger. From a reverse engineering perspective, we need to learn the most basic instructions, and as and when we come across new instructions, it does you good to look them up in Intel’s instructions reference manual to understand what they do.

Stack Operations

A stack is a memory area that is used by programs to store temporary data related to function calls. The two most basic instructions that manipulate the stack are PUSH and POP. There are other instructions as well, like the CALL and RET that manipulate the stack, which is important as well, which we talk about later. Apart from these, there are other stack manipulation instructions like ENTER and LEAVE, which you can read about using Intel’s reference manual.

Now the stack works in a LIFO (last in, first out) manner, where data is pushed/added onto the top of the stack using the PUSH instruction, and data is popped/removed from the top of the stack using the POP instruction; that is, the last item pushed in is the first item removed out. The general format of the PUSH and POP instructions are shown in Listing 16-5.

PUSH <register>/<immediate_value>/<indirect_memory_address>

POP <register>/<indirect_memory_address>

Listing 16-5

General Format of PUSH and POP Instructions

Both PUSH and POP (and other stack manipulation instructions as well) use the ESP register as an implicit, indirect memory operand based off which it pushes its <operand> value to the stack. At any point in time, the ESP points to the topmost address of the stack. As a PUSH instruction is executed, it decrements the address stored in ESP by a value of 4 and then pushes its operand data into the location at this address. For example, if the ESP is currently 0x40004, a PUSH instruction decrement it to 0x40000. Did you notice something strange here? We said when you push something to the stack, the ESP decrements and not increments. This is because though the stack moves up, the actual stack in memory moves from high memory to lower memory range, as illustrated in Figure 16-12.

Similarly, when a POP is executed, the address in ESP is automatically incremented by 4, simulating popping/removal of data from the stack. For example, if the ESP is 0x40000, a POP instruction copy the contents at address 0x40000 into the operand location and increments ESP to 0x40004.

As an example, have a look at Figure 16-12 that shows how the stack expands and contracts and the way ESP pointer moves when PUSH and POP instructions are executed.

There are other variations of the PUSH instruction like PUSHF and PUSHFD, which don’t require an explicit operand that it pushes onto the stack, as these instructions implicitly indicate an operand: the flags register. Both save the flags registers to the stack. Similarly, their POP instruction counterparts have variants POPF and POPFD, which pop the data at the top of the stack to the flag registers.

Shift Instructions

The logical shift shifts the bits in an operand by a specific count, either in the left or right direction. There are two shift instructions: the left-Shift (SHL) and the right-Shift (SHR).

The SHR instruction follows the following format, SHR <operand>,<num>. The <operand> is the one in which the instruction shifts the bits in a specific direction, and it can be a register or memory operand. The <num> tells the operand how many bytes to shift. The <num> operand value can be either an immediate value or supplied via the CL register.

Figure 16-13 shows an example where the AL register holds the value 1011, and the instruction executed is SHR AL,1. As seen, each bit of the AL register is shifted by a value of 1 in the right direction. The rightmost bit is transferred to the CF flag register and the void left by the leftmost bits are filled with 0.

Similarly, the SHL instruction shifts every bit of its operand in the left direction. As a result, the leftmost bit is pushed out of AL, which is stored in CF. The void in the rightmost bit(s) is filled with a value of 0.

If you go back to the example in the figure, the decimal equivalent of the content of AL register is 1011; that is, the value 11 before the right-Shift. If you shift it right by 1, the value is 101; that is, 5. If you again execute the same instruction moving it right by 1 bit field value, it becomes 10; that is, 2. As you can see every right-Shift divides the value of the contents you are right-Shifting by 2 and this is what SHR does and it is used a lot. If you generalize it into a mathematical formula , a SHR <operand>,<num> is equivalent to <operand> = <operand>/(2 ^^ <num>).

Similarly, the SHL instruction also works in the same manner as, except that every left-Shift multiplies the content you are shifting by a value of 2. If you generalize it into a mathematical formula, a SHL <operand>, <num> is equivalent to <operand> = <operand> * (2 ^^ <num>).

Rotate Instructions

The rotate instructions work like the shift operation. But in this case the byte that is pushed out of the operand at one end is placed back at the other end as illustrated in Figure 16-14.

The format of Rotate instructions is similar to shift instruction. We have two rotate instructions ROR and ROL. Rotate Right; that is, ROR follows the format ROR <operand>, <num> and the ROL instruction follows the format ROL <operand>, <num>. Again <operand> and <num> mean the same as in SHR instruction.

Control Flow Instructions

The Control Flow Instructions alter the linear flow of the execution of the instructions in a program. These instructions come up in assembly as a result of using loops and if/else branches, switch statements, goto in high-level languages which we generally used to branch/modify the execution path of the program based on various conditions. The general format of any control flow instruction takes a <target address> as its operand to which it transfer/branch its execution post its execution.

Control flow instructions can largely be categorized as conditional branch and unconditional branch instructions, which we cover in the next set of sections.

Unconditional Branch Instructions

An unconditional branch instruction like the name says unconditionally branches out and transfers control of the execution of the process to the target address. The three most popular unconditional branch instructions are CALL, JMP and RET.

The JMP instruction follows the format jmp <target_address>, where the operand <target_address> is the target address of the instruction, which can either be a register, absolute immediate value, or an indirect memory address. When this instruction executes the EIP is set to the <target_address> transferring execution control of the program to this <target_address>.

The CALL instruction comes up in assembly when we make function/subroutine calls in our high-level languages and the RET instruction comes up in assembly as the last instruction in a function call, to return the execution control back and out of the function call. Just like the unconditional JMP instruction, the CALL instruction follows the format CALL <target_address>, which transfers the control of the program to the <target_address> by setting the EIP to this address. The instruction also saves the address of the next instruction located right after it into the stack frame, which is also known as the return address, so that when the execution control returns from the function call, the execution of the program resumes from where it branched off the CALL instruction. This is also illustrated in Figure 16-15.

As you can see in the figure, on the left side of the figure, the currently executing instruction at Addr_3, which is the CALL instruction when executed transfer control to the target Addr_33. After execution of this CALL instruction, EIP is set to Addr_33 transferring control of the program to the instruction at this address. Also, the address of the next instruction after the CALL instruction Addr_4 is pushed to the stack, which is the return address.

Now when the control (EIP) reaches the RET instruction at Addr_36, and it gets executed, the CPU update the EIP with the value at the top of the stack pointed to by the ESP, and then increments the ESP by 4 (basically popping/removing the value at the top of the stack). Hence you can say after executing the RET instruction, the control goes to the address that is pointed to by the ESP.

Do note that unlike a CALL instruction, a jump instruction does not push the return address to the stack.

Conditional Branch Instructions

A conditional branch instruction uses the same general format as its unconditional counterpart, but it jumps to its <target_address> only if certain conditions are met. The various jump conditions that need to be satisfied by these instructions are present in the various status flags of the flags register. The jump conditions are usually set by CMP, TEST, and other comparison instructions, which are executed before these conditional branch instructions are executed.

Table 16-4 lists some of the conditional branch instructions available, and the various conditions it checks in the flags register to make the jump.

Table 16-4

Various Conditional Branch Instructions and the Flags They Need Set To Make A Jump

Instruction	Description
JZ	Jumps if ZF is 1
JNZ	Jumps if ZF is 0
JO	Jumps if OF is 1
JNO	Jumps if OF is 0
JS	Jumps if SF is 1
JNS	Jumps if SF is 0
JC	Jumps if CF is 1
JNC	Jumps if CF is 0
JP	Jumps if PF is 1
JNP	Jumps if PF is 0

Loops

Loops are another form of control flow instruction that loop or iterate over a set of instructions by using a counter set in one of its implicit operands, the ECX register. A loop instruction uses the following format: LOOP <target_address>. Before the loop can start, the ECX register is set with the loop count value, which defines the iterations that the loop needs to run. Every time the LOOP instruction executes, it decrements the ECX register (the counter) by 1 and jumps to the <target_address> until the ECX register reaches 0.

You may encounter other variations of the LOOP instructions LOOPE, LOOPZ, LOOPNE, and LOOPNZ. The instructions LOOPE/LOOPZ iterates till ECX is 0 and ZF flag is 1. The instructions LOOPNE/LOOPNZ iterates till ECX is 0 and ZF is 1.

Other Instructions and Reference Manual

In the sections, we went through some of the important and frequently encountered instructions in assembly, but the actual no instructions are far huger in number. Whenever you encounter a new instruction, or when you want to obtain more information about an instruction, searching on the web is a good first step. There are enough resources out there with various examples that should help you understand what an instruction should do and how to use it with various operands.

Debugger Basics

A debugger is software that troubleshoots other applications. Debuggers help programmers to execute programs in a controlled manner, not presenting to you the current state of the program, its memory, its register state, and so forth, but also allowing you to modify this state of the program while it is dynamically executing.

There are two types of debuggers based on the code that needs to be debugged: source-level debuggers and machine-language debuggers . Source-level debuggers debug programs at a high-level language level and are popularly used by software developers to debug their applications. But unlike programs that have their high-level source code available for reference, we do not have the source code of malware when we debug them. Instead, what we have are compiled binary executables at our disposal. To debug them, we use machine language binary debuggers like OllyDbg and IDA, which is the subject of our discussion here and which is what we mean here on when we refer to debuggers.

These debuggers allow us to debug the machine code by disassembling and presenting to us the machine code in assembly language format and allowing us to step and run through this code in a controlled manner. Using a debugger, we can also change the execution flow of the malware as per our needs.

OllyDbg vs. IDA Pro

Now when you launch a program using OllyDbg, by default, the debugger is started. Debugging is like dynamic analysis where the sample is spawned (process created). Hence you see a new process launched with OllyDbg as the parent when you start debugging it with OllyDbg. But when you open a program with IDA by default, it starts as a disassembler, which doesn’t require you to spawn a new process for the sample. If you want to start the debugger, you can then make IDA do it, which spawns a new process for the sample to debug it. Hence IDA is very beneficial if you only want to disassemble the program without wanting to run it. Of course, do note that you can use IDA as a debugger as well.

Also, IDA comes with various disassembly features that let you visualize the code in various styles, one of the most famous and used features being the graph view, that lets you visualize code in the form of a graph. IDA also comes with the Hex-Rays decompiler, which decompiles the assembly code into C style pseudocode that quickly helps you analyze complex assembly code. Add to this the various plugins and the ability to write scripts using IDA Pro, and you have enough malware reverse engineers who swear by IDA Pro. Do note that IDA Pro is software for purchase, unlike OllyDbg and other debuggers, which are free.

OllyDbg is no slouch, either. Although it lacks many of the features that graph view and the decompiler have, it is a simple and great piece of debugging software that most malware reversers use as a go-to tool when reversing and analyzing malware. OllyDbg has lots of shortcuts that help reverse engineers to quickly debug programs. You can create your plugins as well, and best of all, it is free.

There are other debuggers and disassemblers out there, both paid and free, that have incorporated various features of both OllyDbg and IDA Pro. For example, x64Dbg is a great debugger that is free, provides a graph view similar to IDA Pro, and integrates the Sandbox decompiler. Binary Ninja is another great disassembler/debugger. Ghidra is the latest entry to this list. New tools come up every day, and it is best if we are aware of all the latest tools and how to use them. No one debugger or disassembler provides all the best features. You must combine all of them to improve your productivity while reversing malware samples.

Exploring OllyDbg

Let’s start by exploring OllyDbg 2.0 debugger, which we have installed in our analysis VM in Chapter 2. Before we use OllyDbg, we need to make sure some settings are enabled. After you start the debugger, go to the Options menu and select Options and then change the setting for starting a program, making sure you select the Entry point of main module option under Start, as seen in Figure 16-16. This setting makes sure that while OllyDbg starts debugging a new program, it stops/breaks at the entry point of the PE file, it is debugging.

Another option you should disable in OllyDbg is the SFX option. You should uncheck all the options in the SFX tab, as seen in Figure 16-17.

You can now use the File ➤ Open option in the menu to open Sample-16-1 from our samples repo, which has been compiled off a simple Hello World C program located in Sample-16-1.c in the samples repo. With the program loaded, OllyDbg should present you with a UI that looks like Figure 16-18.

As you can see, the main UI of OllyDbg has five main windows: the Disassembly window, Information window, Register window, Memory window, and the Stack window. The following is a description of the various windows.

• Disassembly Window

  Displays the disassembled code. As seen in Figure 16-19, this window has four columns. The first column shows the address of the instruction, the second the instruction opcode (machine code), the third column shows assembly language mnemonic for the disassembled opcode, and the fourth column gives a description/comment of the instruction whenever possible. The Disassembly window also highlights the instruction in black for the instruction that is currently going to be executed, which is also obtained by the value of the EIP register.

• Register window

  Displays the registers and their values, including the flags register.

• Information window

  Displays information for an instruction you click from the Disassembly window.

• Memory window

  You can browse the memory and view its content using this window.

• Stack window

  Displays theaddress and contents of the stack, as seen in Figure 16-20. The current top of the stack; that is, the value in the ESP is highlighted in black in this window. The first column in this window indicates the stack address. The second column displays the data/value at the stack address. The third column displays the ASCII equivalent of the stack value. The last column displays the information/analysis figured by the debugger for the data at that stack address.

Basic Debugging Steps

All kinds of debuggers have a provision that lets you run, execute, and step through the code. To this end, OllyDbg, like various other debuggers, provides various debugging options that are easily accessible through its various buttons under the menu bar, also shown in Figure 16-21.

Hovering the mouse over the button opens a small information message displaying to you what the button does. The same functionality can also be reached using the Debug menu bar option. The following is a description of some of these buttons. Some of the other buttons are described later.

Stepping Into and Stepping Over

Stepping is a method using which we can execute instructions one at a time. There are two ways to step through instructions: step over and step into. Both when used work in the same way, unless a CALL instruction is encountered. A CALL instruction transfer execution to the target address of a function call or an API. If you step into a CALL instruction, the debugger takes you to the first instruction of the function, which is the target address of the CALL instruction. But instead, if you step over a CALL instruction, the debugger executes all the instructions of the function called by the CALL instruction, without making you step through all of it and instead takes you to the next instruction after the CALL instruction. This feature lets you bypass stepping through instructions in function calls.

For example, malware programs call many Win32 APIs. You don’t want to step into/through the instructions inside the Win32 APIs that it calls, since it is pretty much pointless. You already know what these Win32 APIs do. Instead, you want to bypass stepping through the instructions in these APIs, which you can do by stepping over CALLs made to these Win32 APIs.

We can use the stepping functionality using the step into and step over buttons, as seen in Figure 16-21. Alternatively, you can use the F7 and F8 shortcut keys to step into and step over instructions. Let’s try this out using OllyDbg and Sample-16-1.

If you already have Sample-16-1.exe loaded into OllyDbg from our previous exercises, you can reload/restart it. Post loading, OllyDbg stops at the entry point of the main module, which is 0x401040, as seen in Figure 16-19. In the same figure, you can also see that the EIP is also set to 0x401040. Now step over this instruction by using the F8 key. As seen in Figure 16-22, the instruction at 0x401040 executes, and the control transfers over to the next instruction. You can also see that the EIP has now been updated to 0x401041.

Now continue this process, stepping over instructions until we encounter the instruction at address 0x40109E, which has a CALL instruction , as seen in Figure 16-23.

Now, if you step over at this instruction, OllyDbg jump straight to 0x4010A3, bypassing the execution of all instructions inside the function call pointed to by the CALL’s target 0x401000.

But now restart the program from scratch and instead step into using F7 at this CALL instruction at 0x40109E, and as you in Figure 16-24, OllyDbg transfers control to the first instruction in the function call, jumping to the target of the CALL instruction.

Run to Cursor

step into and step over execute a single instruction at a time. What if we want to execute the instructions up to a certain instruction without having to step through instructions one by one. You can do this by using the Run to Cursor debugging option, where you can take your cursor to the instruction in the Disassembly window and highlight the instruction by clicking it. You can now press F4 for the Run to Cursor option.

To try this out, restart debugging Sample-16-1.exe by either using the Ctrl+F2 option or the Restart button from Figure 16-21. OllyDbg starts the program and breaks/stops at the starting address 0x401040. Now scroll to the instruction at address 0x40109E, click it, and then press F4. What do you see? The debugger run/execute all the instructions up and until 0x40109E and then stops/breaks. You can also see that the EIP is now set to 0x40109E.

Do note that Run to Cursor does not work for a location that is not in the execution path. It similarly won’t work for a previously executed instruction that no longer fall in the execution path of the program if it continues execution. For example, for our hello world program Sample-16-1.exe, after you have executed till 0x40109E, you cannot Run to Cursor at 0x40901D; that is, the previous instruction unless you restart the debugger.

Run

Run executes the debugger till it encounters a breakpoint (covered later), or the program exits or an exception is encountered. F9 is the shortcut for Run. Alternatively, you can use the button shown in the menu bar, as seen in Figure 16-21. You can also use the Debug Menu option from the menu bar to use the Run option.

Now restart the debugger for Sample-16-1.exe using Ctrl+F2. Once the program stops/breaks at 0x401040, which is the first instruction in the main module, you can now click F9, and the program executes until it reaches its end and terminates. Had you put a breakpoint in the debugger at some instruction or had the process encountered an exception, it has paused execution at those points.

Execute Till Return

Execute Till Return executes all instructions up and until it encounters a RET instruction. You can use this option by using the fast access button the menu bar, as seen in Figure 16-21 or the shortcut key combination of Ctrl+F9.

Execute Till User Code

You need this feature when you are inside a DLL module and want to get out of the DLL into the user compiled code, which is the main module of the program you are debugging. You can use this option by using the fast access button the menu bar, as seen in Figure 16-21 or the shortcut key combination of Alt+F9. If this feature does not work, you need to manually debug till you reach the user compiled code in the main module of the program.

Jump to Address

You can go/jump to a specified address in the program that is being debugged in OllyDbg using Ctrl+G. The address to which you want to jump into can be either an address in the Disassembly window or the Memory window from Figure 16-18. Using the keyboard shortcut prompt you a window which says Enter the expression to follow. You can type in the address you want to jump to and then press Enter to go to the address.

Note that you won’t execute any instructions during this step. It only takes your cursor to the address you input. There won’t be any change in the EIP register or any other register or memory.

As an example, if you have Sample-16-1.exe loaded in OllyDbg, go to the Disassembly window and click Ctrl+G and key in 0x40109E. It automatically takes your cursor to this instruction address and displays instructions around this address. Similarly, if you go to the Memory window and repeat the same process, keying in the same address, it loads the memory contents at this address in the Memory window, which in this case are instruction machine code bytes.

Breakpoint

Breakpoints are features provided by debuggers that allow you to specify pausing/stopping points in the program. Breakpoints give us the luxury to pause the execution of the program at various locations of our choices conditionally or unconditionally and allow us to inspect the state of the process at these points. There are four main kinds of breakpoints: software, conditional, hardware, and memory breakpoints.

A breakpoint can be used on instructions or memory locations.

• A breakpoint against an instruction tells the debugger to pause/stop/break the execution of the process when control reaches that instruction.

• You can also place a breakpoint on a memory location/address, which instructs the debugger to pause/stop/break the execution of the process when data (instruction or non-instruction) at that memory location is accessed. Accessed here can be split into either read, written into, or executed operations.

In the next set of sections, let’s check how we can use these breakpoints using OllyDbg. We cover conditional breakpoints later.

Software Breakpoints

Software breakpoints implement the breakpoint without the help of any special hardware but instead relies on modifying the underlying data or the properties of the data on which it wants to apply a breakpoint.

Let’s try out software breakpoints on instructions. Restart the debugger against Sample-16-1.exe using Ctrl+F2 or the Restart button from Figure 16-21. OllyDbg starts the process and stop/break at the entry point 0x401040, like in Figure 16-19. Scroll down to instruction at address 0x40109E. You can now place a software breakpoint at this instruction by using the F2 key or double-clicking this instruction or right-clicking and selecting Breakpoints ➤ Toggle, which should highlight the instruction in red as seen in Figure 16-25. Note that setting a breakpoint on an instruction doesn’t change the EIP, which is still at 0x401040.

Now execute the program using F9 or the Run fast access button from Figure 16-21, and you see that the debugger has executed all the instructions up and until the instruction 0x40109E and paused execution at this instruction because you have set a breakpoint at this instruction. To confirm, you can also see that the EIP is now at 0x40109E. This is almost the same as Run to Cursor, but unlike Run to Cursor, you can set a breakpoint once, and it always stops execution of the program whenever execution touches that instruction.

Hardware Breakpoints

One of the drawbacks of software breakpoints is that implementing this functionality modifies the value and properties of the instruction or data location that it intends to break on. This can open these breakpoints to easy scanning-based detection by malware that checks if any of the underlying data has been modified. This makes for easy debugging armoring checks by malware.

Hardware breakpoints counter the drawback by using dedicated hardware registers to implement the breakpoint. They don’t modify either the state, value, or properties of the instruction/data that we want to set a breakpoint on.

From a debugger perspective setting a hardware breakpoint compared to a software breakpoint differs in the method/UI used to set the breakpoint; otherwise, you won’t notice any difference internally on how the breakpoint functionality operates. But do note that software breakpoints can be slower than hardware breakpoints. At the same time, you can only set a limited number of hardware breakpoints because the dedicated hardware registers to implement them are small.

To set a hardware breakpoint on an instruction in the Disassembly window or any raw data in the Data window, you can right-click it select Breakpoint ➤ Hardware, which should open a window like Figure 16-26 seen in the next section. As you can see in this window, you can set a hardware breakpoint for the underlying data either on its execution, access (read/written to), or Write. For example, if the underlying data is an instruction in the Disassembly window on which you want to apply a hardware breakpoint, you can select the Execution option, which breaks the execution of the process at this instruction address, when the execution control reaches this instruction.

In the next section, we talk about memory breakpoints and explore a hands-on exercise on how to set a memory breakpoint using hardware.

Memory Breakpoint

In our previous sections, we explored an exercise that set breakpoints on an instruction. But you can also set a breakpoint on a data at a memory location, where the data may or may not be an instruction. These breakpoints are called memory breakpoints , and they instruct the debugger to break the execution of the process when the data that we set a memory breakpoint has been accessed or executed (depending on the options you set for the memory breakpoint).

From a malware reversing perspective, memory breakpoints can be useful to pinpoint decryption loops that pick up data from an address and write the unpacked/uncompressed data to a location. There are other similarly useful use-cases as well.

You can set a memory breakpoint both in software and hardware. Do note that setting a software memory breakpoint on a memory location relies on modifying the attributes of the underlying pages that contain the memory address on which you want to break. It does this internally by applying the PAGE_GUARD modifier on the page containing the memory you want to set a memory breakpoint on. When any memory address inside that page is now accessed, the system generates STATUS_GUARD_PAGE_VIOLATION exception, which is picked up and handled by OllyDbg.

Alternatively, you can also use hardware breakpoints for memory, but again do remember hardware breakpoints are limited in number. Either way, use memory breakpoints sparingly, especially for software.

Let’s now try our hands on an exercise that sets a hardware memory breakpoint. Let’s get back to Sample-16-2.exe and load it in OllyDbg. In this sample, the encrypted data is located at 0x402000, which is accessed by the instructions in the decryption loop and decrypted and written to another location. Let’s go to the address 0x402000 in the Memory window, by clicking Ctrl+G and enter the address 0x402000. You can then right-click the first byte at address 0x402000 and select Breakpoint ➤ Hardware, which presents you the window, as seen in Figure 16-26. You can select the options; that is, Access and Byte, which tells the debugger to set a hardware breakpoint on the Byte at 0x402000 if the data at that address is accessed (read or written).

After you set the breakpoint, the Memory window should look like Figure 16-27, where the specific memory location is highlighted in red like with instruction breakpoints we set earlier. The red color represents a breakpoint set on that byte.

Now run the debugger using the F9 key, and as you can see in Figure 16-28, the debugger breaks at the very next instruction after the instruction, which accessed the memory location 0x402000. You can see that the instruction at the address 0x401012 accesses the memory location 0x402000, and the debugger breaks after executing that instruction.

While in OllyDbg, you can apply hardware memory breakpoints up to a DWORD in size. You can even place a software memory breakpoint by selecting a full memory chunk, right-clicking, and selecting Breakpoint ➤ Memory Breakpoint. We have set a memory breakpoint on the entire memory chunk from 0x402000 to 0x402045 in Sample-16-2.exe, which ends up being highlighted, as seen in Figure 16-29.

You can also set both software and hardware breakpoints using IDA. We leave that as an exercise for you to explore in the next section.

Exploring IDA Debugger

Let’s now explore IDA Pro to debug our samples. Open the same program Sample-16-1.exe from our previous section in IDA. When we open our sample for analysis using IDA using the File ➤ Open menu option, you are asked for an option if you want to analyses the file as a Portable executable for 80386 or Binary file. Since we already know that the file is a PE executable, we can select the first options, as seen in Figure 16-30.

Before we can start analyzing the sample, let’s set some more stuff up using the Options ➤ General option in the menu bar, which should open the IDA Options window, as seen in Figure 16-31. The first thing that we want to configure is the ability to see the addresses of instructions and their machine code, which by default, IDA doesn’t show. To enable this option, select the Line prefixes (graph) option and then update the Number of opcode bytes (graph) field with a value of 8, as seen in Figure 16-31.

With the option set, the analysis window should look like Figure 16-32.

Now by default, when you launch a program for analysis using IDA, it launches the disassembler and not the debugger. The debugger is only launched when you explicitly start the debugger. To start the debugger, go to the Debugger ➤ Select Debugger option in the menu, which should open the Select a debugger window like that allows you to select the debugger you want to use to debug the program. You can select Local Windows debugger and then click OK, as seen in Figure 16-33.

You can now set up the other debugger options by going to Debugger ➤ Debugger setup in the menu which should open the window in Figure 16-34, and select the Suspend on process entry point option, which instructs the debugger to start the process and break/stop the execution of the process at its entry point, just like how we did with OllyDbg.

Like the shortcut in OllyDbg, you can then press F9 to start debugging, which should now look like Figure 16-35 seen.

The layout is quite similar to OllyDbg. The same Disassembly window, Register window, Memory window, and Stack window are present in IDA like it did with OllyDbg in Figure 16-18. We closed two other windows—thread window and modules window—that opened on the right and then readjusted their window sizes to arrive at the OllyDbg type look.

As an exercise, you can try to debug the samples in IDA like the way we did in OllyDbg in the previous section. Try stepping in/out of instructions. Set breakpoints. IDA Pro is a more complex tool with various features. The power of IDA Pro comes up when you can use all its features. A good resource to use to learn IDA Pro in depth is The IDA Pro Book by Chris Eagle (No Starch Press, 2011), which should come in handy.

Notations in OllyDbg and IDA

Both OllyDbg and IDA disassemble in the same manner, but the way they present us, the disassembled data is slightly different from each other. Both carry out some analysis on the disassembled assembly code and try to beautify the output assembly code, trying to make it more readable to us. The beautification process might involve replacing raw memory addresses and numbers with human-readable names, function names, variable names, and so forth. You can also see automatically generated analysis/comments in the Disassembly window, Stacks window, and Register window in OllyDbg. Sometimes even the view of the actual disassembly is also altered. But sometimes you need to remove all this extra analysis and beautification so that you can see the unadulterated assembly instructions so that you understand what’s happening with the instructions.

Let’s now look at some of these beautification and analysis modifications done by both OllyDbg and IDA Pro and how we can undo them to look at the raw assembly underneath it.

Local Variable and Parameter Names

Both OllyDbg and IDA automatically rename the local variables and parameters for the functions. OllyDbg names the local variables with the LOCAL. prefix, while IDA names the local variables using the var_ prefix. Similarly, in OllyDbg, the arguments passed to the functions are named using the ARG prefix, while in IDA, they are represented using the arg_ prefix.

You can now open Sample-16-3 using OllyDbg and go to the address at 0x401045 using Ctrl+G, which is the start of a function. As you can see in Figure 16-36, OllyDbg has disassembled the machine code at this address, analyzed and beautified the assembly code it generates, renamed the local variables in the function, and the arguments passed to the function to produce the output.

Now carry out the same steps and open the same sample using IDA and go to the same address as. As seen in Figure 16-37, IDA has beautified the assembly in its own way, renaming the local variables and the arguments passed to functions.

Compare the generated assembly from both OllyDbg and IDA Pro, see how they vary. Repeat this process for various other pieces of code at other address locations and compare how the analyzed assembly output varies between OllyDbg and IDA.

Now that you know that both tools modify the generated assembly code and beautify them and pepper it with its analysis, let’s now investigate how to undo this analysis.

Undoing Debugger Analysis

As seen in Figure 16-38, to undo the assembly analysis in OllyDbg you can right on any instruction in the Disassembly window and select Analysis ➤ Remove analysis, where if you select Remove analysis from selection, it only undo the analysis on the instruction on which you right-clicked on the cursor, while Remove analysis from module undoes it for the entire module.

Try the by removing analysis at the instruction at address 0x401045 for Sample-16-3 from the previous section, and you see that OllyDbg replace that instruction at this address with the code in Listing 16-13. Notice that it has replaced LOCAL.1 with the EBP-4. As you remember, EBP is a pointer that points to a function’s stack frame, and EBP-4, in this case, indicates a local variable inside the function.

00401045      MOV DWORD PTR SS:[EBP-4],EAX

Listing 16-13

Instruction at Address 0x401045 in Sample-16-3 After Removing OllyDbg Analysis

Similarly, to remove the assembly analysis in IDA, you need to click the variable name or argument and then press the letter H, as shown in Figure 16-39.

Removing the analysis at address 0x401045 for var_4, converts the assembly instruction to the one in Listing 16-14.

00401045 mov     [ebp-4], eax

Listing 16-14

Instruction at Address 0x401045 in Sample-16-3 After Removing IDA’s Analysis

As you can see, IDA removes the analysis to convert the local variable var_4 as [ebp-4], while OllyDbg from earlier converts LOCAL.1 as DWORD PTR SS:[EBP-4]. Well, both are the same. OllyDbg adds SS, which is the stack segment register. You see in disassembly other segments registers like DS, ES, but let’s not bother about these. Another thing you notice is that OllyDbg adds DWORD PTR, which tells that the variable is a DWORD in size.

As an exercise, undo the analysis at various points in the code and compare the unanalyzed code between OllyDbg and IDA. Extend this exercise to various other samples that you have as well.

Now that we have an understanding of how to use OllyDbg and IDA Pro to both disassemble and debug programs, in the next section, we start exploring various tricks that we can use to identify various high-level language constructs from chunks of assembly code. The ability to identify high-level language code from the assembly easily helps us analyze assembly code and understand its functionality.

Identifying The Stack Frame

Every function has its own block of space on the stack called the stack frame that is used by the function to hold the parameters passed to the function, its local variables. The frame also holds other book-keeping data that allows it to clean itself up after the function has finished execution and returns, and set the various registers to point to the earlier stack frame.

There are two functions: func_a() and func_b(). Both func_a() and func_b() have their own local variables. When func_a() invokes func_b() it passes arguments to func_b(). Also when func_a() invokes func_b() and the control of execution transfers to func_b(), each of these functions have their own stack frames on the stack, as seen in Figure 16-40.

Please note from the figure that though the stack is shown as moving up, the stack always grows from higher address to lower address, So the memory locations at the top have an address that is lower than the ones it.

Identifying a Function Epilogue and Prologue

Now the three instructions form the function prologue, but there can be other combinations of instructions as well. Identifying this sequence of instructions helps us identify the start of functions in assembly code.

Sometimes you may not see these exact sets of instructions in the function epilogue. Instead, you might see instructions like LEAVE, which instead carries out the operations conducted by multiple of the instructions seen in the function epilogue.

Identifying Local Variables

Open Sample-16-4.exe using OllyDbg and go to the instruction at address 0x401000, which is the start of the main() function, as seen in Figure 16-41.

How do you identify the local variables that are part of this function? The easiest way is to let OllyDbg do the work for us. OllyDbg uses the LOCAL. Prefix for all its local variables in a function. There are three local variables: LOCAL.1, LOCAL.2, and LOCAL.3, thereby indicating the presence of three local variables on the stack. Usually, the local variables are accessed using the memory indirect operators’ square brackets [] that also tells us when these variables are being accessed to be read or written into. If you look at the disassembly and map it to our C program in Listing 16-18, LOCAL.1 map to the variable a, LOCAL.2 maps to b and LOCAL.3 maps to c.

Now the method relies on OllyDbg successfully analyzing the sample, but there are various times when OllyDbg analysis fails, and it doesn’t identify the local variables in a function, thereby failing to identify any of the local variables. You no longer have this LOCAL prefix from OllyDbg. How do you identify these local variables then?

You learned earlier that every function has a stack frame, and while a function is being accessed, the EBP pointer is set to point to the currently executing function’s stack frame. Any access to local variables in the currently executing function’s stack frame is always done using the EBP or the ESP as a reference and using an address that is lesser than the EBP; that is, the EBP in the stack frame, which means it looks something like EBP-X.

As an example, take the same sample you have running in OllyDbg and disable analysis for this module (like you learned earlier). The code should now look like Figure 16-42.

As you can see, LOCAL.1, LOCAL.3, and LOCAL.2 are referenced using [EBP-4], [EBP-0C], and [EBP-8]. All are references against the EBP pointer, and lesser than the EBP; that is, the EBP in the stack, thereby indicating that the variable at these memory address [EBP-4], [EBP-8] and [EBP-0C] are local variables of the function.

If you step over through the process for Sample-16-4.exe in OllyDbg to the instruction at address 0x401022 inside the main() function. In Figure 16-41, you see what the stack looks like for the function and what these local variable references look like, as seen in Figure 16-43.

Identifying Pointers

You can open Sample-16-5 using OllyDbg and go to the start of the main() function located at address 0x401000, as seen in Figure 16-44.

But how do you identify a pointer variable? Now, if you go back to our section on x86 instructions, you know that LEA loads an address into another variable, which in our use case, we are loading the address of a local variable LOCAL.1 into EAX. But then we store this address we have stored in EAX into another local variable LOCAL.2, which all translates to LOCAL.2 = EAX = [LOCAL.1]. Remember from the C programming language that addresses are stored in pointers. Since from the instructions, we finally store an address of LOCAL.1 into LOCAL.2, LOCAL.2 is a local variable that is a pointer.

So, to identify pointers, try to locate address loading instructions like LEA and locate the variables in which the addresses are stored, which should indicate that the variables that store addresses are pointers.

Identifying Global Variables

Open Sample-16-6 in OllyDbg and go to the address 0x401000, which is the start of the main() function, as seen in Figure 16-45.

We exploited the hard work put in by OllyDbg analysis to figure out the presence of a global variable. But there’s another manual way to figure this out as well. Just click the instruction that accesses the LOCAL.1 variable, and the Information Display window show you the address of this variable as 0x19FF44 (please note that address might be different on your system). In the Information Display window, it also says that this address is on the stack, so your job is done, but let’s figure this out the hard and long way. We also have the address of the other variable as 0x402000.

Let’s check out a feature called the memory map in OllyDbg. You can open the memory map by going to View ➤ Memory map in the OllyDbg menu or by using the Alt+M keyboard shortcut, which opens a window, as seen in Figure 16-46.

If you go back to our earlier chapters, this memory map window looks very similar to the view of memory in Process Hacker. If you notice the OllyDbg tags, it clearly states the various memory blocks that represent the stack, indicating which is the stack and the other memory blocks. If you compare the two addresses we obtained earlier, 0x19FF44 and 0x402000, we can easily figure out by their locations in the memory blocks, that one is located on the stack and the in one of the other segments like the data segment (i.e., global).

Identifying Array on Stack

We have compiled this program into Sample-16-7 in our samples repo. The main() of the compiled code is located at 0x412130. Let’s load the program using OllyDbg. Figure 16-47 shows the disassembly at the main() function, on which we have removed OllyDbg analysis.

Mapping back to our source code, you can see that there is a loop indicated by the return arrow from 0x412193 to 0x412176. Now you can see the first element of the source array is located at address EBP-14, while that of the destination at EBP-28. The index we used in the program is an integer and is assigned an address of EBP-34. The elements of the array are integers, so each element takes a space of 4 bytes. Figure 16-48 shows the layout of the array in the memory.

At the disassembly level, an array may not be identified when they are initialized, and each element looks like a regular data type–like integer. Arrays can only be identified when the elements in the array are getting accessed. To identify the presence of an array, identify if the elements of an array are accessed using an offset or index against a single element or reference point in memory.

Let’s look back to the disassembly at the instruction at 0x41218B in Figure 16-47. Let’s look at the second operand of the instruction, which is [ECX*4+EBP-14] where EBP-14 is the address of the first element of the source array. Trace back the value stored in ECX to the instruction at 0x412188, which is the value of the local variable [EBP-34]. At each iteration of the loop, the value of this index [EBP-34] is incremented by 1. But if you come back to the instruction at 0x41218B, we use this index value from ECX (i.e., from EBF-34) in every single iteration, but always against the same local variable at EBP-14. The only constant here is the local variable EBP-14, with the variance being ECX ([EBP-34]), thereby indicating that the constant reference variable EBP-34 is an array index variable.

If you refer to the image, the operand accesses the first element of the array in the first iteration and second element in the second iteration and the third one in the third iteration. If you refer to the instruction in 0x41218F in Figure 16-47, you find the same pattern, but instead, elements are being written into the destination array at EBP-28, the same way the source array is accessed earlier.

Identifying Structures on Stack

Open Sample-16-8 using OllyDbg and go to the start of the main() function at 0x401000, as seen in Figure 16-49.

The amount of space allocated for a structure is by adding up the elements of the structure, including padding.

In assembly code, the elements of a structure are accessed in most cases by using the topmost member of a structure as a reference point/member/index. The address of the rest of the elements in the structure is done by adding offsets against this base reference member of the structure.

Now in the figure, LOCAL.3 is a local variable as identified by OllyDbg, and this local variable corresponds to the variable s1 inside main(). So to identify a structure in the assembly code, identify if multiple other assembly instructions are accessing data locations on the stack by using a single variable as a reference point.

The Block (2) and Block (3) addresses are composed and accessed by all of them using the address of LOCAL.3 as a reference index address, all indicating that LOCAL.3 is some kind of complex data structure variable, like a structure or a union and the various other addresses composed/referenced off it are its members.

To figure out the size of the member variables, you need to figure out the size of the data access from the instructions. In the assembly case, the various data members are assigned values considering DWORD as the size; hence the members are 4 bytes in size. Now you might point out that the second data member char c is a character and hence should be only 1 byte in size. This is where padding comes in. A compiler pads the extra 3 bytes in the structure for various purposes, including efficiency, giving you the illusion in the assembly that the variable is 4 bytes in size.

Function Call Parameter Identification

We have compiled the C program into Sample-16-9 in our samples repo. Load Sample-16-9 in OllyDbg and go to the main() function at address 0x401000 in OllyDbg and see how the parameters are passed to the function, as seen in Figure 16-50

If we map the instruction back to our C code LOCAL.1 maps to variable a and LOCAL.2 maps to variable b. These variables are passed as parameters to the sum() function. The instruction at 0x401022 calls the function sum() using its address 0x401036. Parameters are passed to the sum() function by pushing them to the stack. But if you notice the order, the second parameter is pushed first, followed by the first parameter. You can execute step by step till the call instruction 0x401022 and see the location of the parameters on the stack.

Now, if we step into the sum() function using the F7 keyboard shortcut, you notice that the address of the instruction right after the call instruction at 0x401022; that is, 0x401027 is pushed to the stack. This address is the return address from the sum() function back to the main() function, after sum() has finished execution.

Now let us get into the sum() function at 0x401036 and see how it accessed the parameters passed onto it, as seen in Figure 16-51.

OllyDbg has again analyzed this function for us. Identifying the arguments passed to the function is made super easy by OllyDbg as it has done all the analysis and hard work. It has represented the first parameter passed to it with ARG.1 and the second with ARG.2. It has also identified the total from the C code in Listing 16-24 as LOCAL.1. But the LOCAL.1 here is local to this sum() function, and is different from LOCAL.1 in the main() we saw in Figure 16-50. Job done!

But let’s try to figure this out the hard way, just in case OllyDbg fails to analyze the code. The EBP is used as a reference point in the currently executing function's stack frame, and any references to its local variables and arguments passed to it are accessed using the EBP. The arguments passed to the function are placed in the stack below the EBP of the function’s stack frame, which means it can be accessed using EBP+X. Though we said it is below the EBP, we still referenced it using + X. The reason is though the stack moves up, it moves from a higher memory address range to a lower. So, the EBP is at a lower address than its arguments placed below it on the stack, which is at a higher address range.

Now in the sample, remove analysis for the instructions at 0x401040 and 0x401043. As seen in Figure 16-52, the ARG.1 and ARG.2 are de-analyzed by OllyDbg to reveal their true assembly as EBP+8 and EBP+0x0C, thereby proving to us this other method of identifying arguments passed to functions.

Identifying Branch Conditions

Conditions are the steering factors for the flow of execution in programs. In high-level languages, if/else, switches, and so forth, are constructs that test for conditions and alter the execution flow to different branches based on the outcome of their tests. In assembly, you are not going to see these. Instead, the test for conditions are carried out by instructions like CMP, ADD, and SUB and based on the results of the test instructions, which update various status flags in the flags register, various conditional JUMP instructions like jnz, jns, and so forth, branch and alter the execution flow to various points in the code.

Load Sample-16-10.exe using OllyDbg and go to address 0x401000, which is the start of the main() function, as seen in Figure 16-53.

It is very easy to identify the presence of conditional branch instructions. All you need is to look for some sort of comparison instruction and then a branch instruction that tests for conditions in the flags register. We see both instructions here. One is the CMP in Block 2, which does the comparison. The other is the subsequent JNE in Block 4, which branches to different portions of the code based on the test results of the previous CMP, which then update the flags register. The two blocks of code that map to the if and the else branches can be identified in Block 5 and Block 6.

Using OllyDbg, we had to manually figure out the various branches and blocks, but IDA Pro makes it easy to identify branch instructions using its graph view. IDA has two modes to view disassembly: the text view and the graph view. The text view is the linear view, like how OllyDbg shows, while the graph view displays the code in the form of flowcharts. You can switch between the views by right-clicking disassembling and choosing the right view.

Figure 16-54 shows the same code but in graph view using IDA. As you can see, it is easy to identify branch conditions using graph view. Green arrows identify the possible direction or conditional jumps while red arrows are the ones where branching does not happen.

Identifying Loops

Every programming language uses loops with the help of for and while constructs. Malware makes use of loops for various reasons—iterating through processes, files, or other objects, encrypting or decrypting data, even to implement a pseudo sleep, and so forth. Hence it is important for malware analysts to look out for loops in disassembly code since they might point to some kind of special functionality used by the malware.

Loops in assembly language are identified by a backward jump in the execution flow; that is, the target of the jump is a lower address compared to the instruction making the jump. The jump should be a near jump; that is, not to another memory block or segment. The jump can be either conditional or unconditional. Also, loops are not meant to run forever. So there has to be a condition for exiting the loop. If a LOOP instruction creates a loop, then the value of ECX determines the exit condition. In other cases, exit conditions are determined by the presence of instructions like CMP and conditional jump instructions like JNZ, JNE, and so forth. So to identify loops, look for a combination of some kind of immediate short backward jump and some kind of comparison and conditional jump instructions.

Open Sample-16-11.exe using OllyDbg and go to the start of the main() function at 0x401000, which should look like Figure 16-55.

As you can see in the figure, you see a short backward jump at 0x401029 to 0x401012. You then see a comparison instruction at 0x401015, and then immediately, the next instruction at 0x401018 is a conditional jump instruction JG. The body of the loop can be identified by the address of the instruction where the unconditional backward jump-starts and the address of the backward jump instruction itself. Loop identified!

Now there is another easier way to identify loops, and that is allowing OllyDbg to analyze the sample. As you can see in the figure, OllyDbg shows you the loop and its body using the arrow line connecting the unconditional jump instruction at 0x401029, and the jump target 0x401012, which we have pointed out in the figure. Job done!

IDA also analyzes the sample to show loops. With IDA Pro’s graph view, you can identify a loop, similar to how you identify loops in a graph (something that we have learned in graph theory in our college days), as seen in Figure 16-56.

There are more complex loops where there are loops inside loops. Sometimes the number of iterations in the loop can be quite high, and debugging each item may be frustrating. You can run past the entire loop by setting a breakpoint on the exits of the loop. Also, there might be several conditions and comparisons in the body of a loop to exit the loop.

In our sample from Figure 16-55, you can exit the loop at 0x40102B, as indicated by the JG 0x40102B conditional jump instruction earlier at address 0x401018. You should similarly locate all the exit points in a loop and set breakpoints on all of them if you are not interested in iterating through the loop while debugging it and want to exit it early.

Color Coding

Different kinds of color coding schemes are provided by various disassemblers to make the code more readable. In OllyDbg, you can right-click any of the Windows and scroll down to Appearance under which you can select Colors or Highlighting. For example, under the Highlighting menu, you can choose the Christmas tree coloring scheme, which is our personal favorite. Similarly, with IDA, you can go to Options ➤ Colors to select various coloring and appearance options for instructions, variables, and other assembly constructs.

Labels and Comments

Both IDA and OllyDbg have options to label or name an address. When another instruction in the code references that address you previously labeled, the label that you used for that address open in the Information window.

Having the ability to label addresses of the start of the functions or the address of certain code blocks with specific names is a great way for you to tag certain code blocks based on functionality. For example, when you are analyzing malware code that implements decryption or encryption functionality inside a function of its, you can label the start address of that function with your own name, like EncryptionFunction or DecryptionFunction. Now when you click any other instruction in the program that references these function addresses, you see the label names that you gave these function addresses earlier in the Information window.

To apply a label, in OllyDbg, you can click any instruction in the Disassembly window and press the : key, which opens the input box that lets you enter your label for that address location, as seen in Figure 16-57.

In IDA, you can also apply a label to a variable, register, or an address by clicking it and then pressing the letter N, as seen in Figure 16-58.

IDA and OllyDbg also both provide options to comment on instructions in the Disassembly window, which gets saved by the debugger so that next time you reanalyze the same sample, you can look at the comments you added at various instructions.

In OllyDbg to leave a comment on an instruction you can click the instruction in the Disassembly window and press the ; key and enter your comment, which should open a window, where you can enter your comments and click enter. The entered comment opens in the fourth/comment column of the Disassembly window, as seen in Figure 16-59.

Tracking Variables

When you are reading a large piece of disassembled code, you like to know where the variables in the code are used, where they are getting changed, and so forth. Let’s load Sample-16-2.exe in IDA and go to the start of the main() function at 0x401000. We switched to text view, but you can do the same in graph view as well. As seen in Figure 16-60, var_4 is used in the disassembly analysis. If you click var_4 located at any of the instructions, IDA highlight (in yellow) all the other instances of var_4, thereby allowing you to track this variable in the code.

Skipping Compiler Stub and Library Code

You have noticed that we have asked you to go to the main() function in all the examples we demonstrated till now and not the entry point of the PE file. Do you know why we did that? Isn’t the entry point the start of the main() function? When we compile a program, the compiler inserts some code of its own. This code is present in the entry point of the executable and goes all the way to the main() function, which has been written by the programmer. This code is called the compiler stub.

The code in the compiler stub varies from compiler to compiler and even between versions of the same compiler. The compiler stub is present in any executable, whether benign or malware, as long as it’s been generated from a compiler. It’s a waste of time to look at the compiler code since it is present in both benign and malware executables.

Compiler stubs can have specific patterns, and the main function can also be located by parsing the compiler stub. IDA’s FLIRT signatures are there for your help. They can take you across the compiler stub when you open an executable in IDA, thereby helping you get past this unwanted compiler code and into the true functionality of the sample you are analyzing, saving precious time.

Condensing Instructions With Algebra

We saw many of the instructions like MOV, ADD, SUB, INC in assembly, and all these can be represented with arithmetic equations. For example, MOV EAX,9 can be represented as EAX=EAX+9. Similarly, INC EAX can be translated to EAX=EAX+1.

Now this one boils down to an arithmetic equation. If we further solve the equation, this reduces to EAX=004020F0 + LOCAL.1. So translations help us simplify a complex set of instructions into simpler algebraic equations. Once you have translated the, you can add it as a comment so that you can refer the comment back if you were to pass through the same instructions later while debugging the code. This is especially useful if you are analyzing decryption and obfuscation loops in malware that involve multiple instructions that involve various arithmetic instructions that modify data.

Using Decompilers

Disassembly is a process of converting the raw machine code bytes into a more readable assembly language. But assembly language is not as easy to read as high-level languages like C, Java, and so forth. But we can use a process called decompilation , which can convert the machine code back to high-level language code (which is even better for reverser engineers).

There are various tools that can decompile code (Hex-Rays decompiler, Sandman decompiler, Cutter, Ghidra, x64Dbg) that integrate the Sandman decompiler into its UI. x64Dbg is another great debugger that looks and works just like OllyDbg, and the integration of the Sandman decompiler into its UI makes it even better. The best part of it all is that it’s free!

Now coming back to Hex-Rays decompiler, it is an IDA Pro plugin that can convert x86 or x64 disassembly into high-level C-type pseudocode but note that this is a plugin that you must purchase. Let’s put the Hex-Rays decompiler to action. You can open Sample-16-2.exe from the samples repo using IDA and decompile its main() function, which starts at address 0x401000, the decompiled output which you can see in Figure 16-61.

Blocks and Flowcharts

It is extremely hard to read a large piece of Disassembly Code and figure out what it’s doing. No assembly code executes linearly. There are branches taken, calls made, all of which break the linear execution flow. A better way to view the disassembly instructions and understand its execution flow is to use a debugger graph view.

IDA Pro tool provides this graph view feature, which analyzes the assembly code and breaks it into multiple blocks and presents it into a graph view, showing the various execution flows across these blocks. IDA Pro figures out the start and end of the blocks based on various conditions like branches from the jump and call instructions, execution transfer to an instruction from another remote instruction that is not the instruction linearly behind it. Apart from IDA Pro, other debuggers also provide graph view, including OllyDbg using a plugin called OllyGraph, but none of them are as fancy as the IDA Pro one.

We showed the graph view earlier in the chapter, but let’s look at it in action again. You can open the Sample-16-2.exe file in IDA, which then shows you the list of functions it has recognized from the code. It is displayed in the Functions window, as seen in Figure 16-62.

From the functions, sub_401000 starts at 0x401000 which is also the main() function of our sample. If you double-click this function, it opens a new Disassembly window called IDA View-A for this function in graph view, as seen in Figure 16-63.

IDA has broken up the main() function of Sample-16-2.exe into seven blocks. It also shows the control execution flow among these blocks. This is much easier to read and understand and figure out the various branches taken by the instructions in the function than if you were to read the same assembly instructions for this function if it is displayed linearly.

References (XREF)

References or XREF is a feature provided by disassemblers that given a piece of code, instruction, data, the debugger point to other locations in the code that references that piece of code. For example, you have a function call, and as you know, every function call starts at an address. Using references, you can figure out all the other locations/instructions in the code that references that function address. Another example is you have a piece of data in the code, say a global variable, which has an address. Using references, you can figure out all the other locations/instructions in the code that references that global variable’s address.

Let’s take a hands-on exercise to show references in action. For our exercise, please take Sample-16-12 from the samples repo, which is a GandCrab malware sample. Now start by loading this sample in BinText to list all the strings in this sample. You can verify from the strings listed that one of the strings in the sample is -DECRYPT.txt.

You can view the same strings using IDA too. Load the sample in IDA and go to View ➤ Open subviews ➤ Strings, which opens a new window that displays all the strings in the sample, as seen in Figure 16-64.

Now inside the String window shown by IDA, right-click and select Setup, which should bring up the Setup strings window, as seen in Figure 16-65, where we can set up the various options for IDA that decides what kind of strings are displayed by IDA. Select all the options, as seen in the figure.

As seen, we have checked all the options and set the minimum length of string to be displayed as three. That should give us good visibility into all the strings in the sample. With that set, the strings are seen in Figure 16-65.

You can see the string -DECRYPT.txt that we were also able to locate previously using BinText. Let’s try to analyze this particular string, which most probably is related to the ransomware's ransom note.

IDA tells us that an instance of this string is located at address 0x41A0D8. If you double-click the row having this string, you get more details on the different locations in the sample where this string is referenced, as seen in Figure 16-66.

As seen in the screenshot, IDA says that the string has been referenced at the offset 9F from the start of function at 0x4074B9, which in the end translates to 0x4074B9 + 0x9F, which is 0x407558. If you click the XREF, as seen in the figure, it takes you to the address 0x407558 located inside function 0x4074B9, where this string is referenced.

If you want to see the entire flow of code that leads to the specific instruction at address 0x407558 that references this string, you can simply right-click the string DECRYPT-txt in Figure 16-66 and choose the Xrefs graph to option, which show you graph like in Figure 16-67.

As you can see in the figure, -DECRYPT.txt is referenced by code inside the function that starts at 0x4074B9, which in turn is called by another function that starts at 0x407949, which in turn has been called by another function at address 0x407BD3 and so on.

The references to strings features are also available in OllyDbg, but the procedure is slightly different. In OllyDbg, you need to go to the Disassembler window. Right-click inside it and then select Search for ➤ All reference strings, which should open a new window the strings from the file, as seen in Figure 16-68.

The first column is the address where the string has been referred, which is 0x407558, which is what we discovered in IDA.

Do note that if you have not disabled ASLR as per the requirements of the analysis VM setup we discussed in Chapter 2, these addresses we are showing might vary while you open it on your VM.

Like how we found references to data/strings, we can extend it to find references to functions/subroutines, individual instructions, and so forth. For example, if you go to the function 0x4074B9 in the IDA Disassembly window and switch to text view, you see the XREF to the function, as seen in Figure 16-69.

IDA is saying that this function has been referenced at offset 0x85 inside another function located at 0x407949, which all added up is address 0x4079CE. Since this is a reference to code, it is called CODE XREF by IDA, as seen in the figure. Similar to how you built the XREF graph for data earlier, you can right-click the start of the function and select Xref graphs to to display the graph view of how the execution flows to this function.

The references to code can be done in OllyDbg too. With the same Sample-16-12.exe opened using OllyDbg, go to the location 0x4074B9 in the Disassembler window, select the instruction at this address, right-click and go to Find references to ➤ Selected command or instead use the keyboard shortcut Ctrl+R, which opens the Reference window, as seen in the right side of Figure 16-70, which shows the other instructions/code in the sample that references this address.

As seen in the image, the instruction at 0x4074B9 has been referenced from the instruction at 0x4079CE.

References to API calls

Malware uses Win32 APIs extensively to carry out their malicious intentions like injecting code into other processes, stealing information, connecting over the network, and so forth. Our APIMiner tool could figure out what APIs are used during execution. But from a reverse engineering point of view, these Win32 APIs are called from somewhere within the malware code/functions. Using XREF, you can also figure the code blocks or functions which invoke various Win32 APIs used by the sample.

Using Sample-16-12 from our previous exercise, using IDA, you can list the APIs/functions that are imported by either going to View ➤ Open Subviews ➤ Names in the menu bar, which shows you a table that lists all the imports, as seen in Figure 16-71.

If you double-click any of the Win32 APIs listed in any of the rows in the table, it takes you to the XREF window for that API. Click the CreateFileW API in the figure, and as seen in Figure 16-72, it shows us the XREF for CreateFileW API.

As you can see in the figure, it shows multiple locations in the malware’s sample code where CreateFileW is invoked: sub_401261 + 40, sub_40303E + DE, and so on. If you want to see the graph for the XREF, you can right-click the API name CreateFileW and choose Xrefs graph to, just like we did for strings.

You can repeat the same process using OllyDbg as well by right-clicking inside the Disassembly window, and then selecting Search for ➤ All intermodular calls, which should bring up a window like in Figure 16-73, that lists all the Win32 APIs and all its references in the sample code that invokes those Win32 APIs. As you can see, one of the instructions in the malware code that invokes the CreateFileW API is the instruction at address 0x4012A1, which maps to the same address that IDA shows in its XREF in the figure, sub_401261 + 40.

Observing API Calls and Parameters

While debugging malware, you are going to encounter a lot of Win32 APIs that are used by them. While either analyzing or reversing malware, it is important to know the various arguments passed to these Win32 APIs and to figure out the values returned by them, since this tells us more about the functionality and state of the malware. In our analysis chapters, we could obtain both the result returned, and the parameters passed using APIMiner.

Similarly, with debuggers, including OllyDbg, you can also obtain the same information. As an exercise, check out Sample-16-13 using OllyDbg and go to the instruction located at address 0x411A8E call VirtualAlloc API.

As seen in Figure 16-74, at the instruction before it CALLs VirtualAlloc Win32 API, OllyDbg can recognize the API call and also the various arguments passed to this API, which can be seen in the stack window. OllyDbg is even able to recognize the parameter names of the APIs—Address, size, AllocType, and Protect, which are the parameters passed on to the VirtualAlloc API. If the debugger is not able to guess the parameter names, you need to visit MSDN and correlate with the values in the stack.

Now when it comes to figuring out the output or return value of the Win32 API, you need to step over the CALL instruction so that EIP is at the next instruction after the CALL instruction. In this case, the return value of VirtualAlloc, which is the address allocated by it, is placed in the EAX register.

Do note that different APIs return the output in different locations. Some might return output in memory locations that are passed as parameters to the stack. Some might use registers. Some other kinds of results are stored in memory buffers, which you must inspect in the Memory window.

Breaking on Win32 APIs

When reversing, analysts often prefer to skip part of malware code and look at what’s interesting to us. For example, if you want to analyze the network activity of malware, you can skip analyzing/reversing the rest of the malware code and instead set a breakpoint on the Network Win32 APIs like HttpSendRequest(), Send(), Recv().

If you execute the program after setting the breakpoints at APIS, the debugger stop/pause execution when these APIs are finally involved by some malware code. You can then find out the part of the malware code which has invoked the API and then can further analyze that specific piece of malware code.

As an exercise, let’s look at the Sample-16-13 from the samples repo. This sample calls the VirtualAlloc Win32 API to allocate memory. Instead of stepping through every single instruction in the sample to figure out the sample code that involves the API, you can instead go to this VirtualAlloc API by pressing Ctrl+G and type in the API name and then press Enter, as shown in Figure 16-75.

While entering the API name, you get suggestions which you can select, which in our case here it is KERNELBASE.VirtualAlloc, which is the second option in Figure 16-75. Alternatively, you can press enter on any of the options shown, which take you to the location of the API in the corresponding DLL, where you can manually set a breakpoint by using F2.

After setting the breakpoint, when we continue execution of the sample now, we break at the first instance when VirtualAlloc is involved by our sample, as seen in Figure 16-76.

VirtualAlloc internally calls VirtualAllocEx API . The breakpoint breaks at the first instruction at the start of the API call (i.e., prologue). If you execute until the end of the API call (that is, the RET instruction at address 0xDCE7A18), you see the results of the API, which are stored in the EAX register.

Now our main goal is to go to the code in the sample, which involved this Win32 API. To do this, you can use the Execute till user code option in the Debug menu or press Alt+F9 key, which should take you straight to the next instruction in the sample code’s main module that invoked this VirtualAlloc API, which is 0x411A94, as seen in Figure 16-77. As you can see, 0x411A8E is the location in the Sample-16-13.exe that invokes the VirtualAlloc API.

Conditional Breakpoints

Do note that there are a lot of calls to a single Win32 API in a sample. If we simply put a breakpoint at a Win32 API, we break at all the instances of that API, and every time we have to go back to the main sample code to figure out the functionality of the malware code that involved the API and why it invoked the API. Sometimes the malware code might not invoke the Win32 API directly, but via some other Win32 API only if there was a way to break on an API only when it met certain conditions. In comes the conditional breakpoint feature in debuggers.

Now back to the VirtualAlloc API. If we have set a breakpoint on this API, it technically sets a breakpoint on the first instruction on the API. At the very first instruction of the VirtualAlloc function, which is also the first instruction of the function prologue, the ESP points to return address of the caller, the ESP + 4 points to the first argument/parameter passed to the API, ESP + 8 to the second parameter and so on, as seen in Figure 16-78.

Let’s say we want to break on the VirtualAlloc API, only if the Size Parameter passed to the VirtualAlloc API is 0x1000. The Size argument is the second parameter on the stack at ESP + 8. To take this value into consideration while setting the breakpoint, you can set a conditional breakpoint at VirtualAlloc, by right-clicking the first instruction in KERNELBASE.VirtualAlloc and selecting Breakpoint ➤ Conditional in OllyDbg. Alternatively, you can use the keyboard shortcut Shift+F4 to set a conditional breakpoint. You can then place the expression [ESP+8]==1000 as a conditional breakpoint, as seen in Figure 16-79, which tells the debugger to pause execution at this breakpoint only if the value at the address location ESP + 8 is 0x1000, which translates to Size Parameter == 1000.

Conditional breakpoints like are very useful to discard the unimportant API calls and only break on execution if it meets certain more specific conditions. Conditional breakpoints should be used in combination with Dynamic Analysis tools like APIMiner, which we can run right before we can use a debugger to reverse a sample. Using APIMiner lets you know the various Win32 APIs and the number of times those APIs are called and the various arguments that are passed to it. Armed with this knowledge, you can specify conditional breakpoints based on the various argument values used by the sample we next want to debug.

Debugger Events

Debuggers provide us the option to pause the execution of the process we are debugging at various process events, pretty much like a breakpoint, thereby allowing us to catch these events and analyze the state of a program. This feature is very useful while analyzing malware because most malware, as you learned in Chapter 10, spawns child processes and new threads for various activities like Code Injection and Process Hollowing.

To enable debugger events, you can go to Options in the menu bar in OllyDbg and select the Events pane, as seen in Figure 16-80, which lists the various events OllyDbg offers to pause execution of the process. While analyzing malware samples you can enable many of these events, especially the one that debugs child processes and pauses on a new thread, as seen in Figure 16-80, that helps you break/pause the execution of the process when the malware creates a new child process or a new thread.

IDA also has similar options as OllyDbg provides, via the Debugger ➤ Debugger Options in the menu, as seen in Figure 16-34.

Patching

A lot of times, malware may refuse to execute on your machine because of some armoring features. For example, if the malware discovers that it is being debugged or analyzed, it might exit early. But with the help of debuggers, we can view all the instructions and functions that implement these armoring checks. Better yet, with the help of debuggers, we can patch/modify the instructions, code, and registers live as the process is executing, thereby allowing us to bypass running these armoring checks.

As an example, check out this Sample C program seen in Listing 16-25, which we have compiled into Sample-16-10. As you can see in the C code, the if branch is always taken, since a is initialized to 3 at the start of the program. Let’s see if we can patch this code dynamically at runtime to make it take the else branch.

Load Sample-16-10 in OllyDbg and set a breakpoint at the instruction at address 0x401018, which is the instruction that decides to either jump into the if branch or else branch, as seen in Figure 16-53.

Now right-click this instruction at 0x0401018 and choose to assemble from the dropdown, which should open the window, as seen in Figure 16-81. Change JNE 00401031 to JUMP 00401031, which converts the earlier conditional jump into an unconditional jump into the address 0x401031, where 0x4010131 is the else branch from our C code in Listing 16-25. Uncheck the Keep Size option and then press the Assemble button.

The disassembly for the instruction we modified/patched now looks like Figure 16-82.

The code has been modified by our assembly patching, which is also highlighted in red. Now execute the program by pressing F9. In Figure 16-83, the else branch is now executed.

Using the patching feature, we not only can modify the instruction code, but also the data contents in memory, the values of various registers, the values of flags register, the return values from Win32 APIs—all of it per our needs.

Call Stack

You saw that setting a breakpoint at a certain location and then executing the code executes the entire code until the breakpoint. Often it might be required for us to know what other functions are executed in between in the call up to our breakpoint instruction/function.

The code has a function call invocation chain as main() -> func_a() -> func_b() -> func_c() as illustrated by seen in Figure 16-84.

The C code has been compiled into Sample-16-14, which you can then load using OllyDbg. Once loaded, set a breakpoint at the start of func_c(), which is the address 0x401067, post which you can run the program by pressing F9, which should then break/stop execution at func_c() where we have set our breakpoint.

Now a call stack is a feature of the debugger that shows the entire chain of function calls that has led to the current instruction getting executed. Now since we have hit the breakpoint inside function func_c, we are currently paused inside func_c right at the first instruction of this function.

Now go to the menu bar of OllyDbg and select Call Stack. Alternatively, you can use the keyboard shortcut Alt+K, which opens a new window called Call Stack. It shows the call stack and the entire call stack chain from the main() up to func_c(), as seen in Figure 16-85.

Summary

Dynamic analysis and static analysis are super-fast ways to analyze and classify a sample. But sometimes they may not be enough for various reasons, including the presence of armoring in the samples we are analyzing and also for the need to dissect deeper into a sample to write better detection. In comes the process of reverse engineering to dive deeper into samples and debug them.

In this chapter, you learned what reverse engineering means and the various processes involved in reversing a sample. We started by learning the basics of the x86 Instruction format and run through various important sets of instructions that we encounter while reversing malware samples. We then explored what debuggers mean and how to use them using OllyDbg and IDA as our reference debugger examples.

Using debuggers, we then did various exercises in which you learned how to identify high-level code constructs in the assembly code. Identifying high-level code constructs in the assembly code helps us speed up the analysis of the assembly code while reversing samples.

You also learned various other additional features debuggers to better present the assembly code and explore ways to tag the assembly code for our future reference. Finally, you learned various other advanced debugging tricks, including using XREFs and patching assembly code that are part of useful tricks reverse engineers use to reverse malware samples.