In this chapter, we will cover various anti-reverse engineering techniques that malware authors use to protect their code against unauthorized analysts who want to understand its functionality. We will familiarize ourselves with various approaches, from detecting the debugger and other analysis tools to breakpoint detection, VM detection, and even attacking the anti-malware tools and products.

We will also cover the VM and sandbox-detection techniques that malware authors use to avoid spam detection, as well as automatic malware-detection techniques that are implemented in various enterprises. As these anti-reverse engineering techniques are widely used by malware authors, it's very important to understand how to detect them and bypass them to be able to analyze complex or highly obfuscated malware.

The chapter is divided into the following sections: 

• Exploring debugger detection
• Handling debugger breakpoints evasion
• Escaping the debugger
• Obfuscation and anti-disassemblers
• Detecting and evading behavioral analysis tools
• Detecting sandboxes and virtual machines

Exploring debugger detection

For malware authors to keep their operations going without being interrupted by antivirus products or any takedown operations, they have to fight back and equip their tools with various anti-reverse engineering techniques. Debuggers are the most common tools that malware analysts use to dissect malware and reveal its functionality. Therefore, malware authors implement various anti-debugging tricks to keep their Command & Control servers (C&Cs) hidden and their configurations, exfiltrated data, and their communication with the malware well protected.

Direct check for debugger presence

Windows includes lots of ways to detect the presence of a debugger. There are multiple APIs that help detect whether the current process is being debugged or not, as follows:

• IsDebuggerPresent
• CheckRemoteDebuggerPresent
• NtQueryInformationProcess (with the ProcessDebugPort (7) argument)

These APIs access a flag in the process environment block (PEB) called BeingDebugged that is set to True when the process is running under a debugger. To access this flag, malware can execute the following instructions:

mov     eax, dword ptr fs:[30h]   ; PEB
cmp     byte ptr [eax+2], 1 ; PEB.BeingDebugged
jz      <debugger_detected>

These are mostly direct ways to check for the presence of a debugger. However, there are also other ways to detect them, such as by observing the differences in the process loading, thread loading, or the initialization phase between a process running with a debugger and another process running without a debugger. One of these techniques involves using NtGlobalFlag.

The common way to bypass them is by overwriting IsDebuggerPresent or CheckRemoteDebuggerPresent APIs with NOP instructions or by setting a breakpoint at the start of each of these APIs for monitoring and changing the return values.

Detecting a debugger through an environment change

NtGlobalFlag is a flag at offset 0x68 of the PEB in 32-bit systems and 0xBC in 64-bit systems. During normal execution, this flag is set to zero when the process is running without the presence of a debugger, but when a debugger is attached to the process, this flag is set with the following three values:

• FLG_HEAP_ENABLE_TAIL_CHECK (0x10)
• FLG_HEAP_ENABLE_FREE_CHECK (0x20)
• FLG_HEAP_VALIDATE_PARAMETERS (0x40)

The initial value of NtGlobalFlag can be changed from the registry. However, in the default situation, malware can check for the presence of a debugger using these flags by executing the following instructions:

mov eax, fs:[30h] ;Process Environment Block
mov al, [eax+68h] ;NtGlobalFlag
and al, 70h       ;Other flags can also be checked this way
cmp al, 70h       ;0x10 | 0x20 | 0x40
je <debugger_detected>

The following flags can be used in the x64 environment:

push 60h
pop rsi
gs:lodsq ;Process Environment Block
mov al, [rsi*2+rax-14h] ;NtGlobalFlag
and al, 70h
cmp al, 70h
je <debugger_detected>

This is just one of many ways in which the differences in the environment between processes running and not running under a debugger can be detected.

Detecting a debugger using parent processes

One last technique worth mentioning is that processes can detect whether they were created by a debugger by checking the parent process's name. Windows OS sets the process ID and the parent process ID in the process information. Using the parent process ID, you can check whether it was created normally (for example, by using Explorer.exe or iexplore.exe) or whether it has been created by a debugger (for example, by detecting the presence of the dbg substring in its name).

There are two common techniques for malware to get the parent process ID, listed as follows:

• Looping through the list of running processes using CreateToolhelp32Snapshot, Process32First and Process32Next (as we saw in Chapter 4, Inspecting Process Injection and API Hooking, with process injection). These APIs not only return the process name and ID, but also more information, such as the parent process ID that the malware is looking for.  
• Using the undocumented NtQueryInformationProcess API. Given ProcessBasicInformation as an argument, this API can return the parent process ID. Even though this API could be altered in later versions of Windows, it's still widely used by malware to get process information, as shown in the following screenshot:
  

After getting the parent process ID, the next step is to get the process name or the filename to check whether it's the name of a common debugger or whether it includes any dbg or debugger substrings in its name. There are two common ways to get the process name from its ID, as shown in the following list:

• Looping through the processes the same way to get the parent process ID, but this time attackers get the process name by providing the parent process ID that they got earlier.
• Using the GetProcessImageFileNameA API to get the filename of a process given its handle. To do this, they need to execute the OpenProcess API in order to get permission to access this process to query for information (by using PROCESS_QUERY_INFORMATION as the requested permissions argument). This API returns the process filename, which can be checked later to detect whether it's a debugger.

Handling debugger breakpoints evasion

Another way to detect debuggers or evade them is to detect their breakpoints. Whether they are software breakpoints (like INT3), hardware breakpoints, single-step breakpoints (trap flag), or memory breakpoints, malware can detect them and possibly remove them to escape reverse engineer control. 

Detecting software breakpoints (INT3)

This type of breakpoint is the easiest to use, as well the easiest to detect. As we stated in Chapter 1, A Crash Course in CISC/RISC and Programming Basics, this breakpoint modifies the instruction bytes by replacing the first byte with 0xCC (the INT3 instruction), which creates an exception (an error) that gets delivered to the debugger to handle.

Since it modifies the code in memory, it's easy to scan the code section in memory for the INT3 byte. A simple scan will look like this:

The only drawback of this approach is that some C++ compilers write INT3 instructions after the end of each function as filler bytes. An INT3 byte (0xCC) can also be found inside some instructions as part of an address or a value, so searching for this byte through the code may not be an effective solution, and could return lots of false positives.

There are two other techniques that are commonly used by malware to scan for an INT3 breakpoint, as shown in the following list:

• Precalculating a checksum (a sum of a group of bytes) for the entire code section and recalculating it again in execution mode. If the value has changed, then it means that there are some bytes that have been changed, either by patching or by setting an INT3 breakpoint. An example would be as follows:

mov esi,<CodeStart>
   mov ecx,<CodeSize>
   xor eax,eax

ChecksumLoop:
   movzx edx,byte [esi]
   add eax,edx
   rol eax,1
   inc esi
   loop .checksum_loop

cmp eax, <Correct_Checksum>
jne <breakpoint_detected>

• Reading the malware sample file and comparing the code section from the file to the memory version of it. If there are any differences between them, this means that the malware has been patched in memory or there is a software breakpoint (INT3) that has been added in the code. This technique is not widely used as it's not effective if the malware sample has its relocation table populated (check Chapter 2, Basic Static and Dynamic Analysis for x86/x64, for more information).

The best solution for software breakpoint detection is to use hardware breakpoints, single-stepping (code tracing) or setting access breakpoints on different places in the code section for any memory read.

Once a memory breakpoint on access gets a hit, you can find the checksum calculating code and deal with it by patching its checksum code itself as you can see in the following screenshot:

In this figure, we have set a breakpoint Memory, on access on the code section. By
executing the program, the application should stop at the address 0x00401033, as this
instruction tried to access the address 0x00401048 where we set our breakpoint. In this manner, we can detect the INT3 scan loop or the checksum calculating loop.

By patching the check at the end of the checksum calculator or the jz/jnz with the opposite check, you can easily bypass this technique. 

Detecting single-stepping breakpoints (trap flag)

Another type of breakpoint detection technique that is widely used is the trap flag detection. When you trace over the instructions one by one, checking the changes they make in memory and on the registers' values, your debugger sets the trap flag in the EFLAGS register, which is responsible for stopping on the next instruction and returning control back to the debugger.

This flag is not trivial to catch because EFLAGS is not directly readable. It's only readable through the pushf instruction, which saves this register value in the stack. Since this flag is always set to False after returning to the debugger, it's hard to check the value of this flag and detect a single-step breakpoint; however, there are multiple ways to detect this behavior. Let's go through the most common examples.

Detecting a trap flag using the SS register

In the x86 architecture, there are multiple registers that are not widely used nowadays. These registers were used in DOS operating systems before virtual memory in the way we know it was introduced, particularly the segment registers. Apart from the FS register (which you already know about), there are other segment registers, such as CS, which was used to point to the code section, DS, which was used to point to the data section, and SS, which was used to point to the stack.

The pop SS instruction is quite special. This instruction is used to get a value from the stack and change the stack segment (or address) according to this value. So if there's any exception happening while executing this instruction, it could lead to confusion (which stack would be used to store the exception information?). Therefore, no exceptions or interrupts are allowed while executing this instruction, including any breakpoints or trap flags.

If you are tracing over this instruction, your debugger will move the cursor, skip the next instruction and jump directly to the instruction after it. It doesn't mean this skipped instruction wasn't executed, it was executed but not interrupted by the debugger.

For example, in the following code, your debugger cursor will move from POP SS to MOV EAX, 1, skipping the PUSHFD instruction, even if it was executed:

PUSH SS
POP SS
PUSHFD     ;your debugger wouldn't stop on this instruction
MOV EAX,1  ;your debugger will automatically stop on this instruction.

The trick here is that, in the previous example, the trap flag will remain set while executing the pushfd instruction, but it won't be allowed to return to the debugger., so the pushfd instruction will push the EFLAGS register to the stack, including the actual value of the trap flag (if it was set, it will show in the EFLAGS register). Then, it's easy for malware to check whether the trap flag is set and detect the debugger. An example of this is shown in the following screenshot:

This is a direct way of checking for code tracing or single-stepping. Another way to detect it is by monitoring the time that passed while executing an instruction or a group of instructions, which is what we will talk about in the next section.

Detecting single-stepping using timing techniques

There are multiple ways to get the exact time with millisecond accuracy from the moment the system is on until the execution of the current instruction. There is an x86 instruction called rdtsc that returns the time in EDX:EAX registers. By calculating the difference between the time before and after executing a certain instruction, any delay will be clearly shown, which represents reverse-engineering tracing through the code. An example of this is shown in the following screenshot:

This instruction is not the only way to get the time at any given moment. There are multiple APIs supported by Windows that help programmers get the exact time, as follows:

• GetLocalTime
• GetSystemTime
• GetTickCount
• KiGetTickCount (in kernel mode)
• QueryPerformanceCounter
• timeGetTime
• timeGetSystemTime

This technique is widely used and more common than the SS segment register trick. The best solution is to patch the instructions. It's easy to detect it if you are already stepping through the instructions; you can patch the code or just set the instruction pointer (EIP/RIP) to make it point to the code after the check.

Evading hardware breakpoints

Hardware breakpoints are based on registers that are not accessible in user mode. Therefore, it's not easy for malware to check these registers and clear them to remove these breakpoints.

For malware to be able to access them, it needs to have them pushed to the stack and pulled out from it again. To do that, many malware families rely on structured exception handling (SEH).

What is structured exception handling?

For any program to handle exceptions, Windows provides a mechanism called SEH. It's based on setting a callback function to handle the exception and then resume execution. If this callback failed to handle the exception, it can pass this exception to the previous callback that was set. If the last callback was unable to handle the exception, the operating system terminates the process and informs the user about the unhandled exception, and often suggests him or her to send it to the developer company.

A pointer to the first callback to be called is stored in the thread information block (TIB) and can be accessed via FS:[0x00]. The structure is a linked list, which means that each item in this list contains the address of the callback function and follows the address of the previous item in the list (the previous callback). The linked list looks like this in the stack:

The setup of the SEH callback generally looks like this:

PUSH <callback_func> // Address of the callback function
PUSH FS:[0]              // Address of the previous callback item in the list
MOV FS:[0],ESP           // Install the new EXCEPTION_REGISTRATION

As you can see, the SEH linked list is mostly saved in the stack. Each item points to the previous one. When an exception occurs, the operating system executes this callback function and passes the necessary information about the exception and the thread state to it (registers, the instruction pointer, and so on). This callback has the ability to modify the registers, the instruction pointer, and the whole thread context. Once the callback returns, the operating system takes the modified thread state and registers (which is called the context) and resumes based on it. The callback function looks like this:

 __cdecl _except_handler(
     struct _EXCEPTION_RECORD *ExceptionRecord,
     void * EstablisherFrame,
     struct _CONTEXT *ContextRecord,
     void * DispatcherContext
     );

The important arguments are the following:

• ExceptionRecord: Contains information related to the exception or the error that has been generated. It contains the exception code number, the address, and other information.
• ContextRecord: This is a structure that represents the state of that thread at the time of the exception. It's a long structure that contains all the registers and other information. A snippet of this structure would look as follows:

struct CONTEXT {
 DWORD ContextFlags;
    DWORD DR[7];
  FLOATING_SAVE_AREA FloatSave;
  DWORD SegGs;
  DWORD SegFs;
  DWORD SegEs;
  DWORD SegDs;
  DWORD Edi;
  ....
 };

There are multiple ways to detect a debugger using SEH. One of them is by detecting and removing hardware breakpoints.

Detecting and removing hardware breakpoints

To detect or remove hardware breakpoints, malware can use SEH to get the thread context, check the values of the DR registers and clear all of them to remove the hardware breakpoints—or at least just check their values and exit if a debugger is detected. The code is as follows:

xor eax, eax
push offset except_callback
push d fs:[eax]
mov fs:[eax], esp
int 3 ;force an exception to occur
...
except_callback:
   mov eax, [esp+0ch] ;get ContextRecord
   mov ecx, [eax+4] ;Dr0
   or ecx, [eax+8] ;Dr1
   or ecx, [eax+0ch] ;Dr2
   or ecx, [eax+10h] ;Dr3
  jne <Debugger_Detected>

Another way to remove hardware breakpoints is to use the GetThreadContext() API to access the current thread (or another thread) context and check for the presence of hardware breakpoints or clear them using the SetThreadContext() API.

The best way to deal with these techniques is to set a breakpoint on GetThreadContext, SetThreadContext, or on the exception callback function to make sure they don't reset or detect your hardware breakpoints.

Memory breakpoints

The last type of breakpoint to talk about is memory breakpoints. It's not common to see an anti memory breakpoints trick, but they can be easily detected by using the ReadProcessMemory API with the malware's ImageBase as an argument and the SizeOfImage as the size. ReadProcessMemory will return False if any page inside the malware is guarded (PAGE_GUARD) or set to no-access protection (PAGE_NOACCESS).

For a malware sample to detect a memory breakpoint on write or execute, it can query any memory page protection using the VirtualQuery API. Alternatively, it can evade them by using VirtualProtect with PAGE_EXECUTE_READWRITE argument.

The best way to deal with these anti-debugging tricks is to set breakpoints on all of these APIs and force them to return the desired result to the malware in order to resume normal execution.

Escaping the debugger

Apart from detecting debuggers and removing their breakpoints, there are multiple tricks that malware uses to escape the debugger's control: escaping the breakpoints, step-into and step-over, or escaping the whole debugging environment altogether. Let's cover some of the most common tricks.

Process injection

We have talked about process injection before, in Chapter 4, Inspecting Process Injection and API Hooking. Process injection is a very well-known technique, not only for man-in-the-browser attacks, but also for escaping the debugged process into a process that is not currently debugged. By injecting code into another process, malware can get out of the debugger's control and execute code before the debugger can attach to it.

A commonly used solution to bypass this trick is to inject an infinite loop into the entry point of the injected code before it gets executed, usually in the injector code either before the WriteProcessMemory call when the code hasn't been injected yet or before CreateRemoteThread, this time in another process's memory.

An infinite loop can be created by writing two bytes (0xEB 0xFE) that represent a jmp instruction to itself, as you can see in the following screenshot:

Next, we are going to talk about another popular technique called the TLS callback. Read on!

TLS callbacks

Many reverse engineers start the debugging phase from the entry point of the malware, which usually makes sense. However, some malicious code can start before the entry point. Some malware families use Thread Local Storage (TLS) to execute code that initializes every thread (which runs before the thread's actual code starts). This gives the malware the ability to escape the debugging and do some preliminary checks, and maybe run most of the malicious code this way while having benign code at the entry point.

In a data directory block of the PE header, there is an entry for TLS. It is commonly stored in the .tls section, and the structure of it looks like this:

The AddressOfCallBacks points to a null-terminated array (the last element is zero) of callback functions, which are to be called after each other, each time a thread is created. Any malware can set its malicious code to start inside the AddressOfCallBacks array and ensure that this code is executed before the entry point.

A solution for this trick is to check the PE header before debugging the malware and set a breakpoint on every callback function registered inside the AddressOfCallBacks field.

Windows events callbacks

Another trick used by malware authors to evade the reverse engineer's single-stepping and breakpoints is by setting callbacks. Callbacks are each called for a specific event (like a mouse click, keyboard keystroke, or a window moving to the front). If you are single-stepping over the malware instructions, the callback would still be executed without you noticing. In addition, if you are setting breakpoints based on the code flow, it will still bypass your breakpoints.

There are so many ways to set callback functions. Therefore, we will just mention two of them here, as follows:

• Using the RegisterClass API: The RegisterClass API creates a window class that can be used to create a window. This API takes a structure called WNDCLASSA as an argument. The WNDCLASSA structure contains all the necessary information related to this window, including the icon, the cursor icon, the style, and most importantly the callback function to receive window events. The code looks as follows:

 MOV     DWORD PTR[WndCls.lpfnWndProc],<WindowCallback>
 LEA     EAX,DWORD PTR SS:[WndCls]
 PUSH    EAX                                     ; pWndClass
 CALL    <JMP.&user32.RegisterClassA>            ; RegisterClassA 

• Using SetWindowLong: Another way to set the window callback is to use SetWindowLong. If you have the window handle (from EnumWindows or FindWindow or other APIs), you can call the SetWindowLong API to change the window callback function. Here is what this code looks like:

PUSH <WindowCallback>
PUSH GWL_DlgProc
PUSH hWnd ;Window Handle
CALL SetWindowLongA

The best solution for this is to set breakpoints on all the APIs that register callbacks or their callback functions. You can check the malware's import table, any calls to GetProcAddress, or other functions that dynamically call an API. 

Obfuscation and anti-disassemblers

Dissemblers are one of the most common tools that are used for reverse engineering, and so they are actively targeted by malware authors. Now, we will take a look at the different techniques that are used in malware to obfuscate its code and make it harder for reverse engineers to analyze it.

Encryption

Encryption is the most common technique as it also protects malware from antivirus static signatures. Malware can encrypt its own code and have a small piece of stub code to decrypt the malicious code before executing it. The malware can also encrypt its own data, such as strings, API names, and their C&Cs.

Dealing with encryption is not always easy. One solution is to execute the malware and dump the memory after it is decrypted. You can dump the process memory using the Sysinternals tool called ProcDump, and the command line looks like:

procdump -ma <process name/pid>

This will dump the whole process and its memory. If you only want the process image, you can use -mm to create a Mini process image. Also, many sandboxes, now can make process dumps of the monitored processes, which will help you get the malware in a
decrypted form.

But for cases like encrypting strings and decrypting each string on demand, you will need to reverse the encryption algorithm and write a script to go through all the calls to the decryption function and use its parameters to decrypt the strings. You can check out Chapter 3, Unpacking, Decryption, and Deobfuscation, for more information on how to write such scripts.

Junk code insertion

Another well-known technique that's used in many samples, and which became increasingly popular from the late 90s and early 2000s, is junk code insertion. With this technique, the malware author inserts lots of code that never gets executed, either after unconditional jumps, calls that never return, or conditional jumps with conditions that would never be met. The main goal of this code is to waste the reverse engineer's time analyzing useless code or make the code graph look more complicated than it actually is.

Another similar technique is to insert ineffective code. This ineffective code could be something like nop, push & pop, inc & dec. A combination of these instructions could look like real code; however, they all compensate for each other, as you can see in the following screenshot:

There are different forms of this junk code, including the expansion of an instruction; for example, inc edx becomes add edx, 3 and sub edx, 2, and so on. This way, it is possible to obfuscate the actual values, such as 0x5a4D ('MZ') or other values that could represent specific functionality for this subroutine.

This technique has been around since the 90s in metamorphic engines, but it's still used by some families to obfuscate their code.

Code transportation

Another trick that's commonly used by malware authors is code transportation. This technique doesn't insert junk code; instead, it rearranges the code inside each subroutine with lots of unconditional jumps, including call and pop or conditional jumps that are always true.

It makes the function graph look very complicated to analyze and wastes the reverse engineer's time. An example of such code can be seen in the following screenshot:

There is a more complicated form of this where malware rearranges the code of each subroutine in the middle of the other subroutines. This form makes it harder for the disassembler to connect each subroutine as it makes it miss the ret instruction at the end of the function and then not consider it as a function.

Some other malware families don't put a ret instruction at the end of the subroutine and substitute it with pop and jmp to hide this subroutine from the disassembler. These are just some of the many forms of code transportation and junk code insertion techniques.

Dynamic API calling with checksum

Dynamic API calling is a famous anti-disassembling trick used by many malware families. The main reason behind using it is that this way, they hide API names from static analysis tools and make it harder to understand what each function inside the malware does.

For a malware author to implement this trick, they need to pre-calculate a checksum for this API name and push this value as an argument to a function that scans export tables of different libraries and searching for an API by this checksum. An example of this is shown in the following screenshot:

The code for resolving the function actually goes through the PE header of the library, loops through the import table, and calculates the checksum of each API to compare it with the given checksum (or hash) that's provided as an argument.

The solution to this approach could require scripting to loop through all known API names and calculate their checksum or executing this function multiple times giving each checksum as input and saving the equivalent API name for it.

Proxy functions and proxy argument stacking

The Nymaim banking trojan took anti-disassembling to another level by adding additional techniques, such as proxy functions and proxy argument stacking.

With the proxy functions technique, malware doesn't directly call the required function; instead, it calls a proxy function that calculates the address of the required function and transfers the execution there. Nymaim included more than 100 different proxy functions with different algorithms (4 or 5 algorithms in total). The proxy function call looks like this:

The proxy function code itself looks like this:

For arguments, Nymaim used a function to push arguments to the stack rather than just using the push instruction. This trick could prevent the disassembler from recognizing the arguments that were given to each function or API. An example of proxy argument stacking is as follows:

This malware included many different forms of the techniques that we introduced in this section, so as long as the main idea is clear, you should be able to understand all of them.

Detecting and evading behavioral analysis tools

There are multiple ways how malware can detect and evade behavioral analysis tools, such as ProcMon, Wireshark, API hooking tools, and so on, even if they don't directly debug the malware or interact with it. In this section, we will talk about two common examples of how it can be done.

Finding the tool process

One of the simplest and most common ways how malware can deal with malware-analysis tools (and antivirus tools as well) is to loop through all the running processes and detect any unwanted entries. Then, it is possible to either terminate or stop them to avoid further analysis.

In Chapter 4, Inspecting Process Injection and API Hooking, we covered how malware can loop through all running processes using the CreateToolhelp32Snapshot, Process32First, and Process32Next APIs. In this anti-reverse engineering trick, the malware uses these APIs in exactly the same way to check the process name against a list of unwanted processes names or their hashes. If there's a match, the malware terminates itself or uses some approach such as calling the TerminateProcess API to kill that process. The following screenshot shows an example of this trick being implemented in Gozi malware:

The following screenshot shows an example of Gozi malware code using the TerminateProcess API to kill a process of its choice:

This trick can be bypassed by renaming the tools you are using before executing them. This simple solution could hide your tools perfectly if you just avoid using any known keywords in the new names, such as dbg, disassembler, AV, and so on.

Searching for the tool window

Another trick would be not to search for the tool's process name, but instead to search for its window name (the window's title). By searching for a program window name, malware can bypass any renaming that could be performed on the process name, which gives it an opportunity to detect new tools as well (mostly, window names are more descriptive than process names).

This trick can be done in the following two ways:

• Using FindWindow: Malware can use either the full window title, such as Microsoft network monitor, or the window class name. The window class name is a name that was given to this window when it was created, and it's different from the title that appears on the window. For example, the OllyDbg window class name is OLLYDBG, while the full title could change based on the process name of the malware under analysis. An example of this is as follows:

push NULL
push .szWindowClassOllyDbg
call FindWindowA
test eax,eax
jnz <debugger_found>

push NULL
push .szWindowClassWinDbg
call FindWindowA
test eax,eax
jnz <debugger_found>

.szWindowClassOllyDbg db "OLLYDBG",0
.szWindowClassWinDbg db "WinDbgFrameClass",0

• Using EnumWindows: Another way to avoid searching for the window class name or dealing with the change of window titles is to just go through all the window names that are accessible and scan their titles, searching for suspicious window names such as Debugger, Monitor, Wireshark, Disassembler, and so on. This is a more flexible way to deal with new tools or tools the malware author forgot to cover. 
  With the EnumWindows API, you need to set a callback to receive all windows. For each top-level window, this callback will receive the handle of this window, from which it can get its name using the GetWindowText API. An example of this is as follows:

The callback function declaration looks like this:

BOOL CALLBACK EnumWindowsProc(
  _In_ HWND   hwnd,
  _In_ LPARAM lParam
);

The hwnd value is the handle of the window, while lParam is a user-defined argument (it's passed by the user to the callback function). Malware can use the GetWindowText API with this handle (hwnd) to get the window title and scan it against a predefined list of keywords.

It's more complicated to modify window titles or classes than actually set breakpoints on these APIs and track the callback function to bypass them. There are plugins for popular tools, such as OllyDbg and IDA, that can help rename their title window to avoid detection (like OllyAdvanced), which you can use as a solution as well.

Detecting sandboxes and virtual machines

Malware authors know that if their malware sample is running on a virtual machine, then it's probably being analyzed by a reverse engineer or it's probably running under the analysis of an automated tool such as a sandbox. There are multiple ways in which malware authors can detect virtual machines and sandboxes. Let's go over some of them now.

Different output between virtual machines and real machines

Nothing is perfect. Therefore, malware authors use certain characteristics of the virtual machines' implementations in some of the assembly instructions. Examples of these are as follows:

• CPUID hypervisor bit: The CPUID instruction returns information about the CPU and provides a leaf/ID of this information in eax. For leaf 0x01 (eax = 1), the CPUID sets bit 31 to 1, indicating that the operating system is running inside a virtual machine or a hypervisor.
• Virtualization brand: With the CPUID instruction, for some virtualization tools, given eax = 0x40000000, it could return the name of the virtualization tool, such as Microsoft HV or VMware in EBX, EDX, and ECX.
• MMX registers: MMX registers are a set of registers that were introduced by Intel that help speed up graphics calculations. Some virtualization tools don't support them. Some malware or packers use them for unpacking in order to detect or avoid running on a virtual machine.

Detecting virtualization processes and services

Virtualization tools mostly install tools on the guest machine to enable clipboard synchronization, drag and drop, mouse synchronization, and other useful features. These tools can be easily detected by scanning for these processes using the CreateToolhelp32Snapshot, Process32First, and Process32Next APIs. Some of these processes are as follows:

• VMware:
  • vmtoolsd.exe
  • vmacthlp.exe
  • VMwareUser.exe
  • VMwareService.exe
  • VMwareTray.exe 
• VirtualBox:
  • VBoxService.exe
  • VBoxTray.exe

Detecting virtualization through registry keys

There are multiple registry keys that can be used to detect virtualization environments. Some of them are related to the hard disk name (which is usually named after the virtualization software), the installed virtualization sync tools, or to other settings for the virtualization process. Some of these registry entries are as follows:

• HKLMSOFTWAREVmware Inc.Vmware Tools
• SYSTEMCurrentControlSetControlVirtualDeviceDrivers
• HKEY_LOCAL_MACHINESYSTEMControlSet001ControlClass{4D36E968-E325-11CE-BFC1-08002BE10318}000ProviderName
• HKEY_LOCAL_MACHINEHARDWAREACPIDSDTVBOX__
• HKEY_LOCAL_MACHINESOFTWAREOracleVirtualBox Guest Additions

Detecting virtual machines using PowerShell

It's not just registry values that reveal lots of information about the virtualization tools—Windows-managed information, which is accessible using PowerShell, can also be used, as shown in the following screenshot:

This information can also be accessed through a WMI query, such as the following:

SELECT * FROM Win32_ComputerSystem WHERE Manufacturer LIKE "%VMware%" AND Model LIKE "%VMware Virtual Platform%"

For Microsoft Hyper-V, it would be as follows:

SELECT * FROM Win32_ComputerSystem WHERE Manufacturer LIKE "%Microsoft Corporation%" AND Model LIKE "%Virtual Machine%"

These techniques make it harder to hide the fact that this malware is running inside virtualization software and not on a real machine.

Detecting sandboxes by using default settings

Sandboxes are sometimes easier to detect. They have lots of default settings that malware authors can use to identify them. The usernames could be default values, such as cuckoo or user. The filesystem could include the same decoy files and the same structure of the files (if not, then the same number of files). These settings can be easily detected for commonly used sandboxes, without even looking at their known tools and processes.

Another way to evade sandboxes is to avoid performing malicious activities in their analysis time window. These sandboxes execute malware for several seconds or minutes and then collect the necessary information before terminating the virtual machine. Some malware families use APIs such as Sleep to skip the execution for quite some time or run it after a machine restart. This trick can help evade sandboxes and ensure that they don't collect important information, such as C&C domains or malware-persistence techniques.

Other techniques

There are lots of other techniques that malware families can use to detect virtualized environments, such as the following:

• Connecting to VirtualBox inter-process communication: \.pipeVBoxTrayIPC
• Detecting other virtualization software files, such as VBoxHook.dll
• Detecting their window title or window class name, such as VBoxTrayToolWndClass or VBoxTrayToolWnd
• The MAC address of their network adapter

This list can be further expanded with many similar techniques and approaches for detecting a virtualized environment.

Summary

In this chapter, we have covered many tricks that malware authors use to detect and evade reverse engineering, from detecting the debugger and its breakpoints to detecting virtual machines and sandboxes, as well as incorporating obfuscation and debugger-escaping techniques. By the end of this chapter, you will be able to analyze more advanced malware equipped with multiple anti-debugging or anti-VM tricks. You will also be able to analyze a highly obfuscated malware implementing lots of anti-disassembling tricks.

In Chapter 6, Understanding Kernel-Mode Rootkits, we are going to enter the operating system's core. We are going to cover the kernel mode and learn how each API call and each operation works internally in the Windows operating system, as well as how rootkits can hook each of these steps to hide malicious activity from antivirus products and the user's eyes.