cdecl is one of the most popular conventions and was described in Chapter 4 when we introduced the stack and function calls. In cdecl, parameters are pushed onto the stack from right to left, the caller cleans up the stack when the function is complete, and the return value is stored in EAX. Example 6-17 shows an example of what the disassembly would look like if the code in Example 6-16 were compiled to use cdecl.

push c
push b
push a
call test
add esp, 12
mov ret, eax

Notice in the highlighted portion that the stack is cleaned up by the caller. In this example, the parameters are pushed onto the stack from right to left, beginning with c.

The popular stdcall convention is similar to cdecl, except stdcall requires the callee to clean up the stack when the function is complete. Therefore, the add instruction highlighted in Example 6-17 would not be needed if the stdcall convention were used, since the function called would be responsible for cleaning up the stack.

The test function in Example 6-16 would be compiled differently under stdcall, because it must be concerned with cleaning up the stack. Its epilogue would need to take care of the cleanup.

stdcall is the standard calling convention for the Windows API. Any code calling these API functions will not need to clean up the stack, since that’s the responsibility of the DLLs that implement the code for the API function.

The fastcall calling convention varies the most across compilers, but it generally works similarly in all cases. In fastcall, the first few arguments (typically two) are passed in registers, with the most commonly used registers being EDX and ECX (the Microsoft fastcall convention). Additional arguments are loaded from right to left, and the calling function is usually responsible for cleaning up the stack, if necessary. It is often more efficient to use fastcall than other conventions, because the code doesn’t need to involve the stack as much.

In addition to using the different calling conventions described so far, compilers may also choose to use different instructions to perform the same operation, usually when the compiler decides to move rather than push things onto the stack. Example 6-18 shows a C code example of a function call. The function adder adds two arguments and returns the result. The main function calls adder and prints the result using printf.

int adder(int a, int b)
{
   return a+b;
}

void main()
{
   int x = 1;
   int y = 2;

   printf("the function returned the number %d
", adder(x,y));
}

The assembly code for the adder function is consistent across compilers and is displayed in Example 6-19. As you can see, this code adds arg_0 to arg_4 and stores the result in EAX. (As discussed in Chapter 4, EAX stores the return value.)

00401730        push    ebp
00401731        mov     ebp, esp
00401733        mov     eax, [ebp+arg_0]
00401736        add     eax, [ebp+arg_4]
00401739        pop     ebp
0040173A        retn

Table 6-1 displays different calling conventions used by two different compilers: Microsoft Visual Studio and GNU Compiler Collection (GCC). On the left, the parameters for adder and printf are pushed onto the stack before the call. On the right, the parameters are moved onto the stack before the call. You should be prepared for both types of calling conventions, because as an analyst, you won’t have control over the compiler. For example, one instruction on the left does not correspond to any instruction on the right. This instruction restores the stack pointer, which is not necessary on the right because the stack pointer is never altered.

00401746   mov     [ebp+var_4], 1
0040174D   mov     [ebp+var_8], 2
00401754   mov     eax, [ebp+var_8]
00401757   push    eax
00401758   mov     ecx, [ebp+var_4]
0040175B   push    ecx
0040175C   call    adder
00401761   add     esp, 8
00401764   push    eax
00401765   push    offset TheFunctionRet
0040176A   call    ds:printf

00401085    mov     [ebp+var_4], 1
0040108C    mov     [ebp+var_8], 2
00401093    mov     eax, [ebp+var_8]
00401096    mov     [esp+4], eax
0040109A    mov     eax, [ebp+var_4]
0040109D    mov     [esp], eax
004010A0    call    adder

004010A5    mov     [esp+4], eax
004010A9    mov     [esp], offset TheFunctionRet
004010B0    call    printf