In order to demonstrate the basic operation of an IDA loader, we introduce the completely fictitious simpleton file format as defined by the following C struct (all values are little-endian):

struct simpleton {
   uint32_t magic; //simpleton magic number: 0x1DAB00C
   uint32_t size;  //size of the code array
   uint32_t base;  //base virtual address and entry point
   uint8_t code[size]; //the actual program code
};

The file format is very straightforward: a magic number file identifier and two integers describing the structure of the file, followed by all of the code contained in the file. Execution of the file begins with the first byte in the code block.

A hexdump of a small simpleton file might look like this:

0000000: 0cb0 da01 4900 0000 0040 0000 31c0 5050  [email protected]
0000010: 89e7 6a10 5457 50b0 f350 cd91 5859 4151  ..j.TWP..P..XYAQ
0000020: 50cd 9166 817f 0213 8875 f16a 3e6a 025b  P..f.....u.j>j.[
0000030: 5853 6a09 516a 3ecd 914b 79f4 5068 6e2f  XSj.Qj>..Ky.Ph//
0000040: 7368 682f 2f62 6989 e350 5389 e150 5153  shh/bin..PS..PQS
0000050: b03b 50cd 91                             .;P..

Several sample loaders are included with the SDK and may be found in the <SDKDIR>/ldr directory. We elect to build our loaders in individual subdirectories alongside the example loaders. In this case we are working in <SDKDIR>/ldr/simpleton. Our loader begins with the following setup:

#include "../idaldr.h"
#define SIMPLETON_MAGIC 0x1DAB00C

struct simpleton {
   uint32_t magic; //simpleton magic number: 0x1DAB00C
   uint32_t size;  //size of the code array
   uint32_t base;  //base virtual address and entry point
};

The idaldr.h header file is a convenience file, included with the SDK (<SDKDIR>/ldr/idaldr.h), which includes several other header files and defines several macros, all of which are commonly used in loader modules.

The next order of business is to declare the required LDSC object, which points to the various functions that implement our loader’s behavior:

int idaapi accept_simpleton_file(linput_t *, char[MAX_FILE_FORMAT_NAME], int);
void idaapi load_simpleton_file(linput_t *, ushort, const char *);
int idaapi save_simpleton_file(FILE *, const char *);

loader_t LDSC = {
  IDP_INTERFACE_VERSION,
  0,                      // loader flags
  accept_simpleton_file,  // test simpleton format.
  load_simpleton_file,    // load file into the database.
  save_simpleton_file,    // simpleton is an easy format to save
  NULL,                   // no special handling for moved segments
  NULL,                   // no special handling for File->New
};

The functions used in this loader are described in the order in which they might be invoked, beginning with the accept_simpleton_loader function shown here:

int idaapi accept_simpleton_file(linput_t *li,
                              char fileformatname[MAX_FILE_FORMAT_NAME], int n) {
   uint32 magic;
   if (n || lread4bytes(li, &magic, false)) return 0;
   if (magic != SIMPLETON_MAGIC) return 0;   //bad magic number found
   qsnprintf(fileformatname, MAX_FILE_FORMAT_NAME, "Simpleton Executable");
   return 1;  //simpleton format recognized
}

The entire purpose of this function is to determine whether the file being opened appears to be a simpleton file. The n parameter is a counter that indicates the number of times that our accept_file function has been called during the current loading process. The intent of this parameter is to allow a loader to recognize multiple related file formats. IDA will invoke your accept_file function with increasing values of n until your function returns 0. For each unique format that your loader recognizes, you should fill in the fileformatname array and return nonzero. In this case, we elect to ignore anything other than the first call (when n is zero) by immediately returning 0. The lread4bytes function, defined in diskio.hpp, is used to read the 4-byte magic number, and it returns 0 if the read completed successfully. A useful feature of lread4bytes is its ability to read bytes in either big-endian or little-endian format, depending on the value of its Boolean third parameter (false reads little-endian; true reads big-endian). This feature can help reduce the number of calls to byte-swapping functions required during the loading process. If the required magic number is located, the final step in accept_simpleton_file is to copy the name of the file format into the fileformatname output parameter prior to returning 1 to indicate that the file format was recognized.

For the simpleton loader, no special processing is required if a user chooses to load a simpleton file using File ▸ New rather than File ▸ Open, so no init_loader_options function is required. Therefore, the next function called in the loading sequence will be load_simpleton_file, which is shown here:

void idaapi load_simpleton_file(linput_t *li, ushort neflags, const char *) {
   simpleton hdr;
   //read the program header from the input file
   lread(li, &hdr, sizeof(simpleton));
   //load file content into the database
   file2base(li, sizeof(simpleton), hdr.base, hdr.base + hdr.size,
             FILEREG_PATCHABLE);
   //create a segment around the file's code section
   if (!add_segm(0, hdr.base, hdr.base + hdr.size, NAME_CODE, CLASS_CODE)) {
      loader_failure();
   }
   //retrieve a handle to the new segment
   segment_t *s = getseg(hdr.base);
   //so that we can set 32 bit addressing mode on (x86 has 16 or 32 bit modes)
   set_segm_addressing(s, 1);  //set 32 bit addressing
   //tell IDA to create the file header comment for us.  Do this
   //only once. This comment contains license, MD5,
   // and original input file name information.
   create_filename_cmt();
   //Add an entry point so that the processor module knows at least one
   //address that contains code.  This is the root of the recursive descent
   //disassembly process
   add_entry(hdr.base, hdr.base, "_start", true);
}

The bulk of the loading process takes place in a loader’s load_file function. Our simple loader performs the following tasks:

The file2base function is a workhorse function for loaders. Its prototype appears here:

int ida_export file2base(linput_t *li, long pos, ea_t ea1, ea_t ea2, int patchable);

This function reads bytes from the provided linput_t beginning at the file position specified by pos. The bytes are loaded into the database beginning at address ea1, up to but not including ea2. The total number of bytes read is calculated as ea2-ea1. The patchable parameter indicates whether IDA should maintain an internal mapping of file offsets to their corresponding locations in the database. To maintain such a mapping, this parameter should be set to FILEREG_PATCHABLE, which allows for the generation of IDA DIF files, as discussed in Chapter 14.

The add_entry function is another important function in the loading process. The disassembly process can begin only with addresses known to contain instructions. For a recursive descent disassembler, such addresses are generally obtained by parsing a file for entry points (such as exported functions). The prototype for add_entry appears here:

bool ida_export add_entry(uval_t ord, ea_t ea, const char *name, bool makecode);

The ord parameter is useful for exported functions that may be exported by ordinal number in addition to function name. If the entry point has no associated ordinal number, ord should be set to the same value as the ea parameter. The ea parameter specifies the effective address of the entry point, while the name parameter specifies the name associated with the entry point. The symbolic name _start is often applied to a program’s initial execution address. The boolean makecode parameter specifies whether the specified address is to be treated as code (true) or not (false). Exported data items, such as LDSC within a loader module, are examples of noncode entry points.

The final function that we have implemented in the simpleton loader, save_simpleton_file, is used to create a simpleton file from the database contents. Our implementation is shown here:

int idaapi save_simpleton_file(FILE *fp, const char *fileformatname) {
   uint32 magic = SIMPLETON_MAGIC;
   if (fp == NULL) return 1;   //special case, success means we can save files
   segment_t *s = getnseg(0);  //get segment zero, the one and only segment
   if (s) {
      uint32 sz = s->endEA - s->startEA;    //compute the segment size
      qfwrite(fp, &magic, sizeof(uint32));  //write the magic value
      qfwrite(fp, &sz, sizeof(uint32));     //write the segment size
      qfwrite(fp, &s->startEA, sizeof(uint32));  //write the base address
      base2file(fp, sizeof(simpleton), s->startEA, s->endEA); //dump the segment
      return 1;  //return success
   }
else {
      return 0;  //return failure
   }
}

A loader_t’s save_file function receives a FILE stream pointer, fp, to which the function should write its output. The fileformatname parameter is the same name filled in by the loader’s accept_file function. As mentioned earlier, the save_file function is called in response to IDA’s File ▸ Produce File ▸ Create EXE File command. In response to this command, IDA initially calls save_file with fp set to NULL. When called in this manner, save_file is being queried as to whether it can produce an output file of the type specified by fileformatname, in which case save_file should return 0 if it cannot create the specified file type or 1 if it can create the specified file. For example, the loader may be able to create a valid output file only if specific information is present within the database.

When called with a valid (non-NULL) FILE pointer, save_file should write a valid output file representation to the provided FILE stream. In such cases, IDA creates the FILE stream after presenting the user with a File Save dialog.

Returning to the save_simpleton_file function, the only truly interesting function used in implementing our save_file capability is the base2file function, which is the output counterpart to the file2base function used in load_simpleton_file. The base2file function simply writes a range of database values to a specified position within a supplied FILE stream.

While the simpleton file format borders on useless, it does serve one purpose, namely that it has allowed us to demonstrate the core functionality of IDA loader modules. The source code for the simpleton loader may be found on the book’s website.

The process for building and installing an IDA loader module is virtually identical to the process for building an IDA plug-in module as discussed in Chapter 17, with only a few minor differences. First, the file extensions used for loaders are .ldw/.l64 on Windows, .llx/.llx64 on Linux platforms, and .lmc/.lmc64 on OS X. Second, this is a matter of personal preference, but when we build loaders, we store the newly created loader binaries into <SDKDIR>/bin/loaders. Third, loader modules are installed by copying the compiled loader binary to <IDADIR>/loaders. The plug-in makefile presented in Example 17-1 is easily adapted to build the simpleton loader by changing the PLUGIN_EXT variable to a LOADER_EXT variable that reflects the proper loader file extensions for each IDA platform, changing all references to idabook_plugin to simpleton, and changing the OUTDIR variable to point to $(IDA)/bin/loaders.

Granted, the majority of network packets do not contain code that can be disassembled. However, if the packets happen to contain evidence of an exploit, the packets may contain binary code that might require disassembly for proper analysis. In order to demonstrate that IDA loaders can be used for many purposes, we now describe the construction of a loader capable of loading a pcap^[132] format packet-capture file into an IDA database. While this may be somewhat over the top, along the way we will demonstrate several more capabilities of IDA’s SDK. No attempt is made here to match the capabilities of tools such as Wireshark^[133] in any way.

The development process for such a loader requires some research into the pcap file format, which reveals that a pcap file is structured with the following rough syntax:

pcap_file: pcap_file_header (pcap_packet)*
pcap_packet: pcap_packet_header pcap_content
pcap_content: (byte)+

A pcap_file_header contains a 32-bit magic number field, as well as other fields describing the content of the file, including the type of packets contained in the file. For the sake of simplification, we assume here that we are dealing only with DLT_EN10MB (10Mb Ethernet packets). In developing the pcap loader, one of our goals is to identify as much header data as possible in order to help users focus on packet content, particularly at the application layer. Our approach for accomplishing this goal is (1) to separate the file header from the packet data by creating a separate segment for each and (2) to identify as many header structures as possible with the packets segment so that the user does not need to manually parse the file content. The discussion that follows focuses only on the load_file component of the pcap loader, because the accept_file function is a simple adaptation of the accept_simpleton_file function changed to recognize the pcap magic number.

In order to highlight header structures, we will need to have some commonly used structures defined in the IDA Structures window during the loading phase. This allows the loader to automatically format groups of bytes as structures when the datatype for those bytes is known. Pcap header structures and various networking-related structures describing Ethernet, IP, TCP, and UDP headers are defined in IDA’s GNU C++ Unix type library; however, in versions of IDA prior to 5.3, the definition for the IP header struct (iphdr) is incorrect. The first step that load_pcap_file takes is to call a helper function we have written named add_types to take care of importing structures into the new database. We examine two possible versions of add_types, one that makes use of the types declared in IDA’s GNU C++ Unix type library and another version in which add_types takes care of all required structure declarations by itself.

The first version loads the GNU C++ Unix type library and then pulls type identifiers from the newly loaded type library. This version of add_types is shown here:

void add_types() {
#ifdef ADDTIL_DEFAULT
   add_til2("gnuunx.til", ADDTIL_SILENT);
#else
   add_til("gnuunx.til");
#endif
   pcap_hdr_struct = til2idb(-1, "pcap_file_header");
   pkthdr_struct = til2idb(-1, "pcap_pkthdr");
   ether_struct = til2idb(-1, "ether_header");
   ip_struct = til2idb(-1, "iphdr");
   tcp_struct = til2idb(-1, "tcphdr");
   udp_struct = til2idb(-1, "udphdr");
}

The add_til functions defined in typinf.hpp are used to load an existing type library file into a database. The add_til function was deprecated in favor of add_til2 with the introduction of IDA version 5.1. These functions are the SDK equivalent of loading a .til file using the Types window discussed in Chapter 8. Once a type library has been loaded, the til2idb function may be utilized to import individual types into the current database. This is the programmatic equivalent of adding a standard structure to the Structures window, which was also described in Chapter 8. The til2idb function returns a type identifier that is required whenever we want to convert a range of bytes into a specific structured datatype. We have chosen to save these type identifiers into global variables (each of type tid_t) in order to provide faster access to types later in the loading process.

Two drawbacks to this first version of add_types are the fact that we need to import an entire type library just to gain access to six datatypes and, as mentioned previously, the built-in IDA definition of a structure may be incorrect, which would lead to problems when we attempt to apply these structures later in the loading process.

The second version of add_types demonstrates the process of building a type library on the fly by parsing actual C-style structure declarations. This version is shown here:

void add_types() {
   til_t *t = new_til("pcap.til", "pcap header types"); //empty type library
   parse_decls(t, pcap_types, NULL, HTI_PAK1); //parse C declarations into library
   sort_til(t);                                //required after til is modified
   pcap_hdr_struct = import_type(t, −1, "pcap_file_header");
   pkthdr_struct = import_type(t, −1, "pcap_pkthdr");
   ether_struct = import_type(t, −1, "ether_header");
   ip_struct = import_type(t, −1, "iphdr");
   tcp_struct = import_type(t, −1, "tcphdr");
   udp_struct = import_type(t, −1, "udphdr");
   free_til(t);                                  //free the temporary library
}

In this case, a temporary, empty type library is created using the new_til function. The new type library is populated by parsing a string (pcap_types) that contains valid C structure definitions for the types required by the loader. The first few lines of the pcap_types string are shown here:

char *pcap_types =
   "struct pcap_file_header {
"
        "int magic;
"
        "short version_major;
"
        "short version_minor;
"
        "int thiszone;
"
        "int sigfigs;
"
        "int snaplen;
"
        "int linktype;
"
   "};
"
   ...

The declaration of pcap_types continues and includes structure definitions for all of the structures required by the pcap loader. In order to simplify the parsing process, we elected to change all data declarations used within the structure definitions to make use of standard C datatypes.

The HTI_PAK1 constant is defined in typeinf.hpp and is one of many HTI_XXX values that may be used to control the behavior of the internal C parser. In this case, structure packing on a 1-byte boundary is being requested. Following modification, a type library is expected to be sorted using sort_til, at which point it is ready to use. The import_type function pulls the requested structure type from the specified type library into the database in a manner similar to til2idb. In this version, again we save the returned type identifier into global variables for use later in the loading process. The function completes by deleting the temporary type library using the free_til function to release the memory consumed by the type library. In this version of add_types, unlike the first version, we have complete control over the datatypes that we choose to import into the database, and we have no need to import entire libraries of structures that we have no intention of using.

As an aside, it is also possible to save the temporary type library file to disk using the store_til function (which should be preceded by a call to compact_til). With so few types to construct, this has little benefit in this case, because it is just as easy to build the structures each time the loader is executed as it is to build and distribute a special-purpose type library that must be properly installed and in the end does not save a significant amount of time.

Turning our attention to the load_pcap_file function, we see the call to add_types to initialize the datatypes, as discussed previously; the creation of a file comment; followed by loading the pcap file header into the database, creating a section around the header bytes, and transforming the header bytes into a pcap_file_header structure:

void idaapi load_pcap_file(linput_t *li, ushort, const char *) {
   ssize_t len;
   pcap_pkthdr pkt;

   add_types();              //add structure templates to database
   create_filename_cmt();    //create the main file header comment
   //load the pcap file header from the database into the file
   file2base(li, 0, 0, sizeof(pcap_file_header), FILEREG_PATCHABLE);
   //try to add a new data segment to contain the file header bytes
   if (!add_segm(0, 0, sizeof(pcap_file_header), ".file_header", CLASS_DATA)) {
      loader_failure();
   }
   //convert the file header bytes into a pcap_file_header
   doStruct(0, sizeof(pcap_file_header), pcap_hdr_struct);
   //... continues

Once again, we see the use of file2base to load content from the newly opened disk file into the database. Once the pcap file header content has been loaded, it gets its own section in the database, and the pcap_file_header structure is applied to all of the header bytes using the doStruct function, declared in bytes.hpp, which is the SDK equivalent of using Edit ▸ Struct Var to convert a contiguous block of bytes into a structure. The doStruct function expects an address, a size, and a type identifier, and it converts size bytes at the given address into the given type.

The load_pcap_file function continues by reading all of the packet content and creating a single .packets section around the packet content, as shown here:

//...continuation of load_pcap_file
   uint32 pos = sizeof(pcap_file_header);    //file position tracker
   while ((len = qlread(li, &pkt, sizeof(pkt))) == sizeof(pkt)) {
      mem2base(&pkt, pos, pos + sizeof(pkt), pos);  //transfer header to database
      pos += sizeof(pkt);       //update position pointer point to packet content
      //now read packet content based on number of bytes of packet that are
      //present
      file2base(li, pos, pos, pos + pkt.caplen, FILEREG_PATCHABLE);
      pos += pkt.caplen;        //update position pointer to point to next header
   }
   //create a new section around the packet content.  This section begins where
   //the pcap file header ended.
   if (!add_segm(0, sizeof(pcap_file_header), pos, ".packets", CLASS_DATA)) {
      loader_failure();
   }
   //retrieve a handle to the new segment
   segment_t *s = getseg(sizeof(pcap_file_header));
   //so that we can set 32 bit addressing mode on
   set_segm_addressing(s, 1);  //set 32 bit addressing
   //...continues

In the preceding code, the mem2base function is new and utilized to transfer content that has already been loaded into memory into the database.

The load_pcap_file function concludes by applying structure templates wherever possible throughout the database. We must apply structure templates after creating the segment; otherwise the act of creating the segment will remove all applied structure templates, negating all of our hard work. The third and final portion of the function is shown here:

//...continuation of load_pcap_file
   //apply headers structs for each packet in the database
   for (uint32 ea = s->startEA; ea < pos;) {
      uint32 pcap = ea;       //start of packet
      //apply pcap packet header struct
      doStruct(pcap, sizeof(pcap_pkthdr), pkthdr_struct);
      uint32 eth = pcap + sizeof(pcap_pkthdr);
      //apply Ethernet header struct
      doStruct(eth, sizeof(ether_header), ether_struct);
      //Test Ethernet type field
      uint16 etype = get_word(eth + 12);
      etype = (etype >> 8) | (etype << 8);  //htons

      if (etype == ETHER_TYPE_IP) {
         uint32 ip = eth + sizeof(ether_header);
         //Apply IP header struct
         doStruct(ip, sizeof(iphdr), ip_struct);
         //Test IP protocol
         uint8 proto = get_byte(ip + 9);
         //compute IP header length
         uint32 iphl = (get_byte(ip) & 0xF) * 4;
         if (proto == IP_PROTO_TCP) {
            doStruct(ip + iphl, sizeof(tcphdr), tcp_struct);
         }
         else if (proto == IP_PROTO_UDP) {
            doStruct(ip + iphl, sizeof(udphdr), udp_struct);
         }
      }
      //point to start of next pcak_pkthdr
      ea += get_long(pcap + 8) + sizeof(pcap_pkthdr);
   }
}

The preceding code simply steps through the database, one packet at a time, and examines a few fields within each packet header in order to determine both the type of structure to be applied and the location of the start of that structure. The following output represents the first few lines of a pcap file that has been loaded into a database using the pcap loader:

.file_header:0000 _file_header    segment byte public 'DATA' use16
.file_header:0000         assume cs:_file_header
.file_header:0000         pcap_file_header <0A1B2C3D4h, 2, 4, 0, 0, 0FFFFh, 1>
.file_header:0000 _file_header    ends
.file_header:0000
.packets:00000018 ; =========================================================
.packets:00000018
.packets:00000018 ; Segment type: Pure data
.packets:00000018 _packets  segment byte public 'DATA' use32
.packets:00000018            assume cs:_packets
.packets:00000018            ;org 18h
.packets:00000018            pcap_pkthdr <<47DF275Fh, 1218Ah>, 19Ch, 19Ch>
.packets:00000028            db 0, 18h, 0E7h, 1, 32h, 0F5h; ether_dhost
.packets:00000028            db 0, 50h, 0BAh, 0B8h, 8Bh, 0BDh; ether_shost
.packets:00000028            dw 8                    ; ether_type
.packets:00000036            iphdr <45h, 0, 8E01h, 0EE4h, 40h, 80h, 6, 9E93h,
                                    200A8C0h, 6A00A8C0h>
.packets:0000004A            tcphdr <901Fh, 2505h, 0C201E522h, 6CE04CCBh, 50h,
                                     18h, 0E01Ah, 3D83h, 0>
.packets:0000005E            db  48h ; H
.packets:0000005F            db  54h ; T
.packets:00000060            db  54h ; T
.packets:00000061            db  50h ; P
.packets:00000062            db  2Fh ; /
.packets:00000063            db  31h ; 1
.packets:00000064            db  2Eh ; .
.packets:00000065            db  30h ; 0

Applying structure templates in this manner, we can expand and collapse any header to show or hide its individual member fields. As displayed, it is fairly easy to observe that the byte at address 0000005E is the first byte of an HTTP response packet.

Having a basic loading capability for pcap files lays the groundwork for developing plug-ins that perform more sophisticated tasks, such as TCP stream reassembly and various other forms of data extraction. Additional work could go into formatting various networking-related structures in a more user-friendly manner, such as displaying readable versions of an IP address and hosting byte-ordered displays for other fields within each header. Such improvements are left as challenges to the reader.