Entry to/Exit from Functions

You can attach an eBPF program to be triggered whenever a kernel function is entered or exited. Many of today’s eBPF examples use the mechanism of kprobes (attached to a kernel function entry point) and kretprobes (function exit). In more recent kernel versions, there is a more efficient alternative called fentry/fexit.³

Note that you can’t guarantee that all functions defined in one kernel version will necessarily be available in future versions unless they are part of a stable API such as the syscall interface.

You can also attach eBPF programs to user space functions with uprobes and uretprobes.

Tracepoints

You can also attach eBPF programs to tracepoints⁴ defined within the kernel. Find the events on your machine by looking under /sys/kernel/debug/tracing/events.

Perf Events

Perf⁵ is a subsystem for collecting performance data. You can hook eBPF programs to all the places where perf data is collected, which can be determined by running perf list on your machine.

Linux Security Module Interface

The LSM interface allows for security policies to be checked before the kernel allows certain operations. You may have come across AppArmor or SELinux that make use of this interface. With eBPF, you can attach custom programs to the same checkpoints, allowing for flexible, dynamic security policies and some new approaches to runtime security tooling.

Network Interfaces—eXpress Data Path

eXpress Data Path (XDP) allows attaching an eBPF program to a network interface, so that it is triggered whenever a packet is received. It can inspect or even modify the packet, and the program’s exit code can tell the kernel what to do with that packet: pass it on, drop it, or redirect it. This can form the basis of some very efficient networking functionality.⁶

Sockets and Other Networking Hooks

You can attach eBPF programs to run when applications open or perform other operations on a network socket, as well as when messages are sent or received. There are also hooks called traffic control or tc within the kernel’s network stack where eBPF programs can run after initial packet processing.

Some features can be implemented with an eBPF program alone, but in many cases we want the eBPF code to receive information from, or pass data to, a user space application. The mechanism that allows data to pass between eBPF programs and user space, or between different eBPF programs, is called maps.

Opensnoop eBPF Code

The eBPF code is written in C, in the file opensnoop.bpf.c. Near the beginning of this file you can see the definitions of two eBPF maps—start and events:

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);
    __type(value, struct args_t);
} start SEC(".maps");
struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(u32));
    __uint(value_size, sizeof(u32));
} events SEC(".maps");

When the ELF object file is created, it contains a section for each map and each program to be loaded into the kernel, and the SEC() macro defines these sections.

As you’ll see when we look into the program, the start map is used to temporarily store the arguments to the syscall—including the name of the file being opened—while the syscall is being processed. The events map⁷ is used for passing event information from the eBPF code in the kernel to the user space executable. This is illustrated in Figure 3-2.

Figure 3-2. Calling open() triggers eBPF programs that store data in opensnoop’s eBPF maps

Later in the opensnoop.bpf.c file, you’ll find two extremely similar functions:

SEC("tracepoint/syscalls/sys_enter_open")
int tracepoint__syscalls__sys_enter_open(struct 
    trace_event_raw_sys_enter* ctx)

and

SEC("tracepoint/syscalls/sys_enter_openat")
int tracepoint__syscalls__sys_enter_openat(struct 
    trace_event_raw_sys_enter* ctx)

There are two different system calls for opening files:⁸ openat() and open(). They are identical except that openat() has an extra argument for a directory file descriptor, and the path name for the file to be opened is taken relative to that directory. Likewise, the two functions in opensnoop are identical except for handling this difference in the arguments.

As you can see, they both take a parameter that is a pointer to a structure called trace_event_raw_sys_enter. You’d find the definition for this structure in the vmlinux header file generated for the particular kernel you’re running on. The art of writing eBPF programs includes working out what structure each program receives as its context, and how to access the information within it.

These two functions use a BPF helper function to retrieve the ID of the process that’s calling this syscall:

u64 id = bpf_get_current_pid_tgid();

The code gets the filename and any flags that were passed to the syscall, and puts them in a structure called args:

args.fname = (const char *)ctx->args[0];
         args.flags = (int)ctx->args[1];

This structure is written into the start map using the current process ID as the key:

bpf_map_update_elem(&start, &pid, &args, 0);

And that’s all that the eBPF programs do on entry to the syscall. But there’s another pair of eBPF programs defined in opensnoop.bpf.c that get triggered when the syscalls exit:

SEC("tracepoint/syscalls/sys_exit_open")
int tracepoint__syscalls__sys_exit_open

This program and its openat() twin share common code in the function trace_exit(). Have you noticed that all the functions called by eBPF programs are prefixed by static __always_inline? That forces the compiler to put the instructions for these functions inline, because in older kernels a BPF program is not allowed to jump to a separate function. Newer kernels and versions of LLVM can support noninlined function calls, but this is a safe way to ensure the BPF verifier stays happy. (Nowadays there is also the concept of a BPF tail call, where execution jumps from one BPF program to another. You can read more about BPF function calls and tail calls in the eBPF documentation.)

The trace_exit() function creates an empty event structure:

struct event event = {};

This will get populated with information about the open/openat syscall that’s coming to a conclusion and sent to user space via the events map.

There should be an entry in the start hash map that corresponds to the current process ID:

ap = bpf_map_lookup_elem(&start, &pid);

This has the information about the filename and flags that was written earlier during the sys_enter_open(at) call. The flags field is an integer stored directly in the structure, so it’s OK to read it directly from the structure:

event.flags = ap->flags;

In contrast, the filename is written into some number of bytes in user space memory, and the verifier needs to be sure that it’s safe for this eBPF program to read that number of bytes from that location in memory. This is done using another helper function, bpf_probe_read_user_str():

bpf_probe_read_user_str(&event.fname, sizeof(event.fname), 
                    ap->fname);

The current command name (that is, the name of the executable that made the open(at) syscall) is also copied into the event structure, using another BPF helper function:

bpf_get_current_comm(&event.comm, sizeof(event.comm));

The event structure gets written into the events perf buffer map:

bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
                    &event, sizeof(event));

The user space code reads event information out of this map. Before we get to that, let’s look briefly at the Makefile.

libbpf-tools Makefile

When you build eBPF code, you get an object file containing the binary definitions of the eBPF programs and maps. You also need an additional user space executable that will load those programs and maps into the kernel, and act as the interface for the user.⁹ Let’s look at the Makefile that builds opensnoop to see how it creates both the eBPF object file and the executable.

Makefiles comprise a set of rules, and the syntax for these can be a bit opaque, so if you’re not familiar with Makefiles and don’t particularly care about the details, please do feel free to skip over this section!

The opensnoop example that we’re looking at is one of a large set of example tools that are all built using one Makefile that you’ll find in the libbpf-tools directory. Not everything in this file is particularly of interest, but there are a few rules I’d like to highlight. The first is a rule that takes a bpf.c file and uses the clang compiler to create a BPF target object file:

$(OUTPUT)/%.bpf.o: %.bpf.c $(LIBBPF_OBJ) $(wildcard %.h) $(AR..
    $(call msg,BPF,$@)
    $(Q)$(CLANG) $(CFLAGS) -target bpf -D__TARGET_ARCH_$(ARCH) 
          -I$(ARCH)/ $(INCLUDES) -c $(filter %.c,$^) -o $@ && 
    $(LLVM_STRIP) -g $@

So, opensnoop.bpf.c gets compiled into $(OUTPUT)/opensnoop.bpf.o. This object file contains the eBPF programs and maps that will get loaded into the kernel.

Another rule uses bpftool gen skeleton to create a skeleton header file from the map and program definitions contained in that bpf.o object file:

$(OUTPUT)/%.skel.h: $(OUTPUT)/%.bpf.o | $(OUTPUT)
   $(call msg,GEN-SKEL,$@)
   $(Q)$(BPFTOOL) gen skeleton $< > $@

The opensnoop.c user space code includes this opensnoop.skel.h header file to get the definitions of the maps that it shares with the eBPF programs in the kernel. This allows the user space and kernel code to know about the layout of the data structures that get stored in these maps.

The following rule compiles the user space code from opensnoop.c into a binary object called $(OUTPUT)/opensnoop.o:

$(OUTPUT)/%.o: %.c $(wildcard %.h) $(LIBBPF_OBJ) | $(OUTPUT)
   $(call msg,CC,$@)
   $(Q)$(CC) $(CFLAGS) $(INCLUDES) -c $(filter %.c,$^) -o $@

Finally, there is a rule that uses cc to link the user space application objects (in our case, opensnoop.o) into a set of executables:

$(APPS): %: $(OUTPUT)/%.o $(LIBBPF_OBJ) $(COMMON_OBJ) | $(OUT...
   $(call msg,BINARY,$@)
   $(Q)$(CC) $(CFLAGS) $^ $(LDFLAGS) -lelf -lz -o $@

Now that you have seen how the eBPF and user space programs are generated separately, let’s look at the user space code.

Opensnoop User Space Code

As I mentioned, the user space code that interacts with eBPF code could be written in pretty much any programming language. The example that we’ll discuss in this section is written in C, but if you’re interested, you could compare it with the original BCC version written in Python, that you’ll find in bcc/tools.

The user space code is in the opensnoop.c file. The first half of the file has #include directives (one of them being the autogenerated opensnoop.skel.h file), various definitions, and the code to handle different command line options, which we won’t dwell on here. Let’s also gloss over functions like print_event() which writes the information about an event to the screen. From an eBPF perspective, all the interesting code is in the main() function.

You will see functions like opensnoop_bpf__open(), opensnoop_bpf__load(), and opensnoop_bpf__attach(). These are all defined in the autogenerated code created by bpftool gen skeleton.¹⁰ This autogenerated code handles all the individual eBPF programs, maps, and attachment points defined in the eBPF object file.

Once opensnoop is up and running, its job is to listen on the events perf buffer and write the information contained in each event to the screen. First, it opens the file descriptor associated with the perf buffer and sets handle_event() as the function to be called when a new event arrives:

pb = perf_buffer__new(bpf_map__fd(obj->maps.events), 
    PERF_BUFFER_PAGES, handle_event, handle_lost_events, 
    NULL, NULL);

Then it polls on buffer events until either a time limit is reached, or the user interrupts the program:

while (!exiting) {
         err = perf_buffer__poll(pb, PERF_POLL_TIMEOUT_MS);
...
}

The data parameter passed to handle_event() points to the event structure that the eBPF program wrote into the map for this event. The user space code can retrieve this information, format it and write it out for the user to see.

As you’ve seen, opensnoop registers eBPF programs that are called every time any application calls the open() or openat() system call. These eBPF programs running in the kernel collect information about the context of that system call—the executable name and process ID—and about the file being opened. This information is written into a map, from which user space can read it and display it to the user.

You’ll find dozens more examples of eBPF tools like this in the libbpf-tools directory, each of which typically instruments one syscall, or a family of related syscalls like open() and openat().

System calls are a stable kernel interface, and they offer a very powerful way to observe what’s happening on a (virtual) machine. But don’t be fooled into thinking that eBPF programming begins and ends at intercepting system calls. There are plenty of other stable interfaces, including LSM and various points in the networking stack, to which eBPF can be attached. If you’re willing to risk or work around changes between kernel versions, the range of places where you can attach eBPF programs is absolutely vast.