Networking protocols place many demands on the memory management facilities of the kernel. These demands include easily manipulating buffers of varying sizes, prepending and appending data to the buffers as the lower layers encapsulate data from higher layers, removing data from buffers (as headers are removed as data packets are passed up the protocol stack), and minimizing the amount of data copied for all these operations. The performance of the networking protocols is directly related to the memory management scheme used within the kernel.

In Chapter 1 we introduced the memory buffer used throughout the Net/3 kernel: the mbuf, which is an abbreviation for “memory buffer.” In this chapter we look in more detail at mbufs and at the functions within the kernel that are used to manipulate them, as we will encounter mbufs on almost every page of the text. Understanding mbufs is essential for understanding the rest of the text.

The main use of mbufs is to hold the user data that travels from the process to the network interface, and vice versa. But mbufs are also used to contain a variety of other miscellaneous data: source and destination addresses, socket options, and so on.

Figure 2.1 shows the four different kinds of mbufs that we’ll encounter, depending on the M_PKTHDR and M_EXT flags in the m_flags member. The differences between the four mbufs in Figure 2.1, from left to right, are as follows:

Figure 2.1. Four different types of mbufs, depending on the m_flags value.

There are numerous additional points we need to make about Figure 2.1:

Figure 2.2 shows an example of two packets on a queue. It is a modification of Figure 1.8. We have placed the UDP datagram onto the interface output queue (showing that the 14-byte Ethernet header has been prepended to the IP header in the first mbuf on the chain) and have added a second packet to the queue: a TCP segment containing 1460 bytes of user data. The TCP data is contained in a cluster and an mbuf has been prepended to contain its Ethernet, IP, and TCP headers. With the cluster we show that the data pointer into the cluster (m_data) need not point to the front of the cluster. We show that the queue has a head pointer and a tail pointer. This is how the interface output queues are handled in Net/3. We have also added the m_ext structure to the mbuf with the M_EXT flag set and have shaded in the unused pkthdr structure of this mbuf.

Figure 2.2. Two packets on a queue: first with 192 bytes of data and second with 1514 bytes of data.

The first mbuf with the packet header for the UDP datagram has a type of MT_DATA, but the first mbuf with the packet header for the TCP segment has a type of MT_HEADER. This is a side effect of the different way UDP and TCP prepend the headers to their data, and makes no difference. Mbufs of these two types are essentially the same. It is the m_flags value of M_PKTHDR in the first mbuf on the chain that indicates a packet header.

Careful readers may note a difference between our picture of an mbuf (the Net/3 mbuf, Figure 2.1) and the picture in [Leffler et al. 1989, p. 290], a Net/1 mbuf. The changes were made in Net/2: adding the m_flags member, renaming the m_act pointer to be m_nextpkt, and moving this pointer to the front of the mbuf.

The difference in the placement of the protocol headers in the first mbuf for the UDP and TCP examples is caused by UDP calling M_PREPEND (Figure 23.15 and Exercise 23.1) while TCP calls MGETHDR (Figure 26.25).

The mbuf functions are in a single C file and the mbuf macros and various mbuf definitions are in a single header, as shown in Figure 2.3.

99 mbufs in use:
        1 mbufs allocated to data
        43 mbufs allocated to packet headers
        17 mbufs allocated to protocol control blocks
        20 mbufs allocated to socket names and addresses
        18 mbufs allocated to socket options
2/34 mapped pages in use
80 Kbytes allocated to network (20% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

m_mtypes[MT_DATA]
m_mtypes[MT_HEADER]
m_mtypes[MT_PCB]
m_mtypes[MT_SONAME]
m_mtypes[MT_SOOPTS]
(see text)
(see text)
m_drops
m_wait
m_drain

There are a few constants that we encounter repeatedly when dealing with mbufs. Their values are shown in Figure 2.7. All are defined in mbuf.h except MCLBYTES, which is defined in /usr/include/machine/param.h.

Figure 2.8 shows the definition of the mbuf structure.

-------------------------------------------------------------------------- mbuf.h
 60 /* header at beginning of each mbuf: */
 61 struct m_hdr {
 62     struct mbuf *mh_next;       /* next buffer in chain */
 63     struct mbuf *mh_nextpkt;    /* next chain in queue/record */
 64     int     mh_len;             /* amount of data in this mbuf */
 65     caddr_t mh_data;            /* pointer to data */
 66     short   mh_type;            /* type of data (Figure 2.10) */
 67     short   mh_flags;           /* flags (Figure 2.9) */
 68 };

 69 /* record/packet header in first mbuf of chain; valid if M_PKTHDR set */
 70 struct pkthdr {
 71     int     len;                /* total packet length */
 72     struct ifnet *rcvif;        /* receive interface */
 73 };

 74 /* description of external storage mapped into mbuf, valid if M_EXT set */
 75 struct m_ext {
 76     caddr_t ext_buf;            /* start of buffer */
 77     void    (*ext_free) ();     /* free routine if not the usual */
 78     u_int   ext_size;           /* size of buffer, for ext_free */
 79 };

 80 struct mbuf {
 81     struct m_hdr m_hdr;
 82     union {
 83         struct {
 84             struct pkthdr MH_pkthdr;    /* M_PKTHDR set */
 85             union {
 86                 struct m_ext MH_ext;    /* M_EXT set */
 87                 char    MH_databuf[MHLEN];
 88             } MH_dat;
 89         } MH;
 90         char    M_databuf[MLEN];    /* !M_PKTHDR, !M_EXT */
 91     } M_dat;
 92 };

 93 #define m_next      m_hdr.mh_next
 94 #define m_len       m_hdr.mh_len
 95 #define m_data      m_hdr.mh_data
 96 #define m_type      m_hdr.mh_type
 97 #define m_flags     m_hdr.mh_flags
 98 #define m_nextpkt   m_hdr.mh_nextpkt
 99 #define m_act       m_nextpkt
100 #define m_pkthdr    M_dat.MH.MH_pkthdr
101 #define m_ext       M_dat.MH.MH_dat.MH_ext
102 #define m_pktdat    M_dat.MH.MH_dat.MH_databuf
103 #define m_dat       M_dat.M_databuf
-------------------------------------------------------------------------- mbuf.h

The mbuf structure is defined as an m_hdr structure, followed by a union. As the comments indicate, the contents of the union depend on the flags M_PKTHDR and M_EXT.

93-103

These 11 #define statements simplify access to the members of the structures and unions within the mbuf structure. We will see this technique used throughout the Net/3 sources whenever we encounter a structure containing other structures or unions.

We previously described the purpose of the first two members in the mbuf structure: the m_next pointer links mbufs together into an mbuf chain and the m_nextpkt pointer links mbuf chains together into a queue of mbufs.

Figure 1.8 differentiated between the m_len member of each mbuf and the m_pkthdr.len member in the packet header. The latter is the sum of all the m_len members of all the mbufs on the chain.

There are five independent values for the m_flags member, shown in Figure 2.9.

We have already described the M_EXT and M_PKTHDR flags. M_EOR is set in an mbuf containing the end of a record. The Internet protocols (e.g., TCP) never set this flag, since TCP provides a byte-stream service without any record boundaries. The OSI and XNS transport layers, however, do use this flag. We will encounter this flag in the socket layer, since this layer is protocol independent and handles data to and from all the transport layers.

The next two flags, M_BCAST and M_MCAST, are set in an mbuf when the packet will be sent to or was received from a link-layer broadcast address or multicast address. These two constants are flags between the protocol layer and the interface layer (Figure 1.3).

The final value, M_COPYFLAGS, specifies the flags that are copied when an mbuf containing a packet header is copied.

Figure 2.10 shows the MT_xxx constants used in the m_type member to identify the type of data stored in the mbuf. Although we tend to think of an mbuf as containing user data that is sent or received, mbufs can contain a variety of different data structures. Recall in Figure 1.6 that an mbuf was used to hold a socket address structure with the destination address for the sendto system call. Its m_type member was set to MT_SONAME.

Not all of the mbuf type values in Figure 2.10 are used in Net/3. Some are historical (MT_HTABLE), and others are not used in the TCP/IP code but are used elsewhere in the kernel. For example, MT_OOBDATA is used by the OSI and XNS protocols, but TCP handles out-of-band data differently (as we describe in Section 29.7). We describe the use of other mbuf types when we encounter them later in the text.

The final column of this figure shows the M_xxx values associated with the piece of memory allocated by the kernel for the different types of mbufs. There are about 60 possible M_xxx values assigned to the different types of memory allocated by the kernel’s malloc function and MALLOC macro. Figure 2.6 showed the mbuf allocation statistics from the netstat -m command including the counters for each MT_xxx type. The vmstat -m command shows the kernel’s memory allocation statistics including the counters for each M_xxx type.

There are more than two dozen macros and functions that deal with mbufs (allocate an mbuf, free an mbuf, etc.). We look at the source code for only a few of the macros and functions, to show how they’re implemented.

Some operations are provided as both a macro and function. The macro version has an uppercase name that begins with M, and the function has a lowercase name that begins with m_. The difference in the two is the standard time-versus-space tradeoff. The macro version is expanded inline by the C preprocessor each time it is used (requiring more code space), but it executes faster since it doesn’t require a function call (which can be expensive on some architectures). The function version, on the other hand, becomes a few instructions each time it is invoked (push the arguments onto the stack, call the function, etc.), taking less code space but more execution time.

---------------------------------------------------------------------- uipc_mbuf.c
134 struct mbuf *
135 m_get(nowait, type)
136 int     nowait, type;
137 {
138     struct mbuf *m;

139     MGET(m, nowait, type);
140     return (m);
141 }
---------------------------------------------------------------------- uipc_mbuf.c

---------------------------------------------------------------------- mbuf.h
154 #define MGET(m, how, type) { 
155     MALLOC((m), struct mbuf *, MSIZE, mbtypes[type], (how)); 
156     if (m) { 
157         (m)->m_type = (type); 
158         MBUFLOCK(mbstat.m_mtypes[type]++;) 
159         (m)->m_next = (struct mbuf *)NULL; 
160         (m)->m_nextpkt = (struct mbuf *)NULL; 
161         (m)->m_data = (m)->m_dat; 
162         (m)->m_flags = 0; 
163     } else 
164         (m) = m_retry((how), (type)); 
165 }
---------------------------------------------------------------------- mbuf.h

MGET(m, M__WAIT, MT_SONAME);
if (m == NULL)
    return(ENOBUFS);

---------------------------------------------------------------------- uipc_mbuf.c
 92 struct mbuf *
 93 m_retry(i, t)
 94 int     i, t;
 95 {
 96     struct mbuf *m;

 97     m_reclaim();
 98 #define m_retry(i, t)   (struct mbuf *)0
 99     MGET(m, i, t);
100 #undef m_retry
101     return (m);
102 }
---------------------------------------------------------------------- uipc_mbuf.c

We encounter the m_pullup function when we show the code for IP, ICMP, IGMP, UDP, and TCP. It is called to guarantee that the specified number of bytes (the size of the corresponding protocol header) are contiguous in the first mbuf of a chain; otherwise the specified number of bytes are copied to a new mbuf and made contiguous. To understand the usage of m_pullup we must describe its implementation and its interaction with both the m_devget function and the mtod and dtom macros. This description also provides additional insight into the usage of mbufs in Net/3.

m_devget Function

When an Ethernet frame is received, the device driver calls the function m_devget to create an mbuf chain and copy the frame from the device into the chain. Depending on the length of the received frame (excluding the Ethernet header), there are four different possibilities for the resulting mbuf chain. The first two possibilities are shown in Figure 2.14.

Figure 2.14. First two types of mbufs created by m_devget.

1. The left mbuf in Figure 2.14 is used when the amount of data is between 0 and 84 bytes. In this figure we assume there are 52 bytes of data: a 20-byte IP header and a 32-byte TCP header (the standard 20-byte TCP header plus 12 bytes of TCP options) but no TCP data. Since the data in the mbuf returned by m_devget starts with the IP header, the realistic minimum value for m_len is 28: 20 bytes for an IP header, 8 bytes for a UDP header, and a 0-length UDP datagram.

   m_devget leaves 16 bytes unused at the beginning of the mbuf. Although the 14-byte Ethernet header is not stored here, room is allocated for a 14-byte Ethernet header on output, should the same mbuf be used for output. We’ll encounter two functions that generate a response by using the received mbuf as the outgoing mbuf: icmp_reflect and tcp_respond. In both cases the size of the received datagram is normally less than 84 bytes, so it costs nothing to leave room for 16 bytes at the front, which saves time when building the outgoing datagram. The reason 16 bytes are allocated, and not 14, is to have the IP header longword aligned in the mbuf.

2. If the amount of data is between 85 and 100 bytes, the data still fits in a packet header mbuf, but there is no room for the 16 bytes at the beginning. The data starts at the beginning of the m_pktdat array and any unused space is at the end of this array. The mbuf on the right in Figure 2.14 shows this example, assuming 85 bytes of data.

3. Figure 2.15 shows the third type of mbuf created by m_devget. Two mbufs are required when the amount of data is between 101 and 207 bytes. The first 100 bytes are stored in the first mbuf (the one with the packet header), and the remainder are stored in the second mbuf. In this example we show a 104-byte datagram. No attempt is made to leave 16 bytes at the beginning of the first mbuf.

   

   Figure 2.15. Third type of mbuf created by m_devget.

4. Figure 2.16 shows the fourth type of mbuf created by m_devget. If the amount of data is greater than or equal to 208 (MINCLBYTES), one or more clusters are used. The example in the figure assumes a 1500-byte Ethernet frame with 2048-byte clusters. If 1024-byte clusters are in use, this example would require two mbufs, each with the M_EXT flag set, and each pointing to a cluster.

   

   Figure 2.16. Fourth type of mbuf created by m_devget.

   #define mtod(m,t)   ((t)((m)->m_data))
   #define dtom(x)     ((struct mbuf *)((int)(x) & ~(MSIZE-1)))

struct mbuf *m;
struct ip *ip;

ip = mtod(m, struct ip *);
ip->ip_v = IPVERSION;

struct mbuf *m;
struct ip *ip;

m = dtom(ip);

     if (m->m_len < sizeof(struct ip) &&
         (m = m_pullup(m, sizeof(struct ip))) == 0)  {
             ipstat.ips_toosmall++;
             goto next;
     }
     ip = mtod(m, struct ip *);

m_pullup and IP Fragmentation and Reassembly

The second use of m_pullup concerns IP reassembly and TCP reassembly. Assume IP receives a packet of length 296, which is a fragment of a larger IP datagram. The mbuf passed from the device driver to IP input looks like the one we showed in Figure 2.16: the 296 bytes of data are stored in a cluster. We show this in Figure 2.17.

Figure 2.17. An IP fragment of length 296.

The problem is that the IP fragmentation algorithm keeps the individual fragments on a doubly linked list, using the source and destination IP address fields in the IP header to hold the forward and backward list pointers. (These two IP addresses are saved, of course, in the head of the list, since they must be put back into the reassembled datagram. We describe this in Chapter 10.) But if the IP header is in a cluster, as shown in Figure 2.17, these linked list pointers would be in the cluster, and when the list is traversed at some later time, the pointer to the IP header (i.e., the pointer to the beginning of the cluster) could not be converted into the pointer to the mbuf. This is the problem we mentioned earlier in this section: the dtom macro cannot be used if m_data points into a cluster, because there is no back pointer from the cluster to the mbuf. IP fragmentation cannot store the links in the cluster as shown in Figure 2.17.

To solve this problem the IP fragmentation routine always calls m__pullup when a fragment is received, if the fragment is contained in a cluster. This forces the 20-byte IP header into its own mbuf. The code looks like

if (m->m_flags & M_EXT) {
    if ((m = m_pullup (m, sizeof (struct ip))) == 0) {
        ipstat.ips_toosmall++;
        goto next;
    }
    ip = mtod(m, struct ip *);
}

Figure 2.18 shows the resulting mbuf chain, after m__pullup is called. m_pullup allocates a new mbuf, prepends it to the chain, and moves the first 40 bytes of data from the cluster into the new mbuf. The reason it moves 40 bytes, and not just the requested 20, is to try to save an additional call at a later time when IP passes the datagram to a higher-layer protocol (e.g., ICMP, IGMP, UDP, or TCP). The magic number 40 (max_protohdr in Figure 7.17) is because the largest protocol header normally encountered is the combination of a 20-byte IP header and a 20-byte TCP header. (This assumes that other protocol suites, such as the OSI protocols, are not compiled into the kernel.)

Figure 2.18. An IP fragment of length 296, after calling m_pullup.

In Figure 2.18 the IP fragmentation algorithm can save a pointer to the IP header contained in the mbuf on the left, and this pointer can be converted into a pointer to the mbuf itself using dtom at a later time.

Figure 2.19 lists the macros and Figure 2.20 lists the functions that we’ll encounter in the code that operates on mbufs. The macros in Figure 2.19 are shown as function prototypes, not as #define statements, to show the data types of the arguments. We will not go through the source code implementation of these routines since they are concerned primarily with manipulating the mbuf data structures and involve no networking issues. Also, there are additional mbuf macros and functions used elsewhere in the Net/3 sources that we don’t show in these two figures since we won’t encounter them in the text.

MCLGET

void MCLGET(struct mbuf *m, int nowait;

MFREE

void MFREE(struct mbuf *m, struct mbuf *n;

MGETHDR

void MGETHDR(struct mbuf *m, int nowait, int type);

MH_ALIGN

void MH_ALIGN(struct mbuf *m, int len);

M_PREPEND

void M_PREPEND(struct mbuf *m, int len, int nowait);

dtom

struct mbuf *dtom(void *x);

mtod

type mtod(struct mbuf *m, type);

m_adj

void m_adj(struct mbuf *m, int len);

m_cat

void m_cat(struct mbuf *m, struct mbuf *n);

m_copy

struct mbuf *m_copy(struct mbuf *m, int Ioffset, int len);

m_copydata

void m_copydata(struct mbuf *m, int offset, int len, caddr_t cp);

m_copyback

void m_copyback(struct mbuf *m, int offset, int len, caddr_t cp);

m_copym

struct mbuf *m_copym(struct mbuf *m, int offset, int len, int nowait);

m_devget

struct mbuf *m_devget(char *buf, int len, int off, struct ifnet *ifp,
                      void (*copy)(const void *, void *, u_int));

m_free

struct mbuf *m_free(struct mbuf *m);

m_freem

void m_freem(struct mbuf *m);

m_get

struct mbuf *m_get(int nowait, int type;

m_getclr

struct mbuf *m_getclr(int nowait, int type);

struct mbuf *m_gethdr(int nowait, int type);

m_pullup

struct mbuf *m_pullup(struct mbuf *m, int len);

In all the prototypes the argument nowait is either M_WAIT or M_DONTWAIT, and the argument type is one of the MT_xxx constants shown in Figure 2.10.

As an example of M_PREPEND, this macro was called when the IP and UDP headers were prepended to the user’s data in the transition from Figure 1.7 to Figure 1.8, causing another mbuf to be allocated. But when this macro was called again (in the transition from Figure 1.8 to Figure 2.2) to prepend the Ethernet header, room already existed in the mbuf for the headers.

This section summarizes the types of data structures we’ll encounter in the Net/3 networking code. Other data structures are used in the Net/3 kernel (interested readers should examine the <sys/queue.h> header), but the following are the ones we’ll encounter in this text.

One obvious advantage with clusters is being able to reduce the number of mbufs required to contain large amounts of data. For example, if clusters were not used, it would require 10 mbufs to contain 1024 bytes of data: the first one with 100 bytes of data, the next eight with 108 bytes of data each, and the final one with 60 bytes of data. There is more overhead involved in allocating and linking 10 mbufs, than there is in allocating a single mbuf containing the 1024 bytes in a cluster. A disadvantage with clusters is the potential for wasted space. In our example it takes 2176 bytes using a cluster (2048+128), versus 1280 bytes without a cluster (10 x 128).

An additional advantage with clusters is being able to share a cluster between multiple mbufs. We encounter this with TCP output and the m_copy function, but describe it in more detail now.

As an example, assume the application performs a write of 4096 bytes to a TCP socket. Assuming the socket’s send buffer was previously empty, and that the receiver’s window is at least 4096, the following operations take place. One cluster is filled with the first 2048 bytes by the socket layer and the protocol’s send routine is called. The TCP send routine appends the mbuf to its send buffer, as shown in Figure 2.24, and calls tcp_output.

Figure 2.24. TCP socket send buffer containing 2048 bytes of data.

The socket structure contains the sockbuf structure, which holds the head of the list of mbufs on the send buffer: so_snd.sb_mb.

Assuming a TCP maximum segment size (MSS) of 1460 for this connection (typical for an Ethernet), tcp_output builds a segment to send containing the first 1460 bytes of data. It also builds an mbuf containing the IP and TCP headers, leaves room for a link-layer header (16 bytes), and passes this mbuf chain to IP output. The mbuf chain ends up on the interface’s output queue, which we show in Figure 2.25.

Figure 2.25. TCP socket send buffer and resulting segment on interface’s output queue.

In our UDP example in Section 1.9, UDP took the mbuf chain containing the datagram, prepended an mbuf for the protocol headers, and passed the chain to IP output. UDP did not keep the mbuf in its send buffer. TCP cannot do this since TCP is a reliable protocol and it must maintain a copy of the data that it sends, until the data is acknowledged by the other end.

In this example tcp_output calls the function m_copy, requesting a copy be made of 1460 bytes, starting at offset 0 from the start of its send buffer. But since the data is in a cluster, m_copy creates an mbuf (the one on the lower right of Figure 2.25) and initializes it to point to the correct place in the existing cluster (the beginning of the cluster in this example). The length of this mbuf is 1460, even though an additional 588 bytes of data are in the cluster. We show the length of the mbuf chain as 1514, accounting for the Ethernet, IP, and TCP headers.

This sharing of clusters prevents the kernel from copying the data from one mbuf into another—a big savings. It is implemented by providing a reference count for each cluster that is incremented each time another mbuf points to the cluster, and decremented each time a cluster is released. Only when the reference count reaches 0 is the memory used by the cluster available for some other use. (See Exercise 2.4.)

For example, when the bottom mbuf chain in Figure 2.25 reaches the Ethernet device driver and its contents have been copied to the device, the driver calls m_freem. This function releases the first mbuf with the protocol headers and then notices that the second mbuf in the chain points to a cluster. The cluster reference count is decremented, but since its value becomes 1, it is left alone. It cannot be released since it is still in the TCP send buffer.

Continuing our example, tcp_output returns after passing the 1460-byte segment to IP, since the remaining 588 bytes in the send buffer don’t comprise a full-sized segment. (In Chapter 26 we describe in detail the conditions under which tcp_output sends data.) The socket layer continues processing the data from the application: the remaining 2048 bytes are placed into an mbuf with a cluster, TCP’s send routine is called again, and this new mbuf is appended to the socket’s send buffer. Since a full-sized segment can be sent, tcp_output builds another mbuf chain with the protocol headers and the next 1460 bytes of data. The arguments to m_copy specify a starting offset of 1460 bytes from the start of the send buffer and a length of 1460 bytes. This is shown in Figure 2.26, assuming the mbuf chain is again on the interface output queue (so the length of the first mbuf in the chain reflects the Ethernet, IP, and TCP headers).

Figure 2.26. Mbuf chain to send next 1460-byte TCP segment.

This time the 1460 bytes of data come from two clusters: the first 588 bytes are from the first cluster in the send buffer and the next 872 bytes are from the second cluster in the send buffer. It takes two mbufs to describe these 1460 bytes, but again m_copy does not copy the 1460 bytes of data—it references the existing clusters.

We encounter the m_copy function about a dozen times throughout the text. Although the name implies that a physical copy is made of the data, if the data is contained in a cluster, an additional reference is made to the cluster instead.

Mbufs are far from perfect and they are berated regularly. Nevertheless, they form the basis for all the Berkeley-derived networking code in use today.

A research implementation of the Internet protocols by Van Jacobson [Partridge 1993] has done away with the complex mbuf data structures in favor of large contiguous buffers. [Jacobson 1993] claims a speed improvement of one to two orders of magnitude, although many other changes were made besides getting rid of mbufs.

The complexity of mbufs is a tradeoff that avoids allocating large fixed buffers that are rarely filled to capacity. At the time mbufs were being designed, a VAX-11/780 with 4 megabytes of memory was a big system, and memory was an expensive resource that needed to be carefully allocated. Today memory is inexpensive, and the focus has shifted toward higher performance and simplicity of code.

The performance of mbufs is also dependent on the amount of data stored in the mbuf. [Hutchinson and Peterson 1991] show that the amount of time required for mbuf processing is nonlinear with respect to the amount of data.

We’ll encounter mbufs in almost every function in the text. Their main purpose is to hold the user data that travels from the process to the network interface, and vice versa, but mbufs are also used to contain a variety of other miscellaneous data: source and destination addresses, socket options, and so on.

There are four types of mbufs, depending whether the M_PKTHDR and M_EXT flags are on or off:

We looked at the source code for a few of the mbuf macros and functions, but did not present the source code for all the mbuf routines. Figures 2.19 and 2.20 provide the function prototypes and descriptions of all the mbuf routines that we encounter in the text.

We looked at the operation of two functions that we’ll encounter: m_devget, which is called by many network device drivers to store a received frame; and m_pullup, which is called by all the input routines to place the required protocol headers into contiguous storage in an mbuf.

The clusters (external buffers) pointed to by an mbuf can be shared by m_copy. This is used, for example, by TCP output, because a copy of the data being transmitted must be maintained by the sender until that data is acknowledged by the other end. Sharing clusters through reference counts is a performance improvement over making a physical copy of the data.