D is a block-structured language similar in layout to awk. A program consists of one or more clauses that take the following form:

probe
/ optional predicates /
{
        optional action statements;
}

Each clause describes one or more probes to enable, an optional predicate, and any actions to associate with the probe specification. When a D program contains several clauses that enable the same probe, the clauses are executed in the order in which they appear in the program. For example:

syscall::read:entry
{
        printf("A");
}

syscall::read:entry
{
        printf(" B");
        exit(1);
}

The above script contains two clauses; each clause enables the read(2) system call entry probe. When this script is executed, the system is modified dynamically to insert our tracing actions into the read() system call. When any application next makes a read() call, the first clause is executed, causing the character “A” to be displayed. The next clause is executed immediately after the first, and the sequence “B” is also displayed. The exit(1) call terminates the tracing session, an action that in turn causes the enabled probes and their actions to be removed. The system then returns to its default state. Executing the script we see this:

sol10# ./read.d
A B

The preceding explanation is a huge simplification of what actually happens when we execute a D script. The important thing to note here is the dynamic nature of the modifications that are made when a D script is executed. The modifications made to the system (the “instrumentation”) exist just for the lifetime of the script. When no DTrace scripts are running, the system acts just as if DTrace were not installed.

By default, DTrace provides tens of thousands of probes that you can enable to gain unparalleled insight into the behavior of a system (use dtrace -l to list them all). Each probe can be referred to by a unique numerical ID or by a more commonly used human-readable one that consists of four colon-separated fields. These are defined as follows:

provider:module:function:name

Note two key facts about probe specifications:

Table 10.1 lists examples of valid probe descriptions.

Although it isn’t necessary to specify all the fields in a probe, the examples in this book do so in order to remove any ambiguity about which probes are being enabled. Also note that a comma-separated list of probes can be used to associate multiple probes with the same predicate and actions.

In previous examples we saw the syscall provider being used to ask questions concerning system call usage. Exactly what is a provider and what is its relationship to a probe? A provider creates the probes that are essentially the individual system points at which we ask questions. There are a number of providers, each able to instrument a different part of the system.

The following providers are of special interest to us:

The following providers leverage the sdt provider to grant powerful observability into key Solaris functional areas:

The syscall example used earlier is simple and powerful. However, the output quickly becomes voluminous and overwhelming with thousands of lines generated in seconds. It rapidly becomes difficult to discern patterns of activity in the data, such as might be perceived in a view of all system calls sorted by count. Historically, we would have generated our data and post-processed this by using tools such as awk(1) or perl(1), but that approach is laborious and time wasting. DTrace enables us to succinctly specify how to group vast amounts of data so that we can easily observe such patterns. The mechanism that does this is termed an aggregation. We use aggregations to refine our initial script.

syscall:::entry
{
        @sys[probefunc] = count();
}

And here is the output now.

sol10# ./truss.d
dtrace: script './truss.d' matched 225 probes
^C

  <output elided>
  fcntl
  xstat                                                           1113
  lwp_park                                                        2767
  setcontext                                                      4593
  lwp_sigmask                                                     4599
  write                                                           7429
  setitimer                                                       8234
  writev                                                          8444
  ioctl                                                          17718
  pollsys                                                       135603
  read                                                          141379

Instead of seeing every system call as it is made, we are now presented with a table of system calls sorted by count: over 330, 000 system calls presented in several lines!

The concept of an aggregation is simple. We want to associate the value of a function with an arbitrary element in an array. In our example, every time a system call probe is fired, the name of the system call is used (using the probefunc built-in variable) to index an associative array. The result of the count() function is then stored in this element of the array (this simply adds 1 to an internal variable for the index in the array and so effectively keeps a running total of the number of times this system call has been entered). In that way, we do not focus on data at individual probe sites but succinctly collate large volumes of data.

An aggregation can be split into two basic components: on the left side, a named associative array that is preceded by the @ symbol; on the right side, an aggregating function.

@name [ keys ] = function();

An aggregating function has the special property that it produces the same result when applied to a set of data as when applied to subsets of that data and then again to that set of results. A simple example of this is finding the minimum value of the set [5, 12, 4, 7, 18]. Applying the min() function to the whole set gives the result of 4. Equally, computing the minimum value of two subsets [5, 12] and [4, 7, 18] produces 5 and 4. Applying min() again to [5, 4] yields 4.

Several aggregating functions in DTrace and their results are listed below.

The following example stores in the appropriate bucket the size of the memory requested in the call to malloc().

pid$1:libc:malloc:entry
{
        @["malloc sizes"] = quantize(arg0);
}

sol10# ./malloc.d 658
dtrace: script './malloc.d' matched 1 probe
^C

  malloc sizes
           value  ------------- Distribution ------------- count
               2 |                                         0
               4 |@@                                       405
               8 |@@@@                                     886
              16 |@@@@@@@                                  1673
              32 |@                                        205
              64 |@@@@@@                                   1262
             128 |@@@@@@@                                  1600
             256 |@@@@@@@                                  1632
             512 |                                         3
            1024 |                                         5
            2048 |@@@@                                     866
            4096 |                                         10
            8192 |@@@                                      586
           16384 |                                         0

The example shows that 1673 memory allocations between the size of 16 and 31 bytes were requested. The @ character indicates the relative size of each bucket.

pid$1:libc:malloc:entry
{
        @["malloc sizes"] = lquantize(arg0,4,8,1);
}

sol10# ./lmalloc.d 658
dtrace: script './lmalloc.d' matched 1 probe
^C

  malloc sizes
           value  ------------- Distribution -------------  count
             < 4 |                                          6
               4 |@                                         423
               5 |                                          0
               6 |@                                         400
               7 |                                          0
            >= 8 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    18452

Having looked at aggregations, we now come to the two basic data types provided by D: associative arrays and scalar variables. An associative array stores data elements that can be accessed with an arbitrary name, known as a key or an index. This differs from normal, fixed-size arrays in a number of different ways:

Associative arrays in D commonly keep a history of events that have occurred in the past to use in controlling flow in scripts. The following example uses an associative array, arr, to keep track of the largest writes made by applications.

syscall::write:entry
/arr[execname] < arg2/
{
        printf("%d byte write: %s
", arg2, execname);
        arr[execname] = arg2;
}

The actions of the clause are executed if the write size, stored in arg2, is larger than that stored in the associative array arr for a given application. If the predicate evaluates to true, then this is the largest write seen for this application. The actions record this by first printing the size of the write and then by updating the element in the array with the new maximum write size.

D is similar to languages such as C in its implementation of scalar variables, but a few differences need to be highlighted. The first thing to note is that in the D language, variables do not have to be declared in advance of their use, much the same as in awk(1) or perl(1). A variable comes into existence when it first has a value assigned to it; its type is inferred from the assigned value (you are allowed to declare variables in advance but doing so isn't necessary). There is no explicit memory management in D, much as in the Java programming language. The storage for a variable is allocated when the variable is declared, and deallocated when the value of 0 is assigned to the variable.

The D language provides three types of variable scope: global, thread-local, and clause-local. Thread-local variables provide separate storage for each thread for a given variable and are referenced with the self-> prefix.

fbt:ufs:ufs_write:entry
{
        self->in = timestamp;
}

In the clause above, every different thread that executes the ufs_write() function has its own copy of a variable named in. Its type is the same as the timestamp built-in variable, and it holds the value that the timestamp built-in variable had when the thread started executing the actions in the clause. This is a nanosecond value since an arbitrary time in the past.

A common use of thread-local variables is to highlight a sequence of interest for a given thread and also to associate data with a thread during that sequence. The following example uses the sched provider to record, by application, all the time that a specified user (UID 1003) spent executing.

sched:::on-cpu
/uid == 1003/
{
        self->ts = timestamp;
}

sched:::off-cpu
/self->ts/
{
        @time[execname] = sum(timestamp - self->ts);
        self->ts = 0;
}

The above D script contains two clauses. The first one uses the sched:::oncpu probe to enable a probe at every point in the kernel where a thread can be placed onto a processor and run. The predicate attached to this probe specifies that the actions are only to be executed if the uid of the thread is 1003. The action merely stores the current timestamp in nanoseconds by assigning the timestamp built-in variable to a thread-local variable, self->ts.

The second clause uses the sched:::off-cpu probe to enable a probe at every location in the kernel where a thread can be taken off the CPU. The self->ts variable in the predicate ensures that only threads owned by uid 1003 that have already been through the sched:::on-cpu probe shall execute the following actions. Why couldn’t we just predicate on uid == 1003 as in the first clause? Well, we want to ensure that any thread executing the following actions has already been through the first clause so that its self->ts variable is set. If it hasn't been set, we will end up storing a huge value in the @time aggregation because self->ts will be 0! Using a thread-local variable in predicates like this to control flow in a D script is a common technique that we frequently use in this book.

sol10# ./sched.d
dtrace: script './sched.d' matched 6 probes
^C

  cat                                                          1247134
  xterm                                                        1830446
  ksh                                                          3909045
  stdlibfilt                                                   5499630
  make                                                        60092218
  sh                                                         158049977
  sed                                                        162507340
  CC                                                         304925644
  ir2hf                                                      678855289
  sched                                                     2600916929
  ube_ipa                                                   2851647754
  ccfe                                                      5879225939
  ube                                                       7942433397

The preceding example can be enhanced with the profile provider to produce output at a given periodic rate. To produce output every 5 seconds, we can just add the following clause:

profile:::tick-5s
{
              printa(@time);
              trunc(@time);
}

The profile provider sets up a probe that fires every 5 seconds on a single CPU. The two actions used here are commonly used when periodically displaying aggregation data:

In DTrace, probe arguments are made available through one or two mechanisms, depending on which provider is responsible for the probe:

For an example of argument usage, let’s look at a script based on the fbt provider. The Solaris kernel, like any other program, is made up of many functions that offer well-defined interfaces to perform specific operations. We often want to ask pertinent questions upon entry to a function, such as, What was the value of its third argument? or upon exit from a function, What was the return value? For example:

fbt:ufs:ufs_read:entry
/uid == 1003/
{
        self->path = stringof(args[0]->v_path);
        self->ts = timestamp;
}

fbt:ufs:ufs_read:return
/self->path != NULL/
{
        @[self->path] = max(timestamp - self->ts);
        self->path = 0;
        self->ts = 0;
}

This example looks at all the reads performed through ufs file systems by a particular user (UID 1003) and, for each file, records the maximum time taken to carry out the read call. A few new things require further explanation.

The name of the file being read from is stored in the thread-local variable, self->path, with the following statement:

self->path = stringof(args[0]->v_path);

The main point to note here is the use of the args[] array to reference the first argument (args[0]) of the ufs_read function. Using MDB, we can inspect the arguments of ufs_read:

sol10# mdb -k
Loading modules: [ unix krtld genunix dtrace specfs ufs ip sctp usba uhci fctl s1394 nca
lofs audiosup nfs random sppp crypto ptm ipc ]
> ufs_read::nm -f ctype
C Type
int (*)(struct vnode *, struct uio *, int, struct cred *, struct caller_context *)

The first argument to ufs_read() is a pointer to a vnode structure (struct vnode *). The path name of the file that is represented by that vnode is stored in the v_path member of the vnode structure and can be accessed through args[0]->v_path. Using MDB again, we inspect the type of the v_path member variable.

> ::print -t struct vnode v_path
char *v_path

The v_path member is a character pointer and needs to be converted to DTrace’s native string type. In DTrace a string is a built-in data type. The stringof() action is one of many features that allow easy manipulation of strings. It converts the char * representation of v_path into the DTrace string type.

If the arg0 built-in variable had been used, a cast would be required and would be written as this:

self->path = stringof(((struct vnode *)arg0)->v_path);

The predicate associated with the ufs_read:return probe ensures that its actions are only executed for files with a non-NULL path name. The action then uses the path name stored in the self->path variable to index an aggregation, and the max() aggregating function tracks the maximum time taken for reads against this particular file. For example:

sol10# ./ufs.d
dtrace: script './ufs.d' matched 2 probes
^C

  /lib/ld.so.1                                                    3840
  /usr/share/lib/zoneinfo/GB                                      5523
  <output elided>
  /usr/share/man/man1/ls.1                                     2599836
  /./usr/bin/more                                              3941344
  /./usr/bin/tbl                                               3988087
  /usr/share/lib/pub/eqnchar                                   4397573
  /usr/share/lib/tmac/an                                       5054675
  /./usr/bin/nroff                                             7004599
  /./usr/bin/neqn                                              7021088
  /./usr/bin/col                                               9989462
  /usr/share/man/windex                                       13742938
  /./usr/bin/man                                              17179129

Now let’s look at a syscall-based example of return probe use. The following script exploits the fact that syscall probes have their return value stored in arg0 and the error code for the call stored in the errno built-in variable.

syscall::open*:entry
{
        self->path = arg0;
}
syscall::open*:return
/self->path != NULL && (int)arg0 == -1 && errno == EACCES/
{
        printf("UID %d permission denied to open %s
",
            uid, copyinstr(self->path));
        self->path = 0;
}

The first clause enables probes for the open(2) and open64(2) system calls. It then stores the address of the buffer, which contains the file name to open, in the thread-local variable self->path.

The second clause enables the corresponding syscall return probes. The conditions of interest are laid out in the predicate:

If the above conditions are all true, then a message is printed specifying the UID that induced the condition and the file for which permissions were lacking .

sol10# ./open.d
UID 39079 permission denied to open /etc/shadow

Finally, a note regarding the copyinstr() action used in the second clause above: All probes, predicates, and associated actions are executed in the kernel, and therefore any data that originates in userland must be copied into the kernel to be used. The buffer that contains the file name to be opened in our example is a buffer that resides in a userland application. For the contents to be printed, the buffer must be copied to the kernel address space and converted into a DTrace string type; this is what copyinstr() does.

DTrace gives us the freedom to observe interactions across many different subsystems. The following slightly larger script demonstrates how we can follow all the work done in userland and the kernel by a given application function. We can use dtrace -p to attach to and instrument a running process. For example, we can use a script that looks at the function getgr_lookup() in the name services cache daemon. The getgr_lookup() function is called to translate group IDs and group names. Note that here we are interested in the principle of examining a particular function; the actual program and function chosen here are irrelevant.

#pragma D option flowindent

pid$target:a.out:getgr_lookup:entry
{
        self->in = 1;
}

pid$target:::entry,
pid$target:::return
/self->in/
{
        printf("(pid)
");
}

fbt:::entry,
fbt:::return
/self->in/
{
        printf("(fbt)
");
}

pid$target:a.out:getgr_lookup:return
/self->in/
{
        self->in = 0;
        exit(0);
}

The #pragma flowindent directive at the start of the script means that indentation will be increased on entry to a function and reduced on the same function’s return. Showing function calls in a nested manner like this makes the output much more readable.

The pid provider instruments userland applications. The process to be instrumented is specified with the $target macro argument, which always expands to the PID of the process being traced when we attach to the process by using the -p option to dtrace(1M).

The second clause enables all the entry and return probes in the nscd process, and the third clause enables every entry and return probe in the kernel. The predicate in both of these clauses specifies that we are only interested in executing the actions if the thread-local self->in variable is set. This variable is set to 1 when nscd’s getgr_lookup() function is entered and set to 0 on exit from this function (that is, when the return probe is fired). For example:

sol10# dtrace -s ./nscd.d -p 'pgrep nscd`
dtrace: script './nscd.d' matched 43924 probes
CPU FUNCTION
  0  -> getgr_lookup                           (pid)
  0    -> mutex_lock                           (pid)
  0      -> mutex_lock_impl                    (pid)
  0      <- mutex_lock_impl                    (pid)
  0    <- mutex_lock                           (pid)
  0    -> _xstat                               (pid)
  0      -> copyout                            (fbt)
  0      <- kcopy                              (fbt)
  0      -> syscall_mstate                     (fbt)
  0        -> gethrtime_unscaled               (fbt)
  0        <- gethrtime_unscaled               (fbt)
  0      <- syscall_mstate                     (fbt)
  0      -> syscall_entry                      (fbt)
 <output elided>
  0      <- syscall_exit                       (fbt)
  0      -> syscall_mstate                     (fbt)
  0        -> gethrtime_unscaled               (fbt)
  0        <- gethrtime_unscaled               (fbt)
  0      <- syscall_mstate                     (fbt)
  0    <- _xstat                               (pid)
  0    -> get_hash                             (pid)
  0      -> abs                                (pid)
  0        -> copyout                          (fbt)
  0        <- kcopy                            (fbt)
  0      <- abs                                (pid)
  0    <- get_hash                             (pid)
  0    -> memcpy                               (pid)
  0      -> copyout                            (fbt)
  0      <- kcopy                              (fbt)
  0    <- memcpy                               (pid)
  0    -> mutex_unlock                         (pid)
  0    <- mutex_unlock                         (pid)
  0   | getgr_lookup:return                    (pid)
  0  <- getgr_lookup

DTrace provides a very useful feature by which we can access symbols defined in the Solaris kernel from within a D script. We can use the backquote character (`) to refer to kernel symbols, and this information can be used to great advantage when we are exploring the behavior of a Solaris kernel. For example, a variable named mpid is declared in the Solaris kernel source to keep track of the last PID that was allocated. It is declared in uts/common/os/pid.c as follows:

static pid_t mpid;

The following script uses this variable to calculate the rate of process creation on the system and to output a message if it exceeds a given amount (10 processes per second in this case):

dtrace:::BEGIN
{
        cnt = 'mpid;
}
profile:::tick-1s
/'mpid < cnt+10/
{
        cnt = 'mpid;
}

profile:::tick-1s
/'mpid >= cnt+10/
{
        printf("High process creation rate: %d/sec
", 'mpid - cnt);
        cnt = 'mpid;
}

The first clause uses the BEGIN probe from the dtrace provider to initialize a global variable (cnt) to the current value of the mpid kernel variable.

The BEGIN, END, and ERROR probes are special probes that belong to the dtrace provider. These probes are essentially virtual probes in that they aren’t associated with any code location or timer source. The BEGIN probe fires before any other probes when we start the tracing session and allows us to perform tasks such as data initialization. The END probe is called when the tracing session is terminated either with a Control-C or an explicit call to the exit() action. Its main function is to print data collected during the execution of the script. The ERROR probe is less commonly used; it is called upon abnormal termination of the script.

Both of the next two clauses in the previous example enable the profile:::tick-1s probe. The probe fires every second, and the two clauses are executed in the order specified in the script. The important thing to note is that the predicates in the two clauses contain mutually exclusive logic, which ensures that only one of them will be true at any one time—either ten processes have been created in the last second or they haven’t!

The predicate in the first profile:::tick-1s clause specifies that its actions should only be executed if fewer than ten processes have been created (the 'mpid variable is within ten of its value one second ago as stored in the cnt variable). If fewer than ten processes have been created in the last second, the cnt variable is updated with the current value of mpid.

The actions in the second clause are executed when more than ten processes have been created. If cnt has already been updated in the first clause, then the predicate will be false and the actions are not executed (a message is then printed with the growth rate, and the cnt variable is updated). For example:

sol10# ./scope.d
High process creation rate: 30/sec
High process creation rate: 31/sec
High process creation rate: 35/sec
High process creation rate: 35/sec
High process creation rate: 44/sec
High process creation rate: 44/sec
High process creation rate: 20/sec

DTrace defines numerous actions, only a small percentage of which are used in this book. Actions that you may see used include normalize(), stack(), and ustack().

The find(1) application is at the top of the list here. The walk() routine is listed multiple times because it is recursively called to walk a file tree.

Mustang is the first release to contain built-in DTrace probes, but support for the DTrace jstack() action was actually first introduced in the JavaTM 2 Platform Standard Edition 5.0 Update Release 1. The DTrace jstack() action prints mixed-mode stack traces including both Java method and native function names. As an example of its use, consider the following application, which periodically sleeps to mimic hanging behavior:

public class dtest{
    int method3(int stop){
        try{Thread.sleep(500);}
        catch(Exception ex){}
        return stop++;
    }
    int method2(int stop){
        int result = method3(stop);
        return result + 1;
    }
    int method1(int arg){
        int stop=0;
        for(int i=1; i>0; i++){
            if(i>arg){stop=i=1;}
            stop=method2(stop);
        }
        return stop;
    }
    public static void main(String[] args) {
        new dtest().method1(10000);
    }
}

To find the cause of the hang, the user would want to know the chain of native and Java method calls in the currently executing thread. The expected chain would be something like:

<chain of initial VM functions> ->
dtest.main -> dtest.method1 -> dtest.method2 -> dtest.method3 ->
java/lang/Thread.sleep -> <chain of VM sleep functions> ->
<Kernel pool functions>

The following D script (usestack.d) uses the DTrace jstack() action to print the stack trace:

#!/usr/sbin/dtrace -s

BEGIN { this->cnt = 0; }
syscall::pollsys:entry
/pid == $1 && tid == 1/
{
   this->cnt++;
   printf("
	TID: %d", tid);
   jstack(50);
}

syscall:::entry
/this->cnt == 1/
{
   exit(0);
}

And the stack trace itself appears as follows:

$ usejstack.d 1344 | c++filt
CPU     ID                    FUNCTION:NAME
 0    316                    pollsys:entry
   TID: 1
   libc.so.1'__pollsys+0xa
   libc.so.1'poll+0x52
   libjvm.so'int os_sleep(long long,int)+0xb4
   libjvm.so'int os::sleep(Thread*,long long,int)+0x1ce
   libjvm.so'JVM_Sleep+0x1bc
   java/lang/Thread.sleep
   dtest.method3
   dtest.method2
   dtest.method1
   dtest.main
   StubRoutines (1)
   libjvm.so'void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArgu-
ments*,Thread*)+0x1b5
   libjvm.so'void os::os_exception_wrapper(void(*)(JavaValue*,methodHandle*,JavaCallAr-
guments*,Thread*),JavaValue*,methodHandle*,Ja
vaCallArguments*,Thread*)+0x18
   libjvm.so'void JavaCalls::call(JavaValue*,methodHandle,JavaCallArgu-
ments*,Thread*)+0x2d
   libjvm.so'void jni_invoke_static(JNIEnv_*,JavaValue*,_jobject*,JNICallType,_
jmethodID*,JNI_ArgumentPush er*,Thread*)+0x214
   libjvm.so'jni_CallStaticVoidMethod+0x244
   java'main+0x642
   StubRoutines (1)

The command line shows that the output from this script was piped to the c++filt utility, which demangles C++ mangled names making the output easier to read. The DTrace header output shows that the CPU number is 0, the probe number is 316, the thread ID (TID) is 1, and the probe name is pollsys:entry, where pollsys is the name of the system call. The stack trace frames appear from top to bottom in the following order: two system call frames, three VM frames, five Java method frames, and the remaining frames are VMframes.

It is also worth noting that the DTrace jstack action will run on older releases, such as the Java 2 Platform, Standard Edition version 1.4.2, but hexadecimal addresses will appear instead of Java method names. Such addresses are of little use to application developers.

In addition to the jstack() action, it is also possible for pre-Mustang users to add DTrace probes to their release with the help of VM Agents. A VM agent is a shared library that is dynamically loaded into the VM at startup.

VM agents are available for the following releases:

To obtain the agents, visit the DVM java.net project website at

https://solaris10-dtrace-vm-agents.dev.java.net/

and follow the “Documents and Files” link. The file dvm.zip contains both binary and source code versions of the agent libraries.

The following diagram shows an abbreviated view of the resulting directory structure once dvm.zip has been extracted:

          build   make   src    test
            |
       ---------------------
         |      |     |      |
        amd64  i386  sparc  sparcv9
         |      |     |      |
        lib    lib   lib    lib

Each lib directory contains the pre-built binaries dvmti.jar, libdvmpi.so, and libdvmti.so. If you prefer to compile the libraries yourself, the included README file contains all necessary instructions.

Once unzipped, the VM must be able to find the native libraries on the filesystem. This can be accomplished either by copying the libraries into the release with the other shared libraries, or by using a platform-specific mechanism to help a process find it, such as LD_LIBRARY_PATH. In addition, the agent library itself must be able to find all the external symbols that it needs. The ldd utility can be used to verify that a native library knows how to find all required externals.

Both agents accept options to limit the probes that are available, and default to the least possible performance impact. To enable the agents for use in your own applications, run the java command with one of the following additional options:

For additional options, consult the DVM agent README. Both agents have their limitations, but dvmpi has more, and we recommend using the Java Standard Edition 5.0 Development Kit (JDK 5.0) and the dvmti agent if possible.

When using the agent-based approach, keep in mind that:

Section C.1 provides a D script for testing DVM probes. The DVM agent provider interface, shown in Section C.2, lists all probes provided by dvmpi and dvmti.

The io probes are listed in Table 10.2, and the arguments are described in Sections 10.6.1.1 through 10.6.1.3.

typedef struct bufinfo {
              int b_flags;                    /* flags */
              size_t b_bcount;                /* number of bytes */
              caddr_t b_addr;                 /* buffer address */
              uint64_t b_blkno;               /* expanded block # on device */
              uint64_t b_lblkno;              /* block # on device */
              size_t b_resid;                 /* # of bytes not transferred */
              size_t b_bufsize;               /* size of allocated buffer */
              caddr_t b_iodone;               /* I/O completion routine */
              dev_t b_edev;                   /* extended device */
} bufinfo_t;
                                                               See /usr/lib/dtrace/io.d

typedef struct devinfo {
              int dev_major;                  /* major number */
              int dev_minor;                  /* minor number */
              int dev_instance;               /* instance number */
              string dev_name;                /* name of device */
              string dev_statname;            /* name of device + instance/minor */
              string dev_pathname;            /* pathname of device */
} devinfo_t;
                                                               See /usr/lib/dtrace/io.d

typedef struct fileinfo {
              string fi_name;                 /* name (basename of fi_pathname) */
              string fi_dirname;              /* directory (dirname of fi_pathname) */
              string fi_pathname;             /* full pathname */
              offset_t fi_offset;             /* offset within file */
              string fi_fs;                   /* filesystem */
              string fi_mount;                /* mount point of file system */
} fileinfo_t;
                                                               See /usr/lib/dtrace/io.d

The vminfo provider probes correspond to the fields in the “vm” named kstat: A probe provided by vminfo fires immediately before the corresponding vm value is incremented. Table 10.4 lists the probes available from the VM provider. A probe takes the following arguments:

For example, if you should see the following paging activity with vmstat, indicating page-in from the swap device, you could drill down to investigate.

# vmstat -p 3
     memory           page          executable      anonymous      filesystem
   swap  free  re  mf  fr  de  sr  epi  epo  epf  api  apo  apf  fpi  fpo  fpf
 1512488 837792 160 20 12   0   0    0    0    0 8102    0    0   12   12   12
 1715812 985116 7  82   0   0   0    0    0    0 7501    0    0   45    0    0
 1715784 983984 0   2   0   0   0    0    0    0 1231    0    0   53    0    0
 1715780 987644 0   0   0   0   0    0    0    0 2451    0    0   33    0    0

$ dtrace -n anonpgin'{@[execname] = count()}'
dtrace: description 'anonpgin' matched 1 probe
  svc.startd                                                       1
  sshd                                                             2
  ssh                                                              3
  dtrace                                                           6
  vmstat                                                          28
  filebench                                                      913

The VM probes are described in Table 10.4.

The sched probes are described in Table 10.5.

typedef struct cpuinfo {
              processorid_t cpu_id;           /* CPU identifier */
              psetid_t cpu_pset;              /* processor set identifier */
              chipid_t cpu_chip;              /* chip identifier */
              lgrp_id_t cpu_lgrp;             /* locality group identifer */
              processor_info_t cpu_info;      /* CPU information */
} cpuinfo_t;

The lockstat provider makes available probes that can be used to discern lock contention statistics or to understand virtually any aspect of locking behavior. The lockstat(1M) command is actually a DTrace consumer that uses the lockstat provider to gather its raw data.

The lockstat provider makes available two kinds of probes: content-event probes and hold-event probes.

The lockstat provider makes available probes that correspond to the different synchronization primitives in Solaris; these primitives and the probes that correspond to them are discussed in the remainder of this chapter.

This following section lists all probes published by the hotspot provider.