Contributions from Mike Shapiro, Matthew Simmons, and Eric Schrock
In this chapter, we take a tour of MDB basics, from startup through elements (command syntax, expressions, symbols, and other core concepts), via simple procedures illustrated by examples.
MDB is available on Solaris systems as two commands that share common features:mdb and kmdb. You can use the mdb command interactively or in scripts to debug live user processes, user process core files, kernel crash dumps, the live operating system, object files, and other files. You can use the kmdb command to debug the live operating system kernel and device drivers when you also need to control and halt the execution of the kernel. To start mdb, execute the mdb(1) command.
The following example shows how mdb can be started to examine a live kernel.
sol8# mdb -k
Loading modules: [ unix krtld genunix specfs dtrace ufs ip sctp usba uhci s1394 fcp fctl
emlxs nca lofs zfs random nfs audiosup sppp crypto md fcip logindmux ptm ipc ]
>
To start mdb with a kernel crash image, specify the namelist and core image names on the command line.
sol8# cd /var/crash/myserver sol8# ls /var/crash/* bounds unix.1 unix.3 unix.5 unix.7 vmcore.1 vmcore.3 vmcore.5 vmcore.7 unix.0 unix.2 unix.4 unix.6 vmcore.0 vmcore.2 vmcore.4 vmcore.6 sol8# mdb -k unix.1 vmcore.1 Loading modules: [ unix krtld genunix specfs dtrace ufs ip sctp usba uhci s1394 fcp fctl emlxs nca lofs zfs random nfs audiosup sppp crypto md fcip logindmux ptm ipc ] >
To start mdb with a process target, enter either a command to execute or a process ID with the -p option.
# mdb /usr/bin/ls > # mdb -p 121 Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libuutil.so.1 ]
To start kmdb, boot the system or execute the mdb command with the -K option as described in Chapter 14.
The MDB debugger lets us interact with the target program and the memory image of the target. The syntax is an enhanced form of that used with debuggers like adb, in which basic form is expressed as value and a command.
[value] [,count ] command
The language syntax is designed around the concept of computing the value of an expression (typically a memory address in the target), and applying a command to that expression. A command in MDB can be of several forms. It can be a macro file, a metacharacter, or a dcmd pipeline. A simple command is a metacharacter or dcmd followed by a sequence of zero or more blank-separated words. The words are typically passed as arguments. Each command returns an exit status that indicates it succeeded, failed, or was invoked with invalid arguments.
For example, if we wanted to display the contents of the word at address fec4b8d0, we could use the / metacharacter with the word X as a format specifier, and optionally a count specifying the number of iterations.
> fec4b8d0 /X lotsfree: lotsfree: f5e > fec4b8d0,4 /X lotsfree: lotsfree: f5e 7af 3d7 28
MDB retains the notion of dot (.) as the current address or value, retained from the last successful command. A command with no supplied expression uses the value of dot for its argument.
> /X lotsfree: lotsfree: f5e > . /X lotsfree: lotsfree: f5e
A pipeline is a sequence of one or more simple commands separated by |. Unlike the shell, dcmds in MDB pipelines are not executed as separate processes. After the pipeline has been parsed, each dcmd is invoked in order from left to right. The full definition of a command involving pipelines is as follows.
[expr] [,count ] pipeline [words...]
Each dcmd’s output is processed and stored as described in “dcmd Pipelines” in Section 13.2.8. After the left-hand dcmd is complete, its processed output is used as input for the next dcmd in the pipeline. If any dcmd does not return a successful exit status, the pipeline is aborted.
For reference, Table 13.1 lists the full set of expression and pipeline combinations that form commands.
Table 13.1. General MDB Command Syntax
Description | |
|---|---|
| basic |
| set dot, run once |
| set dot, repeat |
| repeat |
| set dot, last pipeline, run once |
| last pipeline, repeat |
| set dot, last pipeline, repeat |
| shell escape |
Arithmetic expansion is performed when an MDB command is preceded by an optional expression representing a numerical argument for a dcmd. A list of common expressions is summarized in Tables 13.2, 13.3, and 13.4.
Table 13.2. Arithmetic Expressions
Operator | Expression |
|---|---|
|
|
| IEEE floating point |
| little-endian character const |
| variable lookup |
| symbol lookup |
| the value of expr |
| the value of dot |
| last dot used by dcmd |
| dot+increment |
| dot-increment (increment is effected by the last formatting dcmd) |
Table 13.3. Unary Operators
Expression | |
|---|---|
| logical NOT |
| bitwise NOT |
| integer negation |
| object-file pointer dereference |
| object-file typed dereference |
| object-file sized dereference |
| virtual-address pointer dereference |
| virtual-address typed dereference |
| virtual-address sized dereference |
[csil] is char-, short-, int-, or long-sized |
Table 13.4. Binary Operators
Operator | Description |
|---|---|
| integer multiplication |
| integer division |
| left rounded up to next right multiple |
| integer addition |
| integer subtraction |
| bitwise left shift |
| bitwise right shift (logical) |
| logical equality |
| logical inequality |
| bitwise AND |
| bitwise XOR |
| bitwise OR |
An example of a simple expression is adding an integer to an address.
> d7c662e0+0t8/X 0xd7c662e8: d2998b80 > d7c662e0+0t8::print int 0xd7c662e8: d2998b80
MDB can reference memory or objects according to the value of a symbol of the target. A symbol is the name of either a function or a global variable in the target.
For example, you compute the address of the kernel’s global variable lotsfree by entering it as an expression, and display it by using the = metacharacter. You display the value of the lotsfree symbol by using the / metacharacter.
> lotsfree=X fec4b8d0 > lotsfree/D lotsfree: lotsfree: 3934
Symbol names can be resolved from kernel and userland process targets. In the kernel, the resolution of the symbol names can optionally be defined with a scope by specifying the module or object file name. In a process, symbols’ scope can be defined by library or object file names. They take the form shown in Table 13.5.
The target typically searches the primary executable’s symbol tables first, then one or more of the other symbol tables. Notice that ELF symbol tables contain only entries for external, global, and static symbols; automatic symbols do not appear in the symbol tables processed by MDB.
Additionally, MDB provides a private user-defined symbol table that is searched before any of the target symbol tables are searched. The private symbol table is initially empty and can be manipulated with the ::nmadd and ::nmdel dcmds.
The ::nm -P option displays the contents of the private symbol table. The private symbol table allows the user to create symbol definitions for program functions or data that were either missing from the original program or stripped out.
> ::nm
Value Size Type Bind Other Shndx Name
0x00000000|0x00000000|NOTY |LOCL |0x0 |UNDEF |
0xfec40038|0x00000000|OBJT |LOCL |0x0 |14 |_END_
0xfe800000|0x00000000|OBJT |LOCL |0x0 |1 |_START_
0xfec00000|0x00000000|NOTY |LOCL |0x0 |10 |__return_from_main
...
These definitions are then used whenever MDB converts a symbolic name to an address, or an address to the nearest symbol. Because targets contain multiple symbol tables and each symbol table can include symbols from multiple object files, different symbols with the same name can exist. MDB uses the backquote “`” character as a symbol-name scoping operator to allow the programmer to obtain the value of the desired symbol in this situation.
The /, , ?, and = metacharacters denote the special output formatting dcmds. Each of these dcmds accepts an argument list consisting of one or more format characters, repeat counts, or quoted strings. A format character is one of the ASCII characters shown in Table 13.6.
Format characters read or write and format data from the target. They are combined with the formatting metacharacters to read, write, or search memory. For example, if we want to display or set the value of a memory location, we could represent that location by its hexadecimal address or by its symbol name. Typically, we use a metacharacter with a format or a dcmd to indicate what we want MDB to do with the memory at the indicated address.
In the following example, we display the address of the kernel’s lotsfree symbol. We use the = metacharacter to display the absolute value of the symbol, lotsfree and the X format to display the address in 32-bit hexadecimal notation.
> lotsfree=X
fec4b8d0
In a more common example, we can use the / metacharacter to format for display the value at the address of the lotsfree symbol.
> lotsfree/D
lotsfree:
lotsfree: 4062
Optionally, a repeat count can be supplied with a format. A repeat count is a positive integer preceding the format character and is always interpreted in base 10 (decimal). A repeat count can also be specified as an expression enclosed in square brackets preceded by a dollar sign ($[ ]). A string argument must be enclosed in double-quotes (“ ”). No blanks are necessary between format arguments.
> lotsfree/4D
lotsfree:
lotsfree: 3934 1967 983 40
If MDB is started in writable (-w) mode, then write formats are enabled. Note that this should be considered MDB’s dangerous mode, especially if operating on live kernels or applications. For example, if we wanted to rewrite the value indicated by lotsfree to a new value, we could use the W write format with a valid MDB value or arithmetic expression as shown in the summary at the start of this section. For example, the W format writes the 32-bit value to the given address. In this example, we use an integer value, represented by the 0t arithmetic expression prefix.
> lotsfree/W 0t5000
lotsfree:
lotsfree: f5e
A complete list of format strings can be found with the ::formats dcmd.
> ::formats
+ - increment dot by the count (variable size)
- - decrement dot by the count (variable size)
B - hexadecimal int (1 byte)
C - character using C character notation (1 byte)
D - decimal signed int (4 bytes)
E - decimal unsigned long long (8 bytes)
...
A summary of the common formatting characters and the required metacharacters is shown in Table 13.7 through Table 13.9.
Table 13.7. Metacharacters and Formats for Reading
Metacharacter | Description | ||
|---|---|---|---|
| value is immediate or | ||
| format VA from . (dot) | ||
format PA from. | |||
| format primary object file, using VA from. | ||
| format value of. | ||
Format | Description | Format | Description |
| hex |
| dot += increment |
| char (C-encoded) |
| dot -= increment |
| unsigned |
| dot -= incr*count |
| octal |
| newline |
| char (raw) |
| newline |
| signed |
| tab |
| hex, swap endianness |
| whitespace |
| octal |
| tab |
| signed octal |
| dot as symbol+offset |
| decimal |
| address and instruction |
| signed |
| instruction |
| hex, swap endianness |
| string (C-encoded) |
| octal |
| string (raw) |
| signed octal |
| unsigned |
| unsigned |
| double |
| hex |
| octal |
| decoded |
| hex |
| float |
| binary |
|
|
| signed |
| symbol |
| signed octal |
| symbol |
| decoded |
Table 13.8. Metacharacters and Formats for Writing
Description | |
|---|---|
| value is immediate or |
| write virtual addresses |
write physical addresses | |
| write object file |
| Description |
| write low byte of each value, starting at dot |
| write low 2 bytes of each value, starting at dot |
| write low 4 bytes of each value, starting at dot |
| write all 8 bytes of each value, starting at dot |
Table 13.9. Metacharacters and Formats for Searching
Metacharacter | Description |
|---|---|
[/?][lLM] value [mask] | value and mask are immediate or |
/ | search virtual addresses |
search physical addresses | |
? | search object file |
Format | Description |
l (2) | search for 2-byte value, optionally masked |
L (4) | search for 4-byte value, optionally masked |
M (8) | search for 8-byte value, optionally masked |
The metacharacters we explored in the previous section are actually forms of dcmds. The more general form of a dcmd is ::name, where name is the command name, as summarized by the following:
::{module`}d
expr>var write the value of expr into var
A list of dcmds can be obtained with ::dcmds. Alternatively, the ::dmods command displays information about both dcmds and walkers, conveniently grouped per MDB module.
> ::dmods -l
genunix
...
dcmd pfiles - print process file information
dcmd pgrep - pattern match against all processes
dcmd pid2proc - convert PID to proc_t address
dcmd pmap - print process memory map
dcmd project - display kernel project(s)
dcmd prtconf - print devinfo tree
dcmd ps - list processes (and associated thr,lwp)
dcmd ptree - print process tree
...
Help on individual dcmds is available with the help dcmd. Yes, almost everything in MDB is implemented as a dcmd!
> ::help ps
NAME
ps - list processes (and associated thr,lwp)
SYNOPSIS
::ps [-fltzTP]
ATTRIBUTES
Target: kvm
Module: genunix
Interface Stability: Unstable
For example, we can optionally use ::ps as a simple dcmd with no arguments.
> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 fffffffffbc23640 sched
R 3 0 0 0 0 0x00020001 ffffffff812278f8 fsflush
R 2 0 0 0 0 0x00020001 ffffffff81228520 pageout
R 1 0 0 0 0 0x42004000 ffffffff81229148 init
R 1782 1 1782 1782 1 0x42000000 ffffffff8121cc38 lockd
R 524 1 524 524 0 0x42000000 ffffffff8b7fd548 dmispd
R 513 1 513 513 0 0x42010000 ffffffff87bd2878 snmpdx
R 482 1 7 7 0 0x42004000 ffffffff87be90b8 intrd
R 467 1 466 466 0 0x42010000 ffffffff87bd8020 smcboot
Optionally, we could use the same ::ps dcmd with an address supplied in hexadecimal.
> ffffffff87be90b8::ps S PID PPID PGID SID UID FLAGS ADDR NAME R 482 1 7 7 0 0x42004000 ffffffff87be90b8 intrd > ffffffff87be90b8::ps -ft S PID PPID PGID SID UID FLAGS ADDR NAME R 482 1 7 7 0 0x42004000 ffffffff87be90b8 /usr/perl5/bin/perl / usr/lib/intrd T 0xffffffff8926d4e0 <TS_SLEEP>
A walker is used to traverse a connect set of data. Walkers are a type of plugin that is coded to iterate over the specified type of data. In addition to the ::dcmds dcmd, the ::walkers dcmd lists walkers.
> ::walkers
Client_entry_cache - walk the Client_entry_cache cache
DelegStateID_entry_cache - walk the DelegStateID_entry_cache cache
File_entry_cache - walk the File_entry_cache cache
HatHash - walk the HatHash cache
...
For example, the ::proc walker could be used to traverse set of process structures (proc_ts). Many walkers also have a default data item to walk if none is specified.
> ::walk proc
fffffffffbc23640
ffffffff812278f8
ffffffff81228520
...
There are walkers to traverse common generic data structure indexes. For example, simple linked lists can be traversed with the ::list walker, and AVL trees with the ::avl walker.
> ffffffff9a647ae0::walk avl ffffffff9087a990 fffffe85ad8aa878 fffffe85ad8aa170 ... > fffffffffbc23640::list proc_t p_prev fffffffffbc23640 ffffffff81229148 ffffffff81228520 ...
MDB provides a compatibility mode that can interpret macros built for adb. A macro file is a text file containing a set of commands to execute. Macro files typically automate the process of displaying a simple data structure. These older macros can therefore be used with either tool. The development of macros is discouraged, since they are difficult to construct and maintain. Following is an example of using a macro to display a data structure.
> d8126310$<ce
ce instance structure
0xd8126310: dip instance dev_regs
d8c8e840 d84b65c8 d2999900
...
Walkers and dcmds can build on each other, combining to do more powerful things by placement into an mdb “pipeline.”
The purpose of a pipeline is to pass a list of values, typically virtual addresses, from one dcmd or walker to another. Pipeline stages might map a pointer from one type of data structure to a pointer to a corresponding data structure, sort a list of addresses, or select the addresses of structures with certain properties.
MDB executes each dcmd in the pipeline in order from left to right. The leftmost dcmd executes with the current value of dot or with the value specified by an explicit expression at the start of the command. When a | operator is encountered, MDB creates a pipe (a shared buffer) between the output of the dcmd to its left and the MDB parser, and an empty list of values.
To give you a taste of the power of pipelines, here’s an example, running against the live kernel. The ::pgrep dcmd allows you to find all processes matching a pattern, the thread walker walks all of the threads in a process, and the ::findstack dcmd gets a stack trace for a given thread. Connecting them into a pipeline, you can yield the stack traces of all sshd threads on the system (note that the middle one is swapped out). MDB pipelines are quite similar to standard UNIX pipelines and afford debugger users a similar level of power and flexibility.
> ::pgrep sshd S PID PPID PGID SID UID FLAGS ADDR NAME R 100174 1 100174 100174 0 0x42000000 0000030009216790 sshd R 276948 100174 100174 100174 0 0x42010000 000003002d9a9860 sshd R 276617 100174 100174 100174 0 0x42010000 0000030013943010 sshd > ::pgrep sshd | ::walk thread 3000c4f0c80 311967e9660 30f2ff2c340 > ::pgrep sshd | ::walk thread | ::findstack stack pointer for thread 3000c4f0c80: 2a10099d071 [ 000002a10099d071 cv_wait_sig_swap+0x130() ] 000002a10099d121 poll_common+0x530() 000002a10099d211 pollsys+0xf8() 000002a10099d2f1 syscall_trap32+0x1e8() stack pointer for thread 311967e9660: 2a100897071 [ 000002a100897071 cv_wait_sig_swap+0x130() ] stack pointer for thread 30f2ff2c340: 2a100693071 [ 000002a100693071 cv_wait_sig_swap+0x130() ] 000002a100693121 poll_common+0x530() 000002a100693211 pollsys+0xf8() 000002a1006932f1 syscall_trap32+0x1e8()
The full list of built-in dcmds can be obtained with the ::dmods dcmd.
> ::dmods -l mdb
mdb
dcmd $< - replace input with macro
dcmd $<< - source macro
dcmd $> - log session to a file
dcmd $? - print status and registers
dcmd $C - print stack backtrace
...
MDB can pipe output to UNIX commands with the ! pipe. A common task is to use grep to filter output from a dcmd. We’ve shown the output from ::ps for illustration; actually, a handy ::pgrep command handles this common task.
> ::ps !grep inet
R 255 1 255 255 0 0x42000000 ffffffff87be9ce0 inetd
The MDB environment exploits the Compact Type Format (CTF) information in debugging targets. This provides symbolic type information for data structures in the target; such information can then be used within the debugging environment.
Several dcmds consume CTF information, most notably ::print. The ::print dcmd displays a target data type in native C representation. The following example shows ::print in action.
/* process ID info */
struct pid {
unsigned int pid_prinactive :1;
unsigned int pid_pgorphaned :1;
unsigned int pid_padding :6; /* used to be pid_ref, now an int */
unsigned int pid_prslot :24;
pid_t pid_id;
struct proc *pid_pglink;
struct proc *pid_pgtail;
struct pid *pid_link;
uint_t pid_ref;
};
See sys/proc.h
> ::print -t "struct pid"
{
unsigned pid_prinactive :1
unsigned pid_pgorphaned :1
unsigned pid_padding :6
unsigned pid_prslot :24
pid_t pid_id
struct proc *pid_pglink
struct proc *pid_pgtail
struct pid *pid_link
uint_t pid_ref
}
The ::print dcmd is most useful to print data structures in their typed format. For example, using a pipeline we can look up the address of the p_pidp member of the supplied proc_t structure and print its structure’s contents.
> ::pgrep inet
S PID PPID PGID SID UID FLAGS ADDR NAME
R 1595 1 1595 1595 0 0x42000400 d7c662e0 inetd
> d7c662e0::print proc_t p_pidp |::print -t "struct pid"
{
unsigned pid_prinactive :1 = 0
unsigned pid_pgorphaned :1 = 0x1
unsigned pid_padding :6 = 0
unsigned pid_prslot :24 = 0xae
pid_t pid_id = 0x63b
struct proc *pid_pglink = 0xd7c662e0
struct proc *pid_pgtail = 0xd7c662e0
struct pid *pid_link = 0
uint_t pid_ref = 0x3
}
The ::print command also understands how to traverse more complex data structures. For example, here we traverse an element of an array.
> d7c662e0::print proc_t p_user.u_auxv[9]
{
p_user.u_auxv[9].a_type = 0x6
p_user.u_auxv[9].a_un = {
a_val = 0x1000
a_ptr = 0x1000
a_fcn = 0x1000
}
}
Several other dcmds, listed below, use the CTF information. Starting with Solaris 9, the kernel is compiled with CTF information, making type information available by default. Starting with Solaris 10, CTF information is also available in userland, and by default some of the core system libraries contain CTF. The CTF-related commands are summarized in Table 13.10.
Table 13.10. CTF-Related dcmds
dcmd | Description |
|---|---|
addr::print [type] [field...] | Use CTF info to print out a full structure or particular fields thereof. |
::sizeof type ::offsetof type field ::enum enumname | Get information about a type. |
addr::array [type count] [var] | Walk the count elements of an array of type |
addr::list type field [var] | Walk a circular or NULL-terminated list of type |
::typegraph addr::whattype addr::istype type addr::notype | Use the type inference engine—works on non-debug text. |
A variable is a variable name, a corresponding integer value, and a set of attributes. A variable name is a sequence of letters, digits, underscores, or periods. A variable can be assigned a value with > dcmd and read with < dcmd. Additionally, the variable can be the ::typeset dcmd, and its attributes can be manipulated with the ::typeset dcmd. Each variable’s value is represented as a 64-bit unsigned integer. A variable can have one or more of the following attributes:
Read-only (cannot be modified by the user)
Persistent (cannot be unset by the user)
Tagged (user-defined indicator)
The following examples shows assigning and referencing a variable.
> 0t27>myvar > <myvar=D 27 > $v myvar = 1b . = 1b 0 = f5e b = fec00000 d = 85737 e = fe800000 m = 464c457f t = 1a3e70
The CPU’s registers are also exported as variables.
> ::vars
uesp = 0
eip = 0
myvar = 1b
cs = 0
savfp = 0
ds = 0
trapno = 0
es = 0
. = 1b
0 = f5e
1 = 0
2 = 0
ss = 0
9 = 0
fs = 0
gs = 0
_ = 0
eax = 0
b = fec00000
d = 85737
e = fe800000
eflags = 0
ebp = 0
m = 464c457f
ebx = 0
t = 1a3e70
ecx = 0
hits = 0
edi = 0
edx = 0
err = 0
esi = 0
esp = 0
savpc = 0
thread = 0
Commands for working with variables are summarized in Table 13.11.
Table 13.11. Variables
Variable | Description |
|---|---|
| Most recent value |
| Most recent count for |
| Base VA of the data section |
| Size of the data |
| VA of entry point |
| Event callback match count |
| Magic number of primary object file, or zero |
| Size of text section |
| TID of current representative thread |
Variables can be combined with arithmetic expressions and evaluated to construct more complex pipelines, in which data is manipulated between stages. In a simple example, we might want to iterate only over processes that have a uid of zero. We can easily iterate over the processes by using a pipeline consisting of a walker and type information, which prints the cr_uids for every process.
> ::walk proc | ::print proc_t p_cred->cr_uid
cr_uid = 0
cr_uid = 0x19
cr_uid = 0x1
cr_uid = 0
...
Adding an expression allows us to select only those that match a particular condition. The ::walk dcmd takes an optional variable name, in which to place the value of the walk. In this example, the walker sets the value of myvar and also pipes the output of the same addresses into ::print, which extracts the value of proc_t->p_cred->cr_uid. The ::eval dcmd prints the variable myvar only when the expression is true; in this case when the result of the previous dcmd (the printed value of cr_uid) is equal to 1. The statement given to ::eval to execute retrieves the value of the variable myvar and formats it with the K format (uint_ptr_t).
> ::walk proc myvar |::print proc_t p_cred->cr_uid |::grep .==1 |::eval <myvar=K fec1d280 d318d248 d318daa8 d318e308 ... > ::walk proc myvar | ::print proc_t p_cred->cr_uid |::grep .==1 |::eval <myvar=K |::print -d proc_t p_pidp->pid_id p_pidp->pid_id = 0t4189 p_pidp->pid_id = 0t4187 p_pidp->pid_id = 0t4067 p_pidp->pid_id = 0t4065 ...
MDB can control and interact with live mdb processes or kmdb kernel targets. Typical debugging operations include starting, stopping, and stepping the target. We discuss more about controlling kmdb targets in Chapter 14. The common commands for controlling targets are summarized in Table 13.12.
Table 13.12. Debugging Target dcmds
dcmd | Description |
|---|---|
::status | Print summary of current target. |
$r ::regs | Display current register values for target. |
$c ::stack $C | Print current stack trace ( |
addr[, b] ::dump [-g sz] [-e] | Dump at least |
addr::dis | Disassemble text, starting around |
[ addr ] :b [ addr ] ::bp [+/-dDestT] [-c cmd] [-n count] sym ... addr [cmd ... ] | Set breakpoint at |
$b ::events [-av] $b [-av] | Display all breakpoints. |
addr ::delete [id | all] addr :d [id | all] | Delete a breakpoint at |
| Delete all breakpoints. |
::cont [SIG] :c [SIG] | Continue the target program, and wait for it to terminate. |
id ::evset [+/-dDestT] [-c cmd] [-n count] id ... | Modify the properties of one or more software event specifiers. |
::next [SIG] :e [SIG] | Step the target program one instruction, but step over subroutine calls. |
::step [branch | over | out] [SIG] :s SIG :u SIG | Step the target program one instruction. |
addr [, len]::wp [+/-dDestT] [-rwx] [-ip] [-c cmd] [-n count] addr [, len]:a [cmd... ] addr [, len]:p [cmd... ] addr [, len]:w [cmd... ] | Set a watchpoint at the specified address. |
We can print a stack of the current address with the $c command or with $C, which also prints the stack frame address for each stack level.
> $c atomic_add_32+8(0) nfs4_async_inactive+0x3b(dc1c29c0, 0) nfs4_inactive+0x41() fop_inactive+0x15(dc1c29c0, 0) vn_rele+0x4b(dc1c29c0) snf_smap_desbfree+0x59(dda94080) > $C d2a58828 atomic_add_32+8(0) d2a58854 nfs4_async_inactive+0x3b(dc1c29c0, 0) d2a58880 nfs4_inactive+0x41() d2a5889c fop_inactive+0x15(dc1c29c0, 0) d2a588b0 vn_rele+0x4b(dc1c29c0) d2a588c0 snf_smap_desbfree+0x59(dda94080)
We can print a stack of the current address with the $c command or with $C, which also prints the stack frame address for each stack level.
> ::regs (or $r) %cs = 0x0158 %eax = 0x00000000 %ds = 0xd9820160 %ebx = 0xde453000 %ss = 0x0000 %ecx = 0x00000001 %es = 0xfe8d0160 %edx = 0xd2a58de0 %fs = 0xfec30000 %esi = 0xdc062298 %gs = 0xfe8301b0 %edi = 0x00000000 %eip = 0xfe82ca58 atomic_add_32+8 %ebp = 0xd2a58828 %esp = 0xd2a58800 %eflags = 0x00010282 id=0 vip=0 vif=0 ac=0 vm=0 rf=1 nt=0 iopl=0x0 status=<of,df,IF,tf,SF,zf,af,pf,cf> %uesp = 0xfe89ab0d %trapno = 0xe %err = 0x2
We can dissasemble instructions in the target with the ::dis dcmd.
> atomic_add_32+8::dis
atomic_add_32: movl 0x4(%esp),%eax
atomic_add_32+4: movl 0x8(%esp),%ecx
atomic_add_32+8: lock addl %ecx,(%eax)
atomic_add_32+0xb: ret
Note that in this example combined with the registers shown in Section 13.3.2, the contents of %eax from $r is zero, causing the movl instruction to trap with a NULL pointer reference at atomic_add_32+4.
We can set breakpoints in MDB by using :b. Typically, we pass a symbol name to :b (the name of the function of interest).
We can start the target program and then set a breakpoint for the printf function.
> printf:b > :r mdb: stop at 0x8050694 mdb: target stopped at: PLT:printf: jmp *0x8060980
In this example, we stopped at the first symbol matching “printf”, which is actually in the procedure linkage table (PLT) (see the Linker and Libraries manual for a description of how dynamic linking works in Solaris). To match the printf we likely wanted, we can increase the scope of the symbol lookup. The :c command continues execution until the next breakpoint or until the program finishes.
> libc`printf:b > :c mdb: stop at libc.so.1`printf mdb: target stopped at: libc.so.1`printf: pushl %ebp
Table 13.13. GDB-to-MDB Migration
MDB | Description | ||
|---|---|---|---|
Starting Up | |||
|
| Start debugging a command or running process. GDB will treat numeric arguments as pids, while MDB explicitly requires the | |
|
| Debug a corefile associated with | |
Exiting | |||
|
| Both programs also exit on Ctrl-D. | |
Getting Help | |||
| |||
|
| List all the available walkers or dcmds, as well as get help on a specific dcmd (MDB). Another useful trick is | |
Running Programs | |||
|
| Run the program with the given arguments. If the target is currently running or is a corefile, MDB will restart the program if possible. | |
|
| Forcibly kill and release target. | |
|
| Display current environment. | |
|
| Set an environment variable. | |
|
| Get a specific environment variable. | |
|
| Execute the given shell command. | |
Breakpoints and Watchpoints | |||
| |||
|
| Set a breakpoint at the given address or function. | |
| — | Break at the given line of the file. MDB does not support source-level debugging. | |
| — | Set a conditional breakpoint. MDB doesn’t support conditional breakpoints, though you can get a close approximation with the | |
|
| Set a watchpoint on the given region of memory. | |
| |||
|
| Display active watchpoints and breakpoints. MDB shows you signal events as well. | |
|
| Delete the given breakpoint or watchpoints. | |
Program Stack | |||
|
| Display stack backtrace for the current thread. | |
— |
| Display a stack for a given thread. In the kernel, thread is the address of | |
| — | Display information about the current frame. MDB doesn’t support the debugging data necessary to maintain the frame abstraction. | |
Execution Control | |||
| |||
|
| Continue target. | |
|
| Step to the next machine instruction. MDB does not support stepping by source lines. | |
|
| Step over the next machine instruction, skipping any function calls. | |
|
| Continue until returning from the current frame. | |
|
| Jump to the given location. In MDB, | |
Display | |||
|
| Print the given expression. In GDB you can specify variable names as well as addresses. For MDB, you give a particular address and then specify the type to display (which can include dereferencing of members, etc.). | |
|
| Print data in a precise format. See | |
|
| Disassemble text at the given address or the current PC if no address is specified. | |
pipeline [!word...] [;] basic
expr pipeline [!word...] [;] set dot, run once
expr, expr pipeline [!word...] [;] set dot, repeat
,expr pipeline [!word...] [;] repeat
expr [!word...] [;] set dot, last pipeline, run once
,expr [!word...] [;] last pipeline, repeat
expr, expr [!word...] [;] set dot, last pipeline, repeat
!word... [;] shell escape
Arithmetic integer 0i binary, 0o octal, 0t decimal, 0x hex 0t[0-9]+.[0-9]+ IEEE floating point 'cccccccc' Little-endian character const <identifier variable lookup identifier symbol lookup (expr) the value of expr . the value of dot & last dot used by dcmd + dot+increment ^ dot-increment increment is effected by the last formatting dcmd. Unary Ops #expr logical NOT ~expr bitwise NOT -expr integer negation %expr object file pointer dereference %/[csil]/expr object file typed dereference %/[1248]/expr object file sized dereference *expr virtual address pointer dereference */[csil]/expr virtual address typed dereference */[1248]/expr virtual address sized dereference [csil] is char-, short-, int-, or long-sized Binary Ops expr * expr integer multiplication expr % expr integer division left # right left rounded up to next right multiple expr + expr integer addition expr - expr integer subtraction expr << expr bitwise left shift expr >> expr bitwise right shift (logical) expr == expr logical equality expr != expr logical inequality expr & expr bitwise AND expr ^ expr bitwise XOR expr | expr bitwise OR
0 Most recent value [/?=]ed.
9 Most recent count for $< dcmd
b base VA of the data section
d size of the data
e VA of entry point
hits Event callback match count
m magic number of primary object file, or zero
t size of text section
thread TID of current representative thread.
registers are exported as variables (g0, g1, ...)
/ format VA from .
format PA from .
? format primary object file, using VA from .
= format value of .
B (1) hex + dot += increment
C (1) char (C-encoded) - dot -= increment
V (1) unsigned ^ (var) dot -= incr*count
b (1) octal N newline
c (1) char (raw) n newline
d (2) signed T tab
h (2) hex, swap endianness r whitespace
o (2) octal t tab
q (2) signed octal a dot as symbol+offset
u (2) decimal I (var) address and instruction
D (4) signed i (var) instruction
H (4) hex, swap endianness S (var) string (C-encoded)
O (4) octal s (var) string (raw)
Q (4) signed octal E (8) unsigned
U (4) unsigned F (8) double
X (4) hex G (8) octal
Y (4) decoded time32_t J (8) hex
f (4) float R (8) binary
K (4|8) hex uintptr_t e (8) signed
P (4|8) symbol g (8) signed octal
p (4|8) symbol y (8) decoded time64_t
[/?][vwWZ] value... value is immediate or $[expr]
/ write virtual addresses
write physical addresses
? write object file
v (1) write low byte of each value, starting at dot
w (2) write low 2 bytes of each value, starting at dot
W (4) write low 4 bytes of each value, starting at dot
Z (8) write all 8 bytes of each value, starting at dot
[/?][lLM] value [mask] value and mask are immediate or $[expr]
/ search virtual addresses
search physical addresses
? search object file
l (2) search for 2-byte value, optionally masked
L (4) search for 4-byte value, optionally masked
M (8) search for 8-byte value, optionally masked
::status
Print summary of current target.
$r / ::regs
Display current register values for target.
$c / ::stack / $C
Print current stack trace ($C: with frame pointers).
addr[,b]::dump [-g sz] [-e]
Dump at least b bytes starting at address addr. -g sets
the group size -- for 64-bit debugging, '-g 8' is useful.
addr::dis
Disassemble text, starting around addr.
[ addr ] :b
[ addr ] ::bp [+/-dDestT] [-c cmd] [-n count] sym ... addr [cmd ... ]
Set breakpoint at addr.
$b
::events [-av]
$b [-av]
Display all the breakpoints.
addr ::delete [id | all]
addr :d [id | all]
Delete a breakpoint at addr.
:z
Deletes all breakpoints
::cont [SIG]
:c [SIG]
Continue the target program, and wait for it to terminate
id ::evset [+/-dDestT] [-c cmd] [-n count] id ...
Modify the properties of one or more software event specifiers.
::next [SIG]
:e [SIG]
Step the target program one instruction, but step over subroutine calls.
::step [branch | over | out] [SIG]
:s SIG
:u SIG
Step the target program one instruction.
addr [,len]::wp [+/-dDestT] [-rwx] [-ip] [-c cmd] [-n count]
addr [,len]:a [cmd... ]
addr [,len]:p [cmd... ]
addr [,len]:w [cmd... ]
Set a watchpoint at the specified address.
addr::print [type] [field...]
Use CTF info to print out a full structure, or
particular fields thereof.
::sizeof type / ::offsetof type field / ::enum enumname
Get information about a type
addr::array [type count] [var]
Walk the count elements of an array of type 'type'
starting at address.
addr::list type field [var]
Walk a circular or NULL-terminated list of type 'type',
which starts at addr and uses 'field' as its linkage.
::typegraph / addr::whattype / addr::istype type / addr::notype
bmc's type inference engine -- works on non-debug
0tpid::pid2proc
Convert the process ID 'pid' (in decimal) into a proc_t ptr.
as::as2proc
Convert a 'struct as' pointer to its associated proc_t ptr.
vn::whereopen
Find all processes with a particular vnode open.
::pgrep pattern
Print out proc_t ptrs which match pattern.
[procp]::ps
Process table, or (with procp) the line for particular proc_t.
::ptree
Print out a ptree(1)-like indented process tree.
procp::pfiles
Print out information on a process' file descriptors.
[procp]::walk proc
walks all processes, or the tree rooted at procp
threadp::findstack
Print out a stack trace (with frame pointers) for threadp.
[threadp]::thread
Give summary information about all threads or a particular thread.
[procp]::walk thread
Walk all threads, or all threads in a process (with procp).
[sobj]::wchaninfo [-v]
Get information on blocked-on condition variables. With
sobj, info about that wchan. With -v, lists all threads
blocked on the wchan.
sobj::rwlock
Dump out a rwlock, including detailed blocking information.
sobj::walk blocked
Walk all threads blocked on sobj, a synchronization object.
::cpuinfo [-v]
Give information about CPUs on the system and what they
are doing. With '-v', show threads on the run queues.
::cpupart
Give information about CPU partitions (psrset(1m)s).
addr::cpuset
Print out a cpuset as a list of included CPUs.
[cpuid]::ttrace
Dump out traptrace records, which are generated in DEBUG
kernels. These include all traps and various other events of
interest.
::walk cpu
Walk all cpu_ts on the system.
::memstat
Display memory usage summary.
pattern::kgrep [-d dist|-m mask|-M invmask]
Search the kernel heap for pointers equal to pattern.
addr::whatis [-b]
Try to identify what a given kernel address is. With
'-b', give bufctl address for the buffer (see
$<bufctl_audit, below).
::kmastat
Give statistics on the kmem caches and vmem arenas in the system
::kmem_cache
Information about the kmem caches on the system
[cachep]::kmem_verify
Validate all buffers in the system, checking for corruption.
With cachep, shows the details of a particular cache.
threadp::allocdby / threadp::freedby
Show buffers that were last allocated/freed by a particular
thread, and are still in that state.
::kmalog [fail | slab]
Dump out the transaction log, showing recent kmem activity.
With fail/slab, outputs records of allocation failures and
slab creations (which are always enabled)
::findleaks [-dvf]
Find memory leaks, coalesced by stack trace.
::bufctl [-v]
Print a summary line for a bufctl -- can also filter them
-v dumps out a kmem_bufctl_audit_t.
::walk cachename
Print out all allocated buffers in the cache named cachename.
[cp]::walk kmem/[cp]::walk freemem/[cp]::walk bufctl/[cp]::walk freectl
Walk {allocated,freed}{buffers,bufctls} for all caches,
or the particular kmem_cache_t cp.
flt ::fltbp [+/-dDestT] [-c cmd] [-n count] flt ...
Trace the specified machine faults.
signal :i
Ignore the specified signal and allow it to be delivered
transparently to the target.
$i
Display the list of signals that are ignored by the debugger and
will be handled directly by the target.
$l
Print the LWPID of the representative thread if the target is a user process.
$L
Print the LWPIDs of each LWP in the target if the target is a user
process.
::kill
:k
Forcibly terminate the target if it is a live user process.
::run [args ... ]
:r [args ... ]
Start a new target program running with the specified arguments and
attach to it.
[signal] ::sigbp [+/-dDestT] [-c cmd] [-n count] SIG ...
[signal] :t [+/-dDestT] [-c cmd] [-n count] SIG ...
Trace delivery of the specified signals.
::step [branch | over | out] [SIG]
:s SIG
:u SIG
Step the target program one instruction.
[syscall] ::sysbp [+/-dDestT] [-io] [-c cmd] [-n count] syscall ...
Trace entry to or exit from the specified system calls.
::help dcmd
gives help text for 'dcmd'
::dmods -l [module...]
Lists dcmds and walkers grouped by the dmod which provides them
::status
Print summary of current target.
$r
::regs
Display current register values for target.
$c
::stack
$C
Print current stack trace ($C: with frame pointers).
addr[,b]
::dump [-g sz] [-e]
Dump at least b bytes starting at address addr. -g sets the group size;
for 64-bit debugging, -g 8 is useful.
addr::dis
Disassemble text, starting around addr.
[ addr ] :b
[ addr ] ::bp [+/-dDestT] [-n count] sym ... addr
Set breakpoint at addr.
$b
Display all the breakpoints.
::branches
Display the last branches taken by the CPU. (x86 only)
addr ::delete [id | all]
addr :d [id | all]
Delete a breakpoint at addr.
:z
Delete all breakpoints.
function ::call [arg [arg ...]]
Call the specified function, using the specified arguments.
[cpuid] ::cpuregs [-c cpuid]
Display the current general-purpose register set.
[cpuid] ::cpustack [-c cpuid]
Print a C stack backtrace for the specified CPU.
::cont
:c
Continue the target program.
$M
List the macro files that are cached by kmdb for use with the $< dcmd
::next
:e
Step the target program one instruction, but step over subroutine calls.
::step [branch | over | out]
Step the target program one instruction.
$<systemdump
Initiate a panic/dump.
::quit [-u]
$q
Cause the debugger to exit. When the -u option is used,
the system is resumed and the debugger is unloaded.
addr [,len]::wp [+/-dDestT] [-rwx] [-ip] [-n count]
addr [,len]:a [cmd ...]
addr [,len]:p [cmd ...]
addr [,len]:w [cmd ...]
Set a watchpoint at the specified address.