Home Page Icon
Home Page
Table of Contents for
Two. Observability Infrastructure
Close
Two. Observability Infrastructure
by Brendan Gregg, Jim Mauro, Richard McDougall
Solaris
Copyright
Dedication
Foreword
Preface
About These Books
Solaris™ Internals
Solaris™ Performance and Tools
Intended Audience
OpenSolaris
How the Books Are Organized
Volume 1: Solaris™ Internals
Volume 2: Solaris™ Performance and Tools
Updates and Related Material
Notational Conventions
A Note from the Authors
About the Authors
Acknowledgments
The Solaris™ Internals Community Authors
The Reviewers
Personal Acknowledgments from Richard
Personal Acknowledgments from Jim
Personal Acknowledgements from Brendan
One. Observability Methods
1. Introduction to Observability Tools
1.1. Observability Tools
1.1.1. Kstat Tools
1.1.2. Process Tools
1.1.3. Forensic Tools
1.2. Drill-Down Analysis
1.3. About Part One
1.3.1. Chapter Layout
2. CPUs
2.1. Tools for CPU Analysis
2.2. vmstat Tool
2.3. CPU Utilization
2.4. CPU Saturation
2.5. psrinfo Command
2.6. uptime Command
2.7. sar Command
2.7.1. sar Default Output
2.7.2. sar -q
2.7.3. Capturing Historical Data
2.8. Clock Tick Woes
2.9. mpstat Command
2.10. Who Is Using the CPU?
2.11. CPU Run Queue Latency
2.12. CPU Statistics Internals
2.12.1. usr, sys, idl Times
2.12.2. Load Averages
2.12.3. pr_pctcpu Field
2.13. Using DTrace to Explain Events from Performance Tools
2.14. DTrace Versions of runq-sz, %runocc
2.15. DTrace Probes for CPU States
3. Processes
3.1. Tools for Process Analysis
3.2. Process Statistics Summary: prstat
3.2.1. Thread Summary: prstat -L
3.2.2. Process Microstates: prstat -m
3.2.3. Sorting by a Key: prstat -s
3.2.4. User Summary: prstat -t
3.2.5. Project Summary: prstat -J
3.2.6. Zone Summary: prstat -Z
3.3. Process Status: ps
3.3.1. /usr/bin/ps Command
3.3.2. /usr/ucb/ps
3.4. Tools for Listing and Controlling Processes
3.4.1. Process Tree: ptree
3.4.2. Grepping for Processes: pgrep
3.4.3. Killing Processes: pkill
3.4.4. Temporarily Stop a Process: pstop
3.4.5. Making a Process Runnable: prun
3.4.6. Wait for Process Completion: pwait
3.4.7. Reap a Zombie Process: preap
3.5. Process Introspection Commands
3.5.1. Process Stack: pstack
3.5.2. Process Memory Map: pmap -x
3.5.3. Process File Table: pfiles
3.5.4. Execution Time Statistics for a Process: ptime
3.5.5. Process Signal Disposition: psig
3.5.6. Process Libraries: pldd
3.5.7. Process Flags: pflags
3.5.8. Process Credentials: pcred
3.5.9. Process Arguments: pargs
3.5.10. Process Working Directory: pwdx
3.6. Examining User-Level Locks in a Process
3.7. Tracing Processes
3.7.1. Using truss to Trace Processes
3.7.2. Using apptrace to Trace Processes
3.7.3. Using DTrace to Trace Process Functions
3.7.4. Using DTrace to Aggregate Process Functions
3.7.5. Using DTrace to Peer Inside Processes
3.7.6. Using DTrace to Sample Stack Backtraces
3.8. Java Processes
3.8.1. Process Stack on a Java Virtual Machine: pstack
3.8.2. JVM Profiling
3.8.3. Tuning Java Garbage Collection
3.8.4. Using DTrace on Java Applications
4. Disk Behavior and Analysis
4.1. Terms for Disk Analysis
4.2. Random vs. Sequential I/O
4.2.1. Demonstration of Sequential Disk Activity
4.2.2. Demonstration of Random Disk Activity
4.3. Storage Arrays
4.4. Sector Zoning
4.5. Max I/O Size
4.6. iostat Utility
4.7. Disk Utilization
4.8. Disk Saturation
4.9. Disk Throughput
4.10. iostat Reference
4.10.1. iostat Default
4.10.2. iostat -D
4.10.3. iostat -l n
4.10.4. iostat -n
4.10.5. iostat -x
4.10.6. iostat -p, -P
4.10.7. iostat -e
4.10.8. iostat -E
4.11. Reading iostat
4.11.1. Event Size Ratio
4.11.2. Service Times
4.12. iostat Internals
4.13. sar -d
4.14. Trace Normal Form (TNF) Tracing for I/O
4.15. DTrace for I/O
4.15.1. io Probes
4.15.2. I/O Size One-Liners
4.15.3. A More Elaborate Example
4.15.4. I/O Size Aggregation
4.15.5. I/O Seek Aggregation
4.15.6. I/O File Names
4.16. Disk I/O Time
4.16.1. Simple Disk Event
4.16.2. Concurrent Disk Events
4.16.3. Adaptive Disk I/O Time Algorithm
4.16.4. Other Response Times
4.16.5. Time by Layer
4.17. DTraceToolkit Commands
4.17.1. iotop Script
4.17.2. iosnoop Script
4.17.3. Plotting Disk Activity
4.17.4. Plotting Concurrent Activity
4.18. DTraceTazTool
5. File Systems
5.1. Layers of File System and I/O
5.2. Observing Physical I/O
5.3. File System Latency
5.4. Causes of Read/Write File System Latency
5.4.1. Disk I/O Wait
5.4.2. Block or Metadata Cache Misses
5.4.3. I/O Breakup
5.4.4. Locking in the File System
5.4.5. Metadata Updates
5.5. Observing File System “Top End” Activity
5.6. File System Caches
5.6.1. Page Cache
5.6.2. Bypassing the Page Cache with Direct I/O
5.6.3. The Directory Name Lookup Cache
5.6.4. Block Buffer Cache
5.6.5. UFS Inode Cache
5.6.6. Monitoring UFS Caches with fcachestat
5.7. NFS Statistics
5.7.1. NFS Client Statistics: nfsstat -c
5.7.2. NFS Server Statistics: nfsstat -s
6. Memory
6.1. Tools for Memory Analysis
6.2. vmstat(1M) Command
6.3. Types of Paging
6.3.1. File I/O Paging: “Good” Paging
6.3.2. Anonymous Memory Paging: “Bad” Paging
6.3.3. Per-Zone Paging Statistics
6.4. Physical Memory Allocation
6.4.1. Total Physical Memory
6.4.2. Free Memory
6.4.3. Using the memstat Command in MDB
6.5. Relieving Memory Pressure
6.6. Scan Rate as a Memory Health Indicator
6.6.1. Using prstat to Estimate Memory Slowdowns
6.7. Process Virtual and Resident Set Size
6.8. Using pmap to Inspect Process Memory Usage
6.9. Calculating Process Memory Usage with ps and pmap
6.10. Displaying Page-Size Information with pmap
6.11. Using DTrace for Memory Analysis
6.11.1. Using DTrace to Estimate Memory Slowdowns
6.12. Obtaining Memory Kstats
6.13. Using the Perl Kstat API to Look at Memory Statistics
6.14. System Memory Allocation Kstats
6.15. Kernel Memory with kstat
6.16. System Paging Kstats
6.17. Observing MMU Performance Impact with trapstat
6.18. Swap Space
6.18.1. Swap Allocation
6.18.2. Swap Statistics
6.18.3. Swap Summary: swap -s
6.18.4. Listing Physical Swap Devices: swap -l
6.18.5. Determining Swapped-Out Threads
6.18.6. Monitoring Physical Swap Activity
6.18.7. MemTool prtswap
6.18.8. Display of Swap Reservations with pmap
7. Networks
7.1. Terms for Network Analysis
7.2. Packets Are Not Bytes
7.3. Network Utilization
7.4. Network Saturation
7.5. Network Errors
7.6. Misconfigurations
7.7. Systemwide Statistics
7.7.1. netstat Command
7.7.2. kstat Command
7.7.3. nx.se Tool
7.7.4. nicstat Tool
7.7.5. SNMP
7.7.6. checkcable Tool
7.7.7. ping Tool
7.7.8. traceroute Tool
7.7.9. snoop Tool
7.7.10. TTCP
7.7.11. pathchar Tool
7.7.12. ntop Tool
7.7.13. NFS Client Statistics: nfsstat -c
7.7.14. NFS Server Statistics: nfsstat -s
7.8. Per-Process Network Statistics
7.8.1. tcptop Tool
7.8.2. tcpsnoop Tool
7.9. TCP Statistics
7.9.1. TCP Statistics Internals
7.9.2. TCP Statistics from Kstat
7.9.3. TCP Statistics Reference
7.9.4. TCP Statistics from DTrace
7.10. IP Statistics
7.10.1. IP Statistics Internals
7.10.2. IP Statistics from Kstat
7.10.3. IP Statistics Reference
7.10.4. IP Statistics from DTrace
7.11. ICMP Statistics
7.11.1. ICMP Statistics from Kstat
7.11.2. ICMP Statistics Reference
7.11.3. ICMP Statistics from DTrace
7.11.4. Tracing Raw Network Functions
8. Performance Counters
8.1. Introducing CPU Caches
8.2. cpustat Command
8.2.1. Cache Hit Ratio, Cache Misses
8.2.2. Listing PICs and Events
8.2.3. PIC Examples: UltraSPARC IIi
8.2.4. PIC Examples: The UltraSPARC T1 Processor
8.2.5. Event Multiplexing
8.2.6. Using cpustat with Multiple CPUs
8.2.7. Cycles per Instruction
8.2.8. PIC Examples: UltraSPARC IV
8.3. cputrack Command
8.4. busstat Command
8.4.1. Listing Supported Buses
8.4.2. Listing Bus Events
8.4.3. Monitoring Bus Events
8.4.4. Event Multiplexing
8.4.5. Example: UltraSPARC T1
9. Kernel Monitoring
9.1. Tools for Kernel Monitoring
9.2. Profiling the Kernel and Drivers
9.2.1. Profiling the Kernel with lockstat -I
9.3. Analyzing Kernel Locks
9.3.1. Adaptive Locks
9.3.2. Spin Locks
9.3.3. Reader/Writer Locks
9.3.4. Thread Locks
9.3.5. Analyzing Locks with lockstat
9.4. DTrace lockstat Provider
9.5. DTrace Kernel Profiling
9.6. Interrupt Statistics: vmstat -i
9.7. Interrupt Analysis: intrstat
Two. Observability Infrastructure
10. Dynamic Tracing
10.1. Introduction to DTrace
10.2. The Basics
10.2.1. D Program Structure
10.2.2. Providers and Probes
10.2.3. Aggregations
10.2.4. Variables
10.2.5. Probe Arguments
10.2.6. Mixing Providers
10.2.7. Accessing Global Kernel Data
10.2.8. Assorted Actions of Interest
10.3. Inspecting Java Applications with DTrace
10.3.1. Inspecting Applications with the DTrace jstack Action
10.3.2. Adding Probes to Pre-Mustang Releases
10.4. DTrace Architecture
10.5. Summary
10.6. Probe Reference
10.6.1. The I/O Provider
10.6.1.1. bufinfo_t structure
10.6.1.2. devinfo_t
10.6.1.3. fileinfo_t
10.6.2. Virtual Memory Provider Probes
10.6.3. The Sched Provider
10.6.3.1. Arguments
10.6.4. DTrace Lockstat Provider
10.6.4.1. Adaptive Lock Probes
10.6.4.2. Spin Lock Probes
10.6.4.3. Thread Locks
10.6.4.4. Readers/Writer Lock Probes
10.6.5. The Java Virtual Machine Provider
10.6.5.1. VM Life Cycle Probes
10.6.5.2. Thread Life Cycle Probes
10.6.5.3. Class-Loading Probes
10.6.5.4. Garbage Collection Probes
10.6.5.5. Method Compilation Probes
10.6.5.6. Monitor Probes
10.6.5.7. Application Tracking Probes
10.6.5.8. The hotspot_jni Provider
10.7. MDB Reference
11. Kernel Statistics
11.1. C-Level Kstat Interface
11.1.1. Data Structure Overview
11.1.2. Getting Started
11.1.3. Data Types
11.1.3.1. KSTAT_TYPE_RAW
11.1.3.2. KSTAT_TYPE_NAMED
11.1.3.3. KSTAT_TYPE_TIMER
11.1.3.4. KSTAT_TYPE_INTR
11.1.3.5. KSTAT_TYPE_IO
Accumulated Time and Queue Length Statistics
11.1.4. Kstat Names
11.1.5. Functions
11.1.6. Management of Chain Updates
11.1.7. Putting It All Together
11.2. Command-Line Interface
11.2.1. The kstat Command
11.2.2. Real-World Example That Uses kstat and nawk
11.3. Using Perl to Access kstats
11.3.1. The Tied-Hash Interface to the kstat Facility
11.3.2. The update() Method
11.3.3. 64-Bit Values
11.3.4. Getting Started with Perl
11.3.5. netstatMulti Implemented in Perl
11.4. Snooping a Program’s kstat Use with DTrace
11.5. Adding Statistics to the Solaris Kernel
11.5.1. A kstat Provider Walkthrough
11.5.2. I/O Statistics
11.6. Additional Information
Three. Debugging
12. The Modular Debugger
12.1. Introduction to the Modular Debugger
12.1.1. MDB
12.1.2. MDB Features
12.1.3. Terms
12.2. MDB Concepts
12.2.1. Building Blocks
12.2.2. Targets
12.2.3. Debugger Commands
12.2.4. Walker
12.2.5. Debugger Modules
12.2.6. Macros
12.2.7. Modularity
13. An MDB Tutorial
13.1. Invoking MDB
13.1.1. Logging Output to a File
13.2. MDB Command Syntax
13.2.1. Expressions
13.2.2. Symbols
13.2.3. Formatting Metacharacters
13.2.4. Formatting Characters
13.2.5. dcmds
13.2.6. Walkers
13.2.7. Macros
13.2.8. Pipelines
13.2.9. Piping to UNIX Commands
13.2.10. Obtaining Symbolic Type Information
13.2.11. Variables
13.2.12. Walkers, Variables, and Expressions Combined
13.3. Working with Debugging Targets
13.3.1. Displaying Stacks
13.3.2. Displaying Registers
13.3.3. Disassembling the Target
13.3.4. Setting Breakpoints
13.4. GDB-to-MDB Reference
13.5. dcmd and Walker Reference
13.5.1. Commands
13.5.2. Comments
13.5.3. Expressions
13.5.4. Symbols
13.5.5. dcmds
13.5.6. Variables
13.5.7. Read Formats
13.5.8. Write Formats
13.5.9. Search Formats
13.5.10. General dcmds
13.5.11. Target-Related dcmds
13.5.12. CTF-Related
13.5.13. Kernel: proc-Related
13.5.14. Kernel: Thread-Related
13.5.15. Kernel: Synchronization-Related
13.5.16. Kernel: CPU-Related
13.5.17. Kernel: Memory-Related
13.5.18. Kernel: kmem-Related
13.5.19. Process: Target-Related
13.5.20. Kernel: kmdb-Related
14. Debugging Kernels
14.1. Working with Kernel Cores
14.1.1. Locating and Attaching the Target
14.1.2. Examining Kernel Core Summary Information
14.1.3. Examining the Message Buffer
14.1.4. Obtaining a Stack Trace of the Running Thread
14.1.5. Which Process?
14.1.6. Disassembling the Suspect Code
14.1.7. Displaying General-Purpose Registers
14.1.8. Navigating the Stack Backtrace
14.1.9. Looking at the Status of the CPUs
14.1.10. Traversing Stack Frames in SPARC Architectures
14.1.11. Listing Processes and Process Stacks
14.1.12. Global Memory Summary
14.1.13. Listing Network Connections
14.1.14. Listing All Kernel Threads
14.1.15. Other Notable Kernel dcmds
14.2. Examining User Process Stacks within a Kernel Image
14.2.1. Enabling Process Pages in a Dump
14.2.2. Invoking MDB to Examine the Kernel Image
14.2.3. Locating the Target Process
14.2.4. Extracting the User-Mode Stack Frame Pointers
14.3. Switching MDB to Debug a Specific Process
14.3.1. Constructing the Process Stack
14.3.2. Examining the Process Memory
14.4. kmdb, the Kernel Modular Debugger
14.4.1. Diagnosing with kmdb and moddebug
14.4.1.1. Starting kmdb from the Console
14.4.2. Booting with the Kernel Debugger
14.4.3. Configuring a tty Console on x86
14.4.4. Investigating Hangs
14.4.5. Collecting Information about Panics
14.4.6. Working with Debugging Targets
14.4.7. Setting Breakpoints
14.4.8. Forcing a Crash Dump with halt -d
14.4.9. Forcing a Dump with kmdb
14.5. Kernel Built-In MDB dcmds
Appendices
A. Tunables and Settings
A.1. Tunable Parameters in Solaris
A.1.1. /etc/default Directory
A.1.2. prctl Command
A.1.3. /etc/system File
A.1.4. driver.conf File
A.1.5. ndd Command
A.1.6. routeadm(1)
A.2. System V IPC Tunables for Databases
B. DTrace One-Liners
B.1. DTrace One-Liners
B.2. DTrace Longer One-Liners
C. Java DTrace Scripts
C.1. dvm_probe_test.d
C.2. DVM Agent Provider Interface
D. Sample Perl Kstat Utilities
D.1. A Simple Kstat Walker
D.2. A Perl Version of Uptime
D.3. A Network Statistics Utility
D.4. A Performance Utility for CPU, Memory, Disk, and Net
Bibliography
Search in book...
Toggle Font Controls
Playlists
Add To
Create new playlist
Name your new playlist
Playlist description (optional)
Cancel
Create playlist
Sign In
Email address
Password
Forgot Password?
Create account
Login
or
Continue with Facebook
Continue with Google
Sign Up
Full Name
Email address
Confirm Email Address
Password
Login
Create account
or
Continue with Facebook
Continue with Google
Prev
Previous Chapter
9. Kernel Monitoring
Next
Next Chapter
10. Dynamic Tracing
Part Two. Observability Infrastructure
Chapter 10
, “
Dynamic Tracing
”
Chapter 11
, “
Kernel Statistics
”
Add Highlight
No Comment
..................Content has been hidden....................
You can't read the all page of ebook, please click
here
login for view all page.
Day Mode
Cloud Mode
Night Mode
Reset