fluent-plugin-perf-tools 0.1.0

data/perf-tools/man/man8/funcgraph.8

@@ -0,0 +1,166 @@
+.TH funcgraph 8  "2014-07-29" "USER COMMANDS"
+.SH NAME
+funcgraph \- trace kernel function graph, showing child function calls and times. Uses Linux ftrace.
+.SH SYNOPSIS
+.B funcgraph
+[\-aCDhHPtT] [\-m maxdepth] [\-p PID] [\-L TID] [\-d secs] funcstring
+.SH DESCRIPTION
+This is an exploratory tool that shows the graph of child function calls
+for a given kernel function. This can cost moderate overhead to execute, and
+should only be used to understand kernel behavior before using other, lower
+overhead tools. This is a proof of concept using Linux ftrace capabilities
+on older kernels.
+
+The output format is the same as the ftrace function graph trace format,
+described in the kernel source under Documentation/trace/ftrace.txt.
+Note that the output may be shuffled when different CPU buffers are read;
+check the CPU column for changes, or include timestamps (-t) and post sort.
+
+The "-d duration" mode leaves the trace data in the kernel buffer, and
+only reads it at the end. If the trace data is large, beware of exhausting
+buffer space (/sys/kernel/debug/tracing/buffer_size_kb) and losing data.
+
+Also beware of feedback loops: tracing tcp* functions over an ssh session,
+or writing ext4* functions to an ext4 file system. For the former, tcp
+trace data could be redirected to a file (as in the usage message). For
+the latter, trace to the screen or a different file system.
+
+WARNING: This uses dynamic tracing of kernel functions, and could cause
+kernel panics or freezes. Test, and know what you are doing, before use.
+Also see the OVERHEAD section.
+
+Since this uses ftrace, only the root user can use this tool.
+.SH REQUIREMENTS
+FTRACE CONFIG, which you may already have enabled and available on recent
+kernels.
+.SH OPTIONS
+.TP
+\-a
+All info. Same as \-HPt. (But no -T, which isn't available in older kernels.)
+.TP
+\-C
+Function durations measure on-CPU time only (exclude sleep time).
+.TP
+\-d seconds
+Set the duration of tracing, in seconds. Trace output will be buffered and
+printed at the end. This also reduces overheads by buffering in-kernel,
+instead of printing events as they occur.
+
+The ftrace buffer has a fixed size per-CPU (see
+/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,
+try increasing that size.
+.TP
+\-D
+Do not show function duration times.
+.TP
+\-h
+Print usage message.
+.TP
+\-H
+Print column headers.
+.TP
+\-m
+Max depth to trace functions. By default, unlimited (0). This feature is only
+available for newer Linux kernel versions.
+.TP
+\-p PID
+Only trace kernel functions when this process ID is on-CPU.
+.TP
+\-L TID
+Only trace kernel functions when this thread ID is on-CPU.
+.TP
+\-P
+Show process names and process IDs with every line of output.
+.TP
+\-t
+Show timestamps on every line of output.
+.TP
+\-T
+Tail mode: decorate function return lines with the name of the function. This
+option may not be available for older kernels.
+.TP
+funcstring
+A function name to trace, which may include file glob style wildcards ("*") at
+the beginning or ending of a string only. Eg, "vfs*" means match "vfs" followed
+by anything. Since the output is verbose, you probably only want to trace
+single functions, and not use wildcards.
+.SH EXAMPLES
+.TP
+Trace calls to do_nanosleep(), showing child functions and durations:
+#
+.B funcgraph do_nanosleep
+.TP
+Same as above, but include column headers:
+#
+.B funcgraph -H do_nanosleep
+.TP
+Same as above, but include timestamps and process names as well:
+#
+.B funcgraph -HtP do_nanosleep
+.TP
+Trace all vfs_read() kernel function calls, and child functions, for PID 198 only:
+#
+.B funcgraph \-p 198 vfs_read
+.TP
+Trace all vfs_read() kernel function calls, and child functions, for 1 second then write to a file.
+#
+.B funcgraph \-d 1 vfs_read > out
+.SH FIELDS
+The output format depends on the kernel version, and headings can be printed
+using \-H. The format is the same as the ftrace function trace format, described
+in the kernel source under Documentation/trace/ftrace.txt.
+
+Typical fields are:
+.TP
+TIME
+(Shown with \-t.) Time of event, in seconds.
+.TP
+CPU
+The CPU this event occurred on.
+.TP
+TASK/PID
+(Shown with \-P.) The process name (which could include dashes), a dash, and the process ID.
+.TP
+DURATION
+Elapsed time during the function call, inclusive of children. This is also
+inclusive of sleep time, unless -C is used. The time is either displayed on
+the return of a function ("}"), or for a leaf function (no children), on the
+same line.
+
+If the trace output begins with some returns that lack entries, their durations
+may not be trusted. This is usually only the case for the first dozen or so
+lines.
+.TP
+FUNCTION CALLS
+Entries and returns from kernel functions.
+.SH OVERHEAD
+This tool causes moderate to high overheads. Use with caution for
+exploratory purposes, then switch to lower overhead techniques based on
+findings. It's expected that the kernel will run at least 50% slower while
+this tool is running -- even while no output is being generated. This is
+because ALL kernel functions are traced, and filtered based on the function
+of interest. When output is generated, it can generate many lines quickly
+depending on the traced event. Such data will cause performance overheads.
+This also works without buffering by default, printing function events
+as they happen (uses trace_pipe), context switching and consuming CPU to do
+so. If needed, you can try the "-d secs" option, which buffers events
+instead, reducing overhead. If you think the buffer option is losing events,
+try increasing the buffer size (buffer_size_kb).
+
+It's a good idea to use funccount(8) first, which is lower overhead, to
+help you select which functions you may want to trace using funcgraph(8).
+.SH SOURCE
+This is from the perf-tools collection:
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+funccount(8), functrace(8), kprobe(8)

data/perf-tools/man/man8/funcslower.8

@@ -0,0 +1,129 @@
+.TH funcslower 8  "2014-07-30" "USER COMMANDS"
+.SH NAME
+funcslower \- trace kernel functions slower than a threshold (microseconds). Uses Linux ftrace.
+.SH SYNOPSIS
+.B funcslower
+[\-aChHPt] [\-p PID] [\-L TID] [\-d secs] funcstring latency_us
+.SH DESCRIPTION
+This uses the Linux ftrace function graph profiler to time kernel functions
+and filter them based on a latency threshold. Latency outliers can be studied
+this way, confirming their presence, duration, and rate. This tool
+is a proof of concept using Linux ftrace capabilities on older kernels.
+
+The output format is based on the ftrace function graph trace format,
+described in the kernel source under Documentation/trace/ftrace.txt. Use the
+\-H option to print column headings.
+Note that the output may be shuffled when different CPU buffers are read;
+check the CPU column for changes, or include timestamps (-t) and post sort.
+
+WARNING: This uses dynamic tracing of kernel functions, and could cause
+kernel panics or freezes. Test, and know what you are doing, before use.
+
+Since this uses ftrace, only the root user can use this tool.
+.SH REQUIREMENTS
+FTRACE function graph, which you may already have enabled and available on
+recent kernels. And awk.
+.SH OPTIONS
+.TP
+\-a
+All info. Same as \-HPt.
+.TP
+\-C
+Function durations measure on-CPU time only (exclude sleep time).
+.TP
+\-d seconds
+Set the duration of tracing, in seconds. Trace output will be buffered and
+printed at the end. This also reduces overheads by buffering in-kernel,
+instead of printing events as they occur.
+
+The ftrace buffer has a fixed size per-CPU (see
+/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,
+try increasing that size.
+.TP
+\-h
+Print usage message.
+.TP
+\-H
+Print column headers.
+.TP
+\-p PID
+Only trace kernel functions when this process ID is on-CPU.
+.TP
+\-L TID
+Only trace kernel functions when this thread ID is on-CPU.
+.TP
+\-P
+Show process names and process IDs with every line of output.
+.TP
+\-t
+Show timestamps on every line of output.
+.TP
+funcstring
+A function name to trace, which may include file glob style wildcards ("*") at
+the beginning or ending of a string only. Eg, "vfs*" means match "vfs" followed
+by anything. Since the output is verbose, you probably only want to trace
+single functions, and not use wildcards.
+.TP
+latency_us
+Minimum function duration to trace, in units of microseconds. This is filtered
+in-kernel.
+.SH EXAMPLES
+.TP
+Trace calls to vfs_read(), showing events slower than 10 ms:
+#
+.B funcslower vfs_read 10000
+.TP
+Same as above, but include column headers, event timestamps, and process names:
+#
+.B funcslower -HPt vfs_read 10000
+.TP
+Trace slow vfs_read()s for PID 198 only:
+#
+.B funcslower \-p 198 vfs_read 10000
+.SH FIELDS
+The output format depends on the kernel version, and headings can be printed
+using \-H. The format is the same as the ftrace function trace format, described
+in the kernel source under Documentation/trace/ftrace.txt.
+
+Typical fields are:
+.TP
+TIME
+(Shown with \-t.) Time of event, in seconds.
+.TP
+CPU
+The CPU this event occurred on.
+.TP
+TASK/PID
+(Shown with \-P.) The process name (which could include dashes), a dash, and the process ID.
+.TP
+DURATION
+Elapsed time during the function call, inclusive of children. This is also
+inclusive of sleep time, unless -C is used.
+.TP
+FUNCTION CALLS
+Kernel function returns.
+.SH OVERHEAD
+OVERHEADS: Timing and filtering is performed in-kernel context, costing
+lower overheads than post-processing in user space. If you trace frequent
+events (eg, pick a common function and a low threshold), you might want to
+try the "-d secs" option, which buffers events in-kernel instead of printing
+them live.
+
+It's a good idea to start with a high threshold (eg, "100000" for 100 ms) then
+to decrease it. If you start low instead, you may start printing too many
+events.
+.SH SOURCE
+This is from the perf-tools collection:
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+funccount(8), functrace(8), funcgraph(8), kprobe(8)

data/perf-tools/man/man8/functrace.8

@@ -0,0 +1,123 @@
+.TH functrace 8  "2014-07-20" "USER COMMANDS"
+.SH NAME
+functrace \- trace kernel function calls matching specified wildcards. Uses Linux ftrace.
+.SH SYNOPSIS
+.B functrace
+[\-hH] [\-p PID] [\-L TID] [\-d secs] funcstring
+.SH DESCRIPTION
+This tool provides a quick way to capture the execution of kernel functions,
+showing basic details including as the process ID, timestamp, and calling
+function.
+
+WARNING: This uses dynamic tracing of (what can be many) kernel functions,
+and could cause kernel panics or freezes. Test, and know what you are doing,
+before use.
+
+Also beware of feedback loops: tracing tcp* functions over an ssh session,
+or writing ext4* functions to an ext4 file system. For the former, tcp
+trace data could be redirected to a file (as in the usage message). For
+the latter, trace to the screen or a different file system.
+
+SEE ALSO: kprobe(8), which can dynamically trace a single function call or
+return, and examine CPU registers and return values.
+
+Since this uses ftrace, only the root user can use this tool.
+.SH REQUIREMENTS
+FTRACE CONFIG, which you may already have enabled and available on recent
+kernels.
+.SH OPTIONS
+.TP
+\-d seconds
+Set the duration of tracing, in seconds. Trace output will be buffered and
+printed at the end. This also reduces overheads by buffering in-kernel,
+instead of printing events as they occur.
+
+The ftrace buffer has a fixed size per-CPU (see
+/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,
+try increasing that size.
+.TP
+\-h
+Print usage message.
+.TP
+\-H
+Print column headers.
+.TP
+\-p PID
+Only trace kernel functions when this process ID is on-CPU.
+.TP
+\-L TID
+Only trace kernel functions when this thread ID is on-CPU.
+.TP
+funcstring
+A function name to trace, which may include file glob style wildcards ("*") at
+the beginning or ending of a string only. Eg, "vfs*" means match "vfs" followed
+by anything.
+.SH EXAMPLES
+.TP
+Trace calls to do_nanosleep():
+#
+.B functrace do_nanosleep
+.TP
+Trace calls to all kernel functions ending in "*sleep":
+#
+.B functrace '*sleep'
+.TP
+Trace all "vfs*" kernel function calls for PID 198:
+#
+.B functrace \-p 198 'vfs*'
+.TP
+Trace all "tcp*" kernel function calls, and output to a file until Ctrl-C:
+#
+.B functrace 'tcp*' > out
+.TP
+Trace all "tcp*" kernel function calls, output to a file, for 1 second (buffered):
+#
+.B functrace \-d 1 'tcp*' > out
+.SH FIELDS
+The output format depends on the kernel version, and headings can be printed
+using \-H. The format is the same as the ftrace function trace format, described
+in the kernel source under Documentation/trace/ftrace.txt.
+
+Typical fields are:
+.TP
+TASK-PID
+The process name (which could include dashes), a dash, and the process ID.
+.TP
+CPU#
+The CPU ID, in brackets.
+.TP
+||||
+Kernel state flags. For example, on Linux 3.16 these are for irqs-off,
+need-resched, hardirq/softirq, and preempt-depth.
+.TP
+TIMESTAMP
+Time of event, in seconds.
+.TP
+FUNCTION
+Kernel function name.
+.SH OVERHEAD
+This can generate a lot of trace data quickly, depending on the
+frequency of the traced events. Such data will cause performance overheads.
+This also works without buffering by default, printing function events
+as they happen (uses trace_pipe), context switching and consuming CPU to do
+so. If needed, you can try the "\-d secs" option, which buffers events
+instead, reducing overhead. If you think the buffer option is losing events,
+try increasing the buffer size (buffer_size_kb).
+
+It's a good idea to use funccount(8) first, which is lower overhead, to
+help you select which functions you may want to trace using functrace(8).
+.SH SOURCE
+This is from the perf-tools collection:
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+funccount(8), kprobe(8)

data/perf-tools/man/man8/iolatency.8

@@ -0,0 +1,116 @@
+.TH iolatency 8  "2014-07-12" "USER COMMANDS"
+.SH NAME
+iolatency \- summarize block device I/O latency as a histogram. Uses Linux ftrace.
+.SH SYNOPSIS
+.B iolatency
+[\-hQT] [\-d device] [\-i iotype] [interval [count]]
+.SH DESCRIPTION
+This shows the distribution of latency, allowing modes and latency outliers
+to be identified and studied. For more details of block device I/O, use
+iosnoop(8).
+
+This is a proof of concept tool using ftrace, and involves user space
+processing and related overheads. See the OVERHEAD section.
+
+NOTE: Due to the way trace buffers are switched per interval, there is the
+possibility of losing a small number of I/O (usually less than 1%). The
+summary therefore shows the general distribution, but may be slightly
+incomplete. If 100% of I/O must be studied, use iosnoop(8) and post-process.
+Also note that I/O may be missed when the trace buffer is full: see the
+interval section in OPTIONS.
+
+Since this uses ftrace, only the root user can use this tool.
+.SH REQUIREMENTS
+FTRACE CONFIG, and the tracepoints block:block_rq_issue and
+block:block_rq_complete, which you may already have enabled and available on
+recent Linux kernels. And awk.
+.SH OPTIONS
+.TP
+\-d device
+Only show I/O issued by this device. (eg, "202,1"). This matches the DEV
+column in the iolatency output, and is filtered in-kernel.
+.TP
+\-i iotype
+Only show I/O issued that matches this I/O type. This matches the TYPE column
+in the iolatency output, and wildcards ("*") can be used at the beginning or
+end (only). Eg, "*R*" matches all reads. This is filtered in-kernel.
+.TP
+\-h
+Print usage message.
+.TP
+\-Q
+Include block I/O queueing time. This uses block I/O queue insertion as the
+start tracepoint (block:block_rq_insert), instead of block I/O issue
+(block:block_rq_issue).
+.TP
+\-T
+Include timestamps with each summary output.
+.TP
+interval
+Interval between summary histograms, in seconds.
+
+During the interval, trace output will be buffered in-kernel, which is then
+read and processed for the summary. This buffer has a fixed size per-CPU (see
+/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,
+try increasing that size (the bufsize_kb setting in iolatency). With the
+default setting (4 Mbytes), I'd expect this to happen around 50k I/O per
+summary.
+.TP
+count
+Number of summaries to print.
+.SH EXAMPLES
+.TP
+Default output, print a summary of block I/O latency every 1 second:
+# 
+.B iolatency
+.TP
+Include block I/O queue time:
+.B iolatency \-Q
+.TP
+Print 5 x 1 second summaries:
+#
+.B iolatency 1 5
+.TP
+Trace reads only:
+#
+.B iolatency \-i '*R*'
+.TP
+Trace I/O issued to device 202,1 only:
+#
+.B iolatency \-d 202,1
+.SH FIELDS
+.TP
+>=(ms)
+Latency was greater than or equal-to this value, in milliseconds.
+.TP
+<(ms)
+Latency was less than this value, in milliseconds.
+.TP
+I/O
+Number of block device I/O in this latency range, during the interval.
+.TP
+Distribution
+ASCII histogram representation of the I/O column.
+.SH OVERHEAD
+Block device I/O issue and completion events are traced and buffered
+in-kernel, then processed and summarized in user space. There may be
+measurable overhead with this approach, relative to the block device IOPS.
+
+The overhead may be acceptable in many situations. If it isn't, this tool
+can be reimplemented in C, or using a different tracer (eg, perf_events,
+SystemTap, ktap.)
+.SH SOURCE
+This is from the perf-tools collection.
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+iosnoop(8), iostat(1)

data/perf-tools/man/man8/iosnoop.8

@@ -0,0 +1,169 @@
+.TH iosnoop 8  "2014-07-12" "USER COMMANDS"
+.SH NAME
+iosnoop \- trace block I/O events as they occur. Uses Linux ftrace.
+.SH SYNOPSIS
+.B iosnoop
+[\-hQst] [\-d device] [\-i iotype] [\-p pid] [\-n name] [duration]
+.SH DESCRIPTION
+iosnoop prints block device I/O events as they happen, with useful details such
+as PID, device, I/O type, block number, I/O size, and latency.
+
+This traces disk I/O at the block device interface, using the block:
+tracepoints. This can help characterize the I/O requested for the storage
+devices and their resulting performance. I/O completions can also be studied
+event-by-event for debugging disk and controller I/O scheduling issues.
+
+NOTE: Use of a duration buffers I/O, which reduces overheads, but this also
+introduces a limit to the number of I/O that will be captured. See the duration
+section in OPTIONS.
+
+Since this uses ftrace, only the root user can use this tool.
+.SH REQUIREMENTS
+FTRACE CONFIG, and the tracepoints block:block_rq_insert, block:block_rq_issue,
+and block:block_rq_complete, which you may already have enabled and available on
+recent Linux kernels. And awk.
+.SH OPTIONS
+.TP
+\-d device
+Only show I/O issued by this device. (eg, "202,1"). This matches the DEV
+column in the iosnoop output, and is filtered in-kernel.
+.TP
+\-i iotype
+Only show I/O issued that matches this I/O type. This matches the TYPE column
+in the iosnoop output, and wildcards ("*") can be used at the beginning or
+end (only). Eg, "*R*" matches all reads. This is filtered in-kernel.
+.TP
+\-p PID
+Only show I/O issued by this PID. This filters in-kernel. Note that I/O may be
+issued indirectly; for example, as the result of a memory allocation, causing
+dirty buffers (maybe from another PID) to be written to storage.
+
+With the \-Q
+option, the identified PID is more accurate, however, LATms now includes
+queueing time (see the \-Q option).
+.TP
+\-n name
+Only show I/O issued by processes with this name. Partial strings and regular
+expressions are allowed. This is a post-filter, so all I/O is traced and then
+filtered in user space. As with PID, this includes indirectly issued I/O,
+and \-Q can be used to improve accuracy (see the \-Q option).
+.TP
+\-h
+Print usage message.
+.TP
+\-Q
+Use block I/O queue insertion as the start tracepoint (block:block_rq_insert),
+instead of block I/O issue (block:block_rq_issue). This makes the following
+changes: COMM and PID are more likely to identify the origin process, as are
+\-p PID and \-n name; STARTs shows queue insert; and LATms shows I/O
+time including time spent on the block I/O queue.
+.TP
+\-s
+Include a column for the start time (issue time) of the I/O, in seconds.
+If the \-Q option is used, this is the time the I/O is inserted on the block
+I/O queue.
+.TP
+\-t
+Include a column for the completion time of the I/O, in seconds.
+.TP
+duration
+Set the duration of tracing, in seconds. Trace output will be buffered and
+printed at the end. This also reduces overheads by buffering in-kernel,
+instead of printing events as they occur.
+
+The ftrace buffer has a fixed size per-CPU (see
+/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,
+try increasing that size (the bufsize_kb setting in iosnoop). With the
+default setting (4 Mbytes), I'd expect this to happen around 50k I/O.
+.SH EXAMPLES
+.TP
+Default output, print I/O activity as it occurs:
+# 
+.B iosnoop
+.TP
+Buffer for 5 seconds (lower overhead) and write to a file:
+# 
+.B iosnoop 5 > outfile
+.TP
+Trace based on block I/O queue insertion, showing queueing time:
+#
+.B iosnoop -Q
+.TP
+Trace reads only:
+#
+.B iosnoop \-i '*R*'
+.TP
+Trace I/O issued to device 202,1 only:
+#
+.B iosnoop \-d 202,1
+.TP
+Include I/O start and completion timestamps:
+#
+.B iosnoop \-ts
+.TP
+Include I/O queueing and completion timestamps:
+#
+.B iosnop \-Qts
+.TP
+Trace I/O issued when PID 181 was on-CPU only:
+#
+.B iosnoop \-p 181
+.TP
+Trace I/O queued when PID 181 was on-CPU (more accurate), and include queue time:
+#
+.B iosnoop \-Qp 181
+.SH FIELDS
+.TP
+COMM
+Process name (command) for the PID that was on-CPU when the I/O was issued, or
+inserted if \-Q is used. See PID. This column is truncated to 12 characters.
+.TP
+PID
+Process ID which was on-CPU when the I/O was issued, or inserted if \-Q is
+used. This will usually be the
+process directly requesting I/O, however, it may also include indirect I/O. For
+example, a memory allocation by this PID which causes dirty memory from another
+PID to be flushed to disk.
+.TP
+TYPE
+Type of I/O. R=read, W=write, M=metadata, S=sync, A=readahead, F=flush or FUA (force unit access), D=discard, E=secure, N=null (not RWFD).
+.TP
+DEV
+Storage device ID.
+.TP
+BLOCK
+Disk block for the operation (location, relative to this device).
+.TP
+BYTES
+Size of the I/O, in bytes.
+.TP
+LATms
+Latency (time) for the I/O, in milliseconds.
+.SH OVERHEAD
+By default, iosnoop works without buffering, printing I/O events
+as they happen (uses trace_pipe), context switching and consuming CPU to do
+so. This has a limit of about 10,000 IOPS (depending on your platform), at
+which point iosnoop will be consuming 1 CPU. The duration mode uses buffering,
+and can handle much higher IOPS rates, however, the buffer has a limit of
+about 50,000 I/O, after which events will be dropped. You can tune this with
+bufsize_kb, which is per-CPU. Also note that the "-n" option is currently
+post-filtered, so all events are traced.
+
+The overhead may be acceptable in many situations. If it isn't, this tool
+can be reimplemented in C, or using a different tracer (eg, perf_events,
+SystemTap, ktap.)
+.SH SOURCE
+This is from the perf-tools collection.
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+iolatency(8), iostat(1), lsblk(8)