fluent-plugin-perf-tools 0.1.0

data/perf-tools/man/man8/killsnoop.8

@@ -0,0 +1,100 @@
+.TH killsnoop 8  "2014-09-15" "USER COMMANDS"
+.SH NAME
+killsnoop \- trace kill() syscalls with process and signal details. Uses Linux ftrace.
+.SH SYNOPSIS
+.B killsnoop
+[\-hst] [\-d secs] [\-p pid] [\-n name]
+.SH DESCRIPTION
+This traces kill() syscalls, showing which process killed which pid and
+returns the returncode (0 for success, -1 for error).
+
+This implementation is designed to work on older kernel versions, and without
+kernel debuginfo. It works by dynamic tracing of the return value of kill()
+and associating it with the previous kill() syscall return.
+This approach is kernel version specific, and may not work on your version.
+It is a workaround, and proof of concept for ftrace, until more kernel tracing
+functionality is available.
+
+Since this uses ftrace, only the root user can use this tool.
+.SH REQUIREMENTS
+FTRACE and KPROBE CONFIG, syscalls:sys_enter_kill and
+syscalls:sys_exit_kill kernel tracepoints (you may already have these
+on recent kernels) and awk.
+.SH OPTIONS
+.TP
+\-d secs
+Set the duration of tracing, in seconds. Trace output will be buffered and
+printed at the end. This also reduces overheads by buffering in-kernel,
+instead of printing events as they occur.
+
+The ftrace buffer has a fixed size per-CPU (see
+/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,
+try increasing that size.
+.TP
+\-h
+Print usage message.
+.TP
+\-n name
+Only show processes matching this process name. Partial strings and regular
+expressions are allowed. This is post-filtered using awk.
+.TP
+\-p PID
+Only trace this process ID. This is filtered in-kernel.
+.TP
+\-s
+Use human readable signal names, instead of signal numbers.
+.TP
+\-t
+Include timestamps, in seconds.
+.SH EXAMPLES
+.TP
+Trace all kill() syscalls with details:
+#
+.B killsnoop
+.TP
+Trace kill() syscalls with readable signal names, and times:
+#
+.B killsnoop -st
+.TP
+Track kill() syscalls for processes named "httpd":
+#
+.B killsnoop -n httpd
+.SH FIELDS
+.TP
+TIMEs
+Time of open() completion, in units of seconds.
+.TP
+COMM
+Process name (if known) of the process that issued the signal.
+.TP
+PID
+Process ID that issued the signal.
+.TP
+TPID
+Target PID for the signal.
+.TP
+SIGNAL
+Signal number sent to the target process, or name if -s is used.
+.TP
+RETURN
+Return status: 0 for success, -1 for failure.
+.SH OVERHEAD
+This reads and kill() syscalls as they occur. For high rates of kills (> 500/s),
+the overhead may begin to be measurable, however, the rate is unlikely to get
+this high. And if it is: you should investigate why. Test yourself. You can
+also use the \-d mode to buffer output, reducing overheads.
+.SH SOURCE
+This is from the perf-tools collection.
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Martin Probst
+.SH SEE ALSO
+tpoint(8), execsnoop(8), opensnoop(8)

data/perf-tools/man/man8/kprobe.8

@@ -0,0 +1,162 @@
+.TH kprobe 8  "2014-07-20" "USER COMMANDS"
+.SH NAME
+kprobe \- trace a given kprobe definition. Kernel dynamic tracing. Uses Linux ftrace.
+.SH SYNOPSIS
+.B kprobe
+[\-FhHsv] [\-d secs] [\-p PID] [\-L TID] kprobe_definition [filter]
+.SH DESCRIPTION
+This will create, trace, then destroy a given kprobe definition. See
+Documentation/trace/kprobetrace.txt in the Linux kernel source for the
+syntax of a kprobe definition, and "kprobe -h" for examples. With this tool,
+the probe alias is optional (it will become to kprobe:<funcname> if not
+specified).
+
+WARNING: This uses dynamic tracing of kernel functions, and could cause
+kernel panics or freezes, depending on the function traced. Test in a lab
+environment, and know what you are doing, before use.
+
+Also beware of feedback loops: tracing tcp functions over an ssh session,
+or writing ext4 functions to an ext4 file system. For the former, tcp
+trace data could be redirected to a file (as in the usage message). For
+the latter, trace to the screen or a different file system.
+
+SEE ALSO: functrace(8), which can perform basic tracing (event only) of
+multiple kernel functions using wildcards.
+
+Since this uses ftrace, only the root user can use this tool.
+.SH REQUIREMENTS
+FTRACE and KPROBES CONFIG, which you may already have enabled and available on
+recent kernels.
+.SH OPTIONS
+.TP
+\-F
+Force. Trace despite warnings. By default the specified kernel function must
+exist in the available_filter_functions file. This option overrides this check.
+This might expose you to more unsafe functions, which could cause kernel
+panics or freezes when traced.
+.TP
+\-d seconds
+Set the duration of tracing, in seconds. Trace output will be buffered and
+printed at the end. This also reduces overheads by buffering in-kernel,
+instead of printing events as they occur.
+
+The ftrace buffer has a fixed size per-CPU (see
+/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,
+try increasing that size.
+.TP
+\-h
+Print usage message.
+.TP
+\-H
+Print column headers.
+.TP
+\-s
+Print kernel stack traces after each event.
+.TP
+\-v
+Show the kprobe format file only (do not trace), identifying possible variables
+for use in a custom filter.
+.TP
+\-p PID
+Only trace kernel functions when this process ID is on-CPU.
+.TP
+\-L TID
+Only trace kernel functions when this thread ID is on-CPU.
+.TP
+kprobe_definition
+A full kprobe definition, as documented by Documentation/trace/kprobetrace.txt
+in the Linux kernel source. Note that the probe alias name is optional with
+kprobe(8), and if not specified, the tracepoint will become kprobe:<funcname>.
+See the EXAMPLES section.
+.TP
+filter
+An ftrace filter definition.
+.SH EXAMPLES
+These examples may need modification to match your kernel version's function
+names and platform's register usage. If using platform specific registers
+becomes too painful in practice, consider a kernel debuginfo-based tracer,
+which can trace variables names instead. For example, perf_events.
+.TP
+Trace do_sys_open() entry:
+#
+.B kprobe p:do_sys_open
+.TP
+Trace do_sys_open() return:
+#
+.B kprobe r:do_sys_open
+.TP
+Trace do_sys_open() return value:
+#
+.B kprobe 'r:do_sys_open $retval'
+.TP
+Trace do_sys_open() return value, with a custom probe alias "myopen":
+#
+.B kprobe 'r:myopen do_sys_open $retval'
+.TP
+Trace do_sys_open() file mode:
+#
+.B kprobe 'p:myopen do_sys_open mode=%cx:u16'
+.TP
+Trace do_sys_open() file mode for PID 81:
+#
+.B kprobe -p 81 'p:myopen do_sys_open mode=%cx:u16'
+.TP
+Trace do_sys_open() with filename string:
+#
+.B kprobe 'p:myopen do_sys_open filename=+0(%si):string'
+.TP
+Trace do_sys_open() for filenames ending in "stat":
+#
+.B kprobe 'p:myopen do_sys_open fn=+0(%si):string' 'fn ~ """*stat"""'
+.TP
+Trace tcp_retransmit_skb() and show kernel stack traces, showing the path that led to it (can help explain why):
+#
+.B kprobe \-s 'p:myprobe tcp_retransmit_skb'
+.SH FIELDS
+The output format depends on the kernel version, and headings can be printed
+using \-H. The format is the same as the ftrace function trace format, described
+in the kernel source under Documentation/trace/ftrace.txt.
+
+Typical fields are:
+.TP
+TASK-PID
+The process name (which could include dashes), a dash, and the process ID.
+.TP
+CPU#
+The CPU ID, in brackets.
+.TP
+||||
+Kernel state flags. For example, on Linux 3.16 these are for irqs-off,
+need-resched, hardirq/softirq, and preempt-depth.
+.TP
+TIMESTAMP
+Time of event, in seconds.
+.TP
+FUNCTION
+Kernel function name.
+.SH OVERHEAD
+This can generate a lot of trace data quickly, depending on the
+frequency of the traced events. Such data will cause performance overheads.
+This also works without buffering by default, printing function events
+as they happen (uses trace_pipe), context switching and consuming CPU to do
+so. If needed, you can try the "\-d secs" option, which buffers events
+instead, reducing overhead. If you think the buffer option is losing events,
+try increasing the buffer size (buffer_size_kb).
+
+It's a good idea to use funccount(8) first, which is lower overhead, to
+help you select which functions you may want to trace using kprobe(8).
+.SH SOURCE
+This is from the perf-tools collection:
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+functrace(8), funccount(8)

data/perf-tools/man/man8/opensnoop.8

@@ -0,0 +1,113 @@
+.TH opensnoop 8  "2014-07-20" "USER COMMANDS"
+.SH NAME
+opensnoop \- trace open() syscalls with file details. Uses Linux ftrace.
+.SH SYNOPSIS
+.B opensnoop
+[\-htx] [\-d secs] [\-p pid] [\-L tid] [\-n name] [filename]
+.SH DESCRIPTION
+This traces open() syscalls, showing the file name (pathname) and returned file
+descriptor number (or \-1, for error).
+
+This implementation is designed to work on older kernel versions, and without
+kernel debuginfo. It works by dynamic tracing of the return value of getname()
+as a string, and associating it with the following open() syscall return.
+This approach is kernel version specific, and may not work on your version.
+It is a workaround, and proof of concept for ftrace, until more kernel tracing
+functionality is available.
+
+Since this uses ftrace, only the root user can use this tool.
+.SH REQUIREMENTS
+FTRACE and KPROBE CONFIG, the syscalls:sys_exit_open tracepoint, and the
+getname() kernel function. You may already have these enabled and available
+on recent Linux kernels. And awk.
+.SH OPTIONS
+.TP
+\-d secs
+Set the duration of tracing, in seconds. Trace output will be buffered and
+printed at the end. This also reduces overheads by buffering in-kernel,
+instead of printing events as they occur.
+
+The ftrace buffer has a fixed size per-CPU (see
+/sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing,
+try increasing that size.
+.TP
+\-h
+Print usage message.
+.TP
+\-n name
+Only show processes matching this process name. Partial strings and regular
+expressions are allowed. This is post-filtered using awk.
+.TP
+\-p PID
+Only trace this process ID. This is filtered in-kernel.
+.TP
+\-L TID
+Only trace this thread ID. This is filtered in-kernel.
+.TP
+\-t
+Include timestamps, in seconds.
+.TP
+\-x
+Only print failed open()s.
+.TP
+filename
+Only show open()s which match this filename. Partial strings and regular
+expressions are allowed. This is post-filtered using awk.
+.SH EXAMPLES
+.TP
+Trace all open() syscalls with details:
+#
+.B opensnoop
+.TP
+Only trace open()s for PID 81:
+#
+.B opensnoop -p 81
+.TP
+Trace failed open() syscalls:
+#
+.B opensnoop -x
+.TP
+Trace open() syscalls for filenames containing "conf":
+#
+.B opensnoop conf
+.TP
+Trace open() syscalls for filenames ending in "log":
+#
+.B opensnoop 'log$'
+.SH FIELDS
+.TP
+TIMEs
+Time of open() completion, in units of seconds.
+.TP
+COMM
+Process name (if known).
+.TP
+PID
+Process ID.
+.TP
+FD
+File descriptor. If this is a successful open, the file descriptor number is
+shown. If this is unsuccessful, -1 is shown. Numbers beginning with 0x are
+hexadecimal.
+.TP
+FILE
+Filename (pathname) used by the open() syscall.
+.SH OVERHEAD
+This reads and open() syscalls and getname() kernel functions as they occur.
+For high rates of opens (> 500/s), the overhead may begin to be measurable.
+Test yourself. You can use the \-d mode to buffer output, reducing overheads.
+.SH SOURCE
+This is from the perf-tools collection.
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+execsnoop(8), strace(1)

data/perf-tools/man/man8/perf-stat-hist.8

@@ -0,0 +1,111 @@
+.TH perf-stat-hist 8  "2014-07-07" "USER COMMANDS"
+.SH NAME
+perf-stat-hist \- histogram summary of tracepoint values. Uses Linux perf_events.
+.SH SYNOPSIS
+.B perf-stat-hist
+[-h] [-b buckets|-P power] [-m max] tracepoint variable [seconds]
+.SH DESCRIPTION
+This is a proof-of-concept showing in-kernel histograms using Linux perf_events
+(aka the "perf" command), on older kernels where perf_events does not have
+this native capability.
+
+These histograms show the distribution of variable, allowing details
+including multiple modes and outliers to be studied.
+
+This uses multiple counting tracepoints with different filters, one for each
+histogram bucket. While this is summarized in-kernel, the use of multiple
+tracepoints does add addiitonal overhead. Hopefully, in the
+future this this functionality will be provided in an efficient way from
+perf_events itself, at which point this tool can be deleted or rewritten.
+.SH REQUIREMENTS
+Linux perf_events: add linux-tools-common, run "perf", then add any additional
+packages it requests. Also uses awk.
+.SH OPTIONS
+.TP
+\-h
+Usage message.
+.TP
+\-b buckets
+Specify a list of bucket points for the histogram as a string (eg, "10 500
+1000"). The histogram will include buckets for less-than the minimum, and
+greater-than-or-equal-to the maximum.  If a single value is specified, two
+statistics only are gathered: for less-than and for greater-than-or-equal-to.
+The overhead is relative to the number of buckets, so only specifying a
+single value costs the lowest overhead.
+.TP
+\-P power
+Power for power-of histogram. By default, a power-of-4 histogram is created.
+This and the \-b option are exclusive.
+.TP
+\-m max
+Max value for power-of histograms.
+.TP
+tracepoint
+Tracepoint specification. Eg, syscalls:sys_enter_read.
+.TP
+variable
+The tracepoint variable name to summarize. To see what are available, cat the
+format file under /sys/kernel/debug/tracing/events/*/*/format.
+.TP
+seconds
+Number of seconds to trace. If not specified, this runs until Ctrl-C.
+.SH EXAMPLES
+.TP
+Trace read() syscalls until Ctrl-C, and show histogram of requested size:
+#
+.B perf\-stat\-hist syscalls:sys_enter_read count
+.TP
+Trace read() syscall completions until Ctrl-C, and show histogram of successful returned size:
+#
+.B perf\-stat\-hist syscalls:sys_exit_read ret
+.TP
+Trace read() return sizes for 10 seconds, showing histogram:
+#
+.B perf\-stat\-hist syscalls:sys_exit_read ret 10
+.TP
+Trace network transmits until Ctrl-C, and show histogram of packet size:
+#
+.B perf\-stat\-hist net:net_dev_xmit len
+.TP
+Trace read() return sizes, using a power-of-10 histogram:
+.B perf\-stat\-hist \-P 10 syscalls:sys_exit_read ret
+.TP
+Trace read() return sizes, using a power-of-2 histogram, and a max of 1024:
+.B perf\-stat\-hist \-P 2 \-m 1024 syscalls:sys_exit_read ret
+.TP
+Trace read() return sizes, using the specified bucket points:
+.B perf\-stat\-hist \-b """10 50 100 5000""" syscalls:sys_exit_read ret
+.TP
+Trace read() return sizes, and bifurcate statistics by the value 10:
+.B perf-stat-hist \-b 10 syscalls:sys_exit_read ret
+.SH FIELDS
+.TP
+Range
+Range of the histogram bucket, in units of the variable specified.
+.TP
+Count
+Number of occurrences (tracepoint events) of the variable in this range.
+.TP
+Distribution
+ASCII histogram representation of the Count column.
+.SH OVERHEAD
+While the counts are performed in-kernel, there is one tracepoint used per
+histogram bucket, so the overheads are higher than usual (relative to the
+number of buckets) than function counting using perf stat. The lowest
+overhead is when \-b is used to specify one bucket only, bifurcating
+statistics.
+.SH SOURCE
+This is from the perf-tools collection.
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+perf(1)

data/perf-tools/man/man8/reset-ftrace.8

@@ -0,0 +1,49 @@
+.TH reset-ftrace 8  "2014-07-07" "USER COMMANDS"
+.SH NAME
+reset-ftrace \- reset state of ftrace, disabling all tracing. Written for Linux ftrace.
+.SH SYNOPSIS
+.B reset-ftrace
+[\-fhq]
+.SH DESCRIPTION
+This resets the state of various ftrace files, and shows the before and after
+state.
+
+This may only be of use to ftrace hackers who, in the process of developing
+ftrace software, often get the subsystem into a partially active state, and
+would like a quick way to reset state. Check the end of this script for the
+actually files reset, and add more if you need.
+
+WARNING: Only use this if and when you are sure that there are no other active
+ftrace sessions on your system, as otherwise it will kill them.
+.SH REQUIREMENTS
+FTRACE CONFIG.
+.SH OPTIONS
+.TP
+\-f
+Force. If the ftrace lock file exists (/var/tmp/.ftrace-lock), delete it.
+.TP
+\-h
+Print usage message.
+.TP
+\-q
+Quiet. Run, but don't print any output.
+.SH EXAMPLES
+.TP
+Reset various ftrace files:
+#
+.B reset-ftrace
+.SH SOURCE
+This is from the perf-tools collection.
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+perf(1)

data/perf-tools/man/man8/syscount.8

@@ -0,0 +1,96 @@
+.TH syscount 8  "2014-07-07" "USER COMMANDS"
+.SH NAME
+syscount \- count system calls. Uses Linux perf_events.
+.SH SYNOPSIS
+.B syscount
+[\-chv] [\-t top] {\-p PID|\-d seconds|command}
+.SH DESCRIPTION
+This is a proof-of-concept using perf_events capabilities for older kernel
+versions, that lack custom in-kernel aggregations. Once they exist, this
+script can be substantially rewritten and improved (lower overhead).
+.SH REQUIREMENTS
+Linux perf_events: add linux-tools-common, run "perf", then
+add any additional packages it requests. Also needs awk.
+.SH OPTIONS
+.TP
+\-c
+Show counts by syscall name. This mode (without -v) uses in-kernel counts, which
+have lower overhead than the default mode.
+.TP
+\-h
+Usage message.
+.TP
+\-v
+Verbose: include PID.
+.TP
+\-p PID
+Trace this process ID only.
+.TP
+\-d seconds
+Duration of trace in seconds.
+.TP
+command
+Run and trace this command.
+.SH EXAMPLES
+.TP
+Trace and summarize syscalls by process name:
+#
+.B syscount
+.TP
+Trace and summarize syscalls by syscall name (lower overhead):
+#
+.B syscount \-c
+.TP
+Trace for 5 seconds, showing by process name:
+#
+.B syscount \-d 5
+.TP
+Trace PID 932 only, and show by syscall name (lower overhead):
+#
+.B syscount \-cp 923
+.TP
+Execute the """ls""" command, and show by syscall name:
+#
+.B syscount -c ls
+.SH FIELDS
+.TP
+PID
+Process ID.
+.TP
+COMM
+Process command name.
+.TP
+SYSCALL
+Syscall name.
+.TP
+COUNT
+Number of syscalls during tracing.
+.SH OVERHEAD
+Modes that report syscall names only (\-c, \-cp PID, \-cd secs) have
+lower overhead, since they use in-kernel counts. Other modes which report
+process IDs (\-cv) or process names (default) create a perf.data file for
+post processing, and you will see messages about it doing this. Beware of
+the file size (test for short durations, or use \-c to see counts based on
+in-kernel counters), and gauge overheads based on the perf.data size.
+
+Note that this script delibrately does not pipe perf record into
+perf script, which would avoid perf.data, because it can create a feedback
+loop where the perf script syscalls are recorded. Hopefully there will be a
+fix for this in a later perf version, so perf.data can be skipped, or other
+kernel features to aggregate by process name in-kernel directly (eg, via
+eBPF, ktap, or SystemTap).
+.SH SOURCE
+This is from the perf-tools collection.
+.IP
+https://github.com/brendangregg/perf-tools
+.PP
+Also look under the examples directory for a text file containing example
+usage, output, and commentary for this tool.
+.SH OS
+Linux
+.SH STABILITY
+Unstable - in development.
+.SH AUTHOR
+Brendan Gregg
+.SH SEE ALSO
+iosnoop(8), iolatency(8), iostat(1)