fluent-plugin-perf-tools 0.1.0

data/perf-tools/examples/reset-ftrace_example.txt

@@ -0,0 +1,88 @@
+Demonstrations of reset-ftrace, the Linux ftrace tool.
+
+
+You will probably never need this tool. If you kill -9 an ftrace-based tool,
+leaving the kernel in a tracing enabled state, you could try using this tool
+to reset ftrace and disable tracing. Make sure no other ftrace sessions are
+in use on your system, or it will kill those.
+
+Here's an example:
+
+# ./opensnoop
+Tracing open()s. Ctrl-C to end.
+ERROR: ftrace may be in use by PID 2197 /var/tmp/.ftrace-lock
+
+I tried to run opensnoop, but there's a lock file for PID 2197. Checking if it
+exists:
+
+# ps -fp 2197
+UID        PID  PPID  C STIME TTY          TIME CMD
+#
+
+No.
+
+I also know that no one is using ftrace on this system. So I'll use reset-ftrace
+to clean up this lock file and ftrace state:
+
+# ./reset-ftrace 
+ERROR: ftrace lock (/var/tmp/.ftrace-lock) exists. It shows ftrace may be in use by PID 2197.
+Double check to see if that PID is still active. If not, consider using -f to force a reset. Exiting.
+
+... except it's complaining about the lock file too. I'm already sure that this
+PID doesn't exist, so I'll add the -f option:
+
+# ./reset-ftrace -f
+Reseting ftrace state...
+
+current_tracer, before:
+     1	nop
+current_tracer, after:
+     1	nop
+
+set_ftrace_filter, before:
+     1	#### all functions enabled ####
+set_ftrace_filter, after:
+     1	#### all functions enabled ####
+
+set_ftrace_pid, before:
+     1	no pid
+set_ftrace_pid, after:
+     1	no pid
+
+kprobe_events, before:
+kprobe_events, after:
+
+Done.
+
+The output shows what has been reset, including the before and after state of
+these files.
+
+Now I can try iosnoop again:
+
+# ./iosnoop
+Tracing block I/O. Ctrl-C to end.
+COMM             PID    TYPE DEV      BLOCK        BYTES     LATms
+supervise        1689   W    202,1    17039664     4096       0.58
+supervise        1689   W    202,1    17039672     4096       0.47
+supervise        1694   W    202,1    17039744     4096       0.98
+supervise        1694   W    202,1    17039752     4096       0.74
+supervise        1684   W    202,1    17039760     4096       0.63
+[...]
+
+Fixed.
+
+Note that reset-ftrace currently only resets a few methods of enabling
+tracing, such as set_ftrace_filter and kprobe_events. Static tracepoints could 
+be enabled individually, and this script currently doesn't find and disable
+those.
+
+
+Use -h to print the USAGE message:
+
+# ./reset-ftrace -h
+USAGE: reset-ftrace [-fhq]
+                 -f              # force: delete ftrace lock file
+                 -q              # quiet: reset, but say nothing
+                 -h              # this usage message
+  eg,
+       reset-ftrace              # disable active ftrace session

data/perf-tools/examples/syscount_example.txt

@@ -0,0 +1,297 @@
+Demonstrations of syscount, the Linux perf_events version.
+
+
+The first mode I use is "-c", where it behaves like "strace -c", but for the
+entire system (all procesess) and with much lower overhead:
+
+# ./syscount -c
+Tracing... Ctrl-C to end.
+^Csleep: Interrupt
+SYSCALL              COUNT
+accept                   1
+getsockopt               1
+setsid                   1
+chdir                    2
+getcwd                   2
+getpeername              2
+getsockname              2
+setgid                   2
+setgroups                2
+setpgid                  2
+setuid                   2
+getpgrp                  4
+getpid                   4
+rename                   4
+setitimer                4
+setrlimit                4
+setsockopt               4
+statfs                   4
+set_tid_address          5
+readlink                 6
+set_robust_list          6
+nanosleep                7
+newuname                 7
+faccessat                8
+futex                   10
+clock_gettime           16
+newlstat                20
+pipe                    20
+epoll_wait              24
+getrlimit               25
+socket                  27
+connect                 29
+exit_group              30
+getppid                 31
+dup2                    34
+wait4                   51
+fcntl                   58
+getegid                 72
+getgid                  72
+getuid                  72
+geteuid                 75
+perf_event_open        100
+munmap                 121
+gettimeofday           216
+access                 266
+ioctl                  340
+poll                   348
+sendto                 374
+mprotect               414
+brk                    597
+rt_sigaction           632
+recvfrom               664
+lseek                  749
+newfstatat            2922
+openat                2925
+newfstat              3229
+newstat               4334
+open                  4534
+fchdir                5845
+getdents              5854
+read                  7673
+close                 7728
+select                9633
+rt_sigprocmask       19886
+write                34581
+
+While tracing, the write() syscall was executed 34,581 times.
+
+This mode uses "perf stat" to count the syscalls:* tracepoints in-kernel.
+
+
+You can add a duration (-d) and limit the number shown (-t):
+
+# ./syscount -cd 5 -t 10
+Tracing for 5 seconds. Top 10 only...
+SYSCALL              COUNT
+gettimeofday          1009
+write                 3583
+read                  8174
+openat               21550
+newfstat             21558
+open                 21824
+fchdir               43098
+getdents             43106
+close                43694
+newfstatat          110936
+
+While tracing for 5 seconds, the newfstatat() syscall was executed 110,936
+times.
+
+
+Without the -c, syscount shows syscalls by process name:
+
+# ./syscount -d 5 -t 10
+Tracing for 5 seconds. Top 10 only...
+[ perf record: Woken up 66 times to write data ]
+[ perf record: Captured and wrote 16.513 MB perf.data (~721455 samples) ]
+COMM                COUNT
+stat                  450
+perl                  537
+catalina.sh          1700
+postgres             2094
+run                  2362
+:6946                4764
+ps                   5961
+sshd                45796
+find                61039
+
+So processes named "find" called 61,039 syscalls during the 5 seconds of
+tracing.
+
+Note that this mode writes a perf.data file. This is higher overhead for a
+few reasons:
+
+- all data is passed from kernel to user space, which eats CPU for the memory
+  copy. Note that it is buffered in an efficient way by perf_events, which
+  wakes up and context switches only a small number of times: 66 in this case,
+  to hand 16 Mbytes of trace data to user space.
+- data is post-processed in user space, eating more CPU.
+- data is stored on the file system in the perf.data file, consuming available
+  storage.
+
+This will be improved in future kernels, but it is difficult to improve this
+much further in existing kernels. For example, using a pipe to "perf script"
+instead of writing perf.data can have issues with feedback loops, where
+perf traces itself. This syscount version goes to lengths to avoid tracing
+its own perf, but 
+right now with existing functionality in older kernels. The trip via perf.data
+is necessary
+
+
+Running without options shows syscalls by process name until Ctrl-C:
+
+# ./syscount 
+Tracing... Ctrl-C to end.
+^C[ perf record: Woken up 39 times to write data ]
+[ perf record: Captured and wrote 9.644 MB perf.data (~421335 samples) ]
+COMM                COUNT
+apache2                 8
+apacheLogParser        13
+platformservice        16
+snmpd                  16
+ntpd                   21
+multilog               66
+supervise              84
+dirname               102
+echo                  102
+svstat                108
+cut                   111
+bash                  113
+grep                  132
+xargs                 132
+redis-server          190
+sed                   192
+setuidgid             294
+stat                  450
+perl                  537
+catalina.sh          1275
+postgres             1736
+run                  2352
+:7396                4527
+ps                   5925
+sshd                20154
+find                28700
+
+Note again it is writing a perf.data file to do this.
+
+
+The -v option adds process IDs:
+
+# ./syscount -v
+Tracing... Ctrl-C to end.
+^C[ perf record: Woken up 48 times to write data ]
+[ perf record: Captured and wrote 12.114 MB perf.data (~529276 samples) ]
+PID    COMM                COUNT
+3599   apacheLogParser         3
+7977   xargs                   3
+7982   supervise               3
+7993   xargs                   3
+3575   apache2                 4
+1311   ntpd                    6
+3135   postgres                6
+3600   apacheLogParser         6
+3210   platformservice         8
+6503   sshd                    9
+7978   :7978                   9
+7994   run                     9
+7968   :7968                  11
+7984   run                    11
+1451   snmpd                  16
+3040   svscan                 17
+3066   postgres               17
+3133   postgres               24
+3134   postgres               24
+3136   postgres               24
+3061   multilog               29
+3055   supervise              30
+7979   bash                   31
+7977   echo                   34
+7981   dirname                34
+7993   echo                   34
+7968   svstat                 36
+7984   svstat                 36
+7975   cut                    37
+7991   cut                    37
+9857   bash                   37
+7967   :7967                  40
+7983   run                    40
+7972   :7972                  41
+7976   xargs                  41
+7988   run                    41
+7992   xargs                  41
+7969   :7969                  42
+7976   :7976                  42
+7985   run                    42
+7992   run                    42
+7973   :7973                  43
+7974   :7974                  43
+7989   run                    43
+7990   run                    43
+7973   grep                   44
+7989   grep                   44
+7975   :7975                  45
+7991   run                    45
+7970   :7970                  51
+7986   run                    51
+7981   catalina.sh            52
+7974   sed                    64
+7990   sed                    64
+3455   postgres               66
+7971   :7971                  66
+7987   run                    66
+7966   :7966                  96
+7966   setuidgid              98
+3064   redis-server          110
+7970   stat                  150
+7986   stat                  150
+7969   perl                  179
+7985   perl                  179
+7982   run                   341
+7966   catalina.sh           373
+7980   postgres              432
+7972   ps                   1971
+7988   ps                   1983
+9832   sshd                37511
+7979   find                51040
+
+Once you've found a process ID of interest, you can use "-c" and "-p PID" to
+show syscall names. This also switches to "perf stat" mode for in-kernel
+counts, and lower overhead:
+
+# ./syscount -cp 7979
+Tracing PID 7979... Ctrl-C to end.
+^CSYSCALL              COUNT
+brk                     10
+newfstat              2171
+open                  2171
+newfstatat            2175
+openat                2175
+close                 4346
+fchdir                4346
+getdents              4351
+write                25482
+
+So the most frequent syscall by PID 7979 was write().
+
+
+Use -h to print the USAGE message:
+
+# ./syscount -h
+USAGE: syscount [-chv] [-t top] {-p PID|-d seconds|command}
+       syscount                  # count by process name
+                -c               # show counts by syscall name
+                -h               # this usage message
+                -v               # verbose: shows PID
+                -p PID           # trace this PID only
+                -d seconds       # duration of trace
+                -t num           # show top number only
+                command          # run and trace this command
+  eg,
+        syscount                 # syscalls by process name
+        syscount -c              # syscalls by syscall name
+        syscount -d 5            # trace for 5 seconds
+        syscount -cp 923         # syscall names for PID 923
+        syscount -c ls           # syscall names for "ls"
+
+See the man page and example file for more info.

data/perf-tools/examples/tcpretrans_example.txt

@@ -0,0 +1,93 @@
+Demonstrations of tcpretrans, the Linux ftrace version.
+
+
+Tracing TCP retransmits on a busy server:
+
+# ./tcpretrans 
+TIME     PID    LADDR:LPORT          -- RADDR:RPORT          STATE
+05:16:44 3375   10.150.18.225:53874  R> 10.105.152.3:6001    ESTABLISHED
+05:16:44 3375   10.150.18.225:53874  R> 10.105.152.3:6001    ESTABLISHED
+05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED
+05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED
+05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED
+05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED
+05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED
+05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED
+05:16:54 4028   10.150.18.225:6002   R> 10.150.30.249:1710   ESTABLISHED
+05:16:55 0      10.150.18.225:47115  R> 10.71.171.158:6001   ESTABLISHED
+05:16:58 0      10.150.18.225:44388  R> 10.103.130.120:6001  ESTABLISHED
+05:16:58 0      10.150.18.225:44388  R> 10.103.130.120:6001  ESTABLISHED
+05:16:58 0      10.150.18.225:44388  R> 10.103.130.120:6001  ESTABLISHED
+05:16:59 0      10.150.18.225:56086  R> 10.150.32.107:6001   ESTABLISHED
+05:16:59 0      10.150.18.225:56086  R> 10.150.32.107:6001   ESTABLISHED
+^C
+Ending tracing...
+
+This shows TCP retransmits by dynamically tracing the kernel function that does
+the retransmit. This is a low overhead approach.
+
+The PID may or may not make sense: it's showing the PID that was on-CPU,
+however, retransmits are often timer-based, where it's the kernel that is
+on-CPU.
+
+The STATE column shows the TCP state for the socket performing the retransmit.
+The "--" column is the packet type. "R>" for retransmit.
+
+
+Kernel stack traces can be included with -s, which may show the type of
+retransmit:
+
+# ./tcpretrans -s
+TIME     PID    LADDR:LPORT          -- RADDR:RPORT          STATE
+06:21:10 19516  10.144.107.151:22    R> 10.13.106.251:32167  ESTABLISHED
+ => tcp_fastretrans_alert
+ => tcp_ack
+ => tcp_rcv_established
+ => tcp_v4_do_rcv
+ => tcp_v4_rcv
+ => ip_local_deliver_finish
+ => ip_local_deliver
+ => ip_rcv_finish
+ => ip_rcv
+ => __netif_receive_skb
+ => netif_receive_skb
+ => handle_incoming_queue
+ => xennet_poll
+ => net_rx_action
+ => __do_softirq
+ => call_softirq
+ => do_softirq
+ => irq_exit
+ => xen_evtchn_do_upcall
+ => xen_do_hypervisor_callback
+
+This looks like a fast retransmit (inclusion of tcp_fastretrans_alert(), and
+being based on receiving an ACK, rather than a timer).
+
+
+The -l option will include TCP tail loss probe events (TLP; see
+http://lwn.net/Articles/542642/). Eg:
+
+# ./tcpretrans -l
+TIME     PID    LADDR:LPORT          -- RADDR:RPORT          STATE       
+21:56:06 0      10.100.155.200:22    R> 10.10.237.72:18554   LAST_ACK    
+21:56:08 0      10.100.155.200:22    R> 10.10.237.72:18554   LAST_ACK    
+21:56:10 16452  10.100.155.200:22    R> 10.10.237.72:18554   LAST_ACK    
+21:56:10 0      10.100.155.200:22    L> 10.10.237.72:46408   LAST_ACK    
+21:56:10 0      10.100.155.200:22    R> 10.10.237.72:46408   LAST_ACK    
+21:56:12 0      10.100.155.200:22    R> 10.10.237.72:46408   LAST_ACK    
+21:56:13 0      10.100.155.200:22    R> 10.10.237.72:46408   LAST_ACK    
+^C
+Ending tracing...
+
+Look for "L>" in the type column ("--") for TLP events.
+
+
+Use -h to print the USAGE message:
+
+# ./tcpretrans -h
+USAGE: tcpretrans [-hs]
+                  -h       # help message
+                  -s       # print stack traces
+   eg,
+       tcpretrans          # trace TCP retransmits

data/perf-tools/examples/tpoint_example.txt

@@ -0,0 +1,210 @@
+Demonstrations of tpoint, the Linux ftrace version.
+
+
+Let's trace block:block_rq_issue, to see block device (disk) I/O requests:
+
+# ./tpoint block:block_rq_issue 
+Tracing block:block_rq_issue. Ctrl-C to end.
+       supervise-1692  [001] d... 7269912.982162: block_rq_issue: 202,1 W 0 () 17039656 + 8 [supervise]
+       supervise-1696  [000] d... 7269912.982243: block_rq_issue: 202,1 W 0 () 12862264 + 8 [supervise]
+           cksum-12994 [000] d... 7269913.317924: block_rq_issue: 202,1 R 0 () 9357056 + 72 [cksum]
+           cksum-12994 [000] d... 7269913.319013: block_rq_issue: 202,1 R 0 () 2977536 + 144 [cksum]
+           cksum-12994 [000] d... 7269913.320217: block_rq_issue: 202,1 R 0 () 2986240 + 216 [cksum]
+           cksum-12994 [000] d... 7269913.321677: block_rq_issue: 202,1 R 0 () 620344 + 56 [cksum]
+           cksum-12994 [001] d... 7269913.329309: block_rq_issue: 202,1 R 0 () 9107912 + 88 [cksum]
+           cksum-12994 [001] d... 7269913.340133: block_rq_issue: 202,1 R 0 () 3147008 + 248 [cksum]
+           cksum-12994 [001] d... 7269913.354551: block_rq_issue: 202,1 R 0 () 11583488 + 256 [cksum]
+           cksum-12994 [001] d... 7269913.379904: block_rq_issue: 202,1 R 0 () 11583744 + 256 [cksum]
+[...]
+^C
+Ending tracing...
+
+Great, that was easy!
+
+perf_events can do this as well, and is better in many ways, including a more
+efficient buffering strategy, and multi-user access. It's not that easy to do
+this one-liner in perf_events, however. An equivalent for recent kernels is:
+
+perf record --no-buffer -e block:block_rq_issue -a -o - | PAGER=cat stdbuf -oL perf script -i -
+
+Older kernels, use -D instead of --no-buffer. Even better is to set the buffer
+page size to a sufficient grouping (using -m), to minimize overheads, at the
+expense of liveliness of updates. Note that stack traces (-g) don't work on
+my systems with this perf one-liner, however, they do work with tpoint -s.
+
+
+Column headings can be printed using -H:
+
+# ./tpoint -H block:block_rq_issue 
+Tracing block:block_rq_issue. Ctrl-C to end.
+# tracer: nop
+#
+# entries-in-buffer/entries-written: 0/0   #P:2
+#
+#                              _-----=> irqs-off
+#                             / _----=> need-resched
+#                            | / _---=> hardirq/softirq
+#                            || / _--=> preempt-depth
+#                            ||| /     delay
+#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
+#              | |       |   ||||       |         |
+       supervise-1697  [000] d... 7270545.340856: block_rq_issue: 202,1 W 0 () 12862464 + 8 [supervise]
+       supervise-1697  [000] d... 7270545.341256: block_rq_issue: 202,1 W 0 () 12862472 + 8 [supervise]
+       supervise-1690  [000] d... 7270545.342363: block_rq_issue: 202,1 W 0 () 17040368 + 8 [supervise]
+[...]
+
+They are also documented in the Linux kernel source under:
+Documentation/trace/ftrace.txt.
+
+
+How about stacks traces for those block_rq_issue events? Adding -s:
+
+# ./tpoint -s block:block_rq_issue
+Tracing block:block_rq_issue. Ctrl-C to end.
+       supervise-1691  [000] d... 7269511.079179: block_rq_issue: 202,1 W 0 () 17040232 + 8 [supervise]
+       supervise-1691  [000] d... 7269511.079188: <stack trace>
+ => blk_peek_request
+ => do_blkif_request
+ => __blk_run_queue
+ => queue_unplugged
+ => blk_flush_plug_list
+ => blk_finish_plug
+ => ext4_writepages
+ => do_writepages
+ => __filemap_fdatawrite_range
+ => filemap_flush
+ => ext4_alloc_da_blocks
+ => ext4_rename
+ => vfs_rename
+ => SYSC_renameat2
+ => SyS_renameat2
+ => SyS_rename
+ => system_call_fastpath
+           cksum-7428  [000] d... 7269511.331778: block_rq_issue: 202,1 R 0 () 9006848 + 208 [cksum]
+           cksum-7428  [000] d... 7269511.331784: <stack trace>
+ => blk_peek_request
+ => do_blkif_request
+ => __blk_run_queue
+ => queue_unplugged
+ => blk_flush_plug_list
+ => blk_finish_plug
+ => __do_page_cache_readahead
+ => ondemand_readahead
+ => page_cache_async_readahead
+ => generic_file_read_iter
+ => new_sync_read
+ => vfs_read
+ => SyS_read
+ => system_call_fastpath
+           cksum-7428  [000] d... 7269511.332631: block_rq_issue: 202,1 R 0 () 620992 + 200 [cksum]
+           cksum-7428  [000] d... 7269511.332639: <stack trace>
+ => blk_peek_request
+ => do_blkif_request
+ => __blk_run_queue
+ => queue_unplugged
+ => blk_flush_plug_list
+ => blk_finish_plug
+ => __do_page_cache_readahead
+ => ondemand_readahead
+ => page_cache_sync_readahead
+ => generic_file_read_iter
+ => new_sync_read
+ => vfs_read
+ => SyS_read
+ => system_call_fastpath
+^C
+Ending tracing...
+
+Easy. Now I can read the ancestry to understand what actually lead to issuing
+a block device (disk) I/O.
+
+
+Here's insertion onto the block I/O queue (better matches processes):
+
+# ./tpoint -s block:block_rq_insert
+Tracing block:block_rq_insert. Ctrl-C to end.
+           cksum-11908 [000] d... 7269834.882517: block_rq_insert: 202,1 R 0 () 736304 + 256 [cksum]
+           cksum-11908 [000] d... 7269834.882528: <stack trace>
+ => __elv_add_request
+ => blk_flush_plug_list
+ => blk_finish_plug
+ => __do_page_cache_readahead
+ => ondemand_readahead
+ => page_cache_sync_readahead
+ => generic_file_read_iter
+ => new_sync_read
+ => vfs_read
+ => SyS_read
+ => system_call_fastpath
+[...]
+
+
+You can also add tracepoint filters. To see what variables you can use, use -v:
+
+# ./tpoint -v block:block_rq_issue 
+name: block_rq_issue
+ID: 942
+format:
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:dev_t dev;	offset:8;	size:4;	signed:0;
+	field:sector_t sector;	offset:16;	size:8;	signed:0;
+	field:unsigned int nr_sector;	offset:24;	size:4;	signed:0;
+	field:unsigned int bytes;	offset:28;	size:4;	signed:0;
+	field:char rwbs[8];	offset:32;	size:8;	signed:1;
+	field:char comm[16];	offset:40;	size:16;	signed:1;
+	field:__data_loc char[] cmd;	offset:56;	size:4;	signed:1;
+
+print fmt: "%d,%d %s %u (%s) %llu + %u [%s]", ((unsigned int) ((REC->dev) >> 20)), ((unsigned int) ((REC->dev) & ((1U << 20) - 1))), REC->rwbs, REC->bytes, __get_str(cmd), (unsigned long long)REC->sector, REC->nr_sector, REC->comm
+
+
+Now I'll add a filter to check that the rwbs field (I/O type) includes an "R",
+making it a read:
+
+# ./tpoint -s block:block_rq_insert 'rwbs ~ "*R*"'
+           cksum-11908 [000] d... 7269839.919098: block_rq_insert: 202,1 R 0 () 736560 + 136 [cksum]
+           cksum-11908 [000] d... 7269839.919107: <stack trace>
+ => __elv_add_request
+ => blk_flush_plug_list
+ => blk_finish_plug
+ => __do_page_cache_readahead
+ => ondemand_readahead
+ => page_cache_async_readahead
+ => generic_file_read_iter
+ => new_sync_read
+ => vfs_read
+ => SyS_read
+ => system_call_fastpath
+[...]
+
+
+Use -h to print the USAGE message:
+
+# ./tpoint -h
+USAGE: tpoint [-hHsv] [-d secs] [-p PID] [-L TID] tracepoint [filter]
+       tpoint -l
+                 -d seconds      # trace duration, and use buffers
+                 -p PID          # PID to match on events
+                 -L TID          # thread id to match on events
+                 -v              # view format file (don't trace)
+                 -H              # include column headers
+                 -l              # list all tracepoints
+                 -s              # show kernel stack traces
+                 -h              # this usage message
+
+Note that these examples may need modification to match your kernel
+version's function names and platform's register usage.
+   eg,
+       tpoint -l | grep open
+                                 # find tracepoints containing "open"
+       tpoint syscalls:sys_enter_open
+                                 # trace open() syscall entry
+       tpoint block:block_rq_issue
+                                 # trace block I/O issue
+       tpoint -s block:black_rq_issue
+                                 # show kernel stacks
+
+See the man page and example file for more info.