Kernel Trace Systems
Here are some links to information about different kernel tracing systems:
Contents
General Purpose tracing systems
Some major Linux general-purpose tracing systems are:
- ptrace - ability to trace syscall entry and exit, and signal delivery, to a process (also used for debugging a process)
- see "man ptrace" and "man strace"
- Ftrace
- Ftrace Function Graph ARM - presentations and patches by Tim Bird to add function graph and duration tracing to ARM systems
- The presentation has good introductory material on ftrace, as well as links to additional resources
- tracer for kernel functions
- can also be used for debugging or analyzing latencies and performance issues
- in mainline since 2.6.27
- See Measuring Function Duration with FTrace - outline of presentation by Tim Bird for Linux Symposium 2009
- Ftrace Function Graph ARM - presentations and patches by Tim Bird to add function graph and duration tracing to ARM systems
- System Tap - System Tap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system
- LTTng - Linux Trace Toolkit, next generation
- LKST - Linux Kernel State Tracer
Special Purpose tracing systems
There are some other notable special-purpose kernel tracing systems:
- KFT - Kernel Function Trace - traces functions to show function durations and call graphs
- latency trace - RT-preempt tool for measuring interrupt and mutex latency
- The latency tracer is embedded in the RT-preempt patch - see Realtime Preemption and RT-preempt Article
- block tracer (blktrace) - allows you to see exactly what is going on in the block layer for a given queue
- Introduction by Jens Axboe: Introduction
- Execellent presentation: blktrace.pdf
- Guide to using is at: blktrace guide
- This appears to have been mainlined as of 2.6.17
- Timeline utility (blktrace post-processing tool): blktrace timeline utility
- delay accounting patches - collect statistics about the delays that are experienced by each task on the system
Trace Infrastructure
- KProbes - grew out of dprobes, with information at: dprobes
- see an excellent tutorial at: kprobes
- The mainline version of the KProbes supports x86,Alpha and PPC64 architectures. A MIPS implementation has been completed on the 2.6.16 kernel and tested on the Toshiba TX49 platform. Patch is available in the Patch Archive.
- [would be nice to have some djprobe stuff here]
Sampling Systems
Note that profile systems (or "sampling systems") are slightly different, in that they involve sampling instead of event tracing. Some major ones for Linux are:
- top - provides a dynamic real-time view of a running system, including processes
- see "man top"
- see also ksysguard, Gnome system Monitor
- OProfile - system-wide profiler for Linux systems
- see oprofile
- also: oprofile at IBM
- BootChart - samples bootup and provides visualization of process startup and system utilization
- see Bootchart
Related facilities
- in-kernel statistics infrastructure - proposal for a generic implementation of statistics facilities inside the kernel
- see inkernel stats
- perfmon2 - interfaces to hardware performance monitoring features of the CPU
- see perfmon
- inotify - inotify
Other Systems
Here are some systems I haven't classified yet:
- Datastreams - a system for creating and monitoring tracepoints - see datastreams
Collaboration Efforts
Some trace system project leaders are trying to collaborate: see Tracing Collaboration Project