Difference between revisions of "Kernel Trace Systems"

From eLinux.org
Jump to: navigation, search
(General Purpose tracing systems: remove MoinMoin-isms)
(Sampling Systems)
 
(7 intermediate revisions by 3 users not shown)
Line 6: Line 6:
 
*ptrace - ability to trace syscall entry and exit, and signal delivery, to a process (also used for debugging a process)
 
*ptrace - ability to trace syscall entry and exit, and signal delivery, to a process (also used for debugging a process)
 
** see "man ptrace" and "man strace"
 
** see "man ptrace" and "man strace"
 +
* [[Ftrace]]
 +
** [[Ftrace Function Graph ARM]] - presentations and patches by Tim Bird to add function graph and duration tracing to ARM systems
 +
*** The presentation has good introductory material on ftrace, as well as links to additional resources
 +
** tracer for kernel functions
 +
** can also be used for debugging or analyzing latencies and performance issues
 +
** in mainline since 2.6.27
 +
** See [[Measuring Function Duration with FTrace]] - outline of presentation by Tim Bird for Linux Symposium 2009
 
* [[System Tap]] - System Tap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system
 
* [[System Tap]] - System Tap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system
 
*LTTng - [[Linux Trace Toolkit]], next generation
 
*LTTng - [[Linux Trace Toolkit]], next generation
Line 13: Line 20:
  
 
There are some other notable special-purpose kernel tracing systems:
 
There are some other notable special-purpose kernel tracing systems:
* KFT - Kernel Function Trace - traces functions to show function durations and call graphs
+
* KFT - [[Kernel Function Trace]] - traces functions to show function durations and call graphs
**see [[Kernel Function Trace]]
 
 
*latency trace - RT-preempt tool for measuring interrupt and mutex latency
 
*latency trace - RT-preempt tool for measuring interrupt and mutex latency
 
**The latency tracer is embedded in the RT-preempt patch - see [[Realtime Preemption]] and [http://lwn.net/Articles/97811/ RT-preempt Article]
 
**The latency tracer is embedded in the RT-preempt patch - see [[Realtime Preemption]] and [http://lwn.net/Articles/97811/ RT-preempt Article]
Line 43: Line 49:
 
** see [http://oprofile.sourceforge.net/about/ oprofile]
 
** see [http://oprofile.sourceforge.net/about/ oprofile]
 
** also: [http://www-128.ibm.com/developerworks/linux/library/l-oprof.html oprofile at IBM]
 
** also: [http://www-128.ibm.com/developerworks/linux/library/l-oprof.html oprofile at IBM]
* !BootChart - samples bootup and provides visualization of process startup and system utilization
+
* BootChart - samples bootup and provides visualization of process startup and system utilization
** see [[Boot Chart]]
+
** see [[Bootchart]]
  
 
== Related facilities ==
 
== Related facilities ==
Line 62: Line 68:
  
 
Some trace system project leaders are trying to collaborate: see [[Tracing Collaboration Project]]
 
Some trace system project leaders are trying to collaborate: see [[Tracing Collaboration Project]]
 +
 +
[[Category:Linux tracing]]

Latest revision as of 14:01, 28 July 2010

Here are some links to information about different kernel tracing systems:

General Purpose tracing systems

Some major Linux general-purpose tracing systems are:

  • ptrace - ability to trace syscall entry and exit, and signal delivery, to a process (also used for debugging a process)
    • see "man ptrace" and "man strace"
  • Ftrace
    • Ftrace Function Graph ARM - presentations and patches by Tim Bird to add function graph and duration tracing to ARM systems
      • The presentation has good introductory material on ftrace, as well as links to additional resources
    • tracer for kernel functions
    • can also be used for debugging or analyzing latencies and performance issues
    • in mainline since 2.6.27
    • See Measuring Function Duration with FTrace - outline of presentation by Tim Bird for Linux Symposium 2009
  • System Tap - System Tap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system
  • LTTng - Linux Trace Toolkit, next generation
  • LKST - Linux Kernel State Tracer

Special Purpose tracing systems

There are some other notable special-purpose kernel tracing systems:

  • KFT - Kernel Function Trace - traces functions to show function durations and call graphs
  • latency trace - RT-preempt tool for measuring interrupt and mutex latency
  • block tracer (blktrace) - allows you to see exactly what is going on in the block layer for a given queue
  • delay accounting patches - collect statistics about the delays that are experienced by each task on the system

Trace Infrastructure

  • KProbes - grew out of dprobes, with information at: dprobes
    • see an excellent tutorial at: kprobes
    • The mainline version of the KProbes supports x86,Alpha and PPC64 architectures. A MIPS implementation has been completed on the 2.6.16 kernel and tested on the Toshiba TX49 platform. Patch is available in the Patch Archive.
  • [would be nice to have some djprobe stuff here]

Sampling Systems

Note that profile systems (or "sampling systems") are slightly different, in that they involve sampling instead of event tracing. Some major ones for Linux are:

  • top - provides a dynamic real-time view of a running system, including processes
  • OProfile - system-wide profiler for Linux systems
  • BootChart - samples bootup and provides visualization of process startup and system utilization

Related facilities

  • in-kernel statistics infrastructure - proposal for a generic implementation of statistics facilities inside the kernel
  • perfmon2 - interfaces to hardware performance monitoring features of the CPU
  • inotify - inotify

Other Systems

Here are some systems I haven't classified yet:

  • Datastreams - a system for creating and monitoring tracepoints - see datastreams

Collaboration Efforts

Some trace system project leaders are trying to collaborate: see Tracing Collaboration Project