Linux Trace Toolkit

@) Preliminary Draft @) under construction

Introduction
The Linux Trace Toolkit is used to examine the flow of execution (between processes, kernel threads, and interrupts) in a Linux system. This is useful for analyzing where delays occur in the system, and to see how processes interact (especially with regard to scheduling, interrupts, synchronization primitives, etc.)

* Note that as of 2.6.14, LTT is being replaced with LTTng. See LTTng

See also the announcement here: Patch: Linux Trace Toolkit Viewer/Next Generation announcement (LTTV/LTTng)

Rationale
Tracing is useful for analyzing stuff.

LTTng
See the QUICKSTART guide at: Quickstart

Sample i386 trace for LTTng
Here is a zip file containing a sample trace on an i386 machine: i386-ltt-trace.zip

0.9.6 Release (Nov 21, 2004)
I found it a bit difficult to get the appropriate set of patches for LTT. After a bit of frustration, I built my own release for Nov. 23, 2004. The release is 0.9.6 (no 'pre') and it works with the Linux kernel 2.6.9. The tar file for the release is in the PatchArchive.

Older stuff
The latest stable release is 0.9.5a, but this release is over 2 years old.

Here is the patch set I finally used, for kernel 2.6.8.1:
 * message with ltt-2.6.8.1-relayfs.patch.bz2 by Mathieu Desnoyers
 * message with ltt-2.6.8.1-ltt.patch.bz2 by Mathieu Desnoyers

getting attachments from ltt-dev archive
I had to use 'munpack' to extract the compressed patches from the mail messages in the archive. 'munpack' is part of the package 'mpack', which is available at: munpack

Patch
Here are some patches used by Tim Bird at Sony, targeted at MIPS, but known to also work on other platforms. This tarball contains both an all-in-one patch and a series file with discreet sub-patches.


 * [[media:Ltt-2.6.11-from-TimBird.tgz]]


 * Patches for Linux kernel version 2.6.8.1, and !TraceToolkit tarfile are here:
 * ltt-2.6.8.1.tar.gz
 * TraceToolkit-0.9.6pre3-plus.tar.gz

These are bundles with multiple sub-patches and sub-tars. To install them, do the following:
 * untar kernel patches:
 * apply patches:
 * untar !TraceToolkit stuff
 * unpack and apply patch:
 * untar !TraceToolkit stuff
 * unpack and apply patch:
 * unpack and apply patch:

Now, follow the build and usage instructions for the software in the include docs (!TraceToolkit/Help/index.html). For cross-compiling, use the instructions on this wiki page.

Building the software
Apply the ltt and relayfs patches to your kernel:

make menuconfig You can leave klog debugging support turned off.
 * configure kernel with LTT
 * turn on RelayFS and LTT

Compile the user-space tracedaemon program:

I couldn't figure out if the tracevisualizer tools handle cross-compilation correctly (by which I mean that you can natively compile the tracevisualizer but cross-compile the tracedaemon). Instead of blindly trying configure tricks, I instead used the instructions from Karim's book "Building Embedded Linux Systems". In a nutshell, the instructions go something like this:

Overview: * install source for user-space stuff * configure for native build * hand-build tracedaemon specifying a cross-compiler * build rest of user-space suite using native compiler * install programs as appropriate

Detail (in my case):

* install !TraceToolkit-0.9.6pre3, apply "plus" patch ** tpm -f user-space.pl -o !TraceToolkit-0.9.6pre3 * cd !TraceToolkit-0.9.6pre3 * configure --prefix=/home/tbird/work/ltt/tools * make -C !LibUserTrace CC= !UserTrace.o * make -C !LibUserTrace CC= LDFLAGS="-static" * make -C Daemon CC= LDFLAGS="-static" * cp Daemon/tracedaemon Daemon/Scripts/trace Daemon/Scripts/tracecore Daemon/Scripts/traceu /usr/sbin * make * make install

Using the software
In April, 2004, Karim wrote: lkml thread

Future Work
Here is a list of things that could be worked on for this feature:
 * online reference needs to be updated
 * e.g. no mention of relayfs
 * project seems fairly quiet, and unmaintained
 * see if LTT patch should be refactored to take into account new kprobe support in the kernel

LKML Reaction to tracing
In Sep 2002, there was a thread about tracing, where some major kernel developers expressed their concerns about tracing infrastructure in the kernel.


 * Ingo Molnar

... so use the power of the GPL-ed kernel and keep your patches separate, releasing them for specific stable kernel ranches (or even development kernels).


 * Linus Torvalds