eLinux.org - User contributions [en]

Talk:Session: Developer's Diary: It's About Time ELCE 2011

2012-11-07T08:19:55Z

Peter Huewe: Created page with "work on formatting :/--~~~~"

work on formatting :/--[[User:Peter Huewe|Peter Huewe]] ([[User talk:Peter Huewe|talk]]) 08:19, 7 November 2012 (UTC)

Session: Developer's Diary: It's About Time ELCE 2011

2012-11-07T08:19:17Z

Peter Huewe: Added first 10 minures of transcript-

== Session Details ==
; Event : [[ELCE 2011]], Prague
; Date : 28.10.2011
; Presenter : Wolfram Sang
; Organization: Pengutronix
; Slides : [[Media:elce11_sang.pdf|PDF]]
; Video : [http://free-electrons.com/pub/video/2011/elce/elce-2011-sang-developer-diary.webm | HD] [http://free-electrons.com/pub/video/2011/elce/elce-2011-sang-developer-diary-450p.webm | SD ] thanks to [http://free-electrons.com/blog/elce-2011-videos/ | Free Electrons]
; Duration : 00:49:00

=== Abstract ===
A Talk about timeouts and delays.
=== Biography ===

=== Notes ===

== Transcript ==
; Transcribed by: Peter Huewe
; Verified by: 1 -

''Introduction of and by Wolfram Sang''

My talks which I usually entitle “Developers Diary” I want to share experiences I made in my everyday work I want to [show you] problems I encountered, things I've found out [and] try to give best practices.
As Linux in general is a very huge project and I pretty much want [...]

A lot of people think “oh my problem [fits here] as well” - so if you have something to add, feel free to do so.
I hope at the end of the talk we're all a bit enlightened than before.

This time I want to talk about dealing with time in device drivers and I chose to talk about two groups:

The first I will be talking about [are] delays, when you deal with hardware and sometime you activate something and have to wait for a certain amount of time until something happens.

And the second will be timeouts, you do something and if it not does something in a certain time you'll have to deal with it.

Both are not very complicated matters [on their own] but let's have a look:

-
The basics of delays:

There are two kinds, usually,
One [called] busy waiting: the cpu burns cycles while waiting
and the other one is
blocking: you say “okay I’m going to wait, let me go to sleep, and you’ll come back later so the cpu can do something in between”.

[''Poll:''] Who knew this before?
Okay that's what I kinda expected, so I'm not telling you rocket science. I know that.

[''Poll:'']Who is writing device drivers for the linux kernel here? [''few hands rise'']
Great! That's the audience I was hoping for.
So let's get a bit interactive - [''Q''] What kind of delay functions do you know
[''responses by the audience'']
- schedule
Does it delay?
- udelay
- mdelay
- sleep – which ones
- usleep
-msleep
we're getting into discussion already
-usleep_range
-ndelay ?
- nanosleep?
- msleep_interruptible
Not bad – was it all?
[''Guy in the audience joking''] a for cycle loop
Is there an sdelay`?

So let’start with the solution:
- ndelay
- udelay
- mdelay
luckily no sdelay
no usleep
- usleep_range which you can give a time frame
- msleep
- msleep_interruptible
- ssleep

But it's already interesting to see that here, as you just recognized here are quite a bunch of people doing kernel code and there seems to be some agreements and disagreements of what is available.

And now think about implicit things which might happen in the back[ground] [...]

There can be things happening you're not expecting or whatever so it's really pays off to have a look at this once in a while so you get the foundations right so you're really knowing what you're doing when you're using such functions.

Let’s have a short look at the delay headers. [''opens include/linux/delay.h'']

I’ll call this talk a little bit ARM specific - I hope nobody will be angry about being ARM specific, but it basically applies to all platform - because one of the thing that’s interesting is that the delay functions for ARM are architecture specific.

So this is a general Linux header file – and you can see here in those lines that [...] for mdelay there is a fallback so if the architecture does not define it basically falls back to udelay.

There’s also an ndelay, the general fall back if the architecture does not define one is based on udelay – so that udelay seems to be a pretty important function in all that.

We see msleep, msleep_interruptible, usleep_range and ssleep, they are all defined here, so they are independent of the architecture. If we look at the architecture specific implementation of udelay, which we just found out is a very core function, you see that in the arm case it’s not even a function but the first […] encounter with a define, which tracks if it’s a constant value or not.

That’s the places where you have to look if you want to dive in deeper.

--------------------

So how I got interested in the topic was we had a customer problem and he said “yeah the system is working fine but when we press the touch the system becomes unusable and further analysis have shown that 25% of the cpu was busy doing something”.

I first expected an interrupt storm or whatever but that couldn’t be found out so I started looking at code, especially at the touchscreen driver and this was the thing I found. [''slide shows the function in drivers/input/touchscreen/wm9712.c'']

This is a delay inside the driver, which takes a value from the [lookup]table and [the] array offset is given to the function, which is (as you can see on the next slide) more or less a module parameter.
It says “okay we need that to get the correct value. The delay can be set by setting the delay to n where n is the array position of the delay in the array table below.”
This is not very transparent to the user; it is said “long delays for 1ms are supported for completeness but are not recommended” – that’s a very useful comment but it’s totally invisible for the user using the module.

That’s what happened to our customer – I think it was also caused little bit broken hardware – he didn’t get the touch[screen] working so here’s the “mighty table” which is feeding the udelay.
The standard is this one [indicating 200us], so it will in general burn like 200 us, our customer ended up being here [indicating 3ms] – so on a regular basis for every poll this routine was nearly burning 3ms – and this was indeed the cause for the total cpu loss, so it just lost all the cpu power doing nothing.
Sadly this is still in the kernel – because once the problem was found out, the customer said “yeah we can fix that”, and took the hardware away from me, so I think they did some nasty stuff to get it working – so this is still present in the kernel, sadly.

So this gave me the idea – udelay is a pretty dangerous function.

0:00 - 1:00:

1:00 - 2:00:

2:00 - 3:00:

3:00 - 4:00:

4:00 - 5:00:

5:00 - 6:00:

6:00 - 7:00:

7:00 - 8:00:

8:00 - 9:00:

9:00 - 10:00:

10:00 - 11:00:

11:00 - 12:00:

12:00 - 13:00:

13:00 - 14:00:

14:00 - 15:00:

15:00 - 16:00:

16:00 - 17:00:

17:00 - 18:00:

18:00 - 19:00:

19:00 - 20:00:

20:00 - 21:00:

21:00 - 22:00:

22:00 - 23:00:

23:00 - 24:00:

24:00 - 25:00:

25:00 - 26:00:

26:00 - 27:00:

27:00 - 28:00:

28:00 - 29:00:

29:00 - 30:00:

30:00 - 31:00:

31:00 - 32:00:

32:00 - 33:00:

33:00 - 34:00:

34:00 - 35:00:

35:00 - 36:00:

36:00 - 37:00:

37:00 - 38:00:

38:00 - 39:00:

39:00 - 40:00:

40:00 - 41:00:

41:00 - 42:00:

42:00 - 43:00:

43:00 - 44:00:

44:00 - 45:00:

45:00 - 46:00:

46:00 - 47:00:

47:00 - 48:00:

48:00 - 49:00:

49:00 - 50:00:

50:00 - 51:00:

51:00 - 52:00:

52:00 - 53:00:

53:00 - 54:00:

54:00 - 55:00:

55:00 - 56:00:

56:00 - 57:00:

57:00 - 58:00:

58:00 - 59:00:

59:00 - 60:00:

Session: Developer's Diary: It's About Time ELCE 2011

2012-11-06T12:28:01Z

Peter Huewe: Created page with "== Session Details == ; Event : ELCE 2011, Prague ; Date : 28.10.2011 ; Presenter : Wolfram Sang ; Organization: Pengutronix ; Slides : PDF ; Vi..."

ELCE 2011 Presentations redirect

2012-11-06T12:21:49Z

Peter Huewe: Added link to transcript

Presenters, Demo-ers, Participants:
Thanks very much for your participation in Linux Foundation's [http://events.linuxfoundation.org/events/embedded-linux-conference-europe Embedded Linux Conference Europe 2011].

This page is for collecting the presentations that were made at the conference. During and
after the conference we will collect materials from the presenters and place them here.
Please watch this page if you are interested in a particular presentation - and if it
doesn't show up, please [[Special:EmailUser/Wmat | send me and email]] and we'll try to track it down.

== Videos ==

Once again, through the diligent work of the [http://free-electrons.com/ Free Electrons] team, all videos for ELCE2011 can be found at [http://free-electrons.com/blog/elce-2011-videos/ ELCE2011 Videos].

== Instructions ==
'''Presenters:''' Please post your technical conference presentations on this page.
(See Instructions below the tables)

= Table of Presentations =

NOTE: If you add a wikilink to your presentation and attempt to upload it via the link, it may fail. If it does, use the [[Special:Upload]] page to upload your file.

== Keynotes ==
{| border="1" cellspacing="0" cellpadding="4"
|- bgcolor="#c0e0e0"
|+ '''Keynotes'''
|- bgcolor="#c0e0e0"
| align="center" | '''Presenter(s)'''
| align="center" | '''Session Description'''
| align="center" | '''Presentation'''
|-
| Jim Zemlin, Executive Director of The Linux Foundation
| Imagine a World Without Linux
|
|-
| Linus Torvalds, Alan Cox, Thomas Gleixner, Paul McKenney
| Kernel Developer Panel
| No presentation.
|-
| Antti Aumo, President of Global Solutions at Ixonos
| Re-Defining the Cloud Phone
|
|-
| Dirk Hohndel, Chief Linux and Open Source Technologist at Intel
| Reflection on 20 Years of Linux
|
|-
| Jonathan Corbet, Editor at LWN
| The Kernel Report: 20th Anniversary Edition
|
|-
| Wim Coekaerts, Senior Vice President, Linux and Vistualization Engineering at Oracle
| Engineered Systems With Linux
|
|-
| Matt Jones, Vice President, GENIVI Alliance & Technical Lead - Next Generation Infotainment, Jaguar Land Rover
| Linux for In Car Infotainment
|
|}

<br />

== Presenters ==
{| border="1" cellspacing="0" cellpadding="4"
|- bgcolor="#c0e0e0"
|+ '''Presentations'''
|- bgcolor="#c0e0e0"
| align="center" | '''Presenter(s)'''
| align="center" | '''Session Description'''
| align="center" | '''Presentation'''
|-
|- bgcolor="#a0c0c0"
| colspan="3" | Day 1, 10:45am
|-
| Zach Pfeffer (Linaro)
| Linaro's Android Platform
| [[Media:Linaro Android Presentation 5.pdf|PDF]]
|-
| Thomas Gleixner (Linutronix)
| Another Mile Down the RT Road
|
|-
| Jessica Zhang (Intel)
| The Yocto Project Eclipse PlugIn: An Effective IDE Environment for both Embedded Application and System Developers
| [[Media:ELCE Yocto Plugin 2011 latest.pdf|PDF]]
|- bgcolor="#a0c0c0"
| colspan="3" | Day 1, 11:45am
|-
| Satoru Ueda (Sony/Japan OSS Promotion Forum)
| Contributing to the Community? Does your manager support you?
|
|-
| Benjamin Zores (Alcatel-Lucent)
| Embedded Linux Optimization Techniques: How Not To Be Slow
| [[Media:ELCE_2011_-_BZ_-_Embedded_Linux_Optimization_Techniques_-_How_Not_to_Be_Slow.pdf|PDF]]
|-
| Ohad Ben-Cohen (Texas Instruments/Wizery)
| Asymmetric Multiprocessing using VirtIO (was: "Remote Processor Messaging")
| [[Media:AMP_ELCE2011.pdf|PDF]]
|- bgcolor="#a0c0c0"
| colspan="3" | Day 1, 2:00pm
|-
| Jeff Osier-Mixon (Intel)
| Collaborative Initiatives in Embedded Linux
| [[Media:ELCE2011-Collaborative-Initiatives.pdf|PDF]]
|-
| Karim Yaghmour (Opersys, Inc.)
| Leveraging Android's Linux Heritage
|
|-
| Pierre Tardy (Intel)
| Using pytimechart For Real World Analysis
| [[Media:elce11_tardy.pdf|PDF]]
|-
|- bgcolor="#a0c0c0"
| colspan="3" | Day 1, 3:00pm
|-
|-
| Arnd Bergmann (Linaro)
| Optimizations for Cheap Flash Media
|
|-
| Vitaly Wool (Sony Ericsson)
| Saving Power With Wi-Fi: How to Prolong Your Battery Life and Still Stay Connected
| [[Media:elce11_wool.odp|ODP]]
|-
| David Stewart (Intel)
| Developing Embedded Linux Devices Using the Yocto Project and What's New in 1.1
| [[Media:elce11_stewart.pdf|PDF]]
|- bgcolor="#a0c0c0"
| colspan="3" | Day 1, 4:15pm
|-
| Tetsuyuki Kobayashi (Kyoto Micro Computer)
| Android is NOT Just "Java On Linux"
| [[Media:elce11_kobayashi.pdf|PDF]]
|-
| Thomas Petazzoni (Free Electrons)
| Using Buildroot for a Real Project
| [[Media:Using-buildroot-real-project.pdf|PDF]]
|-
| Rajesh Lal (Nokia)
| Qt Quick: The Most Advanced UI Technology for Mobile
| Talk was canceled
|-
|- bgcolor="#a0c0c0"
| colspan="3" | Day 1, 5:15pm
|-
|-
| Tim Bird (Sony)
| Status of Embedded Linux BoFs
| [[media:Status-of-Embedded-Linux-2011-10-ELCE.pdf|Status-of-Embedded-Linux-2011-10-ELCE.pdf]]
|-
| Lauro Ramos Venancio (Instituto Nokia de Technologia) & Samuel Ortiz (Intel)
| The Linux NFC Subsystem
| [[Media:elce11_venancio_ortiz.pdf|PDF]]
|-
| David Anders (Texas Instruments)
| Board BringUp: LCD and Display Interfaces
| [[media:elce-2011-anders.pdf|PDF]] [[media:elce-2011-anders.odp|ODP]] [[BoardBringupLCD|Resource Page]]
|- bgcolor="#a0c0c0"
| colspan="3" | Day 2, 10:15am
|-
| Grant Likely (Secret Lab)
| Device Tree Status Report
|
|-
| Laurent Pinchart (Ideas on Board)
| Success Story of the Open Source Camera Stack: The Nokia N9 Case
| [[Media:elce11_pinchart.pdf|PDF]]
|-
| Avinash Mahadeva & Vishwanth Sripathy (Texas Instruments)
| SOC Power Management - Debugging and Optimization Techniques
| [[Media:Omap-power-management-debugging.pdf|PDF]]
|-
| Tia Cassett (Qualcomm) & Mike Chalupa (BSquare)
| Android Development with the Snapdragon Processor
|
|-
|- bgcolor="#a0c0c0"
| colspan="3" | Day 2, 11:15am
|-
| Rafael J. Wysocki (Faculty of Physics, U. Warsaw/SUSE Labs)
| Power Management Using PM Domains on SH7372
| [[Media:elce11_wysocki.pdf|PDF]]
|-
| Sascha Hauer (Pengutronix e.K.)
| A Generic Clock Framework in the Kernel: Why We Need It & Why We Still Don't Have It
| [[Media:elce11_hauer.pdf|PDF]]
|-
| Ruud Derwig (Synopsys)
| Android Platform Optimizations
| [[Media:Android_Platform_Optimizations_SNPS_20111027.pdf|PDF]]
|-
| Tia Cassett & David Brown (Qualcomm)
| Kernel Development Using the Dragonboard with the Snapdragon Processor
|
|-
|- bgcolor="#a0c0c0"
| colspan="3" | Day 2, 3:00pm
|-
| Inki Dae (Samsung Electronics)
| DRM Driver Development For Embedded Systems
| [[Media:elce11_dae.pdf|PDF]]
|-
| Lorenzo Pieralisi (ARM Ltd.)
| Consolidating Linux Power Management on ARM Multiprocessor Systems
| [[Media:elce11_pieralisi.pdf|PDF]]
|-
| Thomas Petazzoni (Free Electrons)
| Using Qt For Non-Graphical Applications
| [[Media:qt-for-non-graphical-applications.pdf|PDF]]
|-
| David Anders (Texas Instruments)
| PandaBoard Workshop: Booting the PandaBoard
|
|-
|- bgcolor="#a0c0c0"
| colspan="3" | Day 2, 4:00pm
|-
| Marek Szyprowski & Kyungmin Park (Samsung)
| ARM DMA-Mapping Framework Redesign and IOMMU Integration
| [[Media:elce11_szyprowski_park.pdf|PDF]]
|-
| Keerthy Jagadeesh & Vishwanath Sripathy (Texas Instruments)
| Thermal Framework for ARM based SOCs
|
|-
| Marc Titinger (ST Microelectronics)
| Efficient JTAG-Based Linux Kernel Debugging
| [[Media:elce11_titinger.pdf|PDF]]
|-
| David Anders (Texas Instruments)
| PandaBoard Workshop: PandaBoard Expansion I/O
|
|- bgcolor="#a0c0c0"
| colspan="3" | Day 2, 5:00pm
|-
| Tsugikazu Shibata (NEC & Linux Foundation Board Member)
| Toward the Long Term Stable Kernel Tree for The Embedded Industry
| [[Media:LTSI_ELCE2011.pdf|PDF]]
|-
| Lisko Lappalainen (MontaVista Software)
| Secure Virtualization in Automotive
|
|-
| Jeff Osier-Mixon (Intel)
| Yocto Project Community BoFs
|
|-
| Luca Coelho (Texas Instruments)
| PandaBoard Workshop: WLAN Kernel Hacking with PandaBoard
|
|-
|- bgcolor="#a0c0c0"
| colspan="3" | Day 3, 10:15am
|-
| Andrea Gallo (ST-Ericsson)
| ARM Linux Kernel Alignment & Benefits for Snowball
| [[Media:elce11_gallo.pdf|PDF]]
|-
| Liam Girdwood & Peter Ujfalusi (Texas Instruments)
| Smart Audio: Next-Generation A SoC For Smart Phones
|
|-
| Pawel Moll (ARM Ltd)
| Linux on Non-Existing SoCs
|
|- bgcolor="#a0c0c0"
| colspan="3" | Day 3, 11:15am
|-
| Koen Kooi (The Angstrom Distribution)
| Integrating Systemd: Booting Userspace in Less Than 1 Second
| [[Media:elce11_koen.pdf|PDF]]
|-
| Sylvain Leroy & Philippe Thierry
| Grsecurity in Embedded Linux Used in Android
| [[Media:elce11_thierry_leroy.pdf|PDF]]
|-
| MyungJoo Ham (Samsung))
| Charger Manager: Aggregating Chargers, Fuel Gauges and Batteries
| [[Media:elce11_ham.pdf|PDF]]
|- bgcolor="#a0c0c0"
| colspan="3" | Day 3, 2:30pm
|-
| Frank Rowand (Sony)
| How Linux PREEMPT_RT Works
| [[Media:elce11_rowand.pdf|PDF]]
|-
| Catalin Marinas (ARM Ltd.)
| Linux Support For the Large Physical Address Extensions
| [[Media:elce11_marinas.pdf|PDF]]
|-
| Jim Huang (0xlab)
| Build Community Android Distribution and Ensure the Quality
| [[Media:community-android.pdf|PDF]]
|- bgcolor="#a0c0c0"
| colspan="3" | Day 3, 3:45pm
|-
|-
| Till Jaeger (JBB Rechtsanwalte)
| The Case AVM v. Cybits: The GPL and Embedded Systems
|
|-
| Darren Hart (Intel)
| Tuning Linux For Embedded Systems: When Less Is More
| [[Media:elce11_hart.pdf|PDF]]
|-
| Wolfram Sang (Pengutronix e.K.)
| [[Session: Developer's Diary: It's About Time ELCE 2011| Developer's Diary: It's About Time]]
| [[Media:elce11_sang.pdf|PDF]]
|}

<br />

== Workshops ==
{| border="1" cellspacing="0" cellpadding="4"
|- bgcolor="#c0e0e0"
|+ '''Workshops'''
|- bgcolor="#c0e0e0"
| align="center" | '''Presenter(s)'''
| align="center" | '''Session Description'''
| align="center" | '''Presentation'''
|-
| Chris Simmonds, Freelance Embedded Linux Consultant
| Workshop 1: Outside the Box: An Introduction to Embedded Linux and Hardware Interfacing Using the Snowball Board
| [[Media:Outside-the-box-elce-2011.tar.gz|tar.gz]]
|-
| Karim J. Yaghmour
| Workshop 2: Embedded Android Workshop
|
|-

|}

<br />
== Instructions for Presenters ==
Please create a link in the table for your presentation, copying the style of other links.
(You may need to create an account in order to edit the wiki or upload files.)

When you have created the link, click on it to upload the file containing your slides.
[[Category:ELCE]]
[[Category:2011]]
[[Category:Events]]
[[Category:Presentations]]

LTTng

2012-04-26T22:33:55Z

Peter Huewe: /* External Links */

Linux Trace Toolkit Viewer/Next Generation

LTTV is a modular viewer/analysis tool specifically designed to deal with very
large traces generated by a production system.

It comes with a Linux kernel tracer, Linux Trace Toolkit Next Generation
(LTTng), which builds on the existing LTT tracepoints and RelayFS delivery
mechanism but is a complete rewrite of LTT tracing module and daemon.

The design aims at facilitating contributions from the community. We know that
the vast quantity of analysis that can be performed on trace data is
practically unlimited, so we want to make it as easy and fun as possible to add
those to the project.

You can get it at http://www.lttng.org. Follow the Quickstart guide to know how
to get it from Debian packages, RPM or sources.

LTTV key features :

*Support for large traces (it has been tested with 15GB traces). I has been designed from the start to deal with huge traces.
*Information from several tracefiles can be combined in a single view on the fly.
*Deals with traces coming from any architecture size or endianness.
*Text command line interface supporting plugins for trace batch analysis.
*Graphical interface supporting visualisation plugins.
*Flexible event filter.
*Modular architecture :
- dynamically loadable plugins : each specific view/analysis becomes a plugin.
- module dependencies architecture for maximum functionality reuse and easier testing.
*Addition of new instrumentations (or any kind of trace point) becomes easier with an event description parser and a tracing code generator (genevent).

You can also get LTTng from http://www.lttng.org . Available in Debian, RPM and
sources packages. See the Quickstart guide.

LTTng key features :

*Easy addition of new instrumentations by supporting the genevent code generator.
*Very precise timestamps on events by using the processor cycle counter as an unique monotonic time reference.
*Minimal impact on the traced system : instrumentation does not disable interrupts or take locks. It uses RCU and cmpxchg atomic operations instead.
*Non maskable interrupts (NMI) reentrancy.
*Supports many nestable types : structures, unions, arrays, sequences.
*Supports host dependent types.
*Makes dynamic alignment of trace data with a low overhead.
*Integration with LTTV viewer so tracing can be controlled directly from the graphical interface.
*Can record many (n) independent traces at once.
*Modular architecture.

http://lwn.net/Articles/166952/

== External Links ==
* [http://lwn.net/Articles/491510/ lwn.net: LTTng 2.0: Tracing for power users and developers - part 1]
* [http://lwn.net/Articles/492296/ lwn.net: LTTng 2.0: Tracing for power users and developers - part 2]

[[Category:Linux]]

LTTng

2012-04-26T22:33:35Z

Peter Huewe: Added Links

Linux Trace Toolkit Viewer/Next Generation

LTTV is a modular viewer/analysis tool specifically designed to deal with very
large traces generated by a production system.

It comes with a Linux kernel tracer, Linux Trace Toolkit Next Generation
(LTTng), which builds on the existing LTT tracepoints and RelayFS delivery
mechanism but is a complete rewrite of LTT tracing module and daemon.

The design aims at facilitating contributions from the community. We know that
the vast quantity of analysis that can be performed on trace data is
practically unlimited, so we want to make it as easy and fun as possible to add
those to the project.

You can get it at http://www.lttng.org. Follow the Quickstart guide to know how
to get it from Debian packages, RPM or sources.

LTTV key features :

*Support for large traces (it has been tested with 15GB traces). I has been designed from the start to deal with huge traces.
*Information from several tracefiles can be combined in a single view on the fly.
*Deals with traces coming from any architecture size or endianness.
*Text command line interface supporting plugins for trace batch analysis.
*Graphical interface supporting visualisation plugins.
*Flexible event filter.
*Modular architecture :
- dynamically loadable plugins : each specific view/analysis becomes a plugin.
- module dependencies architecture for maximum functionality reuse and easier testing.
*Addition of new instrumentations (or any kind of trace point) becomes easier with an event description parser and a tracing code generator (genevent).

You can also get LTTng from http://www.lttng.org . Available in Debian, RPM and
sources packages. See the Quickstart guide.

LTTng key features :

*Easy addition of new instrumentations by supporting the genevent code generator.
*Very precise timestamps on events by using the processor cycle counter as an unique monotonic time reference.
*Minimal impact on the traced system : instrumentation does not disable interrupts or take locks. It uses RCU and cmpxchg atomic operations instead.
*Non maskable interrupts (NMI) reentrancy.
*Supports many nestable types : structures, unions, arrays, sequences.
*Supports host dependent types.
*Makes dynamic alignment of trace data with a low overhead.
*Integration with LTTV viewer so tracing can be controlled directly from the graphical interface.
*Can record many (n) independent traces at once.
*Modular architecture.

http://lwn.net/Articles/166952/

== External Links ==
[http://lwn.net/Articles/491510/ lwn.net: LTTng 2.0: Tracing for power users and developers - part 1]
[http://lwn.net/Articles/492296/ lwn.net: LTTng 2.0: Tracing for power users and developers - part 2]

[[Category:Linux]]

Perf

2012-04-15T14:18:50Z

Peter Huewe:

== Installation ==

=== Documentation ===
The documentation is written in asciidoc syntax and needs asciidoc installed to be converted into html/manpages.
You can create the documentation of perf by typing 'make' in 'tools/perf/Documentation'.

== Usage ==
Perf has a [[git]]-like interface, so
perf help
shows you the basic usage and a list of the most common subcommands and e.g.
perf top help
shows you the help for 'perf top' command

= References and external links =
* [https://perf.wiki.kernel.org/ Perf Wiki at Kernel.org]

* [https://events.linuxfoundation.org/events/collaboration-summit/de-melo Collaboration Submit 2012: The Linux Perf Tools: Overview and Current Developments] [https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_melo.pdf slides] [http://video.linux.com/videos/the-linux-perf-tools-overview-and-developments video]
* [https://events.linuxfoundation.org/events/collaboration-summit/weisbecker Collaboration Submit 2012: Stacktraces / Callchains in Perf Events: What's Now? What's Next?] [https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_weisbecker.pdf slides] [http://video.linux.com/videos/stacktraces--callchains-in-perf-events video]

System Tap

2012-04-14T01:35:41Z

Peter Huewe: /* Links */

This page has information about [[System Tap]], which is of interest to embedded developers,
because tracers are a useful tool for diagnosing problems during product development.

== Introduction ==
SystemTap is a flexible and extensible system for adding trace collection and
analysis to a running Linux kernel.

SystemTap is designed to be very flexible (allowing for the insertion of
arbitrary C code), yet also easy-to-use (most trace statements are written
in a simple scripting language, with useful data collection and aggregation
routines available in (essentially) library form).

A key aspect of SystemTap is that it is intended to allow you to create
a trace set (a "tapset"), and run it on a running Linux system, with
no modification or re-compilation of the system required. To do this, it
uses the kernel [http://www-users.cs.umn.edu/~boutcher/kprobes/ KProbes] interface and loadable kernel modules to dynamically
add probe points and newly generated code to the running kernel.

== Open Source Projects/Mailing Lists ==
The main SystemTap site is at: http://sourceware.org/systemtap/

The SystemTap mail list archives are at: http://sourceware.org/ml/systemtap/

The tutorial, which gives a good overview of the system, is at:
http://sourceware.org/systemtap/tutorial/

== Miscellaneous notes ==
=== Probe types ===
There are several types of probes:
* kprobe & kretprobe, for dynamically insterted probes
* timers
* static instrumentation markers
* performance counter events

In the future, there may be:
* user-space probes,
* user-space return probes, and
* watchpoint probes (kernel & user)
* and more

== See Also ==
Note that SystemTap is one of the major tracing systems for the Linux kernel.

There is work afoot (as of spring 2006) to try to collaborate on different
parts of the tracing problem, between some of the major tracing projects.
See the [[Tracing Collaboration Project]] page for more information.

=== ARM Support ===
System Tap works on ARM & OMAP platforms instructions are available [http://omappedia.org/wiki/Systemtap here]

== Some Performance measurements ==
Jian Gui writes (in July 2006 on the [[System Tap]] mailing list):

<pre>
Hi, we've tested the overhead of systemtap/LKET with some benchmarks
on a ppc64 machine.

It shows the overhead of systemtap/LKET is acceptable generally.
But it will also cause significant overhead for some benchmark of
special behavior, e.g. dbench. Dbench calls kill() in a very high
frequency to check whether a task is complete, thus leads to a high
overhead.

We categorized the event hooks into five groups in the testing:
grp1 - syscall.entry, process
grp2 - syscall.return, process
grp3 - iosyscall, ioscheduler, scsi, aio, process
grp4 - tskdispatch, pagefault, netdev, process
grp5 - syscall.entry, syscall.return, process

All the results are
(score1 - score2)/score2 * 100%, where:
score1: the benchmark score when probed by systemtap
score2: the benchmark score without probing

dbench (<3% is noise)
--------------------
grp1 -14.4%
grp2 -33.1%
grp3 -7.92%
grp4 -13.6%
grp5 -43.3%

specjbb (<3% is noise)
---------------------
grp 1 -0.87%
grp 2 -0.67%
grp 4 +0.47%
grp 5 +0.05%

tiobench (<3% is noise)
----------------------
grp1 sequential reads +1.45%
sequential writes -6.98%
random reads +0.57%
random writes -2.11%
grp2 sequential reads +0.11%
sequential writes -5.81%
random reads +0.03%
random writes -2.11%
grp3 sequential reads +1.42%
sequential writes -6.98%
random reads +0.51%
random writes -2.11%
grp4 sequential reads +1.38%
sequential writes -5.81%
random reads +0.60%
random writes -2.11%
grp5 sequential reads +0.22%
sequential writes -8.14%
random reads -0.10%
random writes -1.05%

Rawiobench (<3% is noise)
------------------------
grp1 sequential aioread() 0%
sequential aiowrite() 0%
random aioread() 0%
random aiowrite() 0%
grp2 sequential aioread() 0%
sequential aiowrite() 0%
random aioread() 0%
random aiowrite() -0.82%
grp3 sequential aioread() 0%
sequential aiowrite() 0%
random aioread() 0%
random aiowrite() 0%
grp4 sequential aioread() 0%
sequential aiowrite() 0%
random aioread() +0.79%
random aiowrite() -0.82%
grp5 sequential aioread() 0%
sequential aiowrite() -6.41%
random aioread() +0.79%
random aiowrite() 0%

Test environment:
Machine: Open Power 720/ 8 cpus/ 2 cores/ 6GB RAM (tiobench use 1G)
Software: RHEL4-U3GA/ 2.6.17.2/ systemtap-20060718/ elfutils-0.122-0.4

</pre>

== Links ==
* [https://events.linuxfoundation.org/events/collaboration-summit/stone SystemTap Sans Kernel: A Pure Userspace Backend] [https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_jstone.pdf Slides] [http://video.linux.com/videos/systemtap-sans-kernel-a-pure-userspace-backend Video]

[[Category:Development Tools]]

System Tap

2012-04-14T01:35:20Z

Peter Huewe: Added videos

This page has information about [[System Tap]], which is of interest to embedded developers,
because tracers are a useful tool for diagnosing problems during product development.

== Introduction ==
SystemTap is a flexible and extensible system for adding trace collection and
analysis to a running Linux kernel.

SystemTap is designed to be very flexible (allowing for the insertion of
arbitrary C code), yet also easy-to-use (most trace statements are written
in a simple scripting language, with useful data collection and aggregation
routines available in (essentially) library form).

A key aspect of SystemTap is that it is intended to allow you to create
a trace set (a "tapset"), and run it on a running Linux system, with
no modification or re-compilation of the system required. To do this, it
uses the kernel [http://www-users.cs.umn.edu/~boutcher/kprobes/ KProbes] interface and loadable kernel modules to dynamically
add probe points and newly generated code to the running kernel.

== Open Source Projects/Mailing Lists ==
The main SystemTap site is at: http://sourceware.org/systemtap/

The SystemTap mail list archives are at: http://sourceware.org/ml/systemtap/

The tutorial, which gives a good overview of the system, is at:
http://sourceware.org/systemtap/tutorial/

== Miscellaneous notes ==
=== Probe types ===
There are several types of probes:
* kprobe & kretprobe, for dynamically insterted probes
* timers
* static instrumentation markers
* performance counter events

In the future, there may be:
* user-space probes,
* user-space return probes, and
* watchpoint probes (kernel & user)
* and more

== See Also ==
Note that SystemTap is one of the major tracing systems for the Linux kernel.

There is work afoot (as of spring 2006) to try to collaborate on different
parts of the tracing problem, between some of the major tracing projects.
See the [[Tracing Collaboration Project]] page for more information.

=== ARM Support ===
System Tap works on ARM & OMAP platforms instructions are available [http://omappedia.org/wiki/Systemtap here]

== Some Performance measurements ==
Jian Gui writes (in July 2006 on the [[System Tap]] mailing list):

<pre>
Hi, we've tested the overhead of systemtap/LKET with some benchmarks
on a ppc64 machine.

It shows the overhead of systemtap/LKET is acceptable generally.
But it will also cause significant overhead for some benchmark of
special behavior, e.g. dbench. Dbench calls kill() in a very high
frequency to check whether a task is complete, thus leads to a high
overhead.

We categorized the event hooks into five groups in the testing:
grp1 - syscall.entry, process
grp2 - syscall.return, process
grp3 - iosyscall, ioscheduler, scsi, aio, process
grp4 - tskdispatch, pagefault, netdev, process
grp5 - syscall.entry, syscall.return, process

All the results are
(score1 - score2)/score2 * 100%, where:
score1: the benchmark score when probed by systemtap
score2: the benchmark score without probing

dbench (<3% is noise)
--------------------
grp1 -14.4%
grp2 -33.1%
grp3 -7.92%
grp4 -13.6%
grp5 -43.3%

specjbb (<3% is noise)
---------------------
grp 1 -0.87%
grp 2 -0.67%
grp 4 +0.47%
grp 5 +0.05%

tiobench (<3% is noise)
----------------------
grp1 sequential reads +1.45%
sequential writes -6.98%
random reads +0.57%
random writes -2.11%
grp2 sequential reads +0.11%
sequential writes -5.81%
random reads +0.03%
random writes -2.11%
grp3 sequential reads +1.42%
sequential writes -6.98%
random reads +0.51%
random writes -2.11%
grp4 sequential reads +1.38%
sequential writes -5.81%
random reads +0.60%
random writes -2.11%
grp5 sequential reads +0.22%
sequential writes -8.14%
random reads -0.10%
random writes -1.05%

Rawiobench (<3% is noise)
------------------------
grp1 sequential aioread() 0%
sequential aiowrite() 0%
random aioread() 0%
random aiowrite() 0%
grp2 sequential aioread() 0%
sequential aiowrite() 0%
random aioread() 0%
random aiowrite() -0.82%
grp3 sequential aioread() 0%
sequential aiowrite() 0%
random aioread() 0%
random aiowrite() 0%
grp4 sequential aioread() 0%
sequential aiowrite() 0%
random aioread() +0.79%
random aiowrite() -0.82%
grp5 sequential aioread() 0%
sequential aiowrite() -6.41%
random aioread() +0.79%
random aiowrite() 0%

Test environment:
Machine: Open Power 720/ 8 cpus/ 2 cores/ 6GB RAM (tiobench use 1G)
Software: RHEL4-U3GA/ 2.6.17.2/ systemtap-20060718/ elfutils-0.122-0.4

</pre>

== Links ==
* [https://events.linuxfoundation.org/events/collaboration-summit/stone SystemTap Sans Kernel: A Pure Userspace Backend] [https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_jstone.pdf Slides][http://video.linux.com/videos/systemtap-sans-kernel-a-pure-userspace-backend Video]

[[Category:Development Tools]]

User:Peter Huewe

2012-04-14T01:32:31Z

Peter Huewe:

Thanks for visiting my page.

I'm a 26 year old Embedded Linux Software Developer from Germany.

Don't hesitate to contact me if you have any questions.
I'm always open to new ideas and interesting projects.

Contact:
* Email: broeggle <at> scheissprovider <dot> de
* Email: PeterHuewe <at> linux <dot> com
* Jabber: broeggle <at> jabber <dot> ccc <dot> de
* ICQ : 137 1111 59

List of articles I'm currently working on / will be working on:
* [[Debugging by printing]]
* [[Debugging by querying]]
* [[Debugging OOPS]]
* [[JTAG]]
* [[MagicSysRq]]
* [[gcov]]
* [[profiling]]
* [[tracing]]
* [[Kernel Debug Support]]
* [[KGDB]]\[[KDB]]
* [[FTrace]] -> [[kernelshark]]
* [[LTTng]]
* [[oprofile]]
* [[System Tap]]
* [[Kprobes]]
* [[perf]]

* [[Debugging Portal]]

Please feel free to help me ;)

Currently missing articles on elinux.org
[[Coreboot]]

Links:
https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_hiramatsu.pdf

Tracing

2012-04-14T01:31:19Z

Peter Huewe:

* https://events.linuxfoundation.org/events/collaboration-summit/rostedt
* https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_rostedt.pdf
* http://video.linux.com/videos/removing-stop-machine-from-the-tracing-infrastructure

* https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_zannoni_hees.pdf
* http://video.linux.com/videos/dtrace

Tracing

2012-04-14T01:30:09Z

Peter Huewe: Created page with "* https://events.linuxfoundation.org/events/collaboration-summit/rostedt * https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_rostedt.pdf * http://video.linux.com/vid..."

Perf

2012-04-14T01:29:07Z

Peter Huewe: Created page with "https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_melo.pdf http://video.linux.com/videos/the-linux-perf-tools-overview-and-developments https://events.linuxfoundatio..."

https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_melo.pdf
http://video.linux.com/videos/the-linux-perf-tools-overview-and-developments
https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_weisbecker.pdf
http://video.linux.com/videos/stacktraces--callchains-in-perf-events

Debugging Tips

2012-04-03T21:37:55Z

Peter Huewe:

* See the [[Debugging_Portal]] page
* See the [[Kernel Debugging Tips]] page
* See also [[Debugging Makefiles]]
* [[Debugging by printing]]
* Debug user-space initialization:
** If you get a panic - "not syncing: Attempted to kill init!" it can be for many different reasons. Try setting CONFIG_DEBUG_USER=y in your .config and pass 'user_debug=255' in the kernel command line. That will give you a more verbose output about why user space programs crash. Thanks to Daniel Mack on the linux-arm-kernel mailing list for this tip.
[[Category: Tips and Tricks]]

Talk:Debugging by printing

2012-04-03T21:35:23Z

Peter Huewe: Created page with "TODO: """We should probably add the scripts/bootgraph.pl information to that page (or to the initcall_debug page). I don't think very many people are using that yet.""" Consoli..."

TODO:
"""We should probably add the scripts/bootgraph.pl information
to that page (or to the initcall_debug page). I don't think very many people
are using that yet."""

Consolidate remaining stuff from debugging tips

Kernel Debugging Tips

2012-04-03T21:33:21Z

Peter Huewe: Added links to content consolidated with Debugging_by_printing

Here are some miscellaneous tips for debugging a kernel:

== Using printks ==
To add your own debug message to the kernel, you can place a "printk()" in the kernel code.

See [[Debugging_by_printing#Usage| Debugging by printing -> Usage]] for more details.

=== Log levels ===
Each kernel message can be pre-pended with a tag indicating the importance of the message.

See [[Debugging_by_printing#Log_Levels| Debugging by printing -> Log_Levels]] for more details.

=== Adding timing information ===
Sometimes, it is useful to add timing information to the printk values, so you can see when a particular event occurred. The kernel includes an feature for doing
this called printk times.

See the help for CONFIG_PRINTK_TIMES in the file lib/Kconfig.debug for more information on this feature. This option is found on the "Kernel Hacking" menu when configuring the kernel.

The timestamps which are inserted into the printk output consist of seconds and microseconds, as absolute values from the start of machine operation (or from the start of kernel timekeeping).

There is also tool in the kernel source which will convert the timestamp values to relative values (so you can see the interval between events). This tools is called show_delta and is located in the kernel 'scripts' directory.

See [[Printk Times]] for more information.

=== Viewing log messages ===
The <tt>klogd</tt> program will extract messages from the kernel log buffer, and send them to the system log (which winds up in /var/log/messages on most systems). This command runs in the background on most desktop or server systems, and continually transfers messages from the kernel log buffer to the system log.

You can view the contents of the log buffer directly, using the <tt>dmesg</tt> command. Note that by default <tt>dmesg</tt> displays the messages from the buffer, but does not remove them. So this command can be run multiple times to view the kernel printk messages. See the dmesg man page for more things you can do with this tool.

=== Controlling console output ===
In order to have the kernel boot be less "noisy", or in order to boot more quickly, it is sometimes useful to control the amount of messages displayed to the console during boot. You can do this by setting the kernel log level at boot time via a kernel command line option. See the "loglevel=" argument in <tt>Documentation/kernel-parameters.txt</tt>.

You can turn off all messages using the kernel command line option "quiet". See [[Disable Console]] for information on how much time this can save at boot up.

Note that even if the log level is changed, or "quiet" is used, although the printk messages are not print to console, they are still entered into the log buffer, and they can still be extracted and displayed later using the <tt>dmesg</tt> command.

=== Changing the size of the printk buffer ===
See [[Debugging_by_printing#Internals_.2F_Changing_the_size_of_the_printk_buffer | Debugging by printing -> Internals / Changing the size of the printk buffer ]]

== Using kernel symbols ==
You can look up the source code for a function address using your toolchain's addr2line program.
See [[Find a kernel function line]] or [[Addr2line for kernel debugging]].

== Using a kernel debugger ==
You can use the in-kernel debugger: [[KDB]]

Also, you can use QEMU and gdb (and a high-level IDE like eclipse).

See [http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu] for a great article on using Eclipse (with the CDT plugin) to debug the Linux kernel.

== Debugging early boot problems ==
See [[Debugging_by_printing#Debugging_early_boot_problems]]

== Triggering a kernel event ==
=== Overloading the sync system call ===
Sometimes, it is nice to trigger an event to happen in the kernel from user space. Instead of creating infrastructure to handle a /proc event, an ioctl() or making a new syscall, it can be quick and easy to just overload an existing function. One function not used very often is sync. (I have found that the sync system call is not normally called by user space programs (or during standard linux booting).

It is quite easy to put a hook to your own kernel program in the sys_sync() routine (located in fs/sync.c) and cause it to execute by issuing 'sync' from the shell command line. This is handy as a temporary mechanism to test things that you have put in the kernel.

[[Category:Development Tools]]

== Interpreting an Oops message ==
When the kernel encounters an internal fault, it will print an Oops message.
Here are some tips on using the Oops message to find the source of the problem.
* See [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/oops-tracing.txt;hb=HEAD Documentation/oops-tracing.txt]
* See [http://vmlinux.org/jocke/linux/howto-find-oops-location.shtml HOWTO find oops location] by Denis Vlasenko

== Compilation tricks for the kernel ==
Sometimes, you want to modify how the compiler builds an individual kernel file. The following are tips for doing tasks related to this.
=== Build an individual file ===
You can build an individual output object file, with:
make fs/buffer.o

This will build JUST fs/buffer.o (if it needs rebuilding) and not the entire kernel. To force it to need re-building, use 'touch' on the associated source file:
touch fs/buffer.c

=== Create the preprocessed file for an individual source file ===
Using the same technique, you can create the preprocessed file for a C source file. This is useful if you're having trouble tracking down macro expansion or where defines/prototypes are coming from exactly.
make fs/buffer.i

=== Create the assembly file for an individual source file ===
Using the same technique, you can create the assembly file for a C source file. This is useful to get an idea what actual machine instructions are generated from the C source code.
make fs/buffer.s

Another way to get the raw assembly, is to dump the object file using 'objdump'
objdump -d fs/buffer.o > fs/buffer.disassem

This will produce a disassembly of the object file, which should show how the assembly was translated into machine instructions.

If the object has been compiled with debug symbols (using '-g'), then you might get more information using the '-S' option with objdump:
objdump -S -d fs/buffer.o >fs/buffer.disassem

You can also request that the toolchain show mixed source and assembly, by passing extra flags:
make EXTRA_CFLAGS="-g -Wa,-a,-ad -fverbose-asm" fs/buffer.o >fs/buffer.mixed

=== Alter the flags for a compilation ===
Sometimes, you need to alter the compilation flags for an individual file. There are two ways to do this. One is to add the extra flags on the make command line:

make EXTRA_CFLAGS="-g -finstrument-functions" fs/buffer.o

This will work if the flags can be appended to the regular set of C flags used for compiling the object.

However, if you need to do something more complicated, like removing or modifying flags, then you can build your own command line by hand. To do this, it is easiest to have 'make' produce the default compilation command (which will be several lines long), then copy, paste and edit it, to run on the command line directly. To see the exact compile commands used to compile a particular object, use the V=1 option with the kernel build system:

make V=1 fs/buffer.o

For me, this produced something like this:

mipsel-linux-gcc -Wp,-MD,fs/.buffer.o.d -nostdinc -isystem /home/usr/local/mipsel-linux-glibc/bin/../lib/gcc/mipsisa32el-linux/3.4.3/include -D__KERNEL__ -Iinclude -Iinclude2 -I/home/tbird/work/linux/include -I/home/tbird/work/linux/fs -Ifs -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -O2 -fomit-frame-pointer -g -I/home/tbird/work/linux/ -I /home/tbird/work/linux/include/asm/gcc -G 0 -mno-abicalls -fno-pic -pipe -finline-limit=100000 -mabi=32 -march=mips32r2 -Wa,-32 -Wa,-march=mips32r2 -Wa,-mips32r2 -Wa,--trap -I/home/tbird/work/linux/include/asm-mips/ati -Iinclude/asm-mips/ati -I/home/tbird/work/linux/include/asm-mips/mach-generic -Iinclude/asm-mips/mach-generic -Wdeclaration-after-statement -DKBUILD_BASENAME=buffer -DKBUILD_MODNAME=buffer -c -o fs/buffer.o /home/tbird/work/linux/fs/buffer.c

Debugging by printing

2012-04-03T21:30:21Z

Peter Huewe:

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.
The k in printk is used to specifically remind kernel developers that the environment is different.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals / Changing the size of the printk buffer ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

'''Note:''' dmesg reads by default a buffer of max 16392 bytes, so if you use a larger logbuffer you have to invoke dmesg with the ''-s'' parameter e.g.:
### CONFIG_LOG_BUF_SHIFT 17 = 128k
$ dmesg -s 128000

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg / Clearing the buffer ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== Disabling printk messages at compile time ==
There is a configuration option which allows you to turn off all the printk messages in the
whole kernel (''CONFIG_PRINTK''). This reduces the size the kernel, usually by at least 100k,
since all message strings are not compiled into the kernel binary image.

However, it also means you get absolutely no output from the kernel while it is running. Disabling kernel
printk messages is usually the last thing you do when you are tuning your kernel for size.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)

Some page related to printk:
* [[Printk Times]] - has information about how to turn on timestamps for each printk message
** [[Printk_Times_Sample_4| printk time stamps sample]]
* [[Printk_Size_Info|printk size information]]
* [[Do Printk]] - has information about a method of disabling printk messages on a per-module basis

{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Kernel Debugging Tips

2012-04-03T21:28:21Z

Peter Huewe: Consolidated with Debugging_by_printing

Debugging by printing

2012-04-03T21:26:17Z

Peter Huewe: /* Internals */

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals / Changing the size of the printk buffer ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

'''Note:''' dmesg reads by default a buffer of max 16392 bytes, so if you use a larger logbuffer you have to invoke dmesg with the ''-s'' parameter e.g.:
### CONFIG_LOG_BUF_SHIFT 17 = 128k
$ dmesg -s 128000

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg / Clearing the buffer ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== Disabling printk messages at compile time ==
There is a configuration option which allows you to turn off all the printk messages in the
whole kernel (''CONFIG_PRINTK''). This reduces the size the kernel, usually by at least 100k,
since all message strings are not compiled into the kernel binary image.

However, it also means you get absolutely no output from the kernel while it is running. Disabling kernel
printk messages is usually the last thing you do when you are tuning your kernel for size.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)

Some page related to printk:
* [[Printk Times]] - has information about how to turn on timestamps for each printk message
** [[Printk_Times_Sample_4| printk time stamps sample]]
* [[Printk_Size_Info|printk size information]]
* [[Do Printk]] - has information about a method of disabling printk messages on a per-module basis

{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T21:25:30Z

Peter Huewe: /* Internals */ added dmesg -s

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

'''Note:''' dmesg reads by default a buffer of max 16392 bytes, so if you use a larger logbuffer you have to invoke dmesg with the ''-s'' parameter e.g.:
### CONFIG_LOG_BUF_SHIFT 17 = 128k
$ dmesg -s 128000

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg / Clearing the buffer ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== Disabling printk messages at compile time ==
There is a configuration option which allows you to turn off all the printk messages in the
whole kernel (''CONFIG_PRINTK''). This reduces the size the kernel, usually by at least 100k,
since all message strings are not compiled into the kernel binary image.

However, it also means you get absolutely no output from the kernel while it is running. Disabling kernel
printk messages is usually the last thing you do when you are tuning your kernel for size.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)

Some page related to printk:
* [[Printk Times]] - has information about how to turn on timestamps for each printk message
** [[Printk_Times_Sample_4| printk time stamps sample]]
* [[Printk_Size_Info|printk size information]]
* [[Do Printk]] - has information about a method of disabling printk messages on a per-module basis

{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T21:19:50Z

Peter Huewe: /* Dmesg */

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg / Clearing the buffer ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== Disabling printk messages at compile time ==
There is a configuration option which allows you to turn off all the printk messages in the
whole kernel (''CONFIG_PRINTK''). This reduces the size the kernel, usually by at least 100k,
since all message strings are not compiled into the kernel binary image.

However, it also means you get absolutely no output from the kernel while it is running. Disabling kernel
printk messages is usually the last thing you do when you are tuning your kernel for size.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)

Some page related to printk:
* [[Printk Times]] - has information about how to turn on timestamps for each printk message
** [[Printk_Times_Sample_4| printk time stamps sample]]
* [[Printk_Size_Info|printk size information]]
* [[Do Printk]] - has information about a method of disabling printk messages on a per-module basis

{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T21:17:24Z

Peter Huewe: Printk was consolidated with Debugging_by_printing - so can be removed

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== Disabling printk messages at compile time ==
There is a configuration option which allows you to turn off all the printk messages in the
whole kernel (''CONFIG_PRINTK''). This reduces the size the kernel, usually by at least 100k,
since all message strings are not compiled into the kernel binary image.

However, it also means you get absolutely no output from the kernel while it is running. Disabling kernel
printk messages is usually the last thing you do when you are tuning your kernel for size.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)

Some page related to printk:
* [[Printk Times]] - has information about how to turn on timestamps for each printk message
** [[Printk_Times_Sample_4| printk time stamps sample]]
* [[Printk_Size_Info|printk size information]]
* [[Do Printk]] - has information about a method of disabling printk messages on a per-module basis

{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging Portal

2012-04-03T21:16:29Z

Peter Huewe: Since printk and Debugging by printing are consolidated move both into one line

This page is the new portal for all debugging related stuff.
It currently deals with Kernel topics, but feel free to add user space debugging too.

<div style="width=100%; background:#FFF;">
{| width="100%" cellspacing="5" cellpadding="0" valign="top" style="background:inherit;"
| valign="top" style="margin: 0 0 0.5em 1em; border:1px solid #aaa; text-align:left; width: 50%;" cellpadding="5" |
<div style="background: #eeeeee; border: 1px solid #2C547A; padding: 5px; margin: 3px; font-weight:bold;text-align:center;font-size:120%;">Kernel Debugging</div>
<div style="margin: 0 0 0 1em;">
;[[Debugging by printing]] / [[Printk]]
;[[Kernel Debugging Tips]]
;[[Kgdb]]
;[[KDB]]
;[[Debugging The Linux Kernel Using Gdb]]
;[[MagicSysRq]]
;External Links:
*[http://tree.celinuxforum.org/CelfPubWiki/ELC2009Presentations?action=AttachFile&do=get&target=DebuggingWithJtagCelf2009.pdf Embedded Linux JTAG debugging (CELF presentation)]
*[http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu]

</div>
| valign="top" style="margin: 0 0 0.5em 1em; border:1px solid #aaa; text-align:left; width: 50%;" cellpadding="5"|
<div style="background: #eeeeee; border: 1px solid #2C547A; padding: 5px; margin: 3px; font-weight:bold;text-align:center;font-size:120%;">Kernel Tracing and Profiling</div>
<div style="margin: 0 0 0 1em;">
;[[System Tap]]
;[[Kernel Trace Systems]]
;[[Linux Trace Toolkit]]
;[[LTTng]]
;[[Ftrace]]
*[[Measuring Function Duration with FTrace]]
*[[Ftrace Function Graph ARM]]
;[[Using Kernel Function Trace]]
;[[Linux Kernel State Tracer]]

</div>
|}
</div>
[[Category:Development Tools]]
[[Category:Tips and Tricks]]

Printk

2012-04-03T21:15:16Z

Peter Huewe: Consolidated with Debugging_by_printing

#REDIRECT [[Debugging_by_printing]]

Debugging by printing

2012-04-03T21:14:15Z

Peter Huewe: Moved / Added section about other printk pages here

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== Disabling printk messages at compile time ==
There is a configuration option which allows you to turn off all the printk messages in the
whole kernel (''CONFIG_PRINTK''). This reduces the size the kernel, usually by at least 100k,
since all message strings are not compiled into the kernel binary image.

However, it also means you get absolutely no output from the kernel while it is running. Disabling kernel
printk messages is usually the last thing you do when you are tuning your kernel for size.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)

Some page related to printk:
* [[Printk Times]] - has information about how to turn on timestamps for each printk message
** [[Printk_Times_Sample_4| printk time stamps sample]]
* [[Printk_Size_Info|printk size information]]
* [[Do Printk]] - has information about a method of disabling printk messages on a per-module basis

{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T21:12:27Z

Peter Huewe: /* Dynamic Debugging */

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== Disabling printk messages at compile time ==
There is a configuration option which allows you to turn off all the printk messages in the
whole kernel (''CONFIG_PRINTK''). This reduces the size the kernel, usually by at least 100k,
since all message strings are not compiled into the kernel binary image.

However, it also means you get absolutely no output from the kernel while it is running. Disabling kernel
printk messages is usually the last thing you do when you are tuning your kernel for size.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T21:11:22Z

Peter Huewe: Added / moved section about disabling the messages

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== Disabling printk messages at compile time ==
There is a configuration option which allows you to turn off all the printk messages in the
whole kernel (''CONFIG_PRINTK''). This reduces the size the kernel, usually by at least 100k,
since all message strings are not compiled into the kernel binary image.

However, it also means you get absolutely no output from the kernel while it is running. Disabling kernel
printk messages is usually the last thing you do when you are tuning your kernel for size.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T21:08:07Z

Peter Huewe:

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T21:07:47Z

Peter Huewe: Added some stuff from [printk] here

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

Most of the conversion specifiers supported by the user-space library routine printf() are also available in the kernel; there are some notable additions, including "%pf", which will print the symbol name in place of the numeric pointer value, if available.

The supported format strings are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T21:02:52Z

Peter Huewe: Added IBM Document

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

The supported format strings are also quite similar to the standard ones
supported in printf and are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
* [http://www.ibm.com/developerworks/linux/library/l-kernel-logging-apis/index.html Kernel logging: APIs and implementation, Tim Jones (for IBM)] (nice article)
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T20:59:18Z

Peter Huewe: /* References and external links */

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

The supported format strings are also quite similar to the standard ones
supported in printf and are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* [http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T20:58:16Z

Peter Huewe: Added KERN_CONT

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

The supported format strings are also quite similar to the standard ones
supported in printf and are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|-
|KERN_CONT
|"<c>"
|"continued" line of log printout (only done after a line that had no enclosing \n) [http://lwn.net/Articles/252651/]
|pr_cont
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

''KERN_CONT'' and ''pr_cont'' are special cases since they do not specify an urgency but rather indicate a 'continued message' e.g.:
printk(KERN_ERR "Doing something was ");
/* <100 lines of whatever>*/
if (success)
printk(KERN_CONT "successful\n");
else
printk(KERN_CONT "NOT successful\n");

-> "Doing something was successful"
'''Important Note:''' ''KERN_CONT'' and ''pr_cont'' should only be used by core/arch code during early bootup (a continued line is not SMP-safe otherwise).[http://lwn.net/Articles/252651/]

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* []http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Kernel Debugging Tips

2012-04-03T20:45:24Z

Peter Huewe: Moved to Debugging by Printing

Here are some miscellaneous tips for debugging a kernel:

== Using printks ==
To add your own debug message to the kernel, you can place a "printk()" in the kernel code.

These messages are logged to an internal buffer, and eventually displayed by the kernel, being emitted to whatever is the active console, at the time of display. Since the console is not initialized until part-way through the boot sequence, several messages build up in the log until the console is initialize, at which point all buffered messages are displayed at once.

After the messages are displayed on the kernel console, they are left in the log buffer (which is a ring buffer), and they can be (and usually are) retrieved later by various commands.

printk is a variable-argument function, just like printf. Note that there is no "printf()" function for the kernel. "printk()" is used to specifically remind kernel developers that the environment is different.

printk can use many standard C-style printf format characters, but not all of them. See the kernel source for examples of what you can do.

=== Log levels ===
Each kernel message can be pre-pended with a tag indicating the importance of the message. The message tags are shown in the file include/linux/kernel.h. The available tags are:
KERN_EMERG, KERN_ALERT, KERN_CRIT, KERN_WARNING, KERN_NOTICE, KERN_INFO, KERN_DEBUG.

Note that these tags are defined as simple strings, with angle brackets and a number:
#define KERN_INFO "<6>"

You could place the tag onto front of the string without using the #define, but that is considered very bad form, and is discouraged.

The way the tag is placed on the string is with C string concatenation:
printk(KERN_DEBUG "This is a debug message\n");

Note that there is no comma between the tag macro and the message string.

=== Adding timing information ===
Sometimes, it is useful to add timing information to the printk values, so you can see when a particular event occurred. The kernel includes an feature for doing
this called printk times.

See the help for CONFIG_PRINTK_TIMES in the file lib/Kconfig.debug for more information on this feature. This option is found on the "Kernel Hacking" menu when configuring the kernel.

The timestamps which are inserted into the printk output consist of seconds and microseconds, as absolute values from the start of machine operation (or from the start of kernel timekeeping).

There is also tool in the kernel source which will convert the timestamp values to relative values (so you can see the interval between events). This tools is called show_delta and is located in the kernel 'scripts' directory.

See [[Printk Times]] for more information.

=== Viewing log messages ===
The <tt>klogd</tt> program will extract messages from the kernel log buffer, and send them to the system log (which winds up in /var/log/messages on most systems). This command runs in the background on most desktop or server systems, and continually transfers messages from the kernel log buffer to the system log.

You can view the contents of the log buffer directly, using the <tt>dmesg</tt> command. Note that by default <tt>dmesg</tt> displays the messages from the buffer, but does not remove them. So this command can be run multiple times to view the kernel printk messages. See the dmesg man page for more things you can do with this tool.

=== Controlling console output ===
In order to have the kernel boot be less "noisy", or in order to boot more quickly, it is sometimes useful to control the amount of messages displayed to the console during boot. You can do this by setting the kernel log level at boot time via a kernel command line option. See the "loglevel=" argument in <tt>Documentation/kernel-parameters.txt</tt>.

You can turn off all messages using the kernel command line option "quiet". See [[Disable Console]] for information on how much time this can save at boot up.

Note that even if the log level is changed, or "quiet" is used, although the printk messages are not print to console, they are still entered into the log buffer, and they can still be extracted and displayed later using the <tt>dmesg</tt> command.

=== Changing the size of the printk buffer ===
The size of the buffer that printk writes to is referred to in the kernel source as the "log buffer".

You can make the log buffer larger when you build the kernel. This is useful for when additional printk data (such as extra messages inserted for debugging) is produced by the kernel and it overflows the regular buffer. The log buffer is a ring buffer, so later messages will overwrite the earliest messages in the buffer if too much data is written before the log buffer is drained (by, e.g. klogd or dmesg)

Step 1: Change the kernel log buffer size:

On the General Setup menu, change "Kernel log buffer size" to a larger number. This corresponds to LOG_BUF_SHIFT, which is the power-of-two size of the log buffer.
By default, this is 14, which yeilds a log buffer size of 16KB. Changing this to 17 will yield a log buffer size of 128KB.

Set the option, then re-build and install the kernel for booting on the target board.

Step 2: Tell dmesg to retrieve the larger buffer:
Even though the buffer is larger in the kernel, dmesg will still default to only reading 16K, unless told otherwise. To read the full buffer, use the '-s' option
like so:
$dmesg -s 128000

== Using kernel symbols ==
You can look up the source code for a function address using your toolchain's addr2line program.
See [[Find a kernel function line]] or [[Addr2line for kernel debugging]].

== Using a kernel debugger ==
You can use the in-kernel debugger: [[KDB]]

Also, you can use QEMU and gdb (and a high-level IDE like eclipse).

See [http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu] for a great article on using Eclipse (with the CDT plugin) to debug the Linux kernel.

== Debugging early boot problems ==
See [[Debugging_by_printing#Debugging_early_boot_problems]]

== Triggering a kernel event ==
=== Overloading the sync system call ===
Sometimes, it is nice to trigger an event to happen in the kernel from user space. Instead of creating infrastructure to handle a /proc event, an ioctl() or making a new syscall, it can be quick and easy to just overload an existing function. One function not used very often is sync. (I have found that the sync system call is not normally called by user space programs (or during standard linux booting).

It is quite easy to put a hook to your own kernel program in the sys_sync() routine (located in fs/sync.c) and cause it to execute by issuing 'sync' from the shell command line. This is handy as a temporary mechanism to test things that you have put in the kernel.

[[Category:Development Tools]]

== Interpreting an Oops message ==
When the kernel encounters an internal fault, it will print an Oops message.
Here are some tips on using the Oops message to find the source of the problem.
* See [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/oops-tracing.txt;hb=HEAD Documentation/oops-tracing.txt]
* See [http://vmlinux.org/jocke/linux/howto-find-oops-location.shtml HOWTO find oops location] by Denis Vlasenko

== Compilation tricks for the kernel ==
Sometimes, you want to modify how the compiler builds an individual kernel file. The following are tips for doing tasks related to this.
=== Build an individual file ===
You can build an individual output object file, with:
make fs/buffer.o

This will build JUST fs/buffer.o (if it needs rebuilding) and not the entire kernel. To force it to need re-building, use 'touch' on the associated source file:
touch fs/buffer.c

=== Create the preprocessed file for an individual source file ===
Using the same technique, you can create the preprocessed file for a C source file. This is useful if you're having trouble tracking down macro expansion or where defines/prototypes are coming from exactly.
make fs/buffer.i

=== Create the assembly file for an individual source file ===
Using the same technique, you can create the assembly file for a C source file. This is useful to get an idea what actual machine instructions are generated from the C source code.
make fs/buffer.s

Another way to get the raw assembly, is to dump the object file using 'objdump'
objdump -d fs/buffer.o > fs/buffer.disassem

This will produce a disassembly of the object file, which should show how the assembly was translated into machine instructions.

If the object has been compiled with debug symbols (using '-g'), then you might get more information using the '-S' option with objdump:
objdump -S -d fs/buffer.o >fs/buffer.disassem

You can also request that the toolchain show mixed source and assembly, by passing extra flags:
make EXTRA_CFLAGS="-g -Wa,-a,-ad -fverbose-asm" fs/buffer.o >fs/buffer.mixed

=== Alter the flags for a compilation ===
Sometimes, you need to alter the compilation flags for an individual file. There are two ways to do this. One is to add the extra flags on the make command line:

make EXTRA_CFLAGS="-g -finstrument-functions" fs/buffer.o

This will work if the flags can be appended to the regular set of C flags used for compiling the object.

However, if you need to do something more complicated, like removing or modifying flags, then you can build your own command line by hand. To do this, it is easiest to have 'make' produce the default compilation command (which will be several lines long), then copy, paste and edit it, to run on the command line directly. To see the exact compile commands used to compile a particular object, use the V=1 option with the kernel build system:

make V=1 fs/buffer.o

For me, this produced something like this:

mipsel-linux-gcc -Wp,-MD,fs/.buffer.o.d -nostdinc -isystem /home/usr/local/mipsel-linux-glibc/bin/../lib/gcc/mipsisa32el-linux/3.4.3/include -D__KERNEL__ -Iinclude -Iinclude2 -I/home/tbird/work/linux/include -I/home/tbird/work/linux/fs -Ifs -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -O2 -fomit-frame-pointer -g -I/home/tbird/work/linux/ -I /home/tbird/work/linux/include/asm/gcc -G 0 -mno-abicalls -fno-pic -pipe -finline-limit=100000 -mabi=32 -march=mips32r2 -Wa,-32 -Wa,-march=mips32r2 -Wa,-mips32r2 -Wa,--trap -I/home/tbird/work/linux/include/asm-mips/ati -Iinclude/asm-mips/ati -I/home/tbird/work/linux/include/asm-mips/mach-generic -Iinclude/asm-mips/mach-generic -Wdeclaration-after-statement -DKBUILD_BASENAME=buffer -DKBUILD_MODNAME=buffer -c -o fs/buffer.o /home/tbird/work/linux/fs/buffer.c

Kernel Debugging Tips

2012-04-03T20:44:35Z

Peter Huewe: Undo revision 113000 by Peter Huewe (talk)

Here are some miscellaneous tips for debugging a kernel:

== Using printks ==
To add your own debug message to the kernel, you can place a "printk()" in the kernel code.

These messages are logged to an internal buffer, and eventually displayed by the kernel, being emitted to whatever is the active console, at the time of display. Since the console is not initialized until part-way through the boot sequence, several messages build up in the log until the console is initialize, at which point all buffered messages are displayed at once.

After the messages are displayed on the kernel console, they are left in the log buffer (which is a ring buffer), and they can be (and usually are) retrieved later by various commands.

printk is a variable-argument function, just like printf. Note that there is no "printf()" function for the kernel. "printk()" is used to specifically remind kernel developers that the environment is different.

printk can use many standard C-style printf format characters, but not all of them. See the kernel source for examples of what you can do.

=== Log levels ===
Each kernel message can be pre-pended with a tag indicating the importance of the message. The message tags are shown in the file include/linux/kernel.h. The available tags are:
KERN_EMERG, KERN_ALERT, KERN_CRIT, KERN_WARNING, KERN_NOTICE, KERN_INFO, KERN_DEBUG.

Note that these tags are defined as simple strings, with angle brackets and a number:
#define KERN_INFO "<6>"

You could place the tag onto front of the string without using the #define, but that is considered very bad form, and is discouraged.

The way the tag is placed on the string is with C string concatenation:
printk(KERN_DEBUG "This is a debug message\n");

Note that there is no comma between the tag macro and the message string.

=== Adding timing information ===
Sometimes, it is useful to add timing information to the printk values, so you can see when a particular event occurred. The kernel includes an feature for doing
this called printk times.

See the help for CONFIG_PRINTK_TIMES in the file lib/Kconfig.debug for more information on this feature. This option is found on the "Kernel Hacking" menu when configuring the kernel.

The timestamps which are inserted into the printk output consist of seconds and microseconds, as absolute values from the start of machine operation (or from the start of kernel timekeeping).

There is also tool in the kernel source which will convert the timestamp values to relative values (so you can see the interval between events). This tools is called show_delta and is located in the kernel 'scripts' directory.

See [[Printk Times]] for more information.

=== Viewing log messages ===
The <tt>klogd</tt> program will extract messages from the kernel log buffer, and send them to the system log (which winds up in /var/log/messages on most systems). This command runs in the background on most desktop or server systems, and continually transfers messages from the kernel log buffer to the system log.

You can view the contents of the log buffer directly, using the <tt>dmesg</tt> command. Note that by default <tt>dmesg</tt> displays the messages from the buffer, but does not remove them. So this command can be run multiple times to view the kernel printk messages. See the dmesg man page for more things you can do with this tool.

=== Controlling console output ===
In order to have the kernel boot be less "noisy", or in order to boot more quickly, it is sometimes useful to control the amount of messages displayed to the console during boot. You can do this by setting the kernel log level at boot time via a kernel command line option. See the "loglevel=" argument in <tt>Documentation/kernel-parameters.txt</tt>.

You can turn off all messages using the kernel command line option "quiet". See [[Disable Console]] for information on how much time this can save at boot up.

Note that even if the log level is changed, or "quiet" is used, although the printk messages are not print to console, they are still entered into the log buffer, and they can still be extracted and displayed later using the <tt>dmesg</tt> command.

=== Changing the size of the printk buffer ===
The size of the buffer that printk writes to is referred to in the kernel source as the "log buffer".

You can make the log buffer larger when you build the kernel. This is useful for when additional printk data (such as extra messages inserted for debugging) is produced by the kernel and it overflows the regular buffer. The log buffer is a ring buffer, so later messages will overwrite the earliest messages in the buffer if too much data is written before the log buffer is drained (by, e.g. klogd or dmesg)

Step 1: Change the kernel log buffer size:

On the General Setup menu, change "Kernel log buffer size" to a larger number. This corresponds to LOG_BUF_SHIFT, which is the power-of-two size of the log buffer.
By default, this is 14, which yeilds a log buffer size of 16KB. Changing this to 17 will yield a log buffer size of 128KB.

Set the option, then re-build and install the kernel for booting on the target board.

Step 2: Tell dmesg to retrieve the larger buffer:
Even though the buffer is larger in the kernel, dmesg will still default to only reading 16K, unless told otherwise. To read the full buffer, use the '-s' option
like so:
$dmesg -s 128000

== Using kernel symbols ==
You can look up the source code for a function address using your toolchain's addr2line program.
See [[Find a kernel function line]] or [[Addr2line for kernel debugging]].

== Using a kernel debugger ==
You can use the in-kernel debugger: [[KDB]]

Also, you can use QEMU and gdb (and a high-level IDE like eclipse).

See [http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu] for a great article on using Eclipse (with the CDT plugin) to debug the Linux kernel.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

1) using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.

2) figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== Triggering a kernel event ==
=== Overloading the sync system call ===
Sometimes, it is nice to trigger an event to happen in the kernel from user space. Instead of creating infrastructure to handle a /proc event, an ioctl() or making a new syscall, it can be quick and easy to just overload an existing function. One function not used very often is sync. (I have found that the sync system call is not normally called by user space programs (or during standard linux booting).

It is quite easy to put a hook to your own kernel program in the sys_sync() routine (located in fs/sync.c) and cause it to execute by issuing 'sync' from the shell command line. This is handy as a temporary mechanism to test things that you have put in the kernel.

[[Category:Development Tools]]

== Interpreting an Oops message ==
When the kernel encounters an internal fault, it will print an Oops message.
Here are some tips on using the Oops message to find the source of the problem.
* See [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/oops-tracing.txt;hb=HEAD Documentation/oops-tracing.txt]
* See [http://vmlinux.org/jocke/linux/howto-find-oops-location.shtml HOWTO find oops location] by Denis Vlasenko

== Compilation tricks for the kernel ==
Sometimes, you want to modify how the compiler builds an individual kernel file. The following are tips for doing tasks related to this.
=== Build an individual file ===
You can build an individual output object file, with:
make fs/buffer.o

This will build JUST fs/buffer.o (if it needs rebuilding) and not the entire kernel. To force it to need re-building, use 'touch' on the associated source file:
touch fs/buffer.c

=== Create the preprocessed file for an individual source file ===
Using the same technique, you can create the preprocessed file for a C source file. This is useful if you're having trouble tracking down macro expansion or where defines/prototypes are coming from exactly.
make fs/buffer.i

=== Create the assembly file for an individual source file ===
Using the same technique, you can create the assembly file for a C source file. This is useful to get an idea what actual machine instructions are generated from the C source code.
make fs/buffer.s

Another way to get the raw assembly, is to dump the object file using 'objdump'
objdump -d fs/buffer.o > fs/buffer.disassem

This will produce a disassembly of the object file, which should show how the assembly was translated into machine instructions.

If the object has been compiled with debug symbols (using '-g'), then you might get more information using the '-S' option with objdump:
objdump -S -d fs/buffer.o >fs/buffer.disassem

You can also request that the toolchain show mixed source and assembly, by passing extra flags:
make EXTRA_CFLAGS="-g -Wa,-a,-ad -fverbose-asm" fs/buffer.o >fs/buffer.mixed

=== Alter the flags for a compilation ===
Sometimes, you need to alter the compilation flags for an individual file. There are two ways to do this. One is to add the extra flags on the make command line:

make EXTRA_CFLAGS="-g -finstrument-functions" fs/buffer.o

This will work if the flags can be appended to the regular set of C flags used for compiling the object.

However, if you need to do something more complicated, like removing or modifying flags, then you can build your own command line by hand. To do this, it is easiest to have 'make' produce the default compilation command (which will be several lines long), then copy, paste and edit it, to run on the command line directly. To see the exact compile commands used to compile a particular object, use the V=1 option with the kernel build system:

make V=1 fs/buffer.o

For me, this produced something like this:

mipsel-linux-gcc -Wp,-MD,fs/.buffer.o.d -nostdinc -isystem /home/usr/local/mipsel-linux-glibc/bin/../lib/gcc/mipsisa32el-linux/3.4.3/include -D__KERNEL__ -Iinclude -Iinclude2 -I/home/tbird/work/linux/include -I/home/tbird/work/linux/fs -Ifs -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -O2 -fomit-frame-pointer -g -I/home/tbird/work/linux/ -I /home/tbird/work/linux/include/asm/gcc -G 0 -mno-abicalls -fno-pic -pipe -finline-limit=100000 -mabi=32 -march=mips32r2 -Wa,-32 -Wa,-march=mips32r2 -Wa,-mips32r2 -Wa,--trap -I/home/tbird/work/linux/include/asm-mips/ati -Iinclude/asm-mips/ati -I/home/tbird/work/linux/include/asm-mips/mach-generic -Iinclude/asm-mips/mach-generic -Wdeclaration-after-statement -DKBUILD_BASENAME=buffer -DKBUILD_MODNAME=buffer -c -o fs/buffer.o /home/tbird/work/linux/fs/buffer.c

Kernel Debugging Tips

2012-04-03T20:43:50Z

Peter Huewe: Moved to Debugging by Printing

Here are some miscellaneous tips for debugging a kernel:

== Using printks ==
To add your own debug message to the kernel, you can place a "printk()" in the kernel code.

These messages are logged to an internal buffer, and eventually displayed by the kernel, being emitted to whatever is the active console, at the time of display. Since the console is not initialized until part-way through the boot sequence, several messages build up in the log until the console is initialize, at which point all buffered messages are displayed at once.

After the messages are displayed on the kernel console, they are left in the log buffer (which is a ring buffer), and they can be (and usually are) retrieved later by various commands.

printk is a variable-argument function, just like printf. Note that there is no "printf()" function for the kernel. "printk()" is used to specifically remind kernel developers that the environment is different.

printk can use many standard C-style printf format characters, but not all of them. See the kernel source for examples of what you can do.

=== Log levels ===
Each kernel message can be pre-pended with a tag indicating the importance of the message. The message tags are shown in the file include/linux/kernel.h. The available tags are:
KERN_EMERG, KERN_ALERT, KERN_CRIT, KERN_WARNING, KERN_NOTICE, KERN_INFO, KERN_DEBUG.

Note that these tags are defined as simple strings, with angle brackets and a number:
#define KERN_INFO "<6>"

You could place the tag onto front of the string without using the #define, but that is considered very bad form, and is discouraged.

The way the tag is placed on the string is with C string concatenation:
printk(KERN_DEBUG "This is a debug message\n");

Note that there is no comma between the tag macro and the message string.

=== Adding timing information ===
Sometimes, it is useful to add timing information to the printk values, so you can see when a particular event occurred. The kernel includes an feature for doing
this called printk times.

See the help for CONFIG_PRINTK_TIMES in the file lib/Kconfig.debug for more information on this feature. This option is found on the "Kernel Hacking" menu when configuring the kernel.

The timestamps which are inserted into the printk output consist of seconds and microseconds, as absolute values from the start of machine operation (or from the start of kernel timekeeping).

There is also tool in the kernel source which will convert the timestamp values to relative values (so you can see the interval between events). This tools is called show_delta and is located in the kernel 'scripts' directory.

See [[Printk Times]] for more information.

=== Viewing log messages ===
The <tt>klogd</tt> program will extract messages from the kernel log buffer, and send them to the system log (which winds up in /var/log/messages on most systems). This command runs in the background on most desktop or server systems, and continually transfers messages from the kernel log buffer to the system log.

You can view the contents of the log buffer directly, using the <tt>dmesg</tt> command. Note that by default <tt>dmesg</tt> displays the messages from the buffer, but does not remove them. So this command can be run multiple times to view the kernel printk messages. See the dmesg man page for more things you can do with this tool.

=== Controlling console output ===
In order to have the kernel boot be less "noisy", or in order to boot more quickly, it is sometimes useful to control the amount of messages displayed to the console during boot. You can do this by setting the kernel log level at boot time via a kernel command line option. See the "loglevel=" argument in <tt>Documentation/kernel-parameters.txt</tt>.

You can turn off all messages using the kernel command line option "quiet". See [[Disable Console]] for information on how much time this can save at boot up.

Note that even if the log level is changed, or "quiet" is used, although the printk messages are not print to console, they are still entered into the log buffer, and they can still be extracted and displayed later using the <tt>dmesg</tt> command.

=== Changing the size of the printk buffer ===
The size of the buffer that printk writes to is referred to in the kernel source as the "log buffer".

You can make the log buffer larger when you build the kernel. This is useful for when additional printk data (such as extra messages inserted for debugging) is produced by the kernel and it overflows the regular buffer. The log buffer is a ring buffer, so later messages will overwrite the earliest messages in the buffer if too much data is written before the log buffer is drained (by, e.g. klogd or dmesg)

Step 1: Change the kernel log buffer size:

On the General Setup menu, change "Kernel log buffer size" to a larger number. This corresponds to LOG_BUF_SHIFT, which is the power-of-two size of the log buffer.
By default, this is 14, which yeilds a log buffer size of 16KB. Changing this to 17 will yield a log buffer size of 128KB.

Set the option, then re-build and install the kernel for booting on the target board.

Step 2: Tell dmesg to retrieve the larger buffer:
Even though the buffer is larger in the kernel, dmesg will still default to only reading 16K, unless told otherwise. To read the full buffer, use the '-s' option
like so:
$dmesg -s 128000

== Using kernel symbols ==
You can look up the source code for a function address using your toolchain's addr2line program.
See [[Find a kernel function line]] or [[Addr2line for kernel debugging]].

== Using a kernel debugger ==
You can use the in-kernel debugger: [[KDB]]

Also, you can use QEMU and gdb (and a high-level IDE like eclipse).

See [http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu] for a great article on using Eclipse (with the CDT plugin) to debug the Linux kernel.

== Triggering a kernel event ==
=== Overloading the sync system call ===
Sometimes, it is nice to trigger an event to happen in the kernel from user space. Instead of creating infrastructure to handle a /proc event, an ioctl() or making a new syscall, it can be quick and easy to just overload an existing function. One function not used very often is sync. (I have found that the sync system call is not normally called by user space programs (or during standard linux booting).

It is quite easy to put a hook to your own kernel program in the sys_sync() routine (located in fs/sync.c) and cause it to execute by issuing 'sync' from the shell command line. This is handy as a temporary mechanism to test things that you have put in the kernel.

[[Category:Development Tools]]

== Interpreting an Oops message ==
When the kernel encounters an internal fault, it will print an Oops message.
Here are some tips on using the Oops message to find the source of the problem.
* See [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/oops-tracing.txt;hb=HEAD Documentation/oops-tracing.txt]
* See [http://vmlinux.org/jocke/linux/howto-find-oops-location.shtml HOWTO find oops location] by Denis Vlasenko

== Compilation tricks for the kernel ==
Sometimes, you want to modify how the compiler builds an individual kernel file. The following are tips for doing tasks related to this.
=== Build an individual file ===
You can build an individual output object file, with:
make fs/buffer.o

This will build JUST fs/buffer.o (if it needs rebuilding) and not the entire kernel. To force it to need re-building, use 'touch' on the associated source file:
touch fs/buffer.c

=== Create the preprocessed file for an individual source file ===
Using the same technique, you can create the preprocessed file for a C source file. This is useful if you're having trouble tracking down macro expansion or where defines/prototypes are coming from exactly.
make fs/buffer.i

=== Create the assembly file for an individual source file ===
Using the same technique, you can create the assembly file for a C source file. This is useful to get an idea what actual machine instructions are generated from the C source code.
make fs/buffer.s

Another way to get the raw assembly, is to dump the object file using 'objdump'
objdump -d fs/buffer.o > fs/buffer.disassem

This will produce a disassembly of the object file, which should show how the assembly was translated into machine instructions.

If the object has been compiled with debug symbols (using '-g'), then you might get more information using the '-S' option with objdump:
objdump -S -d fs/buffer.o >fs/buffer.disassem

You can also request that the toolchain show mixed source and assembly, by passing extra flags:
make EXTRA_CFLAGS="-g -Wa,-a,-ad -fverbose-asm" fs/buffer.o >fs/buffer.mixed

=== Alter the flags for a compilation ===
Sometimes, you need to alter the compilation flags for an individual file. There are two ways to do this. One is to add the extra flags on the make command line:

make EXTRA_CFLAGS="-g -finstrument-functions" fs/buffer.o

This will work if the flags can be appended to the regular set of C flags used for compiling the object.

However, if you need to do something more complicated, like removing or modifying flags, then you can build your own command line by hand. To do this, it is easiest to have 'make' produce the default compilation command (which will be several lines long), then copy, paste and edit it, to run on the command line directly. To see the exact compile commands used to compile a particular object, use the V=1 option with the kernel build system:

make V=1 fs/buffer.o

For me, this produced something like this:

mipsel-linux-gcc -Wp,-MD,fs/.buffer.o.d -nostdinc -isystem /home/usr/local/mipsel-linux-glibc/bin/../lib/gcc/mipsisa32el-linux/3.4.3/include -D__KERNEL__ -Iinclude -Iinclude2 -I/home/tbird/work/linux/include -I/home/tbird/work/linux/fs -Ifs -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -O2 -fomit-frame-pointer -g -I/home/tbird/work/linux/ -I /home/tbird/work/linux/include/asm/gcc -G 0 -mno-abicalls -fno-pic -pipe -finline-limit=100000 -mabi=32 -march=mips32r2 -Wa,-32 -Wa,-march=mips32r2 -Wa,-mips32r2 -Wa,--trap -I/home/tbird/work/linux/include/asm-mips/ati -Iinclude/asm-mips/ati -I/home/tbird/work/linux/include/asm-mips/mach-generic -Iinclude/asm-mips/mach-generic -Wdeclaration-after-statement -DKBUILD_BASENAME=buffer -DKBUILD_MODNAME=buffer -c -o fs/buffer.o /home/tbird/work/linux/fs/buffer.c

Debugging by printing

2012-04-03T20:42:26Z

Peter Huewe: Moved Debugging Early boot problems from Debugging tipps here.

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

The supported format strings are also quite similar to the standard ones
supported in printf and are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== Debugging early boot problems ==
=== Accessing the printk buffer after a silent hang on boot ===
Sometimes, if the kernel hangs early in the boot process, you get no messages on the console before the hang. However, there may still be messages in the printk buffer, which can give you an idea of where the problem is.

The kernel starts putting messages into the printk buffer as soon as it starts. They stay buffered there until the console code has a chance to initialize the console device (often the serial port for embedded devices). Even though these messages are not printed before the hang, it is still possible in some circumstances
to dump the printk buffer and see the messages.

The tricky parts of doing this are:

# using a warm reset (if possible) so the memory contents are not lost when transitioning from the stuck kernel to the bootloader. You ''can'' do a cold boot, but if the memory is left unpowered for very long, you will start to see memory corruption.
# figuring out the address to use in the bootloader, based on the address of __log_buf in System.map. You will probably need to subtract the value of CONFIG_PAGE_OFFSET from the __log_buf address. However, there may be other offsets present as well (such as TEXT_OFFSET). Sometimes you can find the buffer by dumping the memory in a suspected area and locating the kernel messages visually in the dump. Note that the mapping offset between the kernel map of memory and the bootloader map of memory should not change. So once you figure it out you are set.

Quinn Jensen writes:

Something I've found handy when the console is silent is to dump the printk buffer from the boot loader. To do it you have to know how your boot loader maps memory compared to the kernel.

==== Redboot example on a Freescale ADS board ====
Quinn says: Here's what I do with Redboot on i.MX31:

fgrep printk_buf System.map

this shows the linked address of the printk_buf, e.g.:

c02338f0 b printk_buf.16194

The address "c02338f0" is in kernel virtual, which, in the case of i.MX31 ADS, redboot will have mapped to 0x802338f0. So, after resetting the target board, but without letting it try to boot again, at the redboot prompt:

dump -b 0x802338f0 -l 10000

And you see the printk buffer that never got flushed to the UART. Knowing what's there can be '''very''' useful in debugging your console.

==== U-boot example on an OMAP OSK board ====
Tim Bird tried these steps and they worked:

grep __log_buf System.map

or

grep __log_buf /proc/kallsyms

These show:

c0352d88 B __log_buf

In the case of the OSK, this address maps to 0x10352d88. So I reset the target board and use the following:

OMAP5912 OSK # md 10352d88
10352d88: 4c3e353c 78756e69 72657620 6e6f6973 <5>Linux version
10352d98: 362e3220 2e32322e 612d3631 6e5f706c 2.6.22.16-alp_n
10352da8: 7428206c 64726962 6d697440 6b736564 l (tbird@timdesk
10352db8: 2e6d612e 796e6f73 6d6f632e 67282029 .am.sony.com) (g
10352dc8: 76206363 69737265 33206e6f 342e342e cc version 3.4.4
10352dd8: 34232029 45525020 54504d45 65755420 ) #4 PREEMPT Tue
...

==== Grub ====
Grub also supports a dump command which you can invoke from the grub prompt.
dump [ -s offset ] [-n length] { FILE | (mem) }

=== Using CONFIG_DEBUG_LL and printascii() (ARM only) ===
If the kernel fails before the serial console is enabled, you can use CONFIG_DEBUG_LL to add extra debug output routines to the kernel. These are printascii, printch and printhex. These routines print directly to the serial port, bypassing the console code, and are available earlier in machine initialization.

Here is an e-mail exchange seen on the linux-embedded mailing list (with answer by George Davis):[http://www.mail-archive.com/linux-embedded@vger.kernel.org/msg00223.html]
<pre>
> When we try to boot a 2.6.21 kernel after uncompressing the kernel the boot process dies somehow.
> We've figured out so far that the kernel dies somewhere between the gunzip and start_kernel.

Try enabling DEBUG_LL to see if it's an machine ID error. If you see:

Error: unrecognized/unsupported processor variant.

or:

Error: unrecognized/unsupported machine ID...

Then you either don't have proper processor support enabled for your target or your bootloader is passing in the wrong machine number.

If you still don't see anything, try hacking printk.c to call printascii() (enabled for the DEBUG_LL case) to print directly to the serial port w/o a driver, etc.,. You can find more details on these low-level debugging hacks via a little googling...
</pre>

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* []http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

Debugging by printing

2012-04-03T20:11:55Z

Peter Huewe: /* References and external links */ Added LWN Link

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

The supported format strings are also quite similar to the standard ones
supported in printf and are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

You can also specify an argument on the kernel command line to enable this, or
you can enable it any time at runtime by doing the following:

$echo 1 >/sys/module/printk/parameters/time

Also, there are tools available to use the information to show relative times
between printks (scripts/show_delta) and create graphs of durations in the
kernel (scripts/bootgraph.pl)

See [[Printk Times]] for more details

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
* []http://lwn.net/Articles/487437/ LWN.net: The perils of pr_info()]
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]

MagicSysRq

2012-03-28T21:04:28Z

Peter Huewe:

----
'''Note''': This article is currently only a draft and is a part of a series of articles I'm going to publish the next few months - if you want to contribute to it, please feel free to. However it would be nice if you could coordinate your efforts with [[User:Peter_Huewe|me]]

This article is still work in progress and still needs a lot of effort.
----
== Introduction ==
MagicSysRq is a special key combination which lets the user perform certain
low level tasks of the kernel and is especially useful when the whole system
seems to be hung.
MagicSysRq e.g. makes it possible to, more or less, cleanly shutdown a system,
which doesn't even respond to CTRL+ALT+DEL any more.

The MagicSysRq functions are invoked by pressing ALT+SysRq+Function Key where
function key describes one of the following.

TODO:Insert table here

==Example==

The probably most used combination of MagicSysRq functions is the one RSEIUB,
which can be easyly remebered by the mnemonic trick ``Raising Elephants is so
utterly boring'' \footnote{``You can still find several references the
mnemonic ''Raising Skinny Elephants is utterly boring'' on the internet -
however it is probably better to do the sync after the tasks exited/were
killed}
- although we're dealing with penguins here ;)
According to the table above this performs, in order:
R Put Keyboard in Raw Mode
E terminate all tasks
I kill all tasks
S Sync
U remount all filesystems read-only
B Reboot

== Troubleshooting ==
You can easily and safely check whether MagicSysRq works on your System by
pressing
Alt+SysRq+H - on a console you should now see a short help text of the
MagicSysRq Feature or at least print this to `dmesg`.

If you don't see the help text, check whether CONFIG\_MAGIC\_SYSRQ=y is set in
your Kernel Config.
If it is set, please check the output of 'cat /proc/sys/kernel/sysrq' which
should report 1 if MagicSysRq is enabled. To enable it just do a simple echo 1
> /proc/sys/kernel/sysrq

== Important Files ==
/proc/sys/kernel/sysrq - enable/disable MagicSysRq at runtime
/proc/sysrq-trigger - trigger MagicSysRq from the console (e.g. useful over
ssh)

TODO Content:
* Explanation of remaining keys
* using it over serial line -> BREAK+key, in kermit ctrl-\ b KEY
* Interaction with KDB
* Interaction with kexec / crashkernel

== External links ==
* [http://en.wikipedia.org/wiki/Magic_SysRq_key Wikipedia Article about Magic_SysRq_key]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/sysrq.txt;hb=HEAD Kernel Documentation/sysrq.txt]

[[Category:Development Tools]]
[[Category:Tips and Tricks]]

MagicSysRq

2012-03-28T21:03:41Z

Peter Huewe:

----
'''Note''': This article is currently only a draft and is a part of a series of articles I'm going to publish the next few months - if you want to contribute to it, please feel free to. However it would be nice if you could coordinate your efforts with [[User:Peter_Huewe|me]
This article is still work in progress and still needs a lot of effort.
]
----
== Introduction ==
MagicSysRq is a special key combination which lets the user perform certain
low level tasks of the kernel and is especially useful when the whole system
seems to be hung.
MagicSysRq e.g. makes it possible to, more or less, cleanly shutdown a system,
which doesn't even respond to CTRL+ALT+DEL any more.

The MagicSysRq functions are invoked by pressing ALT+SysRq+Function Key where
function key describes one of the following.

TODO:Insert table here

==Example==

The probably most used combination of MagicSysRq functions is the one RSEIUB,
which can be easyly remebered by the mnemonic trick ``Raising Elephants is so
utterly boring'' \footnote{``You can still find several references the
mnemonic ''Raising Skinny Elephants is utterly boring'' on the internet -
however it is probably better to do the sync after the tasks exited/were
killed}
- although we're dealing with penguins here ;)
According to the table above this performs, in order:
R Put Keyboard in Raw Mode
E terminate all tasks
I kill all tasks
S Sync
U remount all filesystems read-only
B Reboot

== Troubleshooting ==
You can easily and safely check whether MagicSysRq works on your System by
pressing
Alt+SysRq+H - on a console you should now see a short help text of the
MagicSysRq Feature or at least print this to `dmesg`.

If you don't see the help text, check whether CONFIG\_MAGIC\_SYSRQ=y is set in
your Kernel Config.
If it is set, please check the output of 'cat /proc/sys/kernel/sysrq' which
should report 1 if MagicSysRq is enabled. To enable it just do a simple echo 1
> /proc/sys/kernel/sysrq

== Important Files ==
/proc/sys/kernel/sysrq - enable/disable MagicSysRq at runtime
/proc/sysrq-trigger - trigger MagicSysRq from the console (e.g. useful over
ssh)

TODO Content:
* Explanation of remaining keys
* using it over serial line -> BREAK+key, in kermit ctrl-\ b KEY
* Interaction with KDB
* Interaction with kexec / crashkernel

== External links ==
* [http://en.wikipedia.org/wiki/Magic_SysRq_key Wikipedia Article about Magic_SysRq_key]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/sysrq.txt;hb=HEAD Kernel Documentation/sysrq.txt]

[[Category:Development Tools]]
[[Category:Tips and Tricks]]

User:Peter Huewe

2012-03-18T17:53:19Z

Peter Huewe:

Thanks for visiting my page.

I'm a 26 year old Embedded Linux Software Developer from Germany.

Don't hesitate to contact me if you have any questions.
I'm always open to new ideas and interesting projects.

Contact:
Email: broeggle <at> scheissprovider <dot> de
PeterHuewe <at> linux <dot> com

Jabber: broeggle <at> jabber <dot> ccc <dot> de

ICQ : 137 1111 59

List of articles I'm currently working on / will be working on:
* [[Debugging by printing]]
* [[Debugging by querying]]
* [[Debugging OOPS]]
* [[JTAG]]
* [[MagicSysRq]]
* [[gcov]]
* [[profiling]]
* [[tracing]]
* [[Kernel Debug Support]]
* [[KGDB]]\[[KDB]]
* [[FTrace]] -> [[kernelshark]]
* [[LTTng]]
* [[oprofile]]
* [[System Tap]]
* [[Kprobes]]
* [[perf]]

* [[Debugging Portal]]

Please feel free to help me ;)

Currently missing articles on elinux.org
[[Coreboot]]

BeaconBoard

2012-03-18T17:51:21Z

Peter Huewe: /* RGB LED */ Added RGB Led driver for Kernel 3.2 for Beagle and Pandaboard

= Description =

Beacon Board is a small accessory board that is primarily intended for use with the [[PandaBoard]] or [[BeagleBoard]] however it can be wired into any development platform that supports GPIOs

[[File:BeaconBoard.jpg|400px]]

= Sponsors =

The Beacon Board is sponsored by:

* [http://tincantools.com TinCanTools]
* [http://boardzoo.com BoardZoo]
* Designed by [[User:Prpplague| David Anders]]
* Layout by Chris Auten
* Assembly by [http://www.circuitco.com/ CircuitCo]

= Features =

* Single tricolored RGB LED ([[media:rgb-led.pdf|datasheet]])
* WS2801 RGB LED PWM controller ([[media:ws2801.pdf|datasheet]])
* 74HC595 Serial Shift Register ([[media:74hc595.pdf|datasheet]])
* 7 Segment Display ([[media:7seg-display.pdf|datasheet]])
* 74AVC8T245 Level Shifter ([[media:74AVC8T245.pdf|datasheet]])
* I2C EEPROM ([[media:cat24aa01.pdf|datasheet]])

= References =

* [http://www.protostack.com/blog/2011/06/atmega168a-pulse-width-modulation-pwm/ PWM Tutorial]
* [http://conductiveresistance.com/interactive-595-shift-register-simulator/ 595 Tutorial]
* [http://en.wikipedia.org/wiki/Seven-segment_display_character_representations 7 Segment Display Character Mapping]

= Projects =
* [http://youtu.be/jP0X9_z2pTQ 7 Segment Display Video]

= Design =

* [[media:beacon-revb1.pdf|Schematic]]

= Interface =
<br>

== Connectors ==
* Male Header Major Leage TSHS-114-D-06-A-T-LF
* Female Header Major League LSWSS-114-D-02-T-LF
* Digikey Part - http://search.digikey.com/us/en/products/1-534998-4/A26493-ND/352964

== 7 Segment Display ==
{|border=1
!Pin
!Function
!Beagle
!Panda
!Comment
|-
|14
|CLOCK
|GPIO161
|GPIO139
|
|-
|16
|LATCH
|GPIO162
|GPIO137
|
|-
|10
|DATA
|GPIO145
|GPIO138
|
|}
<br>

== RGB LED ==
{|border=1
!Pin
!Function
!Beagle
!Panda
!Comment
|-
|12
|DATA
|GPIO158
|GPIO136
|
|-
|18
|CLOCK
|GPIO159
|GPIO135
|
|}
<br>

= Software =
== 7 Segment Display ==
* Driver for Linux Kernel 3.0
* Userspace Shell Script
** [[media:do7-panda.sh|PandaBoard for busybox ash]]
** [[media:do7-beagle.sh|BeagleBoard for bash]]

== RGB LED ==
* [[media:kernel-ws2801-driver.patch| WS2801 Driver for Linux Kernel 3.0]]
* [[media:0001-LEDS-add-initial-support-for-WS2801-controller-Panda.patch‎| WS2801 Driver for Linux Kernel 3.2 (PandaBoard)]]
* [[media:0001-LEDS-add-initial-support-for-WS2801-controller-Beagl.patch‎| WS2801 Driver for Linux Kernel 3.2 (BeagleBoard)]]
* Userspace shell script

== Angstrom ==

* [http://git.angstrom-distribution.org/cgi-bin/cgit.cgi/meta-texasinstruments/commit/?id=7bab703f4ac28a1982eb9b16eea97330579404c6 Commit info]

[[Category:BeagleBoard]]
[[Category:BeagleBoard Expansion Boards]]
[[Category:TinCanTools]]

== Projects using BeaconBoard ==

* [https://gitorious.org/av500/bacon-lunch Bacon Lunch] ([http://www.youtube.com/watch?v=OHhKzCKFINQ video])

File:0001-LEDS-add-initial-support-for-WS2801-controller-Panda.patch

2012-03-18T17:47:35Z

Peter Huewe: LEDS: add initial support for WS2801 controller (PandaBoard) This adds initial support for the WS2801 RGB LED controller on the beaconboard for the Panda Board

LEDS: add initial support for WS2801 controller (PandaBoard)

This adds initial support for the WS2801 RGB LED controller on the
beaconboard for the Panda Board

File:0001-LEDS-add-initial-support-for-WS2801-controller-Beagl.patch

2012-03-18T17:46:51Z

Peter Huewe: This adds initial support for the WS2801 RGB LED controller on the beaconboard for the BeagleBoard v2: - Make it compile on 3.2 and up

This adds initial support for the WS2801 RGB LED controller on the
beaconboard for the BeagleBoard

v2:
- Make it compile on 3.2 and up

Main Page

2012-03-18T16:08:29Z

Peter Huewe: Added Debugging Portal

{|
| nowrap style="vertical-align: top; font: bold xx-large sans-serif;" |
Embedded Linux Wiki
|}


Welcome to the eLinux wiki! The purpose of this wiki is to preserve and present information about the
development and use of Linux in embedded systems. To use this wiki, click on one of the portal links below.



== Development Portals ==
{| width="100%"
|-
|style="vertical-align:top" |
<div style="margin: 0; margin-top:10px; margin-right:10px; padding: 0 1em 1em 1em; align:right;">
<div style="margin-left: 25px;">

{|style="width: 100%; margin:0; padding:0; border-collapse: collapse;"
| style="width: 25%;" | [[image:Welly1.jpg|link=Boot Time]] [[Boot Time]]
| style="width: 25%;" | [[image:cfcard.jpg|link=Memory Management]] [[Memory Management]]
| style="width: 25%;" | [[image:Padlock2.jpg|link=Security]] [[Security]]
| style="width: 25%;" | [[image:Event.jpg|link=Events]] [[Events]]
|-
| [[image:Multimedia.png|link=Multimedia]] [[Multimedia]]
| [[image:Power.png|link=Power Management]] [[Power Management]]
| [[image:Blimp.jpg|link=System Size]] [[System Size]]
| [[image:Skull.jpg|link=Hardware Hacking]] [[Hardware Hacking]]
|-
| [[image:Filecab.jpg|link=File Systems]] [[File Systems]]
| [[image:clockface.jpg|link=Real Time]] [[Real Time]]
| [[image:Board2.jpg|link=Resource Management]] [[Resource Management]]
| [[image:Board.jpg|link=Development Platforms]] [[Development Platforms]]
|-
| [[image:Icon-network.png|link=:Category:Networking]] [[:Category:Networking|Networking]]
| [[image:Processor.png|link=Firmware]] [[Firmware]]
| [[image:justice-scales.png|link=Legal Issues]] [[Legal Issues]]
| [[image:Toolbox.jpg|link=Toolbox]] [[Toolbox]]
|-
| [[image:Android.png|link=Android Portal]] [[Android Portal]]
| [[image:icon-inspector2.png|link=Debugging Portal]] [[Debugging Portal]]
| [[image:Category-box.png|link=:Category:Categories]] [[:Category:Categories]]
| <br />
| <br />
|}

</div>
</div>

<div style="margin:0; margin-top:10px; margin-right:10px; border:1px solid #dfdfdf; padding:0 1em 1em 1em; background-color:#ffffcc; align:right; ">
'''ELC 2012 Presentations Page'''

Presentations can be added to the [[ELC_2012_Presentations | ELC 2012 Presentations]] page.
</div>



<div style="margin: 0; margin-top:10px; margin-right:10px; border: 1px solid #dfdfdf; padding: 0 1em 1em 1em; align:right;">
'''Embedded Linux Information'''
<div style="margin-left: 25px;">
{|style="width: 100%; margin:0; padding:0; border-collapse: collapse;"
|style="width: 20%;"|[[image:Productsico.jpg|link=Products]] [[Products]]
|style="width: 20%;"|[[image:Companysico.jpg|link=Companies]] [[Companies]]
|style="width: 20%;"|[[image:Vendorsico.jpg|link=Vendors]] [[Vendors]]
|style="width: 20%;"|[[image:Processorsico.png|link=Processors]] [[Processors]]
|-
|[[image:communityico.jpg|link=Community]] [[Community]]
|[[image:communityico.jpg|link=Experts]] [[Experts]]
|[[image:at-work.jpg|link=Jobs]] [[Jobs]]
|[[image:Companysico.jpg|link=Board and Chip Vendors]] [[Board and Chip Vendors]]
|}
</div>
</div>

== Technology Watch List ==
See the [[Technology Watch List]] for a list of interesting
projects and their current status in the Linux development
community.

== eLinux.org Information and Usage tips ==
The main portals of the site take you to lists of resources or collections of
information, you can use to tackle problems in the particular area referred to.
For example, if you have a problem with boot up time of your embedded Linux system,
click on "Boot Time".
You can also see a list of [[Special:Allpages|all the pages on this site]].

Please read [[ELinuxWiki:Policies & Guidelines|Policies & Guidelines]] for a detail list of site policies.

The following icons are for general information about this site, and related resources.
<div style="margin-left: 50px;">
{|style="width: 100%; margin:0; padding:0; border-collapse: collapse;"
|style="width: 25%;"|
[[image:mp_info.png|16px|link=Help:About]] [[Help:About|About]]<br/>
[[image:mp_help.png|16px|link=Help:Contents]] [[Help:Contents|Site Policy]]
|style="width: 25%;"|
[[image:mp_man.png|16px|link=Help:Editing]] [[Help:Editing|Editing Help]]<br/>
[[image:mp_wanted.png|16px|link=Wanted]] [[Wanted|Wanted Pages]]
|style="width: 25%;"|
[[image:mp_mail.png|16px|link=eLinuxWiki:Mailing List]] [[eLinuxWiki:Mailing List|Mailing Lists]]<br/>
[[image:mp_irc.png|16px|link=eLinuxWiki:Irc]] [[eLinuxWiki:Irc|IRC]]
|style="width: 25%;"|
[[image:mp_admin.png|16px|link=eLinuxWiki:Glossary]] [[eLinuxWiki:Glossary|Glossary]]<br/>
[[image:mp_twitter.png|16px|link=http://twitter.com/elinux]] [http://twitter.com/elinux Twitter Follow]
|}
</div>
We hope this information is helpful in your development tasks.

If you see something wrong, please change it. If you know something more about an issue, please
add it. Please [[volunteer editor tasks|help to extend]] this wiki. Thank you!

----
To experiment with this wiki try [[Sandbox]]. See the [http://meta.wikipedia.org/wiki/MediaWiki_User's_Guide User's Guide] for usage and configuration help.

[[Category:Community| ]]
[[Category:Categories| ]]

File:Icon-inspector2.png

2012-03-18T16:07:56Z

Peter Huewe: uploaded a new version of "File:Icon-inspector2.png": Icon created by Regional Computing Centre Erlangen (RRZE) and licenced under CC-BY-SA. Retrieved from http://rrze-icon-set.berlios.de/gallery.html#I

The image shows an Inspector with a magnifying glass.
Icon created by Regional Computing Centre Erlangen (RRZE) and licenced under CC-BY-SA.
Retrieved from http://rrze-icon-set.berlios.de/gallery.html#I

File:Icon-inspector2.png

2012-03-18T16:05:40Z

File:Icon-inspector2.png

2012-03-18T16:00:51Z

Peter Huewe: The image shows an Inspector with a magnifying glass. Icon created by Regional Computing Centre Erlangen (RRZE) and licenced under CC-BY-SA. Retrieved from http://rrze-icon-set.berlios.de/gallery.html#I

Debugging Portal

2012-03-18T15:49:15Z

Peter Huewe: fixed tables

This page is the new portal for all debugging related stuff.
It currently deals with Kernel topics, but feel free to add user space debugging too.

<div style="width=100%; background:#FFF;">
{| width="100%" cellspacing="5" cellpadding="0" valign="top" style="background:inherit;"
| valign="top" style="margin: 0 0 0.5em 1em; border:1px solid #aaa; text-align:left; width: 50%;" cellpadding="5" |
<div style="background: #eeeeee; border: 1px solid #2C547A; padding: 5px; margin: 3px; font-weight:bold;text-align:center;font-size:120%;">Kernel Debugging</div>
<div style="margin: 0 0 0 1em;">
;[[Debugging by printing]]
;[[Printk]]
;[[Kernel Debugging Tips]]
;[[Kgdb]]
;[[KDB]]
;[[Debugging The Linux Kernel Using Gdb]]
;[[MagicSysRq]]
;External Links:
*[http://tree.celinuxforum.org/CelfPubWiki/ELC2009Presentations?action=AttachFile&do=get&target=DebuggingWithJtagCelf2009.pdf Embedded Linux JTAG debugging (CELF presentation)]
*[http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu]

</div>
| valign="top" style="margin: 0 0 0.5em 1em; border:1px solid #aaa; text-align:left; width: 50%;" cellpadding="5"|
<div style="background: #eeeeee; border: 1px solid #2C547A; padding: 5px; margin: 3px; font-weight:bold;text-align:center;font-size:120%;">Kernel Tracing and Profiling</div>
<div style="margin: 0 0 0 1em;">
;[[System Tap]]
;[[Kernel Trace Systems]]
;[[Linux Trace Toolkit]]
;[[LTTng]]
;[[Ftrace]]
*[[Measuring Function Duration with FTrace]]
*[[Ftrace Function Graph ARM]]
;[[Using Kernel Function Trace]]
;[[Linux Kernel State Tracer]]

</div>
|}
</div>
[[Category:Development Tools]]
[[Category:Tips and Tricks]]

Debugging Portal

2012-03-18T15:44:38Z

Peter Huewe: Created page with "This page is the new portal for all debugging related stuff. It currently deals with Kernel topics, but feel free to add user space debugging too. <div style="width=100%; backgr..."

This page is the new portal for all debugging related stuff.
It currently deals with Kernel topics, but feel free to add user space debugging too.

<div style="width=100%; background:#FFF;">
{| width="100%" cellspacing="5" cellpadding="0" valign="top" style="background:inherit;"
| valign="top" style="margin: 0 0 0.5em 1em; border:1px solid #aaa; text-align:left; width: 33%;" cellpadding="5" |
<div style="background: #eeeeee; border: 1px solid #2C547A; padding: 5px; margin: 3px; font-weight:bold;text-align:center;font-size:120%;">Kernel Debugging</div>
<div style="margin: 0 0 0 1em;">
;[[Debugging by printing]]
;[[Printk]]
;[[Kernel Debugging Tips]]
;[[Kgdb]]
;[[KDB]]
;[[Debugging The Linux Kernel Using Gdb]]
;[[MagicSysRq]]
;External Links:
*[http://tree.celinuxforum.org/CelfPubWiki/ELC2009Presentations?action=AttachFile&do=get&target=DebuggingWithJtagCelf2009.pdf Embedded Linux JTAG debugging (CELF presentation)]
*[http://issaris.blogspot.com/2007/12/download-linux-kernel-sourcecode-from.html Debugging the Linux kernel using Eclipse/CDT and Qemu]

</div>
| valign="top" style="margin: 0 0 0.5em 1em; border:1px solid #aaa; text-align:left; width: 33%;" cellpadding="5"|
<div style="background: #eeeeee; border: 1px solid #2C547A; padding: 5px; margin: 3px; font-weight:bold;text-align:center;font-size:120%;">Kernel Tracing and Profiling</div>
<div style="margin: 0 0 0 1em;">
;[[System Tap]]
;[[Kernel Trace Systems]]
;[[Linux Trace Toolkit]]
;[[LTTng]]
;[[Ftrace]]
*[[Measuring Function Duration with FTrace]]
*[[Ftrace Function Graph ARM]]
;[[Using Kernel Function Trace]]
;[[Linux Kernel State Tracer]]

</div>

</div>

HP t5325

2012-03-17T03:52:10Z

Peter Huewe: Added short writeup of getting vanilla linux to work on hp t5325

== Vanilla Kernel on t5325 ==
In order to boot a vanilla kernel on the t5325 be sure to set these u-boot environment variables.
setenv arcNumber 2846
setenv mainlineLinux yes
saveenv
reset

Otherwise the board will simply hang during boot.
Please be sure to verify that the ''Machine'' is reported like this
[ 0.000000] Machine: HP t5325 Thin Client
and '''not'''
[ 0.000000] Machine: Marvell DB-88F6281-BP Development Board

Debugging by printing

2012-03-07T23:07:19Z

Peter Huewe: Updated article about debugging by printing

Probably the simplest way to get some debug information from your kernel code is by printing out various information with the kernel's equivalent of printf - the printk function and its derivatives.

== Usage ==
printk works more or less the same way as printf in userspace, so if you ever debugged your userspace program using printf, you are ready to do the same with your kernel code, e.g. by adding:
printk("My Debugger is Printk\n");

This wasn't that difficult, was it?

Usually you would print out some more interesting information like
printk("Var1 %d var2 %d\n", var1, var2);
just like in userspace.

In order to see the kernel messages, just use the
$ dmesg
command in one of your shells - this one will print out the whole kernel log
buffer to you.

The supported format strings are also quite similar to the standard ones
supported in printf and are quite extensively documented in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]

However please '''note''': always use ''%zu'', ''%zd'' or ''%zx'' for printing
''size_t'' and ''ssize_t'' values. ssize_t and size_t are quite common values in the kernel, so
please use the ''%z'' to avoid annoying compile warnings.

----
'''Author's practical advice:'''<br>
If you want to debug an oops (e.g caused by releasing a resource twice) in your driver and you don't have a clue where the oops happens, simply add this line
printk(KERN_ALERT "DEBUG: Passed %s %d \n",__FUNCTION__,__LINE__);
after each possibly offending statement.
Recompile and (re-)load the module and trigger the error condition - your log now shows you the last line that was successfully executed before the oops happened.

Of course you should '''remove''' these 'rude' statements before shipping your module ;)
----

== Log Levels ==
If you look into real kernel code you will always see something like:
printk(KERN_ERR "something went wrong, return code: %d\n",ret);

where "''KERN_ERR''" is one of the eight different log levels defined in
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] and specifes the severity of the error message.

Note that there is '''NO''' comma between the ''KERN_ERR'' and the format string (as the preprocessor concatenates both strings)

The log levels are:
{| class="wikitable"
|-
!Name
!String
!Meaning
!alias function
|-
|KERN_EMERG
|"<0>"
|Emergency messages, system is about to crash or is unstable
|pr_emerg
|-
|KERN_ALERT
|"<1>"
|Something bad happened and action must be taken immediately
|pr_alert
|-
|KERN_CRIT
|"<2>"
|A critical condition occurred like a serious hardware/software failure
|pr_crit
|-
|KERN_ERR
| "<3>"
|An error condition, often used by drivers to indicate difficulties with the hardware
|pr_err
|-
|KERN_WARNING
|"<4>"
|A warning, meaning nothing serious by itself but might indicate problems
|pr_warning
|-
|KERN_NOTICE
|"<5>"
|Nothing serious, but notably nevertheless. Often used to report security events.
|pr_notice
|-
|KERN_INFO
|"<6>"
|Informational message e.g. startup information at driver initialization
|pr_info
|-
|KERN_DEBUG
|"<7>"
|Debug messages
|pr_debug, pr_devel if DEBUG is defined
|}

The pr_* macros (with exception of pr_debug) are simple shorthand definitions
in ''include/linux/printk.h''
for their respective printk call and should probably be used in newer drivers.

''pr_devel'' and ''pr_debug'' are replaced with ''printk(KERN_DEBUG ...'' if the kernel was
compiled with ''DEBUG'', otherwise replaced with an empty statement.

For drivers the pr_debug should not be used anymore (use dev_dbg instead).

If you don't specify a log level in your message it defaults to
''DEFAULT_MESSAGE_LOGLEVEL'' (usually ''"<4>"''=''KERN_WARNING'')
which can be set via the ''CONFIG_DEFAULT_MESSAGE_LOGLEVEL'' kernel config
option (''make menuconfig-> Kernel Hacking -> Default message log level'')

The log level is used by the kernel to determine the importance of
a message and to decide whether it should be presented to the user
immediately, by printing it to the current console (where console could also
be a serial line or even a printer, not an xterm).

For this the kernel compares the log level of the message to the
''console_loglevel'' (a kernel variable) and if the priority is higher (i.e. a
lower value) than the ''console_loglevel'' the message will be printed to the current console.

To determine your current ''console_loglevel'' you simply enter:
$ cat /proc/sys/kernel/printk
7 4 1 7
current default minimum boot-time-default

The first integer shows you your current console_loglevel; the second the
default log level that you have seen above.

To change your current console_loglevel simply write to this file, so in order
to get all messages printed to the console do a simple
# echo 8 > /proc/sys/kernel/printk
and every kernel message will appear on your console.

Another way to change the console log level is to use ''dmesg'' with the ''-n''
parameter
# #set console_loglevel to print KERN_WARNING (4) or more severe messages
# dmesg -n 5
Only messages with a value lower ('''not''' lower equal) than the
console_loglevel will be printed.

You can also specify the console_loglevel at boot time using the ''loglevel''
boot parameter. (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=Documentation/kernel-parameters.txt;hb=HEAD#l1334 Documentation/kernel-parameters.txt] for more details)

----
'''Author's practical advice:'''<br>
Of course you should always specify an appropriate log level for your messages,
but for '''debugging''', I guess most developers leave the
console_loglevel unchanged and simply use KERN_ERR or KERN_CRIT to ensure the message reaches the console.

pr_err("REMOVE ME: my debug statement that I swear to remove when I'm done\");

Please make sure to remove these 'inappropriately' tagged messages before
shipping the module ;)
----

== Rate limiting and one time messages ==
Occasionally you have to insert a message in a section which gets called
quite often. This not only might have a severe performance impact, it also
could overwrite and spam your kernel buffer so it should be avoided.

As always the kernel already provides you with nice functions that solve your
problems:
printk_ratelimited(...)
and
printk_once(...)

''printk_once'' is fairly trivial - no matter how often you call it, it prints once and never again.

''printk_ratelimited'' is a little bit more complicated - it prints by default not more than 10
times in every 5 seconds (for each function it is called in).

If you need other values for the maximum burst count and the timeout, you can
always setup your own ratelimit using the ''DEFINE_RATELIMIT_STATE'' macro and
the ''__ratelimit'' function - see the implementation of
''printk_ratelimited'' for an example.

Be sure to ''#include <kernel/ratelimit.h>'' in order to use
''printk_ratelimited()''

Both functions have also their ''pr_*'' equivalents like
''pr_info_ratelimited'' for ''printk_ratelimited(KERN_INFO...'' and
''pr_crit_once'' for ''printk_once(KERN_CRIT...''

'''Note: both did not work as expected in my tests here, will probably investigate further'''

== Printk from userspace ==
Sometimes, especially when doing automated testing, it is quite useful to
insert some messages in the kernel log buffer in order to annotate what's
going on.

It is as simple as
# echo "Hello Kernel-World" > /dev/kmsg

Of course this messages gets the default log level assigned, if you want e.g.
to issue a KERN_CRIT message you have to use the string representation of the
log level - in this case "<2>"

# echo "<2>Writing critical printk messages from userspace" >/dev/kmsg

The message will appear like any other '''kernel''' message - there is '''no way''' to distinguish them!

'''Note:''' Don't confuse this with printf - we are printing a kernel message
from userspace here.

== Internals ==
Printk is implemented by using a ring buffer in the kernel with a size of
''__LOG_BUF_LEN'' bytes where ''__LOG_BUF_LEN'' equals ''(1 <<
CONFIG_LOG_BUF_SHIFT)'' (see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] for details).

You can specify the size of the buffer in your kernel config by setting
''CONFIG_LOG_BUF_SHIFT'' to an appropriate value (e.g. 17 for 128Kb) (''make menuconfig -> General Setup -> Kernel log buffer size'').

Using a ring buffer implies that older messages get overwritten once the
buffer fills up, but this is only a minor drawback compared to the robustness of this
solution (i.e. minimum memory footprint, callable from every context, not many
resources wasted if nobody reads the buffer, no filling up of disk space/ram when some kernel process goes
wild and spams the buffer, ...).
Using a reasonably large buffer size should give you enough time to read your
important messages before they are overwritten.

The kernel log buffer is accessible for reading from userspace by ''/proc/kmsg''.
''/proc/kmsg'' behaves more or less like a FIFO and blocks until new messages
appear.

'''Please note''' - reading from ''/proc/kmsg'' consumes the messages in the
ring buffer so they may not be available for other programs. It is usually a
good idea to let ''klogd'' or ''syslog'' do this job and read the content
of the buffer via ''dmesg''.

== Pros and Cons==
The main advantage of printk over other debugging solutions is that it requires no sophisticated setup and can be called anywhere from any time.
Printk can be called while holding a lock, from interrupt and process context, is SMP safe and does not need any special preparation. It is just there and just works.
The only precondition is that you have some kind of working console to display the messages.

For the early stages in the boot process, where no console is available yet,
there is a special function named ''early_printk'', this function writes
directly to the VGA buffer or a serial line but otherwise works just like
printk -- you have to enable this function by setting ''CONFIG_EARLY_PRINTK''
in your kernel config (''make menuconfig -> Kernel Hacking -> Early printk'').

The major drawback is that printk is quite static, so you have to figure out what you want to trace beforehand and if you want to trace something different you have to recompile your code - which can become quite cumbersome. (And of course printk is not interactive at all, so you can't modify any variables or the like.)

The other drawback is that printing usually consumes quite some processing power and io time, so if you're trying to debug a timing critical section or a timing bug, you're probably out of luck.

== NetConsole ==
Sometimes you are in the unlucky situation that the machine crashes or hangs and you have no way to access the console afterwards, e.g. the graphic driver hangs and and the kernel dies soon after.
In this case you could either hook up a serial line to your crashing target machine (if a serial port is available) or use the kernels netconsole feature to enable printk logging via UDP.

In order to use it you have to enable the ''CONFIG_NETCONSOLE'' kernel config
option (''make menuconfig -> Device Drivers -> Network device support -> Network core driver support -> Network console logging support'') and
configure it appropriately by using the ''netconsole'' configuration parameter

netconsole=[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
where
src-port source for UDP packets (defaults to 6665)
src-ip source IP to use (interface address)
dev network interface (eth0)
tgt-port port for logging agent (6666)
tgt-ip IP address for logging agent
tgt-macaddr ethernet MAC address for logging agent (broadcast)

e.g. specify at kernel commandline (in your bootloader)
linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc
to send messages from 10.0.0.1 port 4444 via eth1 to 10.0.0.2 port 9353 with the MAC 12:34:56:78:9a:bc

or while loading the module e.g
# insmod netconsole netconsole=@/,@10.0.0.2/
to send messages via broadcast to 10.0.0.2 port 6666

On the other machine you can now simply fire up
# netcat -u -l -p <port>
e.g.
$ netcat -u -l -p 6666
and see the printk messages from your target dribbling in.

If you don't see any messages you might have to set the console_loglevel to a
higher value (see above) or test the connection via telnet e.g. from the target type
$ telnet 10.0.0.2 6666

see
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
for more details.

== Misc ==
=== Dmesg ===
dmesg -c
clears the dmesg buffer.
Sometimes it is nice to start with a blank buffer, so you will only see new
messages when you invoke ''dmesg''

=== Printk Timestamps ===
CONFIG_PRINTK_TIMES
Setting this kernel config option prepends every printk statement with a
timestamp representing the time since boot. This is particularly useful to get
a general idea about the timings of your code.

=== Printing buffers as hex ===
If you want to print a buffer as hex within the kernel, don't reinvent the
wheel use ''printk_hex_dump_bytes()'' instead.

print_hex_dump_bytes(const char *prefix_str, int prefix_type, const void *buf, size_t len)
this function prints a buffer as hex values to the kernel log buffer (with level KERN_DEBUG)
Example:
Kernel Code:
char mybuf[] = "abcdef";
print_hex_dump_bytes("", DUMP_PREFIX_NONE, mybuf, ARRAY_SIZE(mybuf));

dmesg output:
61 62 63 64 65 66 00 abcdef.

If you need something more sophisticated and flexible maybe have a look at
''print_hex_dump()'' and ''hex_dump_to_buffer()''

=== Dynamic Debugging ===
It is also possible to enable/disable debug information at runtime using the
dynamic debug functionality of the kernel.
For this the ''CONFIG_DYNAMIC_DEBUG'' kernel config option must be set.
See
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt] for more information.

== References and external links ==
* [[Linux Kernel Development - by Robert Love|Linux Kernel Development]], Robert Love, 3rd Edition, Chapter 18 Debugging
* [[Linux Device Drivers]], Corbet, Rubini and Kroah-Hartmann, 3rd Edition, Chapter 4 Section 2
* [[Essential Linux Device Drivers]]

* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/printk-formats.txt;hb=HEAD Documentation/printk-formats.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/dynamic-debug-howto.txt;hb=HEAD Documentation/dynamic-debug-howto.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/networking/netconsole.txt;hb=HEAD Documentation/networking/netconsole.txt]
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=kernel/printk.c;hb=HEAD kernel/printk.c] Implementation of printk and others
* [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=include/linux/printk.h;hb=HEAD include/linux/printk.h] printk header file
* [http://www.embedded-bits.co.uk/2010/printk-format-specifiers/ Blog Entry about the different %p format specifiers]
* [[Printk]]
{{Reflist}}

[[Category:Development Tools]]
[[Category:Tips and Tricks]]
[[Category:Printk]]