Difference between revisions of "Talk:Kernel dynamic memory analysis"

From eLinux.org
Jump to: navigation, search
(Extra rings)
(Problem with absolute paths)
Line 54: Line 54:
 
  readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms2.o': No such file
 
  readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms2.o': No such file
 
  ... [lots more 'No such file' errors]
 
  ... [lots more 'No such file' errors]
 +
 +
==== Answer ====
 +
 +
Fixed
  
 
=== Description of how to get trace ===
 
=== Description of how to get trace ===

Revision as of 19:40, 7 October 2012

2012-10-05

Here is some feedback from testing on 2012-10-05:

These items are not in any order

Install issue

There's no 'install' mechanism for the tool, and the tool depends visualize_mem_tree.py being on the python path or in the same directory as the trace_analyze.py script. I tried hard-linking trace_analyze to a private 'bin' directory, but got a python error doing this. Using a symlink worked OK. If visualize_mem_tree.py is not used separately, it might be worth considering integrating everything into a single script. (or not - I'm not sure).

Answer

trace_analyze and visualize_mem_tree are now merged. (should we add an 'install' mechanism anyway?)

Extra rings

The chart for my kernel shows some unexpected rings. Specifically, I'm building in /home/tbird/work/mem/build/panda-torvalds My chart has an inner ring of 'build' (showing 100% of memory), a second ring of 'panda-torvalds' (showing 100% of memory), then a third ring where directories start to get broken out.

See this chart: Linux-2012-10-05.png

Is this correct?

Answer

Fixed

Overlapping annotations

The annotations showing the name and size of the ring segments overlap sometimes, making some of them difficult or impossible to read. See above chart.

Availability to show plot interactively

The tool by default produces a png in the current directory. It would be nice to be able to control the output format and location, or allow the user to directly enter matplotlibs interactive viewer. (i.e. pylab.show()).

Problem with absolute paths

I tried using the tool with an absolute path, and it had problems:

$ trace_analyze.py -k /home/tbird/work/mem/build/panda-torvalds/
No trace log file specified: will report on static size only
Reading symbol map at /home/tbird/work/mem/build/panda-torvalds
Creating tree from compiled symbols at /home/tbird/work/mem/build/panda-torvalds
readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms1.o': No such file
readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms2.o': No such file
... [lots more 'No such file' errors]

Answer

Fixed

Description of how to get trace

The instructions should includes steps for how to get the trace for the dynamic analysis. It looks like a comprehensive section is started in Documentation/memory_accounting section 3.2.1, so maybe this feedback is premature. Maybe a few short steps on the wiki page for people to get their first trace would be good.

Here's what I did:

  • cloned git repository from https://github.com/ezequielgarcia/kmem-probe-framework
  • built kernel
  • generated first chart:
    • $ trace_analyze.py -k ../../build/panda-torvalds -r -a static
    • $ trace_analyze.py -k ../../build/panda-torvalds -r -a static -b fs
  • check that kernel config has CONFIG_FTRACE=y
    • If not, set and rebuild kernel
  • adjust kernel command line to add tracing of kmem events
    • Add: "trace_buf_size=10000000 trace_event=kmem:kmalloc,kmem:kmem_cache_alloc,kmem:kfree,kmem:kmem_cache_free" to kernel command line
  • boot kernel
  • mount debugfs:
$ mount -t debugfs none /debug
  • capture trace data:
$ cat /debug/tracing/trace >/tmp/kmem.log
  • move to host:
$ ttc cp target:/tmp/kmem.log .
  • generate more charts
    • $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current ; mv linux.png linux-current.png
    • $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current -b fs ; mv fs.png fs-current.png

Note: Make sure you do a 'make clean' if you reconfigure and rebuild the kernel, before running trace_analyze.py again.

sub-sub-dirs doesn't work

I don't know if it's intended to work or not, but I tried to get a ring chart for a sub-sub-directory, and it didn't work


$ trace_analyze.py -a static -k ../../build/panda-torvalds/ -b fs/ubifs
No trace log file specified: will report on static size only
Reading symbol map at ../../build/panda-torvalds
Creating tree from compiled symbols at ../../build/panda-torvalds/fs/ubifs
[WARNING] Duplicate data entry! __func__.25429
[WARNING] Duplicate data entry! __func__.25886
...[some lines omitted]
[WARNING] Duplicate data entry! __func__.26112
Cleaning tree
..
  build
    panda-torvalds
      fs
        ubifs
          tnc_misc.o
          lprops.o
          scan.o
          ... [ some lines omitted]
          tnc_commit.o
          io.o
          dir.o
          debug.o
          ioctl.o
          sb.o
          orphan.o
          log.o

[WARNING] Can't find first branch 'fs/ubifs'
Creating ringchart file at fs/ubifs.png
Traceback (most recent call last):
  File "/a/home/tbird/bin/trace_analyze.py", line 654, in <module>
    main()
  File "/a/home/tbird/bin/trace_analyze.py", line 651, in main
    visualize_mem_tree(tree, opts.attr, filename)
  File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 148, in visualize_mem_tree
    rings = create_child_rings(tree, size_attr=size_attr)
  File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 98, in create_child_rings
    max_size = getattr(tree.size(), size_attr)()
AttributeError: 'NoneType' object has no attribute 'size'