Talk:Kernel dynamic memory analysis

From eLinux.org
Jump to: navigation, search

2012-10-05

Here is some feedback from testing on 2012-10-05:

These items are not in any order

Install issue

There's no 'install' mechanism for the tool, and the tool depends visualize_mem_tree.py being on the python path or in the same directory as the trace_analyze.py script. I tried hard-linking trace_analyze to a private 'bin' directory, but got a python error doing this. Using a symlink worked OK. If visualize_mem_tree.py is not used separately, it might be worth considering integrating everything into a single script. (or not - I'm not sure).

Answer

trace_analyze and visualize_mem_tree are now merged. (should we add an 'install' mechanism anyway?)

Extra rings

The chart for my kernel shows some unexpected rings. Specifically, I'm building in /home/tbird/work/mem/build/panda-torvalds My chart has an inner ring of 'build' (showing 100% of memory), a second ring of 'panda-torvalds' (showing 100% of memory), then a third ring where directories start to get broken out.

See this chart: Linux-2012-10-05.png

Is this correct?

Answer

Fixed

Overlapping annotations

The annotations showing the name and size of the ring segments overlap sometimes, making some of them difficult or impossible to read. See above chart.

Availability to show plot interactively

The tool by default produces a png in the current directory. It would be nice to be able to control the output format and location, or allow the user to directly enter matplotlibs interactive viewer. (i.e. pylab.show()).

Answer

Done. There's now a --rings-show (-i) option for interactive plotting: pylab.show(). Also, without this option a png file is generated at ./{selected_branch}_{selected_attr}.png. (e.g. ./fs_sysfs_static.png)

Warning: the interactive plot sometimes show annotations out-of-bounds. Tried to fix a while ago, but failed. Will re-try a bit.

Problem with absolute paths

I tried using the tool with an absolute path, and it had problems:

$ trace_analyze.py -k /home/tbird/work/mem/build/panda-torvalds/
No trace log file specified: will report on static size only
Reading symbol map at /home/tbird/work/mem/build/panda-torvalds
Creating tree from compiled symbols at /home/tbird/work/mem/build/panda-torvalds
readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms1.o': No such file
readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms2.o': No such file
... [lots more 'No such file' errors]

Answer

Fixed

Description of how to get trace

The instructions should includes steps for how to get the trace for the dynamic analysis. It looks like a comprehensive section is started in Documentation/memory_accounting section 3.2.1, so maybe this feedback is premature. Maybe a few short steps on the wiki page for people to get their first trace would be good.

Here's what I did:

  • cloned git repository from https://github.com/ezequielgarcia/kmem-probe-framework
  • built kernel
  • generated first chart:
    • $ trace_analyze.py -k ../../build/panda-torvalds -r -a static
    • $ trace_analyze.py -k ../../build/panda-torvalds -r -a static -b fs
  • check that kernel config has CONFIG_FTRACE=y
    • If not, set and rebuild kernel
  • adjust kernel command line to add tracing of kmem events
    • Add: "trace_buf_size=10000000 trace_event=kmem:kmalloc,kmem:kmem_cache_alloc,kmem:kfree,kmem:kmem_cache_free" to kernel command line
  • boot kernel
  • mount debugfs:
$ mount -t debugfs none /debug
  • capture trace data:
$ cat /debug/tracing/trace >/tmp/kmem.log
  • move to host:
$ ttc cp target:/tmp/kmem.log .
  • generate more charts
    • $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current ; mv linux.png linux-current.png
    • $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current -b fs ; mv fs.png fs-current.png

Note: Make sure you do a 'make clean' if you reconfigure and rebuild the kernel, before running trace_analyze.py again.

sub-sub-dirs doesn't work

I don't know if it's intended to work or not, but I tried to get a ring chart for a sub-sub-directory, and it didn't work


$ trace_analyze.py -a static -k ../../build/panda-torvalds/ -b fs/ubifs
No trace log file specified: will report on static size only
Reading symbol map at ../../build/panda-torvalds
Creating tree from compiled symbols at ../../build/panda-torvalds/fs/ubifs
[WARNING] Duplicate data entry! __func__.25429
[WARNING] Duplicate data entry! __func__.25886
...[some lines omitted]
[WARNING] Duplicate data entry! __func__.26112
Cleaning tree
..
  build
    panda-torvalds
      fs
        ubifs
          tnc_misc.o
          lprops.o
          scan.o
          ... [ some lines omitted]
          tnc_commit.o
          io.o
          dir.o
          debug.o
          ioctl.o
          sb.o
          orphan.o
          log.o

[WARNING] Can't find first branch 'fs/ubifs'
Creating ringchart file at fs/ubifs.png
Traceback (most recent call last):
  File "/a/home/tbird/bin/trace_analyze.py", line 654, in <module>
    main()
  File "/a/home/tbird/bin/trace_analyze.py", line 651, in main
    visualize_mem_tree(tree, opts.attr, filename)
  File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 148, in visualize_mem_tree
    rings = create_child_rings(tree, size_attr=size_attr)
  File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 98, in create_child_rings
    max_size = getattr(tree.size(), size_attr)()
AttributeError: 'NoneType' object has no attribute 'size'

Answer

Fixed

There are no threads on this page yet.