Difference between revisions of "Talk:Kernel dynamic memory analysis"
(Created page with "== 2012-10-05 == Here is some feedback from testing on 2012-10-05: These items are not in any order * Install issue There's no 'install' mechanism for the tool, and the tool de...") |
(→Availability to show plot interactively) |
||
(12 intermediate revisions by 2 users not shown) | |||
Line 4: | Line 4: | ||
These items are not in any order | These items are not in any order | ||
− | + | === Install issue === | |
There's no 'install' mechanism for the tool, and the tool depends | There's no 'install' mechanism for the tool, and the tool depends | ||
visualize_mem_tree.py being on the python path or in the same | visualize_mem_tree.py being on the python path or in the same | ||
Line 14: | Line 14: | ||
I'm not sure). | I'm not sure). | ||
− | + | ==== Answer ==== | |
+ | |||
+ | trace_analyze and visualize_mem_tree are now merged. | ||
+ | (should we add an 'install' mechanism anyway?) | ||
+ | |||
+ | === Extra rings === | ||
The chart for my kernel shows some unexpected rings. Specifically, | The chart for my kernel shows some unexpected rings. Specifically, | ||
I'm building in /home/tbird/work/mem/build/panda-torvalds | I'm building in /home/tbird/work/mem/build/panda-torvalds | ||
Line 25: | Line 30: | ||
Is this correct? | Is this correct? | ||
− | + | ==== Answer ==== | |
+ | |||
+ | Fixed | ||
+ | |||
+ | === Overlapping annotations === | ||
The annotations showing the name and size of the ring segments overlap | The annotations showing the name and size of the ring segments overlap | ||
sometimes, making some of them difficult or impossible to read. | sometimes, making some of them difficult or impossible to read. | ||
See above chart. | See above chart. | ||
− | + | === Availability to show plot interactively === | |
The tool by default produces a png in the current directory. It would | The tool by default produces a png in the current directory. It would | ||
be nice to be able to control the output format and location, or allow | be nice to be able to control the output format and location, or allow | ||
Line 36: | Line 45: | ||
(i.e. pylab.show()). | (i.e. pylab.show()). | ||
− | + | ==== Answer ==== | |
+ | |||
+ | Done. There's now a --rings-show (-i) option for interactive plotting: pylab.show(). | ||
+ | Also, without this option a png file is generated at ./{selected_branch}_{selected_attr}.png. | ||
+ | (e.g. ./fs_sysfs_static.png) | ||
+ | |||
+ | Warning: the interactive plot sometimes show annotations out-of-bounds. | ||
+ | Tried to fix a while ago, but failed. Will re-try a bit. | ||
+ | |||
+ | === Problem with absolute paths === | ||
I tried using the tool with an absolute path, and it had problems: | I tried using the tool with an absolute path, and it had problems: | ||
$ trace_analyze.py -k /home/tbird/work/mem/build/panda-torvalds/ | $ trace_analyze.py -k /home/tbird/work/mem/build/panda-torvalds/ | ||
Line 45: | Line 63: | ||
readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms2.o': No such file | readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms2.o': No such file | ||
... [lots more 'No such file' errors] | ... [lots more 'No such file' errors] | ||
+ | |||
+ | ==== Answer ==== | ||
+ | |||
+ | Fixed | ||
+ | |||
+ | === Description of how to get trace === | ||
+ | The instructions should includes steps for how to get the trace for the dynamic analysis. | ||
+ | It looks like a comprehensive section is started in Documentation/memory_accounting | ||
+ | section 3.2.1, so maybe this feedback is premature. Maybe a few short steps on the | ||
+ | wiki page for people to get their first trace would be good. | ||
+ | |||
+ | Here's what I did: | ||
+ | * cloned git repository from https://github.com/ezequielgarcia/kmem-probe-framework | ||
+ | * built kernel | ||
+ | * generated first chart: | ||
+ | ** $ trace_analyze.py -k ../../build/panda-torvalds -r -a static | ||
+ | ** $ trace_analyze.py -k ../../build/panda-torvalds -r -a static -b fs | ||
+ | * check that kernel config has CONFIG_FTRACE=y | ||
+ | ** If not, set and rebuild kernel | ||
+ | * adjust kernel command line to add tracing of kmem events | ||
+ | ** Add: "trace_buf_size=10000000 trace_event=kmem:kmalloc,kmem:kmem_cache_alloc,kmem:kfree,kmem:kmem_cache_free" to kernel command line | ||
+ | * boot kernel | ||
+ | * mount debugfs: | ||
+ | $ mount -t debugfs none /debug | ||
+ | * capture trace data: | ||
+ | $ cat /debug/tracing/trace >/tmp/kmem.log | ||
+ | * move to host: | ||
+ | $ ttc cp target:/tmp/kmem.log . | ||
+ | * generate more charts | ||
+ | ** $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current ; mv linux.png linux-current.png | ||
+ | ** $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current -b fs ; mv fs.png fs-current.png | ||
+ | |||
+ | ''Note: Make sure you do a 'make clean' if you reconfigure and rebuild the kernel, before running trace_analyze.py again.'' | ||
+ | |||
+ | === sub-sub-dirs doesn't work === | ||
+ | I don't know if it's intended to work or not, but I tried to get a ring chart for | ||
+ | a sub-sub-directory, and it didn't work | ||
+ | |||
+ | <pre> | ||
+ | |||
+ | $ trace_analyze.py -a static -k ../../build/panda-torvalds/ -b fs/ubifs | ||
+ | No trace log file specified: will report on static size only | ||
+ | Reading symbol map at ../../build/panda-torvalds | ||
+ | Creating tree from compiled symbols at ../../build/panda-torvalds/fs/ubifs | ||
+ | [WARNING] Duplicate data entry! __func__.25429 | ||
+ | [WARNING] Duplicate data entry! __func__.25886 | ||
+ | ...[some lines omitted] | ||
+ | [WARNING] Duplicate data entry! __func__.26112 | ||
+ | Cleaning tree | ||
+ | .. | ||
+ | build | ||
+ | panda-torvalds | ||
+ | fs | ||
+ | ubifs | ||
+ | tnc_misc.o | ||
+ | lprops.o | ||
+ | scan.o | ||
+ | ... [ some lines omitted] | ||
+ | tnc_commit.o | ||
+ | io.o | ||
+ | dir.o | ||
+ | debug.o | ||
+ | ioctl.o | ||
+ | sb.o | ||
+ | orphan.o | ||
+ | log.o | ||
+ | |||
+ | [WARNING] Can't find first branch 'fs/ubifs' | ||
+ | Creating ringchart file at fs/ubifs.png | ||
+ | Traceback (most recent call last): | ||
+ | File "/a/home/tbird/bin/trace_analyze.py", line 654, in <module> | ||
+ | main() | ||
+ | File "/a/home/tbird/bin/trace_analyze.py", line 651, in main | ||
+ | visualize_mem_tree(tree, opts.attr, filename) | ||
+ | File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 148, in visualize_mem_tree | ||
+ | rings = create_child_rings(tree, size_attr=size_attr) | ||
+ | File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 98, in create_child_rings | ||
+ | max_size = getattr(tree.size(), size_attr)() | ||
+ | AttributeError: 'NoneType' object has no attribute 'size' | ||
+ | </pre> | ||
+ | |||
+ | ==== Answer ==== | ||
+ | |||
+ | Fixed |
Latest revision as of 14:51, 7 October 2012
Contents
2012-10-05
Here is some feedback from testing on 2012-10-05:
These items are not in any order
Install issue
There's no 'install' mechanism for the tool, and the tool depends visualize_mem_tree.py being on the python path or in the same directory as the trace_analyze.py script. I tried hard-linking trace_analyze to a private 'bin' directory, but got a python error doing this. Using a symlink worked OK. If visualize_mem_tree.py is not used separately, it might be worth considering integrating everything into a single script. (or not - I'm not sure).
Answer
trace_analyze and visualize_mem_tree are now merged. (should we add an 'install' mechanism anyway?)
Extra rings
The chart for my kernel shows some unexpected rings. Specifically, I'm building in /home/tbird/work/mem/build/panda-torvalds My chart has an inner ring of 'build' (showing 100% of memory), a second ring of 'panda-torvalds' (showing 100% of memory), then a third ring where directories start to get broken out.
Is this correct?
Answer
Fixed
Overlapping annotations
The annotations showing the name and size of the ring segments overlap sometimes, making some of them difficult or impossible to read. See above chart.
Availability to show plot interactively
The tool by default produces a png in the current directory. It would be nice to be able to control the output format and location, or allow the user to directly enter matplotlibs interactive viewer. (i.e. pylab.show()).
Answer
Done. There's now a --rings-show (-i) option for interactive plotting: pylab.show(). Also, without this option a png file is generated at ./{selected_branch}_{selected_attr}.png. (e.g. ./fs_sysfs_static.png)
Warning: the interactive plot sometimes show annotations out-of-bounds. Tried to fix a while ago, but failed. Will re-try a bit.
Problem with absolute paths
I tried using the tool with an absolute path, and it had problems:
$ trace_analyze.py -k /home/tbird/work/mem/build/panda-torvalds/ No trace log file specified: will report on static size only Reading symbol map at /home/tbird/work/mem/build/panda-torvalds Creating tree from compiled symbols at /home/tbird/work/mem/build/panda-torvalds readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms1.o': No such file readelf: Error: 'home/tbird/work/mem/build/panda-torvalds/.tmp_kallsyms2.o': No such file ... [lots more 'No such file' errors]
Answer
Fixed
Description of how to get trace
The instructions should includes steps for how to get the trace for the dynamic analysis. It looks like a comprehensive section is started in Documentation/memory_accounting section 3.2.1, so maybe this feedback is premature. Maybe a few short steps on the wiki page for people to get their first trace would be good.
Here's what I did:
- cloned git repository from https://github.com/ezequielgarcia/kmem-probe-framework
- built kernel
- generated first chart:
- $ trace_analyze.py -k ../../build/panda-torvalds -r -a static
- $ trace_analyze.py -k ../../build/panda-torvalds -r -a static -b fs
- check that kernel config has CONFIG_FTRACE=y
- If not, set and rebuild kernel
- adjust kernel command line to add tracing of kmem events
- Add: "trace_buf_size=10000000 trace_event=kmem:kmalloc,kmem:kmem_cache_alloc,kmem:kfree,kmem:kmem_cache_free" to kernel command line
- boot kernel
- mount debugfs:
$ mount -t debugfs none /debug
- capture trace data:
$ cat /debug/tracing/trace >/tmp/kmem.log
- move to host:
$ ttc cp target:/tmp/kmem.log .
- generate more charts
- $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current ; mv linux.png linux-current.png
- $ trace_analyze.py -k ../../build/panda-torvalds -k kmem.log -r -a current -b fs ; mv fs.png fs-current.png
Note: Make sure you do a 'make clean' if you reconfigure and rebuild the kernel, before running trace_analyze.py again.
sub-sub-dirs doesn't work
I don't know if it's intended to work or not, but I tried to get a ring chart for a sub-sub-directory, and it didn't work
$ trace_analyze.py -a static -k ../../build/panda-torvalds/ -b fs/ubifs No trace log file specified: will report on static size only Reading symbol map at ../../build/panda-torvalds Creating tree from compiled symbols at ../../build/panda-torvalds/fs/ubifs [WARNING] Duplicate data entry! __func__.25429 [WARNING] Duplicate data entry! __func__.25886 ...[some lines omitted] [WARNING] Duplicate data entry! __func__.26112 Cleaning tree .. build panda-torvalds fs ubifs tnc_misc.o lprops.o scan.o ... [ some lines omitted] tnc_commit.o io.o dir.o debug.o ioctl.o sb.o orphan.o log.o [WARNING] Can't find first branch 'fs/ubifs' Creating ringchart file at fs/ubifs.png Traceback (most recent call last): File "/a/home/tbird/bin/trace_analyze.py", line 654, in <module> main() File "/a/home/tbird/bin/trace_analyze.py", line 651, in main visualize_mem_tree(tree, opts.attr, filename) File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 148, in visualize_mem_tree rings = create_child_rings(tree, size_attr=size_attr) File "/a/home/tbird/work/tiny/tools/kmem-probe-framework/post-process/visualize_mem_tree.py", line 98, in create_child_rings max_size = getattr(tree.size(), size_attr)() AttributeError: 'NoneType' object has no attribute 'size'
Answer
Fixed