AoEForEmbeddedLinuxDevEnvEngText

[original file is http://elinux.org/upload/3/38/AoEForEmbeddedLinuxDevEnv-CELF200905shigeoka-jp.pdf]

[translated by ikoma]

[see original slide for figures]

- CELF Technical Jamboree #27

May, 22, 2009 -

-- Copyright c Hitachi Ltd. 2008 All rights reserved. --

(slide 1)
CE Linux Forum Japan Technical Jamboree #27 2009/5/22

ATA over Ethernet for Embedded Linux Development Environment

Tomohiko Shigeoka Embedded System Platform Research Laboratory, Hitachi Ltd.

(slide 2)Background

 * NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development environment


 * When burnt onto devices, file systems are built on flash memory
 * File system other than extN is often used
 * cramfs/squashfs/jffs2/ubifs/ etc.

link title
 * Modifying files placed on flash memory is not easy (in many cases)
 * Writing with ROM Writer, JTAG-ICE etc. is cumbersome
 * If failed, further troublesome

(slide 3)Typical Development Environment using NFS

 * Developing software on host system using cross tool chain


 * A file system on the host is used as the root file system on the target via NFS


 * Later written onto flash memory by generating file system image

(Figure)

(slide 4)Issues when using NFS

 * NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ...
 * Different software configuration in operation (after development) may cause mysterious behavior
 * Debugging problems attributed to file system is impossible/hard


 * (+) Sun publicly announced NFSv2 in 1984; NFS implemented on Linux since kernel circa 1.2

(slide 5)Networking the Block Device Layer

 * Network connection in block device layer, not file system layer
 * Same file system as on flash memory can be used


 * Network connectable block devices:
 * ATA over Ethernet (AoE)
 * iSCSI (internet SCSI)
 * Network Block Device (NBD)

(slide 6)ATA over Ethernet (AoE) Overview

 * Low cost SAN technology lead by coraid.com


 * Light-weight protocol to communicate by loading ATA command/data on Ethernet frames (does not use TCP/IP)


 * Client side driver mainlined since kernel 2.6.11 (drivers/block/aoe/)


 * Configuration commands (aoetools) available on client side


 * Server side is special hardware or software implementation - userland daemon, kernel driver etc.
 * vblade, kvblade, qaoed, etc.


 * Server side is not required to be an ATA


 * Supported on numbers of OSes and bootloaders

(slide 7)System Structure Diagram of AoE

 * vblade daemon on server publishes disk image file on HDD

# vblade 0 0 eth0 image_file


 * AoE driver on client presents an image published on server as a block device to file system layer

# mount -t ext2 /dev/etherd/e0.0 /mnt

(Figure)

(slide 8)iSCSI Overview

 * SAN technology getting popular in enterprise arena


 * Protocol to communicate by loading SCSI commands/data on TCP/IP


 * Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c)


 * Block devices can be used with the same interface with ordinary SCSI devices (/dev/sda etc.)


 * On client side, configuration with userland tool (Open-iSCSI) is required


 * Server side runs on high-end special hardware as well as various software implementation (iSCSI Enterprise Target, etc.)

(slide 9)System Structure Diagram of iSCSI

 * iSCSI Target on server publishes a disk image file or a partition on HDD


 * iSCSI Initiator on client connects to the target, and presents the disk image as a SCSI disk block device

(Figure)

(slide 10)Network Block Device (NBD) Overview

 * Treats server data via TCP/IP as block device


 * Protocol irrelevant to ATA (compression supported)


 * Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c)

# nbd-client server 2000 /dev/nb0
 * On client side, connection management daemon (nbd-client) required

# nbd-server 2000 image_file
 * Server is implemented as software (nbd-server)

(Figure)

(slide 11)Use of rootfs with each Method

 * Only AoE support the same style as NFS
 * AoE root patch required(http://support.coraid.com/support/linux/contrib/mcmullan/aoeroot-2.6.15.diff)
 * With this patch, AoE server to be root must be specified in kernel config
 * Can not be specified in kernel command line (root=???)
 * Setting with command (aoetools) is not required


 * Setting up with command is required for iSCSI or NBD
 * Setting up using initramfs/initrd
 * Direct mounting as rootfs is not possible; must add initramfs/initrd with setting up
 * --> this is why we use AoE here

(slide 12)MTD Emulation

 * Typical file systems (ext3/cramfs etc.) assume block devices


 * jffs2 etc. assumes MTD, not block devices

# insmod block2mtd.ko block2mtd=/dev/hda1 # mount -t jffs2 /dev/mtdblock0 /mnt
 * MTD interface can be used with block2mtd

(Figure)

(slide 13)AoE for Embedded Development Environment

 * Software developed with cross tool chain in the same manner as when NFS used


 * File system image must be created to be executed on target


 * File system image to be written on flash memory can be directly used as root file system via network with AoE

(Figure)

(slide 14)Evaluation of AoE Use

 * Evaluation items (Comparison among AoE/NFS/Flash memory)
 * Object size of target kernel
 * Memory usage at boot with identical rootfs
 * Comparison of behavior with compression file system (cramfs)
 * read time of a highly compressible file (all zero) and a less compressible file (random data) of the same data volume


 * Evaluation environment
 * Target: OSK (TI OMAP 192MHz, RAM 32M), kernel 2.6.20.20, 10BASE-T
 * Host: x86_64(3.4GHz), kernel 2.4.21(RHEL3), 1000BASE-T

(slide 15)Evaluation of AoE Use (Size/Memory Usage)

 * Target kernel object size comparison

+--+-+-+


 * AoE         |   NFS   |     Flash memory    |

+---+--+-+---+-+


 * file system|  aoe    |   NFS   |file system|   mtd   |


 * (cramfs) |          | client  | (cramfs)  |         |

+---+--+-+---+-+


 * 10,433 |  28,326 |         |    10,433 | 73.401  |

+---+--+        +---+-+


 * 38,759 | 322,503 |             83,834 |

+--+-+-+


 * size of build-in.o in each directory (bytes)


 * Memory usage comparison
 * After booting up with an identical rootfs (busybox), logging in and executing free command

+---++++-+


 * | total |  used  |  free  | buffers |

+---++++-+


 * AoE | 30,220 |  5,000 | 25,220 |     724 |

+---++++-+


 * NFS | 29,888 |  4,352 | 25,536 |       0 |

+---++++-+


 * Flash | 30,188 | 5,040 | 25,148 |     736 |

+---++++-+


 * free command output (Kbytes)

(slide 16)Evaluation of AoE Use (Behavior)

 * File read time (CPU time)
 * "echo 3 > /proc/sys/vm/drop_caches", then read out with "time wc -c"

+---+++


 * |  all zero data    |     random data    |




 * | real | user | sys | real | user | sys  |

+---+--+--+--+--+--+--+


 * AoE | 1.57 | 0.71 | 0.63 | 7.09 | 0.66 | 1.14 |

+---+--+--+--+--+--+--+


 * NFS | 6.28 | 0.72 | 0.88 | 6.39 | 0.72 | 0.86 |

+---+--+--+--+--+--+--+


 * Flash | 1.44 | 0.72 | 0.61 | 3.23 | 0.69 | 0.88 |

+---+--+--+--+--+--+--+


 * unit: second (read 5MB; avarage of 10 trials)


 * Data transfer volume
 * Measured as the difference between /proc/diskstats and /proc/net/dev

+---+--+--+


 * |     all zero data       |       random data        |




 * | eth recv  | sector read |  eth recv  | sector read |

+---++-++-+


 * AoE |    352,040 |     339,968 |  5,770,820 |   5,570,560 |

+---++-++-+


 * NFS |  5,666,068 |           - |  5,663,699 |           - |

+---++-++-+


 * Flash |         - |     339,968 |          - |   5,570,560 |

+---++-++-+


 * unit: byte (when read 5MB)

(slide 17)Consideration

 * Kernel CPU time reflects the difference of compression processing
 * Expanding highly compressible data (all zero) does not use CPU as much as random data (Assuming that data transfer does not need so much CPU power; if data transfer needs much CPU power, volume of memory copy would be also reflected)
 * In case of NFS, it is independent of data contents as there is no expanding (data transfer volume is also uniform)


 * Data transfer volume is reflects the data contents to be transferred
 * Transferring compressed data causes highly compressible data is transferred as less volume
 * NFS does not compress data, so data transfer volume is constant

(slide 18)Summary

 * AoE is usable even for embedded Linux systems


 * AoE make some debugging impossible with NFS possible


 * Object size is smaller with AoE than with NFS


 * AoE user-friendliness is still evolving

(slide 19)References

 * http://www.coraid.com/RESOURCES/AoE-Protocol-Definition


 * http://sourceforge.net/projects/aoetools/vblade


 * http://wwww.open-iscsi.org/


 * http://iscsitarget.sourceforge.net/


 * http://nbd.sourceforge.net/