AoEForEmbeddedLinuxDevEnvEngText

From eLinux.org
Jump to: navigation, search

[original file is http://elinux.org/upload/3/38/AoEForEmbeddedLinuxDevEnv-CELF200905shigeoka-jp.pdf]

[translated by ikoma]

[see original slide for figures]


CELF Technical Jamboree #27

May, 22, 2009




Copyright c Hitachi Ltd. 2008 All rights reserved.



(slide 1)

CE Linux Forum Japan Technical Jamboree #27 2009/5/22


ATA over Ethernet for Embedded Linux Development Environment

Tomohiko Shigeoka Embedded System Platform Research Laboratory, Hitachi Ltd.


(slide 2)Background

  • NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development environment
  • When burnt onto devices, file systems are built on flash memory
    • File system other than extN is often used
    • cramfs/squashfs/jffs2/ubifs/ etc.
  • Modifying files placed on flash memory is not easy (in many cases)
    • Writing with ROM Writer, JTAG-ICE etc. is cumbersome
    • If failed, further troublesome

link title

(slide 3)Typical Development Environment using NFS

  • Developing software on host system using cross tool chain
  • A file system on the host is used as the root file system on the target via NFS
  • Later written onto flash memory by generating file system image
              (Figure)

(slide 4)Issues when using NFS

  • NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ...
    • Different software configuration in operation (after development) may cause mysterious behavior
    • Debugging problems attributed to file system is impossible/hard


(+) Sun publicly announced NFSv2 in 1984; NFS implemented on Linux since kernel circa 1.2

(slide 5)Networking the Block Device Layer

  • Network connection in block device layer, not file system layer
    • Same file system as on flash memory can be used
  • Network connectable block devices:
    • ATA over Ethernet (AoE)
    • iSCSI (internet SCSI)
    • Network Block Device (NBD)

(slide 6)ATA over Ethernet (AoE) Overview

  • Low cost SAN technology lead by coraid.com
  • Light-weight protocol to communicate by loading ATA command/data on Ethernet frames (does not use TCP/IP)
  • Client side driver mainlined since kernel 2.6.11 (drivers/block/aoe/)
  • Configuration commands (aoetools) available on client side
  • Server side is special hardware or software implementation - userland daemon, kernel driver etc.
    • vblade, kvblade, qaoed, etc.
  • Server side is not required to be an ATA
  • Supported on numbers of OSes and bootloaders

(slide 7)System Structure Diagram of AoE

  • vblade daemon on server publishes disk image file on HDD
 # vblade 0 0 eth0 image_file
  • AoE driver on client presents an image published on server as a block device to file system layer
 # mount -t ext2 /dev/etherd/e0.0 /mnt


    (Figure)

(slide 8)iSCSI Overview

  • SAN technology getting popular in enterprise arena
  • Protocol to communicate by loading SCSI commands/data on TCP/IP
  • Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c)
  • Block devices can be used with the same interface with ordinary SCSI devices (/dev/sda etc.)
  • On client side, configuration with userland tool (Open-iSCSI) is required
  • Server side runs on high-end special hardware as well as various software implementation (iSCSI Enterprise Target, etc.)

(slide 9)System Structure Diagram of iSCSI

  • iSCSI Target on server publishes a disk image file or a partition on HDD
  • iSCSI Initiator on client connects to the target, and presents the disk image as a SCSI disk block device
    (Figure)

(slide 10)Network Block Device (NBD) Overview

  • Treats server data via TCP/IP as block device
  • Protocol irrelevant to ATA (compression supported)
  • Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c)
  • On client side, connection management daemon (nbd-client) required
 # nbd-client server 2000 /dev/nb0
  • Server is implemented as software (nbd-server)
 # nbd-server 2000 image_file
   (Figure)


(slide 11)Use of rootfs with each Method

  • Setting up with command is required for iSCSI or NBD
    • Setting up using initramfs/initrd
    • Direct mounting as rootfs is not possible; must add initramfs/initrd with setting up
--> this is why we use AoE here

(slide 12)MTD Emulation

  • Typical file systems (ext3/cramfs etc.) assume block devices
  • jffs2 etc. assumes MTD, not block devices
  • MTD interface can be used with block2mtd
 # insmod block2mtd.ko block2mtd=/dev/hda1
 # mount -t jffs2 /dev/mtdblock0 /mnt
     (Figure)


(slide 13)AoE for Embedded Development Environment

  • Software developed with cross tool chain in the same manner as when NFS used
  • File system image must be created to be executed on target
  • File system image to be written on flash memory can be directly used as root file system via network with AoE
    (Figure)


(slide 14)Evaluation of AoE Use

  • Evaluation items (Comparison among AoE/NFS/Flash memory)
    • Object size of target kernel
    • Memory usage at boot with identical rootfs
    • Comparison of behavior with compression file system (cramfs)
      • read time of a highly compressible file (all zero) and a less compressible file (random data) of the same data volume
  • Evaluation environment
    • Target: OSK (TI OMAP 192MHz, RAM 32M), kernel 2.6.20.20, 10BASE-T
    • Host: x86_64(3.4GHz), kernel 2.4.21(RHEL3), 1000BASE-T

(slide 15)Evaluation of AoE Use (Size/Memory Usage)

  • Target kernel object size comparison

+----------------------+---------+---------------------+

| AoE | NFS | Flash memory |

+-----------+----------+---------+-----------+---------+

|file system| aoe | NFS |file system| mtd |

| (cramfs) | | client | (cramfs) | |

+-----------+----------+---------+-----------+---------+

| 10,433 | 28,326 | | 10,433 | 73.401 |

+-----------+----------+ +-----------+---------+

| 38,759 | 322,503 | 83,834 |

+----------------------+---------+---------------------+

size of build-in.o in each directory (bytes)
  • Memory usage comparison
    • After booting up with an identical rootfs (busybox), logging in and executing free command

+-------+--------+--------+--------+---------+

| | total | used | free | buffers |

+-------+--------+--------+--------+---------+

| AoE | 30,220 | 5,000 | 25,220 | 724 |

+-------+--------+--------+--------+---------+

| NFS | 29,888 | 4,352 | 25,536 | 0 |

+-------+--------+--------+--------+---------+

| Flash | 30,188 | 5,040 | 25,148 | 736 |

+-------+--------+--------+--------+---------+

free command output (Kbytes)

(slide 16)Evaluation of AoE Use (Behavior)

  • File read time (CPU time)
    • "echo 3 > /proc/sys/vm/drop_caches", then read out with "time wc -c"

+-------+--------------------+--------------------+

| | all zero data | random data |

| +------+------+------+------+------+------+

| | real | user | sys | real | user | sys |

+-------+------+------+------+------+------+------+

| AoE | 1.57 | 0.71 | 0.63 | 7.09 | 0.66 | 1.14 |

+-------+------+------+------+------+------+------+

| NFS | 6.28 | 0.72 | 0.88 | 6.39 | 0.72 | 0.86 |

+-------+------+------+------+------+------+------+

| Flash | 1.44 | 0.72 | 0.61 | 3.23 | 0.69 | 0.88 |

+-------+------+------+------+------+------+------+

unit: second (read 5MB; avarage of 10 trials)


  • Data transfer volume
    • Measured as the difference between /proc/diskstats and /proc/net/dev

+-------+--------------------------+--------------------------+

| | all zero data | random data |

| +------------+-------------+------------+-------------+

| | eth recv | sector read | eth recv | sector read |

+-------+------------+-------------+------------+-------------+

| AoE | 352,040 | 339,968 | 5,770,820 | 5,570,560 |

+-------+------------+-------------+------------+-------------+

| NFS | 5,666,068 | - | 5,663,699 | - |

+-------+------------+-------------+------------+-------------+

| Flash | - | 339,968 | - | 5,570,560 |

+-------+------------+-------------+------------+-------------+

unit: byte (when read 5MB)

(slide 17)Consideration

  • Kernel CPU time reflects the difference of compression processing
    • Expanding highly compressible data (all zero) does not use CPU as much as random data (Assuming that data transfer does not need so much CPU power; if data transfer needs much CPU power, volume of memory copy would be also reflected)
    • In case of NFS, it is independent of data contents as there is no expanding (data transfer volume is also uniform)
  • Data transfer volume is reflects the data contents to be transferred
    • Transferring compressed data causes highly compressible data is transferred as less volume
    • NFS does not compress data, so data transfer volume is constant

(slide 18)Summary

  • AoE is usable even for embedded Linux systems
  • AoE make some debugging impossible with NFS possible
  • Object size is smaller with AoE than with NFS
  • AoE user-friendliness is still evolving

(slide 19)References