From eLinux.org
< R-Car‎ | Virtualization
Revision as of 07:59, 15 June 2018 by Kbingham (talk | contribs) (Host Side)
Jump to: navigation, search

Device Pass-Through Using VFIO

Platform Device Pass-Through Prototype

This is a proof-of-concept showing how to provide guest access to an R-Car GPIO controller block on the Renesas Salvator-X and Salvator-XS boards.

Host Side

  • Configure workarounds for missing functionality:
$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
  • Unbind GPIO6 from the "gpio-rcar" driver:
$ echo e6055400.gpio > /sys/bus/platform/drivers/gpio_rcar/unbind
  • Bind GPIO6 to the "vfio-platform" driver, for pass-through to guests:
$ echo vfio-platform > /sys/bus/platform/devices/e6055400.gpio/driver_override
$ echo e6055400.gpio > /sys/bus/platform/drivers/vfio-platform/bind
iommu: Adding device e6055400.gpio to group 0
vfio-platform e6055400.gpio: Adding kernel taint for vfio-noiommu group on device

Guest Side

  • When starting QEMU, specify pass-through of the GPIO6 platform device. The device specifier consists of three parts:
    the device's node name, or
    the full path to the device's node in sysfs,
    the manufacturer part of the device's compatible value,
    the model part of the device's compatible value.


-device vfio-platform,host=$1a,manufacturer=$2,model=$3
-device vfio-platform,host=e6055400.gpio,manufacturer=renesas,model=rcar-gen3-gpio
-device vfio-platform,sysfsdev=$1b,manufacturer=$2,model=$3
-device vfio-platform,sysfsdev=/sys/bus/platform/devices/e6055400.gpio,manufacturer=renesas,model=rcar-gen3-gpio

Full command line:

$ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \
        -kernel /path/to/guest/kernel/Image \
        -device vfio-platform,host=e6055400.gpio,manufacturer=renesas,model=rcar-gen3-gpio

Guest kernel output:

Booting Linux on physical CPU 0x0000000000 [0x411fd073]
Linux version 4.16.0-arm64-virt-00010-g8f416425ed2e81bb ...
gpio_rcar c000000.e6055400.gpio: driving 32 GPIOs

Host kernel output:

vfio-platform e6055400.gpio: reset
vfio-platform e6055400.gpio: vfio-noiommu device opened by user (qemu-system-aar:3003)
  • Analysis of "/sys/firmware/devicetree/base" using "dtx_diff" shows that "e6055400.gpio@0" has been added as a subnode to the existing "platform@c000000" node:
--- /sys/firmware/devicetree/base.orig
+++ /sys/firmware/devicetree/base
@@ -124,6 +124,14 @@
                compatible = "qemu,platform", "simple-bus";
                interrupt-parent = <0x8001>;
                ranges = <0x0 0x0 0xc000000 0x2000000>;
+               e6055400.gpio@0 {
+                       #gpio-cells = <0x2>;
+                       compatible = "renesas,rcar-gen3-gpio";
+                       gpio-controller;
+                       interrupts = <0x0 0x70 0x4>;
+                       reg = <0x0 0x50>;
+               };

        pmu {
  • Export GPIOs used for LEDs (LED4/5/6), and turn all LEDs off:
# base=$(cat /sys/class/gpio/gpiochip*/base)
# for i in 11 12 13; do
    echo $(($base + $i)) > /sys/class/gpio/export
    echo low > /sys/class/gpio/gpio$(($base + $i))/direction
  • Cycle through all LEDs, letting them blink once:
# for i in 11 12 13; do
    echo high > /sys/class/gpio/gpio$(($base + $i))/direction
    sleep 1
    echo low > /sys/class/gpio/gpio$(($base + $i))/direction
  • Change the first GPIO from output to input, and configure edge detection to enable its interrupt:
# echo in > /sys/class/gpio/gpio$(($base + 11))/direction
# echo both > /sys/class/gpio/gpio$(($base + 11))/edge
# grep gpio /proc/interrupts
  5:          0     GIC-0 144 Level     c000000.e6055400.gpio
 42:          0  c000000.e6055400.gpio  11 Edge      gpiolib
  • Monitor the GPIO interrupt (press button SW20 to trigger):
# while /bin/true; do grep gpiolib /proc/interrupts; done | uniq -c
      71298  42:          0  c000000.e6055400.gpio  11 Edge      gpiolib
        388  42:          1  c000000.e6055400.gpio  11 Edge      gpiolib
        247  42:          2  c000000.e6055400.gpio  11 Edge      gpiolib
        139  42:          3  c000000.e6055400.gpio  11 Edge      gpiolib
        586  42:          4  c000000.e6055400.gpio  11 Edge      gpiolib
  • Shut the guest down:
# poweroff

Guest kernel output:

reboot: Power down

Host kernel output:

vfio-platform e6055400.gpio: reset

The GPIO module has been reset, turning on all three LEDs.

Future Work

  • IOMMU groups

Serial Pass through

Creating a serial pass-through will allow you to test VFIO platform pass-through using a serial loopback (USB-A -> USB Micro-B) between a host and guest using the R-Car platform.

Host Side

  • Utilise a host kernel configured as stated in the section Host Side above
  • To work around the lack of dynamic clock handling within QEmu, a [HACK] patch is provided to hardcode the SCIF clocks:
 git://git.kernel.org/pub/scm/linux/kernel/git/kbingham/rcar.git virt/serial-passthrough
  • Configure workarounds for no IOMMU support
$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
  • Unbind the SCIF1 from the "sh-sci" (rcar-gen3-scif compatible) driver:
$ echo e6e68000.serial > /sys/bus/platform/drivers/sh-sci/unbind
  • Rebind the SCIF1 to the vfio-platform driver:
$ echo vfio-platform > /sys/bus/platform/devices/e6e68000.serial/driver_override
$ echo e6e68000.serial > /sys/bus/platform/drivers/vfio-platform/bind

Guest Side

Build QEMU using:

   Repository: https://github.com/kbingham/qemu.git
   Branch: "rcar3/serial/passthrough"

Launch with the Serial passthrough:

   -device vfio-platform,host=e6e68000.serial,manufacturer=renesas,model=rcar-gen3-scif \

Full Q-Emu command:

qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \
   -kernel /path/to/guest/kernel/Image \
   -device vfio-platform,host=e6e68000.serial,manufacturer=renesas,model=rcar-gen3-scif \
   -append "loglevel=7 root=/dev/nfs nfsroot=,nolock,v3 rw ip=dhcp"

Ensure that SLiRP is configured in your QEmu build to support networking, and NFS roots.

Serial loopback test


* Compile and install the sertest utility in both the host, and guest root filesystem.

We can utilise a USB-A -> USB Micro-B cable as a serial-loopback to test the SCIF device passthrough.

  • Connect a USB cable between the Debug Serial 1 port, and a free available USB2 Type A socket.
  • Launch a guest with SCIF pass through as above in Guest Side
  • 'In' the *guest*:
 $ sertest --slave -s 9600 /dev/ttySC0
  • 'On' the *host*:
 $ sertest --master -s 9600 /dev/ttyUSB0