Difference between revisions of "R-Car/Virtualization/VFIO"
(→Host Side) |
(IOMMU groups work when using a device tied to an IOMMU (e.g. SATA)) |
||
(2 intermediate revisions by the same user not shown) | |||
Line 3: | Line 3: | ||
Device Pass-Through Using VFIO | Device Pass-Through Using VFIO | ||
− | = | + | = GPIO Pass-Through Prototype = |
This is a proof-of-concept showing how to provide guest access to an R-Car GPIO controller block on the Renesas [[R-Car/Boards/Salvator-X|Salvator-X]] and [[R-Car/Boards/Salvator-XS|Salvator-XS]] boards. | This is a proof-of-concept showing how to provide guest access to an R-Car GPIO controller block on the Renesas [[R-Car/Boards/Salvator-X|Salvator-X]] and [[R-Car/Boards/Salvator-XS|Salvator-XS]] boards. | ||
Line 11: | Line 11: | ||
* Build and boot a host kernel using: | * Build and boot a host kernel using: | ||
** Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git | ** Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git | ||
− | ** Branch: "topic/rcar3-virt-gpio-passthrough- | + | ** Branch: "topic/rcar3-virt-gpio-passthrough-v3" |
** Config: "renesas_defconfig" | ** Config: "renesas_defconfig" | ||
Line 37: | Line 37: | ||
* Build a guest kernel using: | * Build a guest kernel using: | ||
** Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git | ** Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git | ||
− | ** Branch: "topic/rcar3-virt-gpio-passthrough- | + | ** Branch: "topic/rcar3-virt-gpio-passthrough-v3" |
** Config: "virt_defconfig" | ** Config: "virt_defconfig" | ||
* Build QEMU using: | * Build QEMU using: | ||
** Repository: https://github.com/geertu/qemu.git | ** Repository: https://github.com/geertu/qemu.git | ||
− | ** Branch: "topic/rcar3-virt-gpio-passthrough- | + | ** Branch: "topic/rcar3-virt-gpio-passthrough-v3" |
− | * When starting QEMU, specify pass-through of the GPIO6 platform device. The device specifier consists of | + | * When starting QEMU, specify pass-through of the GPIO6 platform device. The device specifier consists of either the device's node name, or the full path to the device's node in sysfs. |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
I.e.: | I.e.: | ||
<pre> | <pre> | ||
− | + | -device vfio-platform,host=e6055400.gpio | |
− | -device vfio-platform,host=e6055400. | + | -device vfio-platform,sysfsdev=/sys/bus/platform/devices/e6055400.gpio |
− | |||
− | -device vfio-platform,sysfsdev=/sys/bus/platform/devices/e6055400. | ||
</pre> | </pre> | ||
Line 65: | Line 55: | ||
$ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ | $ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ | ||
-kernel /path/to/guest/kernel/Image \ | -kernel /path/to/guest/kernel/Image \ | ||
− | -device vfio-platform,host=e6055400. | + | -device vfio-platform,host=e6055400.gpio |
</pre> | </pre> | ||
Line 71: | Line 61: | ||
<pre> | <pre> | ||
Booting Linux on physical CPU 0x0000000000 [0x411fd073] | Booting Linux on physical CPU 0x0000000000 [0x411fd073] | ||
− | Linux version 4. | + | Linux version 4.18.0-rc6-arm64-virt-00012-g33dccc007797d60a ... |
... | ... | ||
gpio_rcar c000000.e6055400.gpio: driving 32 GPIOs | gpio_rcar c000000.e6055400.gpio: driving 32 GPIOs | ||
Line 87: | Line 77: | ||
--- /sys/firmware/devicetree/base.orig | --- /sys/firmware/devicetree/base.orig | ||
+++ /sys/firmware/devicetree/base | +++ /sys/firmware/devicetree/base | ||
− | @@ - | + | @@ -126,6 +126,14 @@ |
compatible = "qemu,platform", "simple-bus"; | compatible = "qemu,platform", "simple-bus"; | ||
interrupt-parent = <0x8001>; | interrupt-parent = <0x8001>; | ||
Line 94: | Line 84: | ||
+ e6055400.gpio@0 { | + e6055400.gpio@0 { | ||
+ #gpio-cells = <0x2>; | + #gpio-cells = <0x2>; | ||
− | + compatible = "renesas,rcar-gen3-gpio"; | + | + #interrupt-cells = <0x2>; |
+ | + compatible = "renesas,gpio-r8a7795", "renesas,rcar-gen3-gpio"; | ||
+ gpio-controller; | + gpio-controller; | ||
+ | + interrupt-controller; | ||
+ interrupts = <0x0 0x70 0x4>; | + interrupts = <0x0 0x70 0x4>; | ||
+ reg = <0x0 0x50>; | + reg = <0x0 0x50>; | ||
Line 159: | Line 151: | ||
The GPIO module has been reset, turning on all three LEDs. | The GPIO module has been reset, turning on all three LEDs. | ||
− | |||
− | |||
− | |||
= Serial Pass through = | = Serial Pass through = | ||
Line 213: | Line 202: | ||
* 'On' the *host*: | * 'On' the *host*: | ||
$ sertest --master -s 9600 /dev/ttyUSB0 | $ sertest --master -s 9600 /dev/ttyUSB0 | ||
+ | |||
+ | |||
+ | = SATA Pass-Through = | ||
+ | |||
+ | This shows how to provide guest access to an R-Car SATA controller block on the Renesas [[R-Car/Boards/Salvator-X|Salvator-X]] and [[R-Car/Boards/Salvator-XS|Salvator-XS]] boards. | ||
+ | |||
+ | == Host Side == | ||
+ | |||
+ | * Connect a SATA device to the SATA port on Salvator-X(S) (R-Car H3 only!). | ||
+ | |||
+ | * Build and boot a host kernel according to section [[R-Car/Virtualization/VFIO#Host_Side|Host Side]] above. | ||
+ | |||
+ | * Unbind SATA from the "sata_rcar" driver (actual SATA device output may vary): | ||
+ | <pre> | ||
+ | $ echo ee300000.sata > /sys/bus/platform/drivers/sata_rcar/unbind | ||
+ | ata1.00: disabled | ||
+ | sd 0:0:0:0: [sda] Synchronizing SCSI cache | ||
+ | sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00 | ||
+ | sd 0:0:0:0: [sda] Stopping disk | ||
+ | sd 0:0:0:0: [sda] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00 | ||
+ | </pre> | ||
+ | |||
+ | * Bind SATA to the "vfio-platform" driver, for pass-through to guests: | ||
+ | <pre> | ||
+ | $ echo vfio-platform > /sys/bus/platform/devices/ee300000.sata/driver_override | ||
+ | $ echo ee300000.sata > /sys/bus/platform/drivers/vfio-platform/bind | ||
+ | </pre> | ||
+ | |||
+ | == Guest Side == | ||
+ | |||
+ | * Build a guest kernel and QEMU according to section [[R-Car/Virtualization/VFIO#Guest_Side|Guest Side]] above. | ||
+ | |||
+ | * Launch QEMU: | ||
+ | <pre> | ||
+ | $ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ | ||
+ | -kernel /path/to/guest/kernel/Image \ | ||
+ | -device vfio-platform,host=ee300000.sata | ||
+ | </pre> | ||
+ | |||
+ | Guest kernel output (actual SATA device output may vary): | ||
+ | <pre> | ||
+ | Booting Linux on physical CPU 0x0000000000 [0x411fd073] | ||
+ | Linux version 4.18.0-rc6-arm64-virt-00012-g33dccc007797d60a ... | ||
+ | ... | ||
+ | scsi host0: sata_rcar | ||
+ | ata1: SATA max UDMA/133 irq 5 | ||
+ | ... | ||
+ | ata1: link resume succeeded after 1 retries | ||
+ | ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300) | ||
+ | ata1.00: ATA-7: Maxtor 6L160M0, BANC1G10, max UDMA/133 | ||
+ | ata1.00: 320173056 sectors, multi 0: LBA48 NCQ (not used) | ||
+ | ata1.00: configured for UDMA/133 | ||
+ | scsi 0:0:0:0: Direct-Access ATA Maxtor 6L160M0 1G10 PQ: 0 ANSI: 5 | ||
+ | sd 0:0:0:0: [sda] 320173056 512-byte logical blocks: (164 GB/153 GiB) | ||
+ | sd 0:0:0:0: [sda] Write Protect is off | ||
+ | sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA | ||
+ | sda: sda1 | ||
+ | sd 0:0:0:0: [sda] Attached SCSI disk | ||
+ | ... | ||
+ | </pre> | ||
+ | |||
+ | Host kernel output: | ||
+ | <pre> | ||
+ | vfio-platform ee300000.sata: Using IPMMU context 1 | ||
+ | vfio-platform ee300000.sata: reset | ||
+ | </pre> | ||
+ | |||
+ | * Analysis of "/sys/firmware/devicetree/base" using "dtx_diff" shows that "ee300000.sata@0" has been added as a subnode to the existing "platform@c000000" node: | ||
+ | <pre> | ||
+ | --- /sys/firmware/devicetree/base.orig | ||
+ | +++ /sys/firmware/devicetree/base | ||
+ | @@ -126,6 +126,12 @@ | ||
+ | compatible = "qemu,platform", "simple-bus"; | ||
+ | interrupt-parent = <0x8001>; | ||
+ | ranges = <0x0 0x0 0xc000000 0x2000000>; | ||
+ | + | ||
+ | + ee300000.sata@0 { | ||
+ | + compatible = "renesas,sata-r8a7795", "renesas,rcar-gen3-sata"; | ||
+ | + interrupts = <0x0 0x70 0x4>; | ||
+ | + reg = <0x0 0x200000>; | ||
+ | + }; | ||
+ | }; | ||
+ | |||
+ | pmu { | ||
+ | </pre> | ||
+ | |||
+ | * Depending on the SATA device, you can mount a partition. | ||
+ | |||
+ | * Shut the guest down: | ||
+ | <pre> | ||
+ | # poweroff | ||
+ | </pre> | ||
+ | |||
+ | Guest kernel output: | ||
+ | <pre> | ||
+ | reboot: Power down | ||
+ | </pre> | ||
+ | |||
+ | Host kernel output: | ||
+ | <pre> | ||
+ | vfio-platform ee300000.sata: reset | ||
+ | vfio-platform ee300000.sata: Reusing IPMMU context 0 | ||
+ | </pre> |
Latest revision as of 10:46, 25 July 2018
Device Pass-Through Using VFIO
GPIO Pass-Through Prototype
This is a proof-of-concept showing how to provide guest access to an R-Car GPIO controller block on the Renesas Salvator-X and Salvator-XS boards.
Host Side
- Build and boot a host kernel using:
- Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git
- Branch: "topic/rcar3-virt-gpio-passthrough-v3"
- Config: "renesas_defconfig"
- Configure workarounds for missing functionality:
$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
- Unbind GPIO6 from the "gpio-rcar" driver:
$ echo e6055400.gpio > /sys/bus/platform/drivers/gpio_rcar/unbind gpio gpiochip6: REMOVING GPIOCHIP WITH GPIOS STILL REQUESTED
- Bind GPIO6 to the "vfio-platform" driver, for pass-through to guests:
$ echo vfio-platform > /sys/bus/platform/devices/e6055400.gpio/driver_override $ echo e6055400.gpio > /sys/bus/platform/drivers/vfio-platform/bind iommu: Adding device e6055400.gpio to group 0 vfio-platform e6055400.gpio: Adding kernel taint for vfio-noiommu group on device
Guest Side
- Build a guest kernel using:
- Repository: https://git.kernel.org/pub/scm/linux/kernel/git/geert/renesas-drivers.git
- Branch: "topic/rcar3-virt-gpio-passthrough-v3"
- Config: "virt_defconfig"
- Build QEMU using:
- Repository: https://github.com/geertu/qemu.git
- Branch: "topic/rcar3-virt-gpio-passthrough-v3"
- When starting QEMU, specify pass-through of the GPIO6 platform device. The device specifier consists of either the device's node name, or the full path to the device's node in sysfs.
I.e.:
-device vfio-platform,host=e6055400.gpio -device vfio-platform,sysfsdev=/sys/bus/platform/devices/e6055400.gpio
Full command line:
$ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ -kernel /path/to/guest/kernel/Image \ -device vfio-platform,host=e6055400.gpio
Guest kernel output:
Booting Linux on physical CPU 0x0000000000 [0x411fd073] Linux version 4.18.0-rc6-arm64-virt-00012-g33dccc007797d60a ... ... gpio_rcar c000000.e6055400.gpio: driving 32 GPIOs ...
Host kernel output:
vfio-platform e6055400.gpio: reset vfio-platform e6055400.gpio: vfio-noiommu device opened by user (qemu-system-aar:3003)
- Analysis of "/sys/firmware/devicetree/base" using "dtx_diff" shows that "e6055400.gpio@0" has been added as a subnode to the existing "platform@c000000" node:
--- /sys/firmware/devicetree/base.orig +++ /sys/firmware/devicetree/base @@ -126,6 +126,14 @@ compatible = "qemu,platform", "simple-bus"; interrupt-parent = <0x8001>; ranges = <0x0 0x0 0xc000000 0x2000000>; + + e6055400.gpio@0 { + #gpio-cells = <0x2>; + #interrupt-cells = <0x2>; + compatible = "renesas,gpio-r8a7795", "renesas,rcar-gen3-gpio"; + gpio-controller; + interrupt-controller; + interrupts = <0x0 0x70 0x4>; + reg = <0x0 0x50>; + }; }; pmu {
- Export GPIOs used for LEDs (LED4/5/6), and turn all LEDs off:
# base=$(cat /sys/class/gpio/gpiochip*/base) # for i in 11 12 13; do echo $(($base + $i)) > /sys/class/gpio/export echo low > /sys/class/gpio/gpio$(($base + $i))/direction done
- Cycle through all LEDs, letting them blink once:
# for i in 11 12 13; do echo high > /sys/class/gpio/gpio$(($base + $i))/direction sleep 1 echo low > /sys/class/gpio/gpio$(($base + $i))/direction done
- Change the first GPIO from output to input, and configure edge detection to enable its interrupt:
# echo in > /sys/class/gpio/gpio$(($base + 11))/direction # echo both > /sys/class/gpio/gpio$(($base + 11))/edge # grep gpio /proc/interrupts 5: 0 GIC-0 144 Level c000000.e6055400.gpio 42: 0 c000000.e6055400.gpio 11 Edge gpiolib
- Monitor the GPIO interrupt (press button SW20 to trigger):
# while /bin/true; do grep gpiolib /proc/interrupts; done | uniq -c 71298 42: 0 c000000.e6055400.gpio 11 Edge gpiolib 388 42: 1 c000000.e6055400.gpio 11 Edge gpiolib 247 42: 2 c000000.e6055400.gpio 11 Edge gpiolib 139 42: 3 c000000.e6055400.gpio 11 Edge gpiolib 586 42: 4 c000000.e6055400.gpio 11 Edge gpiolib ^C
- Shut the guest down:
# poweroff
Guest kernel output:
reboot: Power down
Host kernel output:
vfio-platform e6055400.gpio: reset
The GPIO module has been reset, turning on all three LEDs.
Serial Pass through
Creating a serial pass-through will allow you to test VFIO platform pass-through using a serial loopback (USB-A -> USB Micro-B) between a host and guest using the R-Car platform.
Host Side
- Utilise a host kernel configured as stated in the section Host Side above
- To work around the lack of dynamic clock handling within QEmu, a [HACK] patch is provided to hardcode the SCIF clocks:
git://git.kernel.org/pub/scm/linux/kernel/git/kbingham/rcar.git virt/serial-passthrough
- Configure workarounds for no IOMMU support
$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
- Unbind the SCIF1 from the "sh-sci" (rcar-gen3-scif compatible) driver:
$ echo e6e68000.serial > /sys/bus/platform/drivers/sh-sci/unbind
- Rebind the SCIF1 to the vfio-platform driver:
$ echo vfio-platform > /sys/bus/platform/devices/e6e68000.serial/driver_override $ echo e6e68000.serial > /sys/bus/platform/drivers/vfio-platform/bind
Guest Side
Build QEMU using:
Repository: https://github.com/kbingham/qemu.git Branch: "rcar3/serial/passthrough"
Launch with the Serial passthrough:
-device vfio-platform,host=e6e68000.serial,manufacturer=renesas,model=rcar-gen3-scif \
Full Q-Emu command:
qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ -kernel /path/to/guest/kernel/Image \ -device vfio-platform,host=e6e68000.serial,manufacturer=renesas,model=rcar-gen3-scif \ -append "loglevel=7 root=/dev/nfs nfsroot=192.168.0.39:/path/to/nfsroot/,nolock,v3 rw ip=dhcp"
Ensure that SLiRP is configured in your QEmu build to support networking, and NFS roots.
Serial loopback test
Dependencies:
https://github.com/geertu/sertest * Compile and install the sertest utility in both the host, and guest root filesystem.
We can utilise a USB-A -> USB Micro-B cable as a serial-loopback to test the SCIF device passthrough.
- Connect a USB cable between the Debug Serial 1 port, and a free available USB2 Type A socket.
- Launch a guest with SCIF pass through as above in Guest Side
- 'In' the *guest*:
$ sertest --slave -s 9600 /dev/ttySC0
- 'On' the *host*:
$ sertest --master -s 9600 /dev/ttyUSB0
SATA Pass-Through
This shows how to provide guest access to an R-Car SATA controller block on the Renesas Salvator-X and Salvator-XS boards.
Host Side
- Connect a SATA device to the SATA port on Salvator-X(S) (R-Car H3 only!).
- Build and boot a host kernel according to section Host Side above.
- Unbind SATA from the "sata_rcar" driver (actual SATA device output may vary):
$ echo ee300000.sata > /sys/bus/platform/drivers/sata_rcar/unbind ata1.00: disabled sd 0:0:0:0: [sda] Synchronizing SCSI cache sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00 sd 0:0:0:0: [sda] Stopping disk sd 0:0:0:0: [sda] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00
- Bind SATA to the "vfio-platform" driver, for pass-through to guests:
$ echo vfio-platform > /sys/bus/platform/devices/ee300000.sata/driver_override $ echo ee300000.sata > /sys/bus/platform/drivers/vfio-platform/bind
Guest Side
- Build a guest kernel and QEMU according to section Guest Side above.
- Launch QEMU:
$ qemu-system-aarch64 -enable-kvm -M virt -cpu cortex-a57 -m 1024 -nographic \ -kernel /path/to/guest/kernel/Image \ -device vfio-platform,host=ee300000.sata
Guest kernel output (actual SATA device output may vary):
Booting Linux on physical CPU 0x0000000000 [0x411fd073] Linux version 4.18.0-rc6-arm64-virt-00012-g33dccc007797d60a ... ... scsi host0: sata_rcar ata1: SATA max UDMA/133 irq 5 ... ata1: link resume succeeded after 1 retries ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata1.00: ATA-7: Maxtor 6L160M0, BANC1G10, max UDMA/133 ata1.00: 320173056 sectors, multi 0: LBA48 NCQ (not used) ata1.00: configured for UDMA/133 scsi 0:0:0:0: Direct-Access ATA Maxtor 6L160M0 1G10 PQ: 0 ANSI: 5 sd 0:0:0:0: [sda] 320173056 512-byte logical blocks: (164 GB/153 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sda1 sd 0:0:0:0: [sda] Attached SCSI disk ...
Host kernel output:
vfio-platform ee300000.sata: Using IPMMU context 1 vfio-platform ee300000.sata: reset
- Analysis of "/sys/firmware/devicetree/base" using "dtx_diff" shows that "ee300000.sata@0" has been added as a subnode to the existing "platform@c000000" node:
--- /sys/firmware/devicetree/base.orig +++ /sys/firmware/devicetree/base @@ -126,6 +126,12 @@ compatible = "qemu,platform", "simple-bus"; interrupt-parent = <0x8001>; ranges = <0x0 0x0 0xc000000 0x2000000>; + + ee300000.sata@0 { + compatible = "renesas,sata-r8a7795", "renesas,rcar-gen3-sata"; + interrupts = <0x0 0x70 0x4>; + reg = <0x0 0x200000>; + }; }; pmu {
- Depending on the SATA device, you can mount a partition.
- Shut the guest down:
# poweroff
Guest kernel output:
reboot: Power down
Host kernel output:
vfio-platform ee300000.sata: reset vfio-platform ee300000.sata: Reusing IPMMU context 0