R-Car/Virtualization
Virtualization on Renesas R-Car Platforms
Virtualization with Xen hypervisor
Xen supports R-Car chips out of the box. Example instructions on how to run Xen on R-Car H3/H2 can be found @ XenProject Wiki:
Virtualization with QEMU
QEMU is a generic and open source machine emulator and virtualizer. It emulates CPUs and provides a set of emulated devices to support running unmodified guest operating systems. When running on Linux hosts, it can use the host's hardware virtualization extensions (if supported) through the KVM API to run guests at near-native speed.
Quickstart
- Install QEMU (e.g. "apt-get install qemu-system-arm")
- Download a guest image (e.g. "openwrt-arm64-qemu-virt.Image" from OpenWRT)
- Start QEMU with
$ qemu-system-aarch64 -m 1024 -cpu cortex-a57 -M virt -nographic -kernel openwrt-arm64-qemu-virt.Image
and press <ENTER> to enjoy your new ARM64 system!
Installing QEMU
When not using a binary distribution that provides a QEMU package, QEMU can be cross-compiled from sources, manually or with the help of a cross-compilation build system such as BuildRoot or OpenEmbedded.
Manual cross-compilation
QEMU supports a wide variety of options, in particular related to emulation of architectures or various classes of devices. The instructions given here only enable a minimal set of options to reduce the required dependencies. We only enable the aarch64-softmmu target architecture, FDT for ARM Linux guest support, KVM for hardware virtualization acceleration and SDL for display acceleration.
The following packages are needed on the host to cross-compile QEMU.
- pkg-config
- Python 2.x
The following packages are needed in the host build environment to cross-compile and on the target to run QEMU.
- DTC (Device Tree Compiler)
- GLib 2.x
- Pixman
- SDL (1.x or 2.x)
- zlib
Other options can be enabled as needed and may require extra dependencies.
After obtaining the QEMU sources, the build must be setup. The following environment variables must be set.
- CROSS_COMPILE
- Cross-compilation toolchain prefix (e.g. aarch64-linux-gnu-).
- PKG_CONFIG
- Path to the pkg-config script for the cross-compilation environment (typically ${HOST_DIR}/bin/pkg-config).
- The script must set the PKG_CONFIG_LIBDIR environment variable and execute the pkg-config binary.
- SDL_CONFIG
- Path to the sdl-config script (typically ${STAGING_DIR}/usr/bin/sdl-config).
Additionally, if not using the host system Python, the following environment variables must be set.
- PYTHON
- Path to the host build environment Python 2 binary.
- PYTHONPATH
- Search file for Python module files for the Python interpreter referenced by $PYTHON.
You can then run the configure script.
$ ./configure \ --prefix=/usr --cross-prefix=${CROSS_COMPILE} \ --target-list=aarch64-softmmu \ --enable-attr --enable-fdt --enable-kvm \ --enable-sdl --enable-system --enable-tools \ --audio-drv-list= \ --disable-bluez --disable-brlapi --disable-bsd-user \ --disable-cap-ng --disable-curl --disable-curses \ --disable-docs --disable-libiscsi --disable-linux-aio \ --disable-rbd --disable-seccomp --disable-slirp \ --disable-sparse --disable-spice --disable-strip \ --disable-usb-redir --disable-vde --disable-virtfs \ --disable-vnc --disable-werror --disable-xen
QEMU can then be compiled and installed and installed to the destination directory $DESTDIR.
$ make $ make DESTDIR=${DESTDIR} install
Running QEMU
Recent kernels may access ARM64 registers that require a recent version of QEMU. If your kernel crashes with
Code: ........ ........ ........ ........ (d5380740) Internal error: undefined instruction: 0 [#1] SMP ... pc : __cpuinfo_store_cpu+0x68/0x1b0
you have to upgrade QEMU, or comment out the following two lines in arch/arm64/kernel/cpuinfo.c:__cpuinfo_store_cpu():
info->reg_id_aa64mmfr2 = read_cpuid(ID_AA64MMFR2_EL1); info->reg_id_aa64zfr0 = read_cpuid(ID_AA64ZFR0_EL1);
Using KVM
When using an arm64 platform as the host system, guest performance can be improved drastically by making use of the Kernel-based Virtual Machine (KVM).
Host Kernel Configuration
Make sure your host kernel has the following options enabled:
CONFIG_VIRTUALIZATION=y CONFIG_KVM=y
"renesas_defconfig" should be fine.
Firmware Support
To use KVM, the firmware on your board must start Linux in hypervisor (EL2 / HYP) mode.
If your host kernel prints:
CPU: All CPU(s) started at EL2 ... kvm [1]: Hyp mode initialized successfully
during boot up, everything is fine, and you can skip the next section.
If your host kernel prints:
CPU: All CPU(s) started at EL1 ... kvm [1]: HYP mode not available
during bootup, HYP mode is not available, and KVM cannot be used, unless you first replace the firmware of your board with a version that supports HYP mode.
Enabling HYP Support
To build your own ARM Trusted Firmware with hypervisor mode support:
$ git clone https://github.com/renesas-rcar/arm-trusted-firmware.git $ cd arm-trusted-firmware $ make CROSS_COMPILE=aarch64-linux-gnu- PLAT=rcar RCAR_DRAM_SPLIT=3 RCAR_BL33_EXECUTION_EL=1 LSI=H3 # or LSI=M3
Replace the BL2 and BL31 binaries in your firmware package by the generated "build/rcar/release/bl2.srec" and "build/rcar/release/bl31.srec", and follow your normal firmware upgrade procedure.
Running QEMU with KVM
$ qemu-system-aarch64 -enable-kvm -m 1024 -cpu cortex-a57 -M virt -nographic -kernel openwrt-arm64-qemu-virt.Image
and press <ENTER> to enjoy a faster new ARM64 system!
Note that the above command requires running on a Cortex-A57 CPU core. When running on a Cortex-A53 CPU core in a big.LITTLE configuration, the command will fail with:
kvm_init_vcpu failed: Invalid argument
Just retry, or force your luck by offlining all Cortex-A53 cores first:
for i in $(grep -lr arm,cortex-a53 /sys/bus/cpu/devices/cpu*/of_node/compatible); do echo 0 > $(dirname $(dirname $i))/online done
Note: The same is true for running with '-cpu cortex-a53', with a53 and a57 exchanged.
Device Pass-Through
See VFIO
Managing virtual machines with libvirt
See Libvirt.
Building your own guest kernel
Kernel Config
The kernel config file used for the OpenWRT guest image is a good starting point. You can extract it from '/proc/config.gz' on the running system.
RAM Disk
You can extract the initramfs from an OpenWRT guest image using 'binwalk'. Make sure to truncate the initramfs file after the first 256-byte boundary after "TRAILER!!!", else the kernel will crash with
Kernel panic - not syncing: broken padding