Difference between revisions of "Kernel build bundle"

From eLinux.org
Jump to: navigation, search
(tools that pass data around)
(kernel build system)
 
(4 intermediate revisions by the same user not shown)
Line 1: Line 1:
This page describes a standard proposal for a "kernel build bundle".
+
This page describes a proposed standard for a "kernel build bundle".
  
This is used to describe the build artifacts from a kernel build, that can be referenced and passed
+
This is used to describe the build artifacts from a kernel build, that can be
to other tools - mainly a provisioning tool, to install a kernel into a test system.
+
referenced and passed to other tools - mainly a provisioning tool, to install
 +
a kernel into a test system.
  
 
This is not to be confused with boot standards, although those are related.
 
This is not to be confused with boot standards, although those are related.
  
It is anticipated that a kernel build system will produce a kernel build bundle (or a metadata file
+
It is anticipated that a kernel build system will produce a kernel build bundle
that references the kernel build artifacts), and that a test framework might use this to actually
+
(or a metadata file that references the kernel build artifacts), and that a test
provision a system.
+
framework might use this to actually provision a system.
  
 
= pre-existing work and standards =
 
= pre-existing work and standards =
 
== kernel build systems ==
 
== kernel build systems ==
 
=== kernel build system ===
 
=== kernel build system ===
* Linux kernel kbuild system (Makefiles and kbuild)
+
The Linux kernel has a 'make install' target, that will populate a directory
 +
with artifacts from a build (or run a script to do so)
 +
 
 +
The logic for this is implement in the Makefiles scattered throughout the
 +
kernel source tree.
 +
 
 +
By default, 'make install' will run the script install.sh in the respective
 +
boot directory for the configured architecture.  That is /arch/x86/boot/install.sh, etc.
 +
 
 +
[should document this more]
  
 
=== kernelci ===
 
=== kernelci ===
Line 27: Line 37:
  
 
It also has a modules.tar.xz file in the top directory.
 
It also has a modules.tar.xz file in the top directory.
 +
 +
==== Bundle json file ====
 +
Here is a "bundle file:
 +
https://s3.amazonaws.com/storage.lkft.org/rootfs/oe-sumo/20211006/juno/bundle.json
 +
It has the following fields:
 +
* kernel_config - name of kernel config file (that was used for this build
 +
* dtb - name of the dtb file
 +
* kernel - name of the kernel image file for this bundle
 +
* rootfs - names of possible rootfs image files for this bundle
 +
** ext4gz - name of the ext4.gz file for the rootfs
 +
** tarxz - name of the tar.xz file for the rootfs
 +
* modules - name of the archive file containing the modules
 +
* modules_compression - type of compression for the mdoules file (eg. 'targz')
 +
 +
Daniel Diaz provided a link to a tool that takes an online directory, and creates
 +
a bundle of files that can be used to install the kernel.  It's called 'dir2bundle'
 +
and it's here:
 +
https://gitlab.com/Linaro/lkft/rootfs/dir2bundle
 +
 +
According to the description, 'dir2bundle' reads the contents of an images/ directory and
 +
create a bundle.json file.  It can read the contents of an online directory, or a
 +
local directory, to create the file.
  
 
=== tuxmake ===
 
=== tuxmake ===
Tuxmake is a tool to write
+
Tuxmake is a tool to handle kernel builds in a reproducible way.
 +
 
 +
See https://www.tuxmake.org/metadata/ for an (incomplete) description of their metadata
 +
Here is an online directory with build results.  But this build didn't work, so the build
 +
artifacts are not there.
 +
 
 +
https://builds.tuxbuild.com/1wK6XBsWrtdspwuwyXX9cocbnXR/
  
 
== kernel installation systems ==
 
== kernel installation systems ==

Latest revision as of 12:31, 21 October 2021

This page describes a proposed standard for a "kernel build bundle".

This is used to describe the build artifacts from a kernel build, that can be referenced and passed to other tools - mainly a provisioning tool, to install a kernel into a test system.

This is not to be confused with boot standards, although those are related.

It is anticipated that a kernel build system will produce a kernel build bundle (or a metadata file that references the kernel build artifacts), and that a test framework might use this to actually provision a system.

pre-existing work and standards

kernel build systems

kernel build system

The Linux kernel has a 'make install' target, that will populate a directory with artifacts from a build (or run a script to do so)

The logic for this is implement in the Makefiles scattered throughout the kernel source tree.

By default, 'make install' will run the script install.sh in the respective boot directory for the configured architecture. That is /arch/x86/boot/install.sh, etc.

[should document this more]

kernelci

The kernelci system builds lots of kernels from mainline trees, with lots of different configurations.

Here is an example output manifest from one build: https://storage.kernelci.org/broonie-regulator/for-next/v5.15-rc1-lava-bisection-9108-pass/arm64/defconfig/gcc-8/artifacts.json

This has a description of information for this online directory: https://storage.kernelci.org/broonie-regulator/for-next/v5.15-rc1-lava-bisection-9108-pass/arm64/defconfig/gcc-8

which has sub-directories: config, dtbs, kernel, and logs

It also has a modules.tar.xz file in the top directory.

Bundle json file

Here is a "bundle file: https://s3.amazonaws.com/storage.lkft.org/rootfs/oe-sumo/20211006/juno/bundle.json It has the following fields:

  • kernel_config - name of kernel config file (that was used for this build
  • dtb - name of the dtb file
  • kernel - name of the kernel image file for this bundle
  • rootfs - names of possible rootfs image files for this bundle
    • ext4gz - name of the ext4.gz file for the rootfs
    • tarxz - name of the tar.xz file for the rootfs
  • modules - name of the archive file containing the modules
  • modules_compression - type of compression for the mdoules file (eg. 'targz')

Daniel Diaz provided a link to a tool that takes an online directory, and creates a bundle of files that can be used to install the kernel. It's called 'dir2bundle' and it's here: https://gitlab.com/Linaro/lkft/rootfs/dir2bundle

According to the description, 'dir2bundle' reads the contents of an images/ directory and create a bundle.json file. It can read the contents of an online directory, or a local directory, to create the file.

tuxmake

Tuxmake is a tool to handle kernel builds in a reproducible way.

See https://www.tuxmake.org/metadata/ for an (incomplete) description of their metadata Here is an online directory with build results. But this build didn't work, so the build artifacts are not there.

https://builds.tuxbuild.com/1wK6XBsWrtdspwuwyXX9cocbnXR/

kernel installation systems

There are multiple ways of installing a kernel so that it will boot on a board. Some of these installating methods are persistent (e.g. installed to a filesystem on the device, or a partition on the device), and some are ephemeral. Some examples of this later category are placing the kernel in a tftp directory on a host, or transferring a kernel into ram and executing it using adb.

  • tftp boot
  • etherboot
  • /boot directory on filesystem on device (used by grub)
  • in it's own flash partition
  • in it's own disk partition

tools that pass data around

bootline 'custom test tools'

Bootline has a tool called "custom_tests_tool" that can send the materials for kernelci job to a server:

See this tool: https://github.com/bootlin/custom_tests_tool/blob/master/ctt.py

Some notes:

  • CTTLauncher instance creates an 'artifacts' map, which has:
    • kernel
    • dtb
    • modules
    • rootfs

For any files mentioned in the _cfg, the launch() method can send the file to the REMOTE_ROOT, and change the reference to be local-relative (prepended with 'file://'), before submitting the job.