Ci-rt survey response

ci-rt survey response

ci-rt survey response provided by Manuel Traut

r4d is no test-system. It's just a tool for managing test-racks and control them by libvirt.

I filled out the survey for 'ci-rt' the testsystem used by RT_PREEMPT developers to test different kernels with different configs on various hardwares. The ci-rt testsystem uses r4d to control the hardware.

diagram missing element

If you have an element that is not featured in this diagram, please let us know.

Dynamically creating test jobs based on a test-description.

Survey Questions

• What is the name of your test framework? ci-rt (ci-rt.linutronix.de)

Does your test framework:

source code access

• access source code repositories for the software under test? yes, via git
• access source code repositories for the test software? yes, via git
• include the source for the test software? test-scripts can be part of the test-description, binaries (e.g. cyclictest) needs to be installed on the DUT.
• provide interfaces for developers to perform code reviews? no
• detect that the software under test has a new version? yes
  • if so, how? (e.g. polling a repository, a git hook, scanning a mail list, etc.) git hook.
• detect that the test software has a new version? the test-description includes a git-ref of the test-software to be used.

test definitions

Does your test system:

• have a test definition repository?
  • if so, what data format or language is used (e.g. yaml, json, shell script) Yes, it's a folder/file layout, the files are either lists that refer to other files, key-value pairs, kernel-patches, or kconfig overlays and kernel configs.

Does your test definition include:

• source code (or source code location)? No
• dependency information? No
• execution instructions? Yes, you can specify if a kernel variant should be deployed to a target and run e.g. cyclictest with different loads; or if it should be just bootet on a target, or just compiled.
• command line variants? Yes, for kernel params and cyclictest.
• environment variants? Yes, for kernel compile.
• setup instructions? No.
• cleanup instructions? No.
  • if anything else, please describe: ci-rt is able to build and test different variants of kernels based on different kernel configs, overlays and architectures.

Does your test system:

• provide a set of existing tests?
  • if so, how many? kernel compile, kernel boot, cyclictest, generictest

build management

Does your test system:

• build the software under test (e.g. the kernel)? Yes and analyze compiler warnings/errors.
• build the test software? No.
• build other software (such as the distro, libraries, firmware)? No.
• support cross-compilation? Yes.
• require a toolchain or build system for the SUT? Yes, toolchains.
• require a toolchain or build system for the test software? N/A.
• come with pre-built toolchains? No. We recommend to use the toolchains that are in Debian/stretch.
• store the build artifacts for generated software? ci-rt generates a compile script per kernel variant. This is stored.
  • in what format is the build metadata stored (e.g. json)? Debian packages.
  • are the build artifacts stored as raw files or in a database? raw-files. Additionally artifacts that are needed to reproduce the tests without ci-rt are stored in a DB that is accesible by a web-application.
    • if a database, what database? postgresql with row-based authentication.

Test scheduling/management

Does your test system:

• check that dependencies are met before a test is run? There is a sanity check for the test-description. Also the requirements on the compile boxes (cross-toolchains) and on the target (cyclictest, ..) are checked.
• schedule the test for the DUT? Yes.
  • select an appropriate individual DUT based on SUT or test attributes? Yes.
  • reserve the DUT? Yes.
  • release the DUT? Yes.
• install the software under test to the DUT? Yes.
• install required packages before a test is run? No.
• require particular bootloader on the DUT? (e.g. grub, uboot, etc.) No.
• deploy the test program to the DUT? Yes. (some scripts)
• prepare the test environment on the DUT? Yes.
• start a monitor (another process to collect data) on the DUT? No.
• start a monitor on external equipment? No.
• initiate the test on the DUT? Yes.
• clean up the test environment on the DUT? Yes.

DUT control

The DUT control system in ci-rt is r4d.

Does your test system:

• store board configuration data? Yes.
  • in what format? R4D
• store external equipment configuration data?
  • in what format? No.
• power cycle the DUT? Yes.
• monitor the power usage during a run? No.
• gather a kernel trace during a run? Yes.
• claim other hardware resources or machines (other than the DUT) for use during a test? No.
• reserve a board for interactive use (ie remove it from automated testing)? Yes.
• provide a web-based control interface for the lab? Yes, via Jenkins
• provide a CLI control interface for the lab? Yes, via Jenkins

Run artifact handling

Does your test system:

• store run artifacts: yes
  • in what format? junit-xml
• put the run meta-data in a database? yes
  • if so, which database? postgresql
• parse the test logs for results? yes - kernel-compile, kernel-boot
• convert data from test logs into a unified format? yes
  • if so, what is the format? junit-xml
• evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)?
• do you have a common set of result names: (e.g. pass, fail, skip, etc.)
  • if so, what are they? cyclictest threshold, no common name.
• How is run data collected from the DUT?
  • e.g. by pushing from the DUT, or pulling from a server? by Jenkins stashes.
• How is run data collected from external equipment? serial bootlog is recorded via r4d/libvirt
• Is external equipment data parsed? Yes, serial bootlog is analyzed.

User interface

Does your test system:

• have a visualization system? Yes, tomcat hosted web-application.
• show build artifacts to users? Yes.
• show run artifacts to users? Yes.
• do you have a common set of result colors? yes
  • if so, what are they? green / red
• generate reports for test runs? Yes.
• notify users of test results by e-mail? Yes. And Administrators about problems with the build-infrastructure. E.g. target is not booting.
• can you query (aggregate and filter) the build meta-data? No.
• can you query (aggregate and filter) the run meta-data? No. Only graphical compare of cyclictest results.
• what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, etc.) HTML, jsp
• what language or data format is used for reports? (e.g. PDF, excel, etc.) txt, junit-xml
• does your test system have a CLI control tool? yes
  • what is it called? jenkins-cli

Languages:

Examples: json, python, yaml, C, javascript, etc.

• what is the base language of your test framework core? groovy
• What languages or data formats is the user required to learn? the format of the ci-rt test-description

Can a user do the following with your test framework:

• manually request that a test be executed (independent of a CI trigger)? yes
• see the results of recent tests? yes
• set the pass criteria for a test? yes, for cyclictest
  • set the threshold value for a benchmark test? yes, for cyclictest
  • set the list of testcase results to ignore? no
• provide a rating for a test? (e.g. give it 4 stars out of 5) no
• customize a test? yes
  • alter the command line for the test program? yes
  • alter the environment of the test program? yes
  • specify to skip a testcase? yes
  • set a new expected value for a test? yes
  • edit the test program source? no
• customize the notification criteria? no
  • customize the notification mechanism (eg. e-mail, text) no
• generate a custom report for a set of runs? no
• save the report parameters to generate the same report in the future? no

Requirements

Does your test framework:

• require minimum software on the DUT? yes
  • If so, what? (e.g. POSIX shell or some other interpreter, specific libraries, command line tools, etc.) openjdk8
• require minimum hardware on the DUT (e.g. memory) no
• require agent software on the DUT? (e.g. extra software besides production software) no
  • If so, what agent?
• is there optional agent software or libraries for the DUT? no
• require external hardware in your labs? Serialserver, Powercontrol

APIS

Does your test framework:

• use existing APIs or data formats to interact within itself, or with 3rd-party modules? junit-xml
• have a published API for any of its sub-module interactions (any of the lines in the diagram)?
  • Please provide a link or links to the APIs?

• DUT Control:
  • R4D - https://github.com/ci-rt/r4d
  • libvirt - https://github.com/ci-rt/libvirt-debian

Sorry - this is kind of open-ended...

• What is the nature of the APIs you currently use?

Are they:

• • RPCs? SOAP
  • Unix-style? (command line invocation, while grabbing sub-tool output)
  • compiled libraries?
  • interpreter modules or libraries?
  • web-based APIs?
  • something else?

Relationship to other software:

• what major components does your test framework use? Jenkins, R4D, libvirt, tomcat, postgresql

• does your test framework interoperate with other test frameworks or software? no
  • which ones?

Overview

Please list the major components of your test system.

• Jenkins - job trigger, visualization, notification
• ci-rt-lib (jenkins, groovy lib) - variant management, test scheduling, kernel build, kernel deploy, log/result retrieval
• libvirt/r4d - access serial console of DUT and power control DUTs
• webapplication - visualization, provide test results public (without having jenkins accesible)
• pyjutest - run commands and produce junit-xml including stdout/err return value, runtime