Kerneltests survey response

From eLinux.org
Jump to: navigation, search

kerneltests survey response

kerneltests survey response provided by Guenter Roeck

Survey Questions

  • What is the name of your test framework? kerneltests.org

Does your test framework:

source code access

  • access source code repositories for the software under test? yes
  • access source code repositories for the test software? no
  • include the source for the test software? yes
  • provide interfaces for developers to perform code reviews? no.
  • detect that the software under test has a new version? yes
    • if so, how? git hooks
  • detect that the test software has a new version? No.

test definitions

Does your test system:

  • have a test definition repository? No
    • if so, what data format or language is used? n/a

Does your test definition include:

  • source code (or source code location)? https://github.com/groeck/linux-build-test
  • dependency information? Not sure I understand what that is.
  • execution instructions? yes
  • command line variants? no
  • environment variants? not under user control.
  • setup instructions? yes
  • cleanup instructions? no
    • if anything else, please describe:

Does your test system:

  • provide a set of existing tests? Not sure I understand the question.
    • if so, how many?

build management

Does your test system:

  • build the software under test (e.g. the kernel)? yes
  • build the test software? no
  • build other software (such as the distro, libraries, firmware)? no
  • support cross-compilation? yes
  • require a toolchain or build system for the SUT? yes
  • require a toolchain or build system for the test software? yes
  • come with pre-built toolchains? Not published due to repository (github) size limitations, but yes
  • store the build artifacts for generated software? no
    • in what format is the build metadata stored?
    • are the build artifacts stored as raw files or in a database?
      • if a database, what database?

Test scheduling/management

Does your test system:

  • check that dependencies are met before a test is run? n/a
  • schedule the test for the DUT?

DUT questions are n/a: There is no DUT other than qemu, and qemu runs do not need to be scheduled.


    • select an appropriate individual DUT based on SUT or test attributes? n/a
    • reserve the DUT? n/a
    • release the DUT? n/a
  • install the software under test to the DUT?
  • install required packages before a test is run?
  • require particular bootloader on the DUT? n/a
  • deploy the test program to the DUT?
  • prepare the test environment on the DUT?
  • start a monitor (another process to collect data) on the DUT?
  • start a monitor on external equipment?
  • initiate the test on the DUT?
  • clean up the test environment on the DUT?

DUT control

Does your test system:

  • store board configuration data? n/a
    • in what format?
  • store external equipment configuration data? n/a
    • in what format?
  • power cycle the DUT? n/a?
  • monitor the power usage during a run?
  • gather a kernel trace during a run?
  • claim other hardware resources or machines (other than the DUT) for use

during a test?

  • reserve a board for interactive use (ie remove it from automated

testing)?

  • provide a web-based control interface for the lab?
  • provide a CLI control interface for the lab?

DUT question are n/a. All tests are run on qemu

Run artifact handling

Does your test system:

  • store run artifacts yes
    • in what format? database
  • put the run meta-data in a database? yes
    • if so, which database? the database used by buildbot.
  • parse the test logs for results? yes
  • convert data from test logs into a unified format? no
    • if so, what is the format?
  • evaluate pass criteria for a test? yes
  • do you have a common set of result names? yes
    • if so, what are they? per buildbot: SUCCESS,WARNINGS,FAILURE,EXCEPTION,RETRY,SKIPPED
  • How is run data collected from the DUT?
  • How is run data collected from external equipment? n/a
  • Is external equipment data parsed? n/a


User interface

Does your test system:

  • have a visualization system? no
  • show build artifacts to users? no
  • show run artifacts to users? yes
  • do you have a common set of result colors? yes
    • if so, what are they? per buildbot: green - passed, orange - warnings, red - failed, blue - aborted
  • generate reports for test runs? no
  • notify users of test results by e-mail? no
  • can you query (aggregate and filter) the build meta-data? no
  • can you query (aggregate and filter) the run meta-data? no
  • what language or data format is used for online results presentation? html
  • what language or data format is used for reports? (e.g. PDF, excel, etc.) n/a
  • does your test system have a CLI control tool? no
    • what is it called?

Languages:

  • what is the base language of your test framework core? python
  • What languages or data formats is the user required to learn?


Can a user do the following with your test framework:

  • manually request that a test be executed (independent of a CI trigger)? yes
  • see the results of recent tests? yes
  • set the pass criteria for a test? no
    • set the threshold value for a benchmark test? no
    • set the list of testcase results to ignore? no
  • provide a rating for a test? (e.g. give it 4 stars out of 5) no
  • customize a test? no
    • alter the command line for the test program? no
    • alter the environment of the test program? no
    • specify to skip a testcase? no
    • set a new expected value for a test? no
    • edit the test program source? no
  • customize the notification criteria? no
    • customize the notification mechanism? no
  • generate a custom report for a set of runs? no
  • save the report parameters to generate the same report in the future? no

Requirements

Does your test framework:

  • require minimum software on the DUT? n/a
    • If so, what?
  • require minimum hardware on the DUT (e.g. memory) n/a
    • If so, what?
  • require agent software on the DUT? n/a
    • If so, what agent?
  • is there optional agent software or libraries for the DUT?
  • require external hardware in your labs?

APIS

Does your test framework:

  • use existing APIs or data formats to interact within itself, or with 3rd-party modules? no
  • have a published API for any of its sub-module interactions? no
    • Please provide a link or links to the APIs?
  • What is the nature of the APIs you currently use? Whatever buildbot uses

Relationship to other software:

  • what major components does your test framework use?

buildbot, and its dependencies.

  • Buildbot: 0.8.14
  • Twisted: 18.7.0
  • Jinja: 2.10
  • Python: 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609]
  • does your test framework interoperate with other test frameworks or software? No.
    • which ones?

Overview

Please list the major components of your test system.

Per buildbot manual.

Additional Data

Web site: https://kerneltests.org/