Kselftest survey response

From eLinux.org
Jump to: navigation, search

Kselftest survey response

Kselftest survey response provided by Shuah Khan

Survey Questions

  • What is the name of your test framework? Kernel Selftest (Kselftest)


Does your test framework:

source code access

  • access source code repositories for the software under test Yes
  • access source code repositories for the test software? Yes
  • include the source for the test software? Yes
  • provide interfaces for developers to perform code reviews? Yes
  • detect that the software under test has a new version? no

This question isn't applicable to Kselftest. Kselftest sources are included in the Kernel sources. Individual tests are expected to detect if the kernel they are running on supports the features, modules, and other dependencies for the test code to run. If kernel doesn't have the support, individual tests skip the test and report skip.

    • if so, how?
  • detect that the test software has a new version? n/a

This question isn't applicable to Kselftest. Running Kselftest from the latest mainline Linux repo is recommended for best coverage.

test definitions

Does your test system:

  • have a test definition repository? Yes. C, Shell, Python. Simply put, any scripting language currently allowed in the Linux Kernel sources.

Does your test definition include:

  • source code (or source code location)? Linux Kernel Repo

Kselftest git:

  • dependency information?
  • execution instructions?
  • command line variants?
  • environment variants?
  • setup instructions?
  • cleanup instructions?

All of the above: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/dev-tools/kselftest.rst

Does your test system:

  • provide a set of existing tests? 'Yes
    • how many? This ia hard to answer for Kselftest. It contains several tests for various kernel features. There are about 45 major area tests with severla sub-tests and individual tests.

build management

Does your test system:

  • build the software under test (e.g. the kernel)? No.
  • build the test software? Yes.
  • build other software (such as the distro, libraries, firmware)? No.
  • support cross-compilation? Yes.
  • require a toolchain or build system for the SUT? Linux Kernel build tools are used. Barring a few libraries, if kernel can build, Kselftest can build.
  • require a toolchain or build system for the test software? Linux Kernel build tools are used. Barring a few libraries, if kernel can build, Kselftest can build.
  • come with pre-built toolchains? No.
  • store the build artifacts for generated software? Yes
    • in what format is the build metadata stored? n/a
    • are the build artifacts stored as raw files or in a database? n/a

Test scheduling/management

Does your test system:

  • check that dependencies are met before a test is run?
  • schedule the test for the DUT?
    • select an appropriate individual DUT based on SUT or test attributes?
    • reserve the DUT?
    • release the DUT?
  • install the software under test to the DUT?
  • install required packages before a test is run?
  • require particular bootloader on the DUT? (e.g. grub, uboot, etc.)
  • deploy the test program to the DUT?
  • prepare the test environment on the DUT?
  • start a monitor (another process to collect data) on the DUT?
  • start a monitor on external equipment?
  • initiate the test on the DUT?
  • clean up the test environment on the DUT?

This section isn't applicable to Kselftest.

DUT control

Does your test system:

  • store board configuration data?
    • in what format?
  • store external equipment configuration data?
    • in what format?
  • power cycle the DUT?
  • monitor the power usage during a run?
  • gather a kernel trace during a run?
  • claim other hardware resources or machines (other than the DUT) for use during a test?
  • reserve a board for interactive use (ie remove it from automated testing)?
  • provide a web-based control interface for the lab?
  • provide a CLI control interface for the lab.

This section isn't applicable to Kselftest.

Run artifact handling

Does your test system:

  • store run artifacts
    • in what format?
  • put the run meta-data in a database?
    • if so, which database?
  • parse the test logs for results?
  • convert data from test logs into a unified format?
    • if so, what is the format?
  • evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)?
  • do you have a common set of result names: (e.g. pass, fail, skip, etc.)
    • if so, what are they?
  • How is run data collected from the DUT?
    • e.g. by pushing from the DUT, or pulling from a server?
  • How is run data collected from external equipment?
  • Is external equipment data parsed?

This section isn't applicable to Kselftest.

User interface

Does your test system:

  • have a visualization system?
  • show build artifacts to users?
  • show run artifacts to users?
  • do you have a common set of result colors?
    • if so, what are they?
  • generate reports for test runs?
  • notify users of test results by e-mail?
  • can you query (aggregate and filter) the build meta-data?
  • can you query (aggregate and filter) the run meta-data?
  • what language or data format is used for online results presentation? (e.g. HTML, Javascript, xml, etc.)
  • what language or data format is used for reports? (e.g. PDF, excel, etc.)
  • does your test system have a CLI control tool?
    • what is it called?

This section isn't applicable to Kselftest.

Languages:

  • what is the base language of your test framework core? C
  • What languages or data formats is the user required to learn? None

Can a user do the following with your test framework:

  • manually request that a test be executed (independent of a CI trigger)?
  • see the results of recent tests?
  • set the pass criteria for a test?
    • set the threshold value for a benchmark test?
    • set the list of testcase results to ignore?
  • provide a rating for a test? (e.g. give it 4 stars out of 5)
  • customize a test?
    • alter the command line for the test program?
    • alter the environment of the test program?
    • specify to skip a testcase?
    • set a new expected value for a test?
    • edit the test program source?
  • customize the notification criteria?
    • customize the notification mechanism (eg. e-mail, text)
  • generate a custom report for a set of runs?
  • save the report parameters to generate the same report in the future?

This section isn't applicable to Kselftest.

Requirements

Does your test framework:

  • require minimum software on the DUT? no
    • If so, what?
  • require minimum hardware on the DUT? no
    • If so, what?
  • require agent software on the DUT? no
    • If so, what agent?
  • is there optional agent software or libraries for the DUT?
  • require external hardware in your labs?

This section is hard for Kselftest. Default run doesn't require any additional hardware.

APIS

Does your test framework:

  • use existing APIs or data formats to interact within itself, or with 3rd-party modules? No
  • have a published API for any of its sub-module interactions?
    • Please provide a link or links to the APIs? Refer to Kselftest documentation in the kernel.

Sorry - this is kind of open-ended...

  • What is the nature of the APIs you currently use?

This section isn't applicable to Kselftest.

Relationship to other software:

  • what major components does your test framework use?
  • does your test framework interoperate with other test frameworks or software?
    • which ones?

This section isn't applicable to Kselftest.

Overview

Please list the major components of your test system.

Please list your major components here:

  • Kselftest framework for build/install/Tap13 compliant reporting
  • Individual tests
  • Packaging and install tools

Additional Data