Difference between revisions of "Test Execution Notes"

From eLinux.org
Jump to: navigation, search
(Undo revision 500916 by Tim Bird (talk))
(Tag: Undo)
Line 37: Line 37:
 
= Notes by test system =
 
= Notes by test system =
 
== Fuego ==
 
== Fuego ==
 +
 +
=== test locations ==
 +
* test source location: each test is in it's own directory under fuego-core/tests/<test-name>
 +
* on the target, test materials are placed in $BOARD_TESTIDR/fuego.$TESTDIR
 +
** BOARD_TESTDIR is usually something like: /home/fuego
 +
** TESTDIR is the name of the test: eg. Functional.hello_world
 +
** so the final path is something like: /home/fuego/fuego.Functional.hello_world
  
 
=== test script names and locations ===
 
=== test script names and locations ===
Every test is in a directory
+
* each test has a script called 'fuego_test.sh'
 +
** this script is run on the host
 +
* often there is a test on the target, which is executed by the test on the host
 +
** there is no convention for the name of the target-side script
 +
** it is referenced inside the host-side 'fuego_test.sh'
  
 
=== APIs ===
 
=== APIs ===

Revision as of 16:26, 3 October 2019

This page has notes about aspects of the execution control for different systems that perform test execution.

Here are some of the elements of a the test execution API:

  • test API - what functions can the test call?
    • file operations
    • service operations (start, stop)
    • log access
    • results output functions
  • host/target abstraction - is the test executed from host or directly on target
  • DUT preparation/control: - this might be in a separate standard
    • kernel installation / provisioning
    • console access
    • log access
    • test activation - how is the test activated or started?
      • e.g. installed and activated as part of boot
      • e.g. a command is executed as a program?
      • e.g. is the test built into the kernel and activated via some trigger mechanism (e.g. via /proc or /sys or an ioctl)?
  • test retrieval, build, deployment
  • test execution:
    • test name, or standard for test invocation name
    • ex: 'make test' - a standard make target name?
    • runtest.sh? - a standard shell script name?
    • <testname>.sh - a standard shell script name, based on test meta-data?
  • test phases?
  • test selection / skiplists - how are test selected for inclusion or exclusion in a test run
  • test variables - how does the test control execution
    • make variables
    • environment variables
    • command line arguments

Does this include test scheduling? (no), but it does include test selection.

Does this include test building (no) - see Test Building notes


Notes by test system

Fuego

= test locations

  • test source location: each test is in it's own directory under fuego-core/tests/<test-name>
  • on the target, test materials are placed in $BOARD_TESTIDR/fuego.$TESTDIR
    • BOARD_TESTDIR is usually something like: /home/fuego
    • TESTDIR is the name of the test: eg. Functional.hello_world
    • so the final path is something like: /home/fuego/fuego.Functional.hello_world

test script names and locations

  • each test has a script called 'fuego_test.sh'
    • this script is run on the host
  • often there is a test on the target, which is executed by the test on the host
    • there is no convention for the name of the target-side script
    • it is referenced inside the host-side 'fuego_test.sh'

APIs

Provided by board library: fuego_board_function_lib.sh

Is a library of shell functions.

board-side functions:

  • set_init_manager()
  • detect_logger_service()
  • exec_service_on_target()
  • detect_active_eth_device()
  • get_service_status()

host-side functions: See http://fuegotest.org/wiki/Test_Script_APIs

test variables

Provided in environment variables, prefixed by test name:

  • ex: FUNCTIONAL_HELLO_ARGS
  • ex:

Required minimum Linux command set

Tools builtin or on the PATH

  • cat, df, find, free, grep, head, logger, logread, mkdir, mount, ps, rm ,rmdir, sync, tail, tee, touch, true, umount, uname, uptime, xargs, [

Required command arguments:

  • mkdir -p, rm -rf, grep -f,

Tools at specific paths:

  • /sbin/reboot, /bin/true, /sbin/route

Data files:

  • /var/log/messages, /var/log/syslog
  • /proc/interrupts, /proc/drop_caches, /proc/oom_score_adj

(optional)

  • proc/config.gz

Require arguments:

LKFT

LAVA

CKI

kselftest

Location of tests

  • source of tests is in: <kernel_repository>/tools/testing/selftests/<area>
  • test binaries are placed in: $INSTALL_PATH/kselftest
    • they are organized by test area, which corresponds to the directory in which they reside in the kernel source repsoitory

test executable names

There is no standard, but the names appear to be obtainable with 'make -s -C tools/testing/selftests/<area> emit_tests'

A script is generated which runs all the test, sourcing 'kselftest/runner.sh' and calling all the test names with the function 'run_many', generating TAP output based on the return code for each test.

LTP

  • test selection via 'scenario' or 'runtest' files (located in <topdir>/runtest)
  • test skiplist support via runltp -S option
    • ex: runltp ... -S skiplist.txt ...
  • test command line argument control is via the runtest file

Standards

Makefile targets

  • Makefile targets for software testing:
    • make test
    • make check

sources

If a program uses Makefile to build and install the software under test, then it should provide the following Makefile targets, to support testing of the software.

"make test"

or

"make check"

See https://www.gnu.org/software/make/manual/make.html#Goals (section "9.2 Arguments to Specify the Goals")

See also: https://www.gnu.org/software/make/manual/make.html#Standard-Targets

board-local test APIs