Difference between revisions of "Test Execution Notes"
(Tag: Undo) |
|||
Line 37: | Line 37: | ||
= Notes by test system = | = Notes by test system = | ||
== Fuego == | == Fuego == | ||
+ | |||
+ | === test locations == | ||
+ | * test source location: each test is in it's own directory under fuego-core/tests/<test-name> | ||
+ | * on the target, test materials are placed in $BOARD_TESTIDR/fuego.$TESTDIR | ||
+ | ** BOARD_TESTDIR is usually something like: /home/fuego | ||
+ | ** TESTDIR is the name of the test: eg. Functional.hello_world | ||
+ | ** so the final path is something like: /home/fuego/fuego.Functional.hello_world | ||
=== test script names and locations === | === test script names and locations === | ||
− | + | * each test has a script called 'fuego_test.sh' | |
+ | ** this script is run on the host | ||
+ | * often there is a test on the target, which is executed by the test on the host | ||
+ | ** there is no convention for the name of the target-side script | ||
+ | ** it is referenced inside the host-side 'fuego_test.sh' | ||
=== APIs === | === APIs === |
Revision as of 16:26, 3 October 2019
This page has notes about aspects of the execution control for different systems that perform test execution.
Here are some of the elements of a the test execution API:
- test API - what functions can the test call?
- file operations
- service operations (start, stop)
- log access
- results output functions
- host/target abstraction - is the test executed from host or directly on target
- DUT preparation/control: - this might be in a separate standard
- kernel installation / provisioning
- console access
- log access
- test activation - how is the test activated or started?
- e.g. installed and activated as part of boot
- e.g. a command is executed as a program?
- e.g. is the test built into the kernel and activated via some trigger mechanism (e.g. via /proc or /sys or an ioctl)?
- test retrieval, build, deployment
- test execution:
- test name, or standard for test invocation name
- ex: 'make test' - a standard make target name?
- runtest.sh? - a standard shell script name?
- <testname>.sh - a standard shell script name, based on test meta-data?
- test phases?
- test selection / skiplists - how are test selected for inclusion or exclusion in a test run
- test variables - how does the test control execution
- make variables
- environment variables
- command line arguments
Does this include test scheduling? (no), but it does include test selection.
Does this include test building (no) - see Test Building notes
Notes by test system
Fuego
= test locations
- test source location: each test is in it's own directory under fuego-core/tests/<test-name>
- on the target, test materials are placed in $BOARD_TESTIDR/fuego.$TESTDIR
- BOARD_TESTDIR is usually something like: /home/fuego
- TESTDIR is the name of the test: eg. Functional.hello_world
- so the final path is something like: /home/fuego/fuego.Functional.hello_world
test script names and locations
- each test has a script called 'fuego_test.sh'
- this script is run on the host
- often there is a test on the target, which is executed by the test on the host
- there is no convention for the name of the target-side script
- it is referenced inside the host-side 'fuego_test.sh'
APIs
Provided by board library: fuego_board_function_lib.sh
Is a library of shell functions.
board-side functions:
- set_init_manager()
- detect_logger_service()
- exec_service_on_target()
- detect_active_eth_device()
- get_service_status()
host-side functions: See http://fuegotest.org/wiki/Test_Script_APIs
test variables
Provided in environment variables, prefixed by test name:
- ex: FUNCTIONAL_HELLO_ARGS
- ex:
Required minimum Linux command set
Tools builtin or on the PATH
- cat, df, find, free, grep, head, logger, logread, mkdir, mount, ps, rm ,rmdir, sync, tail, tee, touch, true, umount, uname, uptime, xargs, [
Required command arguments:
- mkdir -p, rm -rf, grep -f,
Tools at specific paths:
- /sbin/reboot, /bin/true, /sbin/route
Data files:
- /var/log/messages, /var/log/syslog
- /proc/interrupts, /proc/drop_caches, /proc/oom_score_adj
(optional)
- proc/config.gz
Require arguments:
LKFT
LAVA
CKI
kselftest
- test selection via TARGETS make variable
- test skiplist support via SKIP_TARGETS make variable
- patch submitted 9/24/2019 - need to check mainline status - see https://lore.kernel.org/linux-kselftest/20190925132421.23572-1-cristian.marussi@arm.com/
Location of tests
- source of tests is in: <kernel_repository>/tools/testing/selftests/<area>
- test binaries are placed in: $INSTALL_PATH/kselftest
- they are organized by test area, which corresponds to the directory in which they reside in the kernel source repsoitory
test executable names
There is no standard, but the names appear to be obtainable with 'make -s -C tools/testing/selftests/<area> emit_tests'
A script is generated which runs all the test, sourcing 'kselftest/runner.sh' and calling all the test names with the function 'run_many', generating TAP output based on the return code for each test.
LTP
- test selection via 'scenario' or 'runtest' files (located in <topdir>/runtest)
- test skiplist support via runltp -S option
- ex: runltp ... -S skiplist.txt ...
- test command line argument control is via the runtest file
Standards
Makefile targets
- Makefile targets for software testing:
- make test
- make check
sources
If a program uses Makefile to build and install the software under test, then it should provide the following Makefile targets, to support testing of the software.
"make test"
or
"make check"
See https://www.gnu.org/software/make/manual/make.html#Goals (section "9.2 Arguments to Specify the Goals")
See also: https://www.gnu.org/software/make/manual/make.html#Standard-Targets