Difference between revisions of "Test Definition Project"
(→candidates) |
(→Test elements comparison tables) |
||
(12 intermediate revisions by the same user not shown) | |||
Line 15: | Line 15: | ||
See [[Test definition survey]] | See [[Test definition survey]] | ||
− | + | See [[Test Execution Notes]] | |
= Test definition elements = | = Test definition elements = | ||
Line 31: | Line 31: | ||
* field or item name | * field or item name | ||
* language (C, Python, sh, etc.) | * language (C, Python, sh, etc.) | ||
− | * file format (json, xml, etc.) | + | * file format (json, xml, etc.) - see [[Test definition files Note]] |
* allowed values | * allowed values | ||
** including APIs, for code items | ** including APIs, for code items | ||
Line 51: | Line 51: | ||
* Test format version | * Test format version | ||
* Package manifest | * Package manifest | ||
+ | |||
+ | See [[Test meta-data Notes]] | ||
== Pre-requisites == | == Pre-requisites == | ||
Line 78: | Line 80: | ||
== Dependencies == | == Dependencies == | ||
Things that can be installed or modified | Things that can be installed or modified | ||
+ | * kernel configuration (eg. kselftest config fragments) | ||
* packages | * packages | ||
Line 165: | Line 168: | ||
* file names | * file names | ||
* data format | * data format | ||
+ | |||
+ | = Test elements comparison tables = | ||
+ | |||
+ | {| class="wikitable" | ||
+ | !Element | ||
+ | !description | ||
+ | !Fuego | ||
+ | !Linaro | ||
+ | !Yocto | ||
+ | !0day | ||
+ | !CKI | ||
+ | |- | ||
+ | |name | ||
+ | |name of the test | ||
+ | |test.yaml:name (also directory under fuego-core/tests) Includes test type (Functional or Benchmark) (example: Functional.LTP) | ||
+ | |<test>.yaml:metadata:name (also directory under test-definitions/automated/linux, and <testname>.sh and <testname>.yaml in that dir) | ||
+ | |???, (directory in /usr/lib/<testname>/ptest) | ||
+ | |executable name under tests/, <testname>.yaml under tests/ and under jobs/, shell script <testname> under pack/ | ||
+ | |directory path under repository top (example: "kernel/filesystems/loopdev/sanity" (runs iozone)) | ||
+ | | | ||
+ | |- | ||
+ | |description | ||
+ | |description of the test | ||
+ | |test.yaml:description | ||
+ | |<test>.yaml:metata:description | ||
+ | |- | ||
+ | |license | ||
+ | |license of the test software | ||
+ | |test.yaml:license | ||
+ | |- | ||
+ | |author | ||
+ | |author of the test | ||
+ | |test.yaml:author | ||
+ | |- | ||
+ | |version | ||
+ | |version of the test software | ||
+ | |test.yaml:version | ||
+ | |- | ||
+ | |maintainer | ||
+ | |person who maintains this test definition/wrapper | ||
+ | |test.yaml:maintainer | ||
+ | |<test>.yaml:metadata:maintainer | ||
+ | |- | ||
+ | |release??? | ||
+ | |release number for this test definition | ||
+ | |test.yaml:fuego_version | ||
+ | |test.yaml:metadata:version | ||
+ | |- | ||
+ | |} | ||
= APIs = | = APIs = | ||
Line 172: | Line 224: | ||
What languages are supported? | What languages are supported? | ||
+ | == test framework APIs == | ||
+ | These are the APIS between the test and the test framework | ||
− | == candidates == | + | === API library candidates === |
+ | * functions.sh (Fuego) | ||
+ | * beakerlib (Beaker) | ||
+ | * rhts_lib(sp?) (CKI) | ||
+ | * (other system's function libraries??) | ||
+ | |||
+ | === test framework functions === | ||
+ | (need to peruse the list above for individual functions) | ||
+ | |||
+ | === candidates === | ||
+ | |||
+ | == board-local test APIs == | ||
+ | These are the APIs between the test and the DUT system itself, that are | ||
+ | not part of the DUT system (that is, which call a test framework library). | ||
+ | |||
+ | === API library candidates === | ||
There are candidates for libraries that libraries that have test APIs: | There are candidates for libraries that libraries that have test APIs: | ||
* beakerlib (Beaker) | * beakerlib (Beaker) | ||
* fuego_board_function_lib.sh (Fuego) | * fuego_board_function_lib.sh (Fuego) | ||
* sh_test_lib (LAVA? or LKFT?) | * sh_test_lib (LAVA? or LKFT?) | ||
+ | |||
+ | === local functions === | ||
+ | [put list of board-local functions here] | ||
+ | |||
+ | === candidates === | ||
+ | |||
+ | == external equipment APIS == | ||
+ | These are APIs between the test and external equipment, such as external BUS controllers, power measurement | ||
+ | devices, external tracers and logic analyzers, video or audio grabbers, or other external hardware. | ||
+ | |||
+ | Question: Does this include the API to server hardware, like netperf server? |
Latest revision as of 21:14, 10 October 2019
The Test Definition Project is consists of work to categorize and harmonize the attribute of open source tests, so that tests and test artifacts can more easily be shared between different test projects.
Tim Bird is working on a "Test Definition Standards" document, to be presented and discussed at various meetings in the Fall of 2019.
This page has information that is being retained on this wiki as resources for this work.
Contents
Presentations
- See Harmonizing Open Source test definitions by Tim Bird, at Linaro Connect 2019 in Bangkok, Thailand, April 2019
Resources
Test definition elements
Categories
- Information about a test
- Pre-requisites
- Dependencies
- Instructions
- Output format
- Test variables
- Results analysis
- Visualization control
For individual elements of the above:
- field or item name
- language (C, Python, sh, etc.)
- file format (json, xml, etc.) - see Test definition files Note
- allowed values
- including APIs, for code items
- groupings
Information about a test (meta-data)
Also known as test meta-data. Does not affect the test execution, but provides information about it.
Surveyed fields:
- Name
- Description
- License
- Version
- test program version
- test wrapper version
- Author
- Maintainer
- Test format version
- Package manifest
Pre-requisites
- required machine attributes
- memory
- cpus
- storage
- architecture
- (specific hardware - for scheduling)
- network, bus, device
- required kernel attributes
- config value (= kernel feature)
- kernel module
- required distribution attributes
- distribution
- distro version
- logging system
- init system
- installed package
- installed program
- installed file
- test environment
- root permissions
- file system type
- tags
Dependencies
Things that can be installed or modified
- kernel configuration (eg. kselftest config fragments)
- packages
Test control
test variables
- params
skiplists
- how does each test handle skiplists
- Tests that are know to handle skiplists:
- LTP
- xfstests
- xfstests also has a mechanism for selecting individual tests (which Fuego does for LTP with LTP_one_test)
- (does kselftest? - they have a skip mechanism based on CONFIG fragments, is it generalized)
expected duration (timeout)
Expected duration is something a test needs to communicate to the test harness (test manager and scheduler), so that the harness can automatically detect if the test or machine has hung.
The difficulty here is that actual duration may be affected by a lot of factors. So coming up with a value that will work in all circumstances is difficult.
Some systems, like Jenkins, measure the duration of previous runs, but (to my knowledge), do not use that information to stop an instance of a test.
Instructions
- source location
- build instructions
- run instructions
- setup instructions
- teardown/cleanup instructions
test library
The test library is a set of functions or capabilities that are available to a test to perform utility operations on a target device.
A category of library functions is a set of programs that a test may utilize on a target, to perform operations.
Fuego has a list of core programs that it tries to constrain itself to. All other programs must be specified in as dependencies in the test_pre_check function, or in NEED_ variables.
Here is Fuego's minimal Linux command list:
- cat, df, find, free, grep, head, (logger or logread), mkdir, mount, ps, rm, rmdir, sync, tail, tee, touch, true, umount, uname, uptime, xargs, [
- /sbin/reboot, /sbin/true, /sbin/logread
Depending on the distribution, certain files must be present:
- /var/log/syslog (optional)
Certain psuedo-files are required by the Fuego core:
- /proc/interrupts, /proc/sys/vm/drop_caches, /proc/$$/oom_score_adj
Certain commands have required minimum arguments that must be supported:
- mkdir -p, rm -rf, grep -Fv
One program to consider for characterizing the hardware on a platform is 'lshw'. This may be something that a test manager runs, to characterize a DUT (the information might be used for scheduling, or test selection, or to modify the parameters to a test). Or, it could be something a test itself runs, to alter its own execution.
Output format
- parser
- results output API
Results analysis
- pass criteria
- ignore lists
- pass/fail counts
- expected values
- threshholds
Visualization control
- key fields
- chart types
- thresholds
- groupings
- summaries
Things that are NOT test part of a test definition
- build artifact
- results artifact
- board scheduling API
- lab management API (unless test performs board reboot, provision, hardware control, etc.)
- trigger API
- scheduling API
element expression
- file formats
- file names
- data format
Test elements comparison tables
Element | description | Fuego | Linaro | Yocto | 0day | CKI | |
---|---|---|---|---|---|---|---|
name | name of the test | test.yaml:name (also directory under fuego-core/tests) Includes test type (Functional or Benchmark) (example: Functional.LTP) | <test>.yaml:metadata:name (also directory under test-definitions/automated/linux, and <testname>.sh and <testname>.yaml in that dir) | ???, (directory in /usr/lib/<testname>/ptest) | executable name under tests/, <testname>.yaml under tests/ and under jobs/, shell script <testname> under pack/ | directory path under repository top (example: "kernel/filesystems/loopdev/sanity" (runs iozone)) | |
description | description of the test | test.yaml:description | <test>.yaml:metata:description | ||||
license | license of the test software | test.yaml:license | |||||
author | author of the test | test.yaml:author | |||||
version | version of the test software | test.yaml:version | |||||
maintainer | person who maintains this test definition/wrapper | test.yaml:maintainer | <test>.yaml:metadata:maintainer | ||||
release??? | release number for this test definition | test.yaml:fuego_version | test.yaml:metadata:version |
APIs
- API between test and test framework
- API between test and target system
- API between test and external equipment
What languages are supported?
test framework APIs
These are the APIS between the test and the test framework
API library candidates
- functions.sh (Fuego)
- beakerlib (Beaker)
- rhts_lib(sp?) (CKI)
- (other system's function libraries??)
test framework functions
(need to peruse the list above for individual functions)
candidates
board-local test APIs
These are the APIs between the test and the DUT system itself, that are not part of the DUT system (that is, which call a test framework library).
API library candidates
There are candidates for libraries that libraries that have test APIs:
- beakerlib (Beaker)
- fuego_board_function_lib.sh (Fuego)
- sh_test_lib (LAVA? or LKFT?)
local functions
[put list of board-local functions here]
candidates
external equipment APIS
These are APIs between the test and external equipment, such as external BUS controllers, power measurement devices, external tracers and logic analyzers, video or audio grabbers, or other external hardware.
Question: Does this include the API to server hardware, like netperf server?