Test Definition Project

The Test Definition Project is consists of work to categorize and harmonize the attribute of open source tests, so that tests and test artifacts can more easily be shared between different test projects.

Tim Bird is working on a "Test Definition Standards" document, to be presented and discussed at various meetings in the Fall of 2019.

This page has information that is being retained on this wiki as resources for this work.

Presentations

• See Harmonizing Open Source test definitions by Tim Bird, at Linaro Connect 2019 in Bangkok, Thailand, April 2019

Resources

See Test definition survey

Test definition elements

Categories

• Information about a test
• Pre-requisites
• Dependencies
• Instructions
• Output format
• Test variables
• Results analysis
• Visualization control

For individual elements of the above:

• field or item name
• language (C, Python, sh, etc.)
• file format (json, xml, etc.)
• allowed values
  • including APIs, for code items
• groupings

Information about a test

Also known as test meta-data. Does not affect the test execution, but provides information about it.

Surveyed fields

• Name
• Description
• License
• Version
  • test program version
  • test wrapper version
• Author
• Maintainer
• Test format version
• Package manifest

Pre-requisites

• required machine attributes
  • memory
  • cpus
  • storage
  • architecture
  • (specific hardware - for scheduling)
    • network, bus, device
• required kernel attributes
  • config value (= kernel feature)
  • kernel module
• required distribution attributes
  • distribution
  • distro version
  • logging system
  • init system
  • installed package
  • installed program
  • installed file
• test environment
  • root permissions
  • file system type
• tags

Dependencies

Things that can be installed or modified

• packages

Test control

• test variables
  • params
• skiplists
• timeouts

Instructions

• source location
• build instructions
• run instructions
• setup instructions
• teardown/cleanup instructions

Output format

• parser
• results output API

Results analysis

• pass criteria
  • ignore lists
  • pass/fail counts
  • expected values
  • threshholds

Visualization control

• key fields
• chart types
• thresholds
• groupings

Things that are NOT test part of a test definition

• build artifact
• results artifact
• board scheduling API
• lab management API (unless test performs board reboot, provision, hardware control, etc.)
• trigger API
• scheduling API

element expression

• file formats
• file names
• data format

APIs

• API between test and test framework
• API between test and target system
• API between test and external equipment

What languages are supported?

candidates

• beakerlib
• fuego_board_function_lib.sh
• shelltestlib