Test Standards
Start collecting info for testing standards here:
- https://tools.ietf.org/html/rfc2119 - ietf MUST, SHALL, MAY, etc. wording standards
Things we'd like to standardize in open source testing:
Contents
Terminology and Framework
- test nomenclature (test glossary)
- CI loop diagram
Test definition
- fields
- file format (json, xml, etc.)
- meta-data
- visualization control
- ex: chart_config.json
- instructions
Pass Criteria
- what tests can be skipped etc.
Test dependencies
- how to specify test dependencies
- ex: assert_define ENV_VAR_NAME
- ex: kernel_config
Test Execution API (E)
- test API
- host/target abstraction
- kernel installation
- file operations
- console access
- command execution
- test retrieval, build, deployment
- test execution:
- ex: 'make test'
- test execution:
- test phases
Run Artifacts
- logs
- data files (audio, video)
- monitor results (power log, trace log)
- snapshots
Results Format
- test log output format
- counts
- subtest results
- Candidate formats:
Build Artifacts
- test package format
- meta-data for each test
- test results
- baseline expected results for particular tests on particular platforms
- what tests can be skipped etc.