Difference between revisions of "Test Standards"
(→Test definition) |
|||
Line 9: | Line 9: | ||
* CI loop diagram | * CI loop diagram | ||
− | = Test | + | = Test Definition = |
* fields | * fields | ||
* file format (json, xml, etc.) | * file format (json, xml, etc.) |
Revision as of 15:54, 30 October 2018
This page will be used to collect information about test standards.
Contents
meta-documents
- https://tools.ietf.org/html/rfc2119 - IETF MUST, SHALL, MAY, etc. wording standards
Things we'd like to standardize in open source testing:
Terminology and Framework
- test nomenclature (test glossary)
- CI loop diagram
Test Definition
- fields
- file format (json, xml, etc.)
- meta-data
- visualization control
- ex: chart_config.json
- instructions
Pass Criteria
- what tests can be skipped etc.
Test dependencies
- how to specify test dependencies
- ex: assert_define ENV_VAR_NAME
- ex: kernel_config
Test Execution API (E)
- test API
- host/target abstraction
- kernel installation
- file operations
- console access
- command execution
- test retrieval, build, deployment
- test execution:
- ex: 'make test'
- test execution:
- test phases
Run Artifacts
- logs
- data files (audio, video)
- monitor results (power log, trace log)
- snapshots
Results Format
- test log output format
- counts
- subtest results
- Candidate formats:
Build Artifacts
- test package format
- meta-data for each test
- test results
- baseline expected results for particular tests on particular platforms
- what tests can be skipped etc.