Difference between revisions of "Test Standards"
(→Terminology and Framework) |
|||
Line 1: | Line 1: | ||
− | + | This page will be used to collect information about test standards. | |
− | * https://tools.ietf.org/html/rfc2119 - | + | |
+ | = meta-documents = | ||
+ | * https://tools.ietf.org/html/rfc2119 - IETF MUST, SHALL, MAY, etc. wording standards | ||
Things we'd like to standardize in open source testing: | Things we'd like to standardize in open source testing: |
Revision as of 15:30, 30 October 2018
This page will be used to collect information about test standards.
Contents
meta-documents
- https://tools.ietf.org/html/rfc2119 - IETF MUST, SHALL, MAY, etc. wording standards
Things we'd like to standardize in open source testing:
Terminology and Framework
- test nomenclature (test glossary)
- CI loop diagram
Test definition
- fields
- file format (json, xml, etc.)
- meta-data
- visualization control
- ex: chart_config.json
- instructions
Pass Criteria
- what tests can be skipped etc.
Test dependencies
- how to specify test dependencies
- ex: assert_define ENV_VAR_NAME
- ex: kernel_config
Test Execution API (E)
- test API
- host/target abstraction
- kernel installation
- file operations
- console access
- command execution
- test retrieval, build, deployment
- test execution:
- ex: 'make test'
- test execution:
- test phases
Run Artifacts
- logs
- data files (audio, video)
- monitor results (power log, trace log)
- snapshots
Results Format
- test log output format
- counts
- subtest results
- Candidate formats:
Build Artifacts
- test package format
- meta-data for each test
- test results
- baseline expected results for particular tests on particular platforms
- what tests can be skipped etc.