Difference between revisions of "Test Standards"

From eLinux.org
Jump to: navigation, search
(Created page with "Start collecting info for testing standards here: * https://tools.ietf.org/html/rfc2119 - ietf MUST, SHALL, MAY, etc. wording standards Things we'd like to standardize in ope...")
 
(Terminology and Framework)
Line 5: Line 5:
 
= Terminology and Framework =
 
= Terminology and Framework =
 
* test nomenclature (test glossary)
 
* test nomenclature (test glossary)
 +
* CI loop diagram
  
 
= Test definition =
 
= Test definition =

Revision as of 15:29, 30 October 2018

Start collecting info for testing standards here:

Things we'd like to standardize in open source testing:

Terminology and Framework

  • test nomenclature (test glossary)
  • CI loop diagram

Test definition

  • fields
  • file format (json, xml, etc.)
  • meta-data
  • visualization control
    • ex: chart_config.json
  • instructions

Pass Criteria

    • what tests can be skipped etc.

Test dependencies

  • how to specify test dependencies
    • ex: assert_define ENV_VAR_NAME
    • ex: kernel_config

Test Execution API (E)

  • test API
  • host/target abstraction
    • kernel installation
    • file operations
    • console access
    • command execution
  • test retrieval, build, deployment
    • test execution:
      • ex: 'make test'
  • test phases

Run Artifacts

  • logs
  • data files (audio, video)
  • monitor results (power log, trace log)
  • snapshots

Results Format

Build Artifacts

  • test package format
    • meta-data for each test
    • test results
    • baseline expected results for particular tests on particular platforms
    • what tests can be skipped etc.