Test Standards

This page will be used to collect information about test standards.

meta-documents

• https://tools.ietf.org/html/rfc2119 - IETF MUST, SHALL, MAY, etc. wording standards

A survey of existing test systems was conducted in the Fall of 2018. The survey and results are here: Test Stack Survey

---

Here are some things we'd like to standardize in open source automated testing:

Terminology and Framework

• test nomenclature (test glossary)
• CI loop diagram

Test Definition

• fields
• file format (json, xml, etc.)
• meta-data
• visualization control
  • ex: chart_config.json
• instructions

Pass Criteria

• what tests can be skipped (this is more part of test execution and control)
• what test results can be ignored (xfail)
• min required pass counts, max allowed failures
• thresholds for measurement results
  • requires testcase id, number and operator

Test dependencies

• how to specify test dependencies
  • ex: assert_define ENV_VAR_NAME
  • ex: kernel_config

Test Execution API (E)

• test API
• host/target abstraction
  • kernel installation
  • file operations
  • console access
  • command execution
• test retrieval, build, deployment
  • test execution:
    • ex: 'make test'
• test phases

Run Artifacts

• logs
• data files (audio, video)
• monitor results (power log, trace log)
• snapshots

Results Format

• test log output format
  • counts
  • subtest results
  • Candidate formats:
    • https://testanything.org/

Build Artifacts

• test package format
  • meta-data for each test
  • test results
  • baseline expected results for particular tests on particular platforms

• • what tests can be skipped etc.