Test Result Codes

From eLinux.org
Revision as of 15:07, 13 November 2018 by Tim Bird (talk | contribs) (Created page with "A result code is a set of values from which a valid result is enumerated. The result or "status" of the test indicated the test outcome. This status is usually restricted to...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

A result code is a set of values from which a valid result is enumerated. The result or "status" of the test indicated the test outcome.

This status is usually restricted to on of a set of possible values. Due to different kinds of problems that can arise during testing, the result codes consist of more than just PASS and FAIL, which indicate success or failure for whatever the test is testing for.

This page is used to document the most common result codes, to try to harmonize the industry usage of these.

survey

LTP

  • TPASS
  • TFAIL
  • TSKIP
  • TBROK

Fuego

See http://fuegotest.org/wiki/run.json, the 'status' field:

  • PASS - a testcase, test set or test suite completed successfully
  • FAIL - a testcase, test set or test suite was unsuccessful
  • ERROR - a test did not execute properly (e.g. the test program did not run correctly)
  • SKIP - a test was not executed, usually due to invalid configuration (missing some pre-requisite)

Jenkins

  • Stable - color: blue; everything passed
    • many people install the 'greenballs' plugin for this to be green
  • Unstable - color: grey; Test were successfully executed but found failures.
  • Failed - color: red; Problem with compilation / configuration / runtime error.
  • Aborted - color: grey; Build time-out, someone intentionally stopped the run in the middle.
  • Not executed yet - color: grey; Test has not been executed yet

pytest

buildbot

  • SUCCESS: Value: 0; color: green; a successful run.
  • WARNINGS: Value: 1; color: orange; a successful run, with some warnings.
  • FAILURE: Value: 2; color: red; a failed run, due to problems in the build itself, as opposed to a Buildbot misconfiguration or bug.
  • SKIPPED: Value: 3; color: white; a run that was skipped – usually a step skipped by doStepIf (see Common Parameters)
  • EXCEPTION: Value: 4; color: purple; a run that failed due to a problem in Buildbot itself.
  • RETRY: Value: 5; color: purple; a run that should be retried, usually due to a worker disconnection.
  • CANCELLED: Value: 6; color: pink; a run that was cancelled by the user.

See http://docs.buildbot.net/latest/developer/results.html