Difference between revisions of "Test Glossary"

From eLinux.org
Jump to: navigation, search
Line 3: Line 3:
  
 
== Terms ==
 
== Terms ==
* Actual Value - the value that was seen for an operation performed by a test
+
* '''Actual Value''' - the value that was seen for an operation performed by a test
* Bisection - automatic testing of SUT variations to find the source of a problem
+
* '''Bisection''' - automatic testing of SUT variations to find the source of a problem
* Boot - to start the target from an off state.  The end of 'boot' is the point of time when a test can be started, which will be different depending on whether a test is testing the bootloader, the kernel or some user-space software.
+
* '''Boot''' - to start the target from an off state.  The end of 'boot' is the point of time when a test can be started, which will be different depending on whether a test is testing the bootloader, the kernel or some user-space software.
* Build artifact - item created during build of the software under test
+
* '''Build artifact''' - item created during build of the software under test
* Build manager (build server) - a machine that performs builds of the software under test
+
* '''Build manager''' - a machine or process that performs builds of the software under test.  Also known as a build server, when it is a machine.
* Dependency - indicates a pre-requisite that must be filled in order for a test to run
+
* '''Dependency''' - indicates a pre-requisite that must be filled in order for a test to run
 
** Some examples: must have root access, must have 100 meg of memory, some program or package must be installed, etc.
 
** Some examples: must have root access, must have 100 meg of memory, some program or package must be installed, etc.
* Device Under Test (DUT) - the product, board or device that is being tested. It consists of hardware and software under test. (See also 'board', 'target')
+
* '''Device Under Test''' (DUT) - the product, board or device that is being tested. It consists of hardware and software under test. (See also 'board', 'target')
 
** This glossary uses 'target' as the more generic term for a thing that hosts the system being tested (for example, it could be a Virtual Machine)
 
** This glossary uses 'target' as the more generic term for a thing that hosts the system being tested (for example, it could be a Virtual Machine)
* Deploy - put the test program or system software on the target
+
* '''Deploy''' - put the test program or system software on the target
* Expected value - the value that was expected for an operation performed by a test
+
* '''Expected value''' - the value that was expected for an operation performed by a test
* Lab - a collection of resources for testing one or more targets (also 'board farm')
+
* '''Lab''' - a collection of resources for testing one or more targets (also 'board farm')
* Log - one of the run artifacts - output from the test program or test framework
+
* '''Log''' - one of the run artifacts - output from the test program or test framework
* Log Parsing - extracting information from a log into a machine-processable format (possibly into a common format)
+
* '''Log Parsing''' - extracting information from a log into a machine-processable format (possibly into a common format)
* Monitor - a program or process to watch some attribute (e.g. power) while the test is running
+
* '''Monitor (noun)''' - a program or process to watch some attribute (e.g. power) while the test is running
** This can be on or off the DUT.
+
** A monitor could be running either on or off the target, and it could be collecting data from either on or off the target.
* Notification - communication based on results of test (triggered by results and including results)
+
* '''Notification''' - communication based on results of test (triggered by results and including results)
* Pass criteria - set of constraints indicating pass/fail conditions for a test
+
* '''Pass criteria''' - set of constraints indicating pass/fail conditions for a test
* PDU - Power Distribution Unit - a piece of hardware used to control power to one or more DUTs (used by LAVA)
+
* '''PDU - Power Distribution Unit''' - a piece of hardware used to control power to one or more DUTs (used by LAVA)
* Report generation - collecting run data and putting it into a formatted output
+
* '''Report generation''' - collecting run data and putting it into a formatted output
* Request (noun) - a request to execute a test
+
* '''Request (noun)''' - a request to execute a test
* Result - the status indicated by a test - pass/fail (or something else) for a Run
+
* '''Result''' - the status indicated by a test - pass/fail (or something else) for a Run
* Results query - Selection and filtering of data from runs, to find patterns
+
* '''Results query''' - Selection and filtering of data from runs, to find patterns
* Run (noun) - an execution instance of a test (in Jenkins, a build)
+
* '''Run (noun)''' - an execution instance of a test (in Jenkins, a build)
* Run artifact - item created during a run of the test program
+
* '''Run artifact''' - item created during a run of the test program
* Serial console - the Linux console connected over a serial connection
+
* '''Serial console''' - the Linux console connected over a serial connection
* Software under test (SUT) - the software being tested
+
* '''Software under test''' - the software being tested
* System under test - see Target, DUT - The complete system being tested, including hardware and software
+
* '''System under test''' - The complete system being tested, including hardware and software. See Target, DUT.
* Target - The board, device or virtual machine in which tests are performed.  Usually this is thought of to consist of both the Device under test and the accompanying software under test.
+
* '''Target''' - The board, device or virtual machine in which tests are performed.  Usually this is thought of to consist of both the Device Under Test and the accompanying Software Under Test.
* Target Manager - a process or machine that controls and schedules a target (see also DUT controller, DUT supervisor, DUT scheduler)
+
* '''Target Manager''' - a process or machine that controls and schedules a target (see also DUT controller, DUT supervisor, DUT scheduler)
 
** This includes things like booting the target, controlling networking to the target, and moving data to and from the target.
 
** This includes things like booting the target, controlling networking to the target, and moving data to and from the target.
* Test agent - software running on the DUT that assists in test operations (e.g. test deployment, execution, log gathering, debugging
+
* '''Test agent''' - software running on the DUT that assists in test operations (e.g. test deployment, execution, log gathering, debugging
 
** One example would be 'adb', for Android-based systems)
 
** One example would be 'adb', for Android-based systems)
* Test definition - meta-data and software that comprise a particular test
+
* '''Test definition''' - meta-data and software that comprise a particular test
* Test plan - the definition and meta-data for a group of tests run together (e.g. sequentially on the same target).  This always includes the list of tests, but may also include per-test data such as timeout, test variables or reporting/notification instructions.
+
* '''Test plan''' - the definition and meta-data for a group of tests run together (e.g. sequentially on the same target).  This always includes the list of tests, but may also include per-test data such as timeout, test variables or reporting/notification instructions.
* Test program - a script or binary on the target that performs the test
+
* '''Test program''' - a script or binary on the target that performs the test
* Test scheduler - program for scheduling tests (selecting a target for a test, reserving it, releasing it)
+
* '''Test scheduler''' - program for scheduling tests (selecting a target for a test, reserving it, releasing it)
* Test setup - refers to the sequence of operations required to prepare the target or the lab environment for the test
+
* '''Test setup''' - refers to the sequence of operations required to prepare the target or the lab environment for the test
 
** This might include other major operations, such as deploying the system software to the target.
 
** This might include other major operations, such as deploying the system software to the target.
* Test software - source and/or binary that implements the test
+
* '''Test software''' - source and/or binary that implements the test
* Transport (noun) - the method of communicating and transferring data between the test system and the target
+
* '''Transport (noun)''' - the method of communicating and transferring data between the test system and the target
* Trigger (noun) - an event that causes the CI loop to start
+
* '''Trigger (noun)''' - an event that causes the CI loop to start
* Variant - arguments or data that affect the execution and output of a test (e.g. test program command line; Fuego calls this a 'spec')
+
* '''Variant''' - arguments or data that affect the execution and output of a test (e.g. test program command line; Fuego calls this a 'spec')
* Visualization - allowing the viewing of test artifacts, in aggregated form (e.g. multiple runs plotted in a single diagram)
+
* '''Visualization''' - allowing the viewing of test artifacts, in aggregated form (e.g. multiple runs plotted in a single diagram)
  
 
== Contested or confusing terms ==
 
== Contested or confusing terms ==
* Device Under Test - was confusing to enterprise testers and those using VMS.  Argument was that they have no hardware under test, and that "Device"
+
* '''Device Under Test''' - was confusing to enterprise testers and those using VMS.  Argument was that they have no hardware under test, and that "Device"
 
is confusing, and is treated as just another resource for the test.
 
is confusing, and is treated as just another resource for the test.
* Deploy - put the test program or system software on the target.  Putting the system software on the target is much different from putting the test program on the target, so usage of "deploy" is ambiguous.
+
* '''Deploy''' - put the test program or system software on the target.  Putting the system software on the target is much different from putting the test program on the target, so usage of "deploy" is ambiguous.
  
 
== Candidate terms ==
 
== Candidate terms ==
* Board - the board hardware under test.  See also Device Under Test and Target.
+
* '''Board''' - the board hardware under test.  See also Device Under Test and Target.
* Feature - an attribute of a DUT or SUT or test environment that can be used to match tests.  Requiring a DUT to have a particular feature could be a test dependency. (used by labgrid)
+
* '''Feature''' - an attribute of a DUT or SUT or test environment that can be used to match tests.  Requiring a DUT to have a particular feature could be a test dependency. (used by labgrid)
* Device type - The name of a set of DUTs that have identical or similar features, such that any one of them can be used to run a test (used by LAVA)
+
* '''Device type''' - The name of a set of DUTs that have identical or similar features, such that any one of them can be used to run a test (used by LAVA)
 
** Tim's comment: Some examples would be good.  I'm not sure I like this.  Is there a related term for the set of boards that have a particular type?  (e.g. something that refers to the pool of boards, rather than the characteristics of the set?  Maybe DUT pool?)
 
** Tim's comment: Some examples would be good.  I'm not sure I like this.  Is there a related term for the set of boards that have a particular type?  (e.g. something that refers to the pool of boards, rather than the characteristics of the set?  Maybe DUT pool?)
* Interactive DUT access - the ability to take a board out of automated testing service, for use in interactive testing or debugging sessions (or for some other reason.  "DUT-offlining"?  "DUT reservation"?)
+
* '''Interactive DUT access''' - the ability to take a board out of automated testing service, for use in interactive testing or debugging sessions (or for some other reason.  "DUT-offlining"?  "DUT reservation"?)
  
 
== Rejected terms ==
 
== Rejected terms ==
* Test Profile - same thing as Test Definition.  Test Profile is used by Phoronix Test Suite.
+
* '''Test Profile''' - same thing as Test Definition.  Test Profile is used by Phoronix Test Suite.
* DUT Supervisor - provides connection to the DUT and abstraction for DUT management actions (e.g. dut_boot, dut_login, dut_exec, dut_copyfrom, dut_copyto commands - additional software on NanoPi) (used by SLAV).  This glossary uses "Target manager" instead.
+
* '''DUT Supervisor''' - provides connection to the DUT and abstraction for DUT management actions (e.g. dut_boot, dut_login, dut_exec, dut_copyfrom, dut_copyto commands - additional software on NanoPi) (used by SLAV).  This glossary uses "Target manager" instead.
* DUT controller - program and hardware for controlling a DUT (reboot, provision, etc.).  Confusing because of DUT confusion.
+
* '''DUT controller''' - program and hardware for controlling a DUT (reboot, provision, etc.).  Confusing because of DUT confusion.
* DUT scheduler - program for managing access to a DUT (take online/offline, make available for interactive use) (Confusing because of DUT confusion)
+
* '''DUT scheduler''' - program for managing access to a DUT (take online/offline, make available for interactive use) (Confusing because of DUT confusion)
* Provision (verb) - arrange the DUT and the lab environment (including other external hardware) for a test
+
* '''Provision (verb)''' - arrange the DUT and the lab environment (including other external hardware) for a test
 
** This may include installing the SUT to the device under test and booting the DUT.
 
** This may include installing the SUT to the device under test and booting the DUT.
 +
 +
= External Test System Glossaries =
 +
* Fuego: http://fuegotest.org/wiki/Glossary
 +
* LAVA: https://validation.linaro.org/static/docs/v2/glossary.html

Revision as of 16:44, 13 November 2018

This is a list of terms that are used by various test system in the Linux ecosystem, and by the Test Standards that are discussed and proposed on this site.

Terms

  • Actual Value - the value that was seen for an operation performed by a test
  • Bisection - automatic testing of SUT variations to find the source of a problem
  • Boot - to start the target from an off state. The end of 'boot' is the point of time when a test can be started, which will be different depending on whether a test is testing the bootloader, the kernel or some user-space software.
  • Build artifact - item created during build of the software under test
  • Build manager - a machine or process that performs builds of the software under test. Also known as a build server, when it is a machine.
  • Dependency - indicates a pre-requisite that must be filled in order for a test to run
    • Some examples: must have root access, must have 100 meg of memory, some program or package must be installed, etc.
  • Device Under Test (DUT) - the product, board or device that is being tested. It consists of hardware and software under test. (See also 'board', 'target')
    • This glossary uses 'target' as the more generic term for a thing that hosts the system being tested (for example, it could be a Virtual Machine)
  • Deploy - put the test program or system software on the target
  • Expected value - the value that was expected for an operation performed by a test
  • Lab - a collection of resources for testing one or more targets (also 'board farm')
  • Log - one of the run artifacts - output from the test program or test framework
  • Log Parsing - extracting information from a log into a machine-processable format (possibly into a common format)
  • Monitor (noun) - a program or process to watch some attribute (e.g. power) while the test is running
    • A monitor could be running either on or off the target, and it could be collecting data from either on or off the target.
  • Notification - communication based on results of test (triggered by results and including results)
  • Pass criteria - set of constraints indicating pass/fail conditions for a test
  • PDU - Power Distribution Unit - a piece of hardware used to control power to one or more DUTs (used by LAVA)
  • Report generation - collecting run data and putting it into a formatted output
  • Request (noun) - a request to execute a test
  • Result - the status indicated by a test - pass/fail (or something else) for a Run
  • Results query - Selection and filtering of data from runs, to find patterns
  • Run (noun) - an execution instance of a test (in Jenkins, a build)
  • Run artifact - item created during a run of the test program
  • Serial console - the Linux console connected over a serial connection
  • Software under test - the software being tested
  • System under test - The complete system being tested, including hardware and software. See Target, DUT.
  • Target - The board, device or virtual machine in which tests are performed. Usually this is thought of to consist of both the Device Under Test and the accompanying Software Under Test.
  • Target Manager - a process or machine that controls and schedules a target (see also DUT controller, DUT supervisor, DUT scheduler)
    • This includes things like booting the target, controlling networking to the target, and moving data to and from the target.
  • Test agent - software running on the DUT that assists in test operations (e.g. test deployment, execution, log gathering, debugging
    • One example would be 'adb', for Android-based systems)
  • Test definition - meta-data and software that comprise a particular test
  • Test plan - the definition and meta-data for a group of tests run together (e.g. sequentially on the same target). This always includes the list of tests, but may also include per-test data such as timeout, test variables or reporting/notification instructions.
  • Test program - a script or binary on the target that performs the test
  • Test scheduler - program for scheduling tests (selecting a target for a test, reserving it, releasing it)
  • Test setup - refers to the sequence of operations required to prepare the target or the lab environment for the test
    • This might include other major operations, such as deploying the system software to the target.
  • Test software - source and/or binary that implements the test
  • Transport (noun) - the method of communicating and transferring data between the test system and the target
  • Trigger (noun) - an event that causes the CI loop to start
  • Variant - arguments or data that affect the execution and output of a test (e.g. test program command line; Fuego calls this a 'spec')
  • Visualization - allowing the viewing of test artifacts, in aggregated form (e.g. multiple runs plotted in a single diagram)

Contested or confusing terms

  • Device Under Test - was confusing to enterprise testers and those using VMS. Argument was that they have no hardware under test, and that "Device"

is confusing, and is treated as just another resource for the test.

  • Deploy - put the test program or system software on the target. Putting the system software on the target is much different from putting the test program on the target, so usage of "deploy" is ambiguous.

Candidate terms

  • Board - the board hardware under test. See also Device Under Test and Target.
  • Feature - an attribute of a DUT or SUT or test environment that can be used to match tests. Requiring a DUT to have a particular feature could be a test dependency. (used by labgrid)
  • Device type - The name of a set of DUTs that have identical or similar features, such that any one of them can be used to run a test (used by LAVA)
    • Tim's comment: Some examples would be good. I'm not sure I like this. Is there a related term for the set of boards that have a particular type? (e.g. something that refers to the pool of boards, rather than the characteristics of the set? Maybe DUT pool?)
  • Interactive DUT access - the ability to take a board out of automated testing service, for use in interactive testing or debugging sessions (or for some other reason. "DUT-offlining"? "DUT reservation"?)

Rejected terms

  • Test Profile - same thing as Test Definition. Test Profile is used by Phoronix Test Suite.
  • DUT Supervisor - provides connection to the DUT and abstraction for DUT management actions (e.g. dut_boot, dut_login, dut_exec, dut_copyfrom, dut_copyto commands - additional software on NanoPi) (used by SLAV). This glossary uses "Target manager" instead.
  • DUT controller - program and hardware for controlling a DUT (reboot, provision, etc.). Confusing because of DUT confusion.
  • DUT scheduler - program for managing access to a DUT (take online/offline, make available for interactive use) (Confusing because of DUT confusion)
  • Provision (verb) - arrange the DUT and the lab environment (including other external hardware) for a test
    • This may include installing the SUT to the device under test and booting the DUT.

External Test System Glossaries