Difference between revisions of "Test Glossary"

From eLinux.org
Jump to: navigation, search
 
(14 intermediate revisions by the same user not shown)
Line 2: Line 2:
 
that are discussed and proposed on this site.
 
that are discussed and proposed on this site.
  
== Terms ==
+
= Terms =
* Actual Value - the value that was seen for an operation performed by a test
+
* '''Actual Value''' - the value that was seen for an operation performed by a test
* Bisection - automatic testing of SUT variations to find the source of a problem
+
* '''Bisection''' - automatic testing of SUT variations to find the source of a problem
* Boot - to start the target (DUT) from an off state.  The end of 'boot' is the point of time when a test can be started, which will be different depending on whether a test is testing the bootloader, the kernel or some user-space software.
+
* '''Boot''' - to start the target from an off state.  The end of 'boot' is the point of time when a test can be started, which will be different depending on whether a test is testing the bootloader, the kernel or some user-space software.
* Build artifact - item created during build of the software under test
+
* '''Build artifact''' - item created during build of the software under test
* Build manager (build server) - a machine that performs builds of the software under test
+
* '''Build manager''' - a machine or process that performs builds of the software under test.  Also known as a build server, when it is a machine.
* Dependency - indicates a pre-requisite that must be filled in order for a test to run (e.g. must have root access, must have 100 meg of memory, some program must be installed, etc.)
+
* '''Continuous Integration Loop''' - the process of automatically testing software as it is integrated. Usually, a commit to a source repository (the trigger) will cause a build, deploy, instantiation and test of the software, with notification of results to interested parties. Also "CI Loop".
* Device under test (DUT) - the product, board or device that is being tested (consists of hardware under test and software under test) (also 'board', 'target')
+
* '''Dependency''' - indicates a pre-requisite that must be filled in order for a test to run
 +
** Some examples: test requires root access, test needs 100 meg of memory, some program or package must be installed, etc.
 +
* '''Device Under Test''' (DUT) - the product, board or device that is being tested. It consists of hardware and software under test. (See also 'board', 'target')
 
** This glossary uses 'target' as the more generic term for a thing that hosts the system being tested (for example, it could be a Virtual Machine)
 
** This glossary uses 'target' as the more generic term for a thing that hosts the system being tested (for example, it could be a Virtual Machine)
* Deploy - put the test program or system software on the target
+
* '''Deploy''' - put the test program or system software on the target
* Expected value - the value that was expected for an operation performed by a test
+
* '''Expected value''' - the value that was expected for an operation performed by a test
* Lab - a collection of resources for testing one or more targets (also 'board farm')
+
* '''Lab''' - a collection of resources for testing one or more targets (also 'board farm')
* Log - one of the run artifacts - output from the test program or test framework
+
* '''Log''' - one of the run artifacts - output from the test program or test framework
* Log Parsing - extracting information from a log into a machine-processable format (possibly into a common format)
+
* '''Log Parsing''' - extracting information from a log into a machine-processable format (possibly into a common format)
* Monitor - a program or process to watch some attribute (e.g. power) while the test is running
+
* '''Monitor (noun)''' - a program or process to watch some attribute (e.g. power) while the test is running
** This can be on or off the DUT.
+
** A monitor could be running either on or off the target, and it could be collecting data from either on or off the target.
* Notification - communication based on results of test (triggered by results and including results)
+
* '''Notification''' - communication based on results of test (triggered by results and including results)
* Pass criteria - set of constraints indicating pass/fail conditions for a test
+
* '''Pass criteria''' - set of constraints indicating pass/fail conditions for a test
* PDU - Power Distribution Unit - a piece of hardware used to control power to one or more DUTs (used by LAVA)
+
* '''PDU - Power Distribution Unit''' - a piece of hardware used to control power to one or more DUTs (used by LAVA)
* Provision (verb) - arrange the DUT and the lab environment (including other external hardware) for a test
+
* '''Report generation''' - collecting run data and putting it into a formatted output
** This may include installing the SUT to the device under test and booting the DUT.
+
* '''Request (noun)''' - a request to execute a test
* Report generation - collecting run data and putting it into a formatted output
+
* '''Result''' - the status indicated by a test - pass/fail (or something else) for a Run
* Request (noun) - a request to execute a test
+
* '''Results query''' - Selection and filtering of data from runs, to find patterns
* Result - the status indicated by a test - pass/fail (or something else) for a Run
+
* '''Run (noun)''' - an execution instance of a test (in Jenkins, a build)
* Results query - Selection and filtering of data from runs, to find patterns
+
* '''Run artifact''' - item created during a run of the test program
* Run (noun) - an execution instance of a test (in Jenkins, a build)
+
* '''Serial console''' - the Linux kernel console connected over a serial connection
* Run artifact - item created during a run of the test program
+
** Often, the kernel serial console is the most reliable way to receive test output data.  This is particularly true if the kernel is the software under test and the test is a boot test which might fail part-way through the boot.
* Serial console - the Linux console connected over a serial connection
+
* '''Software under test''' - the software being tested
* Software under test (SUT) - the software being tested
+
* '''System under test''' - The complete system being tested, including hardware and software. See Target, DUT.
* System under test - see Target, DUT - The complete system being tested, including hardware and software
+
* '''Target''' - The board, device or virtual machine in which tests are performed.  Usually this is thought of to consist of both the Device Under Test and the accompanying Software Under Test.
* Target - The board, device or virtual machine in which tests are performed.  Usually this is thought of to consist of both the Device under test and the accompanying software under test.
+
* '''Target Manager''' - a process or machine that controls and schedules a target (see also DUT controller, DUT supervisor, DUT scheduler)
* Target Manager - a process or machine that controls and schedules a target (see also DUT controller, DUT supervisor, DUT scheduler)
+
** This entity does things like booting the target, controlling networking to the target, moving data to and from the target, installing software on the target, and providing access to a target's serial console.
** This includes things like booting the target, controlling networking to the target, and moving data to and from the target.
+
* '''Test agent''' - software running on the DUT that assists in test operations (e.g. test deployment, execution, log gathering, debugging
* Test agent - software running on the DUT that assists in test operations (e.g. test deployment, execution, log gathering, debugging
 
 
** One example would be 'adb', for Android-based systems)
 
** One example would be 'adb', for Android-based systems)
* Test definition - meta-data and software that comprise a particular test
+
* '''Test definition''' - meta-data and software that comprise a particular test
* Test plan - the definition and meta-data for a group of tests run together (e.g. sequentially on the same target).  This always includes the list of tests, but may also include per-test data such as timeout, test variables or reporting/notification instructions.
+
* '''Test plan''' - the definition and meta-data for a group of tests run together (e.g. sequentially on the same target).  This always includes the list of tests, but may also include per-test data such as timeout, test variables or reporting/notification instructions.
* Test program - a script or binary on the target that performs the test
+
* '''Test program''' - a script or binary on the target that performs the test
* Test scheduler - program for scheduling tests (selecting a target for a test, reserving it, releasing it)
+
* '''Test scheduler''' - program for scheduling tests (selecting a target for a test, reserving it, releasing it)
* Test software - source and/or binary that implements the test
+
* '''Test setup''' - refers to the sequence of operations required to prepare the target or the lab environment for the test
* Transport (noun) - the method of communicating and transferring data between the test system and the target
+
** This might include other major operations, such as deploying the system software to the target.
* Trigger (noun) - an event that causes the CI loop to start
+
* '''Test software''' - source and/or binary that implements the test
* Variant - arguments or data that affect the execution and output of a test (e.g. test program command line; Fuego calls this a 'spec')
+
* '''Transport (noun)''' - the method of communicating and transferring data between the test system and the target
* Visualization - allowing the viewing of test artifacts, in aggregated form (e.g. multiple runs plotted in a single diagram)
+
* '''Trigger (noun)''' - an event that causes the CI loop to start
 +
* '''Variant''' - arguments or data that affect the execution and output of a test (e.g. test program command line; Fuego calls this a 'spec')
 +
* '''Visualization''' - allowing the viewing of test artifacts, in aggregated form (e.g. multiple runs plotted in a single diagram)
  
 
== Contested or confusing terms ==
 
== Contested or confusing terms ==
* Device under test - was confusing to enterprise testers and those using VMS.  Argument was that they have no hardware under test, and that "Device"
+
* '''Device Under Test''' - was confusing to enterprise testers and those using virtual machines.  Argument was that they have no hardware under test, and that "Device" is confusing, and is treated as just another resource for the test.
is confusing, and is treated as just another resource for the test.
+
* '''Deploy''' - put the test program or system software on the target.  Putting the system software on the target is much different from putting the test program on the target, so usage of "deploy" is ambiguous.
* Deploy - put the test program or system software on the target.  Putting the system software on the target is much different from putting the test program on the target, so usage of "deploy" is ambiguous.
 
  
 
== Candidate terms ==
 
== Candidate terms ==
* Feature - an attribute of a DUT or SUT or test environment that can be used to match tests.  Requiring a DUT to have a particular feature could be a test dependency. (used by labgrid)
+
* '''Board''' - the board hardware under test.  See also Device Under Test and Target.
* Device type - The name of a set of DUTs that have identical or similar features, such that any one of them can be used to run a test (used by LAVA)
+
* '''Feature''' - an attribute of a DUT or SUT or test environment that can be used to match tests.  Requiring a DUT to have a particular feature could be a test dependency.
** Tim's comment: Some examples would be good.  I'm not sure I like this.  Is there a related term for the set of boards that have a particular type?  (e.g. something that refers to the pool of boards, rather than the characteristics of the set?  Maybe DUT pool?)
+
** this is called 'feature' by labgrid
* Interactive DUT access - the ability to take a board out of automated testing service, for use in interactive testing or debugging sessions (or for some other reason. "DUT-offlining""DUT reservation"?)
+
** this is called a 'device tag' by LAVA
 +
** this was called a 'capability' by JTA (the precursor to Fuego)
 +
* '''Device type''' - The name of a set of DUTs that have identical or similar features, such that any one of them can be used to run a test (used by LAVA)
 +
** Tim Bird's comment: Some examples would be good.  I'm not sure I like this.  Is there a related term for the set of boards that have a particular type?  (e.g. something that refers to the set of boards, rather than the characteristics of the set?) Maybe board pool or target pool?
 +
** Jan Lubbe's comment: We may want to have the ability to put one DUT into several (overlapping) groups, so having 'labels' like the Jenkins executors may
 +
be a better fit. Otherwise 'device types', 'groups' or 'classes' come to mind.
 +
* '''Interactive DUT access''' - the ability to take a board out of automated testing service, for use in interactive testing or debugging sessions (or for some other reason.)
 +
** other possible terms: 'target-offlining''target reservation'?
  
 
== Rejected terms ==
 
== Rejected terms ==
* Test Profile - same thing as Test Definition.  Test Profile is used by Phoronix Test Suite.
+
Note: reason for rejection is in ''italics''.
* DUT Supervisor - provides connection to the DUT and abstraction for DUT management actions (e.g. dut_boot, dut_login, dut_exec, dut_copyfrom, dut_copyto commands - additional software on NanoPi) (used by SLAV).  This glossary uses "Target manager" instead.
+
 
* DUT controller - program and hardware for controlling a DUT (reboot, provision, etc.).  Confusing because of DUT confusion.
+
* '''Test Profile''' - ''same thing as Test Definition.'' Test Profile is used by Phoronix Test Suite.
* DUT scheduler - program for managing access to a DUT (take online/offline, make available for interactive use) (Confusing because of DUT confusion)
+
* '''DUT Supervisor''' - provides connection to the DUT and abstraction for DUT management actions (e.g. dut_boot, dut_login, dut_exec, dut_copyfrom, dut_copyto commands - additional software on NanoPi) (used by SLAV).  ''This glossary uses "Target manager" instead.''
 +
* '''DUT controller''' - program and hardware for controlling a DUT (reboot, provision, etc.).  ''Confusing because of DUT confusion. Subsumed in "Target Manager"''
 +
* '''DUT scheduler''' - program for managing access to a DUT (take online/offline, make available for interactive use) ''Confusing because of DUT confusion. Subsumed in "Target Manager".''
 +
* '''Provision (verb)''' - arrange the DUT and the lab environment (including other external hardware) for a test
 +
** This may include installing the SUT to the device under test and booting the DUT.
 +
** ''Not enough people used this for it to be useful for communication.''
 +
 
 +
= External Test System Glossaries =
 +
* Fuego: http://fuegotest.org/wiki/Glossary
 +
* LAVA: https://validation.linaro.org/static/docs/v2/glossary.html
 +
== Generic terminology lists ==
 +
* https://www.softwaretestinghelp.com/software-testing-terms-complete-glossary/
 +
* Here's one that overly comprehensive: [http://glossary.istqb.org/search/ The International Software Testing Qualification Board Glossary]

Latest revision as of 18:07, 9 January 2019

This is a list of terms that are used by various test system in the Linux ecosystem, and by the Test Standards that are discussed and proposed on this site.

Terms

  • Actual Value - the value that was seen for an operation performed by a test
  • Bisection - automatic testing of SUT variations to find the source of a problem
  • Boot - to start the target from an off state. The end of 'boot' is the point of time when a test can be started, which will be different depending on whether a test is testing the bootloader, the kernel or some user-space software.
  • Build artifact - item created during build of the software under test
  • Build manager - a machine or process that performs builds of the software under test. Also known as a build server, when it is a machine.
  • Continuous Integration Loop - the process of automatically testing software as it is integrated. Usually, a commit to a source repository (the trigger) will cause a build, deploy, instantiation and test of the software, with notification of results to interested parties. Also "CI Loop".
  • Dependency - indicates a pre-requisite that must be filled in order for a test to run
    • Some examples: test requires root access, test needs 100 meg of memory, some program or package must be installed, etc.
  • Device Under Test (DUT) - the product, board or device that is being tested. It consists of hardware and software under test. (See also 'board', 'target')
    • This glossary uses 'target' as the more generic term for a thing that hosts the system being tested (for example, it could be a Virtual Machine)
  • Deploy - put the test program or system software on the target
  • Expected value - the value that was expected for an operation performed by a test
  • Lab - a collection of resources for testing one or more targets (also 'board farm')
  • Log - one of the run artifacts - output from the test program or test framework
  • Log Parsing - extracting information from a log into a machine-processable format (possibly into a common format)
  • Monitor (noun) - a program or process to watch some attribute (e.g. power) while the test is running
    • A monitor could be running either on or off the target, and it could be collecting data from either on or off the target.
  • Notification - communication based on results of test (triggered by results and including results)
  • Pass criteria - set of constraints indicating pass/fail conditions for a test
  • PDU - Power Distribution Unit - a piece of hardware used to control power to one or more DUTs (used by LAVA)
  • Report generation - collecting run data and putting it into a formatted output
  • Request (noun) - a request to execute a test
  • Result - the status indicated by a test - pass/fail (or something else) for a Run
  • Results query - Selection and filtering of data from runs, to find patterns
  • Run (noun) - an execution instance of a test (in Jenkins, a build)
  • Run artifact - item created during a run of the test program
  • Serial console - the Linux kernel console connected over a serial connection
    • Often, the kernel serial console is the most reliable way to receive test output data. This is particularly true if the kernel is the software under test and the test is a boot test which might fail part-way through the boot.
  • Software under test - the software being tested
  • System under test - The complete system being tested, including hardware and software. See Target, DUT.
  • Target - The board, device or virtual machine in which tests are performed. Usually this is thought of to consist of both the Device Under Test and the accompanying Software Under Test.
  • Target Manager - a process or machine that controls and schedules a target (see also DUT controller, DUT supervisor, DUT scheduler)
    • This entity does things like booting the target, controlling networking to the target, moving data to and from the target, installing software on the target, and providing access to a target's serial console.
  • Test agent - software running on the DUT that assists in test operations (e.g. test deployment, execution, log gathering, debugging
    • One example would be 'adb', for Android-based systems)
  • Test definition - meta-data and software that comprise a particular test
  • Test plan - the definition and meta-data for a group of tests run together (e.g. sequentially on the same target). This always includes the list of tests, but may also include per-test data such as timeout, test variables or reporting/notification instructions.
  • Test program - a script or binary on the target that performs the test
  • Test scheduler - program for scheduling tests (selecting a target for a test, reserving it, releasing it)
  • Test setup - refers to the sequence of operations required to prepare the target or the lab environment for the test
    • This might include other major operations, such as deploying the system software to the target.
  • Test software - source and/or binary that implements the test
  • Transport (noun) - the method of communicating and transferring data between the test system and the target
  • Trigger (noun) - an event that causes the CI loop to start
  • Variant - arguments or data that affect the execution and output of a test (e.g. test program command line; Fuego calls this a 'spec')
  • Visualization - allowing the viewing of test artifacts, in aggregated form (e.g. multiple runs plotted in a single diagram)

Contested or confusing terms

  • Device Under Test - was confusing to enterprise testers and those using virtual machines. Argument was that they have no hardware under test, and that "Device" is confusing, and is treated as just another resource for the test.
  • Deploy - put the test program or system software on the target. Putting the system software on the target is much different from putting the test program on the target, so usage of "deploy" is ambiguous.

Candidate terms

  • Board - the board hardware under test. See also Device Under Test and Target.
  • Feature - an attribute of a DUT or SUT or test environment that can be used to match tests. Requiring a DUT to have a particular feature could be a test dependency.
    • this is called 'feature' by labgrid
    • this is called a 'device tag' by LAVA
    • this was called a 'capability' by JTA (the precursor to Fuego)
  • Device type - The name of a set of DUTs that have identical or similar features, such that any one of them can be used to run a test (used by LAVA)
    • Tim Bird's comment: Some examples would be good. I'm not sure I like this. Is there a related term for the set of boards that have a particular type? (e.g. something that refers to the set of boards, rather than the characteristics of the set?) Maybe board pool or target pool?
    • Jan Lubbe's comment: We may want to have the ability to put one DUT into several (overlapping) groups, so having 'labels' like the Jenkins executors may

be a better fit. Otherwise 'device types', 'groups' or 'classes' come to mind.

  • Interactive DUT access - the ability to take a board out of automated testing service, for use in interactive testing or debugging sessions (or for some other reason.)
    • other possible terms: 'target-offlining'? 'target reservation'?

Rejected terms

Note: reason for rejection is in italics.

  • Test Profile - same thing as Test Definition. Test Profile is used by Phoronix Test Suite.
  • DUT Supervisor - provides connection to the DUT and abstraction for DUT management actions (e.g. dut_boot, dut_login, dut_exec, dut_copyfrom, dut_copyto commands - additional software on NanoPi) (used by SLAV). This glossary uses "Target manager" instead.
  • DUT controller - program and hardware for controlling a DUT (reboot, provision, etc.). Confusing because of DUT confusion. Subsumed in "Target Manager"
  • DUT scheduler - program for managing access to a DUT (take online/offline, make available for interactive use) Confusing because of DUT confusion. Subsumed in "Target Manager".
  • Provision (verb) - arrange the DUT and the lab environment (including other external hardware) for a test
    • This may include installing the SUT to the device under test and booting the DUT.
    • Not enough people used this for it to be useful for communication.

External Test System Glossaries

Generic terminology lists