Difference between revisions of "Test Stack Layers"

From eLinux.org
Jump to: navigation, search
(add SLAV stack)
 
(add list of operations)
Line 104: Line 104:
 
* collect artifacts
 
* collect artifacts
 
* interpret results
 
* interpret results
 +
 +
= Tim Bird (Fuego stack) =
 +
This section may get long, but here goes:
 +
 +
Fuego has several distinct sections:
 +
== board layer ==
 +
* defines how to communicate with board (e.g. ssh, serial)
 +
** verbs are put, get, cmd
 +
* defines how to control board
 +
** verb is hardware_reboot
 +
* defined how to access board functions (system logger)
 +
* defines SDK for board
 +
 +
== test layer ==
 +
* Tests are defined with shell scripts
 +
* Tests are built from source
 +
 +
== user interface ==
 +
Jenkins is used for job/board scheduling
 +
* Triggers are left as an exercise for the user
 +
 +
== what's left to the user? ==
 +
* detecting that the software under test has changed
 +
* building the software under test
 +
* deploying the software under test to the DUT (provisioning)
 +
 +
= uncategorized list of actions =
 +
* reboot the DUT
 +
* power on to DUT
 +
* power off to DUT
 +
* read serial console of DUT
 +
* build test software
 +
* build software under test
 +
** build kernel under test
 +
** build rootfs under test
 +
* install software under test (provision)
 +
** install kernel (update tftp, switch sdcard, install to /boot, etc.)
 +
** install rootfs (update nfs area, write to sdcard, write to partitions, etc.)
 +
* deploy test
 +
** install test software
 +
* collect log from test / send log from test
 +
* parse log from test
 +
* present test results to user (web interface)
 +
** present table of past results
 +
** present chart of past results
 +
* detect software under test has changed
 +
* schedule test for DUT
 +
* reserve DUT
 +
* match job request with DUT
 +
* remove test from DUT
 +
* determine pass/fail from test log
 +
** determine if metric is over/under threshold
 +
** determine if required tests passed (allow individual test cases to be ignored)
 +
* detect if DUT is working
 +
* store test artifacts
 +
** store test results

Revision as of 14:31, 14 August 2018

Here is some information on standards for test stack layers.

Different people have different approaches to the different sub-tasks for an automated testing stack.

Andrew Murray (the Farm stack)

Andrew posted some thoughts on different layers (using a farm metaphor) Here is the original message: https://lists.yoctoproject.org/pipermail/automated-testing/2017-November/000134.html

Grass

The Grass - This is the bottom layer which provides the bare minimum software abstraction to the fundamental capabilities of the farm. This consists of physical devices and suitable drivers for them. The interface to higher layers is of the language of turn power port 2 on, bridge relay 5 on, etc. The goal here is for us to be able pick up some hardware off the shelf (e.g. a APC power switch) and to have a 'driver' already available from this community that fits into this view of the world. In order to achieve this it would be necessary to define categories of devices (power, relay, serial), the verbs (power cycle, power off, etc) and a way of addressing physical devices. I believe the LabGrid stack has a good model in this respect.

Cow

The Cow - This is the middle layer which uses the bottom layer and provides an abstraction of a board, this is where boards are defined and related to the physical devices. This layer would manage exclusive access to boards and self-tests. Users of this layer could find out what boards are available (access control), what their capabilities are and access to those capabilities. The language used here may be turn board on, get serial port, press BOOT switch, etc. The goal here is that we can work as a community to create definition of boards - a bit like device tree at a high physical level.

Dairy

The Dairy - This is the top layer which uses the middle layer. This is the application layer, applications can focus on their value-add which may be running tests (LAVA) or providing remote access to boards (perhaps with additional features such as 'workspaces' with prebuilt and predeployed software images).

Pawel Wieczorrk (SLAV stack)

From Pawel's presentation in ELC 2018: PDF (See also ELC_2018_Presentations for a link to the video.

In the SLAV stack, there is a DUT controller board for each DUT. They use the Muxpi as the DUT controller board.

Dryad

Manages single DUT

  • Fully aware of its capabilities
  • Requires only two interfaces
    • Power supply
    • Network connection (Ethernet)

A command line tool called 'stm' is used to multiplex an SDCard between the DUT and the test server. stm is also used to turn power on and off to a board.

A command line tool called 'fota' is used to ...?

Boruta

Manages access to multiple boards in a farm. Dryad farm management system

  • Schedules requests
  • Priority
  • Device groups
  • Delayed access
  • Provides convenient access to selected Dryad

This supports reservations for both the automated test system, and the

Boruta:

  • allocates boards (reserves them?)
* matches requirements
* prepares environment
* sets up tunnel

Weles

Welese is a lightweight testing framework, built on top of Boruta:

It provides a LAVA-like interface.

It has a YAML job definition that has actions executed on the DUT.

  • Deploy
  • Boot
  • Test
  • Collect logs and data

Weles does the following:

  • parse the YAML job definition
  • collect assets
  • request DUT
  • perform tests

Perun

Perun does OS image testing

  • schedules verification (based on new set of OS images)

Actions:

  • Crawl URL
  • report changes
  • submit weles jobs
  • collect artifacts
  • interpret results

Tim Bird (Fuego stack)

This section may get long, but here goes:

Fuego has several distinct sections:

board layer

  • defines how to communicate with board (e.g. ssh, serial)
    • verbs are put, get, cmd
  • defines how to control board
    • verb is hardware_reboot
  • defined how to access board functions (system logger)
  • defines SDK for board

test layer

  • Tests are defined with shell scripts
  • Tests are built from source

user interface

Jenkins is used for job/board scheduling

  • Triggers are left as an exercise for the user

what's left to the user?

  • detecting that the software under test has changed
  • building the software under test
  • deploying the software under test to the DUT (provisioning)

uncategorized list of actions

  • reboot the DUT
  • power on to DUT
  • power off to DUT
  • read serial console of DUT
  • build test software
  • build software under test
    • build kernel under test
    • build rootfs under test
  • install software under test (provision)
    • install kernel (update tftp, switch sdcard, install to /boot, etc.)
    • install rootfs (update nfs area, write to sdcard, write to partitions, etc.)
  • deploy test
    • install test software
  • collect log from test / send log from test
  • parse log from test
  • present test results to user (web interface)
    • present table of past results
    • present chart of past results
  • detect software under test has changed
  • schedule test for DUT
  • reserve DUT
  • match job request with DUT
  • remove test from DUT
  • determine pass/fail from test log
    • determine if metric is over/under threshold
    • determine if required tests passed (allow individual test cases to be ignored)
  • detect if DUT is working
  • store test artifacts
    • store test results