Difference between revisions of "Test Stack Layers"

From eLinux.org
Jump to: navigation, search
(Tim Bird (Fuego stack))
Line 119: Line 119:
 
* defines how to communicate with board (e.g. ssh, serial)
 
* defines how to communicate with board (e.g. ssh, serial)
 
** verbs are put, get, cmd
 
** verbs are put, get, cmd
** also:  
+
** also: transport_connect, transport_disconnect
  
 
== test layer ==
 
== test layer ==

Revision as of 16:14, 14 August 2018

Here is some information on standards for test stack layers.

Different people have different approaches to the different sub-tasks for an automated testing stack.

Andrew Murray (the Farm stack)

Andrew posted some thoughts on different layers (using a farm metaphor) Here is the original message: https://lists.yoctoproject.org/pipermail/automated-testing/2017-November/000134.html

Grass

The Grass - This is the bottom layer which provides the bare minimum software abstraction to the fundamental capabilities of the farm. This consists of physical devices and suitable drivers for them. The interface to higher layers is of the language of turn power port 2 on, bridge relay 5 on, etc. The goal here is for us to be able pick up some hardware off the shelf (e.g. a APC power switch) and to have a 'driver' already available from this community that fits into this view of the world. In order to achieve this it would be necessary to define categories of devices (power, relay, serial), the verbs (power cycle, power off, etc) and a way of addressing physical devices. I believe the LabGrid stack has a good model in this respect.

Cow

The Cow - This is the middle layer which uses the bottom layer and provides an abstraction of a board, this is where boards are defined and related to the physical devices. This layer would manage exclusive access to boards and self-tests. Users of this layer could find out what boards are available (access control), what their capabilities are and access to those capabilities. The language used here may be turn board on, get serial port, press BOOT switch, etc. The goal here is that we can work as a community to create definition of boards - a bit like device tree at a high physical level.

Dairy

The Dairy - This is the top layer which uses the middle layer. This is the application layer, applications can focus on their value-add which may be running tests (LAVA) or providing remote access to boards (perhaps with additional features such as 'workspaces' with prebuilt and predeployed software images).

Pawel Wieczorrk (SLAV stack)

From Pawel's presentation in ELC 2018: PDF (See also ELC_2018_Presentations for a link to the video.

In the SLAV stack, there is a DUT controller board for each DUT. They use the Muxpi as the DUT controller board.

Dryad

Manages single DUT

  • Fully aware of its capabilities
  • Requires only two interfaces
    • Power supply
    • Network connection (Ethernet)

A command line tool called 'stm' is used to multiplex an SDCard between the DUT and the test server. stm is also used to turn power on and off to a board.

A command line tool called 'fota' is used to ...?

Boruta

Manages access to multiple boards in a farm. Dryad farm management system

  • Schedules requests
  • Priority
  • Device groups
  • Delayed access
  • Provides convenient access to selected Dryad

This supports reservations for both the automated test system, and the

Boruta:

  • allocates boards (reserves them?)
* matches requirements
* prepares environment
* sets up tunnel

Weles

Welese is a lightweight testing framework, built on top of Boruta:

It provides a LAVA-like interface.

It has a YAML job definition that has actions executed on the DUT.

  • Deploy
  • Boot
  • Test
  • Collect logs and data

Weles does the following:

  • parse the YAML job definition
  • collect assets
  • request DUT
  • perform tests

Perun

Perun does OS image testing

  • schedules verification (based on new set of OS images)

Actions:

  • Crawl URL
  • report changes
  • submit weles jobs
  • collect artifacts
  • interpret results

Tim Bird (Fuego stack)

This section may get long, but here goes:

Fuego has several distinct areas if execution:

board layer

  • defines how to control board hardware
    • verb is hardware_reboot
  • defined how to access board functions (system logger)
  • defines SDK for board

transport layer

Defined in board file:

  • defines how to communicate with board (e.g. ssh, serial)
    • verbs are put, get, cmd
    • also: transport_connect, transport_disconnect

test layer

  • Tests are defined with shell scripts
  • Tests are built from source
  • the 'ftc' tool is used to execute a test, which consists of several phases:
    • pre_check - check that DUT has required attributes and is ready (matching)
    • build - build the test software
    • deploy - deploy the test software
    • run - execute the test
    • post_process - retrieve results
    • log_compare - analyze results


user interface

Jenkins is used for job/board scheduling

what's left to the user?

  • detecting that the software under test has changed
    • initiating the test (trigger)
  • building the software under test
  • deploying the software under test to the DUT (provisioning)

uncategorized list of actions

  • reboot the DUT
  • power on to DUT
  • power off to DUT
  • read serial console of DUT
  • build test software
    • generate test data
    • generate test variations
  • build software under test
    • build kernel under test
    • build rootfs under test
  • install software under test (provision)
    • install kernel (update tftp, switch sdcard, install to /boot, etc.)
    • install rootfs (update nfs area, write to sdcard, write to partitions, etc.)
  • switch sdcard to host
  • switch sdcard to DUT
  • deploy test
    • install test software
  • collect log from test / send log from test
  • parse log from test
  • present test results to user (web interface)
    • present table of past results
    • present chart of past results
  • detect software under test has changed
  • schedule test for DUT
  • reserve DUT
  • match job request with DUT
  • remove test from DUT
  • determine pass/fail from test log
    • determine if metric is over/under threshold
    • determine if required tests passed (allow individual test cases to be ignored)
  • detect if DUT is working
  • store test artifacts
    • store test results
  • notify user of test failure
  • notify user of board failure
  • bisect software under test
  • press/hold button on DUT


Here are a bunch of miscellaneous ones:

  • update expected results for a test
  • monitor power while test is running
  • validate data from test
  • disconnect/reconnect power during test
  • disconnect/reconnect bus (USB, CAN, i2C, etc.) during test
  • disconnect/reconnect network during test
  • change battery load during test
  • monitor temparature during test
  • capture video during test
  • capture audio during test
  • capture trace during test
  • control user interface during test
  • install host software needed for test
  • set up network server for test

tool operations

Operations supported by different tools:

ttc

  • console - access the target serial console
  • cp - Copy files to or from the target.
  • fsbuild - Build root filesystem for use on target.
  • fsinstall - Install root filesystem for use on target.
  • get_config - Install kernel config for target in the $KBUILD_OUTPUT directory
  • get_kernel - Install kernel sources for target in the $KERNEL_SRC directory
  • info - Show information about a target.
  • kbuild - Build kernel from source.
  • kinstall - Install kernel for use on target.
  • list - Show a list of available targets.
  • login - access a network login on the target.
  • off - Turn off target board.
  • on - Turn on target board.
  • pos - Show power status of target board.
  • reboot - Reboot target board.
  • release - Release a reservation of a target.
  • reserve - Reserve a target for use.
  • reset - Reset target board.
  • rm - Remove files from the target.
  • run - Run a command on the target.
  • set_config - Set one or more individual config options
  • setenv - Prepare environment for building for target.
  • status - Show status of target, including reservations. (not implemented yet)
  • wait_for - Wait for a condition to be true.

labgrid

labgrid-client

  • monitor - Monitor events from the coordinator
  • resources - List available resources
  • places - List available places
  • show - Ahow a place and related resources
  • create - Add a new place
  • delete - Delete an existing place
  • acquire - Acquire a place
  • release - Release a place
  • env - Generate a labgrid environement for a place
  • power - Change or get a place's power status
    • power get
    • power on
    • power off
    • power status
  • console - Connect to the console
  • fastboot - Run fastboot
  • bootstrap - Start a bootloader
  • io - Interact with Onewire devices

lava-tool?