Test Execution Notes

This page has notes about aspects of the execution control for different systems that perform test execution.

Here are some of the elements of a the test execution API:

• test API - what functions can the test call?
  • file operations
  • service operations (start, stop)
  • log access
  • results output functions
• host/target abstraction - is the test executed from host or directly on target
• DUT preparation/control: - this might be in a separate standard
  • kernel installation / provisioning
  • console access
  • log access
  • test activation - how is the test activated or started?
    • e.g. installed and activated as part of boot
    • e.g. a command is executed as a program?
    • e.g. is the test built into the kernel and activated via some trigger mechanism (e.g. via /proc or /sys or an ioctl)?
• test retrieval, build, deployment
• test execution:
  • test name, or standard for test invocation name
  • ex: 'make test' - a standard make target name?
  • runtest.sh? - a standard shell script name?
  • <testname>.sh - a standard shell script name, based on test meta-data?
• test phases?
• test selection / skiplists - how are test selected for inclusion or exclusion in a test run
• test variables - how does the test control execution
  • make variables
  • environment variables
  • command line arguments

Does this include test scheduling? (no), but it does include test selection.

Does this include test building (no) - see Test Building notes

board-local test APIs

Fuego

test script names and locations

Every test is in a directory

APIs

Provided by board library: fuego_board_function_lib.sh

Is a library of shell functions.

board-side functions:

• set_init_manager()
• detect_logger_service()
• exec_service_on_target()
• detect_active_eth_device()
• get_service_status()

host-side functions: See http://fuegotest.org/wiki/Test_Script_APIs

test variables

Provided in environment variables, prefixed by test name:

• ex: FUNCTIONAL_HELLO_ARGS
• ex:

Required minimum Linux command set

Tools builtin or on the PATH

• cat, df, find, free, grep, head, logger, logread, mkdir, mount, ps, rm ,rmdir, sync, tail, tee, touch, true, umount, uname, uptime, xargs, [

Required command arguments:

• mkdir -p, rm -rf, grep -f,

Tools at specific paths:

• /sbin/reboot, /bin/true, /sbin/route

Data files:

• /var/log/messages, /var/log/syslog
• /proc/interrupts, /proc/drop_caches, /proc/oom_score_adj

(optional)

• proc/config.gz

Require arguments:

kselftest

• test selection via TARGETS make variable
• test skiplist support via SKIP_TARGETS make variable
  • patch submitted 9/24/2019 - need to check mainline status - see https://lore.kernel.org/linux-kselftest/20190925132421.23572-1-cristian.marussi@arm.com/

LTP

• test selection via 'scenario' or 'runtest' files (located in <topdir>/runtest)
• test skiplist support via runltp -S option
  • ex: runltp ... -S skiplist.txt ...
• test command line argument control is via the runtest file