Testing

This page has lists of resources for Linux testing.

Board Farm Page

See the Board Farm page

Automated Testing page

The Automated Testing page has information about issues and initiatives related to automated testing of embedded Linux.

Presentations from events

LPC 2016 testing mini-summit (Nov 2016)

The 2016 Linux Plumbers Conference had a mini-summit on Testing and Fuzzing, as well as other sessions dedicated to open source testing issues. Here are some links related to that event:

• Testing and Fuzzing agenda
  • etherpad from the session
• presentations:
  • Testing Kernels in the Cloud - by Theodore Ts'o
  • All About the perf_fuzzer - by Vince Weaver
  • Applying mutation testing for testing RCU: Progress report - by Iftekhar Ahmed
  • Integrating and running all the kernel tests - by Yannick Brosseau
  • Linux Kernel Testing: Where are we? (PDF) - by Guenter Roeck
  • Syzkaller: Future development - by Dmitry Vyukov
  • Trinity: Next directions - by Dave Jones
  • Increase Test Coverage in a Linux-based OS (Google doc) - By Guillermo Ponce and Victor Rodriguez
    • BOF notes

collect presentation links above

Japan Jamboree 58-1 (Oct 2016)

• presentations:
  • Fuego: Introduction, Status, and Future directions (PDF) - by Tim Bird, Sony
  • Case Study (1) - by Kusakabe-san, Fujitsu TEN
  • Fuego LTP Improvements (PDF) - by Danial Sangorrin, Toshiba]
  • Continuous Integration and Autotest Environment using Fuego (PDF)(from ELCE 2016) - by Kengo Ibe, Kotaro Hashimoto, Mitsubishi Electric
  • Current State of the Test Activities in Renesas Using Fuego (PDF) - by Masao Nomura, Renesas Electronics Corporation

ELC Europe 2016 (Oct 2016)

• test frameworks:
  • KernelCI needs you! (PDF) - by Mark Brown, Linaro, and Kevin Hilman, BayLibre
  • Continuous Integration and Autotest Environment Using Fuego (PDF) - by Kenji Tadano & Kengo Ibe, Mitsubishi Electric

• board farms:
  • Herd Your Baords, Become a Farmer by Geert Uytterhoeven, Glider bvba
  • Automated Testing Laboratory for Embedded Linux Distributions (PDF) - by Pawel Wieczorek, Samsung R&D Institute Poland
  • Building a Board Farm: Continuous Integration and Remote Control (PDF) - by Antoine Tenart & Quentin Schulz, Free Electrons

Previous ELC and ELCE talks on testing

• ELCE 2010 - Embedded Software Testing (PDF) - by Arnout Vandecappelle

Testing Standards

Start collecting info for testing standards here:

• https://tools.ietf.org/html/rfc2119 - ietf MUST, SHALL, MAY, etc. wording standards

Things we'd like to standardize in open source testing:

• test nomenclature (test glossary)
• how to specify test pre-requisites
  • ex: assert_define ENV_VAR_NAME
  • ex: kernel_config
• test API
• host/target abstraction
  • kernel installation
  • file operations
  • console access
  • command execution
• test retrieval, build, deployment
  • test execution:
    • ex: 'make test'
• test log output format
  • counts
  • subtest results
  • Candidate formats:
    • https://testanything.org/
• test package format
  • meta-data for each test
  • test results
  • baseline expected results for particular tests on particular platforms
  • what tests can be skipped etc.