Testing

This page has lists of resources for Linux testing.

Presentations from events

LPC 2016 testing mini-summit (Nov 2016)

The 2016 Linux Plumbers Conference had a mini-summit on Testing and Fuzzing, as well as other sessions dedicated to open source testing issues. Here are some links related to that event:

• Testing and Fuzzing agenda
  • etherpad from the session
• presentations:
  • Testing Kernels in the Cloud - by Theodore Ts'o
  • All About the perf_fuzzer - by Vince Weaver
  • Applying mutation testing for testing RCU: Progress report - by Iftekhar Ahmed
  • Integrating and running all the kernel tests - by Yannick Brosseau
  • Linux Kernel Testing: Where are we? (PDF - by Guenter Roeck
  • Syzkaller: Future development - by Dmitry Vyukov
  • Trinity: Next directions - by Dave Jones
  • Increase Test Coverage in a Linux-based OS (Google doc) - By Guillermo Ponce and Victor Rodriguez
    • BOF notes

collect presentation links above

Japan Jamboree 58-1 (Oct 2016)

• presentations:
  • Fuego: Introduction, Status, and Future directions (PDF) - by Tim Bird, Sony
  • Case Study (1) - by Kusakabe-san, Fujitsu TEN
  • Fuego LTP Improvements (PDF) - by Danial Sangorrin, Toshiba]
  • Continuous Integration and Autotest Environment using Fuego (PDF)(from ELCE 2016) - by Kengo Ibe, Kotaro Hashimoto, Mitsubishi Electric
  • Current State of the Test Activities in Renesas Using Fuego (PDF) - by Masao Nomura, Renesas Electronics Corporation

ELC Europe 2016 (Oct 2016)

• test frameworks:
  • KernelCI needs you! (PDF) - by Mark Brown, Linaro, and Kevin Hilman, BayLibre
  • Continuous Integration and Autotest Environment Using Fuego (PDF) - by Kenji Tadano & Kengo Ibe, Mitsubishi Electric

• board farms:
  • Herd Your Baords, Become a Farmer by Geert Uytterhoeven, Glider bvba
  • Automated Testing Laboratory for Embedded Linux Distributions (PDF) - by Pawel Wieczorek, Samsung R&D Institute Poland
  • Building a Board Farm: Continuous Integration and Remote Control (PDF) - by Antoine Tenart & Quentin Schulz, Free Electrons

Previous ELC and ELCE talks on testing

• ELCE 2010 - Embedded Software Testing (PDF) - by Arnout Vandecappelle

Testing Standards

Start collecting info for testing standards here:

• https://tools.ietf.org/html/rfc2119 - ietf MUST, SHALL, MAY, etc. wording standards

Things we'd like to standardize in open source testing:

• test nomenclature (test glossary)
• how to specify test pre-requisites
  • ex: assert_define ENV_VAR_NAME
  • ex: kernel_config
• test API
• host/target abstraction
  • kernel installation
  • file operations
  • console access
  • command execution
• test retrieval, build, deployment
  • test execution:
    • ex: 'make test'
• test log output format
  • counts
  • subtest results
  • Candidate formats:
    • https://testanything.org/
• test package format
  • meta-data for each test
  • test results
  • baseline expected results for particular tests on particular platforms
  • what tests can be skipped etc.