Automated Testing microsummit Linaro Connect BKK19
Linaro invited Tim Bird and Daniel Sangorrin from the Fuego project to Linaro Connect 2019 in Bangkok. At thie event, a "Testing microsummit" meeting was organized by Dan Rue.
Linaro Connect was held April 1-5, 2019 in Bangkok. There were a lot of sessions (listed below) around Linux testing, LAVA, and Fuego.
We also held an impromptu micro-summit (minutes below), where we all locked ourselves in a room together for the day and discussed LAVA, Fuego, test definitions, and yes, pdudaemon.
Finally, Maria Högberg (cc'd) has graciously arranged for a monthly meeting that we can use to coordinate across projects throughout the year - please send her an email if you'd like to be added. The first one is scheduled for Thursday, May 9th at 13:00 UTC.
Linaro Connect 2019 BBK - sessions on testing
Selected Sessions (videos should be available within a week or two at the referenced URLs):
- EAS Unit Testing
- LAVA Users Forum
- KEYNOTE: Open Source QA - what will it take to get to the next level
- Experiences and lessons we learned using kselftest and potential improvements
- LAVA community enabled testing
- Harmonizing open source test definitions
- KernelCI New Generation
- What is this Fuego thing and where is it going?
- Automating test results analysis using neural networks
- Scheduling in CI/CD systems
- Bootloader testing in LAVA
- How to integrate Fuego automated testing tool in your CI loop
Date: Wednesday, April 4, 2019
- Daniel Sangorrin (Toshiba)
- Tim Bird (Sony)
- Carlos Hernandez (TI)
- Kat Cosgrove (jfrog)
- Naresh Kamboju
- Antonio Tercerio
- Daniel Diaz
- Charles Oliveira
- Chase Qi
- Milosz Wasilewski
- Stevan Radakovic
- Dave Pigott
- Maria Högbergadding
- Luca Di Stefano
- Steve McIntyre
- Matt Hart
- Remi Duraffort
- Fathi Boudra
- Anders Roxell
- Dan Rue
- Wider community agreement on standardizing on pdudaemon
- Linaro lab doesn’t actually use pdudaemon
- Running pdudaemon @ Linaro
- [Dave] Queueing vs running immediately
- [matt] we can just add it to pdu daemon if this is a problem
- [Matt] added in a PR
- Dave will write a proposal for power control API
- There will be an abstraction layer for power control to turn a device on or off.
- [Dave] What about pressing buttons, turning on/off usb ports, etc
- [Tim] let’s not deal with composition right now, and only components
- [Carlos] we do need to solve the problem… once we have an interface for power, we need a similar interface for relays.
- Combinations of things may still make it complicated. For example, hold a button while pressing power.
- Core pdu api actions: on, off, status
- Core relay api actions: on, off, ???
- LAVA lab uses cambrionix USB hubs
- Also use good usb shielded cables
- Sony uses an open hardware board per DUT
- Tim gave a subset of his Thursday talk; see his Thursday talk for the full details.
- Jenkins based
- Test execution system
- Steps broken into discrete phases like build, deploy, run, process, etc.
- Board and platform management are abstracted
- Not serial console based; typically uses SSH on a board that’s already provisioned (imaged)
- Contains functional and benchmark tests
- Fuego is a linux distribution, distributed as a container, self contained
- Testing driven from host, using ssh connection to DUT
- Is container privileged?
- [tim] yes
- [dave] Dynamic device detection is possible without privileged mode
- Designed for embedded linux testing
- IoT possible
- By default, builds tests, though LTP has a way of checking to see if it’s on disk and skipping the build
- Can also just skip the build phase so long as the build is in the location that is expected on target
- [long discussion about skip lists, known issues, and (lack of) test documentation]
- Fuego-core has skip logic inside the test folder (ltp for example)
- Test-definitions has similar
- Additionally, lkft uses known issues in SQUAD to annotate known failures
- Remi gave a brief LAVA architecture overview
- Apache2, lava-server-gunicorn, postgresql, lava-logs, lava-master
- Lava-slave, lava-run, devices under test
- One lava-server, multiple dispatchers, multiple DUTs per dispatcher
- Lava-run is ephemeral process that communicates with DUT during a job run
- Device-type defines e.g. a raspberry pi
- Device defines an instance of a raspberry pi, with specifics about which port its plugged into, what serial port is it, etc.
- [daniel s] can i skip the deploy?
- Yes, it’s supported
- [daniel s] parsing in the dispatcher, where is that done?
- [steve] we recommend parsing gets done on DUT
- [remi] interestedin junit, tap13 parsing in LAVA so that a parser on DUT isn’t necessary
- [remi] goal is to not have to parse on DUT and do as much as we can on dispatcher
- Multinode jobs, used for networking, controlling lab hardware using lxc, etc
- Remi mentioned a feature to run a container as a part of a test run, which could help with jobs that are more complex, and eliminate a lot of existing multi-node jobs (making it easier)
Running Fuego tests in Lava
- Fuego operates a lot like an "interactive" LAVA test
- Are several solutions here: One that seems good is to use the new feature to run a container as part of a test.
Running Linaro tests in Fuego
- Fuego already has Functional.linaro, which runs a Linaro test using testrunner.
Tim Bird proposing `run-test.sh` as a naming convention for the name of the test program for each package
- In ptest, you can build a test package, which you can install and execute.
- More controversially, why don’t you rename all the <testname>.sh scripts to run-test.sh
- Tim Bird proposing a standard variable name for the location of kernel config path
- LTP implements ‘KCONFIG_PATH’
How do we decide such standards?
- Proposal @ Automated testing list
- Tap13 is just a website. Maybe we need something similar to publish these things on
Tim Bird on Test Names
- Standardizing on test names would be nice so that we can compare across projects
- Related to test case definition discussion
- A good first step is to investigate how tests are named by various projects, starting with LTP.
- How different are test case names for LTP?
Carlos Hernandez - Test case definition standards (proposal) Provide everything needed to run any test from any test framework.
- Test Name
- Test Execution Engine (e.g. LAVA, Fuego, VATF)
- Test script path
- Agreement to start a public monthly meeting. Maria will organize.
Linaro graciously sponsored Tim and Daniel's attendance at LCC BKK19.