Automated Testing Summit 2019
This is a public planning page for the Automated Testing Summit
Please see: https://events.linuxfoundation.org/events/ats-2019/
Contents
2019 meeting
Coordinated by: Tim Bird This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything.
See the Linux Foundation Event page for details.
Mailing list
Inter-project discussions on Automated Testing occur on the "Automated Testing" mailing list of the Yocto Project
Date and Venue
- Location: Lyon, France
- Venue: Lyon Convention Center, 50 Quai Charles de Gaulle
- Date: October 31, 2019
- Rooms: Pasteur Lounge and Rhone 3AB
- Time: 9:00 am to 17:15 (5:15 pm)
- co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe
- ELCE and OSSEU run Oct 28-30
- see https://events.linuxfoundation.org/events/embedded-linux-conference-europe-2019/
Attendance is by open to the public, and we are projecting the attendee fee to be $250, with this fee waived for speakers.
Summit Pre-planning
Here is a list of action items to do to prepare for the summit:
- organize program committee (done)
- get sponsors (done)
- create sponsorship prospectus (done)
- contact candidate sponsors (done)
- hold "Call for Proposals" (done)
- need to determine CFP dates (done)
- determine agenda (done)
- organize program committee (done)
- advertise for attendees (done)
- reserve venue, catering, AV rentals (done)
- create signage, web site, registration system (done)
- hold event (done)
For private planning information, see the private planning wiki page
Call for Presentations
- CFP Open: Tuesday, May 28
- CFP Close: Friday, August 13
- Notifications Out: September 2
- Schedule Out: September 4
- Slides Due: October 24, 2019
- Event Date: October 31, 2019
Suggested Topics
- Board Farms and Test Labs - hardware, software, useful practices
- Layers and Interfaces - components of your automated CI loop and protocols between them
- Results Reporting - results aggregation, visualization, notifications, etc.
- Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc.
- Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding
- Scalability - supporting big data, or handling distributed builds or tests
- Ease of use - making testing and test development accessible to non-QA folk (developers, end-users)
Pre-meeting work
Here's what Tim thinks we need to do:
- Refine the test stack reference diagram
- Organize committees for specific APIs
- designate a head for each committee
Sponsors
Sponsorship was provided by:
- Sony - Platinum sponsor
- Core Embedded Linux Project (of the Linux Foundation) - Gold sponsor
- Linaro - Bronze sponsor
- Civil Infrastructure Platform - Bronze sponsor
Schedule
Summit Artifacts
documents and presentations
- for presentations - see below
Key decisions
- all systems to make a kcidb client (upload results)
- extend the kcidb schema
- https://github.com/kernelci/kcidb
- priority (over test definition unification work)
- use LTP metadata format as initial standard
- add meta-data convert to kselftest (Tim)
- build a prototype to move boards between LAVA and labgrid (Jan)
- want to collect more information for "Hardware Design for Testing"
- Contributions are Welcome: https://github.com/SmithChart/Designing-for-Automated-Testing/
- get something working for board management layer for poweron/poweroff/serial-console
- continue work on test definition common schema and central repository (Tim)
- add information about hardware you find useful, or have designed please add to Board_Farm
- focus on meeting at Plumbers next year
- continue using automated-testing@lists.yoctoproject.org
Action items from meeting
Action items:
- post slides to this page
- send minutes
Presentations
Session | Presenter | Slides |
---|---|---|
Status updates | ||
The bugs are too fast, and why we can't catch them (and a report on recent meetups) | Kevin Hilman | File:ATS 2019.pdf |
LKFT status update | Milosz Wasilewski | File:2019 ATS LKFT status.pdf |
Fuego Status Update | Tim Bird | File:ATS2019-Fuego-Status-Update.pdf |
KernelCI Status Update | Kevin Hilman | File:ATS 2019.pdf (starting at slide 15) |
CKI Status Update | Veronika Kabatova | File:Cki-overview-ats.pdf |
SLAV Status Update | Paweł Wieczorek | File:SLAV Status Update.pdf |
Collaboration philosophy and issues | ||
Open Testing Philosophy | Kevin Hilman | File:ATS 2019.pdf (starting at slide 23) |
Working together to Build a Modular CI Ecosystem | Tim Bird | File:ATS2019- buiding-a-modular-ecosystem.pdf |
Board/LAB management track | ||
Labgrid - Real World Examples | Jan Lübbe | File:ATS2019-jlu-labgrid.pdf |
New Ways Out of the Struggle of Testing Embedded Devices | Chris Fiege | File:ATS2019-cfi-embedded-testing handout.pdf |
Beaker Project: Automated Testing at Red Hat | Tomas Klohna | Beaker at Slides.com File:ATS19 - Beaker Project - Automated Testing at Red Hat.pdf |
SLAV: Test Stack Abstraction Layers | Paweł Wieczorek | File:Test stack abstraction layers.pdf |
Test systems/ Test definitions track | ||
Bootloader testing in LAVA | Remi Duraffort | File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf |
Test Metadata Extraction | Cyril Hrubis | File:Test metadata extraction.pdf |
Test Plan Templating in LAVA | Milosz Wasilewski | File:Testjob templating for LAVA.pdf |
A Survey of Open Source Test Definitions | Tim Bird | File:ATS2019-survey-of-test-definitions.pdf |
QA systems and issues | ||
How AGL Tests its distros and what challenges we face | Jan-Simon Möller | <need file> |
A Guide to CIP Testing | Chris Paterson & Michael Adler | File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf |