https://elinux.org/api.php?action=feedcontributions&user=Pawiecz&feedformat=atomeLinux.org - User contributions [en]2024-03-29T13:24:22ZUser contributionsMediaWiki 1.31.0https://elinux.org/index.php?title=Automated_Testing_Summit_2019&diff=503996Automated Testing Summit 20192019-11-13T12:06:11Z<p>Pawiecz: Add SLAV Status Update slides</p>
<hr />
<div>This is a public planning page for the Automated Testing Summit<br />
<br />
Please see: https://events.linuxfoundation.org/events/ats-2019/<br />
<br />
= 2019 meeting =<br />
Coordinated by: Tim Bird<br />
This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything.<br />
<br />
See the [https://events.linuxfoundation.org/events/ats-2019/ Linux Foundation Event page] for details.<br />
<br />
== Mailing list ==<br />
Inter-project discussions on Automated Testing occur on the "Automated Testing" mailing list of the Yocto Project<br />
* See https://lists.yoctoproject.org/listinfo/automated-testing<br />
<br />
= Date and Venue =<br />
* Location: Lyon, France<br />
* Venue: Lyon Convention Center, 50 Quai Charles de Gaulle<br />
* Date: October 31, 2019<br />
* Rooms: Pasteur Lounge and Rhone 3AB<br />
* Time: 9:00 am to 17:15 (5:15 pm)<br />
<br />
** co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe<br />
** ELCE and OSSEU run Oct 28-30<br />
** see https://events.linuxfoundation.org/events/embedded-linux-conference-europe-2019/<br />
<br />
Attendance is by open to the public, and we are projecting the attendee fee to be $250, with this<br />
fee waived for speakers.<br />
<br />
= Summit Pre-planning =<br />
Here is a list of action items to do to prepare for the summit:<br />
* organize program committee (''done'')<br />
* get sponsors (''done'')<br />
** create sponsorship prospectus (''done'')<br />
** contact candidate sponsors (''done'')<br />
* hold "Call for Proposals" (''done'')<br />
** need to determine CFP dates (''done'')<br />
* determine agenda (''done'')<br />
** organize program committee (''done'')<br />
* advertise for attendees (''done'')<br />
* reserve venue, catering, AV rentals (''done'')<br />
* create signage, web site, registration system (''done'')<br />
* hold event (''done'')<br />
<br />
For private planning information, see [http://embeddedlinuxconference.com/cgi-bin/elcwiki.cgi/ATS_Planning the private planning wiki page]<br />
<br />
== Call for Presentations ==<br />
* CFP Open: Tuesday, May 28<br />
* CFP Close: Friday, August 13<br />
* Notifications Out: September 2<br />
* Schedule Out: September 4<br />
* Slides Due: October 24, 2019<br />
* Event Date: October 31, 2019<br />
<br />
Suggested Topics<br />
<br />
* Board Farms and Test Labs - hardware, software, useful practices<br />
* Layers and Interfaces - components of your automated CI loop and protocols between them<br />
* Results Reporting - results aggregation, visualization, notifications, etc.<br />
* Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc.<br />
* Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding<br />
* Scalability - supporting big data, or handling distributed builds or tests<br />
* Ease of use - making testing and test development accessible to non-QA folk (developers, end-users)<br />
<br />
= Pre-meeting work =<br />
Here's what Tim thinks we need to do:<br />
* Refine the test stack reference diagram<br />
* Organize committees for specific APIs<br />
** designate a head for each committee<br />
<br />
= Sponsors =<br />
Sponsorship was provided by:<br />
* '''Sony''' - Platinum sponsor<br />
* '''Core Embedded Linux Project''' (of the Linux Foundation) - Gold sponsor<br />
* '''Linaro''' - Bronze sponsor<br />
* '''Civil Infrastructure Platform''' - Bronze sponsor<br />
<br />
= Schedule =<br />
See https://ats19.sched.com/<br />
<br />
= Summit Artifacts =<br />
''documents and presentations''<br />
* for presentations - see below<br />
<br />
== Key decisions ==<br />
* all systems to make a kcidb client (upload results)<br />
** extend the kcidb schema<br />
** https://github.com/kernelci/kcidb<br />
** priority (over test definition unification work)<br />
* use LTP metadata format as initial standard<br />
** add meta-data convert to kselftest (Tim)<br />
* build a prototype to move boards between LAVA and labgrid (Jan)<br />
* want to collect more information for "Hardware Design for Testing"<br />
** Contributions are Welcome: https://github.com/SmithChart/Designing-for-Automated-Testing/<br />
* get something working for board management layer for poweron/poweroff/serial-console<br />
* continue work on test definition common schema and central repository (Tim)<br />
* add information about hardware you find useful, or have designed please add to [[Board_Farm]]<br />
* focus on meeting at Plumbers next year<br />
* continue using automated-testing@lists.yoctoproject.org<br />
<br />
== Action items from meeting ==<br />
Action items:<br />
* post slides to this page<br />
* send minutes<br />
<br />
== Presentations ==<br />
{| class="wikitable"<br />
|-<br />
! Session !! Presenter !! Slides<br />
|- bgcolor="#a0c0c0"<br />
| colspan="3" align="center" | Status updates and collaboration issues<br />
|-<br />
| The bugs are too fast, and why we can't catch them (report on recent meetups) || Kevin Hilman || [[File:ATS_2019.pdf]]<br />
|-<br />
| LKFT status update || Milosz Wasilewski || <need file><br />
|-<br />
| Fuego Status Update|| Tim Bird || [[File:ATS2019-Fuego-Status-Update.pdf]]<br />
|-<br />
| KernelCI Status Update || Kevin Hilman || [[File:ATS_2019.pdf]] (starting at slide 15)<br />
|-<br />
| CKI Status Update || Veronika Kabatova || <need file><br />
|-<br />
| SLAV Status Update || Paweł Wieczorek || [[File:SLAV_Status_Update.pdf]]<br />
|- bgcolor="#a0c0c0"<br />
| colspan="3" align="center" | Collaboration philosophy and issues<br />
|-<br />
| Open Testing Philosophy || Kevin Hilman || [[File:ATS_2019.pdf]] (starting at slide 23)<br />
|- <br />
| Working together to Build a Modular CI Ecosystem|| Tim Bird || [[File:ATS2019- buiding-a-modular-ecosystem.pdf]]<br />
|- bgcolor="#a0c0c0"<br />
| colspan="3" align="center" | Board/LAB management track <br />
|-<br />
| Labgrid - Real World Examples || Jan Lübbe || <need file><br />
|-<br />
| New Ways Out of the Struggle of Testing Embedded Devices || Chris Fiege || [[File:ATS2019-cfi-embedded-testing_handout.pdf]]<br />
|-<br />
| Beaker Project: Automated Testing at Red Hat|| Tomas Klohna || [https://redhat.slides.com/tklohna/deck?token=soUy6UhB Beaker at Slides.com] [[File:ATS19 - Beaker Project - Automated Testing at Red Hat.pdf]]<br />
|-<br />
| SLAV: Test Stack Abstraction Layers || Paweł Wieczorek || [[File:Test_stack_abstraction_layers.pdf]]<br />
|- bgcolor="#a0c0c0"<br />
| colspan="3" align="center" | Test systems/ Test definitions track<br />
|-<br />
| Bootloader testing in LAVA || Remi Duraffort || [[File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf]]<br />
|-<br />
| Test Metadata Extraction|| Cyril Hrubis || [[File:Test metadata extraction.pdf]]<br />
|-<br />
| Test Plan Templating in LAVA || Milosz Wasilewski || <need file><br />
|-<br />
| A Survey of Open Source Test Definitions || Tim Bird || [[File:ATS2019-survey-of-test-definitions.pdf]]<br />
|- bgcolor="#a0c0c0"<br />
| colspan="3" align="center" | QA systems and issues<br />
|-<br />
| How AGL Tests its distros and what challenges we face || Jan-Simon Möller || <need file><br />
|-<br />
| A Guide to CIP Testing || Chris Paterson & Michael Adler || [[File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf]]<br />
|-<br />
|}</div>Pawieczhttps://elinux.org/index.php?title=File:SLAV_Status_Update.pdf&diff=503991File:SLAV Status Update.pdf2019-11-13T12:04:20Z<p>Pawiecz: </p>
<hr />
<div></div>Pawieczhttps://elinux.org/index.php?title=Automated_Testing_Summit_2019&diff=503186Automated Testing Summit 20192019-10-31T19:34:43Z<p>Pawiecz: /* Presentations */</p>
<hr />
<div>This is a public planning page for the Automated Testing Summit<br />
<br />
Please see: https://events.linuxfoundation.org/events/ats-2019/<br />
<br />
= 2019 meeting =<br />
Coordinated by: Tim Bird<br />
This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything.<br />
<br />
See the [https://events.linuxfoundation.org/events/ats-2019/ Linux Foundation Event page] for details.<br />
<br />
== Mailing list ==<br />
Inter-project discussions on Automated Testing occur on the "Automated Testing" mailing list of the Yocto Project<br />
* See https://lists.yoctoproject.org/listinfo/automated-testing<br />
<br />
= Date and Venue =<br />
* Location: Lyon, France<br />
* Venue: Lyon Convention Center, 50 Quai Charles de Gaulle<br />
* Date: October 31, 2019<br />
* Rooms: Pasteur Lounge and Rhone 3AB<br />
* Time: 9:00 am to 17:15 (5:15 pm)<br />
<br />
** co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe<br />
** ELCE and OSSEU run Oct 28-30<br />
** see https://events.linuxfoundation.org/events/embedded-linux-conference-europe-2019/<br />
<br />
Attendance is by open to the public, and we are projecting the attendee fee to be $250, with this<br />
fee waived for speakers.<br />
<br />
= Summit Pre-planning =<br />
Here is a list of action items to do to prepare for the summit:<br />
* organize program committee (''in progress'')<br />
* get sponsors<br />
** create sponsorship prospectus (''almost done'')<br />
** contact candidate sponsors (''in progress'')<br />
* hold "Call for Proposals"<br />
** need to determine CFP dates (''done'')<br />
* determine agenda<br />
** organize program committee (''in progress'')<br />
* advertise for attendees<br />
* reserve venue, catering, AV rentals (''done'')<br />
* create signage, web site, registration system<br />
<br />
For private planning information, see [http://embeddedlinuxconference.com/cgi-bin/elcwiki.cgi/ATS_Planning the private planning wiki page]<br />
<br />
== Call for Presentations ==<br />
* CFP Open: Tuesday, May 28<br />
* CFP Close: Friday, August 13<br />
* Notifications Out: September 2<br />
* Schedule Out: September 4<br />
* Slides Due: October 24, 2019<br />
* Event Date: October 31, 2019<br />
<br />
Suggested Topics<br />
<br />
* Board Farms and Test Labs - hardware, software, useful practices<br />
* Layers and Interfaces - components of your automated CI loop and protocols between them<br />
* Results Reporting - results aggregation, visualization, notifications, etc.<br />
* Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc.<br />
* Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding<br />
* Scalability - supporting big data, or handling distributed builds or tests<br />
* Ease of use - making testing and test development accessible to non-QA folk (developers, end-users)<br />
<br />
<br />
= Pre-meeting work =<br />
Here's what Tim thinks we need to do:<br />
* Refine the test stack reference diagram<br />
* Organize committees for specific APIs<br />
** designate a head for each committee<br />
<br />
= Sponsors =<br />
Sponsorship is being provided by:<br />
* ''(To be announced)''<br />
<br />
= Schedule =<br />
''(To be announced)''<br />
<br />
= Summit Artifacts =<br />
''(To be uploaded after the event)''<br />
<br />
== Key decisions ==<br />
* all systems to make a kcidb client (upload results)<br />
** extend the kcidb schema<br />
** https://github.com/kernelci/kcidb<br />
** priority (over test definition unification work)<br />
* use LTP metadata format as initial standard<br />
** add meta-data convert to kselftest (Tim)<br />
* build a prototype to move boards between LAVA and labgrid (Jan)<br />
* want to collect more information for "Hardware Design for Testing"<br />
** Contributions are Welcome: https://github.com/SmithChart/Designing-for-Automated-Testing/<br />
* get something working for board management layer for poweron/poweroff/serial-console<br />
* continue work on test definition common schema and central repository (Tim)<br />
* add information about hardware you find useful, or have designed please add to [[Board_Farm]]<br />
* focus on meeting at Plumbers next year<br />
* continue using automated-testing@lists.yoctoproject.org<br />
<br />
== Action items from meeting ==<br />
Action items:<br />
* post slides to this page<br />
* send minutes<br />
<br />
== Presentations ==<br />
{| class="wikitable"<br />
|-<br />
! Session !! Presenter !! Slides<br />
|-<br />
| A Guide to CIP Testing || Chris Paterson & Michael Adler || [[File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf|thumb|A Guide to CIP Testing]]<br />
|-<br />
| Update on previous conferences, KernelCI update || Kevin Hilman || [[File:ATS_2019.pdf]]<br />
|-<br />
| Bootloader testing in LAVA || Remi Duraffort || [[File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf|thumb|A Guide to CIP Testing]]<br />
|-<br />
| New Ways Out of the Struggle of Testing Embedded Devices || Chris Fiege || [[File:ATS2019-cfi-embedded-testing_handout.pdf|thumb|Embedded Testing]]<br />
|-<br />
| SLAV: Test Stack Abstraction Layers || Paweł Wieczorek || [[File:Test_stack_abstraction_layers.pdf]]<br />
|-<br />
|}<br />
<br />
== Videos ==</div>Pawieczhttps://elinux.org/index.php?title=Automated_Testing_Summit_2019&diff=503181Automated Testing Summit 20192019-10-31T19:33:58Z<p>Pawiecz: /* Presentations */</p>
<hr />
<div>This is a public planning page for the Automated Testing Summit<br />
<br />
Please see: https://events.linuxfoundation.org/events/ats-2019/<br />
<br />
= 2019 meeting =<br />
Coordinated by: Tim Bird<br />
This is now a full-blow Linux Foundation event, with sponsors, a CFP, program and everything.<br />
<br />
See the [https://events.linuxfoundation.org/events/ats-2019/ Linux Foundation Event page] for details.<br />
<br />
== Mailing list ==<br />
Inter-project discussions on Automated Testing occur on the "Automated Testing" mailing list of the Yocto Project<br />
* See https://lists.yoctoproject.org/listinfo/automated-testing<br />
<br />
= Date and Venue =<br />
* Location: Lyon, France<br />
* Venue: Lyon Convention Center, 50 Quai Charles de Gaulle<br />
* Date: October 31, 2019<br />
* Rooms: Pasteur Lounge and Rhone 3AB<br />
* Time: 9:00 am to 17:15 (5:15 pm)<br />
<br />
** co-located (following) Embedded Linux Conference Europe and Open Source Summit Europe<br />
** ELCE and OSSEU run Oct 28-30<br />
** see https://events.linuxfoundation.org/events/embedded-linux-conference-europe-2019/<br />
<br />
Attendance is by open to the public, and we are projecting the attendee fee to be $250, with this<br />
fee waived for speakers.<br />
<br />
= Summit Pre-planning =<br />
Here is a list of action items to do to prepare for the summit:<br />
* organize program committee (''in progress'')<br />
* get sponsors<br />
** create sponsorship prospectus (''almost done'')<br />
** contact candidate sponsors (''in progress'')<br />
* hold "Call for Proposals"<br />
** need to determine CFP dates (''done'')<br />
* determine agenda<br />
** organize program committee (''in progress'')<br />
* advertise for attendees<br />
* reserve venue, catering, AV rentals (''done'')<br />
* create signage, web site, registration system<br />
<br />
For private planning information, see [http://embeddedlinuxconference.com/cgi-bin/elcwiki.cgi/ATS_Planning the private planning wiki page]<br />
<br />
== Call for Presentations ==<br />
* CFP Open: Tuesday, May 28<br />
* CFP Close: Friday, August 13<br />
* Notifications Out: September 2<br />
* Schedule Out: September 4<br />
* Slides Due: October 24, 2019<br />
* Event Date: October 31, 2019<br />
<br />
Suggested Topics<br />
<br />
* Board Farms and Test Labs - hardware, software, useful practices<br />
* Layers and Interfaces - components of your automated CI loop and protocols between them<br />
* Results Reporting - results aggregation, visualization, notifications, etc.<br />
* Defining standards - propose or discuss conventions for test definitions, results formats, layer interoperability (e.g. protocols), etc.<br />
* Existing Projects - describe your test system or framework and any challenges you overcame or that are still outstanding<br />
* Scalability - supporting big data, or handling distributed builds or tests<br />
* Ease of use - making testing and test development accessible to non-QA folk (developers, end-users)<br />
<br />
<br />
= Pre-meeting work =<br />
Here's what Tim thinks we need to do:<br />
* Refine the test stack reference diagram<br />
* Organize committees for specific APIs<br />
** designate a head for each committee<br />
<br />
= Sponsors =<br />
Sponsorship is being provided by:<br />
* ''(To be announced)''<br />
<br />
= Schedule =<br />
''(To be announced)''<br />
<br />
= Summit Artifacts =<br />
''(To be uploaded after the event)''<br />
<br />
== Key decisions ==<br />
* all systems to make a kcidb client (upload results)<br />
** extend the kcidb schema<br />
** https://github.com/kernelci/kcidb<br />
** priority (over test definition unification work)<br />
* use LTP metadata format as initial standard<br />
** add meta-data convert to kselftest (Tim)<br />
* build a prototype to move boards between LAVA and labgrid (Jan)<br />
* want to collect more information for "Hardware Design for Testing"<br />
** Contributions are Welcome: https://github.com/SmithChart/Designing-for-Automated-Testing/<br />
* get something working for board management layer for poweron/poweroff/serial-console<br />
* continue work on test definition common schema and central repository (Tim)<br />
* add information about hardware you find useful, or have designed please add to [[Board_Farm]]<br />
* focus on meeting at Plumbers next year<br />
* continue using automated-testing@lists.yoctoproject.org<br />
<br />
== Action items from meeting ==<br />
Action items:<br />
* post slides to this page<br />
* send minutes<br />
<br />
== Presentations ==<br />
{| class="wikitable"<br />
|-<br />
! Session !! Presenter !! Slides<br />
|-<br />
| A Guide to CIP Testing || Chris Paterson & Michael Adler || [[File:2019-10-31 ATS-Lyon CIP Testing Overview v3.pdf|thumb|A Guide to CIP Testing]]<br />
|-<br />
| Update on previous conferences, KernelCI update || Kevin Hilman || [[File:ATS_2019.pdf]]<br />
|-<br />
| Bootloader testing in LAVA || Remi Duraffort || [[File:ATS-Lyon-Bootloader-testing-in-LAVA.pdf|thumb|A Guide to CIP Testing]]<br />
|-<br />
| New Ways Out of the Struggle of Testing Embedded Devices || Chris Fiege || [[File:ATS2019-cfi-embedded-testing_handout.pdf|thumb|Embedded Testing]]<br />
|-<br />
| SLAV: Test Stack Abstraction Layers || Paweł Wieczorek || [[Test_stack_abstraction_layers.pdf]]<br />
|-<br />
|}<br />
<br />
== Videos ==</div>Pawieczhttps://elinux.org/index.php?title=File:Test_stack_abstraction_layers.pdf&diff=503176File:Test stack abstraction layers.pdf2019-10-31T19:33:38Z<p>Pawiecz: Talk from the Automated Testing Summit 2019 in Lyon by Paweł Wieczorek
This session will cover the benefits and consequences of having custom hardware in a testing laboratory as well as using generic, off-the-shelf parts. Paweł will discuss API desi...</p>
<hr />
<div>== Summary ==<br />
Talk from the Automated Testing Summit 2019 in Lyon by Paweł Wieczorek<br />
<br />
This session will cover the benefits and consequences of having custom hardware in a testing laboratory as well as using generic, off-the-shelf parts. Paweł will discuss API design decisions for testing laboratories focusing on SLAV test lab stack case. The presentation will also include lessons learned from shifting to a user-centric perspective and possible outcome of such a process.</div>Pawiecz