Difference between revisions of "Buildbot survey response"

From eLinux.org
Jump to: navigation, search
(formating)
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
= Buildbot survey response =
 
= Buildbot survey response =
Buildbot survey response provided by Alice Ferrazzi
+
Buildbot survey response provided by Alice Ferrazzi, updates Krzysztof Kozlowski
  
  
Line 34: Line 34:
  
 
Does your test system:
 
Does your test system:
* provide a set of existing tests?
+
* provide a set of existing tests? '''no'''
 
** if so, how many?   
 
** if so, how many?   
  
Line 46: Line 46:
 
* require a toolchain or build system for the SUT? '''yes'''
 
* require a toolchain or build system for the SUT? '''yes'''
 
* require a toolchain or build system for the test software? '''no'''
 
* require a toolchain or build system for the test software? '''no'''
* come with pre-built toolchains? '''yes (part of Gentoo)'''
+
* come with pre-built toolchains? '''no'''
* store the build artifacts for generated software?  
+
* store the build artifacts for generated software? '''yes'''
** in what format is the build metadata stored (e.g. json)?  
+
** in what format is the build metadata stored (e.g. json)? '''internal'''
** are the build artifacts stored as raw files or in a database?
+
** are the build artifacts stored as raw files or in a database? '''raw files'''
 
*** if a database, what database? '''everything supported by sqlalchemy'''
 
*** if a database, what database? '''everything supported by sqlalchemy'''
  
Line 56: Line 56:
 
* check that dependencies are met before a test is run?  
 
* check that dependencies are met before a test is run?  
 
* schedule the test for the DUT? '''yes'''
 
* schedule the test for the DUT? '''yes'''
** select an appropriate individual DUT based on SUT or test attributes?  
+
** select an appropriate individual DUT based on SUT or test attributes? '''yes''' (can be configured)
 
** reserve the DUT? '''yes'''
 
** reserve the DUT? '''yes'''
 
** release the DUT? '''yes'''
 
** release the DUT? '''yes'''
* install the software under test to the DUT?  
+
* install the software under test to the DUT? '''yes/no''' (can be configured)
* install required packages before a test is run? '''yes/only kernel packages'''
+
* install required packages before a test is run? '''yes/no''' (can be configured)
* require particular bootloader on the DUT? (e.g. grub, uboot, etc.) '''Working with qemu as now, real hardware is still in progress'''
+
* require particular bootloader on the DUT? (e.g. grub, uboot, etc.) '''no''' (depends on the setups, e.g. whether with QEMU or, real hardware)
* deploy the test program to the DUT?  
+
* deploy the test program to the DUT? '''yes/no''' (can be configured)
* prepare the test environment on the DUT?
+
* prepare the test environment on the DUT? '''yes/no''' (can be configured)
* start a monitor (another process to collect data) on the DUT?
+
* start a monitor (another process to collect data) on the DUT? '''yes/no''' (can be configured)
* start a monitor on external equipment?
+
* start a monitor on external equipment? '''yes/no''' (can be configured)
 
* initiate the test on the DUT? '''yes'''
 
* initiate the test on the DUT? '''yes'''
 
* clean up the test environment on the DUT? '''yes'''
 
* clean up the test environment on the DUT? '''yes'''
Line 71: Line 71:
 
==== DUT control ====
 
==== DUT control ====
 
Does your test system:
 
Does your test system:
* store board configuration data?  
+
* store board configuration data? '''no'''
 
** in what format?  
 
** in what format?  
* store external equipment configuration data?
+
* store external equipment configuration data? '''no'''
 
** in what format?
 
** in what format?
* power cycle the DUT?
+
* power cycle the DUT? '''no'''
* monitor the power usage during a run?
+
* monitor the power usage during a run? '''no'''
* gather a kernel trace during a run?
+
* gather a kernel trace during a run? '''no'''
* claim other hardware resources or machines (other than the DUT) for use during a test?
+
* claim other hardware resources or machines (other than the DUT) for use during a test? '''no'''
* reserve a board for interactive use (ie remove it from automated testing)?
+
* reserve a board for interactive use (ie remove it from automated testing)? '''no'''
* provide a web-based control interface for the lab? '''Buildbot web interface'''
+
* provide a web-based control interface for the lab? '''Buildbot web interface''' (limited, only stop test, pause the worker)
 
* provide a CLI control interface for the lab? '''yes, Buildbot-worker'''
 
* provide a CLI control interface for the lab? '''yes, Buildbot-worker'''
 
'''Gentoo kernel CI is working with Qemu container, real hardware is still in progress.'''
 
  
 
==== Run artifact handling ====
 
==== Run artifact handling ====
Line 96: Line 94:
 
* evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? '''yes'''
 
* evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? '''yes'''
 
* do you have a common set of result names: (e.g. pass, fail, skip, etc.) '''yes'''
 
* do you have a common set of result names: (e.g. pass, fail, skip, etc.) '''yes'''
** if so, what are they? '''Gentoo Kernel CI is using [http://docs.buildbot.net/latest/developer/results.html Buildbot test results]'''
+
** if so, what are they? ''' [http://docs.buildbot.net/latest/developer/results.html Buildbot test results]'''
  
 
* How is run data collected from the DUT?
 
* How is run data collected from the DUT?
Line 103: Line 101:
  
 
* How is run data collected from external equipment?
 
* How is run data collected from external equipment?
* Is external equipment data parsed?  
+
* Is external equipment data parsed?
  
 
==== User interface ====
 
==== User interface ====
Line 122: Line 120:
  
 
* does your test system have a CLI control tool? '''yes'''
 
* does your test system have a CLI control tool? '''yes'''
** what is it called? '''buildbot"
+
** what is it called? '''buildbot'''
  
 
==== Languages: ====
 
==== Languages: ====
Line 133: Line 131:
 
* see the results of recent tests? '''yes'''
 
* see the results of recent tests? '''yes'''
 
* set the pass criteria for a test? '''yes'''
 
* set the pass criteria for a test? '''yes'''
** set the threshold value for a benchmark test? '''not sure'''
+
** set the threshold value for a benchmark test? '''yes/no''' (can be configured)
** set the list of testcase results to ignore? '''not sure'''
+
** set the list of testcase results to ignore? '''yes/no''' (can be configured)
 
* provide a rating for a test? (e.g. give it 4 stars out of 5) '''no'''
 
* provide a rating for a test? (e.g. give it 4 stars out of 5) '''no'''
 
* customize a test? '''yes'''
 
* customize a test? '''yes'''
Line 153: Line 151:
 
** If so, what?  
 
** If so, what?  
  
* require agent software on the DUT?  '''yes'''
+
* require agent software on the DUT?  '''yes/no'''
** If so, what agent? '''buildbot-worker'''
+
** If so, what agent? '''buildbot-worker or any access (UART, SSH) if buildbot-worker is not running on DUT'''  
 
* is there optional agent software or libraries for the DUT? '''no'''
 
* is there optional agent software or libraries for the DUT? '''no'''
 
* require external hardware in your labs? '''no'''
 
* require external hardware in your labs? '''no'''
Line 166: Line 164:
 
* What is the nature of the APIs you currently use?
 
* What is the nature of the APIs you currently use?
 
Are they:
 
Are they:
** RPCs?
+
** RPCs? '''No'''
** Unix-style?  
+
** Unix-style? '''No'''
** compiled libraries?
+
** compiled libraries? '''No'''
** interpreter modules or libraries?
+
** interpreter modules or libraries? '''yes (python API)'''
** web-based APIs?
+
** web-based APIs? '''yes (REST API and websocket event queue system)'''
** something else?
+
** something else? '''No'''
  
 
==== Relationship to other software: ====
 
==== Relationship to other software: ====
* what major components does your test framework use? '''Buildbot'''
+
* what major components does your test framework use? '''Buildbot''', '''Python'''
* does your test framework interoperate with other test frameworks or software? '''yes'''
 
** which ones? '''Buildbot'''
 
  
 
== Overview ==
 
== Overview ==

Latest revision as of 05:57, 18 October 2018

Buildbot survey response

Buildbot survey response provided by Alice Ferrazzi, updates Krzysztof Kozlowski


Survey Questions

  • What is the name of your test framework? Buildbot

Does your test framework:

source code access

  • access source code repositories for the software under test? yes
  • access source code repositories for the test software? yes
  • include the source for the test software? yes
  • provide interfaces for developers to perform code reviews? yes
  • detect that the software under test has a new version? yes
    • if so, how? git polling
  • detect that the test software has a new version? experimental

test definitions

Does your test system:

  • have a test definition repository? yes
    • if so, what data format or language is used? Python/Bash

Does your test definition include:

  • source code (or source code location)? yes. See https://github.com/buildbot/buildbot/
  • dependency information? 'yes
  • execution instructions? yes
  • command line variants? yes
  • environment variants? yes
  • setup instructions? yes
  • cleanup instructions? yes
    • if anything else, please describe:


Does your test system:

  • provide a set of existing tests? no
    • if so, how many?

build management

Does your test system:

  • build the software under test (e.g. the kernel)? yes
  • build the test software? yes
  • build other software (such as the distro, libraries, firmware)? yes
  • support cross-compilation? 'yes
  • require a toolchain or build system for the SUT? yes
  • require a toolchain or build system for the test software? no
  • come with pre-built toolchains? no
  • store the build artifacts for generated software? yes
    • in what format is the build metadata stored (e.g. json)? internal
    • are the build artifacts stored as raw files or in a database? raw files
      • if a database, what database? everything supported by sqlalchemy

Test scheduling/management

Does your test system:

  • check that dependencies are met before a test is run?
  • schedule the test for the DUT? yes
    • select an appropriate individual DUT based on SUT or test attributes? yes (can be configured)
    • reserve the DUT? yes
    • release the DUT? yes
  • install the software under test to the DUT? yes/no (can be configured)
  • install required packages before a test is run? yes/no (can be configured)
  • require particular bootloader on the DUT? (e.g. grub, uboot, etc.) no (depends on the setups, e.g. whether with QEMU or, real hardware)
  • deploy the test program to the DUT? yes/no (can be configured)
  • prepare the test environment on the DUT? yes/no (can be configured)
  • start a monitor (another process to collect data) on the DUT? yes/no (can be configured)
  • start a monitor on external equipment? yes/no (can be configured)
  • initiate the test on the DUT? yes
  • clean up the test environment on the DUT? yes

DUT control

Does your test system:

  • store board configuration data? no
    • in what format?
  • store external equipment configuration data? no
    • in what format?
  • power cycle the DUT? no
  • monitor the power usage during a run? no
  • gather a kernel trace during a run? no
  • claim other hardware resources or machines (other than the DUT) for use during a test? no
  • reserve a board for interactive use (ie remove it from automated testing)? no
  • provide a web-based control interface for the lab? Buildbot web interface (limited, only stop test, pause the worker)
  • provide a CLI control interface for the lab? yes, Buildbot-worker

Run artifact handling

Does your test system:

  • store run artifacts yes
    • in what format? raw files
  • put the run meta-data in a database? yes
    • if so, which database? anything supported by sqlalchemy
  • parse the test logs for results? yes
  • convert data from test logs into a unified format? no
    • if so, what is the format?
  • evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? yes
  • do you have a common set of result names: (e.g. pass, fail, skip, etc.) yes
  • How is run data collected from the DUT?

Buildbot-worker/Buildbot master

    • e.g. by pushing from the DUT, or pulling from a server?
  • How is run data collected from external equipment?
  • Is external equipment data parsed?

User interface

Does your test system:

  • have a visualization system? yes
  • show build artifacts to users? yes
  • show run artifacts to users? yes
  • do you have a common set of result colors? yes
  • generate reports for test runs? yes
  • notify users of test results by e-mail? yes
  • can you query (aggregate and filter) the build meta-data? yes
  • can you query (aggregate and filter) the run meta-data? yes
  • what language or data format is used for online results presentation? custom html/javascript app
  • what language or data format is used for reports? Depend from the report tool used
  • does your test system have a CLI control tool? yes
    • what is it called? buildbot

Languages:

  • what is the base language of your test framework core? Python
  • What languages or data formats is the user required to learn? Python


Can a user do the following with your test framework:

  • Can a user manually request that a test be executed? yes
  • see the results of recent tests? yes
  • set the pass criteria for a test? yes
    • set the threshold value for a benchmark test? yes/no (can be configured)
    • set the list of testcase results to ignore? yes/no (can be configured)
  • provide a rating for a test? (e.g. give it 4 stars out of 5) no
  • customize a test? yes
    • alter the command line for the test program? no
    • alter the environment of the test program? yes
    • specify to skip a testcase? yes
    • set a new expected value for a test? yes
    • edit the test program source? yes
  • customize the notification criteria? yes
    • customize the notification mechanism (eg. e-mail, text) yes
  • generate a custom report for a set of runs? yes
  • save the report parameters to generate the same report in the future? yes

Requirements

Does your test framework:

  • require minimum software on the DUT?
  • require minimum hardware on the DUT (e.g. memory)
    • If so, what?
  • require agent software on the DUT? yes/no
    • If so, what agent? buildbot-worker or any access (UART, SSH) if buildbot-worker is not running on DUT
  • is there optional agent software or libraries for the DUT? no
  • require external hardware in your labs? no

APIS

Does your test framework:

  • use existing APIs or data formats to interact within itself, or with 3rd-party modules? yes [1]
  • have a published API for any of its sub-module interactions (any of the lines in the diagram)? yes
    • Please provide a link or links to the APIs? [2]
  • What is the nature of the APIs you currently use?

Are they:

    • RPCs? No
    • Unix-style? No
    • compiled libraries? No
    • interpreter modules or libraries? yes (python API)
    • web-based APIs? yes (REST API and websocket event queue system)
    • something else? No

Relationship to other software:

  • what major components does your test framework use? Buildbot, Python

Overview

Please list your major components here:

  • Build/Test Management (Buildbot)
  • Test Scheduling (Buildbot)
  • Results Management (Buildbot)
  • View/Interact (Buildbot)

Additional Data