Difference between revisions of "Buildbot survey response"

From eLinux.org
Jump to: navigation, search
(Update Buildbot (and remove Gentoo specifics as they are not part of Buildbot), Buildbot is generic CI framework so some questions depend how people configure it by themself (marked as "can be configured"))
Line 34: Line 34:
  
 
Does your test system:
 
Does your test system:
* provide a set of existing tests?
+
* provide a set of existing tests? '''no'''
 
** if so, how many?   
 
** if so, how many?   
  
Line 46: Line 46:
 
* require a toolchain or build system for the SUT? '''yes'''
 
* require a toolchain or build system for the SUT? '''yes'''
 
* require a toolchain or build system for the test software? '''no'''
 
* require a toolchain or build system for the test software? '''no'''
* come with pre-built toolchains? '''yes (part of Gentoo)'''
+
* come with pre-built toolchains? '''no'''
* store the build artifacts for generated software?  
+
* store the build artifacts for generated software? '''yes'''
** in what format is the build metadata stored (e.g. json)?  
+
** in what format is the build metadata stored (e.g. json)? '''internal'''
** are the build artifacts stored as raw files or in a database?
+
** are the build artifacts stored as raw files or in a database? '''raw files'''
 
*** if a database, what database? '''everything supported by sqlalchemy'''
 
*** if a database, what database? '''everything supported by sqlalchemy'''
  
Line 56: Line 56:
 
* check that dependencies are met before a test is run?  
 
* check that dependencies are met before a test is run?  
 
* schedule the test for the DUT? '''yes'''
 
* schedule the test for the DUT? '''yes'''
** select an appropriate individual DUT based on SUT or test attributes?  
+
** select an appropriate individual DUT based on SUT or test attributes? '''yes''' (can be configured)
 
** reserve the DUT? '''yes'''
 
** reserve the DUT? '''yes'''
 
** release the DUT? '''yes'''
 
** release the DUT? '''yes'''
* install the software under test to the DUT?  
+
* install the software under test to the DUT? '''yes/no''' (can be configured)
* install required packages before a test is run? '''yes/only kernel packages'''
+
* install required packages before a test is run? '''yes/no''' (can be configured)
* require particular bootloader on the DUT? (e.g. grub, uboot, etc.) '''Working with qemu as now, real hardware is still in progress'''
+
* require particular bootloader on the DUT? (e.g. grub, uboot, etc.) '''no''' (setups with QEMU, real hardware)
* deploy the test program to the DUT?  
+
* deploy the test program to the DUT? '''yes/no''' (can be configured)
* prepare the test environment on the DUT?
+
* prepare the test environment on the DUT? '''yes/no''' (can be configured)
* start a monitor (another process to collect data) on the DUT?
+
* start a monitor (another process to collect data) on the DUT? '''yes/no''' (can be configured)
* start a monitor on external equipment?
+
* start a monitor on external equipment? '''yes/no''' (can be configured)
 
* initiate the test on the DUT? '''yes'''
 
* initiate the test on the DUT? '''yes'''
 
* clean up the test environment on the DUT? '''yes'''
 
* clean up the test environment on the DUT? '''yes'''
Line 71: Line 71:
 
==== DUT control ====
 
==== DUT control ====
 
Does your test system:
 
Does your test system:
* store board configuration data?  
+
* store board configuration data? '''no'''
 
** in what format?  
 
** in what format?  
* store external equipment configuration data?
+
* store external equipment configuration data? '''no'''
 
** in what format?
 
** in what format?
* power cycle the DUT?
+
* power cycle the DUT? '''no'''
* monitor the power usage during a run?
+
* monitor the power usage during a run? '''no'''
* gather a kernel trace during a run?
+
* gather a kernel trace during a run? '''no'''
* claim other hardware resources or machines (other than the DUT) for use during a test?
+
* claim other hardware resources or machines (other than the DUT) for use during a test? '''no'''
* reserve a board for interactive use (ie remove it from automated testing)?
+
* reserve a board for interactive use (ie remove it from automated testing)? '''no'''
* provide a web-based control interface for the lab? '''Buildbot web interface'''
+
* provide a web-based control interface for the lab? '''Buildbot web interface''' (limited, only stop test, pause the worker)
 
* provide a CLI control interface for the lab? '''yes, Buildbot-worker'''
 
* provide a CLI control interface for the lab? '''yes, Buildbot-worker'''
  
Line 122: Line 122:
  
 
* does your test system have a CLI control tool? '''yes'''
 
* does your test system have a CLI control tool? '''yes'''
** what is it called? '''buildbot"
+
** what is it called? '''buildbot'''
  
 
==== Languages: ====
 
==== Languages: ====
Line 133: Line 133:
 
* see the results of recent tests? '''yes'''
 
* see the results of recent tests? '''yes'''
 
* set the pass criteria for a test? '''yes'''
 
* set the pass criteria for a test? '''yes'''
** set the threshold value for a benchmark test? '''not sure'''
+
** set the threshold value for a benchmark test? '''yes/no''' (can be configured)
** set the list of testcase results to ignore? '''not sure'''
+
** set the list of testcase results to ignore? '''yes/no''' (can be configured)
 
* provide a rating for a test? (e.g. give it 4 stars out of 5) '''no'''
 
* provide a rating for a test? (e.g. give it 4 stars out of 5) '''no'''
 
* customize a test? '''yes'''
 
* customize a test? '''yes'''
Line 153: Line 153:
 
** If so, what?  
 
** If so, what?  
  
* require agent software on the DUT?  '''yes'''
+
* require agent software on the DUT?  '''yes/no'''
** If so, what agent? '''buildbot-worker'''
+
** If so, what agent? '''buildbot-worker or any access (UART, SSH) if buildbot-worker is not running on DUT'''  
 
* is there optional agent software or libraries for the DUT? '''no'''
 
* is there optional agent software or libraries for the DUT? '''no'''
 
* require external hardware in your labs? '''no'''
 
* require external hardware in your labs? '''no'''
Line 174: Line 174:
  
 
==== Relationship to other software: ====
 
==== Relationship to other software: ====
* what major components does your test framework use? '''Buildbot'''
+
* what major components does your test framework use? '''Buildbot''', '''Python'''
 
* does your test framework interoperate with other test frameworks or software? '''yes'''
 
* does your test framework interoperate with other test frameworks or software? '''yes'''
 
** which ones? '''Buildbot'''
 
** which ones? '''Buildbot'''

Revision as of 23:08, 17 October 2018

Buildbot survey response

Buildbot survey response provided by Alice Ferrazzi


Survey Questions

  • What is the name of your test framework? Buildbot

Does your test framework:

source code access

  • access source code repositories for the software under test? yes
  • access source code repositories for the test software? yes
  • include the source for the test software? yes
  • provide interfaces for developers to perform code reviews? yes
  • detect that the software under test has a new version? yes
    • if so, how? git polling
  • detect that the test software has a new version? experimental

test definitions

Does your test system:

  • have a test definition repository? yes
    • if so, what data format or language is used? Python/Bash

Does your test definition include:

  • source code (or source code location)? yes. See https://github.com/buildbot/buildbot/
  • dependency information? 'yes
  • execution instructions? yes
  • command line variants? yes
  • environment variants? yes
  • setup instructions? yes
  • cleanup instructions? yes
    • if anything else, please describe:


Does your test system:

  • provide a set of existing tests? no
    • if so, how many?

build management

Does your test system:

  • build the software under test (e.g. the kernel)? yes
  • build the test software? yes
  • build other software (such as the distro, libraries, firmware)? yes
  • support cross-compilation? 'yes
  • require a toolchain or build system for the SUT? yes
  • require a toolchain or build system for the test software? no
  • come with pre-built toolchains? no
  • store the build artifacts for generated software? yes
    • in what format is the build metadata stored (e.g. json)? internal
    • are the build artifacts stored as raw files or in a database? raw files
      • if a database, what database? everything supported by sqlalchemy

Test scheduling/management

Does your test system:

  • check that dependencies are met before a test is run?
  • schedule the test for the DUT? yes
    • select an appropriate individual DUT based on SUT or test attributes? yes (can be configured)
    • reserve the DUT? yes
    • release the DUT? yes
  • install the software under test to the DUT? yes/no (can be configured)
  • install required packages before a test is run? yes/no (can be configured)
  • require particular bootloader on the DUT? (e.g. grub, uboot, etc.) no (setups with QEMU, real hardware)
  • deploy the test program to the DUT? yes/no (can be configured)
  • prepare the test environment on the DUT? yes/no (can be configured)
  • start a monitor (another process to collect data) on the DUT? yes/no (can be configured)
  • start a monitor on external equipment? yes/no (can be configured)
  • initiate the test on the DUT? yes
  • clean up the test environment on the DUT? yes

DUT control

Does your test system:

  • store board configuration data? no
    • in what format?
  • store external equipment configuration data? no
    • in what format?
  • power cycle the DUT? no
  • monitor the power usage during a run? no
  • gather a kernel trace during a run? no
  • claim other hardware resources or machines (other than the DUT) for use during a test? no
  • reserve a board for interactive use (ie remove it from automated testing)? no
  • provide a web-based control interface for the lab? Buildbot web interface (limited, only stop test, pause the worker)
  • provide a CLI control interface for the lab? yes, Buildbot-worker

Gentoo kernel CI is working with Qemu container, real hardware is still in progress.

Run artifact handling

Does your test system:

  • store run artifacts yes
    • in what format? raw files
  • put the run meta-data in a database? yes
    • if so, which database? anything supported by sqlalchemy
  • parse the test logs for results? yes
  • convert data from test logs into a unified format? no
    • if so, what is the format?
  • evaluate pass criteria for a test (e.g. ignored results, counts or thresholds)? yes
  • do you have a common set of result names: (e.g. pass, fail, skip, etc.) yes
  • How is run data collected from the DUT?

Buildbot-worker/Buildbot master

    • e.g. by pushing from the DUT, or pulling from a server?
  • How is run data collected from external equipment?
  • Is external equipment data parsed?

User interface

Does your test system:

  • have a visualization system? yes
  • show build artifacts to users? yes
  • show run artifacts to users? yes
  • do you have a common set of result colors? yes
  • generate reports for test runs? yes
  • notify users of test results by e-mail? yes
  • can you query (aggregate and filter) the build meta-data? yes
  • can you query (aggregate and filter) the run meta-data? yes
  • what language or data format is used for online results presentation? custom html/javascript app
  • what language or data format is used for reports? Depend from the report tool used
  • does your test system have a CLI control tool? yes
    • what is it called? buildbot

Languages:

  • what is the base language of your test framework core? Python
  • What languages or data formats is the user required to learn? Python


Can a user do the following with your test framework:

  • Can a user manually request that a test be executed? yes
  • see the results of recent tests? yes
  • set the pass criteria for a test? yes
    • set the threshold value for a benchmark test? yes/no (can be configured)
    • set the list of testcase results to ignore? yes/no (can be configured)
  • provide a rating for a test? (e.g. give it 4 stars out of 5) no
  • customize a test? yes
    • alter the command line for the test program? no
    • alter the environment of the test program? yes
    • specify to skip a testcase? yes
    • set a new expected value for a test? yes
    • edit the test program source? yes
  • customize the notification criteria? yes
    • customize the notification mechanism (eg. e-mail, text) yes
  • generate a custom report for a set of runs? yes
  • save the report parameters to generate the same report in the future? yes

Requirements

Does your test framework:

  • require minimum software on the DUT?
  • require minimum hardware on the DUT (e.g. memory)
    • If so, what?
  • require agent software on the DUT? yes/no
    • If so, what agent? buildbot-worker or any access (UART, SSH) if buildbot-worker is not running on DUT
  • is there optional agent software or libraries for the DUT? no
  • require external hardware in your labs? no

APIS

Does your test framework:

  • use existing APIs or data formats to interact within itself, or with 3rd-party modules? yes [1]
  • have a published API for any of its sub-module interactions (any of the lines in the diagram)? yes
    • Please provide a link or links to the APIs? [2]
  • What is the nature of the APIs you currently use?

Are they:

    • RPCs?
    • Unix-style?
    • compiled libraries?
    • interpreter modules or libraries?
    • web-based APIs?
    • something else?

Relationship to other software:

  • what major components does your test framework use? Buildbot, Python
  • does your test framework interoperate with other test frameworks or software? yes
    • which ones? Buildbot

Overview

Please list your major components here:

  • Build/Test Management (Buildbot)
  • Test Scheduling (Buildbot)
  • Results Management (Buildbot)
  • View/Interact (Buildbot)

Additional Data