Difference between revisions of "Buildroot:ReproducibleBuilds"

From eLinux.org
Jump to: navigation, search
(22 June 2019: Journal Entry)
(Meetings: 2019-W26)
Line 8: Line 8:
 
Weekly meetings on [https://appear.in/buildroot appear.in/buildroot] every Tuesdays at 14:30 UTC.
 
Weekly meetings on [https://appear.in/buildroot appear.in/buildroot] every Tuesdays at 14:30 UTC.
  
 +
* 2019-W26
 +
** Fix reproducible issues
 +
*** tar problem
 +
*** GZIP environment variable
 +
*** Nothing has been done yet.
 +
*** Priority for coming week.
 +
** autobuilder scripts
 +
*** Changes requested on Builder class series. Almost done.
 +
**** This will conflict with a patch from Thomas.
 +
**** Include Thomas's patches in the Builder class series.
 +
*** No changes requested on the reason patch, but it depends on the Builder series.
 +
**** Put it in a series together with the Builder class, so it can be applied together.
 +
** Atharva wrote in his journal that he was "waiting for feedback"
 +
*** He was not actually waiting, but working on the separate/different output directories.
 +
*** This should have been mentioned in the journal, that's what it is for.
 +
** Environment variable for KCONFIG_PROBABILITY.
 +
*** Atharva just made a local branch that fixes KCONFIG_PROBABILITY to 1.
 +
*** Good enough.
 +
*** It could be useful to have an option that makes genrandconfig behave predictably, cfr. KCONFIG_SEED
 +
**** Something that Atharva can do when there's nothing else to do.
 +
** ELC-E
 +
*** Atharva will submit an abstract for a talk at ELC-E, deadline is June 30.
 +
*** Draft should be shared with Yann and Arnout soon so they can review.
 +
*** Atharva will apply for travel funding at ELC-E as well, perhaps the Buildroot Association can contribute funding as well.
 
* 2019-W25
 
* 2019-W25
 
** autobuilder scripts:
 
** autobuilder scripts:

Revision as of 08:03, 25 June 2019

As part of a Google Summer of Code project, Atharva Lele works on reproducible builds.

Away time

Arnout is away on: 10/6; some time in July; 15-19/8.

Meetings

Weekly meetings on appear.in/buildroot every Tuesdays at 14:30 UTC.

  • 2019-W26
    • Fix reproducible issues
      • tar problem
      • GZIP environment variable
      • Nothing has been done yet.
      • Priority for coming week.
    • autobuilder scripts
      • Changes requested on Builder class series. Almost done.
        • This will conflict with a patch from Thomas.
        • Include Thomas's patches in the Builder class series.
      • No changes requested on the reason patch, but it depends on the Builder series.
        • Put it in a series together with the Builder class, so it can be applied together.
    • Atharva wrote in his journal that he was "waiting for feedback"
      • He was not actually waiting, but working on the separate/different output directories.
      • This should have been mentioned in the journal, that's what it is for.
    • Environment variable for KCONFIG_PROBABILITY.
      • Atharva just made a local branch that fixes KCONFIG_PROBABILITY to 1.
      • Good enough.
      • It could be useful to have an option that makes genrandconfig behave predictably, cfr. KCONFIG_SEED
        • Something that Atharva can do when there's nothing else to do.
    • ELC-E
      • Atharva will submit an abstract for a talk at ELC-E, deadline is June 30.
      • Draft should be shared with Yann and Arnout soon so they can review.
      • Atharva will apply for travel funding at ELC-E as well, perhaps the Buildroot Association can contribute funding as well.
  • 2019-W25
    • autobuilder scripts:
      • first patches to autobuilder script applied and deployed; first results trickling in! Wee! :-)
    • reason for failure is still unknown'
      • add a reason file in the result dir,
      • tweak the PHP code to report that if available, and fallback to the curent behaviour if missing
    • we need an autobuilder instance that has diffoscope installed, to get more intersting results
      • Yann wil look at doing that in his instance (or spawn another one)
    • Builder class
      • initial big ptch for proof of concept pushed; comments from Arnout
      • introducing the class really needs a big patch (bonus point if it can be made mechanical):
        • move functions in the class,
        • add the self parameter
        • call functions from foo() to self.foo()
        • instanciate the object
      • then migrate variables one by one from kwargs to object members, to stop duplicating code
    • Atharva will shift his working hours ahead after college starts. Will work from around 12:00Z. College will start end of June/first week of July,
  • 2019-W24
    • reprotest
      • When building under reprotest, building tar fails: https://pastebin.com/2UbQSuu4 - maybe some issue with uid mapping?
      • For now, leave it alone, we can revisit later.
      • Reprotest already does two builds and compares the results. It's pretty invasive in terms of what it expects from the environment.
      • Probably better to use reprotest as inspiration and do the same from autobuild-run.
    • disorderfs
      • It uses a FUSE filesystem to randomize the order in which files are listed.
      • Also didn't succeed, autobuild-run fails because the output directory isn't removed.
      • Also the second build failed, because a file was changed while it was being tarred.
      • Cfr. https://pastebin.com/pzGfF1c9
      • For now, leave it alone, we can revisit later.
    • Next steps: choose between:
      • Continue on reprotest and disorderfs
      • Improve autobuild-run script, e.g. build in two different directories
      • Improve reporting on the autobuild website.
    • For next week:
      • Collect the review feedback which has not been implemented yet
      • Introduce Builder class in autobuild-run
      • Use this to store the output_dir
      • This makes it easy to do two builds with different output_dir
      • In parallel, mark failures as reason=reproducible
        • Add a 'reason' file in the build results
        • Use that in PHP script
  • 2019-W23
    • Initial round-up of autobuild scripts patches
      • Basically, look OK-ish
      • Not bisectable because reverse order
      • Re-spin in correct order, but still split for ease of review
      • Ultimately, to be committed squashed together
    • diffoscope is silent on success, and so is cmp -> diffoscope_result.txt can be used to determine if reason should be set to reproducible.
    • Atharva will evaluate the reprotest and disorderfs projects to see if they can be useful for our reproducible tests.
    • Atharva should add a journal (log) to this wiki page two or three times a week
  • 2019-W22
    • As discussed on IRC, diffoscope only needs to be done if cmp detects differences. However, it doesn't take long anyway, and it *will* report if there is a difference.
    • diffoscope must be done on output/target/ and target/images, but autobuilders don't enable any images. So when doing a reproducible test, a tarball must be generated.
      • Manually try this, to be sure that it also looks inside the generated images.
      • Enable one / all target filesystems to check this manually.
      • Disable BR2_REPRODUCIBLE for this test, so there actually are some differences.
    • diffoscope has a lot of dependencies, we don't want all of these on the autobuilders
      • Try what the output is if the external tools are not installed
      • autobuilder script should fall back on cmp if diffoscope is not installed
    • Start patching autobuilder script to do a reproducible test.
      • Randomly enable BR2_REPRODUCIBLE, e.g. 10% of the times
      • Do the same build a second time. Only variation is time.
      • Run diffoscope on the result.
  • 2019-W21
    • Confirmed that starting from next week, work is full-time on GSoC (end of exams)
    • Review of the Yocto implementation
      • differences: Yocto is a distribution, so has a cache of the output, while buildroot does not
      • SOURCE_DATE_EPOCH and TZ: already done (depends on BR2_REPRODUCIBLE)
    • Doing similar in Buildroot:
      • Do a first build with a successfull config from autobuilders, after enabling BR2_REPRODUCIBLE
      • Then mv $(O)/target to $(O)/target-1; make clean; make
      • And then run diffoscope target-1 target/
    • Identify diffoscope dependencies to run it in autobuilders (eventually)
      • How to save and present the result on autobuilder site?
  • 2019-W20
    • introductions
    • confirm overal actions and planning

Yocto's Implementation

  • Shared State Mechanism: If input metadata hashes are same, outputs are reused. If inputs have changed, tools from Reproducible-Builds to be used. Further development yet to be done.
  • At this stage, binary contents should be same. However file timestamps (due to package managers) may be different.
  • Static Timezone value: Bugzilla
  • Adapted SOURCE_DATE_EPOCH: Bugzilla, Source-Date-Epoch - Reproducible Builds
  • Archives generated with deterministic metadata (using archive tools' arguments)
  • Remove non-deterministic data from rootfs

Diffoscope Dependencies

  • Depends on: python3, PyPI modules: libarchive-c, python-magic
  • External tools requied: Rscript, abootimg, apktool, bsdtar, bzip2, cbfstool, cd-iccdump, cmp, compare, convert, db_dump, diff, docx2txt, dumpxsb, enjarify, fdtdump, ffprobe, getfacl, ghc, gifbuild, gpg, gzip, identify, img2txt, isoinfo, javap, js-beautify, lipo, llvm-bcanalyzer, llvm-dis, lsattr, lz4, msgunfmt, nm, objcopy, objdump, ocamlobjinfo, odt2txt, oggDump, otool, pdftotext, pedump, pgpdump, ppudump, procyon, ps2ascii, readelf, showttf, sng, sqlite3, ssconvert, ssh-keygen, stat, tcpdump, unsquashfs, wasm2wat, xxd, xz, zipinfo, zipnote
  • This has tools used to compare a lot of file formats that probably aren't generated (like android APKs, Windows/Mac executables) in a Buildroot run. We can exclude those.
  • APT packages (available in Ubuntu, Debian): abootimg, acl, apktool, binutils-multiarch, bzip2, caca-utils, colord, coreutils, db-util, default-jdk-headless | default-jdk | java-sdk, device-tree-compiler, diffutils, docx2txt, e2fsprogs, enjarify, ffmpeg, fontforge-extras, fp-utils, genisoimage, gettext, ghc, ghostscript, giflib-tools, gnumeric, gnupg, gzip, imagemagick, jsbeautifier, libarchive-tools, llvm, lz4 | liblz4-tool, mono-utils, ocaml-nox, odt2txt, oggvideotools, openssh-client, pgpdump, poppler-utils, procyon-decompiler, r-base-core, sng, sqlite3, squashfs-tools, tcpdump, unzip, xmlbeans, xxd | vim-common, xz-utils, zip

Sample Diffoscope Output

  • Minimal config build (make defconfig; make). Will run diffoscope on a build from Autobuilder config tomorrow.
  • Builds run about 10 minutes apart.
  • Moved first build to target, and rerun. Then run diffoscope target-1 target > diff.txt
  • diffoscope log: https://paste.ubuntu.com/p/VpMbW4qQQP/
  • Except for a time record in the busybox binary, all other differences seem to be only timestamps of file generation.

Planning

  • Week 20: study how yocto does it
  • Week 21: ...
  • Week 22: do two builds in autobuild-run script
  • Week 23: revisit patches to autobuild-run
  • Week 24: autobuild-run: different output directories; report with reason=reproducible
  • Week 25: improve how reproducible results are shown on http://autobuild.buildroot.org
  • Week 26: Fix the reproducible issues found until now
  • Week 27: Further extend autobuild-run script with more variation
  • Week 28: Tooling to understand where the differences come from (e.g. which package)
  • Week 29: More tooling to analyse differences
  • Week 30: More tooling to analyse differences

Progress Journal

  • 06/06/2019:
    • Setup Gitlab account to track issues and progress
    • Pushed dev branch to Gitlab, as well as re-spun commits and pushed
  • 07/06/2019:
    • Submitted v2 patches to mailing list
    • Submitted patch to enable BR2_REPRODUCIBLE
    • Started evaluating reprotest and its working
  • 08/06/2019:
  • 09/06/2019 - 11/06/2019:
    • Finished v3 patches, sent to mailing list
    • Evaluated reprotest & disorderfs, discussed with Arnout during meeting
    • Less work done than possible due to lack of planning, now have planned till week 30
  • 12/06/2019:
    • Explored PHP components of Autobuilder website to identify what and how to modify
    • Learned basic syntax and working of PHP since I've never worked with it before
    • Brushed up on using classes in python because it has been a while since I worked using classes
    • Now it's easier for me to implement the Builder class
  • 13/06/2019:
    • Worked on implementing and transitioning to Builder class
    • Will push code to Gitlab tomorrow after removing errors
  • 14/06/2019:
    • First step of transitioning to Builder class done, pushed to Gitlab
    • Testing it thoroughly before sending to mailing list
    • Created and sent reproducible-v4 to mailing list, Gitlab
  • 15/06/2019 - 18/06/2019:
    • Worked on implementing builder class
    • Pushed proof of concept to Gitlab
    • Received feedback and now revising my patches
  • 20/06/2019:
    • Finished work on Builder class, waiting for feedback from Mentors
  • 21/06/2019:
    • Going to work on reason file in autobuild-run
    • Sent Builder class to mailist list for feedback
  • 22/06/2019:
    • Worked on implementing reason-file, pushed to Gitlab
    • Waiting for merge of builder-class to send to mailing list
    • Waiting for feedback from Mentors

Code

Atharva Lele's on-going work to make the autobuilder scripts reproducible-aware (check the reproducible-vN branches).

GSoC Proposal

The proposal PDF can be found here: PDF