Difference between revisions of "Buildroot:ReproducibleBuilds"
(→Meetings: 2019-W26) |
Atharvalele (talk | contribs) (23-27 June 2019: Journal Entries) |
||
Line 204: | Line 204: | ||
** Waiting for merge of builder-class to send to mailing list | ** Waiting for merge of builder-class to send to mailing list | ||
** Waiting for feedback from Mentors | ** Waiting for feedback from Mentors | ||
+ | |||
+ | * 23/06/2019 - 25/06/2019: | ||
+ | ** Working on different output directories | ||
+ | |||
+ | * 26/06/2019: | ||
+ | ** Revising builder-class with feedback | ||
+ | ** Having a bit of trouble with merging Thomas' patches | ||
+ | |||
+ | * 27/06/2019: | ||
+ | ** Worked on ELC-E 2019 talk abstract | ||
+ | ** Worked further on builder-class, testing ongoing.. | ||
== Code == | == Code == |
Revision as of 11:41, 27 June 2019
As part of a Google Summer of Code project, Atharva Lele works on reproducible builds.
Contents
Away time
Arnout is away on: 10/6; some time in July; 15-19/8.
Meetings
Weekly meetings on appear.in/buildroot every Tuesdays at 14:30 UTC.
- 2019-W26
- Fix reproducible issues
- tar problem
- GZIP environment variable
- Nothing has been done yet.
- Priority for coming week.
- autobuilder scripts
- Changes requested on Builder class series. Almost done.
- This will conflict with a patch from Thomas.
- Include Thomas's patches in the Builder class series.
- No changes requested on the reason patch, but it depends on the Builder series.
- Put it in a series together with the Builder class, so it can be applied together.
- Changes requested on Builder class series. Almost done.
- Atharva wrote in his journal that he was "waiting for feedback"
- He was not actually waiting, but working on the separate/different output directories.
- This should have been mentioned in the journal, that's what it is for.
- Environment variable for KCONFIG_PROBABILITY.
- Atharva just made a local branch that fixes KCONFIG_PROBABILITY to 1.
- Good enough.
- It could be useful to have an option that makes genrandconfig behave predictably, cfr. KCONFIG_SEED
- Something that Atharva can do when there's nothing else to do.
- ELC-E
- Atharva will submit an abstract for a talk at ELC-E, deadline is June 30.
- Draft should be shared with Yann and Arnout soon so they can review.
- Atharva will apply for travel funding at ELC-E as well, perhaps the Buildroot Association can contribute funding as well.
- Fix reproducible issues
- 2019-W25
- autobuilder scripts:
- first patches to autobuilder script applied and deployed; first results trickling in! Wee! :-)
- reason for failure is still unknown'
- add a reason file in the result dir,
- tweak the PHP code to report that if available, and fallback to the curent behaviour if missing
- we need an autobuilder instance that has diffoscope installed, to get more intersting results
- Yann wil look at doing that in his instance (or spawn another one)
- Builder class
- initial big ptch for proof of concept pushed; comments from Arnout
- introducing the class really needs a big patch (bonus point if it can be made mechanical):
- move functions in the class,
- add the self parameter
- call functions from foo() to self.foo()
- instanciate the object
- then migrate variables one by one from kwargs to object members, to stop duplicating code
- Atharva will shift his working hours ahead after college starts. Will work from around 12:00Z. College will start end of June/first week of July,
- autobuilder scripts:
- 2019-W24
- reprotest
- When building under reprotest, building tar fails: https://pastebin.com/2UbQSuu4 - maybe some issue with uid mapping?
- For now, leave it alone, we can revisit later.
- Reprotest already does two builds and compares the results. It's pretty invasive in terms of what it expects from the environment.
- Probably better to use reprotest as inspiration and do the same from autobuild-run.
- disorderfs
- It uses a FUSE filesystem to randomize the order in which files are listed.
- Also didn't succeed, autobuild-run fails because the output directory isn't removed.
- Also the second build failed, because a file was changed while it was being tarred.
- Cfr. https://pastebin.com/pzGfF1c9
- For now, leave it alone, we can revisit later.
- Next steps: choose between:
- Continue on reprotest and disorderfs
- Improve autobuild-run script, e.g. build in two different directories
- Improve reporting on the autobuild website.
- For next week:
- Collect the review feedback which has not been implemented yet
- Introduce Builder class in autobuild-run
- Use this to store the output_dir
- This makes it easy to do two builds with different output_dir
- In parallel, mark failures as reason=reproducible
- Add a 'reason' file in the build results
- Use that in PHP script
- reprotest
- 2019-W23
- Initial round-up of autobuild scripts patches
- Basically, look OK-ish
- Not bisectable because reverse order
- Re-spin in correct order, but still split for ease of review
- Ultimately, to be committed squashed together
- diffoscope is silent on success, and so is cmp -> diffoscope_result.txt can be used to determine if reason should be set to reproducible.
- Atharva will evaluate the reprotest and disorderfs projects to see if they can be useful for our reproducible tests.
- Atharva should add a journal (log) to this wiki page two or three times a week
- Initial round-up of autobuild scripts patches
- 2019-W22
- As discussed on IRC, diffoscope only needs to be done if cmp detects differences. However, it doesn't take long anyway, and it *will* report if there is a difference.
- diffoscope must be done on output/target/ and target/images, but autobuilders don't enable any images. So when doing a reproducible test, a tarball must be generated.
- Manually try this, to be sure that it also looks inside the generated images.
- Enable one / all target filesystems to check this manually.
- Disable BR2_REPRODUCIBLE for this test, so there actually are some differences.
- diffoscope has a lot of dependencies, we don't want all of these on the autobuilders
- Try what the output is if the external tools are not installed
- autobuilder script should fall back on cmp if diffoscope is not installed
- Start patching autobuilder script to do a reproducible test.
- Randomly enable BR2_REPRODUCIBLE, e.g. 10% of the times
- Do the same build a second time. Only variation is time.
- Run diffoscope on the result.
- 2019-W21
- Confirmed that starting from next week, work is full-time on GSoC (end of exams)
- Review of the Yocto implementation
- differences: Yocto is a distribution, so has a cache of the output, while buildroot does not
- SOURCE_DATE_EPOCH and TZ: already done (depends on BR2_REPRODUCIBLE)
- Doing similar in Buildroot:
- Do a first build with a successfull config from autobuilders, after enabling BR2_REPRODUCIBLE
- Then mv $(O)/target to $(O)/target-1; make clean; make
- And then run diffoscope target-1 target/
- Identify diffoscope dependencies to run it in autobuilders (eventually)
- How to save and present the result on autobuilder site?
- 2019-W20
- introductions
- confirm overal actions and planning
Yocto's Implementation
- Wiki page: Reproducible Builds
- Shared State Mechanism: If input metadata hashes are same, outputs are reused. If inputs have changed, tools from Reproducible-Builds to be used. Further development yet to be done.
- At this stage, binary contents should be same. However file timestamps (due to package managers) may be different.
- Static Timezone value: Bugzilla
- Adapted SOURCE_DATE_EPOCH: Bugzilla, Source-Date-Epoch - Reproducible Builds
- Archives generated with deterministic metadata (using archive tools' arguments)
- Remove non-deterministic data from rootfs
- Diffoscope data on their shared states: yocto-reproduciblebuilds-data
Diffoscope Dependencies
- Depends on: python3, PyPI modules: libarchive-c, python-magic
- External tools requied: Rscript, abootimg, apktool, bsdtar, bzip2, cbfstool, cd-iccdump, cmp, compare, convert, db_dump, diff, docx2txt, dumpxsb, enjarify, fdtdump, ffprobe, getfacl, ghc, gifbuild, gpg, gzip, identify, img2txt, isoinfo, javap, js-beautify, lipo, llvm-bcanalyzer, llvm-dis, lsattr, lz4, msgunfmt, nm, objcopy, objdump, ocamlobjinfo, odt2txt, oggDump, otool, pdftotext, pedump, pgpdump, ppudump, procyon, ps2ascii, readelf, showttf, sng, sqlite3, ssconvert, ssh-keygen, stat, tcpdump, unsquashfs, wasm2wat, xxd, xz, zipinfo, zipnote
- This has tools used to compare a lot of file formats that probably aren't generated (like android APKs, Windows/Mac executables) in a Buildroot run. We can exclude those.
- APT packages (available in Ubuntu, Debian): abootimg, acl, apktool, binutils-multiarch, bzip2, caca-utils, colord, coreutils, db-util, default-jdk-headless | default-jdk | java-sdk, device-tree-compiler, diffutils, docx2txt, e2fsprogs, enjarify, ffmpeg, fontforge-extras, fp-utils, genisoimage, gettext, ghc, ghostscript, giflib-tools, gnumeric, gnupg, gzip, imagemagick, jsbeautifier, libarchive-tools, llvm, lz4 | liblz4-tool, mono-utils, ocaml-nox, odt2txt, oggvideotools, openssh-client, pgpdump, poppler-utils, procyon-decompiler, r-base-core, sng, sqlite3, squashfs-tools, tcpdump, unzip, xmlbeans, xxd | vim-common, xz-utils, zip
Sample Diffoscope Output
- Minimal config build (make defconfig; make). Will run diffoscope on a build from Autobuilder config tomorrow.
- Builds run about 10 minutes apart.
- Moved first build to target, and rerun. Then run diffoscope target-1 target > diff.txt
- diffoscope log: https://paste.ubuntu.com/p/VpMbW4qQQP/
- Except for a time record in the busybox binary, all other differences seem to be only timestamps of file generation.
Planning
- Week 20: study how yocto does it
- Week 21: ...
- Week 22: do two builds in autobuild-run script
- Week 23: revisit patches to autobuild-run
- Week 24: autobuild-run: different output directories; report with reason=reproducible
- Week 25: improve how reproducible results are shown on http://autobuild.buildroot.org
- Week 26: Fix the reproducible issues found until now
- Week 27: Further extend autobuild-run script with more variation
- Week 28: Tooling to understand where the differences come from (e.g. which package)
- Week 29: More tooling to analyse differences
- Week 30: More tooling to analyse differences
Progress Journal
- 06/06/2019:
- Setup Gitlab account to track issues and progress
- Pushed dev branch to Gitlab, as well as re-spun commits and pushed
- 07/06/2019:
- Submitted v2 patches to mailing list
- Submitted patch to enable BR2_REPRODUCIBLE
- Started evaluating reprotest and its working
- 08/06/2019:
- Worked on v3 patches, their current form pushed to Gitlab for now
- Submitted patch to add Buildroot to Reproducible Builds: Who's involved (Merged and visible on website)
- 09/06/2019 - 11/06/2019:
- Finished v3 patches, sent to mailing list
- Evaluated reprotest & disorderfs, discussed with Arnout during meeting
- Less work done than possible due to lack of planning, now have planned till week 30
- 12/06/2019:
- Explored PHP components of Autobuilder website to identify what and how to modify
- Learned basic syntax and working of PHP since I've never worked with it before
- Brushed up on using classes in python because it has been a while since I worked using classes
- Now it's easier for me to implement the Builder class
- 13/06/2019:
- Worked on implementing and transitioning to Builder class
- Will push code to Gitlab tomorrow after removing errors
- 14/06/2019:
- First step of transitioning to Builder class done, pushed to Gitlab
- Testing it thoroughly before sending to mailing list
- Created and sent reproducible-v4 to mailing list, Gitlab
- 15/06/2019 - 18/06/2019:
- Worked on implementing builder class
- Pushed proof of concept to Gitlab
- Received feedback and now revising my patches
- 20/06/2019:
- Finished work on Builder class, waiting for feedback from Mentors
- 21/06/2019:
- Going to work on reason file in autobuild-run
- Sent Builder class to mailist list for feedback
- 22/06/2019:
- Worked on implementing reason-file, pushed to Gitlab
- Waiting for merge of builder-class to send to mailing list
- Waiting for feedback from Mentors
- 23/06/2019 - 25/06/2019:
- Working on different output directories
- 26/06/2019:
- Revising builder-class with feedback
- Having a bit of trouble with merging Thomas' patches
- 27/06/2019:
- Worked on ELC-E 2019 talk abstract
- Worked further on builder-class, testing ongoing..
Code
Atharva Lele's on-going work to make the autobuilder scripts reproducible-aware (check the reproducible-vN branches).
GSoC Proposal
The proposal PDF can be found here: PDF