Buildroot:DeveloperDaysELCE2012

What is Buildroot ?
Buildroot is a set of Makefiles and patches that makes it easy to generate a complete embedded Linux system. Buildroot can generate any or all of a cross-compilation toolchain, a root filesystem, a kernel image and a bootloader image. Buildroot is useful mainly for people working with small or embedded systems, using various CPU architectures (x86, ARM, MIPS, PowerPC, etc.) : it automates the building process of your embedded system and eases the cross-compilation process.

Meeting in Barcelona
The Buildroot community is organizing a meeting on Saturday November 3rd and Sunday November 4th, for Buildroot developers and contributors. This meeting will be a mixture of discussion and hacking session around the Buildroot project. This meeting takes place right before the Embedded Linux Conference Europe, in order to make it easy for participants to attend both events.

Event sponsored by Fluendo and Synopsys
The meeting room is sponsored by Fluendo. Thanks!

The Saturday dinner is sponsored by Synopsys. Thanks!

Action points

 * Accept perl series.
 * Accept qemu series (after further review).
 * Remove compiler on target.
 * Remove development files on target.
 * Remove documentation on target.
 * Explain in the documentation why the above was done.
 * Put httpd daemons outside BUSYBOX_SHOW_OTHERS.
 * Move BUSYBOX_SHOW_OTHERS conditions to depends on inside the package Config.in
 * busybox: disable mount-nfs in the default config.
 * simplify patch logic: if pkg-version directory exists, use that; otherwise, use pkg-*.patch.
 * Automate selection of correct FP and other compiler options based on subarchitecture.
 * Move download helpers to a script.
 * Samuel merges documentation patches and sends pull requests to Peter.
 * Ask Yann for help regarding the comments in the AVAILABLE framework.
 * Document better that you need to do make clean; make instead of make target-clean.
 * Rename output/target to output/tmproot.
 * Arnout reposts the save-configuration simplification patches without out-of-buildroot-tree support.

Details of the discussion

 * 1) Perl.  The perl patch series (now at v11) looks ready to be integrated, but we have our doubts about cpanminus. The problems are: it has separate download infrastructure, which blocks some buildroot features (e.g. make source); it doesn't integrate with the legal-info.  However, CPAN is pretty important if you really want to use perl.  The decision was to accept cpanminus as well, but add some warnings that some things don't work.
 * 2) Qemu.  Adding this opens the door for more people to do stuff that buildroot wasn't meant for in the first place (i.e. embedded).  However, even in embedded qemu could be relevant.  Things like the compiler on the target are much worse for doing crazy things.  Giving new packages a chance is OK; we have to be much more careful with things that have global impact (like including development files on target).  Conclusion: it's OK to accept qemu.
 * 3) Compiler on target.  It leads to newbies using buildroot in an inappropriate way (i.e. compile on the target).  Also it only works with the internal toolchain.  And if we remove it, we can also remove "Development files on target", which simplifies some of the build infrastructure.  Also, if you have a compiler on the target, then you can just as well use a distro.  Conclusion: we remove it.  Also a bunch of other packages can be removed (autotools, vala, ...)
 * 4) Documentation on target: if you need this, use a distro.  So we remove it.
 * 5) We need to explain in the documentation why we removed this stuff.
 * 6) Available stuff: i.e. solve the problem of combination of select and depends. Solving it in Kconfig itself seems to be difficult: there were some patches but the kconfig people didn't like it; Yann and Thomas looked at Kconfig and it looks difficult to modify.  The AVAILABLE solution has two problems: it adds verbosity, and the recursive dependencies still should be written in the comments ("libfoo requires WCHAR in the toolchain").  The verbosity is not considered a big issue.  For the comments: they currently don't work perfectly either.  Removing the comments altogether is an option, but then it is difficult for users to find out why a certain package isn't available.  Another possibility is to list the available packages in the documentation (automatically), and mention which toolchain dependencies they have.  Wild idea: consider Config.in as a low-level language, and define our own higher-level language from which we generate Config.in.  Alternatively, have a developer tool that verifies if everything is consistent (i.e. are the comments correct).  One more option: add a big comment in the beginning of the packages list, which says that not all toolchain options are available so not all packages are visible.  Converting the toolchain options from depends to select is not really an option, because changing toolchain options requires a make clean. After more discussion on Sunday, we didn't find a solution. For the time being, we stick to the old system. We ask Yann if he has a brilliant idea of how to deal with the comments.
 * 7) kconfig-frontend: Yann now has a repository for Kconfig, it would be nice to branch our Kconfig from that.  The idea is to still copy it to our repository.
 * 8) BUSYBOX_SHOW_OTHERS: the only reason to have this is to avoid newcomers enabling things like tar etc. which are anyway already in busybox.  But for the web servers, they are clearly different from the busybox web servers (and our default busybox config doesn't even enable a web server).  Also bash is a typical one, because people write bashisms in their scripts.  This discussion is somewhat related to the discussion how to handle packages that provide the same functionality.  But for many packages, it's not even the same (httpd, bash, vim).  But actually, we're trying to solve a non-existent problem here - except for the web servers.  If we generate a list of packages in the documentation, we can add a remark to that that it's also available in busybox.  Also move the condition inside the package Config.in instead of outside and make it a depends on, so it is more visible.
 * 9) packages with multiple variants: python2/3, Qt4/5, gtk2/3, gstreamer0.10/1.0...  This is again similar to the expat/libxml2 issue.  Do we support both python2 and python3? We need to support both versions in most cases. Question is if we want to force that only one of the two is selected, or do we support side-by-side installation.  Conclusion: has to be looked at on a case-by-case basis.  General philosophy: don't try to be too smart, so don't try to force selecting only one of the two if it doesn't make sense to have both.  In cases where a package wants to select one of these things, convert the select into a depends with a comment.  For gstreamer, the complexity of having the two side-by-side is unacceptable, because there are so many fine-grained gstreamer configure options (in the plugins).  However, we expect that everything will move to gstreamer1.0 quite quickly, so for the time being (and only if someone needs it), we can live with a non-perfect solution that just fits the requirements of whoever needs gstreamer1.0.
 * 10) multiple packages providing the same functionality: lua/luajit, jpeg/jpegturbo, expat/libxml2.  These cannot be installed side-by-side, and many packages select e.g. jpeg.  OpenGL will have this issue as well.  A relatively clean solution is to create virtual packages for these, where the user has a choice between the two.  The normal config option would be hidden for these packages.  This still adds complexity in some corner cases, e.g. some package that doesn't work in luajit, but that can probably be solved at the level of that package.
 * 11) Autobuild infrastructure. Thomas is working on scripts that parse the build result and put it in a database.  One important missing feature at the moment is visibility of evolution over time (i.e. distinguish regressions from recurring problems).  Even more important is just being able to see which packages are failing all the time, i.e. for each package how often it failed.  Ideally, we would also want to see how often a package has been built successfully, but when is it successful? Does the entire build have to succeed or that package?  Overall, however, the autobuild infrastructure is considered very useful - when something is wrong, it appears in the autobuild very quickly.
 * 12) * autobuild of the next branch: if only the master is autobuilt, we don't see a lot of problems until after the release. Delaying fixing bugs makes it more difficult to fix them. To support autobuild of the next branch, this should also be made visible in the database so we can distinguish master failures (more important) from next failures.
 * 13) * autobuild database is not suitable to track repeated builds of the same configuration. It needs to keep track of different builds of the same configurations.  But the question is if this is really useful. We would like to check the existing defconfigs at every release, but it doesn't have to be tracked.  So it can be done manually.
 * 14) AVR32. It is end of life as far as Atmel is concerned, but of course there are still products out there using it.  It is not really blocking for buildroot.  For packages which fail to build, we can just add a depends on !BR2_avr32.  But of course, until we have the AVAILABLE mechanism, carrying this kind of dependency can be painful.
 * 15) RPC support. Originally, the idea was to consider RPC support as a toolchain option, i.e. using depends.  However, libtirpc is really a library, comparable to gettext, so it could be enabled automatically with select.  Busybox is special, because we don't want to force RPC just because busybox is selected.  Therefore busybox has a user-selectable option to enable RPC-based applets (mount-nfs).  After discussion, this option was discarded - just remove mount-nfs from busybox (see below).  Other packages will use libtirpc if it was selected (manually or automatically). It's still a question what will happen when dynamically linking against libtirpc when libc also has rpc.
 * 16) A bit related: we'll disable mount-nfs in the default busybox config; if user disables RPC and enables mount-nfs in busybox, he'll get a compilation error.
 * 17) Support for VCS branches in packages: custom packages are often in a VCS and while developing, you want to track a branch of that VCS.  OVERRIDE_SRCDIR doesn't work very well because it means the user has to check everything out at the right version in the right location.  An alternative would be to teach OVERRIDE_SRCDIR about VCSes, so that make foo-rebuild re-syncs with the VCS.
 * 18) There is a general feeling (mainly from Thomas) that buildroot isn't very convenient during development of a project where several people are working on several components. But no solution was found.
 * 19) FP stuff: we now have a single SOFT_FLOAT option, but actually there's a lot more variation: neon, vfpv3, softfp, ....  To support this well, we can add a lot of extra config options to semi-automatically select the correct float options based on the subarchitecture.  There is some cross-dependency with the external toolchains, but it's doable.  There should still be an option to manually override the float options for exotic cases.  ARM thumb adds another layer of complexity.  Someone should take the lead on making this easier.
 * 20) Porting gcc to the generic package infrastructure.  Looks like this is a lot of work, and there's not so much need for it.
 * 21) Getting gcc from git.  It's more convenient to put a tarball somewhere public and use that as an alternative source, similar to avr32.
 * 22) Download helpers are getting too complex.  It should move a script so at least it's more readable.
 * 23) Community organization:
 * 24) * It can be useful to have subsystem maintainers, that are responsible for reviewing and accepting patches. In practice, however, it is difficult to split things in a way that we don't risk duplicating work (i.e. two people in the process of merging the same patch).  However, one clearly defined subsystem is the documentation.  From now on, Samuel Martin is responsible for all documentation patches.  He'll send pull requests to Peter.
 * 25) * Patches should be accepted more aggressively, even if there hasn't been (much) testing of the patch. The autobuilders will catch any issues.  However, it is still important to properly review and comment on coding style issues (i.e. how the features are implemented.  Also, if something really breaks the autobuilder, Thomas can revert it without consulting Peter.
 * 26) Out of tree build: this is a feature that people keep on asking. It is particularly useful for packages with an OVERRIDE_SRCDIR.  But of course, not all packages support out-of-tree build, and not all packages support it in the same way.  For autotools and cmake, it can be automatic; for generic packages, it has to be signalled if out-of-tree build is available.
 * 27) legal-info:
 * 28) * split _LICENSE and _REDISTRIBUTE: patch proposed by Luca is accepted. Even though there is no proprietary package in buildroot that triggers the feature, if it fails to work as expected the user will hopefully report it.
 * 29) * multiple licenses: we don't want to make things very complex. It doesn't have to be machine-readable (we don't need to support machine analysis of the license field), so we don't have to formally decide how the license text must be written.  It does make sense to have some convention, though.  Proposal: "GPLv2+ or BSD-2c for the library, GPLv2+ for the dtc executable".
 * 30) target-clean: purpose is to rebuild the target tree, without rebuilding all packages. The problem is that this looks as if packages are really removed, but there may still be dependencies hidden in the built packages so that the resulting target doesn't work anymore.  One idea is to just not have a target dir anymore, but instead reconstruct it from the staging dir every time the fs is built.  But this is too much overhead in the usual case.  Basically, we prefer to have people do a full make clean if they change anything.  We should update the documentation to explain that this is necessary.  There was an idea to remove the entire target tree, and just regenerate it on every build.  This would simplify the .mk files because for most packages there's no need for INSTALL_TARGET_CMDS anymore (some packages still want to remove stuff), and implicitly does a target-clean (because there is no target dir to begin with), but only for the use case of skeleton changes.  In the end, the conclusion was that we don't want to support target-clean, but we'll document why not.
 * 31) Regarding out-of-buildroot-tree project stuff: we don't really want to advertise keeping stuff outside of the buildroot tree.  Any patches that make that easier will be rejected. But it would be nice to store the result of xxx-menuconfig back to the place where it came from, or warn if that place doesn't exist.
 * 32) API changes: we tend to change the way the packages are used very often, without caring about how this affects users who have a modified buildroot tree. Merge conflicts are not much of an issue, but "invisible" changes can really bite users that don't follow the mailing list.  We should think a bit about how it affects user before randomly renaming "BR2_NEEDS_GETTEXT" to "BR2_NEEDS_EXTERNAL_GETTEXT".

Participants

 * 1) Arnout Vandecappelle, confirmed.  Arriving on Fri Nov 2 20:35 at BCN airport.  Leaving Wed Nov. 8 21:15 at BCN. Staying at HCC Lugano
 * 2) Thomas Petazzoni, confirmed. Arriving on November, 3rd at BCN airport 9:25 AM. Leaving on November 8th from BCN airport at 3:45 PM. Staying at Hotel Fira Palace.
 * 3) Maxime Ripard, confirmed. Arriving on November, 1st at BCN airport. Leaving on November 8th from BCP airport at 3:45 PM. Staying at Hotel Fira Palace.
 * 4) Peter Korsgaard, confirmed, Arriving on Fri Nov 2 21:55 at BCN airport. Leaving Sun Nov 11 17:15 at BCN. Staying at HCC Lugano
 * 5) Luca Ceresoli, confirmed. Arriving on November 3rd at BCN airport 10:05 AM. Leaving on November 7th from BCN airport 19:15 PM. Staying at Hotel Fira Palace. Will arrive around 11:30 AM for the meeting.
 * 6) Samuel Martin, confirmed. Arriving on November, 3rd at Barcelona-Franca Train Station 8:05 AM. Leaving on November 10th from Barcelona-Franca Train Station at 8:43 PM.
 * 7) Simon Dawson, confirmed. Arriving on November 2nd in the evening. Leaving on November 8th early morning.
 * 8) Mischa Jonker, confirmed. From Synopsys.com. Not contributing yet to Buildroot, but interested in using Buildroot for a brand new CPU architecture. Arriving Friday Nov 2nd BCN 21:10 CET, staying at Catalonia Barcelona Plaza (Plaza Espanya), leaving Thursday Nov 8 at 18:40 CET.
 * 9) Maxime Hadjinlian.

Registration
Please contact Thomas Petazzoni (thomas.petazzoni@free-electrons.com) if you would like to attend.

Attending the event is free, but registration is required.

Program

 * Saturday, 10 AM: beginning of the meeting. The first person to arrive can go directly to the front desk and ask for MARIBEL SOCIAS or anyone in the Reservations Department, and then ask for the meeting room booked for Thomas Petazzoni. Thomas Petazzoni will only arrive at 10:30-11 AM.
 * Saturday, 7 PM: end of the meeting
 * Saturday, 8:30 PM: dinner sponsored by Synopsys
 * Sunday, 10 AM: beginning of the meeting
 * Sunday, 7 PM: end of the meeting

Location
The event will take place at Hotel Grumps, in downtown Barcelona. See this map for the location of the hotel, and this map for walking directions between the Fira Palace hotel (location of the ELCE conference) and Hotel Grums.

List of topics to discuss

 * Interaction between Buildroot configuration options to select architecture specifities (architecture variant, soft-float vs. hard-float, etc.) and the multilib variants available in external toolchains. Proposed by User:ThomasPetazzoni.
 * Status on the community organization. Has patchwork improved things? Are patches taken into account in a faster way? Proposed by User:ThomasPetazzoni.
 * What are the next "big" things for Buildroot? Proposed by User:ThomasPetazzoni.
 * Documentation refactoring? Proposed by Samuel Martin.
 * Autobuilder tests: needed improvements, future work. Proposed by User:ThomasPetazzoni.
 * The _AVAILABLE mechanism, and how to properly handle dependency on toolchain options (discuss the proposal made by Yann E. Morin). Proposed by User:ThomasPetazzoni.
 * Discuss (and, if approved, implement) the split between _LICENSE and _REDISTRIBUTE for legal-info. I think this won't take much more than 10 minutes chat + 10 minutes hack. Proposed by Luca Ceresoli.
 * Make it possible to use a git repository as source for toolchain. This doesn't work well today. Proposed by Mischa Jonker
 * Make it possible to do a 'target-clean', as a replacement of uninstall. Will this even work at all? Proposed by Arnout Vandecappelle
 * Does it make sense to port the toolchains to generic-package infrastructure? Asked by Arnout Vandecappelle
 * How to provide licensing info for packages that have more than one license. The main use cases are: a) different parts of the source code have different licenses, b) a file (or the whole project) is distributed under two or more licenses or c) a mix of the two. See the recent thoughts about this issue by Yann Morin: http://lists.busybox.net/pipermail/buildroot/2012-October/060475.html
 * Support for board stuff outside the buildroot tree. Proposed by Arnout Vandecappelle
 * Should we remove the ability to build a toolchain on the target? And also things like development files on target, documentation on target, etc. Proposed by User:ThomasPetazzoni.

List of topics to hack on

 * Reduce the back log of bugs. Proposed by User:ThomasPetazzoni.
 * Reduce the back log of patches. Proposed by User:ThomasPetazzoni.
 * Work on noMMU platform support (Blackfin, Coldfire). Getting Coldfire Qemu working, adding correct support for FLAT binaries. Proposed by User:ThomasPetazzoni.
 * Package upgrades: package python3, gtk3 and latest versions of glib and al. Proposed by User:ThomasPetazzoni.
 * Triage autobuilder failures and fix them. Proposed by Arnout Vandecappelle
 * Improve the documentation. Proposed by Arnout Vandecappelle
 * Work on relocatable toolchain. Proposed by Arnout Vandecappelle
 * Add some infra for setup.py/qmake based packages. Proposed by Samuel Martin
 * Add some infra for make based packages (mainly using the same pattern all over). Proposed by Arnout Vandecappelle
 * Add some infra for Kconfig based packages (linux, barebox, busybox, uclibc, ct-ng, at91bootstrap3). Proposed by Arnout Vandecappelle
 * Make top-level make -j possible. Proposed by Arnout Vandecappelle