Difference between revisions of "Buildroot:DeveloperDaysFOSDEM2019"

From eLinux.org
Jump to: navigation, search
(Meeting agenda)
(Meeting report)
Line 55: Line 55:
 
== Meeting report ==
 
== Meeting report ==
  
... is being written at https://mensuel.framapad.org/p/buildroot_fosdem_2019
+
=== Buildroot Association General Assembly ===
 +
General Assembly took place on February 5, 2019 from 13:00 to 13:15 after being delayed for a day with unanimous agreement of the members present.
 +
* Members present: Yann E. Morin, Thomas Petazzoni, Thomas De Schampheleire, Peter Korsgaard, Arnout Vandecappelle. Members represented: Romain Naour (represented by Yann E. Morin)
 +
* Validation of the accounts of 2018: accepted with 5 votes in favour, one abstinence
 +
* Validation of the activity report of 2018: accepted unanimously
 +
* Election of the board: board consists of Thomas Petazzoni (president), Yann E. Morin (treasurer), Peter Korsgaard. Accepted unanimously.
 +
 
 +
=== Download infra ===
 +
====  source-check [http://lists.busybox.net/pipermail/buildroot/2019-January/240021.html revival] ====
 +
After some discussion, the decision is that the patches are simple enough and the feature is useful enough to apply the patches. The only thing to check is that unsupported download methods (e.g. git) should do a full download.
 +
 
 +
==== git backend ====
 +
How to deal with the reproducibility of tarballs that we create from Git repositories? We currently require a specific version of tar, and also a specific variant of gzip. Some people think this is becoming annoying. Arnout has [http://lists.busybox.net/pipermail/buildroot/2018-November/235739.html proposed] to hash the contents of tarballs instead of the tarballs themselves, by introducing a new type of hash. Yann disagrees. We need to discuss this to reach a consensus. We discussed three options:
 +
# Keep things as they are, and add workaround for various cases to avoid building host-tar and host-gzip when not needed. However, this adds a lot of complexity to Buildroot. In addition, we will have to regularly update the required tar versions, which means refreshing a bunch of hashes.
 +
# Add a custom hash function that takes the hash of the individual files + their metadata. This has a number of disadvantages. We have to make our own, buildroot-specific hash function that is difficult to use outside of builldroot, e.g. to post-verify the legal-info. Also there is a slight risk that our hasher will need to be updated because there is still some inconsistency. An advantage is that we can move things gradually.
 +
# Use cpio instead of tar. cpio format is very well defined and stable. Well, it seems to be, it would have to be verified. Migration is a bit complicated because it's hard to do gradually, so temporary changes to the package .mk file have to be made during migration. But once migrated, things can stay the same, and we have a stable file format. There is one important limitation: cpio doesn't have the --strip-components option, so we would have to create tarbombs (cpiobombs). Also, it is a bit annoying to have cpio archives, people are not used to extracting those.
 +
# Use pax to generate pax archives. A pax archive can be extracted with tar, and the format is apparently stable as well. To make sure the files are renamed, we would compress them with xz instead of gz.
 +
 
 +
Drop support for branches and other unstable refs. It confuses users: when they give a branch name as version, they expect it to be updated on every build, while Buildroot will use the cached tarball. Excluding branch names is a little bit complicated by the fact that we are actually creating branches currently, so to be able to use existing git caches, the git helper should first delete existing branches.
 +
 
 +
=== Infra ===
 +
==== "Config.in.host" for all host packages ====
 +
 
 +
See [http://lists.busybox.net/pipermail/buildroot/2018-December/239298.html 1st] and [http://lists.busybox.net/pipermail/buildroot/2019-January/240064.html 2nd] parts of the discussion.
 +
 
 +
There are two reasons for this feature:
 +
# It allows to have the equivalent of automatic optional dependencies like for the target. The best example for this is host-ninja which requires host-python2 or host-python3, and host-meson which requires host-python3. Currently, if no python is selected for the target, the host-python will default to host-python2, so ninja will depend on host-python2, but then host-meson depends on host-python3, so both will be built. If there was a BR2_PACKAGE_HOST_PYTHON3 option, and host-meson would set this, then the ninja dependency could be based on the BR2_PACKAGE_HOST_PYTHON3 option. There are a few other situations like this but we couldn't remember any.
 +
# It would make sure that the PACKAGES list is really complete. Now, some host-packages are built without being added to PACKAGES. If we would have a full list of PACKAGES, that could be used for some optimisations, e.g. not evaluating .mk files of packages that are not going to be built anyway. However, that's all theory.
 +
 
 +
Note that for the first advantage, it is not needed to have a Config.in.host option for everything. But we do need to have a correct select for the config options that do exist. So we would need to add a check that a package that is built is also selected. Such a check exists already, but it is disabled for host packages. However, having such a partial transition would be really confusing. How does a contributor decide if a new host package requires a Config.in.host option? We could have such a half-baked situation for some time during a transition period, but having it long-term is really not nice.
 +
 
 +
For the second advantage, the list really needs to be accurate. That includes dependencies that are now added by the infra, e.g. cmake, or things handles by support/dependencies. For the infra, we could add "select BR2_PACKAGE_INFRA_CMAKE" to every cmake package Config.in, but that is adding a lot of complexity...
 +
 
 +
After some discussion, we found that the advantages are not strong enough for the added complexity and confusion.
 +
 
 +
Note that a specific use case is still possible: adding configuration options for host packages. For example, it is possible to add a BR2_PACKAGE_HOST_PYTHON_OPENSSL option to enable building the openssl module together with host-python, without adding a BR2_PACKAGE_HOST_PYTHON option.
 +
 
 +
==== Instrumentation hooks ====
 +
 
 +
Correctness vs. cost, especially about the [http://lists.busybox.net/pipermail/buildroot/2019-January/240224.html three file-lists] (host, staging, target)
 +
 
 +
We certainly want the list to be either correct or not to exist.
 +
 
 +
The md5sum approach definitely works, but it is too slow, especially for host/staging, because we have many files in there (e.g. from the toolchain).
 +
 
 +
To get an accurate list, we can just take a stat() on every file before and after the configure/build/install, and check the difference. In fact, find has options to print the relevant stat info. This is not too much overhead; we just run the find that we have now twice instead of once. One little problem is that if the user does 'make foo-configure; make bar; make foo', the changes from bar will be included in the diff for foo - but for such manual intervention, it's OK if the package list is inaccurate. Note that there would still be a problem for packages that install things in extract or patch, but we probably don't cover that at the moment either.
 +
 
 +
For 2019.02, we don't want to make too big changes anymore - especially since the patch series will not be ready tomorrow. However, a simple fix is to just create a list of stat output and take the diff only in the post-install instrumentation. This is definitely an improvement over the current situation and a small patch. It can be included in 2019.02 even if it only gets sent next week.
 +
 
 +
=== Qt5 versions to support: keep 5.6 or a newer LTS? ===
 +
Cfr. http://lists.busybox.net/pipermail/buildroot/2019-January/241108.html
 +
 
 +
For 2019.02, 5.6 will definitely stay the Qt5 LTS, it's too late to change now. But 5.6 will not get any updates anymore, it already didn't get any for a long time.
 +
 
 +
After 2019.02, we will introduce 5.12. The only reason to keep 5.6 is for license reasons, because 5.6 is still LGPLv2+.
 +
 
 +
Just the license is not good enough a reason for us to handle the maintenance for free. If upstream chooses to change the license, the users will either have to handle things on their own or pay for a commercial license.
 +
 
 +
Thus, the plan is to drop 5.6 after 2019.02. We will even drop the version option entirely - we don't have version choices for other packages. 5.12 will be an LTS version. For 5.13 and 5.14, there may be some push for introducing the non-LTS Qt5 version because it has some nice new features. We can decide at that point if we re-introduce the choice, but we prefer to avoid it. In general, we try to use only LTS Qt5 versions in Buildroot, because the maintenance windows don't overlap nicely with Buildroot releases.
 +
 
 +
=== Google Summer of Code. Deadline for applying is "February 6, 2019 at 21:00 (CET)" ===
 +
Ideas have been collected on [https://elinux.org/Buildroot:GSoC2019Ideas]
 +
 
 +
Arnout and Thomas will do application.
 +
 
 +
=== Should init scripts be executable in the Buildroot tree? ===
 +
 
 +
It really doesn't matter, there are arguments for either, as long as it's consistent. We randomly choose that they should not be executable.
 +
 
 +
=== Website ===
 +
It would be great to have a list of companies that use Buildroot on the website, to show that Buildroot really is used in a professional context. We can refer to publicly known information e.g. Tesla, and we can of course ask for companies where it is not known so publicly if they'd like to be mentioned.
 +
 
 +
The "Documentation" part of the website should have a reference to (recent) talks about Buildroot. Angelo will add that.
 +
 
 +
=== systemd's sysusers ===
 +
 
 +
Cfr. [http://patchwork.ozlabs.org/patch/874601/]
 +
 
 +
For sure, we don't want to have a user both in sysusers and in passwd, because then the sysusers entry is useless (it won't be used). So the patch as is is not OK.
 +
 
 +
Long term, we have the following options for dealing with the sysusers. But we only want to add infra for it when there are several packages using it.
 +
 
 +
# Duplicate the sysusers and passwd entries, like we have for service files vs. init scripts. Note that we still need a way to disable the passwd entry in case systemd is used. For now, this can be a condition in the .mk file (which is missing from [http://patchwork.ozlabs.org/patch/874601/ the current patch] but long-term we may want to add infra for it, e.g. a different _SYSUSERS variable name that only gets added to the global users list if not on systemd.
 +
# Generate the sysusers entry from the existing _USERS variable. Again, this requires some kind of identification of which users are to be done in the sysusers way, e.g. by adding an extra variable for it. Note that some packages provide their own sysusers file, in which case we still have to add a condition around the _USERS variable in the .mk file.
 +
# Generate the passwd entry from the sysusers files found in TARGET_DIR (and remove those files). This is probably the most complicated to implement, but it has the advantage that we don't need a USERS entry at all for packages that provide a sysusers file (which will occur more and more in the future). Note that this advantage is not as big as it may seem, since we'd still have to provide an init script.
 +
 
 +
=== Topics that were not discussed ===
 +
* [https://sourceware.org/ml/libc-announce/2019/msg00000.html Glibc-2.29] needs python 3.4 .Do we really want to add host-python3 in toolchain dependencies ? See [https://youtu.be/E6q5WLma-ZA?t=2652 BoF], [https://sourceware.org/ml/libc-alpha/2018-10/msg00215.html patch]
 +
* Top-level parallel build: discussion on outstanding issues
 +
 
 +
=== Patchwork cleanup ===
 +
 
 +
We're right before the -rc1 cut-off. Should we still apply risky patches? Decision to open next now for the risky stuff, but still keep on applying non-risky stuff to master, and cut -rc1 only after the meeting. We will also do more regular merges of master into next - at least once per -rc release.
 +
 
 +
In the end, though, no next branch was opened.
 +
 
 +
Before the developer meeting, there were just over 300 patches in patchwork. After the meeting, there were 172 left.

Revision as of 15:24, 7 February 2019

Buildroot Developers Meeting, 4-6 February 2019, Brussels

The Buildroot Developers meeting is a 3-day event for Buildroot developers and contributors. It allows Buildroot developers and contributors to discuss the hot topics in the Buildroot development, work on patches, and generally meet each other, facilitating further online discussions. Attending the event is free, after registration.

Location and date

The next Buildroot Developers meeting will take place on February 4th, 5th and 6th 2019 in Brussels, right after the FOSDEM conference. The meeting will take place in Google offices, located Chaussée d'Etterbeek 180, 1040 Brussels, very close to the Schuman metro station.

It is possible to join online by going to https://appear.in/buildroot

Sponsors

Google-logo.png

We would like to thank our sponsors:

  • Google, providing the meeting location, with Internet connection, but also free lunch and refreshments for the meeting participants.

Participants

  1. Thomas Petazzoni, arrives Friday 1 at 19:55 at BRU, leaves Wednesday 6 at 21:05 from BRU
  2. Peter Korsgaard
  3. Arnout Vandecappelle
  4. Yann E. MORIN, arrives Friday 1 at 19:55 at BRU, leaves Wednesday 6 at 21:05 from BRU
  5. Matt Weber (via IRC / Google Hangout)
  6. Angelo Compagnucci
  7. Mark Corbin
  8. Thomas De Schampheleire (physically present on Monday and Tuesday, may join remotely on Wednesday)
  9. Titouan Christophe (Monday only)
  10. Adam Duskett
  11. Adrian Perez de Castro (Monday only, flight departs at 17:55)

Meeting agenda

  • Buildroot Association General Assembly
  • Download infra
    • source-check revival
    • git backend:
      • How to deal with the reproducibility of tarballs that we create from Git repositories? We currently require a specific version of tar, and also a specific variant of gzip. Some people think this is becoming annoying. Arnout has proposed to hash the contents of tarballs instead of the tarballs themselves, by introducing a new type of hash. Yann disagrees. We need to discuss this to reach a consensus.
      • Drop support for branches and other unstable refs
  • Infra:
    • "Config.in.host" for all host packages 1st and 2nd parts of the discussion
    • Instrumentation hooks: correctness vs. cost, especially about the three file-lists (host, staging, target)
  • Graphs: speed-ups by Yann
  • Fix SDK: how to handle symlinks that point outside the SDK?
  • Glibc >= 2.29 needs python 3.4 Glibc-2.29: Do we really want to add host-python3 in toolchain dependencies ? See BoF, patch
  • Qt5 versions to support: keep 5.6 or a newer LTS? Cfr. http://lists.busybox.net/pipermail/buildroot/2019-January/241108.html
  • Top-level parallel build: discussion on outstanding issues
  • Google Summer of Code. Deadline for applying is "February 6, 2019 at 21:00 (CET)"
  • Should init scripts be executable in the Buildroot tree?

And of course, patchwork cleanup! Triage into the following categories:

  1. Low-hanging fruit, can probably be applied as-is. This is mostly for version bump and simple new packages.
  2. Already had discussion, just ping and mark as changes requested.
  3. Needs review.
  4. Needs discussion.

Meeting report

Buildroot Association General Assembly

General Assembly took place on February 5, 2019 from 13:00 to 13:15 after being delayed for a day with unanimous agreement of the members present.

  • Members present: Yann E. Morin, Thomas Petazzoni, Thomas De Schampheleire, Peter Korsgaard, Arnout Vandecappelle. Members represented: Romain Naour (represented by Yann E. Morin)
  • Validation of the accounts of 2018: accepted with 5 votes in favour, one abstinence
  • Validation of the activity report of 2018: accepted unanimously
  • Election of the board: board consists of Thomas Petazzoni (president), Yann E. Morin (treasurer), Peter Korsgaard. Accepted unanimously.

Download infra

source-check revival

After some discussion, the decision is that the patches are simple enough and the feature is useful enough to apply the patches. The only thing to check is that unsupported download methods (e.g. git) should do a full download.

git backend

How to deal with the reproducibility of tarballs that we create from Git repositories? We currently require a specific version of tar, and also a specific variant of gzip. Some people think this is becoming annoying. Arnout has proposed to hash the contents of tarballs instead of the tarballs themselves, by introducing a new type of hash. Yann disagrees. We need to discuss this to reach a consensus. We discussed three options:

  1. Keep things as they are, and add workaround for various cases to avoid building host-tar and host-gzip when not needed. However, this adds a lot of complexity to Buildroot. In addition, we will have to regularly update the required tar versions, which means refreshing a bunch of hashes.
  2. Add a custom hash function that takes the hash of the individual files + their metadata. This has a number of disadvantages. We have to make our own, buildroot-specific hash function that is difficult to use outside of builldroot, e.g. to post-verify the legal-info. Also there is a slight risk that our hasher will need to be updated because there is still some inconsistency. An advantage is that we can move things gradually.
  3. Use cpio instead of tar. cpio format is very well defined and stable. Well, it seems to be, it would have to be verified. Migration is a bit complicated because it's hard to do gradually, so temporary changes to the package .mk file have to be made during migration. But once migrated, things can stay the same, and we have a stable file format. There is one important limitation: cpio doesn't have the --strip-components option, so we would have to create tarbombs (cpiobombs). Also, it is a bit annoying to have cpio archives, people are not used to extracting those.
  4. Use pax to generate pax archives. A pax archive can be extracted with tar, and the format is apparently stable as well. To make sure the files are renamed, we would compress them with xz instead of gz.

Drop support for branches and other unstable refs. It confuses users: when they give a branch name as version, they expect it to be updated on every build, while Buildroot will use the cached tarball. Excluding branch names is a little bit complicated by the fact that we are actually creating branches currently, so to be able to use existing git caches, the git helper should first delete existing branches.

Infra

"Config.in.host" for all host packages

See 1st and 2nd parts of the discussion.

There are two reasons for this feature:

  1. It allows to have the equivalent of automatic optional dependencies like for the target. The best example for this is host-ninja which requires host-python2 or host-python3, and host-meson which requires host-python3. Currently, if no python is selected for the target, the host-python will default to host-python2, so ninja will depend on host-python2, but then host-meson depends on host-python3, so both will be built. If there was a BR2_PACKAGE_HOST_PYTHON3 option, and host-meson would set this, then the ninja dependency could be based on the BR2_PACKAGE_HOST_PYTHON3 option. There are a few other situations like this but we couldn't remember any.
  2. It would make sure that the PACKAGES list is really complete. Now, some host-packages are built without being added to PACKAGES. If we would have a full list of PACKAGES, that could be used for some optimisations, e.g. not evaluating .mk files of packages that are not going to be built anyway. However, that's all theory.

Note that for the first advantage, it is not needed to have a Config.in.host option for everything. But we do need to have a correct select for the config options that do exist. So we would need to add a check that a package that is built is also selected. Such a check exists already, but it is disabled for host packages. However, having such a partial transition would be really confusing. How does a contributor decide if a new host package requires a Config.in.host option? We could have such a half-baked situation for some time during a transition period, but having it long-term is really not nice.

For the second advantage, the list really needs to be accurate. That includes dependencies that are now added by the infra, e.g. cmake, or things handles by support/dependencies. For the infra, we could add "select BR2_PACKAGE_INFRA_CMAKE" to every cmake package Config.in, but that is adding a lot of complexity...

After some discussion, we found that the advantages are not strong enough for the added complexity and confusion.

Note that a specific use case is still possible: adding configuration options for host packages. For example, it is possible to add a BR2_PACKAGE_HOST_PYTHON_OPENSSL option to enable building the openssl module together with host-python, without adding a BR2_PACKAGE_HOST_PYTHON option.

Instrumentation hooks

Correctness vs. cost, especially about the three file-lists (host, staging, target)

We certainly want the list to be either correct or not to exist.

The md5sum approach definitely works, but it is too slow, especially for host/staging, because we have many files in there (e.g. from the toolchain).

To get an accurate list, we can just take a stat() on every file before and after the configure/build/install, and check the difference. In fact, find has options to print the relevant stat info. This is not too much overhead; we just run the find that we have now twice instead of once. One little problem is that if the user does 'make foo-configure; make bar; make foo', the changes from bar will be included in the diff for foo - but for such manual intervention, it's OK if the package list is inaccurate. Note that there would still be a problem for packages that install things in extract or patch, but we probably don't cover that at the moment either.

For 2019.02, we don't want to make too big changes anymore - especially since the patch series will not be ready tomorrow. However, a simple fix is to just create a list of stat output and take the diff only in the post-install instrumentation. This is definitely an improvement over the current situation and a small patch. It can be included in 2019.02 even if it only gets sent next week.

Qt5 versions to support: keep 5.6 or a newer LTS?

Cfr. http://lists.busybox.net/pipermail/buildroot/2019-January/241108.html

For 2019.02, 5.6 will definitely stay the Qt5 LTS, it's too late to change now. But 5.6 will not get any updates anymore, it already didn't get any for a long time.

After 2019.02, we will introduce 5.12. The only reason to keep 5.6 is for license reasons, because 5.6 is still LGPLv2+.

Just the license is not good enough a reason for us to handle the maintenance for free. If upstream chooses to change the license, the users will either have to handle things on their own or pay for a commercial license.

Thus, the plan is to drop 5.6 after 2019.02. We will even drop the version option entirely - we don't have version choices for other packages. 5.12 will be an LTS version. For 5.13 and 5.14, there may be some push for introducing the non-LTS Qt5 version because it has some nice new features. We can decide at that point if we re-introduce the choice, but we prefer to avoid it. In general, we try to use only LTS Qt5 versions in Buildroot, because the maintenance windows don't overlap nicely with Buildroot releases.

Google Summer of Code. Deadline for applying is "February 6, 2019 at 21:00 (CET)"

Ideas have been collected on [1]

Arnout and Thomas will do application.

Should init scripts be executable in the Buildroot tree?

It really doesn't matter, there are arguments for either, as long as it's consistent. We randomly choose that they should not be executable.

Website

It would be great to have a list of companies that use Buildroot on the website, to show that Buildroot really is used in a professional context. We can refer to publicly known information e.g. Tesla, and we can of course ask for companies where it is not known so publicly if they'd like to be mentioned.

The "Documentation" part of the website should have a reference to (recent) talks about Buildroot. Angelo will add that.

systemd's sysusers

Cfr. [2]

For sure, we don't want to have a user both in sysusers and in passwd, because then the sysusers entry is useless (it won't be used). So the patch as is is not OK.

Long term, we have the following options for dealing with the sysusers. But we only want to add infra for it when there are several packages using it.

  1. Duplicate the sysusers and passwd entries, like we have for service files vs. init scripts. Note that we still need a way to disable the passwd entry in case systemd is used. For now, this can be a condition in the .mk file (which is missing from the current patch but long-term we may want to add infra for it, e.g. a different _SYSUSERS variable name that only gets added to the global users list if not on systemd.
  2. Generate the sysusers entry from the existing _USERS variable. Again, this requires some kind of identification of which users are to be done in the sysusers way, e.g. by adding an extra variable for it. Note that some packages provide their own sysusers file, in which case we still have to add a condition around the _USERS variable in the .mk file.
  3. Generate the passwd entry from the sysusers files found in TARGET_DIR (and remove those files). This is probably the most complicated to implement, but it has the advantage that we don't need a USERS entry at all for packages that provide a sysusers file (which will occur more and more in the future). Note that this advantage is not as big as it may seem, since we'd still have to provide an init script.

Topics that were not discussed

  • Glibc-2.29 needs python 3.4 .Do we really want to add host-python3 in toolchain dependencies ? See BoF, patch
  • Top-level parallel build: discussion on outstanding issues

Patchwork cleanup

We're right before the -rc1 cut-off. Should we still apply risky patches? Decision to open next now for the risky stuff, but still keep on applying non-risky stuff to master, and cut -rc1 only after the meeting. We will also do more regular merges of master into next - at least once per -rc release.

In the end, though, no next branch was opened.

Before the developer meeting, there were just over 300 patches in patchwork. After the meeting, there were 172 left.