View source for Buildroot:Language package managers and dependencies

This page summarizes the discussion and conclusions around language package managers and downloading dependencies. For example, for go modules, cargo crates, and php compose.

== Problem statement and requirements ==
Language package managers make it possible to express dependencies and make sure the correct version of those dependencies are used.

For Buildroot, the problem is to make sure those dependencies are downloaded. Sometimes the upstream package simply bundles all dependencies (this is called "vendored"), but sometimes they only have them by reference.

Buildroot needs to make sure that:
* the dependencies are downloaded;
* the dependencies are extracted in a way that they're accessible by the language's build system;
* everything is properly hashed so we are sure the correct tarballs were downloaded;
* it is possible to do "make source", unplug the network, and do a build;
* it is possible to use a BR2_PRIMARY_SITE and BR2_PRIMARY_SITE_ONLY;
* it is possible to use BR2_BACKUP_SITE;
* the information produced by "make legal-info" is correct.

An additional nice-to-have is to be able to split off the open source dependencies from the non-redistributable dependencies.


== Possible solutions ==

# Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, and creates a tarball of this bundle.
# Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, and creates a separate tarball containing only the dependencies.
# Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, splits off the non-redistributable dependencies, and creates a separate tarball for the open-source and the non-redistributable dependencies.
# Add a download helper that downloads the base source, runs the package manager to get all dependencies, and creates a separate tarball for each individual dependency.
# Solution 1 + extra legal-info infrastructure that makes it possible to strip off non-redistributable parts when generating the legal-info tarball.

Solution 1 is the simplest to implement. It is the approach taken by http://patchwork.ozlabs.org/project/buildroot/list/?series=159771 for cargo.

Solution 2 is also easy to implement. However:
# the download infra has to "know" how to download the additional tarball (it should not be downloaded when using the normal method, but it should be downloaded for PRIMARY and BACKUP);
# the additional tarball has to be added to the hash file;
# the additional tarball has to be extracted as well.

Solution 3 is like solution 2, but times two.

Solution 4 is a lot more complicated to implement. It is probably doable to use the language itself to wrap the package manager (which typically has a library as well) to gather a list of all dependencies and URLs. Then each of them has to be downloaded and tarred separately. However, to support PRIMARY and BACKUP, we'll need to maintain this list inside Buildroot as well. In addition, we need a separate hash for each tarball.

Solution 5 is somewhat orthogonal to the rest. It has the nice advantage that it can also be applied to cases where the bundling is done without using a language package manager (e.g. for C/C++ code).

== Discussion links ==
http://lists.busybox.net/pipermail/buildroot/2020-September/291002.html