diff options
Diffstat (limited to 'design-documents/043-managing-prebuilt-artifacts.rst')
-rw-r--r-- | design-documents/043-managing-prebuilt-artifacts.rst | 107 |
1 files changed, 107 insertions, 0 deletions
diff --git a/design-documents/043-managing-prebuilt-artifacts.rst b/design-documents/043-managing-prebuilt-artifacts.rst new file mode 100644 index 00000000..3d0debc8 --- /dev/null +++ b/design-documents/043-managing-prebuilt-artifacts.rst @@ -0,0 +1,107 @@ +DD 43: Managing Prebuilt Artifacts and Source-Level Dependencies +################################################################ + +Summary +======= + +This design document defines how the GNU Taler project manages prebuilt +artifacts and other source-level dependencies in repositories +that are part of project. + +Motivation +========== + +Some repositories have source-level dependencies on the build results of other +repositories. While it is possible to build these dependencies from scratch, +we often want to avoid that extra step in order to make building the software +easier and faster. + +Examples are: + +* man-pages built via Sphinx, produced in ``docs.git`` and consumed in ``exchange.git``. +* SPAs used by the exchange, merchant libeufin. +* (formerly): The ``taler-wallet-core-embedded.js`` file used + by the Android repo. + +Another type of source-level dependency is on other **source** files. +Examples for this are: + +* The ``build-common.git`` repository that contains common build-time helpers +* The ``gana.git`` repository + +Requirements +============ + +* We are a free software project, users must always be able to + easily re-build the prebuilt artifacts for themselves. +* We want to reduce reliance on third party infrastructure as much + as possible. +* We want to keep our own infrastructure as slim as possible. +* Builds must be reproducible and remain reproducible in the future. +* Offline builds should be supported as well as possible. +* Ideally, prebuilt artifacts can easily be cached / archived by third parties. + +Proposed Solution +================= + +Instead of using a full-blown artifact management solution from the start, we +make use of Git. Since Git is not very good at managing binary artifacts, +prebuilt files are not managed alongside the source code. Instead, we use +(orphan-)branches) in a (possibly separate) repository to manage them. This +allows us to re-use Git authentication and Git commit signing. + +Due to a historical accident, prebuilt files are currently stored in the +``prebuilt`` branch of the ``wallet-core.git`` repository. This might change +in the future. + +To enable forwards-compatibility, we are implementing the following approach: +The ``taler.net`` HTTP server redirects requests in the form of +``https://artifacts.taler.net/$component/$version/$file`` to +``https://git.taler.net/$repo.git/tree/$version/$file?h=$branch``, where +``$component`` determines ``$repo`` and ``$branch``. + +We are also introducing the new rule that a prebuilt artifact must be a single +archive file (preferable ``.tar.gz.``). + +Repositories that produce artifacts *MUST* have phony Makefile helper targets +to (a) create the prebuilt artifact (b) check out the prebuilt branch as a Git worktree, (c) +move the artifact to the right place in the prebuilt branch and commit/push. + +Repositories that depend on prebuilt components must download them +in their ``./bootstrap`` script. After downloading, the bootstrap +script *SHOULD* verify a checksum of the downloaded artifact. + +FIXME: Link to an example bootstrap file, once the approach from this document +is fully implemented. + +Alternatives +============ + +* The prebuilt branch could be integrated into source repositories that + depend on it via a git submodule. This has the following disadvantages: + + 1. Git submodules are difficult to handle for some developers. We've had issues + in the past where the prebuilt submodule wasn't integrated properly and would + lead to non-reproducible builds + 2. Relying purely on the Git commit hash of the prebuilt branch + makes it very difficult to know which version of the prebuilt + artifact is being used. It also makes it very difficult + to reliable find older versions of the artifact. + 3. Relying on the commit hash instead of a version and artifact hash + couples the artifact to the Git commit. That means that + we can never compact the commit history of the prebuilt + branch of move artifacts around. + +* Set up a full-blown artifact management solution like JFrog Artifactory. + That's costly and requires a lot of admin work. + +* Never rely on prebuilt files. That slows down the build process + and in some cases requires exotic dependencies. + +Drawbacks +========= + +Discussion / Q&A +================ + +(This should be filled in with results from discussions on mailing lists / personal communication.) |