taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

043-managing-prebuilt-artifacts.rst (4312B)

      1 DD 43: Managing Prebuilt Artifacts and Source-Level Dependencies
      2 ################################################################
      3 
      4 .. note::
      5 
      6   History:
      7   
      8   * 2025-11-26: Updated to reflect current state and future plans
      9 
     10 Summary
     11 =======
     12 
     13 This design document defines how the GNU Taler project manages prebuilt
     14 artifacts and other source-level dependencies in repositories
     15 that are part of project.
     16 
     17 Motivation
     18 ==========
     19 
     20 Some repositories have source-level dependencies on the build results of other
     21 repositories.  While it is possible to build these dependencies from scratch,
     22 we often want to avoid that extra step in order to make building the software
     23 easier and faster.
     24 
     25 Examples are:
     26 
     27 * man-pages built via Sphinx, produced in ``docs.git`` and consumed in ``exchange.git``.
     28 * SPAs used by the exchange, merchant libeufin.
     29 * (formerly): The ``taler-wallet-core-embedded.js`` file used
     30   by the Android repo.
     31 
     32 Another type of source-level dependency is on other **source** files.
     33 Examples for this are:
     34 
     35 * The ``build-common.git`` repository that contains common build-time helpers
     36 * The ``gana.git`` repository 
     37 
     38 Requirements
     39 ============
     40 
     41 * We are a free software project, users must always be able to
     42   easily re-build the prebuilt artifacts for themselves.
     43 * We want to reduce reliance on third party infrastructure as much
     44   as possible.
     45 * We want to keep our own infrastructure as slim as possible.
     46 * Builds must be reproducible and remain reproducible in the future.
     47 * Offline builds should be supported as well as possible.
     48 * Ideally, prebuilt artifacts can easily be cached / archived by third parties.
     49 
     50 Proposed Solution
     51 =================
     52 
     53 Instead of using a full-blown artifact management solution from the start, we
     54 make use of Git.  Since Git is not very good at managing binary artifacts,
     55 prebuilt files are not managed alongside the source code. Instead, we use
     56 (orphan-)branches) in a (possibly separate) repository to manage them.  This
     57 allows us to re-use Git authentication and Git commit signing.
     58 
     59 Repositories that produce artifacts *MUST* have phony Makefile helper targets
     60 to (a) create the prebuilt artifact (b) check out the prebuilt branch as a Git worktree, (c)
     61 move the artifact to the right place in the prebuilt branch and commit/push.
     62 
     63 Prebuilt files are consumed via git submodules.  The following rules apply for consumers:
     64 
     65 * The ``./bootstrap`` script of a component must initialize the submodule.
     66 * The ``./bootstrap`` script *SHOULD* fall back gracefully when the
     67   submodule commit of the prebuilt submodule is not available anymore.
     68 * The consuming module *MUST* rely on a particular commit of the prebuilt
     69   branch and the commit *MUST* be tagged to enable sparse checkouts of the submodule.
     70 * Consumers *SHOULD* use sparse checkouts for prebuilt submodules.
     71 
     72 Layout of the prebuilt branch and prebuilt tags:
     73 
     74 * The prebult files for a component are under ``{component_name}/{version}/``
     75 * The ``version`` *MUST* correspond both to the current version number of the
     76   component and a tag of the source repo of ``{component_name}``.
     77 * The prebuilt branch *MUST* have a tag of the form ``prebuilt-{serial}`` for
     78   every version that may be consumed by another component.
     79 * The prebuilt branch *MAY* have additional tags of the form
     80   ``{component_name}/{version}`` to indicate that a particular version of a
     81   particular component was added as a prebuilt version.
     82 
     83 
     84 Alternatives
     85 ============
     86 
     87 * Only maintain one version per module.
     88 
     89   * We were doing this (as of 2025-11-26) and it doesn't work very well,
     90     because the same package (say ``exchange.git``) might depend
     91     on different versions of prebuilt packages (say ``aml-spa`` and ``kyc-spa``).
     92     Thus the prebuilt branch would need a commit with exactly the right combination
     93     of versions, which is hard to do correctly.
     94 
     95   * This is okay for prebuilt branches that only contain one
     96     component, such as taler-docs.git.
     97 
     98 * Set up a full-blown artifact management solution like JFrog Artifactory.
     99   That's costly and requires a lot of admin work.
    100 
    101 * Never rely on prebuilt files.  That slows down the build process
    102   and in some cases requires exotic dependencies.
    103 
    104 Discussion / Q&A
    105 ================
    106 
    107 (This should be filled in with results from discussions on mailing lists / personal communication.)