taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

043-managing-prebuilt-artifacts.rst (4552B)

      1 DD 43: Managing Prebuilt Artifacts and Source-Level Dependencies
      2 ################################################################
      3 
      4 :Design status: Accepted
      5 :Implementation status: N/A
      6 :DD shepherd: TBD
      7 :Historical contributors: Florian Dold
      8 :First published: 2023-05-04
      9 :Last substantive change: 2025-11-26
     10 :Normative references: :doc:`../developer/taler-developer-manual`
     11 
     12 .. note::
     13 
     14   History:
     15 
     16   * 2025-11-26: Updated to reflect current state and future plans
     17 
     18 Summary
     19 =======
     20 
     21 This design document defines how the GNU Taler project manages prebuilt
     22 artifacts and other source-level dependencies in repositories
     23 that are part of project.
     24 
     25 Motivation
     26 ==========
     27 
     28 Some repositories have source-level dependencies on the build results of other
     29 repositories.  While it is possible to build these dependencies from scratch,
     30 we often want to avoid that extra step in order to make building the software
     31 easier and faster.
     32 
     33 Examples are:
     34 
     35 * man-pages built via Sphinx, produced in ``docs.git`` and consumed in ``exchange.git``.
     36 * SPAs used by the exchange, merchant libeufin.
     37 * (formerly): The ``taler-wallet-core-embedded.js`` file used
     38   by the Android repo.
     39 
     40 Another type of source-level dependency is on other **source** files.
     41 Examples for this are:
     42 
     43 * The ``build-common.git`` repository that contains common build-time helpers
     44 * The ``gana.git`` repository
     45 
     46 Requirements
     47 ============
     48 
     49 * We are a free software project, users must always be able to
     50   easily re-build the prebuilt artifacts for themselves.
     51 * We want to reduce reliance on third party infrastructure as much
     52   as possible.
     53 * We want to keep our own infrastructure as slim as possible.
     54 * Builds must be reproducible and remain reproducible in the future.
     55 * Offline builds should be supported as well as possible.
     56 * Ideally, prebuilt artifacts can easily be cached / archived by third parties.
     57 
     58 Proposed Solution
     59 =================
     60 
     61 Instead of using a full-blown artifact management solution from the start, we
     62 make use of Git.  Since Git is not very good at managing binary artifacts,
     63 prebuilt files are not managed alongside the source code. Instead, we use
     64 (orphan-)branches) in a (possibly separate) repository to manage them.  This
     65 allows us to re-use Git authentication and Git commit signing.
     66 
     67 Repositories that produce artifacts *MUST* have phony Makefile helper targets
     68 to (a) create the prebuilt artifact (b) check out the prebuilt branch as a Git worktree, (c)
     69 move the artifact to the right place in the prebuilt branch and commit/push.
     70 
     71 Prebuilt files are consumed via git submodules.  The following rules apply for consumers:
     72 
     73 * The ``./bootstrap`` script of a component must initialize the submodule.
     74 * The ``./bootstrap`` script *SHOULD* fall back gracefully when the
     75   submodule commit of the prebuilt submodule is not available anymore.
     76 * The consuming module *MUST* rely on a particular commit of the prebuilt
     77   branch and the commit *MUST* be tagged to enable sparse checkouts of the submodule.
     78 * Consumers *SHOULD* use sparse checkouts for prebuilt submodules.
     79 
     80 Layout of the prebuilt branch and prebuilt tags:
     81 
     82 * The prebult files for a component are under ``{component_name}/{version}/``
     83 * The ``version`` *MUST* correspond both to the current version number of the
     84   component and a tag of the source repo of ``{component_name}``.
     85 * The prebuilt branch *MUST* have a tag of the form ``prebuilt-{serial}`` for
     86   every version that may be consumed by another component.
     87 * The prebuilt branch *MAY* have additional tags of the form
     88   ``{component_name}/{version}`` to indicate that a particular version of a
     89   particular component was added as a prebuilt version.
     90 
     91 
     92 Alternatives
     93 ============
     94 
     95 * Only maintain one version per module.
     96 
     97   * We were doing this (as of 2025-11-26) and it doesn't work very well,
     98     because the same package (say ``exchange.git``) might depend
     99     on different versions of prebuilt packages (say ``aml-spa`` and ``kyc-spa``).
    100     Thus the prebuilt branch would need a commit with exactly the right combination
    101     of versions, which is hard to do correctly.
    102 
    103   * This is okay for prebuilt branches that only contain one
    104     component, such as taler-docs.git.
    105 
    106 * Set up a full-blown artifact management solution like JFrog Artifactory.
    107   That's costly and requires a lot of admin work.
    108 
    109 * Never rely on prebuilt files.  That slows down the build process
    110   and in some cases requires exotic dependencies.
    111 
    112 Discussion / Q&A
    113 ================
    114 
    115 (This should be filled in with results from discussions on mailing lists / personal communication.)