1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
|
DD 43: Managing Prebuilt Artifacts and Source-Level Dependencies
################################################################
Summary
=======
This design document defines how the GNU Taler project manages prebuilt
artifacts and other source-level dependencies in repositories
that are part of project.
Motivation
==========
Some repositories have source-level dependencies on the build results of other
repositories. While it is possible to build these dependencies from scratch,
we often want to avoid that extra step in order to make building the software
easier and faster.
Examples are:
* man-pages built via Sphinx, produced in ``docs.git`` and consumed in ``exchange.git``.
* SPAs used by the exchange, merchant libeufin.
* (formerly): The ``taler-wallet-core-embedded.js`` file used
by the Android repo.
Another type of source-level dependency is on other **source** files.
Examples for this are:
* The ``build-common.git`` repository that contains common build-time helpers
* The ``gana.git`` repository
Requirements
============
* We are a free software project, users must always be able to
easily re-build the prebuilt artifacts for themselves.
* We want to reduce reliance on third party infrastructure as much
as possible.
* We want to keep our own infrastructure as slim as possible.
* Builds must be reproducible and remain reproducible in the future.
* Offline builds should be supported as well as possible.
* Ideally, prebuilt artifacts can easily be cached / archived by third parties.
Proposed Solution
=================
Instead of using a full-blown artifact management solution from the start, we
make use of Git. Since Git is not very good at managing binary artifacts,
prebuilt files are not managed alongside the source code. Instead, we use
(orphan-)branches) in a (possibly separate) repository to manage them. This
allows us to re-use Git authentication and Git commit signing.
Due to a historical accident, prebuilt files are currently stored in the
``prebuilt`` branch of the ``wallet-core.git`` repository. This might change
in the future.
To enable forwards-compatibility, we are implementing the following approach:
The ``taler.net`` HTTP server redirects requests in the form of
``https://artifacts.taler.net/$component/$version/$file`` to
``https://git.taler.net/$repo.git/tree/$version/$file?h=$branch``, where
``$component`` determines ``$repo`` and ``$branch``.
We are also introducing the new rule that a prebuilt artifact must be a single
archive file (preferable ``.tar.gz.``).
Repositories that produce artifacts *MUST* have phony Makefile helper targets
to (a) create the prebuilt artifact (b) check out the prebuilt branch as a Git worktree, (c)
move the artifact to the right place in the prebuilt branch and commit/push.
Repositories that depend on prebuilt components must download them
in their ``./bootstrap`` script. After downloading, the bootstrap
script *SHOULD* verify a checksum of the downloaded artifact.
FIXME: Link to an example bootstrap file, once the approach from this document
is fully implemented.
Alternatives
============
* The prebuilt branch could be integrated into source repositories that
depend on it via a git submodule. This has the following disadvantages:
1. Git submodules are difficult to handle for some developers. We've had issues
in the past where the prebuilt submodule wasn't integrated properly and would
lead to non-reproducible builds
2. Relying purely on the Git commit hash of the prebuilt branch
makes it very difficult to know which version of the prebuilt
artifact is being used. It also makes it very difficult
to reliable find older versions of the artifact.
3. Relying on the commit hash instead of a version and artifact hash
couples the artifact to the Git commit. That means that
we can never compact the commit history of the prebuilt
branch of move artifacts around.
* Set up a full-blown artifact management solution like JFrog Artifactory.
That's costly and requires a lot of admin work.
* Never rely on prebuilt files. That slows down the build process
and in some cases requires exotic dependencies.
Drawbacks
=========
Discussion / Q&A
================
(This should be filled in with results from discussions on mailing lists / personal communication.)
|
