taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

044-ci-system.rst (4113B)

      1 DD 44: CI System
      2 ################
      3 
      4 :Design status: Accepted
      5 :Implementation status: Implemented
      6 :DD shepherd: TBD
      7 :Historical contributors: Özgür Kesim, Devan Carpenter, Christian Grothoff
      8 :First published: 2023-07-13
      9 :Last substantive change: 2023-12-17
     10 :Implementation evidence: exchange (2023-10-05), merchant (2023-10-12), taler-typescript-core (2024-01-15), libeufin (2024-02-18)
     11 :Normative references: :doc:`../developer/taler-developer-manual`
     12 
     13 .. note::
     14 
     15    Repository-local CI definitions are the current operational source of
     16    truth.  Cross-project builder triggering remains unspecified below.
     17 
     18 Summary
     19 =======
     20 
     21 This documents describes Taler's CI system based on Buildbot.
     22 
     23 This document uses `RFC 2119 <https://tools.ietf.org/html/rfc2119>`_
     24 keywords throughout.
     25 
     26 Motivation
     27 ==========
     28 
     29 With the current CI system there are an array of issues:
     30 
     31 - Central place for all the jobs.
     32 - The central config is poorly organized.
     33 - We should prefer to keep as much CI logic in respective project source repos
     34   as possible.
     35 - Jobs should be split up further to allow for more granular control and
     36   insight.
     37 - Job triggers are unclear.
     38 - The build environments are mutable.
     39 - Non-trivial and error-prone to keep track of environment state.
     40 - Hard to get an overview of what repo is causing a failure, at a quick glance.
     41 - Bad for development workflow on a single project when you are getting
     42   false-negatives all the time.
     43 
     44 Proposed Solution
     45 =================
     46 
     47 General
     48 -------
     49 
     50 Jobs shall be executed inside of containers.
     51 
     52 One build pipeline (aka. "builder") per repo.
     53 
     54 Build steps are generated from directory structure within a given repo.
     55 
     56 Example directory structure:
     57 
     58 ::
     59 
     60   contrib
     61   └── ci
     62       ├── ci.sh
     63       ├── Containerfile
     64       └── jobs
     65           ├── 0-codespell
     66           │   ├── config.ini
     67           │   ├── dictionary.txt
     68           │   └── job.sh
     69           ├── 1-build
     70           │   ├── build.sh
     71           │   └── job.sh
     72           └── 2-docs
     73               ├── docs.sh
     74               └── job.sh
     75 
     76 Job directories **MUST** follow this pattern:
     77 ``<repo_root>/contrib/ci/jobs/<n-job_name>/``
     78 
     79 ``n`` is an integer used for ordering the build steps.
     80 
     81 Job directories **MUST** contain a script named ``job.sh`` which **MAY**
     82 execute other scripts.
     83 
     84 Config files may optionally be created, and MUST be named ``config.ini`` and
     85 placed in the job directory.
     86 
     87 Available config options:
     88 
     89 ::
     90 
     91   [build]
     92   HALT_ON_FAILURE = True|False
     93   WARN_ON_FAILURE = True|False
     94   CONTAINER_BUILD = True|False
     95   CONTAINER_NAME = <string>
     96   CONTAINER_ARCH = <string>
     97 
     98 
     99 Unless *all* jobs specify a "CONTAINER_NAME" in their custom config a
    100 container file **MUST** be present at ``<repo_root>/contrib/ci/Containerfile``.
    101 The container file will be built and used to run all of a repo's jobs
    102 by default.
    103 
    104 All projects SHOULD have a ``build`` step and a ``test`` step, at a minimum.
    105 
    106 Running CI Locally
    107 ------------------
    108 
    109 Running the CI scripts locally can be useful for development and testing.
    110 
    111 Included in each CI directory is a script which simplifies running jobs
    112 in the same way the CI Worker does, in containers, using ``podman``.
    113 
    114 ::
    115 
    116   # Usage:
    117   ./contrib/ci/ci.sh <job-name>
    118 
    119   # For example, if the CI jobs tree looks like this:
    120   ./contrib/ci/jobs
    121     ├── 0-codespell/
    122     ├── 1-build/
    123     ├── 2-test/
    124     ├── 3-docs/
    125     ├── 4-deb-package/
    126     └── 5-deploy-package/
    127 
    128   # Then you can run job '0-codespell' as follows:
    129   ./contrib/ci/ci.sh 0-codespell
    130 
    131   # If you are using podman and have "qemu-user-binfmt" installed
    132   # then you may attempt to run any job under an alternative CPU
    133   # architecture by providing a second argument.
    134   # For example:
    135   ./contrib/ci/ci.sh 0-codespell arm64
    136 
    137 
    138 Additional Builders
    139 -------------------
    140 
    141 To run some tests there is a need for many or most project's sourcecode to be
    142 available in the same environment. This will be a separate builder/pipeline
    143 from the per-repo builders. Triggers for this builder are yet to be
    144 determined.