diff options
Diffstat (limited to 'design-documents/044-ci-system.rst')
-rw-r--r-- | design-documents/044-ci-system.rst | 130 |
1 files changed, 130 insertions, 0 deletions
diff --git a/design-documents/044-ci-system.rst b/design-documents/044-ci-system.rst new file mode 100644 index 00000000..e38632c0 --- /dev/null +++ b/design-documents/044-ci-system.rst @@ -0,0 +1,130 @@ +DD 44: CI System +################ + +Summary +======= + +This documents describes Taler's CI system based on Buildbot. + +This document uses `RFC 2119 <https://tools.ietf.org/html/rfc2119>`_ +keywords throughout. + +Motivation +========== + +With the current CI system there are an array of issues: + +- Central place for all the jobs. +- The central config is poorly organized. +- We should prefer to keep as much CI logic in respective project source repos + as possible. +- Jobs should be split up further to allow for more granular control and + insight. +- Job triggers are unclear. +- The build environments are mutable. +- Non-trivial and error-prone to keep track of environment state. +- Hard to get an overview of what repo is causing a failure, at a quick glance. +- Bad for development workflow on a single project when you are getting + false-negatives all the time. + +Proposed Solution +================= + +General +------- + +Jobs shall be executed inside of containers. + +One build pipeline (aka. "builder") per repo. + +Build steps are generated from directory structure within a given repo. + +Example directory structure: + +:: + + contrib + └── ci + ├── ci.sh + ├── Containerfile + └── jobs + ├── 0-codespell + │ ├── config.ini + │ ├── dictionary.txt + │ └── job.sh + ├── 1-build + │ ├── build.sh + │ └── job.sh + └── 2-docs + ├── docs.sh + └── job.sh + +Job directories **MUST** follow this pattern: +``<repo_root>/contrib/ci/jobs/<n-job_name>/`` + +``n`` is an integer used for ordering the build steps. + +Job directories **MUST** contain a script named ``job.sh`` which **MAY** +execute other scripts. + +Config files may optionally be created, and MUST be named ``config.ini`` and +placed in the job directory. + +Available config options: + +:: + + [build] + HALT_ON_FAILURE = True|False + WARN_ON_FAILURE = True|False + CONTAINER_BUILD = True|False + CONTAINER_NAME = <string> + CONTAINER_ARCH = <string> + + +Unless *all* jobs specify a "CONTAINER_NAME" in their custom config a +container file **MUST** be present at ``<repo_root>/contrib/ci/Containerfile``. +The container file will be built and used to run all of a repo's jobs +by default. + +All projects SHOULD have a ``build`` step and a ``test`` step, at a minimum. + +Running CI Locally +------------------ + +Running the CI scripts locally can be useful for development and testing. + +Included in each CI directory is a script which simplifies running jobs +in the same way the CI Worker does, in containers, using ``podman``. + +:: + + # Usage: + ./contrib/ci/ci.sh <job-name> + + # For example, if the CI jobs tree looks like this: + ./contrib/ci/jobs + ├── 0-codespell/ + ├── 1-build/ + ├── 2-test/ + ├── 3-docs/ + ├── 4-deb-package/ + └── 5-deploy-package/ + + # Then you can run job '0-codespell' as follows: + ./contrib/ci/ci.sh 0-codespell + + # If you are using podman and have "qemu-user-binfmt" installed + # then you may attempt to run any job under an alternative CPU + # architecture by providing a second argument. + # For example: + ./contrib/ci/ci.sh 0-codespell arm64 + + +Additional Builders +------------------- + +To run some tests there is a need for many or most project's sourcecode to be +available in the same environment. This will be a separate builder/pipeline +from the per-repo builders. Triggers for this builder are yet to be +determined. |