taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

commit d26d53b414f30d132b022302f5681463652185e4
parent fa4a7b2502085832bd3dd527d019ed40fbc4f1c8
Author: Özgür Kesim <oec-taler@kesim.org>
Date:   Tue, 30 Dec 2025 14:42:35 +0100

refine structure; add developer section

Diffstat:
Adeveloper/index.rst | 31+++++++++++++++++++++++++++++++
Adeveloper/taler-developer-manual.rst | 1984+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adeveloper/taler-wallet-developer.rst | 663+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Mindex.rst | 29++++++++++++++++-------------
Dtaler-developer-manual.rst | 1984-------------------------------------------------------------------------------
Dtaler-wallet.rst | 657-------------------------------------------------------------------------------
6 files changed, 2694 insertions(+), 2654 deletions(-)

diff --git a/developer/index.rst b/developer/index.rst @@ -0,0 +1,31 @@ +.. + This file is part of GNU TALER. + Copyright (C) 2025 Taler Systems SA + + TALER is free software; you can redistribute it and/or modify it under the + terms of the GNU Affero General Public License as published by the Free Software + Foundation; either version 2.1, or (at your option) any later version. + + TALER is distributed in the hope that it will be useful, but WITHOUT ANY + WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR + A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License along with + TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> + + @author Özgür Kesim + + +Developer Manuals +####################### + +Developers of the various parts of GNU Taler, such as the code for the exchange, +merchant backend, various single-page-application (SPA) frontends +and the wallet find here helpful documentation. + +.. toctree:: + :maxdepth: 1 + :glob: + + taler-developer-manual + taler-wallet-developer diff --git a/developer/taler-developer-manual.rst b/developer/taler-developer-manual.rst @@ -0,0 +1,1984 @@ +.. + This file is part of GNU TALER. + + Copyright (C) 2014-2025 Taler Systems SA + + TALER is free software; you can redistribute it and/or modify it under the + terms of the GNU Affero General Public License as published by the Free Software + Foundation; either version 2.1, or (at your option) any later version. + + TALER is distributed in the hope that it will be useful, but WITHOUT ANY + WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR + A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License along with + TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> + + @author Christian Grothoff + +General Developer Manual +######################## + +.. note:: + + This manual contains information for developers working on GNU Taler + and related components. It is not intended for a general audience. + + +Project Overview +================ + +GNU Taler consists of a large (and growing) number of components +in various Git repositories. The following list gives a first +overview: + +* exchange: core payment processing logic with a REST API, plus various + helper processes for interaction with banks and cryptographic + computations. Also includes the logic for the auditor and an + in-memory "bank" API implementation for testing. + +* libeufin: implementation of the "bank" API using the EBICS protocol + used by banks in the EU. Allows an exchange to interact with + European banks. + +* taler-magnet-bank: implementation of the "bank" API using the Magnet Bank + API.Allows an exchange to interact with Magnet Bank. + +* taler-cyclos: implementation of the "bank" API using the Cyclos API. Allows an exchange to interact with a Cyclos network. + +* depolymerization: implementation of the "bank" API on top of + blockchains, specifically Bitcoin and Ethereum. Allows an exchange + to interact with crypto-currencies. + +* merchant: payment processing backend to be run by merchants, + offering a REST API. + +* wallet-core: platform-independent implementation of a wallet to be run by + normal users. Includes also the WebExtension for various browsers. + Furthermore, includes various single-page apps used by other + components (especially as libeufin and merchant). Also includes + command-line wallet and tools for testing. + +* taler-android: Android Apps including the Android wallet, the + Android point-of-sale App and the Android casher app. + +* taler-ios: iOS wallet App. + +* sync: backup service, provides a simple REST API to allow users to + make encrypted backups of their wallet state. + +* anastasis: key escrow service, provides a simple REST API to allow + users to distribute encryption keys across multiple providers and + define authorization policies for key recovery. + +* taler-mdb: integration of Taler with the multi-drop-bus (MDB) API + used by vending machines. Allows Taler payments to be integrated + with vending machines. + +* gnu-taler-payment-for-woocommerce: payment plugin for the + woocommerce (wordpress) E-commerce solution. + +* twister: man-in-the-middle proxy for tests that require fuzzing a + REST/JSON protocol. Used for some of our testing. + +* challenger: implementation of an OAuth 2.0 provider that can be used + to verify that a user can receive SMS or E-mail at particular addresses. + Used as part of KYC processes of the exchange. + +* taler-mailbox: messaging service used to store and forward payment + messages to Taler wallets. + +* taldir: directory service used to lookup Taler wallet addresses for + sending invoices or payments to other wallets. + +* taler-merchant-demos: various demonstration services operated at + 'demo.taler.net', including a simple shop and a donation page. + +There are other important repositories without code, including: + +* gana: Hosted on git.gnunet.org, this repository defines various + constants used in the GNU Taler project. + +* docs: documentation, including this very document. + +* marketing: various presentations, papers and other resources for + outreach. + +* large-media: very large data objects, such as videos. + +* www: the taler.net website. + +Fundamentals +============ + +Versioning +---------- + +A central rule is to never break anything for any dependency. To accomplish +this, we use versioning, of the APIs, database schema and the protocol. The +database versioning approach is described in the :ref:`Database schema +versioning <DatabaseVersioning>` section. Here, we will focus on API and +protocol versioning. + +The key issue we need to solve with protocols and APIs (and that does not +apply to database versioning) is being able to introduce and remove features +without requiring a flag day where all components must update at the same +time. For this, we use GNU libtool style versioning with MAJOR:REVISION:AGE +and *not* semantic versioning (SEMVER). With GNU libtool style versioning, +first the REVISION should be increased on every change to the respective code. +Then, each time a feature is introduced or deprecated, the MAJOR and AGE +numbers are increased. Whenever an API is actually removed the AGE number is +reduced to match the distance since the removed API was deprecated. Thus, if +some client implements version X of the protocol (including not using any APIs +that have been deprecated), it is compatible for any implementation where +MAJOR is larger or equal to X, and MAJOR minus AGE is smaller or equal to X. +REVISION is not used for expected compatibility issues and merely serves to +uniquely identify each version (in combination with MAJOR). + +To evolve any implementation, it is thus critical to first of all never +just break an existing API or endpoint. The only acceptable modifications +are to return additional information (being aware of binary compatibility!) +or to accept additional optional arguments (again, in a way that does not +break existing users). Thus, the most common way to introduce changes will +be the addition of new endpoints. Breaking existing endpoints is only ever +at best acceptable while in the process of introducing it and if you are +absolutely sure that there are zero users in other components. + +When removing endpoints (or fields being returned), you must first deprecate +the existing API (incrementing MAJOR and AGE) and then wait for all clients, +including all clients in operation (e.g. Android and iOS Apps, e-commerce +integrations, etc.) to upgrade to a protocol implementation above the +deprecated MAJOR revision. Only then you should remove the endpoint and reduce +AGE. + +To document these changes, please try to use ``@since`` annotations in the API +specifications to explain the MAJOR revision when a feature became available, +but most importantly use ``@deprecated X`` annotations to indicate that an API +was deprecated and will be removed once MAJOR minus AGE is above X. When using +an API, use the ``/config`` endpoints to check for compatibility and show a +warning if the version(s) you support and the version(s) offered by the server +are incompatible. + + +Tagging and Package Versioning +------------------------------ + +Release tags are of the form ``v${major}.${minor}.${patch}``. Release tags *should* be +annotated git tags. + +We usually consider Debian packaging files (in ``debian/``) to be part of a release. +When only the Debian packaging files need to be changed, there are two options: + +* Make a new patch release (``v${major}.${minor}.${patch+1}``) +* Make a Debian release: + + * Debian version now includes a revision: ``${major}.${minor}.${patch}-${debrevision}`` + * The tag is Debian-specific: ``debian-${major}.${minor}.${patch}-${debrevision}`` + +All source repos *should* include a ``contrib/bump`` script that automates bumping the\ +version in all relevant source and packaging files. +In the future, we might add an option to the script to only release a packaging bump. +Right now, that process is manual. + +We support tagged and published pre-release versions via tags of the form ``v${major}.${minor}.${patch}-dev.${n}``. +The corresponding Debian version must be ``${major}.${minor}.${patch}~dev${n}``. + +Nightly Debian packages should follow the `Debian conventions <https://wiki.debian.org/Versioning>`__ of ``{upcoming_version}~git{date}.{hash}-{revision}``. + +Testing Tools +------------- + +For full ``make check`` support, install these programs: + +- `jq <https://github.com/stedolan/jq>`__ +- `curl <http://curl.haxx.se>`__ +- `faketime <https://github.com/wolfcw/libfaketime>`__ + +The ``make check`` should be able to function without them, but +their presence permits some tests to run that would otherwise be skipped. + +Manual Testing Database Reset +----------------------------- + +Sometimes ``make check`` will fail with some kind of database (SQL) +error, perhaps with a message like ``OBJECT does not exist`` in the +``test-suite.log`` file, where ``OBJECT`` is the name of a table or function. +In that case, it may be necessary to reset the ``talercheck`` database +with the commands: + +.. code-block:: console + + $ dropdb talercheck + $ createdb talercheck + +This is because, at the moment, there is no support for +doing these steps automatically in the ``make check`` flow. + +(If ``make check`` still fails after the reset, file a bug report as usual.) + +Bug Tracking +------------ + +Bug tracking is done with Mantis (https://www.mantisbt.org/). The bug tracker +is available at `<https://bugs.taler.net>`_. A registration on the Web site is +needed in order to use the bug tracker, only read access is granted without a +login. + +We use the following conventions for the bug states: + +* NEW: Incoming bugs are in 'new' so that management (or developers) + can easily identify those that need to be checked (report correct? + something we want to fix?), prioritized and targeted for releases. + "NEW" bugs are never assigned to a developer. + +* FEEDBACK: When blocked on feedback from reporter or other developer. + Assigned to other developer (but cannot be assigned to reporter, + in this case MAY remain associated with the developer who expects + the feedback). + +* ACKNOWLEDGED: The bug has been reviewed, but no decision about + what action to take has been made yet. Should not be worked on + until management (or a developer) comes up with a plan. + "ACKNOWLEDGED" bugs should NOT be assigned to a developer. + +* CONFIRMED: This is a real issue that should be worked on, but + is not yet actively worked on. If working on this bug requires + other bugs to be fixed first, they should be added as + child-bugs (via relationships). Developers are always welcome + to self-assign bugs that are "CONFIRMED" if they start to work + on a bug. "CONFIRMED" bugs should NOT be assigned to a developer. + +* ASSIGNED: The specific developer the bug is assigned to is + **actively** working on the issue. Developers should strive to + not have more than 5-10 bugs assigned to them at any time. + Only having one assigned to you is totally OK! + Developers should aggressively un-assign bugs that they are + blocked on, cannot make progress on, or are no longer actively + working on (but of course, better *resolve* them before + moving on if possible). If the bug remains open, it probably + should go back to "CONFIRMED" or "ACKNOWLEDGED". + +* RESOLVED: The bug has been fixed in Git. + +* CLOSED: An official release was made with the fix in it. + +When developers want to keep an eye on certain bugs, they should +**monitor** them. Multiple developers can be monitoring a bug, but +it can only be assigned to one. Developers should also keep an +eye on the roadmap (by release), bug categories they care about, +and of course priorities / severities. + +We use **tags** to categorize bugs. Common tags that also imply +some urgency include (in alphabetical order): + +* accounting: issues required for accounting (such as taxes by merchants) +* compliance: issues related to regulatory compliance +* $CUSTOMER: issues requested by a particular customer +* performance: performance problems or ideas for improvement +* security: security issues (including planned improvements to security) +* UX: user experience issues + +These tags **should** be attached to "NEW" bugs if they apply. + + +Code Repositories +----------------- + +Taler code is versioned with Git. For those users without write access, all the +codebases are found at the following URL: + +.. code-block:: none + + git://git.taler.net/<repository> + +A complete list of all the existing repositories is currently found at +`<https://git.taler.net/>`_. + + +Committing code +--------------- + +Before you can obtain Git write access, you must sign the copyright +agreement. As we collaborate closely with GNUnet, we use their +copyright agreement -- with the understanding that your contributions +to GNU Taler are included in the assignment. You can find the +agreement on the `GNUnet site <https://gnunet.org/en/copyright.html>`_. +Please sign and mail it to Christian Grothoff as he currently collects +all the documents for GNUnet e.V. + +To obtain Git access, you need to send us your SSH public key. Most core +team members have administrative Git access, so simply contact whoever +is your primary point of contact so far. You can +find instructions on how to generate an SSH key +in the `Git book <https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key>`_. +If you have been granted write access, you first of all must change the URL of +the respective repository to: + +.. code-block:: none + + ssh://git@git.taler.net/<repository> + +For an existing checkout, this can be done by editing the ``.git/config`` file. + +The server is configured to reject all commits that have not been signed with +GnuPG. If you do not yet have a GnuPG key, you must create one, as explained +in the `GNU Privacy Handbook <https://www.gnupg.org/gph/en/manual/c14.html>`_. +You do not need to share the respective public key with us to make commits. +However, we recommend that you upload it to key servers, put it on your +business card and personally meet with other GNU hackers to have it signed +such that others can verify your commits later. + +To sign all commits, you should run + +.. code-block:: console + + $ git config --global commit.gpgsign true + +You can also sign individual commits only by adding the ``-S`` option to the +``git commit`` command. If you accidentally already made commits but forgot +to sign them, you can retroactively add signatures using: + +.. code-block:: console + + $ git rebase -S + + +Whether you commit to a personal branch (recommended: ``dev/$USER/...``), +a feature branch or to ``master`` should +depend on your level of comfort and the nature of the change. As a general +rule, the code in ``master`` must always build and tests should always pass, at +least on your own system. However, we all make mistakes and you should expect +to receive friendly reminders if your change did not live up to this simple +standard. We plan to move to a system where the CI guarantees this invariant +in the future. + +In order to keep a linear and clean commits history, we advise to avoid +merge commits and instead always rebase your changes before pushing to +the ``master`` branch. If you commit and later find out that new commits were +pushed, the following command will pull the new commits and rebase yours +on top of them. + +.. code-block:: console + + # -S instructs Git to (re)sign your commits + $ git pull --rebase -S + + + +Observing changes +----------------- + +Every commit to the ``master`` branch of any of our public repositories +(and almost all are public) is automatically sent to the +gnunet-svn@gnu.org mailinglist. That list is for Git commits only, +and must not be used for discussions. It also carries commits from +our main dependencies, namely GNUnet and GNU libmicrohttpd. While +it can be high volume, the lists is a good way to follow overall +development. + + +Code generator usage policy +--------------------------- + +We do neither encourage nor discourage the use of tools for code generation. +It is up to the individual developer to decide if a tool is acceptable for +a particular task. But of course, we do encourage you to use FLOSS tools +and we MUST NOT become dependend on non-free software! That said, if you +use tools, you must document their use and in particular satisfy the +`NLnet policy on the use of "AI" <https://nlnet.nl/news/2025/20250829-policy-on-use-of-AI.html>`__. + +Specifically, we ask developers to always put generated code into a *separate* +Git commit and to include the full prompt in the commit message. Naturally, +you may clean up the code generator's output, but then you should do so in +separate Git commits (and of course only merge into master/stable after the +clean up is complete). But do preserve (not squash!) the commit with the +generated code so that it remains documented what the prompts were and which +code is generated. This will go a long way to keep code auditors sane! + + +Communication +------------- + +For public discussions we use the taler@gnu.org mailinglist. All developers +should subscribe to the low-volume Taler mailinglist. There are separate +low-volume mailinglists for gnunet-developers (@gnu.org) and for libmicrohttpd +(@gnu.org). For internal discussions we use https://mattermost.taler.net/ +(invitation only, but also achieved). + + +What to put in bootstrap +------------------------ + +Each repository has a ``bootstrap`` script, which contains commands for the +developer to run after a repository checkout (i.e., after ``git clone`` or +``git pull``). +Typically, this updates and initializes submodules, prepares the tool chain, +and runs ``autoreconf``. +The last step generates the ``configure`` script, whether for immediate use or +for inclusion in the distribution tarball. + +One common submodule is ``contrib/gana``, which pulls from the +`GNUnet GANA repository <https://git.gnunet.org/gana.git/>`__. +For example, in the +`Taler exchange repository <https://git.taler.net/exchange.git>`__, +the bootstrap script eventually runs the ``git submodule update --init`` command +early on, and later runs script ``./contrib/gana-generate.sh``, which +generates files such as ``src/include/taler_signatures.h``. + +Thus, to update that file, you need to: + +- (in GANA repo) Find a suitable (unused) name and number for the Signature + Purposes database. + +- Add it to GANA, in ``gnunet-signatures/registry.rec``. + (You can check for uniqueness with the ``recfix`` utility.) + +- Commit the change, and push it to the GANA Git repo. + +- (in Taler Repo) Run the ``contrib/gana-latest.sh`` script. + +- Bootstrap, configure, do ``make install``, ``make check``, etc. + (Basically, make sure the change does not break anything.) + +- Commit the submodule change, and push it to the Taler exchange Git repo. + +A similar procedure is required for other databases in GANA. +See file ``README`` in the various directories for specific instructions. + + +Debian and Ubuntu Repositories +============================== + +We package our software for Debian and Ubuntu. + +Nightly Repositories +-------------------- + +To try the latest, unstable and untested versions of packages, +you can add the nightly package sources. + +.. code-block:: shell-session + + # For Debian (trixie) + $ curl -sS https://deb.taler.net/apt-nightly/taler-trixie-ci.sources \ + | tee /etc/apt/sources.list.d/taler-trixie-nightly.sources + + +Taler Deployment on gv.taler.net +================================ + +This section describes the GNU Taler deployment on ``gv.taler.net``. ``gv`` +is our server at BFH. It hosts the Git repositories, Web sites, CI and other +services. Developers can receive an SSH account and e-mail alias for the +system, you should contact Javier, Christian or Florian. As with Git, ask +your primary team contact for shell access if you think you need it. + + +DNS +--- + +DNS records for taler.net are controlled by the GNU Taler maintainers, +specifically Christian and Florian, and our system administrator, Javier. If +you need a sub-domain to be added, please contact one of them. + + +User Acccounts +-------------- + +On ``gv.taler.net``, there are three system users that are set up to +serve Taler on the Internet: + +- ``head``: serves ``*.head.taler.net`` and gets automatically + built by Buildbot every 2 hours from the ``sandcastle-ng.git``. + Master key may be reset occasionally + +- ``taler-test``: serves ``*.test.taler.net`` and does *NOT* get + automatically built, and runs more recent tags and/or unreleased + versions of Taler components. Master key may be reset + occasionally. + +- ``demo``: serves ``*.demo.taler.net``. Never automatically built. + Master key is retained. + +Demo Upgrade Procedure +====================== + +#. Login as the ``demo`` user on ``gv.taler.net``. +#. Pull the latest ``sandcastle-ng.git`` code in checkout at ``$HOME/sandcastle-ng``. +#. Run ``systemctl --user restart container-taler-sandcastle-demo.service`` +#. Refer to the sandcastle-ng README (https://git.taler.net/sandcastle-ng.git/about/) + for more info. + + +Upgrading the ``demo`` environment should be done with care, and ideally be +coordinated on the mailing list before. It is our goal for ``demo`` to always +run a "working version" that is compatible with various published wallets. +Please use the :doc:`demo upgrade checklist <checklists/checklist-demo-upgrade>` to make +sure everything is working. +Nginx is already configured to reach the services as exported by the user unit. + + +Tagging components +------------------ + +All Taler components must be tagged with git before they are deployed on the +``demo`` environment, using a tag of the following form: + +.. code-block:: none + + demo-YYYY-MM-DD-SS + YYYY = year + MM = month + DD = day + SS = serial + +Environments and Builders on taler.net +====================================== + +Buildbot implementation +----------------------- + +GNU Taler uses a buildbot implementation (front end at https://buildbot.taler.net) to manage continuous integration. Buildbot documentation is at https://docs.buildbot.net/. + +Here are some highlights: + +- The WORKER is the config that that lives on a shell account on a localhost (taler.net), where this host has buildbot-worker installed. The WORKER executes the commands that perform all end-functions of buildbot. + +- The WORKER running buildbot-worker receives these commands by authenticating and communicating with the buildbot server using parameters that were specified when the worker was created in that shell account with the ``buildbot-worker`` command. + +- The buildbot server's master.cfg file contains FACTORY declarations which specify the commands that the WORKER will run on localhost. + +- The FACTORY is tied to the WORKER in master.cfg by a BUILDER. + +- The master.cfg also allows for SCHEDULER that defines how and when the BUILDER is executed. + +- Our master.cfg file is checked into git, and then periodically updated on a particular account on taler.net (ask Christian for access if needed). Do not edit this file directly/locally on taler.net, but check changes into Git. + + +Best Practices: + +- When creating a new WORKER in the ``master.cfg`` file, leave a comment specifying the server and user account that this WORKER is called from. (At this time, taler.net is the only server used by this implementation, but it's still good practice.) + +- Create a worker from a shell account with this command: ``buildbot-worker create-worker <workername> localhost <username> <password>`` + +Then make sure there is a WORKER defined in master.cfg like: ``worker.Worker("<username>", "<password>")`` + +Test builder +------------ + +This builder (``test-builder``) compiles and starts every Taler component. +The associated worker is run by the ``taler-test`` Gv user, via the SystemD +unit ``buildbot-worker-taler``. The following commands start/stop/restart +the worker: + +.. code-block:: + + systemctl --user start buildbot-worker-taler + systemctl --user stop buildbot-worker-taler + systemctl --user restart buildbot-worker-taler + +.. note:: + the mentioned unit file can be found at ``deployment.git/systemd-services/`` + +Wallet builder +-------------- + +This builder (``wallet-builder``) compiles every Taler component +and runs the wallet integration tests. The associated worker is +run by the ``walletbuilder`` Gv user, via the SystemD unit ``buildbot-worker-wallet``. +The following commands start/stop/restart the worker: + +.. code-block:: + + systemctl --user start buildbot-worker-wallet + systemctl --user stop buildbot-worker-wallet + systemctl --user restart buildbot-worker-wallet + +.. note:: + the mentioned unit file can be found at ``deployment.git/systemd-services/`` + +Documentation Builder +--------------------- + +All the Taler documentation is built by the user ``docbuilder`` that +runs a Buildbot worker. The following commands set the ``docbuilder`` up, +starting with an empty home directory. + +.. code-block:: console + + # Log-in as the 'docbuilder' user. + + $ cd $HOME + $ git clone git://git.taler.net/deployment + $ ./deployment/bootstrap-docbuilder + + # If the previous step worked, the setup is + # complete and the Buildbot worker can be started. + + $ buildbot-worker start worker/ + + +Website Builder +--------------- + + +Taler Websites, ``www.taler.net`` and ``stage.taler.net``, are built by the +user ``taler-websites`` by the means of a Buildbot worker. The following +commands set the ``taler-websites`` up, starting with an empty home directory. + +.. code-block:: console + + # Log-in as the 'taler-websites' user. + + $ cd $HOME + $ git clone git://git.taler.net/deployment + $ ./deployment/bootstrap-sitesbuilder + + # If the previous step worked, the setup is + # complete and the Buildbot worker can be started. + + $ buildbot-worker start worker/ + + +Code coverage +------------- + +Code coverage tests are run by the ``lcovworker`` user, and are also driven +by Buildbot. + +.. code-block:: console + + # Log-in as the 'lcovworker' user. + + $ cd $HOME + $ git clone git://git.taler.net/deployment + $ ./deployment/bootstrap-taler lcov + + # If the previous step worked, the setup is + # complete and the Buildbot worker can be started. + + $ buildbot-worker start worker/ + +The results are then published at ``https://lcov.taler.net/``. + +Producing auditor reports +------------------------- + +Both 'test' and 'demo' setups get their auditor reports compiled +by a Buildbot worker. The following steps get the reports compiler +prepared. + +.. code-block:: console + + # Log-in as <env>-auditor, with <env> being either 'test' or 'demo' + + $ git clone git://git.taler.net/deployment + $ ./deployment/buildbot/bootstrap-scripts/prepare-auditorreporter <env> + + # If the previous steps worked, then it should suffice to start + # the worker, with: + + $ buildbot-worker start worker/ + + +.. _DatabaseVersioning: + +Database schema versioning +-------------------------- + +The PostgreSQL databases of the exchange and the auditor are versioned. +See the ``versioning.sql`` file in the respective directory for documentation. + +Every set of changes to the database schema must be stored in a new +versioned SQL script. The scripts must have contiguous numbers. After +any release (or version being deployed to a production or staging +environment), existing scripts MUST be immutable. + +Developers and operators MUST NOT make changes to database schema +outside of this versioning. All tables of a GNU Taler component should live in their own schema. + + +QA Plans +======== + +.. include:: checklists/qa-1.0.rst + + +Releases +======== + +.. include:: checklists/checklist-release.rst + +Release Process +--------------- + +This document describes the process for releasing a new version of the +various Taler components to the official GNU mirrors. + +The following components are published on the GNU mirrors + +- taler-exchange (exchange.git) +- taler-merchant (merchant.git) +- sync (sync.git) +- taler-mdb (taler-mdb.git) +- libeufin (libeufin.git) +- challenger (challenger.git) +- wallet-core (wallet-core.git) + +Tagging +------- + +Tag releases with an **annotated** commit, like + +.. code-block:: console + + $ git tag -a v0.1.0 -m "Official release v0.1.0" + $ git push origin v0.1.0 + + +Database for tests +------------------ + +For tests in the exchange and merchant to run, make sure that a database +*talercheck* is accessible by *$USER*. Otherwise tests involving the +database logic are skipped. + +.. include:: frags/db-stores-sensitive-data.rst + +Exchange, merchant +------------------ + +Set the version in ``configure.ac``. The commit being tagged should be +the change of the version. + +Tag the current GANA version that works with the exchange and merchant and +checkout that tag of gana.git (instead of master). Otherwise, if there are +incompatible changes in GANA (like removed symbols), old builds could break. + +Update the Texinfo documentation using the files from docs.git: + +.. code-block:: console + + # Get the latest documentation repository + $ cd $GIT/docs + $ git pull + $ make texinfo + # The *.texi files are now in _build/texinfo + # + # This checks out the prebuilt branch in the prebuilt directory + $ git worktree add prebuilt prebuilt + $ cd prebuilt + # Copy the pre-built documentation into the prebuilt directory + $ cp -r ../_build/texinfo . + # Push and commit to branch + $ git commit -a -S -m "updating texinfo" + $ git status + # Verify that all files that should be tracked are tracked, + # new files will have to be added to the Makefile.am in + # exchange.git as well! + $ git push + # Remember $REVISION of commit + # + # Go to exchange + $ cd $GIT/exchange/doc/prebuilt + # Update submodule to point to latest commit + $ git checkout $REVISION + +Finally, the Automake ``Makefile.am`` files may have to be adjusted to +include new ``*.texi`` files or images. + +For bootstrap, you will need to install +`GNU Recutils <https://www.gnu.org/software/recutils/>`_. + +For the exchange test cases to pass, ``make install`` must be run first. +Without it, test cases will fail because plugins can't be located. + +.. code-block:: console + + $ ./bootstrap + $ ./configure # add required options for your system + $ make dist + $ tar -xf taler-$COMPONENT-$VERSION.tar.gz + $ cd taler-$COMPONENT-$VERSION + $ make install check + +Wallet WebExtension +------------------- + +The version of the wallet is in *manifest.json*. The ``version_name`` +should be adjusted, and *version* should be increased independently on +every upload to the WebStore. + +.. code-block:: console + + $ ./configure + $ make dist + +Upload to GNU mirrors +--------------------- + +See https://www.gnu.org/prep/maintain/maintain.html#Automated-FTP-Uploads + +Directive file: + +.. code-block:: none + + version: 1.2 + directory: taler + filename: taler-exchange-0.1.0.tar.gz + symlink: taler-exchange-0.1.0.tar.gz taler-exchange-latest.tar.gz + +Upload the files in **binary mode** to the ftp servers. + + +Creating Debian packages +------------------------ + +Our general setup is based on +https://wiki.debian.org/DebianRepository/SetupWithReprepro + +First, update at least the version of the Debian package in +debian/changelog, and then run: + +.. code-block:: bash + + $ dpkg-buildpackage -rfakeroot -b -uc -us + +in the respective source directory (GNUnet, exchange, merchant) to create the +``.deb`` files. Note that they will be created in the parent directory. This +can be done on gv.taler.net, or on another (secure) machine. +Actual release builds should be done via the Docker images +that can be found in ``deployment.git`` under packaging. + +On ``gv``, we use the ``aptbuilder`` user to manage the reprepro repository. + +Next, the ``*.deb`` files should be copied to gv.taler.net, say to +``/home/aptbuilder/incoming``. Then, run + +.. code-block:: bash + + # cd /home/aptbuilder/apt + # reprepro includedeb bullseye ~/incoming/*.deb + +to import all Debian files from ``~/incoming/`` into the ``bullseye`` +distribution. If Debian packages were build against other distributions, +reprepro may need to be first configured for those and the import command +updated accordingly. + +Finally, make sure to clean up ``~/incoming/`` (by deleting the +now imported ``*.deb`` files). + + + +Continuous integration +====================== + +CI is done with Buildbot (https://buildbot.net/), and builds are +triggered by the means of Git hooks. The results are published at +https://buildbot.taler.net/ . + +In order to avoid downtimes, CI uses a "blue/green" deployment +technique. In detail, there are two users building code on the system, +the "green" and the "blue" user; and at any given time, one is running +Taler services and the other one is either building the code or waiting +for that. + +There is also the possibility to trigger builds manually, but this is +only reserved to "admin" users. + + +Internationalisation +==================== + +Internationalisation (a.k.a "translation") is handled with Weblate (https://weblate.org) via our instance at https://weblate.taler.net/ organised as so-called *strings* in a bunch of Git repositories. + +Who can register +---------------- + +At this time, anyone can register an account at https://weblate.taler.net/ to create translations. Registered users default to the **Users** and **Viewers** privilege levels. + +About privilege levels +---------------------- + +This is the breakdown of privilege levels in Weblate: + +* **Viewers** = Have access to translations and can view them but are not entitled to translate + +* **Users** = Can log in and perform translations (*applies to new users*) + +* **Reviewers** = Can contribute translations to existing *components* and review translations + +* **Managers** = Can create new *components* of existing *projects* + +* **Superusers** = Can create new *projects*, moreover they manage Weblate's settings and repository maintenance, can edit all user accounts, deactivate users, trigger commits from Weblate's local repositories with pushes to Git and perform backups of language files + +How to contribute with your translation +--------------------------------------- + +1 - Log into https://weblate.taler.net + +2 - Navigate to *Projects* and *Browse all projects* + +3 - Choose the *project* you wish to contribute to + +4 - Choose the *component* you wish to contribute to + +5 - Find the language you want to translate into + +6 - Find a phrase and translate it + +7 - While translating, take into consideration at each string to translate + +* the comments + +* the context + +* the string location in the source code + +* the "nearby strings" in Weblate as one of the sorting options on the translation form + +* "other languages" as another sorting option to compare strings in a way translatologists would do + +* if applicable, the history of string changes and the message ID or key of a string + +8 - **Always test your translations in the actual apps** after they have been pushed by the platform to the apps + +9 - Be aware of ambiguities and immediately **report inconsistencies or bugs** by using Mantis bug tickets + +10 - If you want to contribute your translation in a language which does not yet exist in a Taler component, navigate to the *component* you want to translate for and click "Start new translation" to begin. In case you require a privilege upgrade, please contact a *Superuser* with your request. Currently, *Superusers* are Christian, Florian, Martin, and Stefan. + +Translation standards and practices +----------------------------------- + +1 - *Developers*, Weblate *reviewers* and *managers* MUST look into **design documents** for the specific component or software project, like e.g. for the Taler wallets DD53 https://docs.taler.net/design-documents/053-wallet-ui.html and stick close to the documentation, especially the **text to use** and **text to avoid** + +2 - To every ambiguous string *developers* MUST add **context** or **comments** to the strings on Weblate + +3 - *Developers*, Weblate *reviewers* and *managers* SHOULD add **screenshots** where needed to the strings on Weblate + +4 - Weblate *managers* MUST take care of **checks** in the translations overview and keep the repositories consistent and up-to-date, t.i. to perform regularly **synchronisation** between Git and Weblate repositories and vice-versa + +5 - *Weblate admins* MUST set the **commit and push rules** to *manual* only, and when asked, set the license to *GPLv3 or later* + +6 - You MAY also wish to refer to https://docs.weblate.org/ and to the ``README files`` in components like web pages to find out more specific variables if prompted + +How to create a project +----------------------- + +The *GNU Taler* project is probably the correct project for most Weblate components and Taler-related translations. Please contact a *Superuser* if you need another project created. + +How to create a component +------------------------- + +Reference: https://docs.weblate.org/en/weblate-5.13.2/admin/projects.html#component-configuration + +In Weblate, a *component* is a subset of a *project* and each component contains N translations. A component is generally associated with a Git repository. + +To create a component, log into https://weblate.taler.net/ with your credentials for a *Manager* privilege level (or higher) and choose **+ Add** from the upper-right corner. + +What follows is a sort of Wizard. You can find detailed docs at https://docs.weblate.org/. Here are some important notes about connecting your component to the Taler Git repository: + +On the component level, choose *Operations* from the menu bar, select *Settings*, then add at the tab *Basic* these settings: + +* **Component name** - The distinctive name of the software project, e.g. ``Main web site`` + +* **Translation license** - Choose *GNU Affero General Public License v3.0 or Later* + +* **Contributor license agreement** - left empty as the *Translation license* defaults + +* **Source string bug reporting address** - This requires an email address of the Weblate admin or *Superuser* in charge or just ``languages@taler.net`` + +* **Priority** - Generally ``Medium`` + +* **Restricted component** - ``deactivated`` + +* **Share in projects** - ``deactivated`` + +* **Use as a glossary** - ``deactivated`` + +At the tab *Translation* settings are generally: + +* **Turn on suggestions** - ``activated`` + +* **Allow translation propagation** - ``activated`` + +* **Contribute to project translation memory** - ``activated`` + +* **Secondary language** - Leave empty (respectively ``--------`` ) + +At the tab *Version control* ( *https://weblate.taler.net/create/component/vcs/* ) settings require variables as follows: + +* **Version Control System** - Generally ``Git`` + +* **Source code repository** - Generally ``git+ssh://git@git.taler.net/<repositoryname>.git``, check with ``git remote -v`` + +* **Repository branch** - Choose the correct branch to draw from and commit to which is generally ``master`` + +* **Repository push URL** - Generally ``git+ssh://git@git.taler.net/<repositoryname>.git``, check with ``git remote -v`` + +* **Push branch** - Choose the correct branch to push to, this is generally ``master`` + +* **Repository browser** - Normally, this field is left empty. If you are required to have this field pointing to the www URL of the Git repo's file browser, it could be filled with ``https://git.taler.net/<repositoryname>.git/tree/{{filename}}?h={{branch}}#n{{line}}`` where ``<repositoryname>`` gets replaced but ``{{filename}}`` and other items in braces are actual variables in the string + +* **Push on commit** - ``deactivated`` + +* **Age of changes to commit** - 24 (hours after which any pending changes will be committed to the VCS) + +* **Merge style** - Generally ``Rebase``, in line with GNU Taler development procedures + +* **Lock on error** - ``activated`` + +At the tab *Commit messages* you can leave the defaults: + +* **Commit message when translating** has the most important message: +Translated using Weblate ({{ language_name }}) +Currently translated at {{ stats.translated_percent }}% ({{ stats.translated }} of {{ stats.all }} strings) +Translation: {{ project_name }}/{{ component_name }} +Translate-URL: {{ url }} + +At the tab *Files* + +* **File format** - Mostly ``gettext PO file`` for web pages, ``Android String Resource`` for Android app strings, ``XLIFF 1.2 translation file`` for iOS apps, or ``TermBase eXchange file`` for some of the glossaries + +* **Language filter** - Dependent on the respective folder structure of the project, for structures used with gettext PO files mostly ``locale/*/LC_MESSAGES/messages.po`` + +* **Language filter** - Generally ``^[^.]+$`` + +* **Source language** - ``English`` + +* **Monolingual translations / Monolingual base language file** - Leave empty + +* **Edit base file** - ``deactivated`` + +* **Intermediate language file** - Leave empty + +* **Template for new translations** - Dependent on the respective folder structure of the project, for structures used with gettext PO files ``locale/messages.pot`` + +* **Adding new translation** - Generally ``Create new language file`` + +* **Language code style** - Generally ``Default based on the file format`` + +* **Screenshot file mask** - Leave empty (unless we do not dispose of a dedicated screenshot folder with a path relative to repository root) + +GPG signing of translations +--------------------------- + +Weblate signs its GPG commits with the GPG key CD33CE35801462FA5EB0B695F2664BF474BFE502, and the corresponding public key can be found at https://weblate.taler.net/keys/. + +This means that contributions made through Weblate will not be signed with the individual contributor's key when they are checked into the Git repository but with Weblate's public key. + + + +iOS Apps +======== + +.. _Build-iOS-from-source: + +Building Taler Wallet for iOS from source +----------------------------------------- + +The GNU Taler Wallet iOS app is in +`the official Git repository <https://git.taler.net/taler-ios.git>`__. + +Compatibility +^^^^^^^^^^^^^ + +The minimum version of iOS supported is 15.0. +This app runs on all iPhone models at least as new as the iPhone 6S. + + +Building +^^^^^^^^ + +Before building the iOS wallet, you must first checkout the +`quickjs-tart repo <https://git.taler.net/quickjs-tart.git>`__ +and the +`wallet-core repo <https://git.taler.net/wallet-core.git>`__. + +Have all 3 local repos (wallet-core, quickjs-tart, and this one) adjacent at +the same level (e.g. in a "GNU_Taler" folder) +Taler.xcworkspace expects the QuickJS framework sub-project to be at +``../quickjs-tart/QuickJS-rt.xcodeproj``. + +Build wallet-core first: + +.. code-block:: shell-session + + $ cd wallet-core + $ make embedded + $ open packages/taler-wallet-embedded/dist + +then drag or move its product "taler-wallet-core-qjs.mjs" +into your quickjs-tart folder right at the top level. + +Open Taler.xcworkspace, and set scheme / target to Taler_Wallet. Build&run... + +Don't open QuickJS-rt.xcodeproj or TalerWallet.xcodeproj and build anything +there - all needed libraries and frameworks will be built automatically from +Taler.xcworkspace. + + +Android Apps +============ + +Android App Nightly Builds +-------------------------- + +There are currently three Android apps in +`the official Git repository <https://git.taler.net/taler-android.git>`__: + +* Wallet + [`CI <https://git.taler.net/taler-android.git/tree/wallet/.gitlab-ci.yml>`__] +* Merchant PoS Terminal + [`CI <https://git.taler.net/taler-android.git/tree/merchant-terminal/.gitlab-ci.yml>`__] +* Cashier + [`CI <https://git.taler.net/taler-android.git/tree/cashier/.gitlab-ci.yml>`__] + +Their git repositories are `mirrored at Gitlab <https://gitlab.com/gnu-taler/taler-android>`__ +to utilize their CI +and `F-Droid <https://f-droid.org>`_'s Gitlab integration +to `publish automatic nightly builds <https://f-droid.org/docs/Publishing_Nightly_Builds/>`_ +for each change on the ``master`` branch. + +All three apps publish their builds to the same F-Droid nightly repository +(which is stored as a git repository): +https://gitlab.com/gnu-taler/fdroid-repo-nightly + +You can download the APK files directly from that repository +or add it to the F-Droid app for automatic updates +by clicking the following link (on the phone that has F-Droid installed). + + `GNU Taler Nightly F-Droid Repository <fdroidrepos://gnu-taler.gitlab.io/fdroid-repo-nightly/fdroid/repo?fingerprint=55F8A24F97FAB7B0960016AF393B7E57E7A0B13C2D2D36BAC50E1205923A7843>`_ + +.. note:: + Nightly apps can be installed alongside official releases + and thus are meant **only for testing purposes**. + Use at your own risk! + +.. _Build-apps-from-source: + +Building apps from source +------------------------- + +Note that this guide is different from other guides for building Android apps, +because it does not require you to run non-free software. +It uses the Merchant PoS Terminal as an example, but works as well for the other apps +if you replace ``merchant-terminal`` with ``wallet`` or ``cashier``. + +First, ensure that you have the required dependencies installed: + +* Java Development Kit 8 or higher (default-jdk-headless) +* git +* unzip + +Then you can get the app's source code using git: + +.. code-block:: console + + # Start by cloning the Android git repository + $ git clone https://git.taler.net/taler-android.git + + # Change into the directory of the cloned repository + $ cd taler-android + + # Find out which Android SDK version you will need + $ grep -i compileSdkVersion merchant-terminal/build.gradle + +The last command will return something like ``compileSdkVersion 29``. +So visit the `Android Rebuilds <http://android-rebuilds.beuc.net/>`_ project +and look for that version of the Android SDK there. +If the SDK version is not yet available as a free rebuild, +you can try to lower the ``compileSdkVersion`` in the app's ``merchant-terminal/build.gradle`` file. +Note that this might break things +or require you to also lower other versions such as ``targetSdkVersion``. + +In our example, the version is ``29`` which is available, +so download the "SDK Platform" package of "Android 10.0.0 (API 29)" +and unpack it: + +.. code-block:: console + + # Change into the directory that contains your downloaded SDK + $ cd $HOME + + # Unpack/extract the Android SDK + $ unzip android-sdk_eng.10.0.0_r14_linux-x86.zip + + # Tell the build system where to find the SDK + $ export ANDROID_SDK_ROOT="$HOME/android-sdk_eng.10.0.0_r14_linux-x86" + + # Change into the directory of the cloned repository + $ cd taler-android + + # Build the merchant-terminal app + $ ./gradlew :merchant-terminal:assembleRelease + +If you get an error message complaining about build-tools + + > Failed to install the following Android SDK packages as some licences have not been accepted. + build-tools;29.0.3 Android SDK Build-Tools 29.0.3 + +you can try changing the ``buildToolsVersion`` in the app's ``merchant-terminal/build.gradle`` file +to the latest "Android SDK build tools" version supported by the Android Rebuilds project. + +After the build finished successfully, +you will find your APK in ``merchant-terminal/build/outputs/apk/release/``. + +Update translations +------------------- + +Translations are managed with Taler's weblate instance: +https://weblate.taler.net/projects/gnu-taler/ + +To update translations, enter the taler-android git repository +and ensure that the weblate remote exists: + +.. code-block:: console + + $ git config -l | grep weblate + +If it does not yet exist (empty output), you can add it like this: + +.. code-block:: console + + $ git remote add weblate https://weblate.taler.net/git/gnu-taler/wallet-android/ + +Then you can merge in translations commit from the weblate remote: + +.. code-block:: console + + # ensure you have latest version + $ git fetch weblate + + # merge in translation commits + $ git merge weblate/master + +Afterwards, build the entire project from source and test the UI +to ensure that no erroneous translations (missing placeholders) are breaking things. + +Release process +--------------- + +After extensive testing, the code making up a new release should get a signed git tag. +The current tag format is: + +* cashier-$VERSION +* pos-$VERSION +* wallet-$VERSION (where $VERSION has a v prefix) + +.. code-block:: console + + $ git tag -s $APP-$VERSION + +F-Droid +^^^^^^^ +Nightly builds get published automatically (see above) after pushing code to the official repo. +Actual releases get picked up by F-Droid's official repository via git tags. +So ensure that all releases get tagged properly. + +Some information for F-Droid official repository debugging: + +* Wallet: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.wallet.fdroid.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.wallet.fdroid/lastbuild>`__] +* Cashier: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.cashier.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.cashier/lastbuild>`__] +* PoS: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.merchantpos.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.merchantpos/lastbuild>`__] + +Google Play +^^^^^^^^^^^ +Google Play uploads are managed via `Fastlane <https://docs.fastlane.tools/getting-started/android/setup/>`__. +Before proceeding, ensure that this is properly set up +and that you have access to the Google Play API. + +It is important to have access to the signing keys and Google Play access keys +(JSON) and to ensure that the following environment variables are set +correctly and made available to Fastlane: + +.. code-block:: bash + + TALER_KEYSTORE_PATH= + TALER_KEYSTORE_PASS= + TALER_KEYSTORE_WALLET_ALIAS= + TALER_KEYSTORE_WALLET_PASS= + TALER_KEYSTORE_POS_ALIAS= + TALER_KEYSTORE_POS_PASS= + TALER_KEYSTORE_CASHIER_ALIAS= + TALER_KEYSTORE_CASHIER_PASS= + TALER_JSON_KEY_FILE= + +To release an app, enter into its respective folder and run fastlane: + +.. code-block:: console + + $ bundle exec fastlane + +Then select the deploy option. + +All uploads are going to the beta track by default. These can be promoted to +production later or immediately after upload if you feel daring. It is also +important to bump the version and build code with every release. + +.. _Code-coverage: + +Code Coverage +============= + +Code coverage is done with the Gcov / Lcov +(http://ltp.sourceforge.net/coverage/lcov.php) combo, and it is run +nightly (once a day) by a Buildbot worker. The coverage results are +then published at https://lcov.taler.net/ . + + +Coding Conventions +================== + +GNU Taler is developed primarily in C, Kotlin, Python, Swift and TypeScript. + +Components written in C +----------------------- + +These are the general coding style rules for Taler. + +* Baseline rules are to follow GNU guidelines, modified or extended + by the GNUnet style: https://docs.gnunet.org/handbook/gnunet.html#Coding-style + +Naming conventions +^^^^^^^^^^^^^^^^^^ + +* include files (very similar to GNUnet): + + * if installed, must start with "``taler_``" (exception: platform.h), + and MUST live in src/include/ + * if NOT installed, must NOT start with "``taler_``" and + MUST NOT live in src/include/ and + SHOULD NOT be included from outside of their own directory + * end in "_lib" for "simple" libraries + * end in "_plugin" for plugins + * end in "_service" for libraries accessing a service, i.e. the exchange + +* binaries: + + * taler-exchange-xxx: exchange programs + * taler-merchant-xxx: merchant programs (demos) + * taler-wallet-xxx: wallet programs + * plugins should be libtaler_plugin_xxx_yyy.so: plugin yyy for API xxx + * libtalerxxx: library for API xxx + +* logging + + * tools use their full name in GNUNET_log_setup + (i.e. 'taler-exchange-offline') and log using plain 'GNUNET_log'. + * pure libraries (without associated service) use 'GNUNET_log_from' + with the component set to their library name (without lib or '.so'), + which should also be their directory name (i.e. 'util') + * plugin libraries (without associated service) use 'GNUNET_log_from' + with the component set to their type and plugin name (without lib or '.so'), + which should also be their directory name (i.e. 'exchangedb-postgres') + * libraries with associated service) use 'GNUNET_log_from' + with the name of the service, which should also be their + directory name (i.e. 'exchange') + * for tools with ``-l LOGFILE``, its absence means write logs to stderr + +* configuration + + * same rules as for GNUnet + +* exported symbols + + * must start with TALER_[SUBSYSTEMNAME]_ where SUBSYSTEMNAME + MUST match the subdirectory of src/ in which the symbol is defined + * from libtalerutil start just with ``TALER_``, without subsystemname + * if scope is ONE binary and symbols are not in a shared library, + use binary-specific prefix (such as TMH = taler-exchange-httpd) for + globals, possibly followed by the subsystem (TMH_DB_xxx). + +* structs: + + * structs that are 'packed' and do not contain pointers and are + thus suitable for hashing or similar operations are distinguished + by adding a "P" at the end of the name. (NEW) Note that this + convention does not hold for the GNUnet-structs (yet). + * structs that are used with a purpose for signatures, additionally + get an "S" at the end of the name. + +* private (library-internal) symbols (including structs and macros) + + * must not start with ``TALER_`` or any other prefix + +* testcases + + * must be called "test_module-under-test_case-description.c" + +* performance tests + + * must be called "perf_module-under-test_case-description.c" + +Shell Scripts +------------- + +Shell scripts should be avoided if at all possible. The only permissible uses of shell scripts +in GNU Taler are: + +* Trivial invocation of other commands. +* Scripts for compatibility (e.g. ``./configure``) that must run on + as many systems as possible. + +When shell scripts are used, they ``MUST`` begin with the following ``set`` command: + +.. code-block:: console + + # Make the shell fail on undefined variables and + # commands with non-zero exit status. + $ set -eu + +Kotlin +------ + +We so far have no specific guidelines, please follow best practices +for the language. + + +Python +------ + +Supported Python Versions +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Python code should be written and build against version 3.7 of Python. + +Style +^^^^^ + +We use `yapf <https://github.com/google/yapf>`_ to reformat the +code to conform to our style instructions. +A reusable yapf style file can be found in ``build-common``, +which is intended to be used as a git submodule. + +Python for Scripting +^^^^^^^^^^^^^^^^^^^^ + +When using Python for writing small utilities, the following libraries +are useful: + +* ``click`` for argument parsing (should be preferred over argparse) +* ``pathlib`` for path manipulation (part of the standard library) +* ``subprocess`` for "shelling out" to other programs. Prefer ``subprocess.run`` + over the older APIs. + + +Swift +----- + +Please follow best practices for the language. + + +TypeScript +---------- + +Please follow best practices for the language. + + +Testing library +=============== + +This chapter is a VERY ABSTRACT description of how testing is +implemented in Taler, and in NO WAY wants to substitute the reading of +the actual source code by the user. + +In Taler, a test case is an array of ``struct TALER_TESTING_Command``, +informally referred to as ``CMD``, that is iteratively executed by the +testing interpreter. This latter is transparently initiated by the +testing library. + +However, the developer does not have to defined CMDs manually, but +rather call the proper constructor provided by the library. For example, +if a CMD is supposed to test feature ``x``, then the library would +provide the ``TALER_TESTING_cmd_x ()`` constructor for it. Obviously, +each constructor has its own particular arguments that make sense to +test ``x``, and all constructor are thoroughly commented within the +source code. + +Internally, each CMD has two methods: ``run ()`` and ``cleanup ()``. The +former contains the main logic to test feature ``x``, whereas the latter +cleans the memory up after execution. + +In a test life, each CMD needs some internal state, made by values it +keeps in memory. Often, the test has to *share* those values with other +CMDs: for example, CMD1 may create some key material and CMD2 needs this +key material to encrypt data. + +The offering of internal values from CMD1 to CMD2 is made by *traits*. A +trait is a ``struct TALER_TESTING_Trait``, and each CMD contains an array +of traits, that it offers via the public trait interface to other +commands. The definition and filling of such array happens transparently +to the test developer. + +For example, the following example shows how CMD2 takes an amount object +offered by CMD1 via the trait interface. + +Note: the main interpreter and the most part of CMDs and traits are +hosted inside the exchange codebase, but nothing prevents the developer +from implementing new CMDs and traits within other codebases. + +.. code-block:: c + + /* Without loss of generality, let's consider the + * following logic to exist inside the run() method of CMD1 */ + ... + + struct TALER_Amount *a; + /** + * the second argument (0) points to the first amount object offered, + * in case multiple are available. + */ + if (GNUNET_OK != TALER_TESTING_get_trait_amount_obj (cmd2, 0, &a)) + return GNUNET_SYSERR; + ... + + use(a); /* 'a' points straight into the internal state of CMD2 */ + +In the Taler realm, there is also the possibility to alter the behaviour +of supposedly well-behaved components. This is needed when, for example, +we want the exchange to return some corrupted signature in order to +check if the merchant backend detects it. + +This alteration is accomplished by another service called *twister*. The +twister acts as a proxy between service A and B, and can be programmed +to tamper with the data exchanged by A and B. + +Please refer to the Twister codebase (under the ``test`` directory) in +order to see how to configure it. + + +User-Facing Terminology +======================= + +This section contains terminology that should be used and that should not be +used in the user interface and help materials. + +Terms to Avoid +-------------- + +Refreshing + Refreshing is the internal technical terminology for the protocol to + give change for partially spent coins + + **Use instead**: "Obtaining change" + +Charge + Charge has two opposite meanings (charge to a credit card vs. charge a battery). + This can confuse users. + + **Use instead**: "Obtain", "Credit", "Debit", "Withdraw", "Top up" + +Coin + Coins are an internal construct, the user should never be aware that their balance + is represented by coins of different denominations. + + **Use instead**: "(Digital) Cash" or "(Wallet) Balance" + +Consumer + Has bad connotation of consumption. + + **Use instead**: Customer or user. + +Proposal + The term used to describe the process of the merchant facilitating the download + of the signed contract terms for an order. + + **Avoid**. Generally events that relate to proposal downloads + should not be shown to normal users, only developers. Instead, use + "communication with mechant failed" if a proposed order can't be downloaded. + +Anonymous E-Cash + Should be generally avoided, since Taler is only anonymous for + the customer. Also some people are scared of anonymity (which as + a term is also way too absolute, as anonymity is hardly ever perfect). + + **Use instead**: "Privacy-preserving", "Privacy-friendly" + +Payment Replay + The process of proving to the merchant that the customer is entitled + to view a digital product again, as they already paid for it. + + **Use instead**: In the event history, "re-activated digital content purchase" + could be used. (FIXME: this is still not nice.) + +Session ID + See Payment Replay. + +Order + Too ambiguous in the wallet. + + **Use instead**: Purchase + +Fulfillment URL + URL that the serves the digital content that the user purchased + with their payment. Can also be something like a donation receipt. + +Donau + Developer-internal name for the tax authority component. + + **Use instead**: Tax authority + +Terms to Use +------------ + +Auditor + Regulatory entity that certifies exchanges and oversees their operation. + +Exchange Operator + The entity/service that gives out digital cash in exchange for some + other means of payment. + + In some contexts, using "Issuer" could also be appropriate. + When showing a balance breakdown, + we can say "100 Eur (issued by exchange.euro.taler.net)". + Sometimes we may also use the more generic term "Payment Service Provider" + when the concept of an "Exchange" is still unclear to the reader. + +Refund + A refund is given by a merchant to the customer (rather the customer's wallet) + and "undoes" a previous payment operation. + +Payment + The act of sending digital cash to a merchant to pay for an order. + +Purchase + Used to refer to the "result" of a payment, as in "view purchase". + Use sparsingly, as the word doesn't fit for all payments, such as donations. + +Contract Terms + Partially machine-readable representation of the merchant's obligation after the + customer makes a payment. + +Merchant + Party that receives a payment. + +Wallet + Also "Taler Wallet". Software component that manages the user's digital cash + and payments. + + +Developer Glossary +================== + +This glossary is meant for developers. It contains some terms that we usually do not +use when talking to end users or even system administrators. + +.. glossary:: + :sorted: + + absolute time + method of keeping time in :term:`GNUnet` where the time is represented + as the number of microseconds since 1.1.1970 (UNIX epoch). Called + absolute time in contrast to :term:`relative time`. + + aggregate + the :term:`exchange` combines multiple payments received by the + same :term:`merchant` into one larger :term:`wire transfer` to + the respective merchant's :term:`bank` account + + auditor + trusted third party that verifies that the :term:`exchange` is operating correctly + + bank + traditional financial service provider who offers + :term:`wire transfers <wire transfer>` between accounts + + buyer + individual in control of a Taler :term:`wallet`, usually using it to + :term:`spend` the :term:`coins <coin>` on :term:`contracts <contract>` (see also :term:`customer`). + + close + operation an :term:`exchange` performs on a :term:`reserve` that has not been + :term:`emptied <empty>` by :term:`withdraw` operations. When closing a reserve, the + exchange wires the remaining funds back to the customer, minus a :term:`fee` + for closing + + customer + individual that directs the buyer (perhaps the same individual) to make a purchase + + coin + coins are individual token representing a certain amount of value, also known as the :term:`denomination` of the coin + + refresh commitment + data that the wallet commits to during the :term:`melt` stage of the + :term:`refresh` protocol where it + has to prove to the :term:`exchange` that it is deriving the :term:`fresh` + coins as specified by the Taler protocol. The commitment is verified + probabilistically (see: :term:`kappa`) during the :term:`reveal` stage. + + contract + formal agreement between :term:`merchant` and :term:`customer` specifying the + :term:`contract terms` and signed by the merchant and the :term:`coins <coin>` of the + customer + + contract terms + the individual clauses specifying what the buyer is purchasing from the + :term:`merchant` + + denomination + unit of currency, specifies both the currency and the face value of a :term:`coin`, + as well as associated fees and validity periods + + denomination key + (RSA) key used by the exchange to certify that a given :term:`coin` is valid and of a + particular :term:`denomination` + + deposit + operation by which a merchant passes coins to an exchange, expecting the + exchange to credit his bank account in the future using an + :term:`aggregate` :term:`wire transfer` + + drain + process by which an exchange operator takes the profits + (from :term:`fees <fee>`) out of the escrow account and moves them into + their regular business account + + dirty + a :term:`coin` is dirty if its public key may be known to an entity other than + the customer, thereby creating the danger of some entity being able to + link multiple transactions of coin's owner if the coin is not refreshed + + empty + a :term:`reserve` is being emptied when a :term:`wallet` is using the + reserve's private key to :term:`withdraw` coins from it. This reduces + the balance of the reserve. Once the balance reaches zero, we say that + the reserve has been (fully) emptied. Reserves that are not emptied + (which is the normal process) are :term:`closed <close>` by the exchange. + + exchange + Taler's payment service operator. Issues electronic coins during + withdrawal and redeems them when they are deposited by merchants + + expired + Various operations come with time limits. In particular, denomination keys + come with strict time limits for the various operations involving the + coin issued under the denomination. The most important limit is the + deposit expiration, which specifies until when wallets are allowed to + use the coin in deposit or refreshing operations. There is also a "legal" + expiration, which specifies how long the exchange keeps records beyond the + deposit expiration time. This latter expiration matters for legal disputes + in courts and also creates an upper limit for refreshing operations on + special zombie coin + + GNUnet + Codebase of various libraries for a better Internet, some of which + GNU Taler depends upon. + + fakebank + implementation of the :term:`bank` API in memory to be used only for test + cases. + + fee + an :term:`exchange` charges various fees for its service. The different + fees are specified in the protocol. There are fees per coin for + :term:`withdrawing <withdraw>`, :term:`depositing <deposit>`, :term:`melting <melt>`, and + :term:`refunding <refund>`. Furthermore, there are fees per wire transfer + when a :term:`reserve` is :term:`closed <close>` + and for :term:`aggregate` :term:`wire transfers <wire transfer>` + to the :term:`merchant`. + + fresh + a :term:`coin` is fresh if its public key is only known to the customer + + JSON + JavaScript Object Notation (JSON) is a + serialization format derived from the JavaScript language which is + commonly used in the Taler protocol as the payload of HTTP requests + and responses. + + kappa + security parameter used in the :term:`refresh` protocol. Defined to be 3. + The probability of successfully evading the income transparency with the + refresh protocol is 1:kappa. + + libeufin + Kotlin component that implements a regional currency bank and an + adapter to communicate via EBICS with European core banking systems. + + link + specific step in the :term:`refresh` protocol that an exchange must offer + to prevent abuse of the :term:`refresh` mechanism. The link step is + not needed in normal operation, it just must be offered. + + master key + offline key used by the exchange to certify denomination keys and + message signing keys + + melt + step of the :term:`refresh` protocol where a :term:`dirty` :term:`coin` + is invalidated to be reborn :term:`fresh` in a subsequent + :term:`reveal` step. + + merchant + party receiving payments (usually in return for goods or services) + + message signing key + key used by the exchange to sign online messages, other than coins + + order + offer made by the merchant to a wallet; pre-cursor to + a contract where the wallet is not yet fixed. Turns + into a :term:`contract` when a wallet claims the order. + + owner + a coin is owned by the entity that knows the private key of the coin + + relative time + method of keeping time in :term:`GNUnet` where the time is represented + as a relative number of microseconds. Thus, a relative time specifies + an offset or a duration, but not a date. Called relative time in + contrast to :term:`absolute time`. + + recoup + Operation by which an exchange returns the value of coins affected + by a :term:`revocation <revoke>` to their :term:`owner`, either by allowing the owner to + withdraw new coins or wiring funds back to the bank account of the :term:`owner`. + + planchet + precursor data for a :term:`coin`. A planchet includes the coin's internal + secrets (coin private key, blinding factor), but lacks the RSA signature + of the :term:`exchange`. When :term:`withdrawing <withdraw>`, a :term:`wallet` + creates and persists a planchet before asking the exchange to sign it to + get the coin. + + purchase + Refers to the overall process of negotiating a :term:`contract` and then + making a payment with :term:`coins <coin>` to a :term:`merchant`. + + privacy policy + Statement of an operator how they will protect the privacy of users. + + proof + Message that cryptographically demonstrates that a particular claim is correct. + + proposal + a list of :term:`contract terms` that has been completed and signed by the + merchant backend. + + refresh + operation by which a :term:`dirty` :term:`coin` is converted into one or more + :term:`fresh` coins. Involves :term:`melting <melt>` the :term:`dirty` coins and + then :term:`revealing <reveal>` so-called :term:`transfer keys <transfer key>`. + + refund + operation by which a merchant steps back from the right to funds that he + obtained from a :term:`deposit` operation, giving the right to the funds back + to the customer + + refund transaction id + unique number by which a merchant identifies a :term:`refund`. Needed + as refunds can be partial and thus there could be multiple refunds for + the same :term:`purchase`. + + reserve + accounting mechanism used by the exchange to track customer funds + from incoming :term:`wire transfers <wire transfer>`. A reserve is created whenever + a customer wires money to the exchange using a well-formed public key + in the subject. The exchange then allows the customer's :term:`wallet` + to :term:`withdraw` up to the amount received in :term:`fresh` + :term:`coins <coin>` from the reserve, thereby emptying the reserve. If a + reserve is not emptied, the exchange will eventually :term:`close` it. + + Other definition: Funds set aside for future use; either the balance of a customer at the + exchange ready for withdrawal, or the funds kept in the exchange;s bank + account to cover obligations from coins in circulation. + + reveal + step in the :term:`refresh` protocol where some of the transfer private + keys are revealed to prove honest behavior on the part of the wallet. + In the reveal step, the exchange returns the signed :term:`fresh` coins. + + revoke + exceptional operation by which an exchange withdraws a denomination from + circulation, either because the signing key was compromised or because + the exchange is going out of operation; unspent coins of a revoked + denomination are subjected to recoup. + + sharing + users can share ownership of a :term:`coin` by sharing access to the coin&#39;s + private key, thereby allowing all co-owners to spend the coin at any + time. + + spend + operation by which a customer gives a merchant the right to deposit + coins in return for merchandise + + transfer key + special cryptographic key used in the :term:`refresh` protocol, some of which + are revealed during the :term:`reveal` step. Note that transfer keys have, + despite the name, no relationship to :term:`wire transfers <wire transfer>`. They merely + help to transfer the value from a :term:`dirty` coin to a :term:`fresh` coin + + terms + the general terms of service of an operator, possibly including + the :term:`privacy policy`. Not to be confused with the + :term:`contract terms` which are about the specific purchase. + + transaction + method by which ownership is exclusively transferred from one entity + + user + any individual using the Taler payment system + (see :term:`customer`, :term:`buyer`, :term:`merchant`). + + version + Taler uses various forms of versioning. There is a database + schema version (stored itself in the database, see \*-0000.sql) describing + the state of the table structure in the database of an :term:`exchange`, + :term:`auditor` or :term:`merchant`. There is a protocol + version (CURRENT:REVISION:AGE, see GNU libtool) which specifies + the network protocol spoken by an :term:`exchange` or :term:`merchant` + including backwards-compatibility. And finally there is the software + release version (MAJOR.MINOR.PATCH, see https://semver.org/) of + the respective code base. + + wallet + software running on a customer's computer; withdraws, stores and + spends coins + + WebExtension + Cross-browser API used to implement the GNU Taler wallet browser extension. + + wire gateway + API used by the exchange to talk with some real-time gross settlement system + (core banking system, blockchain) to notice inbound credits wire transfers + (during withdraw) and to trigger outbound debit wire transfers (primarily + for deposits). + + wire transfer + a wire transfer is a method of sending funds between :term:`bank` accounts + + wire transfer identifier + Subject of a wire transfer from the exchange to a merchant; + set by the aggregator to a random nonce which uniquely + identifies the transfer. + + withdraw + operation by which a :term:`wallet` can convert funds from a :term:`reserve` to + fresh coins + + zombie + :term:`coin` where the respective :term:`denomination key` is past its + :term:`deposit` :term:`expiration <expired>` time, but which is still (again) valid + for an operation because it was :term:`melted <melt>` while it was still + valid, and then later again credited during a :term:`recoup` process + + + +Developer Tools +=============== + +This section describes various internal programs to make life easier for the +developer. + + +taler-harness +------------- + +**taler-harness deployment gen-coin-config** is a tool to simplify Taler configuration generation. + + +**taler-harness deployment gen-coin-config** +[**-min-amount**=**\ ‌\ *VALUE*] +[**-max-amount**=**\ ‌\ *VALUE*] diff --git a/developer/taler-wallet-developer.rst b/developer/taler-wallet-developer.rst @@ -0,0 +1,663 @@ +.. + This file is part of GNU TALER. + Copyright (C) 2014-2024 Taler Systems SA + + TALER is free software; you can redistribute it and/or modify it under the + terms of the GNU Affero General Public License as published by the Free Software + Foundation; either version 2.1, or (at your option) any later version. + + TALER is distributed in the hope that it will be useful, but WITHOUT ANY + WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR + A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License along with + TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> + + +Wallet Developer Manual +####################### + +.. note:: + + This manual contains information for developers working on the wallet + component of GNU Taler. It is not intended for a general audience. + + +The GNU Taler wallet allows customers to withdraw and spend digital cash. + + +WebExtension Wallet +=================== + +Building from source +-------------------- + +.. code-block:: console + + $ git clone https://git.taler.net/wallet-core.git + $ cd wallet-core + $ ./configure + $ make webex-stable + # Packaged extension now available as: + # dist/taler-wallet-$VERSION.zip + + +Android Wallet +============== + +Please see :ref:`Build-apps-from-source` in the :doc:`taler-developer-manual`. + + +iOS Wallet +========== + +Please see :ref:`Build-iOS-from-source` in the :doc:`taler-developer-manual`. + +.. _command-line-wallet: + +Command-line Wallet +=================== + +This section describes how to use the GNU Taler wallet command line +interface (CLI). + +The the wallet CLI is targeted at developers and operators, but not meant to be +used by customers. It exposes all functionality that the more user-friendly +interfaces (Android app, browser extension) offer. However, it provides more +diagnostics and advanced features as well. + +Building from source +-------------------- + +The easiest way to install the wallet is via NPM. Note that a recent version of +Node.JS (``>=12.20.1``) is required. + +We recommend to install the wallet package on a per-user basis, +thus setting ``$INSTALL_PREFIX`` to a directory in ``$HOME``. + +.. code-block:: console + + $ git clone https://git.taler.net/wallet-core.git + $ cd wallet-core + $ ./bootstrap + $ ./configure --prefix=$INSTALL_PREFIX + $ make && make install + +The wallet command-line interface should then be available as ``taler-wallet-cli`` under ``$INSTALL_PREFIX/bin``. + +Installation via NPM +-------------------- + +The wallet can also obtained via NPM, the Node Package Manager. + +To install the wallet as a global package, run: + +.. code-block:: console + + $ npm install -g taler-wallet + # check if installation was successful + $ taler-wallet-cli --version + +To install the wallet only for your user, run: + +.. code-block:: console + + $ npm install -g --prefix=$HOME/local taler-wallet + # check if installation was successful + $ taler-wallet-cli --version + # If this fails, make sure that $HOME/local/bin is in your $PATH + +To use the wallet as a library in your own project, run: + +.. code-block:: console + + $ npm install taler-wallet + + +Getting Help +------------ + +The wallet CLI comes with built-in help. Invoke the wallet CLI (or any subcommand) with the ``--help`` flag to get help: + +.. code-block:: console + + $ taler-wallet-cli --help + Usage: taler-wallet-cli COMMAND + + Command line interface for the GNU Taler wallet. + + Options: + -h, --help Show this message and exit. + --wallet-db=VALUE location of the wallet database file + --timetravel=VALUE modify system time by given offset in microseconds + --inhibit=VALUE Inhibit running certain operations, useful for debugging and testing. + --no-throttle Don't do any request throttling. + -v, --version + -V, --verbose Enable verbose output. + + Commands: + advanced Subcommands for advanced operations (only use if you know what you're doing!). + api Call the wallet-core API directly. + backup Subcommands for backups + balance Show wallet balance. + deposit Subcommands for depositing money to payto:// accounts + exchanges Manage exchanges. + handle-uri Handle a taler:// URI. + pending Show pending operations. + run-pending Run pending operations. + run-until-done Run until no more work is left. + testing Subcommands for testing GNU Taler deployments. + transactions Show transactions. + +Completing operations +--------------------- + +Note that the CLI does not run as a background daemon. When starting +operations that don't immediately finish, the wallet needs to be run explicitly +to finish any pending tasks: + + +.. code-block:: console + + # Do one attempt to finish all pending operations + $ taler-wallet-cli run-pending + + # Run until all work is done + $ taler-wallet-cli run-until-done + +Resetting the wallet +-------------------- + +The wallet can be reset by deleting its database file. By default, the database file +is ``$HOME/.talerwalletdb.sqlite3``. + + +Handling taler:// URIs +---------------------- + +Many interactions with the Taler wallet happen by scanning QR codes or special +headers on Websites. To emulate this with the command line interface, run the following +command: + +.. code-block:: console + + $ taler-wallet-cli handle-uri $URI + + +Manual withdrawing +------------------ + +.. code-block:: console + + $ taler-wallet-cli advanced withdraw-manually \ + --exchange https://exchange.eurint.taler.net/ \ + --amount EUR:5 + + +P2P push payments +----------------- + +The following code generates a P2P push transaction over 1 CHF +with an expiration time of 30 days (assuming the wallet has a +sufficient balance): + +.. code-block:: console + + $ taler-wallet-cli p2p initiate-push-debit \ + --purse-expiration="30 d" \ + --summary="The summary" \ + CHF:1 + +The final URL can then be found in the transaction list: + +.. code-block:: console + + $ taler-wallet-cli transactions + +Background wallet +----------------- + +A wallet can be launched in the background: + +.. code-block:: console + + $ taler-wallet-cli advanced serve & + +You can then run various Taler operations faster against +this one persistent instance: + +.. code-block:: console + + $ taler-wallet-cli --wallet-connection=wallet-core.sock ... + +Here ``...`` needs to be changed to the commando to run. +Make sure to run + +.. code-block:: console + + $ taler-wallet-cli --wallet-connection=wallet-core.sock \ + run-until-done + +to wait for pending transactions to complete. + + +Testing an exchange deployment +------------------------------ + +The following series of commands can be used to check that an exchange deployment +is functional: + +.. code-block:: console + + # This will now output a payto URI that money needs to be sent to in order to allow withdrawal + # of taler coins + $ taler-wallet-cli advanced withdraw-manually --exchange $EXCHANGE_URL --amount EUR:10.50 + + # Show the status of the manual withdrawal operation + $ taler-wallet-cli transactions + + # Once the transfer has been made, try completing the withdrawal + $ taler-wallet-cli run-pending + + # Check status of transactions and show balance + $ taler-wallet-cli transactions + $ taler-wallet-cli balance + + # Now, directly deposit coins with the exchange into a target account + # (Usually, a payment is made via a merchant. The wallet provides + # this functionality for testing.) + $ taler-wallet-cli deposit create EUR:5 payto://iban/$IBAN + + # Check if transaction was successful. + # (If not, fix issue with exchange and run "run-pending" command again) + $ taler-wallet-cli transactions + + # The wallet can also track if the exchange wired the money to the merchant account. + # The "deposit group id" can be found in the output of the transactions list. + $ taler-wallet-cli deposit track $DEPOSIT_GROUP_ID + + +APIs and Data Formats +===================== + +Envelope Format +--------------- + +All API responses and notifications are returned in the +following envelope: + +.. ts:def:: WalletResponseEnvelope + + type WalletResponseEnvelope = + | WalletSuccess + | WalletError + | WalletNotification + +.. ts:def:: WalletSuccess + + export interface WalletSuccess { + type: "response"; + operation: string; + // ID to correlate success response to request + id: string; + // Result type depends on operation + result: unknown; + } + +.. ts:def:: WalletError + + export interface WalletError { + type: "error"; + operation: string; + // ID to correlate error response to request + id: string; + error: WalletErrorInfo; + } + +.. ts:def:: WalletNotification + + export interface WalletSuccess { + type: "notification"; + + // actual type is WalletNotification, + // to be documented here + payload: any; + } + +.. ts:def:: WalletErrorInfo + + export interface WalletErrorInfo { + // Numeric error code defined defined in the + // GANA gnu-taler-error-codes registry. + talerErrorCode: number; + + // English description of the error code. + talerErrorHint: string; + + // English diagnostic message that can give details + // for the instance of the error. + message: string; + + // Error details, type depends + // on talerErrorCode + details: unknown; + } + +Withdrawal +---------- + +A typical API sequence for *bank-integrated* withdrawals can for example look like this: + +#. ``"getWithdrawalDetailsForUri"`` returns an amount and default exchange +#. ``"getWithdrawalDetailsForAmount"`` returns fee information and that ToS are not accepted + + #. ``"getExchangeTos"`` are shown to the user and return currentEtag + #. ``"setExchangeTosAccepted"`` called with currentEtag after user accepted + +#. ``"acceptWithdrawal"`` after the user confirmed withdrawal with associated fees + +A typical API sequence for *manual* withdrawals can for example look like this: + +#. ``"listExchanges"`` shows a list of exchanges to the user who picks one and an amount +#. ``"getWithdrawalDetailsForAmount"`` returns fee information and that ToS are not accepted + + #. ``"getExchangeTos"`` are shown to the user and return currentEtag + #. ``"setExchangeTosAccepted"`` called with currentEtag after user accepted + +#. ``"acceptManualWithdrawal"`` after the user confirmed withdrawal with associated fees + +Integration Tests +================= + +Integration Test Example +------------------------ + +Integration tests can be done with the low-level wallet commands. To select which coins and denominations +to use, the wallet can dump the coins in an easy-to-process format (`CoinDumpJson <https://git.taler.net/taler-typescript-core.git/tree/packages/taler-util/src/types-taler-wallet.ts#n613>`__). + +The database file for the wallet can be selected with the ``--wallet-db`` +option. This option must be passed to the ``taler-wallet-cli`` command and not +the subcommands. If the database file doesn't exist, it will be created. + +The following example does a simple withdrawal recoup: + +.. code-block:: console + + # Withdraw digital cash + $ taler-wallet-cli --wallet-db=mydb.sqlite3 testing withdraw \ + -b https://bank.int.taler.net/ \ + -e https://exchange.int.taler.net/ \ + -a INTKUDOS:10 + + $ coins=$(taler-wallet-cli --wallet-db=mydb.sqlite3 advanced dump-coins) + + # Find coin we want to revoke + $ rc=$(echo "$coins" | \ + jq -r '[.coins[] | select((.denom_value == "INTKUDOS:5"))][0] | .coin_pub') + + # Find the denom + $ rd=$(echo "$coins" | \ + jq -r '[.coins[] | select((.denom_value == "INTKUDOS:5"))][0] | .denom_pub_hash') + + # Find all other coins, which will be suspended + $ susp=$(echo "$coins" | \ + jq --arg rc "$rc" '[.coins[] | select(.coin_pub != $rc) | .coin_pub]') + + # The exchange revokes the denom + $ taler-exchange-keyup -r $rd + $ taler-deployment-restart + + # Now we suspend the other coins, so later we will pay with the recouped coin + $ taler-wallet-cli --wallet-db=mydb.sqlite3 advanced suspend-coins "$susp" + + # Update exchange /keys so recoup gets scheduled + $ taler-wallet-cli --wallet-db=mydb.sqlite3 exchanges update -f https://exchange.int.taler.net/ + + # Block until scheduled operations are done + $ taler-wallet-cli --wallet-db=mydb.sqlite3 run-until-done + + # Now we buy something, only the coins resulting from recouped will be + # used, as other ones are suspended + $ taler-wallet-cli --wallet-db=mydb.sqlite3 testing test-pay \ + -m https://backend.int.taler.net/ \ + -k sandbox \ + -a "INTKUDOS:1" \ + -s "foo" + $ taler-wallet-cli --wallet-db=mydb.sqlite3 run-until-done + + +To test refreshing, force a refresh: + +.. code-block:: console + + $ taler-wallet-cli --wallet-db=mydb.sqlite3 advanced force-refresh "$coin_pub" + + +To test zombie coins, use the timetravel option. It **must** be passed to the +top-level command and not the subcommand: + +.. code-block:: console + + # Update exchange /keys with time travel, value in microseconds + $ taler-wallet-cli --timetravel=1000000 --wallet-db=mydb.sqlite3 \ + exchanges update -f https://exchange.int.taler.net/ + + +Integration Test and Fault Injection Framework +---------------------------------------------- + +This section describes the current approach to integration testing in the wallet. + +It's all based on a TypeScript harness process, which itself implements +the fault injection proxy (async and in-process)! + +The new approach consists of the following parts: + +1. A strongly typed, convenient helper library to easily set up and run +arbitrary Taler deployments and run test cases. These components plug +together as easily as lego bricks, even with multiple +exchanges/merchants/banks/etc. Logs and clean shutdown (even on SIGINT +or errors) are handled properly. (Support for auditors is still pending +but needed to fully test the wallet.) + +This is how a simple withdrawal and payment test case looks like: +`<https://git.taler.net/taler-typescript-core.git/tree/packages/taler-harness/src/integrationtests/test-payment.ts>`__ + +(What's particularly nice is that all our docs contain TypeScript +definitions for all API request bodies. So just copying them into the +test harness gives us auto-completion and compile-time checks to avoid +typos. The wallet's JSON validation machinery is also re-used.) + +2. A fault injection proxy that can be plugged between the services +and/or the wallet. It runs alongside the test harness, and can thus can +use arbitrary custom logic. There's no dependency for it other than +built-in Node.JS libraries. Simple fault injections are just as easy to +set up as with the twister. + +The following test case (a) logs all requests and responses to the test +harness stdout and (b) at a certain point, starts dropping the next 10 +requests to the exchange (testing the wallet's retry logic): + +`<https://git.taler.net/taler-typescript-core.git/tree/packages/taler-harness/src/integrationtests/test-payment-fault.ts#n165>`__ + +3. All util functionality from JS wallet-core, such as the Taler crypto, +amount/date/etc. handling and JSON parsing/validation (the wallet is now +more modular and easier to use as a library) can be used in the +integration tests, even if a different wallet (Kotlin, whatever) is +tested via the CLI. + +4. A bunch of test cases that use (1)-(3). These are *significantly* +more readable and hackable than other test approaches we had, while +allowing for more complex scenarios. There are still way too few tests +though! + +5. A test runner (written in bash) that runs test cases based on a glob +pattern and reports the results. + +Injecting a fault is as easy as: + +.. code:: ts + + // Set up test case + [...] + + exchangeProxy.addFault({ + beforeResponse(ctx: FaultInjectionResponseContext) { + if (cond1) { // Drop some responses + ctx.dropResponse = true; + return; + } else if (cond2) { // modify some others + ctx.responseBody = Buffer.from(`{"oops": true}`, "utf-8"); + return; + } + // Other things that can be modified: + // - drop/modify the request, not just the response + // - modify headers + // - modify status codes + } + }); + + await doSomethingWithTheWallet(); + + exchangeProxy.clearFault(); + + await doMoreWithTheWallet(); + + +To make the configuration easy, an ``ExchangeService`` (or ``MerchantService``, +``BankService`` etc.) can be wrapped in a ``FaultInjectedExchangeService``, +which implements the ``ExchangeServiceInterface``: + +.. code:: ts + + // create exchange and two merchants + const exchange = await setupExchange(...); + const merchant1 = ...; + const merchant2 = ...; + + // Add exchange to merchant-accepted exchanges. + // This will adjust the config. + merchant1.addExchange(exchange); + + // Wrap exchange in fault injection proxy + const faultInjectedExchange: ExchangeServiceInterface + = new FaultInjectedExchangeService(t, exchange1, 8085); + + // Merchant 2 talks to the exchange over fault injection, + // and thus must use the "twisted" base URL. + merchant2.addExchange(faultInjectedExchange); + + +The package for the integration tests is here: + +`<https://git.taler.net/wallet-core.git/tree/packages/taler-harness>`__ + +The integration tests are run via the ``taler-harness`` tool. + +.. code:: sh + + ./bootstrap && ./configure --prefix=... && make install + taler-harness run-integrationtests + + +Dev Experiments +=============== + +Dev experiments allow simulating certain scenarios that are difficult to +reproduce otherwise. This allows more comprehensive (manual) testing of the +UIs. + +You can enable dev experiments by putting the wallet into dev mode and then +scanning the QR code for a ``taler://dev-experiment`` URI that specifies the +desired dev experiment. + +Faking Protocol Versions +------------------------ + +The ``start-fakeprotover`` dev experiment can be used to fake the protocol +version reported by Taler components. It mocks the ``version`` field in the +response to ``/config`` or ``/keys``. + +Usage: + +.. code:: none + + taler://dev-experiment/start-fakeprotover?base_url=...&fake_ver=... + +Example: + +.. code:: none + + # Fake version 10:0:0 for https://exchange.demo.taler.net/ + taler://dev-experiment/start-fakeprotover?base_url=https%3A%2F%2Fexchange.demo.taler.net%2F&fake_ver=10%3A0%3A0 + + +Faking Transactions +------------------- + + +**Withdrawal Transaction:** + +.. code:: none + + taler://dev-experiment/add-fake-tx?txType=withdrawal&amountEffective=KUDOS:5 + +Options: + +* ``amountEffective``: Mandatory. Effective amount of the withdrawal. +* ``tRel``: Optional (defaults to ``0s``). Relative time that indicates + when in the past the transaction was started (and finished). +* ``exchangeBaseUrl``. Optional (defaults to ``https://exchange.demo.taler.net``). + Exchange base URL used for the withdrwal. + + +**Payment Transaction:** + +.. code:: none + + taler://dev-experiment/add-fake-tx?txType=payment&amountEffective=KUDOS:5 + +Options: + +* ``amountEffective``: Mandatory. Effective amount of the withdrawal. +* ``tRel``: Optional (defaults to ``0s``). Relative time that indicates + when in the past the transaction was started (and finished). +* ``exchangeBaseUrl``. Optional (defaults to ``https://exchange.demo.taler.net``). + Exchange base URL used for the payment. +* ``merchantBaseUrl``. Optional (defaults to ``https://backend.demo.taler.net/``). +* ``merchantName``. Optional (defaults to ``Test Merchant``). Display name + of the merchant in the contract terms of the transaction. +* ``summary``. Optional (defaults to ``Test``). Summary in the contract terms of the + transaction. + +**Peer Push Credit Transaction:** + +.. code:: none + + taler://dev-experiment/add-fake-tx?txType=peer-push-credit&amountEffective=KUDOS:5 + +Options: + +* ``amountEffective``: Mandatory. Effective amount of the withdrawal. +* ``tRel``: Optional (defaults to ``0s``). Relative time that indicates + when in the past the transaction was started (and finished). +* ``exchangeBaseUrl``. Optional (defaults to ``https://exchange.demo.taler.net``). + Exchange base URL used for the payment. +* ``summary``. Optional (defaults to ``Test``). Summary in the contract terms of the + transaction. + +**Peer Push Debit Transaction:** + +.. code:: none + + taler://dev-experiment/add-fake-tx?txType=peer-push-debit&amountEffective=KUDOS:5 + +Options: + +* ``amountEffective``: Mandatory. Effective amount of the withdrawal. +* ``tRel``: Optional (defaults to ``0s``). Relative time that indicates + when in the past the transaction was started (and finished). +* ``exchangeBaseUrl``. Optional (defaults to ``https://exchange.demo.taler.net``). + Exchange base URL used for the payment. +* ``summary``. Optional (defaults to ``Test``). Summary in the contract terms of the + transaction. diff --git a/index.rst b/index.rst @@ -58,6 +58,7 @@ and overall usage of the different Taler components in text and video formats. taler-exchange-manual taler-kyc-manual + taler-challenger-manual .. toctree:: :maxdepth: 1 @@ -72,11 +73,12 @@ and overall usage of the different Taler components in text and video formats. :maxdepth: 1 :hidden: :numbered: - :caption: For Bank Operators + :caption: For Bank Integrators libeufin/index taler-magnet-bank-manual taler-cyclos-manual + depolymerization/index .. toctree:: :maxdepth: 1 @@ -85,10 +87,8 @@ and overall usage of the different Taler components in text and video formats. :caption: For Other Operators taler-auditor-manual - taler-challenger-manual taler-directory-manual taler-mailbox-manual - depolymerization/index .. toctree:: :maxdepth: 1 @@ -97,27 +97,30 @@ and overall usage of the different Taler components in text and video formats. :caption: For Developers core/index - taler-developer-manual - taler-wallet + developer/index design-documents/index .. toctree:: :maxdepth: 1 :hidden: :numbered: - :caption: Licensing + :caption: For System Administrators - global-licensing - fdl-1.3 + manindex + system-administration/index .. toctree:: :maxdepth: 1 :hidden: :numbered: - :caption: Other + :caption: For Lawyers - manindex - system-administration/index - tutorials - genindex + global-licensing + fdl-1.3 + +.. toctree:: + :hidden: + :caption: Other + + genindex diff --git a/taler-developer-manual.rst b/taler-developer-manual.rst @@ -1,1984 +0,0 @@ -.. - This file is part of GNU TALER. - - Copyright (C) 2014-2025 Taler Systems SA - - TALER is free software; you can redistribute it and/or modify it under the - terms of the GNU Affero General Public License as published by the Free Software - Foundation; either version 2.1, or (at your option) any later version. - - TALER is distributed in the hope that it will be useful, but WITHOUT ANY - WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR - A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. - - You should have received a copy of the GNU Affero General Public License along with - TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> - - @author Christian Grothoff - -Developer's Manual -################## - -.. note:: - - This manual contains information for developers working on GNU Taler - and related components. It is not intended for a general audience. - - -Project Overview -================ - -GNU Taler consists of a large (and growing) number of components -in various Git repositories. The following list gives a first -overview: - -* exchange: core payment processing logic with a REST API, plus various - helper processes for interaction with banks and cryptographic - computations. Also includes the logic for the auditor and an - in-memory "bank" API implementation for testing. - -* libeufin: implementation of the "bank" API using the EBICS protocol - used by banks in the EU. Allows an exchange to interact with - European banks. - -* taler-magnet-bank: implementation of the "bank" API using the Magnet Bank - API.Allows an exchange to interact with Magnet Bank. - -* taler-cyclos: implementation of the "bank" API using the Cyclos API. Allows an exchange to interact with a Cyclos network. - -* depolymerization: implementation of the "bank" API on top of - blockchains, specifically Bitcoin and Ethereum. Allows an exchange - to interact with crypto-currencies. - -* merchant: payment processing backend to be run by merchants, - offering a REST API. - -* wallet-core: platform-independent implementation of a wallet to be run by - normal users. Includes also the WebExtension for various browsers. - Furthermore, includes various single-page apps used by other - components (especially as libeufin and merchant). Also includes - command-line wallet and tools for testing. - -* taler-android: Android Apps including the Android wallet, the - Android point-of-sale App and the Android casher app. - -* taler-ios: iOS wallet App. - -* sync: backup service, provides a simple REST API to allow users to - make encrypted backups of their wallet state. - -* anastasis: key escrow service, provides a simple REST API to allow - users to distribute encryption keys across multiple providers and - define authorization policies for key recovery. - -* taler-mdb: integration of Taler with the multi-drop-bus (MDB) API - used by vending machines. Allows Taler payments to be integrated - with vending machines. - -* gnu-taler-payment-for-woocommerce: payment plugin for the - woocommerce (wordpress) E-commerce solution. - -* twister: man-in-the-middle proxy for tests that require fuzzing a - REST/JSON protocol. Used for some of our testing. - -* challenger: implementation of an OAuth 2.0 provider that can be used - to verify that a user can receive SMS or E-mail at particular addresses. - Used as part of KYC processes of the exchange. - -* taler-mailbox: messaging service used to store and forward payment - messages to Taler wallets. - -* taldir: directory service used to lookup Taler wallet addresses for - sending invoices or payments to other wallets. - -* taler-merchant-demos: various demonstration services operated at - 'demo.taler.net', including a simple shop and a donation page. - -There are other important repositories without code, including: - -* gana: Hosted on git.gnunet.org, this repository defines various - constants used in the GNU Taler project. - -* docs: documentation, including this very document. - -* marketing: various presentations, papers and other resources for - outreach. - -* large-media: very large data objects, such as videos. - -* www: the taler.net website. - -Fundamentals -============ - -Versioning ----------- - -A central rule is to never break anything for any dependency. To accomplish -this, we use versioning, of the APIs, database schema and the protocol. The -database versioning approach is described in the :ref:`Database schema -versioning <DatabaseVersioning>` section. Here, we will focus on API and -protocol versioning. - -The key issue we need to solve with protocols and APIs (and that does not -apply to database versioning) is being able to introduce and remove features -without requiring a flag day where all components must update at the same -time. For this, we use GNU libtool style versioning with MAJOR:REVISION:AGE -and *not* semantic versioning (SEMVER). With GNU libtool style versioning, -first the REVISION should be increased on every change to the respective code. -Then, each time a feature is introduced or deprecated, the MAJOR and AGE -numbers are increased. Whenever an API is actually removed the AGE number is -reduced to match the distance since the removed API was deprecated. Thus, if -some client implements version X of the protocol (including not using any APIs -that have been deprecated), it is compatible for any implementation where -MAJOR is larger or equal to X, and MAJOR minus AGE is smaller or equal to X. -REVISION is not used for expected compatibility issues and merely serves to -uniquely identify each version (in combination with MAJOR). - -To evolve any implementation, it is thus critical to first of all never -just break an existing API or endpoint. The only acceptable modifications -are to return additional information (being aware of binary compatibility!) -or to accept additional optional arguments (again, in a way that does not -break existing users). Thus, the most common way to introduce changes will -be the addition of new endpoints. Breaking existing endpoints is only ever -at best acceptable while in the process of introducing it and if you are -absolutely sure that there are zero users in other components. - -When removing endpoints (or fields being returned), you must first deprecate -the existing API (incrementing MAJOR and AGE) and then wait for all clients, -including all clients in operation (e.g. Android and iOS Apps, e-commerce -integrations, etc.) to upgrade to a protocol implementation above the -deprecated MAJOR revision. Only then you should remove the endpoint and reduce -AGE. - -To document these changes, please try to use ``@since`` annotations in the API -specifications to explain the MAJOR revision when a feature became available, -but most importantly use ``@deprecated X`` annotations to indicate that an API -was deprecated and will be removed once MAJOR minus AGE is above X. When using -an API, use the ``/config`` endpoints to check for compatibility and show a -warning if the version(s) you support and the version(s) offered by the server -are incompatible. - - -Tagging and Package Versioning ------------------------------- - -Release tags are of the form ``v${major}.${minor}.${patch}``. Release tags *should* be -annotated git tags. - -We usually consider Debian packaging files (in ``debian/``) to be part of a release. -When only the Debian packaging files need to be changed, there are two options: - -* Make a new patch release (``v${major}.${minor}.${patch+1}``) -* Make a Debian release: - - * Debian version now includes a revision: ``${major}.${minor}.${patch}-${debrevision}`` - * The tag is Debian-specific: ``debian-${major}.${minor}.${patch}-${debrevision}`` - -All source repos *should* include a ``contrib/bump`` script that automates bumping the\ -version in all relevant source and packaging files. -In the future, we might add an option to the script to only release a packaging bump. -Right now, that process is manual. - -We support tagged and published pre-release versions via tags of the form ``v${major}.${minor}.${patch}-dev.${n}``. -The corresponding Debian version must be ``${major}.${minor}.${patch}~dev${n}``. - -Nightly Debian packages should follow the `Debian conventions <https://wiki.debian.org/Versioning>`__ of ``{upcoming_version}~git{date}.{hash}-{revision}``. - -Testing Tools -------------- - -For full ``make check`` support, install these programs: - -- `jq <https://github.com/stedolan/jq>`__ -- `curl <http://curl.haxx.se>`__ -- `faketime <https://github.com/wolfcw/libfaketime>`__ - -The ``make check`` should be able to function without them, but -their presence permits some tests to run that would otherwise be skipped. - -Manual Testing Database Reset ------------------------------ - -Sometimes ``make check`` will fail with some kind of database (SQL) -error, perhaps with a message like ``OBJECT does not exist`` in the -``test-suite.log`` file, where ``OBJECT`` is the name of a table or function. -In that case, it may be necessary to reset the ``talercheck`` database -with the commands: - -.. code-block:: console - - $ dropdb talercheck - $ createdb talercheck - -This is because, at the moment, there is no support for -doing these steps automatically in the ``make check`` flow. - -(If ``make check`` still fails after the reset, file a bug report as usual.) - -Bug Tracking ------------- - -Bug tracking is done with Mantis (https://www.mantisbt.org/). The bug tracker -is available at `<https://bugs.taler.net>`_. A registration on the Web site is -needed in order to use the bug tracker, only read access is granted without a -login. - -We use the following conventions for the bug states: - -* NEW: Incoming bugs are in 'new' so that management (or developers) - can easily identify those that need to be checked (report correct? - something we want to fix?), prioritized and targeted for releases. - "NEW" bugs are never assigned to a developer. - -* FEEDBACK: When blocked on feedback from reporter or other developer. - Assigned to other developer (but cannot be assigned to reporter, - in this case MAY remain associated with the developer who expects - the feedback). - -* ACKNOWLEDGED: The bug has been reviewed, but no decision about - what action to take has been made yet. Should not be worked on - until management (or a developer) comes up with a plan. - "ACKNOWLEDGED" bugs should NOT be assigned to a developer. - -* CONFIRMED: This is a real issue that should be worked on, but - is not yet actively worked on. If working on this bug requires - other bugs to be fixed first, they should be added as - child-bugs (via relationships). Developers are always welcome - to self-assign bugs that are "CONFIRMED" if they start to work - on a bug. "CONFIRMED" bugs should NOT be assigned to a developer. - -* ASSIGNED: The specific developer the bug is assigned to is - **actively** working on the issue. Developers should strive to - not have more than 5-10 bugs assigned to them at any time. - Only having one assigned to you is totally OK! - Developers should aggressively un-assign bugs that they are - blocked on, cannot make progress on, or are no longer actively - working on (but of course, better *resolve* them before - moving on if possible). If the bug remains open, it probably - should go back to "CONFIRMED" or "ACKNOWLEDGED". - -* RESOLVED: The bug has been fixed in Git. - -* CLOSED: An official release was made with the fix in it. - -When developers want to keep an eye on certain bugs, they should -**monitor** them. Multiple developers can be monitoring a bug, but -it can only be assigned to one. Developers should also keep an -eye on the roadmap (by release), bug categories they care about, -and of course priorities / severities. - -We use **tags** to categorize bugs. Common tags that also imply -some urgency include (in alphabetical order): - -* accounting: issues required for accounting (such as taxes by merchants) -* compliance: issues related to regulatory compliance -* $CUSTOMER: issues requested by a particular customer -* performance: performance problems or ideas for improvement -* security: security issues (including planned improvements to security) -* UX: user experience issues - -These tags **should** be attached to "NEW" bugs if they apply. - - -Code Repositories ------------------ - -Taler code is versioned with Git. For those users without write access, all the -codebases are found at the following URL: - -.. code-block:: none - - git://git.taler.net/<repository> - -A complete list of all the existing repositories is currently found at -`<https://git.taler.net/>`_. - - -Committing code ---------------- - -Before you can obtain Git write access, you must sign the copyright -agreement. As we collaborate closely with GNUnet, we use their -copyright agreement -- with the understanding that your contributions -to GNU Taler are included in the assignment. You can find the -agreement on the `GNUnet site <https://gnunet.org/en/copyright.html>`_. -Please sign and mail it to Christian Grothoff as he currently collects -all the documents for GNUnet e.V. - -To obtain Git access, you need to send us your SSH public key. Most core -team members have administrative Git access, so simply contact whoever -is your primary point of contact so far. You can -find instructions on how to generate an SSH key -in the `Git book <https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key>`_. -If you have been granted write access, you first of all must change the URL of -the respective repository to: - -.. code-block:: none - - ssh://git@git.taler.net/<repository> - -For an existing checkout, this can be done by editing the ``.git/config`` file. - -The server is configured to reject all commits that have not been signed with -GnuPG. If you do not yet have a GnuPG key, you must create one, as explained -in the `GNU Privacy Handbook <https://www.gnupg.org/gph/en/manual/c14.html>`_. -You do not need to share the respective public key with us to make commits. -However, we recommend that you upload it to key servers, put it on your -business card and personally meet with other GNU hackers to have it signed -such that others can verify your commits later. - -To sign all commits, you should run - -.. code-block:: console - - $ git config --global commit.gpgsign true - -You can also sign individual commits only by adding the ``-S`` option to the -``git commit`` command. If you accidentally already made commits but forgot -to sign them, you can retroactively add signatures using: - -.. code-block:: console - - $ git rebase -S - - -Whether you commit to a personal branch (recommended: ``dev/$USER/...``), -a feature branch or to ``master`` should -depend on your level of comfort and the nature of the change. As a general -rule, the code in ``master`` must always build and tests should always pass, at -least on your own system. However, we all make mistakes and you should expect -to receive friendly reminders if your change did not live up to this simple -standard. We plan to move to a system where the CI guarantees this invariant -in the future. - -In order to keep a linear and clean commits history, we advise to avoid -merge commits and instead always rebase your changes before pushing to -the ``master`` branch. If you commit and later find out that new commits were -pushed, the following command will pull the new commits and rebase yours -on top of them. - -.. code-block:: console - - # -S instructs Git to (re)sign your commits - $ git pull --rebase -S - - - -Observing changes ------------------ - -Every commit to the ``master`` branch of any of our public repositories -(and almost all are public) is automatically sent to the -gnunet-svn@gnu.org mailinglist. That list is for Git commits only, -and must not be used for discussions. It also carries commits from -our main dependencies, namely GNUnet and GNU libmicrohttpd. While -it can be high volume, the lists is a good way to follow overall -development. - - -Code generator usage policy ---------------------------- - -We do neither encourage nor discourage the use of tools for code generation. -It is up to the individual developer to decide if a tool is acceptable for -a particular task. But of course, we do encourage you to use FLOSS tools -and we MUST NOT become dependend on non-free software! That said, if you -use tools, you must document their use and in particular satisfy the -`NLnet policy on the use of "AI" <https://nlnet.nl/news/2025/20250829-policy-on-use-of-AI.html>`__. - -Specifically, we ask developers to always put generated code into a *separate* -Git commit and to include the full prompt in the commit message. Naturally, -you may clean up the code generator's output, but then you should do so in -separate Git commits (and of course only merge into master/stable after the -clean up is complete). But do preserve (not squash!) the commit with the -generated code so that it remains documented what the prompts were and which -code is generated. This will go a long way to keep code auditors sane! - - -Communication -------------- - -For public discussions we use the taler@gnu.org mailinglist. All developers -should subscribe to the low-volume Taler mailinglist. There are separate -low-volume mailinglists for gnunet-developers (@gnu.org) and for libmicrohttpd -(@gnu.org). For internal discussions we use https://mattermost.taler.net/ -(invitation only, but also achieved). - - -What to put in bootstrap ------------------------- - -Each repository has a ``bootstrap`` script, which contains commands for the -developer to run after a repository checkout (i.e., after ``git clone`` or -``git pull``). -Typically, this updates and initializes submodules, prepares the tool chain, -and runs ``autoreconf``. -The last step generates the ``configure`` script, whether for immediate use or -for inclusion in the distribution tarball. - -One common submodule is ``contrib/gana``, which pulls from the -`GNUnet GANA repository <https://git.gnunet.org/gana.git/>`__. -For example, in the -`Taler exchange repository <https://git.taler.net/exchange.git>`__, -the bootstrap script eventually runs the ``git submodule update --init`` command -early on, and later runs script ``./contrib/gana-generate.sh``, which -generates files such as ``src/include/taler_signatures.h``. - -Thus, to update that file, you need to: - -- (in GANA repo) Find a suitable (unused) name and number for the Signature - Purposes database. - -- Add it to GANA, in ``gnunet-signatures/registry.rec``. - (You can check for uniqueness with the ``recfix`` utility.) - -- Commit the change, and push it to the GANA Git repo. - -- (in Taler Repo) Run the ``contrib/gana-latest.sh`` script. - -- Bootstrap, configure, do ``make install``, ``make check``, etc. - (Basically, make sure the change does not break anything.) - -- Commit the submodule change, and push it to the Taler exchange Git repo. - -A similar procedure is required for other databases in GANA. -See file ``README`` in the various directories for specific instructions. - - -Debian and Ubuntu Repositories -============================== - -We package our software for Debian and Ubuntu. - -Nightly Repositories --------------------- - -To try the latest, unstable and untested versions of packages, -you can add the nightly package sources. - -.. code-block:: shell-session - - # For Debian (trixie) - $ curl -sS https://deb.taler.net/apt-nightly/taler-trixie-ci.sources \ - | tee /etc/apt/sources.list.d/taler-trixie-nightly.sources - - -Taler Deployment on gv.taler.net -================================ - -This section describes the GNU Taler deployment on ``gv.taler.net``. ``gv`` -is our server at BFH. It hosts the Git repositories, Web sites, CI and other -services. Developers can receive an SSH account and e-mail alias for the -system, you should contact Javier, Christian or Florian. As with Git, ask -your primary team contact for shell access if you think you need it. - - -DNS ---- - -DNS records for taler.net are controlled by the GNU Taler maintainers, -specifically Christian and Florian, and our system administrator, Javier. If -you need a sub-domain to be added, please contact one of them. - - -User Acccounts --------------- - -On ``gv.taler.net``, there are three system users that are set up to -serve Taler on the Internet: - -- ``head``: serves ``*.head.taler.net`` and gets automatically - built by Buildbot every 2 hours from the ``sandcastle-ng.git``. - Master key may be reset occasionally - -- ``taler-test``: serves ``*.test.taler.net`` and does *NOT* get - automatically built, and runs more recent tags and/or unreleased - versions of Taler components. Master key may be reset - occasionally. - -- ``demo``: serves ``*.demo.taler.net``. Never automatically built. - Master key is retained. - -Demo Upgrade Procedure -====================== - -#. Login as the ``demo`` user on ``gv.taler.net``. -#. Pull the latest ``sandcastle-ng.git`` code in checkout at ``$HOME/sandcastle-ng``. -#. Run ``systemctl --user restart container-taler-sandcastle-demo.service`` -#. Refer to the sandcastle-ng README (https://git.taler.net/sandcastle-ng.git/about/) - for more info. - - -Upgrading the ``demo`` environment should be done with care, and ideally be -coordinated on the mailing list before. It is our goal for ``demo`` to always -run a "working version" that is compatible with various published wallets. -Please use the :doc:`demo upgrade checklist <checklists/checklist-demo-upgrade>` to make -sure everything is working. -Nginx is already configured to reach the services as exported by the user unit. - - -Tagging components ------------------- - -All Taler components must be tagged with git before they are deployed on the -``demo`` environment, using a tag of the following form: - -.. code-block:: none - - demo-YYYY-MM-DD-SS - YYYY = year - MM = month - DD = day - SS = serial - -Environments and Builders on taler.net -====================================== - -Buildbot implementation ------------------------ - -GNU Taler uses a buildbot implementation (front end at https://buildbot.taler.net) to manage continuous integration. Buildbot documentation is at https://docs.buildbot.net/. - -Here are some highlights: - -- The WORKER is the config that that lives on a shell account on a localhost (taler.net), where this host has buildbot-worker installed. The WORKER executes the commands that perform all end-functions of buildbot. - -- The WORKER running buildbot-worker receives these commands by authenticating and communicating with the buildbot server using parameters that were specified when the worker was created in that shell account with the ``buildbot-worker`` command. - -- The buildbot server's master.cfg file contains FACTORY declarations which specify the commands that the WORKER will run on localhost. - -- The FACTORY is tied to the WORKER in master.cfg by a BUILDER. - -- The master.cfg also allows for SCHEDULER that defines how and when the BUILDER is executed. - -- Our master.cfg file is checked into git, and then periodically updated on a particular account on taler.net (ask Christian for access if needed). Do not edit this file directly/locally on taler.net, but check changes into Git. - - -Best Practices: - -- When creating a new WORKER in the ``master.cfg`` file, leave a comment specifying the server and user account that this WORKER is called from. (At this time, taler.net is the only server used by this implementation, but it's still good practice.) - -- Create a worker from a shell account with this command: ``buildbot-worker create-worker <workername> localhost <username> <password>`` - -Then make sure there is a WORKER defined in master.cfg like: ``worker.Worker("<username>", "<password>")`` - -Test builder ------------- - -This builder (``test-builder``) compiles and starts every Taler component. -The associated worker is run by the ``taler-test`` Gv user, via the SystemD -unit ``buildbot-worker-taler``. The following commands start/stop/restart -the worker: - -.. code-block:: - - systemctl --user start buildbot-worker-taler - systemctl --user stop buildbot-worker-taler - systemctl --user restart buildbot-worker-taler - -.. note:: - the mentioned unit file can be found at ``deployment.git/systemd-services/`` - -Wallet builder --------------- - -This builder (``wallet-builder``) compiles every Taler component -and runs the wallet integration tests. The associated worker is -run by the ``walletbuilder`` Gv user, via the SystemD unit ``buildbot-worker-wallet``. -The following commands start/stop/restart the worker: - -.. code-block:: - - systemctl --user start buildbot-worker-wallet - systemctl --user stop buildbot-worker-wallet - systemctl --user restart buildbot-worker-wallet - -.. note:: - the mentioned unit file can be found at ``deployment.git/systemd-services/`` - -Documentation Builder ---------------------- - -All the Taler documentation is built by the user ``docbuilder`` that -runs a Buildbot worker. The following commands set the ``docbuilder`` up, -starting with an empty home directory. - -.. code-block:: console - - # Log-in as the 'docbuilder' user. - - $ cd $HOME - $ git clone git://git.taler.net/deployment - $ ./deployment/bootstrap-docbuilder - - # If the previous step worked, the setup is - # complete and the Buildbot worker can be started. - - $ buildbot-worker start worker/ - - -Website Builder ---------------- - - -Taler Websites, ``www.taler.net`` and ``stage.taler.net``, are built by the -user ``taler-websites`` by the means of a Buildbot worker. The following -commands set the ``taler-websites`` up, starting with an empty home directory. - -.. code-block:: console - - # Log-in as the 'taler-websites' user. - - $ cd $HOME - $ git clone git://git.taler.net/deployment - $ ./deployment/bootstrap-sitesbuilder - - # If the previous step worked, the setup is - # complete and the Buildbot worker can be started. - - $ buildbot-worker start worker/ - - -Code coverage -------------- - -Code coverage tests are run by the ``lcovworker`` user, and are also driven -by Buildbot. - -.. code-block:: console - - # Log-in as the 'lcovworker' user. - - $ cd $HOME - $ git clone git://git.taler.net/deployment - $ ./deployment/bootstrap-taler lcov - - # If the previous step worked, the setup is - # complete and the Buildbot worker can be started. - - $ buildbot-worker start worker/ - -The results are then published at ``https://lcov.taler.net/``. - -Producing auditor reports -------------------------- - -Both 'test' and 'demo' setups get their auditor reports compiled -by a Buildbot worker. The following steps get the reports compiler -prepared. - -.. code-block:: console - - # Log-in as <env>-auditor, with <env> being either 'test' or 'demo' - - $ git clone git://git.taler.net/deployment - $ ./deployment/buildbot/bootstrap-scripts/prepare-auditorreporter <env> - - # If the previous steps worked, then it should suffice to start - # the worker, with: - - $ buildbot-worker start worker/ - - -.. _DatabaseVersioning: - -Database schema versioning --------------------------- - -The PostgreSQL databases of the exchange and the auditor are versioned. -See the ``versioning.sql`` file in the respective directory for documentation. - -Every set of changes to the database schema must be stored in a new -versioned SQL script. The scripts must have contiguous numbers. After -any release (or version being deployed to a production or staging -environment), existing scripts MUST be immutable. - -Developers and operators MUST NOT make changes to database schema -outside of this versioning. All tables of a GNU Taler component should live in their own schema. - - -QA Plans -======== - -.. include:: checklists/qa-1.0.rst - - -Releases -======== - -.. include:: checklists/checklist-release.rst - -Release Process ---------------- - -This document describes the process for releasing a new version of the -various Taler components to the official GNU mirrors. - -The following components are published on the GNU mirrors - -- taler-exchange (exchange.git) -- taler-merchant (merchant.git) -- sync (sync.git) -- taler-mdb (taler-mdb.git) -- libeufin (libeufin.git) -- challenger (challenger.git) -- wallet-core (wallet-core.git) - -Tagging -------- - -Tag releases with an **annotated** commit, like - -.. code-block:: console - - $ git tag -a v0.1.0 -m "Official release v0.1.0" - $ git push origin v0.1.0 - - -Database for tests ------------------- - -For tests in the exchange and merchant to run, make sure that a database -*talercheck* is accessible by *$USER*. Otherwise tests involving the -database logic are skipped. - -.. include:: frags/db-stores-sensitive-data.rst - -Exchange, merchant ------------------- - -Set the version in ``configure.ac``. The commit being tagged should be -the change of the version. - -Tag the current GANA version that works with the exchange and merchant and -checkout that tag of gana.git (instead of master). Otherwise, if there are -incompatible changes in GANA (like removed symbols), old builds could break. - -Update the Texinfo documentation using the files from docs.git: - -.. code-block:: console - - # Get the latest documentation repository - $ cd $GIT/docs - $ git pull - $ make texinfo - # The *.texi files are now in _build/texinfo - # - # This checks out the prebuilt branch in the prebuilt directory - $ git worktree add prebuilt prebuilt - $ cd prebuilt - # Copy the pre-built documentation into the prebuilt directory - $ cp -r ../_build/texinfo . - # Push and commit to branch - $ git commit -a -S -m "updating texinfo" - $ git status - # Verify that all files that should be tracked are tracked, - # new files will have to be added to the Makefile.am in - # exchange.git as well! - $ git push - # Remember $REVISION of commit - # - # Go to exchange - $ cd $GIT/exchange/doc/prebuilt - # Update submodule to point to latest commit - $ git checkout $REVISION - -Finally, the Automake ``Makefile.am`` files may have to be adjusted to -include new ``*.texi`` files or images. - -For bootstrap, you will need to install -`GNU Recutils <https://www.gnu.org/software/recutils/>`_. - -For the exchange test cases to pass, ``make install`` must be run first. -Without it, test cases will fail because plugins can't be located. - -.. code-block:: console - - $ ./bootstrap - $ ./configure # add required options for your system - $ make dist - $ tar -xf taler-$COMPONENT-$VERSION.tar.gz - $ cd taler-$COMPONENT-$VERSION - $ make install check - -Wallet WebExtension -------------------- - -The version of the wallet is in *manifest.json*. The ``version_name`` -should be adjusted, and *version* should be increased independently on -every upload to the WebStore. - -.. code-block:: console - - $ ./configure - $ make dist - -Upload to GNU mirrors ---------------------- - -See https://www.gnu.org/prep/maintain/maintain.html#Automated-FTP-Uploads - -Directive file: - -.. code-block:: none - - version: 1.2 - directory: taler - filename: taler-exchange-0.1.0.tar.gz - symlink: taler-exchange-0.1.0.tar.gz taler-exchange-latest.tar.gz - -Upload the files in **binary mode** to the ftp servers. - - -Creating Debian packages ------------------------- - -Our general setup is based on -https://wiki.debian.org/DebianRepository/SetupWithReprepro - -First, update at least the version of the Debian package in -debian/changelog, and then run: - -.. code-block:: bash - - $ dpkg-buildpackage -rfakeroot -b -uc -us - -in the respective source directory (GNUnet, exchange, merchant) to create the -``.deb`` files. Note that they will be created in the parent directory. This -can be done on gv.taler.net, or on another (secure) machine. -Actual release builds should be done via the Docker images -that can be found in ``deployment.git`` under packaging. - -On ``gv``, we use the ``aptbuilder`` user to manage the reprepro repository. - -Next, the ``*.deb`` files should be copied to gv.taler.net, say to -``/home/aptbuilder/incoming``. Then, run - -.. code-block:: bash - - # cd /home/aptbuilder/apt - # reprepro includedeb bullseye ~/incoming/*.deb - -to import all Debian files from ``~/incoming/`` into the ``bullseye`` -distribution. If Debian packages were build against other distributions, -reprepro may need to be first configured for those and the import command -updated accordingly. - -Finally, make sure to clean up ``~/incoming/`` (by deleting the -now imported ``*.deb`` files). - - - -Continuous integration -====================== - -CI is done with Buildbot (https://buildbot.net/), and builds are -triggered by the means of Git hooks. The results are published at -https://buildbot.taler.net/ . - -In order to avoid downtimes, CI uses a "blue/green" deployment -technique. In detail, there are two users building code on the system, -the "green" and the "blue" user; and at any given time, one is running -Taler services and the other one is either building the code or waiting -for that. - -There is also the possibility to trigger builds manually, but this is -only reserved to "admin" users. - - -Internationalisation -==================== - -Internationalisation (a.k.a "translation") is handled with Weblate (https://weblate.org) via our instance at https://weblate.taler.net/ organised as so-called *strings* in a bunch of Git repositories. - -Who can register ----------------- - -At this time, anyone can register an account at https://weblate.taler.net/ to create translations. Registered users default to the **Users** and **Viewers** privilege levels. - -About privilege levels ----------------------- - -This is the breakdown of privilege levels in Weblate: - -* **Viewers** = Have access to translations and can view them but are not entitled to translate - -* **Users** = Can log in and perform translations (*applies to new users*) - -* **Reviewers** = Can contribute translations to existing *components* and review translations - -* **Managers** = Can create new *components* of existing *projects* - -* **Superusers** = Can create new *projects*, moreover they manage Weblate's settings and repository maintenance, can edit all user accounts, deactivate users, trigger commits from Weblate's local repositories with pushes to Git and perform backups of language files - -How to contribute with your translation ---------------------------------------- - -1 - Log into https://weblate.taler.net - -2 - Navigate to *Projects* and *Browse all projects* - -3 - Choose the *project* you wish to contribute to - -4 - Choose the *component* you wish to contribute to - -5 - Find the language you want to translate into - -6 - Find a phrase and translate it - -7 - While translating, take into consideration at each string to translate - -* the comments - -* the context - -* the string location in the source code - -* the "nearby strings" in Weblate as one of the sorting options on the translation form - -* "other languages" as another sorting option to compare strings in a way translatologists would do - -* if applicable, the history of string changes and the message ID or key of a string - -8 - **Always test your translations in the actual apps** after they have been pushed by the platform to the apps - -9 - Be aware of ambiguities and immediately **report inconsistencies or bugs** by using Mantis bug tickets - -10 - If you want to contribute your translation in a language which does not yet exist in a Taler component, navigate to the *component* you want to translate for and click "Start new translation" to begin. In case you require a privilege upgrade, please contact a *Superuser* with your request. Currently, *Superusers* are Christian, Florian, Martin, and Stefan. - -Translation standards and practices ------------------------------------ - -1 - *Developers*, Weblate *reviewers* and *managers* MUST look into **design documents** for the specific component or software project, like e.g. for the Taler wallets DD53 https://docs.taler.net/design-documents/053-wallet-ui.html and stick close to the documentation, especially the **text to use** and **text to avoid** - -2 - To every ambiguous string *developers* MUST add **context** or **comments** to the strings on Weblate - -3 - *Developers*, Weblate *reviewers* and *managers* SHOULD add **screenshots** where needed to the strings on Weblate - -4 - Weblate *managers* MUST take care of **checks** in the translations overview and keep the repositories consistent and up-to-date, t.i. to perform regularly **synchronisation** between Git and Weblate repositories and vice-versa - -5 - *Weblate admins* MUST set the **commit and push rules** to *manual* only, and when asked, set the license to *GPLv3 or later* - -6 - You MAY also wish to refer to https://docs.weblate.org/ and to the ``README files`` in components like web pages to find out more specific variables if prompted - -How to create a project ------------------------ - -The *GNU Taler* project is probably the correct project for most Weblate components and Taler-related translations. Please contact a *Superuser* if you need another project created. - -How to create a component -------------------------- - -Reference: https://docs.weblate.org/en/weblate-5.13.2/admin/projects.html#component-configuration - -In Weblate, a *component* is a subset of a *project* and each component contains N translations. A component is generally associated with a Git repository. - -To create a component, log into https://weblate.taler.net/ with your credentials for a *Manager* privilege level (or higher) and choose **+ Add** from the upper-right corner. - -What follows is a sort of Wizard. You can find detailed docs at https://docs.weblate.org/. Here are some important notes about connecting your component to the Taler Git repository: - -On the component level, choose *Operations* from the menu bar, select *Settings*, then add at the tab *Basic* these settings: - -* **Component name** - The distinctive name of the software project, e.g. ``Main web site`` - -* **Translation license** - Choose *GNU Affero General Public License v3.0 or Later* - -* **Contributor license agreement** - left empty as the *Translation license* defaults - -* **Source string bug reporting address** - This requires an email address of the Weblate admin or *Superuser* in charge or just ``languages@taler.net`` - -* **Priority** - Generally ``Medium`` - -* **Restricted component** - ``deactivated`` - -* **Share in projects** - ``deactivated`` - -* **Use as a glossary** - ``deactivated`` - -At the tab *Translation* settings are generally: - -* **Turn on suggestions** - ``activated`` - -* **Allow translation propagation** - ``activated`` - -* **Contribute to project translation memory** - ``activated`` - -* **Secondary language** - Leave empty (respectively ``--------`` ) - -At the tab *Version control* ( *https://weblate.taler.net/create/component/vcs/* ) settings require variables as follows: - -* **Version Control System** - Generally ``Git`` - -* **Source code repository** - Generally ``git+ssh://git@git.taler.net/<repositoryname>.git``, check with ``git remote -v`` - -* **Repository branch** - Choose the correct branch to draw from and commit to which is generally ``master`` - -* **Repository push URL** - Generally ``git+ssh://git@git.taler.net/<repositoryname>.git``, check with ``git remote -v`` - -* **Push branch** - Choose the correct branch to push to, this is generally ``master`` - -* **Repository browser** - Normally, this field is left empty. If you are required to have this field pointing to the www URL of the Git repo's file browser, it could be filled with ``https://git.taler.net/<repositoryname>.git/tree/{{filename}}?h={{branch}}#n{{line}}`` where ``<repositoryname>`` gets replaced but ``{{filename}}`` and other items in braces are actual variables in the string - -* **Push on commit** - ``deactivated`` - -* **Age of changes to commit** - 24 (hours after which any pending changes will be committed to the VCS) - -* **Merge style** - Generally ``Rebase``, in line with GNU Taler development procedures - -* **Lock on error** - ``activated`` - -At the tab *Commit messages* you can leave the defaults: - -* **Commit message when translating** has the most important message: -Translated using Weblate ({{ language_name }}) -Currently translated at {{ stats.translated_percent }}% ({{ stats.translated }} of {{ stats.all }} strings) -Translation: {{ project_name }}/{{ component_name }} -Translate-URL: {{ url }} - -At the tab *Files* - -* **File format** - Mostly ``gettext PO file`` for web pages, ``Android String Resource`` for Android app strings, ``XLIFF 1.2 translation file`` for iOS apps, or ``TermBase eXchange file`` for some of the glossaries - -* **Language filter** - Dependent on the respective folder structure of the project, for structures used with gettext PO files mostly ``locale/*/LC_MESSAGES/messages.po`` - -* **Language filter** - Generally ``^[^.]+$`` - -* **Source language** - ``English`` - -* **Monolingual translations / Monolingual base language file** - Leave empty - -* **Edit base file** - ``deactivated`` - -* **Intermediate language file** - Leave empty - -* **Template for new translations** - Dependent on the respective folder structure of the project, for structures used with gettext PO files ``locale/messages.pot`` - -* **Adding new translation** - Generally ``Create new language file`` - -* **Language code style** - Generally ``Default based on the file format`` - -* **Screenshot file mask** - Leave empty (unless we do not dispose of a dedicated screenshot folder with a path relative to repository root) - -GPG signing of translations ---------------------------- - -Weblate signs its GPG commits with the GPG key CD33CE35801462FA5EB0B695F2664BF474BFE502, and the corresponding public key can be found at https://weblate.taler.net/keys/. - -This means that contributions made through Weblate will not be signed with the individual contributor's key when they are checked into the Git repository but with Weblate's public key. - - - -iOS Apps -======== - -.. _Build-iOS-from-source: - -Building Taler Wallet for iOS from source ------------------------------------------ - -The GNU Taler Wallet iOS app is in -`the official Git repository <https://git.taler.net/taler-ios.git>`__. - -Compatibility -^^^^^^^^^^^^^ - -The minimum version of iOS supported is 15.0. -This app runs on all iPhone models at least as new as the iPhone 6S. - - -Building -^^^^^^^^ - -Before building the iOS wallet, you must first checkout the -`quickjs-tart repo <https://git.taler.net/quickjs-tart.git>`__ -and the -`wallet-core repo <https://git.taler.net/wallet-core.git>`__. - -Have all 3 local repos (wallet-core, quickjs-tart, and this one) adjacent at -the same level (e.g. in a "GNU_Taler" folder) -Taler.xcworkspace expects the QuickJS framework sub-project to be at -``../quickjs-tart/QuickJS-rt.xcodeproj``. - -Build wallet-core first: - -.. code-block:: shell-session - - $ cd wallet-core - $ make embedded - $ open packages/taler-wallet-embedded/dist - -then drag or move its product "taler-wallet-core-qjs.mjs" -into your quickjs-tart folder right at the top level. - -Open Taler.xcworkspace, and set scheme / target to Taler_Wallet. Build&run... - -Don't open QuickJS-rt.xcodeproj or TalerWallet.xcodeproj and build anything -there - all needed libraries and frameworks will be built automatically from -Taler.xcworkspace. - - -Android Apps -============ - -Android App Nightly Builds --------------------------- - -There are currently three Android apps in -`the official Git repository <https://git.taler.net/taler-android.git>`__: - -* Wallet - [`CI <https://git.taler.net/taler-android.git/tree/wallet/.gitlab-ci.yml>`__] -* Merchant PoS Terminal - [`CI <https://git.taler.net/taler-android.git/tree/merchant-terminal/.gitlab-ci.yml>`__] -* Cashier - [`CI <https://git.taler.net/taler-android.git/tree/cashier/.gitlab-ci.yml>`__] - -Their git repositories are `mirrored at Gitlab <https://gitlab.com/gnu-taler/taler-android>`__ -to utilize their CI -and `F-Droid <https://f-droid.org>`_'s Gitlab integration -to `publish automatic nightly builds <https://f-droid.org/docs/Publishing_Nightly_Builds/>`_ -for each change on the ``master`` branch. - -All three apps publish their builds to the same F-Droid nightly repository -(which is stored as a git repository): -https://gitlab.com/gnu-taler/fdroid-repo-nightly - -You can download the APK files directly from that repository -or add it to the F-Droid app for automatic updates -by clicking the following link (on the phone that has F-Droid installed). - - `GNU Taler Nightly F-Droid Repository <fdroidrepos://gnu-taler.gitlab.io/fdroid-repo-nightly/fdroid/repo?fingerprint=55F8A24F97FAB7B0960016AF393B7E57E7A0B13C2D2D36BAC50E1205923A7843>`_ - -.. note:: - Nightly apps can be installed alongside official releases - and thus are meant **only for testing purposes**. - Use at your own risk! - -.. _Build-apps-from-source: - -Building apps from source -------------------------- - -Note that this guide is different from other guides for building Android apps, -because it does not require you to run non-free software. -It uses the Merchant PoS Terminal as an example, but works as well for the other apps -if you replace ``merchant-terminal`` with ``wallet`` or ``cashier``. - -First, ensure that you have the required dependencies installed: - -* Java Development Kit 8 or higher (default-jdk-headless) -* git -* unzip - -Then you can get the app's source code using git: - -.. code-block:: console - - # Start by cloning the Android git repository - $ git clone https://git.taler.net/taler-android.git - - # Change into the directory of the cloned repository - $ cd taler-android - - # Find out which Android SDK version you will need - $ grep -i compileSdkVersion merchant-terminal/build.gradle - -The last command will return something like ``compileSdkVersion 29``. -So visit the `Android Rebuilds <http://android-rebuilds.beuc.net/>`_ project -and look for that version of the Android SDK there. -If the SDK version is not yet available as a free rebuild, -you can try to lower the ``compileSdkVersion`` in the app's ``merchant-terminal/build.gradle`` file. -Note that this might break things -or require you to also lower other versions such as ``targetSdkVersion``. - -In our example, the version is ``29`` which is available, -so download the "SDK Platform" package of "Android 10.0.0 (API 29)" -and unpack it: - -.. code-block:: console - - # Change into the directory that contains your downloaded SDK - $ cd $HOME - - # Unpack/extract the Android SDK - $ unzip android-sdk_eng.10.0.0_r14_linux-x86.zip - - # Tell the build system where to find the SDK - $ export ANDROID_SDK_ROOT="$HOME/android-sdk_eng.10.0.0_r14_linux-x86" - - # Change into the directory of the cloned repository - $ cd taler-android - - # Build the merchant-terminal app - $ ./gradlew :merchant-terminal:assembleRelease - -If you get an error message complaining about build-tools - - > Failed to install the following Android SDK packages as some licences have not been accepted. - build-tools;29.0.3 Android SDK Build-Tools 29.0.3 - -you can try changing the ``buildToolsVersion`` in the app's ``merchant-terminal/build.gradle`` file -to the latest "Android SDK build tools" version supported by the Android Rebuilds project. - -After the build finished successfully, -you will find your APK in ``merchant-terminal/build/outputs/apk/release/``. - -Update translations -------------------- - -Translations are managed with Taler's weblate instance: -https://weblate.taler.net/projects/gnu-taler/ - -To update translations, enter the taler-android git repository -and ensure that the weblate remote exists: - -.. code-block:: console - - $ git config -l | grep weblate - -If it does not yet exist (empty output), you can add it like this: - -.. code-block:: console - - $ git remote add weblate https://weblate.taler.net/git/gnu-taler/wallet-android/ - -Then you can merge in translations commit from the weblate remote: - -.. code-block:: console - - # ensure you have latest version - $ git fetch weblate - - # merge in translation commits - $ git merge weblate/master - -Afterwards, build the entire project from source and test the UI -to ensure that no erroneous translations (missing placeholders) are breaking things. - -Release process ---------------- - -After extensive testing, the code making up a new release should get a signed git tag. -The current tag format is: - -* cashier-$VERSION -* pos-$VERSION -* wallet-$VERSION (where $VERSION has a v prefix) - -.. code-block:: console - - $ git tag -s $APP-$VERSION - -F-Droid -^^^^^^^ -Nightly builds get published automatically (see above) after pushing code to the official repo. -Actual releases get picked up by F-Droid's official repository via git tags. -So ensure that all releases get tagged properly. - -Some information for F-Droid official repository debugging: - -* Wallet: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.wallet.fdroid.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.wallet.fdroid/lastbuild>`__] -* Cashier: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.cashier.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.cashier/lastbuild>`__] -* PoS: [`metadata <https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/net.taler.merchantpos.yml>`__] [`build log <https://f-droid.org/wiki/page/net.taler.merchantpos/lastbuild>`__] - -Google Play -^^^^^^^^^^^ -Google Play uploads are managed via `Fastlane <https://docs.fastlane.tools/getting-started/android/setup/>`__. -Before proceeding, ensure that this is properly set up -and that you have access to the Google Play API. - -It is important to have access to the signing keys and Google Play access keys -(JSON) and to ensure that the following environment variables are set -correctly and made available to Fastlane: - -.. code-block:: bash - - TALER_KEYSTORE_PATH= - TALER_KEYSTORE_PASS= - TALER_KEYSTORE_WALLET_ALIAS= - TALER_KEYSTORE_WALLET_PASS= - TALER_KEYSTORE_POS_ALIAS= - TALER_KEYSTORE_POS_PASS= - TALER_KEYSTORE_CASHIER_ALIAS= - TALER_KEYSTORE_CASHIER_PASS= - TALER_JSON_KEY_FILE= - -To release an app, enter into its respective folder and run fastlane: - -.. code-block:: console - - $ bundle exec fastlane - -Then select the deploy option. - -All uploads are going to the beta track by default. These can be promoted to -production later or immediately after upload if you feel daring. It is also -important to bump the version and build code with every release. - -.. _Code-coverage: - -Code Coverage -============= - -Code coverage is done with the Gcov / Lcov -(http://ltp.sourceforge.net/coverage/lcov.php) combo, and it is run -nightly (once a day) by a Buildbot worker. The coverage results are -then published at https://lcov.taler.net/ . - - -Coding Conventions -================== - -GNU Taler is developed primarily in C, Kotlin, Python, Swift and TypeScript. - -Components written in C ------------------------ - -These are the general coding style rules for Taler. - -* Baseline rules are to follow GNU guidelines, modified or extended - by the GNUnet style: https://docs.gnunet.org/handbook/gnunet.html#Coding-style - -Naming conventions -^^^^^^^^^^^^^^^^^^ - -* include files (very similar to GNUnet): - - * if installed, must start with "``taler_``" (exception: platform.h), - and MUST live in src/include/ - * if NOT installed, must NOT start with "``taler_``" and - MUST NOT live in src/include/ and - SHOULD NOT be included from outside of their own directory - * end in "_lib" for "simple" libraries - * end in "_plugin" for plugins - * end in "_service" for libraries accessing a service, i.e. the exchange - -* binaries: - - * taler-exchange-xxx: exchange programs - * taler-merchant-xxx: merchant programs (demos) - * taler-wallet-xxx: wallet programs - * plugins should be libtaler_plugin_xxx_yyy.so: plugin yyy for API xxx - * libtalerxxx: library for API xxx - -* logging - - * tools use their full name in GNUNET_log_setup - (i.e. 'taler-exchange-offline') and log using plain 'GNUNET_log'. - * pure libraries (without associated service) use 'GNUNET_log_from' - with the component set to their library name (without lib or '.so'), - which should also be their directory name (i.e. 'util') - * plugin libraries (without associated service) use 'GNUNET_log_from' - with the component set to their type and plugin name (without lib or '.so'), - which should also be their directory name (i.e. 'exchangedb-postgres') - * libraries with associated service) use 'GNUNET_log_from' - with the name of the service, which should also be their - directory name (i.e. 'exchange') - * for tools with ``-l LOGFILE``, its absence means write logs to stderr - -* configuration - - * same rules as for GNUnet - -* exported symbols - - * must start with TALER_[SUBSYSTEMNAME]_ where SUBSYSTEMNAME - MUST match the subdirectory of src/ in which the symbol is defined - * from libtalerutil start just with ``TALER_``, without subsystemname - * if scope is ONE binary and symbols are not in a shared library, - use binary-specific prefix (such as TMH = taler-exchange-httpd) for - globals, possibly followed by the subsystem (TMH_DB_xxx). - -* structs: - - * structs that are 'packed' and do not contain pointers and are - thus suitable for hashing or similar operations are distinguished - by adding a "P" at the end of the name. (NEW) Note that this - convention does not hold for the GNUnet-structs (yet). - * structs that are used with a purpose for signatures, additionally - get an "S" at the end of the name. - -* private (library-internal) symbols (including structs and macros) - - * must not start with ``TALER_`` or any other prefix - -* testcases - - * must be called "test_module-under-test_case-description.c" - -* performance tests - - * must be called "perf_module-under-test_case-description.c" - -Shell Scripts -------------- - -Shell scripts should be avoided if at all possible. The only permissible uses of shell scripts -in GNU Taler are: - -* Trivial invocation of other commands. -* Scripts for compatibility (e.g. ``./configure``) that must run on - as many systems as possible. - -When shell scripts are used, they ``MUST`` begin with the following ``set`` command: - -.. code-block:: console - - # Make the shell fail on undefined variables and - # commands with non-zero exit status. - $ set -eu - -Kotlin ------- - -We so far have no specific guidelines, please follow best practices -for the language. - - -Python ------- - -Supported Python Versions -^^^^^^^^^^^^^^^^^^^^^^^^^ - -Python code should be written and build against version 3.7 of Python. - -Style -^^^^^ - -We use `yapf <https://github.com/google/yapf>`_ to reformat the -code to conform to our style instructions. -A reusable yapf style file can be found in ``build-common``, -which is intended to be used as a git submodule. - -Python for Scripting -^^^^^^^^^^^^^^^^^^^^ - -When using Python for writing small utilities, the following libraries -are useful: - -* ``click`` for argument parsing (should be preferred over argparse) -* ``pathlib`` for path manipulation (part of the standard library) -* ``subprocess`` for "shelling out" to other programs. Prefer ``subprocess.run`` - over the older APIs. - - -Swift ------ - -Please follow best practices for the language. - - -TypeScript ----------- - -Please follow best practices for the language. - - -Testing library -=============== - -This chapter is a VERY ABSTRACT description of how testing is -implemented in Taler, and in NO WAY wants to substitute the reading of -the actual source code by the user. - -In Taler, a test case is an array of ``struct TALER_TESTING_Command``, -informally referred to as ``CMD``, that is iteratively executed by the -testing interpreter. This latter is transparently initiated by the -testing library. - -However, the developer does not have to defined CMDs manually, but -rather call the proper constructor provided by the library. For example, -if a CMD is supposed to test feature ``x``, then the library would -provide the ``TALER_TESTING_cmd_x ()`` constructor for it. Obviously, -each constructor has its own particular arguments that make sense to -test ``x``, and all constructor are thoroughly commented within the -source code. - -Internally, each CMD has two methods: ``run ()`` and ``cleanup ()``. The -former contains the main logic to test feature ``x``, whereas the latter -cleans the memory up after execution. - -In a test life, each CMD needs some internal state, made by values it -keeps in memory. Often, the test has to *share* those values with other -CMDs: for example, CMD1 may create some key material and CMD2 needs this -key material to encrypt data. - -The offering of internal values from CMD1 to CMD2 is made by *traits*. A -trait is a ``struct TALER_TESTING_Trait``, and each CMD contains an array -of traits, that it offers via the public trait interface to other -commands. The definition and filling of such array happens transparently -to the test developer. - -For example, the following example shows how CMD2 takes an amount object -offered by CMD1 via the trait interface. - -Note: the main interpreter and the most part of CMDs and traits are -hosted inside the exchange codebase, but nothing prevents the developer -from implementing new CMDs and traits within other codebases. - -.. code-block:: c - - /* Without loss of generality, let's consider the - * following logic to exist inside the run() method of CMD1 */ - ... - - struct TALER_Amount *a; - /** - * the second argument (0) points to the first amount object offered, - * in case multiple are available. - */ - if (GNUNET_OK != TALER_TESTING_get_trait_amount_obj (cmd2, 0, &a)) - return GNUNET_SYSERR; - ... - - use(a); /* 'a' points straight into the internal state of CMD2 */ - -In the Taler realm, there is also the possibility to alter the behaviour -of supposedly well-behaved components. This is needed when, for example, -we want the exchange to return some corrupted signature in order to -check if the merchant backend detects it. - -This alteration is accomplished by another service called *twister*. The -twister acts as a proxy between service A and B, and can be programmed -to tamper with the data exchanged by A and B. - -Please refer to the Twister codebase (under the ``test`` directory) in -order to see how to configure it. - - -User-Facing Terminology -======================= - -This section contains terminology that should be used and that should not be -used in the user interface and help materials. - -Terms to Avoid --------------- - -Refreshing - Refreshing is the internal technical terminology for the protocol to - give change for partially spent coins - - **Use instead**: "Obtaining change" - -Charge - Charge has two opposite meanings (charge to a credit card vs. charge a battery). - This can confuse users. - - **Use instead**: "Obtain", "Credit", "Debit", "Withdraw", "Top up" - -Coin - Coins are an internal construct, the user should never be aware that their balance - is represented by coins of different denominations. - - **Use instead**: "(Digital) Cash" or "(Wallet) Balance" - -Consumer - Has bad connotation of consumption. - - **Use instead**: Customer or user. - -Proposal - The term used to describe the process of the merchant facilitating the download - of the signed contract terms for an order. - - **Avoid**. Generally events that relate to proposal downloads - should not be shown to normal users, only developers. Instead, use - "communication with mechant failed" if a proposed order can't be downloaded. - -Anonymous E-Cash - Should be generally avoided, since Taler is only anonymous for - the customer. Also some people are scared of anonymity (which as - a term is also way too absolute, as anonymity is hardly ever perfect). - - **Use instead**: "Privacy-preserving", "Privacy-friendly" - -Payment Replay - The process of proving to the merchant that the customer is entitled - to view a digital product again, as they already paid for it. - - **Use instead**: In the event history, "re-activated digital content purchase" - could be used. (FIXME: this is still not nice.) - -Session ID - See Payment Replay. - -Order - Too ambiguous in the wallet. - - **Use instead**: Purchase - -Fulfillment URL - URL that the serves the digital content that the user purchased - with their payment. Can also be something like a donation receipt. - -Donau - Developer-internal name for the tax authority component. - - **Use instead**: Tax authority - -Terms to Use ------------- - -Auditor - Regulatory entity that certifies exchanges and oversees their operation. - -Exchange Operator - The entity/service that gives out digital cash in exchange for some - other means of payment. - - In some contexts, using "Issuer" could also be appropriate. - When showing a balance breakdown, - we can say "100 Eur (issued by exchange.euro.taler.net)". - Sometimes we may also use the more generic term "Payment Service Provider" - when the concept of an "Exchange" is still unclear to the reader. - -Refund - A refund is given by a merchant to the customer (rather the customer's wallet) - and "undoes" a previous payment operation. - -Payment - The act of sending digital cash to a merchant to pay for an order. - -Purchase - Used to refer to the "result" of a payment, as in "view purchase". - Use sparsingly, as the word doesn't fit for all payments, such as donations. - -Contract Terms - Partially machine-readable representation of the merchant's obligation after the - customer makes a payment. - -Merchant - Party that receives a payment. - -Wallet - Also "Taler Wallet". Software component that manages the user's digital cash - and payments. - - -Developer Glossary -================== - -This glossary is meant for developers. It contains some terms that we usually do not -use when talking to end users or even system administrators. - -.. glossary:: - :sorted: - - absolute time - method of keeping time in :term:`GNUnet` where the time is represented - as the number of microseconds since 1.1.1970 (UNIX epoch). Called - absolute time in contrast to :term:`relative time`. - - aggregate - the :term:`exchange` combines multiple payments received by the - same :term:`merchant` into one larger :term:`wire transfer` to - the respective merchant's :term:`bank` account - - auditor - trusted third party that verifies that the :term:`exchange` is operating correctly - - bank - traditional financial service provider who offers - :term:`wire transfers <wire transfer>` between accounts - - buyer - individual in control of a Taler :term:`wallet`, usually using it to - :term:`spend` the :term:`coins <coin>` on :term:`contracts <contract>` (see also :term:`customer`). - - close - operation an :term:`exchange` performs on a :term:`reserve` that has not been - :term:`emptied <empty>` by :term:`withdraw` operations. When closing a reserve, the - exchange wires the remaining funds back to the customer, minus a :term:`fee` - for closing - - customer - individual that directs the buyer (perhaps the same individual) to make a purchase - - coin - coins are individual token representing a certain amount of value, also known as the :term:`denomination` of the coin - - refresh commitment - data that the wallet commits to during the :term:`melt` stage of the - :term:`refresh` protocol where it - has to prove to the :term:`exchange` that it is deriving the :term:`fresh` - coins as specified by the Taler protocol. The commitment is verified - probabilistically (see: :term:`kappa`) during the :term:`reveal` stage. - - contract - formal agreement between :term:`merchant` and :term:`customer` specifying the - :term:`contract terms` and signed by the merchant and the :term:`coins <coin>` of the - customer - - contract terms - the individual clauses specifying what the buyer is purchasing from the - :term:`merchant` - - denomination - unit of currency, specifies both the currency and the face value of a :term:`coin`, - as well as associated fees and validity periods - - denomination key - (RSA) key used by the exchange to certify that a given :term:`coin` is valid and of a - particular :term:`denomination` - - deposit - operation by which a merchant passes coins to an exchange, expecting the - exchange to credit his bank account in the future using an - :term:`aggregate` :term:`wire transfer` - - drain - process by which an exchange operator takes the profits - (from :term:`fees <fee>`) out of the escrow account and moves them into - their regular business account - - dirty - a :term:`coin` is dirty if its public key may be known to an entity other than - the customer, thereby creating the danger of some entity being able to - link multiple transactions of coin's owner if the coin is not refreshed - - empty - a :term:`reserve` is being emptied when a :term:`wallet` is using the - reserve's private key to :term:`withdraw` coins from it. This reduces - the balance of the reserve. Once the balance reaches zero, we say that - the reserve has been (fully) emptied. Reserves that are not emptied - (which is the normal process) are :term:`closed <close>` by the exchange. - - exchange - Taler's payment service operator. Issues electronic coins during - withdrawal and redeems them when they are deposited by merchants - - expired - Various operations come with time limits. In particular, denomination keys - come with strict time limits for the various operations involving the - coin issued under the denomination. The most important limit is the - deposit expiration, which specifies until when wallets are allowed to - use the coin in deposit or refreshing operations. There is also a "legal" - expiration, which specifies how long the exchange keeps records beyond the - deposit expiration time. This latter expiration matters for legal disputes - in courts and also creates an upper limit for refreshing operations on - special zombie coin - - GNUnet - Codebase of various libraries for a better Internet, some of which - GNU Taler depends upon. - - fakebank - implementation of the :term:`bank` API in memory to be used only for test - cases. - - fee - an :term:`exchange` charges various fees for its service. The different - fees are specified in the protocol. There are fees per coin for - :term:`withdrawing <withdraw>`, :term:`depositing <deposit>`, :term:`melting <melt>`, and - :term:`refunding <refund>`. Furthermore, there are fees per wire transfer - when a :term:`reserve` is :term:`closed <close>` - and for :term:`aggregate` :term:`wire transfers <wire transfer>` - to the :term:`merchant`. - - fresh - a :term:`coin` is fresh if its public key is only known to the customer - - JSON - JavaScript Object Notation (JSON) is a - serialization format derived from the JavaScript language which is - commonly used in the Taler protocol as the payload of HTTP requests - and responses. - - kappa - security parameter used in the :term:`refresh` protocol. Defined to be 3. - The probability of successfully evading the income transparency with the - refresh protocol is 1:kappa. - - libeufin - Kotlin component that implements a regional currency bank and an - adapter to communicate via EBICS with European core banking systems. - - link - specific step in the :term:`refresh` protocol that an exchange must offer - to prevent abuse of the :term:`refresh` mechanism. The link step is - not needed in normal operation, it just must be offered. - - master key - offline key used by the exchange to certify denomination keys and - message signing keys - - melt - step of the :term:`refresh` protocol where a :term:`dirty` :term:`coin` - is invalidated to be reborn :term:`fresh` in a subsequent - :term:`reveal` step. - - merchant - party receiving payments (usually in return for goods or services) - - message signing key - key used by the exchange to sign online messages, other than coins - - order - offer made by the merchant to a wallet; pre-cursor to - a contract where the wallet is not yet fixed. Turns - into a :term:`contract` when a wallet claims the order. - - owner - a coin is owned by the entity that knows the private key of the coin - - relative time - method of keeping time in :term:`GNUnet` where the time is represented - as a relative number of microseconds. Thus, a relative time specifies - an offset or a duration, but not a date. Called relative time in - contrast to :term:`absolute time`. - - recoup - Operation by which an exchange returns the value of coins affected - by a :term:`revocation <revoke>` to their :term:`owner`, either by allowing the owner to - withdraw new coins or wiring funds back to the bank account of the :term:`owner`. - - planchet - precursor data for a :term:`coin`. A planchet includes the coin's internal - secrets (coin private key, blinding factor), but lacks the RSA signature - of the :term:`exchange`. When :term:`withdrawing <withdraw>`, a :term:`wallet` - creates and persists a planchet before asking the exchange to sign it to - get the coin. - - purchase - Refers to the overall process of negotiating a :term:`contract` and then - making a payment with :term:`coins <coin>` to a :term:`merchant`. - - privacy policy - Statement of an operator how they will protect the privacy of users. - - proof - Message that cryptographically demonstrates that a particular claim is correct. - - proposal - a list of :term:`contract terms` that has been completed and signed by the - merchant backend. - - refresh - operation by which a :term:`dirty` :term:`coin` is converted into one or more - :term:`fresh` coins. Involves :term:`melting <melt>` the :term:`dirty` coins and - then :term:`revealing <reveal>` so-called :term:`transfer keys <transfer key>`. - - refund - operation by which a merchant steps back from the right to funds that he - obtained from a :term:`deposit` operation, giving the right to the funds back - to the customer - - refund transaction id - unique number by which a merchant identifies a :term:`refund`. Needed - as refunds can be partial and thus there could be multiple refunds for - the same :term:`purchase`. - - reserve - accounting mechanism used by the exchange to track customer funds - from incoming :term:`wire transfers <wire transfer>`. A reserve is created whenever - a customer wires money to the exchange using a well-formed public key - in the subject. The exchange then allows the customer's :term:`wallet` - to :term:`withdraw` up to the amount received in :term:`fresh` - :term:`coins <coin>` from the reserve, thereby emptying the reserve. If a - reserve is not emptied, the exchange will eventually :term:`close` it. - - Other definition: Funds set aside for future use; either the balance of a customer at the - exchange ready for withdrawal, or the funds kept in the exchange;s bank - account to cover obligations from coins in circulation. - - reveal - step in the :term:`refresh` protocol where some of the transfer private - keys are revealed to prove honest behavior on the part of the wallet. - In the reveal step, the exchange returns the signed :term:`fresh` coins. - - revoke - exceptional operation by which an exchange withdraws a denomination from - circulation, either because the signing key was compromised or because - the exchange is going out of operation; unspent coins of a revoked - denomination are subjected to recoup. - - sharing - users can share ownership of a :term:`coin` by sharing access to the coin&#39;s - private key, thereby allowing all co-owners to spend the coin at any - time. - - spend - operation by which a customer gives a merchant the right to deposit - coins in return for merchandise - - transfer key - special cryptographic key used in the :term:`refresh` protocol, some of which - are revealed during the :term:`reveal` step. Note that transfer keys have, - despite the name, no relationship to :term:`wire transfers <wire transfer>`. They merely - help to transfer the value from a :term:`dirty` coin to a :term:`fresh` coin - - terms - the general terms of service of an operator, possibly including - the :term:`privacy policy`. Not to be confused with the - :term:`contract terms` which are about the specific purchase. - - transaction - method by which ownership is exclusively transferred from one entity - - user - any individual using the Taler payment system - (see :term:`customer`, :term:`buyer`, :term:`merchant`). - - version - Taler uses various forms of versioning. There is a database - schema version (stored itself in the database, see \*-0000.sql) describing - the state of the table structure in the database of an :term:`exchange`, - :term:`auditor` or :term:`merchant`. There is a protocol - version (CURRENT:REVISION:AGE, see GNU libtool) which specifies - the network protocol spoken by an :term:`exchange` or :term:`merchant` - including backwards-compatibility. And finally there is the software - release version (MAJOR.MINOR.PATCH, see https://semver.org/) of - the respective code base. - - wallet - software running on a customer's computer; withdraws, stores and - spends coins - - WebExtension - Cross-browser API used to implement the GNU Taler wallet browser extension. - - wire gateway - API used by the exchange to talk with some real-time gross settlement system - (core banking system, blockchain) to notice inbound credits wire transfers - (during withdraw) and to trigger outbound debit wire transfers (primarily - for deposits). - - wire transfer - a wire transfer is a method of sending funds between :term:`bank` accounts - - wire transfer identifier - Subject of a wire transfer from the exchange to a merchant; - set by the aggregator to a random nonce which uniquely - identifies the transfer. - - withdraw - operation by which a :term:`wallet` can convert funds from a :term:`reserve` to - fresh coins - - zombie - :term:`coin` where the respective :term:`denomination key` is past its - :term:`deposit` :term:`expiration <expired>` time, but which is still (again) valid - for an operation because it was :term:`melted <melt>` while it was still - valid, and then later again credited during a :term:`recoup` process - - - -Developer Tools -=============== - -This section describes various internal programs to make life easier for the -developer. - - -taler-harness -------------- - -**taler-harness deployment gen-coin-config** is a tool to simplify Taler configuration generation. - - -**taler-harness deployment gen-coin-config** -[**-min-amount**=**\ ‌\ *VALUE*] -[**-max-amount**=**\ ‌\ *VALUE*] diff --git a/taler-wallet.rst b/taler-wallet.rst @@ -1,657 +0,0 @@ -.. - This file is part of GNU TALER. - Copyright (C) 2014-2024 Taler Systems SA - - TALER is free software; you can redistribute it and/or modify it under the - terms of the GNU Affero General Public License as published by the Free Software - Foundation; either version 2.1, or (at your option) any later version. - - TALER is distributed in the hope that it will be useful, but WITHOUT ANY - WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR - A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. - - You should have received a copy of the GNU Affero General Public License along with - TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> - - -Wallet Developer Manual -####################### - -The GNU Taler wallet allows customers to withdraw and spend digital cash. - - -WebExtension Wallet -=================== - -Building from source --------------------- - -.. code-block:: console - - $ git clone https://git.taler.net/wallet-core.git - $ cd wallet-core - $ ./configure - $ make webex-stable - # Packaged extension now available as: - # dist/taler-wallet-$VERSION.zip - - -Android Wallet -============== - -Please see :ref:`Build-apps-from-source` in the :doc:`taler-developer-manual`. - - -iOS Wallet -========== - -Please see :ref:`Build-iOS-from-source` in the :doc:`taler-developer-manual`. - -.. _command-line-wallet: - -Command-line Wallet -=================== - -This section describes how to use the GNU Taler wallet command line -interface (CLI). - -The the wallet CLI is targeted at developers and operators, but not meant to be -used by customers. It exposes all functionality that the more user-friendly -interfaces (Android app, browser extension) offer. However, it provides more -diagnostics and advanced features as well. - -Building from source --------------------- - -The easiest way to install the wallet is via NPM. Note that a recent version of -Node.JS (``>=12.20.1``) is required. - -We recommend to install the wallet package on a per-user basis, -thus setting ``$INSTALL_PREFIX`` to a directory in ``$HOME``. - -.. code-block:: console - - $ git clone https://git.taler.net/wallet-core.git - $ cd wallet-core - $ ./bootstrap - $ ./configure --prefix=$INSTALL_PREFIX - $ make && make install - -The wallet command-line interface should then be available as ``taler-wallet-cli`` under ``$INSTALL_PREFIX/bin``. - -Installation via NPM --------------------- - -The wallet can also obtained via NPM, the Node Package Manager. - -To install the wallet as a global package, run: - -.. code-block:: console - - $ npm install -g taler-wallet - # check if installation was successful - $ taler-wallet-cli --version - -To install the wallet only for your user, run: - -.. code-block:: console - - $ npm install -g --prefix=$HOME/local taler-wallet - # check if installation was successful - $ taler-wallet-cli --version - # If this fails, make sure that $HOME/local/bin is in your $PATH - -To use the wallet as a library in your own project, run: - -.. code-block:: console - - $ npm install taler-wallet - - -Getting Help ------------- - -The wallet CLI comes with built-in help. Invoke the wallet CLI (or any subcommand) with the ``--help`` flag to get help: - -.. code-block:: console - - $ taler-wallet-cli --help - Usage: taler-wallet-cli COMMAND - - Command line interface for the GNU Taler wallet. - - Options: - -h, --help Show this message and exit. - --wallet-db=VALUE location of the wallet database file - --timetravel=VALUE modify system time by given offset in microseconds - --inhibit=VALUE Inhibit running certain operations, useful for debugging and testing. - --no-throttle Don't do any request throttling. - -v, --version - -V, --verbose Enable verbose output. - - Commands: - advanced Subcommands for advanced operations (only use if you know what you're doing!). - api Call the wallet-core API directly. - backup Subcommands for backups - balance Show wallet balance. - deposit Subcommands for depositing money to payto:// accounts - exchanges Manage exchanges. - handle-uri Handle a taler:// URI. - pending Show pending operations. - run-pending Run pending operations. - run-until-done Run until no more work is left. - testing Subcommands for testing GNU Taler deployments. - transactions Show transactions. - -Completing operations ---------------------- - -Note that the CLI does not run as a background daemon. When starting -operations that don't immediately finish, the wallet needs to be run explicitly -to finish any pending tasks: - - -.. code-block:: console - - # Do one attempt to finish all pending operations - $ taler-wallet-cli run-pending - - # Run until all work is done - $ taler-wallet-cli run-until-done - -Resetting the wallet --------------------- - -The wallet can be reset by deleting its database file. By default, the database file -is ``$HOME/.talerwalletdb.sqlite3``. - - -Handling taler:// URIs ----------------------- - -Many interactions with the Taler wallet happen by scanning QR codes or special -headers on Websites. To emulate this with the command line interface, run the following -command: - -.. code-block:: console - - $ taler-wallet-cli handle-uri $URI - - -Manual withdrawing ------------------- - -.. code-block:: console - - $ taler-wallet-cli advanced withdraw-manually \ - --exchange https://exchange.eurint.taler.net/ \ - --amount EUR:5 - - -P2P push payments ------------------ - -The following code generates a P2P push transaction over 1 CHF -with an expiration time of 30 days (assuming the wallet has a -sufficient balance): - -.. code-block:: console - - $ taler-wallet-cli p2p initiate-push-debit \ - --purse-expiration="30 d" \ - --summary="The summary" \ - CHF:1 - -The final URL can then be found in the transaction list: - -.. code-block:: console - - $ taler-wallet-cli transactions - -Background wallet ------------------ - -A wallet can be launched in the background: - -.. code-block:: console - - $ taler-wallet-cli advanced serve & - -You can then run various Taler operations faster against -this one persistent instance: - -.. code-block:: console - - $ taler-wallet-cli --wallet-connection=wallet-core.sock ... - -Here ``...`` needs to be changed to the commando to run. -Make sure to run - -.. code-block:: console - - $ taler-wallet-cli --wallet-connection=wallet-core.sock \ - run-until-done - -to wait for pending transactions to complete. - - -Testing an exchange deployment ------------------------------- - -The following series of commands can be used to check that an exchange deployment -is functional: - -.. code-block:: console - - # This will now output a payto URI that money needs to be sent to in order to allow withdrawal - # of taler coins - $ taler-wallet-cli advanced withdraw-manually --exchange $EXCHANGE_URL --amount EUR:10.50 - - # Show the status of the manual withdrawal operation - $ taler-wallet-cli transactions - - # Once the transfer has been made, try completing the withdrawal - $ taler-wallet-cli run-pending - - # Check status of transactions and show balance - $ taler-wallet-cli transactions - $ taler-wallet-cli balance - - # Now, directly deposit coins with the exchange into a target account - # (Usually, a payment is made via a merchant. The wallet provides - # this functionality for testing.) - $ taler-wallet-cli deposit create EUR:5 payto://iban/$IBAN - - # Check if transaction was successful. - # (If not, fix issue with exchange and run "run-pending" command again) - $ taler-wallet-cli transactions - - # The wallet can also track if the exchange wired the money to the merchant account. - # The "deposit group id" can be found in the output of the transactions list. - $ taler-wallet-cli deposit track $DEPOSIT_GROUP_ID - - -APIs and Data Formats -===================== - -Envelope Format ---------------- - -All API responses and notifications are returned in the -following envelope: - -.. ts:def:: WalletResponseEnvelope - - type WalletResponseEnvelope = - | WalletSuccess - | WalletError - | WalletNotification - -.. ts:def:: WalletSuccess - - export interface WalletSuccess { - type: "response"; - operation: string; - // ID to correlate success response to request - id: string; - // Result type depends on operation - result: unknown; - } - -.. ts:def:: WalletError - - export interface WalletError { - type: "error"; - operation: string; - // ID to correlate error response to request - id: string; - error: WalletErrorInfo; - } - -.. ts:def:: WalletNotification - - export interface WalletSuccess { - type: "notification"; - - // actual type is WalletNotification, - // to be documented here - payload: any; - } - -.. ts:def:: WalletErrorInfo - - export interface WalletErrorInfo { - // Numeric error code defined defined in the - // GANA gnu-taler-error-codes registry. - talerErrorCode: number; - - // English description of the error code. - talerErrorHint: string; - - // English diagnostic message that can give details - // for the instance of the error. - message: string; - - // Error details, type depends - // on talerErrorCode - details: unknown; - } - -Withdrawal ----------- - -A typical API sequence for *bank-integrated* withdrawals can for example look like this: - -#. ``"getWithdrawalDetailsForUri"`` returns an amount and default exchange -#. ``"getWithdrawalDetailsForAmount"`` returns fee information and that ToS are not accepted - - #. ``"getExchangeTos"`` are shown to the user and return currentEtag - #. ``"setExchangeTosAccepted"`` called with currentEtag after user accepted - -#. ``"acceptWithdrawal"`` after the user confirmed withdrawal with associated fees - -A typical API sequence for *manual* withdrawals can for example look like this: - -#. ``"listExchanges"`` shows a list of exchanges to the user who picks one and an amount -#. ``"getWithdrawalDetailsForAmount"`` returns fee information and that ToS are not accepted - - #. ``"getExchangeTos"`` are shown to the user and return currentEtag - #. ``"setExchangeTosAccepted"`` called with currentEtag after user accepted - -#. ``"acceptManualWithdrawal"`` after the user confirmed withdrawal with associated fees - -Integration Tests -================= - -Integration Test Example ------------------------- - -Integration tests can be done with the low-level wallet commands. To select which coins and denominations -to use, the wallet can dump the coins in an easy-to-process format (`CoinDumpJson <https://git.taler.net/taler-typescript-core.git/tree/packages/taler-util/src/types-taler-wallet.ts#n613>`__). - -The database file for the wallet can be selected with the ``--wallet-db`` -option. This option must be passed to the ``taler-wallet-cli`` command and not -the subcommands. If the database file doesn't exist, it will be created. - -The following example does a simple withdrawal recoup: - -.. code-block:: console - - # Withdraw digital cash - $ taler-wallet-cli --wallet-db=mydb.sqlite3 testing withdraw \ - -b https://bank.int.taler.net/ \ - -e https://exchange.int.taler.net/ \ - -a INTKUDOS:10 - - $ coins=$(taler-wallet-cli --wallet-db=mydb.sqlite3 advanced dump-coins) - - # Find coin we want to revoke - $ rc=$(echo "$coins" | \ - jq -r '[.coins[] | select((.denom_value == "INTKUDOS:5"))][0] | .coin_pub') - - # Find the denom - $ rd=$(echo "$coins" | \ - jq -r '[.coins[] | select((.denom_value == "INTKUDOS:5"))][0] | .denom_pub_hash') - - # Find all other coins, which will be suspended - $ susp=$(echo "$coins" | \ - jq --arg rc "$rc" '[.coins[] | select(.coin_pub != $rc) | .coin_pub]') - - # The exchange revokes the denom - $ taler-exchange-keyup -r $rd - $ taler-deployment-restart - - # Now we suspend the other coins, so later we will pay with the recouped coin - $ taler-wallet-cli --wallet-db=mydb.sqlite3 advanced suspend-coins "$susp" - - # Update exchange /keys so recoup gets scheduled - $ taler-wallet-cli --wallet-db=mydb.sqlite3 exchanges update -f https://exchange.int.taler.net/ - - # Block until scheduled operations are done - $ taler-wallet-cli --wallet-db=mydb.sqlite3 run-until-done - - # Now we buy something, only the coins resulting from recouped will be - # used, as other ones are suspended - $ taler-wallet-cli --wallet-db=mydb.sqlite3 testing test-pay \ - -m https://backend.int.taler.net/ \ - -k sandbox \ - -a "INTKUDOS:1" \ - -s "foo" - $ taler-wallet-cli --wallet-db=mydb.sqlite3 run-until-done - - -To test refreshing, force a refresh: - -.. code-block:: console - - $ taler-wallet-cli --wallet-db=mydb.sqlite3 advanced force-refresh "$coin_pub" - - -To test zombie coins, use the timetravel option. It **must** be passed to the -top-level command and not the subcommand: - -.. code-block:: console - - # Update exchange /keys with time travel, value in microseconds - $ taler-wallet-cli --timetravel=1000000 --wallet-db=mydb.sqlite3 \ - exchanges update -f https://exchange.int.taler.net/ - - -Integration Test and Fault Injection Framework ----------------------------------------------- - -This section describes the current approach to integration testing in the wallet. - -It's all based on a TypeScript harness process, which itself implements -the fault injection proxy (async and in-process)! - -The new approach consists of the following parts: - -1. A strongly typed, convenient helper library to easily set up and run -arbitrary Taler deployments and run test cases. These components plug -together as easily as lego bricks, even with multiple -exchanges/merchants/banks/etc. Logs and clean shutdown (even on SIGINT -or errors) are handled properly. (Support for auditors is still pending -but needed to fully test the wallet.) - -This is how a simple withdrawal and payment test case looks like: -`<https://git.taler.net/taler-typescript-core.git/tree/packages/taler-harness/src/integrationtests/test-payment.ts>`__ - -(What's particularly nice is that all our docs contain TypeScript -definitions for all API request bodies. So just copying them into the -test harness gives us auto-completion and compile-time checks to avoid -typos. The wallet's JSON validation machinery is also re-used.) - -2. A fault injection proxy that can be plugged between the services -and/or the wallet. It runs alongside the test harness, and can thus can -use arbitrary custom logic. There's no dependency for it other than -built-in Node.JS libraries. Simple fault injections are just as easy to -set up as with the twister. - -The following test case (a) logs all requests and responses to the test -harness stdout and (b) at a certain point, starts dropping the next 10 -requests to the exchange (testing the wallet's retry logic): - -`<https://git.taler.net/taler-typescript-core.git/tree/packages/taler-harness/src/integrationtests/test-payment-fault.ts#n165>`__ - -3. All util functionality from JS wallet-core, such as the Taler crypto, -amount/date/etc. handling and JSON parsing/validation (the wallet is now -more modular and easier to use as a library) can be used in the -integration tests, even if a different wallet (Kotlin, whatever) is -tested via the CLI. - -4. A bunch of test cases that use (1)-(3). These are *significantly* -more readable and hackable than other test approaches we had, while -allowing for more complex scenarios. There are still way too few tests -though! - -5. A test runner (written in bash) that runs test cases based on a glob -pattern and reports the results. - -Injecting a fault is as easy as: - -.. code:: ts - - // Set up test case - [...] - - exchangeProxy.addFault({ - beforeResponse(ctx: FaultInjectionResponseContext) { - if (cond1) { // Drop some responses - ctx.dropResponse = true; - return; - } else if (cond2) { // modify some others - ctx.responseBody = Buffer.from(`{"oops": true}`, "utf-8"); - return; - } - // Other things that can be modified: - // - drop/modify the request, not just the response - // - modify headers - // - modify status codes - } - }); - - await doSomethingWithTheWallet(); - - exchangeProxy.clearFault(); - - await doMoreWithTheWallet(); - - -To make the configuration easy, an ``ExchangeService`` (or ``MerchantService``, -``BankService`` etc.) can be wrapped in a ``FaultInjectedExchangeService``, -which implements the ``ExchangeServiceInterface``: - -.. code:: ts - - // create exchange and two merchants - const exchange = await setupExchange(...); - const merchant1 = ...; - const merchant2 = ...; - - // Add exchange to merchant-accepted exchanges. - // This will adjust the config. - merchant1.addExchange(exchange); - - // Wrap exchange in fault injection proxy - const faultInjectedExchange: ExchangeServiceInterface - = new FaultInjectedExchangeService(t, exchange1, 8085); - - // Merchant 2 talks to the exchange over fault injection, - // and thus must use the "twisted" base URL. - merchant2.addExchange(faultInjectedExchange); - - -The package for the integration tests is here: - -`<https://git.taler.net/wallet-core.git/tree/packages/taler-harness>`__ - -The integration tests are run via the ``taler-harness`` tool. - -.. code:: sh - - ./bootstrap && ./configure --prefix=... && make install - taler-harness run-integrationtests - - -Dev Experiments -=============== - -Dev experiments allow simulating certain scenarios that are difficult to -reproduce otherwise. This allows more comprehensive (manual) testing of the -UIs. - -You can enable dev experiments by putting the wallet into dev mode and then -scanning the QR code for a ``taler://dev-experiment`` URI that specifies the -desired dev experiment. - -Faking Protocol Versions ------------------------- - -The ``start-fakeprotover`` dev experiment can be used to fake the protocol -version reported by Taler components. It mocks the ``version`` field in the -response to ``/config`` or ``/keys``. - -Usage: - -.. code:: none - - taler://dev-experiment/start-fakeprotover?base_url=...&fake_ver=... - -Example: - -.. code:: none - - # Fake version 10:0:0 for https://exchange.demo.taler.net/ - taler://dev-experiment/start-fakeprotover?base_url=https%3A%2F%2Fexchange.demo.taler.net%2F&fake_ver=10%3A0%3A0 - - -Faking Transactions -------------------- - - -**Withdrawal Transaction:** - -.. code:: none - - taler://dev-experiment/add-fake-tx?txType=withdrawal&amountEffective=KUDOS:5 - -Options: - -* ``amountEffective``: Mandatory. Effective amount of the withdrawal. -* ``tRel``: Optional (defaults to ``0s``). Relative time that indicates - when in the past the transaction was started (and finished). -* ``exchangeBaseUrl``. Optional (defaults to ``https://exchange.demo.taler.net``). - Exchange base URL used for the withdrwal. - - -**Payment Transaction:** - -.. code:: none - - taler://dev-experiment/add-fake-tx?txType=payment&amountEffective=KUDOS:5 - -Options: - -* ``amountEffective``: Mandatory. Effective amount of the withdrawal. -* ``tRel``: Optional (defaults to ``0s``). Relative time that indicates - when in the past the transaction was started (and finished). -* ``exchangeBaseUrl``. Optional (defaults to ``https://exchange.demo.taler.net``). - Exchange base URL used for the payment. -* ``merchantBaseUrl``. Optional (defaults to ``https://backend.demo.taler.net/``). -* ``merchantName``. Optional (defaults to ``Test Merchant``). Display name - of the merchant in the contract terms of the transaction. -* ``summary``. Optional (defaults to ``Test``). Summary in the contract terms of the - transaction. - -**Peer Push Credit Transaction:** - -.. code:: none - - taler://dev-experiment/add-fake-tx?txType=peer-push-credit&amountEffective=KUDOS:5 - -Options: - -* ``amountEffective``: Mandatory. Effective amount of the withdrawal. -* ``tRel``: Optional (defaults to ``0s``). Relative time that indicates - when in the past the transaction was started (and finished). -* ``exchangeBaseUrl``. Optional (defaults to ``https://exchange.demo.taler.net``). - Exchange base URL used for the payment. -* ``summary``. Optional (defaults to ``Test``). Summary in the contract terms of the - transaction. - -**Peer Push Debit Transaction:** - -.. code:: none - - taler://dev-experiment/add-fake-tx?txType=peer-push-debit&amountEffective=KUDOS:5 - -Options: - -* ``amountEffective``: Mandatory. Effective amount of the withdrawal. -* ``tRel``: Optional (defaults to ``0s``). Relative time that indicates - when in the past the transaction was started (and finished). -* ``exchangeBaseUrl``. Optional (defaults to ``https://exchange.demo.taler.net``). - Exchange base URL used for the payment. -* ``summary``. Optional (defaults to ``Test``). Summary in the contract terms of the - transaction.