taler-docs

taler-auditor-manual.rst (44236B)

---

 ..
   This file is part of GNU TALER.
 
   Copyright (C) 2019-2021 Taler Systems SA
 
   TALER is free software; you can redistribute it and/or modify it under the
   terms of the GNU Affero General Public License as published by the Free Software
   Foundation; either version 2.1, or (at your option) any later version.
 
   TALER is distributed in the hope that it will be useful, but WITHOUT ANY
   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
   A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
 
   You should have received a copy of the GNU Affero General Public License along with
   TALER; see the file COPYING.  If not, see <http://www.gnu.org/licenses/>
 
   @author Christian Grothoff
 
 
 Auditor Operator Manual
 #######################
 
 Introduction
 ============
 
 This manual is an early draft that still needs significant editing work
 to become readable.
 
 About GNU Taler
 ---------------
 
 .. include:: frags/about-taler.rst
 
 
 About this manual
 -----------------
 
 This tutorial targets exchange operators, auditors and governments
 who want to run the auditor to verify that a GNU Taler exchange is
 operating correctly.
 
 
 Organizational prerequisites
 ----------------------------
 
 Operating a GNU Taler auditor means that you (henceforth: auditor) have a
 business relationship with (or regulatory authority over) a GNU Taler exchange
 operator (henceforth: exchange).  Your objective is to verify that the
 exchange is operating correctly, and if not to alert the exchange, the
 state or even the public about any misbehavior to limit financial losses
 to other parties.
 
 To perform this duty, you will need at least (read-only) access to the bank
 transactions of the exchange, as well as a continuously synchronized replica
 of the exchange's database.  The general assumption for running the auditor
 is that this is done on a separate system controlled by the auditor. After
 all, the goal is to detect nerfarious activity of the exchange operator,
 which cannot be effectively done on a machine controlled by the exchange
 operator.
 
 For this, every auditor needs to operate a PostgreSQL database.  The data
 collected will include sensitive information about Taler users, including
 withdrawals made by consumers and income received by merchants.  As a result,
 the auditor is expected to provide high confidentiality for the database.  In
 general, the auditor does not have to offer high-availability: the exchange
 operator can continue operations without the auditor, and the auditor can
 catch up with it later when the auditor's systems are restored. However, of
 course any downtime would provide a window of opportunity for fraud and should
 thus be minimized.  Finally, the auditor's copy of the exchange's database can
 be useful as a backup to the exchange in case the exchange experiences a loss
 of its own copies. Thus, business agreements between auditor and exchanges may
 include availability requirements as well.
 
 Then, with the software provided, auditors can verify the cryptographic proofs
 collected by the exchange and detect if any improper bank transactions have been
 made.  There are additional tasks which an auditor should perform.  While this
 manual only focuses on the audit of the exchange's database and wire transfers
 with the existing tools, a proper auditor should also perform the following
 tasks:
 
 - security audit of the source code
 - audit of the operational procedures of the exchange
 - audit of the physical security of the deployment
 - background check of the individuals operating the exchange
 - verification that the exchange properly implements the ``/link`` protocol
   (feature yet to be implemented in common Taler wallets)
 - verification that the exchange properly reports coins issued during
   the refresh protocol (by irregularly refreshing coins withdrawn by
   the auditor and comparing against the exchange's database --- the
   code required to support this is not yet implemented)
 
 
 Architecture overview
 ---------------------
 
 Taler is a pure payment system, not a new crypto-currency. As such, it
 operates in a traditional banking context. In particular, this means that in
 order to receive funds via Taler, the merchant must have a regular bank
 account, and payments can be executed in ordinary currencies such as USD or
 EUR. Similarly, the exchange must interact with a bank. The bank of the
 exchange holds the exchange’s funds in an escrow account.  As a result,
 exchanges operate in a regulated environment, and auditors provide a crucial
 oversight function.
 
 Auditors should generally be independent third parties that verify that the
 exchange operates correctly.  However, an exchange is likely to also run the
 auditing logic, as it is also used to calculate the exchange’s profits, risk
 and liabilities.  Furthermore, it's usually a good idea to not only rely on
 third parties to verify one's own work.
 
 The Taler software stack for an auditor consists of the following
 components:
 
 -  DBMS: PostgreSQL
 
    The auditor requires a DBMS to store a local copy of the transaction history for
    the Taler exchange, as well as for its own internal bookkeeping and checkpointing.
    The DBMS is assumed to be able to assure the auditor of the database invariants (foreign
    key, uniqueness, length restrictions).  Should the exported data from the exchange
    fail to be imported due to constraint violations, this is an immediate serious
    concern that must be addressed manually.  The software only verifies the content
    of a well-formed exchange database (well-formed with respect to SQL).
    For now, the GNU Taler reference implementation
    only supports PostgreSQL, but the code could be easily extended to
    support another DBMS.
 
 -  The auditor Web service
 
    The auditor is expected to provide a public Web service. At this REST API,
    merchants can (probabilistically) provide deposit confirmations, allowing
    the auditor to detect if an exchange is underreporting deposits.
 
    In the future, the Web service should be extended to allow customers and
    merchants to automatically upload cryptographic proof of other violations
    of the specification by the exchange.  However, for now it is assumed that
    the respective cryptographic proofs are reported and verified manually ---
    as with a well-behaved exchange this should obviously be a rare event.
 
    The main binary of this component is the ``taler-auditor-httpd``.
 
 -  The (main) auditor
 
    The main auditor logic checks the various signatures, totals up the
    amounts and checks for arithmetic inconsistencies. It also
    computes the expected bank balance, revenue and risk exposure of the
    exchange operator. The main script of this component is the ``taler-auditor``.
    This script invokes several helper binaries sequentially. Production
    users may want to modify the script to run those binaries in parallel,
    possibly using different privileges (as only the ``taler-helper-auditor-wire``
    needs access to the wire gateway).
 
    The ``taler-helper-auditor-wire`` auditor verifies that the bank
    transactions performed by the exchange
    were done properly.  This component must have access to the bank account
    of the exchange, as well as to a copy of the exchange's database.
 
    The ``taler-auditor`` script invokes the various helpers, each generating
    a JSON report. It then invokes the ``taler-helper-auditor-render.py``
    script to combine those JSON files with a Jinja2 template into a
    LaTeX report.  Finally, ``pdflatex`` is used to generate a PDF report.
 
    The resulting report includes performance data, reports on hard violations
    (resulting in financial losses) and reports on soft violations (such as the
    exchange not performing certain operations in a timely fashion).  The
    report also includes figures on the losses of violations. Careful reading
    of the report is required, as not every detail in the report is necessarily
    indicative of a problem.
 
 
 Installation
 ============
 
 Installing from source
 ----------------------
 
 Please install the following packages before proceeding with the
 exchange compilation.
 
 - Python3 module ``jinja2``
 
 .. include:: frags/list-of-dependencies.rst
 
 -  GNU Taler exchange (from `download directory <http://ftpmirror.gnu.org/taler/>`__,
    see `release announcement <https://mail.gnu.org/archive/cgi-bin/namazu.cgi?query=taler&idxname=info-gnu&max=20&result=normal&sort=date:late>`__)
 
 Except for the last two, these are available in most GNU/Linux
 distributions and should just be installed using the respective package
 manager.
 
 The following instructions will show how to install libgnunetutil and
 the exchange (which includes the code for the auditor).
 
 .. include:: frags/installing-gnunet.rst
 
 .. include:: frags/installing-taler-exchange.rst
 
 .. include:: frags/install-before-check.rst
 
 
 Installing the GNU Taler binary packages on Debian
 --------------------------------------------------
 
 .. include:: frags/installing-debian.rst
 
 To install the Taler auditor, you can now simply run:
 
 .. code-block:: console
 
    # apt install -t sid taler-auditor
 
 For the auditor, you must manually configure access to the exchange database,
 the HTTP reverse proxy (typically with TLS certificates) and offline signing.
 
 Sample configuration files for the HTTP reverse proxy can be found in
 ``/etc/taler-auditor/``.
 
 
 Installing the GNU Taler binary packages on Ubuntu
 --------------------------------------------------
 
 .. include:: frags/installing-ubuntu.rst
 
 To install the Taler exchange, you can now simply run:
 
 .. code-block:: console
 
    # apt install -t focal-fossa taler-auditor
 
 For the auditor, you must manually configure access to the exchange database,
 the HTTP reverse proxy (typically with TLS certificates) and offline signing.
 
 Sample configuration files for the HTTP reverse proxy can be found in
 ``/etc/taler-auditor/``.
 
 System setup
 ============
 
 UNIX accounts
 -------------
 
 For maximum security, you should setup multiple different users (possibly
 on different machines) to run Taler auditor components. While it is possible
 to skip some of these entirely, or to run all of them as the same user, this
 is not recommended for security.  The recommended set of users includes:
 
   * auditor --- runs the main auditing process and HTTP backend
   * sync --- synchronizes the ingres database with the production database
   * helper --- runs taler-auditor-offline download and upload commands
   * auditor-ingres --- imports database from exchange production system
   * auditor-wire --- imports wire transfer data from bank production system
   * offline --- manages the offline key, on a separate *offline* machine
 
 It is suggested that you setup the first five users on the target system(s)
 using:
 
 .. code-block:: console
 
    # add-user --disabled-password $USERNAME
 
 Additionally, there are two canonical system users of relevance (which your
 distribution would typically create for you):
 
   * www-data --- runs the HTTPS frontend (usually nginx or Apache)
   * postgres --- runs the PostgreSQL database
 
 
 Databases and users
 -------------------
 
 We recommend using the following databases for the auditor:
 
   * exchange-ingres --- synchronized exchange database over the network
   * exchange-production --- local copy of exchange database with trusted schema
   * auditor --- auditor production database with current state of the audit
   * libeufin --- local state of the auditor-wire tool for the bank transfer data import
 
 As the *postgres* user, you can create these databases using:
 
 .. code-block:: console
 
    # As the 'postgres' user:
    $ createdb -O auditor-ingres exchange-ingres
    $ createdb -O sync exchange-production
    $ createdb -O auditor auditor
    $ createdb -O auditor-wire libeufin
 
 This will ensure that the correct users have write-access to their
 respective database.  Next, you need to grant read-only access to
 some users to databases owned by other users:
 
 .. code-block:: console
 
    # As the 'auditor-ingres' user:
    $ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO sync;' | psql exchange-ingres
    # As the 'sync' user:
    $ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO auditor;' | psql exchange-production
    # As the 'auditor-wire' user:
    $ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO auditor;' | psql libeufin
 
 
 Configuration
 =============
 
 The auditor's configuration works the same way as the configuration of other
 Taler components.
 This section discusses configuration options related to the auditor.
 
 
 .. include:: frags/configuration-format.rst
 
 
 .. _SetupBaseUrl:
 
 Initial configuration
 ---------------------
 
 You need to tell the Taler auditor configuration where the
 REST API of the auditor will be available to the public:
 
 .. code-block:: ini
 
    # Both for the 'offline' *and* the 'auditor' user:
    [auditor]
    BASE_URL = https://auditor.example.com/
 
 The ``helper`` user that is used to download information from the exchange
 needs to know details about the exchange. Similarly, the ``offline`` user
 needs to check signatures signed with the exchange's offline key. Hence, you
 need to obtain the ``MASTER_PUBLIC_KEY`` from the exchange operator (they need
 to run ``taler-exchange-offline setup``) and the REST endpoint of the exchange
 and configure these:
 
 .. code-block:: ini
 
    # As the 'helper' and 'offline' users:
    [exchange]
    BASE_URL = https://exchange.example.com/
    MASTER_PUBLIC_KEY = $SOMELONGBASE32VALUEHERE
 
 
 .. _AuditorKeys:
 
 Keys
 ----
 
 The auditor works with one signing key to certify that it is auditing
 a particular exchange's denomination keys.  This key can and should
 be kept *offline* (and backed up adequately). As with the exchange's
 offline key, it is only used for a few cryptographic signatures and
 thus the respective code can be run on modest hardware, like a
 Raspberry Pi.
 
 The following values are to be configured in the section ``[auditor]``:
 
 -  ``AUDITOR_PRIV_FILE``: Path to the auditor’s private key file.
 
 Note that the default value here should be fine and there is no clear
 need to change it.  What you do need to do as the ``offine`` user
 is to extract the public key:
 
 .. code-block:: console
 
    # As the 'offline' user:
    $ taler-auditor-offline setup
 
 This public key must then be provided in the configuration file
 of the ``auditor`` user in the ``[auditor]]`` configuration section:
 
 -  ``PUBLIC_KEY``: Public key of the auditor, in Base32 encoding.
    Set from value printed by ``taler-auditor-offline setup``.
 
 You can set this configuration value using:
 
 .. code-block:: ini
 
    # As the 'auditor' and 'helper' users:
    [auditor]
    PUBLIC_KEY = $SOMELONGBASE32VALUEHERE
 
 
 .. _AuditorServing:
 
 Configuring the auditor's REST endpoint
 ---------------------------------------
 
 The auditor can serve HTTP over both TCP and UNIX domain socket.
 
 The following values are to be configured in the section ``[auditor]``:
 
 -  ``serve``: must be set to ``tcp`` to serve HTTP over TCP, or ``unix`` to serve
    HTTP over a UNIX domain socket
 
 -  ``port``: Set to the TCP port to listen on if ``serve`` is ``tcp``.
 
 -  ``unixpath``: set to the UNIX domain socket path to listen on if ``serve`` is
    ``unix``
 
 -  ``unixpath_mode``: number giving the mode with the access permission MASK
    for ``unixpath`` (i.e. 660 = ``rw-rw----``).
 
 
 .. _AuditorBank-account:
 
 Bank account
 ------------
 
 Bank accounts for the auditor (user ``auditor-wire``) are configured in
 exactly the same way as bank accounts for the exchange. See the exchange (and
 LibEuFin) documentation for details.
 
 .. _AuditorDatabaseConfiguration:
 
 Database
 --------
 
 The option ``DB`` under section ``[auditor]`` gets the DB backend’s name the
 exchange is going to use. So far, only ``DB = postgres`` is supported. After
 choosing the backend, it is mandatory to supply the connection string
 (namely, the database name). This is possible in two ways:
 
 -  via an environment variable: ``TALER_AUDITORDB_POSTGRES_CONFIG``.
 
 -  via configuration option ``CONFIG``, under section ``[auditordb-$BACKEND]``.
    For example, the demo exchange is configured as follows:
 
    .. code-block:: ini
 
       [auditor]
       ...
       DB = postgres
       ...
 
       [auditordb-postgres]
       CONFIG = postgres:///auditordemo
 
 If an exchange runs its own auditor, it may use the same database for
 the auditor and the exchange itself.
 
 The ``taler-auditor-dbinit`` tool is used to initialize the auditor's
 tables. After running this tool, the rights to CREATE or DROP tables
 are no longer required and should be removed.
 
 
 Both the ``taler-auditor-httpd`` and the ``taler-auditor`` (and its helpers)
 also need (read-only) access to a (recent, current, synchronized) copy of the
 exchange's database.  The configuration options are the same that are also
 used when configuring the exchange' database:
 
   .. code-block:: ini
 
      [exchange]
      ...
      DB = postgres
      ...
 
      [exchangedb-postgres]
      CONFIG = postgres:///exchangedemo
 
 
 Legal conditions for using the service
 --------------------------------------
 
 .. include:: frags/legal.rst
 
 
 .. _AuditorDeployment:
 
 Deployment
 ==========
 
 .. _Wallets:
 
 Before GNU Taler wallets will happily interact with an exchange, the
 respective auditor's public key (as obtained via ``taler-auditor-offline
 setup`` from the ``offline`` user) must be added under the respective currency
 to the wallet.  This is usually expected to be hard-coded into the Taler
 wallet.
 
 Users can also manually add auditors for a particular currency via a
 Web page offering the respective pairing.
 
 FIXME-DOLD: explain how that Web page works, once it works...
 
 
 .. _AuditorExchange:
 
 Exchange
 --------
 
 The exchange operator must use the ``taler-exchange-offline`` tool to add the
 auditor's public key, base URL and (business) name to the list of approved
 auditors of the exchange. For details, see :ref:`Auditor-configuration` in the
 exchange operator manual.
 
 
 .. _SigningDenominations:
 
 Signing Denominations
 ---------------------
 
 .. index:: maintenance
 
 These steps must be performed *regularly* whenever the exchange is
 deploying new denomination keys. After the exchange operator
 has signed new keys using the ``taler-exchange-offline`` tool,
 each auditor should run:
 
 .. code-block:: console
 
    # As the 'helper' user:
    $ taler-auditor-offline download > input.json
 
 to import the latest set of denomination keys. The key data should then be
 copied to the offline system and there be inspected using:
 
 .. code-block:: console
 
    # As the 'offline' user:
    $ taler-auditor-offline show < input.json
 
 and compared with the data the exchange operator saw when doing the offline
 signature. This process should involve directly the humans operating both
 systems and may require them to establish a trustworthy connection. The
 details how the auditor communicates with the exchange operator are a business
 process that is outside of the scope of this document.
 
 Note that the ``input.json`` does not contain any confidential data.  However,
 signing the wrong keys would be fatal in that it may allow an illegitimate
 exchange to convince users that it is a trustworthy operator and subsequently
 betray the user's trust that is anchored in the existence of a trustworthy
 auditor.
 
 Given the verified JSON input, the auditor can then sign it (typically
 on its offline system) using:
 
 .. code-block:: console
 
    # As the 'offline' user:
    $ taler-auditor-offline sign < input.json > output.json
 
 The resulting ``output.json`` should then be copied to an online system,
 and from there uploaded to the exchange using:
 
 .. code-block:: console
 
    # As the 'helper' user:
    $ taler-auditor-offline upload < output.json
 
 The contents of ``output.json`` can again be public and require no special
 handling.
 
 If the auditor has been correctly added, the exchange’s ``/keys``
 response will contain an entry in the ``auditors`` array mentioning the
 auditor’s URL.
 
 Commands, like ``taler-auditor-offline``, that support the ``-l LOGFILE``
 command-line option, send logging output to standard error by default.
 
 
 .. _AuditorDatabaseInitialization:
 
 Database
 --------
 
 The next key step for the auditor is to configure replication of the
 *exchange*'s database in-house. This should be performed in two steps
 as illustrated in the following figure:
 
 .. image:: images/replication.png
 
 First, the exchange should use standard PostgreSQL replication features to
 enable the auditor to obtain a full copy of the exchange's database.
 Second, the auditor should make a "trusted" local copy, ensuring that it
 never replicates malicious changes using ``taler-auditor-sync``. Both
 of these steps are described in more detail below.
 
 We note that as a result of these steps, the auditor will have three
 databases: its own production primary database (as configured in
 ``auditordb-postgres``), its on production copy of the exchange's database
 (``exchangedb-postgress``), and a third, untrusted "ingres" copy of the
 exchange database.  The untrusted database should run as a separate PostgreSQL
 instance and is only accessed via ``taler-auditor-sync`` and the replication
 mechanism driven by the exchange operator.
 
 
 Ingres replication of the exchange production database
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Ingres operation should be done using the ``auditor-ingres`` user --- or
 depending on the setup parts of the operation may be done by the ``postgres``
 user directly.
 
 The full copy can be obtained in various ways with PostgreSQL.  It is
 possible to use log shipping with streaming replication as described
 in https://www.postgresql.org/docs/13/warm-standby.html, or to use
 logical replication, as described in
 https://www.postgresql.org/docs/13/logical-replication.html. We note
 that asynchronous replication should suffice.
 
 The resulting auditor database should be treated as read-only on the auditor
 side.  The ``taler-exchange-dbinit`` tool can be used to setup the schema, or
 the schema can be replicated using PostgreSQL's standard mechanisms. The same
 applies for schema upgrades: if logical replication is used (which does not
 replicate schema changes), ``taler-exchange-dbinit`` can be used to migrate
 the schema(s) in both the ingres and production copies of the exchange's
 database as well.
 
 On the exchange side, a database user must be created that has the right
 to perform database replication. This is done using:
 
 .. code-block:: console
 
    # As the 'postgres' user of the exchange:
    $ createuser --replication egress
    $ echo "ALTER ROLE egress WITH PASSWORD '$PASSWORD'; | psql
    $ echo "CREATE PUBLICATION $NAME FOR ALL TABLES;" | psql taler-exchange
 
 The exchange must share the password of the publication with the auditor. A
 good ``$NAME`` relates to the auditor's business unit name.  A secure tunnel
 must be setup between the exchange and the auditor, for example using SSH or
 Wireguard.
 
 It is also necessary to edit ``main.cf`` of the exchange and on the auditor
 side to enable logical replication.  If an exchange has multiple auditors, it
 should setup multiple ``egress`` accounts.  The exchange must ensure that
 the following lines are in the ``main.cf`` PostgreSQL configuration (the port
 may differ) to enable replication over the network:
 
 .. code-block::
 
    listen_addresses='*'
    port = 5432
    wal_level= logical
 
 Equally, the auditor must configure logical replication in the ``main.cf``
 PostgreSQL configuration:
 
 .. code-block::
 
    wal_level= logical
 
 Next, the ``postgres`` user of the auditor's system must first initialize the
 local tables:
 
 .. code-block:: ini
 
    # Configure database:
    [exchange]
    DB = "postgres"
    [exchangedb-postgres]
    CONFIG = "postgres:///taler-ingress"
 
 .. code-block:: console
 
    # As the 'ingress' user of the exchange:
    $ taler-exchange-dbinit
 
 To complete the replication, the ``postgres`` user of the auditor's
 system must subscribe:
 
 .. code-block:: console
 
    # As the 'postgres' user of the exchange:
    $ createuser --replication egress
    $ echo "ALTER ROLE egress WITH PASSWORD '$PASSWORD'; | psql
    $ echo "CREATE PUBLICATION $NAME FOR ALL TABLES;" | psql taler-exchange
 
 
 For details, we refer to the PostgreSQL manual.
 
 .. note::
 
    Depending on the replication method used, the exchange may perform
    unexpected changes to the schema or perform ``UPDATE``, ``DELETE`` or
    ``DROP`` operations on the tables.  Hence, the auditor cannot rely upon the
    exchange's primary copy to respect schema constraints, especially as we
    have to presume that the exchange could act maliciously.  Furthermore, it
    is unclear to what degree PostgreSQL database replication mechanisms are
    robust against a malicious master database.  Thus, the auditor should
    isolate its primary copy of the exchange database, including the PostgreSQL
    process, from its actual operational data.
 
 
 Safe replication of the ingres database into the auditor production database
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Using ``taler-auditor-sync`` as the ``sync`` user, the auditor should
 make a second "safe" copy of the exchange's ingres database.
 ``taler-auditor-sync`` basically reads from one exchange database and inserts
 all records found into a second exchange database. If the source database
 violates invariants, the tool halts with an error. This way, records violating
 invariants are never even copied, and in particular schema changes and
 deletions or updates are not propagated into the auditor's production
 database.
 
 While ``taler-auditor-sync`` could in theory be run directly against the
 exchange's production system, this is likely a bad idea due to the high
 latency from the network between auditor and exchange operator. Thus, we
 recommend first making an "untrusted" ingress copy of the exchange's
 production database using standard PostgreSQL tooling, and then using
 ``taler-auditor-sync`` to create a second "safe" copy.  The "safe" copy used
 by the production system should also run under a different UID.
 
 Before ``taler-auditor-sync`` can be used, the target database must be
 initialized with the exchange schema using ``taler-exchange-dbinit``.
 Note that running ``taler-auditor-sync`` requires the use of two
 configuration files, one specifying the options for accessing the source
 database, and a second with the options for accessing the destination
 database.  In both cases, likely only the ``[exchangedb]/CONFIG`` option
 needs to be changed.
 
 To run ``taler-auditor-sync``, you must first configure two configuration
 files that identify the source and destination databases:
 
 .. code-block:: ini
 
    # src.conf
    [exchangedb]
    CONFIG = "postgres:///auditor-ingres/"
 
 .. code-block:: ini
 
    # dst.conf
    [exchangedb]
    CONFIG = "postgres:///auditor/"
 
 Now you should be able to launch the synchronization process. You can run
 the process via systemd in the background. For a first one-off test, you should
 use the ``-t`` option which will cause the process to terminate once the two
 databases are synchronized:
 
 .. code-block:: console
 
    # As the 'sync' user:
    $ taler-auditor-sync -s src.conf -d dst.cfg -t
 
 When the exchange performs garbage collection to ``DELETE`` obsolete records,
 this change should be automatically replicated to the auditors untrusted
 ingress database.  However, as ``taler-auditor-sync`` tries to be "safe",
 it will not replicate those deletions to the auditor's production database.
 Thus, it is necessary to (occasonally) run ``taler-exchange-dbinit -g`` on
 the auditor's production database to garbage collect old data in the
 auditor's production copy.  We note that this does not have to be done
 at the same time when the exchange runs its garbage collection.
 
 
 
 
 .. _Operation:
 
 Operation
 =========
 
 .. _Web service:
 
 Web service
 -----------
 
 The ``taler-auditor-httpd`` runs the required REST API for the auditor.  The
 service must have ``INSERT`` (and ``SELECT``) rights on the
 ``deposit_confirmations`` table in the auditor's database.  We expect that in
 future versions additional rights may be required.
 
 For now, we recommend simply running the ``taler-auditor-httpd`` under the
 ``auditor`` user. However, it is also possible (and might be more secure) to
 create a separate user with more restrictive permissions for this purpose.
 
 As the ``taler-auditor-httpd`` does not include HTTPS-support, it is
 advisable to run it behind a reverse proxy that offers TLS termination.
 
 
 .. _Audit:
 
 Audit
 -----
 
 Performing an audit is done by invoking the ``taler-auditor`` shell script as
 the ``auditor`` user.
 
 The shell script invokes the various helper processes. For additional
 performance and security, one may want to run the various helpers individually
 and with the respective minimal set of access rights (only
 ``taler-helper-auditor-wire`` needs the credentials to query the bank for wire
 transfers, alas if ``auditor-wire`` is used to talk to the bank, this issue is
 already addressed).  The shell script combines the final JSON outputs of the
 various helpers using the ``taler-helper-auditor-render.py`` script into the
 TeX report.  Regardless, the simplest way to obtain a report is to run:
 
 .. code-block:: console
 
    $ taler-audit
 
 This generates a file ``auditor-report.pdf`` (in a temporary directory created
 for this purpose) with all of the issues found and the financial assessment of
 the exchange.  The exact filename will be output to the console upon
 completion.
 
 We note that ``taler-audit`` by default runs in incremental mode. As a result,
 running the commands again will only check the database entries that have been
 added since the last run.
 
 You can use ``taler-auditor-dbinit -r`` to force a full check since the
 beginning of time. However, as this may require excessive time and
 interactions with the bank (which may not even have the wire transfer records
 anymore), this is not recommended in a production setup.
 
 
 Reading the report
 ------------------
 
 The auditor's report needs to be read carefully, as it includes
 several categories of failures of different severity:
 
 - Delayed operations, where an operation was expected to have
   happened, but did not happen yet, possibly because of a
   disagreement in system time or overloading of the system.
   These failures only require action if the delays are
   significant.
 
 - Inconsistencies in the data that have no clear financial
   impact.
 
 - Inconsistencies in the data that show that the exchange
   experienced an unexpected financial loss (such as accepting a coin for
   deposit with an invalid signature).
 
 - Inconsistencies in the data that show that the exchange
   caused some other party to experience a financial loss (such as not wiring
   the correct amount to a merchant).
 
 - Configuration issues (such was wire fees unavailable).
 
 
 .. _AuditorDatabaseUpgrades:
 
 Database upgrades
 -----------------
 
 To upgrade the database between Taler versions can be done by running:
 
 .. code-block:: console
 
    $ taler-auditor-dbinit
    $ taler-exchange-dbinit
 
 In any case, it is recommended that exchange and auditor coordinate closely
 during schema-changing database upgrades as without coordination the database
 replication or ``taler-auditor-sync`` will likely experience problematic
 failures.  In general, we recommend:
 
   * halting the exchange business logic,
   * allowing the replication and ``taler-auditor-sync`` to complete
     (see also the **-t** option of ``taler-auditor-sync``)
   * completing a ``taler-audit`` run against the old schema
   * migrating the exchange schema (``taler-exchange-dbinit``) of
     the master database, possibly the ingres database and the
     auditor's production copy
   * migrating the auditor database (``taler-auditor-dbinit``)
   * resuming database replication between the exchange's master
     database and the auditor's ingres copy
   * resuming ``taler-auditor-sync``
   * resuming the regular exchange and auditor business logic
 
 Regardless, the above is merely the general rule. Please review the specific
 release notes to ensure this procedure is correct for the specific upgrade.
 
 
 Database reset
 ---------------
 
 The auditor database can be reset using:
 
 .. code-block:: console
 
    $ taler-auditor-dbinit -R
 
 However, running this command will result in all data in the database being
 *lost*. Thus, doing so may result in significant commputation (and bandwidth
 consumption with the bank) when the auditor is next launched, as it will
 re-download and re-verify all historic transactions. Hence this should not be
 done in a production system.
 
 
 .. _AuditorRevocations:
 
 Revocations
 -----------
 
 When an auditor detects that the private key of a denomination key pair has
 been compromised, one important step is to revoke the denomination key.  The
 exchange operator includes the details on how to revoke a denomination key, so
 the auditor should only have to report (and possibly enforce) this step.
 For more information, see :ref:`Revocations` in the exchange operator manual.
 
 If all denominations of an exchange are revoked, the exchange includes logic
 to wire back all returned funds to the bank accounts from which they
 originate.  If some denominations remain operational, wallets will generally
 exchange old coins of revoked denominations for new coins -- while providing
 additional information to demonstrate that these coins were not forged from
 the compromised private key but obtained via a legitimate withdraw operation.
 
 
 Failures
 --------
 
 Most audit failures are handled by the auditor's regular reporting functionality,
 creating a (hopefully descriptive) PDF report detailing the problems found.
 
 However, there is one category of errors where this is not possible, which
 concerns arithmetic overflows for amounts. Taler's specification limits amount
 values to at most 2^52. If, during the auditor's calculations, amounts are
 encountered that exceed this threshold, the auditor will not generate a regular
 report, but instead write a log statement explaining where the problem happened
 and exit with a status code of *42*.
 
 The most common expected case when this happens is a corrupted database. This
 could be because the exchange is actively malicious, or more likely due to
 some data corruption.  The audit cannot continue until the corruption has been
 addressed. If it is not possible to restore a fully *correct* version of the
 database, the suggestion is to replace the corrupted (and likely very large)
 amounts with zero (Note: this does not apply to the value of denominations or
 fees, here it is crucial that the correct amounts are restored). While an
 amount of zero would be incorrect, the auditing logic should be able to do its
 calculations with zero instead.
 
 After patching the database, the audit can
 be restarted. A full reset is not required, as the audit transaction is aborted
 when the auditor exits with code *42*.  After restarting, the resulting audit
 report is likely to indicates errors relating to the corrupted fields (such as
 invalid signatures, arithmetic errors by the exchange, etc.), but at least the
 loss/gain calculations will be meaningful and actually indicative of the scope
 of the error created by the corrupted data.
 
 
 
 Auditor implementation guide
 ============================
 
 The auditor implementation is split into five main processes, called
 ``taler-helper-auditor-XXX``.  The split was done to realize the principle of
 least privilege and to enable independent logic to be possibly run in
 parallel.  Only the taler-wire-auditor must have (read-only) access to the
 exchange's bank account, the other components only need access to the
 database.
 
 All auditor subsystems basically start their audit from a certain transaction
 index (``BIG SERIAL``) in the auditor database which identifies where the last
 audit concluded. They then check that the transactions claimed in the
 exchange's database match up internally, including the cryptographic
 signatures and also with respect to amounts adding up. The auditor also
 calculates the exchange's profits and expected bank balances.  Once all
 existing transactions are processed, the auditor processes store the current
 checkpoint in its database and generate a JSON report.
 
 The ``taler-auditor`` shell script calls the five helpers and then
 uses Jinja2 with a TeX template to convert the five individual
 JSON reports into LaTeX and then into PDF.
 
 
 The auditor's database
 ----------------------
 
 The database scheme used by the exchange looks as follows:
 
 .. image:: images/auditor-db.png
 
 
 Invariants checked by the auditor
 ---------------------------------
 
 The auditor verifies a large number of invariants that must hold for a Taler
 exchange.  One objective in the design of the auditor was to check each
 invariant only once, both to minimize cost and to avoid duplicate reporting of
 problems where possible. As a result, not every invariant is checked in every
 pass where it might seem applicable.
 
 
 
 Invariants checked by the taler-helper-auditor-aggregation
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This is from CodeBlau's analysis. A proper write-up is pending.
 CodeBlau reports the following checks:
 
 - arithmetic inconsistencies
 
   - disagreement in fee for deposit between auditor and exchange db
 
   - disagreement in fee for melt between auditor and exchange db
 
   - disagreement in fee for refund between auditor and exchange db
 
   - aggregation of fee is negative
 
   - aggregation (contribution): Expected coin contributions differ:
     coin value without fee, total deposit without refunds
 
   - wire out fee is negative
 
 - coin arithmetic inconsistencies
 
   - refund (merchant) is negative
 
   - refund (balance) is negative
 
   - spend > value
 
 - coin denomination signature invalid
 
 - start date before previous end date
 
 - end date after next start date
 
 - wire out inconsistencies in amount
 
 - row inconsistencies
 
   - wire account given is malformed
 
   - h(wire) does not match wire
 
   - failed to compute hash of given wire data
 
   - database contains wrong hash code for wire details
 
   - no transaction history for coin claimed in aggregation
 
   - could not get coin details for coin claimed in aggregation
 
   - could not find denomination key for coin claimed in aggregation
 
   - coin denomination signature invalid
 
   - target of outgoing wire transfer do not match hash of wire from deposit
 
   - date given in aggregate does not match wire transfer date
 
   - wire fee signature invalid at given time
 
   - specified wire address lacks method
 
   - wire fee unavailable for given time
 
 
 Invariants checked by the taler-helper-auditor-coins
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This is from CodeBlau's analysis. A proper write-up is pending.
 CodeBlau reports the following checks:
 
 - check that all denominations used by the exchange have been signed using
   this auditor's key. All denominations encountered in the database that
   this auditor did not officially sign for are reported (but still included
   in the audit as they obviously may impact the exchange's bank balance).
   Depending on the business situation, this may be normal (say if an exchange
   is changing auditors and newer denominations are no longer supported until
   their end-of-life by the current auditor).
 
 - emergency on denomination over loss
 
   - value of coins deposited exceed value of coins issued
 
 - emergency on number of coins, num mismatch
 
 - arithmetic inconsistencies
 
   - melt contribution vs. fee
 
   - melt (cost)
 
   - refund fee
 
 - row inconsistencies
 
   - revocation signature invalid
 
   - denomination key not found
 
   - denomination key for fresh coin unknown to auditor
 
   - denomination key for dirty coin unknown to auditor
 
   - denomination key for deposited coin unknown to auditor
 
 - coin validity in known_coin, by checking denomination signatures
 
 - coin validity in melt, by checking signatures
 
 - refresh hanging, zero reveals (harmless)
 
 - verify deposit signature
 
 - verify refund signature
 
 - recoup, check coin
 
 - recoup, check signature
 
 - recoup, denomination not revoked
 
 
 
 Invariants checked by the taler-helper-auditor-deposits
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This tool verifies that the deposit confirmations reported by merchants
 directly to the auditor are also included in the database duplicated from the
 exchange at the auditor.  This is to ensure that the exchange cannot defraud
 merchants by simply not reporting deposits to the auditor.
 
 
 Invariants checked by the taler-helper-auditor-reserves
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This is from CodeBlau's analysis. A proper write-up is pending.
 CodeBlau reports the following checks:
 
 - report arithmetic inconsistency
 
   - closing aggregation fee
 
   - global escrow balance
 
 - denomination key validity withdraw inconsistencies
 
 - bad signature losses in withdraw
 
 - bad signature losses in recoup
 
 - bad signature losses in recoup-master
 
 - reserve balance, insufficient, losses and gains
 
 - reserve balance, summary wrong
 
 - reserve not closed after expiration time
 
 - could not determine closing fee / closing-fee unavailable
 
 - denomination key not found for withdraw
 
 - denomination key not in revocation set for recoup
 
 - target account not verified, auditor does not know reserve
 
 - target account does not match origin account
 
 
 Invariants checked by the taler-helper-auditor-wire
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 This auditor is special in that it is the only pass that is required to have
 *read-only* access to the exchange's bank account (privilege separation).  Its
 main role is to verify that the wire transfers in the exchange's database and
 those reported by the bank are identical.
 
 This is from CodeBlau's analysis. A proper write-up is pending.
 CodeBlau reports the following checks:
 
 - check pending
 
 - wire missing
 
 - execution date mismatch
 
 - wire out consistency
 
 - wire transfer not made (yet?)
 
 - receiver account mismatch
 
 - wire amount does not match
 
 - justification for wire transfer not found
 
 - duplicate subject hash
 
 - duplicate wire offset
 
 - incoming wire transfer claimed by exchange not found
 
 - wire subject does not match
 
 - wire amount does not match
 
 - debit account url does not match
 
 - execution date mismatch
 
 - closing fee above total amount
 
 
 
 
 
 Testing the auditor
 -------------------
 
 The main objective of the auditor is to detect inconsistencies. Thus, the
 ``test-auditor.sh`` script deliberately introduces various inconsistencies into
 a synthetic exchange database.  For this, an "normal" exchange database is
 first generated using the ``taler-wallet-cli``.  Then, various fields or rows
 of that database are manipulated, and the auditor is let loose on the modified
 database.  Afterwards, the test verifies that the JSON contains values
 indicating that the auditor found the inconsistencies.  The script also
 verifies that template expansion and LaTeX run work for the JSON output,
 but it does not verify the correctness of the final PDF.
 
 The ``test-auditor.sh`` script is written to maximize code coverage: it should
 cover as many code paths as possible in both the exchange and the auditor.  It
 should also ideally create all interesting possible variations of the exchange
 database fields (within the constraints of the database schema).
 
 In general, ``test-auditor.sh`` runs the tests against an "old" database where
 some transactions are past the due-date (and hence the aggregator would trigger
 wire transfers), as well as a freshly generated exchange database where the
 auditor would not perform any transfers.  Auditor interactions can be made
 before or after the aggregator, depending on what is being tested.
 
 The current script also rudimentarily tests the auditor's resume logic,
 by re-starting the auditor once against a database that the auditor has
 already seen.
 
 
 The ``test-revocation.sh`` script performs tests related to the handling of
 key revocations.
 
 The ``test-sync.sh`` script performs tests related to the ``taler-auditor-sync``
 tool.
 
 .. TODO
 
    More extensive auditor testing where additional transactions
    have been made against the database when the audit is being resumed
    should be done in the future.