\input texinfo @c -*-texinfo-*-
@c %**start of header
@setfilename taler-auditor.info
@documentencoding UTF-8
@ifinfo
@*Generated by Sphinx 5.3.0.@*
@end ifinfo
@settitle Taler Auditor Manual
@defindex ge
@paragraphindent 0
@exampleindent 4
@finalout
@dircategory CATEGORY
@direntry
* MENU ENTRY: (taler-auditor.info). DESCRIPTION
@end direntry
@c %**end of header
@copying
@quotation
GNU Taler 0.9.0, Jan 06, 2024
GNU Taler team
Copyright @copyright{} 2014-2022 Taler Systems SA (GPLv3+ or GFDL 1.3+)
@end quotation
@end copying
@titlepage
@title Taler Auditor Manual
@insertcopying
@end titlepage
@contents
@c %** start of user preamble
@c %** end of user preamble
@ifnottex
@node Top
@top Taler Auditor Manual
@insertcopying
@end ifnottex
@c %**start of body
@anchor{taler-auditor-manual doc}@anchor{0}
@c This file is part of GNU TALER.
@c
@c Copyright (C) 2019-2021 Taler Systems SA
@c
@c TALER is free software; you can redistribute it and/or modify it under the
@c terms of the GNU Affero General Public License as published by the Free Software
@c Foundation; either version 2.1, or (at your option) any later version.
@c
@c TALER is distributed in the hope that it will be useful, but WITHOUT ANY
@c WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
@c A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
@c
@c You should have received a copy of the GNU Affero General Public License along with
@c TALER; see the file COPYING. If not, see
@c
@c @author Christian Grothoff
@menu
* Introduction::
* Installation::
* System setup::
* Configuration::
* Deployment::
* Operation::
* Auditor implementation guide::
* Index::
@detailmenu
--- The Detailed Node Listing ---
Introduction
* About GNU Taler::
* About this manual::
* Organizational prerequisites::
* Architecture overview::
Installation
* Installing from source::
* Installing the GNU Taler binary packages on Debian::
* Installing the GNU Taler binary packages on Ubuntu::
System setup
* UNIX accounts::
* Databases and users::
Configuration
* Configuration format::
* Initial configuration::
* Keys::
* Configuring the auditor’s REST endpoint::
* Bank account::
* Database::
* Legal conditions for using the service::
* Terms of Service::
* Privacy Policy::
* Legal policies directory layout::
* Generating the Legal Terms::
* Adding translations::
* Updating legal documents::
Legal policies directory layout
* Example::
Deployment
* Exchange::
* Signing Denominations::
* Database: Database<2>.
Database
* Ingres replication of the exchange production database::
* Safe replication of the ingres database into the auditor production database::
Operation
* Web service::
* Audit::
* Reading the report::
* Database upgrades::
* Database reset::
* Revocations::
* Failures::
Auditor implementation guide
* The auditor’s database::
* Invariants checked by the auditor::
* Testing the auditor::
Invariants checked by the auditor
* Invariants checked by the taler-helper-auditor-aggregation::
* Invariants checked by the taler-helper-auditor-coins::
* Invariants checked by the taler-helper-auditor-deposits::
* Invariants checked by the taler-helper-auditor-reserves::
* Invariants checked by the taler-helper-auditor-wire::
@end detailmenu
@end menu
@node Introduction,Installation,Top,Top
@anchor{taler-auditor-manual gnu-taler-auditor-operator-manual}@anchor{1}@anchor{taler-auditor-manual introduction}@anchor{2}
@chapter Introduction
This manual is an early draft that still needs significant editing work
to become readable.
@menu
* About GNU Taler::
* About this manual::
* Organizational prerequisites::
* Architecture overview::
@end menu
@node About GNU Taler,About this manual,,Introduction
@anchor{taler-auditor-manual about-gnu-taler}@anchor{3}
@section About GNU Taler
@c This file is part of GNU TALER.
@c
@c Copyright (C) 2014-2023 Taler Systems SA
@c
@c TALER is free software; you can redistribute it and/or modify it under the
@c terms of the GNU Affero General Public License as published by the Free Software
@c Foundation; either version 2.1, or (at your option) any later version.
@c
@c TALER is distributed in the hope that it will be useful, but WITHOUT ANY
@c WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
@c A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
@c
@c You should have received a copy of the GNU Affero General Public License along with
@c TALER; see the file COPYING. If not, see
GNU Taler is an open protocol for an electronic payment system with a
free software reference implementation. GNU Taler offers secure, fast
and easy payment processing using well understood cryptographic
techniques. GNU Taler allows customers to remain anonymous, while
ensuring that merchants can be held accountable by governments. Hence,
GNU Taler is compatible with anti-money-laundering (AML) and
know-your-customer (KYC) regulation, as well as data protection
regulation (such as GDPR).
@node About this manual,Organizational prerequisites,About GNU Taler,Introduction
@anchor{taler-auditor-manual about-this-manual}@anchor{4}
@section About this manual
This tutorial targets exchange operators, auditors and governments
who want to run the auditor to verify that a GNU Taler exchange is
operating correctly.
@node Organizational prerequisites,Architecture overview,About this manual,Introduction
@anchor{taler-auditor-manual organizational-prerequisites}@anchor{5}
@section Organizational prerequisites
Operating a GNU Taler auditor means that you (henceforth: auditor) have a
business relationship with (or regulatory authority over) a GNU Taler exchange
operator (henceforth: exchange). Your objective is to verify that the
exchange is operating correctly, and if not to alert the exchange, the
state or even the public about any misbehavior to limit financial losses
to other parties.
To perform this duty, you will need at least (read-only) access to the bank
transactions of the exchange, as well as a continuously synchronized replica
of the exchange’s database. The general assumption for running the auditor
is that this is done on a separate system controlled by the auditor. After
all, the goal is to detect nerfarious activity of the exchange operator,
which cannot be effectively done on a machine controlled by the exchange
operator.
For this, every auditor needs to operate a PostgreSQL database. The data
collected will include sensitive information about Taler users, including
withdrawals made by consumers and income received by merchants. As a result,
the auditor is expected to provide high confidentiality for the database. In
general, the auditor does not have to offer high-availability: the exchange
operator can continue operations without the auditor, and the auditor can
catch up with it later when the auditor’s systems are restored. However, of
course any downtime would provide a window of opportunity for fraud and should
thus be minimized. Finally, the auditor’s copy of the exchange’s database can
be useful as a backup to the exchange in case the exchange experiences a loss
of its own copies. Thus, business agreements between auditor and exchanges may
include availability requirements as well.
Then, with the software provided, auditors can verify the cryptographic proofs
collected by the exchange and detect if any improper bank transactions have been
made. There are additional tasks which an auditor should perform. While this
manual only focuses on the audit of the exchange’s database and wire transfers
with the existing tools, a proper auditor should also perform the following
tasks:
@itemize -
@item
security audit of the source code
@item
audit of the operational procedures of the exchange
@item
audit of the physical security of the deployment
@item
background check of the individuals operating the exchange
@item
verification that the exchange properly implements the @code{/link} protocol
(feature yet to be implemented in common Taler wallets)
@item
verification that the exchange properly reports coins issued during
the refresh protocol (by irregularly refreshing coins withdrawn by
the auditor and comparing against the exchange’s database — the
code required to support this is not yet implemented)
@end itemize
@node Architecture overview,,Organizational prerequisites,Introduction
@anchor{taler-auditor-manual architecture-overview}@anchor{6}
@section Architecture overview
Taler is a pure payment system, not a new crypto-currency. As such, it
operates in a traditional banking context. In particular, this means that in
order to receive funds via Taler, the merchant must have a regular bank
account, and payments can be executed in ordinary currencies such as USD or
EUR. Similarly, the exchange must interact with a bank. The bank of the
exchange holds the exchange’s funds in an escrow account. As a result,
exchanges operate in a regulated environment, and auditors provide a crucial
oversight function.
Auditors should generally be independent third parties that verify that the
exchange operates correctly. However, an exchange is likely to also run the
auditing logic, as it is also used to calculate the exchange’s profits, risk
and liabilities. Furthermore, it’s usually a good idea to not only rely on
third parties to verify one’s own work.
The Taler software stack for an auditor consists of the following
components:
@itemize -
@item
DBMS: PostgreSQL
The auditor requires a DBMS to store a local copy of the transaction history for
the Taler exchange, as well as for its own internal bookkeeping and checkpointing.
The DBMS is assumed to be able to assure the auditor of the database invariants (foreign
key, uniqueness, length restrictions). Should the exported data from the exchange
fail to be imported due to constraint violations, this is an immediate serious
concern that must be addressed manually. The software only verifies the content
of a well-formed exchange database (well-formed with respect to SQL).
For now, the GNU Taler reference implementation
only supports PostgreSQL, but the code could be easily extended to
support another DBMS.
@item
The auditor Web service
The auditor is expected to provide a public Web service. At this REST API,
merchants can (probabilistically) provide deposit confirmations, allowing
the auditor to detect if an exchange is underreporting deposits.
In the future, the Web service should be extended to allow customers and
merchants to automatically upload cryptographic proof of other violations
of the specification by the exchange. However, for now it is assumed that
the respective cryptographic proofs are reported and verified manually —
as with a well-behaved exchange this should obviously be a rare event.
The main binary of this component is the @code{taler-auditor-httpd}.
@item
The (main) auditor
The main auditor logic checks the various signatures, totals up the
amounts and checks for arithmetic inconsistencies. It also
computes the expected bank balance, revenue and risk exposure of the
exchange operator. The main script of this component is the @code{taler-auditor}.
This script invokes several helper binaries sequentially. Production
users may want to modify the script to run those binaries in parallel,
possibly using different privileges (as only the @code{taler-helper-auditor-wire}
needs access to the wire gateway).
The @code{taler-helper-auditor-wire} auditor verifies that the bank
transactions performed by the exchange
were done properly. This component must have access to the bank account
of the exchange, as well as to a copy of the exchange’s database.
The @code{taler-auditor} script invokes the various helpers, each generating
a JSON report. It then invokes the @code{taler-helper-auditor-render.py}
script to combine those JSON files with a Jinja2 template into a
LaTeX report. Finally, @code{pdflatex} is used to generate a PDF report.
The resulting report includes performance data, reports on hard violations
(resulting in financial losses) and reports on soft violations (such as the
exchange not performing certain operations in a timely fashion). The
report also includes figures on the losses of violations. Careful reading
of the report is required, as not every detail in the report is necessarily
indicative of a problem.
@end itemize
@node Installation,System setup,Introduction,Top
@anchor{taler-auditor-manual installation}@anchor{7}
@chapter Installation
@menu
* Installing from source::
* Installing the GNU Taler binary packages on Debian::
* Installing the GNU Taler binary packages on Ubuntu::
@end menu
@node Installing from source,Installing the GNU Taler binary packages on Debian,,Installation
@anchor{taler-auditor-manual installing-from-source}@anchor{8}
@section Installing from source
Please install the following packages before proceeding with the
exchange compilation.
@itemize -
@item
Python3 module @code{jinja2}
@end itemize
@itemize -
@item
“Sphinx RTD Theme” Python package aka @code{python3-sphinx-rtd-theme}
on Debian-based systems (for GNUnet documentation support, can be
omitted if GNUnet is configured with @code{--disable-documentation})
@item
libsqlite3 >= 3.16.2
@item
GNU libunistring >= 0.9.3
@item
libcurl >= 7.26 (or libgnurl >= 7.26)
@item
libqrencode >= 4.0.0 (Taler merchant only)
@item
GNU libgcrypt >= 1.6 (1.10 or later highly recommended)
@item
libsodium >= 1.0
@item
libargon2 >= 20171227
@item
libjansson >= 2.7
@item
PostgreSQL >= 15, including libpq
@item
GNU libmicrohttpd >= 0.9.71
@item
GNUnet >= 0.20 (from source tarball@footnote{http://ftpmirror.gnu.org/gnunet/})
@item
Python3 with @code{jinja2}
@end itemize
If you are on Debian stable or later, the following command may help you
install these dependencies:
@example
# apt-get install \
libqrencode-dev \
libsqlite3-dev \
libltdl-dev \
libunistring-dev \
libsodium-dev \
libargon2-dev \
libcurl4-gnutls-dev \
libgcrypt20-dev \
libjansson-dev \
libpq-dev \
libmicrohttpd-dev \
python3-jinja2 \
postgresql-15
@end example
@itemize -
@item
GNU Taler exchange (from download directory@footnote{http://ftpmirror.gnu.org/taler/},
see release announcement@footnote{https://mail.gnu.org/archive/cgi-bin/namazu.cgi?query=taler&idxname=info-gnu&max=20&result=normal&sort=date:late})
@end itemize
Except for the last two, these are available in most GNU/Linux
distributions and should just be installed using the respective package
manager.
The following instructions will show how to install libgnunetutil and
the exchange (which includes the code for the auditor).
Before you install GNUnet, you must download and install the dependencies
mentioned in the previous section, otherwise the build may succeed, but could
fail to export some of the tooling required by GNU Taler.
On Ubuntu, you also need to install pkg-config, for example:
@example
$ apt-get install pkg-config
@end example
To install GNUnet, unpack the tarball and change
into the resulting directory, then proceed as follows:
@example
$ ./configure [--prefix=GNUNETPFX]
$ # Each dependency can be fetched from non standard locations via
$ # the '--with-' option. See './configure --help'.
$ make
# make install
# ldconfig
@end example
If you did not specify a prefix, GNUnet will install to @code{/usr/local},
which requires you to run the last step as @code{root}.
The @code{ldconfig} command (also run as @code{root}) makes the
shared object libraries (@code{.so} files)
visible to the various installed programs.
After installing GNUnet, unpack the GNU Taler exchange tarball,
change into the resulting directory, and proceed as follows:
@example
$ ./configure [--prefix=EXCHANGEPFX] \
[--with-gnunet=GNUNETPFX]
$ # Each dependency can be fetched from non standard locations via
$ # the '--with-' option. See './configure --help'.
$ make
# make install
@end example
If you did not specify a prefix, the exchange will install to @code{/usr/local},
which requires you to run the last step as @code{root}. You have to specify
@code{--with-gnunet=/usr/local} if you installed GNUnet to @code{/usr/local} in the
previous step.
Please note that unlike most packages, if you want to run the @code{make check}
command, you should run it only `after' having done @code{make install}. The
latter ensures that necessary binaries are copied to the right place.
In any case, if @code{make check} fails, please consider filing a
bug report with the Taler bug tracker@footnote{https://bugs.taler.net}.
@node Installing the GNU Taler binary packages on Debian,Installing the GNU Taler binary packages on Ubuntu,Installing from source,Installation
@anchor{taler-auditor-manual installing-the-gnu-taler-binary-packages-on-debian}@anchor{9}
@section Installing the GNU Taler binary packages on Debian
To install the GNU Taler Debian packages, first ensure that you have
the right Debian distribution. At this time, the packages are built for
Debian bookworm.
You need to add a file to import the GNU Taler packages. Typically,
this is done by adding a file @code{/etc/apt/sources.list.d/taler.list} that
looks like this:
@example
deb [signed-by=/etc/apt/keyrings/taler-systems.gpg] https://deb.taler.net/apt/debian bookworm main
@end example
Next, you must import the Taler Systems SA public package signing key
into your keyring and update the package lists:
@example
# wget -O /etc/apt/keyrings/taler-systems.gpg \
https://taler.net/taler-systems.gpg
# apt update
@end example
@cartouche
@quotation Note
You may want to verify the correctness of the Taler Systems SA key out-of-band.
@end quotation
@end cartouche
Now your system is ready to install the official GNU Taler binary packages
using apt.
To install the Taler auditor, you can now simply run:
@example
# apt install -t sid taler-auditor
@end example
For the auditor, you must manually configure access to the exchange database,
the HTTP reverse proxy (typically with TLS certificates) and offline signing.
Sample configuration files for the HTTP reverse proxy can be found in
@code{/etc/taler-auditor/}.
@node Installing the GNU Taler binary packages on Ubuntu,,Installing the GNU Taler binary packages on Debian,Installation
@anchor{taler-auditor-manual installing-the-gnu-taler-binary-packages-on-ubuntu}@anchor{a}
@section Installing the GNU Taler binary packages on Ubuntu
To install the GNU Taler Ubuntu packages, first ensure that you have
the right Ubuntu distribution. At this time, the packages are built for
Ubuntu Lunar and Ubuntu Jammy. Make sure to have @code{universe} in your
@code{/etc/apt/sources.list} (after @code{main}) as we depend on some packages
from Ubuntu @code{universe}.
A typical @code{/etc/apt/sources.list.d/taler.list} file for this setup
would look like this for Ubuntu Lunar:
@example
deb [signed-by=/etc/apt/keyrings/taler-systems.gpg] https://deb.taler.net/apt/ubuntu/ lunar taler-lunar
@end example
For Ubuntu Jammy use this instead:
@example
deb [signed-by=/etc/apt/keyrings/taler-systems.gpg] https://deb.taler.net/apt/ubuntu/ jammy taler-jammy
@end example
The last line is crucial, as it adds the GNU Taler packages.
Next, you must import the Taler Systems SA public package signing key
into your keyring and update the package lists:
@example
# wget -O /etc/apt/keyrings/taler-systems.gpg \
https://taler.net/taler-systems.gpg
# apt update
@end example
@cartouche
@quotation Note
You may want to verify the correctness of the Taler Systems key out-of-band.
@end quotation
@end cartouche
Now your system is ready to install the official GNU Taler binary packages
using apt.
To install the Taler exchange, you can now simply run:
@example
# apt install -t focal-fossa taler-auditor
@end example
For the auditor, you must manually configure access to the exchange database,
the HTTP reverse proxy (typically with TLS certificates) and offline signing.
Sample configuration files for the HTTP reverse proxy can be found in
@code{/etc/taler-auditor/}.
@node System setup,Configuration,Installation,Top
@anchor{taler-auditor-manual system-setup}@anchor{b}
@chapter System setup
@menu
* UNIX accounts::
* Databases and users::
@end menu
@node UNIX accounts,Databases and users,,System setup
@anchor{taler-auditor-manual unix-accounts}@anchor{c}
@section UNIX accounts
For maximum security, you should setup multiple different users (possibly
on different machines) to run Taler auditor components. While it is possible
to skip some of these entirely, or to run all of them as the same user, this
is not recommended for security. The recommended set of users includes:
@quotation
@itemize *
@item
auditor — runs the main auditing process and HTTP backend
@item
sync — synchronizes the ingres database with the production database
@item
helper — runs taler-auditor-offline download and upload commands
@item
auditor-ingres — imports database from exchange production system
@item
auditor-wire — imports wire transfer data from bank production system
@item
offline — manages the offline key, on a separate `offline' machine
@end itemize
@end quotation
It is suggested that you setup the first five users on the target system(s)
using:
@example
# add-user --disabled-password $USERNAME
@end example
Additionally, there are two canonical system users of relevance (which your
distribution would typically create for you):
@quotation
@itemize *
@item
www-data — runs the HTTPS frontend (usually nginx or Apache)
@item
postgres — runs the PostgreSQL database
@end itemize
@end quotation
@node Databases and users,,UNIX accounts,System setup
@anchor{taler-auditor-manual databases-and-users}@anchor{d}
@section Databases and users
We recommend using the following databases for the auditor:
@quotation
@itemize *
@item
exchange-ingres — synchronized exchange database over the network
@item
exchange-production — local copy of exchange database with trusted schema
@item
auditor — auditor production database with current state of the audit
@item
libeufin — local state of the auditor-wire tool for the bank transfer data import
@end itemize
@end quotation
As the `postgres' user, you can create these databases using:
@example
# As the 'postgres' user:
$ createdb -O auditor-ingres exchange-ingres
$ createdb -O sync exchange-production
$ createdb -O auditor auditor
$ createdb -O auditor-wire libeufin
@end example
This will ensure that the correct users have write-access to their
respective database. Next, you need to grant read-only access to
some users to databases owned by other users:
@example
# As the 'auditor-ingres' user:
$ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO sync;' | psql exchange-ingres
# As the 'sync' user:
$ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO auditor;' | psql exchange-production
# As the 'auditor-wire' user:
$ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO auditor;' | psql libeufin
@end example
@node Configuration,Deployment,System setup,Top
@anchor{taler-auditor-manual configuration}@anchor{e}
@chapter Configuration
The auditor’s configuration works the same way as the configuration of other
Taler components.
This section discusses configuration options related to the auditor.
@menu
* Configuration format::
* Initial configuration::
* Keys::
* Configuring the auditor’s REST endpoint::
* Bank account::
* Database::
* Legal conditions for using the service::
* Terms of Service::
* Privacy Policy::
* Legal policies directory layout::
* Generating the Legal Terms::
* Adding translations::
* Updating legal documents::
@end menu
@node Configuration format,Initial configuration,,Configuration
@anchor{taler-auditor-manual configuration-format}@anchor{f}
@section Configuration format
All GNU Taler components are designed to possibly share the same
configuration files. When installing a GNU Taler component, the
installation deploys default values in configuration files located
at $@{prefix@}/share/taler/config.d/ where $@{prefix@} is the installation
prefix. Different components must be installed to the same prefix.
In order to override these defaults, the user can write a custom configuration
file and either pass it to the component at execution time using the `-c'
option, or name it taler.conf and place it under $HOME/.config/ which is where
components will look by default. Note that the systemd service files pass @code{-c
/etc/taler/taler.conf}, thus making @code{/etc/taler/taler.conf}
the primary location for the configuration.
A config file is a text file containing sections, and each section
contains maps options to their values. Configuration files follow
basically the INI syntax:
@example
[section1]
value1 = string
value2 = 23
[section2]
value21 = string
value22 = /path22
@end example
Comments start with a hash (@code{#}). Throughout the configuration, it is
possible to use @code{$}-substitution for options relating to names of files or
directories. It is also possible to provide defaults values for those
variables that are unset, by using the following syntax:
@code{$@{VAR:-default@}}. There are two ways a user can set the value
of @code{$}-prefixable variables:
@quotation
@enumerate
@item
by defining them under a @code{[paths]} section:
@end enumerate
@quotation
@example
[paths]
TALER_DEPLOYMENT_SHARED = $@{HOME@}/shared-data
..
[section-x]
path-x = $@{TALER_DEPLOYMENT_SHARED@}/x
@end example
@end quotation
@enumerate 2
@item
or by setting them in the environment:
@end enumerate
@quotation
@example
$ export VAR=/x
@end example
@end quotation
@end quotation
The configuration loader will give precedence to variables set under
@code{[path]} over environment variables.
The utility @code{taler-config}, which gets installed along with the exchange,
can be used get and set configuration values without directly editing the
configuration file. The option @code{-f} is particularly useful to resolve
pathnames, when they use several levels of @code{$}-expanded variables. See
@code{taler-config --help}.
The repository @code{git://git.taler.net/deployment} contains example code
for generating configuration files under @code{deployment/netzbon/}.
@node Initial configuration,Keys,Configuration format,Configuration
@anchor{taler-auditor-manual initial-configuration}@anchor{10}@anchor{taler-auditor-manual setupbaseurl}@anchor{11}
@section Initial configuration
You need to tell the Taler auditor configuration where the
REST API of the auditor will be available to the public:
@example
# Both for the 'offline' *and* the 'auditor' user:
[auditor]
BASE_URL = https://auditor.example.com/
@end example
The @code{helper} user that is used to download information from the exchange
needs to know details about the exchange. Similarly, the @code{offline} user
needs to check signatures signed with the exchange’s offline key. Hence, you
need to obtain the @code{MASTER_PUBLIC_KEY} from the exchange operator (they need
to run @code{taler-exchange-offline setup}) and the REST endpoint of the exchange
and configure these:
@example
# As the 'helper' and 'offline' users:
[exchange]
BASE_URL = https://exchange.example.com/
MASTER_PUBLIC_KEY = $SOMELONGBASE32VALUEHERE
@end example
@node Keys,Configuring the auditor’s REST endpoint,Initial configuration,Configuration
@anchor{taler-auditor-manual auditorkeys}@anchor{12}@anchor{taler-auditor-manual keys}@anchor{13}
@section Keys
The auditor works with one signing key to certify that it is auditing
a particular exchange’s denomination keys. This key can and should
be kept `offline' (and backed up adequately). As with the exchange’s
offline key, it is only used for a few cryptographic signatures and
thus the respective code can be run on modest hardware, like a
Raspberry Pi.
The following values are to be configured in the section @code{[auditor]}:
@itemize -
@item
@code{AUDITOR_PRIV_FILE}: Path to the auditor’s private key file.
@end itemize
Note that the default value here should be fine and there is no clear
need to change it. What you do need to do as the @code{offine} user
is to extract the public key:
@example
# As the 'offline' user:
$ taler-auditor-offline setup
@end example
This public key must then be provided in the configuration file
of the @code{auditor} user in the @code{[auditor]]} configuration section:
@itemize -
@item
@code{PUBLIC_KEY}: Public key of the auditor, in Base32 encoding.
Set from value printed by @code{taler-auditor-offline setup}.
@end itemize
You can set this configuration value using:
@example
# As the 'auditor' and 'helper' users:
[auditor]
PUBLIC_KEY = $SOMELONGBASE32VALUEHERE
@end example
@node Configuring the auditor’s REST endpoint,Bank account,Keys,Configuration
@anchor{taler-auditor-manual auditorserving}@anchor{14}@anchor{taler-auditor-manual configuring-the-auditor-s-rest-endpoint}@anchor{15}
@section Configuring the auditor’s REST endpoint
The auditor can serve HTTP over both TCP and UNIX domain socket.
The following values are to be configured in the section @code{[auditor]}:
@itemize -
@item
@code{serve}: must be set to @code{tcp} to serve HTTP over TCP, or @code{unix} to serve
HTTP over a UNIX domain socket
@item
@code{port}: Set to the TCP port to listen on if @code{serve} is @code{tcp}.
@item
@code{unixpath}: set to the UNIX domain socket path to listen on if @code{serve} is
@code{unix}
@item
@code{unixpath_mode}: number giving the mode with the access permission MASK
for @code{unixpath} (i.e. 660 = @code{rw-rw----}).
@end itemize
@node Bank account,Database,Configuring the auditor’s REST endpoint,Configuration
@anchor{taler-auditor-manual auditorbank-account}@anchor{16}@anchor{taler-auditor-manual bank-account}@anchor{17}
@section Bank account
Bank accounts for the auditor (user @code{auditor-wire}) are configured in
exactly the same way as bank accounts for the exchange. See the exchange (and
LibEuFin) documentation for details.
@node Database,Legal conditions for using the service,Bank account,Configuration
@anchor{taler-auditor-manual auditordatabaseconfiguration}@anchor{18}@anchor{taler-auditor-manual database}@anchor{19}
@section Database
The option @code{DB} under section @code{[auditor]} gets the DB backend’s name the
exchange is going to use. So far, only @code{DB = postgres} is supported. After
choosing the backend, it is mandatory to supply the connection string
(namely, the database name). This is possible in two ways:
@itemize -
@item
via an environment variable: @code{TALER_AUDITORDB_POSTGRES_CONFIG}.
@item
via configuration option @code{CONFIG}, under section @code{[auditordb-$BACKEND]}.
For example, the demo exchange is configured as follows:
@example
[auditor]
...
DB = postgres
...
[auditordb-postgres]
CONFIG = postgres:///auditordemo
@end example
@end itemize
If an exchange runs its own auditor, it may use the same database for
the auditor and the exchange itself.
The @code{taler-auditor-dbinit} tool is used to initialize the auditor’s
tables. After running this tool, the rights to CREATE or DROP tables
are no longer required and should be removed.
Both the @code{taler-auditor-httpd} and the @code{taler-auditor} (and its helpers)
also need (read-only) access to a (recent, current, synchronized) copy of the
exchange’s database. The configuration options are the same that are also
used when configuring the exchange’ database:
@quotation
@example
[exchange]
...
DB = postgres
...
[exchangedb-postgres]
CONFIG = postgres:///exchangedemo
@end example
@end quotation
@node Legal conditions for using the service,Terms of Service,Database,Configuration
@anchor{taler-auditor-manual legal-conditions-for-using-the-service}@anchor{1a}
@section Legal conditions for using the service
@c This file is part of GNU TALER.
@c
@c Copyright (C) 2014-2023 Taler Systems SA
@c
@c TALER is free software; you can redistribute it and/or modify it under the
@c terms of the GNU Affero General Public License as published by the Free Software
@c Foundation; either version 2.1, or (at your option) any later version.
@c
@c TALER is distributed in the hope that it will be useful, but WITHOUT ANY
@c WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
@c A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
@c
@c You should have received a copy of the GNU Affero General Public License along with
@c TALER; see the file COPYING. If not, see
@c
@c @author Christian Grothoff
The service has well-known API endpoints to return its legal conditions to the
user in various languages and various formats. This section describes how to
setup and configure the legal conditions.
@node Terms of Service,Privacy Policy,Legal conditions for using the service,Configuration
@anchor{taler-auditor-manual terms-of-service}@anchor{1b}
@section Terms of Service
The service has an endpoint “/terms” to return the terms of service (in legal
language) of the service operator. Client software show these terms of
service to the user when the user is first interacting with the service.
Terms of service are optional for experimental deployments, if none are
configured, the service will return a simple statement saying that there are
no terms of service available.
To configure the terms of service response, there are two options
in the configuration file for the service:
@itemize -
@item
@code{TERMS_ETAG}: The current “Etag” to return for the terms of service.
This value must be changed whenever the terms of service are
updated. A common value to use would be a version number.
Note that if you change the @code{TERMS_ETAG}, you MUST also provide
the respective files in @code{TERMS_DIR} (see below).
@item
@code{TERMS_DIR}: The directory that contains the terms of service.
The files in the directory must be readable to the service
process.
@end itemize
@node Privacy Policy,Legal policies directory layout,Terms of Service,Configuration
@anchor{taler-auditor-manual privacy-policy}@anchor{1c}
@section Privacy Policy
The service has an endpoint “/pp” to return the terms privacy policy (in legal
language) of the service operator. Clients should show the privacy policy to
the user when the user explicitly asks for it, but it should not be shown by
default. Privacy policies are optional for experimental deployments, if none
are configured, the service will return a simple statement saying that there
is no privacy policy available.
To configure the privacy policy response, there are two options
in the configuration file for the service:
@itemize -
@item
@code{PRIVACY_ETAG}: The current “Etag” to return for the privacy policy.
This value must be changed whenever the privacy policy is
updated. A common value to use would be a version number.
Note that if you change the @code{PRIVACY_ETAG}, you MUST also provide
the respective files in @code{PRIVACY_DIR} (see below).
@item
@code{PRIVACY_DIR}: The directory that contains the privacy policy.
The files in the directory must be readable to the service
process.
@end itemize
@node Legal policies directory layout,Generating the Legal Terms,Privacy Policy,Configuration
@anchor{taler-auditor-manual legal-policies-directory-layout}@anchor{1d}
@section Legal policies directory layout
The @code{TERMS_DIR} and @code{PRIVACY_DIR} directory structures must follow a
particular layout. You may use the same directory for both the terms of
service and the privacy policy, as long as you use different ETAGs. Inside of
the directory, there should be sub-directories using two-letter language codes
like “en”, “de”, or “jp”. Each of these directories would then hold
translations of the current terms of service into the respective language.
Empty directories are permitted in case translations are not available.
Then, inside each language directory, files with the name of the value set as
the @code{TERMS_ETAG} or @code{PRIVACY_ETAG} must be provided. The extension of each
of the files should be typical for the respective mime type. The set of
supported mime types is currently hard-coded in the service, and includes
“.epub”, “.html”, “.md”, “.pdf” and “.txt” files. If other files are present,
the service may show a warning on startup.
@menu
* Example::
@end menu
@node Example,,,Legal policies directory layout
@anchor{taler-auditor-manual example}@anchor{1e}
@subsection Example
A sample file structure for a @code{TERMS_ETAG} of “tos-v0” would be:
@itemize -
@item
TERMS_DIR/en/tos-v0.txt
@item
TERMS_DIR/en/tos-v0.html
@item
TERMS_DIR/en/tos-v0.pdf
@item
TERMS_DIR/en/tos-v0.epub
@item
TERMS_DIR/en/tos-v0.md
@item
TERMS_DIR/de/tos-v0.txt
@item
TERMS_DIR/de/tos-v0.html
@item
TERMS_DIR/de/tos-v0.pdf
@item
TERMS_DIR/de/tos-v0.epub
@item
TERMS_DIR/de/tos-v0.md
@end itemize
If the user requests an HTML format with language preferences “fr” followed by
“en”, the service would return @code{TERMS_DIR/en/tos-v0.html} lacking a version in
French.
@node Generating the Legal Terms,Adding translations,Legal policies directory layout,Configuration
@anchor{taler-auditor-manual generating-the-legal-terms}@anchor{1f}
@section Generating the Legal Terms
The @code{taler-terms-generator} script can be used to generate directories with
terms of service and privacy policies in multiple languages and all required
data formats from a single source file in @code{.rst} format and GNU gettext
translations in @code{.po} format.
To use the tool, you need to first write your legal conditions in English in
reStructuredText (rst). You should find a templates in
@code{$PREFIX/share/terms/*.rst} where @code{$PREFIX} is the location where you
installed the service to. Whenever you make substantive changes to the legal
terms, you must use a fresh filename and change the respective @code{ETAG}. The
resulting file must be called @code{$ETAG.rst} and the first line of the file should be the title of the document.
Once you have written the @code{$ETAG.rst} file in English, you can
generate the first set of outputs:
@example
$ taler-terms-generator -i $ETAG
@end example
Afterwards, you should find the terms in various formats for all configured
languages (initially only English) in @code{$PREFIX/share/terms/}. The generator
has a few options which are documented in its man page.
@node Adding translations,Updating legal documents,Generating the Legal Terms,Configuration
@anchor{taler-auditor-manual adding-translations}@anchor{20}
@section Adding translations
Translations must be available in subdirectories
@code{locale/$LANGUAGE/LC_MESSAGES/$ETAG.po}.
To start translating, you first need to add a new
language:
@example
$ taler-terms-generator -i $ETAG -l $LANGUAGE
@end example
Here, @code{$LANGUAGE} should be a two-letter language
code like @code{de} or @code{fr}. The command will generate
a file @code{locale/$LANGUAGE/LC_MESSAGES/$ETAG.po}
which contains each English sentence or paragraph
in the original document and an initially empty
translation. Translators should update the @code{.po}
file. Afterwards, simply re-run
@example
$ taler-terms-generator -i $ETAG
@end example
to make the current translation(s) available to the
service.
@cartouche
@quotation Note
You must restart the service whenever adding or updating legal documents or their translations.
@end quotation
@end cartouche
@node Updating legal documents,,Adding translations,Configuration
@anchor{taler-auditor-manual updating-legal-documents}@anchor{21}
@section Updating legal documents
When making minor changes without legal implications, edit the @code{.rst} file,
then re-run the step to add a new language for each existing translation to
produce an updated @code{.po} file. Translate the sentences that have changed and
finally run the generator (without @code{-l}) on the ETAG (@code{-i $ETAG}) to
create the final files.
When making major changes with legal implications, you should first rename (or
copy) the existing @code{.rst} file and the associated translation files to a new
unique name. Afterwards, make the major changes, update the @code{.po} files,
complete the translations and re-create the final files. Finally, do not
forget to update the @code{ETAG} configuration option to the new name and to
restart the service.
@node Deployment,Operation,Configuration,Top
@anchor{taler-auditor-manual auditordeployment}@anchor{22}@anchor{taler-auditor-manual deployment}@anchor{23}
@chapter Deployment
@anchor{taler-auditor-manual wallets}@anchor{24}
Before GNU Taler wallets will happily interact with an exchange, the
respective auditor’s public key (as obtained via @code{taler-auditor-offline
setup} from the @code{offline} user) must be added under the respective currency
to the wallet. This is usually expected to be hard-coded into the Taler
wallet.
Users can also manually add auditors for a particular currency via a
Web page offering the respective pairing.
FIXME-DOLD: explain how that Web page works, once it works…
@menu
* Exchange::
* Signing Denominations::
* Database: Database<2>.
@end menu
@node Exchange,Signing Denominations,,Deployment
@anchor{taler-auditor-manual auditorexchange}@anchor{25}@anchor{taler-auditor-manual exchange}@anchor{26}
@section Exchange
The next step is to add the exchange’s master public key and the base URL of
the exchange to the list of exchanges audited by the auditor. This is done
using the @code{taler-auditor-exchange} tool. The tool basically creates the
respective record in the auditor’s database.
If this step is skipped, the auditor will malfunction at all future stages
with a foreign key violation, as it does not know the exchange’s master public
key.
@example
# As the 'auditor' user:
$ taler-auditor-exchange -m $MASTER_PUB -u $EXCHANGE_BASE_URL
@end example
An equivalent step must be performed by the exchange operator. Here, the
exchange operator must use the @code{taler-exchange-offline} tool to add the
auditor’s public key, base URL and (business) name to the list of approved
auditors of the exchange. For details, see Auditor-configuration in the
exchange operator manual.
@node Signing Denominations,Database<2>,Exchange,Deployment
@anchor{taler-auditor-manual signing-denominations}@anchor{27}@anchor{taler-auditor-manual signingdenominations}@anchor{28}
@section Signing Denominations
@geindex maintenance
These steps must be performed `regularly' whenever the exchange is
deploying new denomination keys. After the exchange operator
has signed new keys using the @code{taler-exchange-offline} tool,
each auditor should run:
@example
# As the 'helper' user:
$ taler-auditor-offline download > input.json
@end example
to import the latest set of denomination keys. The key data should then be
copied to the offline system and there be inspected using:
@example
# As the 'offline' user:
$ taler-auditor-offline show < input.json
@end example
and compared with the data the exchange operator saw when doing the offline
signature. This process should involve directly the humans operating both
systems and may require them to establish a trustworthy connection. The
details how the auditor communicates with the exchange operator are a business
process that is outside of the scope of this document.
Note that the @code{input.json} does not contain any confidential data. However,
signing the wrong keys would be fatal in that it may allow an illegitimate
exchange to convince users that it is a trustworthy operator and subsequently
betray the user’s trust that is anchored in the existence of a trustworthy
auditor.
Given the verified JSON input, the auditor can then sign it (typically
on its offline system) using:
@example
# As the 'offline' user:
$ taler-auditor-offline sign < input.json > output.json
@end example
The resulting @code{output.json} should then be copied to an online system,
and from there uploaded to the exchange using:
@example
# As the 'helper' user:
$ taler-auditor-offline upload < output.json
@end example
The contents of @code{output.json} can again be public and require no special
handling.
If the auditor has been correctly added, the exchange’s @code{/keys}
response will contain an entry in the @code{auditors} array mentioning the
auditor’s URL.
Commands, like @code{taler-auditor-offline}, that support the @code{-l LOGFILE}
command-line option, send logging output to standard error by default.
@node Database<2>,,Signing Denominations,Deployment
@anchor{taler-auditor-manual auditordatabaseinitialization}@anchor{29}@anchor{taler-auditor-manual id1}@anchor{2a}
@section Database
The next key step for the auditor is to configure replication of the
`exchange'’s database in-house. This should be performed in two steps
as illustrated in the following figure:
@image{taler-auditor-figures/replication,,,,png}
First, the exchange should use standard PostgreSQL replication features to
enable the auditor to obtain a full copy of the exchange’s database.
Second, the auditor should make a “trusted” local copy, ensuring that it
never replicates malicious changes using @code{taler-auditor-sync}. Both
of these steps are described in more detail below.
We note that as a result of these steps, the auditor will have three
databases: its own production primary database (as configured in
@code{auditordb-postgres}), its on production copy of the exchange’s database
(@code{exchangedb-postgress}), and a third, untrusted “ingres” copy of the
exchange database. The untrusted database should run as a separate PostgreSQL
instance and is only accessed via @code{taler-auditor-sync} and the replication
mechanism driven by the exchange operator.
@menu
* Ingres replication of the exchange production database::
* Safe replication of the ingres database into the auditor production database::
@end menu
@node Ingres replication of the exchange production database,Safe replication of the ingres database into the auditor production database,,Database<2>
@anchor{taler-auditor-manual ingres-replication-of-the-exchange-production-database}@anchor{2b}
@subsection Ingres replication of the exchange production database
Ingres operation should be done using the @code{auditor-ingres} user — or
depending on the setup parts of the operation may be done by the @code{postgres}
user directly.
The full copy can be obtained in various ways with PostgreSQL. It is
possible to use log shipping with streaming replication as described
in @indicateurl{https://www.postgresql.org/docs/13/warm-standby.html}, or to use
logical replication, as described in
@indicateurl{https://www.postgresql.org/docs/13/logical-replication.html}. We note
that asynchronous replication should suffice.
The resulting auditor database should be treated as read-only on the auditor
side. The @code{taler-exchange-dbinit} tool can be used to setup the schema, or
the schema can be replicated using PostgreSQL’s standard mechanisms. The same
applies for schema upgrades: if logical replication is used (which does not
replicate schema changes), @code{taler-exchange-dbinit} can be used to migrate
the schema(s) in both the ingres and production copies of the exchange’s
database as well.
On the exchange side, a database user must be created that has the right
to perform database replication. This is done using:
@example
# As the 'postgres' user of the exchange:
$ createuser --replication egress
$ echo "ALTER ROLE egress WITH PASSWORD '$PASSWORD'; | psql
$ echo "CREATE PUBLICATION $NAME FOR ALL TABLES;" | psql taler-exchange
@end example
The exchange must share the password of the publication with the auditor. A
good @code{$NAME} relates to the auditor’s business unit name. A secure tunnel
must be setup between the exchange and the auditor, for example using SSH or
Wireguard.
It is also necessary to edit @code{main.cf} of the exchange and on the auditor
side to enable logical replication. If an exchange has multiple auditors, it
should setup multiple @code{egress} accounts. The exchange must ensure that
the following lines are in the @code{main.cf} PostgreSQL configuration (the port
may differ) to enable replication over the network:
@example
listen_addresses='*'
port = 5432
wal_level= logical
@end example
Equally, the auditor must configure logical replication in the @code{main.cf}
PostgreSQL configuration:
@example
wal_level= logical
@end example
Next, the @code{postgres} user of the auditor’s system must first initialize the
local tables:
@example
# Configure database:
[exchange]
DB = "postgres"
[exchangedb-postgres]
CONFIG = "postgres:///taler-ingress"
@end example
@example
# As the 'ingress' user of the exchange:
$ taler-exchange-dbinit
@end example
To complete the replication, the @code{postgres} user of the auditor’s
system must subscribe:
@example
# As the 'postgres' user of the exchange:
$ createuser --replication egress
$ echo "ALTER ROLE egress WITH PASSWORD '$PASSWORD'; | psql
$ echo "CREATE PUBLICATION $NAME FOR ALL TABLES;" | psql taler-exchange
@end example
For details, we refer to the PostgreSQL manual.
@cartouche
@quotation Note
Depending on the replication method used, the exchange may perform
unexpected changes to the schema or perform @code{UPDATE}, @code{DELETE} or
@code{DROP} operations on the tables. Hence, the auditor cannot rely upon the
exchange’s primary copy to respect schema constraints, especially as we
have to presume that the exchange could act maliciously. Furthermore, it
is unclear to what degree PostgreSQL database replication mechanisms are
robust against a malicious master database. Thus, the auditor should
isolate its primary copy of the exchange database, including the PostgreSQL
process, from its actual operational data.
@end quotation
@end cartouche
@node Safe replication of the ingres database into the auditor production database,,Ingres replication of the exchange production database,Database<2>
@anchor{taler-auditor-manual safe-replication-of-the-ingres-database-into-the-auditor-production-database}@anchor{2c}
@subsection Safe replication of the ingres database into the auditor production database
Using @code{taler-auditor-sync} as the @code{sync} user, the auditor should
make a second “safe” copy of the exchange’s ingres database.
@code{taler-auditor-sync} basically reads from one exchange database and inserts
all records found into a second exchange database. If the source database
violates invariants, the tool halts with an error. This way, records violating
invariants are never even copied, and in particular schema changes and
deletions or updates are not propagated into the auditor’s production
database.
While @code{taler-auditor-sync} could in theory be run directly against the
exchange’s production system, this is likely a bad idea due to the high
latency from the network between auditor and exchange operator. Thus, we
recommend first making an “untrusted” ingress copy of the exchange’s
production database using standard PostgreSQL tooling, and then using
@code{taler-auditor-sync} to create a second “safe” copy. The “safe” copy used
by the production system should also run under a different UID.
Before @code{taler-auditor-sync} can be used, the target database must be
initialized with the exchange schema using @code{taler-exchange-dbinit}.
Note that running @code{taler-auditor-sync} requires the use of two
configuration files, one specifying the options for accessing the source
database, and a second with the options for accessing the destination
database. In both cases, likely only the @code{[exchangedb]/CONFIG} option
needs to be changed.
To run @code{taler-auditor-sync}, you must first configure two configuration
files that identify the source and destination databases:
@example
# src.conf
[exchangedb]
CONFIG = "postgres:///auditor-ingres/"
@end example
@example
# dst.conf
[exchangedb]
CONFIG = "postgres:///auditor/"
@end example
Now you should be able to launch the synchronization process. You can run
the process via systemd in the background. For a first one-off test, you should
use the @code{-t} option which will cause the process to terminate once the two
databases are synchronized:
@example
# As the 'sync' user:
$ taler-auditor-sync -s src.conf -d dst.cfg -t
@end example
When the exchange performs garbage collection to @code{DELETE} obsolete records,
this change should be automatically replicated to the auditors untrusted
ingress database. However, as @code{taler-auditor-sync} tries to be “safe”,
it will not replicate those deletions to the auditor’s production database.
Thus, it is necessary to (occasonally) run @code{taler-exchange-dbinit -g} on
the auditor’s production database to garbage collect old data in the
auditor’s production copy. We note that this does not have to be done
at the same time when the exchange runs its garbage collection.
@node Operation,Auditor implementation guide,Deployment,Top
@anchor{taler-auditor-manual id2}@anchor{2d}@anchor{taler-auditor-manual operation}@anchor{2e}
@chapter Operation
@menu
* Web service::
* Audit::
* Reading the report::
* Database upgrades::
* Database reset::
* Revocations::
* Failures::
@end menu
@node Web service,Audit,,Operation
@anchor{taler-auditor-manual id3}@anchor{2f}@anchor{taler-auditor-manual web-service}@anchor{30}
@section Web service
The @code{taler-auditor-httpd} runs the required REST API for the auditor. The
service must have @code{INSERT} (and @code{SELECT}) rights on the
@code{deposit_confirmations} table in the auditor’s database. We expect that in
future versions additional rights may be required.
For now, we recommend simply running the @code{taler-auditor-httpd} under the
@code{auditor} user. However, it is also possible (and might be more secure) to
create a separate user with more restrictive permissions for this purpose.
As the @code{taler-auditor-httpd} does not include HTTPS-support, it is
advisable to run it behind a reverse proxy that offers TLS termination.
@node Audit,Reading the report,Web service,Operation
@anchor{taler-auditor-manual audit}@anchor{31}@anchor{taler-auditor-manual id4}@anchor{32}
@section Audit
Performing an audit is done by invoking the @code{taler-auditor} shell script as
the @code{auditor} user.
The shell script invokes the various helper processes. For additional
performance and security, one may want to run the various helpers individually
and with the respective minimal set of access rights (only
@code{taler-helper-auditor-wire} needs the credentials to query the bank for wire
transfers, alas if @code{auditor-wire} is used to talk to the bank, this issue is
already addressed). The shell script combines the final JSON outputs of the
various helpers using the @code{taler-helper-auditor-render.py} script into the
TeX report. Regardless, the simplest way to obtain a report is to run:
@example
$ taler-audit
@end example
This generates a file @code{auditor-report.pdf} (in a temporary directory created
for this purpose) with all of the issues found and the financial assessment of
the exchange. The exact filename will be output to the console upon
completion.
We note that @code{taler-audit} by default runs in incremental mode. As a result,
running the commands again will only check the database entries that have been
added since the last run.
You can use @code{taler-auditor-dbinit -r} to force a full check since the
beginning of time. However, as this may require excessive time and
interactions with the bank (which may not even have the wire transfer records
anymore), this is not recommended in a production setup.
@node Reading the report,Database upgrades,Audit,Operation
@anchor{taler-auditor-manual reading-the-report}@anchor{33}
@section Reading the report
The auditor’s report needs to be read carefully, as it includes
several categories of failures of different severity:
@itemize -
@item
Delayed operations, where an operation was expected to have
happened, but did not happen yet, possibly because of a
disagreement in system time or overloading of the system.
These failures only require action if the delays are
significant.
@item
Inconsistencies in the data that have no clear financial
impact.
@item
Inconsistencies in the data that show that the exchange
experienced an unexpected financial loss (such as accepting a coin for
deposit with an invalid signature).
@item
Inconsistencies in the data that show that the exchange
caused some other party to experience a financial loss (such as not wiring
the correct amount to a merchant).
@item
Configuration issues (such was wire fees unavailable).
@end itemize
@node Database upgrades,Database reset,Reading the report,Operation
@anchor{taler-auditor-manual auditordatabaseupgrades}@anchor{34}@anchor{taler-auditor-manual database-upgrades}@anchor{35}
@section Database upgrades
To upgrade the database between Taler versions can be done by running:
@example
$ taler-auditor-dbinit
$ taler-exchange-dbinit
@end example
In any case, it is recommended that exchange and auditor coordinate closely
during schema-changing database upgrades as without coordination the database
replication or @code{taler-auditor-sync} will likely experience problematic
failures. In general, we recommend:
@quotation
@itemize *
@item
halting the exchange business logic,
@item
allowing the replication and @code{taler-auditor-sync} to complete
(see also the `-t' option of @code{taler-auditor-sync})
@item
completing a @code{taler-audit} run against the old schema
@item
migrating the exchange schema (@code{taler-exchange-dbinit}) of
the master database, possibly the ingres database and the
auditor’s production copy
@item
migrating the auditor database (@code{taler-auditor-dbinit})
@item
resuming database replication between the exchange’s master
database and the auditor’s ingres copy
@item
resuming @code{taler-auditor-sync}
@item
resuming the regular exchange and auditor business logic
@end itemize
@end quotation
Regardless, the above is merely the general rule. Please review the specific
release notes to ensure this procedure is correct for the specific upgrade.
@node Database reset,Revocations,Database upgrades,Operation
@anchor{taler-auditor-manual database-reset}@anchor{36}
@section Database reset
The auditor database can be reset using:
@example
$ taler-auditor-dbinit -R
@end example
However, running this command will result in all data in the database being
`lost', including steps like enabling an exchange using
@code{taler-auditor-exchange}. Thus, doing so may result in significant
commputation (and bandwidth consumption with the bank) when the auditor is
next launched, as it will re-download and re-verify all historic transactions.
Hence this should not be done in a production system.
@node Revocations,Failures,Database reset,Operation
@anchor{taler-auditor-manual auditorrevocations}@anchor{37}@anchor{taler-auditor-manual revocations}@anchor{38}
@section Revocations
When an auditor detects that the private key of a denomination key pair has
been compromised, one important step is to revoke the denomination key. The
exchange operator includes the details on how to revoke a denomination key, so
the auditor should only have to report (and possibly enforce) this step.
For more information, see Revocations in the exchange operator manual.
If all denominations of an exchange are revoked, the exchange includes logic
to wire back all returned funds to the bank accounts from which they
originate. If some denominations remain operational, wallets will generally
exchange old coins of revoked denominations for new coins – while providing
additional information to demonstrate that these coins were not forged from
the compromised private key but obtained via a legitimate withdraw operation.
@node Failures,,Revocations,Operation
@anchor{taler-auditor-manual failures}@anchor{39}
@section Failures
Most audit failures are handled by the auditor’s regular reporting functionality,
creating a (hopefully descriptive) PDF report detailing the problems found.
However, there is one category of errors where this is not possible, which
concerns arithmetic overflows for amounts. Taler’s specification limits amount
values to at most 2^52. If, during the auditor’s calculations, amounts are
encountered that exceed this threshold, the auditor will not generate a regular
report, but instead write a log statement explaining where the problem happened
and exit with a status code of `42'.
The most common expected case when this happens is a corrupted database. This
could be because the exchange is actively malicious, or more likely due to
some data corruption. The audit cannot continue until the corruption has been
addressed. If it is not possible to restore a fully `correct' version of the
database, the suggestion is to replace the corrupted (and likely very large)
amounts with zero (Note: this does not apply to the value of denominations or
fees, here it is crucial that the correct amounts are restored). While an
amount of zero would be incorrect, the auditing logic should be able to do its
calculations with zero instead.
After patching the database, the audit can
be restarted. A full reset is not required, as the audit transaction is aborted
when the auditor exits with code `42'. After restarting, the resulting audit
report is likely to indicates errors relating to the corrupted fields (such as
invalid signatures, arithmetic errors by the exchange, etc.), but at least the
loss/gain calculations will be meaningful and actually indicative of the scope
of the error created by the corrupted data.
@node Auditor implementation guide,Index,Operation,Top
@anchor{taler-auditor-manual auditor-implementation-guide}@anchor{3a}
@chapter Auditor implementation guide
The auditor implementation is split into five main processes, called
@code{taler-helper-auditor-XXX}. The split was done to realize the principle of
least privilege and to enable independent logic to be possibly run in
parallel. Only the taler-wire-auditor must have (read-only) access to the
exchange’s bank account, the other components only need access to the
database.
All auditor subsystems basically start their audit from a certain transaction
index (@code{BIG SERIAL}) in the auditor database which identifies where the last
audit concluded. They then check that the transactions claimed in the
exchange’s database match up internally, including the cryptographic
signatures and also with respect to amounts adding up. The auditor also
calculates the exchange’s profits and expected bank balances. Once all
existing transactions are processed, the auditor processes store the current
checkpoint in its database and generate a JSON report.
The @code{taler-auditor} shell script calls the five helpers and then
uses Jinja2 with a TeX template to convert the five individual
JSON reports into LaTeX and then into PDF.
@menu
* The auditor’s database::
* Invariants checked by the auditor::
* Testing the auditor::
@end menu
@node The auditor’s database,Invariants checked by the auditor,,Auditor implementation guide
@anchor{taler-auditor-manual the-auditor-s-database}@anchor{3b}
@section The auditor’s database
The database scheme used by the exchange looks as follows:
@image{taler-auditor-figures/auditor-db,,,,png}
@node Invariants checked by the auditor,Testing the auditor,The auditor’s database,Auditor implementation guide
@anchor{taler-auditor-manual invariants-checked-by-the-auditor}@anchor{3c}
@section Invariants checked by the auditor
The auditor verifies a large number of invariants that must hold for a Taler
exchange. One objective in the design of the auditor was to check each
invariant only once, both to minimize cost and to avoid duplicate reporting of
problems where possible. As a result, not every invariant is checked in every
pass where it might seem applicable.
@menu
* Invariants checked by the taler-helper-auditor-aggregation::
* Invariants checked by the taler-helper-auditor-coins::
* Invariants checked by the taler-helper-auditor-deposits::
* Invariants checked by the taler-helper-auditor-reserves::
* Invariants checked by the taler-helper-auditor-wire::
@end menu
@node Invariants checked by the taler-helper-auditor-aggregation,Invariants checked by the taler-helper-auditor-coins,,Invariants checked by the auditor
@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-aggregation}@anchor{3d}
@subsection Invariants checked by the taler-helper-auditor-aggregation
This is from CodeBlau’s analysis. A proper write-up is pending.
CodeBlau reports the following checks:
@itemize -
@item
arithmetic inconsistencies
@itemize -
@item
disagreement in fee for deposit between auditor and exchange db
@item
disagreement in fee for melt between auditor and exchange db
@item
disagreement in fee for refund between auditor and exchange db
@item
aggregation of fee is negative
@item
aggregation (contribution): Expected coin contributions differ:
coin value without fee, total deposit without refunds
@item
wire out fee is negative
@end itemize
@item
coin arithmetic inconsistencies
@itemize -
@item
refund (merchant) is negative
@item
refund (balance) is negative
@item
spend > value
@end itemize
@item
coin denomination signature invalid
@item
start date before previous end date
@item
end date after next start date
@item
wire out inconsistencies in amount
@item
row inconsistencies
@itemize -
@item
wire account given is malformed
@item
h(wire) does not match wire
@item
failed to compute hash of given wire data
@item
database contains wrong hash code for wire details
@item
no transaction history for coin claimed in aggregation
@item
could not get coin details for coin claimed in aggregation
@item
could not find denomination key for coin claimed in aggregation
@item
coin denomination signature invalid
@item
target of outgoing wire transfer do not match hash of wire from deposit
@item
date given in aggregate does not match wire transfer date
@item
wire fee signature invalid at given time
@item
specified wire address lacks method
@item
wire fee unavailable for given time
@end itemize
@end itemize
@node Invariants checked by the taler-helper-auditor-coins,Invariants checked by the taler-helper-auditor-deposits,Invariants checked by the taler-helper-auditor-aggregation,Invariants checked by the auditor
@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-coins}@anchor{3e}
@subsection Invariants checked by the taler-helper-auditor-coins
This is from CodeBlau’s analysis. A proper write-up is pending.
CodeBlau reports the following checks:
@itemize -
@item
check that all denominations used by the exchange have been signed using
this auditor’s key. All denominations encountered in the database that
this auditor did not officially sign for are reported (but still included
in the audit as they obviously may impact the exchange’s bank balance).
Depending on the business situation, this may be normal (say if an exchange
is changing auditors and newer denominations are no longer supported until
their end-of-life by the current auditor).
@item
emergency on denomination over loss
@itemize -
@item
value of coins deposited exceed value of coins issued
@end itemize
@item
emergency on number of coins, num mismatch
@item
arithmetic inconsistencies
@itemize -
@item
melt contribution vs. fee
@item
melt (cost)
@item
refund fee
@end itemize
@item
row inconsistencies
@itemize -
@item
revocation signature invalid
@item
denomination key not found
@item
denomination key for fresh coin unknown to auditor
@item
denomination key for dirty coin unknown to auditor
@item
denomination key for deposited coin unknown to auditor
@end itemize
@item
coin validity in known_coin, by checking denomination signatures
@item
coin validity in melt, by checking signatures
@item
refresh hanging, zero reveals (harmless)
@item
verify deposit signature
@item
verify refund signature
@item
recoup, check coin
@item
recoup, check signature
@item
recoup, denomination not revoked
@end itemize
@node Invariants checked by the taler-helper-auditor-deposits,Invariants checked by the taler-helper-auditor-reserves,Invariants checked by the taler-helper-auditor-coins,Invariants checked by the auditor
@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-deposits}@anchor{3f}
@subsection Invariants checked by the taler-helper-auditor-deposits
This tool verifies that the deposit confirmations reported by merchants
directly to the auditor are also included in the database duplicated from the
exchange at the auditor. This is to ensure that the exchange cannot defraud
merchants by simply not reporting deposits to the auditor.
@node Invariants checked by the taler-helper-auditor-reserves,Invariants checked by the taler-helper-auditor-wire,Invariants checked by the taler-helper-auditor-deposits,Invariants checked by the auditor
@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-reserves}@anchor{40}
@subsection Invariants checked by the taler-helper-auditor-reserves
This is from CodeBlau’s analysis. A proper write-up is pending.
CodeBlau reports the following checks:
@itemize -
@item
report arithmetic inconsistency
@itemize -
@item
closing aggregation fee
@item
global escrow balance
@end itemize
@item
denomination key validity withdraw inconsistencies
@item
bad signature losses in withdraw
@item
bad signature losses in recoup
@item
bad signature losses in recoup-master
@item
reserve balance, insufficient, losses and gains
@item
reserve balance, summary wrong
@item
reserve not closed after expiration time
@item
could not determine closing fee / closing-fee unavailable
@item
denomination key not found for withdraw
@item
denomination key not in revocation set for recoup
@item
target account not verified, auditor does not know reserve
@item
target account does not match origin account
@end itemize
@node Invariants checked by the taler-helper-auditor-wire,,Invariants checked by the taler-helper-auditor-reserves,Invariants checked by the auditor
@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-wire}@anchor{41}
@subsection Invariants checked by the taler-helper-auditor-wire
This auditor is special in that it is the only pass that is required to have
`read-only' access to the exchange’s bank account (privilege separation). Its
main role is to verify that the wire transfers in the exchange’s database and
those reported by the bank are identical.
This is from CodeBlau’s analysis. A proper write-up is pending.
CodeBlau reports the following checks:
@itemize -
@item
check pending
@item
wire missing
@item
execution date mismatch
@item
wire out consistency
@item
wire transfer not made (yet?)
@item
receiver account mismatch
@item
wire amount does not match
@item
justification for wire transfer not found
@item
duplicate subject hash
@item
duplicate wire offset
@item
incoming wire transfer claimed by exchange not found
@item
wire subject does not match
@item
wire amount does not match
@item
debit account url does not match
@item
execution date mismatch
@item
closing fee above total amount
@end itemize
@node Testing the auditor,,Invariants checked by the auditor,Auditor implementation guide
@anchor{taler-auditor-manual testing-the-auditor}@anchor{42}
@section Testing the auditor
The main objective of the auditor is to detect inconsistencies. Thus, the
@code{test-auditor.sh} script deliberately introduces various inconsistencies into
a synthetic exchange database. For this, an “normal” exchange database is
first generated using the @code{taler-wallet-cli}. Then, various fields or rows
of that database are manipulated, and the auditor is let loose on the modified
database. Afterwards, the test verifies that the JSON contains values
indicating that the auditor found the inconsistencies. The script also
verifies that template expansion and LaTeX run work for the JSON output,
but it does not verify the correctness of the final PDF.
The @code{test-auditor.sh} script is written to maximize code coverage: it should
cover as many code paths as possible in both the exchange and the auditor. It
should also ideally create all interesting possible variations of the exchange
database fields (within the constraints of the database schema).
In general, @code{test-auditor.sh} runs the tests against an “old” database where
some transactions are past the due-date (and hence the aggregator would trigger
wire transfers), as well as a freshly generated exchange database where the
auditor would not perform any transfers. Auditor interactions can be made
before or after the aggregator, depending on what is being tested.
The current script also rudimentarily tests the auditor’s resume logic,
by re-starting the auditor once against a database that the auditor has
already seen.
The @code{test-revocation.sh} script performs tests related to the handling of
key revocations.
The @code{test-sync.sh} script performs tests related to the @code{taler-auditor-sync}
tool.
@c TODO
@c
@c More extensive auditor testing where additional transactions
@c have been made against the database when the audit is being resumed
@c should be done in the future.
@node Index,,Auditor implementation guide,Top
@unnumbered Index
@printindex ge
@c %**end of body
@bye