diff options
Diffstat (limited to 'texinfo/taler-auditor.texi')
| -rw-r--r-- | texinfo/taler-auditor.texi | 1136 |
1 files changed, 864 insertions, 272 deletions
diff --git a/texinfo/taler-auditor.texi b/texinfo/taler-auditor.texi index e1a0b384..92fee90d 100644 --- a/texinfo/taler-auditor.texi +++ b/texinfo/taler-auditor.texi @@ -3,29 +3,27 @@ @setfilename taler-auditor.info @documentencoding UTF-8 @ifinfo -@*Generated by Sphinx 3.4.3.@* +@*Generated by Sphinx 5.3.0.@* @end ifinfo @settitle Taler Auditor Manual @defindex ge @paragraphindent 0 @exampleindent 4 @finalout -@dircategory CATEGORY +@dircategory Network applications @direntry -* MENU ENTRY: (taler-auditor.info). DESCRIPTION +* GNU Taler Auditor: (taler-auditor.info). External audit for Taler Exchange operation @end direntry -@definfoenclose strong,`,' -@definfoenclose emph,`,' @c %**end of header @copying @quotation -GNU Taler 0.8.0pre0, Apr 28, 2021 +GNU Taler 0.9.4, Apr 12, 2024 GNU Taler team -Copyright @copyright{} 2014-2021 Taler Systems SA (GPLv3+ or GFDL 1.3+) +Copyright @copyright{} 2014-2024 Taler Systems SA (GPLv3+ or GFDL 1.3+) @end quotation @end copying @@ -50,17 +48,17 @@ Copyright @copyright{} 2014-2021 Taler Systems SA (GPLv3+ or GFDL 1.3+) @anchor{taler-auditor-manual doc}@anchor{0} @c This file is part of GNU TALER. @c -@c Copyright (C) 2019-2020 Taler Systems SA +@c Copyright (C) 2019-2021 Taler Systems SA @c @c TALER is free software; you can redistribute it and/or modify it under the -@c terms of the GNU General Public License as published by the Free Software +@c terms of the GNU Affero General Public License as published by the Free Software @c Foundation; either version 2.1, or (at your option) any later version. @c @c TALER is distributed in the hope that it will be useful, but WITHOUT ANY @c WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR -@c A PARTICULAR PURPOSE. See the GNU General Public License for more details. +@c A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. @c -@c You should have received a copy of the GNU General Public License along with +@c You should have received a copy of the GNU Affero General Public License along with @c TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> @c @c @author Christian Grothoff @@ -68,6 +66,7 @@ Copyright @copyright{} 2014-2021 Taler Systems SA (GPLv3+ or GFDL 1.3+) @menu * Introduction:: * Installation:: +* System setup:: * Configuration:: * Deployment:: * Operation:: @@ -88,15 +87,32 @@ Installation * Installing from source:: * Installing the GNU Taler binary packages on Debian:: +* Installing the GNU Taler binary packages on Ubuntu:: + +System setup + +* UNIX accounts:: +* Databases and users:: Configuration * Configuration format:: -* Using taler-config:: +* Initial configuration:: * Keys:: -* Serving:: +* Configuring the auditor’s REST endpoint:: * Bank account:: * Database:: +* Legal conditions for using the service:: +* Terms of Service:: +* Privacy Policy:: +* Legal policies directory layout:: +* Generating the Legal Terms:: +* Adding translations:: +* Updating legal documents:: + +Legal policies directory layout + +* Example:: Deployment @@ -121,7 +137,7 @@ Operation Auditor implementation guide -* The auditor's database:: +* The auditor’s database:: * Invariants checked by the auditor:: * Testing the auditor:: @@ -137,7 +153,7 @@ Invariants checked by the auditor @end menu @node Introduction,Installation,Top,Top -@anchor{taler-auditor-manual gnu-taler-auditor-operator-manual}@anchor{1}@anchor{taler-auditor-manual introduction}@anchor{2} +@anchor{taler-auditor-manual auditor-operator-manual}@anchor{1}@anchor{taler-auditor-manual introduction}@anchor{2} @chapter Introduction @@ -157,6 +173,21 @@ to become readable. @section About GNU Taler +@c This file is part of GNU TALER. +@c +@c Copyright (C) 2014-2023 Taler Systems SA +@c +@c TALER is free software; you can redistribute it and/or modify it under the +@c terms of the GNU Affero General Public License as published by the Free Software +@c Foundation; either version 2.1, or (at your option) any later version. +@c +@c TALER is distributed in the hope that it will be useful, but WITHOUT ANY +@c WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +@c A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. +@c +@c You should have received a copy of the GNU Affero General Public License along with +@c TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> + GNU Taler is an open protocol for an electronic payment system with a free software reference implementation. GNU Taler offers secure, fast and easy payment processing using well understood cryptographic @@ -189,21 +220,21 @@ to other parties. To perform this duty, you will need at least (read-only) access to the bank transactions of the exchange, as well as a continuously synchronized replica -of the exchange's database. The general assumption for running the auditor +of the exchange’s database. The general assumption for running the auditor is that this is done on a separate system controlled by the auditor. After all, the goal is to detect nerfarious activity of the exchange operator, which cannot be effectively done on a machine controlled by the exchange operator. -For this, every auditor needs to operate a Postgres database. The data +For this, every auditor needs to operate a PostgreSQL database. The data collected will include sensitive information about Taler users, including withdrawals made by consumers and income received by merchants. As a result, the auditor is expected to provide high confidentiality for the database. In general, the auditor does not have to offer high-availability: the exchange operator can continue operations without the auditor, and the auditor can -catch up with it later when the auditor's systems are restored. However, of +catch up with it later when the auditor’s systems are restored. However, of course any downtime would provide a window of opportunity for fraud and should -thus be minimized. Finally, the auditor's copy of the exchange's database can +thus be minimized. Finally, the auditor’s copy of the exchange’s database can be useful as a backup to the exchange in case the exchange experiences a loss of its own copies. Thus, business agreements between auditor and exchanges may include availability requirements as well. @@ -211,7 +242,7 @@ include availability requirements as well. Then, with the software provided, auditors can verify the cryptographic proofs collected by the exchange and detect if any improper bank transactions have been made. There are additional tasks which an auditor should perform. While this -manual only focuses on the audit of the exchange's database and wire transfers +manual only focuses on the audit of the exchange’s database and wire transfers with the existing tools, a proper auditor should also perform the following tasks: @@ -237,7 +268,7 @@ verification that the exchange properly implements the @code{/link} protocol @item verification that the exchange properly reports coins issued during the refresh protocol (by irregularly refreshing coins withdrawn by -the auditor and comparing against the exchange's database --- the +the auditor and comparing against the exchange’s database — the code required to support this is not yet implemented) @end itemize @@ -258,8 +289,8 @@ oversight function. Auditors should generally be independent third parties that verify that the exchange operates correctly. However, an exchange is likely to also run the auditing logic, as it is also used to calculate the exchange’s profits, risk -and liabilities. Furthermore, it's usually a good idea to not only rely on -third parties to verify one's own work. +and liabilities. Furthermore, it’s usually a good idea to not only rely on +third parties to verify one’s own work. The Taler software stack for an auditor consists of the following components: @@ -268,7 +299,7 @@ components: @itemize - @item -DBMS: Postgres +DBMS: PostgreSQL The auditor requires a DBMS to store a local copy of the transaction history for the Taler exchange, as well as for its own internal bookkeeping and checkpointing. @@ -278,7 +309,7 @@ fail to be imported due to constraint violations, this is an immediate serious concern that must be addressed manually. The software only verifies the content of a well-formed exchange database (well-formed with respect to SQL). For now, the GNU Taler reference implementation -only supports Postgres, but the code could be easily extended to +only supports PostgreSQL, but the code could be easily extended to support another DBMS. @item @@ -291,7 +322,7 @@ the auditor to detect if an exchange is underreporting deposits. In the future, the Web service should be extended to allow customers and merchants to automatically upload cryptographic proof of other violations of the specification by the exchange. However, for now it is assumed that -the respective cryptographic proofs are reported and verified manually --- +the respective cryptographic proofs are reported and verified manually — as with a well-behaved exchange this should obviously be a rare event. The main binary of this component is the @code{taler-auditor-httpd}. @@ -311,7 +342,7 @@ needs access to the wire gateway). The @code{taler-helper-auditor-wire} auditor verifies that the bank transactions performed by the exchange were done properly. This component must have access to the bank account -of the exchange, as well as to a copy of the exchange's database. +of the exchange, as well as to a copy of the exchange’s database. The @code{taler-auditor} script invokes the various helpers, each generating a JSON report. It then invokes the @code{taler-helper-auditor-render.py} @@ -326,7 +357,7 @@ of the report is required, as not every detail in the report is necessarily indicative of a problem. @end itemize -@node Installation,Configuration,Introduction,Top +@node Installation,System setup,Introduction,Top @anchor{taler-auditor-manual installation}@anchor{7} @chapter Installation @@ -334,6 +365,7 @@ indicative of a problem. @menu * Installing from source:: * Installing the GNU Taler binary packages on Debian:: +* Installing the GNU Taler binary packages on Ubuntu:: @end menu @@ -349,6 +381,18 @@ exchange compilation. @itemize - @item +Python3 module @code{jinja2} +@end itemize + + +@itemize - + +@item +“Sphinx RTD Theme” Python package aka @code{python3-sphinx-rtd-theme} +on Debian-based systems (for GNUnet documentation support, can be +omitted if GNUnet is configured with @code{--disable-documentation}) + +@item libsqlite3 >= 3.16.2 @item @@ -358,10 +402,10 @@ GNU libunistring >= 0.9.3 libcurl >= 7.26 (or libgnurl >= 7.26) @item -libqrencode >= 4.0.0 +libqrencode >= 4.0.0 (Taler merchant only) @item -GNU libgcrypt >= 1.6 +GNU libgcrypt >= 1.6 (1.10 or later highly recommended) @item libsodium >= 1.0 @@ -373,13 +417,40 @@ libargon2 >= 20171227 libjansson >= 2.7 @item -Postgres >= 9.6, including libpq +PostgreSQL >= 15, including libpq @item GNU libmicrohttpd >= 0.9.71 @item -GNUnet >= 0.14.0 (from source tarball@footnote{http://ftpmirror.gnu.org/gnunet/}) +GNUnet >= 0.20 (from source tarball@footnote{http://ftpmirror.gnu.org/gnunet/}) + +@item +Python3 with @code{jinja2} +@end itemize + +If you are on Debian stable or later, the following command may help you +install these dependencies: + +@example +# apt-get install \ + libqrencode-dev \ + libsqlite3-dev \ + libltdl-dev \ + libunistring-dev \ + libsodium-dev \ + libargon2-dev \ + libcurl4-gnutls-dev \ + libgcrypt20-dev \ + libjansson-dev \ + libpq-dev \ + libmicrohttpd-dev \ + python3-jinja2 \ + postgresql-15 +@end example + + +@itemize - @item GNU Taler exchange (from download directory@footnote{http://ftpmirror.gnu.org/taler/}, @@ -397,6 +468,12 @@ Before you install GNUnet, you must download and install the dependencies mentioned in the previous section, otherwise the build may succeed, but could fail to export some of the tooling required by GNU Taler. +On Ubuntu, you also need to install pkg-config, for example: + +@example +$ apt-get install pkg-config +@end example + To install GNUnet, unpack the tarball and change into the resulting directory, then proceed as follows: @@ -432,45 +509,82 @@ which requires you to run the last step as @code{root}. You have to specify @code{--with-gnunet=/usr/local} if you installed GNUnet to @code{/usr/local} in the previous step. -@node Installing the GNU Taler binary packages on Debian,,Installing from source,Installation +Please note that unlike most packages, if you want to run the @code{make check} +command, you should run it only `after' having done @code{make install}. The +latter ensures that necessary binaries are copied to the right place. + +In any case, if @code{make check} fails, please consider filing a +bug report with the Taler bug tracker@footnote{https://bugs.taler.net}. + +@node Installing the GNU Taler binary packages on Debian,Installing the GNU Taler binary packages on Ubuntu,Installing from source,Installation @anchor{taler-auditor-manual installing-the-gnu-taler-binary-packages-on-debian}@anchor{9} @section Installing the GNU Taler binary packages on Debian To install the GNU Taler Debian packages, first ensure that you have the right Debian distribution. At this time, the packages are built for -Sid, which means you should use a system which at least includes -unstable packages in its source list. We recommend using APT pinning -to limit unstable packages to those explicitly requested. To do this, -set your @code{/etc/apt/preferences} as follows: +Debian bookworm. + +You need to add a file to import the GNU Taler packages. Typically, +this is done by adding a file @code{/etc/apt/sources.list.d/taler.list} that +looks like this: @example -Package: * -Pin: release a=stable -Pin-Priority: 700 +deb [signed-by=/etc/apt/keyrings/taler-systems.gpg] https://deb.taler.net/apt/debian bookworm main +@end example -Package: * -Pin: release a=testing -Pin-Priority: 650 +Next, you must import the Taler Systems SA public package signing key +into your keyring and update the package lists: -Package: * -Pin: release a=unstable -Pin-Priority: 600 +@example +# wget -O /etc/apt/keyrings/taler-systems.gpg \ + https://taler.net/taler-systems.gpg +# apt update +@end example + +@cartouche +@quotation Note +You may want to verify the correctness of the Taler Systems SA key out-of-band. +@end quotation +@end cartouche + +Now your system is ready to install the official GNU Taler binary packages +using apt. + +To install the Taler auditor, you can now simply run: + +@example +# apt install -t sid taler-auditor +@end example + +For the auditor, you must manually configure access to the exchange database, +the HTTP reverse proxy (typically with TLS certificates) and offline signing. + +Sample configuration files for the HTTP reverse proxy can be found in +@code{/etc/taler-auditor/}. -Package: * -Pin: release l=Debian-Security -Pin-Priority: 1000 +@node Installing the GNU Taler binary packages on Ubuntu,,Installing the GNU Taler binary packages on Debian,Installation +@anchor{taler-auditor-manual installing-the-gnu-taler-binary-packages-on-ubuntu}@anchor{a} +@section Installing the GNU Taler binary packages on Ubuntu + + +To install the GNU Taler Ubuntu packages, first ensure that you have +the right Ubuntu distribution. At this time, the packages are built for +Ubuntu Lunar and Ubuntu Jammy. Make sure to have @code{universe} in your +@code{/etc/apt/sources.list} (after @code{main}) as we depend on some packages +from Ubuntu @code{universe}. + +A typical @code{/etc/apt/sources.list.d/taler.list} file for this setup +would look like this for Ubuntu Lunar: + +@example +deb [signed-by=/etc/apt/keyrings/taler-systems.gpg] https://deb.taler.net/apt/ubuntu/ lunar taler-lunar @end example -A typical @code{/etc/apt/sources.list} file for this setup -would look like this: +For Ubuntu Mantic use this instead: @example -deb http://ftp.ch.debian.org/debian/ buster main -deb http://security.debian.org/debian-security buster/updates main -deb http://ftp.ch.debian.org/debian/ testing main -deb http://ftp.ch.debian.org/debian/ unstable main -deb https://deb.taler.net/apt/debian sid main +deb [signed-by=/etc/apt/keyrings/taler-systems.gpg] https://deb.taler.net/apt/ubuntu/ mantic taler-mantic @end example The last line is crucial, as it adds the GNU Taler packages. @@ -479,7 +593,8 @@ Next, you must import the Taler Systems SA public package signing key into your keyring and update the package lists: @example -# wget -O - https://taler.net/static/taler-systems.gpg.key | apt-key add - +# wget -O /etc/apt/keyrings/taler-systems.gpg \ + https://taler.net/taler-systems.gpg # apt update @end example @@ -492,52 +607,183 @@ You may want to verify the correctness of the Taler Systems key out-of-band. Now your system is ready to install the official GNU Taler binary packages using apt. -To install the Taler auditor, you can now simply run: +To install the Taler exchange, you can now simply run: @example -# apt install taler-auditor +# apt install -t focal-fossa taler-auditor @end example For the auditor, you must manually configure access to the exchange database, the HTTP reverse proxy (typically with TLS certificates) and offline signing. Sample configuration files for the HTTP reverse proxy can be found in -@code{/etc/taler-exchange/}. +@code{/etc/taler-auditor/}. -@node Configuration,Deployment,Installation,Top -@anchor{taler-auditor-manual configuration}@anchor{a} +@node System setup,Configuration,Installation,Top +@anchor{taler-auditor-manual system-setup}@anchor{b} +@chapter System setup + + +@menu +* UNIX accounts:: +* Databases and users:: + +@end menu + +@node UNIX accounts,Databases and users,,System setup +@anchor{taler-auditor-manual unix-accounts}@anchor{c} +@section UNIX accounts + + +For maximum security, you should setup multiple different users (possibly +on different machines) to run Taler auditor components. While it is possible +to skip some of these entirely, or to run all of them as the same user, this +is not recommended for security. The recommended set of users includes: + +@quotation + + +@itemize * + +@item +auditor — runs the main auditing process and HTTP backend + +@item +sync — synchronizes the ingres database with the production database + +@item +helper — runs taler-auditor-offline download and upload commands + +@item +auditor-ingres — imports database from exchange production system + +@item +auditor-wire — imports wire transfer data from bank production system + +@item +offline — manages the offline key, on a separate `offline' machine +@end itemize +@end quotation + +It is suggested that you setup the first five users on the target system(s) +using: + +@example +# add-user --disabled-password $USERNAME +@end example + +Additionally, there are two canonical system users of relevance (which your +distribution would typically create for you): + +@quotation + + +@itemize * + +@item +www-data — runs the HTTPS frontend (usually nginx or Apache) + +@item +postgres — runs the PostgreSQL database +@end itemize +@end quotation + +@node Databases and users,,UNIX accounts,System setup +@anchor{taler-auditor-manual databases-and-users}@anchor{d} +@section Databases and users + + +We recommend using the following databases for the auditor: + +@quotation + + +@itemize * + +@item +exchange-ingres — synchronized exchange database over the network + +@item +exchange-production — local copy of exchange database with trusted schema + +@item +auditor — auditor production database with current state of the audit + +@item +libeufin — local state of the auditor-wire tool for the bank transfer data import +@end itemize +@end quotation + +As the `postgres' user, you can create these databases using: + +@example +# As the 'postgres' user: +$ createdb -O auditor-ingres exchange-ingres +$ createdb -O sync exchange-production +$ createdb -O auditor auditor +$ createdb -O auditor-wire libeufin +@end example + +This will ensure that the correct users have write-access to their +respective database. Next, you need to grant read-only access to +some users to databases owned by other users: + +@example +# As the 'auditor-ingres' user: +$ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO sync;' | psql exchange-ingres +# As the 'sync' user: +$ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO auditor;' | psql exchange-production +# As the 'auditor-wire' user: +$ echo 'GRANT SELECT ON ALL TABLES IN SCHEMA public TO auditor;' | psql libeufin +@end example + +@node Configuration,Deployment,System setup,Top +@anchor{taler-auditor-manual configuration}@anchor{e} @chapter Configuration -The auditor's configuration works the same way as the configuration of other +The auditor’s configuration works the same way as the configuration of other Taler components. This section discusses configuration options related to the auditor. @menu * Configuration format:: -* Using taler-config:: +* Initial configuration:: * Keys:: -* Serving:: +* Configuring the auditor’s REST endpoint:: * Bank account:: * Database:: +* Legal conditions for using the service:: +* Terms of Service:: +* Privacy Policy:: +* Legal policies directory layout:: +* Generating the Legal Terms:: +* Adding translations:: +* Updating legal documents:: @end menu -@node Configuration format,Using taler-config,,Configuration -@anchor{taler-auditor-manual configuration-format}@anchor{b} +@node Configuration format,Initial configuration,,Configuration +@anchor{taler-auditor-manual configuration-format}@anchor{f} @section Configuration format -In Taler realm, any component obeys to the same pattern to get -configuration values. According to this pattern, once the component has -been installed, the installation deploys default values in -$@{prefix@}/share/taler/config.d/, in .conf files. In order to override -these defaults, the user can write a custom .conf file and either pass -it to the component at execution time, or name it taler.conf and place -it under $HOME/.config/. +All GNU Taler components are designed to possibly share the same +configuration files. When installing a GNU Taler component, the +installation deploys default values in configuration files located +at $@{prefix@}/share/taler/config.d/ where $@{prefix@} is the installation +prefix. Different components must be installed to the same prefix. + +In order to override these defaults, the user can write a custom configuration +file and either pass it to the component at execution time using the `-c' +option, or name it taler.conf and place it under $HOME/.config/ which is where +components will look by default. Note that the systemd service files pass @code{-c +/etc/taler/taler.conf}, thus making @code{/etc/taler/taler.conf} +the primary location for the configuration. A config file is a text file containing sections, and each section -contains its values. The right format follows: +contains maps options to their values. Configuration files follow +basically the INI syntax: @example [section1] @@ -549,14 +795,23 @@ value21 = string value22 = /path22 @end example -Throughout any configuration file, it is possible to use @code{$}-prefixed -variables, like @code{$VAR}, especially when they represent filesystem -paths. It is also possible to provide defaults values for those +Comments start with a hash (@code{#}). Throughout the configuration, it is +possible to use @code{$}-substitution for options relating to names of files or +directories. It is also possible to provide defaults values for those variables that are unset, by using the following syntax: -@code{$@{VAR:-default@}}. However, there are two ways a user can set -@code{$}-prefixable variables: +@code{$@{VAR:-default@}}. There are two ways a user can set the value +of @code{$}-prefixable variables: + +@quotation -by defining them under a @code{[paths]} section, see example below, + +@enumerate + +@item +by defining them under a @code{[paths]} section: +@end enumerate + +@quotation @example [paths] @@ -565,100 +820,71 @@ TALER_DEPLOYMENT_SHARED = $@{HOME@}/shared-data [section-x] path-x = $@{TALER_DEPLOYMENT_SHARED@}/x @end example +@end quotation + + +@enumerate 2 +@item or by setting them in the environment: +@end enumerate + +@quotation @example $ export VAR=/x @end example +@end quotation +@end quotation The configuration loader will give precedence to variables set under -@code{[path]}, though. +@code{[path]} over environment variables. -The utility @code{taler-config}, which gets installed along with the -exchange, serves to get and set configuration values without directly -editing the .conf. The option @code{-f} is particularly useful to resolve +The utility @code{taler-config}, which gets installed along with the exchange, +can be used get and set configuration values without directly editing the +configuration file. The option @code{-f} is particularly useful to resolve pathnames, when they use several levels of @code{$}-expanded variables. See @code{taler-config --help}. -Note that, in this stage of development, the file -@code{$HOME/.config/taler.conf} can contain sections for @emph{all} the -component. For example, both an exchange and a bank can read values from -it. - -The repository @code{git://taler.net/deployment} contains examples of -configuration file used in our demos. See under @code{deployment/config}. - -@quotation - -@strong{Note} - -Expectably, some components will not work just by using default -values, as their work is often interdependent. For example, a -merchant needs to know an exchange URL, or a database name. -@end quotation - -@node Using taler-config,Keys,Configuration format,Configuration -@anchor{taler-auditor-manual using-taler-config}@anchor{c} -@section Using taler-config - - -The tool @code{taler-config} can be used to extract or manipulate -configuration values; however, the configuration use the well-known INI -file format and can also be edited by hand. - -Run - -@example -$ taler-config -s $SECTION -@end example - -to list all of the configuration values in section @code{$SECTION}. - -Run +The repository @code{git://git.taler.net/deployment} contains example code +for generating configuration files under @code{deployment/netzbon/}. -@example -$ taler-config -s $section -o $option -@end example +@node Initial configuration,Keys,Configuration format,Configuration +@anchor{taler-auditor-manual initial-configuration}@anchor{10}@anchor{taler-auditor-manual setupbaseurl}@anchor{11} +@section Initial configuration -to extract the respective configuration value for option @code{$option} in -section @code{$section}. -Finally, to change a setting, run +You need to tell the Taler auditor configuration where the +REST API of the auditor will be available to the public: @example -$ taler-config -s $section -o $option -V $value +# Both for the 'offline' *and* the 'auditor' user: +[auditor] +BASE_URL = https://auditor.example.com/ @end example -to set the respective configuration value to @code{$value}. Note that you -have to manually restart the Taler backend after you change the -configuration to make the new configuration go into effect. - -Some default options will use $-variables, such as @code{$DATADIR} within -their value. To expand the @code{$DATADIR} or other $-variables in the -configuration, pass the @code{-f} option to @code{taler-config}. For example, -compare: +The @code{helper} user that is used to download information from the exchange +needs to know details about the exchange. Similarly, the @code{offline} user +needs to check signatures signed with the exchange’s offline key. Hence, you +need to obtain the @code{MASTER_PUBLIC_KEY} from the exchange operator (they need +to run @code{taler-exchange-offline setup}) and the REST endpoint of the exchange +and configure these: @example -$ taler-config -s ACCOUNT-bank \ - -o WIRE_RESPONSE -$ taler-config -f -s ACCOUNT-bank \ - -o WIRE_RESPONSE +# As the 'helper' and 'offline' users: +[exchange] +BASE_URL = https://exchange.example.com/ +MASTER_PUBLIC_KEY = $SOMELONGBASE32VALUEHERE @end example -While the configuration file is typically located at -@code{$HOME/.config/taler.conf}, an alternative location can be specified -to @code{taler-merchant-httpd} and @code{taler-config} using the @code{-c} -option. - -@node Keys,Serving,Using taler-config,Configuration -@anchor{taler-auditor-manual auditorkeys}@anchor{d}@anchor{taler-auditor-manual keys}@anchor{e} +@node Keys,Configuring the auditor’s REST endpoint,Initial configuration,Configuration +@anchor{taler-auditor-manual auditorkeys}@anchor{12}@anchor{taler-auditor-manual keys}@anchor{13} @section Keys The auditor works with one signing key to certify that it is auditing -a particular exchange's denomination keys. This key can and should -be kept @emph{offline} (and backed up adequately). As with the exchange's +a particular exchange’s denomination keys. This key can and should +be kept `offline' (and backed up adequately). As with the exchange’s offline key, it is only used for a few cryptographic signatures and thus the respective code can be run on modest hardware, like a Raspberry Pi. @@ -670,15 +896,39 @@ The following values are to be configured in the section @code{[auditor]}: @item @code{AUDITOR_PRIV_FILE}: Path to the auditor’s private key file. +@end itemize + +Note that the default value here should be fine and there is no clear +need to change it. What you do need to do as the @code{offine} user +is to extract the public key: + +@example +# As the 'offline' user: +$ taler-auditor-offline setup +@end example + +This public key must then be provided in the configuration file +of the @code{auditor} user in the @code{[auditor]]} configuration section: + + +@itemize - @item @code{PUBLIC_KEY}: Public key of the auditor, in Base32 encoding. Set from value printed by @code{taler-auditor-offline setup}. @end itemize -@node Serving,Bank account,Keys,Configuration -@anchor{taler-auditor-manual auditorserving}@anchor{f}@anchor{taler-auditor-manual serving}@anchor{10} -@section Serving +You can set this configuration value using: + +@example +# As the 'auditor' and 'helper' users: +[auditor] +PUBLIC_KEY = $SOMELONGBASE32VALUEHERE +@end example + +@node Configuring the auditor’s REST endpoint,Bank account,Keys,Configuration +@anchor{taler-auditor-manual auditorserving}@anchor{14}@anchor{taler-auditor-manual configuring-the-auditor-s-rest-endpoint}@anchor{15} +@section Configuring the auditor’s REST endpoint The auditor can serve HTTP over both TCP and UNIX domain socket. @@ -704,17 +954,17 @@ HTTP over a UNIX domain socket for @code{unixpath} (i.e. 660 = @code{rw-rw----}). @end itemize -@node Bank account,Database,Serving,Configuration -@anchor{taler-auditor-manual auditorbank-account}@anchor{11}@anchor{taler-auditor-manual bank-account}@anchor{12} +@node Bank account,Database,Configuring the auditor’s REST endpoint,Configuration +@anchor{taler-auditor-manual auditorbank-account}@anchor{16}@anchor{taler-auditor-manual bank-account}@anchor{17} @section Bank account -Bank accounts for the auditor are configured in exactly the -same way as bank accounts for the exchange. See the exchange -documentation for details. +Bank accounts for the auditor (user @code{auditor-wire}) are configured in +exactly the same way as bank accounts for the exchange. See the exchange (and +LibEuFin) documentation for details. -@node Database,,Bank account,Configuration -@anchor{taler-auditor-manual auditordatabaseconfiguration}@anchor{13}@anchor{taler-auditor-manual database}@anchor{14} +@node Database,Legal conditions for using the service,Bank account,Configuration +@anchor{taler-auditor-manual auditordatabaseconfiguration}@anchor{18}@anchor{taler-auditor-manual database}@anchor{19} @section Database @@ -730,7 +980,7 @@ choosing the backend, it is mandatory to supply the connection string via an environment variable: @code{TALER_AUDITORDB_POSTGRES_CONFIG}. @item -via configuration option @code{CONFIG}, under section @code{[auditordb-BACKEND]}. +via configuration option @code{CONFIG}, under section @code{[auditordb-$BACKEND]}. For example, the demo exchange is configured as follows: @example @@ -747,14 +997,14 @@ CONFIG = postgres:///auditordemo If an exchange runs its own auditor, it may use the same database for the auditor and the exchange itself. -The @code{taler-auditor-dbinit} tool is used to initialize the auditor's +The @code{taler-auditor-dbinit} tool is used to initialize the auditor’s tables. After running this tool, the rights to CREATE or DROP tables are no longer required and should be removed. Both the @code{taler-auditor-httpd} and the @code{taler-auditor} (and its helpers) also need (read-only) access to a (recent, current, synchronized) copy of the -exchange's database. The configuration options are the same that are also -used when configuring the exchange' database: +exchange’s database. The configuration options are the same that are also +used when configuring the exchange’ database: @quotation @@ -769,20 +1019,260 @@ CONFIG = postgres:///exchangedemo @end example @end quotation +@node Legal conditions for using the service,Terms of Service,Database,Configuration +@anchor{taler-auditor-manual legal-conditions-for-using-the-service}@anchor{1a} +@section Legal conditions for using the service + + +@c This file is part of GNU TALER. +@c +@c Copyright (C) 2014-2023 Taler Systems SA +@c +@c TALER is free software; you can redistribute it and/or modify it under the +@c terms of the GNU Affero General Public License as published by the Free Software +@c Foundation; either version 2.1, or (at your option) any later version. +@c +@c TALER is distributed in the hope that it will be useful, but WITHOUT ANY +@c WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +@c A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. +@c +@c You should have received a copy of the GNU Affero General Public License along with +@c TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> +@c +@c @author Christian Grothoff + +The service has well-known API endpoints to return its legal conditions to the +user in various languages and various formats. This section describes how to +setup and configure the legal conditions. + +@node Terms of Service,Privacy Policy,Legal conditions for using the service,Configuration +@anchor{taler-auditor-manual terms-of-service}@anchor{1b} +@section Terms of Service + + +The service has an endpoint “/terms” to return the terms of service (in legal +language) of the service operator. Client software show these terms of +service to the user when the user is first interacting with the service. +Terms of service are optional for experimental deployments, if none are +configured, the service will return a simple statement saying that there are +no terms of service available. + +To configure the terms of service response, there are two options +in the configuration file for the service: + + +@itemize - + +@item +@code{TERMS_ETAG}: The current “Etag” to return for the terms of service. +This value must be changed whenever the terms of service are +updated. A common value to use would be a version number. +Note that if you change the @code{TERMS_ETAG}, you MUST also provide +the respective files in @code{TERMS_DIR} (see below). + +@item +@code{TERMS_DIR}: The directory that contains the terms of service. +The files in the directory must be readable to the service +process. +@end itemize + +@node Privacy Policy,Legal policies directory layout,Terms of Service,Configuration +@anchor{taler-auditor-manual privacy-policy}@anchor{1c} +@section Privacy Policy + + +The service has an endpoint “/pp” to return the terms privacy policy (in legal +language) of the service operator. Clients should show the privacy policy to +the user when the user explicitly asks for it, but it should not be shown by +default. Privacy policies are optional for experimental deployments, if none +are configured, the service will return a simple statement saying that there +is no privacy policy available. + +To configure the privacy policy response, there are two options +in the configuration file for the service: + + +@itemize - + +@item +@code{PRIVACY_ETAG}: The current “Etag” to return for the privacy policy. +This value must be changed whenever the privacy policy is +updated. A common value to use would be a version number. +Note that if you change the @code{PRIVACY_ETAG}, you MUST also provide +the respective files in @code{PRIVACY_DIR} (see below). + +@item +@code{PRIVACY_DIR}: The directory that contains the privacy policy. +The files in the directory must be readable to the service +process. +@end itemize + +@node Legal policies directory layout,Generating the Legal Terms,Privacy Policy,Configuration +@anchor{taler-auditor-manual legal-policies-directory-layout}@anchor{1d} +@section Legal policies directory layout + + +The @code{TERMS_DIR} and @code{PRIVACY_DIR} directory structures must follow a +particular layout. You may use the same directory for both the terms of +service and the privacy policy, as long as you use different ETAGs. Inside of +the directory, there should be sub-directories using two-letter language codes +like “en”, “de”, or “jp”. Each of these directories would then hold +translations of the current terms of service into the respective language. +Empty directories are permitted in case translations are not available. + +Then, inside each language directory, files with the name of the value set as +the @code{TERMS_ETAG} or @code{PRIVACY_ETAG} must be provided. The extension of each +of the files should be typical for the respective mime type. The set of +supported mime types is currently hard-coded in the service, and includes +“.epub”, “.html”, “.md”, “.pdf” and “.txt” files. If other files are present, +the service may show a warning on startup. + +@menu +* Example:: + +@end menu + +@node Example,,,Legal policies directory layout +@anchor{taler-auditor-manual example}@anchor{1e} +@subsection Example + + +A sample file structure for a @code{TERMS_ETAG} of “tos-v0” would be: + + +@itemize - + +@item +TERMS_DIR/en/tos-v0.txt + +@item +TERMS_DIR/en/tos-v0.html + +@item +TERMS_DIR/en/tos-v0.pdf + +@item +TERMS_DIR/en/tos-v0.epub + +@item +TERMS_DIR/en/tos-v0.md + +@item +TERMS_DIR/de/tos-v0.txt + +@item +TERMS_DIR/de/tos-v0.html + +@item +TERMS_DIR/de/tos-v0.pdf + +@item +TERMS_DIR/de/tos-v0.epub + +@item +TERMS_DIR/de/tos-v0.md +@end itemize + +If the user requests an HTML format with language preferences “fr” followed by +“en”, the service would return @code{TERMS_DIR/en/tos-v0.html} lacking a version in +French. + +@node Generating the Legal Terms,Adding translations,Legal policies directory layout,Configuration +@anchor{taler-auditor-manual generating-the-legal-terms}@anchor{1f} +@section Generating the Legal Terms + + +The @code{taler-terms-generator} script can be used to generate directories with +terms of service and privacy policies in multiple languages and all required +data formats from a single source file in @code{.rst} format and GNU gettext +translations in @code{.po} format. + +To use the tool, you need to first write your legal conditions in English in +reStructuredText (rst). You should find a templates in +@code{$PREFIX/share/terms/*.rst} where @code{$PREFIX} is the location where you +installed the service to. Whenever you make substantive changes to the legal +terms, you must use a fresh filename and change the respective @code{ETAG}. The +resulting file must be called @code{$ETAG.rst} and the first line of the file should be the title of the document. + +Once you have written the @code{$ETAG.rst} file in English, you can +generate the first set of outputs: + +@example +$ taler-terms-generator -i $ETAG +@end example + +Afterwards, you should find the terms in various formats for all configured +languages (initially only English) in @code{$PREFIX/share/terms/}. The generator +has a few options which are documented in its man page. + +@node Adding translations,Updating legal documents,Generating the Legal Terms,Configuration +@anchor{taler-auditor-manual adding-translations}@anchor{20} +@section Adding translations + + +Translations must be available in subdirectories +@code{locale/$LANGUAGE/LC_MESSAGES/$ETAG.po}. +To start translating, you first need to add a new +language: + +@example +$ taler-terms-generator -i $ETAG -l $LANGUAGE +@end example + +Here, @code{$LANGUAGE} should be a two-letter language +code like @code{de} or @code{fr}. The command will generate +a file @code{locale/$LANGUAGE/LC_MESSAGES/$ETAG.po} +which contains each English sentence or paragraph +in the original document and an initially empty +translation. Translators should update the @code{.po} +file. Afterwards, simply re-run + +@example +$ taler-terms-generator -i $ETAG +@end example + +to make the current translation(s) available to the +service. + +@cartouche +@quotation Note +You must restart the service whenever adding or updating legal documents or their translations. +@end quotation +@end cartouche + +@node Updating legal documents,,Adding translations,Configuration +@anchor{taler-auditor-manual updating-legal-documents}@anchor{21} +@section Updating legal documents + + +When making minor changes without legal implications, edit the @code{.rst} file, +then re-run the step to add a new language for each existing translation to +produce an updated @code{.po} file. Translate the sentences that have changed and +finally run the generator (without @code{-l}) on the ETAG (@code{-i $ETAG}) to +create the final files. + +When making major changes with legal implications, you should first rename (or +copy) the existing @code{.rst} file and the associated translation files to a new +unique name. Afterwards, make the major changes, update the @code{.po} files, +complete the translations and re-create the final files. Finally, do not +forget to update the @code{ETAG} configuration option to the new name and to +restart the service. + @node Deployment,Operation,Configuration,Top -@anchor{taler-auditor-manual auditordeployment}@anchor{15}@anchor{taler-auditor-manual deployment}@anchor{16} +@anchor{taler-auditor-manual auditordeployment}@anchor{22}@anchor{taler-auditor-manual deployment}@anchor{23} @chapter Deployment -@anchor{taler-auditor-manual wallets}@anchor{17} -Before GNU Taler wallets will happily interact with an exchange, -the respective auditor's public key (to be obtained via @code{taler-auditor-offline setup}) -must be added under the respective currency to the wallet. This -is usually expected to be hard-coded into the Taler wallet. +@anchor{taler-auditor-manual wallets}@anchor{24} +Before GNU Taler wallets will happily interact with an exchange, the +respective auditor’s public key (as obtained via @code{taler-auditor-offline +setup} from the @code{offline} user) must be added under the respective currency +to the wallet. This is usually expected to be hard-coded into the Taler +wallet. Users can also manually add auditors for a particular currency via a Web page offering the respective pairing. -FIXME-DOLD: explain how that Web page works, once it works... +FIXME-DOLD: explain how that Web page works, once it works… @menu * Exchange:: @@ -792,49 +1282,52 @@ FIXME-DOLD: explain how that Web page works, once it works... @end menu @node Exchange,Signing Denominations,,Deployment -@anchor{taler-auditor-manual auditorexchange}@anchor{18}@anchor{taler-auditor-manual exchange}@anchor{19} +@anchor{taler-auditor-manual auditorexchange}@anchor{25}@anchor{taler-auditor-manual exchange}@anchor{26} @section Exchange -The next step is to add the exchange's master public key and the base -URL of the exchange to the list of exchanges audited by the auditor. -This is done using the @code{taler-auditor-exchange} tool. The tool -basically creates the respective record in the auditor's database. +The next step is to add the exchange’s master public key and the base URL of +the exchange to the list of exchanges audited by the auditor. This is done +using the @code{taler-auditor-exchange} tool. The tool basically creates the +respective record in the auditor’s database. -If this step is skipped, the auditor will malfunction at all future -stages with a foreign key violation, as it doesn't know the exchange's -master public key. +If this step is skipped, the auditor will malfunction at all future stages +with a foreign key violation, as it does not know the exchange’s master public +key. @example +# As the 'auditor' user: $ taler-auditor-exchange -m $MASTER_PUB -u $EXCHANGE_BASE_URL @end example -The equivalent step must be performed by the exchange operator. -Here, the exchange operator must use the @code{taler-exchange-offline} -tool to add the auditor's public key, base URL and (business) name -to the list of approved auditors of the exchange. For details, -see Auditor-configuration in the exchange operator manual. +An equivalent step must be performed by the exchange operator. Here, the +exchange operator must use the @code{taler-exchange-offline} tool to add the +auditor’s public key, base URL and (business) name to the list of approved +auditors of the exchange. For details, see Auditor-configuration in the +exchange operator manual. @node Signing Denominations,Database<2>,Exchange,Deployment -@anchor{taler-auditor-manual signing-denominations}@anchor{1a}@anchor{taler-auditor-manual signingdenominations}@anchor{1b} +@anchor{taler-auditor-manual signing-denominations}@anchor{27}@anchor{taler-auditor-manual signingdenominations}@anchor{28} @section Signing Denominations @geindex maintenance -This step must be performed regularly whenever the exchange is +These steps must be performed `regularly' whenever the exchange is deploying new denomination keys. After the exchange operator has signed new keys using the @code{taler-exchange-offline} tool, each auditor should run: @example +# As the 'helper' user: $ taler-auditor-offline download > input.json @end example -to import the latest set of denomination keys. The key data -should then be inspected using: +to import the latest set of denomination keys. The key data should then be +copied to the offline system and there be inspected using: @example +# As the 'offline' user: $ taler-auditor-offline show < input.json @end example @@ -847,13 +1340,14 @@ process that is outside of the scope of this document. Note that the @code{input.json} does not contain any confidential data. However, signing the wrong keys would be fatal in that it may allow an illegitimate exchange to convince users that it is a trustworthy operator and subsequently -betray the user's trust that is anchored in the existence of a trustworthy +betray the user’s trust that is anchored in the existence of a trustworthy auditor. Given the verified JSON input, the auditor can then sign it (typically on its offline system) using: @example +# As the 'offline' user: $ taler-auditor-offline sign < input.json > output.json @end example @@ -861,6 +1355,7 @@ The resulting @code{output.json} should then be copied to an online system, and from there uploaded to the exchange using: @example +# As the 'helper' user: $ taler-auditor-offline upload < output.json @end example @@ -875,27 +1370,27 @@ Commands, like @code{taler-auditor-offline}, that support the @code{-l LOGFILE} command-line option, send logging output to standard error by default. @node Database<2>,,Signing Denominations,Deployment -@anchor{taler-auditor-manual auditordatabaseinitialization}@anchor{1c}@anchor{taler-auditor-manual id1}@anchor{1d} +@anchor{taler-auditor-manual auditordatabaseinitialization}@anchor{29}@anchor{taler-auditor-manual id1}@anchor{2a} @section Database The next key step for the auditor is to configure replication of the -@emph{exchange}'s database in-house. This should be performed in two steps +`exchange'’s database in-house. This should be performed in two steps as illustrated in the following figure: @image{taler-auditor-figures/replication,,,,png} -First, the exchange should use standard Postgres replication features to -enable the auditor to obtain a full copy of the exchange's database. -Second, the auditor should make a "trusted" local copy, ensuring that it +First, the exchange should use standard PostgreSQL replication features to +enable the auditor to obtain a full copy of the exchange’s database. +Second, the auditor should make a “trusted” local copy, ensuring that it never replicates malicious changes using @code{taler-auditor-sync}. Both of these steps are described in more detail below. We note that as a result of these steps, the auditor will have three databases: its own production primary database (as configured in -@code{auditordb-postgres}), its on production copy of the exchange's database -(@code{exchangedb-postgress}), and a third, untrusted "ingres" copy of the -exchange database. The untrusted database should run as a separate Postgres +@code{auditordb-postgres}), its on production copy of the exchange’s database +(@code{exchangedb-postgress}), and a third, untrusted “ingres” copy of the +exchange database. The untrusted database should run as a separate PostgreSQL instance and is only accessed via @code{taler-auditor-sync} and the replication mechanism driven by the exchange operator. @@ -906,11 +1401,15 @@ mechanism driven by the exchange operator. @end menu @node Ingres replication of the exchange production database,Safe replication of the ingres database into the auditor production database,,Database<2> -@anchor{taler-auditor-manual ingres-replication-of-the-exchange-production-database}@anchor{1e} +@anchor{taler-auditor-manual ingres-replication-of-the-exchange-production-database}@anchor{2b} @subsection Ingres replication of the exchange production database -The full copy can be obtained in various ways with Postgres. It is +Ingres operation should be done using the @code{auditor-ingres} user — or +depending on the setup parts of the operation may be done by the @code{postgres} +user directly. + +The full copy can be obtained in various ways with PostgreSQL. It is possible to use log shipping with streaming replication as described in @indicateurl{https://www.postgresql.org/docs/13/warm-standby.html}, or to use logical replication, as described in @@ -919,47 +1418,108 @@ that asynchronous replication should suffice. The resulting auditor database should be treated as read-only on the auditor side. The @code{taler-exchange-dbinit} tool can be used to setup the schema, or -the schema can be replicated using Postgres's standard mechanisms. The same +the schema can be replicated using PostgreSQL’s standard mechanisms. The same applies for schema upgrades: if logical replication is used (which does not replicate schema changes), @code{taler-exchange-dbinit} can be used to migrate -the schema(s) in both the ingres and production copies of the exchange's +the schema(s) in both the ingres and production copies of the exchange’s database as well. -For details, we refer to the Postgres manual. +On the exchange side, a database user must be created that has the right +to perform database replication. This is done using: + +@example +# As the 'postgres' user of the exchange: +$ createuser --replication egress +$ echo "ALTER ROLE egress WITH PASSWORD '$PASSWORD'; | psql +$ echo "CREATE PUBLICATION $NAME FOR ALL TABLES;" | psql taler-exchange +@end example + +The exchange must share the password of the publication with the auditor. A +good @code{$NAME} relates to the auditor’s business unit name. A secure tunnel +must be setup between the exchange and the auditor, for example using SSH or +Wireguard. + +It is also necessary to edit @code{main.cf} of the exchange and on the auditor +side to enable logical replication. If an exchange has multiple auditors, it +should setup multiple @code{egress} accounts. The exchange must ensure that +the following lines are in the @code{main.cf} PostgreSQL configuration (the port +may differ) to enable replication over the network: + +@example +listen_addresses='*' +port = 5432 +wal_level= logical +@end example + +Equally, the auditor must configure logical replication in the @code{main.cf} +PostgreSQL configuration: + +@example +wal_level= logical +@end example + +Next, the @code{postgres} user of the auditor’s system must first initialize the +local tables: + +@example +# Configure database: +[exchange] +DB = "postgres" +[exchangedb-postgres] +CONFIG = "postgres:///taler-ingress" +@end example + +@example +# As the 'ingress' user of the exchange: +$ taler-exchange-dbinit +@end example + +To complete the replication, the @code{postgres} user of the auditor’s +system must subscribe: + +@example +# As the 'postgres' user of the exchange: +$ createuser --replication egress +$ echo "ALTER ROLE egress WITH PASSWORD '$PASSWORD'; | psql +$ echo "CREATE PUBLICATION $NAME FOR ALL TABLES;" | psql taler-exchange +@end example + +For details, we refer to the PostgreSQL manual. @cartouche @quotation Note Depending on the replication method used, the exchange may perform unexpected changes to the schema or perform @code{UPDATE}, @code{DELETE} or @code{DROP} operations on the tables. Hence, the auditor cannot rely upon the -exchange's primary copy to respect schema constraints, especially as we +exchange’s primary copy to respect schema constraints, especially as we have to presume that the exchange could act maliciously. Furthermore, it -is unclear to what degree Postgres database replication mechanisms are +is unclear to what degree PostgreSQL database replication mechanisms are robust against a malicious master database. Thus, the auditor should -isolate its primary copy of the exchange database, including the Postgres +isolate its primary copy of the exchange database, including the PostgreSQL process, from its actual operational data. @end quotation @end cartouche @node Safe replication of the ingres database into the auditor production database,,Ingres replication of the exchange production database,Database<2> -@anchor{taler-auditor-manual safe-replication-of-the-ingres-database-into-the-auditor-production-database}@anchor{1f} +@anchor{taler-auditor-manual safe-replication-of-the-ingres-database-into-the-auditor-production-database}@anchor{2c} @subsection Safe replication of the ingres database into the auditor production database -Using @code{taler-auditor-sync}, the auditor should make a second "safe" copy of -the exchange's ingres database. @code{taler-auditor-sync} basically reads from one -exchange database and inserts all records found into a second exchange -database. If the source database violates invariants, the tool halts with an -error. This way, records violating invariants are never even copied, and in -particular schema changes and deletions or updates are not propagated into the -auditor's production database. +Using @code{taler-auditor-sync} as the @code{sync} user, the auditor should +make a second “safe” copy of the exchange’s ingres database. +@code{taler-auditor-sync} basically reads from one exchange database and inserts +all records found into a second exchange database. If the source database +violates invariants, the tool halts with an error. This way, records violating +invariants are never even copied, and in particular schema changes and +deletions or updates are not propagated into the auditor’s production +database. While @code{taler-auditor-sync} could in theory be run directly against the -exchange's production system, this is likely a bad idea due to the high +exchange’s production system, this is likely a bad idea due to the high latency from the network between auditor and exchange operator. Thus, we -recommend first making an "untrusted" ingress copy of the exchange's -production database using standard Postgres tooling, and then using -@code{taler-auditor-sync} to create a second "safe" copy. The "safe" copy used +recommend first making an “untrusted” ingress copy of the exchange’s +production database using standard PostgreSQL tooling, and then using +@code{taler-auditor-sync} to create a second “safe” copy. The “safe” copy used by the production system should also run under a different UID. Before @code{taler-auditor-sync} can be used, the target database must be @@ -970,17 +1530,42 @@ database, and a second with the options for accessing the destination database. In both cases, likely only the @code{[exchangedb]/CONFIG} option needs to be changed. +To run @code{taler-auditor-sync}, you must first configure two configuration +files that identify the source and destination databases: + +@example +# src.conf +[exchangedb] +CONFIG = "postgres:///auditor-ingres/" +@end example + +@example +# dst.conf +[exchangedb] +CONFIG = "postgres:///auditor/" +@end example + +Now you should be able to launch the synchronization process. You can run +the process via systemd in the background. For a first one-off test, you should +use the @code{-t} option which will cause the process to terminate once the two +databases are synchronized: + +@example +# As the 'sync' user: +$ taler-auditor-sync -s src.conf -d dst.cfg -t +@end example + When the exchange performs garbage collection to @code{DELETE} obsolete records, this change should be automatically replicated to the auditors untrusted -ingress database. However, as @code{taler-auditor-sync} tries to be "safe", -it will not replicate those deletions to the auditor's production database. +ingress database. However, as @code{taler-auditor-sync} tries to be “safe”, +it will not replicate those deletions to the auditor’s production database. Thus, it is necessary to (occasonally) run @code{taler-exchange-dbinit -g} on -the auditor's production database to garbage collect old data in the -auditor's production copy. We note that this does not have to be done +the auditor’s production database to garbage collect old data in the +auditor’s production copy. We note that this does not have to be done at the same time when the exchange runs its garbage collection. @node Operation,Auditor implementation guide,Deployment,Top -@anchor{taler-auditor-manual id2}@anchor{20}@anchor{taler-auditor-manual operation}@anchor{21} +@anchor{taler-auditor-manual id2}@anchor{2d}@anchor{taler-auditor-manual operation}@anchor{2e} @chapter Operation @@ -996,31 +1581,38 @@ at the same time when the exchange runs its garbage collection. @end menu @node Web service,Audit,,Operation -@anchor{taler-auditor-manual id3}@anchor{22}@anchor{taler-auditor-manual web-service}@anchor{23} +@anchor{taler-auditor-manual id3}@anchor{2f}@anchor{taler-auditor-manual web-service}@anchor{30} @section Web service The @code{taler-auditor-httpd} runs the required REST API for the auditor. The service must have @code{INSERT} (and @code{SELECT}) rights on the -@code{deposit_confirmations} table in the auditor's database. We expect that in +@code{deposit_confirmations} table in the auditor’s database. We expect that in future versions additional rights may be required. +For now, we recommend simply running the @code{taler-auditor-httpd} under the +@code{auditor} user. However, it is also possible (and might be more secure) to +create a separate user with more restrictive permissions for this purpose. + As the @code{taler-auditor-httpd} does not include HTTPS-support, it is advisable to run it behind a reverse proxy that offers TLS termination. @node Audit,Reading the report,Web service,Operation -@anchor{taler-auditor-manual audit}@anchor{24}@anchor{taler-auditor-manual id4}@anchor{25} +@anchor{taler-auditor-manual audit}@anchor{31}@anchor{taler-auditor-manual id4}@anchor{32} @section Audit -Performing an audit is done by invoking the @code{taler-auditor} shell script. +Performing an audit is done by invoking the @code{taler-auditor} shell script as +the @code{auditor} user. + The shell script invokes the various helper processes. For additional performance and security, one may want to run the various helpers individually and with the respective minimal set of access rights (only @code{taler-helper-auditor-wire} needs the credentials to query the bank for wire -transfers). The shell script combines the final JSON outputs of the various -helpers using the @code{taler-helper-auditor-render.py} script into the TeX -report. Regardless, the simplest way to obtain a report is to run: +transfers, alas if @code{auditor-wire} is used to talk to the bank, this issue is +already addressed). The shell script combines the final JSON outputs of the +various helpers using the @code{taler-helper-auditor-render.py} script into the +TeX report. Regardless, the simplest way to obtain a report is to run: @example $ taler-audit @@ -1041,11 +1633,11 @@ interactions with the bank (which may not even have the wire transfer records anymore), this is not recommended in a production setup. @node Reading the report,Database upgrades,Audit,Operation -@anchor{taler-auditor-manual reading-the-report}@anchor{26} +@anchor{taler-auditor-manual reading-the-report}@anchor{33} @section Reading the report -The auditor's report needs to be read carefully, as it includes +The auditor’s report needs to be read carefully, as it includes several categories of failures of different severity: @@ -1077,7 +1669,7 @@ Configuration issues (such was wire fees unavailable). @end itemize @node Database upgrades,Database reset,Reading the report,Operation -@anchor{taler-auditor-manual auditordatabaseupgrades}@anchor{27}@anchor{taler-auditor-manual database-upgrades}@anchor{28} +@anchor{taler-auditor-manual auditordatabaseupgrades}@anchor{34}@anchor{taler-auditor-manual database-upgrades}@anchor{35} @section Database upgrades @@ -1103,7 +1695,7 @@ halting the exchange business logic, @item allowing the replication and @code{taler-auditor-sync} to complete -(see also the @strong{-t} option of @code{taler-auditor-sync}) +(see also the `-t' option of @code{taler-auditor-sync}) @item completing a @code{taler-audit} run against the old schema @@ -1111,14 +1703,14 @@ completing a @code{taler-audit} run against the old schema @item migrating the exchange schema (@code{taler-exchange-dbinit}) of the master database, possibly the ingres database and the -auditor's production copy +auditor’s production copy @item migrating the auditor database (@code{taler-auditor-dbinit}) @item -resuming database replication between the exchange's master -database and the auditor's ingres copy +resuming database replication between the exchange’s master +database and the auditor’s ingres copy @item resuming @code{taler-auditor-sync} @@ -1132,7 +1724,7 @@ Regardless, the above is merely the general rule. Please review the specific release notes to ensure this procedure is correct for the specific upgrade. @node Database reset,Revocations,Database upgrades,Operation -@anchor{taler-auditor-manual database-reset}@anchor{29} +@anchor{taler-auditor-manual database-reset}@anchor{36} @section Database reset @@ -1143,14 +1735,14 @@ $ taler-auditor-dbinit -R @end example However, running this command will result in all data in the database being -@emph{lost}, including steps like enabling an exchange using +`lost', including steps like enabling an exchange using @code{taler-auditor-exchange}. Thus, doing so may result in significant commputation (and bandwidth consumption with the bank) when the auditor is next launched, as it will re-download and re-verify all historic transactions. Hence this should not be done in a production system. @node Revocations,Failures,Database reset,Operation -@anchor{taler-auditor-manual auditorrevocations}@anchor{2a}@anchor{taler-auditor-manual revocations}@anchor{2b} +@anchor{taler-auditor-manual auditorrevocations}@anchor{37}@anchor{taler-auditor-manual revocations}@anchor{38} @section Revocations @@ -1163,29 +1755,29 @@ For more information, see Revocations in the exchange operator manual. If all denominations of an exchange are revoked, the exchange includes logic to wire back all returned funds to the bank accounts from which they originate. If some denominations remain operational, wallets will generally -exchange old coins of revoked denominations for new coins -- while providing +exchange old coins of revoked denominations for new coins – while providing additional information to demonstrate that these coins were not forged from the compromised private key but obtained via a legitimate withdraw operation. @node Failures,,Revocations,Operation -@anchor{taler-auditor-manual failures}@anchor{2c} +@anchor{taler-auditor-manual failures}@anchor{39} @section Failures -Most audit failures are handled by the auditor's regular reporting functionality, +Most audit failures are handled by the auditor’s regular reporting functionality, creating a (hopefully descriptive) PDF report detailing the problems found. However, there is one category of errors where this is not possible, which -concerns arithmetic overflows for amounts. Taler's specification limits amount -values to at most 2^52. If, during the auditor's calculations, amounts are +concerns arithmetic overflows for amounts. Taler’s specification limits amount +values to at most 2^52. If, during the auditor’s calculations, amounts are encountered that exceed this threshold, the auditor will not generate a regular report, but instead write a log statement explaining where the problem happened -and exit with a status code of @emph{42}. +and exit with a status code of `42'. The most common expected case when this happens is a corrupted database. This could be because the exchange is actively malicious, or more likely due to some data corruption. The audit cannot continue until the corruption has been -addressed. If it is not possible to restore a fully @emph{correct} version of the +addressed. If it is not possible to restore a fully `correct' version of the database, the suggestion is to replace the corrupted (and likely very large) amounts with zero (Note: this does not apply to the value of denominations or fees, here it is crucial that the correct amounts are restored). While an @@ -1194,14 +1786,14 @@ calculations with zero instead. After patching the database, the audit can be restarted. A full reset is not required, as the audit transaction is aborted -when the auditor exits with code @emph{42}. After restarting, the resulting audit +when the auditor exits with code `42'. After restarting, the resulting audit report is likely to indicates errors relating to the corrupted fields (such as invalid signatures, arithmetic errors by the exchange, etc.), but at least the loss/gain calculations will be meaningful and actually indicative of the scope of the error created by the corrupted data. @node Auditor implementation guide,Index,Operation,Top -@anchor{taler-auditor-manual auditor-implementation-guide}@anchor{2d} +@anchor{taler-auditor-manual auditor-implementation-guide}@anchor{3a} @chapter Auditor implementation guide @@ -1209,15 +1801,15 @@ The auditor implementation is split into five main processes, called @code{taler-helper-auditor-XXX}. The split was done to realize the principle of least privilege and to enable independent logic to be possibly run in parallel. Only the taler-wire-auditor must have (read-only) access to the -exchange's bank account, the other components only need access to the +exchange’s bank account, the other components only need access to the database. All auditor subsystems basically start their audit from a certain transaction index (@code{BIG SERIAL}) in the auditor database which identifies where the last audit concluded. They then check that the transactions claimed in the -exchange's database match up internally, including the cryptographic +exchange’s database match up internally, including the cryptographic signatures and also with respect to amounts adding up. The auditor also -calculates the exchange's profits and expected bank balances. Once all +calculates the exchange’s profits and expected bank balances. Once all existing transactions are processed, the auditor processes store the current checkpoint in its database and generate a JSON report. @@ -1226,23 +1818,23 @@ uses Jinja2 with a TeX template to convert the five individual JSON reports into LaTeX and then into PDF. @menu -* The auditor's database:: +* The auditor’s database:: * Invariants checked by the auditor:: * Testing the auditor:: @end menu -@node The auditor's database,Invariants checked by the auditor,,Auditor implementation guide -@anchor{taler-auditor-manual the-auditor-s-database}@anchor{2e} -@section The auditor's database +@node The auditor’s database,Invariants checked by the auditor,,Auditor implementation guide +@anchor{taler-auditor-manual the-auditor-s-database}@anchor{3b} +@section The auditor’s database The database scheme used by the exchange looks as follows: @image{taler-auditor-figures/auditor-db,,,,png} -@node Invariants checked by the auditor,Testing the auditor,The auditor's database,Auditor implementation guide -@anchor{taler-auditor-manual invariants-checked-by-the-auditor}@anchor{2f} +@node Invariants checked by the auditor,Testing the auditor,The auditor’s database,Auditor implementation guide +@anchor{taler-auditor-manual invariants-checked-by-the-auditor}@anchor{3c} @section Invariants checked by the auditor @@ -1262,11 +1854,11 @@ pass where it might seem applicable. @end menu @node Invariants checked by the taler-helper-auditor-aggregation,Invariants checked by the taler-helper-auditor-coins,,Invariants checked by the auditor -@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-aggregation}@anchor{30} +@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-aggregation}@anchor{3d} @subsection Invariants checked by the taler-helper-auditor-aggregation -This is from CodeBlau's analysis. A proper write-up is pending. +This is from CodeBlau’s analysis. A proper write-up is pending. CodeBlau reports the following checks: @@ -1374,11 +1966,11 @@ wire fee unavailable for given time @end itemize @node Invariants checked by the taler-helper-auditor-coins,Invariants checked by the taler-helper-auditor-deposits,Invariants checked by the taler-helper-auditor-aggregation,Invariants checked by the auditor -@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-coins}@anchor{31} +@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-coins}@anchor{3e} @subsection Invariants checked by the taler-helper-auditor-coins -This is from CodeBlau's analysis. A proper write-up is pending. +This is from CodeBlau’s analysis. A proper write-up is pending. CodeBlau reports the following checks: @@ -1386,9 +1978,9 @@ CodeBlau reports the following checks: @item check that all denominations used by the exchange have been signed using -this auditor's key. All denominations encountered in the database that +this auditor’s key. All denominations encountered in the database that this auditor did not officially sign for are reported (but still included -in the audit as they obviously may impact the exchange's bank balance). +in the audit as they obviously may impact the exchange’s bank balance). Depending on the business situation, this may be normal (say if an exchange is changing auditors and newer denominations are no longer supported until their end-of-life by the current auditor). @@ -1470,7 +2062,7 @@ recoup, denomination not revoked @end itemize @node Invariants checked by the taler-helper-auditor-deposits,Invariants checked by the taler-helper-auditor-reserves,Invariants checked by the taler-helper-auditor-coins,Invariants checked by the auditor -@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-deposits}@anchor{32} +@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-deposits}@anchor{3f} @subsection Invariants checked by the taler-helper-auditor-deposits @@ -1480,11 +2072,11 @@ exchange at the auditor. This is to ensure that the exchange cannot defraud merchants by simply not reporting deposits to the auditor. @node Invariants checked by the taler-helper-auditor-reserves,Invariants checked by the taler-helper-auditor-wire,Invariants checked by the taler-helper-auditor-deposits,Invariants checked by the auditor -@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-reserves}@anchor{33} +@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-reserves}@anchor{40} @subsection Invariants checked by the taler-helper-auditor-reserves -This is from CodeBlau's analysis. A proper write-up is pending. +This is from CodeBlau’s analysis. A proper write-up is pending. CodeBlau reports the following checks: @@ -1541,16 +2133,16 @@ target account does not match origin account @end itemize @node Invariants checked by the taler-helper-auditor-wire,,Invariants checked by the taler-helper-auditor-reserves,Invariants checked by the auditor -@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-wire}@anchor{34} +@anchor{taler-auditor-manual invariants-checked-by-the-taler-helper-auditor-wire}@anchor{41} @subsection Invariants checked by the taler-helper-auditor-wire This auditor is special in that it is the only pass that is required to have -@emph{read-only} access to the exchange's bank account (privilege separation). Its -main role is to verify that the wire transfers in the exchange's database and +`read-only' access to the exchange’s bank account (privilege separation). Its +main role is to verify that the wire transfers in the exchange’s database and those reported by the bank are identical. -This is from CodeBlau's analysis. A proper write-up is pending. +This is from CodeBlau’s analysis. A proper write-up is pending. CodeBlau reports the following checks: @@ -1606,13 +2198,13 @@ closing fee above total amount @end itemize @node Testing the auditor,,Invariants checked by the auditor,Auditor implementation guide -@anchor{taler-auditor-manual testing-the-auditor}@anchor{35} +@anchor{taler-auditor-manual testing-the-auditor}@anchor{42} @section Testing the auditor The main objective of the auditor is to detect inconsistencies. Thus, the @code{test-auditor.sh} script deliberately introduces various inconsistencies into -a synthetic exchange database. For this, an "normal" exchange database is +a synthetic exchange database. For this, an “normal” exchange database is first generated using the @code{taler-wallet-cli}. Then, various fields or rows of that database are manipulated, and the auditor is let loose on the modified database. Afterwards, the test verifies that the JSON contains values @@ -1625,13 +2217,13 @@ cover as many code paths as possible in both the exchange and the auditor. It should also ideally create all interesting possible variations of the exchange database fields (within the constraints of the database schema). -In general, @code{test-auditor.sh} runs the tests against an "old" database where +In general, @code{test-auditor.sh} runs the tests against an “old” database where some transactions are past the due-date (and hence the aggregator would trigger wire transfers), as well as a freshly generated exchange database where the auditor would not perform any transfers. Auditor interactions can be made before or after the aggregator, depending on what is being tested. -The current script also rudimentarily tests the auditor's resume logic, +The current script also rudimentarily tests the auditor’s resume logic, by re-starting the auditor once against a database that the auditor has already seen. |