\input texinfo @c -*-texinfo-*- @c %**start of header @setfilename taler-exchange.info @documentencoding UTF-8 @ifinfo @*Generated by Sphinx 3.4.3.@* @end ifinfo @settitle Taler Exchange Manual @defindex ge @paragraphindent 0 @exampleindent 4 @finalout @dircategory CATEGORY @direntry * MENU ENTRY: (taler-exchange.info). DESCRIPTION @end direntry @definfoenclose strong,`,' @definfoenclose emph,`,' @c %**end of header @copying @quotation GNU Taler 0.9.0, Nov 03, 2022 GNU Taler team Copyright @copyright{} 2014-2022 Taler Systems SA (GPLv3+ or GFDL 1.3+) @end quotation @end copying @titlepage @title Taler Exchange Manual @insertcopying @end titlepage @contents @c %** start of user preamble @c %** end of user preamble @ifnottex @node Top @top Taler Exchange Manual @insertcopying @end ifnottex @c %**start of body @anchor{taler-exchange-manual doc}@anchor{0} @c This file is part of GNU TALER. @c @c Copyright (C) 2014-2020 Taler Systems SA @c @c TALER is free software; you can redistribute it and/or modify it under the @c terms of the GNU Affero General Public License as published by the Free Software @c Foundation; either version 2.1, or (at your option) any later version. @c @c TALER is distributed in the hope that it will be useful, but WITHOUT ANY @c WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR @c A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. @c @c You should have received a copy of the GNU Affero General Public License along with @c TALER; see the file COPYING. If not, see @c @c @author Christian Grothoff @menu * Introduction:: * Installation:: * Configuration: Configuration<2>. * Deployment:: * Testing a deployment:: * Diagnostics:: * Benchmarking:: * Index:: @detailmenu --- The Detailed Node Listing --- Introduction * About GNU Taler:: * About this manual:: * Organizational prerequisites:: * Architecture overview:: * Offline keys:: * Online signing key security:: Online signing key security * Functionality:: * Security goals:: * Setup:: * Configuration:: Installation * Installing from source:: * Installing the GNU Taler binary packages on Debian:: * Installing the GNU Taler binary packages on Trisquel:: * Installing the GNU Taler binary packages on Ubuntu:: Configuration * Configuration format:: * Using taler-config:: * Keying:: * Serving:: * Currency:: * Database:: * Coins (denomination keys): Coins denomination keys. * Sign keys:: * Terms of Service:: * Bank account:: * Auditor configuration:: Terms of Service * Example:: Bank account * Wire fee structure:: Deployment * Launching an exchange:: * Keys generation:: * Private key storage:: * Database upgrades:: Database upgrades * Revocations:: Diagnostics * Internal audits:: * Database Scheme:: @end detailmenu @end menu @node Introduction,Installation,Top,Top @anchor{taler-exchange-manual gnu-taler-exchange-operator-manual}@anchor{1}@anchor{taler-exchange-manual introduction}@anchor{2} @chapter Introduction This manual is an early draft that still needs significant editing work to become readable. @menu * About GNU Taler:: * About this manual:: * Organizational prerequisites:: * Architecture overview:: * Offline keys:: * Online signing key security:: @end menu @node About GNU Taler,About this manual,,Introduction @anchor{taler-exchange-manual about-gnu-taler}@anchor{3} @section About GNU Taler GNU Taler is an open protocol for an electronic payment system with a free software reference implementation. GNU Taler offers secure, fast and easy payment processing using well understood cryptographic techniques. GNU Taler allows customers to remain anonymous, while ensuring that merchants can be held accountable by governments. Hence, GNU Taler is compatible with anti-money-laundering (AML) and know-your-customer (KYC) regulation, as well as data protection regulation (such as GDPR). GNU Taler is not yet production-ready, after following this manual you will have a backend that can process payments in “KUDOS”, but not regular currencies. This is not so much because of limitations in the backend, but because we are not aware of a Taler exchange operator offering regular currencies today. @node About this manual,Organizational prerequisites,About GNU Taler,Introduction @anchor{taler-exchange-manual about-this-manual}@anchor{4} @section About this manual This tutorial targets system administrators who want to install and operate a GNU Taler exchange. @node Organizational prerequisites,Architecture overview,About this manual,Introduction @anchor{taler-exchange-manual organizational-prerequisites}@anchor{5} @section Organizational prerequisites Operating a GNU Taler exchange means that you are operating a payment service provider, which means that you will most likely need a bank license and/or follow applicable financial regulation. GNU Taler payment service providers generally need to ensure high availability and have @emph{really} good backups (synchronous replication, asynchronous remote replication, off-site backup, 24/7 monitoring, etc.). This manual will not cover these aspects of operating a payment service provider. We will assume that you can operate a (high-availability, high-assurance) PostgreSQL database. Furthermore, we expect some moderate familiarity with the compilation and installation of free software packages. You need to understand the cryptographic concepts of private and public keys and must be able to protect private keys stored in files on disk. @cartouche @quotation Note The Taler merchant backend stores private keys and other sensitive business and customer data in the database. The backend operator SHOULD ensure that backup operations are encrypted and secured from unauthorized access. @end quotation @end cartouche @node Architecture overview,Offline keys,Organizational prerequisites,Introduction @anchor{taler-exchange-manual architecture-overview}@anchor{6} @section Architecture overview Taler is a pure payment system, not a new crypto-currency. As such, it operates in a traditional banking context. In particular, this means that in order to receive funds via Taler, the merchant must have a regular bank account, and payments can be executed in ordinary currencies such as USD or EUR. Similarly, the Taler exchange must interact with a bank. The bank of the exchange holds the exchange’s funds in an escrow account. Note that, given the technical burden (XML-based communications, additional cryptography, and a vast variety of standards) due to interact with banks, the exchange uses an intermediary system to talk to its bank. Such intermediary system abstracts the native banking protocol by exposing the @emph{Taler Wire Gateway API}; this way, the exchange can conduct its banking operations in a simplified and JSON-based style. When customers wire money to the escrow account, the bank notifies the exchange about the incoming wire transfers. The exchange then creates a @emph{reserve} based on the subject of the wire transfer. The wallet which knows the secret key matching the wire transfer subject can then withdraw coins from the reserve, thereby draining it. The liability of the exchange against the reserve is thereby converted into a liability against digital coins issued by the exchange. When the customer later spends the coins at a merchant, and the merchant @emph{deposits} the coins at the exchange, the exchange first @emph{aggregates} the amount from multiple deposits from the same merchant and then instructs its bank to make a wire transfer to the merchant, thereby fulfilling its obligation and eliminating the liability. The exchange charges @emph{fees} for some or all of its operations to cover costs and possibly make a profit. @emph{Auditors} are third parties, for example financial regulators, that verify that the exchange operates correctly. The same software is also used to calculate the exchange’s profits, risk and liabilities by the accountants of the exchange. The Taler software stack for an exchange consists of the following components: @itemize - @item HTTP frontend The HTTP frontend interacts with Taler wallets and merchant backends. It is used to withdraw coins, deposit coins, refresh coins, issue refunds, map wire transfers to Taler transactions, inquire about the exchange’s bank account details, signing keys and fee structure. The binary is the @code{taler-exchange-httpd}. @item Crypto-Helpers The @code{taler-exchange-secmod-rsa} and @code{taler-exchange-secmod-eddsa} are two programs that are responsible for managing the exchange's online signing keys. They must run on the same machine as the @code{taler-exchange-httpd} as the HTTP frontend communicates with the crypto helpers using UNIX Domain Sockets. @item Aggregator The aggregator combines multiple deposits made by the same merchant and (eventually) triggers wire transfers for the aggregate amount. The merchant can control how quickly wire transfers are made. The exchange may charge a fee per wire transfer to discourage excessively frequent transfers. The binary is the @code{taler-exchange-aggregator}. @item Closer The @code{taler-exchange-closer} tool check that reserves are properly closed. If a customer wires funds to an exchange and then fails to withdraw them, the closer will (eventually) trigger a wire transfer that sends the customer's funds back to the originating wire account. @item Transfer The @code{taler-exchange-transfer} tool is responsible for actually executing the aggregated wire transfers. It is the only process that needs to have the credentials to execute outgoing wire transfers. The tool uses the Taler Wire Gateway API to execute wire transfers. This API is provided by the Taler Python Bank for stand-alone deployments (like those with @code{KUDOS}) and by LibEuFin. LibEuFin is an adapter which maps the Taler Wire REST API to traditional banking protocols like EBICS and FinTS. @item Wirewatch The @code{taler-exchange-wirewatch} tool is responsible for observing incoming wire transfers to the exchange. It needs to have the credentials to obtain a list of incoming wire transfers. The tool also uses the Taler Wire Gateway API to observe such incoming transfers. It is possible that observing incoming and making outgoing wire transfers is done via different bank accounts and/or credentials. @item Wire adapter A wire adapter is a component that enables exchange to talk to a bank. @enumerate @item The libtalerfakebank implements a bank with a wire adapter API inside of a testcase. @item For the demonstration Web site (or local currencies), the Python bank provides a bank that directly provides the wire adapter API. @item For production, libeufin's Nexus component implements a wire adapter towards the traditional SEPA banking system with IBAN accounts. @end enumerate The client-side wire adapter API is implemented in libtalerbank and is used by the transfer to execute wire transfers and for the auditor to query bank transaction histories. @item DBMS The exchange requires a DBMS to stores the transaction history for the Taler exchange and aggregator, and a (typically separate) DBMS for the Taler auditor. For now, the GNU Taler reference implementation only supports PostgreSQL, but the code could be easily extended to support another DBMS. .. index:: PostgreSQL @item Auditor The auditor verifies that the transactions performed by the exchange were done properly. It checks the various signatures, totals up the amounts and alerts the operator to any inconsistencies. It also computes the expected bank balance, revenue and risk exposure of the exchange operator. The main binary is the @code{taler-auditor}. Aside from the key setup procedures, the most critical setup for deploying an auditor is providing the auditor with an up-to-date copy of the database. @end itemize @node Offline keys,Online signing key security,Architecture overview,Introduction @anchor{taler-exchange-manual offline-keys}@anchor{7} @section Offline keys The exchange (and ideally also auditors) uses a long-term offline master siging key that identifies the operator and is used to authenticate critical information, such as the exchange's bank account and the actual keys the exchange uses online. Interactions with the offline system are performed using the @code{taler-exchange-offline} tool. To use the offline system will require exchange operators to copy JSON files from or to the offline system (say using an USB stick). The offline system does not need any significant amount of computing power, a Raspberry-Pi is perfectly sufficient and the form-factor might be good for safe-keeping! (You should keep a copy of the (encrypted) private offline key on more than one physical medium though.) Exchange operators are strongly advised to secure your private master key and any copies on encrypted, always-offline computers. Again, we assume that you are familiar with good best practices in operational security, including securing key material. @node Online signing key security,,Offline keys,Introduction @anchor{taler-exchange-manual online-signing-key-security}@anchor{8} @section Online signing key security To provide an additional level of protection for the private @emph{online} signing keys used by the exchange, the actual cryptographic signing operations are performed by two helper processes, the @code{taler-exchange-secmod-rsa} and the @code{taler-exchange-secmod-eddsa}. The current implementation does not yet support the use of a hardware security module (HSM). If you have such a device with adequate functionality and are interested in Taler supporting it, please contact the developers for HSM integration support. @menu * Functionality:: * Security goals:: * Setup:: * Configuration:: @end menu @node Functionality,Security goals,,Online signing key security @anchor{taler-exchange-manual functionality}@anchor{9} @subsection Functionality The UNIX domain sockets have mode 0620 (u+rw, g+w). The exchange process MUST be in the same group as the crypto helper processes. The two helper processes will create the required private keys, and allow anyone with access to the UNIX domain socket to sign arbitrary messages with the keys or to inform them about a key being revoked. The helper processes are also responsible for deleting the private keys if their validity period expires or if they are informed about a key having been revoked. @node Security goals,Setup,Functionality,Online signing key security @anchor{taler-exchange-manual security-goals}@anchor{a} @subsection Security goals From a security point of view, the helpers are designed to @emph{only} make it harder for an attacker who took control of the HTTP daemon's account to extract the private keys, limiting the attackers ability to creating signatures to the duration of their control of that account. @cartouche @quotation Note In the future, the helper processes should additionally provide a mechanism to track the total number of signatures they have made for the various keys. @end quotation @end cartouche @node Setup,Configuration,Security goals,Online signing key security @anchor{taler-exchange-manual setup}@anchor{b} @subsection Setup The helper processes should be run under a user ID that is separate from that of the user running the main @code{taler-exchange-httpd} service. For security, it is important that helpers run under a different user ID than the main HTTP frontend, in fact ideally each helper should run under its own user ID. The @code{taler-exchange-httpd} service's will securely communicate with the helpers using UNIX domain sockets. To enable access to the keys, the service's user must be in the group of the helper processes (and no other users should be in that group). @node Configuration,,Setup,Online signing key security @anchor{taler-exchange-manual configuration}@anchor{c} @subsection Configuration The helpers and the HTTP service need both access to the same configuration information. Having divergent configurations may result in run-time failures. It is recommended that the configuration file (@code{-c} option) is simply shared between all of the different processes, even though they run as different system users. The configuration does not contain any sensitive information. @node Installation,Configuration<2>,Introduction,Top @anchor{taler-exchange-manual installation}@anchor{d} @chapter Installation Before installing a Taler exchange, please make sure that your system does not have swap space enabled. Swap space is a security risk that Taler does not try to mitigate against. Please install the following packages before proceeding with the exchange compilation. @itemize - @item Python3 module @code{jinja2} @end itemize @itemize - @item "Sphinx RTD Theme" Python package aka @code{python3-sphinx-rtd-theme} on Debian-based systems (for GNUnet documentation support, can be omitted if GNUnet is configured with @code{--disable-documentation}) @item libsqlite3 >= 3.16.2 @item GNU libunistring >= 0.9.3 @item libcurl >= 7.26 (or libgnurl >= 7.26) @item libqrencode >= 4.0.0 @item GNU libgcrypt >= 1.6 @item libsodium >= 1.0 @item libargon2 >= 20171227 @item libjansson >= 2.7 @item PostgreSQL >= 13, including libpq @item GNU libmicrohttpd >= 0.9.71 @item GNUnet >= 0.16 (from source tarball@footnote{http://ftpmirror.gnu.org/gnunet/}) @item GNU Taler exchange (from download directory@footnote{http://ftpmirror.gnu.org/taler/}, see release announcement@footnote{https://mail.gnu.org/archive/cgi-bin/namazu.cgi?query=taler&idxname=info-gnu&max=20&result=normal&sort=date:late}) @end itemize Except for the last two, these are available in most GNU/Linux distributions and should just be installed using the respective package manager. @menu * Installing from source:: * Installing the GNU Taler binary packages on Debian:: * Installing the GNU Taler binary packages on Trisquel:: * Installing the GNU Taler binary packages on Ubuntu:: @end menu @node Installing from source,Installing the GNU Taler binary packages on Debian,,Installation @anchor{taler-exchange-manual installing-from-source}@anchor{e} @section Installing from source The following instructions will show how to install libgnunetutil and the GNU Taler exchange from source. Before you install GNUnet, you must download and install the dependencies mentioned in the previous section, otherwise the build may succeed, but could fail to export some of the tooling required by GNU Taler. To install GNUnet, unpack the tarball and change into the resulting directory, then proceed as follows: @example $ ./configure [--prefix=GNUNETPFX] $ # Each dependency can be fetched from non standard locations via $ # the '--with-' option. See './configure --help'. $ make # make install # ldconfig @end example If you did not specify a prefix, GNUnet will install to @code{/usr/local}, which requires you to run the last step as @code{root}. The @code{ldconfig} command (also run as @code{root}) makes the shared object libraries (@code{.so} files) visible to the various installed programs. After installing GNUnet, unpack the GNU Taler exchange tarball, change into the resulting directory, and proceed as follows: @example $ ./configure [--prefix=EXCHANGEPFX] \ [--with-gnunet=GNUNETPFX] $ # Each dependency can be fetched from non standard locations via $ # the '--with-' option. See './configure --help'. $ make # make install @end example If you did not specify a prefix, the exchange will install to @code{/usr/local}, which requires you to run the last step as @code{root}. You have to specify @code{--with-gnunet=/usr/local} if you installed GNUnet to @code{/usr/local} in the previous step. Please note that unlike most packages, if you want to run the @code{make check} command, you should run it only @emph{after} having done @code{make install}. The latter ensures that necessary binaries are copied to the right place. Gratuitous editorial note by TTN: I think this is a quirk that we should fix in the long-term as such weirdness might hide other build issues. However, this is probably a minority viewpoint. In any case, if @code{make check} fails, please consider filing a bug report with the Taler bug tracker@footnote{https://bugs.taler.net}. @node Installing the GNU Taler binary packages on Debian,Installing the GNU Taler binary packages on Trisquel,Installing from source,Installation @anchor{taler-exchange-manual installing-the-gnu-taler-binary-packages-on-debian}@anchor{f} @section Installing the GNU Taler binary packages on Debian To install the GNU Taler Debian packages, first ensure that you have the right Debian distribution. At this time, the packages are built for Bullseye. You need to add a file to import the GNU Taler packages. Typically, this is done by adding a file @code{/etc/apt/sources.list.d/taler.list} that looks like this: @example deb https://deb.taler.net/apt/debian bullseye main @end example Next, you must import the Taler Systems SA public package signing key into your keyring and update the package lists: @example # wget -O - https://taler.net/taler-systems.gpg.key | apt-key add - # apt update @end example @cartouche @quotation Note You may want to verify the correctness of the Taler Systems key out-of-band. @end quotation @end cartouche Now your system is ready to install the official GNU Taler binary packages using apt. To install the Taler exchange, you can now simply run: @example # apt install -t sid taler-exchange @end example Note that the package does not perform any configuration work except for setting up the various users and the systemd service scripts. You still must configure at least the database, HTTP reverse proxy (typically with TLS certificates), denomination and fee structure, bank account, auditor(s), offline signing and the terms of service. Sample configuration files for the HTTP reverse proxy can be found in @code{/etc/taler-exchange/}. @node Installing the GNU Taler binary packages on Trisquel,Installing the GNU Taler binary packages on Ubuntu,Installing the GNU Taler binary packages on Debian,Installation @anchor{taler-exchange-manual installing-the-gnu-taler-binary-packages-on-trisquel}@anchor{10} @section Installing the GNU Taler binary packages on Trisquel To install the GNU Taler Trisquel packages, first ensure that you have the right Trisquel distribution. Packages are currently available for Trisquel GNU/Linux 10.0. Simply follow the same instructions provided for Ubuntu 20.04 LTS (Focal Fossa). @node Installing the GNU Taler binary packages on Ubuntu,,Installing the GNU Taler binary packages on Trisquel,Installation @anchor{taler-exchange-manual installing-the-gnu-taler-binary-packages-on-ubuntu}@anchor{11} @section Installing the GNU Taler binary packages on Ubuntu To install the GNU Taler Ubuntu packages, first ensure that you have the right Ubuntu distribution. At this time, the packages are built for Ubuntu 22.04 LTS (Jammy Jellyfish). A typical @code{/etc/apt/sources.list.d/taler.list} file for this setup would look like this: @example deb https://deb.taler.net/apt/ubuntu/ jammy main @end example The last line is crucial, as it adds the GNU Taler packages. Next, you must import the Taler Systems SA public package signing key into your keyring and update the package lists: @example # wget -O /etc/apt/trusted.gpg.d/taler-systems.asc \ https://taler.net/taler-systems.gpg.key # apt update @end example @cartouche @quotation Note You may want to verify the correctness of the Taler Systems key out-of-band. @end quotation @end cartouche Now your system is ready to install the official GNU Taler binary packages using apt. To install the Taler exchange, you can now simply run: @example # apt install -t focal-fossa taler-exchange @end example Note that the package does not perform any configuration work except for setting up the various users and the systemd service scripts. You still must configure at least the database, HTTP reverse proxy (typically with TLS certificates), denomination and fee structure, bank account, auditor(s), offline signing and the terms of service. Sample configuration files for the HTTP reverse proxy can be found in @code{/etc/taler-exchange/}. @node Configuration<2>,Deployment,Installation,Top @anchor{taler-exchange-manual id1}@anchor{12} @chapter Configuration This chapter provides an overview of the exchange configuration. Or at least eventually will do so, for now it is a somewhat wild description of some of the options. @menu * Configuration format:: * Using taler-config:: * Keying:: * Serving:: * Currency:: * Database:: * Coins (denomination keys): Coins denomination keys. * Sign keys:: * Terms of Service:: * Bank account:: * Auditor configuration:: @end menu @node Configuration format,Using taler-config,,Configuration<2> @anchor{taler-exchange-manual configuration-format}@anchor{13} @section Configuration format In Taler realm, any component obeys to the same pattern to get configuration values. According to this pattern, once the component has been installed, the installation deploys default values in $@{prefix@}/share/taler/config.d/, in .conf files. In order to override these defaults, the user can write a custom .conf file and either pass it to the component at execution time, or name it taler.conf and place it under $HOME/.config/. A config file is a text file containing sections, and each section contains its values. The right format follows: @example [section1] value1 = string value2 = 23 [section2] value21 = string value22 = /path22 @end example Throughout any configuration file, it is possible to use @code{$}-prefixed variables, like @code{$VAR}, especially when they represent filesystem paths. It is also possible to provide defaults values for those variables that are unset, by using the following syntax: @code{$@{VAR:-default@}}. However, there are two ways a user can set @code{$}-prefixable variables: by defining them under a @code{[paths]} section, see example below, @example [paths] TALER_DEPLOYMENT_SHARED = $@{HOME@}/shared-data .. [section-x] path-x = $@{TALER_DEPLOYMENT_SHARED@}/x @end example or by setting them in the environment: @example $ export VAR=/x @end example The configuration loader will give precedence to variables set under @code{[path]}, though. The utility @code{taler-config}, which gets installed along with the exchange, serves to get and set configuration values without directly editing the .conf. The option @code{-f} is particularly useful to resolve pathnames, when they use several levels of @code{$}-expanded variables. See @code{taler-config --help}. Note that, in this stage of development, the file @code{$HOME/.config/taler.conf} can contain sections for @emph{all} the component. For example, both an exchange and a bank can read values from it. The repository @code{git://taler.net/deployment} contains examples of configuration file used in our demos. See under @code{deployment/config}. @quotation @strong{Note} Expectably, some components will not work just by using default values, as their work is often interdependent. For example, a merchant needs to know an exchange URL, or a database name. @end quotation @node Using taler-config,Keying,Configuration format,Configuration<2> @anchor{taler-exchange-manual using-taler-002dconfig-exchange}@anchor{14}@anchor{taler-exchange-manual using-taler-config}@anchor{15} @section Using taler-config The tool @code{taler-config} can be used to extract or manipulate configuration values; however, the configuration use the well-known INI file format and can also be edited by hand. Run @example $ taler-config -s $SECTION @end example to list all of the configuration values in section @code{$SECTION}. Run @example $ taler-config -s $section -o $option @end example to extract the respective configuration value for option @code{$option} in section @code{$section}. Finally, to change a setting, run @example $ taler-config -s $section -o $option -V $value @end example to set the respective configuration value to @code{$value}. Note that you have to manually restart the Taler backend after you change the configuration to make the new configuration go into effect. Some default options will use $-variables, such as @code{$DATADIR} within their value. To expand the @code{$DATADIR} or other $-variables in the configuration, pass the @code{-f} option to @code{taler-config}. For example, compare: @example $ taler-config -s ACCOUNT-bank \ -o WIRE_RESPONSE $ taler-config -f -s ACCOUNT-bank \ -o WIRE_RESPONSE @end example While the configuration file is typically located at @code{$HOME/.config/taler.conf}, an alternative location can be specified to @code{taler-merchant-httpd} and @code{taler-config} using the @code{-c} option. @node Keying,Serving,Using taler-config,Configuration<2> @anchor{taler-exchange-manual id2}@anchor{16}@anchor{taler-exchange-manual keying}@anchor{17} @section Keying The exchange works with four types of keys: @itemize - @item master key (kept offline) To create a master key, use: @example $ taler-exchange-offline setup @end example @item sign keys (signs normal messages from the exchange) @item denomination keys (signs electronic coins, see section Coins) @item security module keys (signs sign keys and denomination keys) @end itemize Additionally, the exchange is sometimes concerned with the auditor's public key (to verify messages signed by auditors approved by the exchange operator) and the merchant's public key (to verify refunds are authorized by the merchant). Key options include: @itemize - @item @code{[exchange-offline/MASTER_PRIV_FILE]}: Path to the exchange’s master private file. Only needs to be provided on the offline system where the @code{taler-exchange-offline} command is used. @item @code{[exchange/MASTER_PUBLIC_KEY]}: Must specify the exchange’s master public key. Needed for the exchange to verify information signed by the offline system. @end itemize @node Serving,Currency,Keying,Configuration<2> @anchor{taler-exchange-manual id3}@anchor{18}@anchor{taler-exchange-manual serving}@anchor{19} @section Serving The exchange can serve HTTP over both TCP and UNIX domain socket. The following options are to be configured in the section @code{[exchange]}: @itemize - @item @code{SERVE}: Must be set to @code{tcp} to serve HTTP over TCP, or @code{unix} to serve HTTP over a UNIX domain socket. @item @code{PORT}: Set to the TCP port to listen on if @code{SERVE} is @code{tcp}. @item @code{UNIXPATH}: Set to the UNIX domain socket path to listen on if @code{SERVE} is @code{unix}. @item @code{UNIXPATH_MODE}: Number giving the mode with the access permission mask for the @code{UNIXPATH} (i.e. 660 = @code{rw-rw---}). @end itemize @node Currency,Database,Serving,Configuration<2> @anchor{taler-exchange-manual currency}@anchor{1a}@anchor{taler-exchange-manual id4}@anchor{1b} @section Currency The exchange supports only one currency. This data is set under the respective option @code{CURRENCY} in section @code{[taler]}. @node Database,Coins denomination keys,Currency,Configuration<2> @anchor{taler-exchange-manual database}@anchor{1c}@anchor{taler-exchange-manual id5}@anchor{1d} @section Database The option @code{DB} in section @code{[exchange]} gets the database backend’s name the exchange is going to use. So far, only @code{db = postgres} is supported. After choosing the backend, it is mandatory to supply the connection string (namely, the database name). This is possible in two ways: @itemize - @item via an environment variable: @code{TALER_EXCHANGEDB_POSTGRES_CONFIG}. @item via configuration option @code{CONFIG}, under section @code{[exchangedb-$BACKEND]}. For example, the demo exchange is configured as follows: @end itemize @example [exchange] ... DB = postgres ... [exchangedb-postgres] CONFIG = postgres:///talerdemo @end example Given this database configuration, the database can be initialized using: @example $ taler-exchange-dbinit @end example Note that to run this command, the user must have @code{CREATE TABLE}, @code{CREATE INDEX}, @code{ALTER TABLE} and (in the future possibly even) @code{DROP TABLE} permissions. Those permissions are only required for this step (which may have to be repeated when upgrading a deployment). Afterwards, during normal operation, permissions to @code{CREATE} or @code{ALTER} tables are not required by any of the Taler exchange processes and thus should not be granted. For more information, see manpages/taler-exchange-dbinit.1. Commands, like @code{taler-exchange-dbinit}, that support the @code{-l LOGFILE} command-line option, send logging output to standard error by default. @node Coins denomination keys,Sign keys,Database,Configuration<2> @anchor{taler-exchange-manual coins-denomination-keys}@anchor{1e}@anchor{taler-exchange-manual id6}@anchor{1f} @section Coins (denomination keys) Sections specifying denomination (coin) information start with @code{coin_}. By convention, the name continues with @code{$CURRENCY_[$SUBUNIT]_$VALUE_$REVISION}, i.e. @code{[coin_eur_ct_10_0]} for a 10 cent piece. However, only the @code{coin_} prefix is mandatory. Once configured, these configuration values must not change. The @code{$REVISION} part of the section name should be incremented if any of the coin attributes in the section changes. Each @code{coin_}-section must then have the following options: @itemize - @item @code{VALUE}: How much is the coin worth, the format is CURRENCY:VALUE.FRACTION. For example, a 10 cent piece is "EUR:0.10". @item @code{DURATION_WITHDRAW}: How long can a coin of this type be withdrawn? This limits the losses incurred by the exchange when a denomination key is compromised. @item @code{DURATION_SPEND}: How long is a coin of the given type valid? Smaller values result in lower storage costs for the exchange. @item @code{DURATION_LEGAL}: How long is the coin of the given type legal? @item @code{FEE_WITHDRAW}: What does it cost to withdraw this coin? Specified using the same format as value. @item @code{FEE_DEPOSIT}: What does it cost to deposit this coin? Specified using the same format as value. @item @code{FEE_REFRESH}: What does it cost to refresh this coin? Specified using the same format as value. @item @code{FEE_REFUND}: What does it cost to refund this coin? Specified using the same format as value. @item @code{RSA_KEYSIZE}: How many bits should the RSA modulus (product of the two primes) have for this type of coin. @end itemize See manpages/taler.conf.5 for information on @emph{duration} values (i.e. @code{DURATION_WITHDRAW} and @code{DURATION_SPEND} above, and @code{OVERLAP_DURATION} and @code{DURATION} below). Additionally, there are two global configuration options of note: @itemize - @item @code{[taler-exchange-secmod-rsa/OVERLAP_DURATION]}: What is the overlap of the withdrawal timespan for denomination keys? The value given here must be smaller than any of the @code{DURATION_WITHDRAW} values for any of the coins. @item @code{[taler-exchange-secmod-rsa/LOOKAHEAD_SIGN]}: For how far into the future should denomination keys be pre-generated? This allows the exchange and auditor operators to download, offline-sign, and upload denomination key signatures for denomination keys that will be used in the future by the exchange. @end itemize @geindex maintenance @cartouche @quotation Note We recommend setting the @code{LOOKAHEAD_SIGN} value to at least one year and then to perform the offline-signing procedure at least once every 6 months to ensure that there is sufficient time for wallets to learn the new keys and to avoid unavailability in case this critical maintenance procedure is delayed. @end quotation @end cartouche @cartouche @quotation Note It is crucial that the configuration provided in these sections is identical (!) for the exchange and the crypto helpers. We recommend pointing both users to the same configuration file! @end quotation @end cartouche @node Sign keys,Terms of Service,Coins denomination keys,Configuration<2> @anchor{taler-exchange-manual id7}@anchor{20}@anchor{taler-exchange-manual sign-keys}@anchor{21} @section Sign keys There are three global configuration options of note for sign keys: @itemize - @item @code{[taler-exchange-secmod-eddsa/DURATION]}: How long are sign keys used to sign messages? After this time interval expires, a fresh sign key will be used (key rotation). We recommend using a @code{DURATION} of a few weeks to a few months for sign keys. @item @code{[taler-exchange-secmod-eddsa/OVERLAP_DURATION]}: What is the overlap of the timespan for sign keys? We recommend a few minutes or hours. Must be smaller than @code{DURATION}. @item @code{[taler-exchange-secmod-eddsa/LOOKAHEAD_SIGN]}: For how far into the future should sign keys be pre-generated? This allows the exchange and auditor operators to download, offline-sign, and upload sign key signatures for sign keys that will be used in the future by the exchange. @end itemize @cartouche @quotation Note We recommend setting the @code{LOOKAHEAD_SIGN} value to at least one year and then to perform the offline-signing procedure at least once every 6 months to ensure that there is sufficient time for wallets to learn the new keys and to avoid unavailability in case this critical maintenance procedure is delayed. @end quotation @end cartouche @node Terms of Service,Bank account,Sign keys,Configuration<2> @anchor{taler-exchange-manual terms-of-service}@anchor{22} @section Terms of Service The exchange has an endpoint "/terms" to return the terms of service (in legal language) of the exchange operator. The wallet will show those terms of service to the user when the user is first withdrawing coins. Terms of service are optional for experimental deployments, if none are configured, the exchange will return a simple statement saying that there are no terms of service available. To configure the terms of service response, there are two options in the @code{[exchange]} section: @itemize - @item @code{TERMS_ETAG}: The current "Etag" to return for the terms of service. This value must be changed whenever the terms of service are updated. A common value to use would be a version number. Note that if you change the @code{TERMS_ETAG}, you MUST also provide the respective files in @code{TERMS_DIR} (see below). @item @code{TERMS_DIR}: The directory that contains the terms of service. The files in the directory must be readable to the exchange process. @end itemize The @code{TERMS_DIR} directory structure must follow a particular layout. First, inside of @code{TERMS_DIR}, there should be sub-directories using two-letter language codes like "en", "de", or "jp". Each of these directories would then hold translations of the current terms of service into the respective language. Empty directories are permitted in case translations are not available. Then, inside each language directory, files with the name of the value set as the @code{TERMS_ETAG} must be provided. The extension of each of the files should be typical for the respective mime type. The set of supported mime types is currently hard-coded in the exchange, and includes HTML, PDF and TXT files. If other files are present, the exchange may show a warning on startup. @menu * Example:: @end menu @node Example,,,Terms of Service @anchor{taler-exchange-manual example}@anchor{23} @subsection Example A sample file structure for a @code{TERMS_ETAG} of "v1" would be: @itemize - @item TERMS_DIR/en/v1.txt @item TERMS_DIR/en/v1.html @item TERMS_DIR/en/v1.pdf @item TERMS_DIR/de/v1.txt @item TERMS_DIR/de/v1.html @item TERMS_DIR/de/v1.pdf @item TERMS_DIR/fr/v1.pdf @end itemize If the user requests an HTML format with language preferences "fr" followed by "en", the exchange would return @code{TERMS_DIR/en/v1.html} lacking an HTML version in French. @node Bank account,Auditor configuration,Terms of Service,Configuration<2> @anchor{taler-exchange-manual bank-account}@anchor{24}@anchor{taler-exchange-manual id8}@anchor{25} @section Bank account To configure a bank account in Taler, we need to furnish two pieces of information: @itemize - @item The @code{payto://} URI of the bank account, which uniquely idenfies the account. Examples for such URIs include @code{payto://iban/CH9300762011623852957} for a bank account with an IBAN or @code{payto://x-taler-bank/localhost:8080/2} for the 2nd bank account a the Taler bank demonstrator running at @code{localhost} on port 8080. The first part of the URI following @code{payto://} (“iban” or “x-taler-bank”) is called the wire method. @item The @code{taler-exchange-wirewatch} and @code{taler-exchange-transfer} tools needs to be provided resources for authentication to the respective banking service. The format in which the authentication information is currently a username and password for HTTP basic authentication. @end itemize You can configure multiple accounts for an exchange by creating sections starting with “exchange-account-” for the section name. You can ENABLE for each account whether it should be used, and for what (incoming or outgoing wire transfers): @example [exchange-account-1] # With x-taler-bank (say for PyBank) PAYTO_URI = "payto://x-taler-bank/bank.demo.taler.net/Exchange" # Example using IBAN (for use with LibEuFin) # PAYTO_URI = "payto://iban/CH9300762011623852957" # URL for talking to the bank wire the wire API. WIRE_GATEWAY_URL = https://bank.demo.taler.net/taler-wire-gateway/Exchange # Use for exchange-aggregator (outgoing transfers) ENABLE_DEBIT = YES # Use for exchange-wirewatch (and listed in /wire) ENABLE_CREDIT = YES # Authentication options for exchange bank account go here. # (Next sections have examples of authentication mechanisms) WIRE_GATEWAY_AUTH_METHOD = basic USERNAME = exchange PASSWORD = super-secure @end example The command line tool @code{taler-exchange-offline} must be used to sign the @code{payto://} URI in a way suitable to convince wallets that this is the correct address to wire funds to. For example, the utility may be invoked as follows to enable a wire account: @example $ taler-exchange-offline enable-account payto://iban/CH9300762011623852957 @end example The resulting JSON output must be uploaded to the exchange using @code{taler-exchange-offline upload}. For details, see manpages/taler-exchange-offline.1. @menu * Wire fee structure:: @end menu @node Wire fee structure,,,Bank account @anchor{taler-exchange-manual id9}@anchor{26}@anchor{taler-exchange-manual wire-fee-structure}@anchor{27} @subsection Wire fee structure @geindex wire fee @geindex fee For each wire method (“iban” or “x-taler-bank”) the exchange must know about applicable wire fees. This is also done using the @code{taler-exchange-offline} tool: @example $ taler-exchange-offline wire-fee iban 2040 EUR:0.05 EUR:0.10 EUR:0.15 @end example The above sets the wire fees for wire transfers involving @code{iban} accounts (in Euros) in the year 2040 to 5 cents (wire fee) and 10 cents (closing fee). The tool only supports setting fees that applies for the entire calendar year. We recommend provisioning an exchange with wire fees at least for the next two years. Note that once the fees have been set for a year, they cannot be changed (basically, by signing the fees the exchange makes a legally binding offer to the customers). @geindex maintenance @cartouche @quotation Note Provisioning future wire fees, like provisioning future denomination and signing keys, are key regular maintenance procedures for every exchange operator. We recommend setting automated reminders for this maintenance activity! @end quotation @end cartouche @node Auditor configuration,,Bank account,Configuration<2> @anchor{taler-exchange-manual auditor-configuration}@anchor{28}@anchor{taler-exchange-manual id10}@anchor{29} @section Auditor configuration The exchange must be informed about any auditor that is expected to provision it with auditor signatures. This is also done using the @code{taler-exchange-offline} tool on the offline system. First, the auditor must be configured and provide the exchange operator with its public key and the URL of it's REST API. The exchange operator also needs a human-readable name that may be shown to users to identify the auditor. Given this information, the exchange operator can enable the auditor: @example $ taler-exchange-offline enable-auditor $PUB_KEY $REST_URL "$AUDITOR_NAME" > auditor.json @end example As before, the @emph{auditor.json} file must then be copied from the offline system to a system connected to the exchange and there @code{uploaded} to the exchange. @node Deployment,Testing a deployment,Configuration<2>,Top @anchor{taler-exchange-manual deployment}@anchor{2a}@anchor{taler-exchange-manual id11}@anchor{2b} @chapter Deployment This chapter describes how to deploy the exchange once it has been configured. @menu * Launching an exchange:: * Keys generation:: * Private key storage:: * Database upgrades:: @end menu @node Launching an exchange,Keys generation,,Deployment @anchor{taler-exchange-manual launch}@anchor{2c}@anchor{taler-exchange-manual launching-an-exchange}@anchor{2d} @section Launching an exchange A running exchange requires starting the following processes: @itemize - @item @code{taler-exchange-secmod-rsa} (as special user, sharing group with the HTTPD) @item @code{taler-exchange-secmod-eddsa} (as special user, sharing group with the HTTPD) @item @code{taler-exchange-httpd} (needs database access) @item @code{taler-exchange-aggregator} (only needs database access) @item @code{taler-exchange-closer} (only needs database access) @item @code{taler-exchange-wirewatch} (needs bank account read credentials and database access) @item @code{taler-exchange-transfer} (needs credentials to initiate outgoing wire transfers and database access) @end itemize The crypto helpers must be started before the @code{taler-exchange-httpd} and they should use the same configuration file. For the most secure deployment, we recommend using separate users for each of these processes to minimize information disclosures should any of them be compromised. The helpers do not need access to the PostgreSQL database (and thus also should not have it). The processes that require access to the bank account need to have a configuration file with the respective credentials in it. We recommend using a separate configuration at least for @code{taler-exchange-transfer} which is the @emph{only} process that needs to know the credentials to execute outgoing wire transfers. All of these processes should also be started via a hypervisor like @code{systemd} or @code{gnunet-arm} that automatically re-starts them should they have terminated unexpectedly. If the bank is down (say for maintenance), it is @emph{possible} to halt the @code{taler-exchange-wirewatch} and/or @code{taler-exchange-transfer} processes (to avoid them making requests to the bank API that can only fail) without impacting other operations of the exchange. Naturally, incoming wire transfers will only be observed once @code{taler-exchange-wirewatch} is resumed, and merchants may complain if the disabled @code{taler-exchange-transfer} process causes payment deadlines to be missed. @cartouche @quotation Note The @code{taler-exchange-httpd} does not ship with HTTPS enabled by default. In production, it should be run behind an HTTPS reverse proxy that performs TLS termination on the same system. Thus, it would typically be configured to listen on a UNIX domain socket. The @code{/management} and @code{/auditors} APIs do technically not have to be exposed on the Internet (only to the administrators running @code{taler-exchange-offline}) and should be blocked by the reverse proxy for requests originating from outside of the bank. (However, this is not a strong security assumption: in principle having these endpoints available should do no harm. However, it increases the attack surface.) @end quotation @end cartouche @node Keys generation,Private key storage,Launching an exchange,Deployment @anchor{taler-exchange-manual id12}@anchor{2e}@anchor{taler-exchange-manual keys-generation}@anchor{2f} @section Keys generation Once the configuration is properly set up, all the keys can be signed using the offline key on the offline system by the tool @code{taler-exchange-offline}. To do this, one must first start the crypto helpers and the @code{taler-exchange-httpd} process (the tools for wire transfers may also be started, but do not have to run yet). Next, the @emph{future} key material should be downloaded using: @example $ taler-exchange-offline download > future-keys.json @end example Afterwards, @emph{future-keys.json} contains data about denomination and online signing keys that the exchange operator needs to sign with the offline tool. The file should be copied to the offline system. There, the operator should run: @example $ taler-exchange-offline show < future-keys.json @end example and verify that the output contains the fee structure and key lifetimes they expect to see. They should also note the public keys being shown and communicate those to the @emph{auditors} over a secure channel. Once they are convinced the file is acceptable, they should run: @example $ taler-exchange-offline sign < future-keys.json > offline-sigs.json @end example The @emph{offline-sigs.json} file must then be copied to an online system that is able to again communicate with the exchange. On that system, run: @example $ taler-exchange-offline upload < offline-sigs.json @end example to provision the signatures to the exchange. At this point, the exchange will be able to use those keys, but wallets and merchants may not yet trust them! Thus, the next step is for the auditor to affirm that they are auditing this exchange. Details about this are described in taler-auditor-manual. The simplistic (without using offline keys for the auditor) way to do this would be: @example $ taler-auditor-offline download sign upload @end example For more information, see manpages/taler-auditor-offline.1. @node Private key storage,Database upgrades,Keys generation,Deployment @anchor{taler-exchange-manual private-key-storage}@anchor{30} @section Private key storage Keeping the private keys the helpers create secret is paramount. If the private keys are lost, it is easy to provision fresh keys (with the help of the auditor). Thus, we recommend that the private keys of the crypto helpers are @emph{not} backed up: in the rare event of a disk failure, they can be regenerated. However, we do recommend using RAID (1+1 or 1+1+1) for all disks of the system. @node Database upgrades,,Private key storage,Deployment @anchor{taler-exchange-manual database-upgrades}@anchor{31}@anchor{taler-exchange-manual id13}@anchor{32} @section Database upgrades Currently, there is no way to upgrade the database between Taler versions. The exchange database can be re-initialized using: @example $ taler-exchange-dbinit -r @end example However, running this command will result in all data in the database being lost, which may result in significant financial liabilities as the exchange can then not detect double-spending. Hence this operation must not be performed in a production system. @menu * Revocations:: @end menu @node Revocations,,,Database upgrades @anchor{taler-exchange-manual id14}@anchor{33}@anchor{taler-exchange-manual revocations}@anchor{34} @subsection Revocations When an exchange goes out of business or detects that the private key of a denomination key pair has been compromised, it may revoke some or all of its denomination keys. At this point, the hashes of the revoked keys must be returned as part of the @code{/keys} response under “recoup”. Wallets detect this, and then return unspent coins of the respective denomination key using the @code{/recoup} API. To revoke a denomination key, you need to know the hash of the denomination public key, @code{$HDP}. The @code{$HDP} value is usually included in the security report that is generated when a compromise is detected). Given this value, the key revocation can be approved on the offline system: @example $ taler-exchange-offline revoke-denominatin $HDP > revocation.json @end example The resulting @emph{revocation.json} must be copied to a system connected to the exchange and uploaded to the exchange using the @code{upload} subcommand of @code{taler-exchange-offline}. @cartouche @quotation Note Denomination key revocations should only happen under highly unusual (“emergency”) conditions and not in normal operation. @end quotation @end cartouche @node Testing a deployment,Diagnostics,Deployment,Top @anchor{taler-exchange-manual testing-a-deployment}@anchor{35} @chapter Testing a deployment We recommend testing whether an exchange deployment is functional by using the Taler wallet command line interface. The tool can be used to withdraw and deposit electronic cash via the exchange without having to deploy and operate a separate merchant backend and storefront. For more information, see taler-wallet-cli-manual. @node Diagnostics,Benchmarking,Testing a deployment,Top @anchor{taler-exchange-manual diagnostics}@anchor{36}@anchor{taler-exchange-manual id15}@anchor{37} @chapter Diagnostics This chapter includes various sections on specific topics that might be helpful to understand how the exchange operates. The information may also be helpful for diagnostics. @menu * Internal audits:: * Database Scheme:: @end menu @node Internal audits,Database Scheme,,Diagnostics @anchor{taler-exchange-manual internal-audit}@anchor{38}@anchor{taler-exchange-manual internal-audits}@anchor{39} @section Internal audits While an exchange should use an external auditor to attest to regulators that it is operating correctly, an exchange operator can also use the auditor's logic to perform internal checks. For this, an exchange operator can generally follow the auditor guide. However, instead of using @code{taler-auditor-sync}, an internal audit can and likely should be performed either directly against the production exchange database or against a synchronous copy created using standard database replication techniques. After all, the exchange operator runs this for diagnostics and can generally trust its own database to maintain the database invariants. Running the auditor against a the original the production database (without using @code{taler-auditor-sync}) enables the auditing logic to perform a few additional checks that can detect inconsistencies. These checks are enabled by passing the @strong{-i} option to the @code{taler-auditor} command. As always, the resulting report should be read carefully to see if there are any problems with the setup. Reports are generally created incrementally, with @code{taler-auditor} reporting only incidents and balance changes that were not covered in previous reports. While it is possible to reset the auditor database and to restart the audit from the very beginning, this is generally not recommended as this may be too expensive. @node Database Scheme,,Internal audits,Diagnostics @anchor{taler-exchange-manual database-scheme}@anchor{3a}@anchor{taler-exchange-manual id16}@anchor{3b} @section Database Scheme The exchange database must be initialized using @code{taler-exchange-dbinit}. This tool creates the tables required by the Taler exchange to operate. The tool also allows you to reset the Taler exchange database, which is useful for test cases but should never be used in production. Finally, @code{taler-exchange-dbinit} has a function to garbage collect a database, allowing administrators to purge records that are no longer required. The database scheme used by the exchange looks as follows: @image{taler-exchange-figures/exchange-db,,,,png} @node Benchmarking,Index,Diagnostics,Top @anchor{taler-exchange-manual benchmarking}@anchor{3c}@anchor{taler-exchange-manual exchangebenchmarking}@anchor{3d} @chapter Benchmarking This chapter describes how to run the Taler exchange benchmark. The benchmark can be used to measure the performance of the exchange by running a (possibly large) number of simulated clients against one Taler deployment with a bank, exchange and auditor. For the bank, both a "fakebank" (@code{-f}) and a "Pythonbank" deployment are currently supported. The @code{taler-exchange-benchmark} program can launch all required services and clients, or only launch the parallel clients (@code{-m}), for example for distributed testing over a network. For each @emph{parallel} (@code{-p}) client, a number of @emph{reserves} (@code{-r}) is first established by @strong{transferring} money from a "user" account (42) to the Exchange's account with the respective reserve public key as wire subject. Next, the client will @strong{withdraw} a @emph{number of coins} (@code{-n}) from the reserve and @strong{deposit} them. Additionally, a @emph{fraction} (@code{-R}) of the dirty coins will then be subject to @strong{refreshing}. For some deposits, the auditor will receive @strong{deposit confirmations}. Operations that are not covered today include closing reserves, refunds and recoups. The existing @code{benchmark.conf} file in @code{src/benchmark/} can be used as a starting point for a configuration to run the benchmark. The existing configuration file only requires that the @code{talercheck} database already exists and will launch all required services locally as needed. You can run a first simple benchmark using: @cartouche @quotation Note FIXME-TTN/CG: these instructions are incomplete and untested for the current iteration of the code... @end quotation @end cartouche @example $ createdb talercheck # if it does not yet exist $ taler-exchange-dbinit -c benchmark.conf $ taler-exchange-httpd -c benchmark.conf & $ HTTPD_PID=$! $ taler-exchange-offline -c benchmark.conf \ download sign \ enable-account payto://iban/CH9300762011623852957 \ wire-fee iban EUR:0 EUR:0 EUR:0 \ global-fee EUR:0 EUR:0 EUR:0 EUR:0 4w 4w 6y 4 \ upload $ kill -TERM $HTTPD_PID $ taler-exchange-benchmark -c benchmark.conf -p 4 -r 1 -n 10 @end example This will run 4 parallel clients withdrawing 10 coins from 1 reserve and then depositing those coins. The default refresh probability is 10 percent. Note that the tiny run should only take a few seconds, most of it will be spent in the setup of the original key material. For meaningful runs, all three values should likely be increased. The output of @code{taler-exchange-benchmark} will include for each parallel client the total time spent in each of the major operations, possible repetitions (i.e. if the operation failed the first time), total execution time (operating system and user space) and other details. Naturally, additional instrumentation (including using features of the PostgreSQL database itself) may help discover performance issues. @node Index,,Benchmarking,Top @unnumbered Index @printindex ge @c %**end of body @bye