update benchmarking section in exchange operator manual

1 files changed, 135 insertions, 51 deletions

diff --git a/taler-exchange-manual.rst b/taler-exchange-manual.rst
index ffc1e125..2e6e63b6 100644
--- a/taler-exchange-manual.rst
+++ b/taler-exchange-manual.rst
@@ -351,7 +351,7 @@ The package sources can be find in our
 `download directory <http://ftpmirror.gnu.org/taler/>`__.
 
 .. include:: frags/semver.rst
-             
+
 First, the following packages need to be installed before we can compile the
 backend:
 
@@ -453,7 +453,7 @@ The exchange setup uses the following system groups:
 * ``taler-exchange-secmod``: group for processes with access to online signing keys; this group must have four users: taler-exchange-secmod-rsa, taler-exchange-secmod-cs, taler-exchange-secmod-eddsa and taler-exchange-httpd.
 * ``taler-exchange-offline``: group for the access to the offline private key (only used on the offline host and not used on the online system).
 
-  
+
 The package will deploy systemd service files in
 ``/usr/lib/systemd/system/`` for the various components:
 
@@ -2229,71 +2229,155 @@ not be performed in a production system.
 Benchmarking
 ============
 
-This chapter describes how to run the Taler exchange benchmark.  The benchmark
-can be used to measure the performance of the exchange by running a (possibly
-large) number of simulated clients against one Taler deployment with a bank,
-exchange and auditor.  For the bank, both a "fakebank" (``-f``) and a
-"Pythonbank" deployment are currently supported.  The
-``taler-exchange-benchmark`` program can launch all required services and
-clients, or only launch the parallel clients (``-m``), for example for
-distributed testing over a network.
+This chapter describes how to run various benchmarks against a Taler exchange.
+These benchmark can be used to measure the performance of the exchange by
+running a (possibly large) number of simulated clients against one Taler
+deployment with a bank, exchange and (optionally) auditor.
+
+Real benchmarks that are intended to demonstrate the scalability of GNU Taler
+should not use the tools presented in this section: they may be suitable for
+microbenchmarking and tuning, but the setup is inherently not optimzied for
+performance or realism, both for the load generation and the server side.
+Thus, we do not recommend using these performance numbers to assess the
+scalability of GNU Taler.  That said, the tools can be useful to help identify
+performance issues.
+
+The ``taler-unified-setup.sh`` script can be used to launch all required
+services and clients. However, the resulting deployment is simplistic
+(everything on the local machine, one single-threaded process per service
+type) and not optimized for performance at all. However, this can still be
+useful to assess the performance impact of changes
+to the code or configuration.
+
+The various configuration files used in the code snippets in this section can
+be found in the ``src/benchmark/`` directory of the exchange. These are
+generally intended as starting points.  Note that the configuration files
+ending in ``.edited`` are created by ``taler-unified-setup.sh`` and contain
+some options that are determined at runtime by the setup logic provided by
+``taler-unified-setup.sh``.
+
+
+Choosing a bank
+---------------
+
+For the bank, both a fakebank (``-f``) and libeufin-based (``-ns``)
+bank deployment are currently supported by all benchmark tools and
+configuration templates.
+
+Fakebank is an ultra-fast in-memory implementation of the Taler bank API. It
+is suitable when the goal is to benchmark the core GNU Taler payment system
+and to ignore the real-time gross settlement (RTGS) system typically provided
+by an existing bank.  When using the fakebank, ``taler-unified-setup.sh`` must
+be started with the ``-f`` option and be told to use the right exchange bank
+account from the configuration files via ``-u exchange-account-1``.
+
+.. code-block:: console
+
+    $ dropdb talercheck; createdb talercheck
+    $ taler-unified-setup.sh -emwt -c $CONF -f -u exchange-account-1
+
+
+libeufin is GNU Taler's adapter to the core banking system using the EBICS
+banking protocol standard.  It uses a Postgres database to persist data and is
+thus much slower than fakebank.  If your GNU Taler deployment uses libeufin in
+production, it likely makes sense to benchmark with libeufin.  When using the
+fakebank, ``taler-unified-setup.sh`` must be started with the ``-ns`` options
+(starting libeufin-nexus and libeufin-sandbox) and be told to use the right
+exchange bank account from the configuration files via ``-u
+exchange-account-2``.  Note that ``taler-unified-setup.sh`` currently cannot
+reset a libeufin database, and also will not run if the database is already
+initialized. Thus, you must re-create the database every time before
+running the command:
+
+.. code-block:: console
+
+    $ dropdb talercheck; createdb talercheck
+    $ taler-unified-setup.sh -emwt -c $CONF -ns -u exchange-account-2
+
+
+taler-bank-benchmark
+--------------------
+
+This is the simplest benchmarking tool, simulating only the bank
+interaction.
+
+.. code-block:: console
+
+    $ CONF="benchmark-cs.conf"
+    $ # or with libeufin
+    $ dropdb talercheck; createdb talercheck
+    $ taler-unified-setup.sh -emwt -c "$CONF" -f -u exchange-account-1
+    $ # Once <<READY>>, in another shell (remember to set $CONF):
+    $ time taler-bank-benchmark -c "$CONF" -r 40 -p 4 -P4 -u exchange-account-1 -f
+    $ # or with libeufin
+    $ dropdb talercheck; createdb talercheck
+    $ taler-unified-setup.sh -emwt -c "$CONF" -ns -u exchange-account-2
+    $ # Once <<READY>>, in another shell (remember to set $CONF):
+    $ time taler-bank-benchmark -c "$CONF" -r 40 -p 1 -P1 -u exchange-account-2
 
 For each *parallel* (``-p``) client, a number of *reserves* (``-r``) is first
 established by **transferring** money from a "user" account (42) to the
 Exchange's account with the respective reserve public key as wire subject.
-Next, the client will **withdraw** a *number of coins* (``-n``) from the
-reserve and **deposit** them. Additionally, a *fraction* (``-R``) of the dirty
-coins will then be subject to **refreshing**.  For some deposits, the auditor
-will receive **deposit confirmations**.
+Processing is then handled by *parallel* (``-P``) service workers.
 
-Operations that are not covered today include closing reserves, refunds and
-recoups.
 
-The existing ``benchmark.conf`` file in ``src/benchmark/`` can be used as a
-starting point for a configuration to run the benchmark. The existing
-configuration file only requires that the ``talercheck`` database already
-exists and will launch all required services locally as needed.
+taler-exchange-benchmark
+------------------------
 
-You can run a first simple benchmark using:
-
-.. note::
-   FIXME-CG: these instructions are incomplete and untested for the
-   current iteration of the code...
+This is the benchmarking tool simulates a number of clients withdrawing,
+depositing and refreshing coins.  Operations that are not covered by the
+``taler-exchange-benchmark`` tool today include closing reserves, refunds,
+recoups and P2P payments.
 
 .. code-block:: console
 
-   $ createdb talercheck # if it does not yet exist
-   $ export CIPHER="rsa" # or cs
-   $ export CONF="benchmark-${CIPHER}.conf"
-   $ taler-exchange-dbinit -c "$CONF"
-   $ taler-exchange-secmod-rsa -c "$CONF" &
-   $ taler-exchange-secmod-cs -c "$CONF" &
-   $ taler-exchange-secmod-eddsa -c "$CONF" &
-   $ taler-exchange-httpd -c "$CONF" &
-   $ HTTPD_PID=$!
-   $ taler-exchange-offline -c "$CONF" \
-     download sign \
-     enable-account \
-       payto://iban/CH9300762011623852957?receiver-name=exchange \
-     wire-fee now iban EUR:0 EUR:0 \
-     global-fee now EUR:0 EUR:0 EUR:0 4w 6a 42 \
-     upload
-   $ kill -TERM $(jobs -p)
-   $ taler-exchange-benchmark -c "$CONF" -p 4 -r 1 -n 10
-
-This will run 4 parallel clients withdrawing 10 coins from 1 reserve and then
-depositing those coins. The default refresh probability is 10 percent.  Note
-that the tiny run should only take a few seconds, most of it will be spent in
-the setup of the original key material. For meaningful runs, all three values
-should likely be increased.
+    $ CONF="benchmark-cs.conf" # -rsa also makes sense
+    $ # With fakebank
+    $ dropdb talercheck; createdb talercheck
+    $ taler-unified-setup.sh -aemwt -c "$CONF" -f -u exchange-account-1
+    $ # Once <<READY>>, in another shell (remember to set $CONF):
+    $ taler-exchange-benchmark -c "$CONF".edited -u exchange-account-1 -n 1 -p1 -r 5 -f
+    $ #
+    $ # With libeufin
+    $ dropdb talercheck; createdb talercheck
+    $ taler-unified-setup.sh -aemwt -c "$CONF" -ns -u exchange-account-2
+    $ # Once <<READY>>, in another shell (remember to set $CONF):
+    $ taler-exchange-benchmark -c "$CONF".edited -u exchange-account-2 -L WARNING -n 1 -p1 -r 5
+
+For each *parallel* (``-p``) client, a number of *reserves* (``-r``) is first
+established by **transferring** money from a "user" account (42) to the
+Exchange's account with the respective reserve public key as wire subject.
+Next, the client will **withdraw** a *number of coins* (``-n``) from the
+reserve and **deposit** them. Additionally, a *fraction* (``-R``) of the dirty
+coins will then be subject to **refreshing**.  For some deposits, the auditor
+will receive **deposit confirmations**.
 
 The output of ``taler-exchange-benchmark`` will include for each parallel
 client the total time spent in each of the major operations, possible
 repetitions (i.e. if the operation failed the first time), total execution
 time (operating system and user space) and other details.
 
-Naturally, additional instrumentation (including using features of the
-PostgreSQL database itself) may help discover performance issues.
+
+taler-aggregator-benchmark
+--------------------------
+
+This is another simple benchmark tool that merely prepares an exchange
+database to run a stand-alone benchmark of the ``taler-exchange-aggregator``
+tool.  After preparing a database and running the tool, you can then
+run one or more ``taler-exchange-aggregator`` processes and measure how
+quickly they perform the aggregation work.
+
+.. code-block:: console
+
+    $ CONF=benchmark-rsa.conf
+    $ taler-exchange-dbinit -c "$CONF" --reset
+    $ ./taler-aggregator-benchmark -c "$CONF" -m 500 -r 10 -d 100
+    $ time taler-exchange-aggregator -c "$CONF" --test
+
+This above commands will first create 100 deposits with 10 refunds into each
+of 500 merchant accounts using randomized time stamps.  Afterwards, it will
+time a single aggregator process in ``--test`` mode (asking it to terminate
+as soon as there is no more pending work).
 
 
 FIXMEs