taler-docs

api-auditor.rst (24785B)

---

 ..
   This file is part of GNU TALER.
   Copyright (C) 2018-2024 Taler Systems SA
 
   TALER is free software; you can redistribute it and/or modify it under the
   terms of the GNU Affero General Public License as published by the Free Software
   Foundation; either version 3.0, or (at your option) any later version.
 
   TALER is distributed in the hope that it will be useful, but WITHOUT ANY
   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
   A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
 
   You should have received a copy of the GNU Affero General Public License along with
   TALER; see the file COPYING.  If not, see <http://www.gnu.org/licenses/>
 
   @author Christian Grothoff
 
 ===================
 Auditor RESTful API
 ===================
 
 The API specified here follows the :ref:`general conventions <http-common>`
 for all details not specified in the individual requests.
 The `glossary <https://docs.taler.net/taler-developer-manual.html#developer-glossary>`_
 defines all specific terms used in this section.
 
 
 ---------------
 Version History
 ---------------
 
 The current protocol version is **v2**.
 
 * The auditor SPA is currently targeting protocol version **v2**.
 
 **Version history:**
 
 * ``v2``: Uses extensible ``records`` response envelopes, plural collection
   paths, uniform suppression and deletion operations for diagnostics, and
   serves the auditor WebUI below ``/webui/``.
 
 **Upcoming versions:**
 
 * none anticipated
 
 **Ideas for future version:**
 
 * ``vXXX``: marker for features not yet targeted for release
 
 
 .. _authentication:
 
 --------------
 Authentication
 --------------
 
 Each auditor instance has separate authentication settings for the private API resources
 of that instance.
 
 Currently, the API supports two main authentication methods:
 
 * ``external``:  With this method, no checks are done by the auditor backend.
   Instead, a reverse proxy / API gateway must do all authentication/authorization checks.
 * ``token``: With this method, the client must provide an
   ``Authorization: Bearer secret-token:$TOKEN`` header, where ``$TOKEN`` is
   the configured secret authentication token.
 
 
 -------------
 Auditor WebUI
 -------------
 
 The auditor serves its browser-based administration interface below
 ``/webui/``.  The WebUI resources themselves are public.  Private auditor API
 resources remain subject to the authentication described above.
 
 .. include:: auditor/get-webui.rst
 
 .. _auditor-version:
 
 -------------------------
 Obtaining Auditor Version
 -------------------------
 
 This endpoint exposes protocol and implementation metadata for the auditor,
 including the single exchange master public key configured for this auditor.
 
 .. include:: auditor/get-config.rst
 
 
 .. _deposit-confirmation:
 
 ---------------------
 Deposit Confirmations
 ---------------------
 
 Merchants should probabilistically submit some of the deposit
 confirmations they receive from the exchange to auditors to ensure
 that the exchange does not lie about recording deposit confirmations
 with the exchange. Participating in this scheme ensures that in case
 an exchange runs into financial trouble to pay its obligations, the
 merchants that did participate in detecting the bad behavior can be
 paid out first.
 
 .. include:: auditor/put-deposit-confirmation.rst
 
 
 .. _auditor-monitoring-spa-api:
 
 --------------
 Monitoring API
 --------------
 
 The following entries specify how to access the results of an audit.
 
 For most endpoints, rows may be marked as 'suppressed' to not send them again
 upon subsequent GET requests.  To do this, a
 :ts:type:`GenericAuditorMonitorPatchRequest` object is used in the respective
 PATCH request.
 
 **Details:**
 
   .. ts:def:: GenericAuditorMonitorPatchRequest
 
     interface GenericAuditorMonitorPatchRequest {
 
       // If true, subsequent GET requests will not return this element by default
       suppressed : boolean;
 
     }
 
 
 .. _fee-time-inconsistency-list:
 
 Fee Time Inconsistencies
 ------------------------
 
 This section highlights cases where validity periods associated with wire fees
 the exchange may charge merchants are invalid.  This usually means that the
 validity periods given for the same type of fee are overlapping and it is thus
 unclear which fee really applies.  This is a sign of a serious
 misconfiguration or data corruption as usually the exchange logic should
 prevent such a fee configuration from being accepted.
 
 .. include:: auditor/get-monitoring-fee-time-inconsistency.rst
 
 .. include:: auditor/patch-monitoring-fee-time-inconsistency-SERIAL_ID.rst
 
 .. _emergency-list:
 
 Emergencies
 -----------
 
 This endpoint is used to obtain a list of emergencies.
 
 Emergencies are errors where the total value of coins deposited (of a
 particular denomination) exceeds the total value that the exchange remembers
 issuing.  This usually means that the private keys of the exchange were
 compromised (stolen or factored) and subsequently used to sign coins off the
 books.  If this happens, all coins of the respective denomination that the
 exchange has redeemed so far may have been created by the attacker, and the
 exchange would have to refund all of the outstanding coins from ordinary
 users.  Thus, the risk exposure is the amount of coins in circulation for a
 particular denomination and the maximum loss for the exchange from this type
 of compromise.
 
 The difference between emergencies and emergencies by count is how the auditor
 detected the problem: by comparing amounts, or by counting coins.
 Theoretically, counting coins should always detect an issue first, but given
 the importance of emergencies, the auditor checks both total amounts and total
 numbers of coins (they may differ as coins may be partially deposited).
 
 .. include:: auditor/get-monitoring-emergency.rst
 
 .. include:: auditor/patch-monitoring-emergency-SERIAL_ID.rst
 
 .. _emergency-by-count-list:
 
 Emergencies By Count
 --------------------
 
 This endpoint is used to obtain a list of emergencies by count.
 
 Emergencies are errors where more coins were deposited than the
 exchange remembers issuing.  This usually means that the private keys
 of the exchange were compromised (stolen or factored) and subsequently
 used to sign coins off the books.  If this happens, all coins of the
 respective denomination that the exchange has redeemed so far may have
 been created by the attacker, and the exchange would have to refund
 all of the outstanding coins from ordinary users.  Thus, the risk
 exposure is the amount of coins in circulation for a particular
 denomination and the maximum loss for the exchange from this type of
 compromise.
 
 Emergencies "by count" are cases where this type of money printing was
 detected simply by counting the number of coins the exchange officially put
 into circulation and comparing it to the number of coins that were redeemed.
 If the number of redeemed coins is higher than the number of issued coins, the
 auditor reports an emergency-by-count.
 
 .. include:: auditor/get-monitoring-emergency-by-count.rst
 
 .. include:: auditor/patch-monitoring-emergency-by-count-SERIAL_ID.rst
 
 
 .. _row-inconsistency-list:
 
 Row Inconsistencies
 -------------------
 
 This section highlights inconsistencies in a specific row of a specific table
 of the exchange.  Row inconsistencies are reported from different sources, and
 largely point to some kind of data corruption (or bug). Nothing is implied
 about the seriousness of the inconsistency. Most inconsistencies are detected
 if some signature fails to validate. The affected table is noted in the
 'table' field. A description of the nature of the inconsistency is noted in
 'diagnostic'.
 
 .. include:: auditor/get-monitoring-row-inconsistency.rst
 
 
 .. include:: auditor/patch-monitoring-row-inconsistency-SERIAL_ID.rst
 
 .. _reserve-in-inconsistency-list:
 
 Reserve In Inconsistencies
 --------------------------
 
 This section lists cases where the exchange's and auditor's expectation of
 amounts transferred into a reserve differs.  Basically, the exchange database
 states that a certain reserve was credited for a certain amount via a wire
 transfer, but the auditor disagrees about this basic fact.  This may result in
 either a customer losing funds (by being issued less digital cash than they
 should be) or the exchange losing funds (by issuing a customer more digital
 cash than they should be).
 
 .. include:: auditor/get-monitoring-reserve-in-inconsistency.rst
 
 
 
 
 .. include:: auditor/patch-monitoring-reserve-in-inconsistency-SERIAL_ID.rst
 
 
 .. _purse-not-closed-inconsistencies-list:
 
 Purse Not Closed Inconsistencies
 --------------------------------
 
 This section highlights cases, in which either payer or payee did not finish
 their part of a P2P payment. This caused a purse --– which may contain some
 money --- to reach its expiration date.  However, the exchange failed to
 properly expire the purse, which means the payer did not get their money back.
 The cause is usually that the **taler-exchange-expire** helper is not running
 properly.
 
 
 .. include:: auditor/get-monitoring-purse-not-closed-inconsistencies.rst
 
 
 .. include:: auditor/patch-monitoring-purse-not-closed-inconsistencies-SERIAL_ID.rst
 
 
 Early Aggregations
 ------------------
 
 .. _early-aggregation-list:
 
 
 This endpoint returns cases in which wire transfers are encountered before their
 justifications. This can be harmless, if the justifications appear shortly afterwards.
 
 .. include:: auditor/get-monitoring-early-aggregation.rst
 
 .. include:: auditor/patch-monitoring-early-aggregations-SERIAL_ID.rst
 
 
 
 Pending Deposits
 ----------------
 
 .. _pending-deposits-list:
 
 This endpoint returns cases in which deposits are pending, that is an
 expected wire transfer for a given deposit was not yet found even though
 it is past the wire deadline.
 This can be harmless, if the wire transfers appear shortly afterwards.
 
 .. include:: auditor/get-monitoring-pending-deposits.rst
 
 .. include:: auditor/patch-monitoring-pending-deposits-SERIAL_ID.rst
 
 
 
 
 
 
 .. _reserve-not-closed-inconsistency-list:
 
 Reserve Not Closed Inconsistencies
 ----------------------------------
 
 This section highlights cases, in which reserves were not closed, despite being expired.
 As a result, customers that wired funds to the exchange and then failed to withdraw them
 are not getting their money back. The cause is usually that the **taler-exchange-closer**
 process is not running properly.
 
 .. include:: auditor/get-monitoring-reserve-not-closed-inconsistency.rst
 
 
 
 
 .. include:: auditor/patch-monitoring-reserve-not-closed-inconsistency-SERIAL_ID.rst
 
 
 
 
 .. _reserve-balance-insufficient-inconsistency-list:
 
 Reserve Balance Insufficient Inconsistencies
 --------------------------------------------
 
 This section highlights cases where more coins were withdrawn from a
 reserve than the reserve contained funding for.  This is a serious
 compromise resulting in proportional financial losses to the exchange.
 
 .. include:: auditor/get-monitoring-reserve-balance-insufficient-inconsistency.rst
 
 
 
 
 
 .. include:: auditor/patch-monitoring-reserve-balance-insufficient-inconsistency-SERIAL_ID.rst
 
 
 .. _invalid-signature-losses-list:
 
 Invalid Signature Losses
 ------------------------
 
 This section lists operations that the exchange performed, but for which the
 signatures provided are invalid. Hence the operations are invalid and the
 amount involved could be a loss for the exchange (as the involved parties
 could successfully dispute the resulting transactions).
 
 .. include:: auditor/get-monitoring-bad-sig-losses.rst
 
 
 .. include:: auditor/patch-monitoring-bad-sig-losses-SERIAL_ID.rst
 
 
 .. _coin-inconsistency-list:
 
 Coin Inconsistencies
 --------------------
 
 This section lists cases where the exchange made arithmetic errors found when
 looking at the transaction history of a coin.  The totals sum up the differences
 in amounts that matter for profit/loss calculations of the exchange. When an
 exchange merely shifted money from customers to merchants (or vice versa) without
 any effects on its own balance, those entries are excluded from the total.
 
 .. include:: auditor/get-monitoring-coin-inconsistency.rst
 
 
 .. include:: auditor/patch-monitoring-coin-inconsistency-SERIAL_ID.rst
 
 
 
 
 .. _denominations-without-signatures-list:
 
 Denominations Without Signatures
 --------------------------------
 
 This section highlights denomination keys that lack a proper
 signature from the **taler-auditor-offline** tool. This may be
 legitimate, say in case where the auditor's involvement in the
 exchange business is ending and a new auditor is responsible for
 future denominations. So this must be read with a keen eye on the
 business situation.
 
 .. include:: auditor/get-monitoring-denominations-without-sigs.rst
 
 
 
 .. include:: auditor/patch-monitoring-denominations-without-sigs-SERIAL_ID.rst
 
 
 .. _misattribution-in-inconsistency-list:
 
 Misattribution In Inconsistencies
 ---------------------------------
 
 This section lists cases where the sender account record of an incoming wire
 transfer differs between the exchange and the bank.  This may cause funds to
 be sent to the wrong account should the reserve be closed with a remaining
 balance, as that balance would be credited to the original account.
 
 .. include:: auditor/get-monitoring-misattribution-in-inconsistency.rst
 
 
 
 .. include:: auditor/patch-monitoring-misattribution-in-inconsistency-SERIAL_ID.rst
 
 
 .. _kycauth-in-inconsistency-list:
 
 KYCAUTH In Inconsistencies
 --------------------------
 
 This section lists cases where the exchange and the bank disagree about a
 transfer that established a KYC authentication account.
 
 .. include:: auditor/get-monitoring-kycauth-in-inconsistency.rst
 
 .. include:: auditor/patch-monitoring-kycauth-in-inconsistency-SERIAL_ID.rst
 
 
 
 .. _deposit-confirmations-list:
 
 Deposit Confirmations
 ---------------------
 
 This section contains a list of deposits confirmations that an exchange
 provided to merchants but *failed* to store in its own database.  This is
 indicative of potential fraud by the exchange operator, as the exchange should
 only issue deposit confirmations after storing the respective deposit records
 in its database.  Not storing the deposit data means that the exchange would
 not pay the merchant (pocketing the money) or allow the customer to
 double-spend the money (which is naturally also not good).
 
 Note that entries could appear in this list also because the exchange database
 replication is delayed. Hence, entries that are only a few seconds old might
 not be indicative of an actual problem. If entries in this list are more than
 a few seconds old, the first thing to check is whether or not the database
 replication from the exchange is working properly.
 
 .. include:: auditor/get-monitoring-deposit-confirmations.rst
 
 
 .. include:: auditor/patch-monitoring-deposit-confirmations-SERIAL_ID.rst
 
 
 .. _denomination-key-validity-withdraw-inconsistency-list:
 
 Denomination Key Validity Withdraw Inconsistencies
 --------------------------------------------------
 
 This section highlights cases, where denomination keys were used to sign coins
 withdrawn from a reserve before the denomination was valid or after it was
 already expired for signing.  This doesn't exactly imply any financial loss
 for anyone, it is mostly weird and may have affected the fees the customer
 paid.
 
 .. include:: auditor/get-monitoring-denomination-key-validity-withdraw-inconsistency.rst
 
 .. include:: auditor/patch-monitoring-denomination-key-validity-withdraw-inconsistency-SERIAL_ID.rst
 
 .. _amount-arithmetic-inconsistency-list:
 
 Amount Arithmetic Inconsistencies
 ---------------------------------
 
 This endpoint is used to obtain a list of amount arithmetic inconsistencies.
 
 This section lists cases where the arithmetic of the exchange involving
 amounts disagrees with the arithmetic of the auditor.  Disagreements imply
 that either the exchange made a loss (sending out too much money), or screwed
 a customer (and thus at least needs to fix the financial damage done to the
 customer).  The profitable column is set to true if the arithmetic problem was
 determined to be profitable for the exchange, false if the problem resulted
 in a net loss for the exchange.
 
 
 .. include:: auditor/get-monitoring-amount-arithmetic-inconsistency.rst
 
 
 .. include:: auditor/patch-monitoring-amount-arithmetic-inconsistency-SERIAL_ID.rst
 
 
 .. _wire-format-inconsistency-list:
 
 Wire Format Inconsistencies
 ---------------------------
 
 This section highlights cases where the wire transfer subject
 was used more than once and is thus not unique.  This indicates
 a problem with the bank's implementation of the revenue API, as
 the bank is supposed to warrant uniqueness of wire transfer
 subjects exposed via the revenue API (and bounce non-unique
 transfers).
 
 .. include:: auditor/get-monitoring-wire-format-inconsistency.rst
 
 
 
 
 .. include:: auditor/patch-monitoring-wire-format-inconsistency-SERIAL_ID.rst
 
 
 .. _aml-holds-list:
 
 AML Holds
 ---------
 
 This endpoint is used to obtain a list of the wire transfers the exchange
 aggregated but did not execute.
 
 The exchange aggregates the deposits it owes to a merchant, decides what the
 payout should be, records that decision, and only then makes the wire
 transfer.  Between those two steps the money is still the exchange's.  An
 ``aggregation_tracking`` row is therefore not proof that the merchant was
 paid; a ``wire_out`` row is.  This endpoint lists the transfers that are stuck
 in between, one row per wire transfer identifier, and the row disappears once
 the transfer is made.
 
 The exchange has two legitimate reasons to sit on such funds.  It may not pay
 out to an account whose KYC/AML requirements are still open; and it need not
 make a transfer whose amount would not survive the wire fee, in which case it
 waits for further deposits to the same account.  ``deferral_reason`` says which
 of the two the exchange gave, or ``NONE`` if it gave neither.  The three cases
 are also totalled separately, as the ``total_aml_hold``,
 ``total_small_aggregate`` and ``total_transfer_lag`` balances (see
 :ref:`balances-list`), so that "the exchange is waiting for legitimization
 data", "the exchange is waiting for more deposits" and "the exchange is not
 paying" cannot be confused with each other or with having paid.
 
 Whether a hold is *justified* is a question for a human, not for the auditor:
 the auditor reports what is being withheld and from whom, and leaves the
 judgement to the reader.  What it does check is that the exchange does not get
 to withhold money silently, and that the reason it gives is consistent with
 everything else the auditor can see.
 
 .. note::
 
   ``deferral_reason`` is the exchange's own statement, taken from its
   append-only ``aggregation_deferrals`` table, which is replicated to the
   auditor.  The auditor does not overrule it, but it does check it: the amount
   against its own recomputation of what the aggregate is worth, and a ``KYC``
   claim against the ``legitimization_measures`` and
   ``legitimization_processes`` tables.  If the exchange names a measure that is
   not open, the hold is still reported as a ``KYC`` hold and a row
   inconsistency against ``aggregation_deferrals`` is raised alongside it.  An
   exchange-internal audit (the ``-i`` option of
   **taler-helper-auditor-transfer**) additionally compares the claim with the
   live ``aggregation_transient`` table, which cannot be replicated.
 
 .. include:: auditor/get-monitoring-aml-holds.rst
 
 
 .. include:: auditor/patch-monitoring-aml-holds-SERIAL_ID.rst
 
 
 
 
 .. _closure-lags-list:
 
 Closure Lags
 ------------
 
 This endpoint is used to obtain a list of closure lags.
 
 A closure lag happens if a reserve should have closed a reserve and
 wired (remaining) funds back to the originating account, but did not
 do so on time.  Significant lag may be indicative of fraud, while
 moderate lag is indicative that the systems may be too slow to handle the
 load. Small amounts of lag can occur in normal operation.
 
 If closure lag is experienced, the administrator should check that
 the **taler-exchange-closer** component is operating correctly.
 
 
 .. include:: auditor/get-monitoring-closure-lags.rst
 
 
 .. include:: auditor/patch-monitoring-closure-lags-SERIAL_ID.rst
 
 
 
 
 .. _wire-out-inconsistency-list:
 
 Wire Out Inconsistencies
 ------------------------
 
 This section highlights cases where the exchange wired a different amount to a
 destination account than the auditor expected.
 
 .. include:: auditor/get-monitoring-wire-out-inconsistency.rst
 
 
 
 
 
 .. include:: auditor/patch-monitoring-wire-out-inconsistency-SERIAL_ID.rst
 
 
 .. _reserve-balance-summary-wrong-inconsistency-list:
 
 Reserve Balance Summary Wrong Inconsistencies
 ---------------------------------------------
 
 This section highlights cases, where the exchange's and auditors'
 expectation of the amount of money left in a reserve differs.
 
 .. include:: auditor/get-monitoring-reserve-balance-summary-wrong-inconsistency.rst
 
 .. include:: auditor/patch-monitoring-reserve-balance-summary-wrong-inconsistency-SERIAL_ID.rst
 
 
 .. _row-minor-inconsistencies-list:
 
 Row Minor Inconsistencies
 -------------------------
 
 The section highlights inconsistencies where a row in an exchange table has a
 value that does not satisfy expectations (such as a malformed
 signature). These are cause for concern, but not necessarily point to a
 monetary loss (yet).
 
 .. include:: auditor/get-monitoring-row-minor-inconsistencies.rst
 
 .. include:: auditor/patch-monitoring-row-minor-inconsistencies-SERIAL_ID.rst
 
 
 -------------------------
 Monitoring Auditor Status
 -------------------------
 
 The following entries specify how to access information the auditor keeps to
 properly perform audits.  These tables do not contain inconsistencies, instead
 they store information about balances, reserves, purses etc.  Values in these
 tables should not differ from their respective exchanges' version.
 
 .. _balances-list:
 
 Balances
 --------
 
 Returns the various balances the auditor tracks for the exchange, such as
 coins in circulation, fees earned, losses experienced, etc.
 
 Three of the keys concern money the exchange owes a merchant and has not sent.
 Which one an amount lands in follows the reason the exchange gave for not
 sending it:
 
 ``total_aml_hold``
   The exchange says a KYC/AML requirement against the recipient is still open.
   This is money withheld pending legitimization.
 
 ``total_small_aggregate``
   The exchange says what it has collected for the account so far would not
   cover the wire fee, so it is waiting for further deposits.  This is normal
   operation rather than an irregularity; it has a key of its own so that it
   does not inflate the other two, which are not.
 
 ``total_transfer_lag``
   The exchange gave no reason at all.  This is money the exchange simply has
   not paid out.
 
 All three are current state rather than a running tally: the amount is
 recomputed from the outstanding transfers on every audit run, so it falls again
 as transfers are executed, and a transfer moves between the keys if the
 exchange changes what it says about it.  They are distinct from
 ``total_amount_lag``, which counts deposits the exchange has not aggregated at
 all.  The individual transfers behind all three keys are listed by
 :ref:`aml-holds-list`.
 
 .. include:: auditor/get-monitoring-balances.rst
 
 
 .. _historic-denomination-revenue-list:
 
 Historic Denomination Revenue
 -----------------------------
 
 This endpoint is used to obtain a list of historic denomination revenue, that
 is the profits and losses an exchange has made from coins of a particular
 denomination where the denomination is past its (deposit) expiration and thus
 all values are final.
 
 .. include:: auditor/get-monitoring-historic-denomination-revenue.rst
 
 .. _historic-reserve-summary-list:
 
 Historic Reserve Summary
 ------------------------
 
 This section summarizes historic profits an exchange
 made from reserves and associated reserve-specific
 fees.
 
 .. include:: auditor/get-monitoring-historic-reserve-summary.rst
 
 
 .. _reserves-list:
 
 
 Reserves
 --------
 
 This endpoint is used to obtain a list of open reserves that the auditor is
 currently tracking balances for.
 
 .. include:: auditor/get-monitoring-reserves.rst
 
 
 .. _purses-list:
 
 Purses
 ------
 
 This endpoint is used to obtain information about open purses.
 
 .. include:: auditor/get-monitoring-purses.rst
 
 Progress
 --------
 
 This section contains information about the auditing progress an auditor has made.
 
 .. include:: auditor/get-monitoring-progress.rst
 
 
 Deleting Monitoring Records
 ---------------------------
 
 Deleting a monitoring record is intended for administrative cleanup after the
 underlying issue has been resolved.  Suppression should be preferred when the
 audit history should remain available.
 
 .. include:: auditor/delete-monitoring-records.rst