diff options
Diffstat (limited to 'design-documents/050-libeufin-nexus.rst')
| -rw-r--r-- | design-documents/050-libeufin-nexus.rst | 354 |
1 files changed, 354 insertions, 0 deletions
diff --git a/design-documents/050-libeufin-nexus.rst b/design-documents/050-libeufin-nexus.rst new file mode 100644 index 00000000..6049bda6 --- /dev/null +++ b/design-documents/050-libeufin-nexus.rst @@ -0,0 +1,354 @@ +DD 50: Libeufin-Nexus +##################### + +Summary +======= + +This document proposes a new design for the libeufin Nexus component, +focusing on the user-interaction and where what state is to be stored. + + +Motivation +========== + +The existing Nexus design is overly complex to configure, develop and +maintain. It supports EBICS features we do not need, and lacks key features +(like long-polling) that are absolutely needed. + +.. + long-polling at the TWG is NOT NetzBon-critical, as the TWG is only offered + by the Bank. + +We also have several implementations with Nexus, Bank and Depolymerization +subsystems, and it would be good to combine some of them. + +Requirements +============ + + * Easy to use, high-level abstraction over EBICS details + * Reduce complexity, no multi-account, multi-connection support + * No general EBICS client, Taler-specific logic + * Support for Taler facade, including bouncing of transactions with malformed subject + * Clear separation between configuration and runtime, minimal runtime footprint + * No built-in cron-jobs, background tasks runnable via systemd (one-shot and persistent mode) + * Configuration style same as other GNUnet/Taler components + * No private keys in database, as in other Taler components + * Enable future unified implementation with Depolymerization to share database and REST API logic + +Proposed Solution +================= + +Split up Nexus into four components: + + * nexus-ebics-setup: register account with EBICS server and perform key generation and exchange + * nexus-ebics-fetch: obtain wire transfers and payment status from EBICS server + * nexus-ebics-submit: send payment initiation messages to EBICS server + * nexus-httpd: serve Taler REST APIs (wire gateway API, revenue API) to Taler clients and implement facade logic + +All four components should read a simple INI-style configuration file, +possibly with component-specific sections. + +Configuration file +------------------ + +.. code-block:: shell-session + + [nexus-ebics] + CURRENCY = EUR + HOST_BASE_URL = http://ebics.bank.com/ + HOST_ID = mybank + USER_ID = myuser + PARTNER_ID = myorg + IBAN = MY-IBAN + BIC = MY-BIC + NAME = MY NAME + BANK_PUBLIC_KEYS_FILE = enc-auth-keys.json + CLIENT_PRIVATE_KEYS_FILE = my-private-keys.json + BANK_DIALECT = postfinance # EBICS+ISO20022 style used by the bank. + + [nexus-postgres] + CONFIG = postgres:///libeufin-nexus + + [nexus-ebics-fetch] + FREQUENCY = 30s # used when long-polling is not supported + STATEMENT_LOG_DIRECTORY = /tmp/ebics-messages/ + + [nexus-ebics-submit] + FREQUENCY = 30s # use 0 to always submit immediately (via LISTEN trigger) + + [nexus-httpd] + PORT = 8080 + UNIXPATH = + SERVE = tcp | unix + + [nexus-httpd-wire-gateway-facade] + ENABLED = YES + AUTH_METHOD = token + AUTH_TOKEN = "secret-token:foo" + + [nexus-httpd-revenue-facade] + ENABLED = YES + AUTH_METHOD = token + AUTH_TOKEN = "secret-token:foo" + + +File contents: BANK_PUBLIC_KEYS_FILE +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +JSON with 3 fields: + + * bank_encryption_public_key (base32) + * bank_authentication_public_key (base32) + * accepted (boolean) + +File contents: CLIENT_PRIVATE_KEYS_FILE +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +JSON with: + + * signature_private_key (base32) + * encryption_private_key (base32) + * authentication_private_key (base32) + * submitted_ini (boolean) + * submitted_hia (boolean) + +Database schema +--------------- + +.. code-block:: shell-session + + CREATE TABLE incoming_transactions + (incoming_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY + ,amount taler_amount NOT NULL + ,wire_transfer_subject TEXT + ,execution_time INT8 NOT NULL + ,debit_payto_uri TEXT NOT NULL + ,bank_transfer_id TEXT NOT NULL -- EBICS or Depolymerizer (generic) + ,bounced BOOL DEFAULT FALSE -- to track if we bounced it + ); + + CREATE TABLE outgoing_transactions + (outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY + ,amount taler_amount NOT NULL + ,wire_transfer_subject TEXT + ,execution_time INT8 NOT NULL + ,credit_payto_uri TEXT NOT NULL + ,bank_transfer_id TEXT NOT NULL + ); + + CREATE TABLE initiated_outgoing_transactions + (initiated_outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY -- used as our ID in PAIN + ,amount taler_amount NOT NULL + ,wire_transfer_subject TEXT + ,execution_time INT8 NOT NULL + ,credit_payto_uri TEXT NOT NULL + ,out_transaction_id INT8 REFERENCES outgoing_transactions (out_transaction_id) + ,submitted BOOL DEFAULT FALSE + ,hidden BOOL DEFAULT FALSE -- FIXME: exaplain this. + ,client_request_uuid TEXT NOT NULL UNIQUE + ,failure_message TEXT -- NOTE: that may mix soon failures (those found at initiation time), or late failures (those found out along a fetch operation) + ); + + COMMENT ON COLUMN initiated_outgoing_transactions.out_transaction_id + IS 'Points to the bank transaction that was found via nexus-fetch. If "submitted" is false or nexus-fetch could not download this initiation, this column is expected to be NULL.' + +nexus-ebics-setup +----------------- + +The ebics-setup tool performs the following: + + * Checks if the require configuration options are present and well-formed + (like the file names), if not exits with an error message. Given + --check-full-config, also sanity-check the configuration options of the + other subsystems. + + * Checks if the private keys file exists, if not creates new private keys + with flags "not submitted". + + * If any private key flags are set to "not submitted" or a command-line + override is given (--force-keys-resubmission), attempt to submit the + corresponding client public keys to the bank. If the bank accepts the + client public key, update the flags to "submitted". + + * If a public key was submitted or if a command-line override + (--generate-registration-pdf) is given, generate a PDF with the public key + for the user to print and send to the bank. + + * Checks if the public keys of the bank already exist on disk, if not try to + download them with a flag "not accepted". If downloading fails, display a + message asking the user to register the private keys (and give override + options for re-submission of private keys or re-generation of the + registration PDF). + + * If we just downloaded public keys, display the corresponding public keys + (or fingerprints) to the user and ask the user to interactively confirm to + accept them (or auto-accept via --auto-accept-keys). If the user accepted + the public key, update the flag to "accepted". + +nexus-ebics-fetch +----------------- + + * Fetches by default all incoming and outgoing bank transactions and error + messages and inserts them into the Postgres database tables (including + updating the initiated outgoing transaction table). First, considers the + last transactions in the database and fetches statements from that day + forward (inclusive). Afterwards, fetches reports and when the day rolls + over (before committing any transactions with the next day!) also fetches a + final statement of the previous day, thus ensuring we get a statement + every day plus intra-day reports. + + .. note:: + + (1) "from that day forward (inclusive)" must **not** rely on EBICS returning + the unseen messages: that's because they might **already** be downloaded but + never made it to the database. + + (2) "and when the day rolls over". When does a day roll over? => A day rolls + over when the current time is at least on the day after the transaction with the + most recent timestamp that's stored in the database. + + (3) "Afterwards, fetches reports". This must happen **only after** any possible + previous statement got downloaded. + + To summarize: at any point in time the database must contain (the content of) any + possible statement up to the current time, plus any possible report up to the current + time (in case that's not covered by any statement so far). + + * Bounces transactions with mal-formed wire transfer subjects. + + * Optionally logs EBICS messages to disk, one per file, based on + configuration. Filenames must include the timestamp of the download. The + date must be in the path and the time of day at the beginning of the + filename. This will facilitate easy deletion of logs. + + * Optionally only fetches reports (--only-reports) or statements (--only-statements) + or only error messages (--only-failures). + + * Optionally terminates after one fetch (--transient) or re-fetches based + on the configured frequency. + + * Terminates hard (with error code) if incoming transactions are not in the + expected (configured) currency. + + +nexus-ebics-submit +------------------ + + * Generates a payment initiation message for all client-initiated outgoing + transactions that have not yet been initiated. If the server accepts the + message, sets the initiated flag in the table to true. The EBICS order ID + is set to the lowest initiated_outgoing_transaction_id in the transaction + set modulo 2^20 encoded in BASE36. The payment information ID is set to + the initiated_outgoing_transaction_id of each transaction as a text + string. The message identification is set to the lowest + initiated_outgoing_transaction_id plus ("-") the highest + initiated_outgoing_transaction_id as a text string. + + * Optionally terminates after one fetch (--transient) or re-submits based + on the configured frequency. + + * If configured frequency is zero (default), listens to notifications from + nexus-httpd for insertions of outgoing payment initiation records. + + +nexus-httpd +----------- + + * Offers REST APIs as per configuration. + + * Listens to notifications from nexus-ebics-fetch to run facade-logic and + wake-up long pollers. + + * Offers a *new* REST API to list failed (initiated outgoing) transactions + and allows the user to re-initiate those transactions (by creating new + records in the initiated outgoing transactions table). Also allows the + user to set the "hidden" flag on failed transactions to not show them + anymore. + + +Definition of Done +================== + + * Code implemented + * Testcases migrated (including exchange, merchant, etc.) + * Man pages updated + * Manual updated + * Tested with actual banks (especially error handling and idempotency) + * Tested against server-side EBICS mock (to be resurrected) + * Tested against various ISO 20022 messages + + +Alternatives +============ + + * Only run Taler on top of Bitcoin. + +Drawbacks +========= + + * Uses EBICS. + +Discussion / Q&A +================ +(This should be filled in with results from discussions on mailing lists / personal communication.) + +* From private discussion: bouncing goes inside nexus-fetch as + it saves one database event, makes the HTTPd simpler, and lets + the bouncing happen even when no HTTPd runs. + +* Sign-up PDF is ever only generated if *both* INI & HIA have the "submitted" state. + +* What is 'override option for re-submission of private keys?'. + --force-keys-submission already re-submits the keys but it does not + override them. If the user wants new keys, they can easily remove + the keys file on disk. That makes the CLI shorter. + +* Implementation sticks to the IBAN found in the configuration, **if** the bank + does not show any IBAN related to the EBICS subscriber. + +* from nexus-ebics-submit: "if the server accepts the request, sets the + initiated flag in the table to true". May there be a case where the + server accepted the request, but the client never got any response (some + network issue..), and therefore didn't set the submitted flag, ending + up in submitting the payment twice? Also: flagging the payment _after_ + the bank response, may lead double-submission even if the HTTP talk ended + well: it suffices to crash after having received a "200 OK" response but + before setting the submitted flag to the database. + +* the ebics-submit section mentions the EBICS order ID. The following excerpt + was found however at page 88 of the EBICS 3 specifications: + + ``OrderID is only present if a file is transmitted to the bank relating to an order with an + already existing order number (only allowed for AdminOrderType = HVE or HVS)`` + + Nexus does not support HVE or HVS. + +* As of private communication, the responsibility of submitting idempotent payments + relies on the use of ``request_uid`` (a database column of the initiated payment) + as the ``MsgId`` value of the corresponding pain.001 document. + +* ``submitted`` column of an initiated payment evolved into the following enum: + + .. code-block:: shell-session + + CREATE TYPE submission_state AS ENUM ( + 'unsubmitted' + ,'transient_failure' + ,'permanent_failure' + ,'success' + ,'never_heard_back' + ); + + * ``unsubmitted``: default state when a payment is initiated + * ``transient_failure``: submission failed but can be retried, for example after a network issue. + * ``permanent_failure``: EBICS- or bank-technical error codes were not EBICS_OK (nor any tolerated EBICS code like EBICS_NO_DOWNLOAD_DATA_AVAILABLE), never retry. + * ``never_heard_back``: the payment initiation submission has **been** ``success`` but it was never confirmed by any outgoing transaction (from a camt.5x document) or any pain.002 report. It is responsability of a garbage collector to set this state after a particular time period. + +* the initiated_outgoing_transactions table takes two more columns: + ``last_submission_date``, a timestamp in microseconds, and a + ``submission_counter``. Both of them would serve to decide retry + policies. + +* the ``failure_text`` column at the initiated_outgoing_transactions table + should contain a JSON object that contains any useful detail about the problem. + That *could* be modeled after the Taler `ErrorDetail <https://docs.taler.net/core/api-common.html#tsref-type-ErrorDetail>`_, where at least the error code and the hint fields are provided. |