DD 50: Libeufin-Nexus ##################### Summary ======= This document proposes a new design for the libeufin Nexus component, focusing on the user-interaction and where what state is to be stored. Motivation ========== The existing Nexus design is overly complex to configure, develop and maintain. It supports EBICS features we do not need, and lacks key features (like long-polling) that are absolutely needed. .. long-polling at the TWG is NOT NetzBon-critical, as the TWG is only offered by the Bank. We also have several implementations with Nexus, Bank and Depolymerization subsystems, and it would be good to combine some of them. Requirements ============ * Easy to use, high-level abstraction over EBICS details * Reduce complexity, no multi-account, multi-connection support * No general EBICS client, Taler-specific logic * Support for Taler facade, including bouncing of transactions with malformed subject * Clear separation between configuration and runtime, minimal runtime footprint * No built-in cron-jobs, background tasks runnable via systemd (one-shot and persistent mode) * Configuration style same as other GNUnet/Taler components * No private keys in database, as in other Taler components * Enable future unified implementation with Depolymerization to share database and REST API logic Proposed Solution ================= Split up Nexus into four components: * nexus-ebics-setup: register account with EBICS server and perform key generation and exchange * nexus-ebics-fetch: obtain wire transfers and payment status from EBICS server * nexus-ebics-submit: send payment initiation messages to EBICS server * nexus-httpd: serve Taler REST APIs (wire gateway API, revenue API) to Taler clients and implement facade logic All four components should read a simple INI-style configuration file, possibly with component-specific sections. Configuration file ------------------ .. code-block:: shell-session [nexus-ebics] CURRENCY = EUR HOST_BASE_URL = http://ebics.bank.com/ HOST_ID = mybank USER_ID = myuser PARTNER_ID = myorg SYSTEM_ID = banksys ACCOUNT_NUMBER = DE1234567890 # This value must identify with how the bank calls the bank account BANK_PUBLIC_KEYS_FILE = enc-auth-keys.json CLIENT_PRIVATE_KEYS_FILE = my-private-keys.json ACCOUNT_META_DATA_FILE = ebics-meta.json BANK_DIALECT = postfinance # EBICS+ISO20022 style used by the bank. [nexus-postgres] CONFIG = postgres:///libeufin-nexus [nexus-ebics-fetch] FREQUENCY = 30s # used when long-polling is not supported STATEMENT_LOG_DIRECTORY = /tmp/ebics-messages/ [nexus-ebics-submit] FREQUENCY = 30s # use 0 to always submit immediately (via LISTEN trigger) [nexus-httpd] PORT = 8080 UNIXPATH = SERVE = tcp | unix [nexus-httpd-wire-gateway-facade] ENABLED = YES AUTH_METHOD = token AUTH_TOKEN = "secret-token:foo" [nexus-httpd-revenue-facade] ENABLED = YES AUTH_METHOD = token AUTH_TOKEN = "secret-token:foo" File contents: BANK_PUBLIC_KEYS_FILE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ JSON with 3 fields: * bank_encryption_public_key (base32) * bank_authentication_public_key (base32) * accepted (boolean) File contents: CLIENT_PRIVATE_KEYS_FILE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ JSON with: * signature_private_key (base32) * encryption_private_key (base32) * authentication_private_key (base32) * submitted_ini (boolean) * submitted_hia (boolean) File contents: ACCOUNT_META_DATA_FILE ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ JSON with: * account_holder_iban * bank_code * account_holder_name Database schema --------------- .. code-block:: shell-session CREATE TABLE incoming_transactions (incoming_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY ,amount taler_amount NOT NULL ,wire_transfer_subject TEXT ,execution_time INT8 NOT NULL ,debit_payto_uri TEXT NOT NULL ,bank_transfer_id TEXT NOT NULL -- EBICS or Depolymerizer (generic) ,bounced BOOL DEFAULT FALSE -- to track if we bounced it ); CREATE TABLE outgoing_transactions (outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY ,amount taler_amount NOT NULL ,wire_transfer_subject TEXT ,execution_time INT8 NOT NULL ,credit_payto_uri TEXT NOT NULL ,bank_transfer_id TEXT NOT NULL ); CREATE TABLE initiated_outgoing_transactions (initiated_outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY -- used as our ID in PAIN ,amount taler_amount NOT NULL ,wire_transfer_subject TEXT ,execution_time INT8 NOT NULL ,credit_payto_uri TEXT NOT NULL ,out_transaction_id INT8 REFERENCES outgoing_transactions (out_transaction_id) ,submitted BOOL DEFAULT FALSE ,hidden BOOL DEFAULT FALSE -- FIXME: exaplain this. ,client_request_uuid TEXT NOT NULL UNIQUE ,failure_message TEXT -- NOTE: that may mix soon failures (those found at initiation time), or late failures (those found out along a fetch operation) ); COMMENT ON COLUMN initiated_outgoing_transactions.out_transaction_id IS 'Points to the bank transaction that was found via nexus-fetch. If "submitted" is false or nexus-fetch could not download this initiation, this column is expected to be NULL.' nexus-ebics-setup ----------------- The ebics-setup tool performs the following: * Checks if the require configuration options are present and well-formed (like the file names), if not exits with an error message. Given --check-full-config, also sanity-check the configuration options of the other subsystems. * Checks if the private keys file exists, if not creates new private keys with flags "not submitted". * If any private key flags are set to "not submitted" or a command-line override is given (--force-key-resubmission), attempt to submit the corresponding client public keys to the bank. If the bank accepts the client public key, update the flags to "submitted". * If a public key was submitted or if a command-line override (--generate-registration-pdf) is given, generate a PDF with the public key for the user to print and send to the bank. * Checks if the public keys of the bank already exist on disk, if not try to download them with a flag "not accepted". If downloading fails, display a message asking the user to register the private keys (and give override options for re-submission of private keys or re-generation of the registration PDF). * If we just downloaded public keys, display the corresponding public keys (or fingerprints) to the user and ask the user to interactively confirm to accept them (or auto-accept via --auto-accept-keys). If the user accepted the public key, update the flag to "accepted". * If all of the above has happened and we have accepted public keys from the bank, try to download the list of bank accounts associated with the connection and check that the configured account number is among them. On success, store the associated meta data in the account meta data file. On failure, show an error message with the list of available accounts (also show this list via --show-associated-accounts). If the configured account number is available, show a brief "setup ready" message. nexus-ebics-fetch ----------------- * Fetches by default all incoming and outgoing bank transactions and error messages and inserts them into the Postgres database tables (including updating the initiated outgoing transaction table). First, considers the last transactions in the database and fetches statements from that day forward (inclusive). Afterwards, fetches reports and when the day rolls over (before committing any transactions with the next day!) also fetches a final statement of the previous day, thus ensuring we get a statement every day plus intra-day reports. * Bounces transactions with mal-formed wire transfer subjects. * Optionally logs EBICS messages to disk, one per file, based on configuration. Filenames must include the timestamp of the download. The date must be in the path and the time of day at the beginning of the filename. This will facilitate easy deletion of logs. * Optionally only fetches reports (--only-reports) or statements (--only-statements) or only error messages (--only-failures). * Optionally terminates after one fetch (--transient) or re-fetches based on the configured frequency. * Terminates hard (with error code) if incoming transactions are not in the expected (configured) currency. nexus-ebics-submit ------------------ * Generates a payment initiation message for all client-initiated outgoing transactions that have not yet been initiated. If the server accepts the message, sets the initiated flag in the table to true. The EBICS order ID is set to the lowest initiated_outgoing_transaction_id in the transaction set modulo 2^20 encoded in BASE36. The payment information ID is set to the initiated_outgoing_transaction_id of each transaction as a text string. The message identification is set to the lowest initiated_outgoing_transaction_id plus ("-") the highest initiated_outgoing_transaction_id as a text string. * Optionally terminates after one fetch (--transient) or re-submits based on the configured frequency. * If configured frequency is zero (default), listens to notifications from nexus-httpd for insertions of outgoing payment initiation records. nexus-httpd ----------- * Offers REST APIs as per configuration. * Listens to notifications from nexus-ebics-fetch to run facade-logic and wake-up long pollers. * Offers a *new* REST API to list failed (initiated outgoing) transactions and allows the user to re-initiate those transactions (by creating new records in the initiated outgoing transactions table). Also allows the user to set the "hidden" flag on failed transactions to not show them anymore. Definition of Done ================== * Code implemented * Testcases migrated (including exchange, merchant, etc.) * Man pages updated * Manual updated * Tested with actual banks (especially error handling and idempotency) * Tested against server-side EBICS mock (to be resurrected) * Tested against various ISO 20022 messages Alternatives ============ * Only run Taler on top of Bitcoin. Drawbacks ========= * Uses EBICS. Discussion / Q&A ================ (This should be filled in with results from discussions on mailing lists / personal communication.) From private discussion: bouncing goes inside nexus-fetch as it saves one database event, makes the HTTPd simpler, and lets the bouncing happen even when no HTTPd runs. --