path: root/design-documents/050-libeufin-nexus.rst

DD 50: Libeufin-Nexus
#####################

Summary
=======

This document proposes a new design for the libeufin Nexus component,
focusing on the user-interaction and where what state is to be stored.


Motivation
==========

The existing Nexus design is overly complex to configure, develop and
maintain.  It supports EBICS features we do not need, and lacks key features
(like long-polling) that are absolutely needed.

..
  long-polling at the TWG is NOT NetzBon-critical, as the TWG is only offered
  by the Bank.

We also have several implementations with Nexus, Bank and Depolymerization
subsystems, and it would be good to combine some of them.

Requirements
============

  * Easy to use, high-level abstraction over EBICS details
  * Reduce complexity, no multi-account, multi-connection support
  * No general EBICS client, Taler-specific logic
  * Support for Taler facade, including bouncing of transactions with malformed subject
  * Clear separation between configuration and runtime, minimal runtime footprint
  * No built-in cron-jobs, background tasks runnable via systemd (one-shot and persistent mode)
  * Configuration style same as other GNUnet/Taler components
  * No private keys in database, as in other Taler components
  * Enable future unified implementation with Depolymerization to share database and REST API logic

Proposed Solution
=================

Split up Nexus into four components:

  * nexus-ebics-setup: register account with EBICS server and perform key generation and exchange
  * nexus-ebics-fetch: obtain wire transfers and payment status from EBICS server
  * nexus-ebics-submit: send payment initiation messages to EBICS server
  * nexus-httpd: serve Taler REST APIs (wire gateway API, revenue API) to Taler clients and implement facade logic

All four components should read a simple INI-style configuration file,
possibly with component-specific sections.

Configuration file
------------------

.. code-block:: shell-session

  [nexus-ebics]
  CURRENCY = EUR
  HOST_BASE_URL = http://ebics.bank.com/
  HOST_ID = mybank
  USER_ID = myuser
  PARTNER_ID = myorg
  IBAN = MY-IBAN
  BIC = MY-BIC
  NAME = MY NAME
  BANK_PUBLIC_KEYS_FILE = enc-auth-keys.json
  CLIENT_PRIVATE_KEYS_FILE = my-private-keys.json
  BANK_DIALECT = postfinance # EBICS+ISO20022 style used by the bank.
  
  [nexus-postgres]
  CONFIG = postgres:///libeufin-nexus
  
  [nexus-ebics-fetch]
  FREQUENCY = 30s # used when long-polling is not supported
  STATEMENT_LOG_DIRECTORY = /tmp/ebics-messages/
  
  [nexus-ebics-submit]
  FREQUENCY = 30s # use 0 to always submit immediately (via LISTEN trigger)
  
  [nexus-httpd]
  PORT = 8080
  UNIXPATH =
  SERVE = tcp | unix
  
  [nexus-httpd-wire-gateway-facade]
  ENABLED = YES
  AUTH_METHOD = token
  AUTH_TOKEN = "secret-token:foo"
  
  [nexus-httpd-revenue-facade]
  ENABLED = YES
  AUTH_METHOD = token
  AUTH_TOKEN = "secret-token:foo"


File contents: BANK_PUBLIC_KEYS_FILE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

JSON with 3 fields:

  * bank_encryption_public_key (base32)
  * bank_authentication_public_key (base32)
  * accepted (boolean)

File contents: CLIENT_PRIVATE_KEYS_FILE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

JSON with:

  * signature_private_key (base32)
  * encryption_private_key (base32)
  * authentication_private_key (base32)
  * submitted_ini (boolean)
  * submitted_hia (boolean)

Database schema
---------------

.. code-block:: shell-session

  CREATE TABLE incoming_transactions
    (incoming_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY
    ,amount taler_amount NOT NULL
    ,wire_transfer_subject TEXT
    ,execution_time INT8 NOT NULL
    ,debit_payto_uri TEXT NOT NULL
    ,bank_transfer_id TEXT NOT NULL -- EBICS or Depolymerizer (generic)
    ,bounced BOOL DEFAULT FALSE -- to track if we bounced it
    );
  
  CREATE TABLE outgoing_transactions
    (outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY
    ,amount taler_amount NOT NULL
    ,wire_transfer_subject TEXT
    ,execution_time INT8 NOT NULL
    ,credit_payto_uri TEXT NOT NULL
    ,bank_transfer_id TEXT NOT NULL
    );
  
  CREATE TABLE initiated_outgoing_transactions
    (initiated_outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY -- used as our ID in PAIN
    ,amount taler_amount NOT NULL
    ,wire_transfer_subject TEXT
    ,execution_time INT8 NOT NULL
    ,credit_payto_uri TEXT NOT NULL
    ,out_transaction_id INT8 REFERENCES outgoing_transactions (out_transaction_id)
    ,submitted BOOL DEFAULT FALSE 
    ,hidden BOOL DEFAULT FALSE -- FIXME: exaplain this.
    ,client_request_uuid TEXT NOT NULL UNIQUE
    ,failure_message TEXT -- NOTE: that may mix soon failures (those found at initiation time), or late failures (those found out along a fetch operation)
    );

  COMMENT ON COLUMN initiated_outgoing_transactions.out_transaction_id
    IS 'Points to the bank transaction that was found via nexus-fetch.  If "submitted" is false or nexus-fetch could not download this initiation, this column is expected to be NULL.'

nexus-ebics-setup
-----------------

The ebics-setup tool performs the following:

  * Checks if the require configuration options are present and well-formed
    (like the file names), if not exits with an error message. Given
    --check-full-config, also sanity-check the configuration options of the
    other subsystems.

  * Checks if the private keys file exists, if not creates new private keys
    with flags "not submitted".

  * If any private key flags are set to "not submitted" or a command-line
    override is given (--force-keys-resubmission), attempt to submit the
    corresponding client public keys to the bank. If the bank accepts the
    client public key, update the flags to "submitted".

  * If a public key was submitted or if a command-line override
    (--generate-registration-pdf) is given, generate a PDF with the public key
    for the user to print and send to the bank.

  * Checks if the public keys of the bank already exist on disk, if not try to
    download them with a flag "not accepted". If downloading fails, display a
    message asking the user to register the private keys (and give override
    options for re-submission of private keys or re-generation of the
    registration PDF).

  * If we just downloaded public keys, display the corresponding public keys
    (or fingerprints) to the user and ask the user to interactively confirm to
    accept them (or auto-accept via --auto-accept-keys). If the user accepted
    the public key, update the flag to "accepted".

nexus-ebics-fetch
-----------------

  * Fetches by default all incoming and outgoing bank transactions and error
    messages and inserts them into the Postgres database tables (including
    updating the initiated outgoing transaction table). First, considers the
    last transactions in the database and fetches statements from that day
    forward (inclusive). Afterwards, fetches reports and when the day rolls
    over (before committing any transactions with the next day!) also fetches a
    final statement of the previous day, thus ensuring we get a statement
    every day plus intra-day reports.
    
    .. note::
      
      (1) "from that day forward (inclusive)" must **not** rely on EBICS returning
      the unseen messages: that's because they might **already** be downloaded but
      never made it to the database.
      
      (2) "and when the day rolls over".  When does a day roll over? => A day rolls
      over when the current time is at least on the day after the transaction with the
      most recent timestamp that's stored in the database.

      (3) "Afterwards, fetches reports".  This must happen **only after** any possible
      previous statement got downloaded.

      To summarize: at any point in time the database must contain (the content of) any
      possible statement up to the current time, plus any possible report up to the current
      time (in case that's not covered by any statement so far).

  * Bounces transactions with mal-formed wire transfer subjects.

  * Optionally logs EBICS messages to disk, one per file, based on
    configuration.  Filenames must include the timestamp of the download.  The
    date must be in the path and the time of day at the beginning of the
    filename. This will facilitate easy deletion of logs.

  * Optionally only fetches reports (--only-reports) or statements (--only-statements)
    or only error messages (--only-failures).

  * Optionally terminates after one fetch (--transient) or re-fetches based
    on the configured frequency.

  * Terminates hard (with error code) if incoming transactions are not in the
    expected (configured) currency.


nexus-ebics-submit
------------------

  * Generates a payment initiation message for all client-initiated outgoing
    transactions that have not yet been initiated.  If the server accepts the
    message, sets the initiated flag in the table to true. The EBICS order ID
    is set to the lowest initiated_outgoing_transaction_id in the transaction
    set modulo 2^20 encoded in BASE36.  The payment information ID is set to
    the initiated_outgoing_transaction_id of each transaction as a text
    string.  The message identification is set to the lowest
    initiated_outgoing_transaction_id plus ("-") the highest
    initiated_outgoing_transaction_id as a text string.

  * Optionally terminates after one fetch (--transient) or re-submits based
    on the configured frequency.

  * If configured frequency is zero (default), listens to notifications from
    nexus-httpd for insertions of outgoing payment initiation records.


nexus-httpd
-----------

  * Offers REST APIs as per configuration.

  * Listens to notifications from nexus-ebics-fetch to run facade-logic and
    wake-up long pollers.

  * Offers a *new* REST API to list failed (initiated outgoing) transactions
    and allows the user to re-initiate those transactions (by creating new
    records in the initiated outgoing transactions table). Also allows the
    user to set the "hidden" flag on failed transactions to not show them
    anymore.


Definition of Done
==================

  * Code implemented
  * Testcases migrated (including exchange, merchant, etc.)
  * Man pages updated
  * Manual updated
  * Tested with actual banks (especially error handling and idempotency)
  * Tested against server-side EBICS mock (to be resurrected)
  * Tested against various ISO 20022 messages


Alternatives
============

  * Only run Taler on top of Bitcoin.

Drawbacks
=========

  * Uses EBICS.

Discussion / Q&A
================
(This should be filled in with results from discussions on mailing lists / personal communication.)

* From private discussion: bouncing goes inside nexus-fetch as
  it saves one database event, makes the HTTPd simpler, and lets
  the bouncing happen even when no HTTPd runs.

* Sign-up PDF is ever only generated if *both* INI & HIA have the "submitted" state.

* What is 'override option for re-submission of private keys?'.
  --force-keys-submission already re-submits the keys but it does not
  override them.  If the user wants new keys, they can easily remove
  the keys file on disk.  That makes the CLI shorter.

* Implementation sticks to the IBAN found in the configuration, **if** the bank
  does not show any IBAN related to the EBICS subscriber.

* from nexus-ebics-submit: "if the server accepts the request, sets the
  initiated flag in the table to true".  May there be a case where the
  server accepted the request, but the client never got any response (some
  network issue..), and therefore didn't set the submitted flag, ending
  up in submitting the payment twice?  Also: flagging the payment _after_
  the bank response, may lead double-submission even if the HTTP talk ended
  well: it suffices to crash after having received a "200 OK" response but
  before setting the submitted flag to the database.

* the ebics-submit section mentions the EBICS order ID.  The following excerpt
  was found however at page 88 of the EBICS 3 specifications: 

  ``OrderID is only present if a file is transmitted to the bank relating to an order with an
  already existing order number (only allowed for AdminOrderType = HVE or HVS)``

  Nexus does not support HVE or HVS.

* As of private communication, the responsibility of submitting idempotent payments
  relies on the use of ``request_uid`` (a database column of the initiated payment)
  as the ``MsgId`` value of the corresponding pain.001 document.

* ``submitted`` column of an initiated payment evolved into the following enum:
  
  .. code-block:: shell-session
    
    CREATE TYPE submission_state AS ENUM (
      'unsubmitted'
      ,'transient_failure'
      ,'permanent_failure'
      ,'success'
      ,'never_heard_back'
    );

  * ``unsubmitted``: default state when a payment is initiated
  * ``transient_failure``: submission failed but can be retried, for example after a network issue.
  * ``permanent_failure``: EBICS- or bank-technical error codes were not EBICS_OK (nor any tolerated EBICS code like EBICS_NO_DOWNLOAD_DATA_AVAILABLE), never retry.  
  * ``never_heard_back``: the payment initiation submission has **been** ``success`` but it was never confirmed by any outgoing transaction (from a camt.5x document) or any pain.002 report.  It is responsability of a garbage collector to set this state after a particular time period.

* the initiated_outgoing_transactions table takes two more columns:
  ``last_submission_date``, a timestamp in microseconds, and a
  ``submission_counter``.  Both of them would serve to decide retry
  policies.

* the ``failure_text`` column at the initiated_outgoing_transactions table
  should contain a JSON object that contains any useful detail about the problem.
  That *could* be modeled after the Taler `ErrorDetail <https://docs.taler.net/core/api-common.html#tsref-type-ErrorDetail>`_, where at least the error code and the hint fields are provided.