path: root/design-documents/050-libeufin-nexus.rst

DD 50: Libeufin-Nexus
#####################

Summary
=======

This document proposes a new design for the libeufin Nexus component,
focusing on the user-interaction and where what state is to be stored.


Motivation
==========

The existing Nexus design is overly complex to configure, develop and
maintain.  It supports EBICS features we do not need, and lacks key features
(like long-polling) that are absolutely needed.

..
  long-polling at the TWG is NOT NetzBon-critical, as the TWG is only offered
  by the Bank.

We also have several implementations with Nexus, Bank and Depolymerization
subsystems, and it would be good to combine some of them.

Requirements
============

  * Easy to use, high-level abstraction over EBICS details
  * Reduce complexity, no multi-account, multi-connection support
  * No general EBICS client, Taler-specific logic
  * Support for Taler facade, including bouncing of transactions with malformed subject
  * Clear separation between configuration and runtime, minimal runtime footprint
  * No built-in cron-jobs, background tasks runnable via systemd (one-shot and persistent mode)
  * Configuration style same as other GNUnet/Taler components
  * No private keys in database, as in other Taler components
  * Enable future unified implementation with Depolymerization to share database and REST API logic

Proposed Solution
=================

Split up Nexus into four components:

  * nexus-ebics-setup: register account with EBICS server and perform key generation and exchange
  * nexus-ebics-fetch: obtain wire transfers and payment status from EBICS server
  * nexus-ebics-submit: send payment initiation messages to EBICS server
  * nexus-httpd: serve Taler REST APIs (wire gateway API, revenue API) to Taler clients and implement facade logic

All four components should read a simple INI-style configuration file,
possibly with component-specific sections.

Configuration file
------------------

.. code-block:: shell-session

  [nexus-ebics]
  CURRENCY = EUR
  HOST_BASE_URL = http://ebics.bank.com/
  HOST_ID = mybank
  USER_ID = myuser
  PARTNER_ID = myorg
  SYSTEM_ID = banksys
  ACCOUNT_NUMBER = DE1234567890 # This value must identify with how the bank calls the bank account
  BANK_PUBLIC_KEYS_FILE = enc-auth-keys.json
  CLIENT_PRIVATE_KEY_FILE = my-private-key.json
  ACCOUNT_META_DATA_FILE = ebics-meta.json
  EBICS_DIALECT = postfinance # FIXME: banks differs also in ISO20022, not only EBICS.
  
  [nexus-postgres]
  CONFIG = postgres:///libeufin-nexus
  
  [nexus-ebics-fetch]
  FREQUENCY = 30s # used when long-polling is not supported
  STATEMENT_LOG_DIRECTORY = /tmp/ebics-messages/
  
  [nexus-ebics-submit]
  FREQUENCY = 30s # use 0 to always submit immediately (via LISTEN trigger)
  
  [nexus-httpd]
  PORT = 8080
  UNIXPATH =
  SERVE = tcp | unix
  
  [nexus-httpd-wire-gateway-facade]
  ENABLED = YES
  AUTH_METHOD = token
  AUTH_TOKEN = "secret-token:foo"
  
  [nexus-httpd-revenue-facade]
  ENABLED = YES
  AUTH_METHOD = token
  AUTH_TOKEN = "secret-token:foo"


File contents: BANK_PUBLIC_KEYS_FILE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

JSON with 3 fields:

  * bank_encryption_public_key (base32)
  * bank_authentication_public_key (base32)
  * accepted (boolean)

File contents: CLIENT_PRIVATE_KEY_FILE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

JSON with:

  * signature_private_key (base32)
  * encryption_private_key (base32)
  * authentication_private_key (base32)
  * submitted_ini (boolean)
  * submitted_hia (boolean)

File contents: ACCOUNT_META_DATA_FILE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

JSON with:

  * account_holder_iban
  * bank_code
  * account_holder_name

Database schema
---------------

.. code-block:: shell-session

  CREATE TABLE incoming_transactions
    (incoming_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY
    ,amount taler_amount NOT NULL
    ,wire_transfer_subject TEXT
    ,execution_time INT8 NOT NULL
    ,debit_payto_uri TEXT NOT NULL
    ,bank_transfer_id TEXT NOT NULL -- EBICS or Depolymerizer (generic)
    ,bounced BOOL DEFAULT FALSE -- to track if we bounced it
    );
  
  CREATE TABLE outgoing_transactions
    (outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY
    ,amount taler_amount NOT NULL
    ,wire_transfer_subject TEXT
    ,execution_time INT8 NOT NULL
    ,credit_payto_uri TEXT NOT NULL
    ,bank_transfer_id TEXT NOT NULL
    );
  
  CREATE TABLE initiated_outgoing_transactions
    (initiated_outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY -- used as our ID in PAIN
    ,amount taler_amount NOT NULL
    ,wire_transfer_subject TEXT
    ,execution_time INT8 NOT NULL
    ,credit_payto_uri TEXT NOT NULL
    ,out_transaction_id INT8 REFERENCES outgoing_transactions (out_transaction_id)
    ,submitted BOOL DEFAULT FALSE 
    ,hidden BOOL DEFAULT FALSE -- FIXME: exaplain this.
    ,client_request_uuid TEXT NOT NULL UNIQUE
    ,failure_message TEXT -- NOTE: that may mix soon failures (those found at initiation time), or late failures (those found out along a fetch operation)
    );

  COMMENT ON COLUMN initiated_outgoing_transactions.out_transaction_id
    IS 'Points to the bank transaction that was found via nexus-fetch.  If "submitted" is false or nexus-fetch could not download this initiation, this column is expected to be NULL.'

nexus-ebics-setup
-----------------

The ebics-setup tool performs the following:

  * Checks if the require configuration options are present and well-formed
    (like the file names), if not exits with an error message. Given
    --check-full-config, also sanity-check the configuration options of the
    other subsystems.

..
  Not extremely important, but: try to imitate the 'dry run' option offered
  by the exchange, to avoid having a big config-check upfront, and instead
  spread the check along the execution.


  * Checks if the private keys file exists, if not creates new private keys
    with flags "not submitted".

  * If any private key flags are set to "not submitted" or a command-line
    override is given (--force-key-resubmission), attempt to submit the
    corresponding client public keys to the bank. If the bank accepts the
    client public key, update the flags to "submitted".

  * If a public key was submitted or if a command-line override
    (--generate-registration-pdf) is given, generate a PDF with the public key
    for the user to print and send to the bank.

  * Checks if the public keys of the bank already exist on disk, if not try to
    download them with a flag "not accepted". If downloading fails, display a
    message asking the user to register the private keys (and give override
    options for re-submission of private keys or re-generation of the
    registration PDF).

  * If we just downloaded public keys, display the corresponding public keys
    (or fingerprints) to the user and ask the user to interactively confirm to
    accept them (or auto-accept via --auto-accept-keys). If the user accepted
    the public key, update the flag to "accepted".

  * If all of the above has happened and we have accepted public keys from the
    bank, try to download the list of bank accounts associated with the
    connection and check that the configured account number is among them.  On
    success, store the associated meta data in the account meta data file.  On
    failure, show an error message with the list of available accounts (also
    show this list via --show-associated-accounts). If the configured account
    number is available, show a brief "setup ready" message.


nexus-ebics-fetch
-----------------

  * Fetches by default all incoming and outgoing bank transactions and error
    messages and inserts them into the Postgres database tables (including
    updating the initiated outgoing transaction table). First, considers the
    last transactions in the database and fetches statements from that day
    forward (inclusive). Afterwards, fetches reports and when the day rolls
    over (before committing any transactions with the next day!) also fetches a
    final statement of the previous day, thus ensuring we get a statement
    every day plus intra-day reports.

  * Bounces transactions with mal-formed wire transfer subjects.

  * Optionally logs EBICS messages to disk, one per file, based on
    configuration.  Filenames must include the timestamp of the download.  The
    date must be in the path and the time of day at the beginning of the
    filename. This will facilitate easy deletion of logs.

  * Optionally only fetches reports (--only-reports) or statements (--only-statements)
    or only error messages (--only-failures).

  * Optionally terminates after one fetch (--transient) or re-fetches based
    on the configured frequency.

  * Terminates hard (with error code) if incoming transactions are not in the
    expected (configured) currency.


nexus-ebics-submit
------------------

  * Generates a payment initiation message for all client-initiated outgoing
    transactions that have not yet been initiated.  If the server accepts the
    message, sets the initiated flag in the table to true. The EBICS order ID
    is set to the lowest initiated_outgoing_transaction_id in the transaction
    set modulo 2^20 encoded in BASE36.  The payment information ID is set to
    the initiated_outgoing_transaction_id of each transaction as a text
    string.  The message identification is set to the lowest
    initiated_outgoing_transaction_id plus ("-") the highest
    initiated_outgoing_transaction_id as a text string.

  * Optionally terminates after one fetch (--transient) or re-submits based
    on the configured frequency.

  * If configured frequency is zero (default), listens to notifications from
    nexus-httpd for insertions of outgoing payment initiation records.


nexus-httpd
-----------

  * Offers REST APIs as per configuration.

  * Listens to notifications from nexus-ebics-fetch to run facade-logic and
    wake-up long pollers.

  * Offers a *new* REST API to list failed (initiated outgoing) transactions
    and allows the user to re-initiate those transactions (by creating new
    records in the initiated outgoing transactions table). Also allows the
    user to set the "hidden" flag on failed transactions to not show them
    anymore.


Definition of Done
==================

  * Code implemented
  * Testcases migrated (including exchange, merchant, etc.)
  * Man pages updated
  * Manual updated
  * Tested with actual banks (especially error handling and idempotency)
  * Tested against server-side EBICS mock (to be resurrected)
  * Tested against various ISO 20022 messages


Alternatives
============

  * Only run Taler on top of Bitcoin.

Drawbacks
=========

  * Uses EBICS.

Discussion / Q&A
================
(This should be filled in with results from discussions on mailing lists / personal communication.)

From private discussion: bouncing goes inside nexus-fetch as
it saves one database event, makes the HTTPd simpler, and lets
the bouncing happen even when no HTTPd runs.
--