050-libeufin-nexus.rst (14005B)
1 DD 50: Libeufin-Nexus 2 ##################### 3 4 Summary 5 ======= 6 7 This document proposes a new design for the libeufin Nexus component, 8 focusing on the user-interaction and where what state is to be stored. 9 10 11 Motivation 12 ========== 13 14 The existing Nexus design is overly complex to configure, develop and 15 maintain. It supports EBICS features we do not need, and lacks key features 16 (like long-polling) that are absolutely needed. 17 18 .. 19 long-polling at the TWG is NOT NetzBon-critical, as the TWG is only offered 20 by the Bank. 21 22 We also have several implementations with Nexus, Bank and Depolymerization 23 subsystems, and it would be good to combine some of them. 24 25 Requirements 26 ============ 27 28 * Easy to use, high-level abstraction over EBICS details 29 * Reduce complexity, no multi-account, multi-connection support 30 * No general EBICS client, Taler-specific logic 31 * Support for Taler facade, including bouncing of transactions with malformed subject 32 * Clear separation between configuration and runtime, minimal runtime footprint 33 * No built-in cron-jobs, background tasks runnable via systemd (one-shot and persistent mode) 34 * Configuration style same as other GNUnet/Taler components 35 * No private keys in database, as in other Taler components 36 * Enable future unified implementation with Depolymerization to share database and REST API logic 37 38 Proposed Solution 39 ================= 40 41 Split up Nexus into four components: 42 43 * nexus-ebics-setup: register account with EBICS server and perform key generation and exchange 44 * nexus-ebics-fetch: obtain wire transfers and payment status from EBICS server 45 * nexus-ebics-submit: send payment initiation messages to EBICS server 46 * nexus-httpd: serve Taler REST APIs (wire gateway API, revenue API) to Taler clients and implement facade logic 47 48 All four components should read a simple INI-style configuration file, 49 possibly with component-specific sections. 50 51 Configuration file 52 ------------------ 53 54 .. code-block:: shell-session 55 56 [nexus-ebics] 57 CURRENCY = EUR 58 HOST_BASE_URL = http://ebics.bank.com/ 59 HOST_ID = mybank 60 USER_ID = myuser 61 PARTNER_ID = myorg 62 IBAN = MY-IBAN 63 BIC = MY-BIC 64 NAME = MY NAME 65 BANK_PUBLIC_KEYS_FILE = enc-auth-keys.json 66 CLIENT_PRIVATE_KEYS_FILE = my-private-keys.json 67 BANK_DIALECT = postfinance # EBICS+ISO20022 style used by the bank. 68 69 [libeufin-nexusdb-postgres] 70 CONFIG = postgres:///libeufin-nexus 71 72 [nexus-ebics-fetch] 73 FREQUENCY = 30s # used when long-polling is not supported 74 STATEMENT_LOG_DIRECTORY = /tmp/ebics-messages/ 75 76 [nexus-ebics-submit] 77 FREQUENCY = 30s # use 0 to always submit immediately (via LISTEN trigger) 78 79 [nexus-httpd] 80 PORT = 8080 81 UNIXPATH = 82 SERVE = tcp | unix 83 84 [nexus-httpd-wire-gateway-facade] 85 ENABLED = YES 86 AUTH_METHOD = token 87 AUTH_TOKEN = "secret-token:foo" 88 89 [nexus-httpd-revenue-facade] 90 ENABLED = YES 91 AUTH_METHOD = token 92 AUTH_TOKEN = "secret-token:foo" 93 94 95 File contents: BANK_PUBLIC_KEYS_FILE 96 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 97 98 JSON with 3 fields: 99 100 * bank_encryption_public_key (base32) 101 * bank_authentication_public_key (base32) 102 * accepted (boolean) 103 104 File contents: CLIENT_PRIVATE_KEYS_FILE 105 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 106 107 JSON with: 108 109 * signature_private_key (base32) 110 * encryption_private_key (base32) 111 * authentication_private_key (base32) 112 * submitted_ini (boolean) 113 * submitted_hia (boolean) 114 115 Database schema 116 --------------- 117 118 .. code-block:: shell-session 119 120 CREATE TABLE incoming_transactions 121 (incoming_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY 122 ,amount taler_amount NOT NULL 123 ,wire_transfer_subject TEXT 124 ,execution_time INT8 NOT NULL 125 ,debit_payto_uri TEXT NOT NULL 126 ,bank_transfer_id TEXT NOT NULL -- EBICS or Depolymerizer (generic) 127 ,bounced BOOL DEFAULT FALSE -- to track if we bounced it 128 ); 129 130 CREATE TABLE outgoing_transactions 131 (outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY 132 ,amount taler_amount NOT NULL 133 ,wire_transfer_subject TEXT 134 ,execution_time INT8 NOT NULL 135 ,credit_payto_uri TEXT NOT NULL 136 ,bank_transfer_id TEXT NOT NULL 137 ); 138 139 CREATE TABLE initiated_outgoing_transactions 140 (initiated_outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY -- used as our ID in PAIN 141 ,amount taler_amount NOT NULL 142 ,wire_transfer_subject TEXT 143 ,execution_time INT8 NOT NULL 144 ,credit_payto_uri TEXT NOT NULL 145 ,out_transaction_id INT8 REFERENCES outgoing_transactions (out_transaction_id) 146 ,submitted BOOL DEFAULT FALSE 147 ,hidden BOOL DEFAULT FALSE -- FIXME: exaplain this. 148 ,client_request_uuid TEXT NOT NULL UNIQUE 149 ,failure_message TEXT -- NOTE: that may mix soon failures (those found at initiation time), or late failures (those found out along a fetch operation) 150 ); 151 152 COMMENT ON COLUMN initiated_outgoing_transactions.out_transaction_id 153 IS 'Points to the bank transaction that was found via nexus-fetch. If "submitted" is false or nexus-fetch could not download this initiation, this column is expected to be NULL.' 154 155 nexus-ebics-setup 156 ----------------- 157 158 The ebics-setup tool performs the following: 159 160 * Checks if the require configuration options are present and well-formed 161 (like the file names), if not exits with an error message. Given 162 --check-full-config, also sanity-check the configuration options of the 163 other subsystems. 164 165 * Checks if the private keys file exists, if not creates new private keys 166 with flags "not submitted". 167 168 * If any private key flags are set to "not submitted" or a command-line 169 override is given (--force-keys-resubmission), attempt to submit the 170 corresponding client public keys to the bank. If the bank accepts the 171 client public key, update the flags to "submitted". 172 173 * If a public key was submitted or if a command-line override 174 (--generate-registration-pdf) is given, generate a PDF with the public key 175 for the user to print and send to the bank. 176 177 * Checks if the public keys of the bank already exist on disk, if not try to 178 download them with a flag "not accepted". If downloading fails, display a 179 message asking the user to register the private keys (and give override 180 options for re-submission of private keys or re-generation of the 181 registration PDF). 182 183 * If we just downloaded public keys, display the corresponding public keys 184 (or fingerprints) to the user and ask the user to interactively confirm to 185 accept them (or auto-accept via --auto-accept-keys). If the user accepted 186 the public key, update the flag to "accepted". 187 188 nexus-ebics-fetch 189 ----------------- 190 191 * Fetches by default all incoming and outgoing bank transactions and error 192 messages and inserts them into the Postgres database tables (including 193 updating the initiated outgoing transaction table). First, considers the 194 last transactions in the database and fetches statements from that day 195 forward (inclusive). Afterwards, fetches reports and when the day rolls 196 over (before committing any transactions with the next day!) also fetches a 197 final statement of the previous day, thus ensuring we get a statement 198 every day plus intra-day reports. 199 200 .. note:: 201 202 (1) "from that day forward (inclusive)" must **not** rely on EBICS returning 203 the unseen messages: that's because they might **already** be downloaded but 204 never made it to the database. 205 206 (2) "and when the day rolls over". When does a day roll over? => A day rolls 207 over when the current time is at least on the day after the transaction with the 208 most recent timestamp that's stored in the database. 209 210 (3) "Afterwards, fetches reports". This must happen **only after** any possible 211 previous statement got downloaded. 212 213 To summarize: at any point in time the database must contain (the content of) any 214 possible statement up to the current time, plus any possible report up to the current 215 time (in case that's not covered by any statement so far). 216 217 * Bounces transactions with mal-formed wire transfer subjects. 218 219 * Optionally logs EBICS messages to disk, one per file, based on 220 configuration. Filenames must include the timestamp of the download. The 221 date must be in the path and the time of day at the beginning of the 222 filename. This will facilitate easy deletion of logs. 223 224 * Optionally only fetches reports (--only-reports) or statements (--only-statements) 225 or only error messages (--only-failures). 226 227 * Optionally terminates after one fetch (--transient) or re-fetches based 228 on the configured frequency. 229 230 * Terminates hard (with error code) if incoming transactions are not in the 231 expected (configured) currency. 232 233 234 nexus-ebics-submit 235 ------------------ 236 237 * Generates a payment initiation message for all client-initiated outgoing 238 transactions that have not yet been initiated. If the server accepts the 239 message, sets the initiated flag in the table to true. The EBICS order ID 240 is set to the lowest initiated_outgoing_transaction_id in the transaction 241 set modulo 2^20 encoded in BASE36. The payment information ID is set to 242 the initiated_outgoing_transaction_id of each transaction as a text 243 string. The message identification is set to the lowest 244 initiated_outgoing_transaction_id plus ("-") the highest 245 initiated_outgoing_transaction_id as a text string. 246 247 * Optionally terminates after one fetch (--transient) or re-submits based 248 on the configured frequency. 249 250 * If configured frequency is zero (default), listens to notifications from 251 nexus-httpd for insertions of outgoing payment initiation records. 252 253 254 nexus-httpd 255 ----------- 256 257 * Offers REST APIs as per configuration. 258 259 * Listens to notifications from nexus-ebics-fetch to run facade-logic and 260 wake-up long pollers. 261 262 * Offers a *new* REST API to list failed (initiated outgoing) transactions 263 and allows the user to re-initiate those transactions (by creating new 264 records in the initiated outgoing transactions table). Also allows the 265 user to set the "hidden" flag on failed transactions to not show them 266 anymore. 267 268 269 Definition of Done 270 ================== 271 272 * Code implemented 273 * Testcases migrated (including exchange, merchant, etc.) 274 * Man pages updated 275 * Manual updated 276 * Tested with actual banks (especially error handling and idempotency) 277 * Tested against server-side EBICS mock (to be resurrected) 278 * Tested against various ISO 20022 messages 279 280 281 Alternatives 282 ============ 283 284 * Only run Taler on top of Bitcoin. 285 286 Drawbacks 287 ========= 288 289 * Uses EBICS. 290 291 Discussion / Q&A 292 ================ 293 (This should be filled in with results from discussions on mailing lists / personal communication.) 294 295 * From private discussion: bouncing goes inside nexus-fetch as 296 it saves one database event, makes the HTTPd simpler, and lets 297 the bouncing happen even when no HTTPd runs. 298 299 * Sign-up PDF is ever only generated if *both* INI & HIA have the "submitted" state. 300 301 * What is 'override option for re-submission of private keys?'. 302 --force-keys-submission already re-submits the keys but it does not 303 override them. If the user wants new keys, they can easily remove 304 the keys file on disk. That makes the CLI shorter. 305 306 * Implementation sticks to the IBAN found in the configuration, **if** the bank 307 does not show any IBAN related to the EBICS subscriber. 308 309 * from nexus-ebics-submit: "if the server accepts the request, sets the 310 initiated flag in the table to true". May there be a case where the 311 server accepted the request, but the client never got any response (some 312 network issue..), and therefore didn't set the submitted flag, ending 313 up in submitting the payment twice? Also: flagging the payment _after_ 314 the bank response, may lead double-submission even if the HTTP talk ended 315 well: it suffices to crash after having received a "200 OK" response but 316 before setting the submitted flag to the database. 317 318 * the ebics-submit section mentions the EBICS order ID. The following excerpt 319 was found however at page 88 of the EBICS 3 specifications: 320 321 ``OrderID is only present if a file is transmitted to the bank relating to an order with an 322 already existing order number (only allowed for AdminOrderType = HVE or HVS)`` 323 324 Nexus does not support HVE or HVS. 325 326 * As of private communication, the responsibility of submitting idempotent payments 327 relies on the use of ``request_uid`` (a database column of the initiated payment) 328 as the ``MsgId`` value of the corresponding pain.001 document. 329 330 * ``submitted`` column of an initiated payment evolved into the following enum: 331 332 .. code-block:: shell-session 333 334 CREATE TYPE submission_state AS ENUM ( 335 'unsubmitted' 336 ,'transient_failure' 337 ,'permanent_failure' 338 ,'success' 339 ,'never_heard_back' 340 ); 341 342 * ``unsubmitted``: default state when a payment is initiated 343 * ``transient_failure``: submission failed but can be retried, for example after a network issue. 344 * ``permanent_failure``: EBICS- or bank-technical error codes were not EBICS_OK (nor any tolerated EBICS code like EBICS_NO_DOWNLOAD_DATA_AVAILABLE), never retry. 345 * ``never_heard_back``: the payment initiation submission has **been** ``success`` but it was never confirmed by any outgoing transaction (from a camt.5x document) or any pain.002 report. It is responsability of a garbage collector to set this state after a particular time period. 346 347 * the initiated_outgoing_transactions table takes two more columns: 348 ``last_submission_date``, a timestamp in microseconds, and a 349 ``submission_counter``. Both of them would serve to decide retry 350 policies. 351 352 * the ``failure_text`` column at the initiated_outgoing_transactions table 353 should contain a JSON object that contains any useful detail about the problem. 354 That *could* be modeled after the Taler `ErrorDetail <https://docs.taler.net/core/api-common.html#tsref-type-ErrorDetail>`_, where at least the error code and the hint fields are provided.