050-libeufin-nexus.rst (14842B)
1 DD 50: Libeufin-Nexus 2 ##################### 3 4 :Design status: Superseded 5 :Implementation status: Implemented 6 :DD shepherd: TBD 7 :Historical contributors: Christian Grothoff, Marcello Stanisci, Antoine A 8 :First published: 2023-09-15 9 :Last substantive change: 2024-06-11 10 :Implementation evidence: libeufin (2023-10-18) 11 :Superseded by: Current LibEuFin Nexus implementation and DD 58 transaction identifiers 12 :Normative references: ``libeufin/nexus-manual.rst``, ``manpages/libeufin-nexus.1.rst`` 13 14 .. warning:: 15 16 This document records the design that initiated the Nexus rewrite. It is 17 non-normative: current Nexus uses one command with subcommands, current 18 configuration names differ, and the transaction identifier schema evolved 19 as described by DD 58. Consult the LibEuFin manuals instead. 20 21 Summary 22 ======= 23 24 This document proposes a new design for the libeufin Nexus component, 25 focusing on the user-interaction and where what state is to be stored. 26 27 28 Motivation 29 ========== 30 31 The existing Nexus design is overly complex to configure, develop and 32 maintain. It supports EBICS features we do not need, and lacks key features 33 (like long-polling) that are absolutely needed. 34 35 .. 36 long-polling at the TWG is NOT NetzBon-critical, as the TWG is only offered 37 by the Bank. 38 39 We also have several implementations with Nexus, Bank and Depolymerization 40 subsystems, and it would be good to combine some of them. 41 42 Requirements 43 ============ 44 45 * Easy to use, high-level abstraction over EBICS details 46 * Reduce complexity, no multi-account, multi-connection support 47 * No general EBICS client, Taler-specific logic 48 * Support for Taler facade, including bouncing of transactions with malformed subject 49 * Clear separation between configuration and runtime, minimal runtime footprint 50 * No built-in cron-jobs, background tasks runnable via systemd (one-shot and persistent mode) 51 * Configuration style same as other GNUnet/Taler components 52 * No private keys in database, as in other Taler components 53 * Enable future unified implementation with Depolymerization to share database and REST API logic 54 55 Proposed Solution 56 ================= 57 58 Split up Nexus into four components: 59 60 * nexus-ebics-setup: register account with EBICS server and perform key generation and exchange 61 * nexus-ebics-fetch: obtain wire transfers and payment status from EBICS server 62 * nexus-ebics-submit: send payment initiation messages to EBICS server 63 * nexus-httpd: serve Taler REST APIs (wire gateway API, revenue API) to Taler clients and implement facade logic 64 65 All four components should read a simple INI-style configuration file, 66 possibly with component-specific sections. 67 68 Configuration file 69 ------------------ 70 71 .. code-block:: shell-session 72 73 [nexus-ebics] 74 CURRENCY = EUR 75 HOST_BASE_URL = http://ebics.bank.com/ 76 HOST_ID = mybank 77 USER_ID = myuser 78 PARTNER_ID = myorg 79 IBAN = MY-IBAN 80 BIC = MY-BIC 81 NAME = MY NAME 82 BANK_PUBLIC_KEYS_FILE = enc-auth-keys.json 83 CLIENT_PRIVATE_KEYS_FILE = my-private-keys.json 84 BANK_DIALECT = postfinance # EBICS+ISO20022 style used by the bank. 85 86 [libeufin-nexusdb-postgres] 87 CONFIG = postgres:///libeufin-nexus 88 89 [nexus-ebics-fetch] 90 FREQUENCY = 30s # used when long-polling is not supported 91 STATEMENT_LOG_DIRECTORY = /tmp/ebics-messages/ 92 93 [nexus-ebics-submit] 94 FREQUENCY = 30s # use 0 to always submit immediately (via LISTEN trigger) 95 96 [nexus-httpd] 97 PORT = 8080 98 UNIXPATH = 99 SERVE = tcp | unix 100 101 [nexus-httpd-wire-gateway-facade] 102 ENABLED = YES 103 AUTH_METHOD = token 104 AUTH_TOKEN = "secret-token:foo" 105 106 [nexus-httpd-revenue-facade] 107 ENABLED = YES 108 AUTH_METHOD = token 109 AUTH_TOKEN = "secret-token:foo" 110 111 112 File contents: BANK_PUBLIC_KEYS_FILE 113 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 114 115 JSON with 3 fields: 116 117 * bank_encryption_public_key (base32) 118 * bank_authentication_public_key (base32) 119 * accepted (boolean) 120 121 File contents: CLIENT_PRIVATE_KEYS_FILE 122 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 123 124 JSON with: 125 126 * signature_private_key (base32) 127 * encryption_private_key (base32) 128 * authentication_private_key (base32) 129 * submitted_ini (boolean) 130 * submitted_hia (boolean) 131 132 Database schema 133 --------------- 134 135 .. code-block:: shell-session 136 137 CREATE TABLE incoming_transactions 138 (incoming_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY 139 ,amount taler_amount NOT NULL 140 ,wire_transfer_subject TEXT 141 ,execution_time INT8 NOT NULL 142 ,debit_payto_uri TEXT NOT NULL 143 ,bank_transfer_id TEXT NOT NULL -- EBICS or Depolymerizer (generic) 144 ,bounced BOOL DEFAULT FALSE -- to track if we bounced it 145 ); 146 147 CREATE TABLE outgoing_transactions 148 (outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY 149 ,amount taler_amount NOT NULL 150 ,wire_transfer_subject TEXT 151 ,execution_time INT8 NOT NULL 152 ,credit_payto_uri TEXT NOT NULL 153 ,bank_transfer_id TEXT NOT NULL 154 ); 155 156 CREATE TABLE initiated_outgoing_transactions 157 (initiated_outgoing_transaction_id INT8 GENERATED BY DEFAULT AS IDENTITY -- used as our ID in PAIN 158 ,amount taler_amount NOT NULL 159 ,wire_transfer_subject TEXT 160 ,execution_time INT8 NOT NULL 161 ,credit_payto_uri TEXT NOT NULL 162 ,out_transaction_id INT8 REFERENCES outgoing_transactions (out_transaction_id) 163 ,submitted BOOL DEFAULT FALSE 164 ,hidden BOOL DEFAULT FALSE -- FIXME: exaplain this. 165 ,client_request_uuid TEXT NOT NULL UNIQUE 166 ,failure_message TEXT -- NOTE: that may mix soon failures (those found at initiation time), or late failures (those found out along a fetch operation) 167 ); 168 169 COMMENT ON COLUMN initiated_outgoing_transactions.out_transaction_id 170 IS 'Points to the bank transaction that was found via nexus-fetch. If "submitted" is false or nexus-fetch could not download this initiation, this column is expected to be NULL.' 171 172 nexus-ebics-setup 173 ----------------- 174 175 The ebics-setup tool performs the following: 176 177 * Checks if the require configuration options are present and well-formed 178 (like the file names), if not exits with an error message. Given 179 --check-full-config, also sanity-check the configuration options of the 180 other subsystems. 181 182 * Checks if the private keys file exists, if not creates new private keys 183 with flags "not submitted". 184 185 * If any private key flags are set to "not submitted" or a command-line 186 override is given (--force-keys-resubmission), attempt to submit the 187 corresponding client public keys to the bank. If the bank accepts the 188 client public key, update the flags to "submitted". 189 190 * If a public key was submitted or if a command-line override 191 (--generate-registration-pdf) is given, generate a PDF with the public key 192 for the user to print and send to the bank. 193 194 * Checks if the public keys of the bank already exist on disk, if not try to 195 download them with a flag "not accepted". If downloading fails, display a 196 message asking the user to register the private keys (and give override 197 options for re-submission of private keys or re-generation of the 198 registration PDF). 199 200 * If we just downloaded public keys, display the corresponding public keys 201 (or fingerprints) to the user and ask the user to interactively confirm to 202 accept them (or auto-accept via --auto-accept-keys). If the user accepted 203 the public key, update the flag to "accepted". 204 205 nexus-ebics-fetch 206 ----------------- 207 208 * Fetches by default all incoming and outgoing bank transactions and error 209 messages and inserts them into the Postgres database tables (including 210 updating the initiated outgoing transaction table). First, considers the 211 last transactions in the database and fetches statements from that day 212 forward (inclusive). Afterwards, fetches reports and when the day rolls 213 over (before committing any transactions with the next day!) also fetches a 214 final statement of the previous day, thus ensuring we get a statement 215 every day plus intra-day reports. 216 217 .. note:: 218 219 (1) "from that day forward (inclusive)" must **not** rely on EBICS returning 220 the unseen messages: that's because they might **already** be downloaded but 221 never made it to the database. 222 223 (2) "and when the day rolls over". When does a day roll over? => A day rolls 224 over when the current time is at least on the day after the transaction with the 225 most recent timestamp that's stored in the database. 226 227 (3) "Afterwards, fetches reports". This must happen **only after** any possible 228 previous statement got downloaded. 229 230 To summarize: at any point in time the database must contain (the content of) any 231 possible statement up to the current time, plus any possible report up to the current 232 time (in case that's not covered by any statement so far). 233 234 * Bounces transactions with mal-formed wire transfer subjects. 235 236 * Optionally logs EBICS messages to disk, one per file, based on 237 configuration. Filenames must include the timestamp of the download. The 238 date must be in the path and the time of day at the beginning of the 239 filename. This will facilitate easy deletion of logs. 240 241 * Optionally only fetches reports (--only-reports) or statements (--only-statements) 242 or only error messages (--only-failures). 243 244 * Optionally terminates after one fetch (--transient) or re-fetches based 245 on the configured frequency. 246 247 * Terminates hard (with error code) if incoming transactions are not in the 248 expected (configured) currency. 249 250 251 nexus-ebics-submit 252 ------------------ 253 254 * Generates a payment initiation message for all client-initiated outgoing 255 transactions that have not yet been initiated. If the server accepts the 256 message, sets the initiated flag in the table to true. The EBICS order ID 257 is set to the lowest initiated_outgoing_transaction_id in the transaction 258 set modulo 2^20 encoded in BASE36. The payment information ID is set to 259 the initiated_outgoing_transaction_id of each transaction as a text 260 string. The message identification is set to the lowest 261 initiated_outgoing_transaction_id plus ("-") the highest 262 initiated_outgoing_transaction_id as a text string. 263 264 * Optionally terminates after one fetch (--transient) or re-submits based 265 on the configured frequency. 266 267 * If configured frequency is zero (default), listens to notifications from 268 nexus-httpd for insertions of outgoing payment initiation records. 269 270 271 nexus-httpd 272 ----------- 273 274 * Offers REST APIs as per configuration. 275 276 * Listens to notifications from nexus-ebics-fetch to run facade-logic and 277 wake-up long pollers. 278 279 * Offers a *new* REST API to list failed (initiated outgoing) transactions 280 and allows the user to re-initiate those transactions (by creating new 281 records in the initiated outgoing transactions table). Also allows the 282 user to set the "hidden" flag on failed transactions to not show them 283 anymore. 284 285 286 Definition of Done 287 ================== 288 289 * [x] Code implemented 290 * [ ] Migration of all historical exchange and merchant test cases verified 291 * [x] Man pages updated 292 * [x] Manual updated 293 * [ ] Testing with actual banks, including error handling and idempotency, documented 294 * [ ] Testing against a maintained server-side EBICS mock verified 295 * [ ] Coverage across the intended ISO 20022 message variants documented 296 297 298 Alternatives 299 ============ 300 301 * Only run Taler on top of Bitcoin. 302 303 Drawbacks 304 ========= 305 306 * Uses EBICS. 307 308 Discussion / Q&A 309 ================ 310 (This should be filled in with results from discussions on mailing lists / personal communication.) 311 312 * From private discussion: bouncing goes inside nexus-fetch as 313 it saves one database event, makes the HTTPd simpler, and lets 314 the bouncing happen even when no HTTPd runs. 315 316 * Sign-up PDF is ever only generated if *both* INI & HIA have the "submitted" state. 317 318 * What is 'override option for re-submission of private keys?'. 319 --force-keys-submission already re-submits the keys but it does not 320 override them. If the user wants new keys, they can easily remove 321 the keys file on disk. That makes the CLI shorter. 322 323 * Implementation sticks to the IBAN found in the configuration, **if** the bank 324 does not show any IBAN related to the EBICS subscriber. 325 326 * from nexus-ebics-submit: "if the server accepts the request, sets the 327 initiated flag in the table to true". May there be a case where the 328 server accepted the request, but the client never got any response (some 329 network issue..), and therefore didn't set the submitted flag, ending 330 up in submitting the payment twice? Also: flagging the payment _after_ 331 the bank response, may lead double-submission even if the HTTP talk ended 332 well: it suffices to crash after having received a "200 OK" response but 333 before setting the submitted flag to the database. 334 335 * the ebics-submit section mentions the EBICS order ID. The following excerpt 336 was found however at page 88 of the EBICS 3 specifications: 337 338 ``OrderID is only present if a file is transmitted to the bank relating to an order with an 339 already existing order number (only allowed for AdminOrderType = HVE or HVS)`` 340 341 Nexus does not support HVE or HVS. 342 343 * As of private communication, the responsibility of submitting idempotent payments 344 relies on the use of ``request_uid`` (a database column of the initiated payment) 345 as the ``MsgId`` value of the corresponding pain.001 document. 346 347 * ``submitted`` column of an initiated payment evolved into the following enum: 348 349 .. code-block:: shell-session 350 351 CREATE TYPE submission_state AS ENUM ( 352 'unsubmitted' 353 ,'transient_failure' 354 ,'permanent_failure' 355 ,'success' 356 ,'never_heard_back' 357 ); 358 359 * ``unsubmitted``: default state when a payment is initiated 360 * ``transient_failure``: submission failed but can be retried, for example after a network issue. 361 * ``permanent_failure``: EBICS- or bank-technical error codes were not EBICS_OK (nor any tolerated EBICS code like EBICS_NO_DOWNLOAD_DATA_AVAILABLE), never retry. 362 * ``never_heard_back``: the payment initiation submission has **been** ``success`` but it was never confirmed by any outgoing transaction (from a camt.5x document) or any pain.002 report. It is responsability of a garbage collector to set this state after a particular time period. 363 364 * the initiated_outgoing_transactions table takes two more columns: 365 ``last_submission_date``, a timestamp in microseconds, and a 366 ``submission_counter``. Both of them would serve to decide retry 367 policies. 368 369 * the ``failure_text`` column at the initiated_outgoing_transactions table 370 should contain a JSON object that contains any useful detail about the problem. 371 That *could* be modeled after the Taler `ErrorDetail <https://docs.taler.net/core/api-common.html#tsref-type-ErrorDetail>`_, where at least the error code and the hint fields are provided.