010-exchange-helpers.rst (9850B)
1 DD 10: Exchange crypto helper design 2 #################################### 3 4 :Design status: Accepted 5 :Implementation status: Implemented 6 :DD shepherd: TBD 7 :Historical contributors: Christian Grothoff, Thien-Thi Nguyen 8 :First published: 2020-11-22 9 :Last substantive change: 2021-01-11 10 :Implementation evidence: exchange (2020-11-21, 2020-11-22, 2021-01-17, 2022-01-03) 11 :Normative references: :doc:`../taler-exchange-manual`, :doc:`../core/api-exchange` 12 13 Summary 14 ======= 15 16 A way to minimize the attack surface for extraction of the private online 17 signing keys (RSA, Clause-Schnorr and EdDSA) from the exchange is described. 18 19 20 Motivation 21 ========== 22 23 We want to provide an additional layer of protection for the private online 24 signing keys used by the exchange. The exchange is network-facing, includes an 25 HTTP server, PostgreSQL interaction, JSON parser and quite a bit of other logic 26 which may all be theoretically vulnerable to remote exploitation. Thus, it 27 would be good from a security perspective to protect the private online 28 signing keys via an additional layer of protection. 29 30 31 Requirements 32 ============ 33 34 * The solution should not result in a dramatic loss of performance. 35 * An attacker with a successful arbitrary code execution on the exchange 36 must not be able to extract the private keys. 37 * Ideally, we should be able to determine the number of signatures 38 obtained illicitly by the attacker. 39 * Key management for operators should be simplified to improve usability. 40 * RSA, Clause-Schnorr and EdDSA online signing keys need to be protected. 41 * We should have a way to verify that the keys signed with the offline 42 master private key are those originating from the isolated 43 (software/hardware) security module. 44 45 46 Proposed Solution 47 ================= 48 49 The private keys are created, used and deleted by three helper processes 50 running under a different user ID (UID), creating in effect a software 51 security module. The exchange's HTTP process will be required to interact 52 with those helpers via a UNIX domain socket. 53 54 Socket permission details: 55 56 * The socket will be chmod 0620 (u+rw, g+w) regardless of umask. 57 * That the group is the same group of the crypto helpers must 58 still be ensured by the operator. 59 60 General design details: 61 62 * The helpers will process requests from the exchange to sign and revoke keys. 63 * The helpers will create and destroy the private keys. They will no longer be 64 created on the air-gapped machine with the (offline) master private key. 65 The helpers will tell the exchange when keys are created or deleted/expired. 66 * Each helper will sign freshly generated keys with a security module-specific 67 private key. This key will be verified by the offline signing key process 68 using either manual verification against log output from the security 69 module's start-up routine, or via TOFU. TOFU is considered sufficient, 70 as an adversary breaking into the exchange process during the initial setup, 71 when the exchange is not even yet operational because no keys have ever been 72 provisioned, is considered highly unlikely. Depending on how the exchange 73 is initialized, access to security module logs may or may not be feasible, 74 so TOFU is a good and usable alternative strategy. 75 76 Helper design details: 77 78 * SOCK_DGRAM will be used to avoid needing to parse a data stream. 79 * The helpers will only know about (private) key lifetime. They will not know about 80 details like currency, fee structure, master or auditor signatures. 81 Those will be managed by the HTTP process to keep the helpers minimal. 82 * The helpers will use a single-threaded, GNUnet-scheduler-driven event loop 83 to process incoming requests from the UNIX domain sockets. However, the 84 actual signing will be done by a thread pool of workers that only process 85 signing requests from a work queue. Reference counting is used to avoid 86 releasing private keys while workers are actively using them to sign requests. 87 * The work queue is managed via a pthread-style semaphore. 88 * The master thread is informed about completed work via an ``eventfd()``. 89 * The master thread is responsible for handling revocations, creating future 90 private keys and expiring old keys. Revocations will also be triggered 91 via a new ``/keys`` endpoint. The HTTP server will verify that the revocation 92 is properly signed with the master private key before passing it on to the 93 respective helper. 94 95 Exchange design considerations: 96 97 * The helpers are started by the system, say via systemd, not by the 98 exchange. This simplifies the exchange. The exact number of exchange 99 processes is deployment-specific and has grown since the original design. 100 * Each exchange thread will create its own connection to the helpers, and will 101 block while waiting on the helper to create a signature. This keeps the 102 exchange logic simple and similar to the existing in-line signing calls. 103 Suspending and resuming would be difficult as we currently do not have a 104 way to wait for a UNIX domain socket to resume the MHD logic. 105 If a signal is received while waiting for the helper, the signature operation 106 fails. Signature operations can also fail if the helper is not running or 107 responding with incorrect data. However, signature operations do NOT have a 108 timeout. 109 110 New exchange endpoints: 111 112 * The exchange exposes the corresponding public keys via a GET to the 113 ``/management/keys`` endpoint to the offline signing process. For offline 114 signing, tooling will be provided to first download to a file, then 115 sign based on that file, and then upload the resulting signature back to 116 the exchange. For this, master signatures are POSTed to 117 the exchange at the ``/management/keys`` endpoint. 118 The exchange will keep those signatures in the PostgreSQL database. 119 * The ``/management/auditors`` endpoint enables auditors, and 120 ``/management/auditors/$AUDITOR_PUB/disable`` disables them, using requests 121 signed with the offline master private key. Auditor signatures on 122 denominations are POSTed to 123 ``/auditors/$AUDITOR_PUB/$H_DENOM_PUB``. 124 125 Overall, the result is that except for software updates and the fundamental 126 configuration, the ``taler-exchange-http`` will be updated only via HTTP(S) 127 and not via a signal and new files appearing in the directory hierarchy. 128 All of the more volatile state of the HTTP process will be in the database. 129 Only the helpers continue to keep files on disk. 130 131 132 Alternatives 133 ============ 134 135 * The helpers could have been given the information to validate the signing 136 request. However, without database access, validating the reserve key 137 signature (and others) is pretty useless. Thus, this direction would only 138 complicate the helper (which we want to keep minimal to minimize attack 139 surface) without real benefits. Even validating revocation requests (checking 140 signatures by auditor or master public key) makes no sense, as if an attacker 141 triggers a revocation, we should probably be thankful: That's a white-hat 142 demonstrating that they got control in the least harmful way. 143 * Instead of two helpers, we could have just one helper. But there is limited 144 overlap between the (RSA) denomination key logic and the (EdDSA) signing 145 key logic. Separation may improve security. 146 * We could have proposed a helper per denomination. But as the code of all of 147 these helpers would be identical, this would have no security advantages. 148 * We could have implemented our own event loop and configuration parser, 149 instead of relying on libgnunetutil. But this part of GNUnet is very 150 robust. 151 * We could have had a thread pool reading requests from the exchange clients, 152 instead of a master thread doling out the work. But this would become really 153 complicted with key revocations, and as really only the cryptography should 154 be the bottleneck, performance advantages should be minimal. If IPC ever 155 becomes the issue, then the entire idea of moving signatures to another 156 process would be flawed. 157 * More portable mechanisms (like a ``pipe()``) could be used for signaling 158 instead of ``eventfd()``. But, this can always be implemented if we truly 159 ever have an exchange operator needing support for such a platform. 160 * We could have left the helper single-threaded, to avoid the complications 161 arising from the use of threads. However, given that signing is expected to 162 be a bottleneck of the exchange, this would have had serious performance 163 implications for the entire system. 164 * The helpers could have been started by the exchange. This would have 165 required the helpers use SUID. Allowing the system administrator to start 166 them as they see fit is more flexible with respect to the privilege 167 configuration. Also, this avoid forcing the exchange to manage 168 restarting on crashes and/or crash reporting. 169 170 171 Drawbacks 172 ========= 173 174 * Additional work to properly setup an exchange and to run 175 our automated tests. 176 * Slight (?) performance impact. 177 * UNIX only. Likely Linux-only for now (but fixable). 178 * If exchange receives ANY (not ignored) signal during signing 179 operation, a discrepancy in the number of signatures created 180 between exchange (DB) and the helper will arise. Thus, 181 auditors have to allow for small discrepancies (increasing 182 over time). Note that we only expect the exchange to receive 183 signals if the software is updated or the process is terminated. 184 * If helper is stopped (SIGSTOP), exchange HTTP will itself block 185 (no timeout!). Timeout-based mitigation would additionally increase 186 discrepancies in the count of the number of signatures created. 187 * System administrator must not forget to start helpers, otherwise 188 the exchange will not work (This is not a new problem: same applies 189 for taler-exchange-transfer and other exchange processes). 190 191 192 193 Discussion / Q&A 194 ================ 195 196 (This should be filled in with results from discussions on mailing lists / personal communication.)