api-sync.rst (18259B)
1 .. 2 This file is part of GNU TALER. 3 Copyright (C) 2018-2021 Taler Systems SA 4 5 TALER is free software; you can redistribute it and/or modify it under the 6 terms of the GNU Affero General Public License as published by the Free Software 7 Foundation; either version 2.1, or (at your option) any later version. 8 9 TALER is distributed in the hope that it will be useful, but WITHOUT ANY 10 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 11 A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. 12 13 You should have received a copy of the GNU Affero General Public License along with 14 TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 15 16 @author Christian Grothoff 17 18 .. _sync-api: 19 20 ====================================== 21 Backup and Synchronization RESTful API 22 ====================================== 23 24 The backup and synchronization service uses an EdDSA key 25 to identify the "account" of the user. The key is Crockford 26 Base32-encoded in the URI to access the data and used to sign requests 27 as well as to encrypt the contents (see below). These signatures are 28 provided in detached form as HTTP headers. 29 30 Once the user activates backup or synchronization, the client should 31 display the key as a QR code as well as in text format together 32 with the synchronization service's URL and ask the user to print this 33 key material and keep it safe. 34 35 The actual format of the backup is not relevant for the 36 backup and synchronization service, as the service must only ever see 37 a padded and encrypted version of the data. 38 39 However, there are a few general rules that will apply to 40 any version of the backup. Still, except for the 41 32-byte minimum upload size, the synchronization service 42 itself cannot not enforce these rules. 43 44 * First, the database should be compressed (i.e. gzip), then 45 padded to a power of 2 in kilobytes or a multiple of 46 megabytes, then encrypted and finally protected with 47 an HDKF. 48 * The encryption should use an SHA-512 nonce which 49 is prefixed to the actual database, and combined with 50 the master key to create the encryption symmetric secret. 51 With every revision of the backup (but only real 52 revisions or merge operations), a fresh nonce must be 53 used to ensure that the symmetric secret differs every 54 time. HKDFs are used to derive symmetric key material 55 for authenticated encryption (encrypt-then-mac or a 56 modern AEAD-cipher like Keccak). Given that AES is more 57 easily available and will likey increase the code of 58 the wallet less, AES plus a SHA-512 HMAC should suffice 59 for now. 60 * The client must enable merging databases in a way that is 61 associative and commutative. For most activities, this implies 62 merging lists, applying expirations, dropping duplicates and 63 sorting the result. For deletions (operations by which the user 64 removed records prior to their scheduled expiration), it means 65 keeping a summarizing log of all deletion operations and applying 66 the deletions after each merge. A summarizing log of a deletion 67 operation would combine two deletion operations of the form 68 "delete all transactions smaller than amount X before time T" and 69 "delete all transactions smaller than amount Y before time T" 70 into "delete all transactions smaller than amount max(X,Y) before 71 time T". Similar summarizations should be applied to all 72 deletion operations supported by the client. Deletion operations 73 themselves are associated with an expiration time reflecting the 74 expiration of the longest lasting record that they explicitly 75 deleted. 76 Purchases do not have an expiration time, thus they create 77 a challenge if an indivdiual purchase is deleted. Thus, when 78 an individual purchase is deleted, the client is to keep track 79 of the deletion with a deletion record. The deletion record 80 still includes the purchase amount and purchase date. Thus, 81 when purchases are deleted "in bulk" in a way that would have 82 covered the individual deletion, such deletion records may 83 still be subsumed by a more general deletion clause. In addition 84 to the date and amount, the deletion record should only contain 85 a salted hash of the original purchase record's primary key, 86 so as to minimize information leakage. 87 * The database should contain a "last modified" timestamp to ensure 88 we do not go backwards in time if the synchronization service is 89 malicious. Merging two databases means taking the max of the 90 "last modified" timestamps, not setting it to the current time. 91 The client should reject a "fast forward" database update if the 92 result would imply going back in time. If the client receives a 93 database with a timestamp into the future, it must still 94 increment it by the smallest possible amount when uploading an 95 update. 96 * In general, the merge operation should be implemented in such a way 97 that it deals gracefully with adversarial devices from rogue 98 devices connected to the same account. 99 100 It is assumed that the synchronization service is only ever accessed 101 over TLS, and that the synchronization service is trusted to not build 102 user's location profiles by linking client IP addresses and client 103 keys. 104 105 .. contents:: Table of Contents 106 :local: 107 108 109 .. include:: tos.rst 110 111 ----------------------- 112 Receiving Configuration 113 ----------------------- 114 115 .. http:get:: /config 116 117 Obtain the key configuration settings of the storage service. 118 This specification corresponds to ``current`` protocol being version **2**. 119 120 **Response:** 121 122 Returns a `SyncTermsOfServiceResponse`. 123 124 .. ts:def:: SyncTermsOfServiceResponse 125 126 interface SyncTermsOfServiceResponse { 127 // Name of the service 128 name: "sync"; 129 130 // Maximum backup size supported. 131 storage_limit_in_megabytes: Integer; 132 133 // Fee for an account, per year. 134 annual_fee: Amount; 135 136 // Maximum liability of the provider in case of data loss. 137 liability_limit: Amount; 138 139 // libtool-style representation of the Sync protocol version, see 140 // https://www.gnu.org/software/libtool/manual/html_node/Versioning.html#Versioning 141 // The format is "current:revision:age". 142 version: string; 143 144 // URN of the implementation (needed to interpret 'revision' in version). 145 // @since v2, may become mandatory in the future. 146 implementation?: string; 147 148 } 149 150 .. _sync: 151 152 ---------------------- 153 Recovering Backup Data 154 ---------------------- 155 156 .. http:get:: /backups/${ACCOUNT-KEY} 157 158 Download latest version of the backup. 159 The returned headers must include "Etags" based on 160 the hash of the (encrypted) database. The server must 161 check the client's caching headers and only return the 162 full database if it has changed since the last request 163 of the client. 164 165 This method is generally only performed once per device 166 when the private key and URL of a synchronization service are 167 first given to the client on the respective device. Once a 168 client has made a backup, it should always use the POST method. 169 170 A signature is not required, as (1) the account-key should 171 be reasonably private and thus unauthorized users should not 172 know how to produce the correct request, and (2) the 173 information returned is encrypted to the private key anyway 174 and thus virtually useless even to an attacker who somehow 175 managed to obtain the public key. 176 177 **Response** 178 179 :http:statuscode:`200 OK`: 180 The body contains the current version of the backup 181 as known to the server. 182 183 :http:statuscode:`204 No content`: 184 This is a fresh account, no previous backup data exists at 185 the server. 186 187 :http:statuscode:`304 Not modified`: 188 The version available at the server is identical to that 189 specified in the ``If-None-Match`` header. 190 191 :http:statuscode:`404 Not found`: 192 The backup service is unaware of a matching account. 193 194 :http:statuscode:`410 Gone`: 195 The backup service has closed operations. The body will 196 contain the latest version still available at the server. 197 The body may be empty if no version is available. 198 The user should be urged to find another provider. 199 200 :http:statuscode:`429 Too many requests`: 201 This account has exceeded thresholds for the number of 202 requests. The client should try again later, and may want 203 to decrease its synchronization frequency. 204 205 .. note:: 206 207 "200 OK" responses include an HTTP header 208 "Sync-Signature" with the signature of the 209 client from the original upload, and an 210 "Sync-Previous" with the version that was 211 being updated (unless this is the first revision). 212 "Sync-Previous" is only given to enable 213 signature validation. 214 215 216 --------------------- 217 Uploading Backup Data 218 --------------------- 219 220 .. http:post:: /backups/${ACCOUNT-KEY} 221 222 Upload a new version of the account's database, or download the 223 latest version. The request SHOULD include the ``Expect: 100 Continue`` 224 header. The client then SHOULD wait for ``100 Continue`` before proceeding 225 with the upload, regardless of the size of the upload. 226 227 **Request** 228 229 The request must include a ``If-Match`` header indicating the latest 230 version of the account's database known to the client. If the server 231 knows a more recent version, it will respond with a ``409 conflict`` 232 and return the server's version in the response. The client must 233 then merge the two versions before retrying the upload. Note that 234 a ``409 Conflict`` response will typically be given before the upload, 235 (instead of ``100 continue``), but may also be given after the upload, 236 for example due to concurrent activities from other accounts on the 237 same account! 238 239 The request MUST also include an "Sync-Signature" signing 240 the ``If-Match`` SHA-512 value and the SHA-512 hash of the body with 241 the account private key. 242 243 Finally, the SHA-512 hash of the body MUST also be given in an 244 ``If-None-Match`` header of the request (so that the signature can be verified 245 before the upload is allowed to proceed). 246 247 The uploaded body must have at least 32 bytes of payload (see 248 suggested upload format beginning with an ephemeral key). 249 250 :query paying: 251 Optional argument providing an order identifier. 252 The client is promising that it is already paying on a 253 related order. This will cause the 254 server to delay processing until the respective payment 255 has arrived (if the operation requires a payment). Useful 256 if the server previously returned a ``402 Payment required`` 257 and the client wants to proceed as soon as the payment 258 went through. 259 :query pay: 260 Optional argument, any non-empty value will do, 261 suggested is ``y`` for ``yes``. 262 The client insists on making a payment for the respective 263 account, even if this is not yet required. The server 264 will respond with a ``402 Payment required``, but only 265 if the rest of the request is well-formed (account 266 signature must match). Clients that do not actually 267 intend to make a new upload but that only want to pay 268 may attempt to upload the latest backup again, as this 269 option will be checked before the ``304 Not modified`` 270 case. 271 :query fresh: 272 Optional argument, any non-empty value will do, 273 suggested is ``y`` for ``yes``. 274 The client insists on a fresh order to be generated, say 275 because the one returned before was claimed (but not paid) 276 by another wallet. 277 278 279 **Response** 280 281 :http:statuscode:`204 No content`: 282 The transfer was successful, and the server has registered 283 the new version. 284 285 :http:statuscode:`304 Not modified`: 286 The server is already aware of this version of the client. 287 Returned before ``100 continue`` to avoid upload. 288 289 :http:statuscode:`400 Bad request`: 290 Most likely, the uploaded body is too short (less than 32 bytes). 291 292 :http:statuscode:`402 Payment required`: 293 The synchronization service requires payment before the 294 account can continue to be used. The fulfillment URL 295 should be the ``/$ACCOUNT-KEY`` URL, but can be safely ignored 296 by the client. The contract should be shown to the user 297 in the canonical dialog, possibly in a fresh tab. 298 299 :http:statuscode:`403 Forbidden`: 300 The signature is invalid or missing (or body does not match). 301 302 :http:statuscode:`409 Conflict`: 303 The server has a more recent version than what is given 304 in ``If-Match``. The more recent version is returned. The 305 client should merge the two versions and retry using the 306 given response's "E-Tag" in the next attempt in ``If-Match``. 307 308 :http:statuscode:`410 Gone`: 309 The backup service has closed operations. The body will 310 contain the latest version still available at the server. 311 The body may be empty if no version is available. 312 The user should be urged to find another provider. 313 314 :http:statuscode:`411 Length required`: 315 The client must specify the ``Content-length`` header before 316 attempting upload. While technically optional by the 317 HTTP specification, the synchronization service may require 318 the client to provide the length upfront. 319 320 :http:statuscode:`413 Request entity too large`: 321 The requested upload exceeds the quota for the type of 322 account. The client should suggest to the user to 323 migrate to another backup and synchronization service 324 (like with ``410 Gone``). 325 326 :http:statuscode:`429 Too many requests`: 327 This account has exceeded daily thresholds for the number of 328 requests. The client should try again later, and may want 329 to decrease its synchronization frequency. 330 331 .. note:: 332 333 Responses with a body include an HTTP header 334 "Sync-Signature" with the signature of the 335 client from the original upload, and an 336 "If-Match" with the version that is 337 being updated (unless this is the first revision). 338 339 340 341 --------------------------- 342 Special constraints for Tor 343 --------------------------- 344 345 We might introduce the notion of a "constraint" into the client's 346 database that states that the database is a "Tor wallet". Then, 347 synchronizing a "Tor-wallet" with a non-Tor wallet should trigger a 348 stern warning and require user confirmation (as otherwise 349 cross-browser synchronization may weaken the security of Tor browser 350 users). 351 352 353 ------------------------------------------------ 354 Discovery of backup and synchronization services 355 ------------------------------------------------ 356 357 The client should keep a list of "default" synchronization services 358 per currency (by the currency the synchronization service accepts 359 for payment). If a synchronization service is entirely free, it 360 should be kept in a special list that is always available. 361 362 Extending (or shortening) the list of synchronization services should 363 be possible using the same mechanism that is used to add/remove 364 auditors or exchanges. 365 366 The client should urge the user to make use of a synchronization 367 service upon first withdrawal, suggesting one that is free or 368 accepts payment in the respective currency. If none is available, 369 the client should warn the user about the lack of available 370 backups and synchronization and suggest to the user to find a 371 reasonable service. Once a synchronization service is selected, 372 the client should urge the user to print the respective key 373 material. 374 375 When the client starts the first time on a new device, it should 376 ask the user if he wants to synchronize with an existing client, 377 and if so, ask the user to enter the respective key and the 378 (base) URL of the synchronization service. 379 380 381 ------------------------- 382 Synchronization frequency 383 ------------------------- 384 385 Generally, the client should attempt to synchronize at a randomized 386 time interval between 30 and 300 seconds of being started, unless it 387 already synchronized less than two hours ago already. Afterwards, 388 the client should synchronize every two hours, or after purchases 389 exceed 5 percent of the last bulk amount that the user withdrew. 390 In all cases the exact time of synchronization should be randomized 391 between 30 and 300 seconds of the specified event, both to minimize 392 obvious correlations and to spread the load. 393 394 If the two hour frequency would exceed half of the rate budget offered 395 by the synchronization provider, it should be reduced to remain below 396 that threshold. 397 398 399 ------------------------------- 400 Synchronization user experience 401 ------------------------------- 402 403 The menu should include three entries for synchronization: 404 405 * "synchronize" to manually trigger synchronization, 406 insensitive if no synchronization provider is available 407 * "export backup configuration" to re-display (and possibly 408 print) the synchronization and backup parameters (URL and 409 private key), insensitive if no synchronization 410 provider is available, and 411 * "import backup configuration" to: 412 413 * import another devices' synchronization options 414 (by specifying URL and private key, or possibly 415 scanning a QR code), or 416 * select a synchronization provider from the list, 417 including manual specification of a URL; here 418 confirmation should only be possible if the provider 419 is free or can be paid for; in this case, the 420 client should trigger the payment interaction when 421 the user presses the "select" button. 422 * a special button to "disable synchronization and backup" 423 424 One usability issue here is that we are asking users to deal with a 425 private key. It is likely better to map private keys to trustwords 426 (PEP-style). Also, when putting private keys into a QR code, there is 427 the danger of the QR code being scanned and interpreted as a "public" 428 URL. Thus, the QR code should use the schema 429 ``taler://sync/$SYNC-DOMAIN/$SYNC-PATH#private-key`` where 430 ``$SYNC-DOMAIN`` is the domainname of the synchronization service and 431 ``$SYNC-PATH`` the (usually empty) path. By putting the private key after 432 ``#``, we may succeed in disclosing the value even to eager Web-ish 433 interpreters of URLs. Note that the actual synchronization service 434 must use the HTTPS protocol, which means we can leave out this prefix. 435 436 437 --------------------------- 438 Web Security Considerations 439 --------------------------- 440 441 To ensure that the Taler Web extension (and others) can access the 442 service despite Web "security", all service endpoints must set the 443 header:: 444 445 Access-Control-Allow-Origin: *