api-sync.rst (18631B)
1 .. 2 This file is part of GNU TALER. 3 Copyright (C) 2018-2021 Taler Systems SA 4 5 TALER is free software; you can redistribute it and/or modify it under the 6 terms of the GNU Affero General Public License as published by the Free Software 7 Foundation; either version 2.1, or (at your option) any later version. 8 9 TALER is distributed in the hope that it will be useful, but WITHOUT ANY 10 WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR 11 A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. 12 13 You should have received a copy of the GNU Affero General Public License along with 14 TALER; see the file COPYING. If not, see <http://www.gnu.org/licenses/> 15 16 @author Christian Grothoff 17 18 .. _sync-api: 19 20 ====================================== 21 Backup and Synchronization RESTful API 22 ====================================== 23 24 The backup and synchronization service uses an EdDSA key 25 to identify the "account" of the user. The key is Crockford 26 Base32-encoded in the URI to access the data and used to sign requests 27 as well as to encrypt the contents (see below). These signatures are 28 provided in detached form as HTTP headers. 29 30 Once the user activates backup or synchronization, the client should 31 display the key as a QR code as well as in text format together 32 with the synchronization service's URL and ask the user to print this 33 key material and keep it safe. 34 35 The actual format of the backup is not relevant for the 36 backup and synchronization service, as the service must only ever see 37 a padded and encrypted version of the data. 38 39 However, there are a few general rules that will apply to 40 any version of the backup. Still, except for the 41 32-byte minimum upload size, the synchronization service 42 itself cannot not enforce these rules. 43 44 * First, the database should be compressed (i.e. gzip), then 45 padded to a power of 2 in kilobytes or a multiple of 46 megabytes, then encrypted and finally protected with 47 an HDKF. 48 * The encryption should use an SHA-512 nonce which 49 is prefixed to the actual database, and combined with 50 the master key to create the encryption symmetric secret. 51 With every revision of the backup (but only real 52 revisions or merge operations), a fresh nonce must be 53 used to ensure that the symmetric secret differs every 54 time. HKDFs are used to derive symmetric key material 55 for authenticated encryption (encrypt-then-mac or a 56 modern AEAD-cipher like Keccak). Given that AES is more 57 easily available and will likey increase the code of 58 the wallet less, AES plus a SHA-512 HMAC should suffice 59 for now. 60 * The client must enable merging databases in a way that is 61 associative and commutative. For most activities, this implies 62 merging lists, applying expirations, dropping duplicates and 63 sorting the result. For deletions (operations by which the user 64 removed records prior to their scheduled expiration), it means 65 keeping a summarizing log of all deletion operations and applying 66 the deletions after each merge. A summarizing log of a deletion 67 operation would combine two deletion operations of the form 68 "delete all transactions smaller than amount X before time T" and 69 "delete all transactions smaller than amount Y before time T" 70 into "delete all transactions smaller than amount max(X,Y) before 71 time T". Similar summarizations should be applied to all 72 deletion operations supported by the client. Deletion operations 73 themselves are associated with an expiration time reflecting the 74 expiration of the longest lasting record that they explicitly 75 deleted. 76 Purchases do not have an expiration time, thus they create 77 a challenge if an indivdiual purchase is deleted. Thus, when 78 an individual purchase is deleted, the client is to keep track 79 of the deletion with a deletion record. The deletion record 80 still includes the purchase amount and purchase date. Thus, 81 when purchases are deleted "in bulk" in a way that would have 82 covered the individual deletion, such deletion records may 83 still be subsumed by a more general deletion clause. In addition 84 to the date and amount, the deletion record should only contain 85 a salted hash of the original purchase record's primary key, 86 so as to minimize information leakage. 87 * The database should contain a "last modified" timestamp to ensure 88 we do not go backwards in time if the synchronization service is 89 malicious. Merging two databases means taking the max of the 90 "last modified" timestamps, not setting it to the current time. 91 The client should reject a "fast forward" database update if the 92 result would imply going back in time. If the client receives a 93 database with a timestamp into the future, it must still 94 increment it by the smallest possible amount when uploading an 95 update. 96 * In general, the merge operation should be implemented in such a way 97 that it deals gracefully with adversarial devices from rogue 98 devices connected to the same account. 99 100 It is assumed that the synchronization service is only ever accessed 101 over TLS, and that the synchronization service is trusted to not build 102 user's location profiles by linking client IP addresses and client 103 keys. 104 105 .. contents:: Table of Contents 106 :local: 107 108 109 --------------- 110 Version History 111 --------------- 112 113 The current protocol version is **v2**. 114 115 * No components use Sync at this point, so there are no dependencies. 116 117 **Version history:** 118 119 * ``v2``: add the ``implementation`` field to ``/config`` 120 121 **Upcoming versions:** 122 123 * ``vBLOBS``: changes for blob backups 124 * ``vBACKUP``: changes for incremental backups 125 126 **Ideas for future version:** 127 128 * ``vXXX``: marker for features not yet targeted for release 129 130 .. include:: tos.rst 131 132 ----------------------- 133 Receiving Configuration 134 ----------------------- 135 136 .. http:get:: /config 137 138 Obtain the key configuration settings of the storage service. 139 140 **Response:** 141 142 Returns a `SyncTermsOfServiceResponse`. 143 144 .. ts:def:: SyncTermsOfServiceResponse 145 146 interface SyncTermsOfServiceResponse { 147 // Name of the service 148 name: "sync"; 149 150 // Maximum backup size supported. 151 storage_limit_in_megabytes: Integer; 152 153 // Fee for an account, per year. 154 annual_fee: Amount; 155 156 // Maximum liability of the provider in case of data loss. 157 liability_limit: Amount; 158 159 // libtool-style representation of the Sync protocol version, see 160 // https://www.gnu.org/software/libtool/manual/html_node/Versioning.html#Versioning 161 // The format is "current:revision:age". 162 version: string; 163 164 // URN of the implementation (needed to interpret 'revision' in version). 165 // @since **v2**, may become mandatory in the future. 166 implementation?: string; 167 168 } 169 170 .. _sync: 171 172 ---------------------- 173 Recovering Backup Data 174 ---------------------- 175 176 .. http:get:: /backups/${ACCOUNT-KEY} 177 178 Download latest version of the backup. 179 The returned headers must include "Etags" based on 180 the hash of the (encrypted) database. The server must 181 check the client's caching headers and only return the 182 full database if it has changed since the last request 183 of the client. 184 185 This method is generally only performed once per device 186 when the private key and URL of a synchronization service are 187 first given to the client on the respective device. Once a 188 client has made a backup, it should always use the POST method. 189 190 A signature is not required, as (1) the account-key should 191 be reasonably private and thus unauthorized users should not 192 know how to produce the correct request, and (2) the 193 information returned is encrypted to the private key anyway 194 and thus virtually useless even to an attacker who somehow 195 managed to obtain the public key. 196 197 **Response** 198 199 :http:statuscode:`200 OK`: 200 The body contains the current version of the backup 201 as known to the server. 202 203 :http:statuscode:`204 No content`: 204 This is a fresh account, no previous backup data exists at 205 the server. 206 207 :http:statuscode:`304 Not modified`: 208 The version available at the server is identical to that 209 specified in the ``If-None-Match`` header. 210 211 :http:statuscode:`404 Not found`: 212 The backup service is unaware of a matching account. 213 214 :http:statuscode:`410 Gone`: 215 The backup service has closed operations. The body will 216 contain the latest version still available at the server. 217 The body may be empty if no version is available. 218 The user should be urged to find another provider. 219 220 :http:statuscode:`429 Too many requests`: 221 This account has exceeded thresholds for the number of 222 requests. The client should try again later, and may want 223 to decrease its synchronization frequency. 224 225 .. note:: 226 227 "200 OK" responses include an HTTP header 228 "Sync-Signature" with the signature of the 229 client from the original upload, and an 230 "Sync-Previous" with the version that was 231 being updated (unless this is the first revision). 232 "Sync-Previous" is only given to enable 233 signature validation. 234 235 236 --------------------- 237 Uploading Backup Data 238 --------------------- 239 240 .. http:post:: /backups/${ACCOUNT-KEY} 241 242 Upload a new version of the account's database, or download the 243 latest version. The request SHOULD include the ``Expect: 100 Continue`` 244 header. The client then SHOULD wait for ``100 Continue`` before proceeding 245 with the upload, regardless of the size of the upload. 246 247 **Request** 248 249 The request must include a ``If-Match`` header indicating the latest 250 version of the account's database known to the client. If the server 251 knows a more recent version, it will respond with a ``409 conflict`` 252 and return the server's version in the response. The client must 253 then merge the two versions before retrying the upload. Note that 254 a ``409 Conflict`` response will typically be given before the upload, 255 (instead of ``100 continue``), but may also be given after the upload, 256 for example due to concurrent activities from other accounts on the 257 same account! 258 259 The request MUST also include an "Sync-Signature" signing 260 the ``If-Match`` SHA-512 value and the SHA-512 hash of the body with 261 the account private key. 262 263 Finally, the SHA-512 hash of the body MUST also be given in an 264 ``If-None-Match`` header of the request (so that the signature can be verified 265 before the upload is allowed to proceed). 266 267 The uploaded body must have at least 32 bytes of payload (see 268 suggested upload format beginning with an ephemeral key). 269 270 :query paying: 271 Optional argument providing an order identifier. 272 The client is promising that it is already paying on a 273 related order. This will cause the 274 server to delay processing until the respective payment 275 has arrived (if the operation requires a payment). Useful 276 if the server previously returned a ``402 Payment required`` 277 and the client wants to proceed as soon as the payment 278 went through. 279 :query pay: 280 Optional argument, any non-empty value will do, 281 suggested is ``y`` for ``yes``. 282 The client insists on making a payment for the respective 283 account, even if this is not yet required. The server 284 will respond with a ``402 Payment required``, but only 285 if the rest of the request is well-formed (account 286 signature must match). Clients that do not actually 287 intend to make a new upload but that only want to pay 288 may attempt to upload the latest backup again, as this 289 option will be checked before the ``304 Not modified`` 290 case. 291 :query fresh: 292 Optional argument, any non-empty value will do, 293 suggested is ``y`` for ``yes``. 294 The client insists on a fresh order to be generated, say 295 because the one returned before was claimed (but not paid) 296 by another wallet. 297 298 299 **Response** 300 301 :http:statuscode:`204 No content`: 302 The transfer was successful, and the server has registered 303 the new version. 304 305 :http:statuscode:`304 Not modified`: 306 The server is already aware of this version of the client. 307 Returned before ``100 continue`` to avoid upload. 308 309 :http:statuscode:`400 Bad request`: 310 Most likely, the uploaded body is too short (less than 32 bytes). 311 312 :http:statuscode:`402 Payment required`: 313 The synchronization service requires payment before the 314 account can continue to be used. The fulfillment URL 315 should be the ``/$ACCOUNT-KEY`` URL, but can be safely ignored 316 by the client. The contract should be shown to the user 317 in the canonical dialog, possibly in a fresh tab. 318 319 :http:statuscode:`403 Forbidden`: 320 The signature is invalid or missing (or body does not match). 321 322 :http:statuscode:`409 Conflict`: 323 The server has a more recent version than what is given 324 in ``If-Match``. The more recent version is returned. The 325 client should merge the two versions and retry using the 326 given response's "E-Tag" in the next attempt in ``If-Match``. 327 328 :http:statuscode:`410 Gone`: 329 The backup service has closed operations. The body will 330 contain the latest version still available at the server. 331 The body may be empty if no version is available. 332 The user should be urged to find another provider. 333 334 :http:statuscode:`411 Length required`: 335 The client must specify the ``Content-length`` header before 336 attempting upload. While technically optional by the 337 HTTP specification, the synchronization service may require 338 the client to provide the length upfront. 339 340 :http:statuscode:`413 Request entity too large`: 341 The requested upload exceeds the quota for the type of 342 account. The client should suggest to the user to 343 migrate to another backup and synchronization service 344 (like with ``410 Gone``). 345 346 :http:statuscode:`429 Too many requests`: 347 This account has exceeded daily thresholds for the number of 348 requests. The client should try again later, and may want 349 to decrease its synchronization frequency. 350 351 .. note:: 352 353 Responses with a body include an HTTP header 354 "Sync-Signature" with the signature of the 355 client from the original upload, and an 356 "If-Match" with the version that is 357 being updated (unless this is the first revision). 358 359 360 361 --------------------------- 362 Special constraints for Tor 363 --------------------------- 364 365 We might introduce the notion of a "constraint" into the client's 366 database that states that the database is a "Tor wallet". Then, 367 synchronizing a "Tor-wallet" with a non-Tor wallet should trigger a 368 stern warning and require user confirmation (as otherwise 369 cross-browser synchronization may weaken the security of Tor browser 370 users). 371 372 373 ------------------------------------------------ 374 Discovery of backup and synchronization services 375 ------------------------------------------------ 376 377 The client should keep a list of "default" synchronization services 378 per currency (by the currency the synchronization service accepts 379 for payment). If a synchronization service is entirely free, it 380 should be kept in a special list that is always available. 381 382 Extending (or shortening) the list of synchronization services should 383 be possible using the same mechanism that is used to add/remove 384 auditors or exchanges. 385 386 The client should urge the user to make use of a synchronization 387 service upon first withdrawal, suggesting one that is free or 388 accepts payment in the respective currency. If none is available, 389 the client should warn the user about the lack of available 390 backups and synchronization and suggest to the user to find a 391 reasonable service. Once a synchronization service is selected, 392 the client should urge the user to print the respective key 393 material. 394 395 When the client starts the first time on a new device, it should 396 ask the user if he wants to synchronize with an existing client, 397 and if so, ask the user to enter the respective key and the 398 (base) URL of the synchronization service. 399 400 401 ------------------------- 402 Synchronization frequency 403 ------------------------- 404 405 Generally, the client should attempt to synchronize at a randomized 406 time interval between 30 and 300 seconds of being started, unless it 407 already synchronized less than two hours ago already. Afterwards, 408 the client should synchronize every two hours, or after purchases 409 exceed 5 percent of the last bulk amount that the user withdrew. 410 In all cases the exact time of synchronization should be randomized 411 between 30 and 300 seconds of the specified event, both to minimize 412 obvious correlations and to spread the load. 413 414 If the two hour frequency would exceed half of the rate budget offered 415 by the synchronization provider, it should be reduced to remain below 416 that threshold. 417 418 419 ------------------------------- 420 Synchronization user experience 421 ------------------------------- 422 423 The menu should include three entries for synchronization: 424 425 * "synchronize" to manually trigger synchronization, 426 insensitive if no synchronization provider is available 427 * "export backup configuration" to re-display (and possibly 428 print) the synchronization and backup parameters (URL and 429 private key), insensitive if no synchronization 430 provider is available, and 431 * "import backup configuration" to: 432 433 * import another devices' synchronization options 434 (by specifying URL and private key, or possibly 435 scanning a QR code), or 436 * select a synchronization provider from the list, 437 including manual specification of a URL; here 438 confirmation should only be possible if the provider 439 is free or can be paid for; in this case, the 440 client should trigger the payment interaction when 441 the user presses the "select" button. 442 * a special button to "disable synchronization and backup" 443 444 One usability issue here is that we are asking users to deal with a 445 private key. It is likely better to map private keys to trustwords 446 (PEP-style). Also, when putting private keys into a QR code, there is 447 the danger of the QR code being scanned and interpreted as a "public" 448 URL. Thus, the QR code should use the schema 449 ``taler://sync/$SYNC-DOMAIN/$SYNC-PATH#private-key`` where 450 ``$SYNC-DOMAIN`` is the domainname of the synchronization service and 451 ``$SYNC-PATH`` the (usually empty) path. By putting the private key after 452 ``#``, we may succeed in disclosing the value even to eager Web-ish 453 interpreters of URLs. Note that the actual synchronization service 454 must use the HTTPS protocol, which means we can leave out this prefix. 455 456 457 --------------------------- 458 Web Security Considerations 459 --------------------------- 460 461 To ensure that the Taler Web extension (and others) can access the 462 service despite Web "security", all service endpoints must set the 463 header:: 464 465 Access-Control-Allow-Origin: *