005-wallet-backup-sync.rst (12531B)
1 XX 05: Wallet Backup and Sync 2 ############################# 3 4 :Design status: Superseded 5 :Implementation status: Prototype 6 :DD shepherd: TBD 7 :Historical contributors: Florian Dold, Torsten Grote, Christian Grothoff 8 :First published: 2020-04-27 9 :Last substantive change: 2023-09-15 10 :Superseded by: :doc:`092-incremental-backup-sync` 11 12 .. warning:: 13 14 This document is deprecated and superseded by DD92, which specifies an 15 incremental backup and multi-device synchronization protocol. 16 The multi-device sync described in this document would lead 17 to a bad/unexpected user experience that does not justify the 18 conceptual / implementation complexity. 19 20 The body below is retained for historical context and is non-normative. 21 22 Summary 23 ======= 24 25 This document discusses considerations for backup and synchronization of wallets. 26 27 28 Requirements 29 ============ 30 31 * Backup and sync must not require any synchronous communication between the 32 wallets 33 * Wallets operating (payments/withdrawals/...) for longer periods of time without 34 synchronizing should be handled well 35 * Conflicts should be resolved automatically in pretty much all cases 36 * One wallet can be enrolled with multiple sync servers, and a wallet can 37 join 38 * Other wallets connected to the sync server are trusted. 39 40 Proposed Solution 41 ================= 42 43 The blob stored on the backup/sync server is a compressed and encrypted JSON file. 44 45 The various entity types managed by the wallet are modeled as LWW-Sets (Last Write 46 Wins Set CRDT). Timestamps for inserts/deletes are Lamport timestamps. Concurrent, conflicting insert/delete 47 operations are resolved in favor of "delete". 48 49 The managed entities are: 50 51 * set of exchanges with the data from /keys, /wire 52 * set of directly trusted exchange public keys 53 * set of trusted auditors for currencies 54 * set of reserves together with reserve history 55 * set of accepted bank withdrawal operations 56 * set of coins together with coin history and blinding secret (both for normal withdrawal and refresh) 57 and coin source info (refresh operation, reward, reserve) 58 * set of purchases (contract terms, applied refunds, ...) 59 * assignment of coins to their "primary wallet" 60 61 (Some of these might be further split up to allow more efficient updates.) 62 63 Entities that are **not** synchronized are: 64 65 * purchases before the corresponding order has been claimed 66 * withdrawal operations before they have been accepted by the user 67 68 Entities that **could** be synchronized (to be decided): 69 70 * private keys of other sync accounts 71 * coin planchets 72 * rewards before the corresponding coins have been withdrawn 73 * refresh sessions (not only the "meta data" about the operation, 74 but everything) 75 76 77 Garbage collection 78 ------------------ 79 80 There are two types of garbage collection involved: 81 82 1. CRDT tombstones / other administrative data in the sync blob. These can be deleted 83 after we're sure all wallets enrolled in the sync server have a Lamport timestamp 84 larger than the timestamp of the tombstone. Wallets include their own Lamport timestamp 85 in the sync blob: 86 87 .. code:: javascript 88 89 { 90 clocks: { 91 my_desktop_wallet: 5, 92 my_phone_wallet: 3 93 }, 94 ... 95 } 96 97 All tombstones / overwritten set elements with a timestamp smaller than the 98 smallest clock value can be deleted. 99 100 2. Normal wallet GC. The deletion operations resulting from the wallet garbage 101 collection (i.g. deleting legally expired denomination keys, coins, exchange 102 signing keys, ...) are propagated to the respective CRDT set in the sync 103 blob. 104 105 106 Ghost Entities 107 -------------- 108 109 Sometimes a wallet can learn about an operation that happened in another synced 110 wallet **before** a sync over the sync server happens. An example of this is a 111 deposit operation. When two synced wallets spend the same coin on something, 112 one of them will receive an error from the exchange that proves the coin has 113 been spent on something else. The wallet will add a "ghost entry" for such an 114 event, in order to be able to show a consistent history (i.e. all numbers 115 adding up) to the user. 116 117 When the two wallets sync later, the ghost entry is replaced by the actual 118 purchase entity from the wallet that initiated the spending. 119 120 Ghost entities are not added to the sync state. 121 122 123 Multiple sync servers 124 --------------------- 125 126 When a wallet is connected to multiple sync servers, it automatically 127 propagates changes it received from one sync server to the others. Local 128 changes made by the wallet are propagated to all sync servers. The goal of 129 this is to make the state of the sync servers converge. 130 131 The different sync servers one wallet is enrolled with do not necessarily 132 have the same set of other wallets enrolled. Each sync server has a separate Lamport clock 133 and contains a separate CRDT. 134 135 Backup user flow 136 ================ 137 138 .. graphviz:: 139 140 digraph G { 141 nodesep=0.5; 142 withdrawal [ 143 label = "First\nWithdrawal"; 144 shape = oval; 145 ]; 146 has_backup [ 147 label = "Has backup\nconfigured?"; 148 shape = diamond; 149 ]; 150 app_settings [ 151 label = "App\nSettings"; 152 shape = rect; 153 ]; 154 backup_onboarding [ 155 label = "Backup\nOnboarding"; 156 shape = rect; 157 ]; 158 backup_settings [ 159 label = "Backup Settings\n\n* time of last backup\n* current service"; 160 shape = rect; 161 ]; 162 choose_backup_service [ 163 label = "Choose\nBackup Service"; 164 shape = rect; 165 ]; 166 tos_accepted [ 167 label = "Current ToS\naccepted?"; 168 shape = diamond; 169 ]; 170 tos [ 171 label = "ToS"; 172 shape = rect; 173 ]; 174 payment_required [ 175 label = "Payment\nrequired?"; 176 shape = diamond; 177 ]; 178 payment_confirmation [ 179 label = "Payment\nConfirmation"; 180 shape = rect; 181 ]; 182 backup_secret [ 183 label = "Backup Secret\n\nStore or write down!"; 184 shape = rect; 185 ]; 186 187 withdrawal -> has_backup; 188 has_backup -> backup_onboarding [label="No"]; 189 backup_onboarding -> backup_settings; 190 app_settings -> backup_settings; 191 backup_settings -> backup_settings [label="Disable Backup"]; 192 backup_settings:w -> backup_settings:w [label="Sync now"]; 193 backup_settings -> choose_backup_service; 194 choose_backup_service -> tos_accepted [label="Select Service"]; 195 tos_accepted -> tos [label="No"]; 196 tos_accepted -> payment_required [label="Yes"]; 197 choose_backup_service:w -> choose_backup_service [label="Remove current service"]; 198 choose_backup_service:n -> choose_backup_service:n [headlabel="Add new service", labeldistance=3.5]; 199 tos -> payment_required [label="Accept"]; 200 payment_required -> payment_confirmation [label="Yes"]; 201 payment_confirmation -> backup_secret [label="Paid"]; 202 backup_secret -> backup_settings [dir=both, label="Stored"]; 203 payment_required:s -> backup_secret:w [label="No"]; 204 205 { rank=same; has_backup; backup_onboarding; } 206 { rank=same; withdrawal; app_settings; } 207 { rank=same; tos_accepted; tos; } 208 { rank=same; payment_required; payment_confirmation; } 209 } 210 211 Backup Settings Screen 212 ---------------------- 213 214 * **Backup my wallet** [on/off] 215 * **Backup services** 216 No service active 217 (shows time of last backup per service) 218 * **Show backup secret** 219 You need this secret to restore from backup 220 * option to sync/backup now (hidden in action bar overflow menu) 221 222 Choose Backup Service Screen 223 ---------------------------- 224 This screen can be reached by pressing the **Backup services** setting 225 in the Backup Settings Screen. 226 It lists the currently active service and other services that can be used. 227 The user has the option to add new services to the list. 228 229 A backup service has 230 231 * a name 232 * a base URL 233 * a fee per year in an explicit currency 234 235 Clicking an active service shows the above service information as well as: 236 237 * the service secret that is required to restore from backup 238 * last payment date and next scheduled payment date 239 * option to deactivate the backup service 240 241 Clicking an inactive service allows the user to use the backup service 242 (after accepting ToS and making the payment). 243 244 Terms of Service Screen 245 ----------------------- 246 This screen always appears when a backup provider is selected 247 and the user did not yet accept the current version of its terms of service. 248 249 It shows the terms of service text and an accept checkbox, 250 as well as the usual back button. 251 252 Payment Confirmation Screen 253 --------------------------- 254 This is the same screen that the user sees when doing other purchases. 255 The only difference is that after successful payment, 256 the user will be shown the service secret instead of the transaction list. 257 258 Backup Secret Screen 259 -------------------- 260 After setting up a backup service, 261 the user needs to securely store the secret needed to restore from backup. 262 The secret will be shown as a Taler URI in plain text. 263 This has the form: ``taler://sync/$SYNC-DOMAIN/$SYNC-PATH#$PRIVATE-KEY`` 264 Additionally, the URI will be encoded as a QRcode. 265 Depending on the platform, there should be an option to print or export (PDF) the secret. 266 267 Backup Onboarding 268 ----------------- 269 If no backup service was selected when the user makes the first withdrawal, 270 an onboarding screen will be shown that takes the user to the backup configuration screen. 271 272 Don't lose your money, use a backup service! 273 274 Your wallet comes with a list of backup services 275 that can store an encrypted copy of your wallet. 276 Use one to keep your money safe! 277 278 [Set backup up now] 279 280 References 281 ========== 282 283 * Shapiro, M., Preguiça, N., Baquero, C., & Zawirski, M. (2011). A 284 comprehensive study of convergent and commutative replicated data types. [`PDF <https://hal.inria.fr/inria-00555588/document>`__] 285 286 Discussion / Q&A 287 ================ 288 289 * Why is backup/sync not split into two services / use-cases? 290 291 * For privacy reasons, we can't use some interactive sync service. Thus we 292 use the backup blob as a CRDT that also synchronization for us. 293 294 * Do we need to handle backup/sync state becoming very large e.g. by many transactions 295 and embedded product images potentially exceeding service quota? 296 297 * Do we synchronize the list of other backup enrollments? How 298 do we handle distributing the different private keys for them? 299 300 * If we automatically sync the sync enrollments and the old sync account 301 is compromised, the new sync account would automatically be compromised as well! 302 303 * If every wallet had its own sync key pair, we could select which existing wallets 304 to roll over as well. 305 306 * How do we handle a synced wallet that becomes malicious deleting all coins or purchased products? 307 308 * This needs to balance the genuine need to permanently delete data. 309 310 * Should the sync server allow to fetch previous versions of the sync blob? 311 (If not, how to provide backup functionality?) 312 313 * Should the individual wallets keep tombstones (i.e. entities just marked as deleted) 314 around for some time, or should they delete and "sanitize" (delete data not needed for the CRDT) 315 tombstones as soon as possible? 316 317 * How do we make it easy to remove compromised devices from the sync group 318 and prevent them from getting access to future funds and transactions? 319 320 * We need to remove all sync connections on all connected devices 321 and then individually (and manually) add all devices to the new backup account. 322 323 * If we encrypted the key with each wallet's private sync key, 324 we could choose which wallets we want to migrate to the new sync account. 325 326 * Can we then roll-over wallets to the new account automatically 327 or does it have to be manually on each device to prevent an attacker to roll us over? 328 329 * How are wallets identified for backup/sync? 330 331 * UUID / EdDSA pub and nick name? When nickname clashes, 332 some number is added based on lexical sort of the random id ("phone#1", "phone#2"). 333 334 * How do we explain to users that it can take days for wallet state to synchronize to all devices? 335 336 * How are people supposed to securely store their backup account key(s)? 337 338 * There can be an option to print/export the QR code 339 * They can manually write down the taler:// Uri containing the key. 340 * Maybe encode the key in a different format such as 341 `BIP39 <https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki>`__? 342 343 * Do we have a passphrase for our backup account key(s)? 344 345 * ???