005-wallet-backup-sync.rst (12146B)
1 XX 05: Wallet Backup and Sync 2 ############################# 3 4 .. warning:: 5 6 This document is deprecated. We have decided to first 7 implement backup, and tackle sync later. 8 The multi-device sync described in this document would lead 9 to a bad/unexpected user experience that does not justify the 10 conceptual / implementation complexity. 11 12 Summary 13 ======= 14 15 This document discusses considerations for backup and synchronization of wallets. 16 17 18 Requirements 19 ============ 20 21 * Backup and sync must not require any synchronous communication between the 22 wallets 23 * Wallets operating (payments/withdrawals/...) for longer periods of time without 24 synchronizing should be handled well 25 * Conflicts should be resolved automatically in pretty much all cases 26 * One wallet can be enrolled with multiple sync servers, and a wallet can 27 join 28 * Other wallets connected to the sync server are trusted. 29 30 Proposed Solution 31 ================= 32 33 The blob stored on the backup/sync server is a compressed and encrypted JSON file. 34 35 The various entity types managed by the wallet are modeled LWW-Sets (Last Write 36 Wins Set CRDT). Timestamps for inserts/deletes are are Lamport timestamps. Concurrent, conflicting insert/delete 37 operations are resolved in favor of "delete". 38 39 The managed entities are: 40 41 * set of exchanges with the data from /keys, /wire 42 * set of directly trusted exchange public keys 43 * set of trusted auditors for currencies 44 * set of reserves together with reserve history 45 * set of accepted bank withdrawal operations 46 * set of coins together with coin history and blinding secret (both for normal withdrawal and refresh) 47 and coin source info (refresh operation, reward, reserve) 48 * set of purchases (contract terms, applied refunds, ...) 49 * assignment of coins to their "primary wallet" 50 51 (Some of these might be further split up to allow more efficient updates.) 52 53 Entities that are **not** synchronized are: 54 55 * purchases before the corresponding order has been claimed 56 * withdrawal operations before they have been accepted by the user 57 58 Entities that **could** be synchronized (to be decided): 59 60 * private keys of other sync accounts 61 * coin planchets 62 * rewards before the corresponding coins have been withdrawn 63 * refresh sessions (not only the "meta data" about the operation, 64 but everything) 65 66 67 Garbage collection 68 ------------------ 69 70 There are two types of garbage collection involved: 71 72 1. CRDT tombstones / other administrative data in the sync blob. These can be deleted 73 after we're sure all wallets enrolled in the sync server have a Lamport timestamp 74 larger than the timestamp of the tombstone. Wallets include their own Lamport timestamp 75 in the sync blob: 76 77 .. code:: javascript 78 79 { 80 clocks: { 81 my_desktop_wallet: 5, 82 my_phone_wallet: 3 83 }, 84 ... 85 } 86 87 All tombstones / overwritten set elements with a timestamp smaller than the 88 smallest clock value can be deleted. 89 90 2. Normal wallet GC. The deletion operations resulting from the wallet garbage 91 collection (i.g. deleting legally expired denomination keys, coins, exchange 92 signing keys, ...) are propagated to the respective CRDT set in the sync 93 blob. 94 95 96 Ghost Entities 97 -------------- 98 99 Sometimes a wallet can learn about an operation that happened in another synced 100 wallet **before** a sync over the sync server happens. An example of this is a 101 deposit operation. When two synced wallets spend the same coin on something, 102 one of them will receive an error from the exchange that proves the coin has 103 been spent on something else. The wallet will add a "ghost entry" for such an 104 event, in order to be able to show a consistent history (i.e. all numbers 105 adding up) to the user. 106 107 When the two wallets sync later, the ghost entry is replaced by the actual 108 purchase entity from the wallet that initiated the spending. 109 110 Ghost entities are not added to the sync state. 111 112 113 Multiple sync servers 114 --------------------- 115 116 When a wallet is connected to multiple sync servers, it automatically 117 propagates changes it received from one sync server to the others. Local 118 changes made by the wallet are propagated to all sync servers. The goal of 119 this is to make the state of the sync servers converge. 120 121 The different sync servers one wallet is enrolled with do not necessarily 122 have the same set of other wallet enrolled. Each sync server has a separate Lamport clock 123 and contains a separate CRDT. 124 125 Backup user flow 126 ================ 127 128 .. graphviz:: 129 130 digraph G { 131 nodesep=0.5; 132 withdrawal [ 133 label = "First\nWithdrawal"; 134 shape = oval; 135 ]; 136 has_backup [ 137 label = "Has backup\nconfigured?"; 138 shape = diamond; 139 ]; 140 app_settings [ 141 label = "App\nSettings"; 142 shape = rect; 143 ]; 144 backup_onboarding [ 145 label = "Backup\nOnboarding"; 146 shape = rect; 147 ]; 148 backup_settings [ 149 label = "Backup Settings\n\n* time of last backup\n* current service"; 150 shape = rect; 151 ]; 152 choose_backup_service [ 153 label = "Choose\nBackup Service"; 154 shape = rect; 155 ]; 156 tos_accepted [ 157 label = "Current ToS\naccepted?"; 158 shape = diamond; 159 ]; 160 tos [ 161 label = "ToS"; 162 shape = rect; 163 ]; 164 payment_required [ 165 label = "Payment\nrequired?"; 166 shape = diamond; 167 ]; 168 payment_confirmation [ 169 label = "Payment\nConfirmation"; 170 shape = rect; 171 ]; 172 backup_secret [ 173 label = "Backup Secret\n\nStore or write down!"; 174 shape = rect; 175 ]; 176 177 withdrawal -> has_backup; 178 has_backup -> backup_onboarding [label="No"]; 179 backup_onboarding -> backup_settings; 180 app_settings -> backup_settings; 181 backup_settings -> backup_settings [label="Disable Backup"]; 182 backup_settings:w -> backup_settings:w [label="Sync now"]; 183 backup_settings -> choose_backup_service; 184 choose_backup_service -> tos_accepted [label="Select Service"]; 185 tos_accepted -> tos [label="No"]; 186 tos_accepted -> payment_required [label="Yes"]; 187 choose_backup_service:w -> choose_backup_service [label="Remove current service"]; 188 choose_backup_service:n -> choose_backup_service:n [headlabel="Add new service", labeldistance=3.5]; 189 tos -> payment_required [label="Accept"]; 190 payment_required -> payment_confirmation [label="Yes"]; 191 payment_confirmation -> backup_secret [label="Paid"]; 192 backup_secret -> backup_settings [dir=both, label="Stored"]; 193 payment_required:s -> backup_secret:w [label="No"]; 194 195 { rank=same; has_backup; backup_onboarding; } 196 { rank=same; withdrawal; app_settings; } 197 { rank=same; tos_accepted; tos; } 198 { rank=same; payment_required; payment_confirmation; } 199 } 200 201 Backup Settings Screen 202 ---------------------- 203 204 * **Backup my wallet** [on/off] 205 * **Backup services** 206 No service active 207 (shows time of last backup per service) 208 * **Show backup secret** 209 You need this secret to restore from backup 210 * option to sync/backup now (hidden in action bar overflow menu) 211 212 Choose Backup Service Screen 213 ---------------------------- 214 This screen can be reached by pressing the **Backup services** setting 215 in the Backup Settings Screen. 216 It lists the currently active service and other services that can be used. 217 The user has the option to add new services to the list. 218 219 A backup service has 220 221 * a name 222 * a base URL 223 * a fee per year in an explicit currency 224 225 Clicking an active service shows the above service information as well as: 226 227 * the service secret that is required to restore from backup 228 * last payment date and next scheduled payment date 229 * option to deactivate the backup service 230 231 Clicking an inactive service allows the user to use the backup service 232 (after accepting ToS and making the payment). 233 234 Terms of Service Screen 235 ----------------------- 236 This screen always appears when a backup provider is selected 237 and the user did not yet accept the current version of its terms of service. 238 239 It shows the terms of service text and an accept checkbox, 240 as well as the usual back button. 241 242 Payment Confirmation Screen 243 --------------------------- 244 This is the same screen that the user sees when doing other purchases. 245 The only difference is that after successful payment, 246 the user will be shown the service secret instead of the transaction list. 247 248 Backup Secret Screen 249 -------------------- 250 After setting up a backup service, 251 the user needs to securely store the secret needed to restore from backup. 252 The secret will be shown as a Taler URI in plain text. 253 This has the form: ``taler://sync/$SYNC-DOMAIN/$SYNC-PATH#$PRIVATE-KEY`` 254 Additionally, the URI will be encoded as a QRcode. 255 Depending on the platform, there should be an option to print or export (PDF) the secret. 256 257 Backup Onboarding 258 ----------------- 259 If no backup service was selected when the user makes the first withdrawal, 260 an onboarding screen will be shown that takes the user to the backup configuration screen. 261 262 Don't loose your money, use a backup service! 263 264 Your wallet comes with a list of backup services 265 that can store an encrypted copy of your wallet. 266 Use one to keep your money safe! 267 268 [Set backup up now] 269 270 References 271 ========== 272 273 * Shapiro, M., Preguiça, N., Baquero, C., & Zawirski, M. (2011). A 274 comprehensive study of convergent and commutative replicated data types. [`PDF <https://hal.inria.fr/inria-00555588/document>`__] 275 276 Discussion / Q&A 277 ================ 278 279 * Why is backup/sync not split into two services / use-cases? 280 281 * For privacy reasons, we can't use some interactive sync service. Thus we 282 use the backup blob as a CRDT that also synchronization for us. 283 284 * Do we need to handle backup/sync state becoming very large e.g. by many transactions 285 and embedded product images potentially exceeding service quota? 286 287 * Do we synchronize the list of other backup enrollments? How 288 do we handle distributing the different private keys for them? 289 290 * If we automatically sync the sync enrollments and the old sync account 291 is compromised, the new sync account would automatically be compromised as well! 292 293 * If every wallet had its own sync key pair, we could select which existing wallets 294 to roll over as well. 295 296 * How do we handle a synced wallet that becomes malicious deleting all coins or purchased products? 297 298 * This needs to balance the genuine need to permanently delete data. 299 300 * Should the sync server allow to fetch previous versions of the sync blob? 301 (If not, how to provide backup functionality?) 302 303 * Should the individual wallets keep tombstones (i.e. entities just marked as deleted) 304 around for some time, or should they delete and "sanitize" (delete data not needed for the CRDT) 305 tombstones as soon as possible? 306 307 * How do we make it easy to remove compromised devices from the sync group 308 and prevent them from getting access to future funds and transactions? 309 310 * We need to remove all sync connections on all connected devices 311 and then individually (and manually) add all devices to the new backup account. 312 313 * If we encrypted the key with each wallet's private sync key, 314 we could choose which wallets we want to migrate to the new sync account. 315 316 * Can we then roll-over wallets to the new account automatically 317 or does it have to be manually on each device to prevent an attacker to roll us over? 318 319 * How are wallets identified for backup/sync? 320 321 * UUID / EdDSA pub and nick name? When nickname clashes, 322 some number is added based on lexical sort of the random id ("phone#1", "phone#2"). 323 324 * How do we explain users that it can take days for wallet state to synchronize to all devices? 325 326 * How are people supposed to securely store their backup account key(s)? 327 328 * There can be an option to print/export the QR code 329 * They can manually write down the taler:// Uri containing the key. 330 * Maybe encode the key in a different format such as 331 `BIP39 <https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki>`__? 332 333 * Do we have a passphrase for our backup account key(s)? 334 335 * ???