commit ee0c431687c3cb8f0d8f51fa9f5acf48d42de680
parent 60d01146b4977933feb2a53257c4f1f75cc626ba
Author: Iván Ávalos <avalos@disroot.org>
Date: Mon, 3 Aug 2026 14:23:59 +0200
DD92: document the backup process, schedule and wallet-core API
Also corrects the block nonce size and the blob keying to what the wallet
and the sync server actually implement.
Diffstat:
1 file changed, 192 insertions(+), 19 deletions(-)
diff --git a/design-documents/092-incremental-backup-sync.rst b/design-documents/092-incremental-backup-sync.rst
@@ -137,22 +137,33 @@ reconciliation mechanism that will be discussed in further sections.
Block format
++++++++++++
-Each block will consist of a 2-byte version number, a random 32-byte nonce,
+Each block will consist of a 2-byte version number, a random 24-byte nonce,
and a gzip-compressed JSON object with its length. The block will be padded up
-to the next whole kilobyte for privacy reasons.
+to the next whole kilobyte for privacy reasons. A block whose length is
+already a multiple of a kilobyte is not padded further.
+
+The nonce is 24 bytes because that is exactly what `secretbox`_ takes, which
+lets a block be encrypted under its own nonce.
Encryption will be performed on the block using symmetric authenticated
-encryption via libsodium's `secretbox`_ function, with a 32-bit key derived
-from the wallet's backup encryption key and the hash of the entire plaintext
-block, which in the final implementation should be shareable between any
-wallets that the user wishes to add to the synchronization group.
+encryption via libsodium's `secretbox`_ function, with a 32-byte key derived
+from the wallet's backup encryption key and the nonce of the block, which in
+the final implementation should be shareable between any wallets that the
+user wishes to add to the synchronization group.
+
+.. note::
+
+ The key is derived from the *nonce* rather than from the hash of the
+ plaintext block: the nonce travels with the block, whereas the plaintext
+ hash is only known to whoever can already decrypt it, so deriving from it
+ would make the block undecryptable.
.. code-block:: text
+----------------------------+
| version number (2 byte) |
+----------------------------+
- | nonce (32 byte) |
+ | nonce (24 byte) |
+----------------------------+
| JSON length n (4 byte) |
+----------------------------+
@@ -178,6 +189,14 @@ payload follows the common Taler signing structure with a
``purpose`` field (see :ref:`Signatures` in the API common conventions
for the general format). The specific payloads are:
+.. warning::
+
+ The three purpose numbers below are **not in the GANA registry yet**, so
+ both the server and the wallet spell them out locally. Until they are
+ registered, the wire format depends on whether a given build happened to
+ have them defined, and a mismatch shows up only as a signature that fails
+ to verify.
+
.. sourcecode:: c
/**
@@ -517,10 +536,27 @@ Blob format
Similar to blocks, each blob will consist of 2-byte version number, the 4-byte
data length, the gzipped data, and a padding to the next whole kilobyte. The
blob will be encrypted using a key derived from the wallet's backup encryption
-key and the hash of the unencrypted file.
+key and the hash of the unencrypted file:
-The hash used to index the object in the store will be computer from the
-encrypted blob using SHA-512 and truncated to 32 bytes.
+.. code-block:: text
+
+ key = KDF(32, backup_key, "taler-sync-blob-secret-salt", H(plaintext))
+ uid = H(key)
+
+Every blob therefore has its own key. The 64-byte ``uid``, which is the
+SHA-512 hash of that key, is what indexes the object in the store and is the
+only one of the two the sync server ever learns; the key itself is stored
+*inside the blocks* that reference the blob, where it doubles as the
+reference to the object that has to be fetched.
+
+The key is thus all a wallet needs to both locate and decrypt a blob, which
+is the only thing a block carries. The `secretbox`_ nonce is consequently
+derived from the key as well, as the first 24 bytes of ``H(key)``. Nonce
+reuse cannot occur, because distinct plaintexts derive distinct keys.
+
+Because the key is derived from the plaintext, blobs are content-addressed:
+identical contents yield the same key, UID and ciphertext, so an unchanged
+blob is only ever uploaded once.
.. code-block:: text
@@ -1625,13 +1661,68 @@ denomination. In turn, all the sign and spend operations of the deleted coins
must also be deleted, since they are in the ``coins`` deletion group and thus
contain a reference to a coin.
-.. TODO:
- Backup process
- --------------
+Backup process
+--------------
-.. TODO:
- Backup schedule
- ---------------
+Collecting increments
+~~~~~~~~~~~~~~~~~~~~~
+
+Wallet transactions record what they changed by appending increments to a
+pending buffer, held in the wallet's backup configuration record. The
+recording happens **within the same database transaction that performs the
+change**, so that the change and the increment describing it commit together.
+A wallet can therefore never end up in a state that its backup does not know
+about, however abruptly it is shut down.
+
+Increments are recorded as a flat list; which section of the
+``IncrementSet`` an increment lands in is derived from its
+``type``, so that transaction code never names a section itself and a new
+increment type cannot be added without giving it a home.
+
+A wallet that has not set up backup yet has no encryption key to protect the
+increments with, so recording is a no-op rather than an error.
+
+The backup cycle
+~~~~~~~~~~~~~~~~
+
+One cycle takes whatever increments have accumulated, packs them into a
+block, and appends that block to the account's chain:
+
+1. In a single database transaction, move the pending increments out of the
+ buffer and into an *in-flight block*, storing its nonce, hash, contents
+ and the nonce of the block it is to be appended after.
+2. Upload any blobs the block references, then the block itself.
+3. Once the provider has acknowledged the block, discard the in-flight block
+ and advance the pointer to the last acknowledged block.
+
+The hand-over in step 1 is what makes the cycle resilient: the increments are
+never absent from both the buffer and a block. A wallet that dies at any
+point either finds increments still pending, or finds an in-flight block and
+retries it — under its **original nonce**, which the server answers with
+``304 Not modified`` if the upload did in fact land. Increments are thus
+neither lost nor backed up twice, and a cycle that has packed a block always
+retries it before packing new increments, so the chain stays ordered.
+
+Backup schedule
+---------------
+
+A backup runs at *critical points* of wallet operations, and on a schedule
+otherwise.
+
+A critical point is one past which losing the device loses money or user
+data that cannot be reconstructed. The canonical example is a withdrawal:
+coin secrets are derived from the withdrawal group's seed, so a backup is
+triggered once every planchet has been generated and persisted but **before**
+the exchange is asked to sign them. Past that point the exchange considers
+the coins withdrawn while a wallet restored from an older backup could no
+longer reconstruct them.
+
+Triggering is only a nudge, and deliberately carries no guarantee: it happens
+after the recording transaction has committed, and nothing depends on it
+having run. Independently, a periodic task runs a cycle at a fixed interval,
+which is what catches increments whose trigger never fired — because the
+wallet was killed, was offline, or because the operation that produced them
+has no critical point of its own.
.. TODO:
Restore process
@@ -1641,14 +1732,96 @@ contain a reference to a coin.
Restore schedule
----------------
+Wallet-core API
+---------------
+
+Backup providers and the wallet's backup key are managed through the
+wallet-core API. All requests below are available on every platform.
+
+.. note::
+
+ The request handlers are currently declared but not yet implemented; the
+ collection and scheduling mechanisms described above are.
+
+.. ts:def:: AddBackupProviderRequest
+
+ interface AddBackupProviderRequest {
+ backupProviderBaseUrl: string;
+
+ name: string;
+
+ // Activate the provider. Should only be done after
+ // the user has reviewed the provider.
+ activate?: boolean;
+ }
+
+``addBackupProvider`` registers a sync server. Since a provider may charge an
+annual fee, the response either confirms the provider is ready or hands back
+a ``taler://pay/...`` URI for the account payment:
+
+.. ts:def:: AddBackupProviderResponse
+
+ type AddBackupProviderResponse =
+ | { status: "ok" }
+ | { status: "payment-required"; talerUri?: string };
+
+``removeBackupProvider`` takes a `RemoveBackupProviderRequest` naming the
+provider by base URL and returns an empty object.
+
+.. ts:def:: RemoveBackupProviderRequest
+
+ interface RemoveBackupProviderRequest {
+ backupProviderBaseUrl: string;
+ }
+
+``getBackupInfo`` reports the wallet's backup identity and the state of each
+known provider, including its terms, payment status and the outcome of the
+last backup attempt.
+
+.. ts:def:: BackupInfo
+
+ interface BackupInfo {
+ walletRootPub: string;
+ deviceId: string;
+ providers: ProviderInfo[];
+ }
+
+``getBackupRecovery`` returns the secret needed to restore the wallet on
+another device, along with the providers to fetch the blocks from. It is
+what the user backs up out of band, and what a restoring wallet is fed.
+
+.. ts:def:: BackupRecovery
+
+ interface BackupRecovery {
+ walletRootPriv: string;
+ providers: {
+ name: string;
+ url: string;
+ }[];
+ }
+
+Account keys are not part of any of these payloads: they are derived from the
+wallet root key and the provider's base URL, so each provider sees an
+unlinkable account public key and only the root key has to be preserved.
+
+.. code-block:: text
+
+ account_priv = KDF(32, wallet_root_priv,
+ "taler-sync-account-key-salt", provider_base_url)
+
Definition of done
==================
* [x] Design backup schema.
* [ ] Design incremental sync.
-* [ ] Design backup/restore schedules.
-* [ ] Design wallet-core API.
-* [ ] Wallet-core implementation.
+* [ ] Design backup/restore schedules (partial: backup process and schedule
+ done; restore still missing).
+* [x] Design wallet-core API.
+* [ ] Wallet-core implementation (partial: block and blob encoding, CRDT
+ merge, sync protocol client and signatures, increment collection and the
+ scheduled backup cycle done; the API request handlers, blob upload,
+ fetching and merging remote blocks, item deletion and the remaining
+ increment types still missing).
* [x] Design sync API (+ auth).
* [ ] Server-side implementation (partial: block GET/POST/PUT/DELETE, object
store GET/POST with reference counting, /config and payments done;
