034-wallet-db-migration.rst (4280B)
1 DD 34: Considerations for Wallet Database Migrations 2 #################################################### 3 4 :Design status: Accepted 5 :Implementation status: Implemented 6 :DD shepherd: TBD 7 :Historical contributors: Florian Dold 8 :First published: 2023-01-27 9 :Last substantive change: 2023-01-27 10 :Implementation evidence: taler-typescript-core (2023-07-11, 2026-07-19, 2026-08-06) 11 :Normative references: :doc:`../developer/taler-wallet-developer` 12 13 .. note:: 14 15 The wallet now has both IndexedDB and native SQLite database backends. The 16 The wallet developer manual describes the migration contract; current 17 wallet source is authoritative for backend-specific implementation details. 18 19 Summary 20 ======= 21 22 This design document discusses considerations for wallet database migrations. 23 24 Motivation 25 ========== 26 27 The schema of the GNU Taler Wallet database is evolving over time, either 28 because new features are added or because bugs are fixed. 29 30 Requirements 31 ============ 32 33 * Migrations may not result in data loss and must be automatic. 34 * Migrations of the IndexedDB backend must be compatible with how IndexedDB 35 works. This means that we cannot do arbitrary schema migrations at any time, 36 but need to increment the IndexedDB database version every time we add, 37 remove or change an object store or index. The native SQLite backend has a 38 separate versioned migration mechanism. 39 40 Proposed Solution 41 ================= 42 43 The IndexedDB schema of the wallet database is described in 44 https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/db/indexeddb/schema.ts. 45 The native SQLite schema is described separately in 46 https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/db/sqlite/schema.ts. 47 These schema descriptions are used to initialize and upgrade their respective 48 databases automatically. 49 50 In IndexedDB terminology, the wallet has two databases: 51 52 1. The ``"taler-wallet-meta"`` stores metadata about the current major-version database 53 2. The major-version database (currently ``"taler-wallet-main-v10"``) stores the 54 actual data of the wallet. 55 56 This indirection allows major database migrations to be safe despite the 57 limitations of IndexedDB. The computation that is allowed during an IndexedDB 58 migration is very limited. By migrating to a completely new database, we can 59 keep around the old database until we're sure that the migration has succeeded 60 and, if required, push new code to fix migration errors. 61 62 The IndexedDB backend has three mechanisms to introduce changes: 63 64 1. Major migrations. These migrations introduce a new major-version database and must manually 65 migrate the data from the previous major-version database. This migration should be 66 added in ``db/indexeddb/database.ts#openTalerDatabase``. 67 Major migrations should be used **very** seldomly. It can make sense to implement them 68 as a backup cycle, i.e. implement a backup export from the old version, upgrade to 69 the latest backup version and then re-import into the new major-version database. 70 2. Minor schema migrations: These migrations add or remove object stores or indexes. 71 They are done by adding new elements to the schema descriptions **and** specifying 72 the ``versionAdded`` attribute. This causes an IndexedDB upgrade transaction 73 to be executed. 74 3. Fixups. Fixups change data within or between minor schema versions and 75 contain arbitrary code to make changes to object stores. They are usually used 76 when a new mandatory field is added to an existing object store or some data 77 format changes. Fixups are also useful to retroactively fix bugs 78 introduced by previously deployed wallet versions. 79 They must be added to ``db/indexeddb/fixups.ts#walletDbFixups``. 80 81 The native SQLite backend uses ordered schema migrations in 82 ``db/sqlite/schema.ts``. Migration from the IndexedDB emulation to the native 83 schema is implemented separately in ``db/migration/native.ts``. 84 85 Alternatives 86 ============ 87 88 * Per-object versioning instead of using IndexedDB minor versions 89 * Always use the backup mechanism to upgrade the database 90 91 * Would be overkill for minor migrations 92 93 Drawbacks 94 ========= 95 96 N/A. 97 98 Discussion / Q&A 99 ================ 100 101 (This should be filled in with results from discussions on mailing lists / personal communication.)