taler-docs

Documentation for GNU Taler components, APIs and protocols
Log | Files | Refs | README | LICENSE

034-wallet-db-migration.rst (3216B)

      1 DD 34: Considerations for Wallet Database Migrations
      2 ####################################################
      3 
      4 Summary
      5 =======
      6 
      7 This design document discusses considerations for wallet database migrations.
      8 
      9 Motivation
     10 ==========
     11 
     12 The schema of the GNU Taler Wallet database is evolving over time, either
     13 because new features are added or because bugs are fixed.
     14 
     15 Requirements
     16 ============
     17 
     18 * Migrations may not result in data loss and must be automatic.
     19 * Our schema migration must be compatible with how IndexedDB works. This means that we can't 
     20   do arbitrary schema migrations at any time, but need to increment the IndexedDB database version
     21   every time we add/remove/change an object store or index.
     22 
     23 Proposed Solution
     24 =================
     25 
     26 The schema of the wallet database is described in code in
     27 https://git.taler.net/wallet-core.git/tree/packages/taler-wallet-core/src/db.ts#n1959
     28 (``walletStoresV1``). This schema description is used to initialize and upgrade the
     29 database automatically.
     30 
     31 In IndexedDB terminology, the wallet has two databases:
     32 
     33 1. The ``"taler-wallet-meta"`` stores metadata about the current major-version database
     34 2. The major-version database (currently ``"taler-wallet-main-v9"`` stores the
     35    actual data of the wallet.
     36 
     37 This indirection allows major database migrations to be safe despite the
     38 limitations of IndexedDB.  The computation that is allowed during an IndexedDB
     39 migration is very limited. By migrating to a completely new database, we can
     40 keep around the old database until we're sure that the migration has succeeded
     41 and, if required, push new code to fix migration errors.
     42 
     43 We have three different mechanisms to introduce changes to the database:
     44 
     45 1. Major migrations.  These migrations introduce a new major-version database and must manually
     46    migrate the data from the previons major-version database. This migration should be
     47    added in ``db.ts#openTalerDatabase``.
     48    Major migrations should be used **very** seldomly.  It can make sense to implement them
     49    as a backup cycle, i.e. implement a backup export from the old version, upgrade to
     50    the latest backup version and then re-import into the new major-version database.
     51 2. Minor schema migrations: These migrations add or remove object stores or indexes.
     52    They are done by adding new elements to the schema descriptions **and** specifying
     53    the ``versionAdded`` attribute. This causes an IndexedDB upgrade transaction
     54    to be executed.
     55 3. Fixups. Fixups change data within or between minor schema versions and
     56    contain arbitrary code to make changes to object stores.  They are usually used
     57    when a new mandatory field is added to an existing object store or some data
     58    format changes.  Fixups are also useful to retroactively fix bugs
     59    introduced by previously deployed wallet versions.
     60    They must be added to ``db.ts#walletDbFixups``
     61 
     62 Alternatives
     63 ============
     64 
     65 * Per-object versioning instead of using IndexedDB minor versions
     66 * Always use the backup mechanism to upgrade the database
     67   
     68   * Would be overkill for minor migrations
     69 
     70 Drawbacks
     71 =========
     72 
     73 N/A.
     74 
     75 Discussion / Q&A
     76 ================
     77 
     78 (This should be filled in with results from discussions on mailing lists / personal communication.)