summaryrefslogtreecommitdiff
path: root/design-documents/011-auditor-db-sync.rst
blob: fb2e3bea1e648fe371db332c80c363288f7d13da (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
Design Doc 011: Auditor-Exchange Database Synchronization
#########################################################

Summary
=======

Ways for the auditor to obtain a current copy of the exchange database (to
verify that the exchange is operating correctly) are discussed.


Motivation
==========

The Taler auditor is expected to check that the exchange is operating
correctly.  For this, it needs (read-only) access to the exchange database and
to the bank account of the exchange.  Bank account access is a matter of
setting up an additional user with limited rights with the bank and is out of
the scope of this document.  For database access, the auditor should not trust
the exchange. In particular, the auditor must assume that the exchange may
violate basic database constraints (like foreign keys) or delete/alter records
maliciously. However, we also do not want to complicate the auditor logic to
cope with with violations of well-formedness constraints (like foreign keys or
non-NULL values or size constraints on fields).  Finally, the mechanism by
which the auditor obtains the database must provide a reasonably current
database and the process must perform reasonably well.


Requirements
============

* The solution must allow data to be copied incrementally.
* The solution must tolerate network outages and recover after connectivity
  between exchange and auditor is restored.
* The solution must enable the auditor database to serve as a full backup
  of the exchange's database (even if possibly slightly outdated due to
  asynchronous replication or network outages).
* The solution must scale, in particular if the exchange shards the database,
  the auditor must be also able to use the same kind of sharding and the
  synchronization should be possible per shard.
* The synchronization mechanism must not allow an attacker controlling the
  exchange database to delete or modify arbitrary data from the auditor's copy
  via the synchronization mechanism (in other words, some tables are
  append-only and unalterable).
* The solution must support database schema updates. Those may require some
  downtime and closely coordinated work between exchange and auditor.
* The solution must enable eventual garbage collection at the exchange to
  be permitted and replicated at the auditor (e.g. DELETE on usually append-only
  tables due to a CASCADE from expired denomination keys).
* The synchronization mechanism should raise an alert if the exchange violates basic
  constraints (unexpected schema changes, deletion/motification on append-only
  tables) and then NOT replicate those changes. The auditor's internal asynchronous
  helper may then soft-fail (log and exit) until the exchange has rectified the
  problem (by manual, human intervention resulting in an exchange master database
  that again maintains the required invariants).
  After the corrected master database has been again synchronized with the
  primary copy of the auditor, the auditor's helper is resumed and can continue to
  copy the (now valid) database records into the auditor's internal version.
* A good solution would work independently of the specific database used.


Proposed Solution
=================

* Use "common" incremental database replication (whichever is
  approproate for the exchange database setup, synchronous
  or asynchronous) to make a 1:1 copy of the exchange database
  at the auditor.  This should work for any full-featured
  modern database. This "ingress" copy cannot be trusted, as constraint
  violations or deletions would also be replicated.
* Use helper process to SELECT against the local "ingress" copy (by
  SERIAL ID => make sure all append-only tables have one!)
  to copy append-only tables to a second, "trusted" and fully
  auditor-controlled copy of the database.
  Order (or transactionally group) SELECT
  statements to ensure foreign key constraints are maintained.
  For mutable tables (basically, only current reserve balance)
  do not make another copy, but do have logic to recompute mutable
  tables from other data *if* we need to recover from backup.
* On schema migration, halt exchange, once auditor DB has
  synchronized, update all DB schema (the "ingress" DB schema
  may be updated automatically when the exchange DB schema is
  migrated, but the "trusted" DB of the auditor must most likely
  be manually migrated), then finally resume "ingress" to "trusted"
  helper-based DB synchronization and restart the exchange.
* For GC, simply run GC logic also on auditor's "trusted" copy.
  (The synchronization mechanism will take care of the primary copy,
  and the helper to copy should not be disturbed by the DELETE operations
  anyway.)
* The auditor's "ingress" database should be well isolated from
  the rest of the auditor's system and database
  (different user accounts). The reason is that we should not
  assume that the Postgres replication code is battle-tested with
  malicious parties in mind.
* The canonical Postgres synchronization between exchange and the
  auditor's "ingress" database must use transport security.

The above solution does not gracefully handle mutable tables on which
the exchange performs UPDATE statements, as such updates will not bump
the BIGSERIAL and thus would not be replicated by the helper.  Thus, we
need to consider all tables that the exchange ever performs UPDATE on.
Those are:

* /reserves/ --- the exchange updates the remaining reserve balance;
  here the auditor currently performs a sanity check against
  its own reserve balance calculation.  The proposed way to address
  this is to make this sanity check optional and to be only used if
  the auditor auditor runs against the "primary" exchange database
  (like an internal audit).  This is acceptable, as an inaccurate
  reserve balance is mostly used to raise an early warning and not
  indicative of any actualized financial gains or losses from the
  exchange.
* /deposits/ --- the exchange updates the /tiny/ and /done/ bit
  fields. /tiny/ can be trivially established by the auditor, and
  we can simply avoid the auditor considering that bit. /done/
  was so far only used to enrich the reporting.  The proposed way
  to address the uses of both fields is thus to only use them in
  internal audits (against the primary exchange database). Both
  can be safely ignored by the external audit.
* /prewire/ --- the exchange updates the /finished/ and /failed/
  bits. The entire table is not used by the auditor and its
  main values cannot be validated by the auditor anyway.
* /auditors/ --- the exchange updates the /is_active/ and /last_change/
  fields.  The entire table is of no concern to the auditor.

A good order for replicating the tables should be:

* exchange_sign_keys
* signkey_revocations
* auditors
* denominations
* denomination_revocations
* auditor_denom_sigs
* reserves
* reserves_out
* reserves_in
* reserves_close
* known_coins
* deposits
* refunds
* refresh_commitments
* refresh_transfer_keys
* refresh_revealed_coins
* recoup_refresh
* recoup



Alternatives
============

* Copy the Postgres WAL, filter it for "illegal" operations
  and then apply it at the auditor end.  Disadvantages: WAL
  filtering is not a common operation (format documented?),
  this would be highly Postgres-specific, and would require
  complex work to write the filter.  Also unsure how one
  could later recover gracefully from transient errors
  (say where the exchange recified a bogus DELETE).
* Directly SELECT against the (remote) exchange DB and then
  INSERT/UPDATE at the auditor's local copy. Disadvantages:
  remote SELECT likely very expensive due to high latency.
  Diagnostics more difficult. May expose exchange to additional
  risks from auditor, such as attacks exhausting DB resources
  by running expensive SELECTs.




Drawbacks
=========

* SERIAL IDs required in all tables that are "append-only" / immutable.
* Additional custom logic required to recompute mutable tables
  on-demand.
* Limited ability to cope with mutable tables, imposes restrictions
  on future exchange database evolution.
* Helper logic to SELECT data in batches that will certainly
  maintain invariants may be a bit tricky, but in principle
  the foreign key constraints should form a DAG, simply dictating
  the order in which new entries are to be copied.  It may also
  be that simply running "big" transactions across all tables
  is the answer, to be investigated what performs better.
* A malicious exchange could theoretically send expensive transactions
  to the auditor via the replication mechanism (possibly ones that
  it did not even execute locally itself) to DoS the "ingress"
  database. This would be noticed primarily by load
  monitoring or even the auditor lagging unusually far behind the
  exchange's transaction history. We believe this is acceptable,
  as it would imply highly visible malicious exchange behavior for
  virtually no significant gain.
* The proposed solution does not create a transactional, synchronous
  write-only log as suggested by CodeBlau (see audit report, Section 9.4).
  We believe doing so would be overly costly, both in terms of
  complexity and performance, for limited gains.


Discussion / Q&A
================