Merge branch 'master' of git+ssh://git.taler.net/docs

6 files changed, 421 insertions, 4 deletions

diff --git a/design-documents/000-template.rst b/design-documents/000-template.rst
index c969f862..f620248d 100644
--- a/design-documents/000-template.rst
+++ b/design-documents/000-template.rst
@@ -1,4 +1,25 @@
 Template
 ########
 
-FIXME: define template
+Summary
+=======
+
+Motivation
+==========
+
+Requirements
+============
+
+Proposed Solution
+=================
+
+Alternatives
+============
+
+Drawbacks
+=========
+
+Discussion / Q&A
+================
+
+(This should be filled in with results from discussions on mailing lists / personal communication.)
diff --git a/design-documents/001-new-browser-integration.rst b/design-documents/001-new-browser-integration.rst
index f436a4f7..8dbe2a43 100644
--- a/design-documents/001-new-browser-integration.rst
+++ b/design-documents/001-new-browser-integration.rst
@@ -1,10 +1,30 @@
 Design Doc 001: New Browser Integration
 #######################################
 
-.. note::
+.. warning::
 
-  This design document is currently a draft, it
-  does not reflect any implementation decisions yet.
+  We have decided not to follow through with the proposed solution in this
+  design doc.  We care a lot about a nice upgrade path for when better
+  browser integration becomes available.  Encouraging the ``#taler://`` fragment
+  based integration might lead merchant frontends to **only** support this type
+  of integration.
+
+  Instead, the following path will be taken:
+
+  1. CSS-based presence detection will be removed from the wallet,
+     as there is no satisfactory upgrade path to better mechanisms
+  2. Manual triggering will be implemented as described in this design doc.
+  3. The ``webRequest`` permission that allows ``"Taler: "`` header based
+     browser integration will become opt-in.
+  4. The interactive API will be put on hold.  Instead, SPAs should
+     ask the user to open the wallet popup (and/or render a QR code for mobile wallets).
+  5. To enable easier integration for merchants, the reference merchant backend
+     might include a page to trigger payments, which displays the QR code
+     correctly, does long-polling via JS and serves the ``"Taler: "`` header.
+  6. The presence detection ``taler://`` URI described in this document
+     will **not** be supported, as allowing presence detection might
+     encourage merchants to treat mobile / detached wallets as 2nd class
+     citizens.
 
 Summary
 =======
diff --git a/design-documents/003-tos-rendering.rst b/design-documents/003-tos-rendering.rst
new file mode 100644
index 00000000..5f6d8157
--- /dev/null
+++ b/design-documents/003-tos-rendering.rst
@@ -0,0 +1,84 @@
+Design Doc 003: ToS rendering
+#############################
+
+Summary
+=======
+
+This document describes how terms of service (ToS) as well as other "legal
+agreement documents" are served, represented and rendered.
+
+Motivation
+==========
+
+Different exchanges and backup/sync providers each have their custom legal
+agreement documents.  As we don't know all providers and they are not centrally
+registered anywhere, these documents can't be hardcoded into wallet
+applications.  Instead, these service providers expose endpoints that allow
+downloading the latest version of these legal agreement documents.
+
+These documents must be rendered on a variety of platforms in a user-friendly
+way.
+
+Proposed Solution
+=================
+
+The service providers can output legal agreements in various formats,
+determined via the ``"Accept: "`` request header.  The format provider **must**
+support the ``text/plain`` mime type.  The format provider **must** support
+the ``text/markdown`` mime type.  Except for styling and navigation, the
+content of each format of the same legal agreement document **should** be the
+same.
+
+Legal documents with mime type ``text/markdown`` **should** confirm to the
+`commonmark specification <https://commonmark.org/>`__.
+
+When wallets render ``text/markdown`` legal documents, they **must** disable
+embedded HTML rendering.  Wallets **may** style the markdown rendering to improve
+usability.  For example, they can make sections collabsible or add a nagivation side-bar
+on bigger screens.
+
+It is recommended that the ``text/markdown`` document is used as the "master
+document" for generating the corresponding legal agreement document in other
+formats.  However, service providers can also provide custom versions with more
+appropriate styling, like a logo in the header of a printable PDF document.
+
+Alternatives
+============
+
+We considered and rejected the following alternatives:
+
+* Use only plain text.  This is not user-friendly, as inline formatting (bold,
+  italic), styled section headers, paragraphs wrapped to the screen size,
+  formatted lists and tables are not supported.
+
+* Use HTML.  This has a variety of issues:
+
+  * Service providers might provide HTML that does not render nicely on the
+    device that our wallet application is running on.
+  * Rendering HTML inside the application poses security risks.
+
+* Use a strict subset of HTML.  This would mean we would have to define some
+  standardized subset that all wallet implementations support, which is too
+  much work.  Existing HTML renderers (such as Android's ``Html.fromHTML``)
+  support undocumented subsets that lack features we want, such as ordered
+  lists.  Defining our own HTML subset would also make authoring harder, as it
+  forces authors of legal agreement documents to author in our HTML subset, as
+  conversion tools from other format will not generate output in our HTML
+  subset.
+
+* Use reStructuredText (directly or via Sphinx).  This at first looks like an
+  obvious choice for a master format, since Taler is already using reStructuredText
+  for all its documentation.  But it doesn't work out well, since the only maintained
+  implementation of a parser/renderer is written in Python.  Even with the Python implementation
+  (docutils / Sphinx), we can't convert ``.rst`` to Markdown nicely.
+
+Drawbacks
+=========
+
+* Markdown parsing / rendering libraries can be relatively large.
+
+Discussion / Q&A
+================
+
+* Should the legal agreement endpoints have some mechanism to determine what
+  content types they support?
diff --git a/design-documents/004-wallet-withdrawal-flow.rst b/design-documents/004-wallet-withdrawal-flow.rst
new file mode 100644
index 00000000..e448a773
--- /dev/null
+++ b/design-documents/004-wallet-withdrawal-flow.rst
@@ -0,0 +1,129 @@
+Design Doc 004: Wallet Withdrawal Flow
+######################################
+
+Summary
+=======
+
+This document describes the recommended way of implementing the user experience
+of withdrawing digital cash in GNU Taler wallets.
+
+Motivation
+==========
+
+When digital cash is withdrawn, it is tied to and in custody of an exchange.
+There can be many exchanges offered by different entities,
+each having their custom legal agreement documents and fee structures.
+The user is free to choose an exchange.
+Therefore, the process of withdrawing needs to account for this choice.
+
+Proposed Solution
+=================
+
+There are three screens involved in the process:
+
+1. **Select exchange**:
+   Here the user can pick an exchange from a list of known exchanges
+   or add a new one for immediate use.
+   For details see :doc:`002-wallet-exchange-management`.
+2. **Display an exchange's Terms of Service**:
+   Shows the terms and gives an option to accept them.
+   For details see :doc:`003-tos-rendering`.
+3. **Withdrawal details and confirmation**:
+   This should show the amount to be withdrawn along with its currency,
+   the currently selected exchange and the fee charged by it for the withdrawal.
+
+The user flow between these screens is described in the following graph:
+
+.. graphviz::
+
+   digraph G {
+       rankdir=LR;
+       nodesep=0.5;
+       default_exchange [
+           label = "Has default\nexchange?";
+           shape = diamond;
+       ];
+       tos_changed [
+           label = "ToS\nchanged?";
+           shape = diamond;
+       ];
+       tos_accepted [
+           label = "ToS\naccepted?";
+           shape = diamond;
+       ];
+       accept_tos [
+           label = "Accept\nToS?";
+           shape = diamond;
+       ];
+       withdrawal_action [
+           label = "Withdrawal\nAction";
+           shape = diamond;
+       ];
+       select_exchange [
+           label = "Select\nexchange";
+           shape = rect;
+       ];
+       tos [
+           label = "ToS";
+           shape = rect;
+       ];
+       withdraw [
+           label = "Confirm\nwithdrawal";
+           shape = rect;
+       ];
+       transactions [
+           label = "Transactions";
+           shape = circle;
+       ];
+
+       default_exchange -> tos_changed [label="Yes"];
+       default_exchange -> select_exchange [label="No"];
+       tos_changed -> tos [label="Yes"];
+       tos_changed -> withdraw [label="No"];
+       select_exchange -> tos_accepted;
+       tos_accepted -> tos_changed [label="Yes"];
+       tos_accepted -> tos [label="No"];
+       tos -> accept_tos;
+       accept_tos -> withdraw [label="Yes"];
+       accept_tos -> select_exchange [label="No"];
+       withdraw -> withdrawal_action;
+       withdrawal_action -> transactions [label="Confirm"];
+       withdrawal_action -> select_exchange [label="Change Exchange"];
+
+       { rank=same; tos_accepted; tos_changed; }
+       { rank=same; select_exchange; tos; }
+       { rank=same; withdrawal_action; withdraw; }
+   }
+
+This enables the user to change the current exchange at any time in the process.
+It ensures that the latest version of the exchange's terms of service have been accepted by the user
+before allowing them to confirm the withdrawal.
+
+Some special regional or test currencies might have only a single known exchange.
+For those, the wallet should not offer the option to change an exchange.
+
+Alternatives
+============
+
+We considered and rejected the following alternatives:
+
+* Do not allow more than one exchange to make Taler simpler to use and understand:
+  Taler wants to allow custom exchanges for custom currencies
+  and foster competition between exchanges for the same currency
+  to provide the best possible service to users at the lowest fee.
+* Do not require acceptance to terms of service:
+  Having these terms and prompting the user to accept them
+  is a legal and business requirement in many jurisdictions,
+  so Taler needs to support them.
+  However, Taler encourages exchanges to keep their terms as short and simple as possible.
+
+Discussion / Q&A
+================
+
+* Should wallets pre-set a default exchange for the most common currencies,
+  so that users will not be burdened to understand exchanges and their fee structures
+  when making their first withdrawal?
+  This could increase user retention, but discourage
+* What should happen when an exchange changes its terms of service
+  and the user wants to use the funds stored there,
+  but does not initiate a new withdrawal with that exchange?
diff --git a/design-documents/005-wallet-backup-sync.rst b/design-documents/005-wallet-backup-sync.rst
new file mode 100644
index 00000000..d91c3ff5
--- /dev/null
+++ b/design-documents/005-wallet-backup-sync.rst
@@ -0,0 +1,160 @@
+Design Doc 005: Wallet Backup and Sync
+######################################
+
+.. warning::
+
+  This is an unfinished draft.
+
+Summary
+=======
+
+This document discusses considerations for backup and synchronization of wallets.
+
+
+Requirements
+============
+
+* Backup and sync must not require any synchronous communication between the
+  wallets
+* Wallets operating (payments/withdrawals/...) for longer periods of time without
+  synchronizing should be handled well
+* Conflicts should be resolved automatically in pretty much all cases
+* One wallet can be enrolled with multiple sync servers, and a wallet can
+  join
+* Other wallets connected to the sync server are trusted.
+
+Proposed Solution
+=================
+
+The blob stored on the backup/sync server is a compressed and encrypted JSON file.
+
+The various entity types managed by the wallet are modeled LWW-Sets (Last Write
+Wins Set CRDT).  Timestamps for inserts/deletes are are Lamport timestamps.  Concurrent, conflicting insert/delete
+operations are resolved in favor of "delete".
+
+The managed entities are:
+
+* set of exchanges with the data from /keys, /wire
+* set of directly trusted exchange public keys
+* set of trusted auditors for currencies
+* set of reserves together with reserve history
+* set of accepted bank withdrawal operations
+* set of coins together with coin history and blinding secret (both for normal withdrawal and refresh)
+  and coin source info (refresh operation, tip, reserve)
+* set of purchases (contract terms, applied refunds, ...)
+* assignment of coins to their "primary wallet"
+
+(Some of these might be further split up to allow more efficient updates.)
+
+Entities that are **not** synchronized are:
+
+* purchases before the corresponding order has been claimed
+* withdrawal operations before they have been accepted by the user
+
+Entities that **could** be synchronized (to be decided):
+ 
+* private keys of other sync accounts
+* coin planchets
+* tips before the corresponding coins have been withdrawn
+* refresh sessions (not only the "meta data" about the operation,
+  but everything)
+ 
+
+Garbage collection
+------------------
+
+There are two types of garbage collection involved:
+
+1. CRDT tombstones / other administrative data in the sync blob.  These can be deleted
+   after we're sure all wallets enrolled in the sync server have a Lamport timestamp
+   larger than the timestamp of the tombstone.  Wallets include their own Lamport timestamp
+   in the sync blob:
+
+   .. code:: javascript
+
+     {
+       clocks: {
+         my_desktop_wallet: 5,
+         my_phone_wallet: 3
+       },
+       ...
+     }
+
+   All tombstones / overwritten set elements with a timestamp smaller than the
+   smallest clock value can be deleted.
+
+2. Normal wallet GC.  The deletion operations resulting from the wallet garbage
+   collection (i.g. deleting legally expired denomination keys, coins, exchange
+   signing keys, ...) are propagated to the respective CRDT set in the sync
+   blob.
+
+
+Ghost Entities
+--------------
+
+Sometimes a wallet can learn about an operation that happened in another synced
+wallet **before** a sync over the sync server happens.  An example of this is a
+deposit operation.  When two synced wallets spend the same coin on something,
+one of them will receive an error from the exchange that proves the coin has
+been spent on something else.  The wallet will add a "ghost entry" for such an
+event, in order to be able to show a consistent history (i.e. all numbers
+adding up) to the user.
+
+When the two wallets sync later, the ghost entry is replaced by the actual
+purchase entity from the wallet that initiated the spending.
+
+Ghost entities are not added to the sync state.
+
+
+Multiple sync servers
+---------------------
+
+When a wallet is connected to multiple sync servers, it automatically
+propagates changes it received from one sync server to the others.  Local
+changes made by the wallet are propoagated to all sync servers.  The goal of
+this is to make the state of the sync servers converge.
+
+The different sync servers one wallet is enrolled with do not necessarily
+have the same set of other wallet enrolled.  Each sync server has a separate Lamport clock
+and contains a separate CRDT.
+
+
+References
+==========
+
+* Shapiro, M., Preguiça, N., Baquero, C., & Zawirski, M. (2011). A
+  comprehensive study of convergent and commutative replicated data types. [`PDF <https://hal.inria.fr/inria-00555588/document>`__]
+
+Discussion / Q&A
+================
+
+* Why is backup/sync not split into two services / use-cases?
+
+  * For privacy reasons, we can't use some interactive sync service.  Thus we
+    use the backup blob as a CRDT that also synchronization for us.
+
+* Do we synchronize the list of other backup enrollments?  How
+  do we handle distributing the different private keys for them?
+
+  * If we automatically sync the sync enrollments and the old sync account
+    is compromised, the new sync account would automatically be compromised as well!
+
+  * If every wallet had its own sync key pair, we could select which existing wallets
+    to roll over as well.
+
+* How do we handle a synced wallet that becomes malicious deleting all coins or purchased products?
+
+  * This needs to balance the genuine need to permanently delete data.
+  * Should the sync server allow to fetch previous versions of the sync blob?
+  * Should the individual wallets keep tombstones (i.e. entities just marked as deleted)
+    around for some time, or should they delete and "sanitize" (delete data not needed for the CRDT)
+    tombstones as soon as possible?
+
+* How are wallets identified for backup/sync?
+
+  * UUID / EdDSA pub and nick name?  When nickname clashes,
+    some number is added based on lexical sort of the random id ("phone#1", "phone#2").
+
+* Do we have a passphrase for our backup account key(s)?
+
+  * ???
diff --git a/design-documents/index.rst b/design-documents/index.rst
index 609cc2d0..590cd451 100644
--- a/design-documents/index.rst
+++ b/design-documents/index.rst
@@ -12,3 +12,6 @@ and protocol.
   000-template
   001-new-browser-integration  
   002-wallet-exchange-management
+  003-tos-rendering
+  004-wallet-withdrawal-flow
+  005-wallet-backup-sync