path: root/design-documents/037-wallet-transactions-lifecycle.rst

DD 37: Wallet Transaction Lifecycle
###################################

Summary
=======

This design doc discusses the lifecycle of transactions in wallet-core.

Motivation
==========

The transactions in wallet-core all should have an associated state machine.  All transactions
should have some common actions that work uniformly across all transactions.

Proposed Solution
=================

Common States
-------------

The following states apply to multiple different transactions.  Only pending
and aborting have transaction-specific sub-states, denoted by ``state(substate)``.

``initial``: The initial state is the result a user interaction. A transaction
may have more than one initial state (if it is started in multiple ways) or none
(if it's the result of another process).

``pending``: A pending transaction waits for some external event/service.
The transaction stays pending until its change on the wallet's material balance
is finished. Any pending state can be suspended and resumed.

There are some other distinctions for pending transactions:

* long-polling vs. exponential backoff: A pending transaction is either waiting
  on an external service by making a long-polling request or by repeating requests
  with exponential back-off.
* ``lastError``: A pending transaction is either clean (i.e. the network interaction
  is literally active in transmission or the external service successfully
  communicated that it is not ready yet and this is perfectly normal)
  or has a ``lastError``, which is a ``TalerErrorDetails``
  object with details about what happened during the last attempt to proceed
  with the transaction.

``done``: A transaction that is done does not require any more processing.  It also
never has a ``lastError`` but is considered successful.

``aborting``: Similar to a pending transaction, but instead of taking active steps to
complete the transaction, the wallet is taking active steps to abort it. The ``lastError``
indicates errors the wallet experienced while taking active steps to abort the transaction.

``aborted``: Similar to ``done``, but the transaction was successfully aborted
instead of successfully finished. It will have the information of when (timestamp) it was
aborted and in which pending sub-state the abort action was initiated. Also, we can
include more information information relevant to the transaction in `abortReason`

``suspended``: Similar to a ``aborted`` transaction, but the transaction was could be
resumed and may then still succeed.

``failed``: Similar to ``done``, but the transaction could not even be aborted properly.

``deleted``: A ``deleted`` state is always a final state.  We only use this
state for illustrative purposes. In the implementation, the data associated
with the transaction would be literally deleted.


Common Transitions
------------------

Transitions are actions or other events.

``[action:delete]``: Deleting a transaction
completely deletes the transaction from the database.  Depending on the type of
transaction, some of the other data *resulting* from the transaction might
still survive deletion. For example, deleting a withdrawal transaction does not
delete already successfully withdrawn coins. Deleting is only safe (no money lost)
on initial and final states (failed, aborted, done).

``[action:retry]``: Retrying a transaction *(1.)* stops ongoing long-polling
requests for the transaction *(2.)* resets the retry timeout *(3.)* re-runs the
handler to process the transaction. Retries are always possible the following
states: ``pending(*)`` and ``aborting(*)``.

.. attention::

   Should we show the retry timeout in the UI somewhere?  Should we show it in dev mode?

   SEBASJM: Since the wallet will retry anyway, maybe is better if we replace the "retry"
   button with a "try now" button and a side text "retrying in xxx seconds".

   CG: Instead of a side text, this *might* make a good mouse-over hint for
   a "retry" (or "try now") button. I would not make this overly visible with
   side-text as the information is not that important. The text should also be
   "retrying next at XXX" using an absolute time XXX --- otherwise the UI would
   be way too busy recomputing/updating all of these strings: Using an absolute time,
   we only have to redraw anything once a retry actually happened. Given that
   retries should basically never be > 24h (we can impose a hard cap), the absolute
   time can just be in the format HH:MM:SS (without day).

``[action:abort]``: Aborting a transaction either directly stops processing for the
transaction and puts it in an ``aborted`` state, or starts the necessary steps to
actively abort the transaction (e.g. to avoid losing money) and puts it in an
``aborting`` state.

``[action:suspend]``: Suspends a pending transaction, stopping any associated
network activities, but with a chance of trying again at a later time. This
could be useful if a user needs to save battery power or bandwidth and an
operation is expected to take longer (such as a backup, recovery or very large
withdrawal operation).

``[action:resume]``: Suspended transactions may be resumed, placing them back
into a pending state.

``[action:abort-force]``: Directly puts an ``aborting`` transaction into the
``failed`` state.


Whether aborting or resuming are possible depends on the transaction type, and
usually only one of the two choices should be offered.


.. image:: ../transaction-common-states.png
  :width: 400


Boxed labels indicate an end state in which it is safe to delete the
transaction record since no work is due.

Blue arrows are used for user-triggered actions (via UI buttons).


Common pending sub-states
-------------------------

During the pending state the transaction can go through several sub-states before
reaching a final state. Some of this sub-states are shared between different
transaction types:

``kyc-required``: The transaction can't proceed because the user needs to actively
finish a KYC process. Part of a withdrawal process or peer-to-peer push credit.

``aml-required``: The transaction can't proceed because the user needs to wait for
the exchange operator to conclude an AML investigation by the staff at the exchange.
The user is not expected to take any action and should just wait for the investigation
to conclude. Part of a withdrawal process or peer-to-peer push credit.

``aml-frozen``: The staff at the exchange decided that the account needed to be frozen.
The user should contact the exchange provider's customer service department and
seek resolution (possibly through the courts) to avoid loosing the funds for good.
Part of a withdrawal process or peer-to-peer push credit.


Transaction Type: Withdrawal
----------------------------

* ``pending(bank-register-reserve)``

  Initial state for bank-integrated withdrawals.  The wallet submits the reserve public key
  and selected exchange to the bank (via the bank integration API).  Note that if the
  user aborts at this stage, we do not know if the bank is in the confirmation stage,
  so we must still *try* to abort the transaction at the bank.

  * ``[processed-success] => pending(bank-confirming)``
  * ``[processed-error] => aborted(bank-to-wallet)``
  * ``[action:abort] => aborting(bank-to-wallet)``

* ``pending(bank-confirming)``

  The wallet waits until the bank has confirmed the withdrawal operation;
  usually the user has to complete a 2FA step to *approve* that the money is
  wired to the chosen exchange.  Note that the user's *approve* action is done
  in the bank's user interface and not the wallet's user interface. The wallet
  internally merely *polls* for the success or failure of the approve action.
  The wallet **may** occasionally (after some initial delay, especially on
  failures from the bank-poll to return any result) long-poll for the reserve
  status and, if successful, may then directly jump to
  ``pending(withdrawing-coins)`` if the reserve is filled even if the poll at
  the bank did not return success or failure.

  * ``[bank-poll-success] => pending(exchange-wait-reserve)``
  * ``[bank-poll-denied] => aborted(wallet-to-bank)``
  * ``[exchange-poll-success] => pending(withdrawing-coins)``
  * ``[action:abort] => aborting(wallet-to-bank)``

* ``pending(exchange-wait-reserve)``

  Initial state for manual withdrawals.  Here, the wallet long-polls the
  exchange for the reserve status, waiting for the wire transfer to arrive
  at the exchange.

  * ``[exchange-poll-success] => pending(withdrawing-coins)``
  * ``[action:abort] => aborted(after-wired)``

* ``pending(withdrawing-coins)``

  State where we are finally withdrawing the actual coins. Depending on
  the AML and KYC thresholds, we may at any time transition into a
  holding pattern on the AML or KYC checks of the exchange.

  * ``[processed-success] => done``
  * ``[processed-kyc-required] => pending(kyc-required)``
  * ``[processed-aml-required] => pending(aml-required)``
  * ``[action:abort] => aborted(partially-withdrawn)``

* ``pending(kyc-required)``

  State where the user needs to provide some identity data to pass a KYC
  check.  The wallet only shows the user the link for starting the KYC
  process and long-polls the exchange in anticipation of the user
  completing the KYC requirement.

  * ``[poll-success] => pending(withdrawing-coins)``
  * ``[action:abort] => aborted(partially-withdrawn)``

* ``pending(aml-required)``

  State where the wallet needs to wait for completion of an AML process
  by an AML officer of the exchange. The wallet shows that the AML process
  is pending at the exchange and long-polls for it to complete. The user
  is urged to simply wait.

  * ``[poll-success] => pending(withdrawing-coins)``
  * ``[action:abort] => aborted(partially-withdrawn)``

* ``pending(aml-frozen)``

  State where the wallet informs the user that their funds were frozen
  due to an AML decision.  The user is urged to contact the exchange
  operator's AML department out-of-band.  The wallet long-polls in
  anticipation of the account being unfrozen.

  * ``[poll-success] => pending(withdrawing-coins)``
  * ``[action:abort] => aborted(partially-withdrawn)``

* ``aborting(wallet-to-bank)``

  The user aborted the withdraw operation in the wallet.
  The wallet must now try to signal the bank that the wire
  transfer should no longer be performed.  Note that it
  is possible that the bank registration never succeeded
  (if the user aborted us during ``pending(bank-register-service)``)
  and in this case we get an ``unknown transaction`` failure here.
  It is also theoretically possible that the user approved the
  transaction in the bank while simultaneously aborting in the
  wallet. In this case, we transition to ``aborted(after-wired)``.

  * ``[processed-success] => aborted(wallet-to-bank)``
  * ``[processed-error(already-confirmed)] => aborted(after-wired)``
  * ``[processed-error(unknown-transaction)] => aborted(bank-to-wallet)``

* ``aborted(wallet-to-bank)``: The wallet notified the bank that the withdrawal
  should be aborted, before any money was wired.

* ``aborted(bank-to-wallet)``: The bank notified the wallet that the withdrawal
  was denied on the side of the bank (or the request never succeeded in the
  first place) and won't proceed.

* ``aborted(after-wired)``:

  In this state, the wallet should show to the user that the money from the withdrawal
  reserve will be sent back to the originating bank account after ``$closing_delay``.

* ``aborted(partially-withdrawn)``:

  In this state, the wallet should show how much money arrived into the wallet
  and the rest of the money will be sent back to the originating bank account
  after ``$closing_delay``.

* ``done``

  * ``[action:delete] => deleted``

* ``deleted``

  Withdrawn coins are preserved, as is reserve information for recoup.
  So this mostly removes the entry from the visible transaction history.
  Only once all coins were spent, the withdraw is fully removed.


.. image:: ../transaction-withdrawal-states.png
  :width: 800

Transitions relating to deletion, suspending and resuming are not
shown in the above diagram to keep it on point.


Transaction Type: Payment to Merchant
-------------------------------------

XXX: Also consider re-selection when the wallet accidentally double-spends coins
or the selected coins have expired.  Do we ask the user in this case?

CG: I think no. We correct our balance (after all, we got a proof of
double-spending) and try other coins. If we do not have enough money left, we
abort and simply inform the user that their balance was insufficient to make
the payment after all (very sorry...).

Note that the case of selected coins having expired shouldn't really happen,
as the wallet should have noticed that when is started up, tried to refresh,
and if that already failed should have update the balance with a transaction
history entry saying something like "coins expired, offline too long" or
something like that.

(??) If transaction can be aborted while in-progress then it should return
instructedAmount, effectiveAmount and the affectedAmount (partially effective amount)

* ``initial(download-proposal)``

  Initial state. Download (claim) the proposal from the merchant.

  XXX: Also consider repurchase detection here?

  CG: Well, we could mention that this is a possible transition from
  ``pending(download-proposal)`` to ``deleted`` with a side-effect
  of transitioning the UI into a ``pending(repurchase-session-reset)``
  on a different transaction (which before was in ``done``).

  (??) should not reset another transaction?

* ``pending(proposed)``

  Let the user accept (or refuse) the payment.

  * ``[action:pay-accept] => pending(submit-payment)``
  * ``[action:abort] => deleted`` -- user explicitly decides not
    to proceed
  * ``[action:expired] => deleted`` -- when the offer expires
    before the user decides to make the payment! (We can keep
    pending contracts even in a 'pending transaction' list to
    allow the user to choose to not proceed, but then this
    transition would clean up that list).

* ``pending(submit-payment)``

Submit coin by coin (or in bulk groups) until payment is complete.

  * ``[action:abort] => aborting(refund)``
  * ``[processed-success(auto-refund-enabled)] => pending(refundable)``
  * ``[processed-success(auto-refund-disabled)] => done``
  * ``[processed-error(expired)] => aborting(refresh)`` XXX: If the order is expired but the payment
    succeeded partially before, do we still try an abort-refund? CG: YES, but of course
    we probably should use the ``expired`` transition above a few seconds before the
    offer *actually* expires to avoid this problem in 99.9% of real-world scenarios
    ("prevent last-second payments client-side")

* ``pending(refundable)``

The payment succeed but if auto-refund-check is active it will be checking for refunds

  * ``[auto-refund-timeout] => done``

* ``aborting(refund)``

  * ``[processed-success] => aborted(refunded)``
  * ``[processed-failure] => aborting(refresh)``

* ``aborting(refresh)``

* ``failed(invalid-proposal)``

  The merchant provided a proposal that is invalid (e.g. malformed contract terms or bad signature).

* ``aborted(refunded)``

  * ``[action:delete] => deleted``

* ``deleted``

  When a payment is deleted, associated refunds are always deleted with it



Transaction Type: Refund
------------------------

A refund is a pseudo-transaction that is always associated with a merchant payment transaction.

* ``initial``

  The wallet has downloaded metadata for the refund from the merchant and
  stored it in the database.  The user needs to accept/refuse it.

  * ``[processed-success] => pending``: the refund is accepted
  * ``[processed-abort] => aborted`` : the refund is not accepted

* ``pending``

  A refund is pending when the merchant is getting a non-permanent error from
  the exchange (and relaying that error response to the wallet).

  * ``[processed-success] => done``
  * ``[processed-error] => failed``

* ``failed``

  A failed refund can technically still transition to ``done``, because the wallet
  doesn't query some refund resource, but the purchase for refunds.  Thus, a previously
  failed refund can suddenly transition to ``done``.

  (??) separate transient error vs permanent error ?

Transaction Type: Refresh
-------------------------

XXX: If we have to adjust the refund amount (because a coin has fewer funds on
it than we expect), what is the resulting state of the whole refresh?

CG: first the pending balance is decreased by the reduced amount, and then of
course the final balance. The coin transaction responsible for the reduction
in funds is historic (and we don't have details), so that just changes the total
available balance in the wallet, but without an associated history entry (as we
cannot give details).

* ``pending``

  * ``[processed-success] => done``
  * ``[action:abort] => aborted``: Money that has not been refreshed yet is lost.

* ``done``

(??) should this be like ``pending(exchange-wait-reserve)`` from withdraw?
(??) can refresh trigger kyc?


Transaction Type: Tip
---------------------

* ``initial``

  The wallet has downloaded metadata for the tip from the merchant and
  stored it in the database.  The user needs to accept/refuse it.

  * ``[tip-expired] => failed(expired)``
  * ``[action:accept-tip] => pending(pickup)``
  * ``[action:abort] => aborted``

* ``pending(pickup)``

  * ``[tip-expired] => failed(expired)``
  * ``[processed-kyc-required] => pending(kyc-required)``
  * ``[processed-success] => done``
  * ``[action:abort] => aborted``

* ``pending(kyc-required)``

  * ``[poll-success] => pending(pickup)``


Transaction Type: Deposit
-------------------------

* ``initial``

  Deposit is created, effective amount is removed from balance

  * ``[processed-success] => pending(submit-deposit)``: reserve created
  * ``[action:abort] => aborted``

* ``pending(submit-deposit)``

Submit coin by coin (or in bulk groups) until deposit is completed.

  * ``[action:abort] => aborting(refund)``
  * ``[processed-success] => pending(track)``
  * ``[processed-error] => aborting(refresh)``

* ``pending(track)``

All the coins were submitted, waiting to be wired.

  * ``[poll-success] => done``
  * ``[action:abort] => aborting(partially-wired)``

* ``aborting(refund)``

Trying to get the deposited amount back from the exchange.

  ``[processed-success] => aborting(refresh)``
  ``[processed-error] => aborting(refresh)`` XXX Shouldn't this be some error state?
(??) why it moves to the same state?

* ``aborting(partially-wired)``

Should cancel all pending wire transfer.

  ``[processed-success] => aborting(refund)``
  ``[processed-error] => aborting(refresh)`` XXX Shouldn't this be some error state?


* ``aborting(refresh)``

  ``[processed-success] => aborted``
  ``[processed-error] => failed``

* ``done``


Transaction Type: Peer Push Debit
---------------------------------

Peer Push Debit transactions are created when the user wants to transfer money
to another wallet.

States and transitions:

* ``initial``

  In this state, the user is not yet able to send the payment to somebody else.

  * ``[action:abort] => aborted``: The payment is aborted early, before the wallet even had the chance to create a purse.
    No fees are incurred.
  * ``[processing-success] => pending(purse-created)``: The wallet was able to successfully create a purse.

* ``pending(purse-created)``

  In this state, the user can send / show the ``taler://`` URI or QR code to somebody else.

  * ``[action:abort] => aborting(delete-purse)``: The user aborts the P2P payment. The wallet tries to reclaim money in the purse.
  * ``[purse-timeout] => aborting(refresh)``: The other party was too slow.
  * ``[poll-success] => pending(refundable)``: The other party has accepted the payment.
  * ``[poll-error] => aborting(refresh)``: The exchange claims that there is a permanent error regarding the purse.

* ``aborting(delete-purse)``

  * ``[processed-success] => aborting(refresh)``: The purse was deleted successfully, and refunded coins must be refreshed.
  * ``[processed-failed(already-merged)] => done``: The other party claimed the funds faster that we were able to abort.
  * ``[processed-failed(other)] => aborting(refresh)``:  The exchange reports a permanent error.  We still try to refresh.
  * ``[action:abort-force] => failed``

* ``aborting(refresh)``

  * ``[processed-success] => aborted)``: Refresh group finished. Aborting was successful, money was reclaimed
  * ``[processed-failed] => failed)``: Refresh group failed to complete with a permanent error.
  * ``[action:abort-force] => failed``: XXX will this abort the refresh session or just orphan it?

* ``pending(refundable)``

  * ``[auto-refund-timeout] => done``

* ``aborting(refund)``

  * ``[processed-success] => aborted(refunded)``
  * ``[processed-failure] => aborting(refresh)``



Transaction Type: Peer Push Credit
----------------------------------

Peer Push Credit transactions are created when the user accepts to be paid via
a ``taler://pay-push`` URI.

States and transitions:

* ``initial``

Wallet read the taler:// URI and the transaction was initialized

  * ``[processed-success] => pending(withdrawing)``: Merging the reserve was successful

* ``pending(withdrawing-coins)``

  * ``[processed-kyc-required] => pending(kyc-required)``

* ``pending(kyc-required)``

  * ``[poll-success] => pending(withdrawing-coins)``
  * ``[action:abort] => aborted``: The user will lose the coins they were not able to withdraw yet, unless they
    resume the transaction again. Exchange should return non-withdrawn coins discounting closing fee.

* ``aborted``

  * ``[action:resume] => pending(withdrawing-coins)``
(??) this is not possible since aborted is final

  * ``[action:delete] => deleted``: The user will irrevocable lose coins that were not withdrawn from the reserve yet.
(??) no, i think aborting a push-credit should return to the sender, like a refund

* ``done``

  * ``[action:delete] => deleted``: No money will be lost, the withdrawn coins will be kept



Transaction Type: Peer Pull Credit
----------------------------------

TODO: Also specify variant where account reserve needs to be created / funded first.
?? which is this case?

* ``initial``

Wallet created the transaction

  * ``[action:success] => pending(purse-created)``

* ``pending(purse-created)``

  In this state, the purse is created (already in a merged state, with the initiator providing the reserve).

  * ``[action:success] => pending(wait-deposit)``:
  * ``[action:abort] => aborted``: At this stage, it's safe to just abort.

    CG: is this not 'suspend' (safe to resume!). Also, deletion transitions are missing.

* ``pending(wait-deposit)``

  We're waiting for the other party to pay into the pre-merged purse.

  * ``[action:abort] => aborting(delete-purse)``: At this stage, it's safe to just abort.
  * ``[process-failed(expired)] => failed(expired)``

* ``pending(withdrawing)``

  * ``[processed-success] => done``
  * ``[processed-kyc-required] => pending(kyc-required)``

* ``pending(kyc-required)``

  * ``[poll-success] => pending(withdrawing)``

* ``aborting(delete-purse)``

  * ``[processed-success] => aborted``
  * ``[processed-failed(was-merged)] => done``
  * ``[processed-failed(expired)] => failed(expired)``


Transaction Type: Peer Pull Debit
---------------------------------

* ``initial``

Wallet read the taler:// URI

  * ``[action:success] => pending(invoice-downloaded)``


* ``pending(invoice-downloaded)``

  We've downloaded information about the pull payment and are waiting
  for the user to confirm.

  * ``[action:abort] => aborted``: Safe to abort!
  * ``[action:confirm-pay] => pending(submit-payment)``

* ``pending(submit-payment)``

  The user has confirmed the payment and the wallet tries to deposit
  into the provided purse.

  * ``[action:abort] => aborting(refund)``
  * ``[processed-success(auto-refund-enabled)] => pending(refundable)``
  * ``[processed-success(auto-refund-disabled)] => done``
  * ``[processed-error(expired)] => aborting(refresh)``
  * ``[processed-success] => done``
  * ``[action:abort] => aborting(refresh)``

* ``pending(refundable)``

The payment succeed but if auto-refund-check is active it will be checking for refunds

  * ``[auto-refund-timeout] => done``

* ``aborting(refund)``

  * ``[processed-success] => aborted(refunded)``
  * ``[processed-failure] => aborting(refresh)``


* ``aborting(refresh)``

  XXX Before refreshing, should we not wait until the purse has expired?

  * ``[processed-success] => aborted``
  * ``[processed-failed] => failed``

* ``done``

Alternatives
============

* Each transaction could be treated completely separately; however, uniform
  terminology for actions (and thus button labels) is likely more helpful for
  the user experience.

* We could require user re-approval if fees changed when the available
  denominations change during a *withdraw*.  This would require a different
  state machine on withdraw. We believe the answer can be "no", for two
  reasons: the wallet MUST pick denominations to withdraw with the "most
  long-term" withdraw window (i.e. active denominations that have the longest
  available withdraw durations). So in virtually all normal cases, this will
  just succeed as a sane exchange will have a reasonable duration overlap, and
  in the very few cases it's really the user's fault for going offline in the
  middle of the operation. Plus, even in those few cases, it is highly
  unlikely that the fee would actually change: again most key rotations can be
  expected to be there to rotate the key, and not to adjust the withdraw fee.
  And in the extremely rare case that the user went offline and in the
  meantime the fees did *increase*, it's again unlikely to matter much to the
  user. So special-casing this and testing this is probably not worth it.

* We could require user re-approval if due to expired/invalid coins the coin
  selection (and thus fees) changes during a *deposit*.  Again, expired coins
  should virtually never happen unless a user goes offline for a long time in
  the middle of a purchase (which would be very strange). If deposit fees
  *increase* due to a double-spend detection during payment, we might want to
  have an *optional* dialog ("Balance reduced by X as wallet state was not
  up-to-date (did you restore from backup?).  Consequently, the fees for this
  transactions increased from Y to Z.  [Abort] [Continue] + checkbox: [X] Do
  not ask again."). Probably at best a post-1.0 feature.




Drawbacks
=========

Discussion / Q&A
================

(This should be filled in with results from discussions on mailing lists / personal communication.)