DD 37: Wallet Transaction Lifecycle ################################### Summary ======= This design doc discusses the lifecycle of transactions in wallet-core. Motivation ========== The transactions in wallet-core all should have an associated state machine. All transactions should have some common actions that work uniformly across all transactions. Proposed Solution ================= Common States ------------- The following states apply to multiple different transactions. Only pending and aborting have transaction-specific sub-states, denoted by ``state(substate)``. ``initial``: The initial state is the result a user interaction. A transaction may have more than one initial state (if it is started in multiple ways) or none (if it's the result of another process). ``pending``: A pending transaction waits for some external event/service. The transaction stays pending until its change on the wallet's material balance is finished. Any pending state can be suspended and resumed. There are some other distinctions for pending transactions: * long-polling vs. exponential backoff: A pending transaction is either waiting on an external service by making a long-polling request or by repeating requests with exponential back-off. * ``lastError``: A pending transaction is either clean (i.e. the network interaction is literally active in transmission or the external service successfully communicated that it is not ready yet and this is perfectly normal) or has a ``lastError``, which is a ``TalerErrorDetails`` object with details about what happened during the last attempt to proceed with the transaction. ``done``: A transaction that is done does not require any more processing. It also never has a ``lastError`` but is considered successful. ``aborting``: Similar to a pending transaction, but instead of taking active steps to complete the transaction, the wallet is taking active steps to abort it. The ``lastError`` indicates errors the wallet experienced while taking active steps to abort the transaction. ``aborted``: Similar to ``done``, but the transaction was successfully aborted instead of successfully finished. It will have the information of when (timestamp) it was aborted and in which pending sub-state the abort action was initiated. Also, we can include more information information relevant to the transaction in `abortReason` ``suspended``: Similar to a ``aborted`` transaction, but the transaction was could be resumed and may then still succeed. ``failed``: Similar to ``done``, but the transaction could not even be aborted properly. ``deleted``: A ``deleted`` state is always a final state. We only use this state for illustrative purposes. In the implementation, the data associated with the transaction would be literally deleted. Common Transitions ------------------ Transitions are actions or other events. ``[action:delete]``: Deleting a transaction completely deletes the transaction from the database. Depending on the type of transaction, some of the other data *resulting* from the transaction might still survive deletion. For example, deleting a withdrawal transaction does not delete already successfully withdrawn coins. Deleting is only safe (no money lost) on initial and final states (failed, aborted, done). ``[action:retry]``: Retrying a transaction *(1.)* stops ongoing long-polling requests for the transaction *(2.)* resets the retry timeout *(3.)* re-runs the handler to process the transaction. Retries are always possible the following states: ``pending(*)`` and ``aborting(*)``. .. attention:: Should we show the retry timeout in the UI somewhere? Should we show it in dev mode? SEBASJM: Since the wallet will retry anyway, maybe is better if we replace the "retry" button with a "try now" button and a side text "retrying in xxx seconds". CG: Instead of a side text, this *might* make a good mouse-over hint for a "retry" (or "try now") button. I would not make this overly visible with side-text as the information is not that important. The text should also be "retrying next at XXX" using an absolute time XXX --- otherwise the UI would be way too busy recomputing/updating all of these strings: Using an absolute time, we only have to redraw anything once a retry actually happened. Given that retries should basically never be > 24h (we can impose a hard cap), the absolute time can just be in the format HH:MM:SS (without day). ``[action:abort]``: Aborting a transaction either directly stops processing for the transaction and puts it in an ``aborted`` state, or starts the necessary steps to actively abort the transaction (e.g. to avoid losing money) and puts it in an ``aborting`` state. ``[action:suspend]``: Suspends a pending transaction, stopping any associated network activities, but with a chance of trying again at a later time. This could be useful if a user needs to save battery power or bandwidth and an operation is expected to take longer (such as a backup, recovery or very large withdrawal operation). ``[action:resume]``: Suspended transactions may be resumed, placing them back into a pending state. ``[action:abort-force]``: Directly puts an ``aborting`` transaction into the ``failed`` state. Whether aborting or resuming are possible depends on the transaction type, and usually only one of the two choices should be offered. .. image:: ../transaction-common-states.png :width: 400 Boxed labels indicate an end state in which it is safe to delete the transaction record since no work is due. Blue arrows are used for user-triggered actions (via UI buttons). Common pending sub-states ------------------------- During the pending state the transaction can go through several sub-states before reaching a final state. Some of this sub-states are shared between different transaction types: ``kyc-required``: The transaction can't proceed because the user needs to actively finish a KYC process. Part of a withdrawal process or peer-to-peer push credit. ``aml-required``: The transaction can't proceed because the user needs to wait for the exchange operator to conclude an AML investigation by the staff at the exchange. The user is not expected to take any action and should just wait for the investigation to conclude. Part of a withdrawal process or peer-to-peer push credit. ``aml-frozen``: The staff at the exchange decided that the account needed to be frozen. The user should contact the exchange provider's customer service department and seek resolution (possibly through the courts) to avoid loosing the funds for good. Part of a withdrawal process or peer-to-peer push credit. Transaction Type: Withdrawal ---------------------------- * ``pending(bank-register-reserve)`` Initial state for bank-integrated withdrawals. The wallet submits the reserve public key and selected exchange to the bank (via the bank integration API). Note that if the user aborts at this stage, we do not know if the bank is in the confirmation stage, so we must still *try* to abort the transaction at the bank. * ``[processed-success] => pending(bank-confirming)`` * ``[processed-error] => deleted``: We show a transient warning. * ``[action:abort] => aborting(bank-to-wallet)`` * ``pending(bank-confirming)`` The wallet waits until the bank has confirmed the withdrawal operation; usually the user has to complete a 2FA step to *approve* that the money is wired to the chosen exchange. Note that the user's *approve* action is done in the bank's user interface and not the wallet's user interface. The wallet internally merely *polls* for the success or failure of the approve action. The wallet **may** occasionally (after some initial delay, especially on failures from the bank-poll to return any result) long-poll for the reserve status and, if successful, may then directly jump to ``pending(withdrawing-coins)`` if the reserve is filled even if the poll at the bank did not return success or failure. * ``[bank-poll-success] => pending(exchange-wait-reserve)`` * ``[bank-poll-denied] => deleted``: We show a transient message that the operation was denied in the bank. * ``[exchange-poll-success] => pending(withdrawing-coins)`` * ``[action:abort] => aborting(wallet-to-bank)`` * ``aborting(wallet-to-bank)`` The user aborted the withdraw operation in the wallet. The wallet must now try to signal the bank that the wire transfer should no longer be performed. Note that it is possible that the bank registration never succeeded (if the user aborted us during ``pending(bank-register-service)``) and in this case we get an ``unknown transaction`` failure here. It is also theoretically possible that the user approved the transaction in the bank while simultaneously aborting in the wallet. In this case, we transition to ``suspended(after-wired)`` (treating the ``abort`` action as a ``suspend`` action). * ``[processed-success] => deleted``: We show a transient message that the operation was aborted. * ``[processed-error(already-confirmed)] => suspended(after-wired)``: We keep a transaction history entry reminding the user about when the already wired funds will be returned. * ``[processed-error(unknown-transaction)] => deleted`` * ``suspended(after-wired)`` In this state, the wallet should show to the user that the money from the withdrawal reserve will be sent back to the originating bank account after ``$closing_delay``. Note that the ``resume`` action should be disabled after ``$closing_delay``. * ``[action:delete] => deleted`` * ``[action:resume] => pending(exchange-wait-reserve)`` * ``pending(exchange-wait-reserve)`` Initial state for manual withdrawals. Here, the wallet long-polls the exchange for the reserve status, waiting for the wire transfer to arrive at the exchange. * ``[exchange-poll-success] => pending(withdrawing-coins)`` * ``[action:suspend] => suspended(after-wired)`` * ``suspended(after-wired)`` State where funds were (presumably) wired to the exchange but the wallet was asked to not proceed with the withdraw, but we still resume. * ``[action:resume] => pending(exchange-wait-reserves)`` * ``[action:delete] => deleted`` * ``pending(withdrawing-coins)`` State where we are finally withdrawing the actual coins. Depending on the AML and KYC thresholds, we may at any time transition into a holding pattern on the AML or KYC checks of the exchange. * ``[processed-success] => done`` * ``[processed-kyc-required] => pending(kyc)`` * ``[processed-aml-required] => pending(aml)`` * ``[action:suspend] => suspended(withdrawing-coins)`` * ``pending(kyc)`` State where the user needs to provide some identity data to pass a KYC check. The wallet only shows the user the link for starting the KYC process and long-polls the exchange in anticipation of the user completing the KYC requirement. * ``[poll-success] => pending(withdrawing-coins)`` * ``[action:suspend] => suspended(kyc)`` * ``suspended(kyc)`` State where the user needs to provide some identity data to pass a KYC check, but the long-polling was explicitly stopped. The user can choose to resume or delete. * ``[action:delete] => deleted`` * ``[action:resume] => pending(kyc)`` * ``pending(aml)`` State where the wallet needs to wait for completion of an AML process by an AML officer of the exchange. The wallet shows that the AML process is blocking progress. The message shown should distinguish between a mere pending AML process and an AML freezing decision in terms of the message shown to the user. If the AML decision is pending at the exchange, he user should be urged to simply wait. If the funds were frozen, the wallet informs the user that their funds were frozen due to an AML decision. The user is urged to contact the exchange operator's AML department out-of-band. In any case, the wallet long-polls for the AML decision to be made or change (possibly at a lower frequeny in case of a freeze). * ``[poll-success] => pending(withdrawing-coins)`` * ``[action:suspend] => suspended(aml)`` * ``suspended(aml)`` State where the user needs to await some AML decision by the exchange. The long-polling was explicitly stopped. The user can choose to resume or delete. * ``[action:delete] => deleted`` * ``[action:resume] => pending(aml)`` * ``suspended(withdrawing-coins)`` In this state, the wallet should show how much money arrived into the wallet and the rest of the money will be sent back to the originating bank account after ``$closing_delay``. Note that the ``resume`` action should be disabled after ``$closing_delay``. * ``[action:delete] => deleted`` * ``[action:resume] => pending(exchange-wait-reserve)`` * ``done`` The withdrawal operation is complete. * ``[action:delete] => deleted`` * ``deleted`` Withdrawn coins are preserved, as is reserve information for recoup. So this mostly removes the entry from the visible transaction history. Only once all coins were spent, the withdraw is fully removed. .. image:: ../transaction-withdrawal-states.png :width: 800 Transaction Type: Payment to Merchant ------------------------------------- * ``pending(claim-proposal)`` We received a ``pay`` URI. Download (claim) the proposal from the merchant. Can fail if the proposal was already claimed by someone else. If repurchase detection tells us that we already paid for this product, we go immediately to ``delete`` state for this transaction, but with a side-effect of transitioning the UI into a ``pending(repurchase-session-reset)`` on a *different* transaction (which before was in ``done``). * ``[error: already claimed] => deleted`` -- the proposal was already claimed by someone else; we go directly into the ``deleted`` state and only show a transient warning. * ``[error: invalid proposal] => deleted`` -- the merchant provided a proposal that is invalid (e.g. malformed contract terms or bad signature); we go directly into the ``deleted`` state and only show a transient warning. * ``pending(proposed)`` Let the user accept (or refuse) the payment. * ``[action:pay-accept] => pending(submit-payment)`` * ``[action:pay-refuse] => ``aborting(unclaim)`` -- The user explicitly decided not to proceed (at least not with this wallet). * ``[expired] => deleted`` -- The offer has expired before the user made any decision. (We can keep pending contracts even in a 'pending transaction' list to allow the user to choose to not proceed, but then this transition would clean up that list). Note that we should use this transition at least a few seconds before the offer *actually* expires to avoid encountering an expiration during ``pending(submit-payment)`` in most real-world scenarios. Basically, we should prevent last-second payments to be event attempted client-side. * ``aborting(unclaim)`` Tells the merchant that some *other* wallet is now again free to claim this offer. * ``pending(submit-payment)`` Submit coin-by-coin (or in bulk groups) until payment is complete. * ``[action:abort] => aborting(refund)`` -- The user explicitly decided to abort the process while the payment was happening. Note that if the payment was already completed (and hence the merchant refuses any refunds), it is theoretically possible that pressing the abort button will nevertheless end up in a ``pending(refundable)`` state (and subsequently a ``done`` state) instead! * ``[success] => pending(refundable)`` -- Upon receiving confirmation from the merchant that the purchase was completed. * ``[error(insufficient balance)] => aborting(refresh)`` This transition happens if we detect double-spending and our balance is not sufficient after the double-spending. It is also conceivable (but should be rare) that this transition happens because the offer expired. * ``pending(refundable)`` The payment succeed. We remain in this tate as long as an auto-refund-check is active. If auto refunds are not enabled, we immediately continue to ``done``. * ``[timeout] => done`` -- This happens when the auto refund set by the contract expired. * ``[action:abort] => done`` -- The user may explicitly request to abort the auto-refund processing (for example to enable subsequent deletion before the auto-refund delay expires). * ``aborting(refund)`` The wallet should interact with the merchant to confirm that a refund was approved. * ``[success] => aborted(refunded)`` * ``[failure] => aborting(refresh)`` * ``aborting(refresh)`` The wallet should interact with the exchange to obtain fresh coins for the refunded balance. * ``aborted(refunded)`` The purchase ended with a (partial) refund. The state (and UI) should show the specific provenance of the state, which may include an insufficient balance (due to double-spending being detected during payment), and one or more partial or full refunds. * ``[URI:refund] => aborting(refund)`` -- The previous refund was partial, and we have received an additional refund for this transaction. * ``[action:delete] => deleted`` * ``done`` The purchase is completed. * ``[action:delete] => deleted`` * ``[repurchase] => pending(repurchase-session-reset)`` -- Another offer became pending for this product. * ``pending(repurchase-session-reset)`` The wallet should reset the associated session for the already purchased item. * ``[success] => done`` * ``[action:abort] => done`` -- User aborted the session reset. * ``deleted`` When a payment is deleted, associated refunds are always deleted with it. .. image:: ../transaction-payment-states.png :width: 800 Purple arrows are used to indicate transitions triggered in special ways outside of the user interface for this transaction, such as side-effects of repurchase detection for another purchase or opening of a refund URI. Transaction Type: Refund ------------------------ A refund is a pseudo-transaction that is always associated with a merchant payment transaction. * ``pending(lookup-refund)`` We received a ``refund`` URI. Download refund details (like the amount) from the merchant. If the merchant responds with a permanent failure, we only show a transient warning and delete the transaction. * ``[success] => pending(user-accept)`` * ``[action:suspend] => suspended(lookup-refund)`` * ``[failure] => deleted`` * ``suspended(lookup-refund)`` The user suspended while we were looking up the refund details. Allow resuming or permanent deletion. * ``[action:resume] => pending(lookup-refund)`` * ``[action:delete] => deleted`` * ``pending(user-accept)`` The wallet has downloaded metadata for the refund from the merchant and stored it in the database. The user needs to accept/refuse it. * ``[action:accept] => pending(merchant)``: the refund is accepted * ``[action:delete] => deleted`` : the refund is not accepted * ``pending(merchant)`` A refund is pending in this state while we are waiting for the merchant to get a successful response from the exchange (and relaying that error response to the wallet). * ``[action:suspend] => suspended(merchant)`` * ``[processed-success] => pending(refresh)`` * ``[processed-error] => failed``: we received a permanent failure (such as money already wired to the merchant) * ``pending(refresh)`` The wallet is now refreshing the coins. * ``[processed-success] => done`` * ``[action:suspend] => suspended(refresh)`` * ``[processed-error] => failed``: we received a permanent failure. * ``failed`` The refund failed permanently. .. image:: ../transaction-refund-states.png :width: 800 Transaction Type: Refresh ------------------------- XXX: If we have to adjust the refund amount (because a coin has fewer funds on it than we expect), what is the resulting state of the whole refresh? CG: first the pending balance is decreased by the reduced amount, and then of course the final balance. The coin transaction responsible for the reduction in funds is historic (and we don't have details), so that just changes the total available balance in the wallet, but without an associated history entry (as we cannot give details). * ``pending`` * ``[processed-success] => done`` * ``[action:abort] => aborted``: Money that has not been refreshed yet is lost. * ``done`` (??) should this be like ``pending(exchange-wait-reserve)`` from withdraw? (??) can refresh trigger kyc? Transaction Type: Tip --------------------- * ``initial`` The wallet has downloaded metadata for the tip from the merchant and stored it in the database. The user needs to accept/refuse it. * ``[tip-expired] => failed(expired)`` * ``[action:accept-tip] => pending(pickup)`` * ``[action:abort] => aborted`` * ``pending(pickup)`` * ``[tip-expired] => failed(expired)`` * ``[processed-kyc-required] => pending(kyc-required)`` * ``[processed-success] => done`` * ``[action:abort] => aborted`` * ``pending(kyc-required)`` * ``[poll-success] => pending(pickup)`` Transaction Type: Deposit ------------------------- * ``initial`` Deposit is created, effective amount is removed from balance * ``[processed-success] => pending(submit-deposit)``: reserve created * ``[action:abort] => aborted`` * ``pending(submit-deposit)`` Submit coin by coin (or in bulk groups) until deposit is completed. * ``[action:abort] => aborting(refund)`` * ``[processed-success] => pending(track)`` * ``[processed-error] => aborting(refresh)`` * ``pending(track)`` All the coins were submitted, waiting to be wired. * ``[poll-success] => done`` * ``[action:abort] => aborting(partially-wired)`` * ``aborting(refund)`` Trying to get the deposited amount back from the exchange. ``[processed-success] => aborting(refresh)`` ``[processed-error] => aborting(refresh)`` XXX Shouldn't this be some error state? (??) why it moves to the same state? * ``aborting(partially-wired)`` Should cancel all pending wire transfer. ``[processed-success] => aborting(refund)`` ``[processed-error] => aborting(refresh)`` XXX Shouldn't this be some error state? * ``aborting(refresh)`` ``[processed-success] => aborted`` ``[processed-error] => failed`` * ``done`` Transaction Type: Peer Push Debit --------------------------------- Peer Push Debit transactions are created when the user wants to transfer money to another wallet. States and transitions: * ``initial`` In this state, the user is not yet able to send the payment to somebody else. * ``[action:abort] => aborted``: The payment is aborted early, before the wallet even had the chance to create a purse. No fees are incurred. * ``[processing-success] => pending(purse-created)``: The wallet was able to successfully create a purse. * ``pending(purse-created)`` In this state, the user can send / show the ``taler://`` URI or QR code to somebody else. * ``[action:abort] => aborting(delete-purse)``: The user aborts the P2P payment. The wallet tries to reclaim money in the purse. * ``[purse-timeout] => aborting(refresh)``: The other party was too slow. * ``[poll-success] => pending(refundable)``: The other party has accepted the payment. * ``[poll-error] => aborting(refresh)``: The exchange claims that there is a permanent error regarding the purse. * ``aborting(delete-purse)`` * ``[processed-success] => aborting(refresh)``: The purse was deleted successfully, and refunded coins must be refreshed. * ``[processed-failed(already-merged)] => done``: The other party claimed the funds faster that we were able to abort. * ``[processed-failed(other)] => aborting(refresh)``: The exchange reports a permanent error. We still try to refresh. * ``[action:abort-force] => failed`` * ``aborting(refresh)`` * ``[processed-success] => aborted)``: Refresh group finished. Aborting was successful, money was reclaimed * ``[processed-failed] => failed)``: Refresh group failed to complete with a permanent error. * ``[action:abort-force] => failed``: XXX will this abort the refresh session or just orphan it? * ``pending(refundable)`` * ``[auto-refund-timeout] => done`` * ``aborting(refund)`` * ``[processed-success] => aborted(refunded)`` * ``[processed-failure] => aborting(refresh)`` Transaction Type: Peer Push Credit ---------------------------------- Peer Push Credit transactions are created when the user accepts to be paid via a ``taler://pay-push`` URI. States and transitions: * ``initial`` Wallet read the taler:// URI and the transaction was initialized * ``[processed-success] => pending(withdrawing)``: Merging the reserve was successful * ``pending(withdrawing-coins)`` * ``[processed-kyc-required] => pending(kyc-required)`` * ``pending(kyc-required)`` * ``[poll-success] => pending(withdrawing-coins)`` * ``[action:abort] => aborted``: The user will lose the coins they were not able to withdraw yet, unless they resume the transaction again. Exchange should return non-withdrawn coins discounting closing fee. * ``aborted`` * ``[action:resume] => pending(withdrawing-coins)`` (??) this is not possible since aborted is final * ``[action:delete] => deleted``: The user will irrevocable lose coins that were not withdrawn from the reserve yet. (??) no, i think aborting a push-credit should return to the sender, like a refund * ``done`` * ``[action:delete] => deleted``: No money will be lost, the withdrawn coins will be kept Transaction Type: Peer Pull Credit ---------------------------------- TODO: Also specify variant where account reserve needs to be created / funded first. ?? which is this case? * ``initial`` Wallet created the transaction * ``[action:success] => pending(purse-created)`` * ``pending(purse-created)`` In this state, the purse is created (already in a merged state, with the initiator providing the reserve). * ``[action:success] => pending(wait-deposit)``: * ``[action:abort] => aborted``: At this stage, it's safe to just abort. CG: is this not 'suspend' (safe to resume!). Also, deletion transitions are missing. * ``pending(wait-deposit)`` We're waiting for the other party to pay into the pre-merged purse. * ``[action:abort] => aborting(delete-purse)``: At this stage, it's safe to just abort. * ``[process-failed(expired)] => failed(expired)`` * ``pending(withdrawing)`` * ``[processed-success] => done`` * ``[processed-kyc-required] => pending(kyc-required)`` * ``pending(kyc-required)`` * ``[poll-success] => pending(withdrawing)`` * ``aborting(delete-purse)`` * ``[processed-success] => aborted`` * ``[processed-failed(was-merged)] => done`` * ``[processed-failed(expired)] => failed(expired)`` Transaction Type: Peer Pull Debit --------------------------------- * ``initial`` Wallet read the taler:// URI * ``[action:success] => pending(invoice-downloaded)`` * ``pending(invoice-downloaded)`` We've downloaded information about the pull payment and are waiting for the user to confirm. * ``[action:abort] => aborted``: Safe to abort! * ``[action:confirm-pay] => pending(submit-payment)`` * ``pending(submit-payment)`` The user has confirmed the payment and the wallet tries to deposit into the provided purse. * ``[action:abort] => aborting(refund)`` * ``[processed-success(auto-refund-enabled)] => pending(refundable)`` * ``[processed-success(auto-refund-disabled)] => done`` * ``[processed-error(expired)] => aborting(refresh)`` * ``[processed-success] => done`` * ``[action:abort] => aborting(refresh)`` * ``pending(refundable)`` The payment succeed but if auto-refund-check is active it will be checking for refunds * ``[auto-refund-timeout] => done`` * ``aborting(refund)`` * ``[processed-success] => aborted(refunded)`` * ``[processed-failure] => aborting(refresh)`` * ``aborting(refresh)`` XXX Before refreshing, should we not wait until the purse has expired? * ``[processed-success] => aborted`` * ``[processed-failed] => failed`` * ``done`` Alternatives ============ * Each transaction could be treated completely separately; however, uniform terminology for actions (and thus button labels) is likely more helpful for the user experience. * We could require user re-approval if fees changed when the available denominations change during a *withdraw*. This would require a different state machine on withdraw. We believe the answer can be "no", for two reasons: the wallet MUST pick denominations to withdraw with the "most long-term" withdraw window (i.e. active denominations that have the longest available withdraw durations). So in virtually all normal cases, this will just succeed as a sane exchange will have a reasonable duration overlap, and in the very few cases it's really the user's fault for going offline in the middle of the operation. Plus, even in those few cases, it is highly unlikely that the fee would actually change: again most key rotations can be expected to be there to rotate the key, and not to adjust the withdraw fee. And in the extremely rare case that the user went offline and in the meantime the fees did *increase*, it's again unlikely to matter much to the user. So special-casing this and testing this is probably not worth it. * We could require user re-approval if due to expired/invalid coins the coin selection (and thus fees) changes during a *deposit*. Again, expired coins should virtually never happen unless a user goes offline for a long time in the middle of a purchase (which would be very strange). If deposit fees *increase* due to a double-spend detection during payment, we might want to have an *optional* dialog ("Balance reduced by X as wallet state was not up-to-date (did you restore from backup?). Consequently, the fees for this transactions increased from Y to Z. [Abort] [Continue] + checkbox: [X] Do not ask again."). Probably at best a post-1.0 feature. Drawbacks ========= Discussion / Q&A ================ (This should be filled in with results from discussions on mailing lists / personal communication.)