From 8302beb9ef13811287e16aa27109821b6f660ce0 Mon Sep 17 00:00:00 2001 From: Florian Dold Date: Thu, 6 Aug 2020 15:07:48 +0530 Subject: integration testing and fault injection write-up --- taler-wallet.rst | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) (limited to 'taler-wallet.rst') diff --git a/taler-wallet.rst b/taler-wallet.rst index a4e65f6b..ae0656dc 100644 --- a/taler-wallet.rst +++ b/taler-wallet.rst @@ -921,3 +921,125 @@ There are test cases that require us to modify the communication between the wal * What does the wallet do when the exchange/merchant announce an incompatible protocol version? * What happens if some signature made by the exchange/merchant is garbage? * What if the exchange reports a double-spend and the proof it gives us is invalid? + + + +Integration Test and Fault Injection Framework +---------------------------------------------- + +This section describes the current approach to integration testing in the wallet. + +It's all based on a TypeScript harness process, which itself implements +the fault injection proxy (async and in-process)! + +The new approach consists of the following parts: + +1. A strongly typed, convenient helper library to easily set up and run +arbitrary Taler deployments and run test cases. These components plug +together as easily as lego bricks, even with multiple +exchanges/merchants/banks/etc. Logs and clean shutdown (even on SIGINT +or errors) are handled properly. (Support for auditors is still pending +but needed to fully test the wallet.) + +This is how a simple withdrawal and payment test case looks like: +``__ + +(What's particularly nice is that all our docs contain TypeScript +definitions for all API request bodies. So just copying them into the +test harness gives us auto-completion and compile-time checks to avoid +typos. The wallet's JSON validation machinery is also re-used.) + +2. A fault injection proxy that can be plugged between the services +and/or the wallet. It runs alongside the test harness, and can thus can +use arbitrary custom logic. There's no dependency for it other than +built-in Node.JS libraries. Simple fault injections are just as easy to +set up as with the twister. + +The following test case (a) logs all requests and responses to the test +harness stdout and (b) at a certain point, starts dropping the next 10 +requests to the exchange (testing the wallet's retry logic): + +``__ + +3. All util functionality from JS wallet-core, such as the Taler crypto, +amount/date/etc. handling and JSON parsing/validation (the wallet is now +more modular and easier to use as a library) can be used in the +integration tests, even if a different wallet (Kotlin, whatever) is +tested via the CLI. + +4. A bunch of test cases that use (1)-(3). These are *significantly* +more readable and hackable than other test approaches we had, while +allowing for more complex scenarios. There are still way too few tests +though! + +5. A test runner (written in bash) that runs test cases based on a glob +pattern and reports the results. + +Injecting a fault is as easy as: + +.. code:: ts + + // Set up test case + [...] + + exchangeProxy.addFault({ + beforeResponse(ctx: FaultInjectionResponseContext) { + if (cond1) { // Drop some responses + ctx.dropResponse = true; + return; + } else if (cond2) { // modify some others + ctx.responseBody = Buffer.from(`{"oops": true}`, "utf-8"); + return; + } + // Other things that can be modified: + // - drop/modify the request, not just the response + // - modify headers + // - modify status codes + } + }); + + await doSomethingWithTheWallet(); + + exchangeProxy.clearFault(); + + await doMoreWithTheWallet(); + + +To make the configuration easy, an ExchangeService (or MerchantService, +BankService etc.) can be wrapped in a FaultInjectedExchangeService, +which implements the ExchangeServiceInterface: + +.. code:: ts + + // create exchange and two merchants + const exchange = await setupExchange(...); + const merchant1 = ...; + const merchant2 = ...; + + // Add exchange to merchant-accepted exchanges. + // This will adjust the config. + merchant1.addExchange(exchange); + + // Wrap exchange in fault injection proxy + const faultInjectedExchange: ExchangeServiceInterface + = new FaultInjectedExchangeService(t, exchange1, 8085); + + // Merchant 2 talks to the exchange over fault injection, + // and thus must use the "twisted" base URL. + merchant2.addExchange(faultInjectedExchange); + + +The package for the integration tests is here: + +``__ + +The shortcut to run all integration tests is + +.. code:: sh + + ./bootstrap && ./configure --prefix=... \ + && make install integrationtests + +(From the root of the whole repo. If you're developing tests, it's way +faster to just run "make compile install" once and then use +"./testrunner" from the taler-integrationtests package.) -- cgit v1.2.3