076-paywall-proxy.rst (5661B)
1 DD 76: Paivana - Fighting AI Bots with GNU Taler 2 ################################################ 3 4 Summary 5 ======= 6 7 This design document describes the architecture of an AI Web firewall using GNU 8 Taler, as well as new features that are required for the implementation. 9 10 Motivation 11 ========== 12 13 AI bots are causing enormous amounts of traffic by scraping sites like git 14 forges. They neither respect robots.txt nor 5xx HTTP responses. Solutions like 15 Anubis and IP-based blocking do not work anymore at this point. 16 17 Requirements 18 ============ 19 20 * Must withstand high traffic from bots, requests before a payment happened 21 must be *very* cheap, both in terms of response generation and database 22 interaction. 23 * Should work not just for our paivana-httpd but also for Turnstile-style 24 paywalls that need to work with purely static paywall pages without 25 PHP sessions. 26 27 28 Proposed Solution 29 ================= 30 31 Architecture 32 ------------ 33 34 * paivana-httpd is a reverse proxy that sits between ingress HTTP(S) traffic 35 and the protected upstream service. 36 * paivana-httpd is configured with a particular merchant backend. 37 * A payment template must be set up in the merchant backend (called ``{template_id}`` 38 from here on). 39 40 Steps: 41 42 * Browser visits ``{website}`` 43 (for example, ``https://git.taler.net``) where 44 ``{domain}`` is the domain name of ``{website}``. 45 * paivana-httpd working as a reverse-proxy for 46 ``{website}``. Whenever called for a non-whitelisted 47 URL, it checks for a the presence of a Paivana cookie valid for 48 this client IP address and ``{website}`` at this time. 49 The *Paivana Cookie* is computed as: 50 51 ``cur_time || '-' || H(website || client_ip || paivana_server_secret || cur_time)``. 52 53 * If such a cookie is set and valid, the request is 54 reverse-proxied to upstream. *Stop.* 55 * Otherwise, a static non-cachable paywall page is returned, 56 including a machine-readable ``Paivana`` HTTP header with 57 the ``taler://pay-template/`` URL minus the client-computed 58 ``{paivana_id}`` and fullfillment URL (see below). 59 *Continue.* 60 61 * The browser (rendering the paywall page) generates a random 62 *paivana ID* via JS using the current time (``cur_time``) in seconds 63 since the Epoch and the current URL (``{website}``) plus some 64 freshly generated entropy (``{nonce}``): 65 66 ``paivana_id := cur_time || '-' || H(nonce || website || cur_time)``. 67 68 The same computation could also easily be done by a non-JS client 69 that processes the ``Paivana`` HTTP header (or a GNU Taler wallet 70 running as a Web extension). 71 72 * Based on this paivana ID, a 73 ``taler://pay-template/{merchant_backend}/{template_id}?session_id={paivana_id}&fulfillment_url={website}`` 74 URI is generated and rendered as a QR code and link, prompting 75 the user to pay for access to the ``{website}`` using GNU Taler. 76 77 * The JavaScript in the paywall page running in the browser 78 (or the non-JS client) long-polls 79 on a new ``https://{merchant_backend}/sessions/{paivana_id}`` 80 endpoint that returns when an order with the given session ID has been paid 81 for (regardless of the order ID, which is not known to the browser). 82 * A wallet now needs to instantiate the pay template, passing the 83 ``session_id`` and the ``website`` as an additional inputs 84 to the order creation (the session ID here will work just like 85 existing use of ``session_ids`` in session-bound payments). 86 Similarly, the ``website`` works as the fulfillment URL as usual. 87 * The wallet then must pay for the resulting order 88 by talking to the Merchant backend. 89 * When the long-poller returns and the payment has succeeded, the 90 browser (still rendering the paywall page) also learns the order ID. 91 * The JavaScript of the paywall page (or the non-JS client 92 processing the ``Paivana`` HTTP header) then POSTs the order ID, 93 ``nonce``, ``cur_time`` 94 and ``website`` to ``{domain}/.well-known/pavivana``. 95 * paivana-httpd computes the paivana ID and checks if the given 96 order ID was indeed paid recently for the computed paivana ID. 97 If so, it generates an HTTP response which the Paivana cookie 98 and redirects to the fulfillment URL (which is the original {website}). 99 * The browser reloads the page with the correct 100 Paivana cookie (see first step). 101 102 103 Problems: 104 --------- 105 106 * A smart attacker might still create a lot of orders via the pay-template. 107 108 * Solution A: Don't care, unlikely to happen in the first place. 109 * Solution B: Rate-limit template instantiation on a per-IP basis. 110 111 Implementation: 112 --------------- 113 114 * Merchant backend needs way to lookup order IDs under a ``session_id`` 115 (DONE: e027e729..b476f8ae) 116 * Merchant backend needs way to instantiate templates with 117 a given ``session_id`` and ``fulfillment_url``. This also 118 requires extending the allowed responses for templates in general. 119 * Paivana component needs to be implemented 120 * Wallet-core needs support for a ``session_id`` and 121 ``fulfillment_url`` in pay templates. 122 123 124 Test Plan 125 ========= 126 127 * Deploy it for git.taler.net 128 129 Definition of Done 130 ================== 131 132 N/A 133 134 Alternatives 135 ============ 136 137 * Do not re-use the session ID mechanism but introduce some new concept. 138 This has the drawback of us needing additional tables and indicies, 139 and also the existing use of the session ID is very parallel to this one. 140 141 Drawbacks 142 ========= 143 144 * This exposes an order ID to anyone who knows the session ID. This is 145 clearly not an issue in this context, and for the existing uses of 146 the session ID it also seems clear that knowledge of the session ID 147 requires an attacker to have access that would easily also already 148 give them any order ID, so this seems harmless. 149 150 151 Discussion / Q&A 152 ================