076-paywall-proxy.rst (6053B)
1 DD 76: Paivana - Fighting AI Bots with GNU Taler 2 ################################################ 3 4 Summary 5 ======= 6 7 This design document describes the architecture of an AI Web firewall using GNU 8 Taler, as well as new features that are required for the implementation. 9 10 Motivation 11 ========== 12 13 AI bots are causing enormous amounts of traffic by scraping sites like git 14 forges. They neither respect robots.txt nor 5xx HTTP responses. Solutions like 15 Anubis and IP-based blocking do not work anymore at this point. 16 17 Requirements 18 ============ 19 20 * Must withstand high traffic from bots, requests before a payment happened 21 must be *very* cheap, both in terms of response generation and database 22 interaction. 23 * Should work not just for our paivana-httpd but also for Turnstile-style 24 paywalls that need to work with purely static paywall pages without 25 PHP sessions. 26 27 28 Proposed Solution 29 ================= 30 31 Architecture 32 ------------ 33 34 * paivana-httpd is a reverse proxy that sits between ingress HTTP(S) traffic 35 and the protected upstream service. 36 * paivana-httpd is configured with a particular merchant backend. 37 * A payment template must be set up in the merchant backend (called ``{template_id}`` 38 from here on). 39 40 Steps: 41 42 * Browser visits ``{website}`` 43 (for example, ``https://git.taler.net``) where 44 ``{domain}`` is the domain name of ``{website}``. 45 * paivana-httpd working as a reverse-proxy for 46 ``{website}``. Whenever called for a non-whitelisted 47 URL, it checks for a the presence of a Paivana cookie valid for 48 this client IP address and ``{website}`` at this time. 49 The *Paivana Cookie* is computed as: 50 51 ``cur_time || '-' || crock32(SHA512(website || client_ip || paivana_server_secret || cur_time))``. 52 53 where ``cur_time`` in the prefix is the current time in seconds 54 (to keep it short) while in the hash it is usually binary GNUnet 55 timestamp in network byte order. 56 ``crock32`` is GNUnet's Crockford-inspired base32 encoding. 57 58 * If such a cookie is set and valid, the request is 59 reverse-proxied to upstream. *Stop.* 60 * Otherwise, a static non-cachable paywall page is returned, 61 including a machine-readable ``Paivana`` HTTP header with 62 the ``taler://pay-template/`` URL minus the client-computed 63 ``{paivana_id}`` and fullfillment URL (see below). 64 *Continue.* 65 66 * The browser (rendering the paywall page) generates a random 67 *paivana ID* via JS using the current time (``cur_time``) in seconds 68 since the Epoch and the current URL (``{website}``) plus some 69 freshly generated entropy (``{nonce}``): 70 71 ``paivana_id := cur_time || '-' || b64url(SHA256(nonce || website || cur_time))``. 72 73 Here ``b64url`` is the RFC 7515 base64 URL encoder, used to keep 74 the result short (same reason for the use of SHA-256). 75 The same computation could also easily be done by a non-JS client 76 that processes the ``Paivana`` HTTP header (or a GNU Taler wallet 77 running as a Web extension). 78 79 * Based on this paivana ID, a 80 ``taler://pay-template/{merchant_backend}/{template_id}?session_id={paivana_id}&fulfillment_url={website}`` 81 URI is generated and rendered as a QR code and link, prompting 82 the user to pay for access to the ``{website}`` using GNU Taler. 83 84 * The JavaScript in the paywall page running in the browser 85 (or the non-JS client) long-polls 86 on a new ``https://{merchant_backend}/sessions/{paivana_id}`` 87 endpoint that returns when an order with the given session ID has been paid 88 for (regardless of the order ID, which is not known to the browser). 89 * A wallet now needs to instantiate the pay template, passing the 90 ``session_id`` and the ``fulfillment_url`` as an additional inputs 91 to the order creation (the session ID here will work just like 92 existing use of ``session_ids`` in session-bound payments). 93 Similarly, the ``{website}`` works as the fulfillment URL as usual. 94 * The wallet then must pay for the resulting order 95 by talking to the Merchant backend. 96 * When the long-poller returns and the payment has succeeded, the 97 browser (still rendering the paywall page) also learns the order ID. 98 * The JavaScript of the paywall page (or the non-JS client 99 processing the ``Paivana`` HTTP header) then POSTs the order ID, 100 ``nonce``, ``cur_time`` 101 and ``website`` to ``{domain}/.well-known/pavivana``. 102 * paivana-httpd computes the paivana ID and checks if the given 103 order ID was indeed paid recently for the computed paivana ID. 104 If so, it generates an HTTP response which the Paivana cookie 105 and redirects to the fulfillment URL (which is the original {website}). 106 * The browser reloads the page with the correct 107 Paivana cookie (see first step). 108 109 110 Problems: 111 --------- 112 113 * A smart attacker might still create a lot of orders via the pay-template. 114 115 * Solution A: Don't care, unlikely to happen in the first place. 116 * Solution B: Rate-limit template instantiation on a per-IP basis. 117 118 Implementation: 119 --------------- 120 121 * Merchant backend needs way to lookup order IDs under a ``session_id`` 122 (DONE: e027e729..b476f8ae) 123 * Merchant backend needs way to instantiate templates with 124 a given ``session_id`` and ``fulfillment_url``. This also 125 requires extending the allowed responses for templates in general. 126 * Paivana component needs to be implemented 127 * Wallet-core needs support for a ``session_id`` and 128 ``fulfillment_url`` in pay templates. 129 130 131 Test Plan 132 ========= 133 134 * Deploy it for git.taler.net 135 136 Definition of Done 137 ================== 138 139 N/A 140 141 Alternatives 142 ============ 143 144 * Do not re-use the session ID mechanism but introduce some new concept. 145 This has the drawback of us needing additional tables and indicies, 146 and also the existing use of the session ID is very parallel to this one. 147 148 Drawbacks 149 ========= 150 151 * This exposes an order ID to anyone who knows the session ID. This is 152 clearly not an issue in this context, and for the existing uses of 153 the session ID it also seems clear that knowledge of the session ID 154 requires an attacker to have access that would easily also already 155 give them any order ID, so this seems harmless. 156 157 158 Discussion / Q&A 159 ================