README (23324B)
1 Paivana 2 ======= 3 4 Paivana is an HTTP reverse proxy that gates access to a target website 5 behind a GNU Taler payment. Unpaid visitors receive a paywall page 6 where they can pay using a GNU Taler wallet; once payment is confirmed 7 the request is forwarded transparently to the configured upstream 8 server. 9 10 The sole binary is `paivana-httpd`. 11 12 13 How it works 14 ------------ 15 16 0. `paivana-httpd` learns prices from the Paivana templates configured 17 in the taler-merchant-backend. Paivana templates include a regular 18 expression which determines the set of pages the template applies to. 19 That expression (POSIX extended) is matched against the *entire* 20 URL and is anchored at both ends, so `/premium/` applies to nothing 21 while `.*/premium/.*` applies to every URL containing it. 22 1. An HTTP client accesses a page at `paivana-httpd`. 23 2. If the paywall is enabled for the respective URL and no valid access 24 cookie is present, `paivana-httpd` redirects the browser to 25 a static paywall HTML page (customizable Mustache template) 26 referencing the payment template. 27 The page includes a "Paivana" HTTP header to also facilitate agentic 28 payments. 29 3. The browser computes a unique payment identifier and 30 renders a dynamic payment request (taler:// QR code 31 or link) and long-polls the taler-merchant-backend awaiting 32 completion of the payment. 33 4. The user instructs their Taler wallet to complete the payment. 34 5. The browser notices that the payment is complete and calls back to 35 `POST /.well-known/paivana` (provided by `paivana-httpd`) 36 with a reference to the unique payment identifier. 37 6. `paivana-httpd` verifies the payment with the merchant, sets 38 an access cookie, and redirects the browser to the original URL. 39 The access ends at the `expiration` the browser named in step 3 and 40 repeated in step 5 — it is hashed into the payment identifier, so the 41 two must agree — bounded above by the contract's `max_pickup_time`, 42 which is what the merchant's `max_pickup_delay` on the template sets. 43 7. Requests with a valid cookie are forwarded to the upstream server 44 via libcurl. Both directions are streamed: bytes are passed on as 45 they arrive, so the size of a proxied body is not bounded by memory 46 (see "Architecture notes"). 47 48 The cookie carries a keyed hash over `(expiration time, website, client 49 address)` keyed by a `paivana_secret` derived from the configured 50 `SECRET`. The construction is `GNUNET_CRYPTO_hkdf_gnunet()`, which is 51 HMAC-based but is not itself an HMAC. `SECRET` is required whenever the 52 paywall is on: it is the only input to that hash the client does not 53 supply, so a key invented afresh at each start would invalidate every 54 access anyone had already paid for. Under `-n` no cookie is ever minted 55 and none is needed. 56 57 Keep the same `SECRET` on every `paivana-httpd` serving one site — a 58 cookie issued by one has to verify on the next — and treat it as 59 equivalent to the content itself: whoever holds it can mint access for 60 any visitor, URL and expiry. 61 62 63 Dependencies 64 ------------ 65 66 - GNUnet (libgnunetutil, libgnunetjson, libgnunetcurl) 67 - libmicrohttpd 68 - libcurl >= 7.62.0 69 - libjansson 70 - libgcrypt 71 - GNU Taler: libtalerutil, libtalerjson, libtalermerchant, 72 libtalermhd, libtalertemplating 73 74 That is what the binary links against; `readelf -d` on it is the 75 authority. Paivana does not use libtalerexchange, and compresses 76 nothing itself — that happens inside libtalermhd, which brings zlib 77 with it. 78 79 80 Build 81 ----- 82 83 The project uses Meson but supports a GNU build process. 84 85 ./bootstrap 86 ./configure --prefix=$TARGET 87 make 88 sudo make install 89 90 91 Configuration 92 ------------- 93 94 Paivana reads an INI-style `.conf` file. The only section used is 95 `[paivana]`. A minimal working configuration: 96 97 [paivana] 98 DESTINATION_BASE_URL = https://example.com/ 99 MERCHANT_BACKEND_URL = https://backend.demo.taler.net/instances/sandbox/ 100 MERCHANT_ACCESS_TOKEN = secret-token:sandbox 101 BASE_URL = http://localhost:9967/ 102 SERVE = tcp 103 PORT = 9967 104 105 ### Required keys 106 107 Key Description 108 ---------------------- ----------------------------------------------------- 109 DESTINATION_BASE_URL Upstream server to proxy to once payment is confirmed. 110 MERCHANT_BACKEND_URL Base URL of the Taler merchant backend. 111 MERCHANT_ACCESS_TOKEN Bearer token for all calls to the merchant backend. 112 BASE_URL Public base URL of Paivana. Required unless `-f` 113 is given, in which case it is derived from the 114 forwarding headers (see below). 115 SECRET Key for the access-cookie MAC. Required unless 116 `-n`; see above for why there is no sensible 117 default. It is *not* an input to the Paivana ID, 118 which the browser has to be able to recompute on 119 its own. The Debian package generates one into 120 /etc/paivana/secrets/paivana.secret.conf, which 121 is not world-readable. 122 SERVE `tcp`, `unix` (Unix-domain socket) or `systemd` 123 (socket activation). There is no default: an 124 instance without it exits at startup, and it does 125 so only after the templates have been fetched. 126 PORT TCP port. Required when SERVE = tcp. 127 UNIXPATH Path to bind to. Required when SERVE = unix. 128 UNIXPATH_MODE Access mode of that socket, octal. Required 129 when SERVE = unix — it is what governs who may 130 reach Paivana, since a Unix-domain peer is 131 trusted to report the client address. 132 133 ### Optional keys 134 135 Key Description 136 -------- --------------------------------------------------------------- 137 WHITELIST POSIX extended regular expression; matching request paths 138 are forwarded without payment. Matched against the 139 *entire* path and anchored at both ends, so `/free/` 140 whitelists nothing while `/free/.*` whitelists that 141 subtree. 142 TRUSTED_PROXIES 143 IPv4 networks whose members are reverse proxies trusted to 144 report the client address in `Forwarded` or 145 `X-Forwarded-For`. Only consulted with `-f`, and only 146 needed when more than one proxy is in front. See "Trusted 147 proxies" below. 148 TRUSTED_PROXIES6 149 IPv6 counterpart of TRUSTED_PROXIES. 150 CONNECTION_LIMIT 151 Total number of concurrent client connections to accept, 152 default 512. Divided evenly over the listen sockets that 153 come up, so the process-wide total is what you set -- 154 with no BIND_TO there are two (IPv4 and IPv6). Paivana 155 also spends file descriptors on outbound requests from 156 the same table, so leave headroom below `ulimit -n`. 157 PER_IP_CONNECTION_LIMIT 158 Concurrent connections accepted from any one client 159 address, default 32; 0 disables the check. Set it to 0 160 wherever the peer address is not the client's -- under 161 SERVE = unix or systemd every client shares one peer 162 address, and behind a reverse proxy or a NAT many clients 163 do, so a limit there throttles everyone at once. 164 BIND_TO IP address to bind to; dual-stack wildcard if absent. 165 DESTINATION_UNIXPATH 166 Unix-domain socket to reach the upstream on instead of 167 connecting to the authority in `DESTINATION_BASE_URL`. That 168 URL is still what the request line and `Host` are built 169 from, so it remains required. 170 MERCHANT_BACKEND_UNIX_PATH 171 The same for the merchant backend, relative to 172 `MERCHANT_BACKEND_URL`. An unusable value is a warning and 173 the setting is then ignored, not a startup failure. 174 175 176 Running 177 ------- 178 179 $ paivana-httpd -c /etc/paivana/paivana.conf 180 181 Besides the options GNUnet gives every program (`-c` / `--config`, 182 `-L` / `--log`, `-l` / `--logfile`, `-h`, `-v`), Paivana takes four: 183 184 -n, --no-payment Bypass the paywall entirely — a pure reverse proxy, 185 which is what the test suite runs. No cookie is 186 ever minted and `SECRET` is not required. 187 -g, --global-payment 188 One payment grants access to the whole site rather 189 than to the URL it was made for. The website goes 190 into the cookie's keyed hash as the empty string, 191 and the cookie is scoped to `/` rather than to the 192 page, so cookies minted under one setting do not 193 verify under the other: flipping it invalidates 194 whatever access is outstanding. 195 -f, --respect-forwarded-headers 196 Take the client address from the forwarding 197 headers. Only safe behind a proxy that overwrites 198 them; see "Deployment behind a reverse proxy". 199 -u, --max-upload BYTES 200 Bytes of a request body held in memory at once 201 while relaying it upstream, default 262144. A 202 throughput knob, not a limit: the largest body 203 accepted is MAX_REQUEST_SIZE. For configurations 204 written when these were one number, setting this 205 and not MAX_REQUEST_SIZE still sets both. 206 207 The daemon does not serve requests until it has fetched paywall templates 208 from the merchant backend. If template loading fails, startup is aborted: 209 a template that could not be loaded would otherwise leave everything it 210 covers unpaywalled. Transient backend outages are the service manager's 211 job to ride out -- the shipped `paivana-httpd.service` restarts with an 212 increasing back-off, and does not restart on a configuration error. 213 214 An instance that offers no template at all is refused for the same 215 reason: with nothing to sell, no URL ever matches a paywall and the 216 entire site would be served for free without a word of warning. Serving 217 a site without a paywall is what `-n` is for, and it has to be asked for. 218 219 220 Deployment behind a reverse proxy 221 ---------------------------------- 222 223 The recommended production setup runs Paivana over a Unix socket and 224 places nginx or Apache in front for TLS termination. 225 226 In that setup Paivana **must** be started with `-f` / 227 `--respect-forwarded-headers`. A Unix-domain peer has no address of 228 its own, so without `-f` there is no client address at all: the access 229 cookie cannot be bound to a client, and `POST /.well-known/paivana` 230 fails. `-f` makes Paivana take the client address from the forwarding 231 headers instead, and forward the chain it was given to the upstream 232 rather than replacing it. 233 234 Paivana reads both the RFC 7239 `Forwarded` header and the de-facto 235 `X-Forwarded-*` ones, preferring `Forwarded` where both are present, 236 and emits both upstream — the standardized one for origins that speak 237 it, the de-facto ones for the many that do not. Of the `X-Forwarded-*` 238 family it emits `-For`, `-Proto`, `-Host` and `-Port`; the port is 239 taken from the front end's `X-Forwarded-Port` under `-f`, and otherwise 240 from the authority in `Host` when that names one. 241 242 What the origin sees as `Host` is **not** what the client sent: it is 243 the authority of `DESTINATION_BASE_URL`, because that is the name 244 Paivana connects to. The client's own value survives as 245 `X-Forwarded-Host` (and as the `host` parameter of `Forwarded`). An 246 origin doing virtual hosting must therefore be configured for the 247 `DESTINATION_BASE_URL` authority, and an origin that generates absolute 248 URLs should be told to build them from `X-Forwarded-Host` / 249 `X-Forwarded-Proto` / `X-Forwarded-Port`. If it builds them from 250 `Host` instead, its `Location` values will name Paivana's view of the 251 origin — an internal host and port, which Paivana relays unchanged. 252 That leaks the internal name, and points the client straight at the 253 origin wherever the client can route to it, bypassing the paywall. 254 255 `-f` is only safe if the server in front **writes** the forwarding 256 headers itself, whether by overwriting them or by appending its own 257 element. What it must not do is pass the client's copies through 258 untouched: Paivana believes the rightmost element (see "Trusted 259 proxies"), so a header no hop of yours has written is a header the 260 client filled in, and the client then chooses the identity its access 261 cookie is bound to. The configurations below get this right; if you 262 write your own, note that nginx sets `X-Forwarded-For` only when told 263 to and forwards a client-supplied `Forwarded` verbatim — and 264 `Forwarded` is the one Paivana prefers. 265 266 Conversely, do not pass `-f` to a Paivana that clients can reach 267 directly — there it is the client, not a proxy, that is setting those 268 headers. 269 270 nginx (`/etc/nginx/sites-available/paivana`): 271 272 server { 273 listen 443 ssl; 274 server_name example.com; 275 276 location / { 277 proxy_pass http://unix:/run/paivana/httpd/paivana-http.sock; 278 proxy_set_header Host $host; 279 280 # $remote_addr, not $proxy_add_x_forwarded_for: this is 281 # the outermost hop, so these overwrite rather than 282 # extend what the client claimed. 283 proxy_set_header X-Forwarded-For $remote_addr; 284 proxy_set_header X-Forwarded-Proto $scheme; 285 proxy_set_header X-Forwarded-Host $host; 286 proxy_set_header X-Forwarded-Port $server_port; 287 288 # RFC 7239; preferred by Paivana over the above. The 289 # element is built by a `map` — see the shipped config. 290 proxy_set_header Forwarded \ 291 "$paivana_forwarded_elem;proto=$scheme;host=$host"; 292 } 293 } 294 295 Apache (requires mod_proxy, mod_proxy_http and mod_headers): 296 297 <Location "/"> 298 # mod_proxy appends the real client to any X-Forwarded-For the 299 # client itself sent, so drop the client's copies first. 300 RequestHeader unset X-Forwarded-For 301 RequestHeader unset X-Forwarded-Proto 302 RequestHeader unset X-Forwarded-Host 303 RequestHeader unset X-Forwarded-Port 304 RequestHeader unset Forwarded 305 306 # RFC 7239; Apache emits none of its own. It has to be an 307 # expr= value: %{...}e reads the CGI environment, which is not 308 # populated when mod_headers runs, so the %{REMOTE_ADDR}e form 309 # yields the literal string "(null)". 310 RequestHeader set Forwarded \ 311 "expr=for=%{REMOTE_ADDR};proto=%{REQUEST_SCHEME};host=%{HTTP_HOST}" 312 313 ProxyPass "unix:/run/paivana/httpd/paivana-http.sock|http://example.com/" 314 </Location> 315 316 Ready-made versions of both are shipped in `debian/examples/`, and 317 installed by the Debian package into 318 `/usr/share/doc/paivana-httpd/examples/`. They are examples rather 319 than drop-ins on purpose: the package only *recommends* a web server, 320 so it must not create `/etc/nginx/` or `/etc/apache2/` on a system 321 that has neither (Debian Policy 9.1.1). Copy the one you want into 322 place and enable it yourself. 323 324 325 Trusted proxies 326 --------------- 327 328 `-f` on its own extends trust exactly one hop. The chain is walked 329 from the right and the walk stops at once, so the client is the 330 *rightmost* element — the one the peer we accepted the connection from 331 wrote. Entries a client prepends to its own header sit to the left of 332 that and cannot be promoted. 333 334 What `-f` alone therefore rests on is that the server in front sets or 335 appends those headers itself. Appending is safe here, because the real 336 peer ends up rightmost; passing the client's own headers through 337 unchanged is not, and that is the failure to watch for, since nginx 338 forwards a client-supplied `Forwarded` verbatim and Paivana prefers 339 `Forwarded`. 340 341 `TRUSTED_PROXIES` and `TRUSTED_PROXIES6` are what let the walk step 342 further left, through hops you have listed, when there is more than one 343 proxy in front: 344 345 [paivana] 346 TRUSTED_PROXIES = 10.0.0.0/8;192.168.0.0/16; 347 TRUSTED_PROXIES6 = 2001:db8::/32; 348 349 The walk then steps over each listed proxy in turn; the first element 350 that is not one of them is the client. Each step leftwards is 351 permitted only by the node being stepped over, so an element written by 352 someone you did not list is as far back as the chain can be believed. 353 354 Two things this does *not* do. The address that connected is never 355 matched against these lists — `-f` is what says the peer may speak for 356 a client, and the lists only govern how far past it the walk may go. 357 And an element that names no address (RFC 7239 `unknown`, an obfuscated 358 identifier, a host name) is not skipped either: the walk stops there 359 and Paivana falls back to the socket peer, which under `SERVE = unix` 360 means no client address at all and a `POST /.well-known/paivana` that 361 fails. A front server that emits a `Forwarded` element it cannot fill 362 in is therefore not a degraded paywall but a broken one. 363 364 `X-Forwarded-Proto`, `-Host` and `-Port` are outside all of this: they 365 are read as the leftmost value of the first such field line and the 366 walk never vets them. Under `-f` with no `BASE_URL` they are what 367 Paivana rebuilds its own scheme and authority from, which is the 368 residual reason to care about the front server's configuration even 369 with `TRUSTED_PROXIES` set. 370 371 Syntax notes, inherited from GNUnet's network-policy parser: 372 373 - entries are separated *and terminated* by `;` — a missing trailing 374 semicolon means nothing is parsed; 375 - `TRUSTED_PROXIES6` does not tolerate spaces between entries 376 (`TRUSTED_PROXIES` does); 377 - `0.0.0.0/0` and `::/0` cannot be expressed: they are 378 indistinguishable from the end of the list. 379 380 Anything that parses to an empty list is refused at startup rather 381 than silently trusting nobody. 382 383 Put IPv4 proxies in `TRUSTED_PROXIES`, not in `TRUSTED_PROXIES6` as 384 `::ffff:a.b.c.d`: addresses are folded to their IPv4 form before 385 matching, so a mapped entry would never be hit. 386 387 Set `BASE_URL` in the configuration file to the public HTTPS URL so 388 that redirects and cookie domains are correct. It may be omitted only 389 when `-f` is given: the flag asserts that a reverse proxy in front of 390 Paivana has already enforced a correct `Host`, which is what makes it 391 safe to reconstruct our own URL from the request. Without `-f` the 392 client is assumed to have connected directly, `Host` is whatever it 393 chose to send, and `BASE_URL` is therefore mandatory. 394 395 396 Source layout 397 ------------- 398 399 src/backend/ Main binary and all subsystems 400 paivana-httpd.c Entry point, scheduler, global state, shutdown 401 paivana-httpd_reverse.c Request-proxying state machine (core) 402 paivana-httpd_pay.c POST /.well-known/paivana handler 403 paivana-httpd_cookie.c Access-cookie keyed hash, Paivana ID 404 paivana-httpd_templates.c Paywall template loading and rendering 405 paivana-httpd_helper.c Client IP / base URL helpers 406 paivana-httpd_daemon.c MHD daemon startup 407 paivana_pd.c GNUnet project-data descriptor 408 src/frontend/ The paywall page served to unpaid visitors 409 paywall.en.must.j2 Mustache template source (Jinja2) 410 paywall.js Payment identifier, QR code, long poll 411 generate-paywall.py Renders the Jinja2 source at build time 412 src/include/platform.h GNUnet-style platform header (include first) 413 src/tests/ Automated reverse-proxy and unit tests 414 doc/prebuilt/ Git submodule: taler-docs (man pages) 415 416 417 Architecture notes 418 ------------------ 419 420 Single-threaded event loop: GNUnet scheduler drives both inbound HTTP 421 (libmicrohttpd) and outbound requests (libgnunetcurl / libcurl multi). 422 Running multiple `paivana-httpd` processes on the same port is 423 supported as the main way to scale-up the system. 424 425 Requests and responses are streamed in both directions: each is moved 426 through a fixed-size ring buffer (REQUEST_BUFFER_MAX and 427 RESPONSE_BUFFER_MAX, 256 KiB each by default) rather than assembled 428 whole, so the size of a proxied body is bounded by nothing in Paivana. 429 When the client cannot keep up, Paivana stops reading from the origin; 430 when the origin cannot keep up, it stops reading from the client. The 431 memory an in-flight request costs is therefore the two buffers, and the 432 worst case is that times CONNECTION_LIMIT. 433 434 An upload is still bounded, by MAX_REQUEST_SIZE (1 MiB by default), 435 because accepting one is a policy decision rather than a memory 436 constraint. A response is not bounded at all: an operator who wants to 437 bound what their origin serves can do it at the origin. 438 439 The MHD daemon is not started until paywall templates have been fetched 440 from the merchant backend asynchronously. 441 442 A 1xx interim response is not forwarded — RFC 9110 §15.2 asks a proxy 443 to forward them, and Paivana instead drops them, because the response 444 MHD is handed is a single final one. `103 Early Hints` 445 therefore does not reach clients through Paivana. Its header fields 446 are dropped with it rather than being merged into the final response, 447 which is the part that would be actively harmful. Trailer fields are 448 dropped for the same reason (RFC 9110 §6.5.1 forbids merging them into 449 the header section). 450 451 An origin that accepts the connection but does not produce response 452 *headers* within `UPSTREAM_TIMEOUT` (60 s) yields `504 Gateway 453 Timeout`; one that cannot be reached at all yields `502 Bad Gateway`. 454 The distinction matters because caches and monitoring retry the former 455 and not the latter. That clock is cancelled once the header section 456 ends, and it is the only one that can still produce a status code: 457 after it, the status is already on the wire. 458 459 There is deliberately no ceiling on how long a request may take — a 460 large download legitimately runs for as long as it runs. What is 461 bounded instead is a *stall*: `UPSTREAM_STALL_TIMEOUT` (60 s) is how 462 long the origin may move no bytes in either direction. The clock does 463 not run while Paivana is itself holding the origin back because the 464 client has not drained what has already arrived, so a client on a slow 465 link is never mistaken for a slow origin. 466 467 The MHD connection timeout does not cover any of this: a connection 468 waiting on the origin is suspended, and MHD drops suspended connections 469 from its timeout lists. 470 471 Once the response headers have gone out the status cannot be retracted, 472 so an origin that fails mid-body can only be reported as a framing 473 error: a declared `Content-Length` that is not met, or a chunked 474 response closed without its terminating chunk. Both are required to be 475 treated as failures by RFC 9112 §8.1.2. The exception is an HTTP/1.0 476 client receiving a response of unknown length, where the close *is* the 477 framing and truncation is indistinguishable from success. 478 479 `OPTIONS` carrying `Max-Forwards: 0` is answered by Paivana itself with 480 an `Allow` list, as RFC 9110 §7.6.2 requires of an intermediary; any 481 larger value is decremented before the request is passed on. 482 483 484 License 485 ------- 486 487 GNU Affero General Public License version 3 or later. 488 See COPYING for the full text. 489 490 491 Bug reports 492 ----------- 493 494 Please report bugs at https://bugs.taler.net/.