README (55888B)
1 paivana tests 2 ============= 3 4 This directory contains six test programs: 5 6 reverse_proxy an integration suite for the reverse-proxy side of 7 paivana-httpd, driven by test_reverse_proxy.sh 8 paywall an integration suite for the paywall itself, driven 9 by test_paywall.sh against a real GNU Taler system 10 payment_backend_failure 11 a small integration test that distinguishes an 12 immediate merchant transport failure from an 13 elapsed order deadline 14 client_address a unit test for the client address the access 15 cookie is keyed on (test_client_address.c) 16 cookie_header a unit test for the `Set-Cookie` line paivana emits 17 for that cookie (test_cookie_header.c) 18 cookie_access a unit test for the access decision the cookie 19 value encodes, and for the `paivana_id` the order 20 is created under (test_cookie_access.c) 21 22 The reverse-proxy suite runs paivana-httpd with `-n` (paywall 23 disabled) so no merchant backend is required: it only verifies that the 24 proxy correctly forwards HTTP requests and responses. The paywall 25 suite is the other half -- everything `-n` switches off -- and needs an 26 exchange, a merchant backend and a bank, so it skips where those are 27 not installed. Everything below describes the reverse-proxy suite 28 except the sections at the end. 29 30 What gets built 31 --------------- 32 33 The test suite uses four diverse upstream HTTP server implementations 34 so that paivana is not exercised only against libmicrohttpd peers: 35 36 upstream_mhd C / libmicrohttpd (built always) 37 upstream_go Go (net/http) (built if `go` is found) 38 upstream_rs Rust (std::net) (built if `rustc` is found) 39 upstream_py Python (stdlib) (pure interpreter; needs python3 40 at `make check` time) 41 42 Two further, special-purpose upstreams are also built: 43 44 early_response_upstream 45 C / raw sockets, single-connection. Sends a 413 46 response immediately after reading the request 47 headers, BEFORE consuming the request body — used 48 to exercise paivana's handling of an early upstream 49 response that lands while the client upload is 50 still in flight. Writes the body byte count it 51 observed to a receipt file, as a diagnostic. 52 53 With `--no-drain` it additionally stops reading 54 once it has answered, and parks the connection with 55 the rest of the request queued in a deliberately 56 tiny receive buffer. That is what makes paivana's 57 outbound socket back up: measured on loopback, the 58 upstream holds ~12 KiB unread while ~240 KiB of the 59 request sits undeliverable in paivana's send buffer. 60 A proxy that waited for that write to finish before 61 acting on the response it already holds would 62 deadlock; without `--no-drain` the condition never 63 arises, because the upstream keeps reading. 64 65 stream_upstream 66 C / raw sockets, one process per connection. Serves 67 bodies far larger than memory, at a rate the driver 68 chooses, and in framings a conforming server library 69 will not emit: a declared `Content-Length` that is 70 not delivered, a chunked response with no terminating 71 chunk, a connection that answers and then goes silent 72 for ever. Also reads and verifies a request body, 73 optionally answering before it has finished. 74 75 Bodies are a deterministic function of their own byte 76 offset rather than stored data, so a 200 MiB case 77 costs no disk on either side. Deliberately not a 78 constant byte: a repeated character would pass a 79 comparison that duplicated or dropped a whole aligned 80 block, which is exactly the mistake a ring buffer 81 with wrong wrap arithmetic makes -- and exactly the 82 mistake this caught during development. 83 84 The four canned upstreams all implement the same endpoints (see 85 "Endpoints" below). Two test clients talk to paivana directly: 86 `pipeline_client`, which uses BSD sockets to pipeline requests, and 87 `stream_client`, which verifies a body against the same generated 88 pattern as it arrives and reports what it saw about the *framing* -- 89 whether the response was chunked, what `Content-Length` reached the 90 client, how long the first byte took relative to the last. Verifying 91 incrementally is the point: a body written to a file and compared 92 afterwards says nothing about whether paivana streamed it or assembled 93 it first. 94 95 They all bind 127.0.0.1 and nothing else. They are not hardened in 96 any way -- POST /echo reflects whatever body it is given and GET 97 /large/10485760 hands out 10 MiB per request -- and they have no 98 business being reachable from the network for the duration of `make 99 check'. They also all reject an argument that is not a port in 100 1..65535 rather than defaulting: a silent fallback binds a port the 101 driver is not waiting for, and the failure then surfaces five seconds 102 later as "did not start on port NNNNN", naming the wrong thing. 103 104 Layout of the driver 105 -------------------- 106 107 `test_reverse_proxy.sh` is the single test program automake runs. 108 For each available upstream (mhd / go / py / rs) it: 109 110 1. starts the upstream on its port (see "Ports used"), 111 2. starts paivana-httpd -n pointed at that upstream, 112 3. runs a battery of HTTP tests with curl, wget, and the raw-socket 113 pipelining client, 114 4. stops paivana and moves on to the next upstream. 115 116 Cross-cutting error-path tests (405, 413, 502) are also covered, and 117 the final case restarts paivana pointed at a dead port to exercise 118 upstream-failure handling. 119 120 What each test covers 121 --------------------- 122 123 Per-upstream battery (`run_battery`): 124 125 GET /hello happy-path GET, body proxied unchanged 126 GET /status/201 2xx response status is forwarded intact 127 GET /status/404 4xx response status is forwarded intact 128 GET /status/500 5xx response status is forwarded intact 129 HEAD /hello HEAD method: the status is forwarded 130 HEAD /large/131072 RFC 9110 section 9.3.2's one normative 131 requirement on HEAD -- "MUST NOT send 132 content in the response" -- on a path 133 that yields 128 KiB under GET, so there 134 is something to leak. Read off the 135 socket rather than through curl, which 136 discards a body a HEAD response has no 137 business carrying and would therefore 138 report the bug as a pass. 139 GET /large/131072 128 KiB response body arrives byte for 140 byte, compared against the 'A'..'Z' 141 cycle the upstreams generate rather 142 than merely counted. (Bodies are 143 buffered whole, not streamed: a length 144 that matches says nothing about a 145 buffer reassembled in the wrong order.) 146 POST /echo request body is forwarded unchanged; 147 body round-trip 148 POST /echo (128 KiB) the same in the request direction and 149 at a size that spans several reads: 150 128 KiB of random bytes posted and 151 compared with what comes back. POST 152 /upload below checks only the count the 153 upstream reports, so without this 154 nothing here would notice a request 155 body that arrived complete but corrupt. 156 POST /upload (64 KiB) large random POST upload; upstream 157 reports the byte count it saw 158 PUT /put PUT method + body forwarding 159 PATCH /patch PATCH method + body forwarding 160 (paivana sets CUSTOMREQUEST) 161 DELETE /item/1 DELETE method, 204 No Content 162 OPTIONS /hello OPTIONS method, Allow header survives 163 the round-trip 164 GET /echo-headers paivana adds the reverse-proxy headers 165 X-Forwarded-For, X-Forwarded-Proto, Via 166 Host: rewritten the Host the upstream sees is the 167 authority of DESTINATION_BASE_URL, not 168 the one the client dialed, and is 169 host[:port] and nothing else (RFC 9110 170 §7.2 — no userinfo, no path, no query). 171 Note this only covers the destination 172 URLs paivana will actually accept: 173 TALER_is_web_url() rejects userinfo and 174 IPv6-literal DESTINATION_BASE_URLs at 175 startup, so those cannot be reached 176 from the driver. 177 custom X-Test header arbitrary client request headers are 178 forwarded unchanged 179 X-Upstream response header upstream response headers survive the 180 round-trip back to the client, and the 181 value names the upstream this battery 182 was pointed at -- a restart that 183 silently kept the previous destination 184 would satisfy a presence check 185 186 Forwarding-header tests (run once): 187 188 no -f a client's own X-Forwarded-For / 189 -Proto / -Host must not reach the 190 upstream: paivana is the outermost 191 proxy and replaces them with what it 192 can see for itself. In particular the 193 scheme must come from the transport, 194 not from a header the client wrote -- 195 TALER_mhd_is_https() believes 196 X-Forwarded-Proto, so paivana asks MHD 197 about the TLS session instead. 198 -f, chain extension with -f paivana is behind a trusted 199 proxy: the inbound chain is preserved 200 and paivana's own peer appended to the 201 right, rather than the chain being 202 thrown away. Also covers a repeated 203 X-Forwarded-For arriving as two field 204 lines (RFC 9110 §5.3: one combined 205 header must reach the upstream). 206 -f, trusted -Proto / -Host the values a trusted proxy sent are 207 passed through unchanged. 208 unix socket the deployment the Debian packaging 209 ships. A Unix peer has no address, so 210 with -f the inbound chain is forwarded 211 unadorned (nothing is appended, and no 212 placeholder is invented -- the hop is 213 recorded in Via), and without -f no 214 X-Forwarded-For is emitted at all. 215 This is the only case that reaches the 216 address-less code paths. 217 RFC 7239 Forwarded the standardized header is handled like 218 X-Forwarded-For: extended under -f, 219 replaced without it. Since paivana 220 prefers it when both are present, its 221 for/proto/host must also reach the 222 X-Forwarded-* headers, or an origin that 223 speaks only those would be told the 224 proxy was the client. A chain 225 containing a hop X-Forwarded-For cannot 226 express (§6.3 "unknown") yields no 227 synthesized chain rather than one with a 228 hop silently missing. 229 unix socket, Forwarded unlike X-Forwarded-For, RFC 7239 can 230 name an address-less hop, so paivana's 231 own element reads for=unknown rather 232 than being omitted. 233 TRUSTED_PROXIES startup a policy that parses to nothing usable 234 (missing trailing ';', a /0 network, an 235 address of the wrong family, junk) must 236 abort startup rather than silently 237 trusting nobody; usable ones must start. 238 WHITELIST startup an expression regcomp(3) cannot compile 239 must abort startup rather than leave 240 paivana matching against an 241 uninitialised regex_t; usable ones must 242 start. Two of the refused cases are 243 the anchoring: "a)|(b" and "(a$|^b" do 244 not balance on their own, so wrapping 245 them in "^(%s)$" yields an alternation 246 that has climbed out of the group and a 247 whitelist matching far more than it 248 says. paivana compiles the value bare 249 first for that reason, which is what 250 these two reach. 251 252 The matching itself is still out of 253 reach here: the regexec sits behind the 254 paywall that `-n` switches off, and 255 without `-n` paivana needs a merchant 256 backend to serve it templates before it 257 will start at all. So the anchoring at 258 *match* time -- a WHITELIST of "/free/" 259 waiving payment for every URL merely 260 containing it -- has no end-to-end 261 regression test. 262 263 Cross-cutting tests (run once): 264 265 POST /.well-known/paivana with `-n` the payment endpoint answers 266 501 rather than falling through to the 267 proxy -- the one paywall-side branch 268 `-n` does not shield. Both ways to get 269 it wrong are silent: forwarding the POST 270 would hand the origin payment data it 271 has no business seeing, and claiming the 272 path for every method would shadow 273 whatever the origin serves there, so the 274 GET of the same path is checked to still 275 be forwarded. 276 277 TRACE method unsupported HTTP verb yields 405 Method 278 Not Allowed (paivana rejects it, the 279 upstream is never contacted) 280 2 MiB POST upload request bodies above the 1 MiB 281 MAX_REQUEST_SIZE are rejected with 282 413 Content Too Large 283 curl keep-alive x3 three GETs over one keep-alive TCP 284 connection all succeed 285 wget /hello third-party client interop 286 HTTP/1.1 pipelining (x4) four requests sent back-to-back on a 287 single TCP connection *before* reading 288 any response; responses must come back 289 in the same order and with the correct 290 status codes (200, 201, 200, 404). 291 This specifically tests that paivana's 292 per-request state machine and MHD's 293 keep-alive handling cooperate correctly. 294 upstream down with paivana pointed at a closed port, 295 clients receive 502 Bad Gateway with 296 the built-in "Bad Gateway" HTML body 297 early upstream response against early_response_upstream, a 298 768 KiB POST that the upstream answers 299 with a 413 BEFORE reading the body. 300 Paivana must forward that 413 to the 301 client rather than turning it into a 302 502 — the early-response path, i.e. 303 #UP_DRAINING. 304 Note that the upstream is NOT expected 305 to see the whole body: RFC 9110 §9.3 306 lets a client stop sending once it has 307 a final response, and libcurl does 308 (plain curl against this upstream 309 sends ~128 KiB of the 768 KiB and 310 stops). The receipt is a diagnostic; 311 the test only requires that it appear, 312 i.e. that the exchange finished 313 upstream-side. 314 early response, no drain the same, with --no-drain: having 315 answered, the upstream never reads 316 again, so paivana's outbound socket 317 stays full with a request it can no 318 longer finish sending. It must still 319 answer its own client, promptly, with 320 the upstream's 413. Bounded by 321 timeout(1) rather than curl --max-time 322 because the failure mode is a hang: 323 paivana's stall watchdog would 324 eventually turn it into a truncated 325 response, which must not be allowed to 326 look like a slow pass. The drain-mode 327 case above cannot catch this — the 328 upstream there keeps reading, so the 329 socket never stays full. 330 331 The streaming tests (`test_streaming`) 332 -------------------------------------- 333 334 Everything above would pass equally well against the fully-buffered 335 proxy this replaced: every body in it fits in one buffer. These 336 cases are about what is new — that a body is no longer bounded by 337 memory, and that it starts reaching the client before the origin has 338 finished sending it. 339 340 200 MiB, Content-Length five times the 40 MiB ceiling that used 341 to make this a 502 outright. Body 342 verified byte for byte, and the 343 origin's own Content-Length must reach 344 the client rather than one recomputed 345 from an assembled buffer. 346 200 MiB, chunked the same body without a declared 347 length; must stay chunked to the 348 client instead of being silently 349 converted. 350 chunked to an HTTP/1.0 an HTTP/1.0 client cannot be sent 351 client chunks, so the close of the connection 352 has to be the framing. 353 Range -> 206 Content-Range and the partial body pass 354 through the streamed path. 355 HEAD on a large resource MHD does not run the content reader for 356 a HEAD but does emit the size the 357 response was created with, so the 358 length the equivalent GET would have 359 had now reaches the client (RFC 9110 360 §9.3.2). Buffering could only ever 361 report 0 here. 362 204 / 304 no body either way; the 304 still 363 carries the length of the body it does 364 not send (RFC 9110 §8.6). 365 200 MiB upload, both the request body is streamed too, so 366 framings the origin sees it byte-exact and sees 367 the client's own framing reproduced -- 368 a declared length stays declared, 369 chunked stays chunked. 370 small POST the overwhelmingly common case, which 371 now takes the same path as the large 372 one. 373 chunked upstream stops the status is long gone by the time the 374 mid-stream origin gives up, so the only remaining 375 way to say "incomplete" is to close 376 without the terminating chunk. curl 18 377 is the client noticing. 378 upstream goes quiet MHD will not time this out (a suspended 379 connection is off its timeout lists) 380 and CURLOPT_TIMEOUT is deliberately 381 unset, so paivana's own stall watchdog 382 is the only thing that can end it. 383 upstream never answers distinct from an upstream that is not 384 there, which is a 502: this is a 504, 385 and the time-to-headers clock is what 386 tells them apart. 387 100 abandoned downloads the ownership handshake between MHD's 388 completion notifier and the content 389 reader's free callback runs on every 390 request now, so a mistake in it is a 391 use-after-free or a leak on all 392 traffic. RSS across a hundred 393 abandoned transfers is the cheap 394 detector; ASan is the thorough one. 395 abandoned upload a Content-Length was declared upstream 396 that can no longer be delivered; the 397 origin has to be told the request is 398 broken rather than left waiting. 399 early 413 during a only reachable because the request body 400 200 MiB upload is streamed: with it buffered first the 401 origin could not have answered before 402 seeing all of it. 403 trailers, 1xx what patch 0034 established, re-checked 404 on the streamed path: neither may be 405 merged into a response that has already 406 been queued. 407 408 The congestion tests (`test_congestion`) 409 ---------------------------------------- 410 411 `test_streaming` shows that a large body gets through intact. It does 412 not show that it got through *without being held in memory*, and every 413 case in it would pass against a version that quietly buffered the lot 414 -- so on their own they leave the central claim of the change untested. 415 These are the cases that test it. 416 417 Three things are measured that the client cannot see for itself: 418 419 paivana's VmRSS while a body many times the buffer size is in 420 flight. This is the bound, stated directly. 421 422 How long the *origin* took to write its body, which `stream_upstream` 423 reports per connection on stderr ("served target=... bytes=N ms=M"). 424 A proxy that buffers takes everything at line rate however slowly its 425 own client reads; one that relays can only take what the client has 426 made room for. From the client end the two are indistinguishable, 427 which is why the origin has to report its own timing. 428 429 paivana's CPU time across an interval when nothing is moving. 430 Busy-waiting is the classic failure of a suspend/resume design and is 431 otherwise invisible: the transfer still completes, correctly, with a 432 core pinned for its duration. 433 434 Rate limits (`--read-rate`, `--upload-rate` on the client, `rate=` on 435 the upstream) are what make any of this reproducible. On loopback with 436 both ends going flat out, the kernel socket buffers absorb everything 437 and no ring ever fills. 438 439 The RSS bounds are skipped under `--enable-sanitizers`. ASan's 440 quarantine -- the thing that lets it catch a use-after-free -- holds 441 freed chunks rather than reusing them, so RSS there tracks total bytes 442 moved instead of bytes held: the 64 MiB case grows ~58 MB instrumented 443 against ~0.5 MB not, for identical code. The transfers still run and 444 LSan still watches them; the pacing and CPU assertions are unaffected 445 and are checked in both builds. 446 447 These sizes are deliberately *not* divided by PAIVANA_TEST_SCALE. Each 448 case is rate-limited, so its duration is set by the rate and not by the 449 size, and the sanitised build is no slower for them. Scaling them 450 would also break the pacing assertions: the kernel socket buffers hold 451 a fixed couple of megabytes however small the body is, so at a 452 twentieth of the size the origin legitimately finishes well ahead of 453 the client and "was it throttled" stops having a stable answer. 454 455 64 MiB through a slow peak RSS over baseline must stay within a 456 client few megabytes. Measured: +552 kB across 457 20 samples, against a hard 502 for this 458 size before the change. Checked against a 459 build with the ring cap removed, which 460 grows 12544k -> 78312k for the same body: 461 the bound does detect buffering. 462 upstream pacing the same transfer from the other end: the 463 origin's own elapsed time must track the 464 client's rather than finishing in a 465 fiftieth of it. Measured: 3597 ms to 466 write 64 MiB to a client that read for 467 4000 ms, where buffering would have taken 468 under 100 ms on loopback. 469 upload pacing the same assertion in the request 470 direction, against /sink. 471 32 concurrent throttled the per-request cost is what multiplies, 472 downloads so this is where a bound that holds for 473 one request and not for thirty-two shows. 474 Mixed rates, so the fast ones finish while 475 the slow ones are still going. Measured: 476 10.6 MB of growth for 32 x 8 MiB in 477 flight, about 339 kB each. Runs with 478 PER_IP_CONNECTION_LIMIT lifted, which is 479 otherwise exactly 32 and would have the 480 case measure connection limiting instead. 481 idle transfer an origin dribbling 200 B/s leaves paivana 482 with nothing to do for ~5 s. CPU must 483 stay near zero (measured: 1 jiffy, i.e. 484 10 ms, over 5107 ms), and the first byte 485 must still arrive at once -- measured at 486 1 ms -- rather than after the last. 487 1 KiB receive buffer makes libcurl drain paivana's socket in 488 tiny units, so MHD's content reader is 489 called hundreds of times where the default 490 buffer needs a handful -- each one a 491 chance for the ring to empty and the 492 connection to suspend and resume. Chunked, 493 so MHD's chunk framing is re-entered every 494 time. 495 slow at both ends neither side able to keep up with the 496 other on one request. Both rings spend 497 the transfer alternately full and empty 498 and the two halves of the state machine 499 have to interleave without deadlocking or 500 dropping a byte. 501 502 The base64url cross-check (`test_base64url.sh`) 503 ----------------------------------------------- 504 505 The paivana ID is `<expiration>-<base64url(sha256(...))>`. The daemon 506 builds it with `GNUNET_STRINGS_base64url_encode()`; the browser rebuilds 507 it in `paywall.js` to recognise the payment it has just made. Nothing 508 in either program forces the two encoders to agree, and if they do not, 509 the ID never matches, the payment appears not to go through, and 510 neither side logs anything wrong. 511 512 They have already disagreed twice. Once on the decode side: the daemon 513 emits the RFC 4648 section 5 (URL-safe) alphabet, and the browser fed it 514 to `atob()`, which only knows section 4 and throws on `-` or `_` -- at 515 module scope, so the whole script died and the paywall could not be 516 paid. Once on the encode side: the browser used 517 `Uint8Array.prototype.toBase64`, which is a 2024-25 addition (Firefox 518 133, Safari 18.2, Chrome 140) and is simply not a function on anything 519 older. Two bugs of the same shape in one file is what this test is for. 520 521 `base64url_vectors` prints 369 vectors as the *daemon* produces them -- 522 every length from 0 to 96, so both amounts of padding and none are 523 crossed; every single byte value, because `-` and `_` are only 524 reachable from particular high bit patterns and are exactly the two 525 characters the section 4 alphabet spells differently; and sixteen 526 32-byte blocks, that being the size which actually occurs. 527 `test_base64url.sh` lifts `base64url()` out of `paywall.js` by matching 528 braces -- rather than keeping a copy here, which would be a second 529 implementation to hold in step, and holding implementations in step by 530 hand is the thing that failed -- and compares. It refuses to pass on 531 fewer than 300 vectors, so a generator that broke would fail rather 532 than trivially agree. Skips (77) without node. 533 534 Checked by breaking it: with the alphabet translation removed from 535 `paywall.js`, it reports `paywall.js gave "WH2ix+wRNls", the daemon 536 gives "WH2ix-wRNls"`. 537 538 The client_address unit test 539 ---------------------------- 540 541 `test_client_address.c` covers PAIVANA_HTTPD_resolve_forwarding(), 542 the single walk over the forwarding chain that decides both the client 543 address PAIVANA_HTTPD_get_client_address() hands to the cookie MAC and 544 the scheme and authority PAIVANA_HTTPD_get_base_url() rebuilds the 545 website string from. That function is deliberately pure — it takes 546 the socket peer, the ordered field lines of each forwarding header and 547 the trust configuration, and nothing else — so the whole policy is 548 reachable without an MHD connection; the MHD half is a thin adapter. 549 550 The access cookie is an HMAC over (expiration, website, client 551 address). A host therefore has to produce the *same bytes* however 552 paivana learns its address, or the cookie it was issued silently stops 553 verifying and the visitor is asked to pay again. The test asserts: 554 555 - an X-Forwarded-For value and the socket address of the same host 556 yield identical bytes (including ::ffff:a.b.c.d from a dual-stack 557 listener versus a.b.c.d from a proxy), 558 - alternative spellings of one address are one identity 559 ("::1" / "0:0:0:0:0:0:0:1", upper/lower case hex), 560 - a value that is not a bare IP address is refused rather than 561 turned into an identity of its own (port suffixes, brackets, RFC 562 7239 "unknown"/"_hidden", hostnames, zone ids, junk), 563 - a cookie issued for one host is not accepted for another. 564 565 A table-driven group then covers the walk itself. `-f` means "we are 566 behind a trusted reverse proxy", so the socket peer is trusted 567 implicitly and TRUSTED_PROXIES / TRUSTED_PROXIES6 name the *additional* 568 hops further out; the walk steps leftwards over a node only while that 569 node is trusted and stops at the first one that is not. The rows 570 cover: 571 572 - without `-f`, the socket peer wins even with every forwarding 573 header present, 574 - a single proxy and a single element, in both spellings, 575 - two and three proxies with only some of them listed, and an 576 untrusted node in the middle, which stops the walk where it should, 577 - a chain of nothing but trusted hops, where the leftmost is all 578 there is, 579 - IPv4, bracketed IPv6, IPv6 with a port, RFC 7239 §6.3 "unknown" 580 and an obfuscated identifier, 581 - repeated field lines of one header and a field line that is itself 582 a list (RFC 9110 §5.3), 583 - quoted strings with escapes, and an unterminated one, which used 584 to be read past the end of the header, 585 - malformed, empty and whitespace-only headers, all of which fall 586 back to the socket peer rather than losing it, 587 - `Forwarded` winning where both headers are present, 588 - a chain of 2500 elements, which is refused outright rather than 589 walked: every element used to be located by rescanning the header 590 from byte 0. 591 592 A second table covers what the base URL is built from: `proto=` and 593 `host=` taken from the same element the address came from, the 594 X-Forwarded-Proto/-Host/-Port fallbacks, an X-Forwarded-Host that 595 already carries a port together with an X-Forwarded-Port (which must 596 not yield "example.com:8443:8443"), ports re-rendered rather than 597 echoed, and hosts and schemes that are refused because they are not 598 one. 599 600 A further group covers the rendering back out — a `for=` identifier, 601 an X-Forwarded-For chain, and RFC 7239 §4 values, where a parameter 602 that would otherwise splice a second forwarded-element into a header 603 we build is either quoted or reported as absent. 604 605 A separate group pins the behaviour of GNUnet's 606 GNUNET_STRINGS_parse_ipv{4,6}_policy() that load_trusted_proxies() 607 compensates for: the mandatory trailing ';', the v4/v6 disagreement 608 about spaces, the two ways those parsers return "nothing usable" 609 without returning NULL (a /0 network, which is indistinguishable from 610 the list terminator, and an address of the wrong family), and the one 611 way they return "usable, but not what was written" — a final entry 612 without its ';', or anything after the last ';', is dropped and the 613 prefix reported as success, which is why the loader counts separators. 614 If upstream ever fixes these, this group is what says so. 615 616 The startup validation built on top of that is in the integration 617 suite instead, since it is about whether the daemon comes up. 618 619 It links paivana-httpd_helper.c and paivana-httpd_cookie.c directly 620 and supplies the daemon globals itself, so it needs no MHD connection 621 and no merchant backend. The integration suite cannot cover any of 622 this: with `-n` the cookie path is never reached, so the client 623 address is never computed. 624 625 The cookie unit tests 626 --------------------- 627 628 The same applies to the two cookie tests, and for the same reason: 629 `-n` sets do_forward before the request is looked at, so nothing in 630 the integration suite ever mints or checks a cookie. Both link only 631 paivana-httpd_cookie.c. 632 633 `test_cookie_header.c` is about the header paivana emits, i.e. about 634 whether the credential the client just paid for ever comes back: the 635 `Path` re-encoding (the browser matches against the encoded request 636 path, while the URL paivana holds has been decoded by MHD), the RFC 637 6265 §4.1.1 grammar the attribute has to satisfy, attribute injection 638 through a path containing ';', `Secure`, and the `Max-Age` floor that 639 keeps a sub-second access from being deleted on arrival. 640 641 `test_cookie_access.c` is about the decision made when it does come 642 back. The cookie is a bearer token we hand to the party most 643 interested in widening it, so each of the three things it is minted 644 for -- expiration, website, client address -- is checked to be inside 645 the MAC and re-checked on presentation, including the obvious attempt: 646 reading the expiration off the value and writing a later one. Each of 647 the ways check_cookie() can reject a value has its own case, so that a 648 malformed value ends in a refusal rather than in a read past the end 649 of a string the client chose. A separate group covers the values that 650 are not malformed at all but merely respelled -- a leading '+', a 651 leading space, a leading zero, junk between the seconds and the '-' -- 652 each of which decodes to the same seconds and the same hash as a 653 cookie we really issued, and so is a second live spelling of one 654 credential unless the parser refuses it. `-g` is covered here and 655 nowhere else. 656 657 The `paivana_id` is pinned against a golden vector computed 658 independently from the definition the paywall page implements 659 (src/frontend/paywall.js, makePaivanaId()). Neither side ever sends 660 it; both derive it from their own copy of (nonce, website, expiration) 661 and expect the other to have got the same string, so the two 662 implementations agreeing IS the protocol, and a change on either side 663 that this vector does not survive means every order is created under 664 an id the other side will not look for. 665 666 667 The benchmark (`benchmark.sh`) 668 ------------------------------ 669 670 Not a test -- it asserts nothing about correctness and its result 671 depends on the machine. It answers two questions: how much 672 throughput does putting paivana in front of an origin cost, and how 673 fast is paivana on its own when the paywall turns a client away? 674 675 meson test --benchmark proxy_overhead -C build # the first 676 meson test --benchmark paywall_page -C build # the second 677 678 Both are registered with meson's `benchmark()` rather than `test()`, 679 which is what keeps them out of `make check`: `meson test` does not 680 run benchmarks. They are two entries rather than one so that they 681 skip independently -- the paywall arm needs things the proxy arm does 682 not, and should not be able to take it down with it. Run the script 683 directly for the knobs -- `-c` clients, `-s` page size, `-d` seconds, 684 `-m direct|proxy|paywall|both|all`. It exits 77 when rustc was not 685 available to build upstream_rs, or when the build is sanitized (those 686 timings measure ASan), and 1 if any request failed, since a run with 687 failures has not measured throughput. 688 689 N curl workers fetch a fixed-size page for a fixed time. In `direct` 690 they fetch it straight from upstream_rs, in `proxy` through paivana 691 in front of that same upstream_rs, and in `paywall` they fetch 692 paivana's own 402 page with no upstream in the path at all. For each 693 arm it reports requests, requests/s, MB/s (10^6), the page size and 694 the server processes' CPU time; then the proxy/direct ratio, and the 695 paywall/proxy one. Nothing touches disk: the upstream's page comes 696 from a buffer it fills once at startup -- it used to regenerate it a 697 byte at a time per request, which was real work charged to the arm 698 that has no proxy in it -- the paywall page comes out of paivana's 699 response cache, and the clients discard every body. 700 701 The expected size is measured from a warm-up request rather than 702 assumed, because in paywall mode nothing here knows it up front: it 703 is whatever `paywall.en.must` renders to (50300 bytes as of writing). 704 That measured size is then what every later response is checked 705 against, so a short body counts as a failure rather than as 706 throughput that was not achieved. 707 708 Three properties of the setup shape the number, and all three make 709 the proxy look better than it is, so the reported ratio is a floor: 710 711 - Every request gets a fresh TCP connection in *both* arms. That 712 is forced, not chosen: upstream_rs answers one request per 713 connection and closes, so the direct arm cannot keep-alive at 714 all, while paivana's client side happily would -- MHD strips the 715 upstream's hop-by-hop `Connection: close` and decides the client 716 connection's fate itself. Measured: without the clients sending 717 `Connection: close`, curl's second request through paivana 718 reports num_connects=0 and the same request direct reports 1. 719 So the clients send it, and both arms are charged one TCP setup 720 per request. 721 722 - curl's own process startup (12.5 ms on the machine this was 723 written on -- it links openssl, nghttp2, brotli, zstd, ldap) is 724 amortised by handing each curl invocation a batch of URLs. The 725 batch size converges at run time rather than being computed from 726 the page size, because how long a batch takes depends on the 727 per-worker request rate, which is what is being measured: a size 728 picked up front overshot a 3 s run by 7% at `-c 32`. At a fixed 729 batch of 8 the startup was 44% of the run and the reported rate 730 came out at half the truth. 731 732 Both of the above are per-request constants added to *both* 733 arms, so they pull the ratio toward 1. 734 735 - paivana is single-threaded by construction -- one GNUnet 736 scheduler driving MHD and libcurl -- and upstream_rs spawns a 737 thread per connection, so on a multi-core box the direct arm may 738 use every core and the proxy arm may not. That is a real 739 property of paivana rather than an artefact of the harness, but 740 it does make the ratio a function of the core count, which is 741 why CPU seconds and `nproc` are printed with it. 742 743 A fourth applies to paywall mode only: the paywall page is not the 744 `-s` size, so that arm is comparable to the others in requests/s and 745 not in MB/s. The script prints both page sizes next to the ratio and 746 does not offer a MB/s one. 747 748 For orientation, one run on a 24-core machine at the defaults (8 749 clients, 64 KiB page, 10 s): 750 751 direct 34932 req/s 2289 MB/s upstream_rs 3.07 cores 752 proxy 6816 req/s 447 MB/s paivana 0.97 cores 753 paywall 33993 req/s 1710 MB/s paivana 0.90 cores 754 755 so 0.20x of direct through the proxy, and 4.99x the forwarded rate 756 for the paywall page (50300 bytes). paivana is at 0.97 cores 757 forwarding: it is saturating its single thread, which is the bound 758 that matters. Sweeping the paywall arm shows the same bound from the 759 other side -- 26852 req/s at `-c 4` and 0.81 cores, 33504 at `-c 8`, 760 36049 at `-c 16` and 0.98 cores, 35892 at `-c 32` -- i.e. it stops 761 scaling exactly where the thread runs out, at about 36k req/s. 762 763 Do not quote those figures. They are stable to a couple of percent 764 when the machine is quiet, but an *earlier* set on the same 24-core 765 box read 13375 req/s direct and 4707 through paivana, i.e. 0.35x 766 rather than 0.20x, because the direct arm was then getting only 1.5 767 cores instead of 3.1. The clients are a bash loop forking curl and 768 they compete with the servers for the machine, so under contention 769 the fastest arm loses the most and the ratio flatters the proxy. The 770 run's own CPU numbers are what tell you which regime you were in. 771 772 Why the paywall arm needs a merchant backend, and why a stub is 773 honest here. paivana does not open its listen socket until it has 774 fetched a template from a merchant backend 775 (PAIVANA_HTTPD_load_templates -> templates_ready -> 776 PAIVANA_HTTPD_serve_requests), so paywall mode cannot measure 777 anything without one. It starts `merchant_stub`, which answers the 778 two GETs of that startup exchange -- shaped as 779 merchant_api_get-private-templates{,-TEMPLATE_ID}.c parse them, with 780 the contract test_paywall.sh POSTs to a real backend -- and nothing 781 else. That is the whole of what a real backend would do here: the 782 template is fetched once, the page is rendered locally from it, and 783 the rendered MHD_Response is cached process-wide per (template, language, 784 encoding) in load_paywall(), so from the first measured request onwards a live 785 merchant is exactly as idle as the stub. This benchmark never buys 786 anything, so no code path that can tell the two apart is reached. 787 The stub does check the bearer token, since paivana building that 788 header out of MERCHANT_ACCESS_TOKEN is the one part of the exchange 789 that could silently regress; a 401 makes paivana refuse to start 790 rather than start against a configuration nobody would deploy. To 791 check the claim instead of taking it, set PAIVANA_BENCH_MERCHANT_URL 792 (and PAIVANA_BENCH_MERCHANT_TOKEN) at a live backend carrying a 793 `paivana` template -- named `premium`, or whatever 794 PAIVANA_BENCH_TEMPLATE_ID says. 795 796 Like test_paywall.sh, paywall mode stages `paywall.en.must` into a 797 scratch prefix and points PAIVANA_PREFIX at it, rather than requiring 798 `make install` for a page that lives in the build tree. 799 800 The likeliest cause of failures here is not paivana: one connection 801 per request against a fixed server port pins the 4-tuple for the 802 TIME_WAIT duration, and a few runs back to back can fill the local 803 ephemeral range (28k ports by default against ~13k connections per 804 run). The script says so when it sees transfers that never 805 connected. 806 807 808 Environment variables 809 --------------------- 810 811 The driver script honors: 812 813 PAIVANA_HTTPD path to paivana-httpd (default: the in-tree build) 814 SRCDIR directory containing the upstream sources and the 815 conf template (default: dirname of the script) 816 BUILDDIR directory containing upstream_mhd, pipeline_client, 817 upstream_go, upstream_rs (default: $PWD) 818 KEEP_TMP=1 do not delete the scratch dir on exit 819 PAIVANA_PORT_BASE 820 first port of the block the suite binds 821 (default 18400); see below 822 823 benchmark.sh honors the first four of those, plus: 824 825 PAIVANA_BENCH_PORT_BASE 826 first port of its own block (default 18600) 827 PAIVANA_BENCH_MERCHANT_URL 828 a live merchant backend for paywall mode, instead 829 of starting merchant_stub 830 PAIVANA_BENCH_MERCHANT_TOKEN 831 bearer token for it 832 PAIVANA_BENCH_TEMPLATE_ID 833 template to ask that backend for (default 834 `premium`, which is what merchant_stub serves) 835 836 Ports used 837 ---------- 838 839 Every port is an offset off PAIVANA_PORT_BASE, which defaults to 18400 840 -- the 184xx / 185xx range, chosen to avoid collisions with real 841 services. The suite checks all ten before it starts anything and 842 exits 77 (meson reads that as SKIP) if one of them is taken, naming 843 it. 844 845 That check is not a formality. Readiness used to be "does something 846 accept on this port", which is a different question from "did our 847 child come up": a paivana that lost the bind to a squatter -- most 848 often a stale one of its own from an earlier run -- read as started, 849 and the suite then ran its checks against the wrong process. With a 850 stale paivana of a different vintage they even pass. The startup 851 validation cases are the worst affected, since those decide "refused" 852 from exactly that probe. 853 854 base + 1 (18401) upstream_mhd 855 base + 2 (18402) upstream_go 856 base + 3 (18403) upstream_py 857 base + 4 (18404) upstream_rs 858 base + 5 (18405) early_response_upstream 859 base + 6 (18406) early_response_upstream --no-drain 860 base + 7 (18407) truncating upstream (short-body test) 861 base + 8 (18408) stream_upstream (streaming tests) 862 base + 99 (18499) dead port (for "upstream down" test) 863 base + 100 (18500) paivana-httpd 864 865 Move the base to run the suite in two checkouts at once, or beside a 866 paivana you are debugging: 867 868 PAIVANA_PORT_BASE=18700 meson test -C build reverse_proxy 869 870 benchmark.sh binds its own three ports off a separate base, so it can 871 run beside the suite: PAIVANA_BENCH_PORT_BASE, default 18600, giving 872 18601 for upstream_rs, 18602 for paivana and 18603 for merchant_stub. 873 It checks the ones the chosen `-m` actually needs, and skips the same 874 way. The `paywall_page` benchmark entry passes `-b 18610` so that 875 the two entries cannot collide if anyone runs the benchmarks in 876 parallel, which is not meson's default but is one flag away. 877 878 Endpoints (implemented by every upstream) 879 ----------------------------------------- 880 881 GET /hello text "Hello from <name>\n" 882 GET /status/NNN respond with status NNN and a trivial 883 text body "status NNN\n" 884 GET /large/N N bytes of 'A'..'Z' repeating 885 GET /slow/N sleep N ms, then "slept\n" 886 GET /echo-headers text listing of received request 887 headers, "Key: Value\n" per line 888 POST /echo body is echoed verbatim 889 POST /upload "Received N bytes\n" 890 PUT /put "PUT received N\n" 891 PATCH /patch "PATCH received N\n" 892 DELETE /item* 204 No Content 893 OPTIONS * 204 + Allow: GET, POST, PUT, ... 894 895 Every response also carries an `X-Upstream:` header whose value 896 identifies which server handled it (mhd, go, py, rs); the client 897 test cases use it to confirm that responses are coming back from 898 the expected backend. 899 900 901 Payment-backend failure diagnostics 902 ----------------------------------- 903 904 `test_payment_backend_failure.sh` starts Paivana against a small Python 905 merchant stub through `merchant_fault_proxy.py`. Paivana advertises a public 906 HTTPS merchant URL while its template and order requests use a distinct HTTP 907 URL through the proxy's Unix listener, including the full merchant-instance 908 prefix. The test verifies that the internal URL does not leak into the 909 paywall, and that malformed internal URLs fail at startup. Template and 910 warm-up order requests leave a real persistent merchant connection idle. The 911 fault frontend then resets the next order lookup on that reused connection 912 before forwarding or access-logging it, also covering libcurl's internal fresh 913 retry and Paivana's application retry. Paivana must return promptly as 502 / 914 error 9801 with `merchant_http_status: 0`, while its forced-fresh diagnostic 915 and five separate curl processes all reach the merchant. The test explicitly 916 verifies that none of the failed attempts appears in either the frontend or 917 merchant access log. It then stops the stub to cover a refused port, restarts 918 it to prove a real HTTP response resets the consecutive-failure count, and 919 finally holds an order lookup open long enough to retain the 504 / error 11 920 timeout path. This deterministic version needs neither PostgreSQL nor a full 921 Taler deployment. 922 923 924 The paywall suite 925 ----------------- 926 927 `test_paywall.sh` covers what `-n` hides. It puts a real GNU Taler 928 system behind paivana-httpd -- a fakebank, an exchange and a merchant 929 backend, started with `taler-unified-setup.sh` exactly as the merchant 930 and anastasis suites start theirs -- creates a Paivana template on the 931 merchant instance, buys access with `taler-wallet-cli`, and checks what 932 the daemon does with the result. 31 checks, about 25 seconds. 933 934 The production failure has a real-stack reproduction mode: 935 936 PAIVANA_REPRO_MERCHANT_POOL_FAILURE=1 \ 937 meson test -C build --print-errorlogs paywall 938 939 In this mode the same fault frontend sits between Paivana and the real 940 PostgreSQL-backed merchant, but remains unarmed through startup, withdrawal and 941 payment. Once the wallet has paid a genuine order, the test arms exactly that 942 order path and proves all of the reported production properties together: 943 Paivana returns 502 / code 9801 / merchant status zero; the first reset was on 944 an idle pooled connection; no failed request reached the frontend access log; 945 the forced-fresh diagnostic receives HTTP 200; and five subsequent fresh order 946 status requests all succeed through the same frontend. 947 948 It skips (exit 77) rather than failing when the environment cannot 949 support it: no `taler-unified-setup.sh`, `taler-wallet-cli`, 950 `taler-merchant-httpd`, `jq`, `python3` or PostgreSQL, no built paywall 951 template, or one of its ports already in use. A skip names what was 952 missing. 953 954 Ports. paivana's own two move with `PAIVANA_PORT_BASE` (+110 and 955 +111), but the Taler system's are fixed at 9966 (merchant), 8081 956 (exchange) and 8082 (bank) -- the same ones the merchant suite uses, so 957 the two cannot run at once and this suite skips when they are busy. It 958 also wants a PostgreSQL database named `paivanacheck`, which it creates 959 if it can; `talercheck` is deliberately not reused, since the merchant's 960 own tests would then be clobbering these tables and vice versa. 961 962 The paywall template is staged into a throwaway prefix and reached 963 through `PAIVANA_PREFIX`, so a build tree is enough and `make install` 964 is not required. 965 966 Why the client half is written out by hand. The paywall page computes 967 a payment identifier from (nonce, website, expiration) and the daemon 968 computes the same identifier independently; neither ever sends it to 969 the other, so the two agreeing IS the protocol. `paivana_id.py` 970 re-derives it -- and the Crockford base32 encoding of the nonce -- from 971 the definition `src/frontend/paywall.js` implements, which is what makes 972 this a test of both ends rather than of one end twice. It agrees with 973 the golden vector in `test_cookie_access.c`, which was computed the same 974 way; if you change the derivation, three places have to move together. 975 976 What it covers, in order: 977 978 * the unpaid path: 302 to the paywall, the template named in the 979 Location and the website base64url-encoded in the fragment, the 980 402 page itself, its `Paivana:` pay-template URI and its CSP; 981 * the whitelist, and specifically that a WHITELIST expression is 982 anchored at both ends -- `/echo-headers` waives that path and not 983 `/x/echo-headers` or `/echo-headers/x`. The regexec that decides 984 this sits behind the paywall, so no other test in the tree can 985 reach it; 986 * the redemption endpoint's refusals: a body missing its fields, a 987 nonce of the wrong length, an order the merchant never saw; 988 * a real payment, redeemed for a real access cookie, and that cookie 989 opening the URL it was minted for and no other; 990 * that rewriting the expiration in the cookie value invalidates it 991 (the expiration is the KDF salt) and that a malformed cookie is 992 refused rather than mis-parsed; 993 * an order bought for a DIFFERENT fulfillment URL under the session 994 we then claim. The merchant sells it, the session lookup succeeds, 995 and the only thing between that and a cookie for a page nobody paid 996 for is paivana comparing the contract's fulfillment URL against the 997 website claimed. This is the one case that reaches that comparison: 998 naming another website in the redemption changes the payment 999 identifier, so every simpler attempt is refused earlier, by the 1000 session lookup; 1001 * that redemption is repeatable from anywhere, which is deliberate 1002 (design document 076, "Payment buys access, not a seat"). The 1003 check is here so that a change of mind about it surfaces as a test 1004 failure rather than as a silent change of policy. 1005 1006 The checks were verified not to be vacuous by breaking the code under 1007 them, one property at a time: dropping the `^(...)$` wrapping around 1008 WHITELIST turns the two anchoring cases red (`/x/echo-headers` reaches 1009 the origin); dropping the website from the cookie's keyed hash lets the 1010 paid cookie open `/item` as well; and skipping the fulfillment-URL 1011 comparison lets an order bought for `/elsewhere` mint a cookie for 1012 `/item`.