commit 9f620f17eda8adbd91f13167ec72c9e6d2d8f005
parent b2f7bd8648d36ddf25ad724e80632856fd83bfb8
Author: Florian Dold <dold@taler.net>
Date: Tue, 25 Aug 2026 22:49:11 +0200
paivana-httpd: bound resources and drain gracefully under load
Diffstat:
19 files changed, 1791 insertions(+), 297 deletions(-)
diff --git a/NEWS b/NEWS
@@ -1,4 +1,17 @@
Unreleased:
+ - High-load operation is now fail-fast and bounded. CONNECTION_LIMIT
+ defaults to 384 and may not exceed the select()-safe descriptor budget;
+ PAYMENT_CONNECTION_LIMIT reserves 32 request slots for redemptions, and
+ RELAY_MEMORY_LIMIT validates aggregate streaming-ring memory. Unsafe
+ descriptor, RLIMIT_NOFILE and memory combinations are startup errors.
+
+ - SIGTERM quiesces listeners and drains accepted requests for
+ SHUTDOWN_GRACE_PERIOD (60 s by default). Template discovery, concurrent
+ detail fetches, retained Paivana templates, contract size, installed
+ languages and the process-wide rendered-response cache now have documented
+ bounds. Repeated merchant transport diagnostics are sampled once per
+ minute per failure class with a recovery summary.
+
- The reverse proxy streams both directions. A request or response
body is relayed as it arrives instead of being assembled in memory
first, so its size is no longer bounded by memory -- previously
diff --git a/README b/README
@@ -149,14 +149,21 @@ Paivana reads an INI-style `.conf` file. The only section used is
IPv6 counterpart of TRUSTED_PROXIES.
CONNECTION_LIMIT
Total number of concurrent client connections to accept,
- default 512. Divided evenly over the listen sockets that
+ default 384. Divided evenly over the listen sockets that
come up, so the process-wide total is what you set --
- with no BIND_TO there are two (IPv4 and IPv6). Paivana
- also spends file descriptors on outbound requests from
- the same table, so leave headroom below `ulimit -n`.
- Paivana warns at startup when the soft limit is below twice
- CONNECTION_LIMIT plus a small reserve. The shipped systemd
- service sets `LimitNOFILE=4096` for the default limit.
+ with no BIND_TO there are two (IPv4 and IPv6). The current
+ event loop uses select() and can represent only 1024 file
+ descriptors. Each active request may hold a client and an
+ outbound socket, so the maximum is (1024 - 256 reserve) / 2
+ = 384. The reserve covers listeners, curl pools, resolver,
+ scheduler and transient overlap. Paivana refuses unsafe
+ values or an RLIMIT_NOFILE below the calculated requirement.
+ PAYMENT_CONNECTION_LIMIT
+ Slots reserved within CONNECTION_LIMIT for payment-redemption
+ POSTs, default 32. This leaves 352 ordinary slots and also
+ bounds the merchant lookups that unauthenticated clients can
+ hold for their five-second long poll. Must be at least 1 and
+ smaller than CONNECTION_LIMIT.
PER_IP_CONNECTION_LIMIT
Concurrent connections accepted from any one client
address, default 32; 0 disables the check. Set it to 0
@@ -164,6 +171,20 @@ Paivana reads an INI-style `.conf` file. The only section used is
SERVE = unix or systemd every client shares one peer
address, and behind a reverse proxy or a NAT many clients
do, so a limit there throttles everyone at once.
+ RELAY_MEMORY_LIMIT
+ Aggregate accounting ceiling for ordinary-request streaming
+ rings, default 268435456 bytes (256 MiB). Startup requires
+ (REQUEST_BUFFER_MAX + RESPONSE_BUFFER_MAX) multiplied by
+ (CONNECTION_LIMIT - PAYMENT_CONNECTION_LIMIT) to fit. The
+ defaults account for 176 MiB and retain 80 MiB of this budget
+ for configuration growth; templates, curl and allocator memory
+ remain outside the accounting, which is why the margin matters.
+ SHUTDOWN_GRACE_PERIOD
+ Time accepted requests may finish after SIGTERM, default 60 s;
+ 0 means immediate shutdown. Listeners are quiesced first, so
+ socket activation queues new connections for the replacement.
+ Keep the service manager's stop timeout above this value; the
+ package uses 75 s to leave 15 s for cancellation and cleanup.
BIND_TO IP address to bind to; dual-stack wildcard if absent.
DESTINATION_UNIXPATH
Unix-domain socket to reach the upstream on instead of
@@ -272,6 +293,13 @@ headers.
nginx (`/etc/nginx/sites-available/paivana`):
+An optional per-client concurrency limit belongs here rather than in
+Paivana when a Unix socket is used. The shipped example contains commented
+`limit_conn_zone` / `limit_conn` directives. Its value is a site policy,
+not part of Paivana's descriptor calculation: a low value also combines all
+legitimate users behind the same NAT, and nginx behind another proxy must
+first be configured to trust and recover the real client address.
+
server {
listen 443 ssl;
server_name example.com;
@@ -431,8 +459,17 @@ RESPONSE_BUFFER_MAX, 256 KiB each by default) rather than assembled
whole, so the size of a proxied body is bounded by nothing in Paivana.
When the client cannot keep up, Paivana stops reading from the origin;
when the origin cannot keep up, it stops reading from the client. The
-memory an in-flight request costs is therefore the two buffers, and the
-worst case is that times CONNECTION_LIMIT.
+memory an ordinary in-flight request costs is therefore the two buffers.
+Startup checks their product with the ordinary capacity
+(`CONNECTION_LIMIT - PAYMENT_CONNECTION_LIMIT`) against
+`RELAY_MEMORY_LIMIT`; with the defaults this is 352 * 512 KiB = 176 MiB.
+
+The process currently uses native `fd_set`s in both the MHD and curl
+scheduler integration. `LimitNOFILE` therefore cannot safely raise
+concurrency beyond descriptor 1023. The default 384-connection budget
+charges two descriptors per request and reserves 256 for non-request and
+transient use. Moving to a poll/epoll integration is required before
+raising that ceiling.
An upload is still bounded, by MAX_REQUEST_SIZE (1 MiB by default),
because accepting one is a policy decision rather than a memory
diff --git a/debian/etc/paivana/paivana.conf b/debian/etc/paivana/paivana.conf
@@ -17,6 +17,25 @@ SERVE = systemd
# if left empty. Only used if "SERVE" is 'tcp'.
# BIND_TO =
+# Total accepted connections. The current select()-based event loop can poll
+# descriptors 0..1023 only. At two sockets per active request, 384 consumes
+# 768 descriptor slots and leaves 256 for listeners, curl pools, logs,
+# resolver/scheduler work and transient overlap. Paivana refuses a larger
+# value instead of failing intermittently under load.
+# CONNECTION_LIMIT = 384
+
+# Of the total above, 32 request slots are reserved for payment-redemption
+# POSTs. A merchant lookup may suspend each for up to five seconds, so 32
+# absorbs a normal payment burst without permitting unbounded backend work;
+# the default leaves 352 ordinary proxy slots.
+# PAYMENT_CONNECTION_LIMIT = 32
+
+# MHD sees only the Unix-domain peer in this packaged systemd deployment, not
+# the Internet client. A nonzero per-address limit would therefore throttle
+# every visitor as one client. Apply any optional per-client limit at the
+# trusted nginx/Apache edge, where the real address is known.
+PER_IP_CONNECTION_LIMIT = 0
+
# Largest request body accepted; anything above it is answered 413.
# A policy decision, not a memory one -- bodies are streamed, so this
# does not bound what paivana holds.
@@ -25,12 +44,20 @@ SERVE = systemd
# Bytes of a body held in memory at once while it is relayed, per
# direction per request. Throughput knobs: larger means fewer
# suspend/resume round trips on a fast link and more memory per request
-# in flight, the worst case being these times CONNECTION_LIMIT. There
+# in flight, the validated worst case being their sum times the ordinary
+# capacity (CONNECTION_LIMIT minus PAYMENT_CONNECTION_LIMIT). There
# is deliberately no ceiling on the size of a *response*; bound that at
# the origin if you want one.
# REQUEST_BUFFER_MAX = 262144
# RESPONSE_BUFFER_MAX = 262144
+# Aggregate accounting ceiling for the two relay rings across the 352
+# ordinary request slots. With both 256 KiB defaults the calculated maximum
+# is 176 MiB; the 256 MiB ceiling leaves headroom for templates, rendered
+# responses, merchant JSON, curl and allocator overhead. Startup fails when
+# changed ring/connection values exceed this budget.
+# RELAY_MEMORY_LIMIT = 268435456
+
# How long the origin has to produce response headers before we answer
# 504. Not a bound on the whole request: a large download legitimately
# runs for longer than any useful ceiling, and after the headers have
@@ -43,6 +70,12 @@ SERVE = systemd
# client on a slow link is not mistaken for a slow origin.
# UPSTREAM_STALL_TIMEOUT = 60 s
+# On SIGTERM, stop accepting immediately and give requests whose URI has
+# already been parsed up to 60 seconds to finish. The systemd unit allows 75
+# seconds total, reserving 15 seconds for forced cancellation and cleanup.
+# Zero restores immediate shutdown.
+# SHUTDOWN_GRACE_PERIOD = 60 s
+
# MERCHANT_ACCESS_TOKEN and SECRET live in the file below rather than
# here, because this file is world-readable and those two are not
# things every local account should be able to read. Keep it that way.
diff --git a/debian/examples/nginx-paivana b/debian/examples/nginx-paivana
@@ -8,6 +8,16 @@ map $remote_addr $paivana_forwarded_elem {
default "for=unknown";
}
+# Optional edge-side concurrency control. Paivana cannot enforce a useful
+# per-client-address limit through its Unix socket, because every connection
+# has the same local peer. If your client population is not concentrated
+# behind large NATs, uncomment this zone and the `limit_conn' below. Twenty is
+# an example operational policy, not a Paivana resource formula: lower values
+# reject abusive concurrency sooner but can also reject legitimate users who
+# share one public address. If another proxy is in front, configure nginx's
+# trusted real-IP handling before keying this zone on $binary_remote_addr.
+# limit_conn_zone $binary_remote_addr zone=paivana_clients:10m;
+
server {
listen 80;
listen [::]:80;
@@ -15,6 +25,7 @@ server {
# server_name example.com
location / {
+ # limit_conn paivana_clients 20;
proxy_pass http://unix:/run/paivana/httpd/paivana-http.sock;
proxy_redirect off;
proxy_set_header Host $host;
diff --git a/debian/paivana-httpd.service b/debian/paivana-httpd.service
@@ -31,18 +31,24 @@ RestartMaxDelaySec=300s
# unparseable TRUSTED_PROXIES only fills the journal.
RestartPreventExitStatus=6 9
-# Recycle hourly. This is only tolerable because `SECRET' is now
-# mandatory (paivana-httpd exits 6 without it) and the package
-# generates one: with a per-start random key, every restart would
-# invalidate every access cookie, so an hourly restart meant a customer
-# paying at 10:59 was shown the paywall again at 11:01.
+# Recycle hourly to bound process-lifetime library/cache growth. The 3600 s
+# value is operational hygiene rather than a request deadline. Paivana first
+# quiesces its copy of the socket and drains accepted requests; the socket unit
+# continues queueing new connections for the replacement process. SECRET is
+# mandatory and stable, so the restart does not invalidate paid access.
RuntimeMaxSec=3600s
-# The default CONNECTION_LIMIT is 512, and an active request may consume
-# both an accepted client socket and an outbound origin/merchant socket.
-# Leave room for listeners, resolver activity and libcurl's idle pool too;
-# systemd's common 1024-descriptor soft default is not sufficient.
-LimitNOFILE=4096
+# Paivana's current GNUnet/Taler event loop uses select(), whose fd_set has
+# 1024 entries on this platform. Raising this value would let libraries open
+# descriptors Paivana cannot poll. The default CONNECTION_LIMIT=384 budgets
+# two descriptors per request and leaves the remaining 256 for listeners,
+# scheduler/library state, resolver activity and transient overlap.
+LimitNOFILE=1024
+
+# SHUTDOWN_GRACE_PERIOD defaults to 60 s. The extra 15 s lets Paivana cancel
+# remaining curl work, stop MHD and release templates before systemd sends
+# SIGKILL; keep this value above the configured application grace period.
+TimeoutStopSec=75s
# -f: we are served over a Unix socket by nginx/Apache (see the
# shipped site configs), so the client address has to come from the
# forwarding headers -- a Unix peer has no address of its own, and
@@ -59,9 +65,11 @@ StandardError=journal
# useful while debugging, but turns the production journal into an access log
# for every request Paivana proxies. Keep INFO for Paivana's lifecycle and
# recovery messages while limiting that library source file to actionable
-# severities. GNUNET_FORCE_LOG takes precedence over ExecStart's -L INFO; the
-# final rule deliberately restores INFO for every other source file.
-Environment="GNUNET_FORCE_LOG=;curl.c;;;WARNING/;;;;INFO"
+# severities. The merchant client also emits one generic warning for every
+# failed order poll; Paivana's sampled warning has timing, concurrency and fd
+# context, so suppress the duplicate library warning in production. The final
+# rule deliberately restores INFO for every other source file.
+Environment="GNUNET_FORCE_LOG=;curl.c;;;WARNING/;merchant_api_get-private-orders-ORDER_ID.c;;;ERROR/;;;;INFO"
# Hardening. paivana-httpd needs a listening socket handed to it,
# outbound TCP to the merchant backend and the origin, and read access
diff --git a/src/backend/paivana-httpd.c b/src/backend/paivana-httpd.c
@@ -73,6 +73,33 @@ bool PH_have_trusted_proxies;
*/
#define PH_DEFAULT_REQUEST_BUFFER_MAX (256 * 1024)
+/**
+ * File descriptors deliberately kept outside the client-connection budget.
+ *
+ * The current GNUnet/Taler event-loop integration uses native `fd_set`s, so
+ * descriptors numbered #FD_SETSIZE or higher cannot be serviced even when
+ * RLIMIT_NOFILE is larger. 256 descriptors leave one quarter of the usual
+ * 1024-entry table for listen sockets, the scheduler, logs, resolver work,
+ * libcurl's reusable connections and short-lived overlap while sockets are
+ * being replaced. This is deliberately a safety allowance rather than a
+ * claim that those users always consume exactly 256 descriptors.
+ */
+#define PH_DESCRIPTOR_RESERVE 256U
+
+/**
+ * Worst-case sockets charged to one active client connection: the accepted
+ * client socket and one simultaneous origin or merchant socket.
+ */
+#define PH_DESCRIPTORS_PER_CONNECTION 2U
+
+/**
+ * Largest connection budget that fits the select-based descriptor table
+ * after #PH_DESCRIPTOR_RESERVE has been removed.
+ */
+#define PH_MAX_CONNECTION_LIMIT \
+ ((FD_SETSIZE - PH_DESCRIPTOR_RESERVE) \
+ / PH_DESCRIPTORS_PER_CONNECTION)
+
unsigned long long PH_request_buffer_max = PH_DEFAULT_REQUEST_BUFFER_MAX;
unsigned long long PH_response_buffer_max = 256 * 1024;
@@ -83,10 +110,33 @@ struct GNUNET_TIME_Relative PH_upstream_timeout;
struct GNUNET_TIME_Relative PH_upstream_stall_timeout;
-unsigned int PH_connection_limit = 512;
+unsigned int PH_connection_limit = PH_MAX_CONNECTION_LIMIT;
+
+/**
+ * Thirty-two payment requests are enough to cover a burst while bounding a
+ * malicious client's ability to pin suspended MHD connections for the
+ * five-second merchant long poll. The remaining 352 slots in the default
+ * 384-connection budget stay available for ordinary proxy traffic.
+ */
+unsigned int PH_payment_connection_limit = 32;
unsigned int PH_per_ip_connection_limit = 32;
+/**
+ * Aggregate budget for the two streaming rings of ordinary requests.
+ * 256 MiB admits the defaults (352 * (256 KiB + 256 KiB) = 176 MiB) while
+ * leaving memory for templates, cached responses, merchant replies, libcurl
+ * and allocator overhead. It is a validation budget, not a pre-allocation.
+ */
+unsigned long long PH_relay_memory_limit = 256ULL * 1024 * 1024;
+
+/**
+ * Time accepted requests may finish after SIGTERM. Sixty seconds matches
+ * the two upstream progress timeouts: a healthy request gets a useful chance
+ * to complete, while systemd can still enforce a finite stop deadline.
+ */
+struct GNUNET_TIME_Relative PH_shutdown_grace_period;
+
int PH_global_ret;
int PH_global_cookie;
@@ -113,47 +163,103 @@ static struct GNUNET_CURL_RescheduleContext *merchant_ctx_rc;
*/
static struct GNUNET_CURL_RescheduleContext *proxy_ctx_rc;
+/**
+ * Wall-clock start of graceful shutdown, used to enforce
+ * #PH_shutdown_grace_period.
+ */
+static struct GNUNET_TIME_Absolute shutdown_started;
+
+/**
+ * Poll task waiting for active MHD requests to drain.
+ */
+static struct GNUNET_SCHEDULER_Task *shutdown_poll_task;
+
+/**
+ * Guards the final cleanup against repeated shutdown signals or callbacks.
+ */
+static bool shutdown_cleanup_done;
+
/* *************** General / main code *************** */
/**
- * Warn when the process cannot plausibly sustain its configured client
- * connection limit. A request may hold one inbound socket and one
- * outbound socket (to the origin or merchant) at the same time, while
- * listeners, the scheduler and libcurl's idle connection caches need
- * additional descriptors. Running out during curl_connect() is
- * especially confusing: the merchant API reports it as status zero,
- * just like every other transport failure.
+ * Validate the descriptor and aggregate relay-memory budgets.
+ *
+ * These are startup errors rather than warnings. A configuration that can
+ * allocate an fd which `select()` cannot represent fails intermittently and
+ * most visibly in merchant payment checks, where the client API reports the
+ * transport failure as HTTP status zero.
+ *
+ * @return true if the configured budgets are safe
*/
-static void
-check_file_descriptor_limit (void)
+static bool
+check_resource_limits (void)
{
+ const unsigned int ordinary_limit
+ = PH_connection_limit - PH_payment_connection_limit;
+ const unsigned long long ring_bytes
+ = PH_request_buffer_max + PH_response_buffer_max;
+
+ if (PH_connection_limit > PH_MAX_CONNECTION_LIMIT)
+ {
+ GNUNET_log_config_invalid (
+ GNUNET_ERROR_TYPE_ERROR,
+ "paivana",
+ "CONNECTION_LIMIT",
+ "exceeds the select()-safe maximum: (FD_SETSIZE 1024 - 256"
+ " reserved descriptors) / 2 descriptors per connection = 384");
+ return false;
+ }
+ if (ring_bytes > PH_relay_memory_limit / ordinary_limit)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "REQUEST_BUFFER_MAX (%llu) + RESPONSE_BUFFER_MAX (%llu),"
+ " multiplied by the %u ordinary request slots, exceeds"
+ " RELAY_MEMORY_LIMIT (%llu bytes)\n",
+ PH_request_buffer_max,
+ PH_response_buffer_max,
+ ordinary_limit,
+ PH_relay_memory_limit);
+ return false;
+ }
#if HAVE_SYS_RESOURCE_H
struct rlimit lim;
- unsigned long long recommended;
+ const unsigned long long required
+ = (unsigned long long) PH_DESCRIPTORS_PER_CONNECTION
+ * PH_connection_limit
+ + PH_DESCRIPTOR_RESERVE;
if (0 != getrlimit (RLIMIT_NOFILE,
&lim))
{
- GNUNET_log_strerror (GNUNET_ERROR_TYPE_WARNING,
+ GNUNET_log_strerror (GNUNET_ERROR_TYPE_ERROR,
"getrlimit");
- return;
+ return false;
}
- /* Two descriptors per accepted connection, plus conservative room
- for listeners, logs, resolver activity and reusable idle sockets. */
- recommended = 2ULL * PH_connection_limit + 64ULL;
if ( (RLIM_INFINITY != lim.rlim_cur) &&
- ((unsigned long long) lim.rlim_cur < recommended) )
- GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
- "Open-file soft limit %llu is below the recommended %llu"
- " for CONNECTION_LIMIT=%u; descriptor exhaustion may"
- " cause immediate merchant transport failures. Lower"
- " CONNECTION_LIMIT or raise LimitNOFILE/`ulimit -n'\n",
+ ((unsigned long long) lim.rlim_cur < required) )
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Open-file soft limit %llu is below the required %llu"
+ " for CONNECTION_LIMIT=%u (two descriptors per connection"
+ " plus a 256-descriptor safety reserve)\n",
(unsigned long long) lim.rlim_cur,
- recommended,
+ required,
PH_connection_limit);
+ return false;
+ }
#endif
+ GNUNET_log (GNUNET_ERROR_TYPE_INFO,
+ "Resource budgets: %u connections (%u ordinary, %u payment),"
+ " %llu/%llu relay bytes, %u descriptor slots reserved\n",
+ PH_connection_limit,
+ ordinary_limit,
+ PH_payment_connection_limit,
+ ring_bytes * ordinary_limit,
+ PH_relay_memory_limit,
+ PH_DESCRIPTOR_RESERVE);
+ return true;
}
@@ -277,20 +383,19 @@ load_trusted_proxies (const struct GNUNET_CONFIGURATION_Handle *c,
/**
- * Task run on shutdown
- *
- * @param cls closure
+ * Final cleanup after graceful draining has completed or reached its deadline.
*/
static void
-do_shutdown (void *cls)
+finish_shutdown (void)
{
- (void) cls;
+ if (shutdown_cleanup_done)
+ return;
+ shutdown_cleanup_done = true;
GNUNET_log (GNUNET_ERROR_TYPE_INFO,
- "Shutting down...\n");
- TALER_MHD_daemons_halt ();
+ "Finishing shutdown\n");
+ PAIVANA_HTTPD_daemons_destroy ();
PAIVANA_HTTPD_payment_shutdown ();
PAIVANA_HTTPD_reverse_shutdown ();
- TALER_MHD_daemons_destroy ();
PAIVANA_HTTPD_unload_templates ();
TALER_TEMPLATING_done ();
GNUNET_free (PH_target_server_base_url);
@@ -328,6 +433,87 @@ do_shutdown (void *cls)
/**
+ * Check whether accepted requests have drained.
+ *
+ * A 100 ms poll interval bounds shutdown-completion latency without placing a
+ * callback on every request-completion path. At the 60-second default this
+ * is at most 600 cheap counter reads and does not touch the request list.
+ *
+ * @param cls unused
+ */
+static void
+poll_shutdown_drain (void *cls)
+{
+ struct GNUNET_TIME_Relative elapsed;
+ unsigned int active;
+
+ (void) cls;
+ shutdown_poll_task = NULL;
+ active = PAIVANA_HTTPD_active_requests ();
+ elapsed = GNUNET_TIME_absolute_get_duration (shutdown_started);
+ if (0 == active)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_INFO,
+ "Graceful shutdown drained all requests in %s\n",
+ GNUNET_STRINGS_relative_time_to_string (elapsed,
+ true));
+ finish_shutdown ();
+ return;
+ }
+ if (GNUNET_TIME_relative_cmp (elapsed,
+ >=,
+ PH_shutdown_grace_period))
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
+ "Graceful shutdown deadline reached with %u active"
+ " request%s; terminating them now\n",
+ active,
+ (1 == active) ? "" : "s");
+ finish_shutdown ();
+ return;
+ }
+ shutdown_poll_task = GNUNET_SCHEDULER_add_delayed (
+ GNUNET_TIME_relative_multiply (GNUNET_TIME_UNIT_MILLISECONDS,
+ 100),
+ &poll_shutdown_drain,
+ NULL);
+}
+
+
+/**
+ * Task run on shutdown: close listeners, then allow accepted requests to
+ * complete before final cleanup.
+ *
+ * @param cls closure
+ */
+static void
+do_shutdown (void *cls)
+{
+ unsigned int active;
+
+ (void) cls;
+ shutdown_started = GNUNET_TIME_absolute_get ();
+ active = PAIVANA_HTTPD_begin_drain ();
+ if ( (0 == active) ||
+ (0 == PH_shutdown_grace_period.rel_value_us) )
+ {
+ finish_shutdown ();
+ return;
+ }
+ GNUNET_log (GNUNET_ERROR_TYPE_INFO,
+ "Shutdown quiesced listeners; allowing %u active request%s"
+ " up to %s to finish\n",
+ active,
+ (1 == active) ? "" : "s",
+ GNUNET_STRINGS_relative_time_to_string (
+ PH_shutdown_grace_period,
+ true));
+ shutdown_poll_task = GNUNET_SCHEDULER_add_now (&poll_shutdown_drain,
+ NULL);
+}
+
+
+/**
* Remove trailing slashes from the web URL @a url, in place.
*
* Our configuration syntax prefers base URLs to be written with a
@@ -406,6 +592,9 @@ run (void *cls,
PH_upstream_stall_timeout
= GNUNET_TIME_relative_multiply (GNUNET_TIME_UNIT_SECONDS,
60);
+ PH_shutdown_grace_period
+ = GNUNET_TIME_relative_multiply (GNUNET_TIME_UNIT_SECONDS,
+ 60);
GNUNET_SCHEDULER_add_shutdown (&do_shutdown,
NULL);
if ( (0 == PH_request_buffer_max) ||
@@ -446,6 +635,12 @@ run (void *cls,
GNUNET_SCHEDULER_shutdown ();
return;
}
+ if (! PAIVANA_HTTPD_init_template_languages ())
+ {
+ PH_global_ret = EXIT_NOTINSTALLED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
}
if (! PAIVANA_HTTPD_reverse_init ())
{
@@ -504,6 +699,25 @@ run (void *cls,
if (GNUNET_OK ==
GNUNET_CONFIGURATION_get_value_number (c,
"paivana",
+ "PAYMENT_CONNECTION_LIMIT",
+ &v))
+ {
+ if ( (0 == v) ||
+ (v > UINT_MAX) )
+ {
+ GNUNET_log_config_invalid (GNUNET_ERROR_TYPE_ERROR,
+ "paivana",
+ "PAYMENT_CONNECTION_LIMIT",
+ "must be between 1 and UINT_MAX");
+ PH_global_ret = EXIT_NOTCONFIGURED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
+ PH_payment_connection_limit = (unsigned int) v;
+ }
+ if (GNUNET_OK ==
+ GNUNET_CONFIGURATION_get_value_number (c,
+ "paivana",
"PER_IP_CONNECTION_LIMIT",
&v))
{
@@ -580,8 +794,36 @@ run (void *cls,
PH_request_buffer_max);
PH_max_request_size = PH_request_buffer_max;
}
+ if (GNUNET_OK ==
+ GNUNET_CONFIGURATION_get_value_number (c,
+ "paivana",
+ "RELAY_MEMORY_LIMIT",
+ &v))
+ {
+ if (0 == v)
+ {
+ GNUNET_log_config_invalid (GNUNET_ERROR_TYPE_ERROR,
+ "paivana",
+ "RELAY_MEMORY_LIMIT",
+ "must be at least one byte");
+ PH_global_ret = EXIT_NOTCONFIGURED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
+ PH_relay_memory_limit = v;
+ }
+ }
+ if (PH_payment_connection_limit >= PH_connection_limit)
+ {
+ GNUNET_log_config_invalid (GNUNET_ERROR_TYPE_ERROR,
+ "paivana",
+ "PAYMENT_CONNECTION_LIMIT",
+ "must be smaller than CONNECTION_LIMIT so"
+ " ordinary requests retain capacity");
+ PH_global_ret = EXIT_NOTCONFIGURED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
}
- check_file_descriptor_limit ();
{
struct GNUNET_TIME_Relative st;
@@ -621,6 +863,18 @@ run (void *cls,
}
PH_upstream_stall_timeout = st;
}
+ if (GNUNET_OK ==
+ GNUNET_CONFIGURATION_get_value_time (c,
+ "paivana",
+ "SHUTDOWN_GRACE_PERIOD",
+ &st))
+ PH_shutdown_grace_period = st;
+ }
+ if (! check_resource_limits ())
+ {
+ PH_global_ret = EXIT_NOTCONFIGURED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
}
{
unsigned int n4;
diff --git a/src/backend/paivana-httpd.h b/src/backend/paivana-httpd.h
@@ -198,6 +198,14 @@ extern const struct GNUNET_CONFIGURATION_Handle *PH_cfg;
extern unsigned int PH_connection_limit;
/**
+ * Number of the total connection slots reserved for requests to redeem a
+ * payment, from `PAYMENT_CONNECTION_LIMIT`; 32 by default. Keeping this
+ * separate prevents a burst of ordinary proxy traffic from consuming every
+ * request slot needed to confirm already-paid orders.
+ */
+extern unsigned int PH_payment_connection_limit;
+
+/**
* Number of concurrent connections we accept from any single client
* address, from `PER_IP_CONNECTION_LIMIT`; 0 disables the check, which
* is MHD's default.
@@ -209,6 +217,20 @@ extern unsigned int PH_connection_limit;
extern unsigned int PH_per_ip_connection_limit;
/**
+ * Maximum aggregate bytes the ordinary-request streaming rings may account
+ * for, from `RELAY_MEMORY_LIMIT`; 256 MiB by default. Startup validates the
+ * product of the two per-request ring limits and the ordinary request slots.
+ */
+extern unsigned long long PH_relay_memory_limit;
+
+/**
+ * Time accepted requests may finish after shutdown begins, from
+ * `SHUTDOWN_GRACE_PERIOD`; 60 seconds by default. Zero requests immediate
+ * shutdown, while the packaged service allows another 15 seconds for cleanup.
+ */
+extern struct GNUNET_TIME_Relative PH_shutdown_grace_period;
+
+/**
* How many bytes of a request body we hold in memory at once while
* relaying it upstream, from `REQUEST_BUFFER_MAX` or the `-u` /
* `--max-upload` command-line option; 256 KiB by default.
diff --git a/src/backend/paivana-httpd_daemon.c b/src/backend/paivana-httpd_daemon.c
@@ -69,6 +69,19 @@ struct RequestContext
* We are past the paywall, forward to client.
*/
bool do_forward;
+
+ /**
+ * Admission class charged to this request. Classification waits until the
+ * access handler has both the decoded path and method; the URI callback
+ * alone cannot distinguish a payment POST from a GET to the same path.
+ */
+ enum
+ {
+ AC_UNDECIDED,
+ AC_ORDINARY,
+ AC_PAYMENT,
+ AC_REJECTED
+ } admission_class;
};
@@ -94,6 +107,145 @@ struct RequestContext
*/
static bool have_daemons;
+/**
+ * MHD daemons retained so shutdown can quiesce only their listen sockets
+ * while the Taler scheduler adapter continues driving accepted requests.
+ */
+static struct MHD_Daemon **mhd_daemons;
+
+/**
+ * Length of #mhd_daemons.
+ */
+static unsigned int mhd_daemons_length;
+
+/**
+ * Number of requests whose URI callback has run and whose completion
+ * callback has not. This is the drain condition; half-written request lines
+ * are connections rather than requests and are closed at final cleanup.
+ */
+static unsigned int active_requests;
+
+/**
+ * Ordinary requests admitted against the non-payment part of the connection
+ * budget.
+ */
+static unsigned int ordinary_requests;
+
+/**
+ * Payment redemption requests admitted against the reserved part of the
+ * connection budget.
+ */
+static unsigned int payment_requests;
+
+/**
+ * True after listeners have been quiesced for graceful shutdown.
+ */
+static bool draining;
+
+
+/**
+ * Queue a controlled-overload response and force the connection closed.
+ *
+ * Payment callers receive the machine-readable Taler error used for local
+ * resource exhaustion. Other requests receive a small static HTML body so
+ * overload itself does not allocate or render a template. `Retry-After: 1'
+ * discourages a hot retry loop without claiming a longer outage.
+ *
+ * @param connection client connection
+ * @param payment true for the payment endpoint
+ * @return MHD result
+ */
+static enum MHD_Result
+reply_overloaded (struct MHD_Connection *connection,
+ bool payment)
+{
+ static const char body[] =
+ "<!doctype html><title>Service unavailable</title>"
+ "<p>Paivana is temporarily at capacity. Please retry.</p>";
+ struct MHD_Response *response;
+ enum MHD_Result ret;
+
+ if (payment)
+ response = TALER_MHD_make_error (
+ TALER_EC_GENERIC_OS_RESOURCE_ALLOCATION_FAILURE,
+ "Paivana payment-check capacity is exhausted");
+ else
+ response = MHD_create_response_from_buffer_static (sizeof (body) - 1,
+ body);
+ if (NULL == response)
+ return MHD_NO;
+ if ( (! payment) &&
+ (MHD_YES !=
+ MHD_add_response_header (response,
+ MHD_HTTP_HEADER_CONTENT_TYPE,
+ "text/html; charset=utf-8")) )
+ goto fail;
+ if ( (MHD_YES !=
+ MHD_add_response_header (response,
+ MHD_HTTP_HEADER_CONNECTION,
+ "close")) ||
+ (MHD_YES !=
+ MHD_add_response_header (response,
+ MHD_HTTP_HEADER_RETRY_AFTER,
+ "1")) )
+ goto fail;
+ ret = MHD_queue_response (connection,
+ MHD_HTTP_SERVICE_UNAVAILABLE,
+ response);
+ MHD_destroy_response (response);
+ return ret;
+fail:
+ MHD_destroy_response (response);
+ return MHD_NO;
+}
+
+
+/**
+ * Decide whether a request may consume one of the process-wide slots.
+ *
+ * MHD's connection limit remains the hard transport ceiling. This second
+ * request-level gate stops ordinary requests at CONNECTION_LIMIT minus
+ * PAYMENT_CONNECTION_LIMIT, leaving the remainder available for payment
+ * POSTs. It cannot reserve against clients that connect but never finish a
+ * request line; CLIENT_CONNECTION_TIMEOUT bounds that separate slowloris
+ * case.
+ *
+ * @param rc request context
+ * @param payment whether this is a payment POST
+ * @return true if admitted
+ */
+static bool
+admit_request (struct RequestContext *rc,
+ bool payment)
+{
+ GNUNET_assert (AC_UNDECIDED == rc->admission_class);
+ if (draining)
+ {
+ rc->admission_class = AC_REJECTED;
+ return false;
+ }
+ if (payment)
+ {
+ if (payment_requests >= PH_payment_connection_limit)
+ {
+ rc->admission_class = AC_REJECTED;
+ return false;
+ }
+ payment_requests++;
+ rc->admission_class = AC_PAYMENT;
+ return true;
+ }
+ if (ordinary_requests >=
+ PH_connection_limit - PH_payment_connection_limit)
+ {
+ rc->admission_class = AC_REJECTED;
+ return false;
+ }
+ ordinary_requests++;
+ rc->admission_class = AC_ORDINARY;
+ return true;
+}
+
/**
* Is the request target something we may decide about and then forward
@@ -191,11 +343,22 @@ create_response (void *cls,
bool ok = false;
struct GNUNET_Buffer buf;
char *website;
+ const bool payment
+ = ( (0 == strcmp (url,
+ "/.well-known/paivana")) &&
+ (0 == strcasecmp (meth,
+ MHD_HTTP_METHOD_POST)) );
(void) cls;
memset (&buf,
0,
sizeof (buf));
+ if (AC_UNDECIDED == rc->admission_class)
+ (void) admit_request (rc,
+ payment);
+ if (AC_REJECTED == rc->admission_class)
+ return reply_overloaded (con,
+ payment);
if (! canonical_request_target (rc->url,
url))
{
@@ -208,10 +371,7 @@ create_response (void *cls,
" and empty path segments");
}
if ( (! rc->is_paivana) &&
- (0 == strcmp (url,
- "/.well-known/paivana")) &&
- (0 == strcasecmp (meth,
- MHD_HTTP_METHOD_POST)) )
+ payment )
{
rc->is_paivana = true;
}
@@ -398,6 +558,22 @@ mhd_completed_cb (void *cls,
PAIVANA_HTTPD_reverse_cleanup (rc->hr);
if (NULL != rc->hp)
PAIVANA_HTTPD_payment_destroy (rc->hp);
+ switch (rc->admission_class)
+ {
+ case AC_ORDINARY:
+ GNUNET_assert (ordinary_requests > 0);
+ ordinary_requests--;
+ break;
+ case AC_PAYMENT:
+ GNUNET_assert (payment_requests > 0);
+ payment_requests--;
+ break;
+ case AC_UNDECIDED:
+ case AC_REJECTED:
+ break;
+ }
+ GNUNET_assert (active_requests > 0);
+ active_requests--;
GNUNET_free (rc->url);
GNUNET_free (rc);
*con_cls = NULL;
@@ -430,6 +606,7 @@ mhd_log_callback (void *cls,
(void) cls;
rc = GNUNET_new (struct RequestContext);
+ active_requests++;
rc->connection = connection;
rc->url = GNUNET_strdup (url);
rc->do_forward = (1 == PH_no_check);
@@ -528,10 +705,64 @@ start_daemon (int lsock,
return;
}
have_daemons = true;
+ GNUNET_array_append (mhd_daemons,
+ mhd_daemons_length,
+ mhd);
TALER_MHD_daemon_start (mhd);
}
+unsigned int
+PAIVANA_HTTPD_begin_drain (void)
+{
+ if (! draining)
+ {
+ draining = true;
+ for (unsigned int i = 0; i < mhd_daemons_length; i++)
+ {
+ MHD_socket fd;
+
+ fd = MHD_quiesce_daemon (mhd_daemons[i]);
+ if (MHD_INVALID_SOCKET == fd)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
+ "HTTP listen socket was already quiesced\n");
+ continue;
+ }
+ /* TALER_MHD_daemon_start() has a GNUnet select task whose fdset was
+ built before quiescing and therefore still contains @a fd. Triggering
+ cancels that pending task and schedules a fresh MHD_run(), which will
+ rebuild the set without the listener. Closing first leaves select()
+ holding an invalid descriptor and makes the scheduler abort with
+ EBADF. The trigger task cannot run until this callback returns, so
+ the returned socket is no longer referenced when close() follows. */
+ TALER_MHD_daemon_trigger ();
+ GNUNET_break (0 == close (fd));
+ }
+ }
+ return active_requests;
+}
+
+
+unsigned int
+PAIVANA_HTTPD_active_requests (void)
+{
+ return active_requests;
+}
+
+
+void
+PAIVANA_HTTPD_daemons_destroy (void)
+{
+ TALER_MHD_daemons_halt ();
+ TALER_MHD_daemons_destroy ();
+ GNUNET_array_grow (mhd_daemons,
+ mhd_daemons_length,
+ 0);
+ have_daemons = false;
+}
+
+
void
PAIVANA_HTTPD_serve_requests ()
{
diff --git a/src/backend/paivana-httpd_daemon.h b/src/backend/paivana-httpd_daemon.h
@@ -37,5 +37,28 @@
void
PAIVANA_HTTPD_serve_requests (void);
+/**
+ * Stop accepting new connections while continuing to service accepted
+ * requests.
+ *
+ * @return number of active requests left to drain
+ */
+unsigned int
+PAIVANA_HTTPD_begin_drain (void);
+
+/**
+ * Return the number of requests whose completion callback has not run yet.
+ *
+ * @return active request count
+ */
+unsigned int
+PAIVANA_HTTPD_active_requests (void);
+
+/**
+ * Halt scheduler integration and destroy all MHD daemons after draining.
+ */
+void
+PAIVANA_HTTPD_daemons_destroy (void);
+
#endif
diff --git a/src/backend/paivana-httpd_pay.c b/src/backend/paivana-httpd_pay.c
@@ -171,6 +171,110 @@ static struct PayRequest *ph_tail;
*/
static unsigned int merchant_transport_failures;
+/**
+ * Number of merchant order requests currently in #ph_head. Maintaining the
+ * counter with the DLL makes diagnostics O(1) during a mass outage instead of
+ * walking every suspended request from every completion callback.
+ */
+static unsigned int active_merchant_lookups;
+
+/**
+ * Merchant transport diagnostics are useful immediately and then at most once
+ * per minute per failure class. A minute is short enough for an operator to
+ * see a persistent outage in routine monitoring, while reducing 32 concurrent
+ * five-second failures from hundreds of warnings per minute to two.
+ */
+#define MERCHANT_FAILURE_LOG_INTERVAL \
+ GNUNET_TIME_relative_multiply (GNUNET_TIME_UNIT_MINUTES, 1)
+
+/**
+ * Transport failures that need independently sampled explanations.
+ */
+enum MerchantFailureClass
+{
+ MFC_UNUSABLE_REPLY,
+ MFC_TIMEOUT,
+ MFC_EARLY_TRANSPORT,
+ MFC_COUNT
+};
+
+/**
+ * Sampling state for one #MerchantFailureClass.
+ */
+struct MerchantFailureLogState
+{
+ /** Next time a warning may be emitted. */
+ struct GNUNET_TIME_Absolute next_log;
+
+ /** Failures omitted since the previous emitted warning. */
+ unsigned int suppressed;
+
+ /** Whether this class has emitted its first warning. */
+ bool logged;
+};
+
+/**
+ * Per-class warning sampling state.
+ */
+static struct MerchantFailureLogState failure_logs[MFC_COUNT];
+
+
+/**
+ * Decide whether to emit a merchant failure warning now.
+ *
+ * @param fc failure class
+ * @param[out] suppressed number suppressed since the prior warning
+ * @return true if the caller should log
+ */
+static bool
+merchant_failure_should_log (enum MerchantFailureClass fc,
+ unsigned int *suppressed)
+{
+ struct MerchantFailureLogState *fl = &failure_logs[fc];
+ struct GNUNET_TIME_Absolute now = GNUNET_TIME_absolute_get ();
+
+ if ( (! fl->logged) ||
+ GNUNET_TIME_absolute_cmp (now,
+ >=,
+ fl->next_log) )
+ {
+ *suppressed = fl->suppressed;
+ fl->suppressed = 0;
+ fl->logged = true;
+ fl->next_log = GNUNET_TIME_absolute_add (
+ now,
+ MERCHANT_FAILURE_LOG_INTERVAL);
+ return true;
+ }
+ if (UINT_MAX != fl->suppressed)
+ fl->suppressed++;
+ return false;
+}
+
+
+/**
+ * Clear sampling state after the backend returns an HTTP response.
+ *
+ * @return failures suppressed since the most recent emitted warnings
+ */
+static unsigned int
+reset_merchant_failure_logs (void)
+{
+ unsigned int suppressed = 0;
+
+ for (unsigned int i = 0; i < MFC_COUNT; i++)
+ {
+ if (UINT_MAX - suppressed < failure_logs[i].suppressed)
+ suppressed = UINT_MAX;
+ else
+ suppressed += failure_logs[i].suppressed;
+ }
+ memset (failure_logs,
+ 0,
+ sizeof (failure_logs));
+ return suppressed;
+}
+
/**
* Log process descriptor usage while a merchant transport failure is
@@ -259,11 +363,14 @@ PAIVANA_HTTPD_payment_shutdown ()
GNUNET_CONTAINER_DLL_remove (ph_head,
ph_tail,
ph);
+ GNUNET_assert (active_merchant_lookups > 0);
+ active_merchant_lookups--;
MHD_resume_connection (ph->connection);
/* Note: PAIVANA_HTTPD_payment_destroy()
will be called by the owner of 'ph',
no need to do it here! */
}
+ GNUNET_assert (0 == active_merchant_lookups);
}
@@ -427,7 +534,7 @@ order_status_cb (struct PayRequest *ph,
struct GNUNET_TIME_Relative elapsed;
char *elapsed_s;
char *timeout_s;
- unsigned int active_lookups = 0;
+ unsigned int active_lookups;
elapsed = GNUNET_TIME_absolute_get_duration (
ph->merchant_request_started);
@@ -439,14 +546,13 @@ order_status_cb (struct PayRequest *ph,
timeout_s = GNUNET_strdup (
GNUNET_STRINGS_relative_time_to_string (MERCHANT_ORDER_TIMEOUT,
true));
- for (const struct PayRequest *pos = ph_head;
- NULL != pos;
- pos = pos->next)
- active_lookups++;
+ active_lookups = active_merchant_lookups;
ph->co = NULL;
GNUNET_CONTAINER_DLL_remove (ph_head,
ph_tail,
ph);
+ GNUNET_assert (active_merchant_lookups > 0);
+ active_merchant_lookups--;
MHD_resume_connection (ph->connection);
TALER_MHD_daemon_trigger ();
GNUNET_log (GNUNET_ERROR_TYPE_DEBUG,
@@ -457,14 +563,19 @@ order_status_cb (struct PayRequest *ph,
elapsed_s);
if (0 != osr->hr.http_status)
{
+ unsigned int suppressed = reset_merchant_failure_logs ();
+
if (0 != merchant_transport_failures)
GNUNET_log (GNUNET_ERROR_TYPE_INFO,
"Merchant backend at `%s' answered order `%s' after %u"
- " consecutive lookup%s without an HTTP response\n",
+ " consecutive lookup%s without an HTTP response (%u"
+ " repetitive diagnostic%s suppressed)\n",
PH_merchant_base_url,
ph->order_id,
merchant_transport_failures,
- (1 == merchant_transport_failures) ? "" : "s");
+ (1 == merchant_transport_failures) ? "" : "s",
+ suppressed,
+ (1 == suppressed) ? "" : "s");
merchant_transport_failures = 0;
}
switch (osr->hr.http_status)
@@ -601,14 +712,21 @@ order_status_cb (struct PayRequest *ph,
what tells them apart, being NULL only in the former case. */
if (NULL != osr->hr.reply)
{
+ unsigned int suppressed;
+
merchant_transport_failures = 0;
GNUNET_break_op (0);
- GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
- "Merchant backend at `%s' sent an unusable reply for"
- " order `%s' after %s\n",
- PH_merchant_base_url,
- ph->order_id,
- elapsed_s);
+ if (merchant_failure_should_log (MFC_UNUSABLE_REPLY,
+ &suppressed))
+ GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
+ "Merchant backend at `%s' sent an unusable reply for"
+ " order `%s' after %s (%u similar diagnostic%s"
+ " suppressed)\n",
+ PH_merchant_base_url,
+ ph->order_id,
+ elapsed_s,
+ suppressed,
+ (1 == suppressed) ? "" : "s");
ph->response = TALER_MHD_make_error (TALER_EC_PAIVANA_BACKEND_ERROR,
ph->order_id);
ph->response_status = MHD_HTTP_BAD_GATEWAY;
@@ -616,24 +734,33 @@ order_status_cb (struct PayRequest *ph,
}
if (UINT_MAX != merchant_transport_failures)
merchant_transport_failures++;
- log_file_descriptor_usage ();
if (GNUNET_TIME_relative_cmp (elapsed,
>=,
MERCHANT_ORDER_TIMEOUT))
{
- GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
- "Merchant backend at `%s' returned no HTTP response for"
- " order `%s' by the %s deadline (elapsed %s; %u"
- " concurrent merchant lookup%s including this one; %u"
- " consecutive transport failure%s)\n",
- PH_merchant_base_url,
- ph->order_id,
- timeout_s,
- elapsed_s,
- active_lookups,
- (1 == active_lookups) ? "" : "s",
- merchant_transport_failures,
- (1 == merchant_transport_failures) ? "" : "s");
+ unsigned int suppressed;
+
+ if (merchant_failure_should_log (MFC_TIMEOUT,
+ &suppressed))
+ {
+ log_file_descriptor_usage ();
+ GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
+ "Merchant backend at `%s' returned no HTTP response for"
+ " order `%s' by the %s deadline (elapsed %s; %u"
+ " concurrent merchant lookup%s including this one; %u"
+ " consecutive transport failure%s; %u similar"
+ " diagnostic%s suppressed)\n",
+ PH_merchant_base_url,
+ ph->order_id,
+ timeout_s,
+ elapsed_s,
+ active_lookups,
+ (1 == active_lookups) ? "" : "s",
+ merchant_transport_failures,
+ (1 == merchant_transport_failures) ? "" : "s",
+ suppressed,
+ (1 == suppressed) ? "" : "s");
+ }
/* GENERIC_TIMEOUT's hint ("trying again might help") is the one
that is true once our own deadline was actually reached. */
ph->response = TALER_MHD_make_error (TALER_EC_GENERIC_TIMEOUT,
@@ -642,26 +769,36 @@ order_status_cb (struct PayRequest *ph,
}
else
{
+ unsigned int suppressed;
+
/* The merchant API does not expose CURLcode, so DNS failure,
connection refusal, TLS failure and a dead reused connection
are indistinguishable here. What they have in common is that
they failed before our timeout. Calling that a timeout hid the
most useful fact from both the operator and the client. */
- GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
- "Merchant backend at `%s' returned no HTTP response for"
- " order `%s' after %s, before the %s deadline; this is"
- " an early transport failure (for example DNS, TCP, TLS"
- " or a stale reused connection), not a Paivana timeout"
- " (%u concurrent merchant lookup%s including this one;"
- " %u consecutive transport failure%s)\n",
- PH_merchant_base_url,
- ph->order_id,
- elapsed_s,
- timeout_s,
- active_lookups,
- (1 == active_lookups) ? "" : "s",
- merchant_transport_failures,
- (1 == merchant_transport_failures) ? "" : "s");
+ if (merchant_failure_should_log (MFC_EARLY_TRANSPORT,
+ &suppressed))
+ {
+ log_file_descriptor_usage ();
+ GNUNET_log (GNUNET_ERROR_TYPE_WARNING,
+ "Merchant backend at `%s' returned no HTTP response for"
+ " order `%s' after %s, before the %s deadline; this is"
+ " an early transport failure (for example DNS, TCP, TLS"
+ " or a stale reused connection), not a Paivana timeout"
+ " (%u concurrent merchant lookup%s including this one;"
+ " %u consecutive transport failure%s; %u similar"
+ " diagnostic%s suppressed)\n",
+ PH_merchant_base_url,
+ ph->order_id,
+ elapsed_s,
+ timeout_s,
+ active_lookups,
+ (1 == active_lookups) ? "" : "s",
+ merchant_transport_failures,
+ (1 == merchant_transport_failures) ? "" : "s",
+ suppressed,
+ (1 == suppressed) ? "" : "s");
+ }
ph->response = TALER_MHD_make_error (TALER_EC_PAIVANA_BACKEND_REFUSED,
ph->order_id);
ph->response_status = MHD_HTTP_BAD_GATEWAY;
@@ -803,6 +940,7 @@ PAIVANA_HTTPD_payment_handle (struct PayRequest *ph,
GNUNET_CONTAINER_DLL_insert (ph_head,
ph_tail,
ph);
+ active_merchant_lookups++;
MHD_suspend_connection (ph->connection);
{
enum TALER_ErrorCode ec;
@@ -822,6 +960,8 @@ PAIVANA_HTTPD_payment_handle (struct PayRequest *ph,
GNUNET_CONTAINER_DLL_remove (ph_head,
ph_tail,
ph);
+ GNUNET_assert (active_merchant_lookups > 0);
+ active_merchant_lookups--;
MHD_resume_connection (ph->connection);
TALER_MERCHANT_get_private_order_cancel (ph->co);
ph->co = NULL;
@@ -845,6 +985,8 @@ PAIVANA_HTTPD_payment_destroy (struct PayRequest *ph)
GNUNET_CONTAINER_DLL_remove (ph_head,
ph_tail,
ph);
+ GNUNET_assert (active_merchant_lookups > 0);
+ active_merchant_lookups--;
ph->co = NULL;
}
if (NULL != ph->response)
diff --git a/src/backend/paivana-httpd_templates.c b/src/backend/paivana-httpd_templates.c
@@ -45,18 +45,52 @@ struct Template;
/**
- * Maximum number of rendered paywall responses we cache per template.
+ * Maximum number of rendered paywall responses cached process-wide.
*
- * The key is derived from the client-supplied Accept-Language and
- * Accept-Encoding headers, so without a bound an attacker could send
- * unlimited distinct header values and grow the cache without limit
- * (memory-exhaustion DoS on the cheap pre-payment path). The key is
- * normalised to what we can actually serve — see cache_key_language()
- * and TALER_MHD_can_compress() — which is what keeps every request
- * after the first few a *hit*; this cap only backstops that. On
- * reaching it we evict the least recently used entry.
+ * A cache key is one merchant template, one installed language and one of two
+ * compression outcomes. 256 entries cover the common case of 128 merchant
+ * templates in one language and both encodings, while replacing the old
+ * per-template cap whose aggregate could grow to 16,384 responses. The exact
+ * byte cost depends on rendered choices, so the entry cap bounds it to 256
+ * times the largest rendered response. On reaching it we evict the global
+ * least-recently-used entry.
*/
-#define MAX_RESPONSE_CACHE_ENTRIES 128
+#define MAX_RESPONSE_CACHE_ENTRIES 256
+
+/**
+ * Maximum entries accepted from `GET /private/templates'. 1024 bounds the
+ * temporary ID list received from a shared merchant that may contain many
+ * non-Paivana templates; only #MAX_PAIVANA_TEMPLATES survive startup.
+ */
+#define MAX_DISCOVERED_TEMPLATES 1024
+
+/**
+ * Maximum active Paivana templates. Every request may test their regular
+ * expressions and every template retains contract choices, so 128 bounds
+ * both per-request CPU and long-lived memory.
+ */
+#define MAX_PAIVANA_TEMPLATES 128
+
+/**
+ * Maximum simultaneous merchant template-detail fetches during startup.
+ * Eight keeps startup parallel without consuming the 32-descriptor payment
+ * reserve or producing a large burst against the merchant backend.
+ */
+#define MAX_TEMPLATE_FETCHES 8
+
+/**
+ * Largest compact JSON representation accepted for one template contract.
+ * One MiB is ample for prices and localized summaries while bounding the
+ * contract tree retained for rendering and rejecting accidental bulk data.
+ */
+#define MAX_TEMPLATE_CONTRACT_SIZE (1024 * 1024)
+
+/**
+ * Maximum installed paywall languages recognized for cache normalization.
+ * Thirty-two is far above a practical site translation set and bounds both
+ * startup metadata and the language-selection loop on every cache lookup.
+ */
+#define MAX_PAYWALL_LANGUAGES 32
/**
* How long we give the merchant backend to answer our template
@@ -72,7 +106,7 @@ struct Template;
* exists, so clients connect successfully and then hang forever with
* nothing accepting them. Generous enough for the round-trips a load
* takes (GET /private/templates, then one GET per template, issued in
- * parallel).
+ * batches of at most eight).
*/
#define TEMPLATE_LOAD_TIMEOUT \
GNUNET_TIME_relative_multiply (GNUNET_TIME_UNIT_MINUTES, 2)
@@ -94,6 +128,9 @@ struct ResponseCacheEntry
*/
struct ResponseCacheEntry *prev;
+ /** Merchant template this rendered response belongs to. */
+ struct Template *template;
+
/**
* Language of the response.
*/
@@ -169,19 +206,15 @@ struct Template
struct TALER_MERCHANT_GetPrivateTemplateHandle *gt;
/**
- * Number of entries in the template cache starting at @e rce_head.
+ * Startup fetch state. Detail requests are started in bounded batches
+ * rather than all at once.
*/
- unsigned int rce_length;
-
- /**
- * Kept in a DLL.
- */
- struct ResponseCacheEntry *rce_head;
-
- /**
- * Kept in a DLL.
- */
- struct ResponseCacheEntry *rce_tail;
+ enum
+ {
+ TLS_WAITING,
+ TLS_ACTIVE,
+ TLS_DONE
+ } load_state;
};
@@ -196,6 +229,33 @@ static struct Template *t_head;
*/
static struct Template *t_tail;
+/** Head of the process-wide rendered-response LRU. */
+static struct ResponseCacheEntry *rce_head;
+
+/** Tail of the process-wide rendered-response LRU. */
+static struct ResponseCacheEntry *rce_tail;
+
+/** Number of entries in the process-wide rendered-response LRU. */
+static unsigned int rce_length;
+
+/** Installed language tags for the `paywall' Mustache template. */
+static char *paywall_languages[MAX_PAYWALL_LANGUAGES];
+
+/** Number of entries in #paywall_languages. */
+static unsigned int paywall_languages_length;
+
+/** Template IDs waiting for a detail request to start. */
+static unsigned int pending_template_fetches;
+
+/** Template detail requests currently in flight. */
+static unsigned int active_template_fetches;
+
+/** Paivana templates retained after their contracts were inspected. */
+static unsigned int loaded_paivana_templates;
+
+/** Next waiting template, making bounded fetch dispatch O(T). */
+static struct Template *next_template_fetch;
+
/**
* Handle to get all the templates.
*/
@@ -421,45 +481,94 @@ js_string_literal (const char *s)
/**
- * Number of installed `paywall.*.must' templates, counted once by
- * count_paywall_templates(). UINT_MAX until then.
- */
-static unsigned int paywall_template_count = UINT_MAX;
-
-
-/**
- * Count the installed paywall templates.
+ * Record the language of an installed `paywall.$LANG.must' template.
*
- * TALER_TEMPLATING_build() picks among them by matching the client's
- * `Accept-Language' against the language tag in each file name, and
- * does not report which one it chose. We do not need to know: if there
- * is only one to choose from — the shipped configuration, which ships
- * `paywall.en.must' and nothing else — then every `Accept-Language'
- * whatsoever produces the same body, and the header can be dropped from
- * the cache key entirely.
+ * The scan intentionally uses the same directory and filename grammar as
+ * TALER_TEMPLATING_init(). Directory traversal order is stable across the
+ * two consecutive scans, preserving the templating library's first-template
+ * fallback when no language matches.
*
* @param cls unused
* @param filename file found in the template directory
- * @return #GNUNET_OK to continue the scan
+ * @return #GNUNET_OK to continue, #GNUNET_SYSERR on the language cap
*/
static enum GNUNET_GenericReturnValue
-count_paywall_template (void *cls,
- const char *filename)
+collect_paywall_language (void *cls,
+ const char *filename)
{
const char *base;
+ const char *lang;
+ const char *end;
(void) cls;
base = strrchr (filename,
'/');
base = (NULL == base) ? filename : base + 1;
- if (0 == strncmp (base,
+ if (0 != strncmp (base,
"paywall.",
strlen ("paywall.")))
- paywall_template_count++;
+ return GNUNET_OK;
+ lang = base + strlen ("paywall.");
+ end = strchr (lang,
+ '.');
+ if ( (lang == end) ||
+ (NULL == end) ||
+ (0 != strcmp (end,
+ ".must")) )
+ return GNUNET_OK;
+ if (paywall_languages_length >= MAX_PAYWALL_LANGUAGES)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "More than %u paywall languages are installed; refusing"
+ " startup because language selection and cache variation"
+ " would exceed the documented bound\n",
+ (unsigned int) MAX_PAYWALL_LANGUAGES);
+ return GNUNET_SYSERR;
+ }
+ paywall_languages[paywall_languages_length++]
+ = GNUNET_strndup (lang,
+ end - lang);
return GNUNET_OK;
}
+bool
+PAIVANA_HTTPD_init_template_languages (void)
+{
+ char *dir;
+ char *tdir;
+ int ret;
+
+ GNUNET_assert (0 == paywall_languages_length);
+ dir = GNUNET_OS_installation_get_path (PAIVANA_project_data (),
+ GNUNET_OS_IPK_DATADIR);
+ GNUNET_asprintf (&tdir,
+ "%stemplates",
+ dir);
+ GNUNET_free (dir);
+ ret = GNUNET_DISK_directory_scan (tdir,
+ &collect_paywall_language,
+ NULL);
+ if ( (0 > ret) ||
+ (0 == paywall_languages_length) )
+ {
+ if (0 == paywall_languages_length)
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "No paywall.$LANG.must template is installed in `%s'\n",
+ tdir);
+ GNUNET_free (tdir);
+ return false;
+ }
+ GNUNET_log (GNUNET_ERROR_TYPE_INFO,
+ "%u paywall language%s installed in `%s'\n",
+ paywall_languages_length,
+ (1 == paywall_languages_length) ? "" : "s",
+ tdir);
+ GNUNET_free (tdir);
+ return true;
+}
+
+
/**
* The `Content-Security-Policy' for the paywall page, built once from
* #PH_merchant_base_url. NULL until first needed.
@@ -543,14 +652,9 @@ get_paywall_csp (void)
/**
* Return the language component of the render cache key for @a conn.
*
- * With a single installed paywall template there is nothing to
- * negotiate, so the key does not depend on `Accept-Language' at all and
- * the whole header — an unauthenticated client's free choice, on the
- * pre-payment path — stops being able to force a fresh Mustache render
- * per distinct value. With several installed we cannot tell which one
- * the templating library picked (it has no accessor for that; the
- * proper fix belongs there), so we fall back to keying on the raw
- * header and accept the amplification for that configuration.
+ * Reproduce TALER_TEMPLATING_build()'s language choice from the installed
+ * language list. The cache is therefore keyed by one of at most 32 actual
+ * variants instead of by an unbounded client-controlled header string.
*
* @param conn connection to derive the key component for
* @return the key component, or NULL if `Accept-Language' does not
@@ -559,37 +663,29 @@ get_paywall_csp (void)
static const char *
cache_key_language (struct MHD_Connection *conn)
{
- if (UINT_MAX == paywall_template_count)
+ const char *pattern;
+ const char *best = NULL;
+ double best_q = 0.0;
+
+ GNUNET_assert (0 != paywall_languages_length);
+ pattern = MHD_lookup_connection_value (conn,
+ MHD_HEADER_KIND,
+ MHD_HTTP_HEADER_ACCEPT_LANGUAGE);
+ if (NULL == pattern)
+ pattern = "en";
+ for (unsigned int i = 0; i < paywall_languages_length; i++)
{
- char *dir;
- char *tdir;
-
- paywall_template_count = 0;
- dir = GNUNET_OS_installation_get_path (PAIVANA_project_data (),
- GNUNET_OS_IPK_DATADIR);
- GNUNET_asprintf (&tdir,
- "%stemplates",
- dir);
- GNUNET_free (dir);
- if (0 > GNUNET_DISK_directory_scan (tdir,
- &count_paywall_template,
- NULL))
- {
- /* Cannot tell; assume the worst and keep the old key. */
- GNUNET_break (0);
- paywall_template_count = 2;
- }
- GNUNET_log (GNUNET_ERROR_TYPE_INFO,
- "%u paywall template(s) installed in `%s'\n",
- paywall_template_count,
- tdir);
- GNUNET_free (tdir);
+ double q = TALER_pattern_matches (pattern,
+ paywall_languages[i]);
+
+ if (q <= best_q)
+ continue;
+ best_q = q;
+ best = paywall_languages[i];
}
- if (2 > paywall_template_count)
- return NULL;
- return MHD_lookup_connection_value (conn,
- MHD_HEADER_KIND,
- MHD_HTTP_HEADER_ACCEPT_LANGUAGE);
+ /* This is the same first-loaded fallback as lookup_template() in the
+ templating library. */
+ return (NULL != best) ? best : paywall_languages[0];
}
@@ -617,22 +713,23 @@ load_paywall (struct MHD_Connection *conn,
deflate = (TALER_MHD_CT_DEFLATE ==
TALER_MHD_can_compress (conn,
TALER_MHD_CT_DEFLATE));
- for (struct ResponseCacheEntry *pos = t->rce_head;
+ for (struct ResponseCacheEntry *pos = rce_head;
NULL != pos;
pos = pos->next)
{
- if ( (eq (lang,
+ if ( (t == pos->template) &&
+ (eq (lang,
pos->lang)) &&
(deflate == pos->deflate) )
{
- if (t->rce_head != pos)
+ if (rce_head != pos)
{
/* Hit, move pos to head of DLL for proper LRU eviction */
- GNUNET_CONTAINER_DLL_remove (t->rce_head,
- t->rce_tail,
+ GNUNET_CONTAINER_DLL_remove (rce_head,
+ rce_tail,
pos);
- GNUNET_CONTAINER_DLL_insert (t->rce_head,
- t->rce_tail,
+ GNUNET_CONTAINER_DLL_insert (rce_head,
+ rce_tail,
pos);
}
return MHD_queue_response (conn,
@@ -778,30 +875,31 @@ load_paywall (struct MHD_Connection *conn,
/* '>=', not '>': the insert below is what takes us to the cap, so
testing '>' left the steady state one entry above it. */
- while (t->rce_length >= MAX_RESPONSE_CACHE_ENTRIES)
+ while (rce_length >= MAX_RESPONSE_CACHE_ENTRIES)
{
/* Evict the least recently used entry; the hit path above
promotes to the head, so the tail is the coldest. */
- struct ResponseCacheEntry *old = t->rce_tail;
+ struct ResponseCacheEntry *old = rce_tail;
- GNUNET_CONTAINER_DLL_remove (t->rce_head,
- t->rce_tail,
+ GNUNET_CONTAINER_DLL_remove (rce_head,
+ rce_tail,
old);
- GNUNET_assert (t->rce_length > 0);
- t->rce_length--;
+ GNUNET_assert (rce_length > 0);
+ rce_length--;
MHD_destroy_response (old->paywall);
GNUNET_free (old->lang);
GNUNET_free (old);
}
rce = GNUNET_new (struct ResponseCacheEntry);
+ rce->template = t;
if (NULL != lang)
rce->lang = GNUNET_strdup (lang);
rce->deflate = deflate;
rce->paywall = reply;
rce->http_status = http_status;
- t->rce_length++;
- GNUNET_CONTAINER_DLL_insert (t->rce_head,
- t->rce_tail,
+ rce_length++;
+ GNUNET_CONTAINER_DLL_insert (rce_head,
+ rce_tail,
rce);
return MHD_queue_response (conn,
rce->http_status,
@@ -992,7 +1090,6 @@ static void
drop_template (struct Template *t)
{
GNUNET_assert (NULL == t->gt);
- GNUNET_assert (NULL == t->rce_head);
GNUNET_CONTAINER_DLL_remove (t_head,
t_tail,
t);
@@ -1001,6 +1098,76 @@ drop_template (struct Template *t)
}
+static void start_template_fetches (void);
+
+
+/** Compare template pointers by ID for qsort(). */
+static int
+compare_template_ids (const void *a,
+ const void *b)
+{
+ const struct Template *const *ta = a;
+ const struct Template *const *tb = b;
+
+ return strcmp ((*ta)->template_id,
+ (*tb)->template_id);
+}
+
+
+/**
+ * Sort the retained templates once, after collection and filtering.
+ *
+ * Where expressions overlap, list order chooses the quoted price. Sorting by
+ * ID makes that decision deterministic. Collecting into an array and using
+ * qsort() is O(T log T), unlike the previous sorted DLL insertion's O(T^2).
+ */
+static void
+sort_templates (void)
+{
+ struct Template **templates;
+ unsigned int i = 0;
+
+ if (2 > loaded_paivana_templates)
+ return;
+ templates = GNUNET_malloc (loaded_paivana_templates
+ * sizeof (*templates));
+ for (struct Template *t = t_head; NULL != t; t = t->next)
+ templates[i++] = t;
+ GNUNET_assert (i == loaded_paivana_templates);
+ qsort (templates,
+ loaded_paivana_templates,
+ sizeof (*templates),
+ &compare_template_ids);
+ t_head = NULL;
+ t_tail = NULL;
+ for (i = 0; i < loaded_paivana_templates; i++)
+ {
+ templates[i]->next = NULL;
+ templates[i]->prev = NULL;
+ GNUNET_CONTAINER_DLL_insert_tail (t_head,
+ t_tail,
+ templates[i]);
+ }
+ GNUNET_free (templates);
+}
+
+
+/** Finish template startup once no detail request is queued or active. */
+static void
+finish_template_loading (void)
+{
+ if ( (0 != pending_template_fetches) ||
+ (0 != active_template_fetches) )
+ return;
+ sort_templates ();
+ GNUNET_log (GNUNET_ERROR_TYPE_INFO,
+ "%u Paivana template%s loaded, starting to serve requests\n",
+ loaded_paivana_templates,
+ (1 == loaded_paivana_templates) ? "" : "s");
+ templates_ready ();
+}
+
+
/**
* Callback for a GET /private/templates/$TEMPLATE_ID request.
*
@@ -1012,38 +1179,68 @@ setup_template (
struct Template *t,
const struct TALER_MERCHANT_GetPrivateTemplateResponse *tgr)
{
+ GNUNET_assert (TLS_ACTIVE == t->load_state);
+ GNUNET_assert (active_template_fetches > 0);
+ active_template_fetches--;
t->gt = NULL;
+ t->load_state = TLS_DONE;
switch (tgr->hr.http_status)
{
case MHD_HTTP_OK:
- if (! is_paivana_template (tgr->details.ok.template_contract))
{
- /* One merchant instance serves every kind of template, and
- `template_type' is what says which of them are ours -- as the
- manual promises. Without this check a fixed-order template is
- handed to parse_template(), which fails it for want of
- `choices' and reports "Invalid template X at field choices":
- an error about a template that is simply none of our business,
- and (since a parse failure is fatal) one that keeps paivana
- from starting at all next to a perfectly good paivana
- template. */
- GNUNET_log (GNUNET_ERROR_TYPE_INFO,
- "Ignoring template %s: not a paivana template\n",
- t->template_id);
- drop_template (t);
+ const json_t *contract = tgr->details.ok.template_contract;
+ size_t contract_size;
+
+ if (! is_paivana_template (contract))
+ {
+ /* A shared merchant serves other template types too; they count only
+ against the broad discovery bound and are discarded here. */
+ GNUNET_log (GNUNET_ERROR_TYPE_INFO,
+ "Ignoring template %s: not a paivana template\n",
+ t->template_id);
+ drop_template (t);
+ break;
+ }
+ if (loaded_paivana_templates >= MAX_PAIVANA_TEMPLATES)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Merchant offers more than %u Paivana templates;"
+ " refusing startup to bound regex work and retained"
+ " contract memory\n",
+ (unsigned int) MAX_PAIVANA_TEMPLATES);
+ PH_global_ret = EXIT_NOTCONFIGURED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
+ contract_size = json_dumpb (contract,
+ NULL,
+ 0,
+ JSON_COMPACT);
+ if (contract_size > MAX_TEMPLATE_CONTRACT_SIZE)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Template %s has a %llu-byte contract, exceeding the"
+ " %u-byte startup and retained-memory limit\n",
+ t->template_id,
+ (unsigned long long) contract_size,
+ (unsigned int) MAX_TEMPLATE_CONTRACT_SIZE);
+ PH_global_ret = EXIT_NOTCONFIGURED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
+ if (! parse_template (t,
+ contract))
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Failed to parse template %s, refusing to start\n",
+ t->template_id);
+ PH_global_ret = EXIT_FAILURE;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
+ loaded_paivana_templates++;
break;
}
- if (! parse_template (t,
- tgr->details.ok.template_contract))
- {
- GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
- "Failed to parse template %s, refusing to start\n",
- t->template_id);
- PH_global_ret = EXIT_FAILURE;
- GNUNET_SCHEDULER_shutdown ();
- return;
- }
- break;
default:
GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
"Failed to load template %s from backend"
@@ -1054,13 +1251,58 @@ setup_template (
GNUNET_SCHEDULER_shutdown ();
return;
}
- for (struct Template *p = t_head; NULL != p; p = p->next)
- if (NULL != p->gt)
+ start_template_fetches ();
+ finish_template_loading ();
+}
+
+
+/** Start waiting detail requests until the eight-request window is full. */
+static void
+start_template_fetches (void)
+{
+ while ( (active_template_fetches < MAX_TEMPLATE_FETCHES) &&
+ (0 != pending_template_fetches) )
+ {
+ struct Template *t = next_template_fetch;
+ enum TALER_ErrorCode ec;
+
+ GNUNET_assert (NULL != t);
+ next_template_fetch = t->next;
+ GNUNET_assert (TLS_WAITING == t->load_state);
+ t->gt = TALER_MERCHANT_get_private_template_create (
+ PH_merchant_ctx,
+ PH_merchant_base_url,
+ t->template_id);
+ if (NULL == t->gt)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Could not allocate merchant request for template %s\n",
+ t->template_id);
+ PH_global_ret = EXIT_FAILURE;
+ GNUNET_SCHEDULER_shutdown ();
return;
- /* all templates done, continue with main logic */
- GNUNET_log (GNUNET_ERROR_TYPE_INFO,
- "Templates loaded, starting to serve requests\n");
- templates_ready ();
+ }
+ t->load_state = TLS_ACTIVE;
+ pending_template_fetches--;
+ active_template_fetches++;
+ ec = TALER_MERCHANT_get_private_template_start (t->gt,
+ &setup_template,
+ t);
+ if (TALER_EC_NONE != ec)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Could not start merchant request for template %s: %d\n",
+ t->template_id,
+ (int) ec);
+ TALER_MERCHANT_get_private_template_cancel (t->gt);
+ t->gt = NULL;
+ t->load_state = TLS_DONE;
+ active_template_fetches--;
+ PH_global_ret = EXIT_FAILURE;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
+ }
}
@@ -1077,6 +1319,30 @@ check_templates (
{
(void) cls;
gpt = NULL;
+ /* The merchant client library enforces the same 1024-entry ceiling before
+ filling details.ok. Preserve Paivana's own operator-facing diagnosis by
+ recognizing that parse-failure shape in the raw reply; without it the
+ callback reports only status zero/error 10 and the configured bound is
+ invisible. */
+ if ( (0 == tgr->hr.http_status) &&
+ (NULL != tgr->hr.reply) )
+ {
+ const json_t *templates = json_object_get (tgr->hr.reply,
+ "templates");
+
+ if ( json_is_array (templates) &&
+ (json_array_size (templates) > MAX_DISCOVERED_TEMPLATES) )
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Merchant returned %llu template IDs, exceeding the"
+ " %u-entry startup bound\n",
+ (unsigned long long) json_array_size (templates),
+ (unsigned int) MAX_DISCOVERED_TEMPLATES);
+ PH_global_ret = EXIT_NOTCONFIGURED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
+ }
switch (tgr->hr.http_status)
{
case MHD_HTTP_OK:
@@ -1102,45 +1368,35 @@ check_templates (
templates_ready ();
return;
}
+ if (tgr->details.ok.templates_length > MAX_DISCOVERED_TEMPLATES)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Merchant returned %u template IDs, exceeding the %u-entry"
+ " startup bound\n",
+ tgr->details.ok.templates_length,
+ (unsigned int) MAX_DISCOVERED_TEMPLATES);
+ PH_global_ret = EXIT_NOTCONFIGURED;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
for (unsigned int i = 0; i<tgr->details.ok.templates_length; i++)
{
const struct TALER_MERCHANT_GetPrivateTemplatesTemplateEntry *te
= &tgr->details.ok.templates[i];
struct Template *t;
- struct Template *before;
t = GNUNET_new (struct Template);
t->template_id = GNUNET_strdup (te->template_id);
t->max_pickup_delay = GNUNET_TIME_UNIT_FOREVER_REL;
- t->gt = TALER_MERCHANT_get_private_template_create (PH_merchant_ctx,
- PH_merchant_base_url,
- t->template_id);
- /* Insert sorted by template ID. Where two expressions both match a
- URL, PAIVANA_HTTPD_search_templates() quotes the price of
- whichever template it reaches first, so this list's order is a
- pricing decision. The backend's array arrives in whatever order
- the database returned it -- the SELECT behind
- GET /private/templates has no ORDER BY -- so taking it as given
- (in either direction) makes that decision depend on nothing an
- operator can see or set, and it can differ between two restarts
- with no configuration change. Sorting by ID is an arbitrary
- rule, but it is a rule, and it is one an operator can act on. */
- before = t_head;
- while ( (NULL != before) &&
- (0 > strcmp (before->template_id,
- t->template_id)) )
- before = before->next;
- GNUNET_CONTAINER_DLL_insert_before (t_head,
- t_tail,
- before,
- t);
- GNUNET_assert (
- TALER_EC_NONE ==
- TALER_MERCHANT_get_private_template_start (t->gt,
- &setup_template,
- t));
+ t->load_state = TLS_WAITING;
+ GNUNET_CONTAINER_DLL_insert_tail (t_head,
+ t_tail,
+ t);
}
+ pending_template_fetches = tgr->details.ok.templates_length;
+ next_template_fetch = t_head;
+ start_template_fetches ();
}
@@ -1154,12 +1410,31 @@ PAIVANA_HTTPD_load_templates ()
NULL);
gpt = TALER_MERCHANT_get_private_templates_create (PH_merchant_ctx,
PH_merchant_base_url);
- GNUNET_assert (NULL != gpt);
- GNUNET_assert (
- TALER_EC_NONE ==
- TALER_MERCHANT_get_private_templates_start (gpt,
- &check_templates,
- NULL));
+ if (NULL == gpt)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Could not allocate merchant template-list request\n");
+ PH_global_ret = EXIT_FAILURE;
+ GNUNET_SCHEDULER_shutdown ();
+ return;
+ }
+ {
+ enum TALER_ErrorCode ec;
+
+ ec = TALER_MERCHANT_get_private_templates_start (gpt,
+ &check_templates,
+ NULL);
+ if (TALER_EC_NONE != ec)
+ {
+ GNUNET_log (GNUNET_ERROR_TYPE_ERROR,
+ "Could not start merchant template-list request: %d\n",
+ (int) ec);
+ TALER_MERCHANT_get_private_templates_cancel (gpt);
+ gpt = NULL;
+ PH_global_ret = EXIT_FAILURE;
+ GNUNET_SCHEDULER_shutdown ();
+ }
+ }
}
@@ -1348,23 +1623,23 @@ PAIVANA_HTTPD_unload_templates ()
GNUNET_SCHEDULER_cancel (load_timeout_task);
load_timeout_task = NULL;
}
+ while (NULL != rce_head)
+ {
+ struct ResponseCacheEntry *rce = rce_head;
+
+ GNUNET_assert (rce_length > 0);
+ rce_length--;
+ GNUNET_CONTAINER_DLL_remove (rce_head,
+ rce_tail,
+ rce);
+ MHD_destroy_response (rce->paywall);
+ GNUNET_free (rce->lang);
+ GNUNET_free (rce);
+ }
while (NULL != t_head)
{
struct Template *t = t_head;
- while (NULL != t->rce_head)
- {
- struct ResponseCacheEntry *rce = t->rce_head;
-
- GNUNET_assert (t->rce_length > 0);
- t->rce_length--;
- GNUNET_CONTAINER_DLL_remove (t->rce_head,
- t->rce_tail,
- rce);
- MHD_destroy_response (rce->paywall);
- GNUNET_free (rce->lang);
- GNUNET_free (rce);
- }
GNUNET_CONTAINER_DLL_remove (t_head,
t_tail,
t);
@@ -1380,6 +1655,13 @@ PAIVANA_HTTPD_unload_templates ()
json_decref (t->choices);
GNUNET_free (t);
}
+ for (unsigned int i = 0; i < paywall_languages_length; i++)
+ GNUNET_free (paywall_languages[i]);
+ paywall_languages_length = 0;
+ pending_template_fetches = 0;
+ active_template_fetches = 0;
+ loaded_paivana_templates = 0;
+ next_template_fetch = NULL;
if (NULL != gpt)
{
TALER_MERCHANT_get_private_templates_cancel (gpt);
diff --git a/src/backend/paivana-httpd_templates.h b/src/backend/paivana-httpd_templates.h
@@ -27,10 +27,21 @@
#define PAIVANA_HTTPD_TEMPLATES_H
#include <microhttpd.h>
+#include <stdbool.h>
#include <gnunet/gnunet_util_lib.h>
/**
+ * Discover the installed paywall languages used to normalize response-cache
+ * keys. Must be called after TALER_TEMPLATING_init().
+ *
+ * @return true if at least one and at most the supported maximum are present
+ */
+bool
+PAIVANA_HTTPD_init_template_languages (void);
+
+
+/**
* Load the templates from the merchant backend. Calls
* PAIVANA_HTTPD_serve_requests() upon completion if successful,
* otherwise may initiate shutdown.
diff --git a/src/tests/README b/src/tests/README
@@ -780,8 +780,8 @@ merchant_api_get-private-templates{,-TEMPLATE_ID}.c parse them, with
the contract test_paywall.sh POSTs to a real backend -- and nothing
else. That is the whole of what a real backend would do here: the
template is fetched once, the page is rendered locally from it, and
-the rendered MHD_Response is cached per (language, encoding) in
-load_paywall(), so from the first measured request onwards a live
+the rendered MHD_Response is cached process-wide per (template, language,
+encoding) in load_paywall(), so from the first measured request onwards a live
merchant is exactly as idle as the stub. This benchmark never buys
anything, so no code path that can tell the two apart is reached.
The stub does check the bearer token, since paivana building that
diff --git a/src/tests/merchant_stub.rs b/src/tests/merchant_stub.rs
@@ -30,7 +30,7 @@
//
// Nothing after that exchange touches the backend again. The paywall
// page is rendered locally from the template, and the rendered
-// MHD_Response is then cached per (language, encoding) in
+// MHD_Response is then cached process-wide by (template, language, encoding) in
// load_paywall(), so across the measurement window this process is
// idle and would be equally idle if it were a real merchant. That is
// what makes a stub honest here where test_paywall.sh needs the real
diff --git a/src/tests/meson.build b/src/tests/meson.build
@@ -238,6 +238,16 @@ test(
timeout: 30,
)
+# Startup work and memory bounds for merchant templates and installed paywall
+# languages, using the same lightweight merchant stub as the transport test.
+test(
+ 'template_limits',
+ files('test_template_limits.sh'),
+ env: test_env,
+ depends: [paivana_httpd_exe],
+ timeout: 30,
+)
+
# What the reverse proxy costs: N curl clients at a fixed-size static
# page, first straight at upstream_rs and then through paivana in front
# of it. Registered with benchmark() rather than test() deliberately --
diff --git a/src/tests/payment_backend_stub.py b/src/tests/payment_backend_stub.py
@@ -2,7 +2,9 @@
"""Small merchant HTTP stub for payment-backend failure tests."""
import json
+import os
import sys
+import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlsplit
@@ -10,6 +12,17 @@ from urllib.parse import urlsplit
TOKEN = "secret-token:stub"
TEMPLATE_ID = "premium"
+TEMPLATE_COUNT = int(os.environ.get("PAIVANA_STUB_TEMPLATE_COUNT", "1"))
+CONTRACT_PADDING = int(os.environ.get("PAIVANA_STUB_CONTRACT_PADDING", "0"))
+DETAIL_DELAY = float(os.environ.get("PAIVANA_STUB_DETAIL_DELAY", "0"))
+detail_lock = threading.Lock()
+active_details = 0
+
+
+def template_ids():
+ if TEMPLATE_COUNT == 1:
+ return [TEMPLATE_ID]
+ return [f"{TEMPLATE_ID}-{i:04d}" for i in range(TEMPLATE_COUNT)]
class ReusableServer(ThreadingHTTPServer):
@@ -43,29 +56,41 @@ class Handler(BaseHTTPRequestHandler):
{
"templates": [
{
- "template_id": TEMPLATE_ID,
+ "template_id": template_id,
"template_description": "Paywalled content",
}
+ for template_id in template_ids()
]
},
)
return
- if path == f"/private/templates/{TEMPLATE_ID}":
- self.reply(
- 200,
- {
- "template_description": "Paywalled content",
- "template_contract": {
- "template_type": "paivana",
- "summary": "Access to the article",
- "website_regex": ".*",
- "max_pickup_duration": {"d_us": 3600000000},
- "choices": [
- {"amount": "TESTKUDOS:1", "description": "One article"}
- ],
+ if path.removeprefix("/private/templates/") in template_ids():
+ global active_details
+ with detail_lock:
+ active_details += 1
+ current = active_details
+ print(f"template detail concurrency {current}", flush=True)
+ try:
+ if DETAIL_DELAY:
+ time.sleep(DETAIL_DELAY)
+ self.reply(
+ 200,
+ {
+ "template_description": "Paywalled content",
+ "template_contract": {
+ "template_type": "paivana",
+ "summary": "Access to the article" + "x" * CONTRACT_PADDING,
+ "website_regex": ".*",
+ "max_pickup_duration": {"d_us": 3600000000},
+ "choices": [
+ {"amount": "TESTKUDOS:1", "description": "One article"}
+ ],
+ },
},
- },
- )
+ )
+ finally:
+ with detail_lock:
+ active_details -= 1
return
if path == "/private/orders/timeout-order":
# Longer than Paivana's five-second order deadline. The
diff --git a/src/tests/test_payment_backend_failure.sh b/src/tests/test_payment_backend_failure.sh
@@ -106,7 +106,7 @@ if ! port_is_free "$MERCHANT_PORT" || ! port_is_free "$PAIVANA_PORT"; then
fi
mkdir -p "$SCRATCH/configd" "$SCRATCH/prefix/share/paivana/templates"
-cp "$SRCDIR/../frontend/paywall.en.must.j2" \
+cp "$BUILDDIR/../frontend/paywall.en.must" \
"$SCRATCH/prefix/share/paivana/templates/"
export PAIVANA_BASE_CONFIG="$SCRATCH/configd"
export PAIVANA_PREFIX="$SCRATCH/prefix/"
@@ -120,6 +120,12 @@ MERCHANT_ACCESS_TOKEN = secret-token:stub
SECRET = payment-backend-failure-test
SERVE = tcp
PORT = $PAIVANA_PORT
+BIND_TO = 127.0.0.1
+# Small values make the payment-reservation boundary testable without a large
+# load: two ordinary and two payment slots still satisfy the same accounting
+# relationships as the 352+32 production defaults.
+CONNECTION_LIMIT = 4
+PAYMENT_CONNECTION_LIMIT = 2
EOF
start_stub
@@ -166,4 +172,44 @@ PY
grep -q "by the 5 s deadline" "$SCRATCH/paivana.log" \
|| fail "deadline diagnostic missing from log"
+# Two long polls fill the payment budget. A third redemption must receive an
+# immediate, machine-readable overload response while the first two remain
+# suspended, and repeated timeout diagnostics in the same minute are sampled.
+timeout_pids=()
+for n in 1 2; do
+ curl -sS -o "$SCRATCH/timeout-$n.json" -w '%{http_code}' \
+ -H 'Content-Type: application/json' -X POST \
+ "http://127.0.0.1:$PAIVANA_PORT/.well-known/paivana" \
+ -d "$(redemption_body timeout-order)" \
+ >"$SCRATCH/timeout-$n.status" &
+ timeout_pids+=("$!")
+done
+sleep 0.3
+status="$(curl -sS --max-time 2 -D "$SCRATCH/payment-overload.headers" \
+ -o "$SCRATCH/response.json" -w '%{http_code}' \
+ -H 'Content-Type: application/json' -X POST \
+ "http://127.0.0.1:$PAIVANA_PORT/.well-known/paivana" \
+ -d "$(redemption_body third-order)")" || \
+ fail "over-capacity payment request failed"
+[ "$status" = "503" ] || fail "third payment returned HTTP $status, want 503"
+python3 - "$SCRATCH/response.json" <<'PY' || fail "payment overload JSON is wrong"
+import json
+import sys
+with open(sys.argv[1], encoding="utf-8") as f:
+ body = json.load(f)
+raise SystemExit(0 if body.get("code") == 77 else 1)
+PY
+grep -qi '^Retry-After: 1' "$SCRATCH/payment-overload.headers" || \
+ fail "payment overload response lacks Retry-After: 1"
+grep -qi '^Connection: close' "$SCRATCH/payment-overload.headers" || \
+ fail "payment overload response lacks Connection: close"
+wait "${timeout_pids[0]}" || fail "first concurrent timeout request failed"
+wait "${timeout_pids[1]}" || fail "second concurrent timeout request failed"
+[ "$(cat "$SCRATCH/timeout-1.status")" = 504 ] || \
+ fail "first admitted payment did not complete with 504"
+[ "$(cat "$SCRATCH/timeout-2.status")" = 504 ] || \
+ fail "second admitted payment did not complete with 504"
+[ "$(grep -c 'by the 5 s deadline' "$SCRATCH/paivana.log")" = 1 ] || \
+ fail "timeout diagnostic was not sampled to one warning per minute"
+
echo "payment backend failure diagnostics: OK"
diff --git a/src/tests/test_reverse_proxy.sh b/src/tests/test_reverse_proxy.sh
@@ -271,6 +271,27 @@ function start_paivana() {
fi
}
+# Start the TCP daemon with extra configuration lines. BIND_TO keeps this on
+# one MHD daemon so small connection-limit tests exercise the process-wide
+# arithmetic rather than dividing their tiny budget across IPv4 and IPv6.
+function start_paivana_with_config() {
+ local dest="$1" extra="$2" level="${3:-WARNING}"
+ PAIVANA_DEST="$dest"
+ local cfg="$SCRATCH/paivana-custom.conf"
+ sed -e "s|@DEST@|$dest|g" -e "s|@PORT@|$PAIVANA_PORT|g" \
+ "$SRCDIR/test_reverse_proxy.conf.in" > "$cfg"
+ printf 'BIND_TO = 127.0.0.1\n%s\n' "$extra" >> "$cfg"
+ local log="$LOGDIR/paivana-custom.log"
+ ( exec "$PAIVANA_HTTPD" -c "$cfg" -n -L "$level" ) >"$log" 2>&1 &
+ PAIVANA_PID=$!
+ if ! wait_for_port 127.0.0.1 "$PAIVANA_PORT" "$PAIVANA_PID";
+ then
+ echo "FAIL: custom paivana-httpd did not start" >&2
+ tail -n 40 "$log" >&2
+ exit 1
+ fi
+}
+
function wait_for_unix_socket() {
# Block until the given path exists and is a socket (max ~5s), or
# until the process that was to create it is gone. Same reasoning
@@ -2095,13 +2116,19 @@ function test_forwarded_unix_rfc7239() {
# every refusal an acceptance.
function paivana_with_config_line() {
local line="$1"
+ local nofile="${2:-}"
local cfg="$SCRATCH/startup.conf"
sed -e "s|@DEST@|http://127.0.0.1:$MHD_PORT|g" \
-e "s|@PORT@|$PAIVANA_PORT|g" \
"$SRCDIR/test_reverse_proxy.conf.in" > "$cfg"
printf '%s\n' "$line" >> "$cfg"
local log="$LOGDIR/startup.log"
- ( exec "$PAIVANA_HTTPD" -c "$cfg" -n -f -L ERROR ) >"$log" 2>&1 &
+ if [ -n "$nofile" ];
+ then
+ ( ulimit -n "$nofile"; exec "$PAIVANA_HTTPD" -c "$cfg" -n -f -L ERROR ) >"$log" 2>&1 &
+ else
+ ( exec "$PAIVANA_HTTPD" -c "$cfg" -n -f -L ERROR ) >"$log" 2>&1 &
+ fi
local pid=$!
if wait_for_port 127.0.0.1 "$PAIVANA_PORT" "$pid";
then
@@ -2115,6 +2142,131 @@ function paivana_with_config_line() {
echo "refused"
}
+function test_resource_budget_config() {
+ stop_paivana
+ local r
+
+ # The default ring calculation is exactly
+ # 352 * (262144 + 262144) = 184549376 bytes. Testing one byte on
+ # either side pins the overflow-safe aggregate comparison.
+ for bad in \
+ 'CONNECTION_LIMIT = 385' \
+ 'PAYMENT_CONNECTION_LIMIT = 0' \
+ 'CONNECTION_LIMIT = 32' \
+ 'RELAY_MEMORY_LIMIT = 184549375'
+ do
+ msg "startup refused: $bad"
+ r="$(paivana_with_config_line "$bad")"
+ [ "$r" = "refused" ] || \
+ fail "paivana started with an unsafe resource budget ($bad)"
+ ok
+ done
+
+ for good in \
+ 'CONNECTION_LIMIT = 384' \
+ 'PAYMENT_CONNECTION_LIMIT = 32' \
+ 'RELAY_MEMORY_LIMIT = 184549376' \
+ 'SHUTDOWN_GRACE_PERIOD = 0 s'
+ do
+ msg "startup accepted: $good"
+ r="$(paivana_with_config_line "$good")"
+ [ "$r" = "started" ] || \
+ fail "paivana refused a safe resource budget ($good); log:"$'\n'"$(cat "$LOGDIR/startup.log")"
+ ok
+ done
+
+ # CONNECTION_LIMIT=10 charges 20 descriptors plus the documented
+ # 256-descriptor reserve. The soft limit is a hard startup boundary,
+ # independently of the much larger select() ceiling.
+ msg "startup refused one descriptor below its calculated requirement"
+ r="$(paivana_with_config_line \
+ $'CONNECTION_LIMIT = 10\nPAYMENT_CONNECTION_LIMIT = 2' 275)"
+ [ "$r" = "refused" ] || fail "paivana ignored RLIMIT_NOFILE=275"
+ ok
+
+ msg "startup accepted at its exact calculated descriptor requirement"
+ r="$(paivana_with_config_line \
+ $'CONNECTION_LIMIT = 10\nPAYMENT_CONNECTION_LIMIT = 2' 276)"
+ [ "$r" = "started" ] || \
+ fail "paivana refused RLIMIT_NOFILE=276; log:"$'\n'"$(cat "$LOGDIR/startup.log")"
+ ok
+
+ start_paivana "http://127.0.0.1:$MHD_PORT"
+}
+
+function test_controlled_overload() {
+ stop_paivana
+ start_paivana_with_config "http://127.0.0.1:$MHD_PORT" \
+ $'CONNECTION_LIMIT = 4\nPAYMENT_CONNECTION_LIMIT = 1'
+
+ # Three ordinary requests are the entire non-payment budget. They remain
+ # suspended in Paivana while the single-threaded test origin answers its
+ # slow requests, leaving the fourth MHD connection for an immediate
+ # overload response and then for the reserved payment request.
+ local pids=()
+ for i in 1 2 3;
+ do
+ curl -sS -o /dev/null -w '%{http_code}' \
+ "$(PAIVANA_URL /slow/1000)" >"$SCRATCH/ordinary-$i.status" &
+ pids+=("$!")
+ done
+ sleep 0.3
+
+ msg "ordinary capacity returns a controlled 503"
+ local status
+ status="$(curl -sS --max-time 2 -D "$SCRATCH/overload.headers" \
+ -o "$SCRATCH/overload.body" -w '%{http_code}' \
+ "$(PAIVANA_URL /small)")" || fail "ordinary overload request failed"
+ [ "$status" = 503 ] || fail "ordinary overload status=$status, want 503"
+ grep -qi '^Retry-After: 1' "$SCRATCH/overload.headers" || \
+ fail "ordinary overload response lacks Retry-After: 1"
+ grep -qi '^Connection: close' "$SCRATCH/overload.headers" || \
+ fail "ordinary overload response lacks Connection: close"
+ ok
+
+ msg "the reserved slot still admits the payment endpoint"
+ status="$(curl -sS --max-time 2 -o "$SCRATCH/payment-disabled.body" \
+ -w '%{http_code}' -H 'Content-Type: application/json' -X POST \
+ -d '{}' "$(PAIVANA_URL /.well-known/paivana)")" || \
+ fail "reserved payment request failed"
+ [ "$status" = 501 ] || \
+ fail "reserved payment status=$status, want the -n response 501"
+ ok
+
+ for i in 0 1 2;
+ do
+ wait "${pids[$i]}" || fail "ordinary request $((i + 1)) failed"
+ [ "$(cat "$SCRATCH/ordinary-$((i + 1)).status")" = 200 ] || \
+ fail "ordinary request $((i + 1)) did not complete"
+ done
+ stop_paivana
+ start_paivana "http://127.0.0.1:$MHD_PORT"
+}
+
+function test_graceful_shutdown() {
+ stop_paivana
+ start_paivana_with_config "http://127.0.0.1:$MHD_PORT" \
+ $'CONNECTION_LIMIT = 10\nPAYMENT_CONNECTION_LIMIT = 2\nSHUTDOWN_GRACE_PERIOD = 2 s' INFO
+
+ curl -sS -o /dev/null -w '%{http_code}' \
+ "$(PAIVANA_URL /slow/1000)" >"$SCRATCH/drain.status" &
+ local curl_pid=$!
+ sleep 0.2
+ msg "SIGTERM drains an accepted request within the grace period"
+ kill -TERM "$PAIVANA_PID"
+ wait "$curl_pid" || fail "request was dropped during graceful shutdown"
+ [ "$(cat "$SCRATCH/drain.status")" = 200 ] || \
+ fail "drained request did not retain its 200 response"
+ wait "$PAIVANA_PID" || fail "paivana exited unsuccessfully after draining"
+ PAIVANA_PID=""
+ grep -q 'Graceful shutdown drained all requests' \
+ "$LOGDIR/paivana-custom.log" || \
+ fail "graceful-drain completion was not logged"
+ ok
+
+ start_paivana "http://127.0.0.1:$MHD_PORT"
+}
+
function test_trusted_proxies_config() {
stop_paivana
local r
@@ -2294,6 +2446,9 @@ test_forwarded_unix
test_forwarded_rfc7239
test_forwarded_unix_rfc7239
test_paywall_disabled_endpoint
+test_resource_budget_config
+test_controlled_overload
+test_graceful_shutdown
test_trusted_proxies_config
test_whitelist_config
diff --git a/src/tests/test_template_limits.sh b/src/tests/test_template_limits.sh
@@ -0,0 +1,191 @@
+#!/usr/bin/env bash
+# Exercise the startup limits around merchant templates and installed
+# languages. The production defaults are intentionally tested at their real
+# boundaries; the local stub keeps the run small in wall-clock time.
+
+set -u
+
+SRCDIR="${SRCDIR:-$(cd -- "$(dirname -- "$0")" && pwd)}"
+BUILDDIR="${BUILDDIR:-$PWD}"
+PAIVANA_HTTPD="${PAIVANA_HTTPD:-$BUILDDIR/../backend/paivana-httpd}"
+PORT_BASE="${PAIVANA_PORT_BASE:-18400}"
+# payment_backend_failure owns +120/+121 and Meson runs tests in parallel.
+MERCHANT_PORT=$((PORT_BASE + 130))
+PAIVANA_PORT=$((PORT_BASE + 131))
+SCRATCH="$(mktemp -d -t paivana-template-limits.XXXXXX)"
+STUB_PID=""
+PAIVANA_PID=""
+
+function cleanup() {
+ set +e
+ [ -n "$PAIVANA_PID" ] && kill -TERM "$PAIVANA_PID" 2>/dev/null
+ [ -n "$STUB_PID" ] && kill -TERM "$STUB_PID" 2>/dev/null
+ [ -n "$PAIVANA_PID" ] && wait "$PAIVANA_PID" 2>/dev/null
+ [ -n "$STUB_PID" ] && wait "$STUB_PID" 2>/dev/null
+ [ "${KEEP_TMP:-0}" = 1 ] || rm -rf "$SCRATCH"
+}
+trap cleanup EXIT
+
+function fail() {
+ echo "FAIL: $*" >&2
+ tail -n 80 "$SCRATCH/paivana.log" >&2 2>/dev/null || true
+ tail -n 40 "$SCRATCH/stub.log" >&2 2>/dev/null || true
+ exit 1
+}
+
+function port_is_free() {
+ ! (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null
+}
+
+function wait_for_port() {
+ local port="$1" pid="$2"
+ for _ in $(seq 1 100); do
+ port_is_free "$port" || return 0
+ kill -0 "$pid" 2>/dev/null || return 1
+ sleep 0.05
+ done
+ return 1
+}
+
+function wait_for_exit() {
+ local pid="$1"
+ for _ in $(seq 1 100); do
+ kill -0 "$pid" 2>/dev/null || return 0
+ sleep 0.05
+ done
+ return 1
+}
+
+function stop_case() {
+ set +e
+ [ -n "$PAIVANA_PID" ] && kill -TERM "$PAIVANA_PID" 2>/dev/null
+ [ -n "$STUB_PID" ] && kill -TERM "$STUB_PID" 2>/dev/null
+ [ -n "$PAIVANA_PID" ] && wait "$PAIVANA_PID" 2>/dev/null
+ [ -n "$STUB_PID" ] && wait "$STUB_PID" 2>/dev/null
+ PAIVANA_PID=""
+ STUB_PID=""
+ set -u
+}
+
+PAYWALL_TEMPLATE="$BUILDDIR/../frontend/paywall.en.must"
+[ -x "$PAIVANA_HTTPD" ] || { echo "SKIP: no paivana-httpd"; exit 77; }
+[ -r "$PAYWALL_TEMPLATE" ] || { echo "SKIP: paywall template not built"; exit 77; }
+command -v python3 >/dev/null || { echo "SKIP: no python3"; exit 77; }
+port_is_free "$MERCHANT_PORT" || { echo "SKIP: port $MERCHANT_PORT busy"; exit 77; }
+port_is_free "$PAIVANA_PORT" || { echo "SKIP: port $PAIVANA_PORT busy"; exit 77; }
+
+mkdir -p "$SCRATCH/configd" "$SCRATCH/prefix/share/paivana/templates"
+cp "$PAYWALL_TEMPLATE" "$SCRATCH/prefix/share/paivana/templates/"
+# A visibly distinct second language lets the test verify that cache-key
+# normalization reproduces the templating library's actual choice.
+sed 's/<title>Payment Required<\/title>/<title>Zahlung Erforderlich<\/title>/' \
+ "$PAYWALL_TEMPLATE" \
+ >"$SCRATCH/prefix/share/paivana/templates/paywall.de.must"
+export PAIVANA_BASE_CONFIG="$SCRATCH/configd"
+export PAIVANA_PREFIX="$SCRATCH/prefix/"
+
+cat >"$SCRATCH/paivana.conf" <<EOF
+[paivana]
+DESTINATION_BASE_URL = http://127.0.0.1:9/
+BASE_URL = http://127.0.0.1:$PAIVANA_PORT/
+MERCHANT_BACKEND_URL = http://127.0.0.1:$MERCHANT_PORT/
+MERCHANT_ACCESS_TOKEN = secret-token:stub
+SECRET = template-limit-test
+SERVE = tcp
+BIND_TO = 127.0.0.1
+PORT = $PAIVANA_PORT
+EOF
+
+function start_stub() {
+ local count="$1" padding="$2" delay="$3"
+ : >"$SCRATCH/stub.log"
+ PAIVANA_STUB_TEMPLATE_COUNT="$count" \
+ PAIVANA_STUB_CONTRACT_PADDING="$padding" \
+ PAIVANA_STUB_DETAIL_DELAY="$delay" \
+ python3 "$SRCDIR/payment_backend_stub.py" "$MERCHANT_PORT" \
+ >"$SCRATCH/stub.log" 2>&1 &
+ STUB_PID=$!
+ wait_for_port "$MERCHANT_PORT" "$STUB_PID" || fail "stub did not start"
+}
+
+function start_paivana() {
+ : >"$SCRATCH/paivana.log"
+ "$PAIVANA_HTTPD" -c "$SCRATCH/paivana.conf" -L DEBUG \
+ >"$SCRATCH/paivana.log" 2>&1 &
+ PAIVANA_PID=$!
+}
+
+echo -n "template fetch concurrency is capped at eight ... "
+start_stub 16 0 0.15
+start_paivana
+wait_for_port "$PAIVANA_PORT" "$PAIVANA_PID" || fail "16-template startup failed"
+grep -q 'template detail concurrency 2' "$SCRATCH/stub.log" || \
+ fail "template details were fetched serially instead of concurrently"
+if grep -Eq 'template detail concurrency ([9]|[1-9][0-9]+)' "$SCRATCH/stub.log"; then
+ fail "more than eight template detail requests ran concurrently"
+fi
+echo "OK"
+
+echo -n "Accept-Language variants normalize to the installed language ... "
+for n in $(seq 1 40); do
+ curl -sS -H "Accept-Language: de, x-test-$n;q=0.1" \
+ -o "$SCRATCH/paywall-$n.html" \
+ "http://127.0.0.1:$PAIVANA_PORT/.well-known/paivana/templates/premium-0000" \
+ || fail "paywall language request $n failed"
+ grep -q '<title>Zahlung Erforderlich</title>' "$SCRATCH/paywall-$n.html" || \
+ fail "language request $n did not select the German template"
+done
+echo "OK"
+stop_case
+
+echo -n "a contract over one MiB is refused ... "
+start_stub 1 1048576 0
+start_paivana
+wait_for_exit "$PAIVANA_PID" || fail "oversized-contract startup did not exit"
+wait "$PAIVANA_PID" 2>/dev/null || true
+PAIVANA_PID=""
+grep -q 'exceeding the 1048576-byte' "$SCRATCH/paivana.log" || \
+ fail "oversized contract had no limit diagnostic"
+echo "OK"
+stop_case
+
+echo -n "more than 1024 discovered templates are refused before detail fetches ... "
+start_stub 1025 0 0
+start_paivana
+wait_for_exit "$PAIVANA_PID" || fail "1025-template startup did not exit"
+wait "$PAIVANA_PID" 2>/dev/null || true
+PAIVANA_PID=""
+grep -q 'exceeding the 1024-entry startup bound' "$SCRATCH/paivana.log" || \
+ fail "discovery limit had no diagnostic"
+if grep -q 'template detail concurrency' "$SCRATCH/stub.log"; then
+ fail "details were fetched after the discovery limit was exceeded"
+fi
+echo "OK"
+stop_case
+
+echo -n "more than 128 retained Paivana templates are refused ... "
+start_stub 129 0 0
+start_paivana
+wait_for_exit "$PAIVANA_PID" || fail "129-template startup did not exit"
+wait "$PAIVANA_PID" 2>/dev/null || true
+PAIVANA_PID=""
+grep -q 'more than 128 Paivana templates' "$SCRATCH/paivana.log" || \
+ fail "retained-template limit had no diagnostic"
+echo "OK"
+stop_case
+
+echo -n "more than 32 installed paywall languages are refused ... "
+rm -f "$SCRATCH/prefix/share/paivana/templates/"*
+for n in $(seq -w 0 32); do
+ cp "$PAYWALL_TEMPLATE" \
+ "$SCRATCH/prefix/share/paivana/templates/paywall.l$n.must"
+done
+start_paivana
+wait_for_exit "$PAIVANA_PID" || fail "33-language startup did not exit"
+wait "$PAIVANA_PID" 2>/dev/null || true
+PAIVANA_PID=""
+grep -q 'More than 32 paywall languages' "$SCRATCH/paivana.log" || \
+ fail "language limit had no diagnostic"
+echo "OK"
+
+echo "template startup limits: OK"
