ansible-taler-exchange

README (24524B)

---

 # Ansible Taler Playbooks
 
 ## Installing dependencies
 
 Depending on your local installation, you might need
 to install the following ansible collection:
 
 ```
 $ ansible-galaxy collection install community.postgresql
 ```
 
 Playbooks that touch vaulted variables need the vault password in
 "vault_pass.txt", which is gitignored and shipped only as
 "vault_pass.txt.gpg":
 
 ```
 $ gpg -d vault_pass.txt.gpg > vault_pass.txt
 ```
 
 ## Running the main Playbooks
 
 The canonical playbooks are run via shell scripts in the top-level
 directory.
 
 The operational helpers pass arguments after their positional arguments
 directly to `ansible-playbook`. For example:
 
 ```
 $ ./deploy spec --check --diff
 $ ./sanction-check spec sanctions.txt --diff
 $ BORG_PASSPHRASE=... ./restore spec archive-name --verbose
 ```
 
 Put the host/group first, followed by the sanction list or restore archive
 where required, then any Ansible options. `extract-borg-key` and
 `setup-pixel-borg` default to `spec` when called without arguments; supply
 an explicit target when passing options. Quote option values containing
 spaces as usual. Check-mode support depends on the playbook; restore
 explicitly rejects check mode. The `test` runner and `stage-activate` do
 not forward Ansible options.
 
 ### Local Podman deployment test
 
 Run `./test` to build a Debian Trixie container, provision the complete test
 deployment, repeat `setup.yml` with `--check --diff`, and exercise the main
 systemd units and HTTP APIs.  The script creates a disposable SSH key and
 removes the container when it exits.
 
 The container deliberately runs privileged because the hardened Taler systemd
 units need capabilities which Podman otherwise removes.  It also uses the
 pregenerated DH parameters and a container-only Certbot substitute which
 creates short-lived self-signed certificates: public ACME validation cannot
 reach the local test domains.  Neither shortcut is used by real deployments.
 
 ### Main setup (restore, deploy)
 
 Database restore is deliberately separate from normal deployment.  It is a
 fresh-host recovery operation, not an idempotent part of `deploy`, and it
 never chooses an archive implicitly.  First list the Borg repository and
 select the exact archive to restore:
 
 ```
 $ BORG_PASSPHRASE=... borg list ssh://borg@pixel.taler-systems.com/~/spec-backup
 $ BORG_PASSPHRASE=... ./restore spec 'archive-name-copied-from-list'
 ```
 
 The first argument must be one exact inventory hostname (not a group or host
 pattern).  The destination PostgreSQL cluster must contain only the standard
 `postgres` database and role.  The restore validates the compressed
 `root/postgres-backup.sql.gz` artifact, rejects dumps made by a newer
 PostgreSQL major version, restores all databases and roles, and verifies the
 exchange database before reporting success.  It does not deploy or start the
 application; inspect the recovered host and then run `deploy` separately.
 
 If SQL restore or verification fails, the partially restored cluster is
 considered tainted and must be reinitialized before another attempt.  The
 temporary database dump is removed, while a private error log is retained in
 `/var/tmp/taler-postgres-restore-*.log` on the target for diagnosis.
 
 The "deploy" script deploys the latest version of a system on a host.
 If you are root@rusty.taler-ops.ch, you may be able to:
 
 ```
 $ ./deploy rusty
 ```
 
 Deployments keep `/var/lib/taler-deployment/incomplete` until configuration
 handlers and post-deployment checks have succeeded. If a run fails or is
 interrupted, the next `deploy` reapplies nginx and enabled monitoring service
 configuration even when the files are unchanged. This recovers restart
 notifications lost by the previous Ansible process. Applications still stay
 stopped on deployment failure; check mode neither clears the marker nor
 restarts services. Normal successful reruns keep the usual change-triggered
 handler behavior.
 
 For TOPS production, replace the "rusty" with "spec" to use the actual secrets
 for the deployment. For this, you first need to decrypt them:
 
 ```
 $ ./contrib/decrypt inventories/host_vars/spec/prod-secrets.yml.gpg
 ```
 
 Make sure to NEVER commit the decrypted production secrets to Git.
 Instead, if you had to edit them, re-encrypt them to all admins:
 
 ```
 $ ./contrib/encrypt inventories/host_vars/spec/prod-secrets.yml
 ```
 
 ### Staging offline master key
 
 The `test-master.priv` offline master key is intentionally committed to this
 repository and referenced by `stage-offline.conf`.  It is a disposable test key
 for the staging exchange, where allowing repository users to sign staging
 configuration is intentional.  It must never be used for production, for an
 exchange holding funds of value, or as the basis for any security assumption.
 
 ### sanction-check
 
 This command imports and checks the latest sanction lists:
 
 ```
 $ ./sanction-check $TARGET $LIST
 ```
 
 where "$TARGET" is a host or group from "inventories/default" and $LIST
 is the path of the sanction list file on the local disk.
 
 Note that "SANCTION_LIST" means something else in the "exchange" role,
 where it is the name of a file under
 "roles/exchange/files/var/lib/taler-exchange/" that is deployed to the
 host and referenced from the exchange configuration.
 
 NOTE: this should still be further automated.
 
 
 ### External node monitoring
 
 To monitor an exchange node through `tsys-infra/monitoring`, first enroll it
 as an external monitoring client using that repository's
 `monitoring/README-external.md`. Retrieve both exported YAML files through
 an authenticated, encrypted channel. This repository configures only the
 exchange node; it does not connect to Sentol or enroll hosts itself.
 
 Place the exports in the exchange inventory host directory:
 
 ```
 inventories/host_vars/$HOST/monitoring-client.yml
 inventories/host_vars/$HOST/monitoring-client-secrets.yml
 ```
 
 The directory uses this repository's inventory hostname, such as `spec`.
 The identity inside the export remains the identity enrolled on Sentol;
 it need not equal that inventory alias. Use the complete exports without
 renaming their fields or combining them with other host variables.
 
 The proxy bind address defaults to `127.0.0.1` when omitted from the bundle.
 Set `monitoring_client.node_exporter.proxy_bind_address` to `"*"` (quoted
 in YAML) to listen on all IPv4 and IPv6 interfaces. This requires IPv6
 support on the node. Alternatively, use a literal IPv4 or IPv6 address
 assigned to the exchange node and reachable from the monitoring server.
 Hostnames and bare wildcard addresses (`0.0.0.0` and `::`) are rejected;
 use `"*"` for both address families. With `"*"`, the proxy and loopback
 backend must use different ports (normally 9100 and 9101). The proxy
 still requires mTLS, and the backend remains loopback-only. Keep the
 enrolled certificate identity unchanged.
 
 Commit both files, encrypting the entire secrets file with Ansible Vault
 before staging it:
 
 ```
 $ chmod 0600 inventories/host_vars/$HOST/monitoring-client-secrets.yml
 $ ansible-vault encrypt inventories/host_vars/$HOST/monitoring-client-secrets.yml
 ```
 
 Ansible uses the existing `vault_pass.txt` configuration. The committed
 secrets file must begin with `$ANSIBLE_VAULT;`; plaintext secrets are
 rejected when monitoring is enabled. Do not commit the plaintext export
 or the vault password. The Ansible controller needs Python 3 and OpenSSL
 for bundle validation. As with other vaulted host variables, Ansible may
 need the vault password even when monitoring management is disabled.
 
 Set this in the host's public inventory configuration, then deploy normally:
 
 ```yaml
 enable_monitoring: true
 ```
 
 ```
 $ ./deploy "$HOST"
 ```
 
 The `monitoring` role installs Debian's node_exporter with a loopback-only
 listener and systemd metrics, a dedicated `node-exporter-proxy` nginx
 service requiring Sentol's mTLS client identity, and buffered Fluent Bit
 JSON Lines forwarding over HTTPS/mTLS. All connection settings and certificate identities
 come from the bundle. TLS material lives in `/etc/taler-monitoring/tls`;
 private keys and the enclosing directory are accessible only to root.
 The root-running Fluent Bit service reads the journal and client key. The role
 requires Debian 13 amd64/arm64 and installs pinned Fluent Bit 5.1.2 from its signed
 upstream APT repository. The bundle declares `logs.protocol: jsonline` and an HTTPS
 `logs.url` ending in `/jsonline`; old RELP bundles fail preflight.
 
 Allow Sentol to reach the exported metrics proxy address and port
 (normally TCP 9100), and allow the node to reach the exported HTTPS log server
 (normally TCP 2514). Firewall and routing configuration is external to
 this role. The dedicated proxy leaves exchange webserver sites intact.
 
 Verify the local services with:
 
 ```
 $ systemctl is-active prometheus-node-exporter node-exporter-proxy fluent-bit
 $ curl --fail http://127.0.0.1:9101/metrics >/dev/null
 $ logger --tag external-monitoring-enrollment 'Exchange monitoring test'
 ```
 
 Use the exported backend address if it differs. Follow the external-client
 runbook to perform an authenticated scrape from Sentol, confirm anonymous
 scrapes fail, and find the test message in VictoriaLogs using the exported identity as
 `authenticated_host`. Fluent Bit metrics are available through the same mTLS proxy
 at `/fluent-bit/metrics`.
 
 Fluent Bit reads only new journal entries on first activation and resumes its saved
 cursor on later starts. Journald uses persistent storage without changing host-specific
 retention. `/var/lib/fluent-bit` retains the cursor and filesystem buffer, with a
 1 GiB output queue by default (`monitoring_fluent_bit_queue_limit`). Retryable failures
 are retried without a count limit; a full queue discards its oldest chunks. Permanent
 HTTP errors can discard records, and VictoriaLogs can accept a batch while rejecting
 individual malformed lines. Watch both collector drop counters and receiver errors.
 Lost responses can cause duplicates; this is not exactly-once or unconditional
 lossless delivery. Fluent Bit's own logs remain local to prevent feedback loops.
 
 For renewal, retrieve both refreshed exports after `make deploy-sentol`,
 replace the local files, encrypt the complete new secrets file before
 staging it, commit both, and run `deploy` again. Do not reuse an old key
 with a renewed certificate. The role validates both pairs before deployment
 and reloads or restarts the affected services when their material changes.
 
 `enable_monitoring` defaults to false. False skips management and leaves
 installed monitoring running. To offboard, follow the Sentol runbook and,
 on the exchange node, stop and disable `node-exporter-proxy` and
 `prometheus-node-exporter` and `fluent-bit`, then remove the proxy unit
 and `/etc/taler-monitoring` (including its private keys) and run
 `systemctl daemon-reload`. Set the flag false and remove both inventory
 exports. This does not revoke previously issued certificates; complete
 the Sentol offboarding steps as well.
 
 ### Setting up backups (TOPS-only for now)
 
 First run:
 
 ```
 $ ./extract-borg-key $DEPLOYMENT
 ```
 
 The resulting SSH public key should be added to the borg-account
 of the host storing the backup. The borg_host inventory setting selects the target
 hostname.
 
 Once the SSH key is deployed and the backup has been initialized
 server-side (see admin-logs/pixel/03-borg.txt), start the daily
 backups via:
 
 ```
 $ ./start-borg-backups $DEPLOYMENT
 ```
 
 This will make a backup basically everything relevant to the
 deployment, **except** the exchange online signing keys. The
 backup will in particular include the system configuration
 and a full (gzip-compressed) snapshot of the database.  Thus,
 the backups should also suffice to diagnose problems.
 
 Backups are set to retain daily snapshots of the last 7 days,
 weekly snapshots for the last 4 weeks, and monthly snapshots
 for the last 6 months.
 
 
 ### Backup (right now)
 
 To run a backup "immediately" (instead of the daily regular
 backups), use:
 
 ```
 $ ./backup $DEPLOYMENT
 ```
 
 
 ### Rebooting (into a new kernel)
 
 This should be done via the 'reboot' playbook which can
 be invoked via the
 
 ```
 $ ./reboot $DEPLOYMENT
 ```
 
 script. The reboot playbook first stops all Taler services,
 then makes a backup, and then reboots. This should help us
 restore to another system in case the host does not come back
 online cleanly.
 
 
 ### Testing Locally
 
 With podman and ansible installed locally one can run:
 
 ```
 $ ./test
 ```
 
 This will begin building the Containerfile in this repo, which is a Debian
 base with systemd and a paswordless ssh server configured. Then container
 will start, binding port 8022 to 127.0.0.1 on the host. Finally
 the setup playbook will be run on the container via ssh.
 
 
 ## Playbooks
 
 ### backup
 
 Runs a backup "right now".
 
 ### borg-ssh-export
 
 Exports the SSH public keys needed at the remote host for backups.
 
 ### borg-start
 
 Enables the borg backup. Should be run after the SSH public keys
 exported via borg-ssh-export have been deployed on the receiving
 host.
 
 ### pixel-borg
 
 Enables receiving (!) backups from pixel. Adds the public key from
 pixel so we accept receiving borg backups from pixel.  Note that
 pixel still needs to be setup to send the backups.
 
 ### reboot
 
 Safely reboots the system by first stopping all Taler services,
 then making a backup and only then actually rebooting it.
 
 ### sanctionlist-check
 
 Imports the latest sanction list and checks all records against
 it.
 
 ### setup
 
 The main script that deploys our entire setup.
 
 ## Roles
 
 ### auditor
 
 Deploys the auditor.
 
 ### backup
 
 Runs the backup script, making a borg backup of the database and other key parts of the system.
 
 ### borg-ssh-export
 
 Exports the SSH public key that must be deployed on the host that is
 to receive the backup.
 
 ### borg-start
 
 Deploys the backup script and the cron job that runs it daily.
 
 ### cert
 
 Issues or renews a LetsEncrypt certificate with certbot and enables the
 nginx sites that use it. Included by the roles that own a vhost, not
 listed in a playbook directly.
 
 ### challenger
 
 Deploys the various challenger services for address verification.
 
 ### common_packages
 
 Installs the base system packages we need on all hosts.
 Sets up Taler package repo and installs Taler packages.
 
 ### database
 
 Installs PostgreSQL and the dependencies needed to administer it through
 Ansible, then ensures the database service is running.  Normal deployment
 never restores a backup.
 
 ### database_restore
 
 Implements the guarded fresh-cluster restore used only by `playbooks/restore.yml`
 and `restore`.
 
 ### devtesting
 
 Sets up a restricted account for faking wire transfers and inspecting
 challenger auth codes. Gated on "dangerously_enable_devtesting" and never
 to be enabled in production.
 
 ### exchange
 
 Deploys the Taler exchange, then hands over to "exchange_$DEPLOYMENT_KIND"
 for the site-specific configuration.
 
 ### exchange_tops
 
 Deployment-specific exchange configuration (denominations, KYC rules, fees,
 legal documents), selected by DEPLOYMENT_KIND.
 
 ### exchange-sanctionlist-import
 
 Imports a new sanction list and checks all existing records against it.
 
 ### libeufin-nexus
 
 Deploys libeufin-nexus which connects us to the bank.
 
 ### monitoring
 
 Provisions external node monitoring when `enable_monitoring` is true.
 
 ### pixel_borg
 
 Configures the host to hosting backups *from* pixel.
 
 ### reboot
 
 Reboots the system.
 
 ### stop_services
 
 Stops all Taler-related services. Useful for emergency stop and
 used as part of the reboot playbook.
 
 ### webserver
 
 Configures Nginx reverse proxy (main service, not individual subdomains).
 
 
 ## Checks
 
 ```
 $ ./contrib/test-fact-helpers.sh   # the /bin helpers that generate local facts
 ```
 
 Also run by the CI job in "contrib/ci/jobs/001-build".
 
 ## Deployment safety and recovery
 
 Normal deployment still upgrades system packages. It stops existing application
 services, targets, timers and sockets before package work, suppresses application
 starts through Debian's policy-rc.d mechanism, and starts applications
 only after all configuration and database migrations finish. Schedule deployment
 as downtime. During the system-wide upgrade, the temporary policy denies Taler
 service actions and delegates other actions to the existing administrator policy
 (or permits them if none existed). Application package tasks use policy_rc_d: 101.
 The original policy is restored even on failure. PostgreSQL and nginx are started
 as needed during provisioning.
 Cold exchange startup may spend several minutes generating denomination keys.
 post_deployment_http_retries defaults to 60 (with a five-second request timeout
 and one-second retry delay); systemd unit checks retain their shorter retry limit.
 
 A failed deployment leaves applications stopped without removing their boot
 enablement. Inspect the failed task, correct the problem, then rerun deploy.
 Do not reboot or manually start applications to bypass an unfinished migration.
 Optional services whose provisioning is skipped are restored if they were active
 when that deployment began. After a failed deployment, also inspect any optional
 services before returning the host to service.
 
 Setting dangerously_enable_devtesting to false revokes the managed login keys,
 sudo access and helper programs, terminates that account's sessions and disables
 the account. Its home-directory data is retained. Setting it true restores the
 managed account settings and access.
 
 The KYC attribute encryption key is preserved in a restricted secret include.
 Do not rotate this key as a substitute for moving it: changing it without a data
 migration would prevent decryption of existing attributes. Auditor authentication
 is stored in a root-only nginx include. Secret tasks suppress output and diffs,
 including when removing secrets from previously public configuration files.
 
 Backup and restore share borg_host and borg_repo inventory defaults, retaining
 ssh://borg@pixel.taler-systems.com/~/spec-backup for existing installations. Set
 borg_repo in host variables to use another already provisioned repository.
 Existing database_restore_borg_repository and database_restore_borg_host overrides
 remain supported. No archives are moved automatically.
 
 The backup script uses /root/.ssh/borg-config and leaves /root/.ssh/config intact.
 A lock protects the entire backup operation. Install the updated backup script
 between backup runs: an already running older script does not acquire this lock. If cron, backup or reboot
 encounters another backup, it exits nonzero (script status 75) without touching
 that run's snapshot. Retry after the existing backup finishes; a reboot playbook
 aborts before rebooting on this failure. Pruning and compaction are skipped after
 an unsuccessful or warning-producing archive creation.
 
 ## Exchange HTTP latency monitoring
 
 Sentol can opt an exchange into latency collection with
 `monitoring_nginx_latency_enabled: true` in its external-host inventory. Its
 exported public bundle then contains `monitoring_client.nginx_latency.enabled:
 true`. Redistribute the bundle and deploy this repository with `enable_monitoring`
 enabled. Missing or false capability values disable the feature; the capability
 must be a boolean and the exchange domain must be a safe hostname.
 
 Fluent Bit tails `/var/log/nginx/<exchange_domain>.tal` and aggregates nginx's
 `rt` field into `nginx_request_duration_seconds` histogram buckets, sum and count.
 The labels are `operation`, `method` and `status`. The checked-in
 `roles/monitoring/files/exchange-latency-routes.json` allowlist is shared by nginx
 GET selection and Fluent Bit classification; update it alongside exchange API
 route changes. Operation names omit identifiers; `/batch-deposit` is `deposit`.
 Unrecognized mutation routes are `other`. Malformed records increment
 `nginx_timing_log_parse_errors_total`; oversized lines are skipped so tailing can
 continue. Fluent Bit diagnostics report oversized lines separately.
 
 POST, PATCH and DELETE keep their existing coverage. The exchange's nginx
 virtual host additionally logs known API GETs, including ordinary requests to
 endpoints that optionally long-poll. Reserve status, deposit tracking, purse
 merge/deposit status, KYC check and KYC info GETs are included only when
 `timeout_ms` is absent or consists entirely of zero digits. Purse requests also
 check the documented `deposit_timeout_ms` argument conservatively. Calls with a
 positive timeout are excluded even when they return immediately.
 
 On these polling-capable routes, repeated timeout arguments, malformed values,
 and queries containing percent escapes or plus signs are excluded. This may omit
 unusual ordinary calls, but prevents encoded polling arguments entering latency
 charts. Query strings are inspected only inside nginx and are not added to timing
 logs. Ordinary `/keys` requests remain included when server trouble makes them
 slow. Unknown GET routes, metrics endpoints, static downloads, HEAD and OPTIONS
 are excluded. Other services' timing-log selection is unaffected.
 
 Generated metrics listen on `127.0.0.1:2021/metrics` and are exposed through the
 existing Sentol-only mTLS proxy at `/nginx/metrics`. Port 2021 must not conflict
 with the configured node exporter or metrics proxy. The usual collector-health
 endpoint on port 2020 and journal forwarding continue separately; timing records
 are consumed locally and never sent to VictoriaLogs.
 
 First enrollment starts at the end of the existing timing log. Later starts
 resume `/var/lib/fluent-bit/exchange-timing.db`; deployment must not remove this
 cursor. Rotation should rename the file and tell nginx to reopen its logs;
 Fluent Bit continues watching the old descriptor for 30 seconds. Rotation while
 the collector is stopped can lose observations: the input follows the active
 filename and does not replay rotated archives. A dedicated logrotate rule retains 14
 daily timing logs with delayed compression, matching Debian's standard nginx
 policy (whose `*.log` pattern does not cover `.tal`). This rule remains installed
 when latency metrics are disabled, since mutation timing logs are still written.
 
 Histogram state is in memory and resets on collector restart. Prometheus retains
 already scraped history, but can lose observations since its last scrape. Unread
 records appear when processed after recovery, which can temporarily distort
 rates. This is operational monitoring, not exact accounting. The Perses dashboard
 uses counter-aware five-minute queries and defers latency alert thresholds until
 a baseline is available. Disabling the capability removes the scrape route and
 collector input after redeployment, while preserving the cursor for later use.
 
 Diagnose collection locally with `curl http://127.0.0.1:2021/metrics` and
 `journalctl -u fluent-bit`; inspect the latency dashboard's scrape and parse-error
 panels on Sentol. Before deployment, configuration is checked with the pinned
 Fluent Bit binary's `--dry-run` mode and nginx's `-t` mode.
 
 ## Regression checks
 
 `python3 contrib/tests/test_deployment_recovery.py` uses the local
 `ansible-taler-test` image built by `./test` (or a Debian systemd image passed
 with `--image`). It runs the real setup failure/retry flow and recovery handlers
 against disposable service fixtures, checking lost notifications, unchanged
 retries, check mode, validation failures and disabled monitoring. It requires
 the `containers.podman` Ansible collection and runs without container networking
 or package downloads. The full `./test` runner includes this regression.
 
 `python3 contrib/tests/test_nginx_latency.py` checks the timing parser and bounded
 route classification (requires Python Jinja2 and Lua). The `tsys-infra/monitoring`
 Podman suite exercises this repository's real monitoring role, nginx GET
 selection, pinned Fluent Bit histograms, mTLS scraping, rotation, cursor recovery,
 and Perses queries across resets and idle windows.
 
 Run python3 contrib/tests/test_backup.py for isolated concurrency, failure,
 cleanup and archive round-trip checks (requires Borg and Python Jinja2).
 The extended ./test also exercises real systemd shutdown, package-start
 suppression, a pre-2.20 Ansible dependency bootstrap, secret permissions,
 devtesting revocation, preservation of unrelated configuration, failed deployment
 and recovery. All fixtures run in the disposable container, never an inventory
 production host. Monitoring regressions also cover Vault-encrypted onboarding
 bundles, preflight failures before application shutdown, authenticated metrics,
 JSON Lines delivery, certificate renewal, idempotence, and check mode.