084-simple-observability.rst (5661B)
1 DD 84: Simple observability 2 ########################### 3 4 :Design status: Draft 5 :Implementation status: Not started 6 :DD shepherd: TBD 7 :Historical contributors: Antoine A 8 :First published: 2026-02-16 9 :Last substantive change: 2026-02-16 10 11 Summary 12 ======= 13 14 We want a simple way to check if our various services are working properly and to be notified when they are not. 15 16 Motivation 17 ========== 18 19 We had some difficulties managing the degraded operation of libeufin-nexus. The systemd service was working correctly but was failling in a loop. There is currenlty no way to detect this and receive alerts. 20 21 We also have the same problem with other services. For example, when the exchange is not configured correctly, wirewatch may fail to retrieve transaction history from libeufin-nexus, but the API would work correctly. We have to check this manually for now. 22 23 Right now we only have uptime check with Uptime Kana and we need a better observability system that can detect this degraded states and also provides some context on the failure for faster remediation. 24 25 This is where observability comes in. It provide the answers to ``is it running`` and ``why is it not``. To do this, you usually setup Grafana and Prometheus, which is the standard maximalist solution, but it has problems: 26 - It is heavy and can actually take up more ressources than all the services its supposed to monitor 27 - It is complicated to set up, maintain, and configure 28 29 Because of these problems, we don't have a solution at the moment, as we are delaying this configuration. 30 31 I think we can have a simpler and lighter solution that would answers ``is it running`` well and ``why is it not`` in a minimalistic way. 32 33 Requirements 34 ============ 35 36 * Easy to implement by services maintainers 37 * Easy to configure 38 * Easy to maintain 39 * Effective at detecting downtime or degraded states 40 41 Proposed Solution 42 ================= 43 44 Health endpoint 45 --------------- 46 47 All services have a least one REST API. This API should expose a health endpoint that would expose the service global status, this means the httpd process but also all the other components it's use (either ``ok`` if everything is fine, or ``degraded`` if it's running but not functioning properly). We are also adding a way to add more context in a less structured way to help with remediation. 48 49 .. ts:def:: HealthStatus 50 51 interface HealthStatus { 52 // Whether the service is running fine or in a degraded way 53 status: "ok" | "degraded"; 54 // Additional context about the service components and processes 55 context: { [key: string]: string }; 56 } 57 58 For libeufin-bank: 59 60 .. code-block:: json 61 62 { 63 "status": "degraded", 64 "context": { 65 "database": "ok", 66 "tan-sms": "ok", 67 "tan-email": "failure" 68 } 69 } 70 71 For libeufin-nexus: 72 73 .. code-block:: json 74 75 { 76 "status": "degraded", 77 "context": { 78 "database": "ok", 79 "ebics-submit": "ok", 80 "ebics-submit-latest": "2012-04-23T18:25:43.511Z", 81 "ebics-submit-latest-success": "2012-04-23T18:25:43.511Z", 82 "ebics-fetch": "failure", 83 "ebics-fetch-latest": "2012-04-23T18:25:43.511Z", 84 "ebics-fetch-latest-success": "2012-03-23T18:25:43.511Z" 85 } 86 } 87 88 For taler-exchange: 89 90 .. code-block:: json 91 92 { 93 "status": "degraded", 94 "context": { 95 "database": "ok", 96 "wirewatch": "failure", 97 "wirewatch-latest": "2012-04-23T18:25:43.511Z" 98 } 99 } 100 101 Next, Uptime Kuma can be configured to retrieve this endpoint and trigger an alert when the status is degraded if the API is in place. 102 The JSON body can be shared in the alert, which makes remediation easier because we have hints as to what is not working well inside the service. 103 104 Logs 105 ---- 106 107 Currently, we also rely on logs to detect failures. To do this, we need to ingest the logs into a monitoring system in order to analyze them and generate alerts. We need to decide whether we want to continue doing this or whether we can choose to use the logs only for remediation and therefore leave them where they are. 108 109 Whenever we want to analyze logs to detect a failure condition, we need to see if it is possible for the system to expose it in its health endpoint. 110 111 Test Plan 112 ========= 113 114 - Add health endpoint to libeufin-bank and libeufin-nexus first 115 - Deploy on demo 116 - Test status update and alerts 117 118 Alternatives 119 ============ 120 121 Prometheus format with Uptime Kuma 122 ---------------------------------- 123 124 Instead of using JSON we could use Prometheus metrics textual format. This would make upgrading to a better observability system easier in the future. 125 126 .. code-block:: text 127 128 # HELP app_status 1=ok, 0.5=degraded, 0=down 129 # TYPE app_status gauge 130 app_status 0.5 131 132 # HELP component_database_status 1=ok, 0=failure 133 # TYPE component_database_status gauge 134 database_status 1.0 135 136 # HELP component_tan_sms_status 1=ok, 0=failure 137 # TYPE component_tan_sms_status gauge 138 tan_sms_status 1.0 139 140 # HELP component_tan_email_status 1=ok, 0=failure 141 # TYPE component_tan_email_status gauge 142 tan_email_status 0.0 143 144 We could then also use the existing Taler Observability API but only provide simple metrics for now. 145 146 Prometheus & Grafana alternative 147 -------------------------------- 148 149 We can try to use other more performant while not simpler alternative like Victoria Metrics. 150 151 Drawbacks 152 ========= 153 154 This does not resolve the issue of system resources and systemd services. 155 It's therefore not sufficient for a complete observability system. However, it is easier to implement for now. 156 157 We could hack our own health endpoint that expose systemd services, postgres database and system status in a similar manner. 158 159 Discussion / Q&A 160 ================