Error reporting and alerting for a hub, with the backend and the channels as typed parameters
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-13 18:51:48 +00:00
docs docs: the openhubs_spl patch against version 6, after curation 2026-10-25 01:22:32 +02:00
src fix: the email handler no longer silences the warnings of every feature; on a root logger nobody set up, a console handler prints what Python printed before 2026-11-13 19:49:37 +01:00
.gitignore feat: scaffold splent_feature_observability with splent feature:create 2026-10-25 00:35:19 +02:00
COPYING feat: scaffold splent_feature_observability with splent feature:create 2026-10-25 00:35:19 +02:00
COPYING.LESSER feat: scaffold splent_feature_observability with splent feature:create 2026-10-25 00:35:19 +02:00
MANIFEST.in feat: scaffold splent_feature_observability with splent feature:create 2026-10-25 00:35:19 +02:00
pyproject.toml chore: release splent-io/splent_feature_observability to v0.3.0 2026-11-13 18:51:48 +00:00
README.md docs: the parameters' types and limits, and the backend's and channels' nodes in the line 2026-10-25 16:27:57 +01:00

splent_feature_observability

Where the errors of a SPLENT product go, and who is told when something breaks. The error backend and the alert channels are parameters of the feature, so each product of the line chooses its own: uvlhub sends its errors to a self-hosted GlitchTip and its alerts by email and to a Matrix room (decision D20).

Choices Parameter
Error backend glitchtip, sentry, email, none OBSERVABILITY_BACKEND
Alert channels any of email, matrix, telegram OBSERVABILITY_ALERT_CHANNELS
Alerted on any of errors, readiness, degraded OBSERVABILITY_ALERT_ON

Errors

Every backend gets the same errors: an exception no handler caught in a request, and every line logged at ERROR or above (a task that failed, a page that could not be drawn, logger.exception anywhere). It also gets what a feature reports with report_degraded, as a warning.

  • glitchtip and sentry are one backend: the Sentry SDK, which GlitchTip speaks, pointed at OBSERVABILITY_DSN. The Flask integration takes the unhandled exceptions, with the endpoint as the transaction, and the logging integration the ERROR lines. Events carry the environment (OBSERVABILITY_ENVIRONMENT, else the app's profile), the release (OBSERVABILITY_RELEASE, else <product>@<version>), the request id and the id of the account signed in. OBSERVABILITY_SAMPLE_RATE is the share of errors sent and OBSERVABILITY_TRACES_SAMPLE_RATE the share of requests traced for performance (off by default).
  • email sends one mail per kind of error (the same exception in the same function, or the same log line) to OBSERVABILITY_EMAIL_TO, through splent_feature_mail's outbox, at most once per OBSERVABILITY_ALERT_DEDUP_MINUTES, saying how many were held back. It carries the type, the message, where it was raised, the request's endpoint and id, and the traceback.
  • none leaves them in the log.

What a report never carries

A report leaves the site, so it carries what finds the bug and nothing that opens an account or says who a user is (scrub.py):

  • no cookies, no request body, no query string, and no header but the content type, the accept header, the host and the request id;
  • no IP address, no email address (replaced by [email] wherever it appears, an exception's message included), no user but its id;
  • no token: a path segment or a word that looks like one (a long run of letters of both cases and digits, or a hex digest) becomes [token], so /reset/<token> arrives as /reset/[token], and what follows Bearer or Basic is removed;
  • no local variables (the SDK is told not to collect them) and no value under a key that names a credential (password, token, api_key, dsn...).

The SDK is started with send_default_pii=False and max_request_body_size="never" as well. The scrubbing is what holds when a feature puts personal data in a message anyway.

The error page

error_reference() is in every template: the id GlitchTip or Sentry gave the error being shown, else the request id. A product's 500.html quotes it, so a user who reports the page gives whoever runs the site something to search for:

<p>{{ _('Reference') }}: <code>{{ error_reference() }}</code></p>

Alerts

An alert is a short message to the people who run the site:

On When
errors An exception no handler caught in a request. Once per kind of error per window, with where it was raised and the request's endpoint and id. With the email backend the email channel is left out of these, since the backend's mail is the alert
readiness A check of /readyz that has failed for OBSERVABILITY_READINESS_GRACE_SECONDS (60), so a deploy that waits a few seconds for its database wakes nobody; again once per window while it fails; and once more when it passes again
degraded What a feature reports with report_degraded (below), once per window per feature and message

The readiness alerts come from the probes: the framework tells the feature which checks failed after every /readyz that something asks for. In a deployment that is the container's healthcheck, which probes it every few seconds. A product without one, or with the probe turned off, gets no readiness alerts.

The same alert goes out once per OBSERVABILITY_ALERT_DEDUP_MINUTES (60), and the next one says how many were held back. The processes of a container agree on it through small files in OBSERVABILITY_STATE_DIR (a directory of the temporary one by default), under a lock, so the four workers of a gunicorn do not send four copies.

Alerts never go from the request: a queue drained by a thread of the process sends them, with an app context of its own, so a slow homeserver delays no page, and an error in a request whose database session is broken does not reach the mail outbox through that session. The framework's task runner is not used on purpose: a task waits for the commit of the request that queued it, which an unhandled error rolls back, and in rq mode for Redis, which may be what is down.

Channels

  • email, to OBSERVABILITY_EMAIL_TO, through splent_feature_mail's MailService: a mail the server refused is retried and shows on /admin/mail. It needs the database, so during an outage of it Matrix and Telegram are the channels that still reach someone.
  • matrix: an m.text message in OBSERVABILITY_MATRIX_ROOM_ID (a room id, !<id>:<server>, not an alias), posted to OBSERVABILITY_MATRIX_HOMESERVER with the access token OBSERVABILITY_MATRIX_TOKEN of an account that has joined the room. A bot account of its own, not a person's.
  • telegram: a message in OBSERVABILITY_TELEGRAM_CHAT_ID, posted by the bot whose token is OBSERVABILITY_TELEGRAM_BOT_TOKEN and which is a member of the chat.

A channel that fails is logged as a warning, with what the service answered and never the credential (the Telegram token is part of the URL it posts to, so neither the URL nor the library's message reaches the log).

For the other features: report_degraded

A feature that catches a failure of a service it depends on and carries on without it says so with the framework's API, which logs a warning and hands the report to this feature when the product has it:

from splent_framework.observability import report_degraded

try:
    deposit = zenodo.create(dataset)
except requests.RequestException as exc:
    report_degraded("zenodo", "Zenodo does not answer.", exc=exc)
    raise

The feature never imports this one, and the call does nothing more than log without it. Keep the message the same every time the problem happens and free of personal data: it is what an alert says and what two reports share to count as one.

Settings

Variable Type Default Bind
OBSERVABILITY_BACKEND enum none product (uvl) glitchtip, sentry, email or none
OBSERVABILITY_DSN secret secret https://<key>@<host>/<project>
OBSERVABILITY_ENVIRONMENT str, [A-Za-z0-9_.-]{1,64} deployment Empty: production, development or testing, after the app's profile
OBSERVABILITY_RELEASE str, [^\s/]{1,200} deployment Empty: <product>@<version>
OBSERVABILITY_SAMPLE_RATE float 1.0 deployment 0 to 1
OBSERVABILITY_TRACES_SAMPLE_RATE float 0.0 deployment 0 to 1
OBSERVABILITY_ALERT_CHANNELS list [] product (uvl) Any of email, matrix, telegram
OBSERVABILITY_ALERT_ON list all three product Any of errors, readiness, degraded
OBSERVABILITY_ALERT_DEDUP_MINUTES int, 1 to 10080 60 deployment
OBSERVABILITY_READINESS_GRACE_SECONDS int, 0 to 3600 60 deployment
OBSERVABILITY_EMAIL_TO list of addresses [] deployment The email backend's and the email channel's addresses
OBSERVABILITY_MATRIX_HOMESERVER url deployment https://matrix.example.org
OBSERVABILITY_MATRIX_ROOM_ID secret secret !<id>:<server>
OBSERVABILITY_MATRIX_TOKEN secret secret
OBSERVABILITY_TELEGRAM_BOT_TOKEN secret secret
OBSERVABILITY_TELEGRAM_CHAT_ID secret secret
OBSERVABILITY_HTTP_TIMEOUT int, 1 to 60 5 deployment Seconds a Matrix or Telegram request may take
OBSERVABILITY_STATE_DIR str deployment Where the processes agree on what was alerted

The backend and the channels are the product's decisions and are reflected in the line's model (uvl = true), since the email ones need splent_feature_mail. In openhubs_spl, observability has a mandatory node of OBSERVABILITY_BACKEND with the alternative errors_glitchtip, errors_sentry, errors_email and errors_none, and an optional one of OBSERVABILITY_ALERT_CHANNELS with the or-group alerts_email, alerts_matrix and alerts_telegram. product:validate selects them from the product's values, and errors_email => mail and alerts_email => mail make a product that mails its errors or alerts without mail fail it. The credentials and the room and chat the alerts go to are secrets: never in a committed file, always in the deployment's .env.

At start the feature refuses a DSN that is not one and a Matrix alias in place of a room id, anywhere. A backend or a channel the product chose without what it needs (glitchtip without a DSN, matrix without a token) stops the start in production, since errors would go nowhere while the site looks watched; elsewhere it is a warning and that one stays off, so a developer's machine runs without the deployment's credentials.

Commands

splent feature:observability status   # where errors go, who alerts reach
splent feature:observability test     # a test alert to every channel, now

test exits with 1 when a channel did not take the alert, so a deployment checks its settings before it needs them. With glitchtip or sentry it also sends a test error and prints its event id.

Enable it in a product

  1. Select observability, an optional platform feature of openhubs_spl, with the nodes of its backend and channels (added by docs/openhubs_spl.uvl.patch, see Settings).
  2. Pin it, and decide the backend and the channels in the product's [tool.splent.config]:
    OBSERVABILITY_BACKEND = "glitchtip"
    OBSERVABILITY_ALERT_CHANNELS = ["email", "matrix"]
    
  3. Put the rest in the deployment's .env.prod:
    OBSERVABILITY_DSN=https://<key>@glitchtip.example.org/<project>
    OBSERVABILITY_EMAIL_TO=ops@example.org
    OBSERVABILITY_MATRIX_HOMESERVER=https://matrix.example.org
    OBSERVABILITY_MATRIX_ROOM_ID=!<id>:example.org
    OBSERVABILITY_MATRIX_TOKEN=<access token of the bot account>
    
  4. After the deploy, splent feature:observability test in the web container.

The image needs sentry-sdk[flask] (a dependency of the feature) for glitchtip and sentry. Without it the feature says so and keeps the errors in the log.

What it does not do yet

The rest of OB-1 to OB-7 belongs to the framework, the product or other features:

  • Errors outside a request and outside a log line at ERROR (a CLI command that raises) are not captured. A task that fails is, by the line the task runner logs.
  • Synthetic probes of the outside services (Zenodo, SMTP, ORCID, TLS expiry, OB-3) and an outside monitor of /readyz. The probes are checks registered with register_readiness_check or tasks on a schedule, which the framework has no runner for yet.
  • X-Request-ID from nginx to the app, the tasks and the outgoing calls, /metrics, and the 502/503/504 pages with the request id (OB-4). The framework reads the header when nginx sets it.
  • Consistency checks and the disk alerts (OB-5), /admin/status (OB-6) and the browser's error beacon with the CSP's report-uri (OB-7).
  • The features that swallow errors today (confirmemail, mail, explore, dataset) calling report_degraded instead (OB-2).

Develop

splent feature:test observability --neutral
splent feature:contract observability --write

No test calls GlitchTip, Sentry, Matrix or Telegram: the SDK's init and the HTTP layer are replaced in the tests.