- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| docs | ||
| src | ||
| .gitignore | ||
| COPYING | ||
| COPYING.LESSER | ||
| MANIFEST.in | ||
| pyproject.toml | ||
| README.md | ||
splent_feature_observability
Where the errors of a SPLENT product go, and who is told when something breaks. The error backend and the alert channels are parameters of the feature, so each product of the line chooses its own: uvlhub sends its errors to a self-hosted GlitchTip and its alerts by email and to a Matrix room (decision D20).
| Choices | Parameter | |
|---|---|---|
| Error backend | glitchtip, sentry, email, none |
OBSERVABILITY_BACKEND |
| Alert channels | any of email, matrix, telegram |
OBSERVABILITY_ALERT_CHANNELS |
| Alerted on | any of errors, readiness, degraded |
OBSERVABILITY_ALERT_ON |
Errors
Every backend gets the same errors: an exception no handler caught in a
request, and every line logged at ERROR or above (a task that failed, a
page that could not be drawn, logger.exception anywhere). It also gets
what a feature reports with report_degraded, as a warning.
- glitchtip and sentry are one backend: the Sentry
SDK, which
GlitchTip speaks, pointed at
OBSERVABILITY_DSN. The Flask integration takes the unhandled exceptions, with the endpoint as the transaction, and the logging integration the ERROR lines. Events carry the environment (OBSERVABILITY_ENVIRONMENT, else the app's profile), the release (OBSERVABILITY_RELEASE, else<product>@<version>), the request id and the id of the account signed in.OBSERVABILITY_SAMPLE_RATEis the share of errors sent andOBSERVABILITY_TRACES_SAMPLE_RATEthe share of requests traced for performance (off by default). - email sends one mail per kind of error (the same exception in the same
function, or the same log line) to
OBSERVABILITY_EMAIL_TO, through splent_feature_mail's outbox, at most once perOBSERVABILITY_ALERT_DEDUP_MINUTES, saying how many were held back. It carries the type, the message, where it was raised, the request's endpoint and id, and the traceback. - none leaves them in the log.
What a report never carries
A report leaves the site, so it carries what finds the bug and nothing that
opens an account or says who a user is (scrub.py):
- no cookies, no request body, no query string, and no header but the content type, the accept header, the host and the request id;
- no IP address, no email address (replaced by
[email]wherever it appears, an exception's message included), no user but its id; - no token: a path segment or a word that looks like one (a long run of
letters of both cases and digits, or a hex digest) becomes
[token], so/reset/<token>arrives as/reset/[token], and what followsBearerorBasicis removed; - no local variables (the SDK is told not to collect them) and no value
under a key that names a credential (
password,token,api_key,dsn...).
The SDK is started with send_default_pii=False and
max_request_body_size="never" as well. The scrubbing is what holds when a
feature puts personal data in a message anyway.
The error page
error_reference() is in every template: the id GlitchTip or Sentry gave
the error being shown, else the request id. A product's 500.html quotes
it, so a user who reports the page gives whoever runs the site something to
search for:
<p>{{ _('Reference') }}: <code>{{ error_reference() }}</code></p>
Alerts
An alert is a short message to the people who run the site:
| On | When |
|---|---|
errors |
An exception no handler caught in a request. Once per kind of error per window, with where it was raised and the request's endpoint and id. With the email backend the email channel is left out of these, since the backend's mail is the alert |
readiness |
A check of /readyz that has failed for OBSERVABILITY_READINESS_GRACE_SECONDS (60), so a deploy that waits a few seconds for its database wakes nobody; again once per window while it fails; and once more when it passes again |
degraded |
What a feature reports with report_degraded (below), once per window per feature and message |
The readiness alerts come from the probes: the framework tells the feature
which checks failed after every /readyz that something asks for. In a
deployment that is the container's healthcheck, which probes it every few
seconds. A product without one, or with the probe turned off, gets no
readiness alerts.
The same alert goes out once per OBSERVABILITY_ALERT_DEDUP_MINUTES (60),
and the next one says how many were held back. The processes of a
container agree on it through small files in OBSERVABILITY_STATE_DIR (a
directory of the temporary one by default), under a lock, so the four
workers of a gunicorn do not send four copies.
Alerts never go from the request: a queue drained by a thread of the
process sends them, with an app context of its own, so a slow homeserver
delays no page, and an error in a request whose database session is broken
does not reach the mail outbox through that session. The framework's task
runner is not used on purpose: a task waits for the commit of the request
that queued it, which an unhandled error rolls back, and in rq mode for
Redis, which may be what is down.
Channels
- email, to
OBSERVABILITY_EMAIL_TO, through splent_feature_mail's MailService: a mail the server refused is retried and shows on/admin/mail. It needs the database, so during an outage of it Matrix and Telegram are the channels that still reach someone. - matrix: an
m.textmessage inOBSERVABILITY_MATRIX_ROOM_ID(a room id,!<id>:<server>, not an alias), posted toOBSERVABILITY_MATRIX_HOMESERVERwith the access tokenOBSERVABILITY_MATRIX_TOKENof an account that has joined the room. A bot account of its own, not a person's. - telegram: a message in
OBSERVABILITY_TELEGRAM_CHAT_ID, posted by the bot whose token isOBSERVABILITY_TELEGRAM_BOT_TOKENand which is a member of the chat.
A channel that fails is logged as a warning, with what the service answered and never the credential (the Telegram token is part of the URL it posts to, so neither the URL nor the library's message reaches the log).
For the other features: report_degraded
A feature that catches a failure of a service it depends on and carries on without it says so with the framework's API, which logs a warning and hands the report to this feature when the product has it:
from splent_framework.observability import report_degraded
try:
deposit = zenodo.create(dataset)
except requests.RequestException as exc:
report_degraded("zenodo", "Zenodo does not answer.", exc=exc)
raise
The feature never imports this one, and the call does nothing more than log without it. Keep the message the same every time the problem happens and free of personal data: it is what an alert says and what two reports share to count as one.
Settings
| Variable | Type | Default | Bind | |
|---|---|---|---|---|
OBSERVABILITY_BACKEND |
enum | none |
product (uvl) |
glitchtip, sentry, email or none |
OBSERVABILITY_DSN |
secret | secret | https://<key>@<host>/<project> |
|
OBSERVABILITY_ENVIRONMENT |
str, [A-Za-z0-9_.-]{1,64} |
deployment | Empty: production, development or testing, after the app's profile |
|
OBSERVABILITY_RELEASE |
str, [^\s/]{1,200} |
deployment | Empty: <product>@<version> |
|
OBSERVABILITY_SAMPLE_RATE |
float | 1.0 |
deployment | 0 to 1 |
OBSERVABILITY_TRACES_SAMPLE_RATE |
float | 0.0 |
deployment | 0 to 1 |
OBSERVABILITY_ALERT_CHANNELS |
list | [] |
product (uvl) |
Any of email, matrix, telegram |
OBSERVABILITY_ALERT_ON |
list | all three | product | Any of errors, readiness, degraded |
OBSERVABILITY_ALERT_DEDUP_MINUTES |
int, 1 to 10080 |
60 |
deployment | |
OBSERVABILITY_READINESS_GRACE_SECONDS |
int, 0 to 3600 |
60 |
deployment | |
OBSERVABILITY_EMAIL_TO |
list of addresses | [] |
deployment | The email backend's and the email channel's addresses |
OBSERVABILITY_MATRIX_HOMESERVER |
url | deployment | https://matrix.example.org |
|
OBSERVABILITY_MATRIX_ROOM_ID |
secret | secret | !<id>:<server> |
|
OBSERVABILITY_MATRIX_TOKEN |
secret | secret | ||
OBSERVABILITY_TELEGRAM_BOT_TOKEN |
secret | secret | ||
OBSERVABILITY_TELEGRAM_CHAT_ID |
secret | secret | ||
OBSERVABILITY_HTTP_TIMEOUT |
int, 1 to 60 |
5 |
deployment | Seconds a Matrix or Telegram request may take |
OBSERVABILITY_STATE_DIR |
str | deployment | Where the processes agree on what was alerted |
The backend and the channels are the product's decisions and are
reflected in the line's model (uvl = true), since the email ones need
splent_feature_mail. In openhubs_spl, observability has a mandatory node
of OBSERVABILITY_BACKEND with the alternative errors_glitchtip,
errors_sentry, errors_email and errors_none, and an optional one of
OBSERVABILITY_ALERT_CHANNELS with the or-group alerts_email,
alerts_matrix and alerts_telegram. product:validate selects them from
the product's values, and errors_email => mail and alerts_email => mail
make a product that mails its errors or alerts without mail fail it. The credentials and the room and chat the alerts go
to are secrets: never in a committed file, always in the deployment's
.env.
At start the feature refuses a DSN that is not one and a Matrix alias in place of a room id, anywhere. A backend or a channel the product chose without what it needs (glitchtip without a DSN, matrix without a token) stops the start in production, since errors would go nowhere while the site looks watched; elsewhere it is a warning and that one stays off, so a developer's machine runs without the deployment's credentials.
Commands
splent feature:observability status # where errors go, who alerts reach
splent feature:observability test # a test alert to every channel, now
test exits with 1 when a channel did not take the alert, so a
deployment checks its settings before it needs them. With glitchtip or
sentry it also sends a test error and prints its event id.
Enable it in a product
- Select
observability, an optional platform feature of openhubs_spl, with the nodes of its backend and channels (added bydocs/openhubs_spl.uvl.patch, see Settings). - Pin it, and decide the backend and the channels in the product's
[tool.splent.config]:OBSERVABILITY_BACKEND = "glitchtip" OBSERVABILITY_ALERT_CHANNELS = ["email", "matrix"] - Put the rest in the deployment's
.env.prod:OBSERVABILITY_DSN=https://<key>@glitchtip.example.org/<project> OBSERVABILITY_EMAIL_TO=ops@example.org OBSERVABILITY_MATRIX_HOMESERVER=https://matrix.example.org OBSERVABILITY_MATRIX_ROOM_ID=!<id>:example.org OBSERVABILITY_MATRIX_TOKEN=<access token of the bot account> - After the deploy,
splent feature:observability testin the web container.
The image needs sentry-sdk[flask] (a dependency of the feature) for
glitchtip and sentry. Without it the feature says so and keeps the errors
in the log.
What it does not do yet
The rest of OB-1 to OB-7 belongs to the framework, the product or other features:
- Errors outside a request and outside a log line at ERROR (a CLI command that raises) are not captured. A task that fails is, by the line the task runner logs.
- Synthetic probes of the outside services (Zenodo, SMTP, ORCID, TLS
expiry, OB-3) and an outside monitor of
/readyz. The probes are checks registered withregister_readiness_checkor tasks on a schedule, which the framework has no runner for yet. X-Request-IDfrom nginx to the app, the tasks and the outgoing calls,/metrics, and the 502/503/504 pages with the request id (OB-4). The framework reads the header when nginx sets it.- Consistency checks and the disk alerts (OB-5),
/admin/status(OB-6) and the browser's error beacon with the CSP'sreport-uri(OB-7). - The features that swallow errors today (confirmemail, mail, explore,
dataset) calling
report_degradedinstead (OB-2).
Develop
splent feature:test observability --neutral
splent feature:contract observability --write
No test calls GlitchTip, Sentry, Matrix or Telegram: the SDK's init and
the HTTP layer are replaced in the tests.