No description
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-08 13:06:52 +01:00
docker images: pinned by version and digest, so every start runs the same bytes 2026-11-08 12:14:19 +01:00
src feat: elasticsearch declares its connection settings in [tool.splent.config_schema], the password and the API key as secrets, and config.py reads no environment 2026-10-31 17:23:48 +01:00
.gitignore fix: ship the compiled translations, so a released feature is actually translated 2026-07-30 18:15:07 +02:00
COPYING License under LGPL-3.0-or-later 2026-10-11 11:47:40 +02:00
COPYING.LESSER License under LGPL-3.0-or-later 2026-10-11 11:47:40 +02:00
MANIFEST.in fix: ship the compiled translations, so a released feature is actually translated 2026-07-30 18:15:07 +02:00
pyproject.toml chore: release splent_io/splent_feature_elasticsearch to v0.3.1 2026-11-08 13:06:52 +01:00
README.md fix: keep the elastic password off the healthcheck's command line 2026-10-23 17:46:45 +02:00

splent_feature_elasticsearch

Reusable Elasticsearch infrastructure for SPLENT products (service archetype). Ships a single-node Elasticsearch 8.x container and a generic, domain-agnostic ElasticsearchService that any feature can consume through the service locator.

What it provides

  • ElasticsearchService (registered in the service locator): available(), index_name(name), ensure_index(name, mappings=None, settings=None), index_document(index, id, body), bulk_index(index, documents), delete_document(index, id), delete_by_query(index, query), refresh_index(name), search(index, query_body), delete_index(name), alias_targets(alias), point_alias(alias, index), health().
  • Docker: docker/docker-compose.dev.yml (host port published, security off) and docker/docker-compose.prod.yml (no port published, security on), both with a healthcheck and a named elasticsearch_data volume.
  • CLI: splent feature:elasticsearch status.

Security

The production node runs with security on and publishes no port: the app reaches it by service name on the product's network, and nothing outside the host reaches it at all. The node takes ELASTICSEARCH_PASSWORD as the password of its elastic superuser, and the app signs in as ELASTICSEARCH_USERNAME with that same password. docker/.env.prod.example ships it as <SET>, so product:deploy generates one and docker compose up refuses to start until it is set. HTTP stays plain inside the product's network, which is the only place it travels.

The node reads the password each time its container is created, so changing it in .env and running docker compose up -d rotates it. That stops holding once someone sets the elastic password through the security API, which then wins over the variable. The healthcheck signs in with the same password, which curl takes from its environment, so it never shows on a command line.

Still open: the app signs in as elastic, the node's superuser, and the product's web container gets the whole .env, that password included. A leak from that container gives control of the whole node (security, snapshots, settings), not only of the product's <prefix>_* indices. The fix is a role limited to those indices and a user or API key of the app's own, with the elastic password left to the operator; the feature does not create them yet. A key made by hand goes in ELASTICSEARCH_API_KEY, and it needs the monitor cluster privilege besides the index ones, because available() and status ask the node who it is.

The development node runs without security, like the rest of the development stack, and ELASTICSEARCH_PASSWORD stays empty there. The CI test job runs the development setup too.

ELASTICSEARCH_API_KEY, when set, wins over the user and the password. That is how a product talks to a cluster it does not run: set it together with ELASTICSEARCH_URL. Credentials never appear in a message the service writes or in what status prints, even when they are written into the URL.

splent feature:elasticsearch status tells a node that does not answer (unreachable) from one that answers and refuses (refused), which after turning security on almost always means the credentials:

  URL:     http://splent_feature_elasticsearch:9200
  Auth:    user elastic
  Enabled: yes
  Indices: uvlhub_splent_*
  Status:  refused
  Reason:  Elasticsearch at http://splent_feature_elasticsearch:9200 rejected the request: AuthenticationException(401, ...)

One node, many products

Each product runs its own node, with its own data volume, on its own network. Index names are still a flat, node-wide namespace, though, and a product may be pointed at a node somebody else runs: two products that both index pages there would overwrite each other's documents and answer searches with each other's material.

Every index name handed to the service is prefixed with ELASTICSEARCH_INDEX_PREFIX, whose shipped value __PRODUCT__ is resolved to the product name by product:env --merge and product:build. A feature keeps using its own short name and never has to know.

es.ensure_index("pages")          # creates innosoft_app_pages
es.index_name("pages")            # "innosoft_app_pages"

splent feature:elasticsearch status prints the namespace in use, which is the quickest way to check that a product is not sharing indices with its neighbour.

An index is a cache and never the authority on visibility. This service stores and returns whatever documents it is given; whether a particular reader may see a particular hit is decided at request time by the feature that owns the material.

Analyzers and mappings

ensure_index takes the index settings as well as the mappings. A mapping that names a custom analyzer needs it in the settings at creation, because an open index takes no new analyzers:

es.ensure_index(
    "pages_v1",
    settings={"analysis": {"analyzer": {"folded": {
        "type": "custom", "tokenizer": "standard",
        "filter": ["lowercase", "asciifolding"],
    }}}},
    mappings={"properties": {"title": {"type": "text", "analyzer": "folded"}}},
)

An index that already exists is left as it is, whatever either argument says.

Deleting an index to build it again with a new mapping leaves search empty until the reindex ends. Instead, build each version under a name of its own and reach it through an alias, which point_alias moves in one atomic step:

es.ensure_index("pages_v2", mappings=MAPPINGS, settings=SETTINGS)
es.bulk_index("pages_v2", documents)
es.refresh_index("pages_v2")
for old in es.point_alias("pages", "pages_v2"):
    es.delete_index(old)

Readers and writers keep using pages throughout. point_alias returns the indices the alias pointed at before, and alias_targets("pages") says where it points now.

An index created before versioning holds the alias's own name, and an alias cannot share it. point_alias("pages", "pages_v1") refuses in that case; replace_index=True deletes that index in the same atomic step, once the new one is built.

Graceful degradation and typed errors

Elasticsearch is a soft dependency. available() never raises, and every other operation raises ElasticsearchError, never a raw client exception. Two subclasses say what happened and what to do about it.

available() is False both when the node does not answer and when it answers and refuses, and a consumer falls back either way. The second is almost always wrong credentials, so it is logged as a warning, at most one every five minutes per node, while a node that is down is only logged at debug.

Exception Meaning Usual response
ElasticsearchUnavailable Disabled, or the node could not be reached. Fall back to a slower search.
ElasticsearchRequestError The node answered and refused the request (malformed query, rejected mapping, missing permission or credentials). Let it surface. Retrying will not help.
from splent_framework.services.service_locator import service_proxy
from splent_io.splent_feature_elasticsearch.services import ElasticsearchUnavailable

es = service_proxy("ElasticsearchService")

try:
    results = es.search("posts", {"query": {"match": {"title": query}}})
except ElasticsearchUnavailable:
    results = fallback_sql_search(query)

Bulk indexing

A full reindex is thousands of documents, and one HTTP round trip each turns a maintenance command into something nobody runs. bulk_index consumes any iterable lazily, sends batches through the _bulk API and reports what happened, so a single bad document does not cost you the other 9999.

report = es.bulk_index(
    "pages",
    ({"id": page.id, "body": {"title": page.title, "body": page.text}} for page in pages),
)
es.refresh_index("pages")
# {"indexed": 9999, "failed": 1, "errors": [{"id": 42, "error": "..."}]}

Only a failure of the request itself raises. Per-document failures come back in the report, with the error list capped so a broken mapping cannot flood a log.

Deleting by query

delete_by_query(index, query) deletes every document the query clause matches and returns how many went:

es.delete_by_query("pages", {"term": {"dataset_id": 7}})   # 12

query is the clause itself, not a whole search body. A document changed while the deletion runs is skipped rather than failing the rest, the deletion is visible to the next search unless refresh=False, and a missing index deletes nothing without raising.

Configuration

Env var Default Purpose
ELASTICSEARCH_URL http://localhost:9200 Node URL. The Docker .env sets http://splent_feature_elasticsearch:9200 for container-to-container access.
ELASTICSEARCH_ENABLED true Feature kill switch. When false all operations degrade immediately.
ELASTICSEARCH_INDEX_PREFIX the product's name Namespace for this product's indices. Declared as __PRODUCT__, resolved to the product name; with neither a prefix nor SPLENT_APP the service refuses to name an index.
ELASTICSEARCH_USERNAME elastic User the app signs in as when a password is set.
ELASTICSEARCH_PASSWORD empty Password of that user, and of the production node's elastic user. Empty in development; <SET> in docker/.env.prod.example.
ELASTICSEARCH_API_KEY empty API key for a cluster this product does not run. Wins over the user and the password.
ELASTICSEARCH_HOST_PORT 9201 Host port published in development only.
ELASTICSEARCH_JAVA_OPTS -Xms512m -Xmx512m Heap for the production node (docker/.env.prod.example).

ELASTICSEARCH_ENABLED, ELASTICSEARCH_INDEX_PREFIX and ELASTICSEARCH_USERNAME are declared in [tool.splent.config]. The URL and the host port, which name the feature's container, live in docker/.env.example; production reads docker/.env.prod.example instead, which asks for the password and carries the heap.

Upgrading from 0.1

  • A production deployment needs ELASTICSEARCH_PASSWORD. product:deploy asks for it and offers a random one; on a host without the CLI, set it in deploy/.env before docker compose up. An existing data volume keeps its indices: the node takes the password on the next start.
  • health() reports auth, status and error besides what it did.
  • available() calls the client's info() instead of ping(). A test that mocks the client and sets ping() has to set info() now.
  • _client() and _translated_errors() stay as they were, but they were never public: ensure_index(settings=...), delete_by_query and the aliases cover what a consumer reached them for.

Tests

splent feature:test splent_feature_elasticsearch

Unit tests mock the client boundary, so no network access is required.