Encrypted, off-host backups of the database and the file volumes, with verification and a restore drill
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-01 12:16:23 +01:00
src refactor: the framework resolves backup's settings from the schema; config.py reads no environment and the contract lists every BACKUP_* key 2026-10-31 17:15:58 +01:00
.gitignore feat: scaffold splent_feature_backup with splent feature:create 2026-10-24 05:14:22 +02:00
COPYING feat: scaffold splent_feature_backup with splent feature:create 2026-10-24 05:14:22 +02:00
COPYING.LESSER feat: scaffold splent_feature_backup with splent feature:create 2026-10-24 05:14:22 +02:00
MANIFEST.in feat: scaffold splent_feature_backup with splent feature:create 2026-10-24 05:14:22 +02:00
pyproject.toml chore: release splent_io/splent_feature_backup to v0.2.0 2026-11-01 12:16:23 +01:00
README.md docs: the parameters' types and binds, and backup in the line 2026-10-25 16:25:27 +01:00

splent_feature_backup

Encrypted backups of a SPLENT product, off the host: the database and the file volumes, with one stamp, kept for as long as the retention says, a command that checks the files the database names against the disk, and a restore drill that proves a backup can be brought back.

A backup is the database, dumped with mariadb-dump --single-transaction (consistent, no lock on the site), and every storage root as a tar. Each part is gzipped and encrypted with age to the public keys you give, and sent to a directory mounted from another machine or to an S3-compatible bucket, with a manifest that lists the parts and their sha256:

20261024T030000Z-daily/
    database.sql.gz.age
    files-protected_uploads.tar.gz.age
    manifest.json

The server holds the public keys only. It writes backups it cannot read, and so does whoever reaches the destination: a restore needs one of the private keys, which stay off the server.

How a backup goes

  1. Dump the database into the staging directory, gzipped, and keep it only if mariadb-dump exited with 0 and the dump ends with its -- Dump completed trailer. Encrypt it, delete the plain copy.
  2. Archive each storage root (BACKUP_PATHS), relative to itself, the same way. Links are kept as links and never followed. A file deleted while the archive is written is counted, not fatal.
  3. Send the parts, then the manifest. A backup is complete exactly when its manifest is there: in a directory the parts are written under .<stamp>.partial and renamed into place in one step, and in a bucket the manifest is the last object.
  4. Prune: a backup older than BACKUP_RETENTION_DAYS goes, except the newest BACKUP_KEEP_MIN, so a backup that stopped running never deletes the last good ones. What an interrupted run left goes once it is a day old.

The database goes first and the files after it. Files only grow while a backup runs, so every row of the dump finds its file in the archive.

Only one backup or restore runs at a time in a container; a second one says it is waiting and waits. Every command exits with 1 when it could not do what it was asked, so cron and the autodeploy agent notice.

What is not backed up

Elasticsearch. Its index is derived from the database and the files, and is rebuilt from them after a restore (splent feature:explore reindex with the explore feature). Redis neither: sessions and queued jobs are transient.

Enable it in a product

  1. Select the feature, an optional feature of plataforma in openhubs_spl. A line that lacks it adds it from a detached workspace, and publishes the model:

    optional
        backup {org 'splent-io', package 'splent_feature_backup'}
    

    It needs no other feature. The database it backs up is the framework's.

  2. Attach it, and tell it which storage roots to archive and which rows name a file, in the product's [tool.splent.config]:

    BACKUP_PATHS = ["/workspace/<product>/protected_uploads"]
    BACKUP_VERIFY_FILES = ["Hubfile.get_full_path:size"]
    
  3. Mount the destination into the web container, in the product's docker/docker-compose.prod.yml, when it is a directory:

    volumes:
      - ${BACKUP_HOST_DIR:?mount the off-host backup disk and name it in BACKUP_HOST_DIR}:/backups
    

    and set BACKUP_DESTINATION=/backups in deploy/.env. An s3:// destination needs no mount.

  4. Generate the key pair on your own machine, never on the server, and keep the private key somewhere safe (a password manager, an offline copy). Put the public key, the age1... line, in deploy/.env.

    age-keygen -o uvlhub-backup.key
    
  5. Schedule the daily backup on the host, as the user the app runs as:

    15 3 * * * docker exec -u splent <product>_web_deploy splent feature:backup run --tag daily
    

Configuration

Declared in [tool.splent.config], with their types in [tool.splent.config_schema], so they reach deploy/.env.example. BACKUP_DESTINATION and BACKUP_AGE_RECIPIENTS are <SET>: the deployment refuses to start until they are set, and product:deploy asks for them. Every one but BACKUP_VERIFY_FILES is of deploy bind, left to each deployment's .env; BACKUP_VERIFY_FILES names the product's models, so it is decided when the product is derived. product:validate checks a value against its type: each recipient must match age1[0-9a-z]+, and each row <Model>.<path>[:<size>] must be made of Python names.

Variable Type Default Meaning
BACKUP_DESTINATION str required A directory mounted from another machine, or s3://bucket/prefix
BACKUP_AGE_RECIPIENTS list required The age public keys (age1...) every backup is encrypted to, separated by commas. Two keys (yours and a deputy's) mean either can restore
BACKUP_AGE_IDENTITY secret empty An age private key, for a restore only, set in the environment of that one command. Never in deploy/.env
BACKUP_PATHS list the protected uploads directory The storage roots archived with the database. Each should be the mount target of a named volume
BACKUP_RETENTION_DAYS int, at least 1 30 Days a backup is kept
BACKUP_KEEP_MIN int, at least 1 7 Backups kept whatever their age
BACKUP_STAGING_DIR str the system's temporary directory Where a backup is put together, and a drill unpacks. It needs room for the compressed backup twice
BACKUP_S3_ENDPOINT str empty, AWS The S3-compatible endpoint, https://...
BACKUP_S3_REGION str empty Its region, when it asks for one
BACKUP_S3_ACCESS_KEY_ID, BACKUP_S3_SECRET_ACCESS_KEY secret empty The bucket's credentials, from the environment only
BACKUP_VERIFY_FILES list empty The rows that name a file, <Model>.<path>[:<size>]

The dump connects as the app's database user, from SQLALCHEMY_DATABASE_URI. A restore into a scratch database creates one, which takes root: MARIADB_ROOT_PASSWORD, which every SPLENT deployment already has.

Commands

splent feature:backup run [--tag daily] [--json]
splent feature:backup list [--json]
splent feature:backup verify [--json]
splent feature:backup restore <stamp|latest> [--identity-file FILE] [--database NAME] [--files-dir DIR] [--keep]
splent feature:backup restore <stamp|latest> --live --yes
splent feature:backup prune

On a production server they run in the web container, as the app's user, which owns the files:

docker exec -u splent <product>_web_deploy splent feature:backup list

Checking the files the database names

verify reads every row BACKUP_VERIFY_FILES names and looks for its file, with the size the row says. <Model> is a model class of any feature, found by its name; <path> an attribute or a method without arguments that gives the path, absolute or relative to the first storage root; :<size>, optional, an attribute with the size in bytes.

  rows checked   1532
  missing        Hubfile 812: /workspace/uvlhub_splent/protected_uploads/hubfile/user_4/dataset_97/uvl/model.uvl

It exits with 1 when a row does not find its file. Run it before the first backup and after any migration of the files.

The restore drill

A backup nobody has restored is a hope. The drill restores one into a scratch database (<database>_restore) and a scratch directory, checks that every row finds its file among the restored ones and that the restored migration revisions are the manifest's, and removes both. It never touches the live database or the live files.

The private key comes from your machine for that one command, on standard input, so it never lands on the server's disk:

ssh <server> 'docker exec -i -u splent <product>_web_deploy splent feature:backup restore latest --identity-file -' < uvlhub-backup.key

--keep leaves the scratch database and files in place to look at them. Run the drill once a month, and before relying on a backup for anything. The drill checks the restored rows with the models of the version that runs, so drill a backup with the version that took it (the manifest names it).

Restoring for real

Restoring loses everything written after the backup. Stop the web container (and pause the autodeploy agent, when the product has one), then run the restore in a one-off container of the same service, which has the same volumes and environment, from the deploy bundle's directory. The private key goes from your machine on standard input, as in the drill:

ssh <server> 'docker exec <product>_web_deploy splent feature:autodeploy pause && docker stop <product>_web_deploy'
ssh <server> 'cd <product>/deploy && docker compose run --rm --no-deps -T -u splent <product>_web_deploy splent feature:backup restore <stamp> --live --yes --identity-file -' < uvlhub-backup.key
ssh <server> 'docker start <product>_web_deploy'
ssh <server> 'docker exec -u splent <product>_web_deploy splent feature:explore reindex && docker exec <product>_web_deploy splent feature:autodeploy resume'

The database is dropped, created again and imported, as root and out of the binary log. Each storage root gets the backup's files, and what was there is moved to .splent-backup-replaced-<stamp>/ inside the root, never deleted: remove it once the site looks right. Run the web container on the version the manifest names, so code and schema match, then reindex the search and resume the agent.

Every part is fetched, checked against its sha256 and decrypted before anything is replaced, so a damaged copy or a wrong key stops the restore while the target is untouched.

Security

  • The server has the public keys only. Whoever takes the server, or the destination, cannot read the backups without a private key.
  • Nothing unencrypted leaves the staging directory, and it is deleted as soon as its encrypted copy exists, on a failure too. The staging directory is 0700, its files 0600.
  • No password is on a command line: the MariaDB client reads it from MYSQL_PWD in its own environment.
  • An S3 key that may list, read, write and delete under the prefix is enough. Turn on versioning or object lock on the bucket: that is what keeps a compromised host from deleting the copies. A directory destination is only off the host if it is another machine's disk; run warns when it is on the same disk as the data, and when a storage root is not a mounted volume.

Develop

splent feature:test backup

The unit tests need no database: the dump is replaced, the destination is a directory under the test's temporary path and the keys are generated per test. The integration test runs the real mariadb-dump against the test database and restores it over itself; run it under the test lock.