- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| src | ||
| .gitignore | ||
| COPYING | ||
| COPYING.LESSER | ||
| MANIFEST.in | ||
| pyproject.toml | ||
| README.md | ||
splent_feature_backup
Encrypted backups of a SPLENT product, off the host: the database and the file volumes, with one stamp, kept for as long as the retention says, a command that checks the files the database names against the disk, and a restore drill that proves a backup can be brought back.
A backup is the database, dumped with mariadb-dump --single-transaction
(consistent, no lock on the site), and every storage root as a tar. Each
part is gzipped and encrypted with age to the
public keys you give, and sent to a directory mounted from another machine
or to an S3-compatible bucket, with a manifest that lists the parts and
their sha256:
20261024T030000Z-daily/
database.sql.gz.age
files-protected_uploads.tar.gz.age
manifest.json
The server holds the public keys only. It writes backups it cannot read, and so does whoever reaches the destination: a restore needs one of the private keys, which stay off the server.
How a backup goes
- Dump the database into the staging directory, gzipped, and keep it only
if
mariadb-dumpexited with 0 and the dump ends with its-- Dump completedtrailer. Encrypt it, delete the plain copy. - Archive each storage root (
BACKUP_PATHS), relative to itself, the same way. Links are kept as links and never followed. A file deleted while the archive is written is counted, not fatal. - Send the parts, then the manifest. A backup is complete exactly when its
manifest is there: in a directory the parts are written under
.<stamp>.partialand renamed into place in one step, and in a bucket the manifest is the last object. - Prune: a backup older than
BACKUP_RETENTION_DAYSgoes, except the newestBACKUP_KEEP_MIN, so a backup that stopped running never deletes the last good ones. What an interrupted run left goes once it is a day old.
The database goes first and the files after it. Files only grow while a backup runs, so every row of the dump finds its file in the archive.
Only one backup or restore runs at a time in a container; a second one says it is waiting and waits. Every command exits with 1 when it could not do what it was asked, so cron and the autodeploy agent notice.
What is not backed up
Elasticsearch. Its index is derived from the database and the files, and
is rebuilt from them after a restore (splent feature:explore reindex with
the explore feature). Redis neither: sessions and queued jobs are
transient.
Enable it in a product
-
Select the feature, an optional feature of
plataformain openhubs_spl. A line that lacks it adds it from a detached workspace, and publishes the model:optional backup {org 'splent-io', package 'splent_feature_backup'}It needs no other feature. The database it backs up is the framework's.
-
Attach it, and tell it which storage roots to archive and which rows name a file, in the product's
[tool.splent.config]:BACKUP_PATHS = ["/workspace/<product>/protected_uploads"] BACKUP_VERIFY_FILES = ["Hubfile.get_full_path:size"] -
Mount the destination into the web container, in the product's
docker/docker-compose.prod.yml, when it is a directory:volumes: - ${BACKUP_HOST_DIR:?mount the off-host backup disk and name it in BACKUP_HOST_DIR}:/backupsand set
BACKUP_DESTINATION=/backupsindeploy/.env. Ans3://destination needs no mount. -
Generate the key pair on your own machine, never on the server, and keep the private key somewhere safe (a password manager, an offline copy). Put the public key, the
age1...line, indeploy/.env.age-keygen -o uvlhub-backup.key -
Schedule the daily backup on the host, as the user the app runs as:
15 3 * * * docker exec -u splent <product>_web_deploy splent feature:backup run --tag daily
Configuration
Declared in [tool.splent.config], with their types in
[tool.splent.config_schema], so they reach deploy/.env.example.
BACKUP_DESTINATION and BACKUP_AGE_RECIPIENTS are <SET>: the deployment
refuses to start until they are set, and product:deploy asks for them.
Every one but BACKUP_VERIFY_FILES is of deploy bind, left to each
deployment's .env; BACKUP_VERIFY_FILES names the product's models, so it
is decided when the product is derived. product:validate checks a value
against its type: each recipient must match age1[0-9a-z]+, and each row
<Model>.<path>[:<size>] must be made of Python names.
| Variable | Type | Default | Meaning |
|---|---|---|---|
BACKUP_DESTINATION |
str | required | A directory mounted from another machine, or s3://bucket/prefix |
BACKUP_AGE_RECIPIENTS |
list | required | The age public keys (age1...) every backup is encrypted to, separated by commas. Two keys (yours and a deputy's) mean either can restore |
BACKUP_AGE_IDENTITY |
secret | empty | An age private key, for a restore only, set in the environment of that one command. Never in deploy/.env |
BACKUP_PATHS |
list | the protected uploads directory | The storage roots archived with the database. Each should be the mount target of a named volume |
BACKUP_RETENTION_DAYS |
int, at least 1 |
30 |
Days a backup is kept |
BACKUP_KEEP_MIN |
int, at least 1 |
7 |
Backups kept whatever their age |
BACKUP_STAGING_DIR |
str | the system's temporary directory | Where a backup is put together, and a drill unpacks. It needs room for the compressed backup twice |
BACKUP_S3_ENDPOINT |
str | empty, AWS | The S3-compatible endpoint, https://... |
BACKUP_S3_REGION |
str | empty | Its region, when it asks for one |
BACKUP_S3_ACCESS_KEY_ID, BACKUP_S3_SECRET_ACCESS_KEY |
secret | empty | The bucket's credentials, from the environment only |
BACKUP_VERIFY_FILES |
list | empty | The rows that name a file, <Model>.<path>[:<size>] |
The dump connects as the app's database user, from
SQLALCHEMY_DATABASE_URI. A restore into a scratch database creates one,
which takes root: MARIADB_ROOT_PASSWORD, which every SPLENT deployment
already has.
Commands
splent feature:backup run [--tag daily] [--json]
splent feature:backup list [--json]
splent feature:backup verify [--json]
splent feature:backup restore <stamp|latest> [--identity-file FILE] [--database NAME] [--files-dir DIR] [--keep]
splent feature:backup restore <stamp|latest> --live --yes
splent feature:backup prune
On a production server they run in the web container, as the app's user, which owns the files:
docker exec -u splent <product>_web_deploy splent feature:backup list
Checking the files the database names
verify reads every row BACKUP_VERIFY_FILES names and looks for its file,
with the size the row says. <Model> is a model class of any feature,
found by its name; <path> an attribute or a method without arguments that
gives the path, absolute or relative to the first storage root; :<size>,
optional, an attribute with the size in bytes.
rows checked 1532
missing Hubfile 812: /workspace/uvlhub_splent/protected_uploads/hubfile/user_4/dataset_97/uvl/model.uvl
It exits with 1 when a row does not find its file. Run it before the first backup and after any migration of the files.
The restore drill
A backup nobody has restored is a hope. The drill restores one into a
scratch database (<database>_restore) and a scratch directory, checks
that every row finds its file among the restored ones and that the
restored migration revisions are the manifest's, and removes both. It
never touches the live database or the live files.
The private key comes from your machine for that one command, on standard input, so it never lands on the server's disk:
ssh <server> 'docker exec -i -u splent <product>_web_deploy splent feature:backup restore latest --identity-file -' < uvlhub-backup.key
--keep leaves the scratch database and files in place to look at them.
Run the drill once a month, and before relying on a backup for anything.
The drill checks the restored rows with the models of the version that
runs, so drill a backup with the version that took it (the manifest names
it).
Restoring for real
Restoring loses everything written after the backup. Stop the web container (and pause the autodeploy agent, when the product has one), then run the restore in a one-off container of the same service, which has the same volumes and environment, from the deploy bundle's directory. The private key goes from your machine on standard input, as in the drill:
ssh <server> 'docker exec <product>_web_deploy splent feature:autodeploy pause && docker stop <product>_web_deploy'
ssh <server> 'cd <product>/deploy && docker compose run --rm --no-deps -T -u splent <product>_web_deploy splent feature:backup restore <stamp> --live --yes --identity-file -' < uvlhub-backup.key
ssh <server> 'docker start <product>_web_deploy'
ssh <server> 'docker exec -u splent <product>_web_deploy splent feature:explore reindex && docker exec <product>_web_deploy splent feature:autodeploy resume'
The database is dropped, created again and imported, as root and out of
the binary log. Each storage root gets the backup's files, and what was
there is moved to .splent-backup-replaced-<stamp>/ inside the root, never
deleted: remove it once the site looks right. Run the web container on the
version the manifest names, so code and schema match, then reindex the
search and resume the agent.
Every part is fetched, checked against its sha256 and decrypted before anything is replaced, so a damaged copy or a wrong key stops the restore while the target is untouched.
Security
- The server has the public keys only. Whoever takes the server, or the destination, cannot read the backups without a private key.
- Nothing unencrypted leaves the staging directory, and it is deleted as
soon as its encrypted copy exists, on a failure too. The staging
directory is
0700, its files0600. - No password is on a command line: the MariaDB client reads it from
MYSQL_PWDin its own environment. - An S3 key that may list, read, write and delete under the prefix is
enough. Turn on versioning or object lock on the bucket: that is what
keeps a compromised host from deleting the copies. A directory
destination is only off the host if it is another machine's disk;
runwarns when it is on the same disk as the data, and when a storage root is not a mounted volume.
Develop
splent feature:test backup
The unit tests need no database: the dump is replaced, the destination is
a directory under the test's temporary path and the keys are generated per
test. The integration test runs the real mariadb-dump against the test
database and restores it over itself; run it under the test lock.