- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| src | ||
| .gitignore | ||
| COPYING | ||
| COPYING.LESSER | ||
| MANIFEST.in | ||
| pyproject.toml | ||
| README.md | ||
splent_feature_zenodo
The zenodo feature of a SPLENT product line.
A SPLENT feature is a Python package that a product composes with others:
its models, routes, services, templates and hooks are registered when the
product starts. What this feature provides and requires is listed in the
[tool.splent.contract] section of pyproject.toml, which
splent feature:contract --write keeps in sync with the source.
This one publishes a hub's datasets on Zenodo, each as a record with its
DOI: it is a publication provider of the dataset feature (dataset's
DepositionProvider, F5-2), next to local_publication. It owns no table
and no page: dataset drives every publication through the provider, and
never names this feature.
What it provides
ZenodoProvider(provider.py), registered with dataset'sregister_deposition_provideraszenodo, whose identifiers are DOIs.createopens a deposition,new_versiona draft of a new version with the files of the one before deleted,uploadsends the deposit's ZIP,publishmints the DOI and answers it with the concept DOI,fetch_recordreads a deposition,delete_unpublisheddeletes a draft (PB-3),discarddrops an edit left open, andupdate_metadataedits, updates and publishes again, discarding the edit when a step fails.- Its errors are dataset's provider errors (PB-4):
ZenodoValidationErroris aProviderValidationError(422),ZenodoAuthErrorandZenodoConfigurationErroraProviderAuthError(502),ZenodoUnavailableErrorandZenodoTimeoutErroraProviderUnavailable(503),ZenodoRateLimitedErroraProviderRateLimited(503, with the Retry-After Zenodo asked for). ZenodoService, the client, registered in the service locator:create_deposition,create_new_deposition,upload_zip,publish_deposition,get_deposition,get_doi,get_concept_doi,update_deposition,discard_deposition,delete_deposition,create_new_version_draft,delete_all_deposition_files,update_draft_metadata,metadata_forandbuild_metadata, plusrecord_url,is_configured()andhealth(). The provider uses the one the locator has, so a refinement of it is the one that talks to Zenodo.GET /zenodo/test, for signed-in users, which answers{"success": true|false, "messages": [...]}. The retry dialog of "My datasets" asks it before offering to publish again. It callshealth(): one read of the deposition list, a page of one, whose answer is reused for a minute. It never writes to the account.
The token
Zenodo takes a personal access token. There is no OAuth flow and no callback
URL to register. Create the token under Account, Applications, Personal
access tokens, with the deposit:write and deposit:actions scopes, in the
account that will own the hub's records. Production (zenodo.org) and the
sandbox (sandbox.zenodo.org) are separate sites, with separate accounts and
separate tokens.
ZENODO_ACCESS_TOKEN is declared as <SET>, so product:deploy asks for it
and the marketplace lists it as required. In a development .env the
placeholder counts as no token: nothing is sent, is_configured() is False
and the retry dialog says Zenodo is not set up.
The token travels in an Authorization: Bearer header, never in the URL,
which ends up in exception messages, tracebacks and proxy logs. Nothing this
feature logs contains it. It can edit every record the account ever
published, so rotate it whenever it may have been seen, and when a hub moves
to a new server.
Sandbox and production
With ZENODO_API_URL empty, the API is production Zenodo in production and
the sandbox anywhere else. Production is what the framework calls production:
SPLENT_ENV, then FLASK_ENV.
Production Zenodo is refused before anything is sent, with a
ZenodoConfigurationError that says how to proceed, unless
ZENODO_ALLOW_PRODUCTION is true, and that holds in production too.
is_configured() and health() answer False in that case. A Zenodo DOI is
permanent, and a test run once minted three, so reaching production Zenodo
has to be a decision of the deployment that makes it, not a default or a
URL. The profile cannot make it: product:build writes SPLENT_ENV=prod
into every deploy, so a staging server runs as production as well.
- The deployment that publishes the hub's real records sets
ZENODO_ALLOW_PRODUCTION=truein its own.env. Not in the product's[tool.splent.config], which every build of the product reads, the staging one included. - A staging server sets
ZENODO_API_URLtohttps://sandbox.zenodo.org/api/deposit/depositions, with a sandbox token. When it starts from a copy of the production.env, it also setsZENODO_ALLOW_PRODUCTIONback tofalseand drops the production token: the flag travels with the copy, and nothing in the code can tell.
The test suite closes the HTTP transport for every test that is not browser
driven (tests/conftest.py), so no test can reach Zenodo, whatever it
mocks.
Settings
Each one is declared, with its default, in [tool.splent.config] and typed
in [tool.splent.config_schema]. The framework resolves them into
app.config, where ZenodoService reads them. A product overrides only
those it decides.
| Setting | Default | What it does |
|---|---|---|
ZENODO_ACCESS_TOKEN |
<SET> |
The token above. Without one nothing is sent. |
ZENODO_API_URL |
empty | The deposition API. Empty means production in production and the sandbox anywhere else. |
ZENODO_ALLOW_PRODUCTION |
false |
Lets the hub reach production Zenodo, in production too. Only in the .env of the deployment that publishes real records. |
ZENODO_CONNECT_TIMEOUT |
5 |
Seconds to wait for Zenodo to accept a connection. |
ZENODO_READ_TIMEOUT |
30 |
Seconds to wait for an answer to any call but the upload. |
ZENODO_UPLOAD_TIMEOUT |
90 |
Seconds to wait for the answer once the ZIP is sent. Zenodo checks the file first, so it grows with the largest dataset. |
ZENODO_MAX_RETRIES |
2 |
Times a read, an update or a delete is sent again. |
ZENODO_KEYWORDS |
none | Keywords every deposit gets after the dataset's own tags. |
ZENODO_COMMUNITY |
empty | A Zenodo community each deposit asks to join. Its curators accept or decline each request. |
ZENODO_ACCESS_RIGHT |
open |
open, or closed for records whose files only their owner sees on Zenodo. |
Each timeout bounds one call, not a publication. See below for what that means for the worker and the proxy in front of the hub.
The generic layer names no product. A hub that wants its records found
together, or under its own community, says so in its own
[tool.splent.config]:
SITE_NAME = "uvlhub"
ZENODO_KEYWORDS = ["uvlhub"]
What a deposit says
What a deposit says comes from dataset's Deposit, which it builds from
the same fields and rules as the DataCite record of the dataset's landing
(fair_metadata.deposit_for, PB-6); test_provider.py checks the two say
the same. This feature adds what is Zenodo's own:
upload_typeis alwaysdataset. The publication type a dataset form asks for describes the article the dataset goes with, which travels as a related identifier (isSupplementTo), and only when its DOI can be read.creatorsare the dataset's authors, or a single "Anonymous" for an anonymous dataset. The anonymity is read from the dataset itself, so a caller that passesanonymous=Falsefor an anonymous dataset still deposits it anonymously.keywordsare the dataset's tags and thenZENODO_KEYWORDS, trimmed, without blanks, each once regardless of case.access_rightcomes fromZENODO_ACCESS_RIGHT,communitiesfromZENODO_COMMUNITY, and the license is the one every dataset of the hub is published under, dataset'sDATASET_LICENSE(dataset 0.6.0 and later; CC-BY-4.0 with an older dataset).- A dataset published through the hub's API carries a note that says so,
with the publishing account's id, since its author credit came from that
client. The note names the hub by
SITE_NAME, which the theme puts inapp.config, or says "the hub" without one. An anonymous record gets no note.
Timeouts, retries and errors
Every call has a timeout. A GET, PUT or DELETE that could not connect, or
got a 429, 502, 503 or 504, is sent again up to ZENODO_MAX_RETRIES times
with a doubling pause, and a Retry-After of up to ten seconds is obeyed. A
POST never is: a second create is a second deposition, and a second publish
may be a second DOI. A read that timed out is not either, since it already
waited as long as the hub will.
These bound each call, not a publication. Since dataset 0.7.0 a
publication runs in the background (the framework's task executor), and
the request waits for it at most dataset's DATASET_PUBLISH_WAIT seconds;
the paragraph below is about the run, and about a product that still runs
an older dataset. A first publication is a create, an upload, a publish and
two reads, for the DOI and the concept DOI, so with the defaults it can wait
about four minutes before any retry: four calls of 5 + 30 s and an upload of
5 + 90 s. With an older dataset that runs in the request, and a gunicorn or
nginx timeout of 120 s cuts it short: a worker killed after the publish
leaves a DOI minted on Zenodo that the hub never stored, so such a product
raises the worker's and the proxy's timeouts for the routes that publish
above that sum. With dataset 0.7.0 the request only waits, those timeouts
only have to be above DATASET_PUBLISH_WAIT, and a run stopped half way
(an RQ worker's timeout, a restart) is found by the next attempt and by
dataset's publication-reconcile, which ask Zenodo first.
Every failure is a ZenodoError. Each carries user_message, a sentence a
page or an API response can show, and http_status, the status a route of
the hub answers with; str() is the detail for the log.
| Error | When | http_status |
|---|---|---|
ZenodoUnavailableError |
No answer, or a 5xx. Worth trying later. | 503 |
ZenodoTimeoutError |
A timeout above (a kind of unavailable). | 503 |
ZenodoRateLimitedError |
429, with retry_after when Zenodo said (a kind of unavailable). |
503 |
ZenodoConfigurationError |
No token, or production outside production. Only an operator can fix it. | 502 |
ZenodoAuthError |
401 or 403: a wrong or revoked token, or one without the scopes (a kind of configuration error). | 502 |
ZenodoValidationError |
400 or 422. user_message names the fields Zenodo refused. |
422 |
ZenodoError |
Anything else, such as a 404 or an answer that is not JSON. | 502 |
ZenodoService exposes them as attributes (ZenodoService.UnavailableError,
ZenodoService.ValidationError and so on), so a caller that gets the service
from the locator catches them without importing this feature.
delete_all_deposition_files fails when a file cannot be listed or deleted,
so a new version is never published with the previous version's file next to
the new one. A publish that Zenodo accepted (202) never fails on its answer,
since the DOI is minted by then: get_doi asks for it.
Use it in a product
splent feature:add splent_feature_zenodo
Develop
splent feature:test splent_feature_zenodo
splent feature:contract splent_feature_zenodo
Release
splent feature:release splent_feature_zenodo --dry-run --bump patch
splent feature:release splent_feature_zenodo
The first command rehearses the release and changes nothing. The second publishes it to the git forge and to PyPI, and a PyPI version is permanent.