- Python 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| docs | ||
| src | ||
| .gitignore | ||
| COPYING | ||
| COPYING.LESSER | ||
| MANIFEST.in | ||
| pyproject.toml | ||
| README.md | ||
splent_feature_dataset_api
The REST API of the datasets and their files, /api/v1/datasets*,
/api/v1/me/datasets and /api/v1/files*, out of
splent_feature_dataset
(milestone H7 of uvlhub 3.0). The routes are the ones dataset served up to
0.10.0, at the same addresses, with the same methods and the same answers:
a client of uvlhub 2.12, the marketplace's first, notices nothing. What is
new comes beside them:
- Scopes per kind of call (AP-4):
datasets:read,datasets:write(upload),datasets:publish(publish and new-version),datasets:transferanddatasets:delete. uvlhub 2.12'sread_datasetandwrite_datasetare aliases that stand for them: a key carried over withwrite_datasetkeeps publishing. - Public reads. What is published is read without a token, a number of
calls a minute per address (apikeys'
APIKEYS_PUBLIC_READS_PER_MINUTE). - Tokens limited to some datasets. The token form and
POST /api/v1/auth/tokens("resources": {"dataset": [4]}) limit a token to some of its owner's lineages; it acts on those alone. - A resource layer (AP-1,
resource.py): each route names the scope, what it loads and who may act on it, and every refusal is v1's own. - Versioned schemas (AP-2,
schemas.py): every body is built from one, and the same schema is what the route tells apidocs it answers. - One way to page (AP-3,
pagination.py):page,per_page(1 to 100), andnextandprev. - A trail of every write (AP-10,
trail.py), with the token that made it, through the framework'srecord_event(kept by splent_feature_audit). - A written versioning policy (AP-7): docs/versioning.md.
- The upload of any file (H11): on a hub without the UVL layer,
/api/v1/datasets/uploadtakes a file of any kind the hub takes, which hubfile's ingest takes or refuses (below). - Descriptions in Markdown (D21): the upload takes
description_markdownbesides 2.12'sdescription, and the answers adddescription_markdownanddescription_format(below). - Transfers through dataset_transfer (DL-17): the five routes of the
transfers ask
splent_feature_dataset_transfer's
TransferService, and are served only in a product that has it (below).
The scopes
Each kind of call asks for its own scope, registered with apikeys from
scopes.py. Only reading is a default; the others, which change things,
are registered with default=False, so a token gets them only when its
creator asks.
| Scope | Lets a token | Default |
|---|---|---|
datasets:read |
Read datasets, their versions, files and transfers | Yes |
datasets:write |
Upload a dataset, as a draft | No |
datasets:publish |
Publish a draft or a new version, which mints its identifier | No |
datasets:transfer |
Offer a dataset to another account, and answer or take back an offer; registered only with dataset_transfer | No |
datasets:delete |
Delete a draft | No |
read_dataset |
uvlhub 2.12's, an alias of datasets:read |
Never offered |
write_dataset |
uvlhub 2.12's, an alias of the four that write | Never offered |
A token limited to some of its owner's datasets acts on those alone, a lineage at a time, and reads the rest only where anyone may.
How a route is written
Each route says what it needs to the resource layer (resource.py), which
resolves the token, loads what the route acts on, asks dataset's access
policy and the token's reach, and refuses the way v1 always did:
@dataset_api_bp.route("/api/v1/datasets/<int:dataset_id>/publish", methods=["POST"])
@trailed("publish")
@api_resource(PUBLISH, DATASET, OWNER)
def api_publish_dataset(dataset):
...
Every body is built from a versioned schema (DatasetV1, LineageV1,
TransferV1...), and the route's openapi attribute, an OpenAPI 3.1
operation as data (docs.py), hands the same schema to apidocs, so the
document cannot drift from the answer. @trailed records every write once
it is answered, whatever the answer, with the framework's record_event:
the action api.<operation> (api.publish), so the trail's screen narrows
to them with api., and the token's id and prefix, never the token.
v1 only grows: a new route, key or optional parameter. What would break a
client goes to a v2 served beside it, and each route of v1 it replaces
answers with Deprecation, Sunset once decided, and a Link to why
(docs/versioning.md). GET /api/v1/files/<id>/metadata,
which no feature can answer since uvl merged the UVL layer, is deprecated
and kept while uvlhub's frozen manifest lists it.
The upload
On a hub with the UVL layer (uvl's FlamapyService, uploads.takes_uvl),
the upload takes a UVL model, byte for byte as uvlhub's did: uvl_file or
file, or its text as uvl_content (or content), UTF-8, .uvl added to
a name without it, at most 1 MB and checked by uvl when a token sends it,
and the answer reads feature_models_created.
On a hub without it, it takes one file of any kind: file or uvl_file,
or its text as content named by filename (dataset's
creation.file_draft). Its bytes and name are kept as they came, and
hubfile's ingest decides, as for every file that comes in: its type by its
name (HUBFILE_FILE_EXTENSIONS; ["*"] takes any), its size (a 413 past
what its type takes) and the type's check. uvl_content there is a 400:
such a hub takes no UVL model. Both answer files_created besides.
The description
A dataset's description is Markdown, or the HTML uvlhub 2.12 stored, kept
as it is until its owner edits it (dataset's descriptions.py, D21). The
upload takes either:
| Key | What it is | Stored as |
|---|---|---|
description_markdown |
Markdown, the new key | markdown, as written |
description |
HTML, as a client of 2.12 sends it | html_legacy, cleaned by dataset's sanitiser |
With both, description_markdown wins, so a client that sends both is read
alike by a hub of 2.12, which knows only description, and by this one.
Each is at most 16,000 characters.
The answers (DatasetV1, OwnDatasetV1) keep description as v1 always
answered it, HTML: the description rendered and cleaned by dataset's
sanitiser, whichever its format, which a 2.12 client draws as it did. After
the keys v1 had they add description_markdown, the Markdown its author
wrote (null for inherited HTML), and description_format, markdown or
html_legacy. A client that shows the Markdown renders and cleans it as the
hub does.
The transfers
/api/v1/datasets/<id>/transfer (POST), /api/v1/datasets/transfers
(GET) and /api/v1/datasets/transfers/<id>/accept, /decline and
/cancel (POST) are transfers.py's. The offers are dataset_transfer's
since dataset 0.14.0, so they are added to the blueprint as each app
registers it (serves_transfers, serve_transfers in __init__.py) when
the product has dataset_transfer, with the endpoints they always had
(dataset_api.api_transfer_dataset_ownership...). A product without it
serves none of them, and registers no datasets:transfer scope;
write_dataset still stands for it, so a key of 2.12 transfers wherever a
hub does.
They keep 2.12's methods, statuses and keys (CO-1). DL-17 changes three things a client can see:
- An offer to an
emailwith no account, or with one that is switched off, and one to auser_idwith no account behind it, is the 202 of any offer, where 2.12 answered 404 "Target account not found" or 400 "The target account is not active.": it is kept, reaches nobody and lapses. - The
to_user_idof an offer to anemailis null for its sender until the account answers (its recipient sees its own id, and an offer to auser_idnames that id, as before), so no answer says whom an address belongs to. - An account makes
DATASET_TRANSFER_MAX_OFFERS_PER_DAYoffers a day; one more is 429transfer_limit, withRetry-After. An offer lapses, unanswered, afterDATASET_TRANSFER_EXPIRY_DAYS:cancelledfrom then on, and answering it is a 400.
Use it in a product
splent feature:add splent_feature_dataset_api
It requires apikeys 0.10.0 and dataset 0.13.0: 0.11.0 stopped serving the
API itself when this feature is installed, 0.12.0 makes the upload of
any file (creation.file_draft) and the neutral names this feature reads
(landing_url(), deposition_id()), and 0.13.0 the description's formats
(description_markdown, description_html). Without this feature dataset still
serves v1, deprecated, in a product with apikeys, without its transfers since
dataset 0.14.0. The transfers take dataset_transfer 0.1.0 or later.
The line's model gets the node and its constraints from
docs/openhubs_spl.uvl.patch: dataset_api => apikeys, auth and dataset, and dataset no longer requires apikeys. The
working copy of openhubs_spl.uvl has it, as an optional feature of
repositorio; the published version 9 does not yet. A hub without an API
leaves both dataset_api and apikeys out. It uses hubfile for the files when
the product has it, and its writes reach the trail when the product has
audit.
Settings
None of its own. apikeys' decide what a token is offered and how many calls a caller without one makes:
| Setting | Of | |
|---|---|---|
APIKEYS_SCOPES |
apikeys | An alias names what it stands for: ["read_dataset", "write_dataset"] offers the five datasets:* |
APIKEYS_DEFAULT_SCOPES |
apikeys | ["read_dataset"] grants datasets:read |
APIKEYS_PUBLIC_READS_PER_MINUTE |
apikeys | 60 by default; 0 asks every caller for a token |
APIKEYS_HEADER_ALIASES |
apikeys | ["X-API-Key"] for the marketplace's client |
DATASET_PUBLISH_WAIT |
dataset | How long publish and new-version wait for the identifier |
What moved, for a product's contract
Each rule keeps its methods; its endpoint goes from dataset.<view> to
dataset_api.<view>, the view's name unchanged (dataset.api_publish_dataset
is dataset_api.api_publish_dataset). uvl's CORS on
/api/v1/datasets/upload for flamapyIDE is by path, so it follows.
Develop
splent feature:test dataset_api --with dataset --with apikeys
splent feature:contract dataset_api
Release
splent feature:release splent_feature_dataset_api --dry-run --bump minor
splent feature:release splent_feature_dataset_api
Release apikeys and dataset first: this feature's neutral app takes their newest releases.