- Python 82.2%
- HTML 8.9%
- JavaScript 6.3%
- CSS 2.6%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| docs | ||
| frontend | ||
| src | ||
| .gitignore | ||
| COPYING | ||
| COPYING.LESSER | ||
| MANIFEST.in | ||
| pyproject.toml | ||
| README.md | ||
splent_feature_dataset
The dataset feature of a SPLENT product line.
A SPLENT feature is a Python package that a product composes with others:
its models, routes, services, templates and hooks are registered when the
product starts. What this feature provides and requires is listed in the
[tool.splent.contract] section of pyproject.toml, which
splent feature:contract --write keeps in sync with the source.
Use it in a product
splent feature:add splent_feature_dataset
It requires auth 1.23.0 or later, and names no file type and no publisher:
its files come from hubfile (0.5.0 or later), its publication from a
provider (zenodo, local_publication), and it works with apikeys, citation,
curation, orcid, profile, public and signup when the product has them. In
the OpenHubs line it is a mandatory feature of repositorio, with
dataset => auth; hubfile, uvl, zenodo, local_publication, curation,
dataset_api, rating, explore, statistics and citation require it.
Configuration
Every setting has a working default, declared in [tool.splent.config].
A product overrides the ones it decides in its own [tool.splent.config].
| Variable | Type | Default | Meaning |
|---|---|---|---|
DATASET_PUBLISHER_NAME |
str | "" |
Publisher, provider and data catalog named in the FAIR metadata (citation's JSON-LD, DataCite, RDF and meta tags, and the deposit), and the site the mails and API messages name. Empty uses SITE_NAME, and without one the catalog's host name |
DATASET_CATALOG_URL |
url | "" |
Address of that catalog. Empty uses the site's own external address: the one the request reached, or PREFERRED_URL_SCHEME and SERVER_NAME outside a request |
DATASET_QR_LOGO |
str, an image path (.png, .jpg, .gif, .webp) |
"" |
Image in the product's static folder drawn in the middle of the PNG and JPG QR codes, for example media/logos/logo.png; citation draws it unless its CITATION_QR_LOGO names another. Empty draws none |
DATASET_LICENSE |
enum | "CC-BY-4.0" |
The license every dataset is published under, as an SPDX identifier: CC-BY-4.0, CC-BY-SA-4.0 or CC0-1.0. The FAIR metadata, the landing page and the deposit name it |
DATASET_HOME_ORDER |
enum (uvl) |
"newest" |
Which datasets the home section puts first: newest, items (those that hold the most items, the count the file type keeps on the dataset: uvl's UVL models) or rating (the best rated, which the rating feature provides) |
DATASET_PUBLICATION_PROVIDER |
str, [a-z][a-z0-9_]{0,31} |
"" |
The provider a publication goes through when its request names none (zenodo, local_publication). Empty: the only one installed |
DATASET_PUBLISH_WAIT |
int, 0 to 3600 |
100 |
Seconds a request that publishes waits for the record before it answers that it is still under way (D5). Keep it under the worker's and the proxy's timeouts. A deployment's setting |
DATASET_PUBLISH_MAX_IN_FLIGHT |
int, 0 to 100 |
3 |
Publications of one account under way at once; one more is answered 429. 0 for no limit. A deployment's setting |
DATASET_DEPOSIT_CONTENT |
enum | "sources" |
What a deposit carries: sources, the dataset's files, or sources_and_derived, also what is derived from them, waited for at most DATASET_PUBLISH_WAIT seconds |
DATASET_PUBLISH_POLICY |
enum (uvl) |
"open" |
Who publishes (PB-8): open, any owner; verified, an owner whose address is confirmed; curated, that, and a review of the accounts the reviewer does not trust. curated needs a reviewer (curation), or the product does not start. See "Who publishes" |
DATASET_PUBLISH_PAUSED |
bool | false |
Holds every publication, whatever the policy: a cutover. API v1 answers 423 publication_paused, the web sends the dataset to review. A deployment's setting |
DATASET_PUBLISH_PAUSED_RETRY_AFTER |
int, 1 to 604800 |
3600 |
The seconds the Retry-After of a paused publication says. A deployment's setting |
DATASET_DESCRIPTION_ALLOWED_TAGS |
list | ["p", "br", "hr", "h4", "h5", "h6", "blockquote", "pre", "ul", "ol", "li", "table", "thead", "tbody", "tr", "th", "td", "a", "strong", "em", "b", "i", "u", "s", "del", "code"] |
The elements a description keeps, Markdown or inherited HTML (see "The description"). The schema's choices are the safe ones (descriptions.SAFE_TAGS: no script, style, iframe or form ever); img is among them, off by default |
DATASET_ORCID_CHECK_URL |
url | "https://pub.orcid.org/v3.0" |
Where whether a credited ORCID iD exists is asked, after the credit is saved, in the background (DL-5). Empty asks nothing: a deployment that cannot reach orcid.org. A deployment's setting |
[tool.splent.config]
DATASET_PUBLISHER_NAME = "Model Archive"
DATASET_CATALOG_URL = "https://models.example.org"
DATASET_QR_LOGO = "media/logos/logo.png"
DATASET_HOME_ORDER = "items"
Each one has its type in [tool.splent.config_schema], which
product:validate checks a product's value against and the framework
resolves at boot. DATASET_HOME_ORDER has uvl = true: the SPL's model
holds a node per order (home_newest, home_most_items, home_most_rated),
the product's value selects one, and the model asks
home_most_rated => rating. So has DATASET_PUBLISH_POLICY
(publish_open, publish_verified, publish_curated), and the model asks
publish_curated => curation. An order no feature of the product provides
shows the newest, with a warning (home.py, whose register_home_order is
how a feature provides one).
The import of a model by URL and its DATASET_IMPORT_ALLOWED_HOSTS and
DATASET_IMPORT_MAX_BYTES went to the uvl feature in 0.6.0, as
UVL_IMPORT_ALLOWED_HOSTS and UVL_IMPORT_MAX_SIZE, and the GET of
/dataset/import no longer fetches anything (UV-1). The deletion window of
§5.5 is gone with DL-2: a draft is deleted whatever its age, and a published
dataset is withdrawn, never deleted.
Pages
The upload and edit form, My datasets, the dataset page and the profile
summary extend shell('app'), so a product that points the application
shell at another layout (SPLENT_SHELL_APP) moves them with it. They name
themselves through the variable title (the profile summary through the
title block), which the shell shows as the page's title.
They are drawn with Bootstrap 5.2 classes, the feather icons of
dataset/_icons.html and assets/css/dataset.css, and need no script of
the shell but Bootstrap's: the form's steps are the feature's own
(assets/js/stepper_widget.js), and My datasets is a plain table that
answers in Bootstrap toasts. Their scripts are the feature's bundles
(Frontend). The description is a textarea of Markdown with a
preview the server renders (below), so no editor ships: TinyMCE (GPL) went
in 0.13.0, its vendored copy and the route that served its runtime under
/dataset/dist/ with it.
What the pages say of publishing and of the files comes from what the product installs, never from a product's copy of a template:
- The provider the publish step names (
publication_provider(), the first the dialog offers): itslabel, and whether itsyncs_metadata. One that does, as zenodo, reads as uvlhub always has: "Synchronized datasets" and "Unsynchronized datasets", "Sync to Zenodo and make public", "This dataset is local. It has not been synchronized with Zenodo yet". One that does not, as local_publication, reads "Published datasets" and "Drafts", "Publish", "Publish on this hub", "This dataset is a draft". A published dataset's own messages (its pending sync, the retry, "View on Zenodo", the edit form's "Published on this hub") name the provider that published it. - The identifier: "DOI" while every provider installed mints DOIs
(
publication_mints_dois()), "identifier" otherwise: My datasets' column ("Dataset DOI" or "Identifier"), the form's "mints its DOI for good", the QR dialog. A published dataset's pill, tooltip and tombstone say DOI or ID as the dataset is (DataSet.identifier_label()). - The files: "2 files", unless the feature that knows them names them
(
dataset.files.nouns): uvl's are models, as uvlhub's always were. The form's "Upload UVL files" is hubfile'shubfile_files_noun.
Each of these strings is translatable, with the Spanish catalog.
The addresses that named Zenodo's sync have neutral ones beside them: a
draft's page is /datasets/<id>/draft/ (dataset.view_draft), which the
pages link, and /datasets/unsynchronized/<id>/ answers the same; the
publish step answers at /datasets/<id>/publish (dataset.publish_dataset)
and at /datasets/sync/<id>. My datasets still posts to the second: nginx
gives it the long timeout a publication waits for
(NGINX_LONG_REQUEST_TIMEOUT), and the first once nginx's location names
it (docs/nginx-publish-timeout.patch). The retry of a metadata change, /datasets/retry-sync/<id>, is only
offered for a provider that syncs metadata, so it keeps its name.
Menus
dataset registers a section of its own, datasets ("Datasets", order 110,
for signed-in accounts, after the framework's main), and two entries in
it in the framework's menu registry.
| Key | Order | Icon | Opens | Current on |
|---|---|---|---|---|
dataset_upload |
10 | upload |
/datasets/upload |
The upload |
dataset_list |
20 | folder |
/datasets/list |
My datasets and the edit form |
An entry whose page the product does not serve is left out. A product
hides or reorders either by key in its [tool.splent.layout.menu]. The
feature fills no sidebar slot: a shell that draws its menus from slots
shows no Datasets section until it draws the registry.
The description (D21, DS-1)
A dataset's description is Markdown, and the HTML uvlhub 2.12 stored is
kept as it is (descriptions.py). The column ds_meta_data.description
holds the text, and description_format what it is:
| Format | What writes it |
|---|---|
markdown |
The upload and edit forms, uvl's import, API v1's description_markdown, and the models by default |
html_legacy |
The migration from 2.12 (TinyMCE's HTML), and API v1's description from a client of 2.12, cleaned on the way in. A row written without the format is this one: the column's default in the database |
- One sanitiser for every path. nh3 with the allowlist
DATASET_DESCRIPTION_ALLOWED_TAGScuts both formats down wherever a description goes: the dataset page and the home section (ds_meta_data.description_html, Markup), the forms' preview, API v1'sdescription, and the deposit a provider gets. Scripts, styles, iframes and templates go with what they hold; any other element it does not name leaves its text; links keep http, https and mailto addresses and getrel="nofollow ugc noopener"; no class, id or style on anything. A Markdown heading starts at h4, under the page's own headings, and a file name (model.uvl,README.md) is no link: only an address with its scheme is. Cleaning twice changes nothing. Markdown reads raw HTML and passes it to the same sanitiser. - The words.
ds_meta_data.description_textis the description as words on one line: what explore indexes and what the citations, the DataCite, JSON-LD and RDF records say (citation's record reads it). - The form. A textarea of Markdown (at most 16,000 characters, which
the TEXT column holds whatever the script) with Write and Preview; the
preview is
POST /datasets/description/preview(dataset. preview_description, signed in, the form's CSRF), which answers{"html"}, the description as the page will draw it, and stores nothing. - Editing an inherited description. The edit form offers its HTML as
the text to edit, and says it is kept as HTML. Sent back as it was
offered, it stays
html_legacy, untouched. Changed, it becomesmarkdown: the HTML left in it shows as before, since Markdown passes it to the same sanitiser, and what is written around it is Markdown. A new version keeps the format of the version it comes from. - API v1. The upload takes
description_markdown, Markdown, anddescription, HTML as a 2.12 client sends it, kept as inherited HTML; with both,description_markdownwins, so a client that sends both is read alike by a hub of 2.12 and by this one. The answers keepdescriptionas HTML, rendered and cleaned, and adddescription_markdown(None for inherited HTML) anddescription_formatafter the keys v1 had (CO-1). - The migration
dataset0010_description_formatadds the column, withhtml_legacyfor every row there is, and cleans the stored descriptions again with today's allowlist;splent feature:explore reindexbrings the index up to date. Its downgrade writes each Markdown description as the HTML it renders to, which a release before 0.13.0 shows, and drops the column.
The page links the publication DOI as https://doi.org/<doi> for the DOI
its field holds (DataSet.publication_url()), or as the http(s) address it
is, and draws no link for anything else. The upload and the edit forms keep
a DOI bare, whether it came as 10.1234/abcd, doi:10.1234/abcd or
https://doi.org/10.1234/abcd, take an http(s) address as it is, and
refuse anything else.
The home section's download menus send the formats ticked in them. A menu
sent with none ticked is answered with a 400 and a message, while a link to
/datasets/download/<id> without formats still downloads every format.
The life of a dataset
A dataset carries its state (status) and who it names (visibility),
and lifecycle.py is the one place that moves them, refusing the moves it
does not list. Whether a dataset is published is read from its state, no
longer from its DOI.
| Status | Means |
|---|---|
draft |
Being prepared. The upload and the edit form leave every dataset here |
submitted, approved |
Sent to review by the hub's policy, and approved by a reviewer: then it is published. Rejected, or taken back by its owner, it is a draft again |
publishing |
Its provider is minting the record: no second publication starts |
published |
Public, with its identifier: a DOI, or the hub's own |
sync_pending |
Published, with metadata changes the provider has not taken yet |
failed |
A publication that did not finish. Publishing again asks the provider first, and goes on from there |
withdrawn |
Taken down: its identifier answers with a tombstone |
visibility is public, anonymous (it names nobody) or
restricted_pending_review (held back for curation). The provider that
holds the record is provider, its record there publisher_record_id,
and what it was published under pid and pid_scheme (F5-3): a DOI
(doi, which ds_meta_data.dataset_doi holds too, for the /doi landings
and API v1), or an identifier of the hub's own (local_publication's,
local). publication_error says why the last publication did not end,
and published_at and updated_at date the rest. A row another feature
inserts without a state (a seeder, a test that gives a dataset a DOI to
publish it) takes the one its metadata implies, the rule the migrations
dataset0002_lifecycle and dataset0005_neutral_pid applied to the rows
already stored: a DOI is its identifier, and a deposition is zenodo's.
splent db:seed makes four sample datasets (seeders.py). On a hub whose
providers mint DOIs (uvlhub) they are published records as uvlhub 2.12 held
them, with the sample DOIs 10.1234/datasetN, whose files uvl's seeder
stores. On any other hub they are drafts: no false DOI, and no published
dataset without a file. Their files, and their metrics, are the seeder's of
the feature that knows them.
Publishing through a provider
A dataset is published by a provider: zenodo deposits it on Zenodo with a
DOI, local_publication publishes it on the hub under an identifier of its
own. publication.py is the interface they implement, DepositionProvider,
and a provider registers itself from its init_feature, as captcha's do
with CaptchaService:
from splent_io.splent_feature_dataset.publication import register_deposition_provider
def init_feature(app):
register_deposition_provider(app, MyProvider)
| Method | What it does |
|---|---|
create(deposit) |
Opens a draft record holding the metadata, and answers its id |
new_version(record_id, deposit) |
Opens a draft of a new version of a published record, without the files of the version before |
upload(record_id, package) |
Puts the deposit's files in the draft |
publish(record_id) |
Mints the identifier: what cannot be taken back. Answers the Record |
fetch_record(record_id) |
The record as the provider holds it: published or not, its identifiers |
delete_unpublished(record_id) |
Deletes a draft that was never published (PB-3) |
discard(record_id) |
Drops the changes of a published record open for editing |
update_metadata(record_id, deposit) |
Sends a published record's new metadata |
landing_url(pid), record_url(record_id) |
Where the record is read: on the hub (None for the /doi landing, or /records/<public_id>/ for an identifier of the hub's own), and at the provider |
deposition_id(record_id) |
The record id as the number API v1's deposition_id holds, for a provider whose ids are numbers (zenodo's); None otherwise |
health(), health_url() |
Whether it answers now, and where a page asks it: /datasets/providers/<name>/health by default, which calls health() and answers {"success", "messages"} |
syncs_metadata |
Whether its records keep a copy of the metadata elsewhere, which a change made on the hub has to reach. By default a provider that mints DOIs does (zenodo) and one of the hub's own identifiers does not (local_publication); a provider that is neither sets it |
register_deposition_provider registers the class in the service locator
under its class name, so a refinement can still replace it with
refine_service, and lists it under its name, which a dataset records. It
raises TypeError for a class that does not subclass the interface, leaves
an abstract method unimplemented, or names no valid name (small letters,
digits and _, from a letter, up to 32) or no pid_scheme, and
RuntimeError for a second class under a name already taken; the same
class registered again, as an app built again does, is accepted. A
provider imports dataset, so its contract requires it
(requires.features = ["dataset>=0.7.0"]) and the line's model says so
(zenodo => dataset), which also makes it load after dataset. dataset
never imports a provider.
What a provider is handed is neutral: a Deposit, built by
fair_metadata.deposit_for from the same fields and rules as the DataCite
record the landing serves (PB-6), and a DepositPackage, the ZIP written
from hubfile's manifest of the dataset's rows (PB-7, hubfile 0.7.0; an older
hubfile's archive otherwise). What it raises is one of the four errors of
PB-4, each with the status a route answers and the message a person reads:
ProviderValidationError (422), ProviderAuthError (502),
ProviderUnavailable and ProviderRateLimited (503, with Retry-After). A
provider never builds a response.
The OpenHubs model has publicacion as an or-group, so a product may
install several: a dataset keeps the one it reached (provider), and its
new versions and metadata go back to it; a new publication takes the one
its request names (provider), else DATASET_PUBLICATION_PROVIDER, else
the only one installed.
publishing.py's PublishDataset is the one orchestrator (PB-1): the web's
publish step, API v1's publish and new-version, and the retries all go
through it. The request claims the dataset (an UPDATE ... WHERE no second
request also wins) and hands the work to the framework's task executor
(TASKS_MODE), then waits for it at most DATASET_PUBLISH_WAIT seconds.
The run packages the files, opens the record, keeps its id (committed at
once: the handle that ties the dataset to the record), uploads and
publishes. A failure before the publish step deletes the draft it opened
(PB-3) and leaves the dataset failed; one of the publish step that does not
say whether it minted leaves the record named, and the next attempt asks
the provider first, so a record is never minted twice.
splent feature:dataset publication-reconcile [--dry-run] is what a
product runs periodically (cron, every quarter of an hour): it completes
the publications minted without the hub learning it, with their concept
DOI, lets go of the drafts of those that stopped, sends the metadata
changes that waited (sync_pending), starts the publications a reviewer
approved that could not start then (publication was paused), and lists what
it could not settle, exiting 1 when something is left.
Who publishes (PB-8, D4)
The orchestrator asks the hub's policy (policy.py) before anything
reaches a provider, whatever asks for the publication.
DATASET_PUBLISH_POLICY |
Who publishes |
|---|---|
open |
Any owner, as before. The web asks a legacy_v2 account to confirm its address first; API v1 asks nothing of the address (CO-1) |
verified |
An owner whose address is confirmed (auth's email_verified_at), on the web and in API v1 |
curated |
As verified; and an owner the reviewer does not trust sends the dataset to review (submitted) instead |
A dataset under review is published once a reviewer approves it
(DataSetService.approve_submission), from approved and in the
background, through the provider its owner chose; a rejection
(reject_submission) takes it back to a draft, and so does its owner
(/datasets/<id>/cancel-submission). While it waits nobody changes it:
what is approved is what is published.
The reviewer is a feature that implements PublicationReviewer and
registers itself with register_publication_reviewer (curation): it says
who is trusted and who reviews, and the accounts that review read what
waits for review. It hears of a dataset sent to review, and of one taken
back, through dataset-submitted and dataset-submission-cancelled. A
product whose policy is curated and that declares no reviewer does not
start, and a reviewer missing at a request refuses rather than publish.
DATASET_PUBLISH_PAUSED holds every publication, for the cutover of a hub
to a new version: API v1 answers 423 and changes nothing; the web sends the
dataset to review when there is a reviewer, and refuses with a 423
otherwise; and what a reviewer approves meanwhile waits, approved, until
the reconciliation starts it once publication opens.
API v1 (PB-2), now dataset_api's (H7)
Since 0.11.0 API v1 (/api/v1/datasets*, /api/v1/me/datasets,
/api/v1/files*) is
splent_feature_dataset_api's:
the same addresses, methods and answers, with the scopes datasets:read,
datasets:write, datasets:publish, datasets:transfer and
datasets:delete (read_dataset and write_dataset stand for them),
public reads of what is published, and tokens limited to some datasets.
This feature no longer requires apikeys.
For one minor this feature still serves v1 itself, deprecated, in a product
that has apikeys and not dataset_api (serves_api_v1), with the endpoints
dataset.api_* it always had and the scopes read_dataset and
write_dataset, and says so in the log at start. In a product with
dataset_api it serves none of them, so no rule is answered twice; in one
without apikeys there is no API. The routes go in 0.12.0: a product that
uses the API adds dataset_api.
The transfer routes (/api/v1/datasets/<id>/transfer and
/api/v1/datasets/transfers*) are no longer among them since 0.14.0: the
offers are dataset_transfer's (see "Its owner"), and dataset_api serves
those routes in a product that has both.
What follows is what v1 answers, whoever serves it.
The upload takes a UVL model on a hub with the UVL layer, as uvlhub's
always did (uvl_file or file, uvl_content or content), and on a hub
without it a file of any kind the hub takes (file or uvl_file, or
content with its filename), which hubfile's ingest takes or refuses,
HUBFILE_FILE_EXTENSIONS deciding (0.12.0). Both answer files_created
next to feature_models_created.
POST /api/v1/datasets/<id>/publish and /new-version keep v1's contract
with the clients that call them, the marketplace's uvlhub.py first (D5,
option a): 200 only with the identifier minted, after the wait; past it a
503 publication_pending with status_url, never a 4xx, which a client
reads as "nothing was minted, send it again"; 202 at once only to a request
with Prefer: respond-async (and Prefer: wait=<seconds> waits less). 409
answers a publication under way, or a key still being processed; 429 an
account with too many publications under way. The keys v1 always answered
stay, and pid, pid_scheme, provider and landing_url follow them.
What the hub's policy holds is a 423, nothing minted, with error and
code naming why and message saying it for a person:
publication_paused (with Retry-After, nothing changed: the frozen rows of
the cutover), email_unconfirmed, publication_under_review (with
dataset_id, status and status_url: published once a curator approves
it) and publication_review_unavailable. A client reads it as a refusal,
as it reads every 4xx. An open hub that is not paused never answers it.
The upload, publish and new-version take an optional Idempotency-Key
(idempotency.py): the same key and the same call get the first answer
back for a day, with Idempotent-Replayed: true, and nothing runs again;
the same key with another call is a 422. An answer that does not settle the
call (a 503, a 202, a 423) keeps nothing, so the same key sent later answers how
the publication ended.
Since 0.13.0 (D21) the upload also takes description_markdown, and the
answers say description_markdown and description_format (see "The
description").
Since 0.9.0 (H5), additive: a publish of a dataset that names no author and
is not anonymous, or that has no file, is a 400 before the provider is
called; new-version keeps every file of the version it comes from, the
upload replacing the one at its path (or the only one) and joining the rest
otherwise, and takes replaces; the upload credits an ORCID no account
here holds, as the depositor declares (D18), where it answered 400 "not
linked to any account"; and an author carries given_name and
family_name.
Who may read, publish and delete a dataset
DatasetAccessPolicy (access.py) is the one rule, which the pages, the
download (/datasets/download/<id>), the QR, the BibTeX and the REST API
of this feature ask, and so do hubfile, uvl, factlabel, rating,
explore and statistics, through the service locator
(DatasetAccessPolicy). It is also the resolver of the framework's file
access for the files owned by splent_feature_dataset.
- A published dataset that is public or anonymous is everyone's to read.
- A draft, a publication under way or failed, or a dataset held for review is its owner's and an administrator's. One waiting for review is also its reviewers' (the accounts the hub's reviewer says review).
- A withdrawn dataset is an administrator's alone.
- Only its owner changes a dataset, and not while it is being published, while it waits for review, or once it is withdrawn.
Anyone else asking for a dataset, its files or what is derived from them
is told it does not exist (404), and one who may read it but not change it
gets a 403. public_clause() and readable_clause(user) are the same rule
as SQL, for the listings, the counters and the exports.
/api/v1/me/datasets lists the token owner's datasets that are not public
yet, with their status and visibility.
Publishing mints an identifier nobody can take back (a DOI at Zenodo), so
it happens only on purpose. The upload and the edit form save a draft,
whatever type they are sent; before publication the type only says whether
the dataset names its authors: dataset_type is draft, anonymous or,
for a published one, published, and uvlhub's zenodo and
zenodo_anonymous are taken as published and anonymous. Publishing is
a step of its own: My datasets' dialog, whose POST to /datasets/sync/<id>
(or /datasets/<id>/publish) carries the form's CSRF token and the
dialog's confirmation (confirm), or the API's
/api/v1/datasets/<id>/publish.
A published dataset never goes back to draft nor changes its anonymity.
An account carried over from an older site (auth 1.23's legacy_v2
accounts, which sign in unconfirmed) confirms its address before the web
publishes, versions or re-syncs anything of it: those routes answer with
auth's confirm_email_first, a 403 with code: "email_unconfirmed" and
where to confirm, which My datasets follows. A token's routes are not
asked, so the tokens such an account holds keep working.
An anonymous dataset names its authors in no output: its authorship is
kept apart, in ds_meta_data.private_authors, which only its owner sees on
the edit form, and DataSet.public_authors() is the one source of the
authors every output names. "Name the authors", a step of its own on the
edit form of an anonymous draft, makes it a named one once its dialog is
confirmed; an edit that asks for it without confirm_named is refused.
The record an anonymous dataset is deposited with names nobody and carries
no account id, whatever the caller asks for.
Withdrawing a published dataset
A published dataset is never deleted: its DOI is permanent at the
publisher. It is withdrawn, and its DOI then answers with a tombstone, 410,
or 451 when the reason is a legal one (copyright, personal_data), with
how to cite it, why and when it went, and a link to the record at its
publisher; a client that asks for JSON-LD, Turtle, DataCite or JSON gets
the same as JSON. Its QR, BibTeX and file pages answer the same status, its
files are no longer served, and dataset-withdrawn lets the search index
drop it. An administrator withdraws from the dataset page; an operator
from the command line. Its owner asks for it with curation, from the same
page (the dataset.view.actions slot), and a curator decides from its
queue; without curation, the owner asks an administrator.
splent feature:dataset withdraw 42 --reason "Requested by its authors"
splent feature:dataset withdraw 42 --code copyright --reason "..." --yes
splent feature:dataset purge 42
purge takes only a withdrawn dataset: its files, and what was derived
from them, go, and so do its description and tags, while its tombstone
stays. A draft is deleted, whatever its age, with its metadata, authors
and files, but not a version other versions were made from. The files of
a deleted draft and of a purged dataset go to hubfile's trash
(trash/ in its storage) once the transaction commits, so a deletion that
fails keeps them. Every lookup by DOI checks that the
dataset is there: metadata a deletion left behind answers 404, and the
migration dataset0004_orphan_metadata removes what it left without a
DOI.
Hiding a published dataset
Its owner can keep a published dataset out of the listings, from My datasets
(POST /datasets/<id>/hidden, with hidden=1, and hidden=0 to bring it back).
It leaves Explore, the home page, the dashboard's figures and rankings, the API
lists, the sitemap and the ZIP of every dataset, with every version of its
lineage. A new version stays out with them.
Its address and its DOI still lead to it, so nothing at its publisher turns false, and its page asks search engines to leave it out. This is not a withdrawal, which takes a dataset down and says why.
DatasetAccessPolicy.public_clause says who may read a dataset, and
listed_clause what the hub lists. A feature that lists datasets uses the
second. dataset-hidden-changed is sent when it changes.
An account that goes
dataset listens to auth's user-deleting and lets go of the account in the
same transaction as its deletion. Its published datasets refuse the
deletion, since a DOI is permanent, and the accounts screen offers
deactivating. Its drafts go with their metadata, authors and files (the
files to the trash once the deletion is committed). Its visits and
downloads stay counted without it, it is no longer named as the API
publisher of a dataset, and the credits linked to it stay as those of a
co-author without an account. The transfer offers it made or received are
dataset_transfer's, which lets go of them on the same signal.
Creating a dataset
The upload form and the REST API create a draft the same way, in
creation.py (and so does the upload form an import by URL opens, uvl's),
and either the whole of it is there or nothing is (DL-3).
DatasetDraftreads and checks what the dataset says of itself, with the same rules wherever it comes from: the title as plain text, the description cleaned, the publication DOI a DOI or an http(s) address, the tags and the authors (AuthorValidator, see "Authors"; API v1 caps them at 25 a request), each one no longer than its column. A field that does not fit is a 400 that names it, before anything is written. The edit form checks every field the same way.CreateDatasettakes the files from hubfile's uploads: the parts of the upload the form names (upload_idandupload_parts[]), or, for the API, the content handed over, staged in an upload of its own in the same transaction. Each file was checked as its type asks when it came in (hubfile's registry of file types: a UVL model that does not parse is refused), so a refused ZIP leaves no draft behind. API v1's upload isimport_drafton a hub with the UVL layer (a UVL model, as uvlhub's) andfile_drafton one without it: a file of any kind the hub takes, its bytes and name as they came, refused as hubfile's ingest says (FileRefused, 413 for one bigger than its type takes).- One transaction makes the metadata, the dataset, its authors and its files. A dataset needs at least one file of a type the hub takes. A rollback takes the files away from the dataset and leaves the upload as it was, so the form can be sent again.
No page empties anything on a GET: the upload zone draws an upload of its
own, and one left behind expires (hubfile's uploads-cleanup).
Authors
One validator, authors.AuthorValidator, checks the credit of every channel
(DL-5): the upload and edit forms, API v1's upload, the import and a new
version. The web used to store its authors unchecked and ask orcid.org
about each ORCID on every edit, failing when it was down; only the API
cleaned the names.
- A name and an affiliation are plain text on one line, without the control and format characters that make a name render as another (the bidirectional overrides), in NFC, at most 120 characters.
- A person's names come apart,
given_nameandfamily_name, and the credit reads "Family, Given"; a credit written "Family, Given" is read apart the same way, and any other (an organization, a single name, one carried over unsplit) is kept as written, its names unknown. The forms ask for the given and the family names; API v1 takesname, or the two. - An ORCID iD is checked offline, its shape and its check digit, and kept as its 19 characters whatever it came as (its URL). The same person twice is refused: the same ORCID, or without one the same name and affiliation.
- An ORCID linked to an account here (orcid's
OrcidService: its holder signed in with it) makes the credit that account's (Author.user_id), and so does the credit the depositor gives themselves ("Add myself", or the profile's author of an upload,myself). The rest are co-authors. - A dataset names one author at least, unless it is anonymous: the forms ask for one, and a publication refuses a dataset without one before its provider is asked, which published it as "Anonymous" when nobody had chosen that (A11-08). An API draft may have none until it is published.
Whether orcid.org knows an iD is not asked on the way in. Once the credit is
saved, the iDs no account holds are looked up in the background
(DATASET_ORCID_CHECK_URL, the task check_orcid) and the answer is kept
for 30 days (dataset_orcid_check): a slow or unreachable orcid.org never
refuses a credit, and the edit form warns about an iD it does not know.
Author.position keeps the order of the credit, which the citations follow.
An author taken out of the list is deleted with it (delete-orphan, and
ds_meta_data_id NOT NULL, ON DELETE CASCADE): every edit used to leave the
authors it replaced with no metadata, and the counts of authors counted
them (A10-04). The JSON-LD and the DataCite metadata name a person by the
names apart too (givenName, familyName), and API v1's authors carry
given_name and family_name after the keys they always had.
Co-authors and their credits
The owner decided (D18 a) that co-authors without an account are published
with their name, affiliation and ORCID, on three conditions (credits.py,
DL-6):
- The depositor's declaration: crediting anyone but oneself takes the
depositor's word that the co-authors know their data is published for
good. The forms ask for it with a checkbox (
authors_declared) whenever the credit names someone new, the publish dialog of a draft that credits co-authors without it (one carried over from uvlhub 2.12) asks for it too, and API v1 takes the upload that sendsauthorsas it (its publish takesauthors_declaredfor a draft made without). The metadata keeps when and by whom (authors_declared_at,authors_declared_by_id). - A notice: an account a published dataset credits, but its owner, is told
by mail when it is published, and when a published one starts crediting
it, with where to claim.
dataset-credits-changedsays, after every change of a credit, who it added and who it took away (added,removed, each{name, orcid, user_id}), for whoever keeps the credits of an account (academic_profile's "My credits"). - A claim: anyone credited, with an account or not, asks from the dataset
page ("Report an authorship or privacy issue",
/datasets/<id>/authorship-claim) to be taken off, corrected or credited. The request goes by mail to the owner and the hub's administrators, with the claimant's address to answer; a correction is then an edit, and a takedown the withdrawal.
Nothing is sent for what uvlhub 2.12 published: its rows are written by
the migration, and no notice or signal goes for them. An account that goes
leaves its credits as those of a co-author without an account
(user-deleting).
New versions
A new version is a version draft (DL-4, lineage.py). The edit page of a
published dataset opens it ("Prepare a new version", /dataset/<id>/new-version):
a dataset of its own, a draft that points at the version it comes from,
with that version's metadata, authors and declaration, and every one of
its files, copied under keys of their own (hubfile's copy_into, with what
is derived from them). Its owner changes it as any draft: the metadata, and
its files, added from the upload zone of the edit form, replaced, and
removed (/dataset/<id>/hubfile/<hid>/remove), each one checked as its type
asks, whatever the case of its extension and with its name as written,
Unicode and all. A draft keeps one file at least, and a file at the place of
one it has is refused: it replaces that file instead.
It is published from My datasets, whose dialog says what it changes in the
files of the version it comes from (LineageService.changes: kept,
replaced, added, removed) before it asks to confirm, through the provider
of the lineage, as a version of its record. A dataset with no file is never
sent to its provider (PublishDataset.check_ready). The old new version
published the uploaded file alone, and a name in upper case or in another
script published a record with no file at all (A01-07).
API v1's new-version is the same draft, given its file and published in
one call: the upload replaces the file at its path, or the only file of a
dataset that has one (as 2.12 did), and joins the others otherwise;
replaces names the file it replaces. A version is made one at a time: a
version draft open on the web refuses the API's, and a version that fails
before anything was minted goes again.
Its owner
A lineage changes owner as a unit: LineageService.change_owner(dataset, new_owner, current_owner=None, commit=True) locks every version (SELECT ... FOR UPDATE, LineageService.lock), refuses unless they all still
belong to current_owner (the dataset's owner by default), and moves them,
with any file still in its old owner's folder (see "Where the files live").
check_owner_change says beforehand whether it could, and why not. With
commit=False the caller commits, so what it records of the move is
committed with it or not at all.
Offering a dataset to another account, and the answer to the offer, are
splent_feature_dataset_transfer's
since 0.14.0 (DL-17): the offers and their table, the inbox, the mails and
the rules. It asks change_owner for the move once the receiving account
accepts. Without it a hub has no transfers at all: this feature keeps no
offers, serves none of API v1's transfer routes, and registers no
TransferService.
The migration dataset0011_transfers_moved drops
dataset_transfer_request, which 7e2d3e88f41c made, when it holds no
offer and dataset_transfer has not taken it over yet; a table with offers
stays, for dataset_transfer to take over with them, and until a product
installs it nothing reads them and an account that made or received one
cannot be deleted. The downgrade makes the table again when it is not there.
Identifiers: public id, lineage and concept
A dataset keeps its sequential id, which the addresses, mails and records
of uvlhub 2.12 carry (D6), and has three more, which migration
dataset0009_public_ids adds and fills for every row:
public_id, a ULID (public_ids.py): the milliseconds of its creation and 80 random bits, 26 characters of Crockford's base 32, unique, given as the dataset is made. It sorts by time and cannot be guessed from another one, so no new address lets anyone walk the drafts by counting (A15-03)./records/<public_id>/is the dataset's canonical landing (CO-5): the page, the FAIR metadata by content negotiation and the tombstone, as/doi/<doi>/answers them, for a dataset that was published; a draft has no landing (404). The id may come in small letters.urls.record_url(dataset)builds it, None for a draft, andDataSetService.find_by_public_idlooks it up.lineage_root_id, the first version of its lineage, its own id for an original and its origin's root for a version, so a lineage is one query away (DL-9) and so are the votes kept on it (DM-8). It has no foreign key: an original names itself, anddataset_origin_idalready keeps a root from going while a version stands on it.concept_pid, the identifier of every version, in the scheme ofpid(Zenodo's concept DOI), compared byte for byte (utf8mb4_bin, DM-15).LineageService.store_concept_doikeeps it withds_meta_data.dataset_concept_doi, which API v1 reads.
A concept DOI answers at /doi/<concept>/ with a 302 to the newest
published version of its lineage, with Link: <...>; rel="latest-version"
(CT-4), as Zenodo resolves it; the CLI's spl:outdated follows it. A DOI
that is neither a version's nor a concept's is still a 404.
local_publication publishes a dataset under its public id (0.2.0), so
/records/<public_id>/ is the landing of its identifiers too, and it serves
no route of its own. The view that handed the ids this feature did not
know to local_publication 0.1.0's (routes.next_view) went in 0.12.0: a
product that pins local_publication pins 0.2.0 or later with it.
The downgrade drops the three columns. The public ids are kept nowhere
else: an upgrade after a downgrade mints others, and the /records/
addresses given out before lead nowhere. Came in 0.10.0.
Where the code is
DataSetService was the whole domain (A01-25). It is split by what each
part does (DL-8), and the models read no request and call no service:
| Module | What it holds |
|---|---|
creation.py |
CreateDataset and DatasetDraft: a new dataset, in one go (DL-3), and API v1's import |
descriptions.py |
The description: its two formats, the renderer, the one sanitiser and its words (D21, DS-1) |
editing.py |
DatasetCommands: the metadata and authors of a dataset, the files of a draft, its deletion, withdrawal and purge, the decisions of a review |
authors.py, credits.py |
The validator of the authors, the ORCID checks, the declaration, the notices and the claims |
lineage.py |
LineageService: the versions, version drafts and the concept DOI, and the change of owner a transfer ends in |
publishing.py, publication.py, policy.py |
PublishDataset, the one orchestrator; the providers' interface; who publishes |
downloads.py |
The formats of the menus, and the ZIPs, from hubfile's storage |
citation.py, routes_citation.py, api_export.py |
The BibTeX, the QR and the citation of a tombstone for a product without citation (deprecated since 0.13.0, see "FAIR metadata and citations"), and what API v1 answers while this feature serves it (dataset_api's schemas since 0.11.0, H7) |
routes_api.py |
API v1 as this feature served it, deprecated: only in a product with apikeys and without dataset_api, gone in 0.12.0 |
urls.py, sizes.py |
A dataset's addresses, in a request or out of one, and sizes as a person reads them |
services.py |
DataSetService: the listings and lookups, and, by their old names, what curation and local_publication ask of it |
DatasetCommands and LineageService are registered in
the service locator, so a product refines them as any service. The metrics
of new metadata (DSMetrics) are made with it, no longer in the middle of
the flush that saves it, which SQLAlchemy warned about. They are uvlhub
2.12's UVL metrics, kept for its rows: this feature reads none of them. What
a dataset holds is counted by the features that know its files, which give
explore its facets (uvl the models, factlabel the features).
DataSet.landing_url() and landing_path() are a dataset's landing;
get_uvlhub_doi() and get_uvlhub_doi_path(), uvlhub's names for them,
stay for one minor. DataSet.deposition_id() reads the record id through
its provider (publisher_record_id), the column of uvlhub 2.12
(ds_meta_data.deposition_id) only for a provider the product no longer
installs.
Where the files live
hubfile keeps them, under keys that name neither the owner nor the file
(its StorageService, which this feature asks the service locator for).
A download, the ZIP a provider gets and the ZIP of every dataset are its
archives of the dataset's files, built from their rows, as uvlhub laid them
out (in uvlhub uvl/<path>, and glencoe/, dimacs/ and splot/ for what
uvl derives); deleting a draft or purging a withdrawn dataset sends its files to
the trash; and a change of owner (LineageService.change_owner, what a
transfer ends in) is the UPDATE of the owner, after moving any file
still in its old owner's folder (the layout of hubfile 0.4) to its key, in
the same transaction. Without hubfile a dataset has no files. That storage
came in hubfile 0.5.0: requires.features_optional asks for that release
when hubfile is there.
Which files the hub takes is hubfile's registry's to say: a replaced file,
a file added to a draft and a new version are checked as their type asks
there, whatever the case of their extension, and a dataset with no file is
never sent to its provider. Copying the files into a version draft and
taking one out of a draft are hubfile 0.8.0's (copy_into, remove, which
runs the removal handlers of the file's type: uvl's lets go of the feature
model); with an older hubfile, the files are stored again from their
content and a file is deleted with its records. The download menus offer the folders of the archive, named
by the features that make them (downloads.py, register_format_label).
uvl.py's is_uvl is deprecated: the REST API v1 is UVL-shaped by its
contract (uvl_content, a model named after the title) and asks it until
the API leaves this feature (dataset_api, H7), and hubfile 0.5.0 imports it
as it loads. Every other place asks the uvl feature.
FAIR metadata and citations
How a dataset is described and cited is
splent_feature_citation's
since 0.13.0 (CT-1, CT-3, CT-4): its JSON-LD, DataCite JSON, Turtle and
RDF/XML from one record with the real dates and the relations of its
versions, its BibTeX and the rest, its QR codes cached with their ETag, and
the signposting of its landing. This feature keeps what every record is
built from (fair_metadata.py): the publisher, DATASET_PUBLISHER_NAME;
the license, DATASET_LICENSE; the identifier and the landing; the article
it supplements; the keywords; and what format its files are in, which the
file types say: the feature that knows a type registers its media type and
the specification its files conform to (fair_metadata.register_format;
uvl registers text/plain and the UVL specification), and a file of a type
nobody registered is application/octet-stream and conforms to nothing.
The deposit a provider gets is built here too (deposit_for, PB-6).
The landing (/doi/<doi>/, /records/<public_id>/) stays this feature's:
the page, the tombstone and the concept DOI's 302. What it says of the
dataset it asks of citation's CitationService when the product has it:
| The landing's | With citation | Without it |
|---|---|---|
Content negotiation (JSON-LD, DataCite, Turtle, RDF/XML, and 2.12's application/x-turtle) |
negotiate and response, with Vary: Accept |
The page |
Link |
link_header: one cite-as, the descriptions, the files, the versions |
<https://doi.org/<doi>>; rel="cite-as" |
The page's rel=canonical and rel=cite-as |
canonical_landing, /doi/<doi>/, and the DOI's address |
DataSet.landing_url() and the DOI's address |
| The page's head | The dataset.view.head slot, which citation fills: its JSON-LD, the links of its descriptions, the meta tags reference managers read |
Nothing more |
| The tombstone's citation | citation_text |
citation.citation, this feature's line |
/doi/<concept DOI> |
latest_for_concept, a 302 |
The same, from the lineage |
| The page's "BibTeX" and its dialogs | citation's "Cite", with every format and the QR | The page's own, at 2.12's address |
uvlhub 2.12's addresses of the BibTeX and the QR are frozen (CO-1), and
since 0.13.0 they are citation's, with the same methods and their views'
names under citation. (dataset.dataset_qr_by_doi is
citation.dataset_qr_by_doi): /doi/<doi>/bibtex,
/datasets/export-bibtex/<id>, /datasets/export-bibtex-content/<id>,
/doi/<doi>/qr and /datasets/<id>/qr, each with its slash where 2.12 had
one. A citation that takes them (0.2.0 and later, TAKES_V212_ADDRESSES)
serves them alone, so no rule is answered twice (serves_citation_routes).
A product without citation, or with an older one, keeps them here for one
minor, deprecated, with this feature's BibTeX and QR (routes_citation.py,
citation.py) and a warning in the log at start; they go in 0.14.0. The
pages link them through citation_endpoint(view), which names citation's
endpoint or this feature's, so the QR of the identifier pill and My
datasets' BibTeX work either way.
The JSON-LD, Dublin Core, Turtle and Link builders of fair_metadata.py
went with 0.13.0: citation's are the ones the landing serves.
build_datacite_json stays one minor, as the record zenodo's tests compare
a deposit with.
The features stacked on top
hubfile, uvl, zenodo, rating, statistics and explore require this
feature, and factlabel builds on hubfile. This feature
requires none of them, so a product line can state hubfile => dataset
without the reverse, which would leave no product it could load. It
reaches them in two ways, and both leave the page or the call working
when the feature is missing.
- Services, through the service locator (
HubfileService,StorageService…), never through an import, and the publication providers, which register themselves (publication.py). - Slots on its pages, which the features above fill from their
hooks.py.
| Slot | Where | Filled by |
|---|---|---|
dataset.home.actions |
Under the top datasets of the home section | explore, with its "Explore more datasets" link |
dataset.form.upload |
The upload step of the upload form | hubfile, with its dropzone and the import from GitHub |
dataset.form.scripts |
On the upload and edit form, ahead of this feature's bundle | hubfile, with the bundle of its dropzone |
dataset.view.meta |
About, the column at the right of the dataset page, after its versions | rating |
dataset.view.workbench |
The top of the main column of the dataset page, over the description | hubfile |
dataset.view.sections |
The main column of the dataset page, under the description | factlabel, with the fact label of the whole dataset |
dataset.view.primary_actions |
The header of the dataset page, beside Download | citation's Cite |
dataset.view.scripts |
The end of the dataset page | hubfile, citation (its Cite dialog) |
dataset.view.actions |
Manage, in About, for the dataset's owner | curation's request to withdraw, dataset_transfer's offer to another account |
dataset.view.head |
The page's head | citation, with the FAIR metadata (see "FAIR metadata and citations") |
dataset.files.nouns |
What the pages call a dataset's files, {"noun", "nouns"} (dataset_files_nouns()) |
uvl, with model and models on a hub of UVL models |
The callbacks of dataset.home.actions and dataset.form.scripts take no
argument, those of dataset.view.* get the dataset, and those of
dataset.form.upload get the files already staged for the upload (uvl's
import from a URL), as [{name, part, size}], and upload_id, the upload
they are in (None: the zone draws one). With nothing in
dataset.form.upload the upload step says the site cannot take files. What
the forms call the files ("UVL files" in uvlhub) and which extensions their
inputs take are hubfile's (hubfile_files_noun, hubfile_accept).
It announces what happens to a dataset with signals, which the features above connect to without dataset naming them:
| Signal | Sent when |
|---|---|
dataset-published, dataset-updated, dataset-withdrawn, dataset-deleted |
The lifecycle moved (explore indexes again, audit records it) |
dataset-submitted, dataset-submission-cancelled |
A dataset went to review, or its owner took it back (curation) |
dataset-viewed, dataset-downloaded |
Somebody read it, which statistics counts |
dataset-credits-changed |
A credit was added or taken away |
The upload contract
The upload step does not know who takes the files. The feature that draws
its zone also registers a provider in window.SPLENT.dataset.uploads,
which assets/js/upload_contract.js defines and the form loads before the
dataset.form.scripts slot, and the steps ask the provider only.
| Member | What it is |
|---|---|
register(provider) |
The provider says it can answer. It sends dataset:upload-ready and then dataset:upload-change on the document |
changed() |
The provider says its files changed. It sends dataset:upload-change |
provider() |
The provider, or null while none has registered |
whenReady(callback) |
Calls back with the provider once one registers, at once if one has |
STATUSES, READY, CHANGE |
The states and the names of the two events |
A provider answers files(), one {name, status, message} per file in
order, with status as text: queued, uploading, success (staged on
the server, ready for the dataset), error or canceled, and message
saying what went wrong in an error. It answers hasErrors(), true while
a file is error or canceled. Continue waits until there is a file and
every file is success, and the form's error box lists the files that
failed until they are removed. The files never travel with the form: the
provider has staged them, and its manifest(), {upload_id, parts}, says
which ones the dataset takes; the form sends them as upload_id and
upload_parts[], and the dataset takes those and nothing else.
requires.features_optional names the features it uses when they are
there and does without otherwise. orcid and signup add their sign-in and
sign-up pages to what the API upload answers a caller that is not signed
in, and profile gives the "Add myself" button of the form its author.
sign_in_links() builds those links, and a feature that sends the same
caller to sign in reuses it, so every answer names the same pages.
The upload also takes the session of a signed-in user, which is how flamapyIDE saves a model. Like the rest of the API it is outside CSRF, which holds while the session cookie is SameSite=Lax: a browser then sends it with no POST from another site. The CORS that lets the IDE post from its own site belongs to the UVL layer, the uvl feature (featuremodel from dataset 0.1.2 to 0.5).
Frontend
The pages' scripts are ES modules in assets/js, which the SPLENT
toolchain builds from frontend/package.json into three bundles, classic
scripts in assets/dist, which the blueprint's asset route serves under
/dataset/dist/, as it serves every feature's:
| Entry | Source | Loaded by | Ships |
|---|---|---|---|
dataset |
assets/js/scripts.js |
the upload and edit form, after the scripts of dataset.form.scripts, with dataset.css |
Mustache 4.2.0, SortableJS 1.15.6 and Tagify 4.38.0 (MIT), and Tagify's stylesheet |
landing |
assets/js/landing.js |
the dataset page | nothing of others |
my_datasets |
assets/js/my_datasets.js |
My datasets | nothing of others |
The three libraries are npm dependencies, locked in
frontend/package-lock.json; nothing of them is vendored or served from a
CDN, and they are byte for byte the copies assets/vendor/npm held. The
form's bundle draws an author's card from the page's x-tmpl-mustache
template with Mustache, reorders the cards with SortableJS, and types the
tags with Tagify, whose stylesheet it puts in dist/dataset.css.
Bootstrap is the shell's: Modal, Tooltip and Toast are imported from
bootstrap, a peerDependency (^5.2.0) that becomes window.bootstrap
in the bundles, so nothing of it ships, and the templates use the shell's
CSS, declared in splent.shell (bootstrap-css); product:validate
checks that the product's shell provides both. The templates run no inline
script. What a script needs from the page comes as JSON beside it: the
form's settings in #dataset-form-config, the CSRF token its requests
carry among them (assets/js/csrf.js adds it to every same-origin request
that changes something), and My datasets' words, in the page's language,
and token in #dataset-list-config.
Two scripts are plain, served as they are: assets/js/layout.js, which the
layout.scripts hook adds to every page and which needs no script of the
shell, and assets/js/upload_contract.js, the registry of the upload
providers, loaded on the form before the features that register in it.
splent feature:compile dataset # build them once, in development mode
splent feature:compile dataset --watch # again on every change
splent feature:compile dataset --verify # the files match assets/meta/manifest.json
splent feature:npm dataset outdated # a newer Mustache, SortableJS or Tagify?
Develop
splent feature:test splent_feature_dataset
splent feature:contract splent_feature_dataset
The tests register a provider of their own, tests/providers.py's
FakeProvider, whose records live in memory, which tells what it was asked
and fails where a test says (provider.fail["publish"] = ...); the
provider fixture registers it as "fake", the default, with DOIs of the
test prefix 10.5072, and puts the app's providers back afterwards. Its
FakeReviewer trusts and lets review the accounts a test names. So the
tests run the same with any provider installed, or none. An autouse fixture
also runs every test on an open hub that is not paused, whatever the
product says (uvlhub is curated), and keeps each test's files under its own
temporary directory. A provider's own tests cannot import these: the
tests package is left out of a feature's wheel, so the neutral app of a
release gate does not have it, and each provider brings its own fakes.
Release
splent feature:release splent_feature_dataset --dry-run --bump patch
splent feature:release splent_feature_dataset
The first command rehearses the release and changes nothing. The second publishes it to the git forge and to PyPI, and a PyPI version is permanent.