- Python 90.2%
- HTML 8.4%
- JavaScript 1.1%
- Mako 0.3%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| docs | ||
| frontend | ||
| src | ||
| .gitignore | ||
| COPYING | ||
| MANIFEST.in | ||
| pyproject.toml | ||
| README.md | ||
splent_feature_uvl
The uvl feature of a SPLENT product line: the UVL
artifact type of a hub. It is everything a hub needs to hold models of the
Universal Variability Language, on top of a core that holds any file
(dataset and hubfile). uvlhub is the hub that selects it, and a hub without
it takes its own kind of files with no UVL anywhere.
A SPLENT feature is a Python package that a product composes with others:
its models, routes, services, templates and hooks are registered when the
product starts. What this feature provides and requires is listed in the
[tool.splent.contract] section of pyproject.toml, which
splent feature:contract --write keeps in sync with the source.
It merges two features, featuremodel (up to 0.4.0) and flamapy (up to 0.3.0), with what was UVL in dataset (0.5), hubfile (0.5) and explore (0.3): see Taking over featuremodel and flamapy. It is GPL-3.0-or-later, as the UVL layer is (decision D8).
Use it in a product
splent feature:add splent_feature_uvl
It requires auth, dataset 0.6.0 or later (its registries of formats and download names, and the import page it no longer serves) and hubfile 0.6.0 or later (the slots of its workbench), and uses explore 0.4.0 or later when the product has it.
The UVL model, a file type of the hub
hubfile keeps a registry of the file types a hub takes (its
artifacts.py). This feature registers the UVL model there, uvl
(artifact.py):
- the upload zone takes the extensions
UVL_EXTENSIONSnames, whatever their case, up toUVL_MAX_SIZE, and a dataset's archive keeps them in itsuvlfolder; - every model that comes in is checked, wherever from (the upload zone, a ZIP, GitHub, the import by URL, the API, a replaced file or a new version): flamapy's parser reads it, as UTF-8 with or without a BOM, and one that does not parse is refused with the line and column of what is wrong. The check is always on (UV-2);
- each model that joins a dataset gets its feature model, in the
transaction that stores it, and is counted in the dataset's
feature_model_count; a model taken out of a dataset that stays (a file removed from a draft or a new version, hubfile'sremove, from hubfile 0.8.0) takes its feature model with it and is counted off; - once the transaction commits, each model is transformed into the formats
UVL_DERIVED_FORMATSkeeps, Glencoe, DIMACS and SPLOT by default, under the model's key in hubfile's storage. A dataset's archive carries them as uvlhub did (glencoe/<folder>/model.json), and/flamapy/to_glencoe,/flamapy/to_cnfand/flamapy/to_splothand them over to whoever may read the dataset.
is_uvl(name) answers, for every place that asks, whether a file is a UVL
model.
splent feature:uvl generate [--force] [--dataset ID] # the missing formats
splent feature:uvl pending [--dataset ID] # how many are missing
The feature model
FeatureModel is one per UVL model, in uvlhub's table, feature_model,
and each file points at it through Hubfile.feature_model_id, a column this
feature adds to hubfile's table. The extension is declared
([tool.splent.refinement.extends]), and this feature's migrations own the
column. API v1 hands feature_model_id out, so the table stays while that
API does (D14).
On the dataset page
hubfile's file workbench shows any file. This feature fills the room it
leaves for a model (hooks.py):
- a model opens in the editor's
uvlmode, under the UVL logo; - the export menu offers the formats the hub makes;
- "Export to LaTeX" shows the model as a uvlhighlight listing, or hands
over a ZIP with the
.texand the uvlhighlight package (José Miguel Horcas', LPPL-1.3c, shipped with its licence) so it compiles:/hubfile/to_latex/<id>and/hubfile/to_latex_zip/<id>, uvlhub's addresses; - "Open with Flamapy IDE" opens the model in
FLAMAPY_IDE_URL.
On a file that is not a model none of it shows. On dataset's upload form, this feature's bundle checks a model's syntax in the browser as it is dropped, before the server checks it again.
Importing a model by URL
flamapyIDE sends the user to /dataset/import?import=<url> (remote_import.py).
The GET names the model and asks to confirm, and fetches nothing, so a page
that sends a signed-in user there makes the hub read nothing. The POST of
that confirmation, with the form's CSRF token, fetches the model through a
safe fetch (safe_fetch.py, UV-1):
- only from the hosts in
UVL_IMPORT_ALLOWED_HOSTS, by their exact name; - never from a host that resolves to an address that is not public (private, loopback, link-local, multicast or reserved), and the connection goes to the address that was checked, never to a second lookup;
- following no redirect;
- at most
UVL_IMPORT_MAX_SIZE, withinUVL_IMPORT_TIMEOUTseconds.
The model is staged in an upload of its own, checked as every file that comes in is, and dataset's upload form opens with it. dataset served this page, fetching on the GET, up to 0.5.
The bridge with flamapyIDE
flamapyIDE (https://ide.flamapy.org) saves a model to the hub with the
user's session, from its own site. This feature owns that bridge (UV-11,
ide.py):
GET /api/v1/auth/statustells the IDE who is signed in, with uvlhub 2.12's contract (CO-1). A visitor gets a 401 withauthenticated: false, a message naming the site and alogin_url, plussignup_urlandorcid_urlwhen the product serves those pages, the ones dataset's upload names. A signed-in user gets a 200 withuser:id,email,name,surname,affiliationandorcid, read from the profile when the product keeps them and null otherwise.- That route and dataset's
POST /api/v1/datasets/uploadanswer the sites inFLAMAPY_IDE_ORIGINSwith CORS and credentials, and no other site. CORS decides which sites may read these answers in a browser, not which ones may send a request.
The IDE's calls only carry the user's cookies if the session cookie allows cross-site requests. The framework sets it SameSite=Lax, so a browser sends it with no call from the IDE's site, and the IDE is answered as a visitor. Relaxing the cookie is not a setting to flip on its own: the upload takes the session outside CSRF, so any site could then post a model as the user.
What dataset calls the files
On a hub whose one file type is the UVL model, dataset's pages call a
dataset's files models, as uvlhub always has ("3 models", "No models in
this dataset yet"): this feature fills dataset's dataset.files.nouns
(dataset 0.12.0 or later) with the words of its workbench panel. A hub
that takes other files too keeps dataset's "files".
What explore narrows by
When the product has explore 0.4.0 or later, this feature registers two of its facets: the number of UVL models of a dataset, counted, which the explore page offers as a range, and the feature model of each file, which a hit on a file carries.
What statistics shows
When the product has statistics (0.5.0 or later), uvl fills its
dashboard's slots (dashboard.py): the number of feature models, also on
the home page's card, the models per dataset, and the datasets with the
most models. Each counts the version each dataset lineage shows, only what
anyone may read. An older statistics counted them itself, through
FeatureModelService.
Configuration
| Variable | Type | Default | Meaning |
|---|---|---|---|
FLAMAPY_IDE_ORIGINS |
list of origins | ["https://ide.flamapy.org"] |
The IDE's sites. Exact origins only: the schema refuses a * or another pattern, since these routes answer with the user's cookies |
FLAMAPY_IDE_URL |
url | "https://ide.flamapy.org" |
The IDE a model opens in. Empty draws no link |
UVL_EXTENSIONS |
list | [".uvl"] |
The extensions of a UVL model, whatever their case |
UVL_MAX_SIZE |
size | "10M" |
The largest UVL model the hub takes (0 for no limit) |
UVL_DERIVED_FORMATS |
list of glencoe, dimacs, splot |
["glencoe", "dimacs", "splot"] |
The formats every model is transformed into |
UVL_IMPORT_ALLOWED_HOSTS |
list of host names | ["ide.flamapy.org", "raw.githubusercontent.com"] |
The hosts a model may be imported from by URL. DATASET_IMPORT_ALLOWED_HOSTS in dataset up to 0.5 |
UVL_IMPORT_MAX_SIZE |
size | "10M" |
The most an imported model may have. 0 keeps the default, since an import always has a cap. DATASET_IMPORT_MAX_BYTES in dataset up to 0.5 |
UVL_IMPORT_TIMEOUT |
int, 1 to 300 |
15 |
The seconds the fetch of an imported model may take in all |
Each one has its type in [tool.splent.config_schema], which
product:validate checks a product's value against and the framework
resolves at boot. All are of derivation bind, the product's decision. A
size is written as 10M, 1G or a number of bytes, an extension with or
without its leading dot, and a host by its exact name, with no pattern.
Taking over featuremodel and flamapy
featuremodel and flamapy are retired: their last releases, featuremodel 0.4.0 and flamapy 0.3.0, stay for the products that pin them, and no product pins them together with this feature, which serves the same routes, the same file type and the same table.
This feature keeps featuremodel's tables as they are, and the id of the
revision that created them, 846acdf9eed2. Its script creates the
feature_model table and hubfiles.feature_model_id only when they are
not there, so a database featuremodel migrated is not migrated again: no
table is created twice and no row moves. This feature's version table,
alembic_splent_feature_uvl, starts empty and records 846acdf9eed2.
A product that moves from featuremodel and flamapy to this feature, in one change:
- take featuremodel and flamapy out. featuremodel's migrations are applied,
so
feature:removeasks for--force. Never roll featuremodel back (db:rollback splent_feature_featuremodelwhile it is declared): its downgrade drops the table and every feature model in it; - add this feature, hubfile 0.6.0 and the releases that follow featuremodel and flamapy out (dataset, explore, statistics);
splent db:upgrade:846acdf9eed2finds the schema and leaves it;- optionally,
splent db:rollback splent_feature_featuremodelonce it is no longer declared: the CLI sees an orphan and only removes its row fromsplent_migrations. Its version table,alembic_splent_feature_featuremodel, is left behind and harmless.
Its downgrade drops the schema only when it holds no feature model, since it cannot tell a table it created from one featuremodel created and filled.
In the OpenHubs model
uvl is a child of tipo_artefacto in repositorio, the or-group of the
kinds of file a hub takes, next to generic_files (hubfile's generic type),
with factlabel and generator as its optional children. A hub may take
both, and one that leaves uvl out takes its own kind of files with no UVL
anywhere. Its constraints are uvl => auth, uvl => dataset and
uvl => hubfile: it extends hubfile's Hubfile and asks dataset who may
read a model, so it loads after both. explore and statistics name no
artifact type, and this feature fills their registries when the product has
them, so neither requires it.
docs/openhubs_spl.uvl.patch is the change of splent_spl_openhubs's
openhubs_spl.uvl that came with this feature, applied since:
- the
uvlgroup's two packages, featuremodel and flamapy, became one node,uvl, with this package, and factlabel and generator stayed its children; featuremodel => dataset | hubfileandflamapy => dataset | hubfilebecameuvl => auth | dataset | hubfile;explore => factlabelwent: explore takes its facets from a registry;- dataset got the nodes of
DATASET_HOME_ORDER(dataset_home_order, withhome_newest,home_most_itemsandhome_most_rated), andhome_most_rated => rating.
The patch also turned statistics => featuremodel into
statistics => uvl, which a later version of the model dropped once
statistics took its slots from a registry. uvlhub validates in its six
phases with DATASET_HOME_ORDER = "items", and a product that asks for the
best rated first without rating does not.
Frontend
The bundle is assets/js/scripts.js, the check of a dropped model in the
browser: hubfile's upload zone runs it (window.SPLENT.hubfile.fileChecks)
before the server checks the file again. The SPLENT toolchain builds it into
assets/dist/uvl.bundle.js (entry uvl, a classic script), which dataset's
upload form loads after hubfile's bundle (dataset.form.scripts).
uvl-parser 0.5.0 is an npm dependency in frontend/package.json, with the
antlr4 4.13.2 it brings, locked in frontend/package-lock.json, and
nothing of them is vendored any more. Both import Node's fs, which the
browser has not got: frontend/webpack.extra.js resolves it to
assets/js/no-fs.js, which says nothing is a file, so uvl-parser parses
the text it is given and logs nothing (its own fs.statSync probe threw and
logged an error on every model, which the copy vendored here had been
patched to keep quiet).
Neither package carries its licence file on npm, so the feature keeps their
repositories' in frontend/licenses/ and names them in
splent.licenseTexts, and the release puts them in third_party.json.
uvl-parser 0.5.0 declares no licence id either: it ships by a
splent.licenseExceptions entry, with its reason and an expiry, until
upstream declares the LGPL-3.0 its repository's LICENSE is.
The workbench's uvl mode is an inline script of
hooks/uvl_workbench_scripts.html that defines it on hubfile's CodeMirror,
which this feature declares it consumes (splent.consumes, CodeMirror
^5.65.16 from hubfile), so product:validate checks the product's hubfile
provides it. The templates use the shell's Bootstrap, its CSS and its
feather icons, declared in splent.shell.
splent feature:compile uvl # build it once, in development mode
splent feature:compile uvl --verify # the files match assets/meta/manifest.json
splent feature:npm uvl outdated # a newer uvl-parser?
Third-party files
Besides what the bundle ships, a few files of others live in
assets/vendor, each with its licence, which the wheel carries too
(MANIFEST.in and [tool.setuptools.package-data]).
| Files | Version | Licence | Used by |
|---|---|---|---|
uvlhighlight |
1.3 | LPPL-1.3c | The LaTeX export's ZIP, so its .tex compiles. A missing package is a 500, not a ZIP that does not compile. Declared in splent.vendored, with the sha256 of its .sty, so the release checks it and lists it in the SBOM; LPPL is a licence the release gate names for review |
uvl/uvl_logo.svg |
See below | The header of the workbench's panel on a UVL model |
The UVL logo is the one uvlhub drew from its own static folder. It comes
with no licence statement, and uvlhub's repository declares none, so it is
shipped as the Universal Variability Language's mark, to name the language
in the panel, and its terms are still to be confirmed with the UVL authors.
The "Open with Flamapy IDE" link draws the IDE's logo from the product's
static folder (media/logos/flamapy_ide.svg), as flamapy did.
The Python side is flamapy 2.5.0 and its parsers (GPL-3.0-or-later and
their own licences), pinned in pyproject.toml.
Develop
splent feature:test splent_feature_uvl
splent feature:contract splent_feature_uvl
Release
splent feature:release splent_feature_uvl --dry-run --bump minor
splent feature:release splent_feature_uvl
The first command rehearses the release and changes nothing. The second publishes it to the git forge and to PyPI, and a PyPI version is permanent.