- Python 69.8%
- JavaScript 15.5%
- HTML 11.2%
- CSS 3.5%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| docs | ||
| frontend | ||
| src | ||
| .gitignore | ||
| COPYING | ||
| COPYING.LESSER | ||
| MANIFEST.in | ||
| pyproject.toml | ||
| README.md | ||
splent_feature_explore
The explore feature of a SPLENT product line.
A SPLENT feature is a Python package that a product composes with others:
its models, routes, services, templates and hooks are registered when the
product starts. What this feature provides and requires is listed in the
[tool.splent.contract] section of pyproject.toml, which
splent feature:contract --write keeps in sync with the source.
Use it in a product
splent feature:add splent_feature_explore
It requires dataset 0.10.0 or later and hubfile, and works with
elasticsearch and rating when the product has them. Its one setting,
EXPLORE_HEADER_SEARCH (a bool, true, the product's decision), says
whether the header carries its search box. In the OpenHubs line it is an
optional feature of repositorio, with explore => dataset and
explore => hubfile, since the index holds datasets and their files.
docs/openhubs_spl.uvl.patch drops explore => elasticsearch, which the
working copy of the model has done (version 15), so a light hub runs no
search node; the published version 9 still has it.
Menus
explore registers one entry in the framework's menu registry.
| Key | Section | Order | Icon | Opens | Shown to |
|---|---|---|---|---|---|
explore |
site |
10 | search |
/explore |
Everyone |
A product hides or reorders it by key in its [tool.splent.layout.menu].
The shell draws it; explore adds no sidebar hook of its own.
What it adds to other pages
- Its stylesheet, for the header search and the explore page
(
layout.head.css). - The header search box and its script (
layout.header.menu,layout.scripts). The box is a Bootstrap dropdown: it lists the datasets and files/searchfinds as the reader types, and the arrow keys go from the box to the results and back. Its icons are feather's, which the application shell brings.EXPLORE_HEADER_SEARCH = falseleaves both out, for a product whose header has the catalog's search (see below). - The "Explore more datasets" link under the top datasets of the home page,
in the
dataset.home.actionsslot that dataset's home section opens. The section names no explore page itself, so it shows no link in a product without this feature. The link's text has a Spanish translation in this feature's own catalog.
The explore page extends shell('app'), so a product that points the
application shell at another layout (SPLENT_SHELL_APP) moves it with it.
The ORCID iD icon of its results ships with the feature, in
assets/vendor/orcid.
What the index holds
The search answers anyone, so it holds what anyone may read: the datasets
dataset's access policy calls public, and their files. A draft is not
indexed, and a withdrawn dataset leaves the index with its files when
dataset sends dataset-withdrawn. The authors of a document are the ones
DataSet.public_authors() names, none for an anonymous dataset.
A dataset's description is Markdown, or the HTML inherited from uvlhub 2.12
(dataset 0.13.0, D21). The index matches its words, dataset's
description_text, in content, so neither its Markdown nor its markup is
searched for, and keeps under description the markup dataset renders and
cleans, description_html, which the explore page draws; the answer cleans
it again as it is served. With an older dataset, the stored HTML and its
text. After an upgrade of dataset, splent feature:explore reindex brings
the documents up to date.
A dataset is one document per lineage (D15). Its versions collapse into the
newest one anyone may read: the document holds that version's metadata,
lists every public version (versions, with version and
versions_count), and is dataset-<id of the first version>, so a dataset
that never had a second version keeps the document it always had. The
files indexed are those of the version shown. A version published, edited
or withdrawn indexes its lineage again, so withdrawing the newest version
shows the one before it, and a lineage nobody may read any version of
leaves the index. A vote changes only the lineage's document.
splent feature:explore reset builds the index again from the database,
in bulk; run it once after upgrading from 0.4.0, whose index held a
document per version. An index already there gets the fields it lacks the
first time a process indexes into it.
Without Elasticsearch
elasticsearch is a feature explore works with, not one it needs
(features_optional): explore asks for its ElasticsearchService by
name, and imports nothing of it. A product without it, or with
ELASTICSEARCH_ENABLED = false, has no index, and /explore, the header
search box (/search) and /api/v1/search answer from the database
(database_search.py), with the same keys and values: each result is the
document the index would hold, one per lineage at its newest public
version, and the files of that version.
- A dataset is found by the words of its title, its description, its tags and its authors (names and affiliations, none of an anonymous dataset), every word in one of them; a file by its name.
- The publication type, the tags, the dates, the year and
only_with_authorsnarrow the datasets and leave the files out, as in the index, andtypenarrows the results, not the counts. - The orders are explore's own: the newest, the oldest, and the best match, which is the datasets whose title holds every word first. The datasets come before the files.
- The facets' ranges and the orders a feature registers (rating's most liked) are values of the documents, which no column holds: the page offers none of them, and the API answers a 400 naming each one asked for.
nextis an offset, which anextof the index is not.
Nothing is indexed then, and splent feature:explore reindex and reset
say there is no index. uvlhub keeps its index.
An index nothing has created yet answers no result rather than a 500
(ignore_unavailable), until the first document goes in or
splent feature:explore reindex builds it.
The search API
/api/v1/search (GET only) checks what it is asked against a declared
schema (schema.py, DC-3) before asking the index, and answers a request
that breaks it with a 400 naming each parameter:
{"error": "invalid_request", "message": "The search request is not valid.",
"fields": {"size": "Must be a whole number from 1 to 50."}}
q takes up to 200 characters, size 1 to 50, tags up to ten, the dates
are ISO, and a parameter left empty counts as not sent. The answer is
{schema_version, results, total, page, size, facets, next}: facets.type
counts the datasets and the files the other filters leave (the type a
reader picks narrows the results after them), and next is where the
following page starts (search_after), past the first 10000 results that
pages reach. Each result keeps the keys uvlhub 2.12 answered with and adds
doi, pid, pid_scheme, identifier_label, landing_path,
published_at (ISO) and size_bytes. The operation is documented in
apidocs.
A result names its identifier as it is: doi holds a DOI and nothing else
(null for an identifier of the hub's own, local_publication's), pid and
pid_scheme the identifier in any scheme, and identifier_label what the
pages call it, DOI or ID, which the result cards and the header search
show. The index keeps the landing's path, never an address: a dataset's
url, and the landing a file's dataset_doi names when its dataset has a
DOI, are built as the search answers, at the host it is asked at. An index
filled by 0.5.0 answers the same, without pid and pid_scheme, until
splent feature:explore reindex refills it.
The service is SearchService. UvlhubSearchService, its name up to
0.5.0, is the same class and stays registered under that name until the
next minor (uvl's seeder asks for it).
With words the results come in the order of the best match (relevance),
and without them the newest first. The orders are a registry too
(register_sort in facets.py): explore has the best match, the newest
and the oldest, and rating adds the most liked first. The page's "Sort by"
lists them, and its "Year" the years a public dataset was created in.
The catalog's search
explore declares the datasets as a source of the framework's search
registry (search_source.py, DC-1): one document per lineage, resolved
against the database whenever the catalog's search (splent_feature_search)
shows a hit, so a dataset withdrawn after its last reindex never shows, and
a find that searches the titles and the tags without an index. A product
that installs search gets the datasets in its box and sets
EXPLORE_HEADER_SEARCH = false.
What the page narrows by
Besides the publication type, the tags, the dates and the authors, the
explore page and /api/v1/search narrow by facets: whole numbers the index
keeps on the documents, which the features that know the files register
(facets.py), so explore names none of them. In uvlhub uvl registers the
number of UVL models of a dataset (models) and the feature model of each
file, and factlabel the number of features (features); a hub of another
kind registers its own, or none, and its page shows no such range. rating
registers the votes (likes, dislikes, net_likes) as facets without a
label, which only the result cards read, so explore names no votes and a
hub without rating shows none.
from splent_io.splent_feature_explore.facets import register_facet
register_facet(
"models", "Number of models", field="number_of_models",
dataset_value=lambda dataset: ..., order=20,
)
A facet with a label is a range of the page, and the API takes its
<key>_min and <key>_max (features_min, models_max in uvlhub, as its
clients send them). One without a label only adds its value to the hits.
The index gets the field of each facet when it is created (index_body),
and a hit on a file names its type as hubfile's registry does
(type_label, "UVL model" in uvlhub). A feature that registers a facet
asks for this release or a later one; an older explore read factlabel's
metrics itself.
Frontend
The explore page's script is assets/js/scripts.js, which the SPLENT
toolchain builds from frontend/package.json into
assets/dist/explore.bundle.js (entry explore, a classic script), loaded
by explore/index.html. It ships Mustache 4.2.0 (MIT), an npm dependency
locked in frontend/package-lock.json, which draws the result cards from
the page's x-tmpl-mustache templates; nothing of it is vendored or served
from a CDN, and it is byte for byte the copy assets/vendor held.
Bootstrap is the shell's: its Tooltip, for the cards drawn after the page
was parsed, is imported from bootstrap, a peerDependency (^5.2.0)
that becomes window.bootstrap in the bundle, so nothing of it ships. The
feather icons the cards and the header search draw are the shell's too,
read from window.feather, and the templates use the shell's CSS: both are
declared in splent.shell (icons, bootstrap-css), so product:validate
checks that the product's shell provides them. The header search's script,
assets/js/header_search.js, is a plain script served as it is, through
layout.scripts, on every page.
splent feature:compile explore # build it once, in development mode
splent feature:compile explore --watch # again on every change
splent feature:compile explore --verify # the files match assets/meta/manifest.json
splent feature:npm explore outdated # a newer Mustache?
Develop
splent feature:test splent_feature_explore
splent feature:contract splent_feature_explore
Release
splent feature:release splent_feature_explore --dry-run --bump patch
splent feature:release splent_feature_explore
The first command rehearses the release and changes nothing. The second publishes it to the git forge and to PyPI, and a PyPI version is permanent.