keepset 0.36.0 → 0.37.1

package/README.md

@@ -57,13 +57,13 @@ An alias that is a path, such as `foo/graphs/bar`, files the entry in a second g
 
 ## The entry format
 
-An entry is a JSON document in `catalogue/entries/`: a descriptor (stable id, name, description, domain, subjects, tags), the category whose normalisation target the entry resolves to, its sources (each with a retrieval date, and for a dead source a pinned Wayback snapshot that travels with the URL because the snapshot timestamp is part of the recipe's reproducibility), its licence and citations, the recipe that turns downloaded bytes into the dataset's native JSON shape, and a crossmap crosswalk mapping that native shape onto the category target. A recipe's archive is no archive, a compressed file (gzip or bzip2), a tar archive (plain, gzip or bzip2 compressed), a zip archive or an Excel workbook, and a member of an archive is selected by path, extension or pattern. The format is defined by Zod schemas, and the generated JSON Schema is published with the package as `keepset/schemas/entry.schema.json`, with the catalogue manifest's beside it. The domain is the entry's topic, its tags are the node fields it carries, its namespace is the source it comes from, and its subjects are the fields of study it is relevant to (for example `operations-research`), as slugs a search filters by (`keepset search --subject operations-research`), a listing page per subject on the site and keywords in the JSON-LD.
+An entry is a JSON document in `catalogue/entries/`: a descriptor (stable id, name, description, domain, subjects, tags), the category whose normalisation target the entry resolves to, its sources (each with a retrieval date, and for a dead source a pinned Wayback snapshot that travels with the URL because the snapshot timestamp is part of the recipe's reproducibility), its licence and citations, the recipe that turns downloaded bytes into the dataset's native JSON shape, and a crossmap crosswalk mapping that native shape onto the category target. A recipe's archive is no archive, a compressed file (gzip or bzip2), a tar archive (plain, gzip or bzip2 compressed), a zip archive or an Excel workbook, a member of an archive is selected by path, extension or pattern, and a format that is several files (the splits of a knowledge graph, the node and link files of an HGB dataset) is read from named parts of one archive. The format is defined by Zod schemas, and the generated JSON Schema is published with the package as `keepset/schemas/entry.schema.json`, with the catalogue manifest's beside it. The domain is the entry's topic, its tags are the node fields it carries, its namespace is the source it comes from, and its subjects are the fields of study it is relevant to (for example `operations-research`), as slugs a search filters by (`keepset search --subject operations-research`), a listing page per subject on the site and keywords in the JSON-LD.
 
 The catalogue itself is a manifest (`catalogue/manifest.json`: id, version, repository) plus the entries and their crosswalks. Contribution is by forking with lineage: a fork is its own catalogue, its manifest names the catalogue and version it was forked from, and entry ids stay stable across forks so two catalogues can be compared entry by entry.
 
 ## Categories and normalisation targets
 
-A category names a normalisation target: the shape every entry in that category resolves to, whatever its native shape. `graph/v1` is `{ meta, nodes, edges }` with unique node ids and edges whose endpoints name nodes; its entries are graph datasets taken from upstream collections such as SuiteSparse, Netzschleuder, SNAP, KONECT and OGB, read by one parser per format (delimited edge lists, Matrix Market, GML, Pajek, UCINet DL full matrices and Graph Drawing contest files), each giving the graph's topology. `tasks/v1` is `{ meta, nodes }`, a set of tasks as the places or cases it offers to visit or serve (a routing benchmark, or an open data register of permits and complaints), each flat with optional coordinates, prize, prize probability, duration, travel time, skill, territory, lifecycle status and times, and deadline, read from tabular sources by the `csv-table` parser (a delimited file, a member of an archive, a sheet of an Excel workbook, or an open data portal read a page at a time) and from ArcGIS feature layers by the `geojson` parser. Its entries come from a survey of open operations-research datasets, imported with a curation that records, for each source, what its download gives and how its columns map onto the node; a dataset whose source has no usable table is imported without a recipe and says why. Any category can be fetched as a collection: a recipe that selects every matching archive member with `members` (by path, by the extensions its name ends with, or by a regular expression its whole path matches) resolves to `{ meta, instances }`, where each instance is a document of the entry's category under the path of the member it came from, the schema derived from the category's own, so a category needs nothing of its own to be collected. The parser reads each member, the crosswalk is applied to each, and a member that does not parse or does not validate fails the fetch and is named. Instance files are read by one parser per format (Solomon and TOPTW time window files, Chao team orienteering files, TSPLIB style orienteering files, VRP-REP XML, VeRoLog 2019 files and customer point lists), which gives each node the columns the file has and each instance a header of its own; a curation says which of them fill the `tasks/v1` node's fields, which pass through under their own names and which are dropped, so what an entry leaves out is written down, and a source whose time quantities are not seconds keeps them under the source's names with the unit recorded as the entry's `timeUnit`. A new category starts by defining its target schema in `src/targets/` and registering it in the category registry; an entry cannot name a category that does not exist. The statically generated site renders every entry with schema.org Dataset JSON-LD, so the catalogue surfaces in the dataset discovery layers.
+A category names a normalisation target: the shape every entry in that category resolves to, whatever its native shape. `graph/v1` is `{ meta, nodes, edges }` with unique node ids and edges whose endpoints name nodes; its entries are graph datasets taken from upstream collections such as SuiteSparse, Netzschleuder, SNAP, KONECT and OGB, read by one parser per format (delimited edge lists, Matrix Market, GML, Pajek, UCINet DL full matrices, Graph Drawing contest files, KONECT edge files, rating and count files, knowledge graph triples and Heterogeneous Graph Benchmark node and link files), each giving the graph's topology and, where the format states them, node and edge types, weights, times and the split each edge belongs to. `tasks/v1` is `{ meta, nodes }`, a set of tasks as the places or cases it offers to visit or serve (a routing benchmark, or an open data register of permits and complaints), each flat with optional coordinates, prize, prize probability, duration, travel time, skill, territory, lifecycle status and times, and deadline, read from tabular sources by the `csv-table` parser (a delimited file, a member of an archive, a sheet of an Excel workbook, or an open data portal read a page at a time) and from ArcGIS feature layers by the `geojson` parser. Its entries come from a survey of open operations-research datasets, imported with a curation that records, for each source, what its download gives and how its columns map onto the node; a dataset whose source has no usable table is imported without a recipe and says why. Any category can be fetched as a collection: a recipe that selects every matching archive member with `members` (by path, by the extensions its name ends with, or by a regular expression its whole path matches) resolves to `{ meta, instances }`, where each instance is a document of the entry's category under the path of the member it came from, the schema derived from the category's own, so a category needs nothing of its own to be collected. The parser reads each member, the crosswalk is applied to each, and a member that does not parse or does not validate fails the fetch and is named. Instance files are read by one parser per format (Solomon and TOPTW time window files, Chao team orienteering files, TSPLIB style orienteering files, VRP-REP XML, VeRoLog 2019 files and customer point lists), which gives each node the columns the file has and each instance a header of its own; a curation says which of them fill the `tasks/v1` node's fields, which pass through under their own names and which are dropped, so what an entry leaves out is written down, and a source whose time quantities are not seconds keeps them under the source's names with the unit recorded as the entry's `timeUnit`. A new category starts by defining its target schema in `src/targets/` and registering it in the category registry; an entry cannot name a category that does not exist. The statically generated site renders every entry with schema.org Dataset JSON-LD, so the catalogue surfaces in the dataset discovery layers.
 
 ## Licensing