fairscape-conversion 0.1.0__py3-none-any.whl

fairscape_conversion/MAPPING-SCHEMA.md

@@ -0,0 +1,216 @@
+# Unified converter mapping format
+
+One mapping format for every fairscape RO-Crate converter. A converter is a
+**plugin**: a folder of CSVs (the mapping data) plus one subclass of
+`core.plugin.PluginBase` holding the workflow-specific algorithms (named
+parsers + step methods; see `docs/NEW-PLUGIN.md`). The core engine
+(`fairscape_conversion/core/`) reads the CSVs and drives the conversion; you never re-read a
+bespoke format to understand a converter — you read these CSVs.
+
+Guiding rule: **if it is data it lives in a CSV; if it is an algorithm it is a
+named parser or hook.** Blank cells are ignored. List-valued cells are
+pipe-joined (`a|b|c`).
+
+Every plugin has `entities.csv` + `properties.csv`. Some plugins add
+`associations.csv` and/or `cv_bases.csv` (currently only c2m2 needs them). None
+of these files are required to be non-empty — an export-only plugin may leave
+half the columns blank.
+
+---
+
+## `entities.csv` — one row per source entity class
+
+How a source node/table/record is recognized and what target entity it becomes.
+Row order is match precedence (first matching row wins).
+
+| column | meaning |
+|---|---|
+| `source_type` | the source `@type` token, C2M2 table name, or model class name to match |
+| `discriminator` | named predicate that further gates the match (blank = match any). e.g. wrroc `instrument_is_workflow`; the datasheet/croissant `ROOT` vs `COMPONENT` position |
+| `target_type` | the target entity/class produced (empty ⇒ node kept as-is or dropped, see `level`) |
+| `target_type_iri` | pipe-joined `@type` array to stamp on the target node, e.g. `prov:Activity\|https://w3id.org/EVI#Computation` |
+| `level` | role tag consumed by hooks: `workflow`, `tool`, `data`, `software`, `root`, `component`, `agent`, `drop` |
+| `id_strategy` | how the target `@id` is set: `keep` (reuse source id), `mint`, `persistent_or_mint`, `persistent_ark_or_mint`, `ontology_term`, `column` |
+| `id_template` | `{token}`-interpolated mint pattern, e.g. `{prefix}-biosample/{local_id}` |
+| `id_column` | source column/field carrying the id (for `persistent_*`, `ontology_term`, `column`) |
+| `keep_identifier` | `TRUE` ⇒ stash the original source `@id` as `identifier` so a reverse trip can restore it |
+| `mint_prefix` | short stem for minted ARKs (wrroc), e.g. `computation-main` |
+| `note` | free text |
+
+`level` semantics: `agent` = kept in the output graph unchanged; `drop` = not
+emitted as an entity (a hook may still harvest data from it); everything else is
+a real target node. The engine mints an `@id` for every non-`agent`/non-`drop`
+node per `id_strategy`.
+
+---
+
+## `properties.csv` — one row per source-property → target-property
+
+One row moves one property, in one or both directions.
+
+| column | meaning |
+|---|---|
+| `source_type` | owning source entity (matches an `entities.source_type`) |
+| `source_property` | source field/column read (blank ⇒ constant or synthesized) |
+| `target_type` | owning target entity |
+| `target_property` | target field written |
+| `import_parser` | named parser, source→target direction (blank ⇒ not imported) |
+| `export_parser` | named parser, target→source direction (blank ⇒ not exported) |
+| `direction` | `both`, `import`, or `export` (which directions this row participates in) |
+| `reverse_primary` | `TRUE` ⇒ for a many-to-one target, this source wins on the reverse trip |
+| `requirement` | `required`, `recommended`, `optional`, `synthesized`, `passthrough`, `nohome` (documentation + passthrough behaviour) |
+| `cardinality` | `scalar` or `list` (target cardinality; `list` accumulates all sources, `scalar` is first-write-wins) |
+| `constant_value` | `{token}`-interpolated literal when `source_property` is blank |
+| `fallback_source` | second source field tried when the first is empty |
+| `fallback_parser` | parser applied to `fallback_source` |
+| `source_table` | CV/lookup table this field resolves against (c2m2) |
+| `param_name` | name for a wrapped `PropertyValue` when `target_property` is `additionalProperty` (c2m2) |
+| `wrap` | wrap the value: `ident_ref`, `property_value`, `ruled_out_pv` (blank = none) |
+| `note` | free text |
+
+**`requirement` = `passthrough`** rows are copied verbatim as extra keys and
+re-emitted on export (fairscape models allow extra keys). **`nohome`** rows are
+documented drops (a target property with no home the other direction).
+
+**Many-to-one merge.** When several `properties.csv` rows share one
+`target_type` + `target_property`, they merge: a `list` target accumulates every
+source's output, a `scalar` target keeps the first non-empty in row order (put
+the `reverse_primary` source first so it wins). This is how d4d collapses
+`intended_uses`, `purposes`, `tasks`, … onto `rai_data_use_cases`.
+
+---
+
+## `associations.csv` — join-table edges (c2m2 only)
+
+C2M2 encodes relationships in link tables (`biosample_disease`, …). Each row is
+one edge rule. The join algorithm itself (build a guid index, look each node up
+by both its natural key and its `@id`) lives in the c2m2 `hooks.py`; this CSV is
+only the rule data.
+
+| column | meaning |
+|---|---|
+| `source_type` | entity the edge attaches to |
+| `assoc_table` | the link/association table |
+| `match_columns` | pipe-joined columns forming the owner's key |
+| `match_resolver` | `entity_key` (owner's natural key) or `ontology_term` (owner is a CV node) |
+| `value_columns` | pipe-joined columns carrying the target reference |
+| `value_resolver` | `ontology_term`, `entity_guid`, or `raw` |
+| `value_entity` | for `entity_guid`: the table whose node the value resolves to; for `ontology_term`: the CV source table |
+| `target_property` | pipe-joined target properties the edge writes |
+| `wrap` | `ident_ref`, `property_value`, `ruled_out_pv` |
+| `filter_column` | gate the rows on this column (blank = no filter) |
+| `filter_endswith` | keep rows whose `filter_column` ends with this (`:1` observed, `:0` ruled-out — **the CV code, never the label**) |
+| `param_name` | name for the wrapped `PropertyValue` when `wrap` = `property_value` (the old `pv_name`) |
+| `note` | free text |
+
+A single association row is written twice in the CSV to express both an edge and
+its back-reference (swap which columns are `match` vs `value`), exactly as the
+original per-table JSON did.
+
+---
+
+## `cv_bases.csv` — controlled-vocabulary → ontology IRI bases (c2m2 only)
+
+The prefix/table → ontology-base lookup that `ontology.resolve_term` used to
+hold as Python dicts. Two kinds of row:
+
+| column | meaning |
+|---|---|
+| `key` | the CURIE prefix (`kind=curie`, e.g. `DOID`) or the CV table (`kind=bare`, e.g. `gene`) |
+| `kind` | `curie` (value carries a CURIE prefix) or `bare` (table's ids are bare accessions) |
+| `iri_base` | IRI base the local id is appended to (e.g. `http://purl.obolibrary.org/obo/DOID_`) |
+| `ontology_name` | human name of the ontology |
+| `curie_prefix` | for `kind=bare`: the CURIE stem to reattach (e.g. `ensembl`) |
+| `note` | free text |
+
+A few resolutions stay in plugin code because they are not simple base lookups:
+the `NCBI:txid…` taxonomy rule and the GlyTouCan-vs-PubChem `compound` split.
+
+---
+
+## Parsers and plugin methods (the Python that a CSV references by name)
+
+**Parsers** all share one signature:
+
+```python
+def parser(value, rule, ctx) -> parsed_value | None
+```
+
+`value` is the source property value (`None` for synthesized fields), `rule` is
+the compiled row (the CSV columns), and `ctx` carries conversion state (the
+current node, the `@id`→node index, the guid map, the root node, and any
+per-plugin extras). Returning `None` means "set nothing." Parsers are registered
+by name in the plugin class's `import_parsers` / `export_parsers` dicts, so the
+loader can reject a typo in a CSV at load time. Shared building blocks live in
+`core/parsers.py`.
+
+**Plugin methods** are the per-workflow orchestration — override them on your
+`PluginBase` subclass (full list + when to override each: `docs/NEW-PLUGIN.md`):
+
+| method | what it does | examples |
+|---|---|---|
+| `load_source` | turn the raw input into `SourceRecord`s (the common override) | example; helpers `records_from_items` / `records_from_graph` |
+| `classify` / `discriminators` | match a record to its `entities.csv` row, gated by named predicates | wrroc workflow-vs-tool discriminator |
+| `pre` | full-control record production: build indexes, mint ARKs, load join/CV tables | c2m2 guid-map + association index; wrroc id minting |
+| `map_record` | **engine default** applies the property rules; override only when a record doesn't map rule-by-rule | d4d many-to-one merge; c2m2 vars pipeline |
+| `link` | cross-node edges added after all nodes exist | wrroc `isPartOf` / `generatedBy` mirroring |
+| `assemble` | build the root, attach children, write preservation files | c2m2 root + datapackage/sqlite copy |
+| `export` | bespoke reverse driver or engine bypass | wrroc/d4d reverse; croissant |
+
+A plugin implements only the steps it needs; the rest default to the engine's
+generic behaviour. Property-rule application is the engine's own code unless a
+plugin genuinely cannot map rule-by-rule — which is the whole point of the
+format.
+
+---
+
+## Worked rows (one per plugin)
+
+**wrroc** `entities.csv` — a workflow-level run becomes an EVI Computation:
+```
+source_type,discriminator,target_type,target_type_iri,level,id_strategy,id_template,id_column,keep_identifier,mint_prefix,note
+CreateAction,instrument_is_workflow,Computation,prov:Activity|https://w3id.org/EVI#Computation,workflow,mint,,,TRUE,computation-main,parent run
+```
+wrroc `properties.csv` — description synthesized to satisfy EVI's min-length:
+```
+source_type,source_property,target_type,target_property,import_parser,export_parser,direction,reverse_primary,requirement,cardinality,constant_value,fallback_source,fallback_parser,source_table,param_name,wrap,note
+CreateAction,description,Computation,description,desc_min10,scalar,both,TRUE,synthesized,scalar,,,,,,,
+```
+
+**d4d** `entities.csv` — a distribution becomes a DigitalObject graph node
+(d4d reuses `target_type_iri` for the additionalType and `id_column` for the
+url field):
+```
+distributions,,DigitalObject,distribution,data,keep,,download_url,FALSE,,each distribution -> DigitalObject in hasPart
+```
+d4d `properties.csv` — one of the many sources merging onto `rai_data_use_cases`:
+```
+D4D,intended_uses,Dataset,rai_data_use_cases,flatten,reverse_flatten,both,TRUE,recommended,list,,,,,,,
+```
+
+**c2m2** `entities.csv` — the biosample table explodes to EVI Samples (c2m2's
+file adds a `precedence` column before `note`; d4d/wrroc's files don't have it):
+```
+biosample,,Sample,,data,persistent_ark_or_mint,{prefix}-biosample/{local_id},persistent_id,FALSE,,20,
+```
+c2m2 `associations.csv` — observed disease edges only:
+```
+source_type,assoc_table,match_columns,match_resolver,value_columns,value_resolver,value_entity,target_property,wrap,filter_column,filter_endswith,param_name,note
+biosample,biosample_disease,biosample_id_namespace|biosample_local_id,entity_key,disease,ontology_term,disease,associatedDisease,ident_ref,association_type,:1,,
+```
+
+**croissant** `entities.csv` — the root crate becomes a Croissant Dataset (export-only):
+```
+ROCrateMetadataElem,ROOT,CroissantDataset,,root,keep,,,FALSE,,0,the crate root -> Croissant Dataset
+```
+croissant `properties.csv` — encodingFormat mapped to a MIME type on export:
+```
+Dataset,format,CroissantFileObject,encodingFormat,,map_format_to_mime_type,export,FALSE,,scalar,,,,,,,
+```
+
+**example** `entities.csv` + `properties.csv` — the minimal plugin
+(`plugins/example/`), a toy memo becoming a schema.org Dataset:
+```
+memo,,Dataset,https://schema.org/Dataset,data,keep,,,FALSE,,one Dataset per memo
+memo,title,Dataset,name,scalar,,import,,required,scalar,,,,,,,
+```

fairscape_conversion/README.md

@@ -0,0 +1,73 @@
+# fairscape-conversion
+
+Your metadata is already written down — as a datasheet, a datapackage, or a
+workflow engine's run output. Convert it into a FAIRSCAPE/EVI RO-Crate
+(`ro-crate-metadata.json`) instead of re-entering it. One command per format.
+
+```bash
+pip install -e .        # from this directory; installs fairscape-conversion
+```
+
+The `fairscape import` / `fairscape export` commands come with the companion
+`fairscape` CLI (not yet published). Without it, every conversion also runs as
+`python -m fairscape_conversion.core.cli convert <format> <import|export> IN [OUT]`.
+
+## Import — get an RO-Crate
+
+Pick the row that matches what you have.
+
+| You have | You need | Run |
+|---|---|---|
+| A Datasheet for Datasets (D4D) | the datasheet as YAML or JSON | `fairscape import d4d datasheet.yaml -o ./crate` |
+| A CFDE C2M2 datapackage | the directory of TSVs + `C2M2_datapackage.json` | `fairscape import c2m2 ./datapackage-dir -o ./crate` |
+| A Workflow Run RO-Crate | its `ro-crate-metadata.json` | `fairscape import wrroc ro-crate-metadata.json -o ./crate` |
+| A finished Cromwell/WDL run | the file from `cromwell run -m metadata.json` | `fairscape import cromwell metadata.json -o ./crate` |
+| A finished Snakemake run | the records JSON from `snakemake --reporter fairscape` | `fairscape import snakemake records.json -o ./crate` |
+| Finished MLflow runs | the tracking store (an `mlruns` dir or tracking URI) and `pip install mlflow` | `fairscape import mlflow ./mlruns --experiment NAME -o ./crate` |
+
+## Export — from an RO-Crate
+
+| You want | Run |
+|---|---|
+| A D4D datasheet | `fairscape export d4d ro-crate-metadata.json` |
+| A Workflow Run RO-Crate | `fairscape export wrroc ro-crate-metadata.json` |
+| An MLCommons Croissant document | `fairscape export croissant ro-crate-metadata.json` |
+
+## Try it — no data needed
+
+Every format ships a real example input and its expected output inside its
+plugin folder, so you can run any conversion right now:
+
+```bash
+fairscape import d4d plugins/d4d/input.yaml -o /tmp/crate
+```
+
+| format | example input | expected output |
+|---|---|---|
+| d4d | `plugins/d4d/input.yaml` (the AI-READI datasheet) | `plugins/d4d/golden.json` |
+| c2m2 | `plugins/c2m2/input-datapackage/` (miniature datapackage) | `plugins/c2m2/golden.json` |
+| wrroc | `plugins/wrroc/input.json` (CWL revsort run crate) | `plugins/wrroc/golden.json` |
+| cromwell | `plugins/cromwell/input.json` (scatter workflow records) | `plugins/cromwell/golden.json` |
+| snakemake | `plugins/snakemake/input.json` (3-rule chain records) | `plugins/snakemake/golden.json` |
+| mlflow | `plugins/mlflow/input.json` (iris experiment records) | `plugins/mlflow/golden.json` |
+| croissant | `plugins/croissant/input.json` (export this crate) | `plugins/croissant/golden.json` |
+
+## From Python
+
+```python
+import yaml
+from fairscape_conversion.plugins import d4d
+
+crate = d4d.convert("import", yaml.safe_load(open("datasheet.yaml")))
+```
+
+Same shape for every format: `wrroc`, `c2m2`, `cromwell`, `snakemake`,
+`mlflow` (`convert("import", ...)`), and `d4d`/`wrroc`/`croissant`
+(`convert("export", crate)`).
+
+## More
+
+- Your format isn't listed → [`docs/NEW-PLUGIN.md`](docs/NEW-PLUGIN.md) —
+  a converter is a folder of two CSVs plus a small plugin class.
+- How the engine works, what's tested → [`docs/INTERNALS.md`](docs/INTERNALS.md)
+  and [`MAPPING-SCHEMA.md`](MAPPING-SCHEMA.md).

fairscape_conversion/__init__.py

@@ -0,0 +1 @@
+"""Unified fairscape RO-Crate converters — one CSV mapping format for all."""

fairscape_conversion/core/__init__.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+"""Unified converter core — one mapping format, one kernel, per-plugin code.
+
+Public API used by the plugins:
+
+    from fairscape_conversion.core import (Plugin, Context, Record, Mapping,
+                            run_pipeline, apply_import_rules, apply_export_rules,
+                            run_import_parser, run_export_parser,
+                            interpolate, roundtrip)
+"""
+
+from . import parsers, roundtrip
+from .engine import (Context, Record, apply_export_rules, apply_import_rules,
+                     interpolate, run_export_parser, run_import_parser,
+                     run_pipeline)
+from .loader import load_mapping
+from .plugin import PluginBase
+from .records import (SourceRecord, classify_node, records_from_graph,
+                      records_from_items, types_of)
+from .registry import Plugin
+from .schema import (AssocRule, CvBase, EntityRule, Mapping, PropertyRule,
+                     split_pipe)
+
+__all__ = [
+    "Plugin", "PluginBase", "Context", "Record", "Mapping",
+    "SourceRecord", "classify_node", "records_from_items", "records_from_graph",
+    "types_of",
+    "EntityRule", "PropertyRule", "AssocRule", "CvBase", "split_pipe",
+    "load_mapping", "run_pipeline",
+    "apply_import_rules", "apply_export_rules",
+    "run_import_parser", "run_export_parser",
+    "interpolate", "parsers", "roundtrip",
+]

fairscape_conversion/core/arks.py

@@ -0,0 +1,30 @@
+"""Deterministic ARK minting — the fairscape identifier scheme, in core.
+
+``ark:{naan}/{prefix}-{slug(name)}-{sha1(source)[:7]}``: the slug is the
+readable half, the hash names the thing (a stable source string — a file path,
+a rule name + outputs, a session id), so re-running a conversion over the same
+source reproduces the same identifiers. This is the scheme nf-fairscape and
+snakemake-report-plugin-fairscape mint, and the one the wrroc plugin composes
+(with a ``wrroc-`` tag folded into the prefix).
+
+Plugins that need a namespace tag put it in ``prefix`` (``wrroc-computation``);
+the minter itself adds nothing.
+"""
+
+import hashlib
+import re
+
+DEFAULT_NAAN = "59853"
+
+
+def slugify(text, max_len=40, fallback="entity"):
+    slug = re.sub(r"[^a-z0-9]+", "-", str(text).lower()).strip("-")
+    return slug[:max_len].strip("-") or fallback
+
+
+def short_hash(text, length=7):
+    return hashlib.sha1(str(text).encode()).hexdigest()[:length]
+
+
+def mint_ark(naan, prefix, name, source, fallback="entity"):
+    return f"ark:{naan}/{prefix}-{slugify(name, fallback=fallback)}-{short_hash(source)}"

fairscape_conversion/core/cli.py

@@ -0,0 +1,54 @@
+#!/usr/bin/env python3
+"""Tiny CLI: ``fairscape_conversion convert <plugin> <import|export> <input> [output]``.
+
+Reads JSON or YAML by extension, dispatches to the named plugin's ``convert``,
+writes JSON (or YAML for a ``.yaml``/``.yml`` output path). The library API
+(``fairscape_conversion.plugins.<name>.convert``) is what the tests and other code call; this
+is only a convenience wrapper.
+"""
+
+from __future__ import annotations
+
+import importlib
+import json
+import sys
+from pathlib import Path
+
+
+def _read(path: Path):
+    text = path.read_text()
+    if path.suffix in (".yaml", ".yml"):
+        import yaml
+        return yaml.safe_load(text)
+    return json.loads(text)
+
+
+def _write(path: Path, data):
+    if path.suffix in (".yaml", ".yml"):
+        import yaml
+        path.write_text(yaml.dump(data, sort_keys=False, allow_unicode=True))
+    else:
+        path.write_text(json.dumps(data, indent=2, default=str))
+
+
+def main(argv=None):
+    argv = argv if argv is not None else sys.argv[1:]
+    if len(argv) < 4 or argv[0] != "convert":
+        print(__doc__)
+        return 1
+    _, plugin_name, direction, in_path, *rest = argv
+    if direction not in ("import", "export"):
+        print(f"direction must be import|export, got {direction!r}")
+        return 1
+    plugin = importlib.import_module(f"fairscape_conversion.plugins.{plugin_name}")
+    result = plugin.convert(direction, _read(Path(in_path)))
+    if rest:
+        _write(Path(rest[0]), result)
+        print(f"wrote {rest[0]}")
+    else:
+        print(json.dumps(result, indent=2, default=str))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

fairscape_conversion/core/engine.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+"""The shared conversion kernel.
+
+Everything workflow-agnostic lives here: applying property rules with named
+parsers, interpolating id/constant templates, and a light ``run_pipeline``
+driver that owns the ``map`` stage (per-record via the overridable
+``map_record`` hook) and calls the plugin's ``pre`` / ``link`` / ``assemble``
+hooks around it.
+
+The kernel is deliberately small. A plugin that does not fit the pipeline (d4d's
+many-to-one root merge) still uses the same ``run_parser`` + rule objects — it
+just composes them itself. That is the unification: one mapping format, one
+parser signature, one kernel, and per-workflow orchestration where it is
+genuinely different.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+
+from .schema import EntityRule, Mapping, PropertyRule
+
+_EMPTY = (None, "", [], {})
+
+
+@dataclass
+class Record:
+    """A source record queued for mapping: its data + the entity rule matched."""
+
+    data: dict
+    rule: EntityRule
+    source_id: str = ""
+
+
+class Context:
+    """Mutable conversion state passed to every parser and hook.
+
+    ``extras`` is the plugin scratchpad (guid maps, fallback agents, association
+    indexes, …). ``node`` / ``node_extras`` are set by the engine per record so
+    parsers can reach the record being mapped and stash side outputs.
+    """
+
+    def __init__(self, plugin, direction: str, source):
+        self.plugin = plugin
+        self.mapping: Mapping = plugin.mapping
+        self.direction = direction          # "import" | "export"
+        self.source = source
+        self.by_id: dict = {}
+        self.root: dict = {}
+        self.records: list[Record] = []
+        self.out_nodes: dict = {}
+        self.order: list = []
+        self.extras: dict = {}
+        self.node: dict = {}
+        self.node_extras: dict = {}
+
+
+# ============================================================================
+# Templates
+# ============================================================================
+
+_TOKEN = re.compile(r"\{([^{}]+)\}")
+
+
+def interpolate(template: str, tokens: dict) -> str:
+    """Replace ``{token}`` with ``tokens[token]`` (unknown tokens left as-is)."""
+    def sub(m):
+        key = m.group(1)
+        return str(tokens.get(key, m.group(0)))
+    return _TOKEN.sub(sub, template)
+
+
+# ============================================================================
+# Parser application
+# ============================================================================
+
+def run_import_parser(ctx: Context, rule: PropertyRule, value):
+    parser = ctx.plugin.import_parsers[rule.import_parser]
+    return parser(value, rule, ctx)
+
+
+def run_export_parser(ctx: Context, rule: PropertyRule, value):
+    parser = ctx.plugin.export_parsers[rule.export_parser]
+    return parser(value, rule, ctx)
+
+
+def _read_source(record: dict, rule: PropertyRule):
+    if rule.source_property:
+        return record.get(rule.source_property)
+    return None
+
+
+def apply_import_rules(record: dict, rules: list[PropertyRule], ctx: Context) -> dict:
+    """Apply import rules to one source record. First rule to set a target wins.
+
+    A rule whose value is empty falls back to ``fallback_source`` (+
+    ``fallback_parser``) when given. ``constant_value`` is interpolated against
+    the record. Side outputs a parser stashes in ``ctx.node_extras`` (e.g.
+    ``parameters``) are merged in at the end. This is the wrroc / c2m2 /
+    croissant semantics; d4d composes rules itself (many-to-one merge).
+    """
+    out: dict = {}
+    ctx.node = record
+    ctx.node_extras = {}
+    for rule in rules:
+        target = rule.target_property
+        if out.get(target) not in _EMPTY:
+            continue
+        if rule.source_property is None and rule.constant_value:
+            result = interpolate(rule.constant_value, record)
+        else:
+            result = run_import_parser(ctx, rule, _read_source(record, rule))
+            if result is None and rule.fallback_source:
+                fb_value = record.get(rule.fallback_source)
+                result = ctx.plugin.import_parsers[
+                    rule.fallback_parser or rule.import_parser](fb_value, rule, ctx)
+        if result is not None:
+            out[target] = result
+    for k, v in ctx.node_extras.items():
+        out.setdefault(k, v)
+    return out
+
+
+def apply_export_rules(node: dict, rules: list[PropertyRule], ctx: Context) -> dict:
+    """Apply export rules to one target-side node. First rule to set wins."""
+    out: dict = {}
+    ctx.node = node
+    for rule in rules:
+        target = rule.source_property
+        if not target or out.get(target) not in _EMPTY:
+            continue
+        result = run_export_parser(ctx, rule, node.get(rule.target_property))
+        if result is not None:
+            out[target] = result
+    return out
+
+
+# ============================================================================
+# Pipeline driver (wrroc / c2m2 / croissant)
+# ============================================================================
+
+def _default_select_rules(ctx: Context, rec: Record) -> list[PropertyRule]:
+    """Default rule set for a record: grouped by the fairscape-side entity.
+
+    Import: every rule producing this record's target type (several source types
+    can share one target's rule set — wrroc Software). Export: every rule reading
+    this record's kind. A plugin whose rules key on the source instead overrides
+    via ``hooks['select_rules']`` (e.g. by-table selection).
+    """
+    if ctx.direction == "import":
+        return ctx.mapping.import_rules_by_target(rec.rule.target_type)
+    return ctx.mapping.export_rules_by_target(rec.rule.target_type)
+
+
+def _default_resolve_id(ctx: Context, rec: Record) -> str:
+    """Default ``@id`` for a mapped node.
+
+    Honours a precomputed ``ctx.extras['guid_map']`` (wrroc mints ARKs up front);
+    otherwise applies the entity rule's ``id_strategy`` for the simple cases.
+    """
+    guid_map = ctx.extras.get("guid_map")
+    if guid_map and rec.source_id in guid_map:
+        return guid_map[rec.source_id]
+    rule = rec.rule
+    if rule.id_strategy in ("keep", "column"):
+        return rec.source_id
+    if rule.id_template:
+        return interpolate(rule.id_template, rec.data)
+    return rec.source_id
+
+
+def _default_map_record(ctx: Context, rec: Record, rules: list[PropertyRule]):
+    """Default per-record mapper: apply the CSV property rules, stamp ``@type``
+    and the round-trip ``identifier``.
+
+    A plugin overrides this (``hooks['map_record']``) when one record does not
+    map rule-by-rule — c2m2's vars pipeline, d4d's many-to-one merge. An
+    override may set ``@id`` itself and may return ``None`` to drop the record.
+    """
+    if ctx.direction == "import":
+        node = apply_import_rules(rec.data, rules, ctx)
+    else:
+        node = apply_export_rules(rec.data, rules, ctx)
+    if rec.rule.target_type_iri:
+        node["@type"] = (rec.rule.target_type_iri if len(rec.rule.target_type_iri) > 1
+                         else rec.rule.target_type_iri[0])
+    if ctx.direction == "import" and rec.rule.keep_identifier and rec.source_id:
+        node["identifier"] = rec.source_id
+    return node
+
+
+def run_pipeline(plugin, direction: str, source, options: dict | None = None):
+    """Drive a pipeline-style conversion: pre -> map (engine) -> link -> assemble.
+
+    The plugin's ``pre`` hook fills ``ctx.records`` (and any indexes in
+    ``ctx.extras``); the engine maps each record with its rule set via
+    ``map_record``; ``link`` adds cross-node edges; ``assemble`` builds and
+    returns the final object. Per-run options (e.g. ``naan``, ``validate``) are
+    seeded into ``ctx.extras``.
+
+    A record whose mapping returns ``None`` is dropped. On a duplicate ``@id``
+    the first-mapped node wins (later records are skipped) — ``pre`` may seed
+    ``ctx.out_nodes``/``ctx.order`` to reserve ids ahead of the loop.
+    """
+    ctx = Context(plugin, direction, source)
+    if options:
+        ctx.extras.update(options)
+    plugin.hook("pre")(ctx)
+
+    select_rules = plugin.hooks.get("select_rules", _default_select_rules)
+    resolve_id = plugin.hooks.get("resolve_id", _default_resolve_id)
+    map_record = plugin.hooks.get("map_record", _default_map_record)
+
+    for rec in ctx.records:
+        node = map_record(ctx, rec, select_rules(ctx, rec))
+        if node is None:
+            continue
+        node.setdefault("@id", resolve_id(ctx, rec))
+        if node["@id"] in ctx.out_nodes:      # first writer wins
+            continue
+        ctx.out_nodes[node["@id"]] = node
+        ctx.order.append(node["@id"])
+
+    plugin.hook("link")(ctx)
+    return plugin.hook("assemble")(ctx)