helios-cortex 0.1.0__tar.gz

helios_cortex-0.1.0/.gitignore

@@ -0,0 +1,21 @@
+tpcds/
+*.parquet
+__pycache__/
+.env
+.env.sh
+/requirements.lock
+.helios-python/
+state/
+*.db-wal
+*.db-shm
+node_modules/
+apps/ui/dist/
+apps/helios/ui/dist/
+apps/helios/ui/src/content/docs/
+*.tsbuildinfo
+.cursor/
+runs/
+run_archive/
+.DS_Store
+*.pptx
+.venv/

helios_cortex-0.1.0/CONTRACTS.md

@@ -0,0 +1,108 @@
+# helios-cortex contracts
+
+This file is helios-cortex's public API. The names in the tables below are the semantic-versioning
+boundary: other packages (helios-model, helios-serve, helios-mcp, plugins and host
+applications) may depend on them, and they change only as the policy below allows. Everything
+else in `helios_cortex` is internal, and can change or disappear in any release.
+
+`tests/test_contracts.py` imports every name listed here from the module given, so the tables
+cannot drift from the code.
+
+## Stability levels
+
+- **Stable.** Changed only under the deprecation policy.
+- **Provisional.** Public, but a named burndown task will reshape it. Depend on it if you need
+  to, and expect to update your code when that task lands. It becomes Stable afterwards.
+- **Planned.** Not written yet. Listed so that the boundary is complete.
+
+## Deprecation policy
+
+helios-cortex is pre-1.0 (0.y.z).
+
+- **While pre-1.0:** a release that removes or changes a Stable name bumps the minor version (y).
+  Where it can, the old spelling keeps working for one more minor release and emits a
+  `DeprecationWarning` naming its replacement. Patch releases (z) never change a Stable name.
+- **From 1.0:** standard semantic versioning. A Stable name is deprecated in a minor release
+  (with a `DeprecationWarning` and a changelog entry) and removed no earlier than the next
+  major release.
+- **What counts as a change:** removing or renaming a name, removing a parameter or field,
+  making an optional parameter required, narrowing an accepted type, or changing the meaning
+  of a return value. Adding optional parameters, fields or methods is not a change.
+- **Provisional names:** may change in any minor release. The changelog says how to migrate.
+
+## Contract
+
+### Configuration
+
+| Module | Names | Level | Notes |
+|---|---|---|---|
+| `helios_cortex.settings` | `Settings`, `ConnectionSettings`, `CatalogSettings`, `LLMSettings` | Stable | Resolution order, the config file and named connections and catalogs. A connection's `identity` (`service`, the default, `impersonate` or `token`) picks the engine identity mode. Core reads `HELIOS_*` and the LLM ecosystem's names (`ANTHROPIC_API_KEY`, `MISTRAL_API_KEY`, `LLM_PROVIDER`). A subclass adds a platform's variables by overriding `profile_sources` (after the conventional environment, before the config file); helios-cloudera's `WorkbenchSettings` reads the Workbench and Cloudera names. `registry_url` (secret) names the model registry's database. `artifact_storage` (`location` plus filesystem options, secrets hidden) puts artifacts on object storage. |
+
+### Semantic model and compiler
+
+| Module | Names | Level | Notes |
+|---|---|---|---|
+| `helios_cortex.ossie.model` | `SemanticModel`, `Dataset`, `Field`, `Relationship`, `Metric` | Stable | The in-memory form of a published Ossie model. |
+| `helios_cortex.ossie` | `validate`, `dump_yaml`, `SPEC_VERSION` | Provisional | The Ossie format (`helios_cortex.ossie.format`): the spec version Helios writes, validation against the vendored schema plus referential checks, and YAML output. Building a document from a discovery proposal is helios-model's (`helios_model.build`). |
+| `helios_cortex.compiler` | `Compiler`, `SemanticRequest`, `Filter`, `Measure`, `Compiled`, `CompileError`, `FilterValueTypeError` | Provisional | `compile(request, dialect)` needs a dialect: the engine's `dialect` or a SQLGlot dialect name; there is no default. |
+
+### Ontology
+
+| Module | Names | Level | Notes |
+|---|---|---|---|
+| `helios_cortex.ontology` | `OntologyGraph`, `GraphNode`, `GraphEdge`, `OntologyGraphError` | Stable | |
+| `helios_cortex.ontology` | `SourceMapping`, `load_mapping`, `load_mappings` | Stable | |
+
+### Index
+
+| Module | Names | Level | Notes |
+|---|---|---|---|
+| `helios_cortex.index` | `IndexStore` | Stable | |
+| `helios_cortex.index` | `OntologyVersionRecord`, `OntologyActivationRecord`, `CrawlRunRecord`, `AssetRecord`, `SegmentRecord`, `MentionRecord`, `EntityRecord`, `EntityLinkRecord`, `RelationshipRecord`, `ClaimRecord`, `ClaimEvidenceRecord`, `EvaluationRecord` | Stable | Rows of the `helios_index` tables. New optional fields are allowed. |
+
+### Storage
+
+| Module | Names | Level | Notes |
+|---|---|---|---|
+| `helios_cortex.artifacts` | `ArtifactStore`, `Activation`, `ActivationConflict`, `ArtifactError`, `ANY` | Provisional | On a local directory or object storage (`ArtifactStore.from_settings`: `[artifact_storage]` else `artifacts_dir`). Published revisions are immutable (named by their YAML's SHA-256, written once); the active revision is the newest entry of `models/<id>/active/`, and `activate(..., expected_sequence=)` is a compare-and-swap (`ActivationConflict`). `read_published_ossie` / `read_revision` work on every backend and verify hashes; methods returning a local `Path` work on a local directory only. Discovery runs live in the same storage under `runs/` (`ArtifactStore.runs`). |
+| `helios_cortex.artifact_storage` | `ArtifactStorage`, `storage_for`, `LocalStorage` | Provisional | `storage_for(location, **options)`: a path or `file://`, `s3://` (`[s3]`), `abfs://` / `abfss://` / `az://` (`[azure]`), `gs://` (`[gcs]`), `ofs://` through Ozone HttpFS (`[ozone]`). The one conditional operation is `create` (write only if absent, atomically), on which immutability and the activation log are built. |
+| `helios_cortex.runs` | `RunStore` | Provisional | Discovery runs, `<run_id>/{harvest,profile,propose,review}.json`: `write`, `load`, `run_ids`, `latest_run_id`, `list_runs`, `detail`. On any storage: `ArtifactStore.runs` (under `runs/`), `RunStore.from_settings` (with the artifacts on object storage, else `runs_dir`), or `RunStore(path)` for a local runs directory. `runs_dir` is a local path, None on object storage. |
+| `helios_cortex.conformance` | `RegistryConformance`, `ArtifactStorageConformance` | Provisional | The tests every backend must pass, Helios's and third parties': subclass one and provide a `registry` or `storage` fixture returning an empty backend. The registry suite checks the `ModelRegistry` rules, including concurrent writers; the storage suite checks the operations (byte-exact content, listing at the prefix boundary, `create` atomic under 10 racing writers) and runs `ArtifactStore` and `RunStore` on the backend. Needs the `[testing]` extra (pytest). New tests may be added in minor releases; a backend that passed may then fail, which is the point. |
+| `helios_cortex.registry` | `ModelRegistry`, `InMemoryRegistry`, `SqlRegistry`, `RegisteredModel`, `SemanticRevision`, `RegistryError`, `RegistryNotReady`, `RevisionConflict`, `DRAFT`, `PUBLISHED` | Provisional | What libraries need to find models: data sources, models (status, creation) and their immutable semantic revisions. Every backend keeps the rules in the module docstring and passes `RegistryConformance`. `InMemoryRegistry` is the reference; `SqlRegistry` stores the records in any SQLAlchemy database by URL (`Settings.registry_url`, default a SQLite file in the data directory), with Alembic migrations in the package (`python -m helios_cortex.registry upgrade`; version table `helios_registry_version`, tables prefixed `helios_`); connections are checked before use, so a database failover does not fail the next call). No ownership: which models a caller sees is the host's `ModelVisibility`. The old console keeps its own SQLite repository until Phase 8, sharing `SemanticRevision`. |
+| `helios_cortex.domain` | `Model`, `DataSource`, `DataSourceReference` | Provisional | Registry records with no ownership (organizations left in 2.6). The registry stores them as they are. |
+
+### Engines and LLM
+
+| Module | Names | Level | Notes |
+|---|---|---|---|
+| `helios_cortex.engines` | `Engine`, `QueryResult`, `engine_for`, `SqlDialect`, `dialect_for` | Provisional | An engine plugin (`helios.engines`) is an `Engine` subclass whose `from_connection(ConnectionSettings)` validates the connection's options and honours its `identity`; `engine_for(connection)` loads the plugin a connection names and builds it. `query(sql, caller=None, *, limit)` runs as the service identity with no caller, and as the caller under the connection's identity mode otherwise; `identity_modes` lists the modes an engine supports. Plugins register under `helios.engines` (`helios_cortex.plugins`). An engine's `dialect` (a `SqlDialect`) renders the compiler's queries and Helios's own statements; plugins subclass it for what SQLGlot lacks (`render_query`, `count_distinct`, `identifier`). The catalog and statistics methods (`databases`, `tables`, `columns`, `row_count`, `recent_queries`) default to `information_schema` and `COUNT(*)`; each takes a keyword-only `caller=None` with the same meaning as `query`'s, so a harvest sees what its caller may see, and a plugin that overrides one must accept and honour it. |
+| `helios_cortex.engines.duckdb_engine` | `DuckDBEngine`, `DuckDBOptions` | Provisional | `engine = "duckdb"`, option `path` (default `:memory:`); `service` mode only. |
+| `helios_cortex.engines.sqlalchemy_engine` | `SqlAlchemyEngine`, `SqlAlchemyOptions` | Provisional | `engine = "sqlalchemy"`, option `url`; needs the `[sqlalchemy]` extra and the database's driver. `service` mode only; introspection through the SQLAlchemy inspector. |
+| `helios_cortex.engines.trino_engine` | `TrinoEngine`, `TrinoOptions`, `TrinoIdentityError` | Provisional | `engine = "trino"`, options `host`, `port`, `catalog`, `schema`, `user`, `password` (basic authentication, https only), `http_scheme`, `verify`; needs the `[trino]` extra. All three identity modes: `service` (the service user), `impersonate` (authenticates as the service user, runs the session as the caller and checks `current_user`), `token` (the caller's JWT). |
+| `helios_cortex.catalog` | `catalog_for`, `capabilities`, `GLOSSARY`, `LINEAGE`, `GlossaryCatalog`, `LineageCatalog`, `ColumnLineage`, `CatalogError` | Provisional | A catalog plugin (`helios.catalogs`) is a class whose `from_catalog(CatalogSettings)` builds it; `catalog_for` loads the one a `[catalogs.<name>]` entry names. Its capabilities are optional and detected from what it implements: `glossary` (`GlossaryCatalog`) and `lineage` (`LineageCatalog.column_lineage` → `ColumnLineage`); a plugin with neither is refused. Tables and columns come from the engine, so no catalog is needed to harvest. Atlas (helios-cloudera) offers both. Harvesting with a catalog is helios-model's (`helios_model.Catalog`, over `helios_model.harvest.TableHarvester`); the glossary is read with the catalog's own credentials, not the caller's. |
+| `helios_cortex.llm` | `LLMClient`, `LLMError`, `LLMTimeoutError`, `llm_from_settings` | Stable | Needs the `[llm]` extra; without it, importing the module says so. |
+
+### Identity and plugins
+
+| Module | Names | Level | Notes |
+|---|---|---|---|
+| `helios_cortex` | `IdentityMode`, `IdentityError` | Stable | `service`, `impersonate`, `token`; `IdentityError` (a `PermissionError`) when a caller lacks the username or token its mode needs. |
+| `helios_cortex.data_authorization` | `DataPolicy`, `DataResource`, `DataAction`, `EngineIdentityAuthorizer`, `PlatformDataAuthorizer`, `PlatformDataDecision`, `DataRestriction`, `DataRestrictionDecision`, `DataAuthorizationDecision`, `DataAuthorizationDenied` | Stable | Platform enforcement plus deny-only host restrictions. The default platform authorizer allows query execution only when `DataResource.identity_mode` runs it as the caller. |
+| `helios_cortex.visibility` | `ModelVisibility`, `AllModels`, `ModelNotVisible`, `visible_models`, `require_visible` | Stable | The host decides which models a caller may list and use; the default shows every model in the registry. A hidden model raises the same `LookupError` as a missing one. |
+| `helios_cortex.audit_sink` | `AuditRecord`, `AuditSink`, `LoggingAuditSink`, `record_safely`, `sql_fingerprint` | Stable | Libraries emit `AuditRecord`s (action, outcome, model, caller username, row count, SQL fingerprint, error); the host's sink stores them. The default logs to `helios_cortex.audit_sink`. New optional fields are allowed. |
+| `helios_cortex` | `Caller` | Stable | The host's caller: `username`, `token` (left out of `repr`, never logged) and free-form `attributes`, all optional. Helios authenticates no one; the engine's identity mode (2.2) decides which field it uses. |
+| `helios_cortex.plugins` | `load`, `installed`, `names`, `missing_configured`, `require_configured`, `PluginInfo`, `PluginError`, `PluginNotFound`, `PluginLoadError`, `ENGINES`, `CATALOGS`, `STORES`, `TOOLS`, `COMMANDS`, `GROUPS` | Provisional | Entry-point groups `helios.engines`, `helios.catalogs`, `helios.stores`, `helios.tools`, `helios.commands`; the group names are Stable. What a plugin object must provide is set per group: engines in 3.5 (an `Engine` subclass with `from_connection`; see `helios_cortex.engines`), catalogs in 3.7 (a class with `from_catalog`; see `helios_cortex.catalog`), stores in 4.x, tools in 5.x, commands in 5.5. Settings that name a missing plugin still load; `require_configured` is the host's startup check. |
+
+## Internal
+
+Everything not in a table above is internal. Most of it is leaving helios-cortex, which is
+why it stays where it is instead of moving under `_internal/` and then moving again:
+
+| Code | Where it goes |
+|---|---|
+| `ontology/parser.py`, the rest of `ontology/mapping.py`, `index/` modules other than the store and records | Reviewed for promotion when helios-crawl is built (Phase 10) |
+| `_pending/` | helios-serve and helios-crawl; import-linter keeps other code from importing it |
+
+## Typing
+
+helios-cortex ships `py.typed`. The type annotations on contract names are part of the contract.

helios_cortex-0.1.0/PKG-INFO

@@ -0,0 +1,38 @@
+Metadata-Version: 2.5
+Name: helios-cortex
+Version: 0.1.0
+Summary: Helios core: shared semantic query and data governance layer
+Author: Helios Team
+License-Expression: Apache-2.0
+Requires-Python: >=3.11
+Requires-Dist: alembic>=1.13
+Requires-Dist: duckdb>=1.3.0
+Requires-Dist: jsonschema>=4.23.0
+Requires-Dist: platformdirs>=4.0
+Requires-Dist: pydantic-settings>=2.6
+Requires-Dist: pydantic>=2.9.0
+Requires-Dist: pyyaml>=6.0.0
+Requires-Dist: sqlalchemy>=2.0
+Requires-Dist: sqlglot>=27.0.0
+Provides-Extra: azure
+Requires-Dist: adlfs>=2024.4; extra == 'azure'
+Provides-Extra: evidence
+Requires-Dist: lancedb>=0.39.0; extra == 'evidence'
+Requires-Dist: sentence-transformers>=6.1.0; extra == 'evidence'
+Provides-Extra: gcs
+Requires-Dist: gcsfs>=2024.6; extra == 'gcs'
+Provides-Extra: llm
+Requires-Dist: httpx>=0.27.0; extra == 'llm'
+Provides-Extra: ontology
+Requires-Dist: linkml-runtime>=1.8.0; extra == 'ontology'
+Provides-Extra: ozone
+Requires-Dist: fsspec>=2024.6; extra == 'ozone'
+Requires-Dist: requests>=2.32; extra == 'ozone'
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
+Provides-Extra: s3
+Requires-Dist: s3fs>=2024.6; extra == 's3'
+Provides-Extra: testing
+Requires-Dist: pytest>=8.0; extra == 'testing'
+Provides-Extra: trino
+Requires-Dist: trino>=0.330; extra == 'trino'

helios_cortex-0.1.0/pyproject.toml

@@ -0,0 +1,79 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "helios-cortex"
+version = "0.1.0"
+description = "Helios core: shared semantic query and data governance layer"
+authors = [{name = "Helios Team"}]
+license = "Apache-2.0"
+requires-python = ">=3.11"
+# The base install: the compiler, semantic model, DuckDB, and the model registry on
+# SQLite (SQLAlchemy and Alembic; the database itself is the standard library).
+# Everything else is an extra; tests/test_optional_dependencies.py keeps it so.
+dependencies = [
+    "duckdb>=1.3.0",
+    "jsonschema>=4.23.0",
+    "platformdirs>=4.0",
+    "pydantic>=2.9.0",
+    "pydantic-settings>=2.6",
+    "pyyaml>=6.0.0", # semantic model files and ontology mappings
+    "sqlglot>=27.0.0",
+    "sqlalchemy>=2.0", # the model registry (SQLite by default) and the sqlalchemy engine
+    "alembic>=1.13", # registry schema migrations
+]
+
+[project.optional-dependencies]
+# LLM providers (helios_cortex.llm), used to propose models.
+llm = [
+    "httpx>=0.27.0",
+]
+# LinkML ontology parsing (helios_cortex.ontology.parser).
+ontology = [
+    "linkml-runtime>=1.8.0",
+]
+# Postgres for the registry (postgresql+psycopg://...) and the sqlalchemy engine. Other
+# databases: install their SQLAlchemy driver (snowflake-sqlalchemy, mysqlclient, ...).
+postgres = [
+    "psycopg[binary]>=3.2",
+]
+# The trino engine plugin (all three identity modes).
+trino = [
+    "trino>=0.330",
+]
+# Object storage for the ArtifactStore (helios_cortex.artifact_storage), by URL scheme.
+s3 = [
+    "s3fs>=2024.6",       # s3:// (AWS S3, MinIO, Ozone's S3 gateway)
+]
+azure = [
+    "adlfs>=2024.4",      # abfs://, abfss://, az:// (ADLS Gen2, Blob storage)
+]
+gcs = [
+    "gcsfs>=2024.6",      # gs://
+]
+ozone = [
+    "fsspec>=2024.6",     # ofs://, through Ozone's HttpFS (WebHDFS) gateway
+    "requests>=2.32",
+]
+# The backend conformance suite (helios_cortex.conformance), for testing a registry or
+# storage backend.
+testing = [
+    "pytest>=8.0",
+]
+# Searching crawled documents. Without these, crawls run and evidence search
+# reports that it is unavailable. sentence-transformers brings PyTorch.
+evidence = [
+    "lancedb>=0.39.0",
+    "sentence-transformers>=6.1.0",
+]
+
+# The built-in engine plugins. Impala and Atlas come with helios-cloudera
+# (see helios_cortex.plugins). trino needs its extra; sqlalchemy needs the database's driver.
+[project.entry-points."helios.engines"]
+duckdb = "helios_cortex.engines.duckdb_engine:DuckDBEngine"
+sqlalchemy = "helios_cortex.engines.sqlalchemy_engine:SqlAlchemyEngine"
+trino = "helios_cortex.engines.trino_engine:TrinoEngine"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/helios_cortex"]

helios_cortex-0.1.0/src/helios_cortex/__init__.py

@@ -0,0 +1,18 @@
+"""helios core library: shared clients and logic for jobs and applications."""
+
+from helios_cortex.caller import Caller, IdentityError, IdentityMode
+
+__all__ = ["Caller", "IdentityError", "IdentityMode"]
+
+
+def __getattr__(name: str) -> str:
+    # __version__ is read from the installed metadata on first use, so importing
+    # helios_cortex opens no files.
+    if name == "__version__":
+        from importlib.metadata import PackageNotFoundError, version
+
+        try:
+            return version("helios-cortex")
+        except PackageNotFoundError:  # running from a source path without an install
+            return "0.0.0+unknown"
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

helios_cortex-0.1.0/src/helios_cortex/_optional.py

@@ -0,0 +1,17 @@
+"""Optional dependencies: the base install is the compiler, semantic model, DuckDB and
+SQLite; anything else comes with an extra (``pip install 'helios-cortex[llm]'``).
+
+A module that needs an extra raises ``missing_extra(...)`` from the ImportError, so
+the message says what to install instead of naming a package the user never chose.
+Base modules import those modules only where they are used.
+"""
+from __future__ import annotations
+
+
+def missing_extra(feature: str, extra: str, error: ImportError) -> ModuleNotFoundError:
+    """The error for ``feature`` when the package ``error`` failed to import is missing."""
+    return ModuleNotFoundError(
+        f"{feature} needs the optional dependency {error.name or 'it imports'}: "
+        f"pip install 'helios-cortex[{extra}]'",
+        name=error.name,
+    )

helios_cortex-0.1.0/src/helios_cortex/_pending/__init__.py

@@ -0,0 +1,8 @@
+"""Feature code that core used to reach into, held here until its package exists.
+
+- health: helios-serve (Phase 6)
+- crawler, crawler_settings, evidence, browse: helios-crawl (Phase 10)
+
+Nothing outside this directory in helios_cortex may import it; import-linter
+enforces that (burndown 1.2). Each module leaves when its package is created.
+"""

helios_cortex-0.1.0/src/helios_cortex/_pending/browse.py

@@ -0,0 +1,181 @@
+"""Browsing what the crawler found, from an ontology class down to passages
+(CR-9): the entities of a class, then one entity's documents and the passages
+that mention it.
+
+Reads each source's latest successful crawl. A finished run never changes, so
+its tables are kept in memory; runs that are no longer the latest are dropped.
+"""
+
+from __future__ import annotations
+
+import posixpath
+from typing import Any
+
+from helios_cortex._pending import evidence
+from helios_cortex.index.records import AssetRecord, EntityRecord, MentionRecord, RelationshipRecord, SegmentRecord
+from helios_cortex.index.store import IndexStore
+
+ASSETS = "helios_index.assets"
+SEGMENTS = "helios_index.segments"
+MENTIONS = "helios_index.mentions"
+ENTITY_LINKS = "helios_index.entity_links"
+RELATIONSHIPS = "helios_index.relationships"
+MAX_DOCUMENTS = 30
+
+_tables: dict[tuple[str, str], list[Any]] = {}
+
+
+def _rows(store: IndexStore, crawl_run_id: str, table: str) -> list[Any]:
+    key = (crawl_run_id, table)
+    if key not in _tables:
+        _tables[key] = store.read(table, {"crawl_run_id": crawl_run_id})
+    return _tables[key]
+
+
+def _latest(store: IndexStore) -> list[Any]:
+    latest = evidence.latest_runs(store)
+    keep = {run.crawl_run_id for run in latest}
+    for key in [k for k in _tables if k[0] not in keep]:
+        del _tables[key]
+    return latest
+
+
+def _entity_view(entity: EntityRecord, source: str) -> dict[str, Any]:
+    return {
+        "entity_id": entity.entity_id,
+        "class": entity.ontology_class,
+        "name": entity.canonical_name,
+        "keys": list(entity.external_ids),
+        "source": source,
+        "crawl_run_id": entity.crawl_run_id,
+    }
+
+
+def class_entities(
+    store: IndexStore, classes: list[str], query: str = "", limit: int = 50
+) -> dict[str, Any]:
+    """Entities of the given ontology classes, most-documented first."""
+    wanted = {c.casefold() for c in classes}
+    needle = query.strip().casefold()
+    found: list[dict[str, Any]] = []
+    for run in _latest(store):
+        documents: dict[str, set[str]] = {}
+        for rel in _rows(store, run.crawl_run_id, RELATIONSHIPS):
+            if isinstance(rel, RelationshipRecord) and (rel.source_kind, rel.target_kind) == ("asset", "entity"):
+                documents.setdefault(rel.target_id, set()).add(rel.source_id)
+        for entity in _rows(store, run.crawl_run_id, evidence.ENTITIES):
+            if not isinstance(entity, EntityRecord) or entity.ontology_class.casefold() not in wanted:
+                continue
+            haystack = " ".join([entity.canonical_name, *entity.external_ids]).casefold()
+            if needle and needle not in haystack:
+                continue
+            found.append(
+                {**_entity_view(entity, run.source), "documents": len(documents.get(entity.entity_id, ()))}
+            )
+    found.sort(key=lambda e: (-e["documents"], e["name"]))
+    return {"total": len(found), "entities": found[: max(1, min(limit, 500))]}
+
+
+def _document_name(asset: AssetRecord) -> str:
+    locator = asset.source_locator or {}
+    path = str(locator.get("key") or locator.get("uri") or locator.get("path") or "")
+    return posixpath.basename(path) or asset.asset_id
+
+
+def entity_documents(store: IndexStore, crawl_run_id: str, entity_id: str) -> dict[str, Any] | None:
+    """One entity with its documents, the passages that mention it (with the
+    character ranges of each mention), related entities and claims."""
+    run = next((r for r in _latest(store) if r.crawl_run_id == crawl_run_id), None)
+    if run is None:
+        return None
+    entities = {
+        e.entity_id: e for e in _rows(store, crawl_run_id, evidence.ENTITIES) if isinstance(e, EntityRecord)
+    }
+    entity = entities.get(entity_id)
+    if entity is None:
+        return None
+
+    relation: dict[str, str] = {}  # asset_id -> About or Mentions (About wins)
+    related: list[dict[str, Any]] = []
+    for rel in _rows(store, crawl_run_id, RELATIONSHIPS):
+        if not isinstance(rel, RelationshipRecord):
+            continue
+        if rel.source_kind == "asset" and rel.target_id == entity_id:
+            if relation.get(rel.source_id) != "About":
+                relation[rel.source_id] = rel.relationship_type
+        elif rel.source_kind == "entity" and entity_id in (rel.source_id, rel.target_id):
+            other = entities.get(rel.target_id if rel.source_id == entity_id else rel.source_id)
+            if other is not None:
+                related.append(
+                    {
+                        "relationship": rel.relationship_type,
+                        "direction": "out" if rel.source_id == entity_id else "in",
+                        "entity_id": other.entity_id,
+                        "class": other.ontology_class,
+                        "name": other.canonical_name,
+                    }
+                )
+
+    links = {
+        link.mention_id: link
+        for link in store.read(ENTITY_LINKS, {"crawl_run_id": crawl_run_id, "entity_id": entity_id})
+    }
+    by_segment: dict[str, list[dict[str, Any]]] = {}
+    for mention in _rows(store, crawl_run_id, MENTIONS):
+        link = links.get(mention.mention_id) if isinstance(mention, MentionRecord) else None
+        if link is None:
+            continue
+        by_segment.setdefault(mention.segment_id, []).append(
+            {
+                "surface_form": mention.surface_form,
+                "start": mention.start_offset,
+                "end": mention.end_offset,
+                "resolved_by": link.resolved_by,
+                "link_type": link.link_type,
+            }
+        )
+    passages: dict[str, list[dict[str, Any]]] = {}
+    for segment in _rows(store, crawl_run_id, SEGMENTS):
+        if isinstance(segment, SegmentRecord) and segment.segment_id in by_segment:
+            passages.setdefault(segment.asset_id, []).append(
+                {
+                    "segment_id": segment.segment_id,
+                    "segment_type": segment.segment_type,
+                    "ordinal": segment.ordinal,
+                    "locator": segment.locator,
+                    "text": segment.text,
+                    "mentions": sorted(
+                        by_segment[segment.segment_id], key=lambda m: (m["start"] is None, m["start"] or 0)
+                    ),
+                }
+            )
+
+    documents = []
+    for asset in _rows(store, crawl_run_id, ASSETS):
+        if not isinstance(asset, AssetRecord):
+            continue
+        if asset.asset_id not in relation and asset.asset_id not in passages:
+            continue
+        documents.append(
+            {
+                "asset_id": asset.asset_id,
+                "name": _document_name(asset),
+                "class": asset.ontology_class,
+                "mime_type": asset.mime_type,
+                "timestamp": asset.semantic_timestamp,
+                "relationship": relation.get(asset.asset_id, "Mentions"),
+                "passages": sorted(passages.get(asset.asset_id, []), key=lambda p: p["ordinal"]),
+            }
+        )
+    documents.sort(
+        key=lambda d: (d["relationship"] != "About", -len(d["passages"]), d["timestamp"] or "", d["name"])
+    )
+    claims = evidence.entity_claims(store, entity_id)
+    return {
+        "entity": _entity_view(entity, run.source),
+        "documents_total": len(documents),
+        "documents": documents[:MAX_DOCUMENTS],
+        "related": sorted(related, key=lambda r: (r["relationship"], r["name"]))[:50],
+        "claims_total": claims["claims_total"],
+        "claims": claims["claims"],
+    }

helios_cortex-0.1.0/src/helios_cortex/_pending/crawler/__init__.py

@@ -0,0 +1,6 @@
+"""Shared crawler definitions: settings (CR-0e). The crawler itself runs as a
+Workbench Job in the Helios project (apps/helios/crawler)."""
+
+from helios_cortex._pending.crawler.settings import EMPTY_SETTINGS, CrawlerSettings, load_preset, ontology_problems, presets
+
+__all__ = ["EMPTY_SETTINGS", "CrawlerSettings", "load_preset", "ontology_problems", "presets"]

helios_cortex-0.1.0/src/helios_cortex/_pending/crawler/presets/index.yaml

@@ -0,0 +1,13 @@
+# Shipped crawler settings presets. Each is <name>.yaml in this directory: a
+# complete settings document a user can load, edit and save as a version.
+presets:
+  retail-returns:
+    title: Retail returns
+    description: >-
+      Store returns over a TPC-DS warehouse: emails, chat threads and PDF return
+      reports. Identifier patterns for RMA, case and ticket numbers, the PDF
+      report's labels, and cue phrases for damage, return reason and refunds.
+
+# Before settings schema 2 one set of rules was built into the crawler. A stored
+# schema 1 document is read as this preset with the document's sections on top.
+schema_1_defaults: retail-returns