kbforge-okfquery 0.1.0__py3-none-any.whl

kbforge_okfquery-0.1.0.dist-info/METADATA

@@ -0,0 +1,244 @@
+Metadata-Version: 2.5
+Name: kbforge-okfquery
+Version: 0.1.0
+Summary: SQL over an OKF v0.2 bundle, via DuckDB
+Author-email: Qing <qingye779@gmail.com>
+License: MIT
+Keywords: duckdb,knowledge-base,okf,open-knowledge-format,sql
+Requires-Python: >=3.12
+Requires-Dist: duckdb>=1.1
+Requires-Dist: pytz>=2024.1
+Requires-Dist: pyyaml>=6
+Description-Content-Type: text/markdown
+
+# okfquery
+
+SQL over an [OKF v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
+bundle, via DuckDB. `okfquery` loads a published bundle into an in-memory DuckDB
+connection and hands the connection back — no daemon, no cache, no committed
+artifact. It depends on no kbforge code at runtime, so it reads any OKF v0.2
+bundle, kbforge-built or not.
+
+## Install and use
+
+```bash
+uv pip install kbforge-okfquery   # or: pip install kbforge-okfquery
+okfquery query "select count(*) from concepts" --bundle path/to/bundle
+okfquery shell --bundle path/to/bundle
+```
+
+`query` runs one SQL statement and prints the result (`--format table|json|csv`).
+`shell` writes a temporary `.duckdb` file, opens the `duckdb` CLI on it, and
+deletes the file when the shell exits.
+
+## Schema
+
+```
+$ okfquery schema
+CREATE TABLE concepts (
+    path          VARCHAR NOT NULL,
+    type          VARCHAR,
+    title         VARCHAR,
+    description   VARCHAR,
+    generated_by  VARCHAR,
+    -- TIMESTAMPTZ, never TIMESTAMP. Not because a naive column would corrupt
+    -- the instant on this package's load path -- `load` binds an aware Python
+    -- datetime through executemany, so the instant survives a naive column
+    -- fine. What a naive column loses is the type: values come back with no
+    -- tzinfo, so every comparison against now() (itself TIMESTAMPTZ) needs a
+    -- cast, and the column asserts UTC by convention with nothing recording
+    -- that it does. §4.4 law 4 exists to force an *aware* stamp; a column that
+    -- cannot hold one throws away exactly the property the law buys.
+    generated_at  TIMESTAMPTZ,
+    facets        JSON,
+    body          VARCHAR
+);
+
+CREATE TABLE sources (
+    path          VARCHAR NOT NULL,
+    -- A real column. kbforge's synthesize.assemble puts the OWNING anchor first
+    -- and grounding anchors after it, and that ordering is the only thing
+    -- telling owner from ground -- OKF has no field for it. Discarding ordinal
+    -- during unnest would destroy the signal.
+    ordinal       INTEGER NOT NULL,
+    id            VARCHAR,
+    resource      VARCHAR,
+    content_hash  VARCHAR
+);
+
+CREATE TABLE links (
+    path          VARCHAR NOT NULL,
+    target        VARCHAR NOT NULL
+);
+
+CREATE TABLE problems (
+    path          VARCHAR NOT NULL,
+    kind          VARCHAR NOT NULL,
+    detail        VARCHAR NOT NULL
+);
+```
+
+`path` is the bundle-relative concept path and the join key throughout.
+`sources.ordinal = 0` is the owning source for a kbforge-produced bundle (the
+convention `synthesize.assemble` writes) — on a foreign bundle it is merely
+source order, with no error to tell you that. `facets` is one JSON column, open
+by construction, so `facets->>'owner'` reaches whatever a connector attached.
+`problems` is additive: a file that failed to parse still gets a `concepts` row
+with NULLs for what could not be read, plus one `problems` row per distinct
+failure — nothing a broken concept says gets silently dropped from an audit.
+
+## Queries, run against the messy test fixture
+
+All four queries below were run as shown against
+`packages/okfquery/tests/fixtures/messy` — 9 `.md` files under `concepts/`, 7 of
+them concepts (`concepts/index.md` and `concepts/log.md` are reserved and
+fenceless, so they are skipped; `concepts/claims/index.md` is reserved-*named*
+but carries frontmatter, so it counts).
+
+**What is built on a given upstream document:**
+
+```bash
+$ okfquery query "
+SELECT c.path, s.ordinal
+FROM concepts c JOIN sources s USING (path)
+WHERE s.resource = 'https://wiki/ok';
+" --bundle packages/okfquery/tests/fixtures/messy
+┌─────────────────────────┬─────────┐
+│          path           │ ordinal │
+│         varchar         │  int32  │
+├─────────────────────────┼─────────┤
+│ concepts/ok/overview.md │       0 │
+└─────────────────────────┴─────────┘
+```
+
+**Staleness by owning system** (`ordinal = 0` picks the owning source, not a
+grounding one — see the ordinal note above):
+
+```bash
+$ okfquery query "
+SELECT split_part(s.id, ':', 1) AS system,
+       count(*) AS concepts,
+       min(c.generated_at) AS oldest
+FROM concepts c JOIN sources s USING (path)
+WHERE s.ordinal = 0
+GROUP BY 1 ORDER BY oldest;
+" --bundle packages/okfquery/tests/fixtures/messy
+┌─────────┬──────────┬──────────────────────────┐
+│ system  │ concepts │          oldest          │
+│ varchar │  int64   │ timestamp with time zone │
+├─────────┼──────────┼──────────────────────────┤
+│ wiki    │        3 │ 2026-08-23 00:00:00+00   │
+└─────────┴──────────┴──────────────────────────┘
+```
+
+Only one system shows up on this fixture: most of the messy bundle's files
+exist to exercise a `problems` kind and never got far enough to have a
+`sources` entry at all, so they don't join into this query — which is correct,
+not a bug in the query.
+
+**Orphans — nothing links to them:**
+
+```bash
+$ okfquery query "
+SELECT c.path FROM concepts c
+WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.target = c.path);
+" --bundle packages/okfquery/tests/fixtures/messy
+┌───────────────────────────────────┐
+│               path                │
+│              varchar              │
+├───────────────────────────────────┤
+│ concepts/badyaml/overview.md      │
+│ concepts/claims/index.md          │
+│ concepts/incomplete/overview.md   │
+│ concepts/nofence/overview.md      │
+│ concepts/notmapping/overview.md   │
+│ concepts/unterminated/overview.md │
+└───────────────────────────────────┘
+```
+
+**Everything that failed to parse** (`--format csv` here, since the default
+table renderer truncates long `detail` strings):
+
+```bash
+$ okfquery query "SELECT path, kind, detail FROM problems ORDER BY path, kind;" \
+    --bundle packages/okfquery/tests/fixtures/messy --format csv
+path,kind,detail
+concepts/badyaml/overview.md,invalid-yaml,frontmatter is not valid YAML: ParserError
+concepts/incomplete/overview.md,bad-links,'links' entry 3 is not a string
+concepts/incomplete/overview.md,bad-sources,'sources' entry 0 has no 'resource' (§5.1 requires one)
+concepts/incomplete/overview.md,bad-timestamp,'generated.at' 'not-a-date' is not an ISO-8601 datetime
+concepts/incomplete/overview.md,missing-required,"missing required OKF keys: title, description"
+concepts/nofence/overview.md,no-frontmatter,file does not open a '---' frontmatter fence
+concepts/notmapping/overview.md,frontmatter-not-mapping,"frontmatter parses to list, not a mapping"
+concepts/unterminated/overview.md,unterminated-frontmatter,frontmatter fence is opened but never closed by a closing '---'
+```
+
+## The mirror, and why both obvious joins are wrong
+
+`--mirror <path>` attaches a `mirror` view over a kbforge mirror directory
+(`read_json_auto('<mirror>/*.json')`) — the same `CanonicalDocument` slots
+`mirror.slot_key` writes, `anchor` struct included. It answers a different
+question than the bundle alone can: not "what does this concept say" but "does
+it still say what its source currently does."
+
+**Both of the joins you'd reach for first are lossy, in different ways:**
+
+- `sources.id = mirror.doc_id` holds for every shipped connector, but
+  `synthesize._source_entry` builds `id` from `anchor.system`/`anchor.native_id`
+  while `doc_id` is the mirror's own field — nothing enforces the two agree for
+  a third-party connector. Where they diverge, this join produces a silent
+  false miss, not an error.
+- `sources.content_hash = mirror.anchor.content_hash` as an **equi-join** is not
+  the sound alternative either. `kbforge.pipeline.run` calls
+  `commit(mirror_path, docs)` right after `kbforge_publish`, and publishing only
+  opens a review request — kbforge never merges. Between publish and merge the
+  mirror already holds the new hash while the bundle still carries the previous
+  run's, so an equi-join on hash returns **zero rows for every concept with a
+  review request open** — exactly the ones an audit most wants to see.
+
+The correct form: join on `id` with a **`LEFT JOIN`**, not an inner join, then
+**compare** hashes instead of joining on them — an inner join silently drops
+every source with no mirror row at all, which is exactly the "never
+published" case the query needs to report. A match means current; a mismatch
+means an update is sitting in review; a source with no mirror row was never
+published. Run for real against the messy fixture (`concepts/ok/overview.md`'s
+two sources — an owning `wiki:ok` and a grounding `notes:ok` — plus two other
+concepts whose sources have no mirror row at all) with a synthetic
+two-document mirror covering only `wiki:ok` and `notes:ok`, one hash left
+matching and one changed to simulate an update stuck in review:
+
+```bash
+$ okfquery query "
+SELECT s.path,
+       s.id,
+       s.content_hash AS bundle_hash,
+       m.anchor.content_hash AS mirror_hash,
+       CASE WHEN m.doc_id IS NULL THEN 'never published'
+            WHEN s.content_hash = m.anchor.content_hash THEN 'current'
+            ELSE 'update in review' END AS status
+FROM sources s LEFT JOIN mirror m ON s.id = m.doc_id
+ORDER BY s.path, s.id;
+" --bundle packages/okfquery/tests/fixtures/messy --mirror /path/to/mirror
+┌─────────────────────────────────┬─────────────────┬─────────────┬─────────────┬──────────────────┐
+│               path              │       id        │ bundle_hash │ mirror_hash │      status      │
+│             varchar             │     varchar     │   varchar   │   varchar   │     varchar      │
+├─────────────────────────────────┼─────────────────┼─────────────┼─────────────┼──────────────────┤
+│ concepts/claims/index.md        │ wiki:claims     │ h-claims    │ NULL        │ never published  │
+│ concepts/incomplete/overview.md │ wiki:incomplete │ NULL        │ NULL        │ never published  │
+│ concepts/ok/overview.md         │ notes:ok        │ h-ground    │ h-ground-v2 │ update in review │
+│ concepts/ok/overview.md         │ wiki:ok         │ h-own       │ h-own       │ current          │
+└─────────────────────────────────┴─────────────────┴─────────────┴─────────────┴──────────────────┘
+```
+
+That does not fix the `id` gap above: a connector whose anchor disagrees with
+its `doc_id` still misses this join silently. There is no workaround for that
+here — only the honest statement of it.
+
+## Not built
+
+- **No full-text search out of the box.** `body` is a plain column; `INSTALL
+  fts` and an index over it is a documented recipe, not a feature.
+- **No remote bundles.** `--bundle` is a local checkout path; reading one over
+  HTTP wants a caching story this package deliberately avoids.
+- **No MCP server.** `load()` is already small enough to wrap in one `query`
+  tool and a schema resource; nothing here does that yet.

kbforge_okfquery-0.1.0.dist-info/RECORD

@@ -0,0 +1,9 @@
+okfquery/__init__.py,sha256=kbuky3_HNMhxsf1pdOo49QiScaRc3UBa20q0Gdf_qeA,446
+okfquery/cli.py,sha256=93mqOdVi7L2VWQTOJztMmpUVs-9ke7b_ZWdcGkAFryI,4986
+okfquery/load.py,sha256=DVeDithpDRiZZxRglakY7NYWxPgwKcyoeo7XcX71-io,5058
+okfquery/parse.py,sha256=DOnEJyLTrDX2qMsUHpe4Tt775r57KkrQ5yw7IqcyAn4,8752
+okfquery/schema.py,sha256=z8E0vQLB5SNNlh3QAJWw9jxt1WAaQYY15vvPwR4QWJk,1856
+kbforge_okfquery-0.1.0.dist-info/METADATA,sha256=SpivqOVFTmZBNdPClLYY52htoKZ6GRzDGbpIrF---6M,12493
+kbforge_okfquery-0.1.0.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+kbforge_okfquery-0.1.0.dist-info/entry_points.txt,sha256=r6vWPNoNezHdt_JlRk-ZTquezvP7lrsKcsrAZK1H89Y,47
+kbforge_okfquery-0.1.0.dist-info/RECORD,,

kbforge_okfquery-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any

kbforge_okfquery-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+okfquery = okfquery.cli:main

okfquery/__init__.py

@@ -0,0 +1,14 @@
+"""SQL over an OKF v0.2 bundle.
+
+`load()` returns a DuckDB connection, deliberately unwrapped: an OKF bundle is
+just a DuckDB database, and anything this package wrapped would be a worse
+version of what DuckDB already offers."""
+
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+from okfquery.load import EmptyMirrorError, load
+from okfquery.schema import SCHEMA_SQL
+
+__all__ = ["EmptyMirrorError", "SCHEMA_SQL", "__version__", "load"]

okfquery/cli.py

@@ -0,0 +1,144 @@
+"""argparse over `load()`. Four verbs, no query logic of its own.
+
+There is deliberately no --format parquet and no --output: DuckDB writes parquet
+from inside the SQL (`COPY (...) TO 'out.parquet'`), and a second export path
+here would be a worse version of a feature already shipping."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import duckdb
+
+from okfquery.load import EmptyMirrorError, load
+from okfquery.schema import SCHEMA_SQL
+
+
+def _bundle_missing(bundle: str) -> bool:
+    """True when `bundle` has no `concepts/` directory to scan.
+
+    `load()` must keep answering an empty bundle (no .md files under an
+    existing `concepts/`) rather than erroring -- that is a real, if boring,
+    OKF bundle. A *missing* `concepts/` is different: it means the path is
+    wrong, and `scan()` silently returning `[]` for it would make `check`
+    report "no problems" and exit 0 against a bundle that was never loaded at
+    all. That is a CLI-level concern, not `load()`'s, so it is caught here.
+    """
+    return not (Path(bundle) / "concepts").is_dir()
+
+
+def _connect(args: argparse.Namespace) -> duckdb.DuckDBPyConnection:
+    mirror = Path(args.mirror) if args.mirror else None
+    return load(Path(args.bundle), mirror=mirror)
+
+
+def _emit(con: duckdb.DuckDBPyConnection, sql: str, fmt: str) -> None:
+    if fmt == "table":
+        print(con.sql(sql))
+        return
+    cursor = con.execute(sql)
+    columns = [d[0] for d in cursor.description]
+    rows = cursor.fetchall()
+    if fmt == "json":
+        print(json.dumps([dict(zip(columns, r)) for r in rows], default=str))
+        return
+    writer = csv.writer(sys.stdout, lineterminator="\n")
+    writer.writerow(columns)
+    writer.writerows(rows)
+
+
+def _shell(args: argparse.Namespace) -> int:
+    if _bundle_missing(args.bundle):
+        print(f"no concepts/ directory under {args.bundle}", file=sys.stderr)
+        return 2
+    # subprocess, NEVER exec. exec replaces this process, so the TemporaryDirectory
+    # finalizer would never run and a database holding every concept body would
+    # outlive the session -- quietly breaking the ephemeral guarantee the whole
+    # design rests on.
+    with tempfile.TemporaryDirectory(prefix="okfquery-") as tmp:
+        database = Path(tmp) / "bundle.duckdb"
+        mirror = Path(args.mirror) if args.mirror else None
+        try:
+            load(Path(args.bundle), mirror=mirror, database=str(database)).close()
+        except EmptyMirrorError as exc:
+            print(exc, file=sys.stderr)
+            return 2
+        try:
+            return subprocess.run(["duckdb", str(database)], check=False).returncode
+        except FileNotFoundError:
+            print(
+                "the `duckdb` CLI is not on PATH. Install it "
+                "(https://duckdb.org/docs/installation/) and run:\n"
+                f"  duckdb {database}\n"
+                "note: that file is deleted when this command exits.",
+                file=sys.stderr,
+            )
+            return 2
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="okfquery", description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    def bundle_args(p: argparse.ArgumentParser) -> None:
+        p.add_argument("--bundle", default=".", help="bundle root (default: .)")
+        p.add_argument("--mirror", default=None, help="kbforge mirror to attach")
+
+    query = sub.add_parser("query", help="run one SQL statement")
+    query.add_argument("sql")
+    query.add_argument("--format", choices=("table", "json", "csv"), default="table")
+    bundle_args(query)
+
+    shell = sub.add_parser("shell", help="open the duckdb CLI on this bundle")
+    bundle_args(shell)
+
+    sub.add_parser("schema", help="print the DDL")
+
+    check = sub.add_parser("check", help="exit 1 if any file failed to parse")
+    bundle_args(check)
+
+    args = parser.parse_args(argv)
+
+    if args.command == "schema":
+        print(SCHEMA_SQL.strip())
+        return 0
+    if args.command == "shell":
+        return _shell(args)
+
+    if _bundle_missing(args.bundle):
+        print(f"no concepts/ directory under {args.bundle}", file=sys.stderr)
+        return 2
+
+    try:
+        con = _connect(args)
+    except EmptyMirrorError as exc:
+        print(exc, file=sys.stderr)
+        return 2
+
+    if args.command == "check":
+        rows = con.execute(
+            "select path, kind, detail from problems order by path, kind"
+        ).fetchall()
+        if not rows:
+            print("no problems")
+            return 0
+        for path, kind, detail in rows:
+            print(f"{path}: {kind}: {detail}", file=sys.stderr)
+        return 1
+
+    try:
+        _emit(con, args.sql, args.format)
+    except duckdb.Error as exc:
+        print(exc, file=sys.stderr)
+        return 2
+    return 0
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())

okfquery/load.py

@@ -0,0 +1,132 @@
+"""Bundle on disk -> a DuckDB connection with the schema filled.
+
+Ephemeral by design: nothing is cached and nothing is written, so the tables
+cannot disagree with the files they came from. That is also why no index is
+committed into a bundle -- an index derived from `ProposedChange.concepts` would
+be a THIRD carrier of the same concept, and kbforge already has two."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import duckdb
+
+from okfquery.parse import is_reserved, parse
+from okfquery.schema import SCHEMA_SQL
+
+
+class EmptyMirrorError(Exception):
+    """A --mirror path holding no document slots. Raised rather than passed
+    through, because read_json_auto's own error on a zero-match glob names a
+    pattern, not the directory the user typed."""
+
+
+def scan(bundle: Path) -> list[Path]:
+    """Concept files, sorted. Reserved OKF artifacts are excluded here and
+    nowhere else, so the count invariant has exactly one definition.
+
+    The glob is `concepts/**/*.md` because that is where kbforge's
+    `synthesize.concept_path` puts everything -- which keeps a bundle repo's own
+    README.md and docs/ out of the tables."""
+    root = bundle / "concepts"
+    if not root.is_dir():
+        return []
+    # errors="replace", not the bare default: a mis-encoded file is exactly the
+    # pathology this tool exists to surface. Raising here would abort the whole
+    # load over one file; U+FFFD replacement characters let it through to
+    # `parse()`, which turns it into a `concepts` row (NULLs where the decode
+    # broke the frontmatter) plus a `problems` row, same as any other bad file.
+    return [
+        p
+        for p in sorted(root.rglob("*.md"))
+        if not is_reserved(p.name, p.read_text("utf-8", errors="replace"))
+    ]
+
+
+def _mirror_view(con: duckdb.DuckDBPyConnection, mirror: Path) -> None:
+    # Shallow, never `**/*.json`: grounding.SIDECAR_DIR keeps flat
+    # {doc_id: content_hash} maps in <mirror>/_grounding/, and unioning those
+    # into a CanonicalDocument view would wreck read_json_auto's inferred schema.
+    slots = sorted(mirror.glob("*.json"))
+    if not slots:
+        raise EmptyMirrorError(
+            f"mirror {mirror} holds no document slots (*.json); "
+            "nothing to attach as the `mirror` view"
+        )
+    pattern = str(mirror / "*.json").replace("'", "''")
+    con.execute(f"CREATE VIEW mirror AS SELECT * FROM read_json_auto('{pattern}')")
+
+
+def load(
+    bundle: Path,
+    mirror: Path | None = None,
+    *,
+    database: str = ":memory:",
+) -> duckdb.DuckDBPyConnection:
+    """An OKF v0.2 bundle as a DuckDB connection.
+
+    Returns the connection itself, unwrapped. An OKF bundle is just a DuckDB
+    database: COPY ... TO 'x.parquet', ATTACH, joining against your own CSV, and
+    INSTALL fts over `body` all work because the real object comes back.
+
+    `database` exists for `okfquery shell`, which needs the same tables in a file
+    the duckdb CLI can open. Everything else takes the default.
+    """
+    bundle = Path(bundle)
+    con = duckdb.connect(database)
+    # Set before anything is inserted: TIMESTAMPTZ values render in the session
+    # timezone, and a bundle should read the same on every machine.
+    con.execute("SET timezone = 'UTC'")
+    con.execute(SCHEMA_SQL)
+
+    concepts: list[tuple] = []
+    sources: list[tuple] = []
+    links: list[tuple] = []
+    problems: list[tuple] = []
+
+    for file in scan(bundle):
+        path = file.relative_to(bundle).as_posix()
+        # Same errors="replace" as scan(), and for the same reason: one
+        # mis-encoded file must produce a row, not a traceback that aborts
+        # every other concept in the bundle along with it.
+        concept = parse(file.read_text("utf-8", errors="replace"))
+        concepts.append(
+            (
+                path,
+                concept.type,
+                concept.title,
+                concept.description,
+                concept.generated_by,
+                concept.generated_at,
+                json.dumps(concept.facets) if concept.facets else None,
+                concept.body,
+            )
+        )
+        for ordinal, source in enumerate(concept.sources):
+            sources.append(
+                (
+                    path,
+                    ordinal,
+                    source["id"],
+                    source["resource"],
+                    source["content_hash"],
+                )
+            )
+        links.extend((path, target) for target in concept.links)
+        problems.extend((path, p.kind, p.detail) for p in concept.problems)
+
+    if concepts:
+        con.executemany(
+            "INSERT INTO concepts VALUES (?, ?, ?, ?, ?, ?, ?, ?)", concepts
+        )
+    if sources:
+        con.executemany("INSERT INTO sources VALUES (?, ?, ?, ?, ?)", sources)
+    if links:
+        con.executemany("INSERT INTO links VALUES (?, ?)", links)
+    if problems:
+        con.executemany("INSERT INTO problems VALUES (?, ?, ?)", problems)
+
+    if mirror is not None:
+        _mirror_view(con, Path(mirror))
+    return con

okfquery/parse.py

@@ -0,0 +1,252 @@
+"""Pure frontmatter parsing for OKF v0.2 concepts.
+
+No filesystem, no clock, no DuckDB: text in, a `Concept` and its `Problem`s out.
+That purity is why every row of the problem ladder is testable from a string
+literal.
+
+This module deliberately does NOT reuse kbforge's `validate._parse_frontmatter`,
+and not only because okfquery imports no kbforge. That function collapses
+no-fence, unterminated-fence, and broken-YAML alike to `{}` -- correct for a gate,
+which needs only "no usable frontmatter" and fails either way. Here, *which* of
+the three occurred is the entire finding: a missing fence is a renderer bug,
+broken YAML is a hand-edit, an unterminated fence is a truncated write. Do not
+"fix" this duplication by importing kbforge's version."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+
+import yaml
+
+RESERVED = frozenset({"index.md", "log.md"})
+"""OKF §8 directory listings and change logs. See `is_reserved` for the rule."""
+
+# The keys OKF owns at the head of a concept. Everything else in the frontmatter
+# is a facet. Mirrors kbforge's synthesize.OKF_OWNED, which is what keeps a
+# source field named `type` out of the facet map on the emit side.
+OKF_OWNED = frozenset({"type", "title", "description", "generated", "sources", "links"})
+
+_REQUIRED = ("type", "title", "description", "generated", "sources")
+
+
+@dataclass(frozen=True)
+class Problem:
+    kind: str
+    detail: str
+
+
+@dataclass
+class Concept:
+    type: str | None = None
+    title: str | None = None
+    description: str | None = None
+    generated_by: str | None = None
+    generated_at: datetime | None = None
+    facets: dict = field(default_factory=dict)
+    body: str = ""
+    sources: list[dict] = field(default_factory=list)
+    links: list[str] = field(default_factory=list)
+    problems: list[Problem] = field(default_factory=list)
+
+
+def is_reserved(basename: str, content: str) -> bool:
+    """Whether a file under `concepts/` is a non-concept OKF artifact.
+
+    Copied from kbforge's `validate._check_strict_okf`, deliberately and with
+    this note: a file is exempt only if its basename is reserved AND it does not
+    open a frontmatter fence, because an `index.md` bearing frontmatter is
+    claiming to be a concept and exempting it would skip every check. Keyed on
+    the raw fence, not on a parse result -- "no frontmatter" and "broken
+    frontmatter" must not look alike."""
+    return basename in RESERVED and not content.lstrip().startswith("---")
+
+
+def _split(content: str) -> tuple[str | None, str, Problem | None]:
+    """(raw frontmatter, body, problem). Raw is None when there is none to parse."""
+    text = content.lstrip()
+    if not text.startswith("---"):
+        return (
+            None,
+            content,
+            Problem("no-frontmatter", "file does not open a '---' frontmatter fence"),
+        )
+    _, _, rest = text.partition("---")
+    raw, sep, body = rest.partition("\n---")
+    if not sep:
+        return (
+            None,
+            content,
+            Problem(
+                "unterminated-frontmatter",
+                "frontmatter fence is opened but never closed by a closing '---'",
+            ),
+        )
+    return raw, body.lstrip("\n"), None
+
+
+def _generated(front: dict, out: Concept) -> None:
+    block = front.get("generated")
+    if not isinstance(block, dict):
+        out.problems.append(
+            Problem(
+                "bad-generated", f"'generated' is {type(block).__name__}, not a mapping"
+            )
+        )
+        return
+    by = block.get("by")
+    out.generated_by = by if isinstance(by, str) and by else None
+    if out.generated_by is None:
+        out.problems.append(
+            Problem("bad-generated", "'generated.by' is missing or empty")
+        )
+    at = block.get("at")
+    if at is None:
+        out.problems.append(Problem("bad-generated", "'generated.at' is missing"))
+        return
+    parsed = _timestamp(at, out)
+    if parsed is not None and parsed.utcoffset() is None:
+        out.problems.append(
+            Problem(
+                "naive-timestamp",
+                f"'generated.at' {at!r} carries no UTC offset (§4.4 law 4 requires "
+                "an aware stamp); read as UTC",
+            )
+        )
+        parsed = parsed.replace(tzinfo=UTC)
+    out.generated_at = parsed
+
+
+def _timestamp(at: object, out: Concept) -> datetime | None:
+    if isinstance(at, datetime):
+        return at
+    try:
+        return datetime.fromisoformat(str(at))
+    except (TypeError, ValueError):
+        out.problems.append(
+            Problem(
+                "bad-timestamp", f"'generated.at' {at!r} is not an ISO-8601 datetime"
+            )
+        )
+        return None
+
+
+def _sources(front: dict, out: Concept) -> None:
+    raw = front.get("sources")
+    if not isinstance(raw, list):
+        out.problems.append(
+            Problem("bad-sources", f"'sources' is {type(raw).__name__}, not a list")
+        )
+        return
+    for i, entry in enumerate(raw):
+        if not isinstance(entry, dict):
+            out.problems.append(
+                Problem(
+                    "bad-sources",
+                    f"'sources' entry {i} is {type(entry).__name__}, not a mapping",
+                )
+            )
+            continue
+        resource = entry.get("resource")
+        if not isinstance(resource, str) or not resource:
+            out.problems.append(
+                Problem(
+                    "bad-sources",
+                    f"'sources' entry {i} has no 'resource' (§5.1 requires one)",
+                )
+            )
+            resource = None
+        # Carried regardless: problems are additive, and dropping the entry
+        # would silently renumber every ordinal after it.
+        out.sources.append(
+            {
+                "id": entry.get("id") if isinstance(entry.get("id"), str) else None,
+                "resource": resource,
+                "content_hash": entry.get("content_hash")
+                if isinstance(entry.get("content_hash"), str)
+                else None,
+            }
+        )
+
+
+def _links(front: dict, out: Concept) -> None:
+    raw = front.get("links")
+    if raw is None:
+        return
+    if not isinstance(raw, list):
+        out.problems.append(
+            Problem("bad-links", f"'links' is {type(raw).__name__}, not a list")
+        )
+        return
+    for entry in raw:
+        if isinstance(entry, str):
+            out.links.append(entry)
+        else:
+            out.problems.append(
+                Problem("bad-links", f"'links' entry {entry!r} is not a string")
+            )
+
+
+def _facets(front: dict) -> dict:
+    """Frontmatter keys OKF does not own. Mirrors synthesize._facets on the emit
+    side: scalars and scalar lists only, so a nested mapping never lands in a
+    column typed for values."""
+    scalar = (str, int, float, bool)
+
+    def ok(v: object) -> bool:
+        if isinstance(v, scalar):
+            return True
+        return isinstance(v, list) and all(isinstance(i, scalar) for i in v)
+
+    return {k: v for k, v in front.items() if k not in OKF_OWNED and ok(v)}
+
+
+def parse(content: str) -> Concept:
+    """One concept file's text into a `Concept` plus the problems it carries.
+
+    Never raises. A file this cannot read still yields a Concept -- with NULLs
+    for what was unreadable -- because dropping it would make an audit tool hide
+    exactly the files most likely to be wrong."""
+    out = Concept()
+    raw, body, problem = _split(content)
+    out.body = body
+    if problem is not None:
+        out.problems.append(problem)
+        return out
+    try:
+        front = yaml.safe_load(raw)
+    except yaml.YAMLError as exc:
+        out.problems.append(
+            Problem(
+                "invalid-yaml",
+                f"frontmatter is not valid YAML: {exc.__class__.__name__}",
+            )
+        )
+        return out
+    if not isinstance(front, dict):
+        out.problems.append(
+            Problem(
+                "frontmatter-not-mapping",
+                f"frontmatter parses to {type(front).__name__}, not a mapping",
+            )
+        )
+        return out
+
+    missing = [k for k in _REQUIRED if front.get(k) is None]
+    if missing:
+        out.problems.append(
+            Problem(
+                "missing-required", f"missing required OKF keys: {', '.join(missing)}"
+            )
+        )
+
+    for key in ("type", "title", "description"):
+        value = front.get(key)
+        setattr(out, key, value if isinstance(value, str) else None)
+    out.facets = _facets(front)
+    if front.get("generated") is not None:
+        _generated(front, out)
+    if front.get("sources") is not None:
+        _sources(front, out)
+    _links(front, out)
+    return out

okfquery/schema.py

@@ -0,0 +1,51 @@
+"""The DDL, defined once.
+
+Explicit CREATE TABLE rather than a dataframe round-trip, for two reasons: an
+empty bundle still answers queries instead of erroring on a missing table, and
+the column types are pinned rather than inferred."""
+
+from __future__ import annotations
+
+SCHEMA_SQL = """
+CREATE TABLE concepts (
+    path          VARCHAR NOT NULL,
+    type          VARCHAR,
+    title         VARCHAR,
+    description   VARCHAR,
+    generated_by  VARCHAR,
+    -- TIMESTAMPTZ, never TIMESTAMP. Not because a naive column would corrupt
+    -- the instant on this package's load path -- `load` binds an aware Python
+    -- datetime through executemany, so the instant survives a naive column
+    -- fine. What a naive column loses is the type: values come back with no
+    -- tzinfo, so every comparison against now() (itself TIMESTAMPTZ) needs a
+    -- cast, and the column asserts UTC by convention with nothing recording
+    -- that it does. §4.4 law 4 exists to force an *aware* stamp; a column that
+    -- cannot hold one throws away exactly the property the law buys.
+    generated_at  TIMESTAMPTZ,
+    facets        JSON,
+    body          VARCHAR
+);
+
+CREATE TABLE sources (
+    path          VARCHAR NOT NULL,
+    -- A real column. kbforge's synthesize.assemble puts the OWNING anchor first
+    -- and grounding anchors after it, and that ordering is the only thing
+    -- telling owner from ground -- OKF has no field for it. Discarding ordinal
+    -- during unnest would destroy the signal.
+    ordinal       INTEGER NOT NULL,
+    id            VARCHAR,
+    resource      VARCHAR,
+    content_hash  VARCHAR
+);
+
+CREATE TABLE links (
+    path          VARCHAR NOT NULL,
+    target        VARCHAR NOT NULL
+);
+
+CREATE TABLE problems (
+    path          VARCHAR NOT NULL,
+    kind          VARCHAR NOT NULL,
+    detail        VARCHAR NOT NULL
+);
+"""