@odla-ai/kg 0.1.0

package/README.md

@@ -0,0 +1,75 @@
+# @odla-ai/kg
+
+An ontology-driven **knowledge-graph builder**. Pluggable source connectors pull
+from the web and data feeds, an LLM extracts entities + relationships against a
+**config-as-data ontology**, and a provenance-preserving graph writer persists
+the result into [odla-db](https://www.npmjs.com/package/@odla-ai/db).
+
+The library is host-agnostic: it runs in Cloudflare Workers, Node, or tests —
+the host supplies the database client, LLM keys, and a KV-like store. For a
+complete deployable consumer (workflows, hourly cron, HTTP API, viewer UI), see
+`examples/kg-demo` in the [odla-ai monorepo](https://github.com/odla-ai/odla-ai).
+
+## Architecture
+
+```
+Connector ──RawItem[]──▶ Extraction ──GraphFragment──▶ writeFragment ──Op[]──▶ odla-db
+ (how data        (LLM forced-tool       (ontology-typed     (Lookup upserts +
+  was acquired)    OR pure mapper)        nodes + edges)       links, idempotent)
+```
+
+- **`RawItem`** — a connector emits either `text` (→ LLM extraction) or a
+  ready-made `fragment` (structured feeds bypass the LLM).
+- **`GraphFragment`** — ontology-typed `nodes[]` + `edges[]`; the single shape
+  the writer consumes, whatever produced it.
+- **`Ontology`** — data, not code. It compiles to the odla-db schema
+  (`toSerializedSchema`), the writer's natural-key + merge-rule maps
+  (`naturalKeysOf`, `mergeRulesOf`, `humanAttrsOf`), and by convention the
+  extractor's tool schema.
+
+Add a source → implement `SourceConnector` + one registry line. Add an
+entity/edge type → edit the ontology data. Neither touches the pipeline.
+
+## Usage
+
+```ts
+import { init } from "@odla-ai/db";
+import {
+  Store, getProvider, makeContext, ingestFresh, writeFragment,
+  toSerializedSchema, type Ontology, type PipelineDeps,
+} from "@odla-ai/kg";
+
+// 1. Your ontology is data (see the OSINT example in examples/kg-demo).
+const ontology: Ontology = { entities: { /* … */ }, edges: { /* … */ } };
+
+// 2. Wire the host services once.
+const db = init({ appId, adminToken, endpoint });
+const provider = getProvider({ LLM_PROVIDER: "claude", ANTHROPIC_API_KEY: key });
+const deps: PipelineDeps = { db, provider, ontology, extract: myExtractor, store: new Store(kv) };
+
+// 3. Run items through the shared pipeline (or call writeFragment directly).
+await ingestFresh(step, deps, freshItems, "ingest");
+```
+
+Key seams the host supplies:
+
+- **`PipelineDeps`** — db (`AdminDb`), LLM `Provider`, the ontology, the
+  extractor (your domain craft), and a `Store` over any `KVLike`.
+- **`ContextConfig`** (→ `makeContext`) — what connectors see: ontology, lazy
+  provider, named secrets.
+- **`StepLike`** — a structural view of Cloudflare's `WorkflowStep`
+  (`asStepLike(step)`); any `do(name, cfg?, fn)` runner works, including a
+  plain inline runner in tests.
+
+## Built-in connectors
+
+`rss`, `url`, `gdelt` (text → LLM extraction) · `fred`, `edgar` (structured →
+straight to the writer) · `web-search` (LLM web-search discovery) · `gov-data`
+(template proving a new source needs no pipeline change).
+
+## Provenance & merging
+
+`writeFragment` reads existing nodes by natural key (one batched query), merges
+scalar attrs per the ontology's rules (`replace` / `prefer` / `earliest` /
+`latest`), never writes `human`-owned attrs, and commits nodes-then-links in a
+single idempotent transaction (deterministic `mutationId`).