@odla-ai/kg 0.2.3 → 0.3.0

package/llms.txt

@@ -1,536 +0,0 @@
-# @odla-ai/kg — LLM context
-
-> Ontology-driven knowledge-graph builder: pluggable source connectors, LLM extraction against a config-as-data ontology, and a provenance-preserving graph writer persisting into odla-db.
-
-> ⚠️ **Early access — pre-1.0.** Agents work from bounded runbooks; humans
-> approve credentials, production changes, releases, and merges. APIs and exact
-> package availability can change. Review the documented guarantees and
-> limitations; this software is MIT-licensed and provided without warranty.
-
-An ontology-driven **knowledge-graph builder**. Pluggable source connectors pull
-from the web and data feeds, an LLM extracts entities + relationships against a
-**config-as-data ontology**, and a provenance-preserving graph writer persists
-the result into [odla-db](https://www.npmjs.com/package/@odla-ai/db).
-
-The library is host-agnostic: it runs in Cloudflare Workers, Node, or tests —
-the host supplies the database client, LLM keys, and a KV-like store. The
-public platform and package manuals describe how to compose it with Workflows,
-scheduled triggers, an HTTP API, and a viewer UI.
-
-## Architecture
-
-```
-Connector ──RawItem[]──▶ Extraction ──GraphFragment──▶ writeFragment ──Op[]──▶ odla-db
- (how data        (LLM forced-tool       (ontology-typed     (Lookup upserts +
-  was acquired)    OR pure mapper)        nodes + edges)       links, idempotent)
-```
-
-- **`RawItem`** — a connector emits either `text` (→ LLM extraction) or a
-  ready-made `fragment` (structured feeds bypass the LLM).
-- **`GraphFragment`** — ontology-typed `nodes[]` + `edges[]`; the single shape
-  the writer consumes, whatever produced it.
-- **`Ontology`** — data, not code. It compiles to the odla-db schema
-  (`toSerializedSchema`), the writer's natural-key + merge-rule maps
-  (`naturalKeysOf`, `mergeRulesOf`, `humanAttrsOf`), and by convention the
-  extractor's tool schema.
-
-Add a source → implement `SourceConnector` + one registry line. Add an
-entity/edge type → edit the ontology data. Neither touches the pipeline.
-
-## Usage
-
-```ts
-import { init } from "@odla-ai/db";
-import {
-  Store, getProvider, makeContext, ingestFresh, writeFragment,
-  toSerializedSchema, type Ontology, type PipelineDeps,
-} from "@odla-ai/kg";
-
-// 1. Your ontology is ordinary application-owned data.
-const ontology: Ontology = { entities: { /* … */ }, edges: { /* … */ } };
-
-// 2. Wire the host services once. Env-var keys work standalone…
-const db = init({ appId, adminToken, endpoint });
-const provider = getProvider({ LLM_PROVIDER: "claude", ANTHROPIC_API_KEY: key });
-// …but platform apps should bring a configured Ai instead (no LLM keys in
-// worker env): createProvider({ provider: "claude", ai }) with the ai from
-// @odla-ai/ai's initFromPlatform. The
-// platform's default model is ignored here; kg's MODEL_TIERS govern verbs.
-const deps: PipelineDeps = { db, provider, ontology, extract: myExtractor, store: new Store(kv) };
-
-// 3. Run items through the shared pipeline (or call writeFragment directly).
-await ingestFresh(step, deps, freshItems, "ingest");
-```
-
-Key seams the host supplies:
-
-- **`PipelineDeps`** — db (`AdminDb`), LLM `Provider`, the ontology, the
-  extractor (your domain craft), and a `Store` over any `KVLike`.
-- **`ContextConfig`** (→ `makeContext`) — what connectors see: ontology, lazy
-  provider, named secrets.
-- **`StepLike`** — a structural view of Cloudflare's `WorkflowStep`
-  (`asStepLike(step)`); any `do(name, cfg?, fn)` runner works, including a
-  plain inline runner in tests.
-
-## Built-in connectors
-
-`rss`, `url`, `gdelt` (text → LLM extraction) · `fred`, `edgar` (structured →
-straight to the writer) · `web-search` (LLM web-search discovery) · `gov-data`
-(template proving a new source needs no pipeline change).
-
-## Provenance & merging
-
-`writeFragment` reads existing nodes by natural key (one batched query), merges
-scalar attrs per the ontology's rules (`replace` / `prefer` / `earliest` /
-`latest`), never writes `human`-owned attrs, and commits nodes-then-links in a
-single idempotent transaction (deterministic `mutationId`).
-
-## API reference (generated from dist/index.d.ts, v0.2.3)
-
-```ts
-import { Value, Op, AdminDb, SerializedAttr, Cardinality, SerializedSchema, Entity } from '@odla-ai/db';
-import { Ai } from '@odla-ai/ai';
-
-/**
- * The output of a connector: one acquired item. Exactly one of `text` or
- * `fragment` is set — `text` goes through LLM extraction, `fragment` is already
- * ontology-typed and goes straight to the graph writer (structured feeds).
- */
-interface RawItem {
-    url: string;
-    title: string;
-    date?: string;
-    sourceName: string;
-    sourceKind: string;
-    watch?: string | null;
-    text?: string;
-    fragment?: GraphFragment;
-}
-/** A graph node = an odla-db entity in a namespace, keyed by a natural key. */
-interface GraphNode {
-    type: string;
-    key: string;
-    attrs: Record<string, Value>;
-}
-/** A graph edge = an odla-db `link` from one node to another by forward label. */
-interface GraphEdge {
-    from: {
-        type: string;
-        key: string;
-    };
-    to: {
-        type: string;
-        key: string;
-    };
-    label: string;
-}
-/** The single shape the graph writer consumes, whatever produced it. */
-interface GraphFragment {
-    nodes: GraphNode[];
-    edges: GraphEdge[];
-}
-/** What an extractor turns a text RawItem into: a fragment plus a prose summary. */
-interface ExtractResult {
-    fragment: GraphFragment;
-    summary: string;
-}
-/** Provider-agnostic LLM usage accounting. */
-interface Usage {
-    calls: number;
-    inputTokens: number;
-    outputTokens: number;
-}
-/** Create zeroed provider-agnostic usage totals for one pipeline run. */
-declare function emptyUsage(): Usage;
-/** Mutate an accumulator by adding call, input-token, and output-token totals. */
-declare function addUsage(into: Usage, more: Usage): void;
-
-/** node type → unique natural-key attribute name. */
-type NaturalKeys = Record<string, string>;
-/**
- * Pure fragment → ops mapping (no read/merge). Node upserts by Lookup natural
- * key, then links — nodes always before links so `link` targets resolve.
- * Used by tests and as the building block for the provenance path below.
- */
-declare function fragmentToOps(fragment: GraphFragment, keys: NaturalKeys): Op[];
-/**
- * Provenance-preserving write. Reads existing nodes by natural key (one batched
- * query), merges scalar attrs per the ontology's rules, drops human-owned
- * attrs, then writes merged node updates + all links in one atomic, idempotent
- * transaction. `mutationId` must be deterministic so odla-db can deduplicate
- * the write on retry; this does not make preceding reads or external effects
- * exactly-once.
- */
-declare function writeFragment(db: AdminDb, fragment: GraphFragment, ontology: Ontology, mutationId: string): Promise<{
-    txId: number;
-    ops: number;
-}>;
-
-/** How a scalar attribute is reconciled when a node is re-discovered. */
-type MergeRule = "replace" | "prefer" | "earliest" | "latest";
-type AttrType = SerializedAttr["type"];
-interface AttrSpec {
-    type: AttrType;
-    unique?: boolean;
-    indexed?: boolean;
-    optional?: boolean;
-    /** The pipeline NEVER writes this attr (hand-edited in a UI); omitted from every update. */
-    human?: boolean;
-    /** Provenance rule for re-discovery. Default "replace". */
-    merge?: MergeRule;
-    /** Shown to the LLM when this attr appears in an extraction schema. */
-    desc?: string;
-}
-interface EntitySpec {
-    /** Unique attr used for Lookup upsert (MERGE key). Must be `unique` in attrs. */
-    naturalKey: string;
-    attrs: Record<string, AttrSpec>;
-    /** FTS fields for fuzzy resolution (P5). */
-    searchable?: string[];
-}
-interface EdgeSpec {
-    from: string;
-    to: string;
-    forwardLabel: string;
-    reverseLabel: string;
-    forwardHas?: Cardinality;
-    reverseHas?: Cardinality;
-}
-interface Ontology {
-    entities: Record<string, EntitySpec>;
-    edges: Record<string, EdgeSpec>;
-}
-/** node type → natural-key attr name (for the graph writer's Lookup refs). */
-declare function naturalKeysOf(o: Ontology): NaturalKeys;
-/** Attr name → merge rule, for every entity type (used by the provenance writer). */
-declare function mergeRulesOf(o: Ontology): Record<string, Record<string, MergeRule>>;
-/** Attr names the pipeline must never write, per entity type. */
-declare function humanAttrsOf(o: Ontology): Record<string, Set<string>>;
-/** Compile the ontology to an odla-db SerializedSchema to POST at /app/:id/schema. */
-declare function toSerializedSchema(o: Ontology): SerializedSchema;
-
-/**
- * Reconcile one scalar attribute when a node is re-discovered. Port of agnacl's
- * provenance rules (utils/markdown.ts). Empty incoming never clobbers; empty
- * existing is filled. Ordering rules compare stringwise (ISO dates sort right).
- */
-declare function mergeScalar(rule: MergeRule, existing: Value | undefined, incoming: Value | undefined): Value | undefined;
-/**
- * Merge an incoming attr set into an existing one per the type's rules. `human`
- * attrs are excluded upstream (never in `incoming`); this only reconciles what
- * the pipeline owns. Returns the attrs to write (update leaves unlisted attrs intact).
- */
-declare function mergeAttrs(rules: Record<string, MergeRule>, existing: Record<string, Value>, incoming: Record<string, Value>): Record<string, Value>;
-
-/** FTS5 search over a namespace's text fields (odla-db QueryOpts.search). */
-declare function searchEntities(db: AdminDb, type: string, q: string, limit?: number): Promise<Entity[]>;
-/**
- * Resolve a name to an existing entity's natural key via FTS — for merging a
- * re-discovery that slugified differently. Conservative: only returns a hit when
- * the top match's name/key normalizes identically to the query.
- */
-declare function resolveName(db: AdminDb, type: string, keyAttr: string, name: string): Promise<string | null>;
-/**
- * Canonicalize a fragment's searchable nodes against existing entities via FTS,
- * remapping node keys (and the edges that reference them) to the canonical key.
- * Used on the lower-volume discovery/enrich paths, not high-volume ingest.
- */
-declare function canonicalizeFragment(db: AdminDb, fragment: GraphFragment, ontology: Ontology): Promise<GraphFragment>;
-
-/**
- * The LLM seam, now a thin adapter over @odla-ai/ai (which was itself
- * generalized from this package's original Provider). Three verbs stay:
- * extract (forced structured tool call), search (web-search-enabled text),
- * write (plain generation). Callers never see vendor or oracle types.
- */
-type ProviderFamily = "claude" | "openai";
-/** Role → model tiers per family — config as data. `cheap` is for gate-style
- * calls, `extract` for structured extraction (and prose writing), `search`
- * for web-search-enabled calls. */
-declare const MODEL_TIERS: Record<ProviderFamily, {
-    cheap: string;
-    extract: string;
-    search: string;
-}>;
-/** A forced structured tool call: schema in, validated object out. */
-interface ExtractCall {
-    system: string;
-    user: string;
-    tool: {
-        name: string;
-        description?: string;
-        parameters: Record<string, unknown>;
-    };
-    model?: string;
-    maxTokens?: number;
-}
-/** A web-search-enabled call: the model searches and returns text (caller tolerant-parses). */
-interface SearchCall {
-    system: string;
-    user: string;
-    model?: string;
-    maxTokens?: number;
-}
-/** A plain text-generation call (no tools, no search) — e.g. digest prose. */
-interface WriteCall {
-    system: string;
-    user: string;
-    model?: string;
-    maxTokens?: number;
-}
-interface Provider {
-    id: ProviderFamily;
-    /** The underlying @odla-ai/ai client — escape hatch for advanced use. */
-    ai: Ai;
-    extract<T = unknown>(c: ExtractCall): Promise<{
-        value: T;
-        usage: Usage;
-    }>;
-    search(c: SearchCall): Promise<{
-        text: string;
-        usage: Usage;
-    }>;
-    write(c: WriteCall): Promise<{
-        text: string;
-        usage: Usage;
-    }>;
-}
-interface LlmConfig {
-    provider: ProviderFamily;
-    anthropicKey?: string;
-    openaiKey?: string;
-    /** Inject a pre-built Ai (tests, custom catalog); keys are ignored when set. */
-    ai?: Ai;
-}
-/** The library primitive: an explicit config in, the three-verb seam out. */
-declare function createProvider(cfg: LlmConfig): Provider;
-/** The env vars the provider factory reads — a structural slice, not the
- * worker Env, so any host (worker, script, test) can supply plain strings. */
-interface LlmEnv {
-    LLM_PROVIDER?: string;
-    ANTHROPIC_API_KEY?: string;
-    OPENAI_API_KEY?: string;
-}
-/** Build the configured provider from host env, failing before use when its key is missing. */
-declare function getProvider(env: LlmEnv): Provider;
-
-/**
- * Tolerant JSON extraction from LLM prose — web-search models return JSON inside
- * explanatory text. Ported from agnacl services/enrichment.ts.
- */
-declare function extractJson<T = unknown>(text: string): T | null;
-/** Tolerantly parse the first JSON-array-shaped span in model prose, or return an empty array. */
-declare function extractJsonArray<T = unknown>(text: string): T[];
-
-/** Ambient services a connector may use. Host-agnostic: secrets come through
- * secret(), never a worker Env, so the library runs anywhere. */
-interface ConnectorContext {
-    now: Date;
-    fetchText(url: string): Promise<string>;
-    secret(name: string): string | undefined;
-    ontology: Ontology;
-    /** The configured LLM provider (lazy — only web-search/discovery connectors need it). */
-    provider(): Provider;
-}
-/**
- * The extensibility seam. A connector emits normalized RawItems — text-bearing
- * (→ LLM extraction) or fragment-bearing (structured → straight to the writer).
- * Adding a source = a new module implementing this + one registry line.
- */
-interface SourceConnector<Cfg = unknown> {
-    id: string;
-    capabilities: {
-        fetch: boolean;
-        discover?: boolean;
-        structured: boolean;
-    };
-    parseConfig(raw: unknown): Cfg;
-    fetch(cfg: Cfg, ctx: ConnectorContext): Promise<RawItem[]>;
-    /** Web-search-style net-new discovery (P4). */
-    discover?(cfg: Cfg, ctx: ConnectorContext): Promise<RawItem[]>;
-}
-/** One unit of work for the ingest workflow: which connector + its config. */
-interface ConnectorJob {
-    kind: string;
-    name: string;
-    watch?: string | null;
-    cfg: unknown;
-}
-
-/** Resolve a built-in source connector by its stable registry id. */
-declare function getConnector(id: string): SourceConnector | undefined;
-/** List registered connector ids in deterministic insertion order. */
-declare function connectorIds(): string[];
-/** Everything a ConnectorContext needs, supplied by the host (worker, script, test). */
-interface ContextConfig {
-    ontology: Ontology;
-    provider: () => Provider;
-    /** Named secrets (API keys, user agents) — e.g. the worker Env, or a plain object. */
-    secrets?: Record<string, string | undefined>;
-    now?: Date;
-}
-/** Create a host-supplied connector context with lazy provider construction. */
-declare function makeContext(cfg: ContextConfig): ConnectorContext;
-
-/** Fetch one URL → a text-bearing RawItem. Used both directly (/trigger) and as a connector. */
-declare function fetchUrl(url: string): Promise<RawItem>;
-
-/**
- * Sources are DATA, not code. Each entry names a connector (`kind`) + its config
- * + scheduling. Add a feed → add an entry; no redeploy of the pipeline. P5 moves
- * this into KV behind a /config admin endpoint so edits don't need a deploy at all.
- */
-interface SourceEntry {
-    kind: string;
-    name: string;
-    enabled?: boolean;
-    cadenceHours?: number;
-    watch?: string | null;
-    url?: string;
-    query?: string;
-    [key: string]: unknown;
-}
-/** Return source entries not explicitly disabled, preserving configured order. */
-declare function enabledSources(sources: SourceEntry[]): SourceEntry[];
-/** Enabled sources whose cadence lands on this UTC hour. */
-declare function jobsForHour(sources: SourceEntry[], hour: number): ConnectorJob[];
-/** Every enabled source, ignoring cadence (manual /trigger?kind=sources). */
-declare function allJobs(sources: SourceEntry[]): ConnectorJob[];
-
-/**
- * The slice of a KV store the pipeline needs. Cloudflare's KVNamespace
- * satisfies it structurally; tests pass an in-memory map. Keeping the library
- * off Workers types is what lets it publish as a plain isomorphic package.
- */
-interface KVLike {
-    get(key: string): Promise<string | null>;
-    put(key: string, value: string, opts?: {
-        expirationTtl?: number;
-    }): Promise<void>;
-}
-/**
- * Operational state in KV (never the graph itself — that lives in odla-db).
- * Ported/trimmed from agnacl services/kv.ts: seen-URL dedup + lastrun stamps.
- */
-declare class Store {
-    private kv;
-    constructor(kv: KVLike);
-    private seenKey;
-    hasSeen(url: string): Promise<boolean>;
-    markSeen(url: string): Promise<void>;
-    /** Drop items whose URL we've already processed (checked BEFORE any LLM work). */
-    filterNew(items: RawItem[]): Promise<RawItem[]>;
-    setLastRun(kind: string, at?: string): Promise<void>;
-    getLastRun(kind: string): Promise<string | null>;
-    /** Per-entity enrichment cooldown so we don't re-search the same node constantly. */
-    isEnrichCooling(key: string): Promise<boolean>;
-    markEnriched(key: string): Promise<void>;
-    /** Runtime source config (edit without redeploy). null = fall back to the static SOURCES. */
-    getConfigSources(): Promise<SourceEntry[] | null>;
-    setConfigSources(list: SourceEntry[]): Promise<void>;
-}
-
-/**
- * Cloudflare Workflow hardening — ported verbatim from agnacl
- * (utils/workflow-helpers.ts). resilientStep adds structured error logging
- * without changing retry semantics; the step configs replace the engine's slow
- * default retry with fast exponential backoff.
- */
-declare function resilientStep<T>(stepName: string, fn: () => Promise<T>): Promise<T>;
-/** I/O-bound steps: KV reads, feed fetches. */
-declare const IO_STEP_CONFIG: {
-    retries: {
-        limit: number;
-        delay: "5 seconds";
-        backoff: "exponential";
-    };
-    timeout: "2 minutes";
-};
-/** Graph write steps. */
-declare const COMMIT_STEP_CONFIG: {
-    retries: {
-        limit: number;
-        delay: "3 seconds";
-        backoff: "exponential";
-    };
-    timeout: "90 seconds";
-};
-/** LLM generation steps (can be slow). */
-declare const LLM_STEP_CONFIG: {
-    retries: {
-        limit: number;
-        delay: "5 seconds";
-        backoff: "exponential";
-    };
-    timeout: "3 minutes";
-};
-/** Retry/timeout shape of the step configs above. */
-interface StepConfig {
-    retries?: {
-        limit: number;
-        delay: string;
-        backoff?: string;
-    };
-    timeout?: string;
-}
-/**
- * Structural view of Cloudflare's WorkflowStep. WorkflowStep.do constrains its
- * result to Rpc.Serializable<T>, which cannot terminate (TS2589) on the
- * recursive Value type now that the @odla/* workspace packages give
- * @odla-ai/db's re-exports real definitions. A live WorkflowStep satisfies
- * this shape at runtime; widen through asStepLike at the workflow boundary.
- */
-interface StepLike {
-    do<T>(name: string, fn: () => Promise<T>): Promise<T>;
-    do<T>(name: string, config: StepConfig, fn: () => Promise<T>): Promise<T>;
-}
-/** Widen a WorkflowStep to StepLike (type-level only, no runtime cost). */
-declare function asStepLike(step: unknown): StepLike;
-
-/** Everything the shared ingest loop needs, supplied by the host. The
- * extractor is the ontology's domain craft (e.g. the OSINT extractor); the
- * library stays ontology-agnostic. */
-interface PipelineDeps {
-    db: AdminDb;
-    provider: Provider;
-    ontology: Ontology;
-    extract: (provider: Provider, item: RawItem) => Promise<ExtractResult | null>;
-    store: Store;
-}
-/**
- * Shared per-item processing: each RawItem is its own step (non-blocking,
- * replay-cached). Structured items (fragment) skip the LLM; text items extract.
- * `resolve` FTS-canonicalizes new nodes against existing ones (discovery path).
- * Used by both the ingest (fetch) and discover (web-search) workflows.
- */
-declare function ingestFresh(step: StepLike, deps: PipelineDeps, fresh: RawItem[], kind: string, resolve?: boolean): Promise<void>;
-
-/**
- * Identity + dedup helpers. The slug / urlHash a node resolves to IS its
- * natural key for Lookup-based upserts, so these must be stable and pure.
- * Ported from agnacl (utils/markdown.ts slugify, services/kv.ts normalizeUrl).
- */
-/** NFKD-normalize, strip accents, lowercase, non-alphanumerics → "-", cap 60. */
-declare function slugify(input: string): string;
-/** Drop tracking params, hash, and trailing slash so the same page maps to one key. */
-declare function normalizeUrl(raw: string): string;
-/** Stable short key for a URL (FNV-1a 32-bit, base36). Used as the `signal` natural key. */
-declare function urlHash(raw: string): string;
-/**
- * Reject descriptive phrases masquerading as people ("three craftsmen",
- * "the founders"). A real name is 2-4 capitalized tokens, no digits.
- * Ported from agnacl utils/names.ts (isLikelyPersonName).
- */
-declare function isLikelyPersonName(name: string): boolean;
-
-/**
- * Dependency-free HTML → text extraction (Workers-isolate friendly: no DOM/jsdom).
- * Ported from agnacl services/content-extractor.ts.
- */
-/** Strip scripts/styles/tags and collapse whitespace; cap length. */
-declare function htmlToText(html: string, maxChars?: number): string;
-/** Fetch a URL and return readable text. Best-effort: returns "" on any failure. */
-declare function fetchText(url: string, maxChars?: number): Promise<string>;
-
-export { type AttrSpec, type AttrType, COMMIT_STEP_CONFIG, type ConnectorContext, type ConnectorJob, type ContextConfig, type EdgeSpec, type EntitySpec, type ExtractCall, type ExtractResult, type GraphEdge, type GraphFragment, type GraphNode, IO_STEP_CONFIG, type KVLike, LLM_STEP_CONFIG, type LlmConfig, type LlmEnv, MODEL_TIERS, type MergeRule, type NaturalKeys, type Ontology, type PipelineDeps, type Provider, type ProviderFamily, type RawItem, type SearchCall, type SourceConnector, type SourceEntry, type StepConfig, type StepLike, Store, type Usage, type WriteCall, addUsage, allJobs, asStepLike, canonicalizeFragment, connectorIds, createProvider, emptyUsage, enabledSources, extractJson, extractJsonArray, fetchText, fetchUrl, fragmentToOps, getConnector, getProvider, htmlToText, humanAttrsOf, ingestFresh, isLikelyPersonName, jobsForHour, makeContext, mergeAttrs, mergeRulesOf, mergeScalar, naturalKeysOf, normalizeUrl, resilientStep, resolveName, searchEntities, slugify, toSerializedSchema, urlHash, writeFragment };
-```