@promptev/context-engine 0.0.2 → 0.0.4

package/dist/skills/context-engine/SKILL.md

@@ -62,6 +62,59 @@ await engine.search(q, { principals: user.groups }); // scoped
   no default. If you hit a `TypeError`, supply real ones — do **not** silence
   it with `principals: () => null`, which disables access control on a public
   port. Anonymous is `() => []`. Trusted is `() => TRUSTED`.
+- **Search picks its own mode unless you name one.** `search({ mode: null })` (the
+  default) adds the graph leg when the deployment has a graph AND something in
+  scope was ingested with `mode: "graph"` — a model cannot know which documents
+  those are, so it is not asked. Pass `mode: "hybrid"` to force the old
+  behaviour. INGESTION is unchanged and stays explicit. An unreachable graph
+  database is skipped rather than waited on, and every graph failure degrades
+  to the hybrid legs instead of failing the search.
+- **The knowledge tool is one call with THIRTEEN actions** — `discover`,
+  `search`, `get_doc`, `get_docs`, `get_chunks`, `list`, `query_meta`,
+  `compute`, `map_reduce`, `get_neighbors`, `traverse`, `find_related`,
+  `community_summary`.
+  `engine.searchKnowledgeBase({ action, principals, scope })` is the
+  same function `createMcpApp` serves. `discover` returns the SCHEMA, not
+  just an inventory, and for EVERY document type. Each listed document carries
+  `structure` — `sheets` (name, column headers, row count; the sheet name is
+  the key `compute` exposes in `dfs`) for a workbook, `sections` and
+  `last_page` for a document with headings, `keys` for JSON, and `chunks` for
+  anything else — plus `document_type` and `mode`. Every list inside a
+  `structure` is capped, with the remainder reported as `more_columns` /
+  `more_sections` / `more_keys` / `more_sheets` (absent when nothing was cut)
+  and, while the remainder is worth fetching, a filled-in `next_action`. Past
+  `MAX_INVITED_ITEMS` the invitation is WITHHELD and replaced by `instead` —
+  a model cannot know what a full list costs until it has paid for it, so
+  offering a sixteen-thousand-column fetch would be the tool's decision, not
+  the caller's; a list that long is a signal to compute over the sheet or read
+  the document, never to list it. **The cap is for the call that did NOT name
+  its documents:** `discover` (and `list`) now accept `document_ids`, filtered
+  in SQL, and a `discover` scoped to specific documents answers their
+  structure WHOLE and narrows `fields_by_type` and `document_types` to them
+  too. The host's `scope` ceiling still wins — a named id outside it is
+  dropped, never reached.
+  Alongside them, `fields_by_type` and `document_types` are BOTH keyed by
+  `(kind, type)` — a `document_type` is a label an LLM wrote, so a PDF invoice
+  and a CSV of invoice rows can share one, and keying on the label alone would
+  merge their fields. `fields_by_type` rows carry `fields` (name, data type,
+  how many in-scope documents) plus `more_fields` (names past the cap);
+  `document_types` rows carry `documents` and `with_fields`. `type` is written
+  only where `extract_structured` was opted into, so it is null on a default
+  deployment — which is why `kind` is there. All of it is ACL- and
+  scope-filtered and masked by an output redaction policy. `list` returns the
+  same page WITHOUT the structure — browsing does not pay for it. `knowledgeToolDefinition()` gives
+  the name, description and JSON Schema if you are wiring it into your own
+  agent loop. `compute` and `mapReduce` accept a host callable that REPLACES
+  the built-in one — that is the seam for your own permission, billing and
+  approval rules, and supplying one makes the action available regardless of
+  the deployment's own gate.
+- **`scope` is REQUIRED on `createMcpApp`** and on
+  `engine.searchKnowledgeBase`: the ceiling of source ids the tool may ever
+  reach. Like `principals` it is host-supplied and NEVER a tool argument — a
+  model can name any source id, and a tool that believed it would cross
+  tenants. Pass `scope: UNSCOPED` only to mean the whole corpus on purpose. A
+  caller narrows within the ceiling; an id outside it is dropped silently, and
+  a request left with nothing returns nothing (never everything).
 - **`acl=null` on `updateDocument` UNRESTRICTS the document.** "Leave it
   alone" is expressed by omitting the argument (the `UNSET` sentinel), because
   `null` already means unrestricted and cannot also mean absent. Same for
@@ -130,11 +183,14 @@ app.route(
 `createFastifyPlugin` (`@promptev/context-engine/fastify`) and
 `createMcpApp` (`@promptev/context-engine/mcp`) take the same two
 arguments. MCP has no request object: `principals` is a zero-argument
-callable resolved fresh on every tool call. The CLI `context-engine mcp`
+callable resolved fresh on every tool call, and `createMcpApp` also requires
+`scope` — the source-id ceiling, resolved the same way. The CLI `context-engine mcp`
 binds `127.0.0.1` and answers as anonymous (`principals=[]`).
 
 Not on a supported framework? Call `engine.search()` / `engine.ingest()`
-from your own handler — the routers are thin wrappers over the same core.
+from your own handler — the routers are thin wrappers over the same core. For
+an agent, `engine.searchKnowledgeBase({ action, principals, scope })` is the
+same one tool the MCP server serves, with no transport in the way.
 
 ## Lifecycle
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@promptev/context-engine",
-  "version": "0.0.2",
+  "version": "0.0.4",
   "description": "Promptev Context Engine — language-agnostic ingestion + hybrid retrieval (TypeScript)",
   "license": "Apache-2.0",
   "author": "Promptev Inc",

package/src/skills/context-engine/SKILL.md

@@ -62,6 +62,59 @@ await engine.search(q, { principals: user.groups }); // scoped
   no default. If you hit a `TypeError`, supply real ones — do **not** silence
   it with `principals: () => null`, which disables access control on a public
   port. Anonymous is `() => []`. Trusted is `() => TRUSTED`.
+- **Search picks its own mode unless you name one.** `search({ mode: null })` (the
+  default) adds the graph leg when the deployment has a graph AND something in
+  scope was ingested with `mode: "graph"` — a model cannot know which documents
+  those are, so it is not asked. Pass `mode: "hybrid"` to force the old
+  behaviour. INGESTION is unchanged and stays explicit. An unreachable graph
+  database is skipped rather than waited on, and every graph failure degrades
+  to the hybrid legs instead of failing the search.
+- **The knowledge tool is one call with THIRTEEN actions** — `discover`,
+  `search`, `get_doc`, `get_docs`, `get_chunks`, `list`, `query_meta`,
+  `compute`, `map_reduce`, `get_neighbors`, `traverse`, `find_related`,
+  `community_summary`.
+  `engine.searchKnowledgeBase({ action, principals, scope })` is the
+  same function `createMcpApp` serves. `discover` returns the SCHEMA, not
+  just an inventory, and for EVERY document type. Each listed document carries
+  `structure` — `sheets` (name, column headers, row count; the sheet name is
+  the key `compute` exposes in `dfs`) for a workbook, `sections` and
+  `last_page` for a document with headings, `keys` for JSON, and `chunks` for
+  anything else — plus `document_type` and `mode`. Every list inside a
+  `structure` is capped, with the remainder reported as `more_columns` /
+  `more_sections` / `more_keys` / `more_sheets` (absent when nothing was cut)
+  and, while the remainder is worth fetching, a filled-in `next_action`. Past
+  `MAX_INVITED_ITEMS` the invitation is WITHHELD and replaced by `instead` —
+  a model cannot know what a full list costs until it has paid for it, so
+  offering a sixteen-thousand-column fetch would be the tool's decision, not
+  the caller's; a list that long is a signal to compute over the sheet or read
+  the document, never to list it. **The cap is for the call that did NOT name
+  its documents:** `discover` (and `list`) now accept `document_ids`, filtered
+  in SQL, and a `discover` scoped to specific documents answers their
+  structure WHOLE and narrows `fields_by_type` and `document_types` to them
+  too. The host's `scope` ceiling still wins — a named id outside it is
+  dropped, never reached.
+  Alongside them, `fields_by_type` and `document_types` are BOTH keyed by
+  `(kind, type)` — a `document_type` is a label an LLM wrote, so a PDF invoice
+  and a CSV of invoice rows can share one, and keying on the label alone would
+  merge their fields. `fields_by_type` rows carry `fields` (name, data type,
+  how many in-scope documents) plus `more_fields` (names past the cap);
+  `document_types` rows carry `documents` and `with_fields`. `type` is written
+  only where `extract_structured` was opted into, so it is null on a default
+  deployment — which is why `kind` is there. All of it is ACL- and
+  scope-filtered and masked by an output redaction policy. `list` returns the
+  same page WITHOUT the structure — browsing does not pay for it. `knowledgeToolDefinition()` gives
+  the name, description and JSON Schema if you are wiring it into your own
+  agent loop. `compute` and `mapReduce` accept a host callable that REPLACES
+  the built-in one — that is the seam for your own permission, billing and
+  approval rules, and supplying one makes the action available regardless of
+  the deployment's own gate.
+- **`scope` is REQUIRED on `createMcpApp`** and on
+  `engine.searchKnowledgeBase`: the ceiling of source ids the tool may ever
+  reach. Like `principals` it is host-supplied and NEVER a tool argument — a
+  model can name any source id, and a tool that believed it would cross
+  tenants. Pass `scope: UNSCOPED` only to mean the whole corpus on purpose. A
+  caller narrows within the ceiling; an id outside it is dropped silently, and
+  a request left with nothing returns nothing (never everything).
 - **`acl=null` on `updateDocument` UNRESTRICTS the document.** "Leave it
   alone" is expressed by omitting the argument (the `UNSET` sentinel), because
   `null` already means unrestricted and cannot also mean absent. Same for
@@ -130,11 +183,14 @@ app.route(
 `createFastifyPlugin` (`@promptev/context-engine/fastify`) and
 `createMcpApp` (`@promptev/context-engine/mcp`) take the same two
 arguments. MCP has no request object: `principals` is a zero-argument
-callable resolved fresh on every tool call. The CLI `context-engine mcp`
+callable resolved fresh on every tool call, and `createMcpApp` also requires
+`scope` — the source-id ceiling, resolved the same way. The CLI `context-engine mcp`
 binds `127.0.0.1` and answers as anonymous (`principals=[]`).
 
 Not on a supported framework? Call `engine.search()` / `engine.ingest()`
-from your own handler — the routers are thin wrappers over the same core.
+from your own handler — the routers are thin wrappers over the same core. For
+an agent, `engine.searchKnowledgeBase({ action, principals, scope })` is the
+same one tool the MCP server serves, with no transport in the way.
 
 ## Lifecycle