@promptev/context-engine 0.0.3 → 0.0.4

package/dist/skills/context-engine/SKILL.md

@@ -74,7 +74,35 @@ await engine.search(q, { principals: user.groups }); // scoped
   `compute`, `map_reduce`, `get_neighbors`, `traverse`, `find_related`,
   `community_summary`.
   `engine.searchKnowledgeBase({ action, principals, scope })` is the
-  same function `createMcpApp` serves. `knowledgeToolDefinition()` gives
+  same function `createMcpApp` serves. `discover` returns the SCHEMA, not
+  just an inventory, and for EVERY document type. Each listed document carries
+  `structure` — `sheets` (name, column headers, row count; the sheet name is
+  the key `compute` exposes in `dfs`) for a workbook, `sections` and
+  `last_page` for a document with headings, `keys` for JSON, and `chunks` for
+  anything else — plus `document_type` and `mode`. Every list inside a
+  `structure` is capped, with the remainder reported as `more_columns` /
+  `more_sections` / `more_keys` / `more_sheets` (absent when nothing was cut)
+  and, while the remainder is worth fetching, a filled-in `next_action`. Past
+  `MAX_INVITED_ITEMS` the invitation is WITHHELD and replaced by `instead` —
+  a model cannot know what a full list costs until it has paid for it, so
+  offering a sixteen-thousand-column fetch would be the tool's decision, not
+  the caller's; a list that long is a signal to compute over the sheet or read
+  the document, never to list it. **The cap is for the call that did NOT name
+  its documents:** `discover` (and `list`) now accept `document_ids`, filtered
+  in SQL, and a `discover` scoped to specific documents answers their
+  structure WHOLE and narrows `fields_by_type` and `document_types` to them
+  too. The host's `scope` ceiling still wins — a named id outside it is
+  dropped, never reached.
+  Alongside them, `fields_by_type` and `document_types` are BOTH keyed by
+  `(kind, type)` — a `document_type` is a label an LLM wrote, so a PDF invoice
+  and a CSV of invoice rows can share one, and keying on the label alone would
+  merge their fields. `fields_by_type` rows carry `fields` (name, data type,
+  how many in-scope documents) plus `more_fields` (names past the cap);
+  `document_types` rows carry `documents` and `with_fields`. `type` is written
+  only where `extract_structured` was opted into, so it is null on a default
+  deployment — which is why `kind` is there. All of it is ACL- and
+  scope-filtered and masked by an output redaction policy. `list` returns the
+  same page WITHOUT the structure — browsing does not pay for it. `knowledgeToolDefinition()` gives
   the name, description and JSON Schema if you are wiring it into your own
   agent loop. `compute` and `mapReduce` accept a host callable that REPLACES
   the built-in one — that is the seam for your own permission, billing and

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@promptev/context-engine",
-  "version": "0.0.3",
+  "version": "0.0.4",
   "description": "Promptev Context Engine — language-agnostic ingestion + hybrid retrieval (TypeScript)",
   "license": "Apache-2.0",
   "author": "Promptev Inc",

package/src/skills/context-engine/SKILL.md

@@ -74,7 +74,35 @@ await engine.search(q, { principals: user.groups }); // scoped
   `compute`, `map_reduce`, `get_neighbors`, `traverse`, `find_related`,
   `community_summary`.
   `engine.searchKnowledgeBase({ action, principals, scope })` is the
-  same function `createMcpApp` serves. `knowledgeToolDefinition()` gives
+  same function `createMcpApp` serves. `discover` returns the SCHEMA, not
+  just an inventory, and for EVERY document type. Each listed document carries
+  `structure` — `sheets` (name, column headers, row count; the sheet name is
+  the key `compute` exposes in `dfs`) for a workbook, `sections` and
+  `last_page` for a document with headings, `keys` for JSON, and `chunks` for
+  anything else — plus `document_type` and `mode`. Every list inside a
+  `structure` is capped, with the remainder reported as `more_columns` /
+  `more_sections` / `more_keys` / `more_sheets` (absent when nothing was cut)
+  and, while the remainder is worth fetching, a filled-in `next_action`. Past
+  `MAX_INVITED_ITEMS` the invitation is WITHHELD and replaced by `instead` —
+  a model cannot know what a full list costs until it has paid for it, so
+  offering a sixteen-thousand-column fetch would be the tool's decision, not
+  the caller's; a list that long is a signal to compute over the sheet or read
+  the document, never to list it. **The cap is for the call that did NOT name
+  its documents:** `discover` (and `list`) now accept `document_ids`, filtered
+  in SQL, and a `discover` scoped to specific documents answers their
+  structure WHOLE and narrows `fields_by_type` and `document_types` to them
+  too. The host's `scope` ceiling still wins — a named id outside it is
+  dropped, never reached.
+  Alongside them, `fields_by_type` and `document_types` are BOTH keyed by
+  `(kind, type)` — a `document_type` is a label an LLM wrote, so a PDF invoice
+  and a CSV of invoice rows can share one, and keying on the label alone would
+  merge their fields. `fields_by_type` rows carry `fields` (name, data type,
+  how many in-scope documents) plus `more_fields` (names past the cap);
+  `document_types` rows carry `documents` and `with_fields`. `type` is written
+  only where `extract_structured` was opted into, so it is null on a default
+  deployment — which is why `kind` is there. All of it is ACL- and
+  scope-filtered and masked by an output redaction policy. `list` returns the
+  same page WITHOUT the structure — browsing does not pay for it. `knowledgeToolDefinition()` gives
   the name, description and JSON Schema if you are wiring it into your own
   agent loop. `compute` and `mapReduce` accept a host callable that REPLACES
   the built-in one — that is the seam for your own permission, billing and