@promptev/context-engine 0.0.3 → 0.0.5

package/dist/skills/context-engine/SKILL.md

@@ -74,7 +74,35 @@ await engine.search(q, { principals: user.groups }); // scoped
   `compute`, `map_reduce`, `get_neighbors`, `traverse`, `find_related`,
   `community_summary`.
   `engine.searchKnowledgeBase({ action, principals, scope })` is the
-  same function `createMcpApp` serves. `knowledgeToolDefinition()` gives
+  same function `createMcpApp` serves. `discover` returns the SCHEMA, not
+  just an inventory, and for EVERY document type. Each listed document carries
+  `structure` — `sheets` (name, column headers, row count; the sheet name is
+  the key `compute` exposes in `dfs`) for a workbook, `sections` and
+  `last_page` for a document with headings, `keys` for JSON, and `chunks` for
+  anything else — plus `document_type` and `mode`. Every list inside a
+  `structure` is capped, with the remainder reported as `more_columns` /
+  `more_sections` / `more_keys` / `more_sheets` (absent when nothing was cut)
+  and, while the remainder is worth fetching, a filled-in `next_action`. Past
+  `MAX_INVITED_ITEMS` the invitation is WITHHELD and replaced by `instead` —
+  a model cannot know what a full list costs until it has paid for it, so
+  offering a sixteen-thousand-column fetch would be the tool's decision, not
+  the caller's; a list that long is a signal to compute over the sheet or read
+  the document, never to list it. **The cap is for the call that did NOT name
+  its documents:** `discover` (and `list`) now accept `document_ids`, filtered
+  in SQL, and a `discover` scoped to specific documents answers their
+  structure WHOLE and narrows `fields_by_type` and `document_types` to them
+  too. The host's `scope` ceiling still wins — a named id outside it is
+  dropped, never reached.
+  Alongside them, `fields_by_type` and `document_types` are BOTH keyed by
+  `(kind, type)` — a `document_type` is a label an LLM wrote, so a PDF invoice
+  and a CSV of invoice rows can share one, and keying on the label alone would
+  merge their fields. `fields_by_type` rows carry `fields` (name, data type,
+  how many in-scope documents) plus `more_fields` (names past the cap);
+  `document_types` rows carry `documents` and `with_fields`. `type` is written
+  only where `extract_structured` was opted into, so it is null on a default
+  deployment — which is why `kind` is there. All of it is ACL- and
+  scope-filtered and masked by an output redaction policy. `list` returns the
+  same page WITHOUT the structure — browsing does not pay for it. `knowledgeToolDefinition()` gives
   the name, description and JSON Schema if you are wiring it into your own
   agent loop. `compute` and `mapReduce` accept a host callable that REPLACES
   the built-in one — that is the seam for your own permission, billing and
@@ -102,6 +130,31 @@ await engine.search(q, { principals: user.groups }); // scoped
   header before the LLM sees a schema, and sweeps the result.
 - Filing a document under an ACL the caller does not hold returns **403**, on
   ingest and on PATCH alike.
+- **`executeTool` cuts its `result` to 8,000 characters and 100 rows by
+  default.** A cut result is `{ _truncated, _original_size }` — a model
+  reading only that prefix will say "not found" about data that was there.
+  Size the budget to the model's window with `resultMaxChars` /
+  `resultMaxRows` (omitted = default, `null` = no limit; rows are per table,
+  and a db tool's own `max_rows`, default 1000, still applies first).
+  `responseMode: "tsv"` works for ANY tool: every array of objects in the
+  result (db `rows`, an HTTP body's list, a function's returned list) becomes
+  a TSV string under the same key — header first, `\N` = NULL, nested values
+  as JSON cells, tab/newline/backslash escaped — and the budget cuts at whole
+  rows, reporting `_result_shaping: { "<dotted.path>": { rows_returned,
+  rows_omitted } }`. Everything else stays JSON. The audit row keeps the
+  original under its own 50KB cap regardless. An http tool can store
+  `response_mode: "tsv"` in its config (every method, `QUERY` included;
+  omitted = `json`); an explicit `responseMode` on the call wins. The
+  conversion alone is `formatResult(result, "tsv")`.
+
+  ```ts
+  const out = await engine.executeTool("db_sales", { query: "SELECT * FROM orders" }, {
+    principals,
+    resultMaxChars: 200_000,
+    resultMaxRows: null,
+    responseMode: "tsv",
+  });
+  ```
 
 ## Search
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@promptev/context-engine",
-  "version": "0.0.3",
+  "version": "0.0.5",
   "description": "Promptev Context Engine — language-agnostic ingestion + hybrid retrieval (TypeScript)",
   "license": "Apache-2.0",
   "author": "Promptev Inc",

package/src/skills/context-engine/SKILL.md

@@ -74,7 +74,35 @@ await engine.search(q, { principals: user.groups }); // scoped
   `compute`, `map_reduce`, `get_neighbors`, `traverse`, `find_related`,
   `community_summary`.
   `engine.searchKnowledgeBase({ action, principals, scope })` is the
-  same function `createMcpApp` serves. `knowledgeToolDefinition()` gives
+  same function `createMcpApp` serves. `discover` returns the SCHEMA, not
+  just an inventory, and for EVERY document type. Each listed document carries
+  `structure` — `sheets` (name, column headers, row count; the sheet name is
+  the key `compute` exposes in `dfs`) for a workbook, `sections` and
+  `last_page` for a document with headings, `keys` for JSON, and `chunks` for
+  anything else — plus `document_type` and `mode`. Every list inside a
+  `structure` is capped, with the remainder reported as `more_columns` /
+  `more_sections` / `more_keys` / `more_sheets` (absent when nothing was cut)
+  and, while the remainder is worth fetching, a filled-in `next_action`. Past
+  `MAX_INVITED_ITEMS` the invitation is WITHHELD and replaced by `instead` —
+  a model cannot know what a full list costs until it has paid for it, so
+  offering a sixteen-thousand-column fetch would be the tool's decision, not
+  the caller's; a list that long is a signal to compute over the sheet or read
+  the document, never to list it. **The cap is for the call that did NOT name
+  its documents:** `discover` (and `list`) now accept `document_ids`, filtered
+  in SQL, and a `discover` scoped to specific documents answers their
+  structure WHOLE and narrows `fields_by_type` and `document_types` to them
+  too. The host's `scope` ceiling still wins — a named id outside it is
+  dropped, never reached.
+  Alongside them, `fields_by_type` and `document_types` are BOTH keyed by
+  `(kind, type)` — a `document_type` is a label an LLM wrote, so a PDF invoice
+  and a CSV of invoice rows can share one, and keying on the label alone would
+  merge their fields. `fields_by_type` rows carry `fields` (name, data type,
+  how many in-scope documents) plus `more_fields` (names past the cap);
+  `document_types` rows carry `documents` and `with_fields`. `type` is written
+  only where `extract_structured` was opted into, so it is null on a default
+  deployment — which is why `kind` is there. All of it is ACL- and
+  scope-filtered and masked by an output redaction policy. `list` returns the
+  same page WITHOUT the structure — browsing does not pay for it. `knowledgeToolDefinition()` gives
   the name, description and JSON Schema if you are wiring it into your own
   agent loop. `compute` and `mapReduce` accept a host callable that REPLACES
   the built-in one — that is the seam for your own permission, billing and
@@ -102,6 +130,31 @@ await engine.search(q, { principals: user.groups }); // scoped
   header before the LLM sees a schema, and sweeps the result.
 - Filing a document under an ACL the caller does not hold returns **403**, on
   ingest and on PATCH alike.
+- **`executeTool` cuts its `result` to 8,000 characters and 100 rows by
+  default.** A cut result is `{ _truncated, _original_size }` — a model
+  reading only that prefix will say "not found" about data that was there.
+  Size the budget to the model's window with `resultMaxChars` /
+  `resultMaxRows` (omitted = default, `null` = no limit; rows are per table,
+  and a db tool's own `max_rows`, default 1000, still applies first).
+  `responseMode: "tsv"` works for ANY tool: every array of objects in the
+  result (db `rows`, an HTTP body's list, a function's returned list) becomes
+  a TSV string under the same key — header first, `\N` = NULL, nested values
+  as JSON cells, tab/newline/backslash escaped — and the budget cuts at whole
+  rows, reporting `_result_shaping: { "<dotted.path>": { rows_returned,
+  rows_omitted } }`. Everything else stays JSON. The audit row keeps the
+  original under its own 50KB cap regardless. An http tool can store
+  `response_mode: "tsv"` in its config (every method, `QUERY` included;
+  omitted = `json`); an explicit `responseMode` on the call wins. The
+  conversion alone is `formatResult(result, "tsv")`.
+
+  ```ts
+  const out = await engine.executeTool("db_sales", { query: "SELECT * FROM orders" }, {
+    principals,
+    resultMaxChars: 200_000,
+    resultMaxRows: null,
+    responseMode: "tsv",
+  });
+  ```
 
 ## Search