@promptev/context-engine 0.0.3 → 0.0.5

package/README.md

@@ -186,7 +186,7 @@ One tool with actions:
 
 | Action | What it is for |
 |---|---|
-| `discover` | what documents there are and what each action can do — call it first |
+| `discover` | what documents there are and what is inside each one — sheets and columns, sections and pages, or JSON keys — plus the extracted fields by document type, a census of the corpus, and what each action can do. Call it first |
 | `search` | passages by meaning or keywords |
 | `get_doc` / `get_docs` | one whole document by id, or several at once |
 | `get_chunks` | walk one long document in order, a piece at a time |
@@ -204,6 +204,30 @@ as off rather than hidden, and called anyway it answers `{ success: false,
 error }` the model can relay. A truncated answer carries the `next_page` call
 that reads the rest.
 
+`discover` answers with the SCHEMA, not an inventory, so one call is enough to
+write one correct `compute` or `query_meta` call instead of three exploratory
+ones. Every listed document carries `document_type`, `mode`, and a `structure`
+whose shape follows the type — `sheets` (name, column headers, row count) for
+a workbook, `sections` and `last_page` for a document with headings, `keys`
+for JSON, and `chunks` as the floor for anything else. Beside them,
+`fields_by_type` lists the extracted structured field names with each field's
+data type and how many in-scope documents carry it, and `document_types` is a
+census of the corpus. Both are keyed by `(kind, type)`: `kind` comes from the
+mime and is always known, while `document_type` is free text an LLM wrote and
+exists only where `extractStructured` was enabled — so a PDF invoice and a
+spreadsheet of invoice rows stay separate rather than merging under one label.
+
+Lists inside a `structure` are capped, with the remainder reported as
+`more_columns` / `more_sections` / `more_keys` / `more_sheets`, because
+`discover` is the first call of a conversation and an unbounded list lands in
+the model's context before it has asked anything. **The cap is for the call
+that did not name its documents:** `discover` and `list` take `document_ids`,
+and a `discover` scoped to specific documents answers their structure whole
+and narrows the field grouping and the census to them too. While the remainder
+is small enough to be worth fetching, the truncated structure carries that
+call already filled in; past that it says what to do instead, because a sheet
+with hundreds of columns is one to compute over rather than read back.
+
 Omit `mode` on a search and it is worked out from the documents in scope — the
 graph leg is added when the deployment has a graph and something in scope was
 ingested that way, and it runs beside the other legs rather than ahead of
@@ -272,6 +296,46 @@ record. Resolve it server-side — on the adapters and the MCP gateway it is an
 still works this release, unscoped and with a `DeprecationWarning`; the next
 release refuses a gated call with no scope.
 
+**Result budget** — the returned `result` is cut to 8,000 characters and 100
+rows unless you say otherwise. Size it to your model's window, or pass `null`
+to lift a limit. `responseMode: "tsv"` works for **any** tool: every array of
+objects in the result — a db tool's `rows`, an HTTP API's `data.items`, a
+function's returned list — comes back as a TSV string under the same key, and
+the budget cuts at whole rows:
+
+```ts
+const out = await engine.executeTool("db_sales", { query: "SELECT * FROM orders" }, {
+  principals: ["user:a"],
+  resultMaxChars: 200_000, // null = no character limit
+  resultMaxRows: null, // per table; a db tool's own max_rows (default 1000) still applies
+  responseMode: "tsv", // -> { success: true, columns: [...], rows: "id\tname\n1\t..." }
+});
+// a cut reports where: { _result_shaping: { rows: { rows_returned: 812, rows_omitted: 188 } } }
+```
+
+In TSV, `\N` is NULL, nested values are compact JSON cells, and tab, newline,
+CR and backslash inside a value are backslash-escaped. Parts of a result that
+aren't arrays of objects stay JSON; a result that *is* one becomes the string
+(wrapped as `{ result, _result_shaping }` when cut). The audit row always keeps
+the original under its own 50KB cap, whatever you pass.
+
+An **http tool** can set its format once, in its config — for every method
+(`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `QUERY`). Omitted, it is `json`; a
+caller's explicit `responseMode` still wins:
+
+```ts
+await engine.registerTool(new ToolConfig({
+  name: "orders",
+  kind: "http",
+  config: { method: "QUERY", url: "https://api.example.com/orders", response_mode: "tsv" }, // "json" (default) | "tsv"
+  acl: ["group:ops"],
+}));
+```
+
+The same conversion is a plain function for anything else:
+`formatResult(result, "tsv")` (and `rowsToTsv(rows)`) from
+`@promptev/context-engine` — `"json"`, the default, returns the input unchanged.
+
 A caller may only register or relabel a tool under principals it holds — filing
 one under another group's ACL is a 403.