@yugabytedb/meko-mcp 2.3.0 → 2.5.0

package/assets/skills/meko-mcp-tools/references/tools-cookbook.md

@@ -68,7 +68,7 @@ memory_add(agent_id="<your-agent-id>",
 
 **What NOT to store via memory_add:**
 - Full conversations (use `conversation_create` + `conversation_add_message`)
-- Large documents (use the UI's Datapack → Actions → Add Knowledge upload flow)
+- Large documents (use `knowledgebase_upload_document`, or the UI's Datapack → Actions → Add Knowledge upload flow)
 - Multi-topic dumps (transcripts, mixed logs + decisions) — store **one coherent fact per memory**, with the identifiers a later search will use
 
 **Common errors:**
@@ -180,7 +180,7 @@ Get exact UUIDs from the `id` field of `memory_search` / `memory_get_all` result
 ```
 conversation_create(agent_id="agent",
     title="Debugging the auth middleware",
-    datapack_id="<uuid from datapack_list>")
+    datapack_id="<uuid from datapack_search>")
 ```
 
 `datapack_id` is required whenever no server-side default resolves for the caller. Omitting it in that state returns `{"error": "datapack_id_required"}` and persists nothing; the same contract applies to `conversation_add_message`, `conversation_get`, `conversation_update`, and `conversation_delete`.
@@ -225,8 +225,8 @@ conversation_get(conversation_id="conv-uuid-123", agent_id="agent",
 ### conversation_list
 
 Target the datapack by `datapack_id` (UUID). It's optional — omit it to use the
-pinned/default datapack, or pass the pinned `datapack_id`. Use `datapack_list`
-to look up the id.
+pinned/default datapack, or pass the pinned `datapack_id`. Use
+`datapack_search(query="<part of the name>")` to look up the id.
 
 ```
 conversation_list(datapack_id="<datapack-uuid>", agent_id="agent", limit=20)
@@ -255,7 +255,7 @@ conversation_delete(conversation_id="conv-uuid-123", agent_id="agent")
 
 ## Knowledge Base Tools
 
-**KB ingestion is a UI-only activity.** Users upload files via the datapack's Actions → **Add Knowledge** dialog (PDF/TXT/MD/JSON/MP4, 5MB each, 10/batch). MCP exposes two KB tools: `knowledgebase_search`, and `knowledgebase_delete_document` for removing a single uploaded file.
+Agents upload files via `knowledgebase_upload_document` and poll `knowledgebase_document_status` for indexing progress; a human uploading through the Cloud UI still uses the datapack's Actions → **Add Knowledge** dialog (PDF/TXT/MD/JSON/CSV/XML/HTML, 5MB Free / 50MB Standard / 60MB Pro+ each, 10/batch) — both paths feed the same index. `knowledgebase_search` queries whatever the KB currently holds. MCP also exposes `knowledgebase_delete_document` for removing a single uploaded file.
 
 See `tools-rag-workflow.md` for the decision table.
 
@@ -294,6 +294,173 @@ knowledgebase_delete_document(document_id="<uuid from a search hit>",
 
 Returns `{"document_id", "chunks_deleted", "s3_deleted"}` — if `s3_deleted` is `false`, say so: the index entries are gone but the stored file may remain. Errors include a numeric `status`: `403` means the user is not the datapack owner, a maintainer, or the file's uploader — report it, don't retry; `409` (`DOCUMENT_PROCESSING`) means the file is still being indexed — wait for indexing to finish, then retry.
 
+### knowledgebase_upload_document
+
+**When to use:** Get a file's bytes into the datapack's KB for later `knowledgebase_search` retrieval — a file already on disk, or one generated mid-conversation. This tool never touches the file's bytes itself; it only requests a presigned S3 URL. Meko registers the file once it lands, so there's no separate confirm call. **The caller POSTs the file to the returned `url` directly** — multipart/form-data, every key in `fields` as a form field (in order), then the file's bytes last as a field named `"file"`.
+
+**Confirm first.** The file goes into the datapack's shared knowledge base, which every member can read. Before calling the tool, tell the user which file and datapack, say the team will see it, and wait for a yes.
+
+**You need a shell that can make the POST** (Python, curl, or PowerShell, below). Without one, as in Claude Desktop by default, don't call the tool: point the user at Datapack → Actions → **Add Knowledge**. An issued URL holds its `size` against the storage quota for up to 15 minutes even if nothing is uploaded.
+
+**After the user confirms and the tool returns, POST the file in the same turn.** The upload is only half-done once you have `url`/`fields`; nothing else completes it for you.
+
+**First, compute `content_hash` and `content_head_base64` — both required — *before* calling the tool at all. `content_head_base64` is compulsory: the tool refuses with `content_head_base64_required` if it's missing or blank, so there's no path that skips the sniff.**
+
+Linux:
+```bash
+CONTENT_HASH=$(sha256sum report.pdf | cut -d' ' -f1)
+CONTENT_HEAD_B64=$(head -c 3072 report.pdf | base64 -w0)
+```
+
+macOS (no `sha256sum` by default — use `shasum`; BSD `base64` has no `-w0`, so strip newlines with `tr` instead):
+```bash
+CONTENT_HASH=$(shasum -a 256 report.pdf | cut -d' ' -f1)
+CONTENT_HEAD_B64=$(head -c 3072 report.pdf | base64 | tr -d '\n')
+```
+
+Windows PowerShell:
+```powershell
+$ContentHash = (Get-FileHash -Algorithm SHA256 -Path "report.pdf").Hash.ToLower()
+$bytes = [System.IO.File]::ReadAllBytes("report.pdf")
+$head = $bytes[0..([Math]::Min(3071, $bytes.Length - 1))]
+$ContentHeadB64 = [Convert]::ToBase64String($head)
+```
+
+Windows cmd.exe (`certutil` ships built in for the hash — prints it on its own line, strip the header/footer lines before use; cmd.exe has **no** built-in way to take just the first 3072 bytes *and* base64-encode them in one step, so use the PowerShell or Python snippet for `content_head_base64` even in a cmd.exe-driven workflow):
+```bat
+certutil -hashfile report.pdf SHA256
+```
+
+Python (cross-platform; streams the file so a large one isn't fully loaded into memory, and reuses the same read to build `content_head_base64` too):
+```python
+import hashlib, base64
+
+h = hashlib.sha256()
+with open(LOCAL_PATH, "rb") as f:
+    head = f.read(3072)                      # also the content_head_base64 sample
+    h.update(head)
+    for chunk in iter(lambda: f.read(1 << 20), b""):
+        h.update(chunk)
+
+content_hash = h.hexdigest()
+content_head_base64 = base64.b64encode(head).decode()
+```
+
+**Then, after the tool call returns `{url, fields}` — a portable snippet (identical on Linux/macOS/Windows, needs `python3` + `requests`), because it loops over whatever `fields` came back instead of hand-enumerating field names, so it can't drift out of sync with what the response actually contains:**
+
+```python
+import json, requests
+resp = json.loads(RESPONSE_JSON)        # the tool's raw JSON string
+if not resp.get("already_uploaded"):
+    with open(LOCAL_PATH, "rb") as f:
+        r = requests.post(resp["url"], data=resp["fields"], files={"file": (FILENAME, f)})
+        r.raise_for_status()
+```
+
+**Linux/macOS, curl (needs `jq`; builds the `-F` flags dynamically from whatever `fields` actually contains — a signer using temporary credentials adds `x-amz-security-token`, and extra policy conditions add their own fields, so a hardcoded field list drops them and S3 answers 403 on a form missing a signed field):**
+
+```bash
+if [ "$(echo "$RESPONSE_JSON" | jq -r '.already_uploaded')" != "true" ]; then
+    URL=$(echo "$RESPONSE_JSON" | jq -r '.url')
+    FIELDS=()
+    while IFS= read -r -d '' key && IFS= read -r -d '' value; do
+        FIELDS+=(-F "${key}=${value}")
+    done < <(echo "$RESPONSE_JSON" | jq -j '.fields | to_entries[] | (.key, "\u0000", .value, "\u0000")')
+    curl -sS --fail "${FIELDS[@]}" -F file=@/local/path/report.pdf "$URL"
+fi
+```
+
+**Windows PowerShell 7+ (`Invoke-RestMethod -Form` takes the whole `fields` object as a hashtable — no per-field enumeration needed at all; use `[ordered]@{}`, not a plain `@{}`, since S3 requires `file` to be the *last* form field and a plain hashtable has no guaranteed enumeration order):**
+
+```powershell
+$resp = $json | ConvertFrom-Json
+if (-not $resp.already_uploaded) {
+    $form = [ordered]@{}
+    $resp.fields.PSObject.Properties | ForEach-Object { $form[$_.Name] = $_.Value }
+    $form["file"] = Get-Item "C:\path\to\report.pdf"
+    Invoke-RestMethod -Uri $resp.url -Method Post -Form $form
+}
+```
+
+**Windows cmd.exe:** cmd.exe has no built-in JSON parsing, so there's no safe way to build the `-F` flags dynamically from whatever `fields` contains without an extra tool. Rather than hardcode a field list that silently drops fields (e.g. `x-amz-security-token` on temporary credentials, or extra policy conditions) and fail with a confusing S3 403, use the PowerShell or Python snippet above even from a cmd.exe-driven workflow.
+
+`size` must be the file's exact length in bytes, and the URL only accepts a body of exactly that size and SHA-256, for 5 minutes. Accepted extensions: pdf, txt, text, md, json, csv, xml, html, htm. `content_head_base64` — the file's first 3072 bytes, or the whole file if shorter, base64-encoded — is **required**: the tool sniffs it against the extension before requesting the URL and rejects an obvious mismatch (e.g. a renamed file) at once as `VALIDATION_ERROR`; a missing or blank one is `content_head_base64_required`. Text files must be UTF-8, because indexing reads UTF-8 only: a Latin-1 or Excel "CSV (Windows)" file is refused with a message to save it as UTF-8 (Excel: "CSV UTF-8"). Samples up to 4096 bytes are accepted and trimmed, so older snippets that send 4096 still work. A mismatch the server catches later surfaces as `state: "abandoned"` from `knowledgebase_document_status`. Idempotent — uploading the same content twice (by SHA-256) returns `already_uploaded: true` with an empty `url` and no `fields`; if that file's indexing never started or failed, the repeat queues it again.
+
+```
+knowledgebase_upload_document(
+    filename="report.pdf",
+    size=42000,
+    content_hash="<sha256 hex of the file's bytes>",
+    conversation_id="<uuid from conversation_create>",
+    content_head_base64="<base64 of the file's first 3072 bytes>",
+    agent_id="agent",
+)
+```
+
+**Response (fresh upload):**
+```json
+{
+  "datapack_id": "...",
+  "url": "https://<bucket>.s3.amazonaws.com/",
+  "fields": {"key": "...", "policy": "...", "...": "..."},
+  "key": "user/datapack/hash/report.pdf",
+  "already_uploaded": false
+}
+```
+
+**Response (identical content already uploaded):**
+```json
+{
+  "datapack_id": "...",
+  "url": "",
+  "key": "user/datapack/hash/report.pdf",
+  "already_uploaded": true,
+  "existing_filename": "report.pdf"
+}
+```
+
+After a fresh upload, POST the file to `url` yourself (see above), then save `key` (as `document_key`) and poll `knowledgebase_document_status` until `state` is terminal. S3 answers a successful POST with `204`. `400 EntityTooSmall` or `EntityTooLarge` means `size` wasn't the file's exact byte count: call the tool again with the right size and upload through the new URL. The earlier URL can never succeed (it only accepts the wrong size) and expires on its own within 5 minutes.
+
+**Common errors:**
+- `conversation_id_required` — `conversation_id` was missing or blank.
+- `datapack_id_required` — no `datapack_id` was given and no tenant default could be resolved.
+- `content_head_base64_required` — `content_head_base64` was missing or blank. Checked after the `filename`/`size`/`content_hash` `VALIDATION_ERROR` checks, but always before a URL is requested — there is no way to skip it.
+- `VALIDATION_ERROR` — missing filename, unsupported extension, missing/malformed `content_hash`, size ≤ 0, invalid or oversized `content_head_base64`, a sample shorter than `min(size, 3072)` bytes or longer than `size` (so `size` is wrong), or the sampled content doesn't match its extension.
+- `FREE_TIER_KB_STORAGE_LIMIT` / `STANDARD_TIER_KB_STORAGE_LIMIT` (`status` 402) — no room left in the storage quota. Tell the user to free space or upgrade instead of retrying: every issued URL holds its `size` against the quota until the file is registered, or for up to 15 minutes if it is never uploaded.
+
+Every error from the server also carries `status`, its HTTP status code.
+- `knowledgebase_upload_document_failed` — unexpected error (see `detail`).
+- The actual S3 POST isn't made through this tool, so its own failures (e.g. the body exceeding the cap the URL was signed under) surface to whatever made that call directly, not from this tool.
+
+---
+
+### knowledgebase_document_status
+
+**When to use:** Track a file uploaded via `knowledgebase_upload_document` from "waiting for bytes" through to indexed.
+
+```
+knowledgebase_document_status(
+    document_key="user/datapack/hash/report.pdf",
+    conversation_id="<uuid from conversation_create>",
+    agent_id="agent"
+)
+```
+
+**Response (still indexing):**
+```json
+{
+  "document_key": "user/datapack/hash/report.pdf",
+  "filename": "report.pdf",
+  "state": "indexing",
+  "message": "The file is registered and being indexed.",
+  "document": {"status": "PROCESSING", "chunks_processed": 3}
+}
+```
+
+Poll until `state` is `completed` or `failed` (or the pre-registration terminals `abandoned`/`oversized`). `NOT_FOUND` means `document_key` is unknown to this datapack — check it was copied verbatim from the upload response. `datapack_id_required` means no `datapack_id` was given and no tenant default could be resolved.
+
+---
+
 ## Artifact Tools
 
 ### artifact_put
@@ -378,14 +545,31 @@ For large files use the `local_path` value to read the file with normal file too
 
 ## Datapack Management Tools
 
-### datapack_create / datapack_list / datapack_describe
+### datapack_create / datapack_search / datapack_list / datapack_describe
 
 ```
 datapack_create(name="analytics_prod", conversation_id="<conversation id>")
-datapack_list(conversation_id="<session conversation id>")  # returns datapack_id for each datapack
+datapack_search(conversation_id="<session conversation id>", query="analytics")    # find an id by part of the name
+datapack_search(conversation_id="<session conversation id>", name="analytics_prod") # exact name
+datapack_list(conversation_id="<session conversation id>")  # small accounts: every datapack in one array
 datapack_describe(datapack_id="<uuid>", include_status=True, conversation_id="<conversation id>")
 ```
 
+Find a datapack id with `datapack_search(query=...)`. It returns one page:
+`{"datapacks": [...], "has_more": true, "next_cursor": "<token>", "total": null}`.
+To get the next page, call it again with the same filters and
+`cursor="<next_cursor>"`. `next_cursor` is `""` on the last page, and also
+when an older server has more matches than this page but cannot continue.
+Do not send an empty cursor; ask for a narrower `query` instead. Treat the
+cursor as opaque. Pass `include_total=True` only when you must show "N of M".
+
+`datapack_list` still returns one array and is fine for an account with a
+handful of datapacks. For an account that can see more than 1000 datapacks
+(the server default limit) it returns `{"error": "too_many_datapacks", ...}`
+instead; switch to `datapack_search`. If the server has no `datapack_search`
+tool (an older server), use `datapack_list`.
+
+
 `datapack_describe` returns these counts:
 
 | Field | Meaning |
@@ -397,7 +581,7 @@ datapack_describe(datapack_id="<uuid>", include_status=True, conversation_id="<c
 | `knowledge_chunk_count` | KB chunks across all documents (for debugging retrieval) |
 | `shared_knowledge_count` | `collective_memory_count + knowledge_base_file_count`; `null` when the file count is `null` |
 
-`knowledge_count` is a deprecated alias of `knowledge_chunk_count`. `datapack_list` doesn't compute counts; call `datapack_describe` for them.
+`knowledge_count` is a deprecated alias of `knowledge_chunk_count`. Neither `datapack_search` nor `datapack_list` computes counts; call `datapack_describe` for them.
 
 ### datapack_update / datapack_delete
 
@@ -420,4 +604,4 @@ Agent and KB lifecycle does not ship as MCP tools. If the user asks to create/li
 - REST: `POST/GET/DELETE /datapacks/:datapack_id/agents` (agents), `POST/GET/DELETE /datapacks/:datapack_id/knowledge-bases` (KB sources).
 - Or the Meko Cloud UI at `cloud.mekodata.ai` → Datapacks → agents / knowledge-bases.
 
-KB ingestion specifically lives in the UI today: Datapack → Actions → **Add Knowledge** (PDF/TXT/MD/JSON/MP4, 5 MB each, 10/batch). Agents querying a populated index use `knowledgebase_search`, and can remove a single uploaded file with `knowledgebase_delete_document` — see `tools-rag-workflow.md`.
+Uploading a *file* into an existing KB is not part of that lifecycle restriction — use `knowledgebase_upload_document` (see above) rather than the UI for anything driven from MCP. Agents querying a populated index use `knowledgebase_search`, and can remove a single uploaded file with `knowledgebase_delete_document` — see `tools-rag-workflow.md`.

package/assets/skills/meko-mcp-tools/references/tools-datapack-workflow.md

@@ -14,7 +14,8 @@ Datapacks are isolated workspaces — each has its own memory store and knowledg
 ## How to obtain a datapack_id
 
 - From `datapack_create` response (`"datapack_id"` field; `"datapack_name"` is also returned)
-- From `datapack_list` (each entry has `"datapack_id"` and `"datapack_name"`)
+- From `datapack_search(query="<part of the name>")` or `datapack_search(name="<exact name>")`: each row in `"datapacks"` has `"datapack_id"` and `"datapack_name"`. Page with `cursor="<next_cursor>"` only when that value is non-empty. An empty `next_cursor` means stop and narrow with `query`.
+- From `datapack_list` on a small account, or on an older server without `datapack_search` (each entry has `"datapack_id"` and `"datapack_name"`). An account that can see more than 1000 datapacks (the server default limit) gets `too_many_datapacks` from an unfiltered `datapack_list`; use `datapack_search`.
 - From `datapack_describe` response (same field names)
 
 ## Datapack lifecycle
@@ -27,13 +28,15 @@ datapack_create(name="sales_analytics",
 -- Returns: {"datapack_id": "dp-uuid-123", "datapack_name": "sales_analytics", ...}
 ```
 
-Provisioning agents and knowledge-base sources on a new datapack is **not** MCP-exposed. Use the Meko control plane:
+Provisioning agents and knowledge-base *sources* (registering/indexing a new source against a datapack) is **not** MCP-exposed. Use the Meko control plane:
 
 - **UI**: `app.mekodata.ai` → Datapacks → select datapack → Agents / Knowledge Bases
-- **REST**: `POST /datapacks/:datapack_id/agents`, `POST /datapacks/:datapack_id/knowledge-bases` (and the Add Knowledge UI for file uploads)
+- **REST**: `POST /datapacks/:datapack_id/agents`, `POST /datapacks/:datapack_id/knowledge-bases`
 
 If the user asks to set those up mid-session, point them at the control plane rather than inventing an MCP call that will fail.
 
+Uploading a *file* into an existing KB is a different, already-MCP-exposed operation — use `knowledgebase_upload_document` (see `tools-cookbook.md`) rather than routing the user to the Add Knowledge UI.
+
 ### Targeting a specific datapack on subsequent calls
 
 ```

package/assets/skills/meko-mcp-tools/references/tools-known-limitations.md

@@ -12,7 +12,7 @@ These are current limitations of the Meko MCP tools. Know them upfront to avoid
 | Standard | 5,000 | 5,000 | 30,000 |
 | Pro / Enterprise | uncapped | uncapped | uncapped |
 
-The Free tier also caps `context_search` and `conversation_add_message` at 1,000 calls each. Embedding turns into the conversation-search cache has its own cap (2,000 turns on Free, 10,000 on Pro). When that cap is reached, `conversation_add_message` still stores the turn and reports `conversation_search_skipped: "quota_exceeded"`; the turn just won't appear in `context_search`'s conversation bucket.
+The Free tier also caps `context_search` and `conversation_add_message` at 1,000 calls each. Embedding turns into the conversation-search cache has its own cap: 2,000 turns on Free and 10,000 on Pro; Standard and Enterprise are uncapped. When that cap is reached, `conversation_add_message` still stores the turn and reports `conversation_search_skipped: "quota_exceeded"`; the turn just won't appear in `context_search`'s conversation bucket.
 
 A quota error can arrive with HTTP 403. It is not an authentication failure; don't reconfigure credentials because of it.
 
@@ -36,6 +36,10 @@ One call returns roughly the **20 most recent rows**; the response's `total` rep
 
 `memory_search` and `memory_get_all` return every agent's rows for the user. `memory_delete_all` deletes only rows in the `agent_id` you pass, and an omitted `agent_id` means the `meko_agent` bucket. A `memory_search(run_id=X)` preview can therefore list rows that `memory_delete_all(run_id=X)` leaves in place. To clear rows written by several agents, delete them by id with `memory_delete_by_id`, or call `memory_delete_all` once per `agent_id` shown in the preview.
 
+## Viewers can read but not write
+
+On a datapack shared with you as a viewer, every conversation and memory write returns `read_only_access` and writes nothing: `conversation_create`, `conversation_add_message`, `conversation_update`, `memory_add`, `memory_update`, `memory_promote`, `artifact_put`, and `track_token_usage`. You can still delete your own memories with `memory_delete_by_id` and `memory_delete_all`, and delete a conversation you started. Searches work as for any other role. `context_search` covers every member's conversations, but for a viewer its conversation results and `conversation_get` return `user_id: null`, so they don't say who wrote a turn. Automatic capture holds a viewer's session and reports `read_only_access` at the next session start.
+
 ## Promoted memories are read-only through MCP
 
 `memory_update` and `memory_delete_by_id` return `memory_is_promoted` for a memory that has been promoted to shared knowledge. Edit or remove promoted content in the Cloud UI Learnings tab.
@@ -52,9 +56,11 @@ The one exception is a single uploaded **file**: `knowledgebase_delete_document`
 
 **Workaround:** For stuck or failed indexes, create a new index with a different name. Stale indexes remain in the database until manually cleaned up by an admin.
 
-## RAG ingestion is UI-only
+## KB index/source management is UI-only
+
+Manually creating a vector index, adding a source to an existing one, or polling raw build status is not exposed via MCP. Use the Cloud UI for those (Datapack → Actions → Add Knowledge).
 
-KB ingestion (creating an index, adding sources, polling build status) is not exposed via MCP. Documents go in through the Cloud UI's **Add Knowledge** flow (Datapack → Actions → Add Knowledge) — PDF/TXT/MD/JSON/MP4 up to 5 MB each, 10 per batch.
+**Adding a file** is a registered MCP tool: call `knowledgebase_upload_document` directly (accepted extensions PDF/TXT/MD/JSON/CSV/XML/HTML, up to 5 MB Free / 50 MB Standard / 60 MB Pro+ per file) to get a presigned URL, then POST the bytes to it yourself; Meko registers and indexes the file once it lands, so no separate index-management call is needed. Confirm with the user first, because every datapack member can read the knowledge base. The Cloud UI's **Add Knowledge** flow (10 files per batch) remains the human-facing path for the same upload.
 
 To **query** a built index, use `knowledgebase_search` — it returns both uploaded documents and memories the user promoted from the Learnings tab.
 
@@ -70,7 +76,7 @@ It works **poorly** for:
 - Data dictionaries — only partial information extracted
 - Structured schemas — relationships between fields lost
 
-**Structured data is not natively supported via MCP.** The options are:
+**Don't store structured data as memories.** The options are:
 
 1. **Write a single narrative summary** as a memory — one call, not row-by-row:
    ```
@@ -79,8 +85,7 @@ It works **poorly** for:
        conversation_id="<session conversation_id>",
        text="Customer Complaints dataset has fields: Complaint ID (int PK), Date Submitted (date), Product (text, e.g. Mortgage/Credit Card), Sub-product (text), Issue (text), Company Response (text), State (text, 2-letter code), ZIP Code (text)")
    ```
-2. **Upload the schema doc** through the Cloud UI's **Add Knowledge** flow (Datapack → Actions → Add Knowledge). Accepts PDF/TXT/MD/JSON — convert CSV to MD or JSON first. That makes it queryable via `knowledgebase_search`.
-3. **Tell the user** that structured-data storage via MCP isn't supported and let them decide how to proceed.
+2. **Upload the file itself to the knowledge base** with `knowledgebase_upload_document` — `.csv`, `.json` and the other accepted types go in as they are — and query it with `knowledgebase_search`. (The Cloud UI's **Add Knowledge** flow does the same for a person.)
 
 Never ingest CSV row-by-row into memory — each row becomes a fragmented fact with lost context.
 

package/assets/skills/meko-mcp-tools/references/tools-memory-vs-conversation.md

@@ -27,11 +27,11 @@ Conversations are backed by Langfuse sessions and traces. They preserve full str
 
 ```
 conversation_create(agent_id="support_bot", title="Pricing discussion",
-    datapack_id="<uuid from datapack_list>")
+    datapack_id="<uuid from datapack_search>")
 -- Returns: {"id": "conv-uuid-here"}
 
 conversation_add_message(conversation_id="conv-uuid-here", agent_id="support_bot",
-    datapack_id="<uuid from datapack_list>",
+    datapack_id="<uuid from datapack_search>",
     input="What are your pricing tiers?",
     output="We offer Starter ($10/mo), Pro ($50/mo), and Enterprise (custom).",
     reasoning="Retrieved pricing page data. No special discounts apply.")
@@ -85,7 +85,7 @@ Agents on Cloud Meko have two distinct read surfaces. Pick the right one for the
 - **Personal memories** — written by `memory_add`. The `agent_id` records the writer but does not restrict reads; memories are scoped **per-user**, so any of that user's agents can read them via MCP, and no other user can.
 - **Team-shared Shared Knowledge** — arrives two ways:
   1. An agent calls `memory_promote` for exact, user-confirmed memory UUIDs, or the user promotes them from the Cloud UI's Learnings tab. Promotion moves the memories into shared knowledge, strips `user_id`, preserves originating `agent_id` as provenance, and evicts the private mem0 records.
-  2. The user uploads a file via Datapack → Actions → **Add Knowledge** in the Cloud UI. PDF/TXT/MD/JSON/MP4 up to 5MB each.
+  2. The user uploads a file via Datapack → Actions → **Add Knowledge** in the Cloud UI. PDF/TXT/MD/JSON/CSV/XML/HTML up to 5MB (Free) / 50MB (Standard) / 60MB (Pro+) each.
 
   Both show up in `knowledgebase_search`, tagged `metadata_filters.source: "memory"` vs other values so the agent can distinguish provenance in responses.
 
@@ -102,7 +102,7 @@ Call `context_search`. One call returns your personal memories (every agent, no
 There is no MCP-side "just put it in a database table" option — raw SQL access isn't part of the MCP surface. Workarounds:
 
 1. **Write a single narrative summary** as a memory: `"The Customer Complaints dataset has fields: Complaint ID (int), Date Submitted (date), Product (text), Sub-product (text), Issue (text)..."` — one call, not row-by-row.
-2. **Suggest the user upload the file** via the Cloud UI's Add Knowledge flow if it's a document (CSV files aren't in the supported list — PDF/TXT/MD/JSON/MP4 are; for CSV, convert to MD or JSON first).
+2. **Suggest the user upload the file** via the Cloud UI's Add Knowledge flow if it's a document (PDF/TXT/MD/JSON/CSV/XML/HTML are supported).
 3. **Never ingest CSV row-by-row** into memory — each row becomes a fragmented fact with lost context.
 
 ## Memory Storage: automatic, not proactive

package/assets/skills/meko-mcp-tools/references/tools-overview.md

@@ -1,13 +1,13 @@
 # Complete Tool Catalog and Decision Tree
 
-**25 Meko data tools covered by this skill.** These are the tools available in production, grouped as: Retrieval (1), Memory (8), Conversation (6), Knowledge Base (2), Datapack (5), Artifacts (2), Observability (1). Combined Meko + AMP endpoints may expose additional infrastructure tools; those are outside this skill's scope.
+**28 Meko data tools covered by this skill.** These are the tools available in production, grouped as: Retrieval (1), Memory (8), Conversation (6), Knowledge Base (4), Datapack (6), Artifacts (2), Observability (1). Combined Meko + AMP endpoints may expose additional infrastructure tools; those are outside this skill's scope.
 
 ## Quick health check before using tools
 
 Before starting a session, verify that the Meko MCP tools are working:
 
 1. `memory_search(query="test", agent_id="<your-session-agent-id>", conversation_id="<session-conversation-id>")` — Confirms memory subsystem is up. Use the `agent_id` and `conversation_id` from the SessionStart `additionalContext`. May fail with "connection already closed" — see `tools-troubleshooting.md`.
-2. `datapack_list(conversation_id="<session-conversation-id>")` — Confirms Meko API connectivity. Returns the datapacks your token has access to.
+2. `datapack_search(conversation_id="<session-conversation-id>", limit=1)`: confirms Meko API connectivity and returns one datapack your token has access to. On a server without `datapack_search`, call `datapack_list(conversation_id="<session-conversation-id>")` instead.
 
 ## Decision tree: which tool do I need?
 
@@ -16,7 +16,8 @@ User wants to...
 ├── Ask "what do you know about X?" --------> context_search (read; memory + KB + past conversations in one call)
 ├── Search personal context? ---------------> memory_search (read)
 ├── Search shared datapack knowledge? ------> knowledgebase_search (read)
-├── Add documents to a knowledge base? -----> point user at Meko UI: Datapack → Actions → Add Knowledge
+├── Add a file to a knowledge base? --------> knowledgebase_upload_document (write)
+│   └── Check its indexing progress? -------> knowledgebase_document_status (read)
 ├── Delete ONE knowledge-base file? --------> knowledgebase_delete_document (destructive — confirm with the user first)
 ├── Store or recall information?
 │   ├── Store a fact/preference/entity? -----------> memory_add (write)
@@ -28,10 +29,11 @@ User wants to...
 │   ├── Upload a generated file (report, CSV, PDF)? -> artifact_put (write)
 │   └── Retrieve a previously uploaded file? -------> artifact_get (read)
 └── Manage datapacks?
+    ├── Find a datapack id by name? -------> datapack_search(query=...) (read)
     └── CRUD datapack? --------------------> datapack_create/list/describe/update/delete
 ```
 
-Agent lifecycle and KB **ingestion / index lifecycle** (adding sources, uploads, re-indexing) are managed outside the MCP surface — typically in the Meko control-plane UI. Deleting a single KB file IS exposed here (`knowledgebase_delete_document`).
+Agent lifecycle and KB **index / source lifecycle** (creating indexes, adding sources, re-indexing) are managed outside the MCP surface — typically in the Meko control-plane UI. Single KB files are handled here: adding one (`knowledgebase_upload_document`, then `knowledgebase_document_status`) and deleting one (`knowledgebase_delete_document`).
 
 ## Retrieval Tools (1)
 
@@ -39,12 +41,14 @@ Agent lifecycle and KB **ingestion / index lifecycle** (adding sources, uploads,
 |------|---------|
 | `context_search(query, conversation_id, agent_id=None, datapack_id=None, limit=10, force_source="auto")` | Default first call for open-ended recall. Queries memory, the knowledge base, and the conversation-history cache in parallel and returns each source in its own bucket: `{"results": {"memory": [...], "knowledge_base": [...], "conversation": {"hits": [...], "thread": [...]}}, "token_savings": {...}}`. A bucket is `null` only when that source wasn't queried: `force_source` excluded it, or the server couldn't build a query vector for it. An empty bucket is ambiguous: either that source found nothing relevant, or it failed. `context_search` turns a failed source into an empty list and records the error only in the trace. Before you tell the user nothing is stored about X, confirm with `memory_search` or `knowledgebase_search`, which return errors. `force_source` is `"auto"` (all three), `"memory"`, `"knowledge_base"`, or `"conversation"`; any other value returns `invalid_force_source`. `limit` bounds the memory and KB buckets only; the conversation bucket returns up to 10 hits plus a thread of whole turns. The conversation bucket is datapack-wide. It returns embedded turns from every user and agent on the datapack, and each hit carries its author's `user_id`. Don't present a hit as this user's own history. |
 
-## Knowledge Base Tools (2)
+## Knowledge Base Tools (4)
 
 | Tool | Purpose |
 |------|---------|
 | `knowledgebase_search(query, conversation_id, datapack_id, agent_id=None, limit=10)` | Hybrid (semantic + keyword) search across KB chunks and promoted memories. `datapack_id` is REQUIRED (no default). Each hit has `id`, `chunk_text`, `document_id`, `metadata_filters`, `distance`, and `match_type` (`semantic`, `keyword`, or `hybrid`). |
 | `knowledgebase_delete_document(document_id, conversation_id, datapack_id="")` | Permanently delete ONE KB file (chunks + metadata; stored-file delete is best-effort — check `s3_deleted`). Irreversible: confirm with the user first. `document_id` comes from `knowledgebase_search` hits (no filename in hits — echo the id + chunk text). Owner/maintainer/uploader only (status 403 otherwise); status 409 while still indexing — wait and retry. |
+| `knowledgebase_upload_document(filename, size, content_hash, conversation_id, content_head_base64, datapack_id=None, agent_id=None)` | Requests a presigned S3 URL for a file (does not upload bytes itself — the caller POSTs to the returned `url`/`fields`). Returns `{datapack_id, url, fields, key, already_uploaded}` (fresh upload) or `{datapack_id, url: "", key, already_uploaded: true, existing_filename}` (dedup). Accepted extensions: pdf/txt/text/md/json/csv/xml/html/htm; `content_head_base64` (base64 of the file's first 3072 bytes, or the whole file if shorter) is **required** and is always sniffed against the extension before requesting the URL — a missing/blank value is rejected as `content_head_base64_required`, this check cannot be skipped. Idempotent by content hash. |
+| `knowledgebase_document_status(document_key, conversation_id, datapack_id=None, agent_id=None)` | Poll indexing progress for a file uploaded via `knowledgebase_upload_document`, using its returned `key` (passed here as `document_key`). `state` is one of `awaiting_upload`/`indexing` (non-terminal) or `completed`/`failed`/`abandoned`/`oversized` (terminal). |
 
 ## Memory Tools (8)
 
@@ -66,21 +70,22 @@ Agent lifecycle and KB **ingestion / index lifecycle** (adding sources, uploads,
 | `conversation_create(agent_id=None, run_id=None, title=None, metadata=None, session_id="", datapack_id=None)` | Create conversation container (Langfuse session) |
 | `conversation_add_message(conversation_id, input, output, agent_id=None, reasoning=None, metadata=None, seed=None, datapack_id=None, plan=None)` | Add a message turn (Langfuse trace). `output` is required. The response may include `extracted_memories` and, when the turn wasn't embedded for conversation search, `conversation_search_skipped` (`quota_exceeded` or `datapack_opted_out`). |
 | `conversation_get(conversation_id, agent_id=None, include_messages=False, limit=100, offset=0, datapack_id=None)` | Retrieve conversation, optionally with messages |
-| `conversation_list(agent_id=None, limit=20, offset=0, conversation_id="", datapack_id=None)` | List the user's conversations within a datapack (`agent_id` does not filter — trace attribution only). `datapack_id` (UUID) is optional — defaults to the pinned/tenant datapack; use `datapack_list` to look it up. `conversation_id` optional (omit for browse). |
+| `conversation_list(agent_id=None, limit=20, offset=0, conversation_id="", datapack_id=None)` | List the user's conversations within a datapack (`agent_id` does not filter — trace attribution only). `datapack_id` (UUID) is optional — defaults to the pinned/tenant datapack; use `datapack_search(query=...)` to look it up. `conversation_id` optional (omit for browse). |
 | `conversation_update(conversation_id, agent_id=None, title=None, metadata=None, datapack_id=None)` | Update title or metadata |
 | `conversation_delete(conversation_id, agent_id=None, datapack_id=None)` | Delete entire conversation (destructive) |
 
-## Datapack Management Tools (5)
+## Datapack Management Tools (6)
 
 | Tool | Purpose |
 |------|---------|
 | `datapack_create(name, conversation_id)` | Create new datapack |
-| `datapack_list(conversation_id, datapack_id=None)` | List all datapacks. `conversation_id` is required for trace nesting; omit `datapack_id` to use the endpoint default. |
+| `datapack_search(conversation_id, query="", name="", limit=20, cursor="", grant="", sort="created", include_total=False, datapack_id=None)` | Find datapacks and page through them. Returns `{datapacks, has_more, next_cursor, total}`. Use `query` for part of a name, `name` for an exact name, and `cursor` (the previous page's `next_cursor`) for the next page. `limit` is 1-100. `grant` is `owner` or `shared`; `sort` is `created` (newest first) or `name`. Pass `include_total=True` only when you must show "N of M". Rows carry no connection fields and no counts. Servers older than this tool don't have it; use `datapack_list` there. |
+| `datapack_list(conversation_id, datapack_id=None, query="", limit=0)` | Return datapacks as one array. Fine for a small account. `query` filters by part of a name; more than 100 matches return a `too_many_datapacks` error. `limit` returns the first matches (at most 100) without saying whether more exist. With neither, an account that can see more than 1000 datapacks (the server default limit) gets a `too_many_datapacks` error; use `datapack_search` then. `conversation_id` is required for trace nesting. |
 | `datapack_describe(conversation_id, datapack_id="", include_status=False)` | Describe a datapack by UUID (default datapack if omitted). Returns `memory_count`, `collective_memory_count` (promoted), `learnings_count` (awaiting a promotion decision), `knowledge_base_file_count`, `knowledge_chunk_count`, and `shared_knowledge_count`. `knowledge_base_file_count` and `shared_knowledge_count` are `null` when the status endpoint is unreachable; `knowledge_count` is a deprecated alias of `knowledge_chunk_count`. |
 | `datapack_update(datapack_id, conversation_id, name=None, description=None, conversation_search_opt_out=None)` | Rename a datapack, update its description, and/or opt out of (or back into) conversation-search caching. Pass at least one field. Opt-out is owner/maintainer only. |
 | `datapack_delete(datapack_id, conversation_id)` | Delete datapack by UUID (irreversible) |
 
-All five datapack tools reject an empty `conversation_id` with `conversation_id_required`.
+All six datapack tools reject an empty `conversation_id` with `conversation_id_required`.
 
 ## Artifact Tools (2)
 
@@ -108,9 +113,9 @@ Some things the Meko platform can do are not wired into the MCP tool surface tod
 | Langfuse project-key generation | `yugabyte/meko` API server | REST: `POST /datapacks/:datapack_id/langfuse/project-keys` |
 | Account / billing / tier management | Meko UI only | Not available via MCP or public REST |
 
-The MCP-exposed `datapack_create` / `datapack_list` / `datapack_describe` / `datapack_update` / `datapack_delete` tools above are a deliberate subset — the common CRUD that an agent reasonably needs mid-conversation. Anything involving agent or KB lifecycle is control-plane territory.
+The MCP-exposed `datapack_create` / `datapack_search` / `datapack_list` / `datapack_describe` / `datapack_update` / `datapack_delete` tools above are a deliberate subset — the common CRUD that an agent reasonably needs mid-conversation. Anything involving agent or KB lifecycle is control-plane territory.
 
-For knowledge-base content on Cloud, the canonical path is **UI upload**: Datapack → Actions → **Add Knowledge** (PDF/TXT/MD/JSON/MP4, 5MB each, 10/batch). `knowledgebase_search` queries the resulting index from MCP.
+For a file already on disk or generated mid-conversation, requesting a URL with `knowledgebase_upload_document` and POSTing to it gets it into the KB without leaving MCP for a human's help. The UI upload path (Datapack → Actions → **Add Knowledge**, PDF/TXT/MD/JSON/CSV/XML/HTML, 5MB Free / 50MB Standard / 60MB Pro+ each, 10/batch) remains the way for a human to add files directly. `knowledgebase_search` queries the resulting index either way.
 
 ## Common Parameter Patterns
 

package/assets/skills/meko-mcp-tools/references/tools-rag-workflow.md

@@ -1,16 +1,19 @@
 # Knowledge Base / RAG workflow
 
-MCP exposes two KB tools: `knowledgebase_search` (retrieval) and
-`knowledgebase_delete_document` (remove ONE uploaded file). Index
-creation and ingestion happen out of band — through the **Meko UI**
-(`cloud.mekodata.ai`).
+MCP exposes four KB tools: `knowledgebase_search` (retrieval),
+`knowledgebase_upload_document` and `knowledgebase_document_status` (add a
+file and track its indexing), and `knowledgebase_delete_document` (remove ONE
+uploaded file). Creating, listing or renaming a knowledge base itself still
+happens out of band — through the **Meko UI** (`cloud.mekodata.ai`).
 
 ## Decision table
 
 | Situation | Use this | Why |
 |---|---|---|
 | Search an existing KB | `knowledgebase_search` | Retrieval over the indexed chunks |
-| Add documents to a KB | **Meko UI** — Datapack → Actions → **Add Knowledge** | File-upload dialog (PDF/TXT/MD/JSON/MP4, 5 MB each, 10/batch). No MCP equivalent |
+| Add a file to a KB | `knowledgebase_upload_document` + one direct S3 POST | Requests a presigned URL, then you POST the file to it; no need to leave MCP |
+| Track a file's indexing progress | `knowledgebase_document_status` | Poll with the `key` from the upload response, passed here as `document_key`, until `state` is terminal |
+| Add documents to a KB from the Cloud UI instead | **Meko UI** — Datapack → Actions → **Add Knowledge** | File-upload dialog (PDF/TXT/MD/JSON/CSV/XML/HTML, 5 MB Free / 50 MB Standard / 60 MB Pro+ each, 10/batch). Same index either way — this is the human-facing path, not required when an agent can upload directly |
 | Delete ONE uploaded file | `knowledgebase_delete_document` | Destructive; `document_id` from search hits; confirm with the user first |
 | Anything else (list / rename / re-index / source lifecycle) | **UI** | No MCP tools for the rest of KB lifecycle |
 
@@ -58,6 +61,59 @@ Lower `distance` is closer. Promoted memories appear as hits whose `metadata_fil
 Empty or nonexistent KB: `{"results": [], "count": 0}` (no error,
 just an empty array).
 
+## knowledgebase_upload_document — putting a file in
+
+This tool never touches the file's bytes itself — it only requests a
+presigned S3 URL. **You POST the file to that URL yourself**, as a
+second step:
+
+```
+knowledgebase_upload_document(
+    filename="report.pdf",
+    size=42000,
+    content_hash="<sha256 hex of the file's bytes>",
+    conversation_id="<uuid from conversation_create>",
+    agent_id="<your agent_id>",
+    content_head_base64="<base64 of the file's first ~4KB>")
+```
+
+Build the POST from every key in the response's `fields`, not a hand-picked list:
+a missing signed field such as `x-amz-security-token` makes S3 answer 403. Use
+the Python, curl, or PowerShell snippet under `knowledgebase_upload_document`
+in `tools-cookbook.md`. Confirm with the user first: the file becomes visible to
+every datapack member.
+
+Required: `filename`, `size` (the exact byte count — the URL accepts no
+other length), `content_hash`, `conversation_id`, `content_head_base64`. `content_hash` is the SHA-256 hex digest of the
+file's bytes, computed by you (e.g. `sha256sum`) — it becomes part of
+the S3 key, so it also drives the dedup check. A response with
+`already_uploaded: true` means identical content already exists; skip
+the S3 POST entirely, there is nothing left to upload. Otherwise POST
+as above: every key in `fields` as a form field (in order), then the
+bytes last as a field named `"file"`. Meko registers the object once
+it lands, so no separate confirm call is needed after the POST. Accepted extensions:
+pdf, txt, text, md, json, csv, xml, html, htm; `content_head_base64` is
+always sniffed against the extension before requesting the URL,
+rejecting an obvious mismatch immediately instead of waiting for the
+watcher — a missing/blank value is rejected as
+`content_head_base64_required`, this sniff cannot be skipped.
+Ingestion is asynchronous — save the response's `key` (as
+`document_key`) and poll `knowledgebase_document_status`.
+
+## knowledgebase_document_status — tracking it
+
+```
+knowledgebase_document_status(
+    document_key="<the upload response's `key`>",
+    conversation_id="<uuid from conversation_create>",
+    agent_id="<your agent_id>")
+```
+
+`state` covers the whole lifecycle: `awaiting_upload` and `indexing`
+are non-terminal (poll again); `completed`, `failed`, `abandoned`, and
+`oversized` are terminal. `NOT_FOUND` means the `document_key` doesn't
+belong to this datapack.
+
 ## UI ingestion — what the user sees
 
 From `cloud.mekodata.ai`:
@@ -65,11 +121,14 @@ From `cloud.mekodata.ai`:
 1. Open a datapack (`/datapacks/<name>`).
 2. Click **Actions** → **Add Knowledge**.
 3. Drag-and-drop or browse for files. Supported: PDF, TXT, MD, JSON,
-   MP4. Per-file limit 5 MB. Up to 10 files per batch.
+   CSV, XML, HTML. Per-file limit 5 MB (Free) / 50 MB (Standard) / 60 MB (Pro+). Up to 10 files per batch.
 4. The UI handles upload, storage, and index build. The user does not
    pick a bucket or provide credentials.
 5. Search results from `knowledgebase_search` become available once
    indexing completes.
 
-If a user asks you "how do I add a document to my knowledge base?",
-point them at this flow.
+If a *human* asks you "how do I add a document to my knowledge base?",
+point them at this flow. If *you* (the agent) have the file's bytes —
+already on disk, or generated mid-conversation — call
+`knowledgebase_upload_document` directly instead of describing the UI
+flow to yourself.

package/assets/skills/meko-mcp-tools/references/tools-troubleshooting.md

@@ -74,9 +74,11 @@ Bursts of requests on one token return:
 | Error | Meaning | What to do |
 |---|---|---|
 | `datapack_access_denied` | The `datapack_id` isn't a UUID, or it isn't owned by or shared with your account. The access check also fails closed on a transient server database error. | Retry once. If it repeats, stop; don't switch to another datapack. If the id came from a pin, ask the user to re-pin with the select-datapack skill. |
-| `datapack_id_required` | The call couldn't resolve a datapack. Conversation reads and writes (including `conversation_list`), `datapack_describe`, and `knowledgebase_delete_document` return it. | Pass the pinned `datapack_id`. Don't drop the parameter to make the call pass. |
+| `datapack_id_required` | The call couldn't resolve a datapack. Conversation reads and writes (including `conversation_list`), `datapack_describe`, `knowledgebase_upload_document`, `knowledgebase_document_status`, and `knowledgebase_delete_document` return it. | Pass the pinned `datapack_id`. Don't drop the parameter to make the call pass. |
 | `not_found` from `memory_get_by_id`, `memory_update`, or `memory_delete_by_id` | The memory doesn't exist **or** belongs to another account. The server returns the same error for both on purpose. | Check the id came from a search in this datapack. |
 | `memory_is_promoted` | The memory was promoted to shared knowledge and is read-only through MCP. | Point the user at the Cloud UI Learnings tab. |
+| `read_only_access` | You're a viewer on this datapack. Viewers can't write conversations or memories, call `memory_promote`, `artifact_put`, or `track_token_usage`, or delete a conversation someone else started. | Don't retry, and don't switch datapacks to get around it. Tell the user; an owner or maintainer can change their role. |
+| `role_lookup_failed` | The server couldn't read your role for this datapack, so it refused the write. | Retry. |
 | `invalid_run_id` | `memory_delete_all` got a `run_id` that isn't a conversation id. | Pass the conversation UUID, or omit `run_id`. |
 | `embedder_error` from `knowledgebase_search` | The server couldn't resolve the embedding setup for this datapack's index. | Report it; retrying won't help. |
 | `invalid_force_source` from `context_search` | `force_source` isn't `auto`, `memory`, `knowledge_base`, or `conversation`. | Omit `force_source`. |