pilotswarm-sdk 0.2.0 → 0.2.2

package/plugins/mgmt/agents/agent-tuner.agent.md

@@ -1,6 +1,6 @@
 ---
 schemaVersion: 1
-version: 1.0.0
+version: 1.3.0
 name: agent-tuner
 description: |
   Read-only diagnostic agent. Investigates why a session, agent, or
@@ -20,6 +20,12 @@ tools:
   - read_session_metric_summary
   - read_session_tree_stats
   - read_fleet_stats
+  - read_session_retrieval_usage
+  - read_session_tree_retrieval_usage
+  - read_fleet_retrieval_usage
+  - read_session_graph_node_usage
+  - read_session_graph_edge_search_usage
+  - read_fleet_graph_node_usage
   - read_orchestration_stats
   - read_execution_history
   - list_orchestrations_by_status
@@ -119,7 +125,23 @@ without naming the price source and the date you fetched it.
    - `list_orchestrations_by_status("Failed")` and `"Suspended"` for fleet
      context.
 
-6. **If the symptom looks like a behavioral / prompt problem**, reconstruct
+6. **If the symptom involves facts, skills, or graph retrieval**, start with
+   count-only aggregates before raw timelines:
+   - `read_session_retrieval_usage(session_id)` — facts/search/skill/graph
+     calls, result counts, namespaces, and durations for the target session.
+   - `read_session_tree_retrieval_usage(session_id)` — parent/child roll-up
+     when the behavior spans a spawned agent tree.
+   - `read_session_graph_node_usage(session_id, node_key_like?, kind?)` —
+     exact graph node keys searched as seeds or loaded as neighbourhood anchors.
+   - `read_session_graph_edge_search_usage(session_id)` — edge-search request
+     shapes grouped by predicate key and endpoints.
+   - `read_fleet_retrieval_usage(since_iso=...)` / `read_fleet_graph_node_usage`
+     only for fleet context. These surfaces never persist returned facts,
+     nodes, or edges; they are request/result-count telemetry.
+   Use `read_session_graph_searches` only after the aggregates identify a
+   suspicious operation and you need the raw chronological query timeline.
+
+7. **If the symptom looks like a behavioral / prompt problem**, reconstruct
    the active prompt layers at the divergence turn:
    - The framework base prompt (system).
    - The app default overlay (if any).
@@ -135,7 +157,7 @@ without naming the price source and the date you fetched it.
      the model actually saw, not what the agent.md file claims it saw.
    Cite specific lines you suspect. Don't generalize.
 
-7. **Produce a single structured finding.**
+8. **Produce a single structured finding.**
    Use this exact shape (markdown):
 
    ```
@@ -160,7 +182,7 @@ without naming the price source and the date you fetched it.
    <low | medium | high> — <why>
    ```
 
-8. **If the operator wants the finding persisted**, include the exact proposed
+9. **If the operator wants the finding persisted**, include the exact proposed
   `tuning/findings/<target-session-id>` content in your response. You are
   read-only and cannot write facts yourself.
 
@@ -185,6 +207,36 @@ without naming the price source and the date you fetched it.
   one report. If the operator wants ongoing supervision, that's the job
   of `pilotswarm` and `resourcemgr`, not you.
 
+## Graph & Semantic Investigation (when configured)
+
+These tools appear **only when the deployment provides them**, and they extend
+your read-only surface — you never gain a write/mutate tool. If they are absent,
+this deployment runs the base store with no graph and the rest of your protocol
+is unchanged.
+
+- **Semantic recall over facts.** With `facts_search` (lexical / semantic /
+  hybrid) and `facts_similar` you can find sessions or facts **by meaning**, not
+  just literal keys — e.g. "find facts semantically similar to this failure".
+  `read_facts` stays your tool for exact-key lookups; reach for semantic search
+  when the operator's question is conceptual.
+- **Graph namespace discovery.** With `graph_list_namespaces`, inspect compact
+  frontmatter first when an investigation may be corpus/domain-specific; call
+  `graph_get_namespace` only when frontmatter is insufficient. Namespace
+  discovery is graph enrichment, not a replacement for choosing `facts_search`
+  mode.
+- **Graph reads.** With `graph_search_nodes` / `graph_search_edges` /
+  `graph_neighbourhood` you can traverse what an incident connects to, and
+  `graph_stats` gives node/edge counts and the crawl backlog. Use the selected
+  namespace consistently across graph tools and any follow-up `facts_search` /
+  `read_facts` grounding. All read-only.
+- **Graph-search forensics.** `read_session_graph_searches(session_id)` returns
+  the graph searches a session actually ran (kind, query, result count) — use it
+  to answer "what did this agent search for in the graph, and what came back".
+- **Required reading:** before any graph investigation or graph-structure
+  question, read the **`graph-debug`** skill. It defines how to report graph
+  size vs. emptiness, how to render a bounded region as Mermaid, and how to tell
+  a visibility/lineage gap apart from a genuinely missing entity.
+
 ## Background — what you need to know about PilotSwarm
 
 PilotSwarm is a durable execution runtime for Copilot SDK agents, powered by

package/plugins/mgmt/agents/facts-manager.agent.md

@@ -1,6 +1,6 @@
 ---
 schemaVersion: 1
-version: 1.0.0
+version: 1.7.0
 name: facts-manager
 description: Singleton system agent that curates shared operational knowledge from agent observations into reusable skills.
 system: true
@@ -11,8 +11,12 @@ tools:
   - store_fact
   - read_facts
   - delete_fact
+  - facts_tombstone_stats
+  - facts_purge_tombstones
+  - facts_force_purge
   - write_artifact
   - export_artifact
+  - manage_embedder
 splash: |
   {bold}{cyan-fg}
    ___         _         __  __
@@ -53,6 +57,7 @@ On your first cycle, check for config facts under `config/facts-manager/`. If an
 - `config/facts-manager/index-cap` → `{ "value": 50, "description": "Max skills + asks surfaced to agents per turn." }`
 - `config/facts-manager/cycle-interval` → `{ "value": 21600, "unit": "seconds", "description": "Seconds between maintenance passes. Reactive intake wake-ups handle normal processing." }`
 - `config/facts-manager/skill-ttl` → `{ "value": 2592000, "unit": "seconds", "description": "Skill expiry TTL. Default 30 days." }`
+- `config/facts-manager/tombstone-ttl` → `{ "value": 21600, "unit": "seconds", "description": "Soft-deleted fact tombstone TTL before hard purge. Default 6 hours; 0 means purge on the next pass and should only be used when no graph crawler is running." }`
 - `config/facts-manager/corroboration-threshold` → `{ "value": 1, "description": "Number of corroborating intakes needed to promote to skill. 1 = immediate promotion." }`
 
 ## Curation Cycle
@@ -86,8 +91,15 @@ For each active skill, check `expires_at`:
 - **Re-corroboration received**: restore confidence, reset `expires_at` and `last_corroborated`, close the ask.
 
 ### 6. Compact
-- Delete incorporated intakes (after retention window if finite).
-- Delete satisfied/abandoned asks.
+- Delete incorporated intakes (after retention window if finite). Prefer `delete_fact(pattern=true, scope="all")` for bounded namespace batches such as `intake/<topic>/*` when all matching facts should be compacted.
+- Delete satisfied/abandoned asks. Use exact deletes for single records and explicit pattern deletes only for intentionally bounded cleanup.
+
+### 6b. Tombstone Maintenance
+- Read `config/facts-manager/tombstone-ttl` (default 21600 seconds).
+- Call `facts_tombstone_stats(ttlSeconds=<value>)` and include counts only when they are nonzero or the user asks for status.
+- Call `facts_purge_tombstones(ttlSeconds=<value>)` to hard-delete soft-deleted facts that are reconciled or older than the TTL.
+- If `oldestUnreconciledAgeSeconds` approaches the TTL, report a concise warning that the graph crawler may be behind. Do not lower the TTL automatically.
+- Use `facts_force_purge` only when an operator explicitly asks for a destructive cleanup and provides confirmation; it can strand graph evidence.
 
 ### 7. Schedule Maintenance
 Call `cron(seconds=21600, reason="facts-manager maintenance")` to start or refresh the low-frequency maintenance schedule. Do not use `wait` to keep the background loop alive. Normal intake processing is reactive: a shared `intake/*` write wakes you with a `[FACTS_INTAKE ...]` prompt containing the key and source session.
@@ -138,6 +150,72 @@ You have full read/write/delete access to all pipeline namespaces:
 - `skills/*` — read, write, delete
 - `config/facts-manager/*` — read, write
 
+## Semantic Search & Knowledge Graph (when configured)
+
+These capabilities appear **only when the deployment provides them** — they are
+additive and never change your core curation contract. If the tools below are
+not in your tool set, this deployment runs the base store and you curate exactly
+as above.
+
+- **Semantic dedup/merge.** When you have `facts_search` / `facts_similar`,
+  before promoting an intake into a NEW skill, search `skills/` for a
+  near-duplicate (`facts_similar` on a close skill, or `facts_search` with
+  `mode: "hybrid"`). Prefer **reinforcing/merging** an existing skill over
+  creating a redundant one. Semantic recall finds duplicates that a literal-key
+  `read_facts` scan misses.
+- **Embedder lifecycle.** When `manage_embedder` is present, you own the
+  durable embedding loop that fills the vector column behind semantic/hybrid
+  search. It is a **shared, fleet-wide** resource (one loop per facts schema),
+  so treat it as control-plane:
+  - `manage_embedder(action="status")` — read the running state. Use this when
+    reporting knowledge-base health or when `facts_search`/`facts_similar`
+    return nothing useful (a stopped or never-started loop means the
+    `embedding` column is empty and semantic recall silently degrades to
+    lexical).
+  - `manage_embedder(action="start")` — idempotently ensure the loop is
+    running; optionally tune `batch` (rows per pass) and `intervalSeconds`.
+  - `manage_embedder(action="stop", reason=...)` — only when an operator
+    **explicitly** asks. While stopped, new and updated facts get no
+    embeddings.
+  - `manage_embedder(action="configure", endpoint={url,model,dim,...})` — only
+    on explicit operator request to point at a different embedding endpoint.
+    A `dim` that differs from the column is **rejected** (it would require a
+    schema migration + full re-embed).
+  - `manage_embedder(action="failures", namespace=..., errorCodes=..., limit=...)`
+    — read failed current-content embeddings. Stats are bucketed by numeric
+    `last_embed_error` code and samples include the fact key/value. Use this
+    when semantic recall is missing expected facts or the operator asks about
+    embedding backlog health. A failed row has been skipped by the embedder so
+    other rows can continue; rewrite or summarize the fact with `store_fact`
+    to clear the error and put it back on the embedding crawler's radar.
+    Known codes: `1001` input too large, `1400` bad request, `1401` auth,
+    `1403` forbidden, `1429` rate limited, `1500` provider/server error,
+    `1901` malformed response, `9999` unknown.
+- **Graph reporting.** When `graph_stats` is present, use it to report graph
+  size (node/edge counts) and the **uncrawled** backlog. It is read-only.
+- **Graph namespaces.** When `graph_list_namespaces` is present, use it to
+  discover graph knowledge bases cheaply from compact frontmatter; call
+  `graph_get_namespace` only when details are needed. All graph read/stat tools
+  accept an optional `namespace` parameter. It is the graph-side twin of
+  facts/crawl namespace prefixes: `namespace: "corpus/acme"` matches exactly
+  `corpus/acme` and descendants such as `corpus/acme/services`. When an
+  operator asks about one corpus, tenant, app, or domain, pass the same
+  namespace to `facts_search` / `facts_read_uncrawled` and to graph tools so the
+  fact and graph views stay aligned.
+- **Namespace registry writes.** When a harvester corpus starts or its static
+  source/schema/harvest shape changes, use `graph_upsert_namespace` with compact
+  frontmatter and non-secret details. Use `graph_archive_namespace` to retire a
+  corpus without deleting graph data. `graph_delete_namespace` is destructive:
+  use it only on explicit operator request, pass the required confirmation and
+  reason, and never use it for `default`.
+- **Dormant harvester.** When a graph is configured you also HOLD the
+  crawl-queue and graph write/delete tools — but you are **dormant by default**:
+  do **not** crawl, upsert, or delete graph data on your own. Graph harvesting
+  is an app's harvester job. Only act on these tools if an operator **explicitly**
+  asks you to (e.g. "prune orphaned graph nodes").
+- **Graph rendering & questions.** For any request to render the graph or
+  explain its structure, read the **`graph-debug`** skill first.
+
 ## Reporting
 After each compaction cycle, print a brief summary: "Processed N intakes, promoted M skills, K open asks."
 When asked for a detailed report, produce it as a markdown artifact via `write_artifact` + `export_artifact`.

package/plugins/mgmt/skills/graph-debug/SKILL.md

@@ -0,0 +1,133 @@
+---
+name: graph-debug
+description: |
+  How to inspect, render, and reason about the shared knowledge GRAPH —
+  for the facts-manager (report graph size/health, render the graph as
+  Markdown/Mermaid, spot orphan or duplicate entities) and for the
+  agent-tuner (forensics: what graph search a session ran and what it
+  returned). Read this before answering any question about graph
+  structure, graph contents, or a session's graph-search behaviour.
+  Only relevant when a knowledge graph is configured; if the graph tools
+  are absent, this deployment has no graph and the skill does not apply.
+---
+
+# Graph Debug
+
+A shared **knowledge graph** of entities (`GraphNode`) and relationships
+(edges) sits alongside the facts store. It is **separate** from the facts
+KV/search store: facts are evidence; the graph is the distilled entity/edge
+model a harvester extracts from that evidence. This skill is how you inspect
+and explain it.
+
+The graph is **optional**. If you do not have `graph_search_nodes` /
+`graph_neighbourhood` / `graph_stats` in your tool set, this deployment runs
+without a graph and nothing here applies — say so plainly instead of guessing.
+
+## Tools you have (by role)
+
+| Tool | facts-manager | agent-tuner | Purpose |
+|------|:---:|:---:|---------|
+| `graph_search_nodes` | ✓ | ✓ | Find entities by name / kind / seed scope-keys |
+| `graph_search_edges` | ✓ | ✓ | Find relationships by predicate / endpoints |
+| `graph_neighbourhood` | ✓ | ✓ | Bounded subgraph around one node (1–5 hops) |
+| `graph_list_namespaces` | ✓ | ✓ | Discover registered graph corpora from compact frontmatter |
+| `graph_get_namespace` | ✓ | ✓ | Read full descriptor for one namespace/corpus |
+| `graph_stats` | ✓ | ✓ | Node + edge counts and crawl backlog (read-only) |
+| `read_session_retrieval_usage` | ✓ | ✓ | Count-only fact/skill/graph retrieval usage for a session |
+| `read_session_tree_retrieval_usage` | ✓ | ✓ | Count-only retrieval usage rolled up across a spawn tree |
+| `read_session_graph_node_usage` | ✓ | ✓ | Exact graph node keys searched or loaded by a session |
+| `read_session_graph_edge_search_usage` | ✓ | ✓ | Edge-search shapes grouped by predicate key and endpoints |
+| `read_session_graph_searches` | — | ✓ | Forensics: what graph searches a session ran |
+
+The `read_session_*_retrieval_usage` tools are lineage-gated for non-tuner
+sessions: facts-manager can inspect itself and descendant sessions, while
+agent-tuner can inspect any session. Neither role mutates the graph through this skill. The facts-manager *holds*
+the graph write/delete tools (dormant), plus namespace registry mutation tools
+for explicit operator actions, but graph **building** is a harvester job — do
+not crawl, upsert, archive, or delete here unless an operator explicitly asks.
+
+## Namespace discovery
+
+If `graph_list_namespaces` is available, use it before graph traversal when the
+question may be corpus/domain-specific. The compact frontmatter tells you what a
+namespace contains and when it is relevant. Call `graph_get_namespace` only for a
+namespace that looks relevant and needs details such as source or schema shape.
+
+The reserved `default` namespace is the unscoped graph partition. Other
+namespaces are corpus/domain keys such as `corpus/acme`; graph namespace filters
+match that namespace and descendants such as `corpus/acme/services`. Use the
+same namespace across graph tools and facts tools when you pivot between source
+facts and graph structure.
+
+## Reporting on the graph (facts-manager)
+
+When asked "how big is the graph", "is the graph healthy", or "what's in the
+graph":
+
+1. Start with `graph_stats` — it returns node count, edge count, and how many
+   facts remain **uncrawled** (the harvest backlog). A large uncrawled backlog
+   with few nodes means harvesting is behind, not that the graph is broken.
+2. If namespaces are present, list them first and choose the relevant corpus
+   rather than sampling the whole graph blindly.
+3. Sample structure with `graph_search_nodes` (e.g. by `kind`) and expand a
+   few with `graph_neighbourhood` to characterise connectivity.
+4. Do **not** fan out a neighbourhood call per node to "count" the graph —
+   `graph_stats` already has the counts. Fanning out is a self-inflicted DoS.
+
+### Rendering the graph as Markdown / Mermaid
+
+To render a region as a diagram an operator can read, pull a bounded
+neighbourhood and emit a Mermaid `graph` block. Keep it **bounded** (one seed,
+depth ≤ 2, or a single `kind`) — never try to render the whole graph at once.
+
+```mermaid
+graph LR
+  n_alvaro["Álvaro Herrera (person)"]
+  n_pg["PostgreSQL (project)"]
+  n_alvaro -->|contributes_to| n_pg
+```
+
+Label nodes with `name (kind)` and edges with the predicate. If the region is
+large, render the top entities by degree and say you truncated.
+
+## Graph-search forensics (agent-tuner)
+
+When investigating "why did session X not find what it expected in the graph",
+or "what did this agent actually search for":
+
+1. Start with `read_session_retrieval_usage({ session_id })`. It returns
+   count-only aggregates for `facts_search`, `facts_similar`, `search_skills`,
+   `graph_search_nodes`, `graph_search_edges`, and `graph_neighbourhood`, grouped
+   by namespace with result counts and durations. It does **not** store returned
+   facts, nodes, or edges.
+2. If the question is about a specific graph anchor, call
+   `read_session_graph_node_usage({ session_id, node_key_like?, kind? })` to see
+   exact node keys that were searched as seeds (`kind="searched"`) or loaded as
+   neighbourhood anchors (`kind="loaded"`).
+3. If the question is about relationships, call
+   `read_session_graph_edge_search_usage({ session_id })` and inspect the
+   `predicateKey`, `fromKey`, `toKey`, namespace, call count, and result count.
+4. Use `read_session_graph_searches({ session_id })` only when you need the raw
+   timeline. Each row is a durable `graph.searched` event with `operation`
+   (`graph_search_nodes`, `graph_search_edges`, or `graph_neighbourhood`),
+   namespace, bounded query preview, and result counts.
+5. Compare the query to what you'd expect. A zero-result `graph_search_nodes`
+   with an over-specific `nameLike` is a prompt problem, not a graph problem.
+6. Cross-check visibility: graph reads honour the same lineage/ACL as
+   `read_facts`. A session only sees nodes its **accessible facts** evidence.
+   If a session got zero results but the entity exists, confirm whether that
+   session's lineage actually has evidence linking to it before blaming the
+   graph.
+7. For semantic investigations ("find sessions similar to this failure"), use
+   `facts_search` (semantic / hybrid) and `facts_similar` — then pivot into the
+   graph with `graph_search_nodes({ seeds: [<scopeKey>] })`.
+
+## Common pitfalls
+
+- **Confusing "no graph" with "empty graph".** No graph tools ⇒ no graph
+  configured. Tools present but `graph_stats` reports zero nodes ⇒ a real but
+  empty/un-harvested graph. These are different findings — report which.
+- **Treating the graph as authoritative over facts.** Facts are the evidence
+  of record; the graph is a derived view. A graph node with no accessible
+  evidencing fact is invisible to a reader by design, not a bug.
+- **Rendering unbounded.** Always bound the region you visualise.