ultimate-pi 0.1.0 → 0.1.3

package/.agents/skills/obsidian-markdown/SKILL.md

@@ -0,0 +1,237 @@
+---
+name: obsidian-markdown
+description: "Write correct Obsidian Flavored Markdown: wikilinks, embeds, callouts, properties, tags, highlights, math, and canvas syntax. Reference this when creating or editing any wiki page. Triggers on: write obsidian note, obsidian syntax, wikilink, callout, embed, obsidian markdown, wikilink format, callout syntax, embed syntax, obsidian formatting, how to write obsidian markdown."
+allowed-tools: Read Write Edit
+---
+
+# obsidian-markdown: Obsidian Flavored Markdown
+
+Reference this skill when writing any wiki page. Obsidian extends standard Markdown with wikilinks, embeds, callouts, and properties. Getting syntax wrong causes broken links, invisible callouts, or malformed frontmatter.
+
+**Cross-reference**: If the kepano/obsidian-skills plugin is installed, prefer its canonical obsidian-markdown skill for authoritative Obsidian syntax reference. Otherwise, use the reference below. See also [github.com/kepano/obsidian-skills](https://github.com/kepano/obsidian-skills).
+
+---
+
+## Wiki Path Resolution
+
+This skill provides Obsidian syntax reference. It does NOT perform file operations itself. When invoked by other skills (wiki-ingest, wiki-save, etc.), those skills handle wiki path resolution via `VAULT_WIKI_PATH`.
+
+---
+
+## Wikilinks
+
+Internal links use double brackets. The filename without extension.
+
+| Syntax | What it does |
+|---|---|
+| `[[Note Name]]` | Basic link |
+| `[[Note Name\|Display Text]]` | Aliased link (shows "Display Text") |
+| `[[Note Name#Heading]]` | Link to a specific heading |
+| `[[Note Name#^block-id]]` | Link to a specific block |
+
+Rules:
+- Case-sensitive on some systems. Match the exact filename.
+- No path needed: Obsidian resolves by filename uniqueness.
+- If two files have the same name, use `[[Folder/Note Name]]` to disambiguate.
+
+---
+
+## Embeds
+
+Embeds use `!` before the wikilink. They display the content inline.
+
+| Syntax | What it does |
+|---|---|
+| `![[Note Name]]` | Embed a full note |
+| `![[Note Name#Heading]]` | Embed a section |
+| `![[image.png]]` | Embed an image |
+| `![[image.png\|300]]` | Embed image with width 300px |
+| `![[document.pdf]]` | Embed a PDF (Obsidian renders natively) |
+| `![[audio.mp3]]` | Embed audio |
+
+---
+
+## Callouts
+
+Callouts are blockquotes with a type keyword. They render as styled alert boxes.
+
+```markdown
+> [!note]
+> Default informational callout.
+
+> [!note] Custom Title
+> Callout with a custom title.
+
+> [!note]- Collapsible (closed by default)
+> Click to expand.
+
+> [!note]+ Collapsible (open by default)
+> Click to collapse.
+```
+
+### All callout types
+
+| Type | Aliases | Use for |
+|------|---------|---------|
+| `note` |: | General notes |
+| `abstract` | `summary`, `tldr` | Summaries |
+| `info` |: | Information |
+| `todo` |: | Action items |
+| `tip` | `hint`, `important` | Tips and highlights |
+| `success` | `check`, `done` | Positive outcomes |
+| `question` | `help`, `faq` | Open questions |
+| `warning` | `caution`, `attention` | Warnings |
+| `failure` | `fail`, `missing` | Errors or failures |
+| `danger` | `error` | Critical issues |
+| `bug` |: | Known bugs |
+| `example` |: | Examples |
+| `quote` | `cite` | Quotations |
+| `contradiction` |: | Conflicting information (wiki convention) |
+
+---
+
+## Properties (Frontmatter)
+
+Obsidian renders YAML frontmatter as a Properties panel. Rules:
+
+```yaml
+---
+type: concept                    # plain string
+title: "Note Title"              # quoted if it contains special chars
+created: 2026-04-08              # date as YYYY-MM-DD (not ISO datetime)
+updated: 2026-04-08
+tags:
+  - tag-one                      # list items use - format
+  - tag-two
+status: developing
+related:
+  - "[[Other Note]]"             # wikilinks must be quoted in YAML
+sources:
+  - "[[source-page]]"
+---
+```
+
+Rules:
+- Flat YAML only. Never nest objects.
+- Dates as `YYYY-MM-DD`, not `2026-04-08T00:00:00`.
+- Lists as `- item`, not inline `[a, b, c]`.
+- Wikilinks in YAML must be quoted: `"[[Page]]"`.
+- `tags` field: Obsidian reads this as the tag list, searchable in vault.
+
+---
+
+## Tags
+
+Two valid forms:
+
+```markdown
+#tag-name             : inline tag anywhere in the body
+#parent/child-tag     : nested tag (shows hierarchy in tag pane)
+```
+
+In frontmatter:
+```yaml
+tags:
+  - research
+  - ai/obsidian
+```
+
+Do not use `#` inside frontmatter tag lists. Just the tag name.
+
+---
+
+## Text Formatting
+
+Standard Markdown plus Obsidian extensions:
+
+| Syntax | Result |
+|---|---|
+| `**bold**` | Bold |
+| `*italic*` | Italic |
+| `~~strikethrough~~` | Strikethrough |
+| `==highlight==` | Highlighted text (yellow in Obsidian) |
+| `` `inline code` `` | Inline code |
+
+---
+
+## Math
+
+Obsidian uses MathJax/KaTeX:
+
+Inline math:
+```markdown
+$E = mc^2$
+```
+
+Block math:
+```markdown
+$$
+\int_0^\infty e^{-x} dx = 1
+$$
+```
+
+---
+
+## Code Blocks
+
+Standard fenced code blocks. Obsidian highlights all common languages:
+
+````markdown
+```python
+def hello():
+    return "world"
+```
+````
+
+---
+
+## Tables
+
+Standard Markdown tables:
+
+```markdown
+| Column A | Column B | Column C |
+|----------|----------|----------|
+| Value    | Value    | Value    |
+| Value    | Value    | Value    |
+```
+
+Obsidian renders tables natively. No plugin needed.
+
+---
+
+## Mermaid Diagrams
+
+Obsidian renders Mermaid natively:
+
+````markdown
+```mermaid
+graph TD
+    A[Start] --> B{Decision}
+    B -->|Yes| C[End]
+    B -->|No| D[Loop]
+    D --> A
+```
+````
+
+Supported: `graph`, `sequenceDiagram`, `gantt`, `classDiagram`, `pie`, `flowchart`.
+
+---
+
+## Footnotes
+
+```markdown
+This sentence has a footnote.[^1]
+
+[^1]: The footnote text goes here.
+```
+
+---
+
+## What NOT to Do
+
+- Do not use `[link text](path/to/note.md)` for internal links: use `[[Note Name]]` instead.
+- Do not use HTML inside callouts: stick to Markdown.
+- Do not use `##` inside a callout body: headings don't render inside callouts.
+- Do not write `tags: [a, b, c]` inline in frontmatter: Obsidian prefers the list format.
+- Do not write ISO datetimes in frontmatter (`2026-04-08T00:00:00Z`): use `2026-04-08`.

package/.agents/skills/posthog-analyst/SKILL.md

@@ -0,0 +1,306 @@
+---
+name: posthog-analyst
+description: >
+  Analyze pi agentic conversations, errors, and misalignments using PostHog MCP.
+  Queries $ai_generation, $ai_span, $ai_trace events, segments by model, detects
+  failure patterns, cross-references the wiki's model-adaptive harness research, and
+  generates model-specific recommendations filed to the wiki. Use this skill when the
+  user wants to analyze agent performance, understand where the harness breaks per
+  model, audit LLM errors or tool failures, improve model profiles based on real data,
+  or run a postmortem on agentic sessions. Trigger on: "posthog analyze",
+  "/posthog-analyze", "analyze agent sessions", "audit llm errors", "postmortem",
+  "what's failing per model", "harness performance analysis", "find misalignments",
+  "model error patterns", "agent conversation audit".
+allowed-tools: Read Write Bash Glob Grep
+compatibility:
+  requires: [PostHog MCP bridge enabled in pi, POSTHOG_PERSONAL_API_KEY set]
+---
+
+# posthog-analyst: Agentic Conversation Analysis via PostHog MCP
+
+You analyze real agentic conversations captured in PostHog to find where the harness breaks — per model. You do NOT auto-fix. You generate specific, actionable recommendations and file them to the wiki as structured analysis pages. The wiki's model-adaptive harness research is your reference for what failure modes to look for and what fixes exist.
+
+## Core Loop
+
+```
+1. VERIFY MCP   → /posthog-mcp-status
+2. QUERY EVENTS → execute-sql against $ai_generation, $ai_span, $ai_trace
+3. SEGMENT      → Group by $ai_model, $ai_agent_name, $ai_stop_reason
+4. DETECT       → Match patterns against known model failure modes (from wiki)
+5. CROSS-REF    → Read wiki model-adaptive-harness pages for fix catalog
+6. RECOMMEND    → Generate per-model recommendations
+7. FILE         → Write findings to vault/wiki/analyses/posthog-YYYY-MM-DD.md
+```
+
+Every analysis gets a wiki page. Knowledge compounds.
+
+## Step 1: Verify PostHog MCP
+
+Run `/posthog-mcp-status` to confirm the bridge is connected. If it fails, tell the user:
+
+> PostHog MCP not connected. Set `POSTHOG_PERSONAL_API_KEY` and restart pi, or run `/posthog-mcp-reload`.
+
+If the user hasn't set up PostHog tracing (no `$ai_*` events being captured), tell them:
+
+> No LLM analytics being captured. Set `POSTHOG_API_KEY` to your project key and restart pi. Events will start flowing immediately.
+
+## Step 2: Query Events
+
+Use the PostHog MCP `execute-sql` tool (or equivalent query tool registered by the bridge). Query the following event types, filtered to this project:
+
+### Core Queries
+
+**A) Error rate by model — `$ai_generation` events:**
+```sql
+SELECT
+  properties.$ai_model as model,
+  countIf(properties.$ai_is_error = true) as errors,
+  count() as total,
+  round(errors / total * 100, 1) as error_pct,
+  avg(properties.$ai_latency) as avg_latency_sec,
+  avg(properties.$ai_total_tokens) as avg_tokens
+FROM events
+WHERE event = '$ai_generation'
+  AND properties.$ai_project_name = 'ultimate-pi'
+  AND timestamp >= now() - INTERVAL {days} DAY
+GROUP BY model
+ORDER BY total DESC
+```
+
+**B) Tool failure rate by model — `$ai_span` events:**
+```sql
+SELECT
+  properties.$ai_model as model,
+  properties.$ai_span_name as tool,
+  countIf(properties.$ai_is_error = true) as errors,
+  count() as total,
+  round(errors / total * 100, 1) as error_pct,
+  avg(properties.$ai_latency) as avg_latency_sec
+FROM events
+WHERE event = '$ai_span'
+  AND properties.$ai_project_name = 'ultimate-pi'
+  AND timestamp >= now() - INTERVAL {days} DAY
+GROUP BY model, tool
+HAVING errors > 0
+ORDER BY error_pct DESC
+```
+
+**C) Stop reason distribution by model — `$ai_generation` events:**
+```sql
+SELECT
+  properties.$ai_model as model,
+  properties.$ai_stop_reason as stop_reason,
+  count() as count,
+  round(count() * 100.0 / sum(count()) OVER (PARTITION BY properties.$ai_model), 1) as pct
+FROM events
+WHERE event = '$ai_generation'
+  AND properties.$ai_project_name = 'ultimate-pi'
+  AND timestamp >= now() - INTERVAL {days} DAY
+GROUP BY model, stop_reason
+ORDER BY model, count DESC
+```
+
+**D) Token/cost efficiency by model — `$ai_trace` events:**
+```sql
+SELECT
+  properties.$ai_model as model,
+  count() as sessions,
+  avg(properties.$ai_total_input_tokens) as avg_input_tokens,
+  avg(properties.$ai_total_output_tokens) as avg_output_tokens,
+  avg(properties.$ai_latency) as avg_session_sec,
+  countIf(properties.$ai_is_error = true) as errored_sessions
+FROM events
+WHERE event = '$ai_trace'
+  AND properties.$ai_project_name = 'ultimate-pi'
+  AND timestamp >= now() - INTERVAL {days} DAY
+GROUP BY model
+ORDER BY sessions DESC
+```
+
+**E) High-error traces — raw error messages by model:**
+```sql
+SELECT
+  properties.$ai_model as model,
+  properties.$ai_error as error_message,
+  properties.$ai_user_prompt as user_prompt,
+  count() as occurrences,
+  min(timestamp) as first_seen,
+  max(timestamp) as last_seen
+FROM events
+WHERE event IN ('$ai_generation', '$ai_trace')
+  AND properties.$ai_project_name = 'ultimate-pi'
+  AND properties.$ai_is_error = true
+  AND timestamp >= now() - INTERVAL {days} DAY
+GROUP BY model, error_message, user_prompt
+ORDER BY occurrences DESC
+LIMIT 50
+```
+
+### Date Range
+
+If the user specifies a date range, use it. Otherwise default to the last 7 days. If they say "all time" or "everything", use 30 days.
+
+If queries return empty: the project may not have tracing active yet. Tell the user and stop.
+
+## Step 3: Segment and Detect Patterns
+
+Group data by model. For each model present, run through this checklist:
+
+### Pattern 1: Premature Completion (GPT Warning)
+**Signals**: High `stop_reason=stop` rate without tool calls, low tokens/turn, sessions end after 1-2 turns
+**Wiki reference**: [[harness-configuration-layers]] — L2 Gate Design (GPT needs hard-gate, L4 Completion Model)
+**What it looks like in data**: Model says "I'm done" after plausible-but-incomplete answer. Stops before verification gate.
+
+### Pattern 2: Tool Loop (any model)
+**Signals**: Same tool called 3+ times in one trace, tool returns errors but agent retries same approach
+**Wiki reference**: [[execution-feedback-loop]] — agent doesn't change strategy on failure
+**What it looks like in data**: Consecutive `$ai_span` events with same tool name, same or similar input, `$ai_is_error=true` on multiple consecutive spans.
+
+### Pattern 3: High Token Waste (any model)
+**Signals**: Sessions with >2x median tokens, high cache-miss rates, large output tokens relative to task complexity
+**Wiki reference**: [[ast-truncation]], [[progressive-disclosure-agents]]
+**What it looks like in data**: `$ai_total_tokens` outliers, `cache_read_input_tokens` near zero despite repeated reads.
+
+### Pattern 4: Model-Specific Misalignment
+**Signals by model** (cross-reference wiki [[harness-configuration-layers]] for dimensions):
+
+| Model | Failure Mode | Data Signal | L1-L4 Layer |
+|-------|--------------|-------------|-------------|
+| GPT variants | Stops early (premature completion) | Stop reason `stop` without verification gate passing | L2 Gate, L4 Completion |
+| GPT variants | Anchors on first-seen content, ignores updates | Low tool retry rate despite errors — agent doubles down | L1 Signal (constraints-first needed) |
+| GPT variants | Confused by nested structures | High error rate on multi-step tasks, tool chains break | L1 Signal (flat structure needed) |
+| Claude/Opus | Overconfident self-assessment | Stop reason `stop`, no errors, but output misses spec requirements | L2 Gate (needs checklist, not self-assessment) |
+| Claude/Opus | Ignores explicit markers if too verbose | Long output, many tokens, but key constraints missed | L1 Signal (too much narrative drowns signals) |
+| Haiku (subagents) | Hallucinates in summaries | `$ai_agent_name` contains `/explore` or `/haiku`, error rate on read | L2 Gate (hard evidence gate needed) |
+
+### Pattern 5: Drift from Spec
+**Signals**: Error messages contain "mismatch", "not found", "unexpected", "did not match". Sessions where later turns undo earlier edits.
+**Wiki reference**: [[verification-drift-detection]], [[grounding-checkpoints]]
+**What it looks like in data**: Sequential edit tool calls that revert each other, errors on verification steps.
+
+## Step 4: Cross-Reference Wiki Fix Catalog
+
+For each detected pattern, read the relevant wiki page to find the fix catalog:
+
+| Pattern | Wiki Page | Fix Type |
+|---------|-----------|----------|
+| Premature completion | [[harness-configuration-layers]] L2, L4 | Add hard gate, falsifiable checklist, enforced self-audit |
+| Tool loops | [[execution-feedback-loop]] | Diff-aware testing, structured error output, max retry gates |
+| Token waste | [[ast-truncation]], [[progressive-disclosure-agents]] | AST truncation, L0-L3 progressive disclosure, cache hints |
+| Model-specific misalignment | [[model-adaptive-harness]], [[harness-configuration-layers]] | Adjust per-model dimension values in model-profiles |
+| Drift from spec | [[verification-drift-detection]], [[grounding-checkpoints]] | Smallest-verifiable-change, re-grounding triggers |
+| Haiku hallucination | [[model-routing-agents]] | Confidence scoring, critical reads stay on frontier |
+| Repeated edit failures | [[fuzzy-edit-matching]], [[inline-post-edit-validation]] | Fuzzy matching, syntax validation before surfacing error |
+
+Read these pages to extract the concrete fix options. Recommendations must reference specific wiki pages with wikilinks.
+
+## Step 5: Generate Recommendations
+
+For each model with detected issues, produce recommendations in this format:
+
+```markdown
+### Model: `$ai_model_value`
+
+**Sessions analyzed**: N | **Error rate**: X% | **Avg tokens/session**: Y
+
+#### Issue 1: [Pattern name]
+- **Evidence**: [specific data — error counts, examples, rates]
+- **Root cause**: [why this model fails this way, with wiki reference]
+- **Recommendation**: [concrete action]
+  - **Layer affected**: L1/L2/L3/L4 (from harness configuration layers)
+  - **What to change**: [specific file, dimension, value]
+  - **Expected impact**: [what should improve]
+
+#### Issue 2: ...
+```
+
+Recommendations are concrete and actionable. They reference specific files to change (e.g., `.pi/skills/wiki-autoresearch/references/model-profiles.md`, specific SKILL.md sections, harness config). They specify which harness layer and which dimension.
+
+**Do NOT auto-apply fixes.** Only recommend. The user decides what to change.
+
+## Step 6: File to Wiki
+
+Write the full analysis to `vault/wiki/analyses/posthog-YYYY-MM-DD.md`:
+
+```yaml
+---
+type: analysis
+title: "PostHog Agent Analysis — YYYY-MM-DD"
+created: YYYY-MM-DD
+updated: YYYY-MM-DD
+tags: [posthog, agent-analysis, model-performance, harness]
+sources:
+  - "PostHog MCP — $ai_generation, $ai_span, $ai_trace events"
+related:
+  - "[[model-adaptive-harness]]"
+  - "[[harness-configuration-layers]]"
+status: complete
+---
+
+# PostHog Agent Analysis — YYYY-MM-DD
+
+## Summary
+[2-3 sentence executive summary. Models analyzed, top finding, most critical recommendation.]
+
+## Data Scope
+- Date range: [start] to [end]
+- Events analyzed: N generations, M spans, T traces
+- Models found: [list]
+- Project filter: ultimate-pi
+
+## Model Breakdown
+
+### Model: [name]
+[Recommendations per the template above]
+
+## Cross-Model Patterns
+[Patterns that appear across multiple models — systemic harness issues, not model-specific]
+
+## Recommendations Summary
+| Priority | Model | Issue | Recommendation | Layer |
+|----------|-------|-------|----------------|-------|
+| Critical | ... | ... | ... | ... |
+| High | ... | ... | ... | ... |
+| Medium | ... | ... | ... | ... |
+
+## Next Steps
+[What to do with these findings. Suggest ADR if recommendations are significant.]
+```
+
+After filing, update `vault/wiki/index.md` (add to analyses if category exists, or note inline), update `vault/wiki/log.md` (append entry at TOP), and update `vault/wiki/hot.md` (add key findings to Recent Context).
+
+## Quick Mode
+
+If the user says "quick posthog analysis" or "just error rates per model", run only queries A and B, produce a summary table, and skip detailed recommendations. Offer to go deep if they want more.
+
+```markdown
+## Quick Summary
+
+| Model | Generations | Errors | Error % | Avg Tokens | Avg Latency |
+|-------|-------------|--------|---------|------------|-------------|
+| claude-sonnet-4-20250514 | 142 | 3 | 2.1% | 8,200 | 4.2s |
+| gpt-4o | 28 | 5 | 17.9% | 12,400 | 6.8s |
+
+**Top failing tools**: edit (4 errors), bash (3 errors), read (1 error)
+
+Want a deep analysis with wiki cross-reference and per-model recommendations?
+```
+
+## Troubleshooting
+
+| Problem | Likely cause | Fix |
+|---------|-------------|-----|
+| `/posthog-mcp-status` fails | MCP bridge not configured | `export POSTHOG_PERSONAL_API_KEY=phx_...` then restart pi |
+| Queries return empty | No tracing data captured | Set `POSTHOG_API_KEY` and generate some agent sessions |
+| `execute-sql` tool not found | MCP tool list not loaded | Run `/posthog-mcp-reload` |
+| Query syntax errors | PostHog uses ClickHouse SQL | Wrap column names in double quotes if they contain special chars |
+| Only one model in results | Only one model used in the date range | Extend date range or note that comparison isn't possible |
+
+## Token Discipline
+
+- Step 1 (mcp-status): ~200 tokens
+- Step 2 (queries): response depends on result size. Use spill-to-file (enabled by default at 12K chars).
+- Step 3-4 (wiki cross-ref): Read only relevant wiki pages. ~300 tokens each.
+- Step 5-6 (recommend + file): ~2,000 tokens for analysis page
+
+Total: ~3,000-5,000 tokens for a typical analysis. If query results are large, PostHog MCP spills them to `/tmp/posthog-mcp/` automatically.

package/.agents/skills/posthog-analyst/evals/evals.json

@@ -0,0 +1,23 @@
+{
+	"skill_name": "posthog-analyst",
+	"evals": [
+		{
+			"id": 1,
+			"prompt": "run a quick posthog analysis on the last 3 days — just error rates per model, nothing deep. I want to know if claude or gpt is having more problems.",
+			"expected_output": "A summary table showing error rates per model from the last 3 days, with a brief verdict on which model has more issues. Follows Quick Mode format: table with error counts and rates, offer to go deeper.",
+			"files": []
+		},
+		{
+			"id": 2,
+			"prompt": "/posthog-analyze for the last 14 days. I'm seeing tool loops where the agent keeps retrying the same failing bash command and it's wasting tokens. Check if this is model-specific and what we should change in the harness to stop it.",
+			"expected_output": "Full analysis: query results showing tool error patterns by model, identification of tool loop pattern, cross-reference with execution-feedback-loop wiki page, model-specific recommendations for gate design changes (L2). Filed to wiki as an analysis page.",
+			"files": []
+		},
+		{
+			"id": 3,
+			"prompt": "do a full postmortem on everything from yesterday. I want to know every misalignment, every error, and what specific changes each model profile needs. File it all to the wiki.",
+			"expected_output": "Comprehensive analysis filed to vault/wiki/analyses/posthog-YESTERDAY.md with per-model breakdown, pattern detection across all 5 patterns, cross-references to wiki model-adaptive-harness pages, and a prioritized recommendations table. Updates index.md, log.md, and hot.md.",
+			"files": []
+		}
+	]
+}