ultimate-pi 0.1.7 → 0.2.2

package/.agents/skills/wiki-autoresearch/SKILL.md

@@ -2,38 +2,94 @@
 name: wiki-autoresearch
 description: >
   Autonomous iterative research loop. Takes a topic, runs web searches, fetches sources,
-  synthesizes findings, and files everything into the wiki as structured pages.
-  Based on Karpathy's autoresearch pattern: program.md configures objectives and constraints,
-  the loop runs until depth is reached, output goes directly into the knowledge base.
+  synthesizes findings, and builds a queryable knowledge graph via Graphify.
+  Based on Karpathy's autoresearch pattern: program.md configures objectives and
+  constraints, the loop runs until depth is reached, output goes into the graph.
   Triggers on: "/wiki-autoresearch", "/autoresearch", "wiki-autoresearch", "autoresearch",
   "research [topic]", "deep dive into [topic]", "investigate [topic]",
   "find everything about [topic]", "research and file", "go research", "build a wiki on".
-allowed-tools: Read Write Edit Glob Grep WebFetch WebSearch
+allowed-tools: Read Write Edit Glob Grep WebFetch WebSearch Bash
 ---
 
-# wiki-autoresearch: Autonomous Research Loop
+# wiki-autoresearch: Autonomous Research Loop with Graphify
 
-You are a research agent. You take a topic, run iterative web searches, synthesize findings, and file everything into the wiki. The user gets wiki pages, not a chat response.
+You are a research agent. You take a topic, run iterative web searches, synthesize
+findings, and build a queryable knowledge graph via Graphify. The user gets a
+graph — not a wall of text.
 
-This is based on Karpathy's autoresearch pattern: a configurable program defines your objectives. You run the loop until depth is reached. Output goes into the knowledge base.
+This is based on Karpathy's autoresearch pattern. You run the loop until depth
+is reached. Output goes into the knowledge graph only — no separate report file.
 
 ---
 
-## Wiki Path Resolution
+## Raw Source Directory
 
-All `wiki/` paths in this skill are relative to the wiki directory inside the Obsidian vault. Resolve before any file operation:
+All fetched sources are saved to `./raw/` for Graphify ingestion:
 
 ```bash
-WIKI_PATH="${WIKI_VAULT_PATH:-${VAULT_WIKI_PATH:-vault/wiki}}"
+RAW_PATH="./raw"
+mkdir -p "$RAW_PATH"
 ```
 
-Use `$WIKI_PATH/` as the prefix for all `wiki/...` file paths. Example: `wiki/sources/` → `$WIKI_PATH/sources/` (default: `vault/wiki/sources/`).
+This folder is the single source of truth for research materials. Graphify reads
+from here to build the knowledge graph.
 
 ---
 
 ## Before Starting
 
-Read `references/program.md` to load the research objectives and constraints. This file is user-configurable. It defines what sources to prefer, how to score confidence, and any domain-specific constraints.
+Read `references/program.md` to load the research objectives and constraints.
+This file defines max rounds, source preferences, confidence thresholds, and
+domain-specific constraints.
+
+### Pre-Flight: Graphify Readiness Check (MANDATORY)
+
+**Do NOT start research until these pass.** A graph that can't be built is wasted work.
+
+```bash
+# 1. Verify graphify CLI is installed
+which graphify || { echo "FATAL: graphify not found. Install: npm install -g graphifyy"; exit 1; }
+
+# 2. Check that graphify's Python has the openai package (required for semantic extraction)
+GRAPHIFY_PYTHON=$(head -1 $(which graphify) | sed 's/^#!//')
+"$GRAPHIFY_PYTHON" -c "import openai; print('openai OK')" 2>/dev/null || {
+  echo "FATAL: openai package missing from graphify's Python ($GRAPHIFY_PYTHON)"
+  echo "Fix: uv pip install openai --python $GRAPHIFY_PYTHON"
+  exit 1
+}
+
+# 3. Verify a valid LLM API key is set (any backend)
+for KEY in OPENAI_API_KEY ANTHROPIC_API_KEY GEMINI_API_KEY; do
+  if [ -n "${!KEY}" ]; then
+    echo "Found: $KEY"
+    break
+  fi
+done
+if [ -z "$OPENAI_API_KEY" ] && [ -z "$ANTHROPIC_API_KEY" ] && [ -z "$GEMINI_API_KEY" ]; then
+  echo "FATAL: No LLM API key set. graphify extract needs one for semantic extraction."
+  echo "Set one of: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY"
+  echo "Or install Ollama locally and use: graphify extract ./raw --out . --backend ollama"
+  exit 1
+fi
+
+# 4. Optionally validate the API key with a quick test call
+# Skip this if you trust the key — adds ~2s latency.
+
+# 5. Record baseline graph state for post-extraction verification
+BASELINE_NODES=$(python3 -c "
+import json, os
+graph_path = 'graphify-out/graph.json'
+if os.path.exists(graph_path):
+    with open(graph_path) as f:
+        g = json.load(f)
+    print(len(g.get('nodes',[])))
+else:
+    print(0)
+" 2>/dev/null || echo 0)
+echo "Baseline graph nodes: $BASELINE_NODES"
+```
+
+**If any check fails, report the exact error and fix command to the user. Do NOT proceed.**
 
 ---
 
@@ -43,7 +99,8 @@ Two paths to a topic. Follow the first one that applies.
 
 ### A. Explicit topic (always respected)
 
-The topic is whatever the user said after the trigger phrase. Extract it by removing the trigger from the user's message.
+The topic is whatever the user said after the trigger phrase. Extract it by
+removing the trigger from the user's message.
 
 **How to extract:**
 - User: `/autoresearch kubernetes operators` → topic: `kubernetes operators`
@@ -54,12 +111,14 @@ The topic is whatever the user said after the trigger phrase. Extract it by remo
 - User: `find everything about eBPF` → topic: `eBPF`
 - User: `build a wiki on Zig allocators` → topic: `Zig allocators`
 
-**Rule**: strip the trigger phrase (`/autoresearch`, `/wiki-autoresearch`, `research`, `deep dive into`, `investigate`, `find everything about`, `go research`, `research and file`, `build a wiki on`), trim whitespace, and treat everything remaining as the topic. Use it verbatim.
+**Rule**: strip the trigger phrase, trim whitespace, treat everything remaining
+as the topic. Use it verbatim.
 
 If the topic is present, skip to the Research Loop. Do NOT ask "what topic."
 
 ### B. No topic given (fallback)
-When the user invokes `/autoresearch`, `/wiki-autoresearch`, `go research`, or `research and file` with NO topic after the trigger, ask: "What topic should I research?"
+When the user invokes the trigger with NO topic after it, ask:
+"What topic should I research?"
 
 ---
 
@@ -72,130 +131,184 @@ Round 1. Broad search
 1. Decompose topic into 3-5 distinct search angles
 2. For each angle: run 2-3 WebSearch queries
 3. For top 2-3 results per angle: WebFetch the page
-4. Extract from each: key claims, entities, concepts, open questions
+4. Save each fetched page to ./raw/ as a markdown file
+5. Extract from each: key claims, entities, concepts, open questions
 
 Round 2. Gap fill
-5. Identify what's missing or contradicted from Round 1
-6. Run targeted searches for each gap (max 5 queries)
-7. Fetch top results for each gap
+6. Identify what's missing or contradicted from Round 1
+7. Run targeted searches for each gap (max 5 queries)
+8. Fetch top results for each gap, save to ./raw/
+9. Run `graphify extract ./raw --out .` to incorporate new sources
+   (NOTE: `graphify update` only works for code files. Research sources are docs
+   and need `graphify extract` for semantic extraction.)
 
 Round 3. Synthesis check (optional, if gaps remain)
-8. If major contradictions or missing pieces still exist: one more targeted pass
-9. Otherwise: proceed to filing
+10. If major contradictions or missing pieces still exist: one more targeted pass
+11. Otherwise: proceed to graph building and synthesis
 
 Max rounds: 3 (as set in program.md). Stop when depth is reached or max rounds hit.
 ```
 
 ---
 
-## Filing Results
-
-After research is complete, create these pages:
+## Building the Knowledge Graph
 
-**wiki/sources/**. One page per major reference found
-- Use source frontmatter (type, source_type, author, date_published, url, confidence, key_claims)
-- Body: summary of the source, what it contributes to the topic
+After all sources are saved to `./raw/`:
 
-**wiki/concepts/**. One page per significant concept extracted
-- Only create a page if the concept is substantive enough to stand alone
-- Check the index first: update existing concept pages rather than creating duplicates
+```bash
+# Build the knowledge graph from all research sources
+# --out . writes graphify-out/ in current dir (same as main project graph)
+# --backend auto-detects from available API keys; use --backend ollama if no cloud key
+graphify extract ./raw --out .
+```
 
-**wiki/entities/**. One page per significant person, org, or product identified
-- Check the index first: update existing entity pages
+### Post-Extraction Verification (MANDATORY)
 
-**wiki/questions/**. One synthesis page titled "Research: [Topic]"
-- This is the master synthesis. Everything comes together here.
-- Sections: Overview, Key Findings, Entities, Concepts, Contradictions, Open Questions, Sources
-- Full frontmatter with related links to all pages created in this session
+**Do NOT skip this. Do NOT delete ./raw/ until this passes.**
 
----
+```bash
+# 1. Verify graph.json exists and contains research-topic nodes
+python3 << 'VERIFY'
+import json, os
+
+GRAPH_PATH = 'graphify-out/graph.json'
+if not os.path.exists(GRAPH_PATH):
+    print("FATAL: graph.json not created. graphify extract failed.")
+    exit(1)
+
+with open(GRAPH_PATH) as f:
+    g = json.load(f)
+
+nodes = g.get('nodes', [])
+# graphify uses 'links' key, not 'edges' — check both
+links = g.get('links', g.get('edges', []))
+
+print(f"Graph: {len(nodes)} nodes, {len(links)} links, {len(g.get('communities',[]))} communities")
+
+# 2. Check that new sources were actually ingested
+# Count nodes whose source_file is in ./raw/
+raw_sources = set()
+for n in nodes:
+    src = n.get('source_file', '')
+    if src.startswith('raw/'):
+        raw_sources.add(src)
+
+print(f"Nodes from ./raw/ sources: {len(raw_sources)} (from {len(raw_sources)} unique files)")
+
+# 3. FATAL if no new nodes were added
+if len(raw_sources) == 0:
+    print("FATAL: Zero nodes from ./raw/ sources. graphify did not ingest research content.")
+    print("Check: LLM API key valid? openai package installed? Files in ./raw/ have content?")
+    print("DO NOT delete ./raw/ — the extraction failed.")
+    exit(1)
+
+# 4. Compare against baseline (BASELINE_NODES env var from pre-flight)
+baseline = int(os.environ.get('BASELINE_NODES', '0'))
+if len(nodes) <= baseline and baseline > 0:
+    print(f"WARNING: Node count ({len(nodes)}) <= baseline ({baseline}). Graph may not have grown.")
+    print("Check: Did graphify run incrementally? Did the old graph merge properly?")
+else:
+    print(f"Node delta: +{len(nodes) - baseline} (baseline was {baseline})")
+
+print("VERIFICATION PASSED — safe to clean up ./raw/")
+VERIFY
+```
 
-## Synthesis Page Structure
+**If verification fails:** Report the exact error. Do NOT delete ./raw/. The user needs
+to fix the graphify environment (API key, openai package, Ollama install) before retrying.
 
-```markdown
----
-type: synthesis
-title: "Research: [Topic]"
-created: YYYY-MM-DD
-updated: YYYY-MM-DD
-tags:
-  - research
-  - [topic-tag]
-status: developing
-related:
-  - "[[Every page created in this session]]"
-sources:
-  - "[[wiki/sources/Source 1]]"
-  - "[[wiki/sources/Source 2]]"
 ---
 
-# Research: [Topic]
+## Cleanup
 
-## Overview
-[2-3 sentence summary of what was found]
+**GUARDED: Only run after Post-Extraction Verification passes.**
 
-## Key Findings
-- Finding 1 (Source: [[Source Page]])
-- Finding 2 (Source: [[Source Page]])
-- ...
+After the graph is built AND verified to contain new research nodes, remove the
+raw source directory. Graphify has already copied everything it needs into
+`graphify-out/`. The `./raw/` folder is transient.
 
-## Key Entities
-- [[Entity Name]]: role/significance
+```bash
+# Guard: only delete if graph was verified
+if python3 -c "
+import json, os
+with open('graphify-out/graph.json') as f:
+    g = json.load(f)
+raw_count = sum(1 for n in g.get('nodes',[]) if n.get('source_file','').startswith('raw/'))
+assert raw_count > 0, 'No raw/ nodes in graph — do not delete'
+print(f'{raw_count} raw-sourced nodes confirmed')
+"; then
+  rm -rf ./raw
+  echo "Cleanup: ./raw/ removed (graph has ingested all data)"
+else
+  echo "SKIPPED cleanup: graph verification failed. ./raw/ preserved for debugging."
+fi
+```
 
-## Key Concepts
-- [[Concept Name]]: one-line definition
+Do this before reporting to the user.
+
+---
 
-## Contradictions
-- [[Source A]] says X. [[Source B]] says Y. [Brief note on which is more credible and why]
+## Synthesis via Graph
 
-## Open Questions
-- [Question that research didn't fully answer]
-- [Gap that needs more sources]
+After the graph is built, query it directly for insights:
 
-## Sources
-- [[Source 1]]: author, date
-- [[Source 2]]: author, date
+```bash
+# Find the god nodes (core concepts)
+graphify explain "$(python3 -c "
+import json
+with open('graphify-out/graph.json') as f:
+    g = json.load(f)
+nodes = sorted(g['nodes'], key=lambda n: n.get('degree', 0), reverse=True)
+print(nodes[0]['label'] if nodes else 'root')
+")"
+
+# Find surprising connections
+graphify query "what are the most surprising cross-domain connections?"
+
+# Check specific relationships
+graphify path "ConceptA" "ConceptB"
 ```
 
+No separate report file is created. The graph IS the research output.
+
 ---
 
-## After Filing
+## After Building
+
+No wiki index, log, hot cache, or report file to update. The graph IS the index.
+The GRAPH_REPORT.md IS the hot cache. Everything is queryable.
 
-1. Update `wiki/index.md`. Add all new pages to the right sections
-2. Append to `wiki/log.md` (at the TOP):
-   ```
-   ## [YYYY-MM-DD] wiki-autoresearch | [Topic]
-   - Rounds: N
-   - Sources found: N
-   - Pages created: [[Page 1]], [[Page 2]], ...
-   - Synthesis: [[Research: Topic]]
-   - Key finding: [one sentence]
-   ```
-3. Update `wiki/hot.md` with the research summary
+The user can:
+- Open `graphify-out/graph.html` for interactive exploration
+- Run `graphify query "question"` for natural language answers
+- Run `graphify path "A" "B"` to trace connections
+- Browse `graphify-out/GRAPH_REPORT.md` for structured analysis
 
 ---
 
 ## Report to User
 
-After filing everything:
+After everything is complete:
 
 ```
 wiki-autoresearch complete: [Topic]
 
-Rounds: N | Searches: N | Pages created: N
+Rounds: N | Sources: N | Graph: N nodes, N links
 
-Created:
-  wiki/questions/Research: [Topic].md (synthesis)
-  wiki/sources/[Source 1].md
-  wiki/concepts/[Concept 1].md
-  wiki/entities/[Entity 1].md
+Output:
+  graphify-out/graph.html         (interactive graph)
+  graphify-out/GRAPH_REPORT.md    (auto-generated analysis)
 
-Key findings:
-- [Finding 1]
-- [Finding 2]
-- [Finding 3]
+God nodes: [top 3-5 concepts]
+Surprising connections: [1-2 highlights]
 
-Open questions filed: N
+Try: graphify query "your question about [topic]"
+```
+
+**IMPORTANT:** graphify uses `links` key (not `edges`) in graph.json. When
+reporting graph size, use the correct key:
+```bash
+python3 -c "import json; g=json.load(open('graphify-out/graph.json')); print(f'{len(g[\"nodes\"])} nodes, {len(g[\"links\"])} links')"
 ```
 
 ---
@@ -204,8 +317,23 @@ Open questions filed: N
 
 Follow the limits in `references/program.md`:
 - Max rounds (default: 3)
+- Max sources per round (default: 8)
 - Max pages per session (default: 15)
 - Confidence scoring rules
 - Source preference rules
 
-If a constraint conflicts with completeness, respect the constraint and note what was left out in the Open Questions section.
+If a constraint conflicts with completeness, respect the constraint and note
+what was left out so the user can query the graph about gaps.
+
+## Failure Modes (Anti-Patterns)
+
+Common failures and how to prevent them:
+
+| Failure | Cause | Prevention |
+|---------|-------|------------|
+| Graph has 0 nodes after extraction | openai package not installed in graphify's Python | Pre-flight check #2 catches this |
+| Graph has old nodes only (no new content) | Invalid LLM API key → semantic extraction failed, but code nodes remained | Pre-flight check #3 catches this |
+| `./raw/` deleted before graph verified | Cleanup ran without post-extraction verification | Guarded cleanup with assertion prevents this |
+| Graph reported as "N edges" but `links` key is correct | graph.json uses `links` not `edges` | Report section uses `links` key explicitly |
+| Incremental extraction misses new files | graphify tracks file paths; deleting and recreating ./raw/ confuses it | Pre-flight records baseline; post-extraction checks node delta |
+| Research content read but not in graph | Semantic extraction failed silently | Post-extraction verification counts raw/-sourced nodes |

package/.agents/skills/wiki-autoresearch/references/program.md

@@ -1,75 +1,41 @@
-# Research Program
-
-This file configures the wiki-autoresearch loop. Edit it to match your domain and research style. The wiki-autoresearch skill reads it before every run.
-
----
-
-## Search Objectives
-
-Default objectives for every research session:
-
-- Find authoritative sources (prefer: .edu, peer-reviewed papers, official documentation, primary sources, established publications)
-- Extract key entities (people, organizations, products, tools)
-- Extract key concepts and frameworks
-- Note contradictions between sources
-- Identify open questions and research gaps
-- Prefer sources from the last 2 years unless the topic is foundational
-
----
-
-## Confidence Scoring
-
-Label every claim with confidence when filing:
-
-- **high**: multiple independent authoritative sources agree
-- **medium**: single good source, or sources partially agree
-- **low**: speculation, opinion pieces, single informal source, or claim not verified
-
-Always note the source date for factual claims. Mark claims from sources older than 3 years as potentially stale.
-
 ---
-
-## Loop Constraints
-
-- Max search rounds per topic: **3**
-- Max wiki pages created per session: **15**
-- Max sources fetched per round: **5**
-- If max pages is reached before the loop completes: file what you have, note what was skipped in Open Questions
-
+type: config
+title: "Autoresearch Program Configuration"
+updated: 2026-05-07
 ---
 
-## Output Style
+# Research Program Configuration
 
-- Declarative, present tense
-- Cite every non-obvious claim: `(Source: [[Page]])`
-- Short pages: under 200 lines. Split if longer.
-- No hedging language ("it seems", "perhaps", "might be")
-- Flag uncertainty explicitly: `> [!gap] This claim needs verification.`
+Edit this file to configure the autoresearch loop's behavior.
 
----
+## Objectives
 
-## Domain Notes
+- Max research rounds: 3
+- Max sources per round: 8
+- Max pages per session: 15
+- Confidence threshold for filing: 0.6
 
-[Add domain-specific instructions here. Examples:]
+## Source Preferences
 
-For AI/tech research:
-- Prefer: arXiv, official GitHub repos, official product documentation, Hacker News discussions with high karma
-- Note: LLM benchmarks are often gamed: treat leaderboard claims as low confidence unless independently verified
+Prefer (in order):
+1. Official documentation & primary sources
+2. Peer-reviewed papers (arxiv, ACM, etc.)
+3. Engineering blogs from recognized teams
+4. GitHub repositories with >100 stars
+5. StackOverflow answers with >10 upvotes
 
-For business/market research:
-- Prefer: company filings, Crunchbase, Bloomberg, verified industry reports
-- Flag: press releases as low confidence without independent verification
+Avoid:
+- AI-generated content farms
+- Medium articles without author credentials
+- Repackaged/aggregator sites
+- Outdated mirrors
 
-For medical/health research:
-- Prefer: PubMed, Cochrane reviews, peer-reviewed clinical trials
-- Always note: sample size, study type (RCT vs observational), and recency
+## Domain Constraints
 
----
+None configured. Add domain-specific constraints below.
 
-## Exclusions
+## Output Format
 
-Do not cite as high-confidence sources:
-- Reddit posts or forums (use as pointers to primary sources only)
-- Social media posts
-- Undated web pages
-- Sources that don't cite their own claims
+Results are filed into `graphify-out/` via the knowledge graph.
+No separate synthesis report — the graph IS the research output.
+Raw sources are stored in `./raw/` for graphify ingestion.

package/.agents/skills/wiki-autoresearch/references/quality-sites.md

@@ -0,0 +1,32 @@
+---
+type: reference
+title: "Quality Sites Reference"
+updated: 2026-05-07
+---
+
+# Quality Sites for Web Research
+
+## Tier 1 — Always Trust (Prefer These)
+- StackOverflow (answers with >10 upvotes)
+- GitHub issues/discussions (from official repos)
+- Engineering blogs (Google, Meta, Netflix, Stripe, etc.)
+- Arxiv papers (cs.* categories)
+- Official documentation sites
+
+## Tier 2 — Use With Caution
+- Dev.to articles (check author credibility)
+- Medium (only authors with verified credentials)
+- Reddit r/programming, r/MachineLearning (highly-voted posts)
+- YouTube talks from recognized conferences
+
+## Tier 3 — Avoid
+- AI-generated content farms
+- "Tutorial" sites that repackage docs
+- Outdated mirrors (>2 years without updates)
+- Sites behind paywalls (can't verify claims)
+- Personal blogs without author background
+
+## Domain-Specific Quality Sites
+
+Add domain-specific trusted sources below:
+- [None configured yet]

package/.env.example

@@ -8,4 +8,8 @@ FIRECRAWL_API_URL="http://localhost:3002"
 VAULT_WIKI_PATH="vault/wiki"
 CURSOR_API_KEY=""
 PI_VCC_CONFIG_PATH=".pi/pi-vcc-config.json"
-# VCC config: points to project-level pi-vcc config (overrideDefaultCompaction, debug)
+# VCC config: points to project-level pi-vcc config (overrideDefaultCompaction, debug)
+
+# graphify headless extract
+OPENAI_API_KEY="sk-your_openai_api_key"
+OPENAI_API_BASE="https://opencode.ai/zen/go/v1"

package/.gitattributes

@@ -0,0 +1 @@
+data/books/**/*.pdf filter=lfs diff=lfs merge=lfs -text

package/.github/workflows/publish-github-packages.yml

@@ -30,6 +30,6 @@ jobs:
           npm pkg set publishConfig.registry='https://npm.pkg.github.com'
 
       - name: Publish package to GitHub Packages
-        run: npm publish
+        run: npm publish --ignore-scripts
         env:
           NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}