flonat-research 0.1.0

package/skills/literature/references/agent-templates.md

@@ -0,0 +1,393 @@
+# Literature Skill: Agent Templates
+
+> Sub-agent prompt templates used in Phases 2, 4, and 5 of the `/literature` skill. Read this when dispatching agents.
+
+---
+
+## Phase 2: Search Agent Templates
+
+### Agent 1: Google Scholar Search
+
+```
+subagent_type: Explore
+prompt: |
+  Search Google Scholar for academic papers on: [TOPIC]
+
+  Focus on: [JOURNALS/FIELDS if specified]
+  Time period: [YEARS if specified]
+
+  Search strategy:
+  1. Start with broad terms capturing the core concept
+  2. Narrow progressively — add specificity based on results
+  3. Try 3-5 different query variations
+
+  Prioritize:
+  - Recent work (past 3 years) first
+  - Highly-cited foundational papers regardless of age
+  - Peer-reviewed over preprints
+
+  Skip these already-known papers: [LIST OF EXISTING CITATION KEYS]
+
+  Return a structured list with for each paper:
+  - Title, authors, year, journal/venue
+  - DOI or URL if found
+  - Brief note on relevance (1 sentence)
+
+  Target: [N] papers. Cast a wide net — duplicates will be removed later.
+```
+
+### Agent 2: Semantic Scholar / arXiv Search
+
+```
+subagent_type: Explore
+prompt: |
+  Search Semantic Scholar and arXiv for academic papers on: [TOPIC]
+
+  [Same structure as Agent 1, but targeting these specific sources]
+  Use WebSearch with site:semanticscholar.org and site:arxiv.org queries.
+
+  Skip these already-known papers: [LIST OF EXISTING CITATION KEYS]
+
+  Return the same structured list format as above.
+  Target: [N] papers.
+```
+
+### Agent 2 (recommended): Cross-Source Search via Pre-Fetched Biblio Data
+
+**This is the primary structured search agent.** The orchestrator (main context) calls `scholarly_search` via the bibliography MCP before spawning this agent, and writes results to a temp file.
+
+**MCP tools are NOT available in sub-agents.** The orchestrator must pre-fetch search results.
+
+**Orchestrator pre-fetch step (run in main context before spawning Agent 2):**
+1. Call `scholarly_search` with query, year_from, year_to, sort_by, limit: 50
+2. If sub-themes exist, run additional `scholarly_search` calls
+3. Optionally call `scholarly_similar_works` for related papers
+4. Write all results to `/tmp/lit-search/bibliography-results.json`
+
+```
+subagent_type: Explore
+prompt: |
+  Read pre-fetched bibliography search results from:
+    /tmp/lit-search/bibliography-results.json
+
+  This file contains structured results from OpenAlex, Scopus, and WoS
+  for papers on: [TOPIC]
+
+  Your tasks:
+  1. Parse the results and extract: title, authors, year, journal, DOI, citation count
+  2. Supplement with WebSearch for papers that may be missing from bibliometric databases
+     (very recent papers, working papers, interdisciplinary work)
+  3. Search Semantic Scholar (site:semanticscholar.org) for additional coverage
+
+  Skip these already-known papers: [LIST OF EXISTING CITATION KEYS]
+
+  Return for each paper: title, authors, year, journal, DOI, citation count, sources found in.
+  Target: [N] papers. Cast a wide net — duplicates will be removed later.
+```
+
+### Agent 3 (optional): Semantic Scholar / arXiv or Domain-Specific
+
+Use when the topic has strong CS/ML overlap (Semantic Scholar) or needs working paper coverage (SSRN, NBER):
+
+```
+subagent_type: Explore
+prompt: |
+  Search Semantic Scholar and arXiv for academic papers on: [TOPIC]
+
+  [Same structure as Agent 1, but targeting these specific sources]
+  Use WebSearch with site:semanticscholar.org and site:arxiv.org queries.
+
+  Skip these already-known papers: [LIST OF EXISTING CITATION KEYS]
+
+  Return the same structured list format as above.
+  Target: [N] papers.
+```
+
+---
+
+## Phase 4: Verification
+
+### Step 1: Batch DOI Pre-Verification via MCP (Direct — No Sub-Agent)
+
+Before spawning verification agents, collect all DOIs from Phase 3 candidates and call the bibliography MCP tool directly:
+
+```
+Call `scholarly_verify_dois` with:
+  dois: ["10.1016/j.ejor.2024.01.001", "10.1287/mnsc.2022.4321", ...]
+
+Results: each DOI gets a status AND a resolved title:
+  - VERIFIED (2+ sources confirm) → CHECK TITLE MATCH before accepting
+  - SINGLE_SOURCE (1 source only) → still needs manual check
+  - NOT_FOUND → needs full manual verification or may be fabricated
+```
+
+**CRITICAL — Title-matching gate:** For every VERIFIED DOI, compare the returned title against the expected title. If they don't match, the DOI is WRONG even though it resolves. This is the most common error pattern — lookup tools return DOIs that are structurally valid (correct journal prefix, plausible suffix) but point to a different paper in the same journal. Example: `10.1111/j.1468-0297.2010.02387.x` vs `02366.x` — both resolve, but to different papers.
+
+Papers with title mismatches go to Step 2. Papers without DOIs always go to manual verification.
+
+### Step 2: Manual Verification for Remaining Papers
+
+Spawn multiple agents in parallel, each verifying a batch of ~5 papers:
+
+```
+subagent_type: general-purpose
+prompt: |
+  Verify that each of the following academic papers exists and has correct metadata.
+  This is for a bibliography — accuracy is critical. Do NOT guess or fabricate details.
+
+  Papers to verify:
+  1. [Title] by [Authors] ([Year]) — [Journal/venue if known]
+  2. [Title] by [Authors] ([Year]) — [Journal/venue if known]
+  3. ...
+  4. ...
+  5. ...
+
+  For EACH paper:
+  1. Search the web to confirm the paper exists
+  2. Verify: authors (ALL authors — full list, exact names and order), title, year, journal, volume, pages
+  3. **Find the correct DOI using these methods IN ORDER of reliability:**
+     a. **Crossref API (most reliable — ALWAYS try this first):** Run:
+        `curl -sL "https://api.crossref.org/works?query.bibliographic=TITLE+AUTHOR&rows=3"`
+        Parse the JSON response — the first result's `DOI` field is the correct DOI.
+        This uses publisher metadata and is far more reliable than web search or MCP tools.
+        **Do NOT use MCP `scholarly_search` for specific known papers** — it returns
+        noisy, irrelevant results. Crossref is the gold standard for targeted lookups.
+     b. **Publisher page:** Visit the journal's website and search for the paper.
+     c. **Web search (last resort):** Search for the paper + "DOI". But DOIs from
+        web search MUST still be verified in step 4.
+  4. **Resolve the DOI** by visiting https://doi.org/[DOI] — confirm the landing
+     page shows the SAME title and authors. If the DOI resolves to a different
+     paper, the metadata is WRONG. Go back to step 3.
+     **NEVER reconstruct DOIs from memory or by guessing suffixes** — this is the
+     most common hallucination pattern (correct journal prefix, wrong suffix).
+  5. Find the abstract
+  6. Note the URL where you found/confirmed it (publisher page preferred)
+  7. **Preprint check:** If the paper was found on arXiv, SSRN, NBER, or any
+     working paper series, search for a published journal or conference version.
+     Check Google Scholar, the DOI, and the author's publication page.
+     Use Crossref API or WebSearch — do NOT call MCP tools (scholarly_search,
+     scholarly_verify_dois, etc.) as MCP tools are not available in sub-agents.
+     - If a published version exists: use that version's metadata instead
+       (journal, year, volume, pages, DOI)
+     - If no published version exists: keep the preprint, but note it as
+       a working paper in the venue field
+
+  **Author accuracy rules:**
+  - List ALL authors — never use "and others" or "et al." in the metadata.
+    Every author must be named.
+  - Beware of author conflation: researchers in the same subfield may share
+    surnames or topics. Confirm authors from the publisher page, not from
+    secondary aggregators.
+  - If you find the paper attributed to different authors on different sources,
+    trust the publisher/DOI landing page over Google Scholar snippets.
+  - **Verify author order and initials** against the publisher page. Common
+    errors: swapped first/last names, wrong middle initials, missing co-authors,
+    or extra authors from a different paper with a similar title.
+
+  **DOI verification is mandatory.** If the DOI does not resolve to the claimed
+  paper, the entry FAILS verification — do not mark it as VERIFIED.
+  Use Crossref API for DOI lookup — never guess or reconstruct DOIs.
+
+  Return for each paper one of:
+  - VERIFIED: [full corrected metadata including DOI and abstract]
+    — include the URL of the publisher page where you confirmed the metadata
+    — if upgraded from preprint, add: "UPGRADED from [preprint source]"
+  - NOT FOUND: [title] — could not confirm existence, do not include
+  - METADATA MISMATCH: [title] — paper exists but DOI/authors don't match
+    search results. Include what you found and flag the discrepancy.
+
+  Be strict. If you cannot find a paper or its details don't match, mark it NOT FOUND
+  or METADATA MISMATCH. Never guess metadata.
+```
+
+---
+
+## Phase 5: PDF Download Agent Template
+
+```
+subagent_type: Bash
+prompt: |
+  Download PDFs for these academic papers. Save each to [PROJECT]/docs/readings/
+  with filename matching the citation key.
+
+  Papers:
+  1. Key: Smith2024 — DOI: 10.1000/example — URL: https://...
+  2. Key: Jones2023 — DOI: 10.1000/example2 — URL: https://...
+  3. ...
+
+  For each paper, try in order:
+  1. Direct PDF link if known
+  2. DOI redirect (https://doi.org/[DOI])
+  3. Search for open access version on author website, SSRN, arXiv, NBER
+
+  Use curl or wget. Create the output directory if needed.
+  Report which downloads succeeded and which failed.
+```
+
+---
+
+## Prompt Templates (User-Facing)
+
+### Find specific references
+```
+Find 5 recent papers on [TOPIC] from [JOURNALS].
+Verify each citation exists and provide BibTeX entries.
+```
+
+### Build comprehensive literature
+```
+Help me synthesize the literature on [TOPIC].
+
+I need:
+1. Top 25 seminal papers (high citations, foundational)
+2. Key themes/debates in this literature
+3. Narrative "story" of how this field developed
+4. Gaps or opportunities for new research
+5. A .bib file with all references
+
+Focus on: [JOURNALS/FIELDS]
+Time period: [YEARS]
+```
+
+### Synthesis prompts
+```
+Organize these papers into themes and write a 2-paragraph summary of each theme.
+```
+
+```
+Based on this literature, what research questions remain unanswered?
+```
+
+---
+
+## Phase 2b: CLI Council Search Template
+
+**When to use:** Broad reviews (20+ papers), interdisciplinary topics, or when Phase 2 coverage seems thin. Each model (Gemini, Codex, Claude) has different training data and recall — and Gemini has live web search.
+
+**Step 1:** Write the search prompt to a temp file:
+
+```bash
+cat > /tmp/lit-council-prompt.txt << 'PROMPT'
+Search for academic papers on: [TOPIC]
+
+Focus: [JOURNALS/FIELDS if specified]
+Time period: [YEARS if specified]
+
+For each paper, provide:
+- Full title
+- ALL authors (full names, never "et al.")
+- Year of publication
+- Journal or venue name
+- DOI if known
+- One sentence on relevance
+
+Prioritize:
+- Highly-cited foundational papers regardless of age
+- Recent work (past 3-5 years)
+- Peer-reviewed over preprints
+- Papers from top journals in the field
+
+Target: [N] papers. Cast a wide net — duplicates will be removed later.
+
+Skip these already-known papers: [LIST OF EXISTING CITATION KEYS]
+PROMPT
+```
+
+**Step 2:** Run the council:
+
+```bash
+cd "$TM/packages/cli-council"
+uv run python -m cli_council \
+    --prompt-file /tmp/lit-council-prompt.txt \
+    --output /tmp/lit-council-result.json \
+    --output-md /tmp/lit-council-report.md \
+    --timeout 180
+```
+
+**Step 3:** Parse results. Read `/tmp/lit-council-report.md` and extract paper lists from:
+- The synthesis (chairman's merged list)
+- Each individual assessment (Assessment A/B/C) — these often contain papers the others missed
+
+Feed all discovered papers into Phase 3 (deduplication) alongside Phase 2 results.
+
+**Expected value:** Gemini finds more recent papers via web search. Codex and Claude recall different foundational works from training. Typical yield: 10-20% more unique papers vs. Phase 2 alone.
+
+---
+
+## Phase 7: CLI Council Synthesis Template (Optional)
+
+For comprehensive reviews, run the narrative synthesis through cli-council to get three independent thematic interpretations.
+
+**Step 1:** Write the paper list to a context file:
+
+```bash
+# Collect all verified paper metadata (titles, abstracts, years, journals)
+# into a single file for context
+cat > /tmp/lit-papers.txt << 'EOF'
+[Paste the verified paper list with titles, authors, years, journals, and abstracts]
+EOF
+```
+
+**Step 2:** Write the synthesis prompt:
+
+```bash
+cat > /tmp/lit-synthesis-prompt.txt << 'PROMPT'
+Synthesise the following academic literature into a structured narrative review.
+
+1. Identify major themes — group papers by approach, finding, or intellectual tradition
+2. Map the intellectual lineage — how did thinking in this area evolve?
+3. Note current debates — where do researchers disagree?
+4. Identify gaps — what questions remain unanswered?
+
+Structure your synthesis as numbered themes, each with:
+- Theme title
+- 2-3 paragraph narrative
+- Key papers in that theme (cite by author and year)
+
+End with a "Gaps and Opportunities" section identifying 3-5 directions for future research.
+PROMPT
+```
+
+**Step 3:** Run the council:
+
+```bash
+cd "$TM/packages/cli-council"
+uv run python -m cli_council \
+    --prompt-file /tmp/lit-synthesis-prompt.txt \
+    --context-file /tmp/lit-papers.txt \
+    --output-md /tmp/lit-synthesis-report.md \
+    --chairman claude \
+    --timeout 180
+```
+
+**Expected value:** Different models identify different thematic groupings and gaps. The chairman synthesis produces a richer narrative than any single model.
+
+---
+
+## Scaling Guide
+
+| Request size | Search agents | CLI Council (Phase 2b) | DOI pre-verify (MCP) | Verification waves | PDF waves |
+|---|---|---|---|---|---|
+| 5 papers | 1 (MCP cross-source) | Skip | 1 call | 1 wave (1 agent) | 1 wave |
+| 10-15 papers | 2 (Scholar + MCP) | Skip | 1 call | 1 wave (2-3 agents) | 1 wave |
+| 20-25 papers | 2-3 | Recommended | 1 call | 1-2 waves | 2 waves |
+| 30+ papers | 3 | Recommended | 1 call (50 DOI limit) | 2+ waves | 3+ waves |
+
+For small requests (< 5 papers), skip sub-agents entirely — call `scholarly_search` and `scholarly_verify_dois` directly, then do verification inline.
+
+**Between waves:** write collected results to the `.bib` file or a scratch markdown file on disk, then proceed to the next wave. This prevents context overflow.
+
+---
+
+## Example Use
+
+"Create a literature review on interactive approaches to multi-criteria decision making, focusing on papers from top journals in the field from 2015-2025. I need about 25 key papers with a .bib file and a narrative summary."
+
+This would trigger:
+1. Pre-search check for existing .bib
+2. 3 parallel search agents (Scholar, Semantic Scholar, domain journals)
+3. Deduplicate ~40 candidates down to ~30
+4. 2 waves of 3 verification agents (5 papers each), results written to disk between waves
+5. 2 waves of 3 PDF download agents
+6. Assemble verified .bib (~25 papers)
+7. Write thematic narrative summary

package/skills/literature/references/bibliometric-apis.md

@@ -0,0 +1,44 @@
+# Bibliometric API Structured Queries
+
+> Four bibliometric sources are available. The **bibliography MCP server** (`packages/mcp-bibliography/`) is the preferred interface — `scholarly_search` queries all enabled sources in one call with automatic DOI-based dedup; `scholarly_verify_dois` batch-verifies DOIs across all sources.
+
+## Bibliography MCP Tools (preferred)
+
+| Tool | What it does | When to use |
+|------|-------------|-------------|
+| `scholarly_search` | Cross-source keyword search (OpenAlex + S2 + Scopus + WoS) with dedup | Phase 2 pre-fetch |
+| `scholarly_similar_works` | ML-based recommendations (S2 Recommendations API) | Phase 2 pre-fetch — finds papers beyond keyword matches |
+| `scholarly_verify_dois` | Batch DOI verification across all sources | Phase 4 verification |
+| `scholarly_citations` | Forward citation graph (papers citing a given paper) | Phase 2.5 snowball — find follow-up work |
+| `scholarly_references` | Backward citation graph (papers referenced by a given paper) | Phase 2.5 snowball — find foundational works |
+| `scholarly_paper_detail` | Full metadata + TLDR + BibTeX + OA PDF link | Phase 3 screening, Phase 6 BibTeX assembly |
+| `scholarly_author_papers` | All papers by an author | Phase 2 pre-fetch — author-based search |
+| `scholarly_source_status` | Check which sources are active | Phase 1 |
+
+## OpenAlex (always available)
+
+**Setup:** `.scripts/openalex/openalex_client.py` + `.scripts/openalex/query_helpers.py`
+
+| Workflow | What it does |
+|----------|-------------|
+| Highly-cited papers | Top-cited papers on a topic (filtered by year) |
+| Author output | Full publication record for a researcher |
+| Institution output | Research output analysis for a university |
+| Publication trends | Year-by-year counts for a topic |
+| Open-access discovery | Find freely downloadable versions |
+| Citation network | Forward citations for a given paper |
+| Batch DOI lookup | Verify metadata for multiple papers |
+
+**Full recipes:** [openalex-workflows.md](openalex-workflows.md) | **API guide:** [openalex-api-guide.md](openalex-api-guide.md)
+
+## Scopus (requires `SCOPUS_API_KEY` + `SCOPUS_INST_TOKEN`)
+
+Query syntax: `TITLE-ABS-KEY("quoted phrases" OR terms)`, subject areas via `SUBJAREA(CODE)`, year filters via `PUBYEAR > N` / `PUBYEAR < N`. Elsevier REST API with `X-ELS-APIKey` + `X-ELS-Insttoken` headers. Provides abstracts, author keywords, and citation counts in COMPLETE view. Pagination via `start`/`count` params (max 25 per page).
+
+**API guide:** [scopus-api-guide.md](scopus-api-guide.md)
+
+## Web of Science (requires `WOS_API_KEY`)
+
+Query syntax: `TS=(topic search)`, year filter via `PY=(YYYY-YYYY)`. Two API tiers: **Starter** (`/documents` endpoint, page-based, max 50/page) and **Expanded** (root endpoint, `firstRecord`-based, max 100/page, includes abstracts). Auth via `X-ApiKey` header. Tier set via `WOS_API_TIER` env var (default: `starter`).
+
+**API guide:** [wos-api-guide.md](wos-api-guide.md)

package/skills/literature/references/cli-council-search.md

@@ -0,0 +1,79 @@
+# CLI Council Search & arXiv Full-Text Reading
+
+> Phase 2b multi-model literature search and arXiv source reading.
+> Referenced from `SKILL.md` — the parent file has a summary + pointer.
+
+## Phase 2b: CLI Council Search (Optional)
+
+**When to use:** Broad literature reviews (20+ papers), interdisciplinary topics, or when maximum recall matters. Each model has different training data and recall — running the same search query through Gemini, Codex, and Claude surfaces papers that any single model would miss. Gemini additionally has live web search.
+
+**How it works:** Run `cli-council` from the `packages/cli-council/` package. The council sends the same search prompt to all three CLI backends in parallel, collects their independent paper lists, then synthesises a merged list.
+
+**Invocation:**
+
+```bash
+cd "packages/cli-council"
+uv run python -m cli_council \
+    --prompt-file /tmp/lit-council-prompt.txt \
+    --output /tmp/lit-council-result.json \
+    --output-md /tmp/lit-council-report.md \
+    --timeout 180
+```
+
+**Prompt template** (write to `/tmp/lit-council-prompt.txt`):
+
+```
+Search for academic papers on: [TOPIC]
+
+Focus: [JOURNALS/FIELDS if specified]
+Time period: [YEARS if specified]
+
+For each paper, provide:
+- Full title
+- ALL authors (never "et al.")
+- Year
+- Journal/venue
+- DOI if known
+
+Prioritize:
+- Highly-cited foundational papers
+- Recent work (past 3-5 years)
+- Peer-reviewed over preprints
+
+Target: [N] papers. Cast a wide net — duplicates will be removed.
+
+Skip these already-known papers: [LIST OF EXISTING CITATION KEYS]
+```
+
+**After the council returns:**
+1. Parse the synthesis and individual assessments from the JSON output
+2. Extract paper lists from each backend's response (Gemini often finds more recent papers via web search; Codex and Claude recall different foundational works)
+3. Feed ALL discovered papers into Phase 3 alongside the Phase 2 results for deduplication
+
+**When to skip:** Small requests (< 10 papers), narrow/well-defined topics, or when Phase 2 already returns sufficient coverage. The standard Phase 2 agents (Scholar + bibliography MCP) remain the primary search mechanism; Phase 2b supplements with model-diversity recall.
+
+---
+
+## Reading Full Paper Text from arXiv
+
+When you need to read the full text of a paper (not just the abstract) — to verify claims, understand methodology, or extract exact phrasing — download the arXiv LaTeX source:
+
+```bash
+# Download source tarball
+curl -L -o /tmp/ARXIV_ID.tar.gz "https://arxiv.org/src/ARXIV_ID"
+
+# Extract
+mkdir -p /tmp/ARXIV_ID && cd /tmp/ARXIV_ID && tar -xzf /tmp/ARXIV_ID.tar.gz
+
+# Find and read the main .tex file
+ls *.tex
+```
+
+This gives you:
+- Full paper text with equations, methodology details, and exact author phrasing
+- The paper's `.bib` or `.bbl` file for cross-referencing their citations
+- Supplementary materials and appendices
+
+**When to use:** Deep verification of a specific paper's claims, extracting precise definitions or theorems, understanding methodology details that abstracts omit, or finding papers cited by the target paper.
+
+**Limitation:** Only works for arXiv papers with source available. For journal-only papers, use `/split-pdf` instead.