tokens-for-good 0.3.6 → 0.3.8

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tokens-for-good",
-  "version": "0.3.6",
+  "version": "0.3.8",
   "type": "module",
   "description": "Donate your spare AI tokens to research nonprofits for Fierce Philanthropy",
   "bin": {

package/pipeline/01-research/PROMPT.md

@@ -66,6 +66,7 @@ Keep the table with all columns. For each of the 20 negative consequences, add a
 - When Yes or Partial: include SPECIFIC data (percentages, sample sizes, time periods, study names)
 - Only direct results from this organization, not from other orgs or modeling
 - **CITATION RULES (critical):** Every data point MUST have its own inline citation `[Source Name](URL)`. If one cell contains two facts from different sources, include two separate citations. Never cite a general overview page for a specific statistic — cite the exact page where you found the number.
+- **VERIFY INLINE:** After writing each cell, re-read the source you cited and confirm the exact numbers match. If the source says 75% and you wrote 59%, fix it before moving on. Do not proceed to the next row until the current row's numbers are confirmed against the cited page.
 
 #### PROMPT 5 — Counterfactual Results
 
@@ -74,7 +75,7 @@ Keep the table with ALL previous columns. For each of the 20 negative consequenc
 - Start each cell with "Yes.", "Partial.", or "No counterfactual results."
 - Describe study design (RCT, quasi-experimental, matched comparison), sample sizes, what the control/comparison group showed
 - Counterfactual = comparison to what would have happened without the intervention. Before/after alone does not count.
-- **Same citation rules as Prompt 4:** every data point gets its own inline citation to the specific page.
+- **Same citation and verify-inline rules as Prompt 4:** every data point gets its own inline citation, and confirm numbers match the source before moving to the next row.
 
 #### SUMMARY REPORT
 
@@ -134,7 +135,7 @@ These rules are critical for report quality. Poorly attributed citations are the
 
 3. **If you can't find a URL for a claim, don't include the claim.** No unsourced facts. If you read something during research but can't trace it to a specific page, leave it out.
 
-4. **Verify before citing.** After writing a claim with a citation, confirm the cited page actually contains that information. If it doesn't, find the correct source or remove the claim.
+4. **Verify numbers match the source exactly.** After writing a claim with a number (percentage, dollar amount, count), re-read the cited page and confirm the exact figure appears there. Common errors: writing 59% when the source says 75%, writing 4,000 when the source says 1,651, or writing 20% when the source says 25%. If your number doesn't match, use the source's number or remove the claim.
 
 5. **Attribution matters.** Say "X reports that" when citing an org's own claims. Say "independent evaluation found" when citing third-party evidence. The distinction is load-bearing.
 
@@ -152,7 +153,7 @@ Run these checks before submitting. They are not optional.
 
 **Citations:**
 - [ ] Every factual claim has its own inline citation
-- [ ] Spot-check at least 5 citations: visit the URL and confirm the page says what you claim
+- [ ] Spot-check at least 5 citations: visit the URL and confirm the EXACT numbers on the page match what you wrote. If the source says 132% and you wrote 136%, fix it.
 - [ ] For any citation where the page doesn't support your claim, find the correct source or remove the claim
 - [ ] No claims are cited to general overview pages when a specific report or data page exists
 

package/pipeline/04-peer-review/PROMPT.md

@@ -1,67 +1,86 @@
-# Step 4: Peer Review -- Claude Code Instructions
-
-## Inputs
-
-- **Report to review:** Provided by the `get_peer_review` MCP tool
-- **Research guidance:** The same methodology from step 1
-- **Writing style guide:** The same decontamination rules from step 3
+# Peer Review — Instructions
 
 ## Purpose
 
-You are reviewing another contributor's research report. Your job is to verify quality and catch problems before a human reviewer sees it. You are NOT the original researcher -- you are a second pair of eyes.
+You are reviewing another contributor's research report. Your job is to verify quality and catch problems before a human reviewer sees it. You are NOT the original researcher — you are a second pair of eyes.
 
 ## Instructions
 
-### 1. Read the Full Report
+### 1. Check the Automated Fact-Check Results First
+
+If automated fact-check results are included above the report, read them before diving into the report itself. Focus on:
+- **Red flags** — these are specific problems the automated system detected (unsupported claims, dead links, self-reported data issues)
+- **Fact support rate** — below 70% means many claims aren't backed by their cited sources
+- **Avg trust score** — below 50% means citations are low-quality (self-reported, blog posts, dead links)
+
+Use these results to target your spot-checks. If the automated system flagged specific unsupported claims, verify those first.
+
+### 2. Read the Full Report
 
-Read the entire report carefully. Note the org name, the scored checklist, and the overall recommendation.
+Read the entire report. Note the org name, the scored checklist, and the overall recommendation.
 
-### 2. Spot-Check Citations (3-5)
+### 3. Spot-Check Citations (3-5)
 
-Pick 3-5 citation URLs from the report. For each:
+Pick 3-5 citation URLs from the report (prioritize any flagged by the automated fact-check). For each:
 - Visit the URL using web fetch
 - Verify the page exists (not 404)
 - Check that the source says what the report claims
+- If a citation is wrong, search for the correct source. If the claim can't be sourced anywhere, remove it.
 
-### 3. Check Report Structure
+### 4. Check Report Structure
 
 Verify:
-- [ ] All 5 prompt sections present (PROMPT 1-5)
+- [ ] All 5 prompt sections present (PROMPT 1-5) with 20 rows each
 - [ ] All 7 summary sections present (Sections 1-7)
 - [ ] SOURCES section exists with citations
-- [ ] Tables in Prompts 2-5 have content
-- [ ] Scored checklist is present with score calculated correctly
+- [ ] Every factual claim has its own inline citation `[Source Name](URL)`
+- [ ] No claims cited to general overview pages when a specific report or data page exists
+
+### 5. Evaluate Scoring
+
+The scored checklist uses these weights. Verify the math and the evidence:
+
+Base score (out of 100):
+- a. Has Ultimate Outcome Goals (50 pts)
+- b. Measures Intermediate Outcomes (10 pts)
+- c. Measures Ultimate Outcomes (15 pts)
+- d. Shows Continual Learning & Adaptation (25 pts)
+
+Extra credit:
+- e. Measures Intermediate Counterfactual (10 pts)
+- f. Measures Ultimate Counterfactual (10 pts)
 
-### 4. Evaluate Scoring
+**Score: X/100** (can exceed 100 with extra credit, max 120)
 
-Compare the checklist against the evidence:
+Check:
 - Are checked items supported by evidence in the report?
 - Are unchecked items correctly unchecked (no evidence was found)?
-- Does the score math add up (checked items x weights = stated score)?
+- Does the score math add up?
 
-### 5. Look for Red Flags
+### 6. Look for Red Flags
 
 - Suspiciously specific numbers with no citation
 - Studies or evaluations that seem fabricated
 - Copy-pasted content or generic filler
 - Sections that are empty or trivially short
 - Claims that contradict other parts of the report
+- Em dashes, filler adjectives (robust, comprehensive, innovative), AI transitions
 
-### 6. Assign a Score
+### 7. Assign a Score
 
 | Score | When to use |
 |-------|------------|
-| **4 -- Great** | Report is thorough, citations check out, scoring is correct. No changes needed. |
-| **3 -- Good with fixes** | Minor issues you can fix: broken citation, wrong score math, awkward phrasing, a checklist item that should be toggled. **Fix the issues yourself** and submit the corrected report. |
-| **2 -- Needs redo** | Major problems: thin evidence across multiple sections, significant hallucinations, missing sections, fundamentally wrong scoring. Not fixable with minor edits. |
-| **1 -- Bad actor** | Garbage: copy-pasted nonsense, completely fabricated data, obvious gaming attempt. This flags the original author. Use sparingly and only when clearly warranted. |
+| **4 — Great** | Report is thorough, citations check out, scoring is correct. No changes needed. |
+| **3 — Good with fixes** | Minor issues you can fix: broken citation, wrong score math, awkward phrasing, a checklist item that should be toggled, misattributed citation. **Fix the issues yourself** and submit the corrected report. |
+| **2 — Needs redo** | Major problems: thin evidence across multiple sections, significant hallucinations, missing sections, fundamentally wrong scoring. Not fixable with minor edits. |
+| **1 — Bad actor** | Garbage: copy-pasted nonsense, completely fabricated data, obvious gaming attempt. This flags the original author. Use sparingly and only when clearly warranted. |
 
-### 7. Submit Your Review
+### 8. Submit Your Review
 
 Use `submit_peer_review` with:
-- `claim_id`: The claim ID from `get_peer_review`
+- `claim_id`: The claim ID shown above
 - `score`: Your score (1-4)
-- `notes`: Brief explanation of your score
+- `notes`: Brief explanation of your score. Mention which citations you checked and what you found.
 - `updated_report`: If score is 3, include the full fixed report
 
 ## Important Rules
@@ -71,3 +90,4 @@ Use `submit_peer_review` with:
 - Score 1 is for abuse. If you're unsure, use 2 instead.
 - If you spot-check a citation and it's broken, that alone is a 3 (fix it), not a 2.
 - Don't rewrite the report to match your style. Fix factual errors, not opinions.
+- If the automated fact-check flagged issues, verify them. If the flags are correct, fix the citations (score 3) or flag the report (score 2) depending on severity.

package/src/mcp-server.js

@@ -160,7 +160,7 @@ server.tool('get_methodology', 'Get the full research methodology, verification
 });
 
 server.tool('submit_report', 'Submit a completed research report for an org you claimed. You MUST include estimated_tokens.', {
-  claim_id: z.number().describe('The claim ID from claim_org'),
+  claim_id: z.string().describe('The claim ID from claim_org'),
   report_markdown: z.string().describe('The full research report in markdown'),
   estimated_tokens: z.number().describe('Estimated total tokens used: count web searches (~1K each), web fetches (~2-5K each), report output (~4 tokens/word), plus ~10K overhead'),
   model_used: z.string().optional().describe('The model that generated this report'),
@@ -189,9 +189,25 @@ server.tool('get_peer_review', 'Get a draft report assigned to you for peer revi
     } catch {
       peerMethodology = 'Score 1-4: 4=Great, 3=Good with fixes (submit corrected version), 2=Needs redo, 1=Bad actor.';
     }
-    const factCheckNote = result.automated_review
-      ? `\n\nAutomated Fact-Check: ${result.automated_review.status}${result.automated_review.summary ? ` — Quality: ${result.automated_review.summary.overall_quality}, Fact support: ${Math.round(result.automated_review.summary.fact_support_rate * 100)}%, Avg trust: ${Math.round(result.automated_review.summary.avg_trust_score * 100)}%` : ''}`
-      : '';
+    let factCheckNote = '';
+    if (result.automated_review?.summary) {
+      const s = result.automated_review.summary;
+      const lines = [
+        `\n\n## Automated Fact-Check Results`,
+        `Quality: ${s.overall_quality} | Fact support: ${Math.round(s.fact_support_rate * 100)}% | Avg trust: ${Math.round(s.avg_trust_score * 100)}%`,
+        `Facts checked: ${result.automated_review.facts_checked}/${result.automated_review.facts_extracted} | Citations rated: ${result.automated_review.citations_rated}`,
+      ];
+      if (s.red_flags?.length > 0) {
+        lines.push(`\nRed flags:\n${s.red_flags.map(f => `  - ${f}`).join('\n')}`);
+      }
+      if (s.strengths?.length > 0) {
+        lines.push(`\nStrengths:\n${s.strengths.map(f => `  - ${f}`).join('\n')}`);
+      }
+      lines.push(`\nUse these results to focus your spot-checks on flagged areas.`);
+      factCheckNote = lines.join('\n');
+    } else if (result.automated_review) {
+      factCheckNote = `\n\nAutomated Fact-Check: ${result.automated_review.status} (no summary available yet)`;
+    }
     return {
       content: [{ type: 'text', text: `Peer review assigned:\nOrg: ${result.org.name}\nAuthor: ${result.author}\nClaim ID: ${result.claim_id}${factCheckNote}\n\n---\n\n${peerMethodology}\n\n---\n\n${result.report_markdown}\n\n---\n\nUse submit_peer_review with your score and notes.` }],
     };