docpluck 1.5.0__tar.gz

docpluck-1.5.0/.claude/skills/docpluck-cleanup/SKILL.md

@@ -0,0 +1,70 @@
+---
+name: docpluck-cleanup
+description: Clean up Docpluck codebase. Sync CLAUDE.md/README.md/ARCHITECTURE.md against actual code, remove dead benchmark scripts, verify LESSONS.md is current, clean temp/ directory, check for stale environment variables, verify .gitignore covers sensitive files, update TODO.md progress. Use /docpluck-cleanup periodically or before releases.
+tags: [python, pdf, docx, html, fastapi, nextjs, docs, cleanup]
+---
+
+## [MANDATORY FIRST ACTION] preflight (do NOT skip, even if orchestrated by /ship)
+
+**Your very first action in this skill, BEFORE reading anything else, is:**
+
+1. Run: `bash ~/.claude/skills/_shared/bin/preflight-filter.sh <this-skill-name>` and print its `🔧 skill-optimize pre-check · ...` heartbeat as your first visible output line.
+2. Initialize `~/.claude/skills/_shared/run-meta/<this-skill-name>.json` per `~/.claude/skills/_shared/preflight.md` step 6 (include `phase_start_sha` from `git rev-parse HEAD`).
+3. Load `~/.claude/skills/_shared/quality-loop/core.md` into working memory (MUST-level rules gated by /ship).
+
+If you skip these steps, /ship will detect the missing heartbeat and FAIL this phase. Do not proceed to the skill body until preflight has run.
+
+# Docpluck Cleanup
+
+You are a codebase janitor for Docpluck. Your job is to keep documentation accurate, remove dead code, and ensure the repo is in a clean state.
+
+## Project Location
+`C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor`
+
+## Cleanup Checklist
+
+### 1. Documentation Sync
+Read the actual code and verify each doc file is accurate:
+
+- **CLAUDE.md** — Does the project structure match? Are the commands correct? Do the critical rules still apply?
+- **README.md** — Does the architecture diagram match reality? Are the tech versions correct?
+- **ARCHITECTURE.md** — Do the system diagrams match actual deployment? Are all tables listed?
+- **API.md** — Do endpoint signatures match `service/app/main.py`? Are normalization steps current with `service/app/normalize.py`?
+- **SETUP_GUIDE.md** — Are all URLs, project IDs, and domain names current?
+- **TODO.md** — Mark completed items. Remove items that are no longer relevant.
+
+### 2. Dead Code Removal
+Check `scripts/` for benchmark scripts that were intermediate steps and no longer needed:
+- `scripts/investigate_stat_loss.py` — diagnostic, can be removed
+- `scripts/investigate_stat_loss2.py` — diagnostic, can be removed
+- `scripts/trace_pipeline.py` — diagnostic, can be removed
+- `scripts/trace_pipeline2.py` — diagnostic, can be removed
+- `scripts/verify_ieee_fp.py` — diagnostic, can be removed
+- `scripts/quick_apa_verify.py` — diagnostic, can be removed
+- `scripts/diagnose_mismatches.py` — diagnostic, can be removed
+- `scripts/benchmark_docling_optimized.py` — Docling was dropped
+
+Keep: `benchmark.py`, `ground_truth_verify.py`, `optimized_extractors.py`, `pdftotext_enhanced.py`, `setup_test_pdfs.py`, `ai_verify.py`, `final_showdown.py`
+
+### 3. Temp Directory
+```bash
+rm -rf temp/
+```
+Contains rendered PNG pages from verification — not needed in repo.
+
+### 4. Environment Variable Audit
+- Check `.env.local` has no real secrets committed
+- Verify `.gitignore` includes: `.env`, `.env.local`, `.env.production`, `.vercel/`, `test-pdfs/`, `temp/`, `__pycache__/`, `node_modules/`, `.next/`
+
+### 5. Stale Dependencies
+- Check `frontend/package.json` for unused deps
+- Check `service/requirements.txt` — should only have fastapi, uvicorn, pdfplumber, python-multipart
+- Verify no pymupdf or pymupdf4llm in requirements (AGPL dropped)
+
+### 6. Memory Files
+Check `C:\Users\filin\.claude\projects\c--Users-filin-Dropbox-Vibe-MetaScienceTools-PDFextractor\memory\` for stale memories that no longer reflect current state.
+
+## Output Format
+Report what was cleaned, what was updated, and what needs manual attention.
+
+## Final step: read ~/.claude/skills/_shared/postflight.md and follow it.

docpluck-1.5.0/.claude/skills/docpluck-deploy/SKILL.md

@@ -0,0 +1,204 @@
+---
+name: docpluck-deploy
+description: Deploy Docpluck to production. Pre-flight checks (Next.js build, Python service health, git status), verify Vercel env vars (DATABASE_URL, AUTH_SECRET, AUTH_GITHUB_ID, AUTH_GOOGLE_ID, EXTRACTION_SERVICE_URL), push to GitHub for auto-deploy, verify Vercel deployment status, check Railway extraction service health, run post-deploy smoke test. Use /docpluck-deploy to deploy or verify deployment.
+tags: [docpluck, nextjs, python, vercel, railway, neon, auth, deploy]
+---
+
+## [MANDATORY FIRST ACTION] preflight (do NOT skip, even if orchestrated by /ship)
+
+**Your very first action in this skill, BEFORE reading anything else, is:**
+
+1. Run: `bash ~/.claude/skills/_shared/bin/preflight-filter.sh <this-skill-name>` and print its `🔧 skill-optimize pre-check · ...` heartbeat as your first visible output line.
+2. Initialize `~/.claude/skills/_shared/run-meta/<this-skill-name>.json` per `~/.claude/skills/_shared/preflight.md` step 6 (include `phase_start_sha` from `git rev-parse HEAD`).
+3. Load `~/.claude/skills/_shared/quality-loop/core.md` into working memory (MUST-level rules gated by /ship).
+
+If you skip these steps, /ship will detect the missing heartbeat and FAIL this phase. Do not proceed to the skill body until preflight has run.
+
+# Docpluck Deploy
+
+Deploy Docpluck to production on Vercel (frontend) and Railway (extraction service).
+
+## Two-Repo Architecture (CRITICAL — read before deploying)
+
+Docpluck is split across **two repos** under `C:\Users\filin\Dropbox\Vibe\MetaScienceTools\`:
+
+| Path | Repo | Visibility | Contains |
+|------|------|------------|----------|
+| `docpluck/` | `giladfeldman/docpluck` | **public** | The `docpluck` Python library only (extraction + normalization + quality + DOCX/HTML). Published to PyPI. |
+| `PDFextractor/` | `giladfeldman/docpluckapp` | **private** | The SaaS app only (Next.js frontend, FastAPI service `service/app/main.py`, Drizzle schema, Auth.js). **No library code duplication** — the service imports `docpluck` via a git pin in `service/requirements.txt`. |
+
+Library changes therefore reach production via TWO steps:
+1. Tag + push the library repo (this updates PyPI, but the app pins by git tag).
+2. Bump the git pin in `PDFextractor/service/requirements.txt` (`docpluck @ git+https://...@v<NEW>`) and redeploy the app.
+
+**Skipping step 2 silently keeps production on the old library.** Deploy pre-flight check 4 below catches this.
+
+## Pre-Flight Checks
+
+Run ALL checks before deploying. Any failure is a blocker.
+
+### 1. Git Status (both repos)
+```bash
+cd C:/Users/filin/Dropbox/Vibe/MetaScienceTools/docpluck && git status --short && git log --oneline -3
+echo "---"
+cd C:/Users/filin/Dropbox/Vibe/MetaScienceTools/PDFextractor && git status --short && git log --oneline -3
+```
+Both working trees must be clean. Library tagged at `v<X.Y.Z>` matching `__version__`.
+
+### 2. Frontend Build
+```bash
+cd C:/Users/filin/Dropbox/Vibe/MetaScienceTools/PDFextractor/frontend && npm run build
+```
+Must pass with 0 errors.
+
+### 3. Python Service Module Check
+```bash
+cd C:/Users/filin/Dropbox/Vibe/MetaScienceTools/PDFextractor/service && python -c "
+from app.main import app
+# Library modules (NOT app.normalize / app.quality — those moved to the docpluck library)
+from docpluck import normalize_text, NormalizationLevel, compute_quality_score, get_version_info
+info = get_version_info()
+print(f'All imports OK; docpluck=={info[\"version\"]} normalize={info[\"normalize_version\"]}')
+"
+```
+
+### 4. Cross-Repo Library Version Sync (CRITICAL)
+
+Verify the app's `service/requirements.txt` git pin matches the library's latest tag. Mismatches mean the deploy will silently ship the OLD library to prod.
+
+```bash
+LIB_VERSION=$(grep '^__version__' C:/Users/filin/Dropbox/Vibe/MetaScienceTools/docpluck/docpluck/__init__.py | grep -oE '[0-9]+\.[0-9]+\.[0-9]+')
+APP_PIN=$(grep -oE 'docpluck.*@v[0-9]+\.[0-9]+\.[0-9]+' C:/Users/filin/Dropbox/Vibe/MetaScienceTools/PDFextractor/service/requirements.txt | grep -oE '[0-9]+\.[0-9]+\.[0-9]+')
+echo "Library __version__: $LIB_VERSION"
+echo "App requirements.txt pin: v$APP_PIN"
+if [ "$LIB_VERSION" != "$APP_PIN" ]; then
+  echo "❌ MISMATCH — bump PDFextractor/service/requirements.txt to docpluck @ git+https://github.com/giladfeldman/docpluck.git@v$LIB_VERSION before deploying"
+  exit 1
+fi
+
+# Also verify the API.md examples are not stale beyond a major version
+API_DOC_VERSION=$(grep -oE 'docpluck_version["\s:]+[0-9]+\.[0-9]+\.[0-9]+' C:/Users/filin/Dropbox/Vibe/MetaScienceTools/PDFextractor/API.md | head -1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+')
+LIB_MAJOR_MINOR=$(echo "$LIB_VERSION" | cut -d. -f1,2)
+DOC_MAJOR_MINOR=$(echo "$API_DOC_VERSION" | cut -d. -f1,2)
+if [ "$LIB_MAJOR_MINOR" != "$DOC_MAJOR_MINOR" ]; then
+  echo "⚠️ API.md examples reference docpluck_version $API_DOC_VERSION; library is at $LIB_VERSION. Update PDFextractor/API.md."
+fi
+echo "✅ Library version sync OK"
+```
+
+### 4. Verify Vercel Environment Variables
+```bash
+cd frontend && vercel env ls
+```
+
+Required variables (all must show as "Encrypted"):
+- `DATABASE_URL` — Neon connection string
+- `AUTH_SECRET` — Auth.js session key
+- `AUTH_TRUST_HOST` — must be `true`
+- `AUTH_GITHUB_ID` — GitHub OAuth client ID
+- `AUTH_GITHUB_SECRET` — GitHub OAuth client secret
+- `AUTH_GOOGLE_ID` — Google OAuth client ID
+- `AUTH_GOOGLE_SECRET` — Google OAuth client secret
+- `EXTRACTION_SERVICE_URL` — Railway service URL
+
+If any are missing, refer to SETUP_GUIDE.md.
+
+## Deploy
+
+### Frontend (Vercel)
+Push to GitHub triggers auto-deploy:
+```bash
+git push origin master
+```
+
+Or manual deploy:
+```bash
+cd frontend && vercel --prod
+```
+
+### Extraction Service (Railway)
+If connected to GitHub, push triggers auto-deploy.
+
+If not connected, deploy via CLI:
+```bash
+cd service && railway up --detach
+```
+
+Note: Railway CLI upload may timeout. If so, connect GitHub repo in Railway dashboard (root dir `/service`).
+
+## Post-Deploy Verification
+
+### 1. Vercel Deployment Status
+```bash
+cd frontend && vercel ls | head -5
+```
+Latest deployment must show `Ready`.
+
+### 2. Frontend Health
+```bash
+curl -s -o /dev/null -w "%{http_code}" https://docpluck.vercel.app/login
+```
+Must return 200.
+
+### 3. Railway Service Health
+```bash
+curl -s https://extraction-service-production-d0e5.up.railway.app/health
+```
+Must return `{"status":"ok",...}`.
+
+### 4. Smoke Test (if service is live)
+```bash
+# Test extraction endpoint directly
+curl -s -X POST https://extraction-service-production-d0e5.up.railway.app/extract \
+  -F "file=@test-pdfs/apa/chan_feldman_2025_cogemo.pdf" | python -c "
+import sys, json
+data = json.load(sys.stdin)
+print(f'Engine: {data[\"metadata\"][\"engine\"]}')
+print(f'Chars: {data[\"metadata\"][\"chars\"]}')
+print(f'Quality: {data[\"quality\"][\"score\"]}')
+assert data['metadata']['chars'] > 10000, 'Too few chars'
+assert data['quality']['score'] >= 80, 'Quality too low'
+print('Smoke test: PASS')
+"
+```
+
+## Rollback
+
+If deployment fails:
+```bash
+# Vercel: rollback to previous deployment
+cd frontend && vercel rollback
+
+# Railway: redeploy from last working commit
+railway service extraction-service
+railway redeploy
+```
+
+## Report Format
+
+```
+## Docpluck Deploy Report
+
+### Pre-Flight
+| Check | Status |
+|-------|--------|
+| Git clean | PASS/FAIL |
+| Frontend build | PASS/FAIL |
+| Service modules | PASS/FAIL |
+| Env vars | X/Y present |
+
+### Deployment
+| Target | Status | URL |
+|--------|--------|-----|
+| Vercel | DEPLOYED/FAILED | https://docpluck.vercel.app |
+| Railway | DEPLOYED/FAILED | https://extraction-service-production-d0e5.up.railway.app |
+
+### Post-Deploy
+| Check | Status |
+|-------|--------|
+| Frontend 200 | PASS/FAIL |
+| Service health | PASS/FAIL |
+| Smoke test | PASS/FAIL/SKIP |
+```
+
+## Final step: read ~/.claude/skills/_shared/postflight.md and follow it.

docpluck-1.5.0/.claude/skills/docpluck-qa/SKILL.md

@@ -0,0 +1,251 @@
+---
+name: docpluck-qa
+description: Comprehensive QA engineer for Docpluck App (PDF + DOCX + HTML extraction SaaS). Runs 211-test Python suite, ESCIcheck 10-PDF AI verification (library + local webapp + production), normalization spot-check, batch extraction, service health, DB, admin API, and deployment checks. When asked for a "DOCX benchmark" or "format parity benchmark" or "--benchmark-docx" or similar, runs the special cross-format benchmark suite (CitationGuard DOCX corpus + DOCX\u2192PDF + PDF\u2192DOCX). Use /docpluck-qa whenever testing, after changes, or before deployment.
+tags: [python, nextjs, pdf, docx, fastapi, drizzle, neon, qa]
+---
+
+# Docpluck QA
+
+## [MANDATORY FIRST ACTION] preflight (do NOT skip, even if orchestrated by /ship)
+
+**Your very first action in this skill, BEFORE reading anything else, is:**
+
+1. Run: `bash ~/.claude/skills/_shared/bin/preflight-filter.sh <this-skill-name>` and print its `🔧 skill-optimize pre-check · ...` heartbeat as your first visible output line.
+2. Initialize `~/.claude/skills/_shared/run-meta/<this-skill-name>.json` per `~/.claude/skills/_shared/preflight.md` step 6 (include `phase_start_sha` from `git rev-parse HEAD`).
+3. Load `~/.claude/skills/_shared/quality-loop/core.md` into working memory (MUST-level rules gated by /ship).
+
+If you skip these steps, /ship will detect the missing heartbeat and FAIL this phase. Do not proceed to the skill body until preflight has run.
+
+You are a QA engineer for Docpluck App, a universal academic PDF text extraction SaaS.
+
+## Project Context
+
+- **App repo (private):** `C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor` (GitHub: giladfeldman/docpluckapp)
+- **Library repo (public):** `C:\Users\filin\Dropbox\Vibe\docpluck` (GitHub: giladfeldman/docpluck, PyPI: docpluck)
+- **Frontend:** Next.js 16 + Auth.js + Drizzle (in `frontend/`), port 6116
+- **Service:** Python FastAPI importing `docpluck` library (in `service/`), port 6117
+- **Database:** Neon Postgres (docpluck project)
+- **ESCIcheck PDFs:** `C:\Users\filin\Dropbox\Vibe\ESCIcheck\testpdfs\Coded already\` (56 PDFs, APA psychology papers)
+- **Test PDFs:** `test-pdfs/` (47 PDFs, 8 citation styles)
+- **Test suite:** `service/tests/` (151 tests across 6 files)
+
+## QA Checklist
+
+Run ALL checks sequentially. Report results in a structured table at the end.
+
+---
+
+### 1. Frontend Build
+```bash
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\frontend && npm run build 2>&1 | tail -20
+```
+Must compile with **0 errors**. Warnings about middleware/turbopack are expected (Next.js 16).
+
+---
+
+### 2. Python Test Suite (CRITICAL — 364+ tests)
+
+Run both the library repo and the service repo suites:
+
+```bash
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck && python -m pytest tests/ -q --tb=short 2>&1 | tail -10
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\service && python -m pytest tests/ -q --tb=short 2>&1 | tail -10
+```
+**All tests must pass.** Any failure indicates a regression.
+
+Library test coverage (`docpluck/tests/`):
+- `test_normalization.py` — All 15 pipeline steps (S0-S9, A1-A6)
+- `test_d5_normalization_audit.py` — **153 tests** (D5 audit, 2026-04-12): comprehensive regression suite for every normalization regex. Covers D5 bug fix, safe regex guard isolation, all A1 sub-rules, A1/S9 interaction, S7/S8/S9 stat protection, A2-A6 edge cases, all 13 stat types near section boundaries, extreme edge cases (section numbers, page numbers, value formats, sequences, Unicode), moderate-risk regex coverage. **This file is the primary defense against silent data corruption — run it on every normalization change.**
+- `test_quality.py` — Scoring, garbled detection, confidence levels
+- `test_extraction.py` — Real PDFs, SMP recovery, 8 citation styles
+- `test_edge_cases.py` — Cross-project lessons (dropped decimals, Unicode soup, column merges)
+- `test_extract_html.py` — 46 tests, block/inline tree-walk, ChanORCID regression
+- `test_extract_docx.py` — 18 tests, mammoth integration, soft breaks, smart quotes
+- `test_benchmark_docx_html.py` — 12 tests, ground-truth passage survival for DOCX/HTML
+- `test_metaesci_followups.py` — D3/D5/D6/D7 regression tests
+
+Service test coverage (`PDFextractor/service/tests/`):
+- `test_api_integration.py` — FastAPI /health and /extract endpoints
+- `test_benchmark.py` — Ground truth regression, idempotency
+
+---
+
+### 2b. D5 Normalization Regression (CRITICAL — 153 tests)
+
+This dedicated check runs the D5 audit test suite that guards against silent data
+corruption in the normalization pipeline. Added 2026-04-12 after MetaESCI found
+that a single regex destroyed ~800-1,200 stat lines across ~1,590 PDFs with zero
+warnings. **This check is mandatory after ANY change to normalize.py.**
+
+```bash
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck && python -m pytest tests/test_d5_normalization_audit.py -v --tb=short 2>&1 | tail -30
+```
+
+**All 153 tests must pass.** Key coverage areas:
+- D5 bug regression (12 tests): all MetaESCI corruption cases must NOT recur
+- Safe regex guards (11 tests): both letter-start and p-value-format guards work
+- A1 sub-rule isolation (17 tests): every stat linebreak repair rule independently
+- A1/S9 interaction (6 tests): stat values protected from page-number stripping
+- All 13 stat types near section boundaries: p, d, g, r, F, t, chi2, eta2, omega2, beta, OR, CI, RR
+- Extreme edge cases (32 tests): section numbers, page numbers, value formats, Unicode
+
+**Regex safety rules** (from D5 lesson):
+1. NEVER use `[^\n]` or `.` as catch-all in `re.sub` patterns
+2. ALWAYS constrain BOTH skipped content AND replacement target (two independent guards)
+3. Test every regex against `stat-value\nsection-number` patterns (18.5% of PDFs)
+4. Prefer narrow character classes (`[a-zA-Z]`) over broad exclusions (`[^\n]`)
+
+---
+
+### 3. Normalization Spot-Check
+```bash
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\service && python -c "
+from docpluck import normalize_text, NormalizationLevel
+
+raw = 'The signi\ufb01cant result was r(261) = \u22120.73, 95%\nCI [\u22120.78; \u22120.67], p\n< .001, d = 484'
+result, report = normalize_text(raw, NormalizationLevel.academic)
+assert 'significant' in result, 'S3 ligature failed'
+assert '-0.73' in result, 'S5 unicode minus failed'
+assert '95% CI' in result, 'A1 stat linebreak failed'
+assert '[-0.78, -0.67]' in result, 'A4 CI delimiter failed'
+assert 'p < .001' in result, 'A1 p-value linebreak failed'
+assert '.484' in result, 'A2 dropped decimal failed'
+print(f'Pipeline: PASS ({len(report.steps_applied)} steps, {len(report.changes_made)} changes)')
+print(f'Version: {report.version}')
+"
+```
+
+---
+
+### 4. SMP Recovery Test
+```bash
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\service && python -c "
+import os
+pdf_path = '../test-pdfs/nature/nathumbeh_2.pdf'
+if os.path.exists(pdf_path):
+    from docpluck import extract_pdf
+    with open(pdf_path, 'rb') as f:
+        content = f.read()
+    text, method = extract_pdf(content)
+    assert text.count('\ufffd') == 0, f'SMP recovery failed: {text.count(chr(0xFFFD))} garbled'
+    assert 'pdfplumber' in method, f'SMP recovery not triggered: {method}'
+    print(f'SMP Recovery: PASS (method={method})')
+else:
+    print('SMP Recovery: SKIP (no test PDF)')
+"
+```
+
+---
+
+### 5. ESCIcheck 10-PDF Verification — Library (CRITICAL)
+
+CRITICAL check: runs 10 ESCIcheck PDFs through the library extract+normalize pipeline, verifies chars, quality score, p-values, method, and sample coherence.
+
+**Full procedure:** [references/check-5-escicheck-library.md](references/check-5-escicheck-library.md)
+
+### 6. ESCIcheck 10-PDF Verification — Local Webapp (CRITICAL)
+
+CRITICAL check: same 10 PDFs through the local webapp /extract endpoint. Verifies HTTP status, engine, quality, and timing.
+
+**Full procedure:** [references/check-6-escicheck-local-webapp.md](references/check-6-escicheck-local-webapp.md)
+
+### 7. Batch Extraction Smoke Test (test-pdfs/)
+
+Walks the test-pdfs/ tree, runs extract_pdf on each, reports failures. Default corpus ~47 PDFs.
+
+**Full procedure:** [references/check-7-batch-smoke.md](references/check-7-batch-smoke.md)
+
+### 8. Service Health Endpoint
+```bash
+curl -s http://localhost:6117/health
+```
+Must return `{"status":"ok","pdftotext":"...","engines":["pdftotext_default"]}`.
+
+---
+
+### 9. Database Connectivity
+```bash
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\frontend && node -e "
+const { neon } = require('@neondatabase/serverless');
+require('dotenv').config({ path: '.env.local' });
+const sql = neon(process.env.DATABASE_URL);
+sql\`SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' ORDER BY table_name\`.then(r => {
+  const tables = r.map(t => t.table_name);
+  const expected = ['user', 'account', 'session', 'verificationToken', 'api_key', 'extraction_cache', 'usage_log'];
+  const missing = expected.filter(t => !tables.includes(t));
+  if (missing.length) console.log('MISSING:', missing.join(', '));
+  else console.log('Database: PASS (7/7 tables)', tables.join(', '));
+}).catch(e => console.log('Database: FAIL', e.message));
+"
+```
+
+---
+
+### 10. Admin API Smoke Test
+```bash
+curl -s http://localhost:6116/api/admin/health | python -c "import sys,json; d=json.load(sys.stdin); print('Admin health:', d.get('service',{}).get('status','?'))"
+curl -s http://localhost:6116/api/admin/stats | python -c "import sys,json; d=json.load(sys.stdin); print('Admin stats:', 'users' in d and 'keys' in d)"
+```
+
+---
+
+### 11. Hard Rules Verification
+
+Asserts 4 hard rules: no -layout flag in pdftotext calls, no AGPL imports, U+2212 normalization present, library version consistency.
+
+**Full procedure:** [references/check-11-hard-rules.md](references/check-11-hard-rules.md)
+
+### 12. Production Deployment (Vercel + Railway)
+```bash
+# Vercel frontend
+curl -s -o /dev/null -w "Vercel: HTTP %{http_code}\n" https://docpluck.vercel.app/login
+
+# Railway extraction service
+curl -s https://extraction-service-production-d0e5.up.railway.app/health | python -c "import sys,json; d=json.load(sys.stdin); print('Railway:', d.get('status','error'), d.get('pdftotext','unknown'))"
+```
+
+---
+
+### 13. ESCIcheck 10-PDF Verification — Production Webapp (CRITICAL)
+
+CRITICAL check: same 10 PDFs through the production Vercel endpoint. Verifies auth, cache behavior, and parity with local.
+
+**Full procedure:** [references/check-13-escicheck-production.md](references/check-13-escicheck-production.md)
+
+## Special Benchmark Mode: DOCX/PDF Parity + MetaESCI Regression
+
+Opt-in cross-format benchmark suite — DOCX corpus integrity, DOCX↔PDF parity via Word COM, PDF↔DOCX parity via pdf2docx, and the 200-DOI MetaESCI regression. Runtime 5-15 min (launches Word). Trigger only when user explicitly asks for "benchmark-docx", "format parity benchmark", or `/docpluck-qa benchmark-docx`.
+
+**Full procedure** (prerequisites, per-benchmark running instructions, pass criteria, interpretation, output format): [references/benchmark-mode.md](references/benchmark-mode.md)
+
+## Report Format
+
+```
+## Docpluck QA Report — [date]
+
+| # | Check | Status | Details |
+|---|-------|--------|---------|
+| 1 | Frontend build | PASS/FAIL | 0 errors |
+| 2 | Python test suite (151) | PASS/FAIL | 151/151 passed |
+| 3 | Normalization spot-check | PASS/FAIL | 15 steps, N changes |
+| 4 | SMP recovery | PASS/FAIL/SKIP | method used |
+| 5 | ESCIcheck 10-PDF (library) | PASS/FAIL | X/10 passed |
+| 6 | ESCIcheck 10-PDF (local webapp) | PASS/FAIL | X/10 passed |
+| 7 | Batch extraction (test-pdfs/) | PASS/FAIL | X/47 succeeded |
+| 8 | Service health | PASS/FAIL | pdftotext version |
+| 9 | Database connectivity | PASS/FAIL | 7/7 tables |
+| 10 | Admin API | PASS/FAIL | health + stats |
+| 11 | Hard rules (4 checks) | PASS/FAIL | no -layout, no AGPL, U+2212, version |
+| 12 | Production health | PASS/FAIL | HTTP codes |
+| 13 | ESCIcheck 10-PDF (production) | PASS/FAIL/SKIP | X/10 passed |
+
+**Overall: X/13 checks passed**
+
+### Issues Found
+- [list any failures with exact error messages and file:line]
+
+### AI Verification Notes (Checks 5, 6, 13)
+For each PDF: file name, chars, quality, p-values found, sample text judgment (coherent/garbled/column-interleaved)
+```
+
+## Final step: read ~/.claude/skills/_shared/postflight.md and follow it.

docpluck-1.5.0/.claude/skills/docpluck-qa/references/benchmark-mode.md

@@ -0,0 +1,106 @@
+# Special Benchmark Mode: DOCX/PDF Parity + MetaESCI Regression
+
+_Extracted from [../SKILL.md](../SKILL.md). Opt-in; 5-15 min runtime._
+
+
+Opt-in cross-format benchmark suite (DOCX corpus, DOCX↔PDF parity via Word COM, PDF↔DOCX via pdf2docx, MetaESCI 200-DOI regression). Takes 5-15 minutes; only run when explicitly requested.
+
+**Full procedure:** [references/benchmark-mode.md](references/benchmark-mode.md)
+
+### Prerequisites
+
+- Microsoft Word installed (for `docx2pdf`, accessed via COM)
+- Python packages: `mammoth`, `beautifulsoup4`, `lxml`, `rapidfuzz`, `pdf2docx`, `docx2pdf`
+- CitationGuard corpus present at `C:\Users\filin\Dropbox\Vibe\CitationGuard\apps\worker\testpdfs\validation\docx\`
+- PDF test corpus at `C:\Users\filin\Dropbox\Vibe\PDFextractor\test-pdfs\`
+
+### Running
+
+Full benchmark (5–15 minutes, launches Word):
+```bash
+cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py
+```
+
+Quick mode (3 files per benchmark, 2–4 minutes):
+```bash
+cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py --quick
+```
+
+Skip Word-based conversion (if Word unavailable):
+```bash
+cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py --skip docx2pdf
+```
+
+Individual benchmarks:
+```bash
+# 1. DOCX corpus (24 files, ~45s)
+python benchmarks/bench_docx_corpus.py --json benchmarks/results/docx_corpus.json
+
+# 2. DOCX → PDF cross-format (20 files via Word COM, ~5-10 min)
+python benchmarks/bench_docx_vs_pdf.py --json benchmarks/results/docx_vs_pdf.json
+
+# 3. PDF → DOCX reverse cross-format (8 files via pdf2docx, ~2-4 min)
+python benchmarks/bench_pdf_vs_docx.py --json benchmarks/results/pdf_vs_docx.json
+
+# 4. MetaESCI regression baseline (200 frozen DOIs, ~70s)
+python benchmarks/bench_metaesci_regression.py --json benchmarks/results/metaesci_regression.json
+```
+
+### What Each Benchmark Validates
+
+**1. DOCX corpus benchmark** (`bench_docx_corpus.py`)
+- Runs `extract_docx()` on all 24 CitationGuard DOCX files
+- Validates: all 20 real papers extract successfully; all 4 corrupted files fail correctly
+- Checks quality score ≥80 and "high" confidence for every real paper
+- Detects garbled extractions (should be 0)
+- Reports total chars, per-file times, aggregate stats
+
+**2. DOCX → PDF cross-format benchmark** (`bench_docx_vs_pdf.py`)
+- For each DOCX: extract text, convert DOCX→PDF via Word (`docx2pdf`), extract PDF text, compare
+- Uses `rapidfuzz.token_set_ratio` to measure similarity (should be ≥ 80%)
+- Uses char-count ratio to detect content loss (should be 0.7–1.3)
+- Both extractors should produce "high" confidence quality scores
+- This is the A/B gold-standard: same content, two extraction paths — asymmetry reveals bugs in either extractor
+
+**3. PDF → DOCX reverse cross-format benchmark** (`bench_pdf_vs_docx.py`)
+- For each PDF: extract text, convert PDF→DOCX via `pdf2docx` (pure Python), extract DOCX text, compare
+- Default selection: 8 PDFs across APA, Vancouver, AMA, Harvard, IEEE, Nature styles
+- Same pass criteria as #2
+
+### Pass Criteria
+
+Benchmark | Threshold
+--- | ---
+DOCX corpus extraction success | 20/20 real papers
+DOCX corpus quality score | avg ≥ 80, all high confidence
+DOCX corpus garbled count | 0
+DOCX→PDF similarity (token_set_ratio) | avg ≥ 80%, min ≥ 80%
+DOCX→PDF char ratio | 0.7–1.3
+PDF→DOCX similarity | avg ≥ 80%, min ≥ 80%
+PDF→DOCX char ratio | 0.7–1.3
+MetaESCI extraction success | 200/200 (zero crashes, zero extraction errors)
+MetaESCI high confidence | ≥ 99% (198/200+)
+MetaESCI avg quality | ≥ 80 / 100
+MetaESCI total U+FFFD | 0
+MetaESCI stat linebreak survivors | 0
+MetaESCI dropped decimal survivors | 0 (watch — current baseline has 21 in 3 files, real gaps pending A1/A2 enhancement)
+
+### Interpreting Results
+
+- **Similarity < 80%**: the two extractors are producing substantially different text for the same content. Investigate word-order or reading-order differences, soft-break handling, or table linearization.
+- **Char ratio far from 1.0**: one extractor is missing content (images, equations, headers, tracked changes). Investigate which format is losing data.
+- **DOCX quality score < PDF quality score**: something is wrong — DOCX should be cleaner than PDF (no column interleaving, no ligatures, no SMP Unicode). This is a red flag for mammoth configuration or the tree-walk.
+- **OMML equations**: known limitation — mammoth drops Office Math objects. Papers with stats inside equations will show lower char counts in DOCX vs PDF.
+
+### Output
+
+Benchmark writes three files to `benchmarks/results/`:
+- `docx_corpus.json` — per-file metrics
+- `docx_vs_pdf.json` — cross-format comparison
+- `pdf_vs_docx.json` — reverse comparison
+- `REPORT.md` — consolidated markdown report
+
+After running, read `benchmarks/results/REPORT.md` and report the summary to the user.
+
+---
+

docpluck-1.5.0/.claude/skills/docpluck-qa/references/check-11-hard-rules.md

@@ -0,0 +1,44 @@
+# 11. Hard Rules Verification
+
+_Extracted from [../SKILL.md](../SKILL.md). Full procedure lives here._
+
+```bash
+cd C:\Users\filin\Dropbox\Vibe\PDFextractor\service && python -c "
+import re
+
+# Rule 1: No -layout flag in pdftotext calls (check library)
+import docpluck.extract as ext_mod
+import inspect
+source = inspect.getsource(ext_mod)
+calls = re.findall(r'subprocess\.run\(\s*\[.*?\]', source, re.DOTALL)
+for call in calls:
+    assert '-layout' not in call, f'BLOCKER: -layout in {call}'
+print('Rule 1 (no -layout): PASS')
+
+# Rule 2: No AGPL imports in library or service
+import docpluck.normalize as norm_mod, docpluck.quality as qual_mod
+for name, mod in [('normalize', norm_mod), ('quality', qual_mod), ('extract', ext_mod)]:
+    src = inspect.getsource(mod)
+    assert 'pymupdf4llm' not in src, f'AGPL import in {name}'
+    assert 'column_boxes' not in src, f'AGPL method in {name}'
+with open('app/main.py') as f:
+    main_src = f.read()
+assert 'pymupdf4llm' not in main_src, 'AGPL import in main.py'
+print('Rule 2 (no AGPL): PASS')
+
+# Rule 3: U+2212 normalization exists in library (check file bytes to avoid encoding issues)
+import docpluck.normalize as _nm_mod
+with open(_nm_mod.__file__, 'rb') as _f:
+    _norm_bytes = _f.read()
+assert b'\\u2212' in _norm_bytes or b'\xe2\x88\x92' in _norm_bytes, 'U+2212 normalization missing'
+print('Rule 3 (U+2212 norm): PASS')
+
+# Rule 4: Library version is consistent
+import docpluck
+assert docpluck.__version__ == '1.4.5', f'Version mismatch: {docpluck.__version__}'
+print(f'Rule 4 (version=1.4.5): PASS')
+"
+```
+
+---
+