scholar-mcp 1.0.0

package/README.md

@@ -0,0 +1,232 @@
+# ScholarMCP
+
+ScholarMCP is an MCP server for literature research workflows in coding agents.
+
+It gives your agent tools to:
+- search papers across multiple sources
+- ingest and parse full-text PDFs
+- extract structured paper details
+- suggest citations and build references
+- validate manuscript citations
+
+## Who this is for
+
+Use this if you want Claude Code, Codex, or any MCP-compatible coding agent to run research tasks directly from chat.
+
+## What you get
+
+- Transports: `stdio` (recommended) and HTTP (`/mcp`)
+- Research providers: Google Scholar, OpenAlex, Crossref, Semantic Scholar
+- Full-text parsing pipeline: `grobid -> sidecar -> simple`
+- Tooling for thesis/paper workflows: ingestion, extraction, references, validation
+
+## Quick Start
+
+### 1. Prerequisites
+
+- Node.js `>=20`
+- `npm` (for install/publish)
+- `pnpm` (for contributors working from source)
+
+### 2. Install as an npm package (recommended)
+
+```bash
+npm install -g scholar-mcp
+```
+
+One-off run without global install:
+
+```bash
+npx -y scholar-mcp --transport=stdio
+```
+
+### 3. Run
+
+Stdio mode:
+
+```bash
+scholar-mcp --transport=stdio
+```
+
+HTTP mode:
+
+```bash
+scholar-mcp --transport=http
+```
+
+Health check (HTTP mode):
+
+```bash
+curl http://127.0.0.1:3000/health
+```
+
+### 4. Run from source (contributors)
+
+```bash
+pnpm install
+pnpm dev:stdio
+```
+
+## Use with Coding Agents
+
+### Claude Code (recommended)
+
+Register from globally installed binary:
+
+```bash
+claude mcp add -s user \
+  -e SCHOLAR_MCP_TRANSPORT=stdio \
+  -e SCHOLAR_REQUEST_DELAY_MS=350 \
+  -e RESEARCH_ALLOW_REMOTE_PDFS=true \
+  -e RESEARCH_ALLOW_LOCAL_PDFS=true \
+  -- scholar_mcp scholar-mcp --transport=stdio
+```
+
+Register without global install:
+
+```bash
+claude mcp add -s user \
+  -e SCHOLAR_MCP_TRANSPORT=stdio \
+  -e SCHOLAR_REQUEST_DELAY_MS=350 \
+  -e RESEARCH_ALLOW_REMOTE_PDFS=true \
+  -e RESEARCH_ALLOW_LOCAL_PDFS=true \
+  -- scholar_mcp npx -y scholar-mcp --transport=stdio
+```
+
+Check status:
+
+```bash
+claude mcp get scholar_mcp
+```
+
+Notes:
+- Keep the `--` before `scholar_mcp` (required by current Claude CLI parsing for multiple `-e` entries).
+- If you need to replace config: `claude mcp remove scholar_mcp -s project`.
+
+### OpenAI Codex App
+
+Add to `~/.codex/config.toml`:
+
+```toml
+[mcp_servers.scholar_mcp]
+command = "npx"
+args = ["-y", "scholar-mcp", "--transport=stdio"]
+
+[mcp_servers.scholar_mcp.env]
+SCHOLAR_MCP_TRANSPORT = "stdio"
+SCHOLAR_REQUEST_DELAY_MS = "350"
+RESEARCH_ALLOW_REMOTE_PDFS = "true"
+RESEARCH_ALLOW_LOCAL_PDFS = "true"
+```
+
+### Generic MCP clients
+
+- `stdio` command:
+  - `scholar-mcp --transport=stdio`
+  - Or: `npx -y scholar-mcp --transport=stdio`
+- HTTP endpoint:
+  1. Start server with `SCHOLAR_MCP_TRANSPORT=http scholar-mcp`
+  2. Connect client to `http://127.0.0.1:3000/mcp`
+  3. Optional auth: set `SCHOLAR_MCP_API_KEY` and send bearer auth header from your client
+
+## MCP Tools
+
+| Tool | Purpose |
+|---|---|
+| `search_literature_graph` | Federated search over OpenAlex/Crossref/Semantic Scholar (+ optional scholar scrape). |
+| `search_google_scholar_key_words` | Keyword search on Google Scholar. |
+| `search_google_scholar_advanced` | Scholar search with author/year/phrase filters. |
+| `get_author_info` | Resolve author profile and top publications. |
+| `ingest_paper_fulltext` | Start async full-text ingestion from DOI/URL/PDF/local path. |
+| `get_ingestion_status` | Poll ingestion job status and parsed summary. |
+| `extract_granular_paper_details` | Extract methods, claims, datasets, metrics, and references. |
+| `suggest_contextual_citations` | Suggest citations from manuscript context. |
+| `build_reference_list` | Generate formatted bibliography and BibTeX. |
+| `validate_manuscript_citations` | Detect missing/uncited/duplicate citation issues. |
+
+## Example Agent Prompts
+
+- "Find 10 recent papers on retrieval-augmented generation and summarize methods and datasets."
+- "Ingest full text for DOI `10.1038/s41467-024-55563-6`, then extract claims and limitations."
+- "Given this draft section, suggest citations in IEEE style and generate BibTeX."
+- "Validate my manuscript citations against this reference list and show missing citations."
+
+## Optional Python Sidecar (better parsing fallback)
+
+Run sidecar:
+
+```bash
+cd python-sidecar
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+uvicorn app:app --host 127.0.0.1 --port 8090
+```
+
+Then set:
+
+```bash
+RESEARCH_PYTHON_SIDECAR_URL=http://127.0.0.1:8090
+```
+
+## Configuration
+
+Most users only need these:
+
+- `SCHOLAR_MCP_TRANSPORT`: `stdio` | `http` | `both` (default: `stdio`)
+- `SCHOLAR_REQUEST_DELAY_MS`: request pacing to reduce rate-limit risk (default: `250`)
+- `RESEARCH_ALLOW_REMOTE_PDFS`: allow remote PDF downloads for ingestion (default: `true`)
+- `RESEARCH_ALLOW_LOCAL_PDFS`: allow local PDF ingestion (default: `true`)
+- `SCHOLAR_MCP_API_KEY`: optional bearer token for HTTP mode
+- `RESEARCH_GROBID_URL`: optional GROBID endpoint
+- `RESEARCH_PYTHON_SIDECAR_URL`: optional sidecar endpoint
+
+The CLI loads `.env` from the current working directory automatically at startup.
+
+Advanced options exist in `src/config.ts` for timeouts, retries, HTTP session capacity/TTL, provider tuning, and cache behavior.
+
+## Troubleshooting
+
+- `Invalid environment variable format` in `claude mcp add`:
+  - Add `--` before the MCP server name (see Claude setup command above).
+- `Unable to resolve a downloadable PDF URL from input` on DOI ingestion:
+  - The DOI landing page may not expose a downloadable PDF.
+  - Retry with `pdf_url` (direct PDF) or `local_pdf_path`.
+- Too many Scholar failures or throttling:
+  - Increase `SCHOLAR_REQUEST_DELAY_MS` (for example `500` to `1000`).
+
+## Dev Verification
+
+```bash
+pnpm check
+pnpm test
+```
+
+## Publish Workflow
+
+```bash
+# 1) update version
+npm version patch
+
+# 2) verify source quality
+pnpm check
+pnpm test
+
+# 3) verify npm package contents and executable bin
+npm run pack:dry-run
+npm pack
+
+# 4) publish to npm
+npm publish
+```
+
+Post-publish smoke test:
+
+```bash
+npx -y scholar-mcp --version
+npx -y scholar-mcp --help
+```
+
+## Usage Notes
+
+Google Scholar may throttle automated traffic. Use conservative request pacing, respect provider terms, and avoid abusive query patterns.

package/dist/cli/args.js

@@ -0,0 +1,57 @@
+const TRANSPORT_OPTIONS = ['stdio', 'http', 'both'];
+const TRANSPORT_SET = new Set(TRANSPORT_OPTIONS);
+const isTransportMode = (value) => TRANSPORT_SET.has(value);
+const parseTransport = (value) => {
+    const normalized = value.trim().toLowerCase();
+    if (!isTransportMode(normalized)) {
+        throw new Error(`Invalid transport "${value}". Expected one of: ${TRANSPORT_OPTIONS.join(', ')}.`);
+    }
+    return normalized;
+};
+export const CLI_USAGE = `ScholarMCP MCP server
+
+Usage:
+  scholar-mcp [--transport <stdio|http|both>]
+  scholar-mcp --help
+  scholar-mcp --version
+
+Options:
+  --transport <mode>  Override SCHOLAR_MCP_TRANSPORT for this run
+  -h, --help          Show help
+  -v, --version       Print package version`;
+export const parseCliArgs = (argv) => {
+    const args = {
+        showHelp: false,
+        showVersion: false
+    };
+    for (let index = 0; index < argv.length; index += 1) {
+        const arg = argv[index]?.trim();
+        if (!arg) {
+            continue;
+        }
+        if (arg === '-h' || arg === '--help') {
+            args.showHelp = true;
+            continue;
+        }
+        if (arg === '-v' || arg === '--version') {
+            args.showVersion = true;
+            continue;
+        }
+        if (arg === '--transport') {
+            const nextValue = argv[index + 1];
+            if (!nextValue) {
+                throw new Error('Missing value after --transport.');
+            }
+            args.transport = parseTransport(nextValue);
+            index += 1;
+            continue;
+        }
+        if (arg.startsWith('--transport=')) {
+            const value = arg.slice('--transport='.length);
+            args.transport = parseTransport(value);
+            continue;
+        }
+        throw new Error(`Unknown argument "${arg}".`);
+    }
+    return args;
+};

package/dist/config.js

@@ -0,0 +1,131 @@
+import { z } from 'zod';
+import { getPackageVersion } from './version.js';
+const numberFromEnv = (defaultValue, min, max) => z.coerce.number().int().min(min).max(max).default(defaultValue);
+const floatFromEnv = (defaultValue, min, max) => z.coerce.number().min(min).max(max).default(defaultValue);
+const booleanFromEnv = (defaultValue) => z.preprocess((value) => {
+    if (typeof value === 'boolean') {
+        return value;
+    }
+    if (typeof value === 'number') {
+        return value !== 0;
+    }
+    if (typeof value === 'string') {
+        const normalized = value.trim().toLowerCase();
+        if (['1', 'true', 'yes', 'on'].includes(normalized)) {
+            return true;
+        }
+        if (['0', 'false', 'no', 'off'].includes(normalized)) {
+            return false;
+        }
+    }
+    return value;
+}, z.boolean().default(defaultValue));
+const defaultServerVersion = getPackageVersion();
+const envSchema = z.object({
+    NODE_ENV: z.enum(['development', 'test', 'production']).default('development'),
+    LOG_LEVEL: z.enum(['debug', 'info', 'warn', 'error']).default('info'),
+    SCHOLAR_MCP_SERVER_NAME: z.string().default('scholar-mcp'),
+    SCHOLAR_MCP_SERVER_VERSION: z.string().default(defaultServerVersion),
+    SCHOLAR_MCP_TRANSPORT: z.enum(['stdio', 'http', 'both']).default('stdio'),
+    SCHOLAR_MCP_HOST: z.string().default('127.0.0.1'),
+    SCHOLAR_MCP_PORT: numberFromEnv(3000, 1, 65535),
+    SCHOLAR_MCP_ENDPOINT_PATH: z.string().default('/mcp'),
+    SCHOLAR_MCP_HEALTH_PATH: z.string().default('/health'),
+    SCHOLAR_MCP_HTTP_SESSION_MODE: z.enum(['stateful', 'stateless']).default('stateful'),
+    SCHOLAR_MCP_HTTP_SESSION_TTL_MS: numberFromEnv(30 * 60 * 1000, 10_000, 24 * 60 * 60 * 1000),
+    SCHOLAR_MCP_HTTP_MAX_SESSIONS: numberFromEnv(200, 1, 5000),
+    SCHOLAR_MCP_ALLOWED_ORIGINS: z.string().optional(),
+    SCHOLAR_MCP_ALLOWED_HOSTS: z.string().optional(),
+    SCHOLAR_MCP_API_KEY: z.string().optional(),
+    SCHOLAR_BASE_URL: z.string().url().default('https://scholar.google.com'),
+    SCHOLAR_LANGUAGE: z.string().default('en'),
+    SCHOLAR_TIMEOUT_MS: numberFromEnv(15000, 1000, 120000),
+    SCHOLAR_RETRY_ATTEMPTS: numberFromEnv(2, 0, 5),
+    SCHOLAR_RETRY_DELAY_MS: numberFromEnv(800, 0, 30000),
+    SCHOLAR_REQUEST_DELAY_MS: numberFromEnv(250, 0, 10000),
+    SCHOLAR_MAX_RESULTS_PER_REQUEST: numberFromEnv(20, 1, 20),
+    RESEARCH_OPENALEX_BASE_URL: z.string().url().default('https://api.openalex.org'),
+    RESEARCH_OPENALEX_API_KEY: z.string().optional(),
+    RESEARCH_CROSSREF_BASE_URL: z.string().url().default('https://api.crossref.org'),
+    RESEARCH_SEMANTIC_SCHOLAR_BASE_URL: z.string().url().default('https://api.semanticscholar.org/graph/v1'),
+    RESEARCH_SEMANTIC_SCHOLAR_API_KEY: z.string().optional(),
+    RESEARCH_TIMEOUT_MS: numberFromEnv(20000, 1000, 120000),
+    RESEARCH_RETRY_ATTEMPTS: numberFromEnv(2, 0, 5),
+    RESEARCH_RETRY_DELAY_MS: numberFromEnv(800, 0, 30000),
+    RESEARCH_REQUEST_DELAY_MS: numberFromEnv(100, 0, 10000),
+    RESEARCH_ALLOW_REMOTE_PDFS: booleanFromEnv(true),
+    RESEARCH_ALLOW_LOCAL_PDFS: booleanFromEnv(true),
+    RESEARCH_GROBID_URL: z.string().url().optional(),
+    RESEARCH_PYTHON_SIDECAR_URL: z.string().url().optional(),
+    RESEARCH_SEMANTIC_ENGINE: z.enum(['cloud-llm', 'none']).default('cloud-llm'),
+    RESEARCH_CLOUD_MODEL: z.string().default('gpt-4.1-mini'),
+    RESEARCH_GRAPH_CACHE_TTL_MS: numberFromEnv(5 * 60 * 1000, 0, 24 * 60 * 60 * 1000),
+    RESEARCH_GRAPH_MAX_CACHE_ENTRIES: numberFromEnv(300, 1, 5000),
+    RESEARCH_GRAPH_PROVIDER_RESULT_MULTIPLIER: numberFromEnv(2, 1, 5),
+    RESEARCH_GRAPH_FUZZY_TITLE_THRESHOLD: floatFromEnv(0.84, 0.6, 0.99)
+});
+const splitCsv = (value) => {
+    if (!value) {
+        return [];
+    }
+    return value
+        .split(',')
+        .map((item) => item.trim())
+        .filter((item) => item.length > 0);
+};
+const normalizePath = (value) => {
+    const withPrefix = value.startsWith('/') ? value : `/${value}`;
+    return withPrefix.length > 1 && withPrefix.endsWith('/')
+        ? withPrefix.slice(0, -1)
+        : withPrefix;
+};
+export const parseConfig = (overrides) => {
+    const mergedEnv = {
+        ...process.env,
+        ...(overrides ?? {})
+    };
+    const env = envSchema.parse(mergedEnv);
+    return {
+        nodeEnv: env.NODE_ENV,
+        logLevel: env.LOG_LEVEL,
+        serverName: env.SCHOLAR_MCP_SERVER_NAME,
+        serverVersion: env.SCHOLAR_MCP_SERVER_VERSION,
+        transport: env.SCHOLAR_MCP_TRANSPORT,
+        host: env.SCHOLAR_MCP_HOST,
+        port: env.SCHOLAR_MCP_PORT,
+        endpointPath: normalizePath(env.SCHOLAR_MCP_ENDPOINT_PATH),
+        healthPath: normalizePath(env.SCHOLAR_MCP_HEALTH_PATH),
+        httpSessionMode: env.SCHOLAR_MCP_HTTP_SESSION_MODE,
+        httpSessionTtlMs: env.SCHOLAR_MCP_HTTP_SESSION_TTL_MS,
+        httpMaxSessions: env.SCHOLAR_MCP_HTTP_MAX_SESSIONS,
+        allowedOrigins: splitCsv(env.SCHOLAR_MCP_ALLOWED_ORIGINS),
+        allowedHosts: splitCsv(env.SCHOLAR_MCP_ALLOWED_HOSTS).map((host) => host.toLowerCase()),
+        apiKey: env.SCHOLAR_MCP_API_KEY,
+        scholarBaseUrl: env.SCHOLAR_BASE_URL,
+        scholarLanguage: env.SCHOLAR_LANGUAGE,
+        scholarTimeoutMs: env.SCHOLAR_TIMEOUT_MS,
+        scholarRetryAttempts: env.SCHOLAR_RETRY_ATTEMPTS,
+        scholarRetryDelayMs: env.SCHOLAR_RETRY_DELAY_MS,
+        scholarRequestDelayMs: env.SCHOLAR_REQUEST_DELAY_MS,
+        scholarMaxResultsPerRequest: env.SCHOLAR_MAX_RESULTS_PER_REQUEST,
+        researchOpenAlexBaseUrl: env.RESEARCH_OPENALEX_BASE_URL,
+        researchOpenAlexApiKey: env.RESEARCH_OPENALEX_API_KEY,
+        researchCrossrefBaseUrl: env.RESEARCH_CROSSREF_BASE_URL,
+        researchSemanticScholarBaseUrl: env.RESEARCH_SEMANTIC_SCHOLAR_BASE_URL,
+        researchSemanticScholarApiKey: env.RESEARCH_SEMANTIC_SCHOLAR_API_KEY,
+        researchTimeoutMs: env.RESEARCH_TIMEOUT_MS,
+        researchRetryAttempts: env.RESEARCH_RETRY_ATTEMPTS,
+        researchRetryDelayMs: env.RESEARCH_RETRY_DELAY_MS,
+        researchRequestDelayMs: env.RESEARCH_REQUEST_DELAY_MS,
+        researchAllowRemotePdfs: env.RESEARCH_ALLOW_REMOTE_PDFS,
+        researchAllowLocalPdfs: env.RESEARCH_ALLOW_LOCAL_PDFS,
+        researchGrobidUrl: env.RESEARCH_GROBID_URL,
+        researchPythonSidecarUrl: env.RESEARCH_PYTHON_SIDECAR_URL,
+        researchSemanticEngine: env.RESEARCH_SEMANTIC_ENGINE,
+        researchCloudModel: env.RESEARCH_CLOUD_MODEL,
+        researchGraphCacheTtlMs: env.RESEARCH_GRAPH_CACHE_TTL_MS,
+        researchGraphMaxCacheEntries: env.RESEARCH_GRAPH_MAX_CACHE_ENTRIES,
+        researchGraphProviderResultMultiplier: env.RESEARCH_GRAPH_PROVIDER_RESULT_MULTIPLIER,
+        researchGraphFuzzyTitleThreshold: env.RESEARCH_GRAPH_FUZZY_TITLE_THRESHOLD
+    };
+};

package/dist/core/logger.js

@@ -0,0 +1,36 @@
+const PRIORITY = {
+    debug: 10,
+    info: 20,
+    warn: 30,
+    error: 40
+};
+export class Logger {
+    minLevel;
+    constructor(minLevel) {
+        this.minLevel = minLevel;
+    }
+    debug(message, context) {
+        this.log('debug', message, context);
+    }
+    info(message, context) {
+        this.log('info', message, context);
+    }
+    warn(message, context) {
+        this.log('warn', message, context);
+    }
+    error(message, context) {
+        this.log('error', message, context);
+    }
+    log(level, message, context) {
+        if (PRIORITY[level] < PRIORITY[this.minLevel]) {
+            return;
+        }
+        const payload = {
+            ts: new Date().toISOString(),
+            level,
+            message,
+            ...(context ? { context } : {})
+        };
+        process.stderr.write(`${JSON.stringify(payload)}\n`);
+    }
+}