@dataville/dataville-mcp 0.1.6 → 0.1.7

package/README.md

@@ -1,6 +1,7 @@
 # dataville-mcp
 
 [![M8ven Score](https://m8ven.ai/badge/mcp/datavilleorg-dataville-mcp-u1eou7?v=6abe1c0b98731d0af6f8978f1b3778e6)](https://m8ven.ai/mcp/datavilleorg-dataville-mcp-u1eou7?s=readme)
+[![Glama MCP server score](https://glama.ai/mcp/servers/datavilleorg/dataville-mcp/badges/score.svg)](https://glama.ai/mcp/servers/datavilleorg/dataville-mcp)
 [![npm](https://img.shields.io/npm/v/@dataville/dataville-mcp)](https://www.npmjs.com/package/@dataville/dataville-mcp)
 [![CI](https://github.com/datavilleorg/dataville-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/datavilleorg/dataville-mcp/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
@@ -20,8 +21,18 @@ Requires a Dataville API key — get one from the [Dataville dashboard](https://
 
 ## Tools
 
-- `list_dataville_sources` — lists the data sources available via `search_dataville`.
-- `search_dataville` — query a data source: `{ source, keywords, params? }`.
+| Tool | What it does |
+| --- | --- |
+| `list_dataville_sources` | Every source, with the SQL tables (if any) you can query for it. |
+| `describe_dataville_source` | One source's keyword format, a working example, and its tables' columns: `{ source }`. |
+| `search_dataville` | Look one thing up by keywords and get the best match: `{ source, keywords, summary? }`. |
+| `query_dataville` | Read-only SQL `SELECT` over the stored tables for lists, filters, counts, joins, and paging with `LIMIT`/`OFFSET`: `{ sql }`. |
+
+All four only read data. `query_dataville` covers every stored source except
+`census` and `news`, which are fetched live per search. It returns at most
+1,000 rows per call and is billed per row returned. It reads Dataville's stored
+copy: some sources are loaded in bulk, others only hold records fetched before,
+so for one specific item `search_dataville` is the reliable route.
 
 ## What you can ask
 
@@ -32,6 +43,8 @@ Once connected, ask in plain language and the client picks the source:
 - "What's the median household income in Travis County, Texas, per the US Census?"
 - "How much protein is in 100 g of cooked lentils, according to USDA FoodData?"
 - "Look up the `requests` package on PyPI — what's the latest version and license?"
+- "List ten Project Gutenberg books by Jane Austen, with their subjects."
+- "List five USDA FoodData entries with 'lentils' in the name, with their categories."
 
 ## Setup
 
@@ -110,6 +123,18 @@ every project instead of just the current one.
 `DATAVILLE_API_BASE_URL` is optional and defaults to `https://api.dataville.com`;
 set it to `http://localhost:5000` to point at a local backend during development.
 
+### Where your API key goes
+
+The server reads `DATAVILLE_API_KEY` from its environment and sends it only as
+an `Authorization: Bearer` header to `DATAVILLE_API_BASE_URL`, which is
+`https://api.dataville.com` unless you change it. It never logs the key or sends
+it anywhere else. The base URL must use `https`. Plain `http` is accepted only
+for `localhost`, `127.0.0.1` and `[::1]`, so the key never crosses the network
+unencrypted.
+
+All four tools are read-only: they search and query Dataville and never create,
+change or delete anything.
+
 ### Running from source
 
 ```bash
@@ -131,6 +156,7 @@ on its own proves nothing.
 | `Using Dataville, what is the latest version of the requests package on PyPI?` | A version you can confirm on pypi.org — and it moves, so it can't come from memory. |
 | `Using Dataville, get the latest SEC filing for AAPL and its revenue.` | A form type, filing date, revenue figure, and a sec.gov link to open. |
 | `Using Dataville, how much protein is in 100g of uncooked quinoa?` | The exact USDA figure, 14.1 g per 100 g. |
+| `Using Dataville SQL, list 5 Project Gutenberg books by Mark Twain.` | Calls `query_dataville` and returns titles with Gutenberg IDs you can open at gutenberg.org/ebooks/<id>. |
 
 Clients show when a tool ran. If you don't see that, say "use dataville" in the
 prompt to make it explicit, and check the answer against the source.
@@ -145,7 +171,8 @@ npm run build  # tsc
 
 ## Releasing
 
-Publishes run from CI via npm trusted publishing (OIDC) — no tokens are stored.
+Publishes run from CI. npm uses trusted publishing (OIDC), so no npm token is
+stored; the MCP Registry step signs in with the `MCP_PRIVATE_KEY` repo secret.
 To cut a release: bump the version, update `CHANGELOG.md`, then publish a GitHub
 Release for the new tag. The `Publish` workflow builds, tests, and publishes to npm,
 then publishes `server.json` to the [MCP Registry](https://registry.modelcontextprotocol.io)

package/dist/client.js

@@ -1,6 +1,6 @@
 const DEFAULT_BASE_URL = "https://api.dataville.com";
 // Keep in sync with package.json version (enforced by a test).
-export const VERSION = "0.1.6";
+export const VERSION = "0.1.7";
 const USER_AGENT = `dataville-mcp/${VERSION}`;
 export class DatavilleApiError extends Error {
     status;
@@ -31,8 +31,29 @@ function getConfig() {
         throw new Error("DATAVILLE_API_KEY is not set. Generate an API key from the Dataville dashboard and set it in your MCP client config.");
     }
     const baseUrl = process.env.DATAVILLE_API_BASE_URL || DEFAULT_BASE_URL;
+    assertSafeBaseUrl(baseUrl);
     return { apiKey, baseUrl };
 }
+const LOCAL_HOSTS = new Set(["localhost", "127.0.0.1", "[::1]"]);
+/**
+ * Every request carries the API key, so refuse a base URL that would send it
+ * in cleartext. Plain http is allowed only for a local backend.
+ */
+function assertSafeBaseUrl(baseUrl) {
+    let url;
+    try {
+        url = new URL(baseUrl);
+    }
+    catch {
+        throw new Error(`DATAVILLE_API_BASE_URL "${baseUrl}" is not a valid URL.`);
+    }
+    if (url.protocol === "https:")
+        return;
+    if (url.protocol === "http:" && LOCAL_HOSTS.has(url.hostname))
+        return;
+    throw new Error(`DATAVILLE_API_BASE_URL must use https (plain http is allowed only for localhost), got "${baseUrl}". ` +
+        "Your API key is sent with every request, so it is not sent over an unencrypted connection.");
+}
 export async function searchDataSource(source, keywords, params) {
     const { apiKey, baseUrl } = getConfig();
     // encodeURIComponent leaves "." alone, and URL resolution collapses a "." or
@@ -55,14 +76,45 @@ export async function searchDataSource(source, keywords, params) {
             "User-Agent": USER_AGENT,
         },
     });
+    return readResponse(response);
+}
+/**
+ * Run a read-only SQL SELECT against Dataville's stored copy of its sources
+ * (DuckDB over the tables listed by describe_dataville_source). The API
+ * rejects anything but SELECT, caps results at 1,000 rows, and times out after
+ * 30 seconds; it bills per row returned.
+ */
+export async function queryDataville(sql) {
+    const { apiKey, baseUrl } = getConfig();
+    const response = await fetch(new URL("/api/v1/query", baseUrl), {
+        method: "POST",
+        headers: {
+            Authorization: `Bearer ${apiKey}`,
+            "User-Agent": USER_AGENT,
+            "Content-Type": "application/json",
+        },
+        body: JSON.stringify({ sql }),
+    });
+    return readResponse(response);
+}
+const FIX_KEY_HINT = "Check the key in your MCP client config, or generate a new one at https://app.dataville.com/api-keys.";
+async function readResponse(response) {
     const body = await response.json().catch(() => undefined);
+    // SQL queries need an account, so an unrecognised key gets a 401 there
+    // rather than the anonymous fallback search uses. The API's own message
+    // ("include your API key") would be misleading — we did include one.
+    if (response.status === 401) {
+        throw new DatavilleAuthError(`Your DATAVILLE_API_KEY was not recognised. ${FIX_KEY_HINT}`);
+    }
     if (!response.ok) {
-        // Errors come back as { status: "error", data: { error: "..." } }. Those
-        // messages are useful to the model — "no results for X", or the full list
-        // of valid sources — so prefer them over a bare status code.
+        // Search errors come back as { status: "error", data: { error: "..." } },
+        // query errors as { status: "error", message: "..." }. Those messages are
+        // useful to the model — "no results for X", the list of valid sources, or
+        // the SQL error — so prefer them over a bare status code.
         const apiMessage = body && typeof body === "object"
             ? body.data?.error ??
-                body.error
+                body.error ??
+                body.message
             : undefined;
         const message = typeof apiMessage === "string" && apiMessage.length > 0
             ? apiMessage
@@ -77,7 +129,7 @@ export async function searchDataSource(source, keywords, params) {
         body.account_state === "anonymous") {
         throw new DatavilleAuthError("Your DATAVILLE_API_KEY was not recognised, so this request fell back to anonymous access " +
             "(much lower rate limits, and usage is not attributed to your account). " +
-            "Check the key in your MCP client config, or generate a new one at https://app.dataville.com/api-keys.");
+            FIX_KEY_HINT);
     }
     return body;
 }

package/dist/server.js

@@ -1,15 +1,30 @@
 import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 import { z } from "zod";
-import { searchDataSource, VERSION } from "./client.js";
-import { DATAVILLE_SOURCES } from "./sources.js";
+import { queryDataville, searchDataSource, VERSION } from "./client.js";
+import { DATAVILLE_SOURCE_DETAILS, DATAVILLE_SOURCES, findSource } from "./sources.js";
+const SQL_TABLES = DATAVILLE_SOURCE_DETAILS.flatMap((s) => s.sqlTables.map((t) => t.name));
+const INSTRUCTIONS = `Dataville serves public datasets (Wikipedia, arXiv, SEC EDGAR, US Census, USDA FoodData, PyPI and more) through one API.
+
+- list_dataville_sources: what sources exist.
+- describe_dataville_source: what keywords a source expects, and the SQL tables and columns it has.
+- search_dataville: look one thing up by keywords; returns the single best match.
+- query_dataville: SQL SELECT over the stored tables, for lists, filters, counts, joins and paging through many rows.`;
+function json(payload) {
+    return { content: [{ type: "text", text: JSON.stringify(payload, null, 2) }] };
+}
+function toolError(error) {
+    const message = error instanceof Error ? error.message : String(error);
+    return { content: [{ type: "text", text: message }], isError: true };
+}
 export function createServer() {
     const server = new McpServer({
         name: "dataville-mcp-server",
         version: VERSION,
-    });
+    }, { instructions: INSTRUCTIONS });
     server.registerTool("list_dataville_sources", {
         title: "List Dataville data sources",
-        description: "List the data sources available through Dataville's search_dataville tool.",
+        description: "List every Dataville data source with a one-line description and the SQL tables (if any) that query_dataville can read for it. " +
+            "Call describe_dataville_source for a source's keyword format and table columns.",
         inputSchema: {},
         // Returns a static list bundled with the server; makes no network calls.
         annotations: {
@@ -18,19 +33,59 @@ export function createServer() {
             idempotentHint: true,
             openWorldHint: false,
         },
-    }, async () => ({
-        content: [{ type: "text", text: JSON.stringify(DATAVILLE_SOURCES, null, 2) }],
-    }));
+    }, async () => json(DATAVILLE_SOURCE_DETAILS.map(({ name, description, sqlTables }) => ({
+        name,
+        description,
+        sql_tables: sqlTables.map((t) => t.name),
+    }))));
+    server.registerTool("describe_dataville_source", {
+        title: "Describe a Dataville data source",
+        description: "Show how to use one data source: what search_dataville expects as keywords (with an example that returns a result), " +
+            "and the SQL tables and columns query_dataville can read for it. Sources with no tables can only be searched.",
+        inputSchema: {
+            source: z.string().describe("Data source name from list_dataville_sources, e.g. 'edgar'"),
+        },
+        // Returns static metadata bundled with the server; makes no network calls.
+        annotations: {
+            readOnlyHint: true,
+            destructiveHint: false,
+            idempotentHint: true,
+            openWorldHint: false,
+        },
+    }, async ({ source }) => {
+        const details = findSource(source);
+        if (!details) {
+            return toolError(`Unknown source "${source}". Valid sources: ${DATAVILLE_SOURCES.map((s) => s.name).join(", ")}.`);
+        }
+        return json({
+            name: details.name,
+            description: details.description,
+            search: {
+                keywords: details.keywords,
+                example: { source: details.name, keywords: details.exampleKeywords },
+            },
+            sql_tables: details.sqlTables,
+            ...(details.sqlTables.length > 0
+                ? { example_sql: `SELECT * FROM ${details.sqlTables[0].name} LIMIT 5` }
+                : { note: "This source is fetched live per search and has no SQL table; use search_dataville." }),
+        });
+    });
     server.registerTool("search_dataville", {
         title: "Search a Dataville data source",
-        description: "Query one of Dataville's data sources (see list_dataville_sources for valid source names) with a keyword string and optional query params.",
+        description: "Look up one thing in a Dataville data source by keywords and get back the single best-matching record " +
+            "(full text in `body`, structured fields in `metadata`). Use describe_dataville_source to see what keywords a source expects. " +
+            "To get many records, filter, or page through results, use query_dataville instead.",
         inputSchema: {
-            source: z.string().describe("Data source name, e.g. 'wikipedia', 'arxiv', 'edgar'"),
-            keywords: z.string().describe("Search keywords or identifier for the query"),
+            source: z.string().describe("Data source name, e.g. 'wikipedia', 'arxiv', 'edgar' (see list_dataville_sources)"),
+            keywords: z.string().describe("Search keywords or identifier, e.g. a title, ticker, or package name"),
+            summary: z
+                .boolean()
+                .optional()
+                .describe("If true, truncate `body` to its first 100 characters to save tokens"),
             params: z
-                .record(z.union([z.string(), z.number(), z.boolean()]))
+                .record(z.string(), z.union([z.string(), z.number(), z.boolean()]))
                 .optional()
-                .describe("Optional additional query parameters"),
+                .describe("Extra query-string parameters passed through to the API as-is. Most callers need none."),
         },
         // A GET against the Dataville API, which fronts external data sources.
         annotations: {
@@ -39,14 +94,45 @@ export function createServer() {
             idempotentHint: true,
             openWorldHint: true,
         },
-    }, async ({ source, keywords, params }) => {
+    }, async ({ source, keywords, summary, params }) => {
+        try {
+            const query = summary === undefined ? params : { ...params, summary };
+            return json(await searchDataSource(source, keywords, query));
+        }
+        catch (error) {
+            return toolError(error);
+        }
+    });
+    server.registerTool("query_dataville", {
+        title: "Query Dataville with SQL",
+        description: "Run a read-only SQL SELECT (DuckDB dialect) over Dataville's stored tables: " +
+            `${SQL_TABLES.join(", ")}. ` +
+            "Use it to list, filter (WHERE … ILIKE '%term%'), sort, count, aggregate, or join across sources, " +
+            "and page through results with LIMIT/OFFSET. Get each table's columns from describe_dataville_source. " +
+            "Only SELECT is allowed; at most 1,000 rows are returned (a missing LIMIT becomes LIMIT 1000); queries time out after 30 s. " +
+            "Billed per row returned, so keep LIMIT as small as the task allows. " +
+            "The tables hold Dataville's stored copy: some sources are loaded in bulk, others only hold records fetched before, " +
+            "so a row missing here may still exist upstream — fall back to search_dataville for a specific item.",
+        inputSchema: {
+            sql: z
+                .string()
+                .min(1)
+                .max(10_000)
+                .describe("A single SELECT statement, e.g. \"SELECT company_name, form_type, filing_date FROM edgar WHERE company_name ILIKE '%apple%' ORDER BY filing_date DESC LIMIT 10\""),
+        },
+        // A POST, but only ever a SELECT: the API rejects anything that writes.
+        annotations: {
+            readOnlyHint: true,
+            destructiveHint: false,
+            idempotentHint: true,
+            openWorldHint: true,
+        },
+    }, async ({ sql }) => {
         try {
-            const result = await searchDataSource(source, keywords, params);
-            return { content: [{ type: "text", text: JSON.stringify(result, null, 2) }] };
+            return json(await queryDataville(sql));
         }
         catch (error) {
-            const message = error instanceof Error ? error.message : String(error);
-            return { content: [{ type: "text", text: message }], isError: true };
+            return toolError(error);
         }
     });
     return server;

package/dist/sources.js

@@ -2,16 +2,165 @@
 // The `name` values must match the source names the API accepts — a mismatch
 // makes the source unreachable. To check against the live API, request an
 // unknown source (e.g. /nosuchsource/foo); the error lists every valid name.
-export const DATAVILLE_SOURCES = [
-    { name: "wikipedia", description: "Wikipedia articles" },
-    { name: "arxiv", description: "arXiv preprints" },
-    { name: "gutenberg", description: "Project Gutenberg public-domain books" },
-    { name: "census", description: "US Census Bureau data" },
-    { name: "fooddata", description: "USDA FoodData Central" },
-    { name: "paperswithcode", description: "Papers with Code — ML papers, code, and benchmarks" },
-    { name: "edgar", description: "SEC EDGAR filings" },
-    { name: "openalex", description: "OpenAlex scholarly works" },
-    { name: "pypi", description: "PyPI package metadata" },
-    { name: "stackexchange", description: "Stack Exchange Q&A" },
-    { name: "news", description: "Front-page news headlines from a historical archive (not live/current news)" },
+//
+// `sqlTables` mirrors the DuckDB views the API's /api/v1/query endpoint
+// exposes (backend services/duckdb.ts). The integration tests check each
+// table and column against the live API.
+export const DATAVILLE_SOURCE_DETAILS = [
+    {
+        name: "wikipedia",
+        description: "Wikipedia articles",
+        keywords: "An article title or topic; the closest matching English Wikipedia article is returned.",
+        exampleKeywords: "Machine learning",
+        sqlTables: [
+            {
+                name: "wiki",
+                columns: [
+                    "id", "pageid", "title", "body", "abstract", "description", "language", "url", "project",
+                    "is_seed", "categories", "license", "infoboxes", "citations", "tables", "main_entity", "image",
+                    "origin", "date_modified", "snapshot_date", "last_fetched",
+                ],
+            },
+        ],
+    },
+    {
+        name: "arxiv",
+        description: "arXiv preprints",
+        keywords: "A topic, title, or arXiv ID.",
+        exampleKeywords: "transformer",
+        sqlTables: [
+            {
+                name: "arxiv",
+                columns: [
+                    "id", "arxiv_id", "title", "authors", "categories", "published", "abs_url", "pdf_url", "body",
+                    "last_updated", "last_fetched",
+                ],
+            },
+        ],
+    },
+    {
+        name: "gutenberg",
+        description: "Project Gutenberg public-domain books",
+        keywords: "A book title, author, or subject.",
+        exampleKeywords: "Alice",
+        sqlTables: [
+            {
+                name: "gutenberg",
+                columns: ["id", "title", "authors", "language", "subjects", "issued", "locc", "bookshelves", "type"],
+            },
+        ],
+    },
+    {
+        name: "census",
+        description: "US Census Bureau data",
+        keywords: "A statistic and a place, e.g. a measure such as population or median household income plus a state or county.",
+        exampleKeywords: "population",
+        sqlTables: [],
+    },
+    {
+        name: "fooddata",
+        description: "USDA FoodData Central",
+        keywords: "A food name, optionally with a nutrient.",
+        exampleKeywords: "apple",
+        sqlTables: [
+            { name: "fooddata", columns: ["fdc_id", "description", "category", "data_type", "nutrients"] },
+        ],
+    },
+    {
+        name: "paperswithcode",
+        description: "Papers with Code — ML papers, code, and benchmarks",
+        keywords: "An ML method or dataset name.",
+        exampleKeywords: "transformer",
+        sqlTables: [
+            {
+                name: "pwc_methods",
+                columns: [
+                    "id", "name", "full_name", "description", "introduced_year", "paper_title", "paper_url", "code_url",
+                    "categories",
+                ],
+            },
+            {
+                name: "pwc_datasets",
+                columns: [
+                    "id", "name", "full_name", "description", "url", "paper_title", "paper_url", "tasks", "subtasks",
+                    "num_papers",
+                ],
+            },
+        ],
+    },
+    {
+        name: "edgar",
+        description: "SEC EDGAR filings",
+        keywords: "A company name or stock ticker; returns its latest filing.",
+        exampleKeywords: "AAPL",
+        sqlTables: [
+            {
+                name: "edgar",
+                columns: [
+                    "id", "accession_number", "cik", "company_name", "form_type", "filing_date", "description",
+                    "document_url", "index_url", "body", "last_fetched",
+                ],
+            },
+        ],
+    },
+    {
+        name: "openalex",
+        description: "OpenAlex scholarly works",
+        keywords: "A topic, title, or DOI.",
+        exampleKeywords: "transformer",
+        sqlTables: [
+            {
+                name: "openalex",
+                columns: [
+                    "id", "openalex_id", "doi", "title", "publication_year", "publication_date", "authors", "venue",
+                    "cited_by_count", "type", "url", "body", "last_fetched",
+                ],
+            },
+        ],
+    },
+    {
+        name: "pypi",
+        description: "PyPI package metadata",
+        keywords: "An exact package name.",
+        exampleKeywords: "requests",
+        sqlTables: [
+            {
+                name: "pypi",
+                columns: [
+                    "id", "name", "version", "summary", "author", "license", "home_page", "requires_python", "keywords",
+                    "body", "last_fetched",
+                ],
+            },
+        ],
+    },
+    {
+        name: "stackexchange",
+        description: "Stack Exchange Q&A",
+        keywords: "A programming question or topic (searches Stack Overflow).",
+        exampleKeywords: "python",
+        sqlTables: [
+            {
+                name: "stackexchange",
+                columns: [
+                    "id", "question_id", "title", "link", "score", "tags", "owner", "site", "is_answered", "answer_count",
+                    "view_count", "creation_date", "body", "last_fetched",
+                ],
+            },
+        ],
+    },
+    {
+        name: "news",
+        description: "Front-page news headlines from a historical archive (not live/current news)",
+        keywords: "A topic to find archived headlines about.",
+        exampleKeywords: "technology",
+        sqlTables: [],
+    },
 ];
+export const DATAVILLE_SOURCES = DATAVILLE_SOURCE_DETAILS.map(({ name, description }) => ({
+    name,
+    description,
+}));
+export function findSource(name) {
+    const wanted = name.trim().toLowerCase();
+    return DATAVILLE_SOURCE_DETAILS.find((s) => s.name === wanted);
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@dataville/dataville-mcp",
-  "version": "0.1.6",
+  "version": "0.1.7",
   "mcpName": "com.dataville/dataville-mcp",
   "description": "MCP server exposing Dataville's data source API as tools for MCP clients",
   "private": false,
@@ -27,12 +27,12 @@
     "test:integration": "tsx --test src/__tests__/*.itest.ts"
   },
   "dependencies": {
-    "@modelcontextprotocol/sdk": "^1.12.0",
-    "zod": "^3.24.1"
+    "@modelcontextprotocol/sdk": "^1.32.1",
+    "zod": "^4.6.5"
   },
   "devDependencies": {
-    "@types/node": "^20.14.0",
+    "@types/node": "^26.6.4",
     "tsx": "^4.19.0",
-    "typescript": "^5.5.0"
+    "typescript": "^7.0.2"
   }
 }