@nstdata-ai/mcp-server 0.0.0-stage → 1.0.2

package/CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+
+## 1.0.2
+
+- Preserve bounded page summaries and storage references when a single crawl result row exceeds the output limit; separate continuation cursors from unread page results.
+- Describe reference inputs, offset units, page bounds and cursor recovery in tool schemas.
+- Default Proxy checks to `https://example.com/`, with an optional target override and no requests during startup.
+- Add credential-free `--help` and `--version` commands; reject unsupported arguments without printing their values.
+
+## 1.0.1
+
+- Provide six Crawl tools and two HTTP/HTTPS Proxy tools in one local stdio server, with independently enabled modules.
+- Support sync/async single-page scraping, bounded site submission, task status, cursor pages and Unicode/binary reference chunks.
+- Prefer storage references over inline content; encode inline binary content explicitly and preserve service identifiers, empty content and truncation flags.
+- Validate inputs before submission; bound deadlines, downloads and output, propagate cancellation and close active connections at shutdown.
+- Reject dot-only task IDs in tool inputs and asynchronous submission acknowledgements.
+- Refuse redirects and automatic retries; distinguish failed tasks from unknown submission outcomes and preserve bounded recovery identifiers.
+- Filter configured credentials and withhold sensitive opaque/binary output; reject malformed UTF-8 text without changing binary bytes.
+- Reuse Crawl SDK 2.0.0 and proxy-checker 1.2.0 transport with locked archive integrity; preserve the target Host header for HTTPS requests.
+- Recognize successful Nstdata client-IP envelopes in Proxy checks without additional requests.
+- Identify invalid environment settings by variable name without exposing their values; keep programmatic configuration errors keyed by field name.
+- Include bilingual setup, tool/workflow references, executable client templates and installed CLI/type/behavior checks.
+- Provide manually dispatched npm releases with environment approval, same-archive verification and public installation checks.

package/CONTRIBUTING.md

@@ -0,0 +1,38 @@
+# Contributing
+
+Use Node.js 22.18+ in the 22.x line, or Node.js 24. Proxy certificate tests also require `openssl` on PATH. Reuse the Crawl SDK for product API calls; reuse `@nstdata-ai/proxy-checker/transport` for Proxy requests. Keep MCP output filtering and tool schemas in this adapter.
+
+```sh
+npm ci --ignore-scripts --no-audit --no-fund
+npm run typecheck
+npm test
+npm run check:package
+```
+
+For offline development, cache the complete locked dependency graph before running `npm ci --offline`. `NSTDATA_SDK_TARBALL` and `NSTDATA_CHECKER_TARBALL` can supply local SDK and transport archives to package checks; their integrity must match the lock. These inputs do not replace source dependency installation. Tests use synthetic or loopback services, not paid endpoints.
+
+## Local package installation
+
+After the checks pass, run `npm pack` to build `nstdata-ai-mcp-server-1.0.2.tgz`. Install it from a separate application directory:
+
+```sh
+npm install --ignore-scripts /path/to/nstdata-ai-mcp-server-1.0.2.tgz
+```
+
+Configure the installed package using the README template; do not point clients at source files.
+
+Cover observable tool inputs/results, failure/empty content, deadlines, cancellation, output limits, reference continuation and process cleanup. Keep stdout exclusively for MCP protocol traffic. Task creation must not gain automatic retries.
+
+Update both READMEs and both language versions of the relevant reference document when public behavior changes. Keep credentials and raw private responses out of examples, fixtures and errors. Package checks install the locked graph outside the source tree, then install the tarball without modifying that lock, verify dependency versions/integrities and validate installed tools, types, configuration templates, packaged links and documented workflows. Runtime and protocol tests also run against the installed CLI, including lifecycle and interrupted-request cases. Release and documentation-check tests run in the source suite. Markdown is actually parsed, and documented npm installation and npx startup run through an isolated loopback registry. Startup checks only initialize and discover tools; they must make no business requests. Third-party registry fixtures are repacked from the already verified installed runtime graph; this does not establish public registry availability. Checks require the complete cache prepared by source installation.
+
+Verify license ownership, dependency availability, compatibility and maintainer authorization before publishing.
+
+## Releases
+
+The `Publish npm` workflow runs only when manually dispatched from `main`, with the stable version in `package.json` and a full reviewed commit SHA. Checks produce one archive and its integrity manifest. The `npm-production` reviewer approves publication of those exact bytes. The workflow finishes when `npm publish` succeeds. Public availability may be delayed; registry integrity, installation and tool discovery are checked separately.
+
+Crawl SDK and Proxy Checker dependencies must be publicly available with the lock's exact integrity before remote CI or release. Configure [npm trusted publishing](https://docs.npmjs.com/trusted-publishers/) for this repository, `release.yml` and `npm-production`, permitting `npm publish`. Require an environment reviewer, allow only `main` and disable administrator bypass. No npm token is needed.
+
+A new npm package must exist before trusted publishing is configured. The scope owner publishes the first reviewed archive using their own npm login and two-factor authentication; use OIDC for subsequent real versions. Configure a new binding close to that release so its first-use deadline is met. Do not create placeholder packages.
+
+If publication fails or the outcome is uncertain, inspect the registry version and integrity before taking further action. Do not retry blindly or modify a published version.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 nstdata-ai
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,61 @@
-# Temporary Holding Version
+# Nstdata MCP Server
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+[中文](README_zh.md)
+
+Use Nstdata Crawl and HTTP/HTTPS proxies from an MCP client. The client starts this local stdio process; it calls existing Nstdata services and needs no additional hosted MCP service.
+
+## Choose a workflow
+
+| Your task | Guide |
+|---|---|
+| Retrieve one webpage | [Single page](docs/mcp.md#single-page) |
+| Crawl bounded website pages | [Site crawl and pagination](docs/mcp.md#site-crawl) |
+| Read referenced text or binary content | [Reference reading](docs/mcp.md#read-a-reference) |
+| Request a target through a proxy | [Proxy setup and tools](docs/proxy.md) |
+| Configure limits or investigate a failure | [Configuration](docs/mcp.md#configuration) · [Results and errors](docs/mcp.md#results-and-errors) |
+
+## Install
+
+Supports Node.js 22 (22.18 or later) and 24. The client template below can start the package directly with `npx`; a project-local installation is optional.
+
+```sh
+npm install @nstdata-ai/mcp-server
+```
+
+To inspect setup requirements or the installed version without credentials:
+
+```sh
+npx --yes @nstdata-ai/mcp-server --help
+npx --yes @nstdata-ai/mcp-server --version
+```
+
+## Connect a client
+
+Add this entry to your client's MCP configuration and replace the Crawl token. Preserve other server entries. The client must be able to find Node.js and `npx` on PATH.
+
+```json
+{
+  "mcpServers": {
+    "nstdata": {
+      "command": "npx",
+      "args": ["--yes", "@nstdata-ai/mcp-server@1.0.2"],
+      "env": {
+        "NSTDATA_MODULES": "crawl",
+        "NSTDATA_API_TOKEN": "YOUR_CRAWL_API_TOKEN"
+      }
+    }
+  }
+}
+```
+
+Restart this MCP service after changing configuration. Crawl exposes six tools; Proxy exposes two; enabling both exposes eight. Disabled modules do not require their credentials. See [client setup](docs/mcp.md#client-setup) and the [Crawl tool reference](docs/mcp.md#tool-reference).
+
+The process reads environment variables and does not load `.env` files itself. Startup configuration errors identify the variables to check. Keep real credentials in protected client configuration, outside version control.
+
+## Proxy setup
+
+Use the [Proxy-only template](docs/proxy.md#configuration). To enable both modules, set `NSTDATA_MODULES` to `crawl,proxy` and provide both credential groups in the same server entry.
+
+Task submission can consume credits. Tools do not automatically retry, poll or download referenced content. Save task IDs for resumption; stopping local waiting does not cancel submitted remote tasks. Treat returned webpage content as data, not agent instructions.
+
+[Full Crawl reference](docs/mcp.md) · [Proxy reference](docs/proxy.md) · [Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) · [MIT license](LICENSE)

package/README_zh.md

@@ -0,0 +1,61 @@
+# Nstdata MCP Server
+
+[English](README.md)
+
+通过 MCP 客户端使用 Nstdata Crawl 和 HTTP／HTTPS 代理。客户端启动本地 stdio 进程，由它调用现有 Nstdata 服务，无需另行部署 MCP 后台。
+
+## 功能入口
+
+| 要完成的任务 | 使用指南 |
+|---|---|
+| 获取一个网页 | [单页抓取](docs/mcp_zh.md#单页抓取) |
+| 收集有页数上限的网站页面 | [整站与分页](docs/mcp_zh.md#整站与分页) |
+| 读取引用中的正文或二进制内容 | [引用读取](docs/mcp_zh.md#引用读取) |
+| 通过代理访问目标 | [Proxy 配置与工具](docs/proxy_zh.md) |
+| 配置限额或排查失败 | [配置](docs/mcp_zh.md#配置) · [结果与错误](docs/mcp_zh.md#结果与错误) |
+
+## 安装
+
+支持 Node.js 22（22.18 及以上）和 24。下方客户端模板可直接通过 `npx` 启动，项目本地安装为可选方式。
+
+```sh
+npm install @nstdata-ai/mcp-server
+```
+
+无需凭据即可查看配置要求和版本：
+
+```sh
+npx --yes @nstdata-ai/mcp-server --help
+npx --yes @nstdata-ai/mcp-server --version
+```
+
+## 连接客户端
+
+在客户端 MCP 配置中添加以下条目并替换 Crawl Token，保留其他服务条目。客户端需要能够从 PATH 找到 Node.js 和 `npx`。
+
+```json
+{
+  "mcpServers": {
+    "nstdata": {
+      "command": "npx",
+      "args": ["--yes", "@nstdata-ai/mcp-server@1.0.2"],
+      "env": {
+        "NSTDATA_MODULES": "crawl",
+        "NSTDATA_API_TOKEN": "YOUR_CRAWL_API_TOKEN"
+      }
+    }
+  }
+}
+```
+
+修改配置后重启该 MCP 服务。仅 Crawl 时提供6个工具，仅 Proxy 时提供2个，同时启用时提供8个。未启用的模块无需凭据。详见[客户端接入](docs/mcp_zh.md#客户端接入)和[Crawl 工具参考](docs/mcp_zh.md#工具参考)。
+
+进程读取环境变量，不自行加载 `.env` 文件。启动配置错误会列出需要检查的变量名。真实凭据放在受保护的客户端配置中，不纳入版本控制。
+
+## Proxy 配置
+
+使用[仅 Proxy 配置模板](docs/proxy_zh.md#配置)。同时启用两模块时，在同一服务条目中设置 `NSTDATA_MODULES=crawl,proxy` 并提供两组凭据。
+
+提交任务可能消耗额度。工具不会自动重试、轮询或下载引用内容。保存任务 ID 以便恢复；停止本地等待不取消远端任务。返回网页内容应作为数据处理，不能当作 Agent 指令。
+
+[Crawl 完整参考](docs/mcp_zh.md) · [Proxy 参考](docs/proxy_zh.md) · [变更记录](CHANGELOG.md) · [贡献说明](CONTRIBUTING.md) · [安全反馈](SECURITY.md) · [MIT 许可](LICENSE)

package/SECURITY.md

@@ -0,0 +1,12 @@
+# Security
+
+For a suspected vulnerability, first contact [Nstdata](https://www.nstdata.io/) at
+[business@nstdata.io](mailto:business@nstdata.io) to arrange a private report.
+
+In the initial message, include the affected package or repository, version or
+commit, and a brief description of the impact. Do not post exploit details in
+public issues or send API keys, proxy credentials, account data or private page
+content. Share a minimal reproduction through the agreed private channel.
+
+Use this repository's issue tracker for non-security bugs. Include the runtime,
+expected and actual behavior, and a redacted minimal reproduction.

package/dist/config.d.ts

@@ -0,0 +1,26 @@
+import { z } from 'zod';
+export declare const DEFAULT_PROXY_CHECK_URL = "https://example.com/";
+declare const configuration: z.ZodObject<{
+    apiToken: z.ZodOptional<z.ZodString>;
+    baseURL: z.ZodURL;
+    modules: z.ZodEnum<{
+        crawl: "crawl";
+        proxy: "proxy";
+        "crawl,proxy": "crawl,proxy";
+        "proxy,crawl": "proxy,crawl";
+    }>;
+    proxyURL: z.ZodOptional<z.ZodURL>;
+    proxyCheckURL: z.ZodOptional<z.ZodURL>;
+    proxyCAFile: z.ZodOptional<z.ZodString>;
+    httpTimeoutMs: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
+    maxResponseBytes: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
+    maxOutputChars: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
+    maxCrawlPages: z.ZodDefault<z.ZodCoercedNumber<unknown>>;
+}, z.core.$strict>;
+export type Config = z.infer<typeof configuration>;
+export declare class EnvironmentConfigError extends Error {
+    constructor(fields: string[]);
+}
+export declare function validateConfig(input: unknown): Config;
+export declare function loadConfig(env?: NodeJS.ProcessEnv): Config;
+export {};

package/dist/config.js

@@ -0,0 +1,94 @@
+import { z } from 'zod';
+import { isAbsolute } from 'node:path';
+export const DEFAULT_PROXY_CHECK_URL = 'https://example.com/';
+const configuration = z.strictObject({
+    apiToken: z.string().trim().min(1).optional(),
+    baseURL: z.url().refine(value => {
+        try {
+            const url = new URL(value);
+            return ['http:', 'https:'].includes(url.protocol) && !url.username && !url.password && !url.search && !url.hash;
+        }
+        catch {
+            return false;
+        }
+    }),
+    modules: z.enum(['crawl', 'proxy', 'crawl,proxy', 'proxy,crawl']),
+    proxyURL: z.url().max(8192).refine(value => {
+        try {
+            const url = new URL(value);
+            decodeURIComponent(url.username);
+            decodeURIComponent(url.password);
+            return ['http:', 'https:'].includes(url.protocol) && url.pathname === '/' && !url.search && !url.hash;
+        }
+        catch {
+            return false;
+        }
+    }).optional(),
+    proxyCheckURL: z.url().max(8192).refine(value => {
+        try {
+            const url = new URL(value);
+            return ['http:', 'https:'].includes(url.protocol) && !url.username && !url.password;
+        }
+        catch {
+            return false;
+        }
+    }).optional(),
+    proxyCAFile: z.string().min(1).max(4096).refine(isAbsolute).optional(),
+    httpTimeoutMs: z.coerce.number().int().min(1).max(600000).default(120000),
+    maxResponseBytes: z.coerce.number().int().min(256).max(67108864).default(8388608),
+    maxOutputChars: z.coerce.number().int().min(1024).max(64000).default(16000),
+    maxCrawlPages: z.coerce.number().int().min(1).max(1000).default(10),
+}).superRefine((config, ctx) => {
+    if (config.modules.includes('crawl') && !config.apiToken)
+        ctx.addIssue({ code: 'custom', path: ['apiToken'], message: 'Required for crawl' });
+    if (config.modules.includes('proxy')) {
+        if (!config.proxyURL)
+            ctx.addIssue({ code: 'custom', path: ['proxyURL'], message: 'Required for proxy' });
+    }
+});
+const environmentVariables = {
+    apiToken: 'NSTDATA_API_TOKEN',
+    baseURL: 'NSTDATA_API_BASE_URL',
+    modules: 'NSTDATA_MODULES',
+    proxyURL: 'NSTDATA_PROXY_URL',
+    proxyCheckURL: 'NSTDATA_PROXY_CHECK_URL',
+    proxyCAFile: 'NSTDATA_PROXY_CA_FILE',
+    httpTimeoutMs: 'NSTDATA_HTTP_TIMEOUT_MS',
+    maxResponseBytes: 'NSTDATA_MAX_RESPONSE_BYTES',
+    maxOutputChars: 'NSTDATA_MAX_OUTPUT_CHARS',
+    maxCrawlPages: 'NSTDATA_MAX_CRAWL_PAGES',
+};
+// Only this error's allowlisted variable names are safe for CLI diagnostics.
+export class EnvironmentConfigError extends Error {
+    constructor(fields) {
+        super(`Invalid configuration: ${fields.map(field => environmentVariables[field] ?? 'environment').join(', ')}`);
+    }
+}
+function parseConfig(input, fromEnvironment) {
+    const result = configuration.safeParse(input);
+    if (!result.success) {
+        const fields = [...new Set(result.error.issues.map(issue => issue.path.join('.')))];
+        if (fromEnvironment)
+            throw new EnvironmentConfigError(fields);
+        throw new Error(`Invalid configuration: ${fields.join(', ')}`);
+    }
+    return { ...result.data, proxyCheckURL: result.data.proxyCheckURL ?? DEFAULT_PROXY_CHECK_URL };
+}
+export function validateConfig(input) {
+    return parseConfig(input, false);
+}
+export function loadConfig(env = process.env) {
+    const modules = env.NSTDATA_MODULES ?? 'crawl';
+    return parseConfig({
+        apiToken: modules.includes('crawl') ? env.NSTDATA_API_TOKEN : undefined,
+        baseURL: modules.includes('crawl') ? env.NSTDATA_API_BASE_URL ?? 'https://api.nstdata.io' : 'https://api.nstdata.io',
+        modules,
+        proxyURL: modules.includes('proxy') ? env.NSTDATA_PROXY_URL : undefined,
+        proxyCheckURL: modules.includes('proxy') ? env.NSTDATA_PROXY_CHECK_URL : undefined,
+        proxyCAFile: modules.includes('proxy') ? env.NSTDATA_PROXY_CA_FILE : undefined,
+        httpTimeoutMs: env.NSTDATA_HTTP_TIMEOUT_MS,
+        maxResponseBytes: env.NSTDATA_MAX_RESPONSE_BYTES,
+        maxOutputChars: env.NSTDATA_MAX_OUTPUT_CHARS,
+        maxCrawlPages: modules.includes('crawl') ? env.NSTDATA_MAX_CRAWL_PAGES : undefined,
+    }, true);
+}

package/dist/modules/crawl-result.d.ts

@@ -0,0 +1,12 @@
+import { type ScrapeDataDto, type RequestOptions } from '@nstdata-ai/crawl';
+export declare function taskFailed(result: {
+    status?: string;
+    success?: boolean;
+}): boolean;
+export declare function requireSubmissionId(result: {
+    id?: string;
+    status?: string;
+    success?: boolean;
+}): void;
+/** References stay lazy; only read_result downloads storage content. */
+export declare function serializeData(data: ScrapeDataDto | null | undefined, options: RequestOptions): Promise<Record<string, unknown> | undefined>;

package/dist/modules/crawl-result.js

@@ -0,0 +1,28 @@
+import { NstDataApiError } from '@nstdata-ai/crawl';
+import { binaryContent } from '../shared/output.js';
+import { identifier } from './crawl-schema.js';
+export function taskFailed(result) {
+    return result.success === false || result.status === 'failed' || result.status === 'expired';
+}
+export function requireSubmissionId(result) {
+    if (taskFailed(result))
+        return;
+    const parsed = identifier.safeParse(result.id);
+    if (!parsed.success || parsed.data !== result.id)
+        throw new NstDataApiError('API error: unusable submission ID');
+}
+/** References stay lazy; only read_result downloads storage content. */
+export async function serializeData(data, options) {
+    if (!data)
+        return undefined;
+    return {
+        markdown: data.markdownRef ? undefined : await data.getMarkdown(options),
+        html: data.htmlRef ? undefined : await data.getHtml(options),
+        pdf: data.pdfRef ? undefined : binaryContent(await data.getPdf(options)),
+        screenshot: data.screenshotRef ? undefined : binaryContent(await data.getScreenshot(options)),
+        rawData: data.rawDataRef ? undefined : binaryContent(await data.getRawData(options)),
+        links: data.links, metadata: data.metadata, truncated: data.truncated,
+        markdownRef: data.markdownRef, htmlRef: data.htmlRef, pdfRef: data.pdfRef,
+        screenshotRef: data.screenshotRef, rawDataRef: data.rawDataRef,
+    };
+}

package/dist/modules/crawl-schema.d.ts

@@ -0,0 +1,74 @@
+import { z } from 'zod';
+export declare const httpURL: z.ZodURL;
+export declare const identifier: z.ZodString;
+export declare const formats: z.ZodDefault<z.ZodArray<z.ZodEnum<{
+    markdown: "markdown";
+    html: "html";
+    pdf: "pdf";
+    screenshot: "screenshot";
+    rawData: "rawData";
+    links: "links";
+}>>>;
+export declare const scrapeOptions: z.ZodDefault<z.ZodObject<{
+    cookies: z.ZodOptional<z.ZodArray<z.ZodObject<{
+        domain: z.ZodOptional<z.ZodString>;
+        httpOnly: z.ZodOptional<z.ZodBoolean>;
+        name: z.ZodString;
+        path: z.ZodOptional<z.ZodString>;
+        secure: z.ZodOptional<z.ZodBoolean>;
+        value: z.ZodString;
+    }, z.core.$strict>>>;
+    proxy: z.ZodOptional<z.ZodObject<{
+        country: z.ZodString;
+    }, z.core.$strict>>;
+    headers: z.ZodOptional<z.ZodRecord<z.ZodString, z.ZodString>>;
+    excludeTags: z.ZodOptional<z.ZodArray<z.ZodString>>;
+    includeTags: z.ZodOptional<z.ZodArray<z.ZodString>>;
+    jsRender: z.ZodOptional<z.ZodObject<{
+        actions: z.ZodOptional<z.ZodArray<z.ZodDiscriminatedUnion<[z.ZodObject<{
+            type: z.ZodLiteral<"click">;
+            selector: z.ZodString;
+            waitSelector: z.ZodOptional<z.ZodString>;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"wait">;
+            milliseconds: z.ZodNumber;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"waitSelector">;
+            selector: z.ZodString;
+            milliseconds: z.ZodOptional<z.ZodNumber>;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"scrollX">;
+            x: z.ZodNumber;
+            text: z.ZodOptional<z.ZodString>;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"scrollY">;
+            y: z.ZodNumber;
+            text: z.ZodOptional<z.ZodString>;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"scrollTo">;
+            selector: z.ZodString;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"fill">;
+            selector: z.ZodString;
+            text: z.ZodString;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"execute">;
+            execute: z.ZodString;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"screenshot">;
+            fullScreenshot: z.ZodOptional<z.ZodBoolean>;
+            particularScreenshot: z.ZodOptional<z.ZodString>;
+        }, z.core.$strict>, z.ZodObject<{
+            type: z.ZodLiteral<"waitForRequestCompletion">;
+            urlPattern: z.ZodString;
+            milliseconds: z.ZodOptional<z.ZodNumber>;
+        }, z.core.$strict>], "type">>>;
+        waitFor: z.ZodOptional<z.ZodNumber>;
+        waitForSelector: z.ZodOptional<z.ZodString>;
+    }, z.core.$strict>>;
+    mobile: z.ZodOptional<z.ZodBoolean>;
+    onlyMainContent: z.ZodOptional<z.ZodBoolean>;
+    removeSelectors: z.ZodOptional<z.ZodArray<z.ZodString>>;
+    targetSelector: z.ZodOptional<z.ZodString>;
+    timeout: z.ZodOptional<z.ZodNumber>;
+}, z.core.$strict>>;

package/dist/modules/crawl-schema.js

@@ -0,0 +1,37 @@
+import { z } from 'zod';
+const text = z.string().min(1).max(8192);
+const selector = text;
+const milliseconds = z.number().int().min(0).max(600000);
+export const httpURL = z.url().max(8192).refine(value => {
+    const url = new URL(value);
+    return ['http:', 'https:'].includes(url.protocol) && !url.username && !url.password;
+}, 'Use an HTTP(S) URL without embedded credentials').describe('Target HTTP(S) URL without embedded credentials. For a site crawl this is the single seed URL.');
+export const identifier = z.string().trim().min(1).max(256)
+    .refine(value => value !== '.' && value !== '..', 'Task IDs must not be . or ..');
+export const formats = z.array(z.enum(['markdown', 'html', 'links', 'pdf', 'screenshot', 'rawData'])).min(1).max(6).default(['markdown']).describe('Requested output formats. Large results may contain storage references instead of inline content.');
+const action = z.discriminatedUnion('type', [
+    z.strictObject({ type: z.literal('click'), selector, waitSelector: selector.optional() }),
+    z.strictObject({ type: z.literal('wait'), milliseconds }),
+    z.strictObject({ type: z.literal('waitSelector'), selector, milliseconds: milliseconds.optional() }),
+    z.strictObject({ type: z.literal('scrollX'), x: z.number().int(), text: text.optional() }),
+    z.strictObject({ type: z.literal('scrollY'), y: z.number().int(), text: text.optional() }),
+    z.strictObject({ type: z.literal('scrollTo'), selector }),
+    z.strictObject({ type: z.literal('fill'), selector, text: z.string().max(8192) }),
+    z.strictObject({ type: z.literal('execute'), execute: text }),
+    z.strictObject({ type: z.literal('screenshot'), fullScreenshot: z.boolean().optional(), particularScreenshot: selector.optional() }),
+    z.strictObject({ type: z.literal('waitForRequestCompletion'), urlPattern: text, milliseconds: milliseconds.optional() }),
+]);
+export const scrapeOptions = z.strictObject({
+    cookies: z.array(z.strictObject({ domain: text.optional(), httpOnly: z.boolean().optional(), name: text,
+        path: text.optional(), secure: z.boolean().optional(), value: z.string().max(8192) })).max(100).optional(),
+    proxy: z.strictObject({ country: z.string().regex(/^[a-zA-Z]{2}$/) }).optional(),
+    headers: z.record(z.string().min(1).max(256), z.string().max(8192)).refine(value => Object.keys(value).length <= 100).optional(),
+    excludeTags: z.array(text).max(100).optional(),
+    includeTags: z.array(text).max(100).optional(),
+    jsRender: z.strictObject({ actions: z.array(action).max(100).optional(), waitFor: z.number().int().min(1000).max(30000).optional(), waitForSelector: selector.optional() }).optional(),
+    mobile: z.boolean().optional(),
+    onlyMainContent: z.boolean().optional(),
+    removeSelectors: z.array(selector).max(100).optional(),
+    targetSelector: selector.optional(),
+    timeout: z.number().int().min(1000).max(60000).optional(),
+}).default({}).describe('Optional Crawl service settings. Set onlyMainContent=false when full-page navigation is needed for link discovery.');

package/dist/modules/crawl.d.ts

@@ -0,0 +1,4 @@
+import { McpServer } from '@modelcontextprotocol/server';
+import { NstDataClient } from '@nstdata-ai/crawl';
+import { type Config } from '../config.js';
+export declare function registerCrawlTools(server: McpServer, client: NstDataClient, config: Config): void;

package/dist/modules/crawl.js

@@ -0,0 +1,102 @@
+import { z } from 'zod';
+import { isUtf8 } from 'node:buffer';
+import { guard, ToolFailure } from '../shared/errors.js';
+import { httpURL, identifier, formats, scrapeOptions } from './crawl-schema.js';
+import { serializeData, taskFailed, requireSubmissionId } from './crawl-result.js';
+export function registerCrawlTools(server, client, config) {
+    server.registerTool('nstdata_crawl_read_result', {
+        description: 'Read a bounded chunk of a Crawl storage reference. Text offsets count Unicode code points; binary offsets count bytes. Each call downloads the reference again, within the response byte limit.',
+        inputSchema: z.strictObject({
+            ref: z.string().min(1).max(4096).describe('Copy the actual value of markdownRef, htmlRef, pdfRef, screenshotRef or rawDataRef into ref; do not use the output field name as an input key.'),
+            format: z.enum(['markdown', 'html', 'pdf', 'screenshot', 'rawData']).describe('Must match the reference field: markdownRef uses markdown, htmlRef uses html, and so on.'),
+            offset: z.number().int().min(0).default(0).describe('Start position: Unicode code points for markdown/html, bytes for binary formats. Use the preceding read result\'s nextOffset to continue.'),
+            length: z.number().int().min(1).max(4000).default(2000).describe('Maximum Unicode code points for markdown/html, or bytes for binary formats, in this chunk. Reduce it if the chunk exceeds the output limit.'),
+        }),
+        annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: true },
+    }, async (args, ctx) => guard(ctx, config, async (options) => {
+        const bytes = Buffer.from(await client.readStorage(args.ref, options));
+        const isText = args.format === 'markdown' || args.format === 'html';
+        if (isText && !isUtf8(bytes))
+            throw new ToolFailure('INVALID_RESPONSE', 'The reference is not valid UTF-8. No text was returned.');
+        const points = isText ? Array.from(bytes.toString('utf8')) : undefined;
+        const total = points?.length ?? bytes.length;
+        const end = Math.min(args.offset + args.length, total);
+        const output = { format: args.format, encoding: isText ? 'utf-8' : 'base64', unit: isText ? 'codePoints' : 'bytes', offset: args.offset,
+            ...(points ? { text: points.slice(args.offset, end).join('') } : { base64: bytes.subarray(args.offset, end).toString('base64') }),
+            total, nextOffset: end < total ? end : undefined };
+        return { content: [{ type: 'text', text: JSON.stringify(output) }], structuredContent: output };
+    }));
+    server.registerTool('nstdata_crawl_start', {
+        description: 'Start a bounded site crawl and return its task ID immediately.',
+        inputSchema: z.strictObject({
+            url: httpURL, formats, options: scrapeOptions,
+            maxPages: z.number().int().min(1).max(config.maxCrawlPages).default(config.maxCrawlPages).describe('Page-count upper bound, not a guarantee of that many pages. Check actual returned URLs and per-page results.'),
+            maxDepth: z.number().int().min(0).max(100).optional().describe('Optional service-side link-discovery depth limit; depth counting follows the Crawl service.'),
+            includeUrls: z.array(z.string().max(8192)).max(100).optional().describe('Optional service-side include filters for discovered links. These are not extra seed URLs or a batch scrape list.'),
+            excludeUrls: z.array(z.string().max(8192)).max(100).optional().describe('Optional service-side exclude filters for discovered links.'),
+            ignoreQuery: z.boolean().optional().describe('Optional service-side query-string handling setting.'),
+        }),
+        annotations: { readOnlyHint: false, destructiveHint: true, idempotentHint: false, openWorldHint: true },
+    }, async (args, ctx) => guard(ctx, config, async (options) => {
+        const { options: advanced, ...request } = args;
+        const output = await client.submitCrawlTask({ ...request, ...advanced }, options);
+        requireSubmissionId(output);
+        return { content: [{ type: 'text', text: JSON.stringify(output) }], structuredContent: { ...output }, isError: taskFailed(output) };
+    }, 'task'));
+    server.registerTool('nstdata_crawl_status', {
+        description: 'Read aggregate crawl progress without polling.',
+        inputSchema: z.strictObject({ crawlId: identifier.describe('Actual id returned by nstdata_crawl_start; reuse the existing task instead of submitting another crawl.') }),
+        annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: true },
+    }, async (args, ctx) => guard(ctx, config, async (options) => {
+        const output = { ...await client.getCrawlStatus(args.crawlId, options), crawlId: args.crawlId };
+        return { content: [{ type: 'text', text: JSON.stringify(output) }], structuredContent: { ...output }, isError: taskFailed(output) };
+    }));
+    server.registerTool('nstdata_crawl_pages', {
+        description: 'Read one page of crawl results without downloading referenced files. On OUTPUT_LIMIT, retry the same input cursor with a smaller limit. A single-row error may retain recovery.page references; handle that page before using recovery.continuation.',
+        inputSchema: z.strictObject({
+            crawlId: identifier.describe('Actual id of the existing site crawl.'),
+            afterCursor: z.string().min(1).max(8192).optional().describe('Copy nextCursor from a successful page read. On OUTPUT_LIMIT, keep the original input cursor. Only use recovery.continuation.nextCursor after handling recovery.page. Mutually exclusive with beforeCursor.'),
+            beforeCursor: z.string().min(1).max(8192).optional().describe('Copy prevCursor from a successful read, or recovery.continuation.prevCursor after handling recovery.page. On OUTPUT_LIMIT keep the original input cursor; do not combine with afterCursor.'),
+            limit: z.number().int().min(1).max(100).default(10).describe('Maximum rows in this response. Reduce down to 1 on OUTPUT_LIMIT; a single row may still exceed the output cap.'),
+        }).refine(args => !(args.afterCursor && args.beforeCursor)),
+        annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: true },
+    }, async ({ crawlId, ...pageOptions }, ctx) => guard(ctx, config, async (options) => {
+        const result = await client.listCrawlPages(crawlId, pageOptions, options);
+        const output = { ...result, crawlId, data: result.data && await Promise.all(result.data.map(async (page) => ({
+                ...page, data: await serializeData(page.data, options),
+            }))) };
+        return { content: [{ type: 'text', text: JSON.stringify(output) }], structuredContent: output };
+    }));
+    server.registerTool('nstdata_crawl_scrape', {
+        description: 'Retrieve a webpage through Nstdata Crawl. Creates a potentially billable scrape task.',
+        inputSchema: z.strictObject({
+            url: httpURL, formats, options: scrapeOptions,
+            mode: z.enum(['sync', 'async']).default('sync').describe('sync waits for the single-page response; async returns a task id to resume with nstdata_crawl_scrape_result. Neither mode retries submission.'),
+        }),
+        annotations: { readOnlyHint: false, destructiveHint: true, idempotentHint: false, openWorldHint: true },
+    }, async (args, ctx) => guard(ctx, config, async (options) => {
+        const { mode, options: advanced, ...common } = args;
+        const request = { ...common, ...advanced };
+        if (mode === 'async') {
+            const submitted = await client.submitScrapeTask(request, options);
+            requireSubmissionId(submitted);
+            return { content: [{ type: 'text', text: JSON.stringify(submitted) }], structuredContent: { ...submitted }, isError: taskFailed(submitted) };
+        }
+        const result = await client.submitScrapeTaskSync(request, options);
+        const output = { id: result.id, status: result.status, success: result.success, errorCode: result.errorCode, errorMessage: result.errorMessage, data: await serializeData(result.data, options) };
+        return { content: [{ type: 'text', text: JSON.stringify(output) }], structuredContent: output, isError: taskFailed(result) };
+    }, 'task'));
+    server.registerTool('nstdata_crawl_scrape_result', {
+        description: 'Resume a submitted single-page task. Processing is not completion; a failed or expired task is reported explicitly.',
+        inputSchema: z.strictObject({ taskId: identifier.describe('Actual id returned by an existing single-page submission; not a site crawl id or a storage reference.') }),
+        annotations: { readOnlyHint: true, idempotentHint: true, openWorldHint: true },
+    }, async (args, ctx) => guard(ctx, config, async (options) => {
+        const result = await client.getScrapeTaskResult(args.taskId, options);
+        const output = { id: result.id, taskId: args.taskId, status: result.status, success: result.success,
+            errorCode: result.errorCode, errorMessage: result.errorMessage,
+            data: await serializeData(result.data, options),
+        };
+        return { content: [{ type: 'text', text: JSON.stringify(output) }], structuredContent: output,
+            isError: taskFailed(result) };
+    }));
+}

package/dist/modules/proxy.d.ts

@@ -0,0 +1,3 @@
+import { type McpServer } from '@modelcontextprotocol/server';
+import { type Config } from '../config.js';
+export declare function registerProxyTools(server: McpServer, config: Config): void;