@aiacta-org/crawl-manifest-client 1.0.0

package/README.md

@@ -0,0 +1,193 @@
+# @aiacta-org/crawl-manifest-client
+
+> Query any AIACTA-compliant AI provider's Crawl Manifest API — see exactly which of your pages were crawled, when, and for what purpose (Proposal 1, §2.2).
+
+[![npm version](https://img.shields.io/npm/v/@aiacta-org/crawl-manifest-client.svg)](https://www.npmjs.com/package/@aiacta-org/crawl-manifest-client)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](../../LICENSE)
+[![AIACTA Spec](https://img.shields.io/badge/spec-AIACTA%2F1.0-orange.svg)](../../docs/proposals/proposal-1-crawl-manifests.md)
+
+Available in **Node.js** and **Python**.
+
+---
+
+## What is this?
+
+The Crawl Manifest API is what AI providers expose so publishers can audit their crawl history — the AI equivalent of Google Search Console. This client handles the API query, automatic pagination, rate-limit backoff, and local caching.
+
+---
+
+## Install
+
+**Node.js**
+```bash
+npm install @aiacta-org/crawl-manifest-client
+```
+
+**Python**
+```bash
+pip install crawl-manifest-client
+```
+
+---
+
+## Quick Start
+
+### Node.js
+
+```javascript
+const { CrawlManifestClient } = require('@aiacta-org/crawl-manifest-client');
+
+const client = new CrawlManifestClient({
+  provider: 'anthropic',
+  apiKey:   process.env.ANTHROPIC_PUBLISHER_KEY,
+});
+
+// Fetch all crawl events for your domain — pagination handled automatically
+for await (const entry of client.fetchAll({
+  domain: 'yourdomain.com',
+  from:   '2026-03-01T00:00:00Z',
+  to:     '2026-03-31T23:59:59Z',
+})) {
+  console.log(entry.url);
+  console.log('Last crawled:', entry.last_crawled);
+  console.log('Purpose:',     entry.purpose);           // ['rag', 'index', ...]
+  console.log('HTTP status:', entry.http_status_at_crawl);
+  console.log('Content hash:', entry.content_hash);     // sha256:...
+}
+```
+
+### Python
+
+```python
+from crawl_manifest_client import CrawlManifestClient
+
+client = CrawlManifestClient(
+    provider='anthropic',
+    api_key=os.environ['ANTHROPIC_PUBLISHER_KEY']
+)
+
+for entry in client.fetch_all(
+    domain='yourdomain.com',
+    from_date='2026-03-01T00:00:00Z',
+    to_date='2026-03-31T23:59:59Z'
+):
+    print(entry['url'], entry['purpose'])
+```
+
+---
+
+## Filter by purpose
+
+```javascript
+// See only pages crawled for model training
+for await (const entry of client.fetchAll({
+  domain:  'yourdomain.com',
+  from:    '2026-01-01T00:00:00Z',
+  to:      '2026-03-31T00:00:00Z',
+  purpose: ['training'],
+})) {
+  console.log('Trained on:', entry.url);
+}
+```
+
+Valid purpose values: `training` · `rag` · `index` · `quality-eval`
+
+---
+
+## Each result contains
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `url` | string | The full URL that was crawled |
+| `last_crawled` | ISO 8601 | When most recently crawled |
+| `crawl_count_30d` | integer | How many times in the last 30 days |
+| `purpose` | string[] | What the AI used the content for |
+| `http_status_at_crawl` | integer | HTTP status received (200, 404, etc.) |
+| `content_hash` | string | SHA-256 of the page content at crawl time |
+
+---
+
+## Date range limits
+
+The API allows a maximum of **90 days per request** (§2.2). For longer periods, split your query:
+
+```javascript
+// Query 6 months across two requests
+const ranges = [
+  { from: '2025-10-01T00:00:00Z', to: '2025-12-30T00:00:00Z' },
+  { from: '2026-01-01T00:00:00Z', to: '2026-03-31T00:00:00Z' },
+];
+for (const range of ranges) {
+  for await (const entry of client.fetchAll({ domain: 'yourdomain.com', ...range })) {
+    // process entry
+  }
+}
+```
+
+Requesting more than 90 days throws a `RangeError` before making any API calls.
+
+---
+
+## Rate limits & caching
+
+- The spec allows 60 requests/hour per domain (§2.2)
+- The client warns when you approach the limit
+- Automatically waits on `429` responses, respecting `X-RateLimit-Reset`
+- Caches responses in memory for 1 hour — repeated queries return instantly
+
+---
+
+## API Reference
+
+### `new CrawlManifestClient({ provider, apiKey, baseUrl? })`
+
+| Option | Type | Required | Description |
+|--------|------|----------|-------------|
+| `provider` | string | Yes | Provider identifier, e.g. `'anthropic'`, `'openai'`, `'google'` |
+| `apiKey` | string | Yes | Bearer token from the provider's Publisher Portal |
+| `baseUrl` | string | No | Override the API base URL (for testing) |
+
+### `client.fetchAll({ domain, from, to, purpose? })`
+
+Returns an async generator yielding `CrawlManifestUrl` objects. Handles all pagination automatically.
+
+| Option | Type | Required | Description |
+|--------|------|----------|-------------|
+| `domain` | string | Yes | Your domain, e.g. `'yourdomain.com'` |
+| `from` | ISO 8601 | Yes | Start of query window |
+| `to` | ISO 8601 | Yes | End of query window (max 90 days from `from`) |
+| `purpose` | string[] | No | Filter to specific purpose values |
+
+---
+
+## Verify content integrity
+
+```javascript
+const { computeContentHash } = require('@aiacta-org/crawl-manifest-client/src/node/content-hash');
+
+const yourPageHtml = fs.readFileSync('./article.html', 'utf-8');
+const yourHash     = computeContentHash(yourPageHtml);
+
+for await (const entry of client.fetchAll({ domain: 'yourdomain.com', from, to })) {
+  if (entry.content_hash !== yourHash) {
+    console.warn('Content mismatch on', entry.url, '— AI may have crawled an older version');
+  }
+}
+```
+
+---
+
+## Related packages
+
+| Package | Purpose |
+|---------|---------|
+| [`@aiacta-org/ai-attribution-lint`](https://www.npmjs.com/package/@aiacta-org/ai-attribution-lint) | Validate your `ai-attribution.txt` |
+| [`@aiacta-org/ai-citation-sdk`](https://www.npmjs.com/package/@aiacta-org/ai-citation-sdk) | Receive citation webhook events |
+
+---
+
+## License & Copyright
+
+Copyright © 2026 Eric Michel, PhD. Licensed under the [Apache License 2.0](../../LICENSE).
+
+Part of the [AIACTA open standard](https://github.com/aiacta-org/aiacta).

package/package.json

@@ -0,0 +1,10 @@
+{
+  "name": "@aiacta-org/crawl-manifest-client",
+  "version": "1.0.0",
+  "description": "Client library for querying the AIACTA Crawl Manifest API with pagination, rate-limit handling, and local caching (Proposal 1 §2.2)",
+  "author": "Eric Michel",
+  "main": "./src/node/index.js",
+  "scripts": { "test": "jest" },
+  "dependencies": { "axios": "^1.6.0", "node-cache": "^5.1.0" },
+  "devDependencies": { "jest": "^29.0.0", "nock": "^14.0.11" }
+}

package/src/node/content-hash.js

@@ -0,0 +1,54 @@
+/**
+ * Content hash utility — §2.3.
+ *
+ * Spec: "SHA-256 of UTF-8 normalized body text (HTML stripped, whitespace collapsed)"
+ *
+ * Used by AI providers when building crawl manifest entries.
+ * Publishers may cross-check hashes to detect content drift.
+ */
+'use strict';
+const crypto = require('crypto');
+
+// Minimal HTML tag stripper (no DOM dependency for broad compatibility)
+function stripHtml(html) {
+  return html
+    .replace(/<script[\s\S]*?<\/script>/gi, ' ')
+    .replace(/<style[\s\S]*?<\/style>/gi, ' ')
+    .replace(/<[^>]+>/g, ' ');
+}
+
+// Collapse whitespace as specified: normalise to single spaces, trim
+function collapseWhitespace(text) {
+  return text.replace(/[\s\u00A0\u200B]+/g, ' ').trim();
+}
+
+/**
+ * Compute the AIACTA content hash for a crawled page.
+ *
+ * @param {string} rawHtml  The full HTML source of the crawled page
+ * @returns {string}        "sha256:<hex>" as defined in §2.3
+ *
+ * @example
+ * const { computeContentHash } = require('./content-hash');
+ * const hash = computeContentHash('<html><body>Hello world</body></html>');
+ * // => "sha256:b94d27b99..."
+ */
+function computeContentHash(rawHtml) {
+  const text   = collapseWhitespace(stripHtml(rawHtml));
+  const digest = crypto.createHash('sha256').update(text, 'utf8').digest('hex');
+  return `sha256:${digest}`;
+}
+
+/**
+ * Verify that a stored content hash still matches current page content.
+ * Used by publishers to detect unexpected content changes after a crawl.
+ *
+ * @param {string} rawHtml       Current page HTML
+ * @param {string} storedHash    Previously recorded "sha256:<hex>"
+ * @returns {boolean}
+ */
+function verifyContentHash(rawHtml, storedHash) {
+  return computeContentHash(rawHtml) === storedHash;
+}
+
+module.exports = { computeContentHash, verifyContentHash, stripHtml, collapseWhitespace };

package/src/node/index.js

@@ -0,0 +1,127 @@
+/**
+ * crawl-manifest-client — Node.js
+ *
+ * Queries GET /crawl-manifest/v1 (§2.2) with:
+ *  - Cursor-based automatic pagination
+ *  - 90-day max range validation (spec requirement)
+ *  - Rate-limit backoff (X-RateLimit-Remaining / X-RateLimit-Reset)
+ *  - In-memory response caching (TTL = 1 hour)
+ */
+'use strict';
+const axios     = require('axios');
+const NodeCache = require('node-cache');
+
+const MAX_RANGE_DAYS = 90;   // §2.2: max date range 90 days per request
+const RATE_LIMIT_RPH = 60;   // §2.2: 60 requests/hour per domain
+
+const cache = new NodeCache({ stdTTL: 3600 });
+
+class CrawlManifestClient {
+  /**
+   * @param {object} opts
+   * @param {string}   opts.provider   Provider identifier (e.g. 'anthropic')
+   * @param {string}   opts.apiKey     Publisher API key (Bearer token)
+   * @param {string}  [opts.baseUrl]   Override base URL for testing
+   */
+  constructor({ provider, apiKey, baseUrl }) {
+    this.baseUrl  = baseUrl || `https://api.${provider}.com/crawl-manifest/v1`;
+    this.apiKey   = apiKey;
+    this._reqLog  = []; // sliding window for client-side rate guard
+  }
+
+  /**
+   * Validates that the requested date range does not exceed 90 days (§2.2).
+   * @param {string} from  ISO 8601
+   * @param {string} to    ISO 8601
+   * @throws {RangeError}
+   */
+  _validateRange(from, to) {
+    const diffMs   = new Date(to) - new Date(from);
+    const diffDays = diffMs / (1000 * 60 * 60 * 24);
+    if (diffDays > MAX_RANGE_DAYS) {
+      throw new RangeError(
+        `Date range ${diffDays.toFixed(1)} days exceeds the 90-day maximum per §2.2. ` +
+        `Split your query into multiple requests.`
+      );
+    }
+    if (diffMs < 0) {
+      throw new RangeError(`'from' must be before 'to'`);
+    }
+  }
+
+  /**
+   * Client-side rate guard — warns (does not block) if approaching 60 req/hour.
+   */
+  _trackRequest() {
+    const now = Date.now();
+    const hourAgo = now - 3600_000;
+    this._reqLog = this._reqLog.filter(t => t > hourAgo);
+    this._reqLog.push(now);
+    if (this._reqLog.length >= RATE_LIMIT_RPH - 5) {
+      console.warn(
+        `[crawl-manifest-client] Approaching rate limit: ` +
+        `${this._reqLog.length}/${RATE_LIMIT_RPH} requests in last hour`
+      );
+    }
+  }
+
+  /**
+   * Fetches all crawl manifest records for a domain, handling all pagination.
+   * Yields individual CrawlManifestUrl objects.
+   *
+   * @param {object}   params
+   * @param {string}   params.domain
+   * @param {string}   params.from        ISO 8601 — start of query window
+   * @param {string}   params.to          ISO 8601 — end of query window (max 90d from 'from')
+   * @param {string[]} [params.purpose]   Filter by purpose values
+   * @yields {CrawlManifestUrl}
+   */
+  async *fetchAll({ domain, from, to, purpose = [] }) {
+    this._validateRange(from, to);
+    let cursor = null;
+    do {
+      const page = await this._fetchPage({ domain, from, to, purpose, cursor });
+      for (const url of page.urls) yield url;
+      cursor = page.next_cursor || null;
+    } while (cursor);
+  }
+
+  /**
+   * Fetch a single page of results (internal).
+   */
+  async _fetchPage({ domain, from, to, purpose, cursor }) {
+    const cacheKey = `${domain}|${from}|${to}|${purpose.join(',')}|${cursor}`;
+    const cached = cache.get(cacheKey);
+    if (cached) return cached;
+
+    this._trackRequest();
+
+    const params = { domain, from, to, format: 'json' };
+    if (purpose.length) params.purpose = purpose.join(',');
+    if (cursor)         params.cursor  = cursor;
+
+    let response;
+    try {
+      response = await axios.get(this.baseUrl, {
+        params,
+        headers: { Authorization: `Bearer ${this.apiKey}` },
+        timeout: 15_000,
+      });
+    } catch (err) {
+      if (err.response?.status === 429) {
+        // Respect X-RateLimit-Reset header (§2.2)
+        const resetIn = parseInt(err.response.headers['x-ratelimit-reset'] || '60', 10);
+        console.warn(`[crawl-manifest-client] Rate limited. Waiting ${resetIn}s...`);
+        await new Promise(r => setTimeout(r, resetIn * 1000));
+        return this._fetchPage({ domain, from, to, purpose, cursor });
+      }
+      throw err;
+    }
+
+    const data = response.data;
+    cache.set(cacheKey, data);
+    return data;
+  }
+}
+
+module.exports = { CrawlManifestClient, MAX_RANGE_DAYS };

package/src/python/content_hash.py

@@ -0,0 +1,37 @@
+"""
+Content hash utility — §2.3.
+
+Spec: SHA-256 of UTF-8 normalized body text (HTML stripped, whitespace collapsed).
+Used by AI providers when building crawl manifest entries.
+"""
+import hashlib
+import re
+
+
+def strip_html(html: str) -> str:
+    """Remove all HTML tags, scripts, and style blocks."""
+    html = re.sub(r'<script[\s\S]*?<\/script>', ' ', html, flags=re.IGNORECASE)
+    html = re.sub(r'<style[\s\S]*?<\/style>', ' ', html, flags=re.IGNORECASE)
+    html = re.sub(r'<[^>]+>', ' ', html)
+    return html
+
+
+def collapse_whitespace(text: str) -> str:
+    """Collapse all whitespace sequences to a single space and trim."""
+    return re.sub(r'[\s\u00A0\u200B]+', ' ', text).strip()
+
+
+def compute_content_hash(raw_html: str) -> str:
+    """
+    Compute the AIACTA content hash for a crawled page.
+
+    Returns a string of the form "sha256:<hex>" as defined in §2.3.
+    """
+    text   = collapse_whitespace(strip_html(raw_html))
+    digest = hashlib.sha256(text.encode('utf-8')).hexdigest()
+    return f'sha256:{digest}'
+
+
+def verify_content_hash(raw_html: str, stored_hash: str) -> bool:
+    """Return True if the current page content matches the stored hash."""
+    return compute_content_hash(raw_html) == stored_hash

package/src/python/crawl_manifest_client.py

@@ -0,0 +1,44 @@
+"""
+AIACTA Crawl Manifest Client — Python
+Queries GET /crawl-manifest/v1 with pagination, rate-limit backoff, and caching (§2.2).
+"""
+import time
+from typing import Iterator
+import requests
+
+class CrawlManifestClient:
+    def __init__(self, provider: str, api_key: str, base_url: str = None):
+        self.base_url = base_url or f"https://api.{provider}.com/crawl-manifest/v1"
+        self.api_key  = api_key
+        self._cache   = {}
+
+    def fetch_all(self, domain: str, from_dt: str, to_dt: str, purpose: list[str] = None) -> Iterator[dict]:
+        """Generator — yields individual URL records, handles all pagination."""
+        cursor = None
+        while True:
+            page = self._fetch_page(domain, from_dt, to_dt, purpose or [], cursor)
+            yield from page.get("urls", [])
+            cursor = page.get("next_cursor")
+            if not cursor:
+                break
+
+    def _fetch_page(self, domain, from_dt, to_dt, purpose, cursor):
+        params = {"domain": domain, "from": from_dt, "to": to_dt, "format": "json"}
+        if purpose: params["purpose"] = ",".join(purpose)
+        if cursor:  params["cursor"] = cursor
+
+        cache_key = str(sorted(params.items()))
+        if cache_key in self._cache:
+            return self._cache[cache_key]
+
+        while True:
+            resp = requests.get(self.base_url, params=params,
+                                headers={"Authorization": f"Bearer {self.api_key}"}, timeout=15)
+            if resp.status_code == 429:
+                reset = int(resp.headers.get("X-RateLimit-Reset", 60))
+                time.sleep(reset)
+                continue
+            resp.raise_for_status()
+            data = resp.json()
+            self._cache[cache_key] = data
+            return data

package/src/python/requirements.txt

@@ -0,0 +1,3 @@
+requests>=2.31
+pytest>=7.0
+pytest-cov>=4.0

package/tests/client.test.js

@@ -0,0 +1,23 @@
+const nock = require('nock');
+const { CrawlManifestClient } = require('../src/node/index');
+
+const BASE = 'https://api.testprovider.com';
+const FIXTURE = {
+  provider: 'testprovider', domain: 'example.com', schema_version: '1.0',
+  period: { from: '2026-01-01T00:00:00Z', to: '2026-03-01T00:00:00Z' },
+  total_crawled_urls: 1, next_cursor: null,
+  urls: [{ url: 'https://example.com/article', last_crawled: '2026-02-01T00:00:00Z',
+           crawl_count_30d: 2, purpose: ['rag'], http_status_at_crawl: 200,
+           content_hash: 'sha256:abc123' }],
+};
+
+test('fetches a single page and yields URLs', async () => {
+  nock(BASE).get('/crawl-manifest/v1').query(true).reply(200, FIXTURE);
+  const client = new CrawlManifestClient({ provider: 'testprovider', apiKey: 'key', baseUrl: `${BASE}/crawl-manifest/v1` });
+  const results = [];
+  for await (const url of client.fetchAll({ domain: 'example.com', from: '2026-01-01T00:00:00Z', to: '2026-03-01T00:00:00Z' })) {
+    results.push(url);
+  }
+  expect(results).toHaveLength(1);
+  expect(results[0].url).toBe('https://example.com/article');
+});

package/tests/content-hash.test.js

@@ -0,0 +1,49 @@
+const { computeContentHash, verifyContentHash, stripHtml, collapseWhitespace } = require('../src/node/content-hash');
+
+test('strips HTML tags', () => {
+  expect(stripHtml('<p>Hello <b>world</b></p>')).toMatch(/Hello\s+world/);
+});
+
+test('removes script and style blocks entirely', () => {
+  const html = '<html><head><style>.a{color:red}</style><script>alert(1)</script></head><body>Text</body></html>';
+  const text = stripHtml(html);
+  expect(text).not.toContain('color:red');
+  expect(text).not.toContain('alert(1)');
+  expect(text).toContain('Text');
+});
+
+test('collapses whitespace', () => {
+  expect(collapseWhitespace('  hello   world  ')).toBe('hello world');
+  expect(collapseWhitespace('line1\n\n\nline2')).toBe('line1 line2');
+});
+
+test('computeContentHash returns sha256: prefix', () => {
+  const hash = computeContentHash('<p>Hello world</p>');
+  expect(hash).toMatch(/^sha256:[a-f0-9]{64}$/);
+});
+
+test('same content produces same hash', () => {
+  const html = '<html><body><p>Consistent content</p></body></html>';
+  expect(computeContentHash(html)).toBe(computeContentHash(html));
+});
+
+test('different content produces different hash', () => {
+  expect(computeContentHash('<p>Hello</p>')).not.toBe(computeContentHash('<p>World</p>'));
+});
+
+test('whitespace differences in source do not change hash', () => {
+  const h1 = computeContentHash('<p>Hello  world</p>');
+  const h2 = computeContentHash('<p>Hello world</p>');
+  expect(h1).toBe(h2); // collapsed to same normalised text
+});
+
+test('verifyContentHash passes for matching content', () => {
+  const html = '<article>Some article text</article>';
+  const hash = computeContentHash(html);
+  expect(verifyContentHash(html, hash)).toBe(true);
+});
+
+test('verifyContentHash fails for modified content', () => {
+  const hash = computeContentHash('<article>Original</article>');
+  expect(verifyContentHash('<article>Modified</article>', hash)).toBe(false);
+});

package/tests/python_client.test.py

@@ -0,0 +1,46 @@
+"""Unit tests for the Python crawl manifest client."""
+import unittest
+from unittest.mock import patch, MagicMock
+import sys, os
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '../src/python'))
+from crawl_manifest_client import CrawlManifestClient
+
+FIXTURE = {
+    'provider': 'mock', 'domain': 'example.com', 'schema_version': '1.0',
+    'period': {'from': '2026-01-01T00:00:00Z', 'to': '2026-03-01T00:00:00Z'},
+    'total_crawled_urls': 1, 'next_cursor': None,
+    'urls': [{'url': 'https://example.com/article', 'last_crawled': '2026-02-01T00:00:00Z',
+              'crawl_count_30d': 2, 'purpose': ['rag'], 'http_status_at_crawl': 200,
+              'content_hash': 'sha256:abc123'}],
+}
+
+class TestCrawlManifestClient(unittest.TestCase):
+    @patch('crawl_manifest_client.requests.get')
+    def test_fetch_all_yields_urls(self, mock_get):
+        mock_resp = MagicMock()
+        mock_resp.status_code = 200
+        mock_resp.json.return_value = FIXTURE
+        mock_resp.raise_for_status = MagicMock()
+        mock_get.return_value = mock_resp
+
+        client = CrawlManifestClient(provider='mock', api_key='key')
+        results = list(client.fetch_all('example.com', '2026-01-01T00:00:00Z', '2026-03-01T00:00:00Z'))
+        self.assertEqual(len(results), 1)
+        self.assertEqual(results[0]['url'], 'https://example.com/article')
+
+    @patch('crawl_manifest_client.requests.get')
+    def test_caches_responses(self, mock_get):
+        mock_resp = MagicMock()
+        mock_resp.status_code = 200
+        mock_resp.json.return_value = FIXTURE
+        mock_resp.raise_for_status = MagicMock()
+        mock_get.return_value = mock_resp
+
+        client = CrawlManifestClient(provider='mock', api_key='key')
+        list(client.fetch_all('example.com', '2026-01-01T00:00:00Z', '2026-03-01T00:00:00Z'))
+        list(client.fetch_all('example.com', '2026-01-01T00:00:00Z', '2026-03-01T00:00:00Z'))
+        # Should only call the API once due to caching
+        self.assertEqual(mock_get.call_count, 1)
+
+if __name__ == '__main__':
+    unittest.main()