@aborruso/ckan-mcp-server 0.3.1

package/openspec/changes/archive/add-automated-tests/tasks.md

@@ -0,0 +1,132 @@
+# Tasks: Add Automated Testing
+
+Ordered list of work items to add automated tests to the project.
+
+## Phase 1: Foundation
+
+- [x] Install Vitest and related dependencies
+- [x] Create vitest.config.ts with TypeScript support
+- [x] Create test directory structure: `tests/unit/`, `tests/integration/`, `tests/fixtures/`
+- [x] Add test scripts to package.json (test, test:watch, test:coverage)
+- [x] Create .gitignore entry for coverage directory
+- [x] Configure coverage settings (threshold: 80%)
+
+## Phase 2: Create Mock Fixtures
+
+- [x] Create `tests/fixtures/responses/status-success.json`
+- [x] Create `tests/fixtures/responses/package-search-success.json`
+- [x] Create `tests/fixtures/responses/package-show-success.json`
+- [x] Create `tests/fixtures/responses/organization-list-success.json`
+- [x] Create `tests/fixtures/responses/organization-show-success.json`
+- [x] Create `tests/fixtures/responses/organization-search-success.json`
+- [x] Create `tests/fixtures/responses/datastore-search-success.json`
+- [x] Create `tests/fixtures/errors/timeout.json`
+- [x] Create `tests/fixtures/errors/not-found.json`
+- [x] Create `tests/fixtures/errors/server-error.json`
+- [x] Document fixture structure in tests/README.md
+
+## Phase 3: Unit Tests for Utils
+
+- [x] Create `tests/unit/formatting.test.ts`
+- [x] Write test for truncateText() with normal text
+- [x] Write test for truncateText() with empty string
+- [x] Write test for truncateText() with text under limit
+- [x] Write test for truncateText() with text over limit
+- [x] Write test for formatDate() with ISO date string
+- [x] Write test for formatDate() with null/undefined
+- [x] Write test for formatBytes() with bytes
+- [x] Write test for formatBytes() with kilobytes
+- [x] Write test for formatBytes() with megabytes
+- [x] Write test for formatBytes() with gigabytes
+- [x] Run tests and verify all pass
+
+- [x] Create `tests/unit/http.test.ts`
+- [x] Write test for makeCkanRequest() successful response
+- [x] Write test for makeCkanRequest() with error (404)
+- [x] Write test for makeCkanRequest() with error (500)
+- [x] Write test for makeCkanRequest() with timeout
+- [x] Write test for makeCkanRequest() URL normalization (trailing slash)
+- [x] Write test for makeCkanRequest() User-Agent header
+- [x] Run tests and verify all pass
+
+## Phase 4: Integration Tests for Tools
+
+### Status Tool
+- [x] Create `tests/integration/status.test.ts`
+- [x] Write test for ckan_status_show with valid server URL
+- [x] Write test for ckan_status_show with invalid URL
+- [ ] Write test for ckan_status_show with server error
+- [ ] Write test for markdown output format
+- [ ] Write test for JSON output format
+- [x] Run tests and verify all pass
+
+### Package Tools
+- [ ] Create `tests/integration/package.test.ts`
+- [ ] Write test for ckan_package_search with basic query
+- [ ] Write test for ckan_package_search with facets
+- [ ] Write test for ckan_package_search with pagination
+- [ ] Write test for ckan_package_search with sorting
+- [ ] Write test for ckan_package_search with filter query
+- [ ] Write test for ckan_package_show with valid package ID
+- [ ] Write test for ckan_package_show with invalid package ID
+- [ ] Write test for markdown output format
+- [ ] Write test for JSON output format
+- [ ] Run tests and verify all pass
+
+### Organization Tools
+- [ ] Create `tests/integration/organization.test.ts`
+- [ ] Write test for ckan_organization_list
+- [ ] Write test for ckan_organization_list with sorting
+- [ ] Write test for ckan_organization_show
+- [ ] Write test for ckan_organization_search
+- [ ] Write test for markdown output format
+- [ ] Write test for JSON output format
+- [ ] Run tests and verify all pass
+
+### DataStore Tool
+- [ ] Create `tests/integration/datastore.test.ts`
+- [ ] Write test for ckan_datastore_search with basic query
+- [ ] Write test for ckan_datastore_search with filters
+- [ ] Write test for ckan_datastore_search with sorting
+- [ ] Write test for ckan_datastore_search with pagination
+- [ ] Write test for markdown output format
+- [ ] Write test for JSON output format
+- [ ] Run tests and verify all pass
+
+## Phase 5: Validation and Documentation
+
+- [x] Run `npm test` and verify all tests pass
+- [x] Run `npm run test:coverage` and check coverage report
+- [ ] Verify coverage meets 80% threshold
+- [ ] Update CLAUDE.md with testing instructions
+- [ ] Update README.md with test section
+- [x] Create test writing guide in tests/README.md
+- [ ] Add example test in test writing guide
+
+## Phase 6: Continuous Integration
+
+- [ ] Check if CI/CD exists (GitHub Actions, etc.)
+- [ ] Add test step to CI pipeline if exists
+- [ ] Add coverage reporting to CI if exists
+- [ ] Verify tests run in CI environment
+- [ ] Configure coverage upload to service (optional)
+
+## Dependencies
+
+- Phase 1 must complete before Phase 2, 3, 4
+- Phase 2 must complete before Phase 4 (fixtures needed)
+- Phase 3 and Phase 4 can be done in parallel after Phase 2
+- Phase 5 depends on completion of Phase 3 and Phase 4
+- Phase 6 is optional and depends on CI infrastructure
+
+## Notes
+
+- Use vi.fn() for mocking axios
+- Keep tests focused and readable
+- Mock realistic CKAN API responses from actual documentation
+- One assertion per test when possible
+- Describe expected behavior clearly in test names
+- Use describe blocks for grouping related tests
+- Use test.only sparingly and remove before committing
+- Run tests in watch mode during development: `npm run test:watch`
+- Generate coverage report: `npm run test:coverage`

package/openspec/project.md

@@ -0,0 +1,113 @@
+# Project Context
+
+## Purpose
+CKAN MCP Server - MCP (Model Context Protocol) server for interacting with open data portals based on CKAN (dati.gov.it, data.gov, demo.ckan.org, etc.).
+
+Exposes MCP tools for:
+- Advanced dataset search with Solr syntax
+- DataStore queries for tabular data analysis
+- Organization and group exploration
+- Complete metadata access
+
+## Tech Stack
+- **TypeScript** (ES2022 target)
+- **Node.js**
+- **esbuild** - ultra-fast build system (preferred over tsc for memory efficiency)
+- **@modelcontextprotocol/sdk** - official MCP SDK
+- **axios** - HTTP client for CKAN API calls
+- **zod** - schema validation for tool inputs
+- **express** - HTTP server (for HTTP transport mode)
+
+## Project Conventions
+
+### Code Style
+- TypeScript strict mode enabled
+- Single-file architecture (src/index.ts)
+- Utility functions for reusable operations (makeCkanRequest, truncateText, formatDate, formatBytes)
+- Zod schemas for strict input validation (reject extra parameters)
+- Italian locale for date formatting (it-IT)
+- Markdown output optimized for human readability; JSON available for programmatic use
+
+### Architecture Patterns
+- **Single-file MCP server** - all logic in src/index.ts
+- **Tool-based architecture** - 7 registered MCP tools for CKAN operations
+- **Dual transport modes**:
+  - stdio (default) - for local MCP client integration
+  - HTTP - for remote access via POST /mcp endpoint
+- **Read-only operations** - no data modification on CKAN
+- **No caching** - fresh API calls for each request
+- **Error handling** - HTTP errors, timeouts, server validation
+
+### Testing Strategy
+- **No automated tests currently** (consider adding for critical tools)
+- Manual testing:
+  - stdio mode: direct integration testing with Claude Desktop
+  - HTTP mode: curl-based testing against localhost endpoint
+- Build validation via `npm run build` before deployment
+
+### Git Workflow
+- **Main branch**: `main`
+- **Commit style**: extremely concise, prioritize brevity over grammar
+- **LOG.md**: date-based entries (YYYY-MM-DD format), most recent at top
+- **GitHub operations**: use `gh` CLI exclusively
+- Update LOG.md for any significant change
+
+## Domain Context
+
+### CKAN Knowledge
+- **CKAN API v3** - standard API across all CKAN portals
+- **Apache Solr** - powers search functionality
+  - Supports field:value syntax, AND/OR/NOT operators, wildcards, ranges
+  - Filter queries (fq) for filtering without affecting score
+  - Facets for statistical aggregations
+  - Sort and pagination (start/rows)
+
+### Output Conventions
+- All tools support `format` parameter: `markdown` (default) or `json`
+- Output truncated to 50000 characters max (hardcoded)
+- Dates formatted in Italian locale
+- File sizes in human-readable format
+
+## Important Constraints
+
+### Environment
+- **WSL considerations** - memory constraints require esbuild instead of tsc
+- Always use `npm run build` (esbuild) not `npm run build:tsc`
+- esbuild bundles internal modules but keeps external deps as-is
+
+### Technical Limits
+- 50000 character output limit (hardcoded, could be configurable)
+- 30-second timeout for CKAN API calls
+- Read-only operations only (security by design)
+- No authentication (public endpoints only)
+- No caching (trade-off: freshness vs performance)
+
+### Build System
+- esbuild config: bundles TypeScript, externalizes runtime deps
+- External deps must be in node_modules: @modelcontextprotocol/sdk, axios, express, zod
+- tsconfig.json mainly for IDE support (esbuild ignores most options)
+
+## External Dependencies
+
+### CKAN Portals
+Major supported portals:
+- 🇮🇹 https://dati.gov.it (Italia)
+- 🇺🇸 https://catalog.data.gov (United States)
+- 🇨🇦 https://open.canada.ca/data (Canada)
+- 🇬🇧 https://data.gov.uk (United Kingdom)
+- 🇪🇺 https://data.europa.eu (European Union)
+- 🌍 https://demo.ckan.org (Official CKAN demo)
+
+### CKAN API Integration
+- All requests use axios with 30s timeout
+- User-Agent: `CKAN-MCP-Server/1.0`
+- URL normalization (remove trailing slash)
+- Response validation (success === true)
+- Error handling: HTTP errors, timeouts, unreachable servers
+
+### Portal Variations
+Each CKAN portal may differ in:
+- DataStore availability
+- Custom dataset fields
+- Available organizations and tags
+- Supported resource formats

package/openspec/specs/documentation-language/spec.md

@@ -0,0 +1,32 @@
+# documentation-language Specification
+
+## Purpose
+TBD - created by archiving change translate-project-to-english. Update Purpose after archive.
+## Requirements
+### Requirement: Project documentation in English
+
+The project SHALL provide all user-facing documentation in English to ensure accessibility for the international open data community.
+
+#### Scenario: English documentation
+Given a CKAN MCP Server project
+When a user reads the documentation
+Then the documentation is in English
+And code examples are in English
+And technical terms follow CKAN API terminology
+
+### Requirement: Preserve multilingual examples
+
+The project SHALL preserve non-English portal names, organization titles, or data values in their original language to accurately reflect real-world CKAN portals.
+
+#### Scenario: Italian portal example
+Given an example using dati.gov.it
+When the example references Italian organizations
+Then organization names remain in Italian (e.g., "Regione Siciliana")
+And descriptions are in English
+
+#### Scenario: Non-English data values
+Given an example query with data filters
+When the data contains non-English values
+Then those values are preserved in their original language
+And the surrounding documentation explains the context in English
+

package/openspec/specs/mcp-resources/spec.md

@@ -0,0 +1,94 @@
+# mcp-resources Specification
+
+## Purpose
+TBD - created by archiving change add-mcp-resources. Update Purpose after archive.
+## Requirements
+### Requirement: Dataset Resource Template
+
+The system SHALL expose a resource template for accessing CKAN dataset metadata via the URI pattern `ckan://{server}/dataset/{id}`.
+
+#### Scenario: Read dataset metadata successfully
+
+- **WHEN** client reads `ckan://dati.gov.it/dataset/vaccini-covid`
+- **THEN** server returns JSON with complete dataset metadata including title, description, resources, organization, tags
+
+#### Scenario: Dataset not found
+
+- **WHEN** client reads `ckan://demo.ckan.org/dataset/nonexistent-id`
+- **THEN** server returns error indicating dataset not found
+
+#### Scenario: Server unreachable
+
+- **WHEN** client reads `ckan://invalid-server.example/dataset/test`
+- **THEN** server returns error indicating server unreachable
+
+### Requirement: Resource Resource Template
+
+The system SHALL expose a resource template for accessing CKAN resource metadata via the URI pattern `ckan://{server}/resource/{id}`.
+
+#### Scenario: Read resource metadata successfully
+
+- **WHEN** client reads `ckan://dati.gov.it/resource/abc-123-def`
+- **THEN** server returns JSON with resource metadata including name, format, URL, size, and download link
+
+#### Scenario: Resource not found
+
+- **WHEN** client reads `ckan://demo.ckan.org/resource/invalid-id`
+- **THEN** server returns error indicating resource not found
+
+### Requirement: Organization Resource Template
+
+The system SHALL expose a resource template for accessing CKAN organization metadata via the URI pattern `ckan://{server}/organization/{name}`.
+
+#### Scenario: Read organization metadata successfully
+
+- **WHEN** client reads `ckan://dati.gov.it/organization/regione-toscana`
+- **THEN** server returns JSON with organization metadata including title, description, and dataset count
+
+#### Scenario: Organization not found
+
+- **WHEN** client reads `ckan://demo.ckan.org/organization/nonexistent-org`
+- **THEN** server returns error indicating organization not found
+
+### Requirement: URI Scheme Parsing
+
+The system SHALL parse `ckan://` URIs extracting server hostname and path components.
+
+#### Scenario: Parse standard URI
+
+- **WHEN** URI is `ckan://dati.gov.it/dataset/test-id`
+- **THEN** server extracts: server=`https://dati.gov.it`, type=`dataset`, id=`test-id`
+
+#### Scenario: Parse URI with www prefix
+
+- **WHEN** URI is `ckan://www.dati.gov.it/dataset/test-id`
+- **THEN** server extracts: server=`https://www.dati.gov.it`, type=`dataset`, id=`test-id`
+
+#### Scenario: Reject malformed URI
+
+- **WHEN** URI is `ckan://invalid` (missing path)
+- **THEN** server returns validation error
+
+### Requirement: Resource Response Format
+
+The system SHALL return resource content as JSON with standard MCP resource response structure.
+
+#### Scenario: JSON response structure
+
+- **WHEN** client reads any valid resource URI
+- **THEN** response contains `contents` array with `uri`, `mimeType` (application/json), and `text` (JSON string)
+
+#### Scenario: Large response truncation
+
+- **WHEN** resource content exceeds CHARACTER_LIMIT
+- **THEN** response is truncated to CHARACTER_LIMIT with truncation indicator
+
+### Requirement: Resource Discovery
+
+The system SHALL list available resource templates when client requests resource list.
+
+#### Scenario: List resource templates
+
+- **WHEN** client calls `resources/listTemplates`
+- **THEN** server returns list of available URI templates with descriptions
+

package/package.json

@@ -0,0 +1,46 @@
+{
+  "name": "@aborruso/ckan-mcp-server",
+  "version": "0.3.1",
+  "description": "MCP server for interacting with CKAN open data portals",
+  "main": "dist/index.js",
+  "type": "module",
+  "scripts": {
+    "build": "esbuild src/index.ts --bundle --platform=node --format=esm --outfile=dist/index.js --external:@modelcontextprotocol/sdk --external:axios --external:express --external:zod",
+    "build:tsc": "node --max-old-space-size=8192 ./node_modules/typescript/bin/tsc",
+    "start": "node dist/index.js",
+    "dev": "npm run build && node dist/index.js",
+    "watch": "esbuild src/index.ts --bundle --platform=node --format=esm --outfile=dist/index.js --external:@modelcontextprotocol/sdk --external:axios --external:express --external:zod --watch",
+    "test": "vitest",
+    "test:watch": "vitest --watch",
+    "test:coverage": "vitest --coverage"
+  },
+  "keywords": [
+    "mcp",
+    "ckan",
+    "open-data",
+    "api",
+    "data-portal"
+  ],
+  "author": "Andrea Borruso <aborruso@gmail.com>",
+  "license": "MIT",
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.0.4",
+    "axios": "^1.7.2",
+    "express": "^4.19.2",
+    "zod": "^3.23.8"
+  },
+  "devDependencies": {
+    "@types/express": "^4.17.21",
+    "@types/node": "^20.12.12",
+    "@vitest/coverage-v8": "^4.0.16",
+    "esbuild": "^0.27.2",
+    "typescript": "^5.4.5",
+    "vitest": "^4.0.16"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "engines": {
+    "node": ">=18.0.0"
+  }
+}

package/spunti.md

@@ -0,0 +1,19 @@
+# Intro
+
+Spunti vari
+
+## Da fare
+
+- farlo totalmente in inglese
+- documentarne l'uso per chatGPT e Claude Code
+- aggiunere agli esempi e al tool poter cercare per frequenza di aggiornamento
+- il tema del peso del ranking dei risultati andrebbe approfondito. Prendere spunto da ckan canada
+
+## Tool con coi confrontarsi
+
+- https://skywork.ai/skypage/en/unlocking-government-data-mcp-server/1980445961155629056
+- https://github.com/ondics/ckan-mcp-server
+
+## Temi aperti
+
+- la data di modifica di una risorsa sembra non esserci realmente, salvo non cambiare i metadti. Se si aggiorna soltanto il csv non ce ne è traccia?

package/tasks/todo.md

@@ -0,0 +1,124 @@
+# Task: Aggiungere tool ckan_organization_search
+
+## Obiettivo
+
+Implementare un nuovo tool MCP dedicato alla ricerca di organizzazioni per nome, con interfaccia più semplice rispetto a `package_search` con sintassi Solr.
+
+## Fasi
+
+### Fase 1: Implementazione tool
+- [ ] Aggiungere `ckan_organization_search` in `src/index.ts` (prima riga 872)
+- [ ] Schema input: `server_url`, `pattern`, `response_format`
+- [ ] Costruire query Solr automatica: `organization:*{pattern}*`
+- [ ] Chiamare `package_search` con `rows=0` e `facet.field=organization`
+- [ ] Output JSON: `{ count: N, organizations: [...] }`
+- [ ] Output markdown: tabella organizzazioni con conteggio dataset
+
+### Fase 2: Testing
+- [ ] Build: `npm run build`
+- [ ] Test con pattern "toscana"
+- [ ] Verificare output JSON
+- [ ] Verificare output markdown
+
+### Fase 3: Documentazione
+- [ ] Aggiungere esempi in `EXAMPLES.md`
+- [ ] Aggiornare `LOG.md` con data e modifiche
+
+## Dettagli Implementazione
+
+**Posizione**: Dopo tool `ckan_organization_show` (circa riga 700) e prima di `ckan_datastore_search`
+
+**Firma tool**:
+```typescript
+server.registerTool(
+  "ckan_organization_search",
+  {
+    title: "Search CKAN Organizations by Name",
+    description: "...",
+    inputSchema: z.object({
+      server_url: z.string().url(),
+      pattern: z.string().min(1).describe("Search pattern (wildcards added automatically)"),
+      response_format: ResponseFormatSchema
+    }).strict(),
+    annotations: {
+      readOnlyHint: true,
+      destructiveHint: false,
+      idempotentHint: true,
+      openWorldHint: true
+    }
+  },
+  async (params) => { ... }
+)
+```
+
+**Logica**:
+1. Query Solr: `organization:*${params.pattern}*`
+2. API call: `package_search` con `rows=0`, `facet.field=["organization"]`, `facet.limit=500`
+3. Parsing facets per estrarre organizzazioni matchate
+4. Output formattato
+
+## Note
+
+- Molto più semplice di costruire query manualmente
+- Risparmio token: restituisce solo organizzazioni, non dataset
+- Pattern matching automatico (l'utente non deve ricordare wildcard Solr)
+
+---
+
+## Review
+
+### Modifiche Completate
+
+**File modificati**:
+1. `src/index.ts` - Aggiunto tool `ckan_organization_search` (righe 680-785)
+2. `EXAMPLES.md` - Aggiunti esempi di utilizzo (righe 54-73)
+3. `LOG.md` - Documentata la nuova feature
+
+**Implementazione**:
+- Tool registrato tra `ckan_organization_show` e `ckan_datastore_search`
+- Schema input: `server_url`, `pattern`, `response_format`
+- Query Solr automatica: `organization:*{pattern}*`
+- Utilizza `package_search` con `rows=0` per efficienza massima
+- Output JSON: `{ count, total_datasets, organizations: [...] }`
+- Output markdown: tabella formattata con org e conteggio dataset
+
+**Testing**:
+- Build completato con successo (47ms, esbuild)
+- Test manuale con pattern "toscana": ✅
+  - 2 organizzazioni trovate
+  - 11.000 dataset totali
+  - Output JSON corretto
+  - Zero dataset scaricati (solo facet)
+
+**Vantaggi**:
+- **UX migliorata**: API semplice vs sintassi Solr
+- **Efficienza**: filtraggio lato server, zero dataset trasmessi
+- **Token saving**: solo metadata organizzazioni
+- **Performance**: wildcard automatici, no costruzione query manuale
+
+### Prossimi Passi
+
+Per utilizzare il tool:
+1. Riavviare Claude Code per caricare il nuovo server MCP
+2. Il tool sarà disponibile come `ckan_organization_search`
+
+### Esempio Output
+
+```json
+{
+  "count": 2,
+  "total_datasets": 11000,
+  "organizations": [
+    {
+      "name": "regione-toscana",
+      "display_name": "Regione Toscana",
+      "dataset_count": 10988
+    },
+    {
+      "name": "autorita-idrica-toscana",
+      "display_name": "Autorità Idrica Toscana",
+      "dataset_count": 12
+    }
+  ]
+}
+```

package/test-urls.js

@@ -0,0 +1,18 @@
+import { getDatasetViewUrl, getOrganizationViewUrl } from './src/utils/url-generator.js';
+
+const serverUrl = 'https://www.dati.gov.it/opendata';
+const pkg = {
+  id: 'f20b37d0cd56764f77e4c707cc62b528eeb48ff24cdb415174a0dd31b63cad0c',
+  name: 'test-dataset'
+};
+
+const org = {
+  id: 'org-id',
+  name: 'comune-di-bari'
+};
+
+console.log('Dataset URL:', getDatasetViewUrl(serverUrl, pkg));
+console.log('Organization URL:', getOrganizationViewUrl(serverUrl, org));
+
+const otherServer = 'https://demo.ckan.org';
+console.log('Other Dataset URL:', getDatasetViewUrl(otherServer, pkg));

package/tmp/test-org-search.js

@@ -0,0 +1,55 @@
+// Test manuale del tool ckan_organization_search
+import axios from 'axios';
+
+async function testOrgSearch() {
+  const serverUrl = 'https://www.dati.gov.it/opendata';
+  const pattern = 'toscana';
+  const query = `organization:*${pattern}*`;
+
+  console.log('Testing organization search...');
+  console.log('Pattern:', pattern);
+  console.log('Query:', query);
+  console.log('');
+
+  try {
+    const url = `${serverUrl}/api/3/action/package_search`;
+    const response = await axios.get(url, {
+      params: {
+        q: query,
+        rows: 0,
+        'facet.field': JSON.stringify(['organization']),
+        'facet.limit': 500
+      },
+      timeout: 30000,
+      headers: {
+        'User-Agent': 'CKAN-MCP-Server/1.0'
+      }
+    });
+
+    const result = response.data.result;
+    const orgFacets = result.search_facets?.organization?.items || [];
+    const totalDatasets = result.count || 0;
+
+    console.log('Organizations found:', orgFacets.length);
+    console.log('Total datasets:', totalDatasets);
+    console.log('');
+
+    const jsonResult = {
+      count: orgFacets.length,
+      total_datasets: totalDatasets,
+      organizations: orgFacets.map((item) => ({
+        name: item.name,
+        display_name: item.display_name,
+        dataset_count: item.count
+      }))
+    };
+
+    console.log(JSON.stringify(jsonResult, null, 2));
+
+  } catch (error) {
+    console.error('Error:', error.message);
+    process.exit(1);
+  }
+}
+
+testOrgSearch();