statcan-mcp-server 0.1.0__tar.gz

statcan_mcp_server-0.1.0/.github/workflows/publish-mcp-registry.yml

@@ -0,0 +1,54 @@
+name: Publish to PyPI & MCP Registry
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  publish-pypi:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+    environment: pypi
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tools
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+
+  publish-registry:
+    needs: publish-pypi
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+      contents: read
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install mcp-publisher
+        run: |
+          curl -L "https://github.com/modelcontextprotocol/registry/releases/latest/download/mcp-publisher_$(uname -s | tr '[:upper:]' '[:lower:]')_$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/').tar.gz" | tar xz mcp-publisher
+
+      - name: Authenticate to MCP Registry
+        run: ./mcp-publisher login github-oidc
+
+      - name: Validate server.json
+        run: ./mcp-publisher validate
+
+      - name: Publish to MCP Registry
+        run: ./mcp-publisher publish

statcan_mcp_server-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,432 @@
+# Contributing to Statistics Canada MCP Server
+
+Thank you for your interest in contributing to the Statistics Canada MCP Server! This project provides tools for interacting with Statistics Canada data APIs through the Model Context Protocol (MCP), enabling LLMs and other clients to access Canadian statistical data.
+
+## 📋 Table of Contents
+
+- [Getting Started](#getting-started)
+- [Development Environment Setup](#development-environment-setup)
+- [Project Architecture](#project-architecture)
+- [Development Workflow](#development-workflow)
+- [Code Standards](#code-standards)
+- [Testing](#testing)
+- [Documentation](#documentation)
+- [Pull Request Process](#pull-request-process)
+- [Types of Contributions](#types-of-contributions)
+- [Future Development](#future-development)
+
+## Getting Started
+
+### Prerequisites
+
+- **Python 3.10+** - Download from [python.org](https://www.python.org/downloads/)
+- **UV** - Fast Python package installer
+- **Git** - For version control
+- Basic familiarity with:
+  - Python async/await patterns
+  - REST APIs
+  - Model Context Protocol (MCP) concepts
+  - Statistics Canada data structure (helpful but not required)
+
+### Quick Setup
+
+1. **Fork and clone the repository**:
+   ```bash
+   git clone https://github.com/YOUR_USERNAME/mcp-statcan.git
+   cd mcp-statcan
+   ```
+
+2. **Install UV** (if not already installed):
+   ```bash
+   curl -fsSL https://astral.sh/uv/install.sh | bash
+   ```
+
+3. **Install dependencies**:
+   ```bash
+   uv pip install fastmcp httpx pydantic
+   # Development dependencies
+   uv pip install pytest black isort
+   ```
+
+## Development Environment Setup
+
+### Environment Variables
+
+
+```bash
+# Database configuration
+STATCAN_DB_FILE=temp_statcan_data.db
+
+# Debug flags
+STATCAN_SERVER_DEBUG=true
+STATCAN_SSL_WARNINGS=false
+STATCAN_SQL_DEBUG=false
+STATCAN_DATA_VALIDATION_WARNINGS=true
+STATCAN_SEARCH_PROGRESS=true
+```
+
+## Project Architecture
+
+### Directory Structure
+
+```
+mcp-statcan/
+├── src/
+│   ├── __init__.py
+│   ├── server.py              # Main FastMCP server entry point
+│   ├── config.py              # Configuration management
+│   ├── api/                   # MCP tools for StatCan API
+│   │   ├── cube_tools.py      # Data cube operations
+│   │   ├── vector_tools.py    # Vector data operations
+│   │   └── metadata_tools.py  # Metadata and code sets
+│   ├── db/                    # Database operations
+│   │   ├── connection.py      # SQLite connection management
+│   │   ├── queries.py         # Database tools registration
+│   │   └── schema.py          # Table schema operations
+│   ├── models/                # Pydantic data models
+│   │   ├── api_models.py      # API request/response models
+│   │   └── db_models.py       # Database models
+│   └── util/                  # Utility functions
+│       ├── coordinate.py      # Coordinate padding utilities
+│       └── sql_helpers.py     # SQL query helpers
+├── pyproject.toml            # Project configuration
+└── README.md                 # Project overview
+```
+
+### Key Components
+
+1. **FastMCP Server** (`server.py`): Main entry point that registers all MCP tools
+2. **API Tools**: Wrappers around Statistics Canada Web Data Service API
+3. **Database Layer**: SQLite integration for data persistence and querying
+4. **Models**: Pydantic models for data validation and serialization
+5. **Configuration**: Environment-based configuration management
+
+## Development Workflow
+
+### Branch Naming Convention
+
+Use descriptive branch names with prefixes:
+
+- `feature/` - New features
+- `fix/` - Bug fixes
+- `docs/` - Documentation updates
+- `refactor/` - Code refactoring
+- `test/` - Test improvements
+
+Examples:
+- `feature/add-windows-installation-guide`
+- `fix/ssl-verification-error`
+- `docs/update-api-examples`
+
+### Making Changes
+
+1. **Create a feature branch**:
+   ```bash
+   git checkout -b feature/your-feature-name
+   ```
+
+2. **Make your changes** following the code standards below
+
+   ```bash
+   git add .
+   git commit -m "Add descriptive commit message"
+   ```
+
+3. **Push and create a pull request**:
+   ```bash
+   git push origin feature/your-feature-name
+   ```
+
+## Code Standards
+
+### Python Style Guidelines
+
+- **Type hints** - Use type hints for all function parameters and return values
+- **Async/await** - Use async functions for API calls and database operations
+- **Error handling** - Use try/except blocks with specific exception types
+- **Logging** - Use the project's logging configuration for debug output
+
+### Code Organization
+
+- **Imports**: Use `isort` for consistent import ordering
+- **Functions**: Keep functions focused and single-purpose
+- **Classes**: Use Pydantic models for data validation
+- **Constants**: Define constants in `config.py`
+
+### Example Code Style
+
+```python
+from typing import List, Optional
+from pydantic import BaseModel
+import httpx
+
+class VectorDataInput(BaseModel):
+    """Input model for vector data requests."""
+    vector_id: int
+    latest_n: int
+    
+async def get_vector_data(
+    input_data: VectorDataInput
+) -> Optional[List[Dict[str, Any]]]:
+    """
+    Retrieve vector data from Statistics Canada API.
+    
+    Args:
+        input_data: Vector data request parameters
+        
+    Returns:
+        List of data points or None if request fails
+        
+    Raises:
+        httpx.HTTPError: If API request fails
+    """
+    try:
+        # Implementation here
+        pass
+    except httpx.HTTPError as e:
+        log_server_debug(f"API request failed: {e}")
+        raise
+```
+
+### Security Considerations
+
+- **Never commit sensitive data** - Use environment variables
+- **SSL verification** - Enable SSL verification in production
+- **Input validation** - Always validate user inputs with Pydantic models
+- **SQL injection prevention** - Use parameterized queries
+
+## Testing
+
+### Running Tests
+
+```bash
+# Run all tests
+pytest
+
+# Run with coverage
+pytest --cov=src
+
+# Run specific test file
+pytest tests/test_api_tools.py
+
+# Run with debug output
+pytest -v -s
+```
+
+### Test Structure
+
+Tests should be organized to mirror the source structure:
+
+```
+tests/
+├── test_api/
+│   ├── test_cube_tools.py
+│   ├── test_vector_tools.py
+│   └── test_metadata_tools.py
+├── test_db/
+│   └── test_queries.py
+└── test_models/
+    └── test_api_models.py
+```
+
+### Writing Tests
+
+- **Unit tests** - Test individual functions and classes
+- **Integration tests** - Test API interactions (with mocking)
+- **Database tests** - Test database operations with temporary databases
+- **Use fixtures** - Create reusable test data with pytest fixtures
+
+Example test:
+
+```python
+import pytest
+from src.models.api_models import VectorLatestNInput
+
+def test_vector_latest_n_input_validation():
+    """Test VectorLatestNInput model validation."""
+    # Valid input
+    valid_input = VectorLatestNInput(vectorId=12345, latestN=5)
+    assert valid_input.vectorId == 12345
+    assert valid_input.latestN == 5
+    
+    # Invalid input
+    with pytest.raises(ValueError):
+        VectorLatestNInput(vectorId="invalid", latestN=5)
+```
+
+## Documentation
+
+### Code Documentation
+
+- **Docstrings** - Use Google-style docstrings for all public functions and classes
+- **Type hints** - Include comprehensive type annotations
+- **Comments** - Add comments for complex logic, not obvious code
+- **API documentation** - Update tool descriptions when adding new MCP tools
+
+### Documentation Updates
+
+When making changes, update relevant documentation:
+
+- **README.md** - For new features or installation changes
+- **API examples** - Add examples in `docs/examples/`
+- **Code comments** - Update docstrings and inline comments
+- **Configuration docs** - Update environment variable documentation
+
+### Example Documentation
+
+```python
+async def search_cubes_by_title(title: str) -> List[CubeMetadata]:
+    """
+    Search for data cubes by title keyword.
+    
+    This function searches Statistics Canada's data cubes using a title
+    keyword and returns matching cube metadata.
+    
+    Args:
+        title: Search keyword for cube titles
+        
+    Returns:
+        List of CubeMetadata objects containing cube information
+        
+    Raises:
+        httpx.HTTPError: If the API request fails
+        ValueError: If the title parameter is empty
+        
+    Example:
+        >>> cubes = await search_cubes_by_title("employment")
+        >>> print(f"Found {len(cubes)} cubes")
+    """
+```
+
+## Pull Request Process
+
+### Before Submitting
+
+1. **Run all checks**:
+   ```bash
+   black src/
+   isort src/
+   pytest
+   ```
+
+2. **Update documentation** if needed
+
+3. **Test your changes** thoroughly
+
+4. **Update IMPLEMENTATIONS.md** if implementing planned features
+
+### Pull Request Template
+
+When creating a pull request, include:
+
+```markdown
+## Description
+Brief description of changes made.
+
+## Type of Change
+- [ ] Bug fix
+- [ ] New feature
+- [ ] Documentation update
+- [ ] Refactoring
+- [ ] Other (please describe)
+
+## Testing
+- [ ] All existing tests pass
+- [ ] New tests added for new functionality
+- [ ] Manual testing completed
+
+## Checklist
+- [ ] Code follows project style guidelines
+- [ ] Self-review completed
+- [ ] Documentation updated
+- [ ] No sensitive data included
+```
+
+### Review Process
+
+1. **Automated checks** - CI/CD pipeline runs tests and formatting checks
+2. **Code review** - Project maintainers review code quality and design
+3. **Testing** - Ensure changes don't break existing functionality
+4. **Documentation** - Verify documentation is updated and accurate
+
+## Types of Contributions
+
+### 🐛 Bug Fixes
+
+- SSL verification issues
+- API endpoint errors
+- Database connection problems
+- Data validation failures
+
+### ✨ New Features
+
+Priority areas from `IMPLEMENTATIONS.md`:
+
+- Windows installation guides
+- Package installer improvements
+- Enhanced tool prompts for LLM efficiency
+- Additional database math tools
+- Graph visualization tools
+- Scheduled report generation
+
+### 📚 Documentation
+
+- API usage examples
+- Installation guides for different platforms
+- Integration documentation
+- Code examples and tutorials
+
+### 🧪 Testing
+
+- Unit test coverage improvements
+- Integration test development
+- Performance testing
+- API mocking improvements
+
+### 🛠️ Infrastructure
+
+- CI/CD pipeline improvements
+- Development environment enhancements
+- Performance optimizations
+- Security improvements
+
+## Future Development
+
+### Current Roadmap
+
+Based on `IMPLEMENTATIONS.md`, priority areas include:
+
+1. **Package Management**: UV/Smithery installer for direct LLM client installation
+2. **Platform Support**: Windows installation guides and setup scripts
+3. **LLM Optimization**: Improved tool prompts to reduce unnecessary API calls
+4. **Database Enhancements**: Math tools and visualization capabilities
+5. **Automation**: Scheduled reporting and data update systems
+
+### Architecture Considerations
+
+- **Multi-agent systems**: Potential A2A + MCP integration
+- **Performance**: Bulk operation optimization
+- **Scalability**: Database connection pooling
+- **Security**: SSL verification and input validation improvements
+
+### Getting Involved
+
+To contribute to future development:
+
+1. **Check IMPLEMENTATIONS.md** for current priorities
+2. **Join discussions** on GitHub issues
+3. **Propose new features** through issue templates
+4. **Review architecture** diagrams and provide feedback
+
+## Questions or Need Help?
+
+- **GitHub Issues**: For bug reports and feature requests
+- **GitHub Discussions**: For general questions and brainstorming
+- **Documentation**: Check the `docs/` directory for detailed guides
+- **Code Examples**: See `docs/examples/` for usage patterns
+
+## License
+
+By contributing to this project, you agree that your contributions will be licensed under the MIT License.
+
+---

statcan_mcp_server-0.1.0/IMPLEMENTATIONS.md

@@ -0,0 +1,144 @@
+# IMPLEMENTATIONS.MD
+
+# 🗺️ Roadmap & Ecosystem Alignment
+*Updated Feb 2026 — informed by MCP Apps, Registry, and ecosystem best practices research*
+
+## Tier 1: Foundational Fixes (Weekend Project)
+
+- [ ] **Create `pyproject.toml`** — Replace `requirements.txt` + manual `uv pip install` with proper Python packaging. Define project name, version (semver), description, dependencies (`fastmcp`, `httpx`, `pydantic`), Python ≥3.10, entry point for the server, and metadata (author, homepage, repo URL, keywords: `mcp`, `statistics-canada`, `statcan`, `open-data`). This enables PyPI publishing and is required for registry listing.
+- [ ] **Enable SSL verification** — SSL is currently disabled for development. Fix httpx SSL settings and document any proxy/corporate certificate workarounds.
+- [ ] **Harden SQL input validation** — Database tools accept raw SQL with "basic" validation. Implement: whitelist of allowed SQL operations (SELECT only), parameterized queries to prevent injection, and query size/timeout limits.
+- [ ] **Add a uv or smithery package installer** — Install packages to Claude or other LLM clients directly instead of having to adjust working directories *(carried from June 1, 2025)*
+
+## Tier 2: Ecosystem Integration (One-Week Sprint)
+
+- [ ] **Publish to PyPI** — With `pyproject.toml` in place, build and upload. Enables `pip install mcp-statcan` and `uvx mcp-statcan`. Update README with simplified installation.
+- [ ] **Register on Official MCP Registry** — Use `mcp-publisher` CLI, generate `server.json` with reverse-DNS naming (`io.github.aryan-jhaveri/mcp-statcan`), reference PyPI package, publish. Currently missing from: Official Registry, Smithery, PulseMCP, Docker Catalog.
+- [ ] **Submit to remaining directories** — PR to `punkpeye/awesome-mcp-servers` (syncs to Glama), register on Smithery.ai, submit to PulseMCP (`pulsemcp.com/submit`), consider Docker MCP Catalog.
+- [ ] **Add GitHub Actions CI/CD** — Linting (ruff), type checking (mypy), tests on every push/PR. Release workflow publishes to PyPI on tagged releases. Registry publishing step using `mcp-publisher` with GitHub OIDC auth. (DONT DO THIS YET - Because I need to polish up on CI/CD)
+- [ ] **Create a Dockerfile** — Slim Python base image for sandboxed deployment. Enables Docker MCP Catalog listing. (DONT DO THIS YET - Because I need to polish up on Docker)
+- [ ] **Create setup/installation guides for Windows** *(carried from June 1, 2025)* (DONT DO THIS YET - Because I need to test on windows virtual machine)
+
+## Tier 3: Quality & Completeness (Two-Week Sprint)
+
+- [ ] **Write tests** — Unit tests per tool function (pytest), mock StatCan API responses. Integration tests via FastMCP in-memory client. Test edge cases: empty results, API timeouts, malformed responses, pagination boundaries. Measure **tool hit rate** (LLM correctly picks right tool for 20 natural-language queries). (Need to research and plan)
+
+- [ ] **Complete StatCan WDS API coverage** — The Web Data Service provides **15 methods**; implement all of them. Missing ones likely include `getAllCubesListLite`, `getCubeMetadata`, `getBulkVectorDataByRange`, `getChangedCubeList`, `getChangedSeriesDataFromVector`, etc. Each tool needs clear name, detailed description (explain StatCan "cubes" and "vectors" for LLMs), and well-defined input schema.
+(There's trade offs between wds and sdmx tools, currently the LLM are fetching and readon one data point at a time maybe )
+
+- [ ] **Add MCP Resources & Prompts** — Currently only exposes tools. Add **resources** for: available StatCan subject categories, StatCan data model explainer (cubes, vectors, coordinates, reference periods), API rate limits/constraints. Add **prompt templates** for: "Find and analyze a StatCan time series", "Compare regional statistics across provinces", "Get the latest economic indicators".
+(Need subject matter experts to review)
+
+- [ ] **Implement cursor-based pagination** — For tools returning large result sets, implement the MCP spec's pagination pattern with opaque cursor tokens and server-determined page sizes. Prevents timeouts on large queries.
+(Need to research and plan and underdstand pagination and cursor tokens)
+
+- [ ] Fix `get_bulk_vector` truncated output exceeding LLM context — better implementation: read heads of fetched data, or always route through db tools *(carried from Jan 7, 2026)*
+
+
+- [ ] Fix `create_table_from_data` not filling DB — LLM needs additional tool call to manually insert data, causes errors and context token exhaustion *(carried from Jan 7, 2026)*
+
+## Tier 4: Differentiation (Month-Long Effort)
+
+- [ ] **Add MCP Apps support for data visualization** — Declare `ui://` resources that render interactive charts/tables. Time series → interactive Chart.js/Plotly chart in sandboxed iframe. Data tables → sortable/filterable HTML table. Use `ext-apps` SDK (`add-app-to-server` agent skill). Text fallback for non-supporting clients. Would make mcp-statcan one of the few data MCP servers with visual output.
+
+
+- [ ] **Support Streamable HTTP transport** — Add HTTP server mode alongside stdio for remote deployment. Unlocks hosting on Cloudflare Workers, Render, Railway. Consider deploying a free public instance for zero-setup access.
+
+
+- [ ] **Add structured output schemas** — Define typed output schemas for each tool's response, enabling downstream tools and MCP Apps UIs to parse results programmatically.
+- [ ] **Implement caching** — StatCan data updates at 8:30 AM ET on business days. Cache API responses with time-based invalidation aligned to this schedule. Reduces StatCan API load, improves response times.
+
+
+- [ ] **Look into SDMX implementation** — Allow Claude to create files or exact URIs for vector and metadata fetching; mix of REST and SDMX tools available *(carried from Jan 7, 2026)*
+
+
+- [ ] **Maybe: Look into A2A + MCP** — (https://arxiv.org/pdf/2506.01804) to create an extended multi-agent system *(carried from June 3, 2025, only for curiosity)*
+
+## Documentation Improvements
+
+- [ ] **Badges section** — PyPI version, CI status, license, MCP registry link
+- [ ] **Tool reference table** — Every tool with parameters, return types, example usage
+- [ ] **CONTRIBUTING.md** with contribution guidelines
+- [ ] **CHANGELOG.md** tracking versions
+- [ ] **Multi-client config examples** — Claude Desktop, Claude Code, Cursor, VS Code Copilot, Windsurf (not just Claude Desktop)
+- [ ] **StatCan explainer section** — What StatCan data is and why it's useful, for international users
+
+---
+
+# 📓 Development Log
+
+## Jan 7, 2026
+
+[x] Adjust and make more detailed tool prompts to prevent the LLM from making separate calls for finding data and then inputting to database.
+
+[x] Need to add db specific math tools. Add additional graph tools if needed.
+
+## January 2, 2026 — Refactor Data Retrieval Pipeline
+
+[x] Identify issue with `get_bulk_vector_data_by_range` returning nested JSON incompatible with DB tools.
+
+[x] **Priority** Shift strategy to **Flatten API Response**: Bulk Tool Flattening → Database Ingestion.
+
+[x] Modify `get_bulk_vector_data_by_range` to return flat list of data points with `vectorId` injected.
+
+[x] Ensure compatibility with `create_table_from_data` for seamless "Fetch → Store" workflow.
+
+## Notes
+- Potential use case: Create scheduled calls for the LLM to create weekly reports for specific data sets.
+
+---
+
+# 🏗️ Server Architecture & Data Flow
+*June 1, 2025*
+
+```mermaid
+flowchart TD
+    A[Claude/MCP Client] -->|MCP Protocol| B[FastMCP Server]
+    
+    B --> C{Tool Type}
+    C -->|API Tools| D[Statistics Canada API]
+    C -->|DB Tools| M[Database Tools]
+    C -->|Metadata Tools| F[Code Sets & Classifications]
+    
+    D --> G[Cube Tools]
+    D --> H[Vector Tools]
+    D --> I[Metadata Tools]
+
+    E[SQLite Database]
+    
+    G -->|get_cube_metadata<br/>search_cubes_by_title<br/>get_data_from_cube| J[StatCan WDS API<br/>statcan.gc.ca/t1/wds/rest]
+    H -->|get_series_info_from_vector<br/>get_data_from_vectors<br/>get_bulk_vector_data| J
+    I -->|get_code_sets<br/>get_changed_cube_list| J
+    
+    J -->|JSON Response| K[API Response Processing]
+    K -->|Flattened Data Points| L{Data Usage}
+    
+    L -->|Return to Client| A
+    L -->|Store in DB| M[Database Tools]
+    
+    M --> N[create_table_from_data]
+    M --> O[insert_data_into_table] 
+    M --> P[query_database]
+    M --> Q[list_tables]
+    M --> R[get_table_schema]
+    
+    N --> E
+    O --> E
+    P --> E
+    Q --> E
+    R --> E
+    
+    E --> S[Dynamic Tables]
+    
+    S -->|SQL Results| T[Formatted Response]
+    T -->|MCP Response| A
+    
+    F -->|get_code_sets| J
+    
+    style A fill:#210d70
+    style B fill:#70190d
+    style E fill:#700d49
+    style L fill:#450d70
+    style T fill:#35700d
+    style J fill:#700d1c
+```

statcan_mcp_server-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Aryan Jhaveri
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.