fd-open-data-mcp 0.2.0__tar.gz

fd_open_data_mcp-0.2.0/PKG-INFO

@@ -0,0 +1,218 @@
+Metadata-Version: 2.4
+Name: fd-open-data-mcp
+Version: 0.2.0
+Summary: Open-data ontology MCP: a semantic concept layer over multi-datasource financial/economic data
+Author: FindDataOfficial
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: click>=8.0
+Requires-Dist: sqlalchemy>=2.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: fastmcp>=2.0
+Requires-Dist: pandas>=2.0
+Requires-Dist: fd-open-data-protocol
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: data
+Requires-Dist: akshare>=1.17; extra == "data"
+Requires-Dist: yfinance>=0.2; extra == "data"
+Requires-Dist: world_bank_data>=0.1; extra == "data"
+Requires-Dist: edgartools>=2.0; extra == "data"
+Requires-Dist: wbgapi>=1.0; extra == "data"
+Requires-Dist: requests>=2.31; extra == "data"
+Requires-Dist: beautifulsoup4>=4.12; extra == "data"
+Requires-Dist: scrapling>=0.4.7; extra == "data"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+
+# fd-open-data-mcp
+
+An **open-data ontology MCP**: a semantic concept layer over multi-datasource
+financial/economic data. You ask for data in **concepts + entities** (e.g.
+"price.close for Moutai", "GDP for China"); the system resolves the concept to
+physical columns across datasources, ranks candidate sources by quality +
+accessibility, fetches from the best one (with failover), caches by concept,
+and refreshes on a per-concept frequency.
+
+It consumes the finddata `fd-*` datasource registries and
+`fd-entities-indicators` **read-only** and adds the unifying layers on top.
+
+## Architecture
+
+```
+CONSUMED (read-only)                  ADDED by fd-open-data-mcp
+ fd-akshare/yfinance/world/             concept_bindings   (column -> concept)
+ cn-report/cn-gov registries            entity_source_identifiers (per-source id)
+ fd-entities-indicators                 source_rankings    (quality × access × freshness)
+   indicator_defs (926 concepts)        semantic_observations (read-through cache)
+   countries/cities/symbols/sw_industries   fetch_log / schedules / executions
+        │
+   TRANSFORMERS: import_catalog, consume_concepts, propose_bindings,
+                 seed_entity_identifiers, generate_refresh_schedules
+        │
+   RUNTIME: read() -> cache hit? : dispatch (ranked, failover) -> cache -> log
+```
+
+Six capabilities (see `openspec/changes/add-fd-open-data-mcp/specs/`):
+`open-data-catalog`, `semantic-layer`, `entity-identity`, `source-ranking`,
+`concept-fetch`, `scheduled-refresh`.
+
+## Install
+
+```bash
+cd /Users/chengsishi/finddata/fd-open-data-mcp
+uv sync                  # base install
+uv sync --extra data     # + akshare / yfinance / world_bank_data (for real fetches)
+```
+
+The DB path defaults to `fd_open_data_mcp/metadata/daas.db`; override with
+`FD_OPEN_DATA_MCP_DATABASE_URL`. `FINDDATA_ROOT` (default: the parent
+`finddata/` dir) locates the `fd-*` providers. `EDGAR_IDENTITY` (an email)
+is required by the SEC before any edgar fetch - the runner refuses to call
+anonymously if it is unset.
+
+## Quickstart
+
+```bash
+# 1. create the ontology tables
+fd-open-data-mcp migrate
+
+# 2. import the catalogs (akshare 673, yfinance 12, cn-gov 11, cn-report 44, edgar 6, ...)
+fd-open-data-mcp import-catalog
+# or one provider:  fd-open-data-mcp import-catalog akshare
+
+# 3. consume the 926 indicator_defs as concepts + propose column->concept bindings
+fd-open-data-mcp consume-concepts
+fd-open-data-mcp propose-bindings
+
+# 4. seed per-source entity identifiers (akshare/yfinance for stocks, worldbank for countries)
+fd-open-data-mcp seed-entities
+
+# 5. generate per-concept refresh schedules from indicator_defs.frequency
+fd-open-data-mcp generate-schedules
+
+# 6. read data by concept + entity (read-through cache + ranked dispatch + failover)
+fd-open-data-mcp read --concept-id 234 --entity-type stock --entity-id 1 --date 2024-07-26
+```
+
+## MCP server
+
+```bash
+fd-open-data-mcp serve          # FastMCP, stdio transport
+```
+
+16 tools: `import_catalog`, `consume_concepts`, `propose_bindings`,
+`list_concepts`, `list_bindings`, `review_bindings`, `confirm_binding`,
+`seed_entity_identifiers`, `resolve_entity`, `add_entity_identifier`,
+`rank_sources`, `read`, `fetch`, `generate_refresh_schedules`,
+`list_schedules`, `run_schedule`.
+
+## Tests
+
+```bash
+uv run --with pytest pytest -q
+```
+
+## Design notes / v1 limitations
+
+- **Propose-and-confirm**: column->concept bindings carry `confidence` +
+  `provenance`; below-threshold bindings are withheld from dispatch (review
+  queue). A real fetch promotes a binding to `sample-confirmed`.
+- **Ranking** is per `(source × concept)`, self-tuning from `fetch_log`
+  (bounded so one failure can't remove a source).
+- **Conflict policy**: one cached value per `(concept, entity, date)` with
+  `source_used` attached; values are never merged across sources.
+- **LLM provider** for meaning-enrichment / cross-language concept mapping is
+  an open question (`design.md`); v1 uses a rule table + `semantic_type` hints.
+- `_build_params` / `_extract_value` in the fetch runner are best-effort; a
+  production runner refines per-function date-format / payload-shape quirks.
+
+See `openspec/changes/add-fd-open-data-mcp/` for the full spec.
+
+---
+
+### Usage Examples
+
+#### Query NBS GDP Data
+```python
+from fd_open_data_mcp.fetch.runner import run_upstream
+
+result = run_upstream(
+    source='nbs-gdp', 
+    command='get_gdp_quarterly',
+    params={'start_year': 2020}
+)
+print(result.head())
+```
+
+#### Query Steel Industry Production
+```python
+result = run_upstream(
+    source='cisa-industry',
+    command='get_steel_production',
+    params={}
+)
+print(result.head())
+```
+
+#### Query Metal Futures Pricing
+```python
+result = run_upstream(
+    source='shfe-metal-futures',
+    command='get_metal_pricing',
+    params={}
+)
+print(result.head())
+```
+
+### CLI Usage
+```bash
+# List all data sources
+fd-open-data-mcp list-sources
+
+# Read specific data
+fd-open-data-mcp read \
+  --source nbs-gdp \
+  --function get_gdp_quarterly \
+  --params '{"start_year": 2020}'
+```
+
+### MCP Server Mode
+```bash
+uv run fd-open-data-mcp serve
+# Then connect from Claude/Codex/etc.
+```
+
+
+---
+
+## Rate Limits & Best Practices
+
+### Recommended Refresh Intervals
+
+| Data Source Type | Refresh Interval | Notes |
+|------------------|------------------|-------|
+| **GDP/Macro** | Weekly | Stable data, updates quarterly/monthly |
+| **Industry Stats** | Daily | Can change frequently |
+| **Futures Prices** | Hourly during market hours | Volatile pricing |
+| **Fund Statistics** | Monthly | Updates monthly |
+| **Market Indices** | Real-time | High volatility |
+
+### API Rate Limiting
+
+- **Government APIs**: Respect 10 requests/minute default limits
+- **Exchange APIs**: Follow exchange-specific rate policies  
+- **Third-party Data**: Check individual terms of service
+- **Recommendation**: Implement exponential backoff on 429 errors
+
+### Caching Strategy
+
+All fetch results are automatically cached based on data frequency:
+- High-frequency data (futures): Cache for 1 hour
+- Medium-frequency data (industry stats): Cache for 24 hours
+- Low-frequency data (GDP, annual reports): Cache for 1 week
+
+Use `fd-open-data-mcp read` to check cache status.

fd_open_data_mcp-0.2.0/README.md

@@ -0,0 +1,188 @@
+# fd-open-data-mcp
+
+An **open-data ontology MCP**: a semantic concept layer over multi-datasource
+financial/economic data. You ask for data in **concepts + entities** (e.g.
+"price.close for Moutai", "GDP for China"); the system resolves the concept to
+physical columns across datasources, ranks candidate sources by quality +
+accessibility, fetches from the best one (with failover), caches by concept,
+and refreshes on a per-concept frequency.
+
+It consumes the finddata `fd-*` datasource registries and
+`fd-entities-indicators` **read-only** and adds the unifying layers on top.
+
+## Architecture
+
+```
+CONSUMED (read-only)                  ADDED by fd-open-data-mcp
+ fd-akshare/yfinance/world/             concept_bindings   (column -> concept)
+ cn-report/cn-gov registries            entity_source_identifiers (per-source id)
+ fd-entities-indicators                 source_rankings    (quality × access × freshness)
+   indicator_defs (926 concepts)        semantic_observations (read-through cache)
+   countries/cities/symbols/sw_industries   fetch_log / schedules / executions
+        │
+   TRANSFORMERS: import_catalog, consume_concepts, propose_bindings,
+                 seed_entity_identifiers, generate_refresh_schedules
+        │
+   RUNTIME: read() -> cache hit? : dispatch (ranked, failover) -> cache -> log
+```
+
+Six capabilities (see `openspec/changes/add-fd-open-data-mcp/specs/`):
+`open-data-catalog`, `semantic-layer`, `entity-identity`, `source-ranking`,
+`concept-fetch`, `scheduled-refresh`.
+
+## Install
+
+```bash
+cd /Users/chengsishi/finddata/fd-open-data-mcp
+uv sync                  # base install
+uv sync --extra data     # + akshare / yfinance / world_bank_data (for real fetches)
+```
+
+The DB path defaults to `fd_open_data_mcp/metadata/daas.db`; override with
+`FD_OPEN_DATA_MCP_DATABASE_URL`. `FINDDATA_ROOT` (default: the parent
+`finddata/` dir) locates the `fd-*` providers. `EDGAR_IDENTITY` (an email)
+is required by the SEC before any edgar fetch - the runner refuses to call
+anonymously if it is unset.
+
+## Quickstart
+
+```bash
+# 1. create the ontology tables
+fd-open-data-mcp migrate
+
+# 2. import the catalogs (akshare 673, yfinance 12, cn-gov 11, cn-report 44, edgar 6, ...)
+fd-open-data-mcp import-catalog
+# or one provider:  fd-open-data-mcp import-catalog akshare
+
+# 3. consume the 926 indicator_defs as concepts + propose column->concept bindings
+fd-open-data-mcp consume-concepts
+fd-open-data-mcp propose-bindings
+
+# 4. seed per-source entity identifiers (akshare/yfinance for stocks, worldbank for countries)
+fd-open-data-mcp seed-entities
+
+# 5. generate per-concept refresh schedules from indicator_defs.frequency
+fd-open-data-mcp generate-schedules
+
+# 6. read data by concept + entity (read-through cache + ranked dispatch + failover)
+fd-open-data-mcp read --concept-id 234 --entity-type stock --entity-id 1 --date 2024-07-26
+```
+
+## MCP server
+
+```bash
+fd-open-data-mcp serve          # FastMCP, stdio transport
+```
+
+16 tools: `import_catalog`, `consume_concepts`, `propose_bindings`,
+`list_concepts`, `list_bindings`, `review_bindings`, `confirm_binding`,
+`seed_entity_identifiers`, `resolve_entity`, `add_entity_identifier`,
+`rank_sources`, `read`, `fetch`, `generate_refresh_schedules`,
+`list_schedules`, `run_schedule`.
+
+## Tests
+
+```bash
+uv run --with pytest pytest -q
+```
+
+## Design notes / v1 limitations
+
+- **Propose-and-confirm**: column->concept bindings carry `confidence` +
+  `provenance`; below-threshold bindings are withheld from dispatch (review
+  queue). A real fetch promotes a binding to `sample-confirmed`.
+- **Ranking** is per `(source × concept)`, self-tuning from `fetch_log`
+  (bounded so one failure can't remove a source).
+- **Conflict policy**: one cached value per `(concept, entity, date)` with
+  `source_used` attached; values are never merged across sources.
+- **LLM provider** for meaning-enrichment / cross-language concept mapping is
+  an open question (`design.md`); v1 uses a rule table + `semantic_type` hints.
+- `_build_params` / `_extract_value` in the fetch runner are best-effort; a
+  production runner refines per-function date-format / payload-shape quirks.
+
+See `openspec/changes/add-fd-open-data-mcp/` for the full spec.
+
+---
+
+### Usage Examples
+
+#### Query NBS GDP Data
+```python
+from fd_open_data_mcp.fetch.runner import run_upstream
+
+result = run_upstream(
+    source='nbs-gdp', 
+    command='get_gdp_quarterly',
+    params={'start_year': 2020}
+)
+print(result.head())
+```
+
+#### Query Steel Industry Production
+```python
+result = run_upstream(
+    source='cisa-industry',
+    command='get_steel_production',
+    params={}
+)
+print(result.head())
+```
+
+#### Query Metal Futures Pricing
+```python
+result = run_upstream(
+    source='shfe-metal-futures',
+    command='get_metal_pricing',
+    params={}
+)
+print(result.head())
+```
+
+### CLI Usage
+```bash
+# List all data sources
+fd-open-data-mcp list-sources
+
+# Read specific data
+fd-open-data-mcp read \
+  --source nbs-gdp \
+  --function get_gdp_quarterly \
+  --params '{"start_year": 2020}'
+```
+
+### MCP Server Mode
+```bash
+uv run fd-open-data-mcp serve
+# Then connect from Claude/Codex/etc.
+```
+
+
+---
+
+## Rate Limits & Best Practices
+
+### Recommended Refresh Intervals
+
+| Data Source Type | Refresh Interval | Notes |
+|------------------|------------------|-------|
+| **GDP/Macro** | Weekly | Stable data, updates quarterly/monthly |
+| **Industry Stats** | Daily | Can change frequently |
+| **Futures Prices** | Hourly during market hours | Volatile pricing |
+| **Fund Statistics** | Monthly | Updates monthly |
+| **Market Indices** | Real-time | High volatility |
+
+### API Rate Limiting
+
+- **Government APIs**: Respect 10 requests/minute default limits
+- **Exchange APIs**: Follow exchange-specific rate policies  
+- **Third-party Data**: Check individual terms of service
+- **Recommendation**: Implement exponential backoff on 429 errors
+
+### Caching Strategy
+
+All fetch results are automatically cached based on data frequency:
+- High-frequency data (futures): Cache for 1 hour
+- Medium-frequency data (industry stats): Cache for 24 hours
+- Low-frequency data (GDP, annual reports): Cache for 1 week
+
+Use `fd-open-data-mcp read` to check cache status.

fd_open_data_mcp-0.2.0/fd_open_data_mcp/__init__.py

@@ -0,0 +1,9 @@
+"""fd-open-data-mcp: open-data ontology MCP.
+
+A semantic concept layer over multi-datasource financial/economic data.
+Consumes the fd-* datasource registries and fd-entities-indicators read-only,
+and adds: a unified catalog, concept<->column bindings, cross-source entity
+identity, ranked dispatch, a read-through concept-keyed cache, and
+frequency-driven auto-refresh.
+"""
+__version__ = "0.2.0"

fd_open_data_mcp-0.2.0/fd_open_data_mcp/adapters/__init__.py

@@ -0,0 +1,74 @@
+"""Per-function adapter registry: concrete param-building + value-extraction per
+``(source, command)``, shared by ``read()`` dispatch and the bulk crawl executor.
+
+Replaces the best-effort ``_build_params`` / ``_extract_value`` in
+``fetch/dispatch.py`` (design.md D4). Seeded by porting per-function logic from the
+``scraw-*`` projects (akshare first - tasks 2.2/2.3). Where no adapter is registered,
+callers fall back to the legacy best-effort mapping (coexistence during migration).
+
+An adapter is registered for a ``(source, command)`` key and implements two methods:
+
+  ``build_params(fn, identifier, date, binding) -> dict``
+      Build the concrete kwargs for the upstream callable. ``fn`` is the
+      ``Function`` row (with ``.parameters``); ``identifier`` is the per-source
+      entity id; ``binding`` is the ``ConceptBinding`` (its ``.column.name`` is used
+      for indicator-style params, e.g. wbgapi ``indicator=column.name``).
+
+  ``extract_value(result, column_name, date) -> value | None``
+      Pull the value for ``(date, column_name)`` from the upstream result.
+
+An optional ``call(command, params)`` method wraps the upstream callable (e.g. with a
+native timeout + retry); ``fetch/runner.py`` opts into it when present.
+
+Dispatch checks ``adapter_for(source, command)``; if present it delegates, otherwise
+it uses the legacy best-effort path. This keeps working reads from regressing while
+adapters are ported one function at a time.
+"""
+from __future__ import annotations
+
+from typing import Any, Optional, Protocol
+
+# Typed loosely to avoid an import cycle with fetch.dispatch / models.
+FunctionLike = Any
+BindingLike = Any
+
+
+class Adapter(Protocol):
+    """Per-function fetch mechanics: param mapping + value extraction."""
+
+    def build_params(
+        self, fn: FunctionLike, identifier: str, date: str, binding: BindingLike,
+    ) -> dict: ...
+
+    def extract_value(
+        self, result: Any, column_name: str, date: str,
+    ) -> Any: ...
+
+
+_REGISTRY: dict[tuple[str, str], Adapter] = {}
+
+
+def register(source: str, command: str, adapter: Adapter) -> Adapter:
+    """Register an adapter for a ``(source, command)`` key (idempotent overwrite)."""
+    _REGISTRY[(source, command)] = adapter
+    return adapter
+
+
+def adapter_for(source: str, command: str) -> Optional[Adapter]:
+    """Return the registered adapter for ``(source, command)``, or ``None``."""
+    return _REGISTRY.get((source, command))
+
+
+def has_adapter(source: str, command: str) -> bool:
+    """Whether a registered adapter exists for ``(source, command)``."""
+    return (source, command) in _REGISTRY
+
+
+def registered() -> list[tuple[str, str]]:
+    """All registered ``(source, command)`` keys (for introspection/debugging)."""
+    return sorted(_REGISTRY.keys())
+
+
+# Load akshare adapters so they register at package import. Imported last so
+# `register` is defined before akshare.py imports it (no cycle).
+from fd_open_data_mcp.adapters import akshare as _akshare_adapters  # noqa: E402,F401

fd_open_data_mcp-0.2.0/fd_open_data_mcp/adapters/agriculture.py

@@ -0,0 +1,4 @@
+
+def run_agriculture(command: str, params: dict):
+ """Stub runner for agriculture. Replace with actual implementation."""
+ raise NotImplementedError(f"{source} runner not yet implemented")