fd-open-data-protocol 0.2.0__tar.gz

fd_open_data_protocol-0.2.0/PKG-INFO

@@ -0,0 +1,232 @@
+Metadata-Version: 2.4
+Name: fd-open-data-protocol
+Version: 0.2.0
+Summary: The open-data datasource protocol: a manifest contract a datasource exposes to be ingested by fd-open-data-mcp (or any consumer).
+Author: FindDataOfficial
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+
+# fd-open-data-protocol
+
+**English** | [中文](README.zh-CN.md)
+
+The **open-data datasource protocol**: a manifest contract a datasource exposes
+(datasource + functions + columns + concept hints + fetch reference) so that
+`fd-open-data-mcp` - or any consumer - can ingest it via `register_datasource`.
+
+**Ship one manifest file -> the datasource is added. No consumer-side wiring.**
+
+## One-click install
+
+This library is a dependency of `fd-open-data-mcp` (pulled in transitively). To
+install the **entire** finddata stack (hub + every datasource + ontology DB):
+
+```bash
+pip install "fd-open-data-mcp[data]" fd-polygon fd-cn-report
+
+fd-open-data-mcp migrate \
+  && fd-open-data-mcp import-catalog \
+  && fd-open-data-mcp consume-concepts \
+  && fd-open-data-mcp propose-bindings \
+  && fd-open-data-mcp seed-entities \
+  && fd-open-data-mcp generate-schedules \
+  && fd-open-data-mcp register-discovered
+
+fd-open-data-mcp serve
+```
+
+## The manifest
+
+A YAML/JSON file (or a Python module exposing `CATALOG`):
+
+```yaml
+version: "1"
+name: my-source
+label: My Source
+ranking_seed: [0.7, 0.7]            # [quality, accessibility] heuristic seed
+functions:
+  - command: get_data
+    frequency: daily
+    parameters: [{name: symbol, type: str, required: true}]
+    columns:
+      - {name: close, type: float, frequency: daily}
+concepts:                           # column -> concept hints (measure/entity_type here)
+  - {column: close, concept: price.close, entity_type: stock, unit: currency, frequency: daily}
+fetch:
+  runner: my-source                  # built-in runner name, OR module: "pkg.mod:run"
+```
+
+See `examples/example_stock.yaml` (declarative) and `examples/example_macro.py`
+(a `DataProvider` class with `run()`).
+
+## Load + validate
+
+```python
+from fd_open_data_protocol.loader import load_catalog
+manifest = load_catalog("examples/example_stock.yaml")
+print(manifest.name, len(manifest.functions))
+```
+
+`load_catalog` accepts a YAML/JSON file path, a `.py` file exposing `CATALOG`,
+a `"pkg.mod"` module path, or a dict.
+
+## Register with fd-open-data-mcp
+
+```bash
+fd-open-data-mcp register-datasource examples/example_stock.yaml
+```
+
+or the MCP tool `register_datasource(path)`.
+
+## Publish a datasource from another project
+
+In your datasource package's `pyproject.toml`:
+
+```toml
+[project.entry-points."fd_open_data_mcp.datasources"]
+my-source = "my_pkg.catalog:CATALOG"
+```
+
+`pip install my-pkg` -> fd-open-data-mcp auto-registers it on `import_catalog`.
+
+## Schema
+
+- **`DatasourceManifest`**: name, label, source_url, scanner_mode, ranking_seed, functions[], concepts[], entities[], entity_definitions[], relationships[], fetch.
+- **`FunctionSpec`**: command, category, description, parameters[], columns[], frequency, verified.
+- **`ColumnSpec`**: name, type, description, meaning, semantic_type, `frequency` + `datasource` (column-level).
+- **`ConceptHint`**: column, concept, `entity_type`, `measure`, unit, frequency, confidence.
+- **`EntitySpec`**: entity_type, coverage ("universe"|"explicit"), codes[] (for explicit coverage).
+- **`Entity`**: entity_type, code, name_en, name_zh, metadata{}, relationships[].
+- **`EntityRelationship`**: target_entity_type, target_code, relation_type, confidence, metadata{}.
+- **`RelationshipSpec`**: relation_type, source_entity_type, target_entity_type, resolver_module.
+- **`FetchRef`**: runner (built-in name) | module (`"pkg.mod:func"`).
+
+`measure` + `entity_type` are **concept-level** (disambiguate GDP-nominal vs
+GDP-PPP; stock close vs fund NAV). Column-level `frequency`/`datasource` support
+composite functions whose columns come from different sources at different cadences.
+
+## Entity Definitions
+
+The protocol supports two ways to declare entities:
+
+### 1. Coverage Declaration (`entities[]`)
+
+Declares which entity types the datasource covers:
+
+```yaml
+entities:
+  - entity_type: stock
+    coverage: explicit
+    codes: [AAPL, MSFT, GOOGL]
+```
+
+- `coverage: "universe"` - datasource can fetch data for all entities of this type
+- `coverage: "explicit"` - datasource only covers the listed codes
+
+### 2. Entity Metadata (`entity_definitions[]`)
+
+Defines canonical entity metadata (names, attributes, relationships):
+
+```yaml
+entity_definitions:
+  - entity_type: stock
+    code: AAPL
+    name_en: Apple Inc.
+    name_zh: 苹果公司
+    metadata:
+      exchange: NASDAQ
+      sector: Technology
+    relationships:
+      - target_entity_type: industry
+        target_code: gics_10
+        relation_type: belongs_to
+```
+
+When included, entities are registered in the ontology database during `register_datasource()`.
+
+### Canonical Entity Types
+
+All `entity_type` values must be from this vocabulary:
+
+| Type | Description | Example IDs |
+|------|-------------|-------------|
+| `country` | ISO codes | CN, US, JP |
+| `city` | Municipalities | beijing, shanghai |
+| `stock` | A-shares | 600000.SH, 000001.SZ |
+| `fund` | ETFs/funds | etf_code, fund_code |
+| `bond` | Bonds | bond_code |
+| `index` | Indices | SH000001, SZ399001 |
+| `future` | Futures | cu2412, rb2401 |
+| `crypto` | Cryptocurrencies | btc, eth |
+| `organization` | General orgs | org_code |
+| `industry` | Classifications | shenwan_1_01, gics_10 |
+| `company` | Public companies | AAPL, TSLA |
+
+## Manifest Declaration Requirement
+
+**Every fd-* datasource package MUST declare a `DatasourceManifest` via one of the following mechanisms:**
+
+1. **Python module**: Expose a `CATALOG` dict in a module (e.g., `catalog.py`) that conforms to the `DatasourceManifest` schema
+2. **YAML/JSON file**: Place a manifest file at the package root (e.g., `catalog.yaml` or `catalog.json`)
+3. **Entry-point declaration**: Register the manifest path in `pyproject.toml` under `[project.entry-points."fd_open_data_mcp.datasources"]`
+
+The declaration **SHALL** be discoverable by `fd-open-data-mcp`'s auto-discovery mechanism (`register-discovered` command). Packages without a CATALOG declaration **SHALL NOT** be considered compliant with the fd-open-data-protocol.
+
+### Recommended Package Structure
+
+```
+my-datasource/
+├── pyproject.toml          # declares entry-point
+└── my_pkg/
+    ├── __init__.py
+    └── catalog.py          # exposes CATALOG = { ... }
+```
+
+### Entry-Point Declaration
+
+In your `pyproject.toml`:
+
+```toml
+[project.entry-points."fd_open_data_mcp.datasources"]
+my-source = "my_pkg.catalog:CATALOG"
+```
+
+After `pip install my-pkg`, the package is automatically discoverable:
+
+```bash
+fd-open-data-mcp register-discovered
+```
+
+### Auto-Discovery Flow
+
+1. **Install package** → `pip install my-datasource`
+2. **Entry-point registered** → setuptools records `my-source = "my_pkg.catalog:CATALOG"`
+3. **Auto-discover** → `fd-open-data-mcp register-discovered` scans all entry-points
+4. **Load manifest** → `load_catalog()` validates and parses the CATALOG dict
+5. **Register to ontology** → `register_datasource()` upserts sources/functions/columns/concepts
+
+### Compliance Checklist
+
+Before publishing a new datasource package, ensure:
+
+- [ ] Package exposes a `CATALOG` dict or manifest file
+- [ ] `pyproject.toml` declares entry-point under `fd_open_data_mcp.datasources` group
+- [ ] CATALOG conforms to `DatasourceManifest` schema (version, name, label, functions[], concepts[], fetch)
+- [ ] `load_catalog()` can successfully parse the manifest
+- [ ] `fd-open-data-mcp register-discovered` discovers and registers the package
+
+### Working Examples
+
+- **fd-world**: `fd_world/catalog.py` + entry-point in `pyproject.toml`
+- **fd-cn-gov**: `fd_cn_gov/catalog.py` + entry-point in `pyproject.toml`
+- **fd-cn-report**: `catalog.py` + entry-point in `pyproject.toml`
+
+## Template
+
+Copy `template/datasource.template.yaml` (declarative) or
+`template/provider_template.py` (a `BaseDataProvider` class with `run()`).

fd_open_data_protocol-0.2.0/README.md

@@ -0,0 +1,219 @@
+# fd-open-data-protocol
+
+**English** | [中文](README.zh-CN.md)
+
+The **open-data datasource protocol**: a manifest contract a datasource exposes
+(datasource + functions + columns + concept hints + fetch reference) so that
+`fd-open-data-mcp` - or any consumer - can ingest it via `register_datasource`.
+
+**Ship one manifest file -> the datasource is added. No consumer-side wiring.**
+
+## One-click install
+
+This library is a dependency of `fd-open-data-mcp` (pulled in transitively). To
+install the **entire** finddata stack (hub + every datasource + ontology DB):
+
+```bash
+pip install "fd-open-data-mcp[data]" fd-polygon fd-cn-report
+
+fd-open-data-mcp migrate \
+  && fd-open-data-mcp import-catalog \
+  && fd-open-data-mcp consume-concepts \
+  && fd-open-data-mcp propose-bindings \
+  && fd-open-data-mcp seed-entities \
+  && fd-open-data-mcp generate-schedules \
+  && fd-open-data-mcp register-discovered
+
+fd-open-data-mcp serve
+```
+
+## The manifest
+
+A YAML/JSON file (or a Python module exposing `CATALOG`):
+
+```yaml
+version: "1"
+name: my-source
+label: My Source
+ranking_seed: [0.7, 0.7]            # [quality, accessibility] heuristic seed
+functions:
+  - command: get_data
+    frequency: daily
+    parameters: [{name: symbol, type: str, required: true}]
+    columns:
+      - {name: close, type: float, frequency: daily}
+concepts:                           # column -> concept hints (measure/entity_type here)
+  - {column: close, concept: price.close, entity_type: stock, unit: currency, frequency: daily}
+fetch:
+  runner: my-source                  # built-in runner name, OR module: "pkg.mod:run"
+```
+
+See `examples/example_stock.yaml` (declarative) and `examples/example_macro.py`
+(a `DataProvider` class with `run()`).
+
+## Load + validate
+
+```python
+from fd_open_data_protocol.loader import load_catalog
+manifest = load_catalog("examples/example_stock.yaml")
+print(manifest.name, len(manifest.functions))
+```
+
+`load_catalog` accepts a YAML/JSON file path, a `.py` file exposing `CATALOG`,
+a `"pkg.mod"` module path, or a dict.
+
+## Register with fd-open-data-mcp
+
+```bash
+fd-open-data-mcp register-datasource examples/example_stock.yaml
+```
+
+or the MCP tool `register_datasource(path)`.
+
+## Publish a datasource from another project
+
+In your datasource package's `pyproject.toml`:
+
+```toml
+[project.entry-points."fd_open_data_mcp.datasources"]
+my-source = "my_pkg.catalog:CATALOG"
+```
+
+`pip install my-pkg` -> fd-open-data-mcp auto-registers it on `import_catalog`.
+
+## Schema
+
+- **`DatasourceManifest`**: name, label, source_url, scanner_mode, ranking_seed, functions[], concepts[], entities[], entity_definitions[], relationships[], fetch.
+- **`FunctionSpec`**: command, category, description, parameters[], columns[], frequency, verified.
+- **`ColumnSpec`**: name, type, description, meaning, semantic_type, `frequency` + `datasource` (column-level).
+- **`ConceptHint`**: column, concept, `entity_type`, `measure`, unit, frequency, confidence.
+- **`EntitySpec`**: entity_type, coverage ("universe"|"explicit"), codes[] (for explicit coverage).
+- **`Entity`**: entity_type, code, name_en, name_zh, metadata{}, relationships[].
+- **`EntityRelationship`**: target_entity_type, target_code, relation_type, confidence, metadata{}.
+- **`RelationshipSpec`**: relation_type, source_entity_type, target_entity_type, resolver_module.
+- **`FetchRef`**: runner (built-in name) | module (`"pkg.mod:func"`).
+
+`measure` + `entity_type` are **concept-level** (disambiguate GDP-nominal vs
+GDP-PPP; stock close vs fund NAV). Column-level `frequency`/`datasource` support
+composite functions whose columns come from different sources at different cadences.
+
+## Entity Definitions
+
+The protocol supports two ways to declare entities:
+
+### 1. Coverage Declaration (`entities[]`)
+
+Declares which entity types the datasource covers:
+
+```yaml
+entities:
+  - entity_type: stock
+    coverage: explicit
+    codes: [AAPL, MSFT, GOOGL]
+```
+
+- `coverage: "universe"` - datasource can fetch data for all entities of this type
+- `coverage: "explicit"` - datasource only covers the listed codes
+
+### 2. Entity Metadata (`entity_definitions[]`)
+
+Defines canonical entity metadata (names, attributes, relationships):
+
+```yaml
+entity_definitions:
+  - entity_type: stock
+    code: AAPL
+    name_en: Apple Inc.
+    name_zh: 苹果公司
+    metadata:
+      exchange: NASDAQ
+      sector: Technology
+    relationships:
+      - target_entity_type: industry
+        target_code: gics_10
+        relation_type: belongs_to
+```
+
+When included, entities are registered in the ontology database during `register_datasource()`.
+
+### Canonical Entity Types
+
+All `entity_type` values must be from this vocabulary:
+
+| Type | Description | Example IDs |
+|------|-------------|-------------|
+| `country` | ISO codes | CN, US, JP |
+| `city` | Municipalities | beijing, shanghai |
+| `stock` | A-shares | 600000.SH, 000001.SZ |
+| `fund` | ETFs/funds | etf_code, fund_code |
+| `bond` | Bonds | bond_code |
+| `index` | Indices | SH000001, SZ399001 |
+| `future` | Futures | cu2412, rb2401 |
+| `crypto` | Cryptocurrencies | btc, eth |
+| `organization` | General orgs | org_code |
+| `industry` | Classifications | shenwan_1_01, gics_10 |
+| `company` | Public companies | AAPL, TSLA |
+
+## Manifest Declaration Requirement
+
+**Every fd-* datasource package MUST declare a `DatasourceManifest` via one of the following mechanisms:**
+
+1. **Python module**: Expose a `CATALOG` dict in a module (e.g., `catalog.py`) that conforms to the `DatasourceManifest` schema
+2. **YAML/JSON file**: Place a manifest file at the package root (e.g., `catalog.yaml` or `catalog.json`)
+3. **Entry-point declaration**: Register the manifest path in `pyproject.toml` under `[project.entry-points."fd_open_data_mcp.datasources"]`
+
+The declaration **SHALL** be discoverable by `fd-open-data-mcp`'s auto-discovery mechanism (`register-discovered` command). Packages without a CATALOG declaration **SHALL NOT** be considered compliant with the fd-open-data-protocol.
+
+### Recommended Package Structure
+
+```
+my-datasource/
+├── pyproject.toml          # declares entry-point
+└── my_pkg/
+    ├── __init__.py
+    └── catalog.py          # exposes CATALOG = { ... }
+```
+
+### Entry-Point Declaration
+
+In your `pyproject.toml`:
+
+```toml
+[project.entry-points."fd_open_data_mcp.datasources"]
+my-source = "my_pkg.catalog:CATALOG"
+```
+
+After `pip install my-pkg`, the package is automatically discoverable:
+
+```bash
+fd-open-data-mcp register-discovered
+```
+
+### Auto-Discovery Flow
+
+1. **Install package** → `pip install my-datasource`
+2. **Entry-point registered** → setuptools records `my-source = "my_pkg.catalog:CATALOG"`
+3. **Auto-discover** → `fd-open-data-mcp register-discovered` scans all entry-points
+4. **Load manifest** → `load_catalog()` validates and parses the CATALOG dict
+5. **Register to ontology** → `register_datasource()` upserts sources/functions/columns/concepts
+
+### Compliance Checklist
+
+Before publishing a new datasource package, ensure:
+
+- [ ] Package exposes a `CATALOG` dict or manifest file
+- [ ] `pyproject.toml` declares entry-point under `fd_open_data_mcp.datasources` group
+- [ ] CATALOG conforms to `DatasourceManifest` schema (version, name, label, functions[], concepts[], fetch)
+- [ ] `load_catalog()` can successfully parse the manifest
+- [ ] `fd-open-data-mcp register-discovered` discovers and registers the package
+
+### Working Examples
+
+- **fd-world**: `fd_world/catalog.py` + entry-point in `pyproject.toml`
+- **fd-cn-gov**: `fd_cn_gov/catalog.py` + entry-point in `pyproject.toml`
+- **fd-cn-report**: `catalog.py` + entry-point in `pyproject.toml`
+
+## Template
+
+Copy `template/datasource.template.yaml` (declarative) or
+`template/provider_template.py` (a `BaseDataProvider` class with `run()`).

fd_open_data_protocol-0.2.0/fd_open_data_protocol/__init__.py

@@ -0,0 +1,8 @@
+"""fd-open-data-protocol: the open-data datasource protocol.
+
+A manifest contract a datasource exposes (datasource + functions + columns +
+concept hints + fetch reference) so that fd-open-data-mcp - or any consumer -
+can ingest it via ``register_datasource``. Ship one manifest file -> the
+datasource is added; no consumer-side wiring.
+"""
+__version__ = "0.2.0"

fd_open_data_protocol-0.2.0/fd_open_data_protocol/constants.py

@@ -0,0 +1,196 @@
+"""Industry classification constants and helpers.
+
+This module defines canonical industry classification systems and provides
+helper functions for parsing industry codes.
+"""
+
+from typing import Optional
+import re
+
+
+# Canonical industry classification systems
+CLASSIFICATION_SYSTEMS = {
+    "shenwan": {
+        "name": "申万行业分类",
+        "name_en": "Shenwan Industry Classification",
+        "levels": 3,
+        "prefix": "shenwan_",
+        "code_pattern": r"^shenwan_(\d)_(\d{2,6})$",
+        "description": "Chinese industry classification by Shenwan Research"
+    },
+    "gics": {
+        "name": "全球行业分类标准",
+        "name_en": "Global Industry Classification Standard",
+        "levels": 4,
+        "prefix": "gics_",
+        "code_pattern": r"^gics_(\d{2,8})$",
+        "description": "Global industry classification by MSCI and S&P"
+    },
+}
+
+
+def parse_industry_code(code: str) -> Optional[dict]:
+    """Parse an industry code and extract classification system and level.
+
+    Args:
+        code: Industry code (e.g., "shenwan_1_01", "gics_50")
+
+    Returns:
+        Dict with keys: system, level, numeric_code, or None if invalid
+
+    Examples:
+        >>> parse_industry_code("shenwan_1_01")
+        {'system': 'shenwan', 'level': 1, 'numeric_code': '01'}
+        >>> parse_industry_code("gics_50")
+        {'system': 'gics', 'level': None, 'numeric_code': '50'}
+    """
+    for system, config in CLASSIFICATION_SYSTEMS.items():
+        pattern = re.compile(config["code_pattern"])
+        match = pattern.match(code)
+        if match:
+            groups = match.groups()
+            if system == "shenwan":
+                # shenwan_1_01 -> level=1, numeric_code=01
+                return {
+                    "system": system,
+                    "level": int(groups[0]),
+                    "numeric_code": groups[1]
+                }
+            elif system == "gics":
+                # gics_50 -> level=None (determined by code length), numeric_code=50
+                numeric = groups[0]
+                # GICS levels: 2-digit=sector, 4-digit=industry group, 6-digit=industry, 8-digit=sub-industry
+                if len(numeric) == 2:
+                    level = 1
+                elif len(numeric) == 4:
+                    level = 2
+                elif len(numeric) == 6:
+                    level = 3
+                elif len(numeric) == 8:
+                    level = 4
+                else:
+                    level = None
+                return {
+                    "system": system,
+                    "level": level,
+                    "numeric_code": numeric
+                }
+    return None
+
+
+def validate_industry_code(code: str) -> bool:
+    """Validate an industry code against canonical patterns.
+
+    Args:
+        code: Industry code to validate
+
+    Returns:
+        True if valid, False otherwise
+
+    Examples:
+        >>> validate_industry_code("shenwan_1_01")
+        True
+        >>> validate_industry_code("invalid_code")
+        False
+    """
+    return parse_industry_code(code) is not None
+
+
+def get_classification_system(code: str) -> Optional[str]:
+    """Get the classification system name from an industry code.
+
+    Args:
+        code: Industry code
+
+    Returns:
+        System name (e.g., "shenwan", "gics") or None if invalid
+
+    Examples:
+        >>> get_classification_system("shenwan_1_01")
+        'shenwan'
+    """
+    result = parse_industry_code(code)
+    return result["system"] if result else None
+
+
+def get_industry_level(code: str) -> Optional[int]:
+    """Get the hierarchy level from an industry code.
+
+    Args:
+        code: Industry code
+
+    Returns:
+        Level (1, 2, 3, or 4) or None if invalid
+
+    Examples:
+        >>> get_industry_level("shenwan_1_01")
+        1
+        >>> get_industry_level("gics_50")
+        1
+    """
+    result = parse_industry_code(code)
+    return result["level"] if result else None
+
+
+def format_industry_code(system: str, level: Optional[int], numeric_code: str) -> str:
+    """Format an industry code from components.
+
+    Args:
+        system: Classification system (e.g., "shenwan", "gics")
+        level: Hierarchy level (for shenwan only)
+        numeric_code: Numeric code portion
+
+    Returns:
+        Formatted industry code
+
+    Examples:
+        >>> format_industry_code("shenwan", 1, "01")
+        'shenwan_1_01'
+        >>> format_industry_code("gics", None, "50")
+        'gics_50'
+    """
+    if system == "shenwan":
+        if level is None:
+            raise ValueError("Shenwan classification requires level parameter")
+        return f"shenwan_{level}_{numeric_code}"
+    elif system == "gics":
+        return f"gics_{numeric_code}"
+    else:
+        raise ValueError(f"Unknown classification system: {system}")
+
+
+# Common industry codes for reference
+COMMON_INDUSTRIES = {
+    "shenwan": {
+        "1": {
+            "01": "农林牧渔",
+            "02": "采掘",
+            "03": "化工",
+            "04": "钢铁",
+            "05": "有色金属",
+            "06": "建筑材料",
+            "07": "建筑装饰",
+            "08": "电气设备",
+            "09": "国防军工",
+            "10": "汽车",
+            "11": "机械设备",
+            "12": "休闲服务",
+            "13": "家用电器",
+            "14": "纺织服装",
+            "15": "轻工制造",
+            "16": "商业贸易",
+            "17": "食品饮料",
+            "18": "医药生物",
+            "19": "公用事业",
+            "20": "交通运输",
+            "21": "房地产",
+            "22": "电子",
+            "23": "通信",
+            "24": "计算机",
+            "25": "传媒",
+            "26": "银行",
+            "27": "非银金融",
+            "28": "综合",
+        }
+    }
+}