fd-cn-gov 0.1.1__tar.gz

fd_cn_gov-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 cli-anything
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fd_cn_gov-0.1.1/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.4
+Name: fd-cn-gov
+Version: 0.1.1
+Summary: Scrapers + datasource registry for Chinese central-government ministry open-information archives.
+Author: cli-anything
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/FindDataOfficial/cn-goverment-datasource
+Project-URL: Repository, https://github.com/FindDataOfficial/cn-goverment-datasource
+Project-URL: Changelog, https://github.com/FindDataOfficial/cn-goverment-datasource/releases
+Project-URL: Issues, https://github.com/FindDataOfficial/cn-goverment-datasource/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: scrapling[fetchers]
+Requires-Dist: sqlalchemy>=2.0
+Dynamic: license-file
+
+# fd-cn-gov
+
+[![PyPI version](https://img.shields.io/pypi/v/fd-cn-gov.svg)](https://pypi.org/project/fd-cn-gov/)
+[![Python versions](https://img.shields.io/pypi/pyversions/fd-cn-gov.svg)](https://pypi.org/project/fd-cn-gov/)
+[![License: MIT](https://img.shields.io/pypi/l/fd-cn-gov.svg)](https://github.com/FindDataOfficial/cn-goverment-datasource/blob/main/LICENSE)
+
+Scrapers + a self-contained datasource registry for **Chinese central-government ministry open-information archives**. Catalog-crawls the public notice / news / data archives of 11 ministries (MOF, PBC, NDRC, MOFCOM, MOHURD, MOT, MOA, SAFE, MNR, MEE, MEM), emitting one JSON record per listed document, and ships a SQLite + JSON registry describing every datasource and its column schema.
+
+Standalone: no monorepo, no MCP server, no `daas.db` dependency. The `MANIFEST` at the top of each scraper is the single source of truth; the bundled `registry.db` / `registry.json` are derived artifacts.
+
+## Ministries
+
+| Name | Label | Seed URL |
+|------|-------|----------|
+| `mee_gsgg_archive` | MEE Notice Archive (生态环境部公示公告) | https://www.mee.gov.cn/ywdt/gsgg/ |
+| `mem_tzgg_archive` | MEM Notice Archive (应急管理部通知公告) | https://www.mem.gov.cn/gk/tzgg/ |
+| `mnr_tzgg_archive` | MNR Notice Archive (自然资源部通知公告) | https://www.mnr.gov.cn/gk/tzgg/ |
+| `moa_govpublic_archive` | MOA GovPublic Archive (农业农村部 机构分类) | https://www.moa.gov.cn/govpublic/ |
+| `mof_gkml_archive` | MOF gkml Archive (财政部信息公开) | https://www.mof.gov.cn/gkml/ |
+| `mofcom_xwfb_archive` | MOFCOM News Preview (商务部新闻发布) | https://www.mofcom.gov.cn/xwfb/index.html |
+| `mohurd_xinwen_archive` | MOHURD Xinwen Archive (住建部新闻动态) | https://www.mohurd.gov.cn/xinwen/ |
+| `mot_shuju_archive` | MOT Data Hub Archive (交通运输部数据) | https://www.mot.gov.cn/shuju/index.html |
+| `ndrc_tzgg_archive` | NDRC Notice Archive (发改委通知公告) | https://www.ndrc.gov.cn/xwdt/tzgg/ |
+| `pbc_xinwen_archive` | PBC News Archive (人民银行新闻发布) | https://www.pbc.gov.cn/goutongjiaoliu/113456/113469/index.html |
+| `safe_whxw_archive` | SAFE News Archive (外汇局外汇新闻) | https://www.safe.gov.cn/safe/whxw/index.html |
+
+Each scraper emits a JSON array of records to stdout with at least `title`, `date` (`YYYY-MM-DD`, from the URL `t<YYYYMMDD>_` token with a `<span>` fallback), and `url` (absolute, the primary key). Some add `section` / `subsection` / `doc_type` / `department`. See `fd-cn-gov describe <name>` for the exact columns of any source.
+
+## Install
+
+```bash
+pip install fd-cn-gov
+```
+
+Requires Python ≥3.10. Dependencies: `scrapling` (HTTP + adaptive parsing), `sqlalchemy`.
+
+### From source
+
+```bash
+pip install git+https://github.com/FindDataOfficial/cn-goverment-datasource.git
+```
+
+## CLI
+
+```bash
+# List the 11 registered sources
+fd-cn-gov list
+
+# Show one source's identity + full column schema
+fd-cn-gov describe mof_gkml_archive
+
+# Crawl one archive (default: 50 pages per sub-archive; prints JSON to stdout)
+fd-cn-gov crawl mof_gkml_archive --max-pages 2 > records.json
+
+# Full crawl, no page cap (use sparingly — see Polite crawling below)
+fd-cn-gov crawl mof_gkml_archive --all > records.json
+
+# Regenerate the bundled registry from the scripts' MANIFESTs
+fd-cn-gov build-registry
+```
+
+## Python API
+
+```python
+import fd_cn_gov
+
+# Discover datasources
+for s in fd_cn_gov.list_sources():
+    print(s.name, s.label, s.url)
+
+# One source + its column schema
+src = fd_cn_gov.get_source("mof_gkml_archive")
+cols = fd_cn_gov.get_columns("mof_gkml_archive")  # -> list[Column]
+#   Column(name='url', type='string', primary_key=True, nullable=False,
+#          description='absolute document URL (.htm/.html/.pdf)',
+#          source_field='a@href', semantic_type='url', ...)
+```
+
+## Registry schema
+
+The bundled `fd_cn_gov/registry/registry.db` is a 3-table SQLite, mirroring the column shapes of the originating `daas.db` for these sources (no foreign keys, no stale-FK footgun):
+
+- **`sources`** — one row per ministry: `id, name, label, url, description, category, category_label, config_json`
+- **`datasource_columns`** — one row per output column: `datasource_id, table_name, column_name, column_type, is_primary_key, is_nullable, description, source_field, unit, semantic_type`
+- **`scraw_configs`** — one row per scraper: `name, url, columns_json`
+
+`registry.json` is a deterministic full dump of the same three tables (`sort_keys`), so the registry is diff-friendly in code review.
+
+To regenerate after editing a `MANIFEST`:
+
+```bash
+fd-cn-gov build-registry
+```
+
+The build is **logically idempotent**: re-running produces an identical `.dump` and a byte-identical `registry.json`. (Only SQLite's file-change-counter header byte differs on each write — unavoidable on any SQLite write; compare via `.dump` or `registry.json` for equality.)
+
+## Polite crawling
+
+These are `.gov.cn` hosts. Each scraper paces requests (`SLEEP ≈ 0.3s` between pages) and defaults to a **50-page cap per sub-archive**. Use `--max-pages N` to bound a crawl further; reserve `--all` for when you genuinely need full history and can afford the time. Do not run multiple `--all` crawls in parallel against the same ministry.
+
+## Project layout
+
+```
+fd-cn-gov/
+├── pyproject.toml
+├── README.md
+├── LICENSE
+└── fd_cn_gov/
+    ├── __init__.py            # public read API: list_sources / get_source / get_columns
+    ├── cli.py                 # fd-cn-gov CLI
+    ├── scraw_contract.py      # ScrawManifest / ScrawColumn dataclasses (vendored, trimmed)
+    ├── build_registry.py      # regenerates registry.db + registry.json from MANIFESTs
+    ├── scripts/               # 11 ministry scrapers, each with a module-level MANIFEST
+    │   ├── mof_gkml_archive.py
+    │   └── ...
+    └── registry/
+        ├── __init__.py        # read API over the bundled DB
+        ├── registry.db        # generated — checked in
+        └── registry.json      # generated — checked in
+```
+
+## License
+
+MIT

fd_cn_gov-0.1.1/README.md

@@ -0,0 +1,124 @@
+# fd-cn-gov
+
+[![PyPI version](https://img.shields.io/pypi/v/fd-cn-gov.svg)](https://pypi.org/project/fd-cn-gov/)
+[![Python versions](https://img.shields.io/pypi/pyversions/fd-cn-gov.svg)](https://pypi.org/project/fd-cn-gov/)
+[![License: MIT](https://img.shields.io/pypi/l/fd-cn-gov.svg)](https://github.com/FindDataOfficial/cn-goverment-datasource/blob/main/LICENSE)
+
+Scrapers + a self-contained datasource registry for **Chinese central-government ministry open-information archives**. Catalog-crawls the public notice / news / data archives of 11 ministries (MOF, PBC, NDRC, MOFCOM, MOHURD, MOT, MOA, SAFE, MNR, MEE, MEM), emitting one JSON record per listed document, and ships a SQLite + JSON registry describing every datasource and its column schema.
+
+Standalone: no monorepo, no MCP server, no `daas.db` dependency. The `MANIFEST` at the top of each scraper is the single source of truth; the bundled `registry.db` / `registry.json` are derived artifacts.
+
+## Ministries
+
+| Name | Label | Seed URL |
+|------|-------|----------|
+| `mee_gsgg_archive` | MEE Notice Archive (生态环境部公示公告) | https://www.mee.gov.cn/ywdt/gsgg/ |
+| `mem_tzgg_archive` | MEM Notice Archive (应急管理部通知公告) | https://www.mem.gov.cn/gk/tzgg/ |
+| `mnr_tzgg_archive` | MNR Notice Archive (自然资源部通知公告) | https://www.mnr.gov.cn/gk/tzgg/ |
+| `moa_govpublic_archive` | MOA GovPublic Archive (农业农村部 机构分类) | https://www.moa.gov.cn/govpublic/ |
+| `mof_gkml_archive` | MOF gkml Archive (财政部信息公开) | https://www.mof.gov.cn/gkml/ |
+| `mofcom_xwfb_archive` | MOFCOM News Preview (商务部新闻发布) | https://www.mofcom.gov.cn/xwfb/index.html |
+| `mohurd_xinwen_archive` | MOHURD Xinwen Archive (住建部新闻动态) | https://www.mohurd.gov.cn/xinwen/ |
+| `mot_shuju_archive` | MOT Data Hub Archive (交通运输部数据) | https://www.mot.gov.cn/shuju/index.html |
+| `ndrc_tzgg_archive` | NDRC Notice Archive (发改委通知公告) | https://www.ndrc.gov.cn/xwdt/tzgg/ |
+| `pbc_xinwen_archive` | PBC News Archive (人民银行新闻发布) | https://www.pbc.gov.cn/goutongjiaoliu/113456/113469/index.html |
+| `safe_whxw_archive` | SAFE News Archive (外汇局外汇新闻) | https://www.safe.gov.cn/safe/whxw/index.html |
+
+Each scraper emits a JSON array of records to stdout with at least `title`, `date` (`YYYY-MM-DD`, from the URL `t<YYYYMMDD>_` token with a `<span>` fallback), and `url` (absolute, the primary key). Some add `section` / `subsection` / `doc_type` / `department`. See `fd-cn-gov describe <name>` for the exact columns of any source.
+
+## Install
+
+```bash
+pip install fd-cn-gov
+```
+
+Requires Python ≥3.10. Dependencies: `scrapling` (HTTP + adaptive parsing), `sqlalchemy`.
+
+### From source
+
+```bash
+pip install git+https://github.com/FindDataOfficial/cn-goverment-datasource.git
+```
+
+## CLI
+
+```bash
+# List the 11 registered sources
+fd-cn-gov list
+
+# Show one source's identity + full column schema
+fd-cn-gov describe mof_gkml_archive
+
+# Crawl one archive (default: 50 pages per sub-archive; prints JSON to stdout)
+fd-cn-gov crawl mof_gkml_archive --max-pages 2 > records.json
+
+# Full crawl, no page cap (use sparingly — see Polite crawling below)
+fd-cn-gov crawl mof_gkml_archive --all > records.json
+
+# Regenerate the bundled registry from the scripts' MANIFESTs
+fd-cn-gov build-registry
+```
+
+## Python API
+
+```python
+import fd_cn_gov
+
+# Discover datasources
+for s in fd_cn_gov.list_sources():
+    print(s.name, s.label, s.url)
+
+# One source + its column schema
+src = fd_cn_gov.get_source("mof_gkml_archive")
+cols = fd_cn_gov.get_columns("mof_gkml_archive")  # -> list[Column]
+#   Column(name='url', type='string', primary_key=True, nullable=False,
+#          description='absolute document URL (.htm/.html/.pdf)',
+#          source_field='a@href', semantic_type='url', ...)
+```
+
+## Registry schema
+
+The bundled `fd_cn_gov/registry/registry.db` is a 3-table SQLite, mirroring the column shapes of the originating `daas.db` for these sources (no foreign keys, no stale-FK footgun):
+
+- **`sources`** — one row per ministry: `id, name, label, url, description, category, category_label, config_json`
+- **`datasource_columns`** — one row per output column: `datasource_id, table_name, column_name, column_type, is_primary_key, is_nullable, description, source_field, unit, semantic_type`
+- **`scraw_configs`** — one row per scraper: `name, url, columns_json`
+
+`registry.json` is a deterministic full dump of the same three tables (`sort_keys`), so the registry is diff-friendly in code review.
+
+To regenerate after editing a `MANIFEST`:
+
+```bash
+fd-cn-gov build-registry
+```
+
+The build is **logically idempotent**: re-running produces an identical `.dump` and a byte-identical `registry.json`. (Only SQLite's file-change-counter header byte differs on each write — unavoidable on any SQLite write; compare via `.dump` or `registry.json` for equality.)
+
+## Polite crawling
+
+These are `.gov.cn` hosts. Each scraper paces requests (`SLEEP ≈ 0.3s` between pages) and defaults to a **50-page cap per sub-archive**. Use `--max-pages N` to bound a crawl further; reserve `--all` for when you genuinely need full history and can afford the time. Do not run multiple `--all` crawls in parallel against the same ministry.
+
+## Project layout
+
+```
+fd-cn-gov/
+├── pyproject.toml
+├── README.md
+├── LICENSE
+└── fd_cn_gov/
+    ├── __init__.py            # public read API: list_sources / get_source / get_columns
+    ├── cli.py                 # fd-cn-gov CLI
+    ├── scraw_contract.py      # ScrawManifest / ScrawColumn dataclasses (vendored, trimmed)
+    ├── build_registry.py      # regenerates registry.db + registry.json from MANIFESTs
+    ├── scripts/               # 11 ministry scrapers, each with a module-level MANIFEST
+    │   ├── mof_gkml_archive.py
+    │   └── ...
+    └── registry/
+        ├── __init__.py        # read API over the bundled DB
+        ├── registry.db        # generated — checked in
+        └── registry.json      # generated — checked in
+```
+
+## License
+
+MIT

fd_cn_gov-0.1.1/fd_cn_gov/__init__.py

@@ -0,0 +1,6 @@
+"""fd_cn_gov — ministry open-information scrapers + datasource registry."""
+
+from fd_cn_gov.registry import list_sources, get_source, get_columns
+
+__all__ = ["list_sources", "get_source", "get_columns"]
+__version__ = "0.1.1"

fd_cn_gov-0.1.1/fd_cn_gov/build_registry.py

@@ -0,0 +1,185 @@
+#!/usr/bin/env python3
+"""Regenerate the bundled registry.db + registry.json from the scripts' MANIFESTs.
+
+The MANIFEST is the source of truth; the DB is derived. Idempotent: re-running
+produces a byte-identical DB (delete-then-insert per source, deterministic
+JSON sort_keys).
+
+Usage: `python -m fd_cn_gov.build_registry` or `fd-cn-gov build-registry`.
+"""
+from __future__ import annotations
+
+import importlib
+import json
+import sqlite3
+from pathlib import Path
+
+# The 11 ministry scrapers, alphabetical by module name (deterministic order).
+SCRIPT_NAMES = [
+    "mee_gsgg_archive",
+    "mem_tzgg_archive",
+    "mnr_tzgg_archive",
+    "moa_govpublic_archive",
+    "mof_gkml_archive",
+    "mofcom_xwfb_archive",
+    "mohurd_xinwen_archive",
+    "mot_shuju_archive",
+    "ndrc_tzgg_archive",
+    "pbc_xinwen_archive",
+    "safe_whxw_archive",
+]
+
+REGISTRY_DIR = Path(__file__).resolve().parent / "registry"
+DB_PATH = REGISTRY_DIR / "registry.db"
+JSON_PATH = REGISTRY_DIR / "registry.json"
+
+
+def _load_manifests() -> list:
+    manifests = []
+    for name in SCRIPT_NAMES:
+        mod = importlib.import_module(f"fd_cn_gov.scripts.{name}")
+        m = getattr(mod, "MANIFEST", None)
+        if m is None or m.name != name:
+            raise RuntimeError(f"{name}: MANIFEST missing or name mismatch")
+        manifests.append(m)
+    return manifests
+
+
+def _create_schema(conn: sqlite3.Connection) -> None:
+    conn.executescript(
+        """
+        DROP TABLE IF EXISTS datasource_columns;
+        DROP TABLE IF EXISTS scraw_configs;
+        DROP TABLE IF EXISTS sources;
+
+        CREATE TABLE sources (
+            id              INTEGER PRIMARY KEY,
+            name            TEXT NOT NULL UNIQUE,
+            label           TEXT NOT NULL,
+            url             TEXT NOT NULL,
+            description     TEXT NOT NULL DEFAULT '',
+            category        TEXT NOT NULL DEFAULT '',
+            category_label  TEXT NOT NULL DEFAULT '',
+            config_json     TEXT NOT NULL DEFAULT '{}'
+        );
+
+        CREATE TABLE datasource_columns (
+            id              INTEGER PRIMARY KEY,
+            datasource_id   INTEGER NOT NULL,
+            table_name      TEXT NOT NULL,
+            column_name     TEXT NOT NULL,
+            column_type     TEXT NOT NULL,
+            is_primary_key  INTEGER NOT NULL DEFAULT 0,
+            is_nullable     INTEGER NOT NULL DEFAULT 1,
+            description     TEXT NOT NULL DEFAULT '',
+            source_field    TEXT NOT NULL DEFAULT '',
+            unit            TEXT NOT NULL DEFAULT '',
+            semantic_type   TEXT NOT NULL DEFAULT ''
+        );
+
+        CREATE TABLE scraw_configs (
+            id              INTEGER PRIMARY KEY,
+            name            TEXT NOT NULL UNIQUE,
+            url             TEXT NOT NULL,
+            columns_json    TEXT NOT NULL DEFAULT '[]'
+        );
+        """
+    )
+
+
+def _write_source(conn: sqlite3.Connection, source_id: int, m) -> None:
+    conn.execute(
+        """INSERT INTO sources
+           (id, name, label, url, description, category, category_label, config_json)
+           VALUES (?,?,?,?,?,?,?,?)""",
+        (
+            source_id, m.name, m.label, m.url, m.description,
+            m.category, m.category_label, m.to_config_json(),
+        ),
+    )
+    cols = m.to_columns_json()
+    for c in cols:
+        conn.execute(
+            """INSERT INTO datasource_columns
+               (datasource_id, table_name, column_name, column_type,
+                is_primary_key, is_nullable, description, source_field,
+                unit, semantic_type)
+               VALUES (?,?,?,?,?,?,?,?,?,?)""",
+            (
+                source_id, c["table_name"], c["column_name"], c["column_type"],
+                c["is_primary_key"], c["is_nullable"], c["description"],
+                c["source_field"], c["unit"], c["semantic_type"],
+            ),
+        )
+    conn.execute(
+        "INSERT INTO scraw_configs (name, url, columns_json) VALUES (?,?,?)",
+        (m.name, m.url, json.dumps(m.to_scraw_columns(), ensure_ascii=False)),
+    )
+
+
+def _dump_json(conn: sqlite3.Connection, manifests: list) -> dict:
+    """Deterministic full dump for registry.json (sort_keys, ensure_ascii=False)."""
+    sources = [
+        {
+            "id": i + 1,
+            "name": m.name,
+            "label": m.label,
+            "url": m.url,
+            "description": m.description,
+            "category": m.category,
+            "category_label": m.category_label,
+            "config": m.to_config(),
+        }
+        for i, m in enumerate(manifests)
+    ]
+    cols = []
+    with conn:
+        for row in conn.execute(
+            """SELECT datasource_id, table_name, column_name, column_type,
+                      is_primary_key, is_nullable, description, source_field,
+                      unit, semantic_type
+               FROM datasource_columns ORDER BY datasource_id, rowid"""
+        ):
+            cols.append(dict(row))
+    scraw = []
+    with conn:
+        for row in conn.execute(
+            "SELECT name, url, columns_json FROM scraw_configs ORDER BY name"
+        ):
+            d = dict(row)
+            d["columns_json"] = json.loads(d["columns_json"])
+            scraw.append(d)
+    return {"sources": sources, "datasource_columns": cols, "scraw_configs": scraw}
+
+
+def build() -> str:
+    manifests = _load_manifests()
+    REGISTRY_DIR.mkdir(parents=True, exist_ok=True)
+    conn = sqlite3.connect(DB_PATH)
+    conn.row_factory = sqlite3.Row
+    try:
+        with conn:
+            _create_schema(conn)
+            for i, m in enumerate(manifests, start=1):
+                _write_source(conn, i, m)
+        dump = _dump_json(conn, manifests)
+    finally:
+        conn.close()
+    JSON_PATH.write_text(
+        json.dumps(dump, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
+        encoding="utf-8",
+    )
+    n_cols = sum(len(m.to_columns_json()) for m in manifests)
+    return (
+        f"built {DB_PATH} + {JSON_PATH}: "
+        f"{len(manifests)} sources, {n_cols} columns"
+    )
+
+
+def main() -> int:
+    print(build())
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

fd_cn_gov-0.1.1/fd_cn_gov/cli.py

@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+"""fd-cn-gov CLI: crawl, list, describe, build-registry.
+
+Usage:
+  fd-cn-gov crawl <name> [--max-pages N | --all]   # run a scraper
+  fd-cn-gov list                                     # list registered sources
+  fd-cn-gov describe <name>                          # show a source + its columns
+  fd-cn-gov build-registry                           # regenerate registry.db/json
+"""
+from __future__ import annotations
+
+import argparse
+import importlib
+import sys
+from typing import Optional
+
+from fd_cn_gov.registry import list_sources, get_source, get_columns
+
+SCRIPT_NAMES = [
+    "mee_gsgg_archive", "mem_tzgg_archive", "mnr_tzgg_archive",
+    "moa_govpublic_archive", "mof_gkml_archive", "mofcom_xwfb_archive",
+    "mohurd_xinwen_archive", "mot_shuju_archive", "ndrc_tzgg_archive",
+    "pbc_xinwen_archive", "safe_whxw_archive",
+]
+
+
+def _cmd_crawl(args) -> int:
+    name = args.name
+    if name not in SCRIPT_NAMES:
+        print(
+            f"error: unknown scraper {name!r}\navailable: {' '.join(SCRIPT_NAMES)}",
+            file=sys.stderr,
+        )
+        return 2
+    # Forward the page-cap flags to the script's argparse via synthesized argv.
+    script_argv = [name]
+    if args.all:
+        script_argv.append("--all")
+    elif args.max_pages is not None:
+        script_argv += ["--max-pages", str(args.max_pages)]
+    mod = importlib.import_module(f"fd_cn_gov.scripts.{name}")
+    old = sys.argv
+    sys.argv = script_argv
+    try:
+        mod.main()
+    finally:
+        sys.argv = old
+    return 0
+
+
+def _cmd_list(args) -> int:
+    for s in list_sources():
+        print(f"{s.name}\t{s.label}\t{s.url}")
+    return 0
+
+
+def _cmd_describe(args) -> int:
+    try:
+        s = get_source(args.name)
+    except KeyError as e:
+        print(f"error: {e}", file=sys.stderr)
+        return 2
+    print(f"name:        {s.name}")
+    print(f"label:       {s.label}")
+    print(f"url:         {s.url}")
+    print(f"category:    {s.category}")
+    print(f"description: {s.description}")
+    print("columns:")
+    for c in get_columns(args.name):
+        pk = " [PK]" if c.primary_key else ""
+        nn = "" if c.nullable else " NOT NULL"
+        print(f"  - {c.name}: {c.type}{pk}{nn}  ({c.semantic_type or '-'})")
+        if c.description:
+            print(f"      {c.description}")
+        if c.source_field:
+            print(f"      source: {c.source_field}")
+    return 0
+
+
+def _cmd_build_registry(args) -> int:
+    from fd_cn_gov.build_registry import build
+    print(build())
+    return 0
+
+
+def build_parser() -> argparse.ArgumentParser:
+    ap = argparse.ArgumentParser(
+        prog="fd-cn-gov",
+        description="Chinese ministry open-information scrapers + registry.",
+    )
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    p_crawl = sub.add_parser("crawl", help="run a scraper; prints JSON records to stdout")
+    p_crawl.add_argument("name", choices=None, help="scraper name (e.g. mof_gkml_archive)")
+    pg = p_crawl.add_mutually_exclusive_group()
+    pg.add_argument("--max-pages", type=int, help="pages per archive (default: script default, usually 50)")
+    pg.add_argument("--all", action="store_true", help="full crawl, no page cap")
+    p_crawl.set_defaults(func=_cmd_crawl)
+
+    p_list = sub.add_parser("list", help="list registered sources")
+    p_list.set_defaults(func=_cmd_list)
+
+    p_desc = sub.add_parser("describe", help="show a source + its column schema")
+    p_desc.add_argument("name", help="scraper name")
+    p_desc.set_defaults(func=_cmd_describe)
+
+    p_build = sub.add_parser("build-registry", help="regenerate registry.db + registry.json from MANIFESTs")
+    p_build.set_defaults(func=_cmd_build_registry)
+    return ap
+
+
+def main(argv: Optional[list[str]] = None) -> int:
+    ap = build_parser()
+    args = ap.parse_args(argv)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())