angineer-docs-core 0.1.0__py3-none-any.whl

angineer_docs_core-0.1.0.dist-info/METADATA

@@ -0,0 +1,194 @@
+Metadata-Version: 2.4
+Name: angineer-docs-core
+Version: 0.1.0
+Summary: AnGIneer docs engine: parse pipeline (source_prep -> MinerU -> PoPo -> structure -> FTS -> vectors -> graph), retrieval and export
+Author: AnGIneer Team
+License: MIT
+Project-URL: Repository, https://github.com/0mao0/angineer-docs-core
+Project-URL: Issues, https://github.com/0mao0/angineer-docs-core/issues
+Keywords: rag,document-parsing,retrieval,knowledge-base,mineru
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: angineer-tree-core<0.2.0,>=0.1.0
+Requires-Dist: angineer-ai-inference<0.3.0,>=0.2.2
+Requires-Dist: pydantic<3.0.0,>=2.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: PyMuPDF>=1.23.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: python-dateutil>=2.8
+Requires-Dist: qdrant-client>=1.12
+Provides-Extra: chroma
+Requires-Dist: chromadb>=0.5.5; extra == "chroma"
+Provides-Extra: text2sql
+Requires-Dist: PyYAML>=6.0; extra == "text2sql"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Dynamic: license-file
+
+# angineer-docs-core
+
+[![PyPI](https://img.shields.io/pypi/v/angineer-docs-core)](https://pypi.org/project/angineer-docs-core/)
+
+AnGIneer 的文档解析入库引擎（纯 Python 库）：把工程规范 / PDF / Office 文档跑成
+「**可读、可查、可算、可引用**」的知识库数据——一条管线覆盖
+**源文件准备 → 格式转换 → MinerU 解析 → PoPo 强化 → 结构化 → 图描述 → SQLite+FTS → 向量索引 → 知识图谱**，
+外加五路检索、text2sql 与产物导出。
+
+| 能力 | 说明 | 靠什么 |
+| :--- | :--- | :--- |
+| 📖 可读 | 高保真解析 PDF，保留结构、公式、图表 | MinerU 端点（HTTP） |
+| 🔍 可查 | 语义检索 + 表格精确查询 + 条款定位（五路召回 + 重排） | canonical 七表 + qdrant |
+| 🧮 可算 | 公式语义、表格插值、条款计算（供上层工具消费） | 结构化的 canonical 产物 |
+| 📎 可引用 | 条文 / 表格 / 图片的块级溯源 | `citation_targets` + 块 id |
+
+> 定位：只做**数据面**（解析 / 建索引 / 图谱 / 检索），**不含 HTTP 服务与前端**。
+> 大算力全部在外部服务：版面解析、视觉结构强化、图描述、embedding、rerank 都走 HTTP 端点（配置见下）。
+
+## 管线（9 个阶段 + 导出）
+
+`hard` = 失败即中止；`soft` = 失败降级继续并在 `parse_stages` 留痕。
+
+| # | 阶段 key | 名称 | 类型 | 依赖 | 做什么 |
+| :-- | :--- | :--- | :--- | :--- | :--- |
+| 1 | `source_prep` | 源文件准备 | hard | — | 落地源文件、登记文档节点、准备解析输入 |
+| 2 | `convert` | 格式转换 | hard | source_prep | 非 PDF（doc/docx/xls…）经 LibreOffice 转 PDF |
+| 3 | `raw_parse` | MinerU 解析 | hard | convert | 调 MinerU 端点（file_parse ZIP 协议），产出 markdown / 图片 / `middle.json` |
+| 4 | `popo` | PoPo 强化 | **soft** | raw_parse | PoPo 做视觉结构强化，产出 enriched blocks（需 VLM 端点） |
+| 5 | `structure` | 结构化 | hard | raw_parse | **Solo 是唯一构建者**；PoPo 信号只作为增强注入（对齐 / 合并 / 层级融合 / 续表） |
+| 6 | `figure_describe` | 图描述 | soft | structure | VLM 描述图表内容（可关） |
+| 7 | `fts` | SQLite + FTS | hard | structure | canonical 七表 + 全文索引（检索数据面） |
+| 8 | `vectors` | 向量索引 | soft | fts | qdrant（默认）/ sqlite / chroma |
+| 9 | `graph` | 知识图谱 | soft | structure | 实体与关系抽取入库 |
+
+第 10 步是产物导出（`step10_export`：按册导 markdown / 图片 / 索引 / 图谱）。
+
+## 安装
+
+```bash
+pip install angineer-docs-core
+# 或钉版本
+pip install "angineer-docs-core @ git+https://github.com/0mao0/angineer-docs-core.git@v0.1.0"
+```
+
+Python 要求 `>=3.10`。运行依赖：`angineer-tree-core`（树表底座）、`angineer-ai-inference`（LLM 客户端）、
+`pydantic>=2`、`PyMuPDF`、`numpy`、`python-dateutil`、`qdrant-client`、`requests`、`python-dotenv`。
+可选 extras：`[chroma]`（旧向量后端）、`[text2sql]`（领域关键词 YAML）。
+
+## 另有外部依赖（不随包提供）
+
+| 依赖 | 用途 | 怎么配 |
+| :--- | :--- | :--- |
+| MinerU 端点 | 版面解析（GPU 大头） | `MINERU_CONFIGS` |
+| PoPo 端点 | 视觉结构强化 | `POPO_CONFIGS` |
+| 图描述端点 | 图表描述 | `FIGURE_DESCRIBE_CONFIGS` |
+| embedding 端点 | 向量化 | `EMBEDDING_CONFIGS` |
+| rerank 端点 | 重排 | `RERANKER_CONFIGS` |
+| Qdrant | 向量库（默认 provider） | `QDRANT_URL`（也可换 sqlite / chroma） |
+| LibreOffice | 非 PDF → PDF | 系统安装，`LIBREOFFICE_BIN` 或 PATH |
+
+## 快速开始
+
+```python
+import os
+
+os.environ["KNOWLEDGE_BASE_DIR"] = "/srv/knowledge"          # 数据根（也可用 ANGINEER_DATA_ROOT）
+os.environ["MINERU_CONFIGS"] = '[{"name":"dgx","url":"https://your-gateway/api/mineru","api_key":"..."}]'
+os.environ["POPO_CONFIGS"]   = '[{"name":"dgx","url":"https://your-gateway/api/popo/v1","api_key":"...","model":"Popo"}]'
+os.environ["DOCS_VECTORSTORE_PROVIDER"] = "qdrant"
+os.environ["QDRANT_URL"] = "http://localhost:6333"
+
+from docs_core.parse_pipeline import ParseOrchestrator
+
+orch = ParseOrchestrator()                                             # 不注入时用包内置的解析记录表
+doc_id = orch.ensure_document("default", "/abs/path/spec.pdf")         # 登记文档节点
+task = orch.create_parse_task("default", doc_id, "/abs/path/spec.pdf") # 后台线程跑全链
+print(task["task_id"])
+
+orch.get_parse_task(task["task_id"])     # 查进度 / 当前阶段
+orch.cancel_parse_task(task["task_id"])  # 协作式取消
+orch.retry_parse_task(doc_id)            # 失败重跑
+```
+
+- 阶段级重跑与校验：`STAGE_REGISTRY` / `resolve_stage_order` / `validate_stage_retry` / `compute_resume_stages`；
+- 检索侧入口：`docs_core.step09_query`（五路召回 + 重排 + text2sql）；
+- 知识库与节点管理：`docs_core.get_docs_service()` / `DocsService`。
+
+## 配置（常用）
+
+| 变量 | 默认 | 说明 |
+| :--- | :--- | :--- |
+| `KNOWLEDGE_BASE_DIR` | — | **数据根（最高优先）**：元库 / 索引 / 图谱 / 文档目录都挂它下面 |
+| `ANGINEER_DATA_ROOT` | — | 数据根（次优先，取 `<root>/knowledge`） |
+| `ANGINEER_REPO_ROOT` | 仓库标记探测 | 仓库树根。独立安装（wheel）里没有仓库树，需要数据根请显式给上面两个之一 |
+| `MINERU_CONFIGS` / `POPO_CONFIGS` / `FIGURE_DESCRIBE_CONFIGS` / `EMBEDDING_CONFIGS` / `RERANKER_CONFIGS` | — | JSON 数组，顺序=优先级，第一项为默认；连接失败/超时自动切下一项 |
+| `DOCS_VECTORSTORE_PROVIDER` | —（必填） | `qdrant` / `sqlite` / `chroma`；不设会直接报错（防静默连空库） |
+| `QDRANT_URL` | — | `provider=qdrant` 时必填 |
+| `LIBREOFFICE_BIN` | PATH 探测 | `soffice` 路径（非 PDF 输入必需） |
+| `POPO_MAX_CONCURRENCY` | 4 | PoPo 强化的并发闸（FIFO 排队） |
+| `FIGURE_DESCRIBE_ENABLED` | 见 `.env.example` | 图描述开关 |
+
+## 包里包含什么
+
+- **`docs_core`**：管线与数据面本体（9 阶段、canonical 存储、五路检索、图谱、导出）；
+- **`popo`**：PoPo fork（上游 MinerU-Popo 的定制版）随包发布，两个要点：
+  - 它是**顶层导入名 `popo`**（上游没有 `__init__.py`，按命名空间包收集），装完 site-packages 里会多出这个目录；
+  - 缺失也不报错——第 4 步按「软阶段」跳过，结构由 Solo 单独构建（质量降级、链路不断）。
+    自定义位置可用 `POPO_REPO_PATH` 指定。
+- 依赖 **`angineer-tree-core`** 与 **`angineer-ai-inference`**，pip 会自动带上。
+
+## 目录结构
+
+```text
+services/docs-core/
+├── src/
+│   ├── docs_core/
+│   │   ├── paths.py                 # 布局：数据根 + 文档目录（纯路径计算，无 IO 副作用）
+│   │   ├── docs_service.py          # 门面：知识库/节点/任务落库与查询的统一入口
+│   │   ├── parse_pipeline.py        # 阶段注册 / 顺序 / 状态机 / 编排器
+│   │   ├── models/                  # 全管线共享契约（canonical 七表类型等）
+│   │   ├── step01_source_prep/      # 源文件准备
+│   │   ├── step02_convert2pdf/      # LibreOffice 转 PDF
+│   │   ├── step03_mineru_parse/     # MinerU 解析 + PoPo 强化（popo_enhance）
+│   │   ├── step04_structure/        # Solo 结构构建 + PoPo 信号消费（popo/ 六件套）
+│   │   ├── step05_sqlite_fts/       # canonical SQLite + FTS（rebuild / store）
+│   │   ├── step06_vectors/          # 向量索引（qdrant / sqlite / chroma + embedding 客户端）
+│   │   ├── step07_graph/            # 知识图谱（实体 / 关系 / 证据包）
+│   │   ├── step08_maintain/         # 巡检与维护
+│   │   ├── step09_query/            # 检索（五路召回）、text2sql、对外协议
+│   │   ├── step10_export/           # 产物导出
+│   │   └── library_registry.py / parse_records_store.py / kb_migrator.py …
+│   └── popo/                        # PoPo fork（本地定制，随包发布；上游同步见 UPSTREAM_SYNC.md）
+├── tests/                           # 88 个测试文件 / 500+ 用例
+└── pyproject.toml
+```
+
+## 不在本库范围
+
+- HTTP / API 服务（由主仓库的 docs-api 实现）、前端与管理台；
+- MinerU / PoPo / VLM / embedding / rerank 的推理服务本身；
+- 知识库数据的业务展示与权限。
+
+## 开发与测试
+
+```bash
+pip install -e ".[dev]"
+python -m pytest tests -q
+```
+
+独立环境（无仓库树、无 `.env`）也能跑：`tests/conftest.py` 会先按 dotenv 规则加载 `.env`（若有），
+再补缺省 —— 向量后端用 `sqlite`、数据根指向临时目录；跨服务用例（需要 docs-api）自动跳过。
+当前基线：**507 passed, 17 skipped**（无外部服务、无网络）。
+
+## 许可
+
+MIT

angineer_docs_core-0.1.0.dist-info/RECORD

@@ -0,0 +1,121 @@
+angineer_docs_core-0.1.0.dist-info/licenses/LICENSE,sha256=mzg-3YeLM4FgEvKAevih2_5oBAlzOKtxN1wmXTpWtX0,1067
+docs_core/__init__.py,sha256=QFC1iq7uujDg_XAeZpgFqKor6q2a-dSrQDf2y4kgGW4,1667
+docs_core/docs_file_io.py,sha256=jPRIMTNiArmD2iXIOWJ2M-GH56IGO4BpKYfnM076JZQ,9558
+docs_core/docs_service.py,sha256=Q2q1ro061dL_fBc1zDnksgwhjk9-0R99Nob2SUT7kcU,59104
+docs_core/kb_migration_audit.py,sha256=QOYcc9sxzc099TGnoCYiJCOl5K3-WEx5X8gE-vjv6wM,1988
+docs_core/kb_migration_store.py,sha256=f9vL4ixlml_im-AfvDRm15anKJGTxtT5Ozg5vAHz8TE,7624
+docs_core/kb_migrator.py,sha256=iIKlUazYuW6zWw1Sma0P0sQL1sHjg5QUsD7y0IP7GuA,43844
+docs_core/library_registry.py,sha256=RFXi4B_Eqe0RkZKte0u1iM1LIKUzG2nDd986oyE8Ws0,18762
+docs_core/parse_pipeline.py,sha256=3CC7W6_347A4LjMyXc--6_7AyVjm61afrFQZu41B3eA,62570
+docs_core/parse_records_store.py,sha256=2Biud9KJNdCwwUTkyQtlYHvekLk3TEwOgQsa9hYtpPE,11112
+docs_core/paths.py,sha256=He6HZoi8NDuzhYr_-XkVnKqwJ9bNgxg6ibXOpDHCMNY,10836
+docs_core/startup_guard.py,sha256=1uR9NO_f_0aA5rDoUlCManzAdpfY2jX22nr7kQqvQAg,8368
+docs_core/models/__init__.py,sha256=cGpz2bcrtRLlHXU-njUDZSRZ29weSkS3OcCv8t0-Zh4,435
+docs_core/models/types.py,sha256=M95lYhB7kB03ioqce_woxW-AyBHzzdlsuWRPg6u96NE,6510
+docs_core/step01_source_prep/__init__.py,sha256=TjTe2Kns9cy4o6B2l9A-hJO7atddhr4K7Dq4W9qq0DY,108
+docs_core/step01_source_prep/source_prep.py,sha256=2h2eFDq4JTvqLvCShPS9Zy_ABJwwubvjdY1v1LIOD2w,2555
+docs_core/step02_convert2pdf/__init__.py,sha256=L338IeyKgFFWSNxWnKGjbKcQIezO0lcIUpF-JCdDyA8,158
+docs_core/step02_convert2pdf/convert2pdf.py,sha256=iRHkkz3dsJduw-wtRgW8X0KVIxeBaM5CX1YZXFnopNU,6850
+docs_core/step03_mineru_parse/__init__.py,sha256=1EIRvODvuvs2cj7jz0XtgDaEnT6HgS0qJkihQR_7LGM,334
+docs_core/step03_mineru_parse/mineru_parser.py,sha256=oPQJDBoIo27svvm13c4dzk_sZA7gN7bPLl-D7BZdmdA,44902
+docs_core/step03_mineru_parse/popo_enhance.py,sha256=D4Tz7JRB7Poj76YdEUUjny1v0IJAia3P7AWHGYuKrWU,18926
+docs_core/step04_structure/__init__.py,sha256=SZ12NffvEPQLclJUi5Nm8WmodrbFU0nwypmWNmAeFaY,695
+docs_core/step04_structure/figure_describer.py,sha256=RI7WPEU-YeAwVOW_aiIPrh6Xj_ciRGbEUArTMSqTgnM,12447
+docs_core/step04_structure/solo2json_pipeline.py,sha256=Z2xmp4UGwBocch7sNL_zZNWqgYa-MMi_d7KlGWNaSNg,27492
+docs_core/step04_structure/solo_engine.py,sha256=X1-Twy8JMqwQP5HRfTLSTgR6Dc1Q899Qbtxp1KVHwoQ,123726
+docs_core/step04_structure/popo/__init__.py,sha256=ZOeC9lRJA3tyr3q0bXV8-REQ9UrRFm6ewsXYMt5W5ss,108
+docs_core/step04_structure/popo/popo_block_merger.py,sha256=pP_W3DKynyH4tXHGVEGCExdV43o-BOvFuWRMWqj4Ilw,15493
+docs_core/step04_structure/popo/popo_signal_aligner.py,sha256=nC8cXcllUOFNviMGC9XodA1zcWpGGPJqg21vTITNtJc,7786
+docs_core/step04_structure/popo/popo_signal_injector.py,sha256=agFFkBnqCKOZGDOyE-uPs79MDCnK_6bHeYxM81Y7wiA,10381
+docs_core/step04_structure/popo/popo_signal_level_fusion.py,sha256=QslgEosbWfuyoIEaSFvUwWkiPsIzgKhiQeejOOwwceM,2389
+docs_core/step04_structure/popo/popo_table_continuation.py,sha256=8r97r6UGTfMwQXVAySVnAVjJX9tBBVCqI-w2KOnbCI8,10136
+docs_core/step04_structure/shared/__init__.py,sha256=85BX9dX4XqmF4yyeaRCVRs-MrDrWm8SpcwRLCNYhJ-4,97
+docs_core/step04_structure/shared/formula_semantics.py,sha256=aJjvBRUraH0pf07Miq41PwluqP4bkTOkUSkddrb_VSk,38403
+docs_core/step04_structure/shared/jsonl_io.py,sha256=-H55p90L-hGfZUEcacTgRuMspxffBnVYpjpQi2Q2Lc0,5488
+docs_core/step04_structure/shared/markdown_projection.py,sha256=fFLPcqd_Pr4dXzF2qpSzvFhf0TqBqA9JeVCuemnnDVo,6143
+docs_core/step04_structure/shared/page_role_classifier.py,sha256=Fxs5bVgVzf6F7mNxUJwvV2G-CRic-gS80gBi6ZZf_7k,8542
+docs_core/step04_structure/shared/row_vocabulary.py,sha256=-cGvmmzf-WQFQp7uhwjGO7oWA_qNHmX2UVEUm8F4Aho,1899
+docs_core/step04_structure/shared/table_caption.py,sha256=JnEs8GC-aZZKmDh_IlYYxo6WWPVN73bNNHxdg53AnMY,2944
+docs_core/step04_structure/shared/table_cells.py,sha256=hxiCtM_s-Eb8qJkjBu9ECuX0V2YYdL167ivEGzgCuoI,9741
+docs_core/step04_structure/shared/table_html_utils.py,sha256=I9JevWBXRusAkdOj53-me3zy6xkPOybGCkEN9BMf7Hs,3228
+docs_core/step04_structure/shared/table_semantics.py,sha256=hHTPoCJrkgpK7g9pX7WYmypXb7rKQZc5FuXW-Rad6Xw,13793
+docs_core/step04_structure/shared/title_level_refiner.py,sha256=gApdXsL8Uq6uQoA3oKVmdPjZmNkzMlPWU5capfcOVAA,7573
+docs_core/step04_structure/shared/title_level_resolver.py,sha256=Yrqd-hhAxjJXyU7AuWSW0zGE4GTXzmZ_Ex46pHlDVMU,8413
+docs_core/step05_sqlite_fts/__init__.py,sha256=wbZs35m55JZ7UjmDSK2Yd1fn9oONbV_bHahyuLw-EEc,201
+docs_core/step05_sqlite_fts/graph_editor.py,sha256=tuMghI-oZQ216FriiS9z6-VHKn82XLmy3mqDqsWCMSA,72750
+docs_core/step05_sqlite_fts/rows_projection.py,sha256=p_bU7mPK1WNs6GM6kZN6PCpaozYbsBfFzaPyIWmw2eU,6226
+docs_core/step05_sqlite_fts/sqlite_index.py,sha256=WKRju2qIMlz_bYoHwpzfBGjQm7xLilkw8ZEYImrcRu8,4416
+docs_core/step05_sqlite_fts/rebuild/__init__.py,sha256=kYccIP0XJL_XozMVSJ2-uni34qowPptCOb5bmTNtIP0,1228
+docs_core/step05_sqlite_fts/rebuild/canonical_builder.py,sha256=ozsDN032SMoA56K5YYb6FZwoGYRhE56D3joWYnSooI4,38151
+docs_core/step05_sqlite_fts/rebuild/graph_rebuilder.py,sha256=6PhpKqiLp9vAOeqMBilfJDs968GkN-793WrSPbY5hhs,11786
+docs_core/step05_sqlite_fts/rebuild/table_semantics.py,sha256=5saIh8po5Ybo4fgJ9U0Vvp7L8P8gBuL1k2tJCUVoDRo,1461
+docs_core/step05_sqlite_fts/rebuild/tag_rules.py,sha256=077hXE1pmEcGb7yQLKi45yE3EM9hwOWSq7U4WV25qTU,903
+docs_core/step05_sqlite_fts/store/__init__.py,sha256=WF3nfsEVmTcrKP4CdkcB2TF8V8PRhKP0jDJgHMrchto,377
+docs_core/step05_sqlite_fts/store/blocks_sql_store.py,sha256=5Mo0lvR25VvwXQGLKeL4bXR9Jy70FEhpMHV85YqT1M8,55100
+docs_core/step05_sqlite_fts/store/canonical_sql_store.py,sha256=PSp0gOfiZdzyELZa1lN0fAMJPytuyBDTOpZnNmZq4jk,62492
+docs_core/step05_sqlite_fts/store/sqlite_utils.py,sha256=gKX3Fp9FBOI4kr1lecODPcUFI9f0ioVArGGSqgZ3buI,2527
+docs_core/step06_vectors/__init__.py,sha256=31Er-xHWv5P4xYLzujYXGGIMpZlKZJQ7pcXzE7cmw4E,1412
+docs_core/step06_vectors/chroma_vector_store.py,sha256=7lgqwlpgN5ktZjo1hKVn5Ei_ib_qU2yVkjAMH9kVbhM,8382
+docs_core/step06_vectors/config.py,sha256=HDPvyD-YK0DEeQlWv_ZS_Bdi9km1CCNa60lEG3PWDWA,4517
+docs_core/step06_vectors/embedding_provider.py,sha256=rNh125htVdFQxWMVgcSoSTEKF25n0ACRgbhOhDX2rlU,12587
+docs_core/step06_vectors/qdrant_vector_store.py,sha256=Dmhm1EC415QCMyo5NU7F4csD7e7CKpsQ_EFg4jCM5rE,19101
+docs_core/step06_vectors/sqlite_vector_store.py,sha256=m1DTEmBlblKtY2Gv1vIoCm9zxZ4eDEifCWvTzYVB2YU,19063
+docs_core/step06_vectors/vector_indexer.py,sha256=ncGJmYJprj0W_fXny3oZ2jJd7KfgyCgGgcwCOPoTuVE,9173
+docs_core/step06_vectors/vector_store.py,sha256=hUhh5zXxKDVerxiflJDQzeuKKqlzSXcxeIyq38ihZ1U,2676
+docs_core/step07_graph/__init__.py,sha256=7mLtnb-NKpIuNXtQoCU_VK-oJvXPx8N3MNpzpj-IM8Q,865
+docs_core/step07_graph/auto_extract.py,sha256=_14pml1ohD_Xrq4jb2b9STZAL2IShPvNPnEkS-F-fGE,1253
+docs_core/step07_graph/config.py,sha256=sMrd2K4ccUZNP60O9ezYQdNvVDlXEAScPdk0BxmnFyU,5478
+docs_core/step07_graph/entity_dedup.py,sha256=s-70kv0l8dPREjcgPtvYpDPUcaBihFVM1AiMoQdT77k,2165
+docs_core/step07_graph/entity_extractor.py,sha256=TR4SvlwUXOS0HR03Uu2NFqywklL0UltuZELA1bffChY,5541
+docs_core/step07_graph/evidence_builder.py,sha256=GSl4PMdttq7KFOebvYnSXFxCEwl-6_Pl5ZwifrCXess,12230
+docs_core/step07_graph/extractor_prompts.py,sha256=t1cY2oTjfkGRJCY0fOlqMogWraX2kLOEmR5sOkXqtVk,6605
+docs_core/step07_graph/graph_orchestrator.py,sha256=ZNjLtPDK8N1lP09MBDnB8N6Yfb7MsZO90ubogQ--ZVU,35539
+docs_core/step07_graph/graph_store.py,sha256=P9B7HstA9ZfJv3gh2tboa6kEwxX4BaIoWRyR9ZJQSO4,54524
+docs_core/step07_graph/push_to_graph.py,sha256=RuG9wW5XuVRg46rpLXm01xIdZYXJ73tG1RrHt1ahEag,4028
+docs_core/step07_graph/question_mapper.py,sha256=t8CSRLYAau3bXxi1BX2HRZvL28WwZ2wZ2-4XOjXDxtA,5723
+docs_core/step07_graph/relation_infer.py,sha256=WHd_djfHOwgL5lKve7lgKjUMRTRMiAnZ5v38zUDqbi0,4367
+docs_core/step08_maintain/__init__.py,sha256=K8_7UIwt3yVsrQMnPTXwy0nOlSvd7-qitoQs2CyJpm4,518
+docs_core/step08_maintain/config.py,sha256=KmPLUDUTHvHxJWcODO2LmxO8HTcT1CarszUgSqSygGA,5082
+docs_core/step08_maintain/report.py,sha256=2VlZdB6wVU3BhWVhvEHRgNUzYj-S5WqacV1Fpq1Gvcs,3636
+docs_core/step08_maintain/runner.py,sha256=F5HfIVJwbXiTZRK-fSR5yaROyvHdcmVA9Y1l_QKMPTc,23828
+docs_core/step09_query/__init__.py,sha256=BQonTsKlimAihrsqaje5r8HjVeZoEScvmB89v-3cxvs,83
+docs_core/step09_query/agent_port.py,sha256=WaxuitQXcb8uEfqz8rN3dLrE1fGXgAJXJXAoGzGSwR8,20252
+docs_core/step09_query/retrieve_service.py,sha256=tryBPybiDcBKRM39WyE9kU92wb4B-7MDVWvSFsngcOg,14624
+docs_core/step09_query/protocols/__init__.py,sha256=YXyLOu0TKj0SE23CQIby-ADFGnox22WWuDcCYSraK8k,682
+docs_core/step09_query/protocols/contracts.py,sha256=Zq-o2d1nBKqlLQ6jpdkZrNrd4es0ij8jjdIuIh-int8,5563
+docs_core/step09_query/protocols/data_port.py,sha256=A29_-2oMahGZ87zf-z-hV3iYBuL6qQ2jBpV3VbcR07E,3399
+docs_core/step09_query/retrieval/__init__.py,sha256=sKxZ-ZYKs4rqaZdK4y7ldY-i2mKregV3YCyVZRQEcXc,998
+docs_core/step09_query/retrieval/clause_resolver.py,sha256=VX3Uy_xAmDRelbmwjONd1tFAJiPKRSvMhGbTjAJoKuQ,9385
+docs_core/step09_query/retrieval/dense_retriever.py,sha256=Uywib7ugVi1vmzQ02yajSBc1LbgspDbgpjE7cbE20Qg,7595
+docs_core/step09_query/retrieval/formula_retriever.py,sha256=COOLuQ2lIUfcKnlW1W5nCVyqwFYhbJ9T_7u0npmKIio,18950
+docs_core/step09_query/retrieval/hybrid_retriever.py,sha256=6xXV-nsX9EgoMkSah3qZS-XykQLnnpmRSl3w7Etg2MI,14804
+docs_core/step09_query/retrieval/query_normalizer.py,sha256=PKGha7Gj3FpoeuhNySnEcKhkFot7g8z-N0okaBtfl1c,15026
+docs_core/step09_query/retrieval/reranker.py,sha256=7a2G52bceH9Dt_STJB8AeTCW_n9VXbsQCrL0XKZ0XqA,3149
+docs_core/step09_query/retrieval/sparse_retriever.py,sha256=m7luCQPwCf2eCf0WMjmYwf696AyuoWupQ-nm9QdcQ3I,14811
+docs_core/step09_query/retrieval/table_retriever.py,sha256=6hMwtK5tR0bVOBtPhEP62TrZY6eAorKayWZwbyIuNKM,21149
+docs_core/step09_query/text2sql/__init__.py,sha256=c7uiIhGLL08ZbgbCQO3JQ5WN5uXrS8ZfG8QA8o9SKhc,431
+docs_core/step09_query/text2sql/schema_linker.py,sha256=tArWYmb_fPxyw0iyv22EU3ArJJdJ8rsaDPQp5Rczsjw,9957
+docs_core/step09_query/text2sql/sql_executor.py,sha256=GgLm3NaQGlybYecWi4U7JhwjLcjkBJt4DBefTEkZp2Y,791
+docs_core/step09_query/text2sql/sql_explainer.py,sha256=i0JlLe4844Fows8hCNccZIezMlprJcaxl85Zkchkyug,587
+docs_core/step09_query/text2sql/sql_generator.py,sha256=VII9nomIh9PrWztxvwLo4MOuiBf3fPjXwjd2oMgBZcA,330
+docs_core/step09_query/text2sql/sql_planner.py,sha256=EZf4cR7YBBDV5B-pW81e3IrDp33ZZsUhDHT5bGRK-ds,2322
+docs_core/step09_query/text2sql/sql_validator.py,sha256=oBTUeZ0u4vRPft4pldhQjZghE6JnS72q8Yv5v5ZfCuc,2260
+docs_core/step10_export/__init__.py,sha256=gPqAC_fQ-Cl1jCbUzFj60LiwuJ3ZPlYXDC-qk1xUq7Y,66
+docs_core/step10_export/export_artifacts.py,sha256=_SMK5nqUnPNRp9BbTteY7VF_67rM84pC_BCaSJu26q4,10702
+popo/data_engine/add_link.py,sha256=H5Y_1qOuoTMsS1lOHGee9Lg5qwSW7d2WWEPWL6HxA2U,34031
+popo/data_engine/api_utils.py,sha256=Q7FnC5hNVRgAs7Dw1mUZfo_BYWFfQq0lANBVAH_zGFE,3397
+popo/data_engine/table_merge_filter.py,sha256=M_KZIe6cLUUmN60eyG8T7R1M6Pi6xGOaFm51ZLCH6gE,12511
+popo/data_engine/table_utils.py,sha256=mItLe5ZUFg4u-DAUwX1gDpFnOIPuPkytsw2gr8kGMYU,10775
+popo/eval/evaluate.py,sha256=LmrtEfsKP4WUNOtPatvDMwNOfgh2DJUBc_d5hy4WXO0,15017
+popo/post_processing/generate_metadata.py,sha256=LUPEISUdLcFvk_5PDfPoZSUUD9W4iJJ1MUerDGiIv1M,6599
+popo/post_processing/get_json_tree.py,sha256=GNGV8KehE64AM8dx4VkPTTJfqA8Mz8bwagNohxbP5_8,9550
+popo/post_processing/inference.py,sha256=vwNrEQeF5Bb7c3ihZ6vo--fpd86Ccn3hkHBiILUz-Z4,30044
+popo/post_processing/label_normalization.py,sha256=aKbq876N1WL7K3sTCC4_2EGJ9YLaaUWDOY8h3L7TY6s,40509
+popo/post_processing/model_utils.py,sha256=fwT8lqRDHcO8ZX6Byu8585So61yKX6etMGUT27mKxOI,8888
+popo/post_processing/run_inference.py,sha256=n4O3e2MDM86dUoqqIaiZdq2_x-2jGePTYeVwW0xqaIU,9834
+popo/post_processing/split_subnode.py,sha256=YcensL_IP5QCndOKXhLIMX8WYfD8K04uWdVYI5sdSdE,3940
+popo/post_processing/table_merge_utils.py,sha256=gtsQwZPYL5YxwwF-jh_B8pzrsu3uy9mVnvoezsL1uPI,11746
+angineer_docs_core-0.1.0.dist-info/METADATA,sha256=LXF2kIQ6FCDw6zrEUiVQQwxQdmLVteq2Z74jir7sHVQ,10524
+angineer_docs_core-0.1.0.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
+angineer_docs_core-0.1.0.dist-info/top_level.txt,sha256=z9GZQhoCi8MCWvthi5W19TgCje6rvu4wZL2kBN51MyE,15
+angineer_docs_core-0.1.0.dist-info/RECORD,,

angineer_docs_core-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

angineer_docs_core-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Rowe Smith
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

angineer_docs_core-0.1.0.dist-info/top_level.txt

@@ -0,0 +1,2 @@
+docs_core
+popo

docs_core/__init__.py

@@ -0,0 +1,71 @@
+"""docs_core — 统一知识管理包：文档处理、知识图谱、检索、文本转SQL、定时维护"""
+
+__version__ = "0.1.0"  # 与 pyproject.toml 保持一致（独立发版的 CI 会校验两者相等）
+
+from .docs_service import DocsService, get_docs_service, docs_service
+
+# 全流水线共享契约（04/05/06/09 + docs_service 根）
+from .models.types import (
+    CanonicalBlock,
+    CanonicalDocument,
+    CanonicalTable,
+    CanonicalChunk,
+    CanonicalOutlineNode,
+    CitationTarget,
+    BoundingBox,
+)
+# 步骤七：知识图谱
+from .step07_graph import (
+    EntityLayer,
+    RelationType,
+    EntitySeed,
+    Confidence,
+    GraphEntity,
+    GraphRelation,
+    GraphStore,
+    GraphOrchestrator,
+    EvidencePacket,
+    build_evidence_packets,
+    QuestionMapper,
+    StructuredQuestion,
+    RelationInferrer,
+    EntityExtractor,
+)
+# 步骤八：维护
+from .step08_maintain import (
+    DreamCycleRunner,
+    DreamCycleReport,
+    DreamCycleConfig,
+    get_config as get_dream_cycle_config,
+)
+
+__all__ = [
+    "DocsService",
+    "get_docs_service",
+    "docs_service",
+    "CanonicalBlock",
+    "CanonicalDocument",
+    "CanonicalTable",
+    "CanonicalChunk",
+    "CanonicalOutlineNode",
+    "CitationTarget",
+    "BoundingBox",
+    "EntityLayer",
+    "RelationType",
+    "EntitySeed",
+    "Confidence",
+    "GraphEntity",
+    "GraphRelation",
+    "GraphStore",
+    "GraphOrchestrator",
+    "EvidencePacket",
+    "build_evidence_packets",
+    "QuestionMapper",
+    "StructuredQuestion",
+    "RelationInferrer",
+    "EntityExtractor",
+    "DreamCycleRunner",
+    "DreamCycleReport",
+    "DreamCycleConfig",
+    "get_dream_cycle_config",
+]

docs_core/docs_file_io.py

@@ -0,0 +1,210 @@
+"""全局文件 IO：结构化结果文件与 JSON 存储（不属于任何步骤，全局共用）。"""
+import json
+import shutil
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+import docs_core.paths as paths
+
+
+class FileStorage:
+    """文件存储管理器（目录布局见 docs_core.paths，本类只负责读写）。"""
+
+    def __init__(self, base_dir: str = None):
+        # base_dir=None 时不固化：paths.library_root 经注册表按组路由（阶段二 data/ 归位）；
+        # 显式 base_dir（测试用）则钉死在该根下。
+        self.base_dir = Path(base_dir) if base_dir is not None else None
+        self.libraries_dir = (self.base_dir / "libraries") if self.base_dir is not None else None
+
+    def save_source_file(
+        self,
+        library_id: str,
+        doc_id: str,
+        content: bytes,
+        original_filename: Optional[str] = None,
+    ) -> str:
+        """保存源文件"""
+        safe_name = Path(original_filename or f"{doc_id}.pdf").name
+        source_dir = paths.get_source_dir(library_id, doc_id, self.base_dir)
+        source_dir.mkdir(parents=True, exist_ok=True)
+        source_path = source_dir / safe_name
+        with open(source_path, "wb") as f:
+            f.write(content)
+        return str(source_path)
+
+    def save_edited_markdown(self, library_id: str, doc_id: str, content: str) -> str:
+        """保存编辑后的 Markdown 文件"""
+        edited_dir = paths.get_edited_dir(library_id, doc_id, self.base_dir)
+        edited_dir.mkdir(parents=True, exist_ok=True)
+        current_path = edited_dir / "current.md"
+        with open(current_path, "w", encoding="utf-8") as f:
+            f.write(content)
+        revision_dir = edited_dir / "history"
+        revision_dir.mkdir(parents=True, exist_ok=True)
+        revision_path = revision_dir / f'{datetime.now().strftime("%Y%m%d%H%M%S")}.md'
+        with open(revision_path, "w", encoding="utf-8") as f:
+            f.write(content)
+        return str(current_path)
+
+    def read_popo_enriched_blocks(self, library_id: str, doc_id: str) -> list:
+        import json as _json
+        path = paths.get_popo_enriched_blocks_path(library_id, doc_id, self.base_dir)
+        if not path.exists():
+            raise FileNotFoundError(f"PoPo enriched blocks not found: {path}")
+        with open(path, "r", encoding="utf-8") as f:
+            return _json.load(f)
+
+    def read_popo_document_tree(self, library_id: str, doc_id: str) -> dict:
+        import json as _json
+        path = paths.get_popo_document_tree_path(library_id, doc_id, self.base_dir)
+        if not path.exists():
+            raise FileNotFoundError(f"PoPo document tree not found: {path}")
+        with open(path, "r", encoding="utf-8") as f:
+            return _json.load(f)
+
+    def read_mineru_blocks(self, library_id: str, doc_id: str) -> List[Dict[str, Any]]:
+        """读取 MinerU 块级结果"""
+        blocks_path = paths.get_mineru_blocks_path(library_id, doc_id, self.base_dir)
+        if not blocks_path.exists():
+            return []
+        try:
+            with open(blocks_path, "r", encoding="utf-8") as f:
+                data = json.load(f)
+            if isinstance(data, list):
+                return [item for item in data if isinstance(item, dict)]
+        except Exception:
+            return []
+        return []
+
+    def read_markdown(self, library_id: str, doc_id: str) -> Optional[str]:
+        """读取 Markdown 文件"""
+        edited_path = paths.get_edited_markdown_path(library_id, doc_id, self.base_dir)
+        parsed_path = paths.get_parsed_markdown_path(library_id, doc_id, self.base_dir)
+        target_path = edited_path if edited_path.exists() else parsed_path
+        if target_path.exists():
+            with open(target_path, "r", encoding="utf-8") as f:
+                return f.read()
+        return None
+
+    def get_latest_source_file(self, library_id: str, doc_id: str) -> Optional[str]:
+        """获取源文件路径"""
+        source_dir = paths.get_source_dir(library_id, doc_id, self.base_dir)
+        if source_dir.exists():
+            files = sorted(
+                [path for path in source_dir.iterdir() if path.is_file()],
+                key=lambda path: path.stat().st_mtime,
+                reverse=True,
+            )
+            if files:
+                return str(files[0])
+        return None
+
+    def _pick_original_source_file(self, source_dir: Path) -> Optional[str]:
+        """优先返回上传的原始文件（非 PDF），避免转换产物把 source_file 顶成 PDF。"""
+        if not source_dir.exists():
+            return None
+        files = sorted(
+            [path for path in source_dir.iterdir() if path.is_file()],
+            key=lambda path: path.stat().st_mtime,
+            reverse=True,
+        )
+        if not files:
+            return None
+        non_pdf = [path for path in files if path.suffix.lower() != ".pdf"]
+        return str(non_pdf[0] if non_pdf else files[0])
+
+    def resolve_pdf_input(self, library_id: str, doc_id: str) -> str:
+        """只读：返回 source 目录下最新 PDF（convert 产物或上传即 PDF）。
+
+        供 raw_parse 输入核查使用；找不到时抛出带指引的 RuntimeError。
+        """
+        source_dir = paths.get_source_dir(library_id, doc_id, self.base_dir)
+        if not source_dir.exists():
+            raise RuntimeError(f"源文件目录不存在: {source_dir}")
+        pdf_files = sorted(
+            [p for p in source_dir.iterdir() if p.suffix.lower() == '.pdf' and p.is_file()],
+            key=lambda p: p.stat().st_mtime, reverse=True,
+        )
+        if not pdf_files:
+            raise RuntimeError(f"未找到 PDF 输入文件（请先运行格式转换）: {source_dir}")
+        return str(pdf_files[0])
+
+    def delete_document(self, library_id: str, doc_id: str) -> bool:
+        """删除文档"""
+        doc_root = paths.get_doc_root(library_id, doc_id, self.base_dir)
+        deleted = False
+        if doc_root.exists():
+            shutil.rmtree(doc_root)
+            deleted = True
+        return deleted
+
+    def list_documents(self, library_id: str) -> List[dict]:
+        """列出知识库中的文档"""
+        documents = []
+        documents_dir = paths.library_root(library_id, self.base_dir) / "documents"
+        if documents_dir.exists():
+            for doc_root in documents_dir.iterdir():
+                if not doc_root.is_dir():
+                    continue
+                source_dir = doc_root / "source"
+                source_files = sorted([file for file in source_dir.glob("*") if file.is_file()])
+                source_file = source_files[0] if source_files else None
+                md_path = doc_root / "parsed" / "content.md"
+                if source_file:
+                    documents.append(
+                        {
+                            "id": doc_root.name,
+                            "filename": source_file.name,
+                            "source_path": str(source_file),
+                            "has_markdown": md_path.exists(),
+                            "created_at": datetime.fromtimestamp(source_file.stat().st_ctime).isoformat(),
+                        }
+                    )
+
+        return documents
+
+    def get_doc_manifest(self, library_id: str, doc_id: str) -> Dict[str, Any]:
+        """获取文档清单"""
+        parsed_dir = paths.get_parsed_dir(library_id, doc_id, self.base_dir)
+        doc_root = parsed_dir.parent
+        source_dir = paths.get_source_dir(library_id, doc_id, self.base_dir)
+        source_file = self._pick_original_source_file(source_dir)
+        parsed_path = paths.get_parsed_markdown_path(library_id, doc_id, self.base_dir)
+        edited_path = paths.get_edited_markdown_path(library_id, doc_id, self.base_dir)
+        assets_path = parsed_dir / "assets"
+        raw_dir = parsed_dir / "raw"
+        mineru_blocks_path = paths.get_mineru_blocks_path(library_id, doc_id, self.base_dir)
+        history_dir = paths.get_edited_dir(library_id, doc_id, self.base_dir) / "history"
+        # 渲染底图 PDF：转换产物或上传即 PDF（优先与原始文件名同名的 PDF）
+        render_pdf_path = None
+        if source_file:
+            same_stem_pdf = Path(source_file).with_suffix(".pdf")
+            if same_stem_pdf.exists():
+                render_pdf_path = same_stem_pdf
+            else:
+                pdf_files = sorted(
+                    [p for p in source_dir.iterdir() if p.suffix.lower() == ".pdf" and p.is_file()],
+                    key=lambda p: p.stat().st_mtime,
+                    reverse=True,
+                )
+                render_pdf_path = pdf_files[0] if pdf_files else None
+        return {
+            "doc_root": str(doc_root),
+            "source_file": source_file,
+            "parsed_markdown": str(parsed_path) if parsed_path.exists() else None,
+            "edited_markdown": str(edited_path) if edited_path.exists() else None,
+            "assets_dir": str(assets_path) if assets_path.exists() else None,
+            "raw_dir": str(raw_dir) if raw_dir.exists() else None,
+            "mineru_blocks": str(mineru_blocks_path) if mineru_blocks_path.exists() else None,
+            "render_pdf": str(render_pdf_path) if render_pdf_path else None,
+            "history_files": sorted([str(path) for path in history_dir.glob("*.md")], reverse=True) if history_dir.exists() else [],
+        }
+
+file_storage = FileStorage()
+
+
+__all__ = [
+    "FileStorage",
+    "file_storage",
+]