knowledge-graph-kit 0.1.0__tar.gz

knowledge_graph_kit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

knowledge_graph_kit-0.1.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: knowledge-graph-kit
+Version: 0.1.0
+Summary: 知识图谱构建管线：文本分块、LLM抽取、实体解析、Neo4j写入
+Author-email: hbue_jerry <lovecpp@foxmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/jerryhbue/property-graph
+Project-URL: Source, https://github.com/jerryhbue/property-graph
+Project-URL: Tracker, https://github.com/jerryhbue/property-graph/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: neo4j>=5.0.0
+Requires-Dist: httpx>=0.28.0
+Dynamic: license-file
+
+# knowledge-graph-kit
+
+知识图谱构建管线：从教材文本中抽取实体和关系，写入 Neo4j 图数据库。
+
+## 安装
+
+```bash
+pip install knowledge-graph-kit
+```
+
+## 环境变量
+
+| 变量 | 说明 | 默认值 |
+|---|---|---|
+| `OPENAI_API_KEY` | OpenAI API 密钥 | — |
+| `OPENAI_BASE_URL` | OpenAI API 地址（可换兼容 API） | — |
+| `LLM_MODEL_NAME` | 模型名称 | `gpt-4o-mini` |
+| `NEO4J_URI` | Neo4j 连接地址 | `bolt://localhost:7687` |
+| `NEO4J_USERNAME` | Neo4j 用户名 | `neo4j` |
+| `NEO4J_PASSWORD` | Neo4j 密码 | `12345678` |
+
+在项目根目录创建 `.env` 文件即可自动加载，或直接在 shell 中设置。
+
+## CLI 命令
+
+安装后提供 4 个命令行工具：
+
+### 1. 文本分块
+
+```bash
+kg-chunker <txt_path>
+```
+
+按章节标题将教材文本拆分为语义块。
+
+### 2. 实体关系抽取
+
+```bash
+kg-extractor <txt_path>
+```
+
+基于本体 schema 引导 LLM 抽取实体和关系，输出 `extraction_result.json`。
+
+可通过 `KG_SCHEMA_PATH` 环境变量指定自定义 ontology 文件。
+
+### 3. 实体解析去重
+
+```bash
+kg-resolver <input.json>
+```
+
+精确/模糊去重实体，清理属性，更新关系引用，输出 `extraction_result_clean.json`。
+
+### 4. 写入 Neo4j
+
+```bash
+kg-neo4j-writer [input.json]
+```
+
+将清洗后的结果写入 Neo4j 图数据库。默认尝试读取 `extraction_result_clean.json` 或 `extraction_result.json`。
+
+## 程序化使用
+
+```python
+from knowledge_graph_kit import chunk_file, Neo4jWriter
+
+# 文本分块
+chunks = chunk_file("教材.txt")
+
+# 配置 OpenAI
+from knowledge_graph_kit.extractor import configure
+configure(api_key="sk-xxx", model="gpt-4o")
+
+# 写入 Neo4j
+writer = Neo4jWriter(
+    uri="bolt://localhost:7687",
+    user="neo4j",
+    password="your-password",
+)
+writer.write_entities(entities)
+writer.write_relations(relations)
+writer.close()
+```
+
+## 管线流程
+
+```
+txt 文件 → 分块(chunker) → LLM抽取(extractor) → 实体解析(resolver) → Neo4j写入(writer)
+```
+
+## License
+
+MIT

knowledge_graph_kit-0.1.0/README.md

@@ -0,0 +1,93 @@
+# knowledge-graph-kit
+
+知识图谱构建管线：从教材文本中抽取实体和关系，写入 Neo4j 图数据库。
+
+## 安装
+
+```bash
+pip install knowledge-graph-kit
+```
+
+## 环境变量
+
+| 变量 | 说明 | 默认值 |
+|---|---|---|
+| `OPENAI_API_KEY` | OpenAI API 密钥 | — |
+| `OPENAI_BASE_URL` | OpenAI API 地址（可换兼容 API） | — |
+| `LLM_MODEL_NAME` | 模型名称 | `gpt-4o-mini` |
+| `NEO4J_URI` | Neo4j 连接地址 | `bolt://localhost:7687` |
+| `NEO4J_USERNAME` | Neo4j 用户名 | `neo4j` |
+| `NEO4J_PASSWORD` | Neo4j 密码 | `12345678` |
+
+在项目根目录创建 `.env` 文件即可自动加载，或直接在 shell 中设置。
+
+## CLI 命令
+
+安装后提供 4 个命令行工具：
+
+### 1. 文本分块
+
+```bash
+kg-chunker <txt_path>
+```
+
+按章节标题将教材文本拆分为语义块。
+
+### 2. 实体关系抽取
+
+```bash
+kg-extractor <txt_path>
+```
+
+基于本体 schema 引导 LLM 抽取实体和关系，输出 `extraction_result.json`。
+
+可通过 `KG_SCHEMA_PATH` 环境变量指定自定义 ontology 文件。
+
+### 3. 实体解析去重
+
+```bash
+kg-resolver <input.json>
+```
+
+精确/模糊去重实体，清理属性，更新关系引用，输出 `extraction_result_clean.json`。
+
+### 4. 写入 Neo4j
+
+```bash
+kg-neo4j-writer [input.json]
+```
+
+将清洗后的结果写入 Neo4j 图数据库。默认尝试读取 `extraction_result_clean.json` 或 `extraction_result.json`。
+
+## 程序化使用
+
+```python
+from knowledge_graph_kit import chunk_file, Neo4jWriter
+
+# 文本分块
+chunks = chunk_file("教材.txt")
+
+# 配置 OpenAI
+from knowledge_graph_kit.extractor import configure
+configure(api_key="sk-xxx", model="gpt-4o")
+
+# 写入 Neo4j
+writer = Neo4jWriter(
+    uri="bolt://localhost:7687",
+    user="neo4j",
+    password="your-password",
+)
+writer.write_entities(entities)
+writer.write_relations(relations)
+writer.close()
+```
+
+## 管线流程
+
+```
+txt 文件 → 分块(chunker) → LLM抽取(extractor) → 实体解析(resolver) → Neo4j写入(writer)
+```
+
+## License
+
+MIT

knowledge_graph_kit-0.1.0/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=69.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "knowledge-graph-kit"
+version = "0.1.0"
+description = "知识图谱构建管线：文本分块、LLM抽取、实体解析、Neo4j写入"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "hbue_jerry", email = "lovecpp@foxmail.com" },
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "python-dotenv>=1.0.0",
+    "openai>=1.0.0",
+    "pydantic>=2.0.0",
+    "neo4j>=5.0.0",
+    "httpx>=0.28.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/jerryhbue/property-graph"
+Source = "https://github.com/jerryhbue/property-graph"
+Tracker = "https://github.com/jerryhbue/property-graph/issues"
+
+[project.scripts]
+kg-chunker = "knowledge_graph_kit.chunker:main"
+kg-extractor = "knowledge_graph_kit.extractor:main"
+kg-resolver = "knowledge_graph_kit.entity_resolver:main"
+kg-neo4j-writer = "knowledge_graph_kit.neo4j_writer:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["knowledge_graph_kit*"]
+
+[tool.setuptools.package-data]
+knowledge_graph_kit = ["schema.txt"]

knowledge_graph_kit-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

knowledge_graph_kit-0.1.0/src/knowledge_graph_kit/__init__.py

@@ -0,0 +1,42 @@
+"""knowledge-graph-kit: 知识图谱构建管线。
+
+提供从文本分块、LLM 抽取、实体解析到 Neo4j 写入的完整管线。
+"""
+
+__version__ = "0.1.0"
+
+from .chunker import Chunk, SectionChunker, chunk_file
+from .entity_resolver import (
+    clean_properties,
+    dedup_exact,
+    fuzzy_align,
+    resolve,
+)
+from .extractor import (
+    Entity,
+    ExtractionResult,
+    Relation,
+    build_system_prompt,
+    extract_from_chunk,
+    merge_results,
+    parse_schema,
+)
+from .neo4j_writer import Neo4jWriter
+
+__all__ = [
+    "Chunk",
+    "SectionChunker",
+    "chunk_file",
+    "Entity",
+    "Relation",
+    "ExtractionResult",
+    "parse_schema",
+    "build_system_prompt",
+    "extract_from_chunk",
+    "merge_results",
+    "dedup_exact",
+    "clean_properties",
+    "fuzzy_align",
+    "resolve",
+    "Neo4jWriter",
+]

knowledge_graph_kit-0.1.0/src/knowledge_graph_kit/chunker.py

@@ -0,0 +1,252 @@
+"""
+chunker.py — Section-Aware Chunking
+
+从教材 txt 文件读取内容，按多级标题（2.1 / 2.1.1 / ...）拆分为语义块，
+每个 Chunk 保留层级元数据，供后续 LLM 抽取实体和关系。
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Optional
+
+# Windows 终端 UTF-8 支持
+if sys.platform == "win32":
+    try:
+        sys.stdout.reconfigure(encoding="utf-8")  # type: ignore[attr-defined]
+    except Exception:
+        pass
+
+
+# ── 正则 ────────────────────────────────────────────────────
+# 匹配 "2.1 基本语法"、"2.1.1标识符"（标题前可有空格也可无空格）
+SECTION_RE = re.compile(r"^(\d+)\.(\d+)(?:\.(\d+))?\s*(.+)$")
+# 匹配 "第2章 ES基础"
+CHAPTER_RE = re.compile(r"^第(\d+)章\s+(.+)$")
+# 匹配 "【示例2-1】" 或 "【示例2-10】"
+EXAMPLE_RE = re.compile(r"【示例(\d+)-(\d+)】")
+# 匹配 "表2-1" 或 "表2-10"
+TABLE_RE = re.compile(r"表(\d+)-(\d+)")
+
+
+# ── 数据结构 ────────────────────────────────────────────────
+
+@dataclass
+class Chunk:
+    """一个语义块"""
+    chunk_id: str               # "2.1" / "2.1.1"
+    title: str                  # "基本语法" / "标识符"
+    level: int                  # 0=章, 1=节, 2=子节
+    parent_id: Optional[str]    # "2" 或 "2.1"
+    text: str                   # 块内容（不含标题行自身）
+    lines: tuple[int, int]      # (start_line, end_line) 1-indexed
+    examples: list[str] = field(default_factory=list)  # ["2-1", "2-2"]
+    tables: list[str] = field(default_factory=list)    # ["2-1", "2-2"]
+    has_code: bool = False      # 是否包含代码示例
+
+
+# ── 分块器 ──────────────────────────────────────────────────
+
+class SectionChunker:
+    """按教材章节层级拆分语义块"""
+
+    def __init__(self, text: str):
+        self.lines = text.splitlines()
+        self._chunks: list[Chunk] = []
+
+    def chunk(self) -> list[Chunk]:
+        """执行分块，返回 Chunk 列表"""
+        self._chunks = []
+        boundaries = self._find_section_boundaries()
+
+        for i, (start, end) in enumerate(boundaries):
+            header_line = self.lines[start]
+            chunk = self._build_chunk(header_line, start, end, boundaries)
+
+            # 章标题使用特殊 ID
+            if chunk.level == 0:
+                chunk.chunk_id = f"ch{chunk.chunk_id}"
+
+            # 提取块内的示例和表格引用
+            body = "\n".join(self.lines[start + 1 : end])
+            chunk.examples = EXAMPLE_RE.findall(body)
+            chunk.tables = TABLE_RE.findall(body)
+            chunk.has_code = "```" in body or "<script>" in body
+
+            self._chunks.append(chunk)
+
+        return self._chunks
+
+    # ── 内部方法 ────────────────────────────────────────────
+
+    def _find_section_boundaries(self) -> list[tuple[int, int]]:
+        """找出所有标题行及其对应的行号区间"""
+        headers: list[tuple[int, int, str]] = []  # (line_no, level, title_text)
+
+        for idx, line in enumerate(self.lines):
+            # 尝试匹配章标题
+            m = CHAPTER_RE.match(line)
+            if m:
+                headers.append((idx, 0, line.strip()))
+                continue
+
+            # 尝试匹配节/子节标题
+            m = SECTION_RE.match(line)
+            if m:
+                # 判断层级
+                if m.group(3) is not None:  # 2.1.1 → level 2
+                    level = 2
+                else:
+                    level = 1
+                headers.append((idx, level, line.strip()))
+
+        # 转换成 (start, end) 区间
+        boundaries: list[tuple[int, int]] = []
+        for i, (start, _, _) in enumerate(headers):
+            end = headers[i + 1][0] if i + 1 < len(headers) else len(self.lines)
+            boundaries.append((start, end))
+
+        return boundaries
+
+    def _build_chunk(
+        self,
+        header_line: str,
+        start: int,
+        end: int,
+        boundaries: list[tuple[int, int]],
+    ) -> Chunk:
+        """从标题行和行区间构建 Chunk"""
+        # 提取层级信息
+        cm = CHAPTER_RE.match(header_line)
+        if cm:
+            ch_num = cm.group(1)
+            title = cm.group(2)
+            # 找前一个 header 作为 parent
+            parent_id = None
+            for s, _ in boundaries:
+                h = self.lines[s]
+                sm = SECTION_RE.match(h)
+                if sm and sm.group(1) == ch_num and sm.group(3) is None:
+                    parent_id = sm.group(1)
+                    break
+            return Chunk(
+                chunk_id=ch_num,
+                title=title,
+                level=0,
+                parent_id=parent_id,
+                text="\n".join(self.lines[start + 1 : end]),
+                lines=(start + 1, end),
+            )
+
+        sm = SECTION_RE.match(header_line)
+        if sm:
+            major = sm.group(1)
+            minor = sm.group(2)
+            sub = sm.group(3)  # None 表示 level 1
+            title = sm.group(4).strip()
+
+            if sub is None:
+                # Level 1: "2.1 基本语法"
+                chunk_id = f"{major}.{minor}"
+                parent_id = f"ch{major}"
+                level = 1
+            else:
+                # Level 2: "2.1.1 标识符"
+                chunk_id = f"{major}.{minor}.{sub}"
+                parent_id = f"{major}.{minor}"
+                level = 2
+
+            return Chunk(
+                chunk_id=chunk_id,
+                title=title,
+                level=level,
+                parent_id=parent_id,
+                text="\n".join(self.lines[start + 1 : end]),
+                lines=(start + 1, end),
+            )
+
+        # 兜底（理论上不会走到这里）
+        return Chunk(
+            chunk_id="unknown",
+            title=header_line.strip(),
+            level=-1,
+            parent_id=None,
+            text="\n".join(self.lines[start + 1 : end]),
+            lines=(start + 1, end),
+        )
+
+
+# ── 入口 ────────────────────────────────────────────────────
+
+def chunk_file(txt_path: str | Path) -> list[Chunk]:
+    """读取 txt 文件并分块"""
+    txt_path = Path(txt_path)
+    if not txt_path.exists():
+        raise FileNotFoundError(f"文件未找到: {txt_path}")
+
+    text = txt_path.read_text(encoding="utf-8")
+    chunker = SectionChunker(text)
+    return chunker.chunk()
+
+
+def safe_print(text: str = "", **kwargs):
+    """安全打印，避免 Windows GBK 编码问题"""
+    try:
+        print(text, **kwargs)
+    except UnicodeEncodeError:
+        print(text.encode("utf-8", errors="replace").decode("utf-8", errors="replace"), **kwargs)
+
+
+def print_summary(chunks: list[Chunk]) -> None:
+    """打印分块摘要（用于验证）"""
+    safe_print(f"\n{'='*60}")
+    safe_print(f"共 {len(chunks)} 个 Chunk")
+    safe_print(f"{'='*60}\n")
+    for c in chunks:
+        line_info = f"L{c.lines[0]}-{c.lines[1]}"
+        parent = f" ← {c.parent_id}" if c.parent_id else ""
+        tags = []
+        if c.examples:
+            tags.append(f"示例:{len(c.examples)}")
+        if c.tables:
+            tags.append(f"表格:{len(c.tables)}")
+        if c.has_code:
+            tags.append("含代码")
+        tag_str = f"  [{', '.join(tags)}]" if tags else ""
+        text_preview = c.text[:60].replace("\n", " ")
+        safe_print(f"  [{c.chunk_id}] (level={c.level}) {line_info}{parent}")
+        safe_print(f"     标题: {c.title}")
+        safe_print(f"     预览: {text_preview}...{tag_str}")
+        safe_print()
+
+
+# ── CLI ─────────────────────────────────────────────────────
+
+def main(argv: list[str] | None = None) -> None:
+    """CLI 入口：分块文本文件并打印摘要。
+
+    用法: kg-chunker <txt_path>
+          kg-chunker （使用 KG_CHUNKER_INPUT 环境变量）
+    """
+    if argv is None:
+        argv = sys.argv[1:]
+
+    if len(argv) >= 1:
+        txt_path = argv[0]
+    elif os.environ.get("KG_CHUNKER_INPUT"):
+        txt_path = os.environ["KG_CHUNKER_INPUT"]
+    else:
+        print("用法: kg-chunker <txt_path>", file=sys.stderr)
+        print("      或设置 KG_CHUNKER_INPUT 环境变量", file=sys.stderr)
+        sys.exit(1)
+
+    chunks = chunk_file(txt_path)
+    print_summary(chunks)
+
+
+if __name__ == "__main__":
+    main()