ragjson 0.1.4__tar.gz → 0.2.0__tar.gz

{ragjson-0.1.4 → ragjson-0.2.0}/.gitignore

@@ -9,4 +9,5 @@ wheels/
 # Virtual environments
 .venv
 
-data/
+data/
+docs/rag.comfig

{ragjson-0.1.4 → ragjson-0.2.0}/CLAUDE.md

@@ -23,30 +23,8 @@ ragjson — 遵循 RAG-Storage-Spec 规范的文件系统 RAG 知识库构建工
 - 蘑菇文件（`.mushroom`）：`rag update` 生成的变更通知（JSONL），供外部索引工具消费
 - 删除机制：源文件删除时 rename `.ragjson` → `.ragjson.deleted`（非状态转换）
 
-## 命令
-
-| 命令 | 说明 |
-|------|------|
-| `rag status [path] [-r]` | 查看知识库状态 |
-| `rag update [path] [-r] [-j N]` | 增量更新（含删除/恢复检测 + 蘑菇生成） |
-| `rag cleanup [path] [-r]` | 清理 .tmp 和 .ragjson.deleted |
-| `rag destroy [path] [-r] [-y]` | 销毁 .rag/ 目录 |
-| `rag export <output> [path] [-r] [-y]` | 导出 .ragjson 为 tar.gz |
 
 ## 规范文档
 
 - `docs/RAG-Storage-Spec.md` — 协议规范（v2.2）
-- `docs/design/mushroom-sync.md` — 蘑菇 + zvec 索引集成设计
-
-## 开发
-
-```bash
-# 安装
-uv pip install -e ".[all]"
-
-# 运行
-rag --help
 
-# 构建
-uv build
-```

ragjson-0.1.4/README.md → ragjson-0.2.0/PKG-INFO

@@ -1,3 +1,15 @@
+Metadata-Version: 2.4
+Name: ragjson
+Version: 0.2.0
+Summary: 遵循RAG-Storage-Spec的知识库构建工具
+Requires-Python: >=3.12
+Requires-Dist: langchain-text-splitters>=0.3
+Requires-Dist: markitdown[all]
+Requires-Dist: openai>=1.0
+Provides-Extra: index
+Requires-Dist: zvec>=0.5; extra == 'index'
+Description-Content-Type: text/markdown
+
 # ragjson
 
 遵循 [RAG-Storage-Spec](docs/RAG-Storage-Spec.md) 规范的基于文件系统的 RAG 知识库构建工具。
@@ -18,7 +30,7 @@
 
 ```bash
 # 全局安装为独立命令（推荐，无需 venv）
-uv tool install ragjson[all]
+uv tool install ragjson
 ```
 
 或在项目虚拟环境中使用：
@@ -26,25 +38,17 @@ uv tool install ragjson[all]
 ```bash
 uv venv
 source .venv/bin/activate
-uv pip install ragjson[all]
+uv pip install ragjson
 ```
 
 本地开发模式（从源码安装）：
 
 ```bash
 uv venv
-uv pip install -e ".[all]"
+uv pip install -e .
 source .venv/bin/activate
 ```
 
-可选的 extras 组合：
-
-| extras | 说明 |
-|--------|------|
-| `ragjson` | 基础安装（切片 + Embedding） |
-| `ragjson[markitdown]` | 增加 markitdown 文档解析 |
-| `ragjson[all]` | 全部依赖 |
-
 ## 配置
 
 在用户 home 目录下创建 `~/.rag.config` 文件（JSON 格式）：
@@ -60,6 +64,10 @@ source .venv/bin/activate
   "splitter": {
     "chunk_size": 1000,
     "chunk_overlap": 200
+  },
+  "vector_database": {
+    "db_url": "zvec:/data/vectordb",
+    "index_type": "flat"
   }
 }
 ```
@@ -72,6 +80,8 @@ source .venv/bin/activate
 | `embedding_model.dimension` | 向量维度 |
 | `splitter.chunk_size` | 切片最大字符数 |
 | `splitter.chunk_overlap` | 切片重叠字符数 |
+| `vector_database.db_url` | 向量数据库目录（collection 在其下的 `zvec-db1/`） |
+| `vector_database.index_type` | 索引类型：`flat` / `hnsw` |
 
 ## 使用方法
 
@@ -119,15 +129,6 @@ rag update ./docs -r -j 4
 | `-j N` | 并行处理的目录数 | CPU 核心数 |
 | `--splitter` | 切片器类型：`recursive` / `markdown` | `recursive` |
 
-### 清理
-
-清除残留的 `.tmp` 临时文件和 `.ragjson.deleted` 文件：
-
-```bash
-rag cleanup ./docs
-rag cleanup ./docs -r  # 递归清理
-```
-
 ### 销毁
 
 删除 `.rag/` 目录及所有元数据（高危操作，需确认）：
@@ -147,6 +148,26 @@ rag export backup.tar.gz          # 仅当前目录
 rag export backup.tar.gz -r       # 递归包含子目录
 ```
 
+### 索引同步
+
+消费 `.mushroom` 变更通知文件，将向量同步到向量数据库（需安装 `ragjson[index]`）：
+
+```bash
+# 同步当前目录
+rag index
+
+# 递归同步子目录
+rag index -r
+```
+
+向量数据库路径和索引类型在 `~/.rag.config` 的 `vector_database` 节配置。
+
+安装索引依赖：
+
+```bash
+uv pip install ragjson[index]
+```
+
 ## 目录结构
 
 执行 `rag update` 后的目录结构示例：
@@ -204,10 +225,25 @@ INITIAL → CHUNKING → CHUNKED → EMBEDDING → EMBEDDED
                                             ↓
     (源文件变更) ←─────────────────── INITIAL (回退)
 
-    源文件删除：任意状态 → rename .ragjson.deleted → (cleanup 物理删除)
+    源文件删除：任意状态 → rename .ragjson → .ragjson.deleted
     源文件恢复：.ragjson.deleted → rename .ragjson → (update 检测 MD5)
 ```
 
+## 项目架构
+
+```
+src/ragjson/
+├── commands/     # CLI 命令（status, update, destroy, export, index）
+├── embed/        # Embedding 生成（OpenAI 兼容 API）
+├── index/        # 向量索引（zvec backend，可扩展）
+├── parser/       # 文档解析（markitdown backend，可扩展）
+├── splitter/     # 文本切片（recursive / markdown）
+├── store/        # 存储层（ragjson 文件、原子操作、mushroom）
+├── config.py     # ~/.rag.config 配置加载
+├── utils.py      # 工具函数（MD5、向量编码、颜色输出）
+└── cli.py        # CLI 入口
+```
+
 ## 开发
 
 ```bash
@@ -217,7 +253,7 @@ cd ragjson
 
 # 创建虚拟环境并安装开发版本
 uv venv
-uv pip install -e ".[all]"
+uv pip install -e ".[index]"
 source .venv/bin/activate
 
 # 运行 CLI

ragjson-0.1.4/PKG-INFO → ragjson-0.2.0/README.md

@@ -1,16 +1,3 @@
-Metadata-Version: 2.4
-Name: ragjson
-Version: 0.1.4
-Summary: 遵循RAG-Storage-Spec的知识库构建工具
-Requires-Python: >=3.12
-Requires-Dist: langchain-text-splitters>=0.3
-Requires-Dist: openai>=1.0
-Provides-Extra: all
-Requires-Dist: markitdown[all]; extra == 'all'
-Provides-Extra: markitdown
-Requires-Dist: markitdown[all]; extra == 'markitdown'
-Description-Content-Type: text/markdown
-
 # ragjson
 
 遵循 [RAG-Storage-Spec](docs/RAG-Storage-Spec.md) 规范的基于文件系统的 RAG 知识库构建工具。
@@ -31,7 +18,7 @@ Description-Content-Type: text/markdown
 
 ```bash
 # 全局安装为独立命令（推荐，无需 venv）
-uv tool install ragjson[all]
+uv tool install ragjson
 ```
 
 或在项目虚拟环境中使用：
@@ -39,25 +26,17 @@ uv tool install ragjson[all]
 ```bash
 uv venv
 source .venv/bin/activate
-uv pip install ragjson[all]
+uv pip install ragjson
 ```
 
 本地开发模式（从源码安装）：
 
 ```bash
 uv venv
-uv pip install -e ".[all]"
+uv pip install -e .
 source .venv/bin/activate
 ```
 
-可选的 extras 组合：
-
-| extras | 说明 |
-|--------|------|
-| `ragjson` | 基础安装（切片 + Embedding） |
-| `ragjson[markitdown]` | 增加 markitdown 文档解析 |
-| `ragjson[all]` | 全部依赖 |
-
 ## 配置
 
 在用户 home 目录下创建 `~/.rag.config` 文件（JSON 格式）：
@@ -73,6 +52,10 @@ source .venv/bin/activate
   "splitter": {
     "chunk_size": 1000,
     "chunk_overlap": 200
+  },
+  "vector_database": {
+    "db_url": "zvec:/data/vectordb",
+    "index_type": "flat"
   }
 }
 ```
@@ -85,6 +68,8 @@ source .venv/bin/activate
 | `embedding_model.dimension` | 向量维度 |
 | `splitter.chunk_size` | 切片最大字符数 |
 | `splitter.chunk_overlap` | 切片重叠字符数 |
+| `vector_database.db_url` | 向量数据库目录（collection 在其下的 `zvec-db1/`） |
+| `vector_database.index_type` | 索引类型：`flat` / `hnsw` |
 
 ## 使用方法
 
@@ -132,15 +117,6 @@ rag update ./docs -r -j 4
 | `-j N` | 并行处理的目录数 | CPU 核心数 |
 | `--splitter` | 切片器类型：`recursive` / `markdown` | `recursive` |
 
-### 清理
-
-清除残留的 `.tmp` 临时文件和 `.ragjson.deleted` 文件：
-
-```bash
-rag cleanup ./docs
-rag cleanup ./docs -r  # 递归清理
-```
-
 ### 销毁
 
 删除 `.rag/` 目录及所有元数据（高危操作，需确认）：
@@ -160,6 +136,26 @@ rag export backup.tar.gz          # 仅当前目录
 rag export backup.tar.gz -r       # 递归包含子目录
 ```
 
+### 索引同步
+
+消费 `.mushroom` 变更通知文件，将向量同步到向量数据库（需安装 `ragjson[index]`）：
+
+```bash
+# 同步当前目录
+rag index
+
+# 递归同步子目录
+rag index -r
+```
+
+向量数据库路径和索引类型在 `~/.rag.config` 的 `vector_database` 节配置。
+
+安装索引依赖：
+
+```bash
+uv pip install ragjson[index]
+```
+
 ## 目录结构
 
 执行 `rag update` 后的目录结构示例：
@@ -217,10 +213,25 @@ INITIAL → CHUNKING → CHUNKED → EMBEDDING → EMBEDDED
                                             ↓
     (源文件变更) ←─────────────────── INITIAL (回退)
 
-    源文件删除：任意状态 → rename .ragjson.deleted → (cleanup 物理删除)
+    源文件删除：任意状态 → rename .ragjson → .ragjson.deleted
     源文件恢复：.ragjson.deleted → rename .ragjson → (update 检测 MD5)
 ```
 
+## 项目架构
+
+```
+src/ragjson/
+├── commands/     # CLI 命令（status, update, destroy, export, index）
+├── embed/        # Embedding 生成（OpenAI 兼容 API）
+├── index/        # 向量索引（zvec backend，可扩展）
+├── parser/       # 文档解析（markitdown backend，可扩展）
+├── splitter/     # 文本切片（recursive / markdown）
+├── store/        # 存储层（ragjson 文件、原子操作、mushroom）
+├── config.py     # ~/.rag.config 配置加载
+├── utils.py      # 工具函数（MD5、向量编码、颜色输出）
+└── cli.py        # CLI 入口
+```
+
 ## 开发
 
 ```bash
@@ -230,7 +241,7 @@ cd ragjson
 
 # 创建虚拟环境并安装开发版本
 uv venv
-uv pip install -e ".[all]"
+uv pip install -e ".[index]"
 source .venv/bin/activate
 
 # 运行 CLI

{ragjson-0.1.4 → ragjson-0.2.0}/pyproject.toml

@@ -7,18 +7,18 @@ packages = ["src/ragjson"]
 
 [project]
 name = "ragjson"
-version = "0.1.4"
+version = "0.2.0"
 description = "遵循RAG-Storage-Spec的知识库构建工具"
 readme = "README.md"
 requires-python = ">=3.12"
 dependencies = [
     "langchain-text-splitters>=0.3",
     "openai>=1.0",
+    "markitdown[all]",
 ]
 
 [project.optional-dependencies]
-markitdown = ["markitdown[all]"]
-all = ["markitdown[all]"]
+index = ["zvec>=0.5"]
 
 [project.scripts]
 rag = "ragjson.cli:main"

{ragjson-0.1.4 → ragjson-0.2.0}/src/ragjson/__init__.py

@@ -1,4 +1,4 @@
 """ragjson - 遵循RAG-Storage-Spec的知识库构建工具"""
 
-__version__ = "0.1.4"
+__version__ = "0.2.0"
 RAGJSON_VERSION = "2.2"

{ragjson-0.1.4 → ragjson-0.2.0}/src/ragjson/cli.py

@@ -70,23 +70,6 @@ def main() -> None:
         help="Text splitter type (default: recursive)",
     )
 
-    # --- rag cleanup [path] [-r] ---
-    cleanup_parser = subparsers.add_parser(
-        "cleanup",
-        help="Clean .tmp files and remove .ragjson.deleted files",
-    )
-    cleanup_parser.add_argument(
-        "path",
-        nargs="?",
-        default=".",
-        help="Directory path to clean up (default: current directory)",
-    )
-    cleanup_parser.add_argument(
-        "-r", "--recursive",
-        action="store_true",
-        help="Recursively process subdirectories",
-    )
-
     # --- rag destroy [path] [-r] ---
     destroy_parser = subparsers.add_parser(
         "destroy",
@@ -135,6 +118,23 @@ def main() -> None:
         help="Skip overwrite confirmation prompt",
     )
 
+    # --- rag index [path] [-r] ---
+    index_parser = subparsers.add_parser(
+        "index",
+        help="Consume .mushroom files and sync vectors to database",
+    )
+    index_parser.add_argument(
+        "path",
+        nargs="?",
+        default=".",
+        help="Directory path to scan for .mushroom files (default: current directory)",
+    )
+    index_parser.add_argument(
+        "-r", "--recursive",
+        action="store_true",
+        help="Recursively scan subdirectories for .mushroom files",
+    )
+
     # Parse and dispatch
     args = parser.parse_args()
 
@@ -144,9 +144,9 @@ def main() -> None:
 
     # Dispatch to command implementations
     from ragjson.commands import (
-        cmd_cleanup,
         cmd_destroy,
         cmd_export,
+        cmd_index,
         cmd_status,
         cmd_update,
     )
@@ -160,12 +160,12 @@ def main() -> None:
             jobs=args.jobs,
             splitter_type=args.splitter,
         )
-    elif args.command == "cleanup":
-        cmd_cleanup(args.path, args.recursive)
     elif args.command == "destroy":
         cmd_destroy(args.path, args.recursive, args.yes)
     elif args.command == "export":
         cmd_export(args.path, args.output, args.recursive, args.yes)
+    elif args.command == "index":
+        cmd_index(args.path, args.recursive)
     else:
         parser.print_help()
         sys.exit(1)

ragjson-0.2.0/src/ragjson/commands/__init__.py

@@ -0,0 +1,15 @@
+"""CLI command implementations — re-exported from submodules."""
+
+from ragjson.commands.destroy import cmd_destroy
+from ragjson.commands.export import cmd_export
+from ragjson.commands.index import cmd_index
+from ragjson.commands.status import cmd_status
+from ragjson.commands.update import cmd_update
+
+__all__ = [
+    "cmd_destroy",
+    "cmd_export",
+    "cmd_index",
+    "cmd_status",
+    "cmd_update",
+]

ragjson-0.2.0/src/ragjson/commands/_helpers.py

@@ -0,0 +1,90 @@
+"""Shared helper functions for CLI commands."""
+
+from pathlib import Path
+
+from ragjson.parser import is_supported_file
+from ragjson.store import (
+    STATUS_CHUNKED,
+    STATUS_CHUNKING,
+    STATUS_EMBEDDED,
+    STATUS_EMBEDDING,
+    STATUS_INITIAL,
+    is_source_changed,
+    ragjson_path_for,
+    read_ragjson,
+)
+
+
+def collect_dirs(base: Path, recursive: bool) -> list[Path]:
+    """Collect directories to scan, optionally recursive."""
+    dirs = [base]
+    if recursive:
+        for child in sorted(base.rglob("*")):
+            if child.is_dir() and child.name != ".rag":
+                dirs.append(child)
+    return dirs
+
+
+def list_supported_files(scan_dir: Path) -> list[Path]:
+    """List supported files directly in a directory (not recursive)."""
+    files = []
+    for child in sorted(scan_dir.iterdir()):
+        if child.is_file() and is_supported_file(child):
+            files.append(child)
+    return files
+
+
+def files_needing_update(scan_dir: Path) -> list[Path]:
+    """Find files in a directory that need processing.
+
+    A file needs update if:
+    - No corresponding .ragjson exists
+    - The .ragjson status is INITIAL (or stuck intermediate state)
+    - The source MD5 has changed
+    """
+    result = []
+
+    for f in list_supported_files(scan_dir):
+        ragjson_path = ragjson_path_for(scan_dir, f.name)
+        data = read_ragjson(ragjson_path)
+
+        if data is None:
+            result.append(f)
+        elif data.processing.status == STATUS_EMBEDDED:
+            if is_source_changed(data, f):
+                result.append(f)
+        elif data.processing.status in (STATUS_CHUNKING, STATUS_EMBEDDING):
+            # Stuck intermediate state — needs recovery
+            result.append(f)
+        elif data.processing.status in (STATUS_INITIAL, STATUS_CHUNKED):
+            result.append(f)
+
+    return result
+
+
+def has_deletions(scan_dir: Path) -> bool:
+    """Check if any .ragjson files have missing source files (need deletion)."""
+    rag_dir = scan_dir / ".rag"
+    if not rag_dir.exists():
+        return False
+
+    for rj_path in rag_dir.glob("*.ragjson"):
+        source_name = rj_path.name.removesuffix(".ragjson")
+        source_path = scan_dir / source_name
+        if not source_path.exists():
+            return True
+    return False
+
+
+def has_recoveries(scan_dir: Path) -> bool:
+    """Check if any .ragjson.deleted files have their source files back."""
+    rag_dir = scan_dir / ".rag"
+    if not rag_dir.exists():
+        return False
+
+    for deleted_path in rag_dir.glob("*.ragjson.deleted"):
+        source_name = deleted_path.name.removesuffix(".ragjson.deleted")
+        source_path = scan_dir / source_name
+        if source_path.exists():
+            return True
+    return False

ragjson-0.2.0/src/ragjson/commands/destroy.py

@@ -0,0 +1,38 @@
+"""rag destroy [path] [-r] [-y] — remove .rag/ directories."""
+
+import shutil
+import sys
+from pathlib import Path
+
+from ragjson.commands._helpers import collect_dirs
+
+
+def cmd_destroy(path: str, recursive: bool = False, yes: bool = False) -> None:
+    """Remove .rag/ directory and all its contents (DESTRUCTIVE)."""
+    target = Path(path).resolve()
+    if not target.is_dir():
+        print(f"Error: Path is not a directory: {target}", file=sys.stderr)
+        sys.exit(1)
+
+    dirs_to_scan = collect_dirs(target, recursive)
+    rag_dirs = [d / ".rag" for d in dirs_to_scan if (d / ".rag").exists()]
+
+    if not rag_dirs:
+        print("No .rag/ directories found.")
+        return
+
+    print(f"WARNING: This will permanently delete {len(rag_dirs)} .rag/ directory(ies):")
+    for rag_dir in rag_dirs:
+        print(f"  {rag_dir}")
+
+    if not yes:
+        answer = input("\nAre you sure? [y/N] ")
+        if answer.lower() not in ("y", "yes"):
+            print("Aborted.")
+            return
+
+    for rag_dir in rag_dirs:
+        shutil.rmtree(rag_dir)
+        print(f"  Destroyed: {rag_dir}")
+
+    print(f"Total: destroyed {len(rag_dirs)} .rag/ directory(ies).")

ragjson-0.2.0/src/ragjson/commands/export.py

@@ -0,0 +1,37 @@
+"""rag export <output> [path] [-r] [-y] — export .ragjson to tar.gz."""
+
+import sys
+import tarfile
+from pathlib import Path
+
+
+def cmd_export(path: str, output: str, recursive: bool = False, yes: bool = False) -> None:
+    """Export .ragjson files into a tar.gz archive."""
+    target = Path(path).resolve()
+    if not target.is_dir():
+        print(f"Error: Path is not a directory: {target}", file=sys.stderr)
+        sys.exit(1)
+
+    output_path = Path(output).resolve()
+
+    if output_path.exists() and not yes:
+        answer = input(f"File already exists: {output_path}\nOverwrite? [y/N] ")
+        if answer.lower() not in ("y", "yes"):
+            print("Aborted.")
+            return
+
+    if recursive:
+        ragjson_files = list(target.rglob(".rag/*.ragjson"))
+    else:
+        ragjson_files = list((target / ".rag").glob("*.ragjson"))
+
+    if not ragjson_files:
+        print("No .ragjson files found to export.")
+        return
+
+    with tarfile.open(output_path, "w:gz") as tar:
+        for rj_file in ragjson_files:
+            arcname = rj_file.relative_to(target)
+            tar.add(rj_file, arcname=arcname)
+
+    print(f"Exported {len(ragjson_files)} ragjson file(s) to: {output_path}")