reindex-cli 0.3.0__tar.gz

reindex_cli-0.3.0/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.4
+Name: reindex-cli
+Version: 0.3.0
+Summary: Compile local files into ReIndex packages.
+Project-URL: Homepage, https://github.com/yechenyan/reIndex
+Project-URL: Documentation, https://github.com/yechenyan/reIndex/tree/main/wiki
+Project-URL: Repository, https://github.com/yechenyan/reIndex
+Project-URL: Issues, https://github.com/yechenyan/reIndex/issues
+Keywords: agents,cli,knowledge,search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Topic :: Utilities
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: docling<3,>=2
+Requires-Dist: httpx<1,>=0.27
+Requires-Dist: pypdfium2<6,>=5
+Requires-Dist: pyyaml<7,>=6
+
+# ReIndex CLI
+
+`rei` builds local files into ReIndex 1.0 packages and connects them to a ReIndex API. `reindex` is an alias.
+
+## Install and initialize
+
+Give an AI Agent this sentence:
+
+> Run `uv tool install --upgrade reindex-cli`, then run `rei init <data-directory> --agent <current-agent>` to create or update the ReIndex skills.
+
+The installation is complete when `rei --help` works directly, without
+`uv run`. `pipx install reindex-cli` and `python -m pip install reindex-cli`
+are also supported. Python 3.12 or newer is required; the Docling dependency
+makes the first installation and first PDF scan larger than a typical CLI.
+
+`<current-agent>` is `codex`, `claude`, `cursor`, or `copilot`. `init` is idempotent, creates or reuses local identity, and installs or safely updates the three bundled skills. It does not scan, push, or download tutorial data.
+
+```bash
+rei init <collection-dir> [--name <name>] [--agent codex]
+rei rename <collection-dir> <new-name>
+```
+
+## Local commands
+
+```bash
+rei inspect <path>
+rei scan <path> [--collection-root <collection-dir>]
+rei check <path>
+```
+
+`inspect` is read-only. `scan` stages, validates, and atomically publishes a package. `check` verifies package resources, protected Node metadata, and current inputs.
+
+For the normal author path, `push` runs the same package check again, so an
+explicit `check` is mainly useful after manually editing cards or in CI:
+
+```bash
+rei init <collection-dir> --name <name> --agent codex
+rei inspect <collection-dir>
+rei scan <collection-dir>
+rei push <collection-dir> --api-url <api-url>
+```
+
+## Remote commands
+
+```bash
+rei set-api <base-url>
+rei push [path]
+rei pull <name> [--output <directory>]
+rei search "<query>" [--remote <name>] [--mode lexical|semantic|hybrid]
+rei get <node-path> [--target card|source|content|asset]
+rei get raw://<path>
+```
+
+`push` synchronously sends the complete validated package and exactly referenced raw sources. `pull` downloads only the complete `.node.md` tree. `get` checks a complete local package, local authoring source, and the SHA-256 cache before downloading one exact resource.
+
+Collection names are the user-facing remote identifier. UUIDs remain internal stable identity. Resource logical paths stay Collection-relative and are scoped by the internal Collection UUID.
+
+## Skills
+
+```bash
+rei skills install --agent codex
+rei skills update --agent codex
+rei skills update --agent codex --force
+```
+
+Normal updates overwrite only unmodified ReIndex-managed copies. A modified skill reports a conflict unless `--force` is explicit.
+
+The CLI does not include or provision a hosted API. Remote commands use the
+configured ReIndex API; without one, the default is the local development
+endpoint at `http://127.0.0.1:8000`.

reindex_cli-0.3.0/README.md

@@ -0,0 +1,70 @@
+# ReIndex CLI
+
+`rei` builds local files into ReIndex 1.0 packages and connects them to a ReIndex API. `reindex` is an alias.
+
+## Install and initialize
+
+Give an AI Agent this sentence:
+
+> Run `uv tool install --upgrade reindex-cli`, then run `rei init <data-directory> --agent <current-agent>` to create or update the ReIndex skills.
+
+The installation is complete when `rei --help` works directly, without
+`uv run`. `pipx install reindex-cli` and `python -m pip install reindex-cli`
+are also supported. Python 3.12 or newer is required; the Docling dependency
+makes the first installation and first PDF scan larger than a typical CLI.
+
+`<current-agent>` is `codex`, `claude`, `cursor`, or `copilot`. `init` is idempotent, creates or reuses local identity, and installs or safely updates the three bundled skills. It does not scan, push, or download tutorial data.
+
+```bash
+rei init <collection-dir> [--name <name>] [--agent codex]
+rei rename <collection-dir> <new-name>
+```
+
+## Local commands
+
+```bash
+rei inspect <path>
+rei scan <path> [--collection-root <collection-dir>]
+rei check <path>
+```
+
+`inspect` is read-only. `scan` stages, validates, and atomically publishes a package. `check` verifies package resources, protected Node metadata, and current inputs.
+
+For the normal author path, `push` runs the same package check again, so an
+explicit `check` is mainly useful after manually editing cards or in CI:
+
+```bash
+rei init <collection-dir> --name <name> --agent codex
+rei inspect <collection-dir>
+rei scan <collection-dir>
+rei push <collection-dir> --api-url <api-url>
+```
+
+## Remote commands
+
+```bash
+rei set-api <base-url>
+rei push [path]
+rei pull <name> [--output <directory>]
+rei search "<query>" [--remote <name>] [--mode lexical|semantic|hybrid]
+rei get <node-path> [--target card|source|content|asset]
+rei get raw://<path>
+```
+
+`push` synchronously sends the complete validated package and exactly referenced raw sources. `pull` downloads only the complete `.node.md` tree. `get` checks a complete local package, local authoring source, and the SHA-256 cache before downloading one exact resource.
+
+Collection names are the user-facing remote identifier. UUIDs remain internal stable identity. Resource logical paths stay Collection-relative and are scoped by the internal Collection UUID.
+
+## Skills
+
+```bash
+rei skills install --agent codex
+rei skills update --agent codex
+rei skills update --agent codex --force
+```
+
+Normal updates overwrite only unmodified ReIndex-managed copies. A modified skill reports a conflict unless `--force` is explicit.
+
+The CLI does not include or provision a hosted API. Remote commands use the
+configured ReIndex API; without one, the default is the local development
+endpoint at `http://127.0.0.1:8000`.

reindex_cli-0.3.0/pyproject.toml

@@ -0,0 +1,46 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "reindex-cli"
+version = "0.3.0"
+description = "Compile local files into ReIndex packages."
+requires-python = ">=3.12"
+readme = "README.md"
+keywords = ["agents", "cli", "knowledge", "search"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Text Processing :: Indexing",
+    "Topic :: Utilities",
+]
+dependencies = [
+    "docling>=2,<3",
+    "httpx>=0.27,<1",
+    "pypdfium2>=5,<6",
+    "pyyaml>=6,<7",
+]
+
+[project.urls]
+Homepage = "https://github.com/yechenyan/reIndex"
+Documentation = "https://github.com/yechenyan/reIndex/tree/main/wiki"
+Repository = "https://github.com/yechenyan/reIndex"
+Issues = "https://github.com/yechenyan/reIndex/issues"
+
+[project.scripts]
+rei = "reindex_cli.cli:main"
+reindex = "reindex_cli.cli:main"
+
+[tool.setuptools.package-dir]
+"" = "src"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+reindex_cli = ["bundled_skills/*/SKILL.md"]

reindex_cli-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

reindex_cli-0.3.0/src/reindex_cli/__init__.py

@@ -0,0 +1,6 @@
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("reindex-cli")
+except PackageNotFoundError:  # Source-only execution without an installed distribution.
+    __version__ = "0.0.0+local"

reindex_cli-0.3.0/src/reindex_cli/__main__.py

@@ -0,0 +1,3 @@
+from reindex_cli.cli import main
+
+raise SystemExit(main())

reindex_cli-0.3.0/src/reindex_cli/api.py

@@ -0,0 +1,69 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import httpx
+
+from reindex_cli.errors import ReIndexError
+
+
+class ApiClient:
+    def __init__(self, base_url: str, timeout: float = 1800.0) -> None:
+        self.base_url = base_url.rstrip("/")
+        self.timeout = timeout
+
+    def health(self) -> dict:
+        with httpx.Client(base_url=self.base_url, timeout=30.0) as client:
+            return self._json(client.get("/health"))
+
+    def push(self, name: str, package: Path, sources: Path) -> dict:
+        with (
+            package.open("rb") as package_stream,
+            sources.open("rb") as source_stream,
+            httpx.Client(base_url=self.base_url, timeout=self.timeout) as client,
+        ):
+            response = client.post(
+                "/v1/push",
+                data={"name": name},
+                files={
+                    "package": ("package.zip", package_stream, "application/zip"),
+                    "sources": ("sources.zip", source_stream, "application/zip"),
+                },
+            )
+            return self._json(response)
+
+    def json(self, path: str, payload: dict) -> dict:
+        with httpx.Client(base_url=self.base_url, timeout=self.timeout) as client:
+            return self._json(client.post(path, json=payload))
+
+    def bytes(self, path: str, payload: dict) -> tuple[bytes, dict[str, str]]:
+        with httpx.Client(base_url=self.base_url, timeout=self.timeout) as client:
+            response = client.post(path, json=payload)
+            self._raise(response)
+            return response.content, dict(response.headers)
+
+    def _json(self, response: httpx.Response) -> dict:
+        self._raise(response)
+        try:
+            value = response.json()
+        except json.JSONDecodeError as error:
+            raise ReIndexError("Server returned invalid JSON") from error
+        if not isinstance(value, dict):
+            raise ReIndexError("Server returned an invalid response")
+        return value
+
+    @staticmethod
+    def _raise(response: httpx.Response) -> None:
+        if response.is_success:
+            return
+        try:
+            body = response.json()
+            detail = body.get("error", body)
+            message = detail.get("message") if isinstance(detail, dict) else None
+        except (json.JSONDecodeError, AttributeError):
+            message = None
+        raise ReIndexError(
+            f"Server request failed ({response.status_code}): "
+            f"{message or response.text or response.reason_phrase}"
+        )

reindex_cli-0.3.0/src/reindex_cli/archives.py

@@ -0,0 +1,97 @@
+from __future__ import annotations
+
+import tempfile
+import zipfile
+from pathlib import Path, PurePosixPath
+
+from reindex_cli.errors import ReIndexError
+from reindex_cli.package.cards import parse_card
+from reindex_cli.util import sha256_file
+
+
+def build_push_archives(context) -> tuple[Path, Path, tempfile.TemporaryDirectory]:
+    temporary = tempfile.TemporaryDirectory(prefix="rei-push-")
+    root = Path(temporary.name)
+    package_zip = root / "package.zip"
+    source_zip = root / "sources.zip"
+    with zipfile.ZipFile(package_zip, "w", zipfile.ZIP_DEFLATED) as bundle:
+        for file in sorted(context.output_dir.rglob("*")):
+            if file.is_file():
+                relative = Path(context.output_dir.name) / file.relative_to(
+                    context.output_dir
+                )
+                bundle.write(file, relative.as_posix())
+    references = raw_references(context.output_dir)
+    with zipfile.ZipFile(source_zip, "w", zipfile.ZIP_DEFLATED) as bundle:
+        for relative, expected in sorted(references.items()):
+            source = context.root / relative
+            if not source.is_file() or sha256_file(source) != expected:
+                temporary.cleanup()
+                raise ReIndexError(f"Missing or changed source: {relative}")
+            bundle.write(source, relative)
+    return package_zip, source_zip, temporary
+
+
+def raw_references(package: Path) -> dict[str, str]:
+    result: dict[str, str] = {}
+    for card_path in package.rglob("*.node.md"):
+        metadata, _body = parse_card(card_path)
+        values = [
+            metadata.get("source"),
+            metadata.get("content"),
+            *metadata.get("assets", []),
+        ]
+        for value in values:
+            if not isinstance(value, dict):
+                continue
+            uri = str(value.get("uri", ""))
+            if not uri.startswith("raw://"):
+                continue
+            relative = _safe(uri.removeprefix("raw://"))
+            digest = str(value.get("sha256", ""))
+            if relative in result and result[relative] != digest:
+                raise ReIndexError(f"Conflicting source hashes: {relative}")
+            result[relative] = digest
+    return result
+
+
+def extract_node_archive(content: bytes, target: Path) -> int:
+    target.mkdir(parents=True, exist_ok=True)
+    archive_path = target / ".download.zip"
+    archive_path.write_bytes(content)
+    count = 0
+    try:
+        with zipfile.ZipFile(archive_path) as bundle:
+            names: set[str] = set()
+            for item in bundle.infolist():
+                path = PurePosixPath(item.filename)
+                if (
+                    item.filename in names
+                    or path.is_absolute()
+                    or ".." in path.parts
+                    or not item.filename.endswith(".node.md")
+                    or (item.external_attr >> 16) & 0o170000 == 0o120000
+                ):
+                    raise ReIndexError(
+                        f"Unsafe or non-Node pull entry: {item.filename}"
+                    )
+                names.add(item.filename)
+                if item.is_dir():
+                    continue
+                destination = target.joinpath(*path.parts)
+                destination.parent.mkdir(parents=True, exist_ok=True)
+                destination.write_bytes(bundle.read(item))
+                parse_card(destination)
+                count += 1
+    finally:
+        archive_path.unlink(missing_ok=True)
+    if not (target / "index.node.md").is_file():
+        raise ReIndexError("Pulled Node tree has no root index.node.md")
+    return count
+
+
+def _safe(value: str) -> str:
+    path = PurePosixPath(value)
+    if path.is_absolute() or ".." in path.parts or not path.parts:
+        raise ReIndexError(f"Unsafe raw path: {value}")
+    return path.as_posix()

reindex_cli-0.3.0/src/reindex_cli/bundled_skills/reindex-create/SKILL.md

@@ -0,0 +1,12 @@
+---
+name: reindex-create
+description: Initialize a ReIndex project when the user asks to set up ReIndex or run rei init/create.
+---
+
+# ReIndex create
+
+1. Resolve the exact directory named by the user.
+2. Prefer `rei init <directory> --agent <current-agent>` for normal setup. It is idempotent and manages skills.
+3. Use `rei create <directory>` only when identity-only initialization was explicitly requested.
+4. Report the Collection name and whether it was created or reused. UUID is internal and normally omitted.
+5. Do not scan, push, delete source files, or replace `.rei/collection.json` during initialization.

reindex_cli-0.3.0/src/reindex_cli/bundled_skills/reindex-data/SKILL.md

@@ -0,0 +1,22 @@
+---
+name: reindex-data
+description: Push, pull, search, and get exact ReIndex data when the user asks to publish or use remote ReIndex knowledge.
+---
+
+# ReIndex data
+
+## Publish
+
+1. Run `rei check <path>` before `rei push <path>`; scan first if stale.
+2. `rei push` synchronously sends the validated package and exactly referenced sources.
+3. Report the user-facing Collection name and ready status, not internal UUID details.
+
+## Find and fetch
+
+1. Use `rei search "<question>"` before downloading large files.
+2. Select the result whose Evidence supports the task.
+3. Use the result's Node path with `rei get <node-path> --target content`; use `source` only when original bytes are required.
+4. Use `rei get raw://<path>` for an explicitly named raw source.
+5. Answer from the fetched file and cite the Collection name and Node or raw path.
+
+`rei pull <name>` downloads the complete Node tree only. It intentionally excludes source, content and assets; fetch exact resources later with `rei get`.

reindex_cli-0.3.0/src/reindex_cli/bundled_skills/reindex-scan/SKILL.md

@@ -0,0 +1,13 @@
+---
+name: reindex-scan
+description: Scan raw files into ReIndex when the user asks to scan, compile, ingest, or convert local data.
+---
+
+# ReIndex scan
+
+1. Run `rei inspect <path>` and review effective inputs, relationships and changes.
+2. Apply only evidence-backed manifest corrections; never delete raw files just to pass validation.
+3. Run `rei scan <path>` and review changes, warnings and generated Node cards.
+4. Edit Markdown card bodies only. The CLI owns YAML frontmatter.
+5. Run `rei check <path>` after manual card edits.
+6. Report the Collection name, Node count, warnings and package location. Passing checks is not human approval.

reindex_cli-0.3.0/src/reindex_cli/cli.py

@@ -0,0 +1,176 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import httpx
+
+from reindex_cli import __version__
+from reindex_cli.collection import create_collection, resolve_collection
+from reindex_cli.config import set_api_url
+from reindex_cli.errors import ReIndexError
+from reindex_cli.skills import AGENTS, manage_skills
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="rei", description="Build and use ReIndex knowledge Collections."
+    )
+    parser.add_argument("--version", action="version", version=__version__)
+    commands = parser.add_subparsers(dest="command", required=True)
+    init = commands.add_parser("init", help="Initialize ReIndex and Agent skills.")
+    init.add_argument("path", type=Path, nargs="?", default=Path.cwd())
+    init.add_argument("--name")
+    init.add_argument("--agent", choices=AGENTS, default="codex")
+    init.add_argument("--codex-home", type=Path)
+    create = commands.add_parser("create", help="Create only local identity state.")
+    create.add_argument("path", type=Path)
+    create.add_argument("--name")
+    rename = commands.add_parser("rename", help="Change the Collection name.")
+    rename.add_argument("path", type=Path)
+    rename.add_argument("name")
+    inspect = commands.add_parser("inspect", help="Inspect inputs without writing.")
+    inspect.add_argument("path", type=Path)
+    scan = commands.add_parser("scan", help="Compile a validated local package.")
+    scan.add_argument("path", type=Path)
+    scan.add_argument("--collection-root", type=Path)
+    check = commands.add_parser("check", help="Validate the current package.")
+    check.add_argument("path", type=Path)
+    skills = commands.add_parser("skills", help="Manage bundled Agent skills.")
+    skill_commands = skills.add_subparsers(dest="skill_command", required=True)
+    for name in ("install", "update"):
+        command = skill_commands.add_parser(name)
+        command.add_argument("--agent", choices=AGENTS, default="codex")
+        command.add_argument("--workspace-root", type=Path, default=Path.cwd())
+        command.add_argument("--codex-home", type=Path)
+        command.add_argument("--force", action="store_true")
+    api = commands.add_parser("set-api", help="Persist the default API URL.")
+    api.add_argument("url")
+    push = commands.add_parser(
+        "push", help="Synchronously publish package and sources."
+    )
+    push.add_argument("path", type=Path, nargs="?", default=Path.cwd())
+    push.add_argument("--api-url")
+    pull = commands.add_parser("pull", help="Download a Node-only ReIndex tree.")
+    pull.add_argument("name")
+    pull.add_argument("--output", type=Path)
+    pull.add_argument("--api-url")
+    pull.add_argument("--force", action="store_true")
+    search = commands.add_parser("search", help="Search a remote Collection.")
+    search.add_argument("query")
+    search.add_argument("--path", type=Path, default=Path.cwd())
+    search.add_argument("--remote")
+    search.add_argument("--api-url")
+    search.add_argument(
+        "--mode", choices=("lexical", "semantic", "hybrid"), default="lexical"
+    )
+    search.add_argument("--limit", type=int, default=10)
+    get = commands.add_parser("get", help="Reuse or fetch one exact resource.")
+    get.add_argument("reference")
+    get.add_argument("--path", type=Path, default=Path.cwd())
+    get.add_argument("--remote")
+    get.add_argument("--api-url")
+    get.add_argument("--target", choices=("card", "source", "content", "asset"))
+    get.add_argument("--asset-ordinal", type=int)
+    get.add_argument("--output", type=Path)
+    return parser
+
+
+def main(argv: list[str] | None = None) -> int:
+    try:
+        args = build_parser().parse_args(argv)
+        output = _execute(args)
+        print(json.dumps(output, ensure_ascii=False, indent=2))
+        return 0
+    except (ReIndexError, OSError, ValueError, httpx.HTTPError) as error:
+        print(
+            json.dumps({"status": "error", "error": str(error)}, ensure_ascii=False),
+            file=sys.stderr,
+        )
+        return 1
+
+
+def _execute(args) -> dict:
+    if args.command == "init":
+        root = args.path.expanduser().resolve()
+        collection = create_collection(root, args.name)
+        skills = manage_skills(
+            args.agent,
+            root,
+            update=True,
+            codex_home=args.codex_home,
+        )
+        return {
+            "status": "ready",
+            **collection,
+            "collection_id": collection["id"],
+            "skills": [vars(item) for item in skills],
+        }
+    if args.command in {"create", "rename"}:
+        name = args.name
+        result = create_collection(args.path.expanduser().resolve(), name)
+        return {
+            "status": "ready",
+            **result,
+            "collection_id": result["id"],
+            "renamed": args.command == "rename",
+        }
+    if args.command == "skills":
+        results = manage_skills(
+            args.agent,
+            args.workspace_root.expanduser().resolve(),
+            update=args.skill_command == "update",
+            force=args.force,
+            codex_home=args.codex_home,
+        )
+        return {"status": "ready", "skills": [vars(item) for item in results]}
+    if args.command == "set-api":
+        return {"status": "ready", "api_url": set_api_url(args.url)}
+    if args.command in {"push", "pull", "search"}:
+        from reindex_cli.remote_ops import (
+            pull_collection,
+            push_collection,
+            search_remote,
+        )
+
+    if args.command == "push":
+        return push_collection(args.path, args.api_url)
+    if args.command == "pull":
+        return pull_collection(
+            args.name,
+            args.output or Path.cwd() / args.name,
+            args.api_url,
+            force=args.force,
+        )
+    if args.command == "search":
+        return search_remote(
+            args.query, args.path, args.remote, args.api_url, args.mode, args.limit
+        )
+    if args.command == "get":
+        from reindex_cli.get_ops import get_resource
+
+        return get_resource(
+            args.reference,
+            args.path,
+            target=args.target,
+            asset_ordinal=args.asset_ordinal,
+            output=args.output,
+            remote=args.remote,
+            api_url=args.api_url,
+        )
+    from reindex_cli.pipeline.runner import (
+        check_collection,
+        inspect_collection,
+        run_scan,
+    )
+
+    context = resolve_collection(
+        args.path, args.collection_root if args.command == "scan" else None
+    )
+    if args.command == "inspect":
+        return inspect_collection(context)
+    if args.command == "scan":
+        return run_scan(context)
+    return check_collection(context)

reindex_cli-0.3.0/src/reindex_cli/collection/__init__.py

@@ -0,0 +1,9 @@
+from .resolver import CollectionContext, resolve_collection
+from .state import create_collection, load_collection
+
+__all__ = [
+    "CollectionContext",
+    "create_collection",
+    "load_collection",
+    "resolve_collection",
+]