deeprefine-cli 0.1.3__py3-none-any.whl

deeprefine_cli-0.1.3.dist-info/METADATA

@@ -0,0 +1,223 @@
+Metadata-Version: 2.4
+Name: deeprefine-cli
+Version: 0.1.3
+Summary: CLI and Cursor skill to refine graphify knowledge graphs with DeepRefine
+Author: HKUST-KnowComp
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/HKUST-KnowComp/DeepRefine
+Project-URL: Documentation, https://github.com/HKUST-KnowComp/DeepRefine
+Project-URL: Repository, https://github.com/HKUST-KnowComp/DeepRefine
+Project-URL: Issues, https://github.com/HKUST-KnowComp/DeepRefine/issues
+Keywords: deeprefine,graphify,knowledge-graph,rag,cursor,agent-skill
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.0
+Dynamic: license-file
+
+# DeepRefine-Skill
+
+Cursor skill and CLI to refine **[graphify](https://github.com/safishamsi/graphify)** knowledge graphs with **[DeepRefine](https://github.com/HKUST-KnowComp/DeepRefine)** (`graphify-out/graph.json` + session query history).
+
+This repository is **standalone**. The DeepRefine model code (`autorefiner`, `atlas_rag`) lives in a separate **DeepRefine** checkout.
+
+---
+
+## Repository layout
+
+```
+DeepRefine-Skill/          ← this repo (pip install -e .)
+├── README.md
+├── SKILL.md
+├── pyproject.toml
+├── deeprefine_skill/
+└── scripts/deeprefine.py
+
+DeepRefine/                ← separate clone (training + Reafiner)
+├── autorefiner/
+├── AutoSchemaKG/
+└── ...
+
+your-kb-project/           ← your data (graphify-out/)
+└── graphify-out/graph.json
+```
+
+Recommended clone layout:
+
+```text
+www/code/
+├── DeepRefine/
+└── DeepRefine-Skill/      # sibling → auto-detected if DEEPREFINE_REPO unset
+```
+
+---
+
+## How graphify and DeepRefine fit together
+
+| Stage | Tool | Input | Output |
+|-------|------|-------|--------|
+| Build | **graphify** | Project files | `graphify-out/graph.json`, report, HTML |
+| Query | **graphify** | Questions | `graphify query "..."` |
+| Refine | **DeepRefine** (this repo) | Graph + **query history** | Updated `graph.json`, logs |
+
+DeepRefine does not build the graph; it edits `graph.json` incrementally so later `graphify query` works better.
+
+---
+
+## Setup
+
+### 1. DeepRefine environment (`atlastune`)
+
+Follow [DeepRefine/README](https://github.com/HKUST-KnowComp/DeepRefine) **Environment** to create `atlastune` and install the main repo:
+
+```bash
+conda activate atlastune
+cd /path/to/DeepRefine
+pip install -e .
+```
+
+### 2. Install this CLI
+
+**From PyPI:**
+
+```bash
+conda activate atlastune
+pip install deeprefine-cli
+```
+
+**From source (development):**
+
+```bash
+pip install -e /path/to/DeepRefine-Skill
+```
+
+Verify:
+
+```bash
+deeprefine --help
+```
+
+> `deeprefine refine` still requires a local [DeepRefine](https://github.com/HKUST-KnowComp/DeepRefine) checkout (`autorefiner`, `atlas_rag`) and running vLLM services — see below.
+
+### 3. Point to DeepRefine (if not cloned as sibling)
+
+Only needed when `DeepRefine` is **not** at `../DeepRefine` relative to this repo and not found by walking up from your cwd:
+
+```bash
+export DEEPREFINE_REPO=/path/to/DeepRefine
+```
+
+Add to `~/.bashrc` if you use a fixed path.
+
+### 4. Start vLLM (before `deeprefine refine`)
+
+From the **DeepRefine** repo:
+
+```bash
+conda activate atlastune
+bash /path/to/DeepRefine/scripts/vllm_serve/qwen3-0.6b-emb.sh
+bash /path/to/DeepRefine/scripts/vllm_serve/qwen3-8b-vllm-reafiner.sh
+```
+
+| Variable | Default |
+|----------|---------|
+| `DEEPREFINE_LLM_URL` | `http://127.0.0.1:8134/v1` |
+| `DEEPREFINE_EMBED_URL` | `http://127.0.0.1:8128/v1` |
+| `DEEPREFINE_MODEL` | `HaoyuHuang2/DeepRefine-v1-8B` |
+| `DEEPREFINE_EMBED_MODEL` | `Qwen/Qwen3-Embedding-0.6B` |
+
+### 5. Install Cursor skill (KB project root)
+
+```bash
+cd /path/to/your-kb-project    # must contain or will contain graphify-out/
+deeprefine cursor install
+```
+
+Same pattern as `graphify cursor install`. Use `--user` for all projects.
+
+---
+
+## Workflow with graphify
+
+**One-time**
+
+```bash
+pip install graphifyy
+graphify cursor install          # in KB project
+pip install -e /path/to/DeepRefine-Skill   # in atlastune
+deeprefine cursor install        # in KB project
+```
+
+**Per session** (KB project root)
+
+1. `/graphify .` or `graphify .` → `graphify-out/graph.json`
+2. `graphify query "..."` or `/graphify query "..."`
+3. `deeprefine history add --query "..."`
+4. `/deeprefine` or `deeprefine refine`
+5. Optional: `graphify query "..."` again to verify
+
+```
+project files ──graphify──► graph.json
+                                 │
+                       graphify query
+                                 │
+                 deeprefine history add
+                                 │
+                 deeprefine refine
+                                 │
+                       graphify query
+```
+
+---
+
+## Command cheat sheet (KB project root)
+
+```bash
+deeprefine history add --query "..."
+deeprefine history list --pending
+deeprefine refine
+deeprefine refine --query "..."
+deeprefine index --rebuild
+```
+
+---
+
+## Where to run commands
+
+| Command | Directory |
+|---------|-----------|
+| `pip install -e .../DeepRefine-Skill` | Any |
+| `pip install -e .../DeepRefine` | DeepRefine repo |
+| `graphify` / `deeprefine cursor install` | **KB project root** |
+| `deeprefine refine` | **KB project root** |
+| vLLM serve scripts | **DeepRefine repo** |
+
+Agent instructions: [SKILL.md](./SKILL.md).
+
+---
+
+## Publish to PyPI (maintainers)
+
+```bash
+cd /path/to/DeepRefine-Skill
+python -m pip install --upgrade build twine
+python -m build
+twine check dist/*
+twine upload dist/*    # needs PyPI token: TWINE_USERNAME=__token__ TWINE_PASSWORD=pypi-...
+```
+
+Test install from TestPyPI first (optional):
+
+```bash
+twine upload --repository testpypi dist/*
+pip install -i https://test.pypi.org/simple/ deeprefine-cli
+```

deeprefine_cli-0.1.3.dist-info/RECORD

@@ -0,0 +1,14 @@
+deeprefine_cli-0.1.3.dist-info/licenses/LICENSE,sha256=K5NjmGZ2iGTxhVTWH1wiFs9uUL6S5RMyj8yoRQlTIbc,1071
+deeprefine_skill/SKILL.md,sha256=BiMmg-xIq2GLkqywmJBJ6-LION5CdClqQjqA6o9vGBQ,1489
+deeprefine_skill/__init__.py,sha256=XIDBdRneslC9k1-ybuVHnpS07dgsTBj8VZAiZ_leAbs,94
+deeprefine_skill/adapter_graphify.py,sha256=I9yudnpUdH_BBs4L090SBYTSY49fnfh0opghryd9-qo,6838
+deeprefine_skill/cli.py,sha256=Ulfq-oyWQoPyiJ2qPr-oZgXlhX9KQs5ByvTfBxkTdlc,6561
+deeprefine_skill/history.py,sha256=ufMUanwfEcJIf22ZkdrAlXj24IW9EIJFZfF2Z0MBBHc,2321
+deeprefine_skill/installers.py,sha256=6PGoFg0z7kdxXEevccBcgrV44CPZ5d1DMNm2BmMi9a0,1455
+deeprefine_skill/paths.py,sha256=jF3QUp53wcwaVBY_4K9Ly1QE622okgrLY80LsGXD5JM,3019
+deeprefine_skill/refine_runner.py,sha256=Pz2z2jw9FSwXPdgtbHkWqUaE6kLeA6sUTiFEvPEAPJg,6435
+deeprefine_cli-0.1.3.dist-info/METADATA,sha256=s_St2jZqmIM2awlDO7imk9I9hWUJ96tMWXldts2icpw,6194
+deeprefine_cli-0.1.3.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+deeprefine_cli-0.1.3.dist-info/entry_points.txt,sha256=zLtVs6vRaVIzaKUGppMuE9cfJXOjksBAbYmRmhKllFE,57
+deeprefine_cli-0.1.3.dist-info/top_level.txt,sha256=owOn5r-Gn0sQbil4xRMfZKOXybukl_jxjPjUu3_IzeM,17
+deeprefine_cli-0.1.3.dist-info/RECORD,,

deeprefine_cli-0.1.3.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

deeprefine_cli-0.1.3.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+deeprefine = deeprefine_skill.cli:main

deeprefine_cli-0.1.3.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 HKUST-KnowComp
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

deeprefine_cli-0.1.3.dist-info/top_level.txt

@@ -0,0 +1 @@
+deeprefine_skill

deeprefine_skill/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: deeprefine
+description: >-
+  Refines a graphify knowledge graph (graphify-out/graph.json) using DeepRefine
+  Reafiner based on session query history. Use when the user runs /deeprefine,
+  asks to improve the graphify KB after Q&A, or wants to patch graph.json from
+  failed retrieval queries.
+disable-model-invocation: true
+---
+
+# DeepRefine (graphify)
+
+Refine a **[graphify](https://github.com/safishamsi/graphify)** `graph.json` with the DeepRefine agent loop.
+
+## Setup
+
+**DeepRefine-Skill** (CLI + this file) is separate from the **DeepRefine** model repo.
+
+```bash
+# 1) DeepRefine env (atlastune) + main package
+conda activate atlastune
+cd /path/to/DeepRefine && pip install -e .
+
+# 2) This skill CLI
+pip install -e /path/to/DeepRefine-Skill
+
+# 3) Optional if DeepRefine is not ../DeepRefine
+export DEEPREFINE_REPO=/path/to/DeepRefine
+
+# 4) Cursor skill in KB project root
+cd /path/to/your-kb-project
+deeprefine cursor install
+```
+
+Start vLLM from DeepRefine: embedding `8128`, refine model `8134` (see DeepRefine `scripts/vllm_serve/`).
+
+## `/deeprefine`
+
+From the **KB project root** (with `graphify-out/graph.json`):
+
+```bash
+deeprefine history add --query "..."   # after graph Q&A
+deeprefine refine                      # all pending queries
+```
+
+Do not hand-edit `graph.json` for refinement.
+
+## Paths
+
+- History: `graphify-out/.deeprefine/history.jsonl`
+- Log: `graphify-out/.deeprefine/refinement_results_*.jsonl`
+- Backup: `graphify-out/.deeprefine/graph.json.bak`

deeprefine_skill/__init__.py

@@ -0,0 +1,3 @@
+"""DeepRefine agent skill: refine graphify graph.json via Reafiner."""
+
+__version__ = "0.1.3"

deeprefine_skill/adapter_graphify.py

@@ -0,0 +1,207 @@
+from __future__ import annotations
+
+import copy
+import json
+import pickle
+import shutil
+from pathlib import Path
+from typing import Any
+
+import networkx as nx
+from networkx.readwrite import json_graph
+
+from atlas_rag.vectorstore.create_graph_index import (
+    build_faiss_index_flat,
+    compute_graph_embeddings,
+    compute_text_embeddings,
+)
+from atlas_rag.vectorstore.embedding_model import BaseEmbeddingModel
+
+
+def load_graphify_json(path: Path) -> tuple[dict[str, Any], nx.DiGraph]:
+    raw = json.loads(path.read_text(encoding="utf-8"))
+    links_key = "links" if "links" in raw else "edges"
+    # node_link_graph returns Graph/ DiGraph depending on attrs; force DiGraph
+    base = json_graph.node_link_graph(raw, edges=links_key, directed=True)
+    if not isinstance(base, nx.DiGraph):
+        kg = nx.DiGraph(base)
+    else:
+        kg = base
+
+    id_to_meta = {n.get("id"): n for n in raw.get("nodes", []) if n.get("id")}
+
+    for nid in list(kg.nodes):
+        meta = id_to_meta.get(nid, {})
+        label = meta.get("label") or meta.get("id") or str(nid)
+        kg.nodes[nid]["id"] = label
+        kg.nodes[nid]["type"] = kg.nodes[nid].get("type") or "entity"
+        kg.nodes[nid]["file_id"] = meta.get("source_file")
+        if "community" in meta:
+            kg.nodes[nid]["community"] = meta["community"]
+
+    for u, v, data in kg.edges(data=True):
+        if "relation" not in data:
+            data["relation"] = data.pop("label", "related_to")
+        data.setdefault("type", "Relation")
+        conf = data.get("confidence", "INFERRED")
+        data.setdefault("confidence", conf)
+
+    return raw, kg
+
+
+def _entity_nodes(kg: nx.DiGraph) -> list[str]:
+    return [
+        n
+        for n in kg.nodes
+        if kg.nodes[n].get("type") != "passage"
+    ]
+
+
+def build_reafiner_data(
+    kg: nx.DiGraph,
+    sentence_encoder: BaseEmbeddingModel,
+    *,
+    normalize_embeddings: bool = False,
+    batch_size: int = 64,
+) -> dict[str, Any]:
+    node_list = _entity_nodes(kg)
+    node_set = set(node_list)
+    edge_list = [(u, v) for u, v in kg.edges if u in node_set and v in node_set]
+    node_list_string = [kg.nodes[n]["id"] for n in node_list]
+    edge_list_string = [
+        f"{kg.nodes[u]['id']} {kg.edges[u, v]['relation']} {kg.nodes[v]['id']}"
+        for u, v in edge_list
+    ]
+
+    node_embeddings, edge_embeddings = compute_graph_embeddings(
+        node_list_string,
+        edge_list_string,
+        sentence_encoder,
+        batch_size=batch_size,
+        normalize_embeddings=normalize_embeddings,
+    )
+
+    node_faiss_index = build_faiss_index_flat(node_embeddings)
+    edge_faiss_index = build_faiss_index_flat(edge_embeddings)
+
+    passage_id = "__deeprefine_passage__"
+    if passage_id not in kg.nodes:
+        kg.add_node(
+            passage_id,
+            id="graphify knowledge graph",
+            type="passage",
+            file_id=None,
+        )
+    text_dict = {passage_id: "graphify knowledge graph"}
+    text_embeddings = compute_text_embeddings(
+        list(text_dict.values()),
+        sentence_encoder,
+        batch_size=8,
+        normalize_embeddings=normalize_embeddings,
+    )
+    text_faiss_index = build_faiss_index_flat(text_embeddings)
+
+    n_nodes = len(node_list)
+    n_edges = len(edge_list)
+    return {
+        "KG": kg,
+        "node_faiss_index": node_faiss_index,
+        "edge_faiss_index": edge_faiss_index,
+        "text_faiss_index": text_faiss_index,
+        "node_embeddings": node_embeddings,
+        "edge_embeddings": edge_embeddings,
+        "text_embeddings": text_embeddings,
+        "node_list": node_list,
+        "edge_list": edge_list,
+        "text_dict": text_dict,
+        "edge_faiss_id_to_list_idx": {i: i for i in range(n_edges)},
+        "node_faiss_id_to_list_idx": {i: i for i in range(n_nodes)},
+        "text_faiss_id_to_list_idx": {0: 0},
+    }
+
+
+def load_or_build_data(
+    graph_path: Path,
+    cache_pkl: Path,
+    sentence_encoder: BaseEmbeddingModel,
+    *,
+    rebuild: bool = False,
+) -> tuple[dict[str, Any], dict[str, Any]]:
+    graph_mtime = graph_path.stat().st_mtime
+    if (
+        not rebuild
+        and cache_pkl.is_file()
+        and cache_pkl.stat().st_mtime >= graph_mtime
+    ):
+        with cache_pkl.open("rb") as f:
+            bundle = pickle.load(f)
+        raw = bundle["graphify_raw"]
+        data = bundle["reafiner_data"]
+        return raw, data
+
+    raw, kg = load_graphify_json(graph_path)
+    data = build_reafiner_data(kg, sentence_encoder)
+    cache_pkl.parent.mkdir(parents=True, exist_ok=True)
+    with cache_pkl.open("wb") as f:
+        pickle.dump({"graphify_raw": raw, "reafiner_data": data}, f)
+    return raw, data
+
+
+def sync_kg_to_graphify(raw: dict[str, Any], kg: nx.DiGraph) -> dict[str, Any]:
+    """Merge refined nx graph back into graphify node-link JSON."""
+    out = copy.deepcopy(raw)
+    links_key = "links" if "links" in out else "edges"
+
+    old_nodes = {n["id"]: n for n in out.get("nodes", [])}
+    new_nodes: list[dict[str, Any]] = []
+    for nid in sorted(kg.nodes, key=str):
+        if kg.nodes[nid].get("type") == "passage":
+            continue
+        base = copy.deepcopy(old_nodes.get(nid, {}))
+        base["id"] = nid
+        base["label"] = kg.nodes[nid].get("id", nid)
+        if kg.nodes[nid].get("file_id"):
+            base["source_file"] = kg.nodes[nid]["file_id"]
+        if "community" in kg.nodes[nid]:
+            base["community"] = kg.nodes[nid]["community"]
+        new_nodes.append(base)
+
+    new_links: list[dict[str, Any]] = []
+    for u, v, edata in kg.edges(data=True):
+        if kg.nodes[u].get("type") == "passage" or kg.nodes[v].get("type") == "passage":
+            continue
+        link: dict[str, Any] = {
+            "source": u,
+            "target": v,
+            "relation": edata.get("relation", "related_to"),
+            "confidence": edata.get("confidence", "INFERRED"),
+        }
+        if link["confidence"] == "INFERRED":
+            link["confidence_score"] = 0.7
+        elif link["confidence"] == "EXTRACTED":
+            link["confidence_score"] = 1.0
+        else:
+            link["confidence_score"] = 0.4
+        new_links.append(link)
+
+    out["nodes"] = new_nodes
+    out[links_key] = new_links
+    return out
+
+
+def save_graphify_json(
+    path: Path,
+    graph_data: dict[str, Any],
+    *,
+    backup_path: Path | None = None,
+) -> None:
+    if backup_path and path.is_file():
+        backup_path.parent.mkdir(parents=True, exist_ok=True)
+        shutil.copy2(path, backup_path)
+    path.write_text(json.dumps(graph_data, indent=2, ensure_ascii=False), encoding="utf-8")
+
+
+def save_bundle(cache_pkl: Path, raw: dict[str, Any], data: dict[str, Any]) -> None:
+    cache_pkl.parent.mkdir(parents=True, exist_ok=True)
+    with cache_pkl.open("wb") as f:
+        pickle.dump({"graphify_raw": raw, "reafiner_data": data}, f)

deeprefine_skill/cli.py

@@ -0,0 +1,192 @@
+"""DeepRefine CLI: `deeprefine cursor install` (graphify-style)."""
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from deeprefine_skill.history import append_history, iter_history, pending_queries
+from deeprefine_skill.installers import install_cursor_skill, uninstall_cursor_skill
+from deeprefine_skill.paths import (
+    env_defaults,
+    find_deeprefine_repo,
+    find_project_root,
+    graphify_paths,
+    setup_import_paths,
+)
+
+
+def _setup_repo_imports() -> None:
+    setup_import_paths(find_deeprefine_repo())
+
+
+def cmd_cursor_install(args: argparse.Namespace) -> int:
+    dest = install_cursor_skill(project=args.project)
+    scope = "project" if args.project else "user"
+    print(f"Installed DeepRefine Cursor skill ({scope}) → {dest}")
+    if args.project:
+        print("Open this folder in Cursor, then use /deeprefine in chat.")
+    return 0
+
+
+def cmd_cursor_uninstall(args: argparse.Namespace) -> int:
+    removed = uninstall_cursor_skill(project=args.project)
+    if removed:
+        scope = "project" if args.project else "user"
+        print(f"Removed DeepRefine Cursor skill ({scope}).")
+    else:
+        print("Skill not installed at the selected scope.")
+    return 0
+
+
+def cmd_install(args: argparse.Namespace) -> int:
+    """Alias for `deeprefine cursor install` (graphify-compatible naming)."""
+    return cmd_cursor_install(args)
+
+
+def cmd_history_add(args: argparse.Namespace) -> int:
+    project = find_project_root()
+    paths = graphify_paths(project)
+    entry = append_history(
+        paths["history"], args.query, source=args.source, refined=False
+    )
+    print(f"Recorded: {entry['id']} → {paths['history']}")
+    return 0
+
+
+def cmd_history_list(args: argparse.Namespace) -> int:
+    project = find_project_root()
+    paths = graphify_paths(project)
+    rows = (
+        pending_queries(paths["history"])
+        if args.pending
+        else list(iter_history(paths["history"]))
+    )
+    if not rows and args.pending:
+        print("No pending queries.")
+        return 0
+    for row in rows:
+        flag = "refined" if row.get("refined") else "pending"
+        print(f"[{flag}] {row.get('id', '?')}: {row.get('query', '')}")
+    return 0
+
+
+def cmd_index(args: argparse.Namespace) -> int:
+    _setup_repo_imports()
+    from deeprefine_skill.adapter_graphify import load_or_build_data
+    from deeprefine_skill.refine_runner import make_clients
+
+    project = find_project_root()
+    paths = graphify_paths(project)
+    cfg = env_defaults()
+    llm, encoder = make_clients(cfg)
+    del llm
+    load_or_build_data(
+        paths["graph_json"],
+        paths["reafiner_pkl"],
+        encoder,
+        rebuild=True,
+    )
+    print(f"Index cache: {paths['reafiner_pkl']}")
+    return 0
+
+
+def cmd_refine(args: argparse.Namespace) -> int:
+    _setup_repo_imports()
+    from deeprefine_skill.refine_runner import refine_from_history
+
+    project = find_project_root(Path(args.project_root) if args.project_root else None)
+    paths = graphify_paths(project)
+    cfg = env_defaults()
+    result = refine_from_history(
+        paths,
+        cfg,
+        query=args.query,
+        rebuild_index=args.rebuild_index,
+    )
+    print("\n--- DeepRefine summary ---")
+    print(f"Queries processed: {result['queries_processed']}")
+    print(f"Graph: {result['graph_path']} ({result['nodes']} nodes, {result['edges']} edges)")
+    print(f"Log: {result['log_path']}")
+    return 0
+
+
+def _add_project_flag(parser: argparse.ArgumentParser) -> None:
+    group = parser.add_mutually_exclusive_group()
+    group.add_argument(
+        "--project",
+        action="store_true",
+        default=None,
+        help="Install to .cursor/skills in the current directory (default for cursor install)",
+    )
+    group.add_argument(
+        "--user",
+        action="store_true",
+        help="Install to ~/.cursor/skills (all projects)",
+    )
+
+
+def _resolve_project(args: argparse.Namespace, *, default_project: bool) -> None:
+    if getattr(args, "user", False):
+        args.project = False
+    elif getattr(args, "project", None) is True:
+        args.project = True
+    else:
+        args.project = default_project
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="deeprefine",
+        description="DeepRefine: refine graphify-out/graph.json using query history",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    # deeprefine cursor install | uninstall
+    p_cursor = sub.add_parser("cursor", help="Cursor IDE integration")
+    cursor_sub = p_cursor.add_subparsers(dest="cursor_cmd", required=True)
+
+    p_ci = cursor_sub.add_parser("install", help="Install /deeprefine skill for Cursor")
+    _add_project_flag(p_ci)
+    p_ci.set_defaults(func=cmd_cursor_install, _default_project=True)
+
+    p_cu = cursor_sub.add_parser("uninstall", help="Remove Cursor skill")
+    _add_project_flag(p_cu)
+    p_cu.set_defaults(func=cmd_cursor_uninstall, _default_project=True)
+
+    # deeprefine install (alias)
+    p_install = sub.add_parser(
+        "install",
+        help="Install Cursor skill (alias: deeprefine cursor install)",
+    )
+    _add_project_flag(p_install)
+    p_install.set_defaults(func=cmd_install, _default_project=True)
+
+    p_hist = sub.add_parser("history", help="Manage query history")
+    hsub = p_hist.add_subparsers(dest="history_cmd", required=True)
+    p_add = hsub.add_parser("add", help="Append a query to history")
+    p_add.add_argument("--query", required=True)
+    p_add.add_argument("--source", default="user")
+    p_add.set_defaults(func=cmd_history_add)
+    p_list = hsub.add_parser("list", help="List history entries")
+    p_list.add_argument("--pending", action="store_true")
+    p_list.set_defaults(func=cmd_history_list)
+
+    p_index = sub.add_parser("index", help="Rebuild FAISS cache from graph.json")
+    p_index.add_argument("--rebuild", action="store_true", default=True)
+    p_index.set_defaults(func=cmd_index)
+
+    p_refine = sub.add_parser("refine", help="Run refinement on pending or given query")
+    p_refine.add_argument("--query", default=None, help="Single query (also recorded)")
+    p_refine.add_argument("--project-root", default=None)
+    p_refine.add_argument("--rebuild-index", action="store_true")
+    p_refine.set_defaults(func=cmd_refine)
+
+    args = parser.parse_args(argv)
+    if hasattr(args, "_default_project"):
+        _resolve_project(args, default_project=args._default_project)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

deeprefine_skill/history.py

@@ -0,0 +1,81 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import time
+from pathlib import Path
+from typing import Any, Iterator
+
+
+def query_id(query: str, entry_id: str | None = None) -> str:
+    """Stable id for a history row (matches append_history id field)."""
+    if entry_id:
+        return entry_id
+    return hashlib.sha256(query.strip().encode("utf-8")).hexdigest()[:16]
+
+
+def _line_id(query: str) -> str:
+    return query_id(query)
+
+
+def append_history(
+    path: Path,
+    query: str,
+    *,
+    source: str = "user",
+    refined: bool = False,
+) -> dict[str, Any]:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    entry = {
+        "id": _line_id(query),
+        "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "query": query.strip(),
+        "source": source,
+        "refined": refined,
+    }
+    with path.open("a", encoding="utf-8") as f:
+        f.write(json.dumps(entry, ensure_ascii=False) + "\n")
+    return entry
+
+
+def iter_history(path: Path) -> Iterator[dict[str, Any]]:
+    if not path.is_file():
+        return
+    with path.open(encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            yield json.loads(line)
+
+
+def pending_queries(path: Path) -> list[dict[str, Any]]:
+    seen: set[str] = set()
+    pending: list[dict[str, Any]] = []
+    for row in iter_history(path):
+        q = row.get("query", "").strip()
+        if not q or row.get("refined") is True:
+            continue
+        qid = row.get("id") or _line_id(q)
+        if qid in seen:
+            continue
+        seen.add(qid)
+        pending.append(row)
+    return pending
+
+
+def mark_refined(path: Path, query_ids: set[str]) -> None:
+    if not path.is_file() or not query_ids:
+        return
+    rows: list[dict[str, Any]] = list(iter_history(path))
+    changed = False
+    for row in rows:
+        qid = row.get("id") or _line_id(row.get("query", ""))
+        if qid in query_ids and not row.get("refined"):
+            row["refined"] = True
+            row["refined_ts"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+            changed = True
+    if changed:
+        with path.open("w", encoding="utf-8") as f:
+            for row in rows:
+                f.write(json.dumps(row, ensure_ascii=False) + "\n")

deeprefine_skill/installers.py

@@ -0,0 +1,47 @@
+from __future__ import annotations
+
+import shutil
+from pathlib import Path
+
+_SKILL_MD_NAME = "SKILL.md"
+
+
+def skill_md_path() -> Path:
+    """SKILL.md bundled in the wheel, or repo root when doing editable install."""
+    bundled = Path(__file__).resolve().parent / _SKILL_MD_NAME
+    if bundled.is_file():
+        return bundled
+    repo_root = Path(__file__).resolve().parents[1]
+    fallback = repo_root / _SKILL_MD_NAME
+    if fallback.is_file():
+        return fallback
+    raise FileNotFoundError(
+        f"Missing {_SKILL_MD_NAME} (expected next to deeprefine_skill/ or repo root)."
+    )
+
+
+def install_cursor_skill(*, project: bool) -> Path:
+    src = skill_md_path()
+    if project:
+        dest_dir = Path.cwd() / ".cursor" / "skills" / "deeprefine"
+    else:
+        dest_dir = Path.home() / ".cursor" / "skills" / "deeprefine"
+    dest_dir.mkdir(parents=True, exist_ok=True)
+    shutil.copy2(src, dest_dir / _SKILL_MD_NAME)
+    return dest_dir / _SKILL_MD_NAME
+
+
+def uninstall_cursor_skill(*, project: bool) -> bool:
+    if project:
+        dest = Path.cwd() / ".cursor" / "skills" / "deeprefine" / _SKILL_MD_NAME
+    else:
+        dest = Path.home() / ".cursor" / "skills" / "deeprefine" / _SKILL_MD_NAME
+    if dest.is_file():
+        dest.unlink()
+        for parent in [dest.parent, dest.parent.parent]:
+            try:
+                parent.rmdir()
+            except OSError:
+                pass
+        return True
+    return False

deeprefine_skill/paths.py

@@ -0,0 +1,92 @@
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+
+def find_skill_root() -> Path:
+    """Root of this repository (DeepRefine-Skill)."""
+    return Path(__file__).resolve().parents[1]
+
+
+def find_deeprefine_repo(start: Path | None = None) -> Path:
+    """
+    DeepRefine source repo (autorefiner/ + AutoSchemaKG/).
+
+    Resolution order:
+    1. DEEPREFINE_REPO environment variable
+    2. Walk up from cwd for autorefiner/ + AutoSchemaKG/
+    3. Sibling ../DeepRefine next to this skill repo
+    """
+    env = os.environ.get("DEEPREFINE_REPO", "").strip()
+    if env:
+        root = Path(env).expanduser().resolve()
+        if (root / "autorefiner").is_dir() and (root / "AutoSchemaKG").is_dir():
+            return root
+        raise FileNotFoundError(
+            f"DEEPREFINE_REPO={env} is not a valid DeepRefine checkout "
+            "(expected autorefiner/ and AutoSchemaKG/)."
+        )
+
+    here = (start or Path.cwd()).resolve()
+    for parent in [here, *here.parents]:
+        if (parent / "autorefiner").is_dir() and (parent / "AutoSchemaKG").is_dir():
+            return parent
+
+    sibling = find_skill_root().parent / "DeepRefine"
+    if (sibling / "autorefiner").is_dir() and (sibling / "AutoSchemaKG").is_dir():
+        return sibling.resolve()
+
+    raise FileNotFoundError(
+        "Could not locate the DeepRefine repository (need autorefiner/ and AutoSchemaKG/).\n"
+        "Clone DeepRefine alongside this repo, or set:\n"
+        "  export DEEPREFINE_REPO=/path/to/DeepRefine"
+    )
+
+
+def find_project_root(start: Path | None = None) -> Path:
+    """User KB project root containing graphify-out/graph.json."""
+    here = (start or Path.cwd()).resolve()
+    for parent in [here, *here.parents]:
+        if (parent / "graphify-out" / "graph.json").is_file():
+            return parent
+    return here
+
+
+def graphify_paths(project_root: Path) -> dict[str, Path]:
+    out = project_root / "graphify-out"
+    deep = out / ".deeprefine"
+    return {
+        "graphify_out": out,
+        "graph_json": out / "graph.json",
+        "history": deep / "history.jsonl",
+        "cache_dir": deep / "cache",
+        "reafiner_pkl": deep / "cache" / "reafiner_data.pkl",
+        "graph_backup": deep / "graph.json.bak",
+    }
+
+
+def setup_import_paths(deeprefine_repo: Path) -> None:
+    import sys
+
+    for p in (deeprefine_repo / "AutoSchemaKG", deeprefine_repo):
+        s = str(p)
+        if s not in sys.path:
+            sys.path.insert(0, s)
+
+
+def env_defaults() -> dict[str, str]:
+    return {
+        "DEEPREFINE_LLM_URL": os.environ.get(
+            "DEEPREFINE_LLM_URL", "http://127.0.0.1:8134/v1"
+        ),
+        "DEEPREFINE_EMBED_URL": os.environ.get(
+            "DEEPREFINE_EMBED_URL", "http://127.0.0.1:8128/v1"
+        ),
+        "DEEPREFINE_MODEL": os.environ.get(
+            "DEEPREFINE_MODEL", "HaoyuHuang2/DeepRefine-v1-8B"
+        ),
+        "DEEPREFINE_EMBED_MODEL": os.environ.get(
+            "DEEPREFINE_EMBED_MODEL", "Qwen/Qwen3-Embedding-0.6B"
+        ),
+    }

deeprefine_skill/refine_runner.py

@@ -0,0 +1,197 @@
+from __future__ import annotations
+
+import json
+import time
+from pathlib import Path
+from typing import Any
+
+from openai import OpenAI
+
+from atlas_rag.llm_generator import GenerationConfig, LLMGenerator
+from atlas_rag.vectorstore.embedding_model import Qwen3Emb
+from autorefiner.src.reafiner import Reafiner, RetrievalStepResult
+
+from .adapter_graphify import (
+    load_or_build_data,
+    save_bundle,
+    save_graphify_json,
+    sync_kg_to_graphify,
+)
+from .history import append_history, mark_refined, query_id
+
+
+def refinement_to_jsonable(
+    sample: dict[str, Any],
+    final_answer: Any,
+    refinement_result: Any,
+) -> dict[str, Any]:
+    base = {"sample": sample, "final_answer": final_answer}
+    if refinement_result is None:
+        base["refinement_result"] = None
+        return base
+
+    hist = []
+    for step in refinement_result.interaction_history:
+        if isinstance(step, RetrievalStepResult):
+            hist.append(
+                {
+                    "num_hops": step.num_hops,
+                    "base_top_k": step.base_top_k,
+                    "query": step.query,
+                    "retrieved_subgraph": step.retrieved_subgraph,
+                    "raw_response": step.raw_response,
+                    "answerable": step.answerable,
+                    "answer": step.answer,
+                }
+            )
+        else:
+            hist.append(str(step))
+
+    base["refinement_result"] = {
+        "query": refinement_result.query,
+        "history_horizon_size": refinement_result.history_horizon_size,
+        "interaction_history": hist,
+        "error_abduction_reason": refinement_result.error_abduction_reason,
+        "original_subgraph": refinement_result.original_subgraph,
+        "refined_subgraph": refinement_result.refined_subgraph,
+        "refinement_action_raw": refinement_result.refinement_action_raw,
+        "refinement_action_count": len(refinement_result.refinement_action_list),
+    }
+    return base
+
+
+def make_clients(cfg: dict[str, str]) -> tuple[LLMGenerator, Qwen3Emb]:
+    llm_client = OpenAI(base_url=cfg["DEEPREFINE_LLM_URL"], api_key="EMPTY")
+    embed_client = OpenAI(base_url=cfg["DEEPREFINE_EMBED_URL"], api_key="EMPTY")
+    llm = LLMGenerator(
+        client=llm_client,
+        model_name=cfg["DEEPREFINE_MODEL"],
+        default_config=GenerationConfig(chat_template_kwargs={"enable_thinking": False}),
+    )
+    encoder = Qwen3Emb(embed_client)
+    return llm, encoder
+
+
+def run_refine(
+    *,
+    graph_path: Path,
+    cache_pkl: Path,
+    backup_path: Path,
+    history_path: Path,
+    log_dir: Path,
+    cfg: dict[str, str],
+    queries: list[dict[str, Any]],
+    rebuild_index: bool = False,
+    base_top_k: int = 5,
+    max_hops: int = 4,
+) -> dict[str, Any]:
+    if not graph_path.is_file():
+        raise FileNotFoundError(f"graphify graph not found: {graph_path}")
+
+    llm, encoder = make_clients(cfg)
+    raw, data = load_or_build_data(
+        graph_path, cache_pkl, encoder, rebuild=rebuild_index
+    )
+
+    reafiner = Reafiner(
+        data=data,
+        sentence_encoder=encoder,
+        llm_generator=llm,
+        base_top_k=base_top_k,
+        max_hops=max_hops,
+        max_triple_num=20,
+        max_triple_num_by_step=[5, 10, 15, 20],
+        history_horizon_size=4,
+        if_gen_answer=False,
+    )
+
+    log_dir.mkdir(parents=True, exist_ok=True)
+    log_path = log_dir / f"refinement_results_{int(time.time())}.jsonl"
+    refined_ids: set[str] = set()
+    summary_rows: list[dict[str, Any]] = []
+    completed = 0
+
+    def _persist() -> None:
+        if completed == 0:
+            return
+        data["KG"] = reafiner.kg
+        nonlocal raw
+        raw = sync_kg_to_graphify(raw, reafiner.kg)
+        save_graphify_json(graph_path, raw, backup_path=backup_path)
+        save_bundle(cache_pkl, raw, data)
+        mark_refined(history_path, refined_ids)
+
+    try:
+        with log_path.open("w", encoding="utf-8") as log_f:
+            for sample in queries:
+                query = sample["query"]
+                qid = query_id(query, sample.get("id"))
+                print(f"\n=== [{qid}] {query}")
+                final_answer, _, refinement_result = reafiner.refine(query=query)
+                record = refinement_to_jsonable(sample, final_answer, refinement_result)
+                log_f.write(json.dumps(record, ensure_ascii=False) + "\n")
+                log_f.flush()
+                n_steps = (
+                    len(refinement_result.interaction_history)
+                    if refinement_result is not None
+                    else 0
+                )
+                rr = record.get("refinement_result") or {}
+                summary_rows.append(
+                    {
+                        "id": qid,
+                        "query": query,
+                        "steps": n_steps,
+                        "action_count": rr.get("refinement_action_count", 0),
+                    }
+                )
+                refined_ids.add(qid)
+                completed += 1
+                print(
+                    f"  steps={n_steps}, nodes={reafiner.kg.number_of_nodes()}, "
+                    f"edges={reafiner.kg.number_of_edges()}"
+                )
+    finally:
+        _persist()
+
+    return {
+        "log_path": str(log_path),
+        "graph_path": str(graph_path),
+        "nodes": reafiner.kg.number_of_nodes(),
+        "edges": reafiner.kg.number_of_edges(),
+        "queries_processed": len(queries),
+        "summary": summary_rows,
+    }
+
+
+def refine_from_history(
+    paths: dict[str, Path],
+    cfg: dict[str, str],
+    *,
+    query: str | None = None,
+    rebuild_index: bool = False,
+) -> dict[str, Any]:
+    if query:
+        entry = append_history(paths["history"], query, source="deeprefine")
+        queries = [entry]
+    else:
+        from .history import pending_queries
+
+        queries = pending_queries(paths["history"])
+        if not queries:
+            raise SystemExit(
+                "No pending queries in history. Use:\n"
+                "  deeprefine history add --query '...'\n"
+                "  deeprefine refine --query '...'"
+            )
+
+    return run_refine(
+        graph_path=paths["graph_json"],
+        cache_pkl=paths["reafiner_pkl"],
+        backup_path=paths["graph_backup"],
+        history_path=paths["history"],
+        log_dir=paths["graphify_out"] / ".deeprefine",
+        cfg=cfg,
+        queries=queries,
+        rebuild_index=rebuild_index,
+    )