gsas2-query 0.3.0__py3-none-any.whl

gsas2_query-0.3.0.dist-info/METADATA

@@ -0,0 +1,333 @@
+Metadata-Version: 2.5
+Name: gsas2-query
+Version: 0.3.0
+Summary: Query-GSAS-II Documentation Assistant — semantic search and AI answers over GSAS-II tutorials and help pages
+Project-URL: Repository, https://github.com/AdvancedPhotonSource/Query-GSAS-II
+Author-email: Pawan Tripathi <ptripathi@anl.gov>, Brian Toby <toby@anl.gov>
+License: MIT
+Keywords: crystallography,documentation,gsas,rag,xrd
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Chemistry
+Requires-Python: >=3.10
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: chromadb>=0.5.0
+Requires-Dist: fastapi>=0.110.0
+Requires-Dist: fastembed>=0.3.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: lxml>=5.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pypdf>=4.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: requests>=2.31.0
+Requires-Dist: uvicorn[standard]>=0.29.0
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.25.0; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: gui
+Requires-Dist: wxpython>=4.2.0; extra == 'gui'
+Provides-Extra: llama-cpp
+Requires-Dist: llama-cpp-python>=0.2.0; extra == 'llama-cpp'
+Description-Content-Type: text/markdown
+
+# gsas2-query — GSAS-II Documentation Assistant
+
+Semantic search + AI answers over the full GSAS-II documentation set:
+**129 HTML pages** (home, help, and all 62 tutorials) plus the
+**Programmer's Guide** and **Powder Crystallography** book PDFs.
+
+Answers include **inline citations** — every `[N]` in the response is a
+clickable link to the exact documentation section that supported that sentence.
+
+All embedding and retrieval runs on your machine — no data is sent externally
+unless you explicitly choose the Anthropic API backend.
+
+---
+
+## Installation
+
+### pip
+
+```bash
+pip install gsas2-query
+```
+
+Or install directly from source:
+
+```bash
+pip install git+https://github.com/AdvancedPhotonSource/Query-GSAS-II.git
+```
+
+### conda
+
+```bash
+conda install -c conda-forge gsas2-query
+```
+
+### Into an existing GSAS-II environment
+
+```bash
+# Activate the GSAS-II conda environment first, then:
+pip install gsas2-query
+```
+
+### Development / editable install
+
+```bash
+git clone https://github.com/AdvancedPhotonSource/Query-GSAS-II.git
+cd Query-GSAS-II
+pip install -e ".[dev]"
+```
+
+---
+
+## Ollama — free local LLM (recommended when llama.cpp is not installed)
+
+```bash
+brew install ollama          # macOS; see https://ollama.com for other platforms
+ollama serve &               # start the local server
+ollama pull llama3           # ~5 GB one-time download
+```
+
+Other model options: `llama3:70b` (better quality, ~40 GB), `mistral` (faster, ~4 GB).
+
+---
+
+## llama-cpp-python — in-process local LLM (auto-selected when installed)
+
+`llama-cpp-python` runs a GGUF model directly inside the Python process — no
+separate server required. It is cross-platform and available from conda-forge.
+
+When `llama-cpp-python` is importable, gsas2-query selects it **automatically**
+without any `LLM_BACKEND` setting.
+
+### Install
+
+```bash
+conda install -c conda-forge llama-cpp-python   # recommended
+# or:
+pip install "gsas2-query[llama_cpp]"
+```
+
+### Download a GGUF model
+
+Download any GGUF-format model from Hugging Face, for example:
+
+```bash
+# Llama 3.1 8B (Q4_K_M quantisation, ~5 GB):
+wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
+```
+
+### Configure
+
+Set the model path in your `.env` or environment:
+
+```env
+LLAMA_CPP_MODEL=/path/to/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
+# Optional tuning:
+# LLAMA_CPP_N_CTX=4096       # context window (default: 4096)
+# LLAMA_CPP_MAX_TOKENS=1500  # max tokens to generate (default: 1500)
+```
+
+---
+
+## First-time setup — index the documentation
+
+Run once, or again when the docs are updated. Fetches ~130 web pages and 2 PDFs,
+embeds everything locally. Takes ~10–20 minutes.
+
+By default the index is stored in `~/.GSASII/gsas_query/chroma_db`
+(see [User data directory](#userdatadirectory) for details).
+
+```bash
+gsas2-query --setup              # all sources (HTML + PDFs)
+gsas2-query --setup --html-only  # skip PDFs, faster (~5 min)
+gsas2-query --setup --reset      # drop index and rebuild from scratch
+```
+
+HTML-only index is 21 Mb. With GSAS-II Programmer's Guide, 41 Mb; with
+textbook as well, 53 Mb.
+
+---
+
+## Usage
+
+### Command-line — single question
+
+```bash
+gsas2-query "How do I set up a sequential refinement?"
+gsas2-query "What parameters control the background in Rietveld?"
+gsas2-query "How do I export a CIF for publication?"
+```
+
+### Command-line — interactive REPL
+
+Multi-turn conversation that remembers previous questions in the session.
+
+```bash
+gsas2-query
+```
+
+```
+GSAS-II Documentation Assistant
+════════════════════════════════════════════════════════════════════════════════
+
+Knowledge base: 3,847 indexed chunks.
+LLM backend: ollama
+Type your question and press Enter. 'clear' resets history, 'quit' exits.
+
+────────────────────────────────────────────────────────────────────────────────
+You: How do I constrain lattice parameters?
+Thinking…
+Assistant: To constrain lattice parameters in GSAS-II, open the Constraints
+tab in the Phase panel [1]…
+
+Sources:
+  [94%] Help: Phase General  ›  Constraints
+         https://advancedphotonsource.github.io/GSAS-II-tutorials/help/phasegeneral.html
+```
+
+Commands inside the REPL: `clear` (reset history), `quit` / `exit` (exit).
+
+### Desktop GUI
+
+Opens a standalone floating dialog — stays open while you work in GSAS-II.
+
+```bash
+gsas2-query --gui
+```
+
+### Embed in GSAS-II Help menu
+
+Add one call to the GSAS-II menu handler (e.g. in `GSASIIctrl.py`):
+
+```python
+def OnDocAssistant(self, event):
+    try:
+        from gsas_query.gui import show_assistant
+        show_assistant(self)          # self = GSAS-II main frame
+    except ImportError:
+        wx.MessageBox(
+            "GSAS-II Assistant not installed.\n"
+            "Run: pip install gsas2-query",
+            "Not available"
+        )
+```
+
+`show_assistant()` is idempotent — calling it a second time raises the existing
+window rather than opening a duplicate.
+
+### Web UI
+
+```bash
+gsas2-query-web                          # serves on 0.0.0.0:8000
+HOST=127.0.0.1 PORT=8765 gsas2-query-web
+```
+
+Then open `http://localhost:8000` in a browser. The web UI shows inline citations
+as clickable superscript links and source chips below each answer.
+
+---
+
+## CLI flags reference
+
+| Flag | Description |
+|---|---|
+| `--setup` | Index all documentation sources |
+| `--setup --reset` | Drop the existing index and rebuild |
+| `--setup --html-only` | Index HTML only, skip PDFs |
+| `--gui` | Open the wxPython desktop assistant |
+| `--backend ollama\|anthropic\|retrieval\|llama_cpp` | Override `LLM_BACKEND` env var |
+| `--model <name>` | Override OLLAMA_MODEL, ANTHROPIC_MODEL, or LLAMA_CPP_MODEL |
+| `--stats` | Show chunk count, backend, and DB path |
+
+---
+
+## LLM backend configuration
+
+Backend selection precedence:
+
+1. `LLM_BACKEND` env var (or `--backend` CLI flag) **always takes effect when set**.
+2. If `LLM_BACKEND` is **not set** and `llama-cpp-python` is importable, `llama_cpp` is selected automatically.
+3. Otherwise the default is `ollama`.
+
+| Backend | Config | Notes |
+|---|---|---|
+| `llama_cpp` (auto) | `LLAMA_CPP_MODEL=/path/to/model.gguf` | In-process, no daemon needed; auto-selected when llama-cpp-python is installed |
+| `ollama` (default) | `OLLAMA_MODEL=llama3` | Free, fully local — no data leaves the network |
+| `anthropic` | `ANTHROPIC_API_KEY=sk-ant-…` | Better answers; queries sent to Anthropic |
+| `retrieval` | — | No LLM — returns raw matched chunks; useful offline or for testing |
+
+```bash
+gsas2-query --backend retrieval "What is Le Bail extraction?"
+gsas2-query --backend ollama --model mistral "How do I index peaks?"
+gsas2-query --backend llama_cpp --model /path/to/model.gguf "How do I refine a structure?"
+```
+
+---
+
+## Inline citations
+
+When using Ollama, llama_cpp, or Anthropic backends, answers contain `[N]` markers inline.
+In the **web UI** these render as clickable superscript links opening the exact
+source section. In the **CLI**, source URLs are listed below the answer with
+relevance scores.
+
+---
+
+## Doc sources
+
+| Category | Count |
+|---|---|
+| Home / installation pages | 22 |
+| Help pages (all sections) | 42 |
+| Tutorials | 62 |
+| Programmer's Guide (PDF, readthedocs) | 1 |
+| Powder Crystallography book (PDF, auto-fetches latest release) | 1 |
+| **Total** | **128 sources** |
+
+All HTML sources are fetched from
+`https://advancedphotonsource.github.io/GSAS-II-tutorials/`.
+The book PDF is fetched from the latest GitHub release of
+[briantoby/PowderCrystallography](https://github.com/briantoby/PowderCrystallography/releases).
+
+---
+
+## Re-indexing when docs update
+
+```bash
+gsas2-query --setup --reset
+```
+
+Or trigger via the web API (requires `ADMIN_KEY` set in `.env`):
+
+```bash
+curl -X POST http://localhost:8000/ingest -H "X-Admin-Key: your-key"
+```
+
+---
+
+## Security and deployment notes
+
+- All embeddings and vector search run locally (sentence-transformers, ChromaDB).
+- Ollama runs entirely on-premises — no queries leave the network.
+- The `anthropic` backend sends question text and retrieved doc chunks to Anthropic.
+  Do not use it in air-gapped or data-sensitive environments.
+- The web server applies per-IP rate limiting (default 30 req/min, configurable
+  via `RATE_LIMIT_RPM` in `.env`).
+- The `/ingest` endpoint is protected by `X-Admin-Key`; leave `ADMIN_KEY` blank
+  to disable remote re-indexing.
+
+---
+
+## User data directory
+
+The ChromaDB index is stored at `~/.GSASII/gsas_query/chroma_db` by default.
+Override with the `GSAS_QUERY_DATA_DIR` environment variable:
+
+```bash
+GSAS_QUERY_DATA_DIR=/data/gsas_query gsas2-query --setup
+```

gsas2_query-0.3.0.dist-info/RECORD

@@ -0,0 +1,16 @@
+gsas_query/__init__.py,sha256=1o5aDYXWQTFxxsDBy3fhyXS5AbPvu66AxIYjxOkfkLo,689
+gsas_query/_embed.py,sha256=RQetKblVR95BpRQkJkYt_DJD3olIK1bZfuruIk2zOks,1478
+gsas_query/_paths.py,sha256=i9fkshKzVTlBypc89AEdmyRgYdpYCZP_n5mXqEA5utE,787
+gsas_query/_web.py,sha256=t0GLAt7XLTYaOWF16rv0T7HmsCt_mtDvKVAxkW3BRiw,301
+gsas_query/app.py,sha256=xt9Sbia_DgmC3hiru5Xev7cxXmsIOcaqxLiHG6yfbWA,4476
+gsas_query/cli.py,sha256=NuwCn6-IIRIbVMAeAmZEQpEtoEFqVTGzyEvOz5oUZAc,6244
+gsas_query/gui.py,sha256=sYOmAPkXpbsffPg62zJfj-Zv2v9_OfJc9SnWLtJZxCY,25910
+gsas_query/ingest.py,sha256=hEePnUW6luihqSMipv-7rTi-Qb4GfVECVlt3qrtUMfc,9664
+gsas_query/rag.py,sha256=oq_y6U5CiAUg_Iwp3RCdUpK0Ess7i0yLFWw59Wueirw,14005
+gsas_query/sources.py,sha256=I-TZonXza-TTkfKpY2y7mM40UW7VrNuE8HGgcYnyquQ,28418
+gsas_query/wordcount.py,sha256=kCCvvoK6ad_GgsjmJQRTRpTqRncMXtJGNxN5POuUeZw,2366
+gsas_query/static/index.html,sha256=xjfb15XUjlo8uVJV18meAUZr29Rol2-ShuLpedL0Cso,17697
+gsas2_query-0.3.0.dist-info/METADATA,sha256=E2emtf0SeEX8KPF05vFIHKPfdGyvsBAk3h6oxRnleyo,10275
+gsas2_query-0.3.0.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+gsas2_query-0.3.0.dist-info/entry_points.txt,sha256=2gcKy0n_QoJZH7wpgZi3PuSh_3R-UXit1ZmpG1V8QfY,91
+gsas2_query-0.3.0.dist-info/RECORD,,

gsas2_query-0.3.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any

gsas2_query-0.3.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+gsas2-query = gsas_query.cli:main
+gsas2-query-web = gsas_query._web:main

gsas_query/__init__.py

@@ -0,0 +1,20 @@
+"""
+GSAS-II Documentation Assistant
+
+Semantic search + AI answers over GSAS-II tutorials, help pages, and PDFs.
+All embedding and retrieval runs locally; the LLM backend is configurable.
+
+Quick start:
+    from gsas_query.rag import answer_question
+    result = answer_question("How do I set up a sequential refinement?", [])
+
+wxPython GUI (for GSAS-II Help menu integration):
+    from gsas_query.gui import show_assistant
+    show_assistant(parent_wx_window)
+"""
+
+from gsas_query._paths import get_chroma_path, get_data_dir, get_static_dir
+from gsas_query.rag import answer_question
+
+__all__ = ["answer_question", "get_chroma_path", "get_data_dir", "get_static_dir"]
+__version__ = "0.3.0"

gsas_query/_embed.py

@@ -0,0 +1,43 @@
+"""Embedding model selection for Query-GSAS.
+
+Default: BAAI/bge-base-en-v1.5 via fastembed (ONNX, no PyTorch required).
+Override: set GSAS_QUERY_EMBED_MODEL=minilm to use all-MiniLM-L6-v2 instead.
+"""
+
+import os
+from functools import lru_cache
+
+EMBED_MODEL_ENV = os.environ.get("GSAS_QUERY_EMBED_MODEL", "bge-base")
+
+BGE_BASE_MODEL = "BAAI/bge-base-en-v1.5"
+
+
+class _FastEmbedWrapper:
+    """ChromaDB-compatible wrapper around a fastembed TextEmbedding model."""
+
+    def __init__(self, model_name: str):
+        from fastembed import TextEmbedding  # type: ignore
+        self._model = TextEmbedding(model_name)
+        self.model_name = model_name
+
+    def __call__(self, input: list[str]) -> list[list[float]]:
+        return [emb.tolist() for emb in self._model.embed(input)]
+
+
+@lru_cache(maxsize=1)
+def get_embedding_function():
+    """Return the configured embedding function (cached after first call)."""
+    if EMBED_MODEL_ENV == "minilm":
+        from chromadb.utils.embedding_functions import DefaultEmbeddingFunction
+        return DefaultEmbeddingFunction()
+
+    try:
+        return _FastEmbedWrapper(BGE_BASE_MODEL)
+    except ImportError:
+        print(
+            "Warning: fastembed is not installed. "
+            "Install with: pip install fastembed\n"
+            "Falling back to all-MiniLM-L6-v2 (lower retrieval quality)."
+        )
+        from chromadb.utils.embedding_functions import DefaultEmbeddingFunction
+        return DefaultEmbeddingFunction()

gsas_query/_paths.py

@@ -0,0 +1,25 @@
+"""Central path management for user data (chroma index) and package assets."""
+
+import os
+from pathlib import Path
+
+# Suppress HuggingFace tokenizer fork warning and OpenMP threading —
+# both must be set before sentence-transformers / loky are imported.
+os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
+os.environ.setdefault("OMP_NUM_THREADS", "1")
+
+
+def get_data_dir() -> Path:
+    """Return (and create) the user data directory for gsas_query."""
+    d = Path(os.environ.get("GSAS_QUERY_DATA_DIR",
+                            Path.home() / ".GSASII" / "query_gsas2"))
+    d.mkdir(parents=True, exist_ok=True)
+    return d
+
+
+def get_chroma_path() -> str:
+    return str(get_data_dir() / "chroma_db")
+
+
+def get_static_dir() -> Path:
+    return Path(__file__).parent / "static"

gsas_query/_web.py

@@ -0,0 +1,16 @@
+"""Entry point for `gsas2-query-web` console script."""
+
+import os
+
+
+def main():
+    import uvicorn
+    from .app import app
+
+    host = os.environ.get("HOST", "0.0.0.0")
+    port = int(os.environ.get("PORT", "8000"))
+    uvicorn.run(app, host=host, port=port)
+
+
+if __name__ == "__main__":
+    main()

gsas_query/app.py

@@ -0,0 +1,154 @@
+"""
+FastAPI web server for the GSAS-II documentation chatbot.
+
+  GET  /         -> chat UI (static/index.html)
+  GET  /health   -> {"status": "ok"}
+  GET  /stats    -> {"chunks_indexed": N, "llm_backend": "..."}
+  POST /chat     -> RAG query, returns {answer, sources, citations}
+  POST /ingest   -> trigger re-indexing (requires X-Admin-Key header)
+"""
+
+import os
+import subprocess
+import sys
+import time
+from typing import Optional
+
+from fastapi import Depends, FastAPI, HTTPException, Request
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import FileResponse, JSONResponse
+from pydantic import BaseModel, Field
+
+from ._paths import get_static_dir
+
+STATIC_DIR = get_static_dir()
+
+app = FastAPI(
+    title="GSAS-II Documentation Assistant",
+    description="RAG chatbot over GSAS-II tutorials and help documentation",
+    version="0.2.0",
+)
+
+origins = os.environ.get("ALLOWED_ORIGINS", "*").split(",")
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=origins,
+    allow_methods=["GET", "POST"],
+    allow_headers=["*"],
+)
+
+
+@app.exception_handler(Exception)
+async def generic_exception_handler(request: Request, exc: Exception):
+    return JSONResponse(status_code=500, content={"detail": str(exc)})
+
+
+# --------------------------------------------------------------------------- #
+# Request / response models                                                    #
+# --------------------------------------------------------------------------- #
+
+class HistoryTurn(BaseModel):
+    role: str
+    content: str
+
+
+class ChatRequest(BaseModel):
+    message: str = Field(..., max_length=2000)
+    history: list[HistoryTurn] = Field(default_factory=list, max_length=20)
+
+
+class Source(BaseModel):
+    title: str
+    section: str
+    url: str
+    category: str
+    relevance: float
+
+
+class Citation(BaseModel):
+    title: str
+    section: str
+    url: str
+    relevance: float = 0.0
+
+
+class ChatResponse(BaseModel):
+    answer: str
+    sources: list[Source]
+    citations: dict[str, Citation]
+    elapsed_ms: int
+
+
+# --------------------------------------------------------------------------- #
+# Simple rate limiting (in-memory, per IP)                                     #
+# --------------------------------------------------------------------------- #
+
+_rate_store: dict[str, list[float]] = {}
+RATE_LIMIT = int(os.environ.get("RATE_LIMIT_RPM", "30"))
+
+
+def check_rate_limit(request: Request):
+    if RATE_LIMIT <= 0:
+        return
+    ip = request.client.host if request.client else "unknown"
+    now = time.time()
+    timestamps = [t for t in _rate_store.get(ip, []) if now - t < 60]
+    if len(timestamps) >= RATE_LIMIT:
+        raise HTTPException(status_code=429, detail="Rate limit exceeded. Please slow down.")
+    timestamps.append(now)
+    _rate_store[ip] = timestamps
+
+
+# --------------------------------------------------------------------------- #
+# Routes                                                                       #
+# --------------------------------------------------------------------------- #
+
+@app.get("/", include_in_schema=False)
+async def serve_ui():
+    return FileResponse(str(STATIC_DIR / "index.html"))
+
+
+@app.get("/health")
+async def health():
+    return {"status": "ok"}
+
+
+@app.get("/stats")
+async def stats():
+    from .rag import _effective_backend, _get_collection
+    try:
+        col = _get_collection()
+        count = col.count()
+    except Exception:
+        count = -1
+    return {"chunks_indexed": count, "llm_backend": _effective_backend()}
+
+
+@app.post("/chat", response_model=ChatResponse)
+async def chat(
+    request: ChatRequest,
+    http_request: Request,
+    _: None = Depends(check_rate_limit),
+):
+    from .rag import answer_question
+
+    t0 = time.monotonic()
+    history = [h.model_dump() for h in request.history]
+    result = answer_question(request.message, history)
+    elapsed = int((time.monotonic() - t0) * 1000)
+
+    return ChatResponse(
+        answer=result["answer"],
+        sources=result["sources"],
+        citations=result.get("citations", {}),
+        elapsed_ms=elapsed,
+    )
+
+
+@app.post("/ingest")
+async def trigger_ingest(http_request: Request):
+    admin_key = os.environ.get("ADMIN_KEY", "")
+    if admin_key and http_request.headers.get("X-Admin-Key") != admin_key:
+        raise HTTPException(status_code=403, detail="Invalid admin key.")
+    subprocess.Popen([sys.executable, "-m", "gsas_query.ingest"])
+    return {"status": "ingestion started in background"}