querdex 0.1.0__tar.gz → 0.2.0__tar.gz

{querdex-0.1.0 → querdex-0.2.0}/.gitignore

@@ -33,3 +33,4 @@ RUNBOOK.md
 .env
 .env.*
 !.env.example
+test_api_key.py

{querdex-0.1.0 → querdex-0.2.0}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.4
 Name: querdex
-Version: 0.1.0
+Version: 0.2.0
 Summary: Reasoning-first document intelligence system
-Author-email: Your Name <you@example.com>
+Author-email: Animay Tiwari <animaytiwari@outlook.com>
 License: MIT
 Keywords: document,indexing,llm,rag,retrieval
 Classifier: Development Status :: 3 - Alpha
@@ -32,6 +32,12 @@ Description-Content-Type: text/markdown
 
 # Querdex
 
+[![PyPI version](https://img.shields.io/pypi/v/querdex)](https://pypi.org/project/querdex/)
+[![Downloads](https://img.shields.io/pypi/dm/querdex)](https://pypistats.org/packages/querdex)
+[![Python versions](https://img.shields.io/pypi/pyversions/querdex)](https://pypi.org/project/querdex/)
+[![CI](https://github.com/its-animay/querdex/actions/workflows/ci.yml/badge.svg)](https://github.com/its-animay/querdex/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)
+
 **Reasoning-first document intelligence system.**
 
 Querdex indexes any document into a hierarchical tree, then uses a two-tier LLM search to answer questions with cited sources. It works without an LLM (keyword heuristics), and optionally plugs in Anthropic or OpenAI for higher-quality results.
@@ -44,6 +50,7 @@ Querdex indexes any document into a hierarchical tree, then uses a two-tier LLM
 - [Installation](#installation)
 - [Quick Start (CLI)](#quick-start-cli)
 - [LLM Setup](#llm-setup)
+- [Structured Extraction](#structured-extraction)
 - [CLI Reference](#cli-reference)
 - [Python API](#python-api)
 - [Supported File Types](#supported-file-types)
@@ -214,6 +221,54 @@ export QUERDEX_LLM_TIER2_MODEL=gpt-4o         # powerful
 
 ---
 
+## Structured Extraction
+
+Pull structured facts out of any indexed document — with **source grounding**: every extraction carries the exact section, page, and character span it came from, so nothing is silently hallucinated.
+
+The schema is defined **by example**, not by code. Describe what you want and (optionally) show one or two examples:
+
+```bash
+querdex extract --doc-id demo \
+  --prompt "Extract revenue figures and executive names" \
+  --examples examples.json \
+  --html review.html
+```
+
+`examples.json`:
+```json
+[
+  {
+    "text": "Alice Chen reported revenue of $5M in Q1.",
+    "extractions": [
+      {"extraction_class": "metric", "extraction_text": "revenue of $5M", "attributes": {"period": "Q1"}},
+      {"extraction_class": "person", "extraction_text": "Alice Chen"}
+    ]
+  }
+]
+```
+
+The classes and attribute keys in your examples define the output schema. Every result is aligned back to the source text (exact → fuzzy matching); model output that cannot be located is kept but flagged `unaligned` so you can review it instead of trusting it.
+
+`--html` writes a self-contained review page: the full document with color-coded highlights per extraction class, toggleable legend, attribute tooltips, and a click-to-jump list of all extractions.
+
+Long documents are chunked and processed in parallel; use `--passes 2` to trade extra LLM calls for higher recall. Without an LLM configured, extraction degrades to literal matching of your example texts.
+
+Python API:
+
+```python
+from querdex.extraction import ExtractionTask, ExtractionExample, ExampleExtraction
+
+task = ExtractionTask(
+    description="Extract revenue figures and executive names",
+    examples=[...],
+)
+run = engine.extract_document("demo", task, passes=1)
+for e in run.extractions:
+    print(e.extraction_class, repr(e.extraction_text), e.section_id, e.char_start, e.alignment)
+```
+
+---
+
 ## CLI Reference
 
 ```
@@ -224,6 +279,7 @@ querdex [--db PATH] <command> [options]
 |---------|-------------|
 | `index <file>` | Index a document. Auto-detects format from extension. |
 | `query` | Query an indexed document. |
+| `extract` | Run schema-by-example structured extraction over an indexed document. |
 | `delete` | Remove a document and all its data from the store. |
 
 ### `index`
@@ -249,6 +305,20 @@ querdex query --doc-id ID --query TEXT [--session-id ID]
 | `--query` | required | Natural language question |
 | `--session-id` | none | Enables multi-turn context (pass same ID across turns) |
 
+### `extract`
+
+```
+querdex extract --doc-id ID --prompt TEXT [--examples FILE] [--passes N] [--html FILE]
+```
+
+| Argument | Default | Description |
+|----------|---------|-------------|
+| `--doc-id` | required | Document to extract from |
+| `--prompt` | required | Natural language description of what to extract |
+| `--examples` | none | JSON file with few-shot examples (defines the output schema) |
+| `--passes` | `1` | Extraction passes; more passes improve recall |
+| `--html` | none | Write an interactive HTML review page to this path |
+
 ### `delete`
 
 ```

{querdex-0.1.0 → querdex-0.2.0}/README.md

@@ -1,5 +1,11 @@
 # Querdex
 
+[![PyPI version](https://img.shields.io/pypi/v/querdex)](https://pypi.org/project/querdex/)
+[![Downloads](https://img.shields.io/pypi/dm/querdex)](https://pypistats.org/packages/querdex)
+[![Python versions](https://img.shields.io/pypi/pyversions/querdex)](https://pypi.org/project/querdex/)
+[![CI](https://github.com/its-animay/querdex/actions/workflows/ci.yml/badge.svg)](https://github.com/its-animay/querdex/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://opensource.org/licenses/MIT)
+
 **Reasoning-first document intelligence system.**
 
 Querdex indexes any document into a hierarchical tree, then uses a two-tier LLM search to answer questions with cited sources. It works without an LLM (keyword heuristics), and optionally plugs in Anthropic or OpenAI for higher-quality results.
@@ -12,6 +18,7 @@ Querdex indexes any document into a hierarchical tree, then uses a two-tier LLM
 - [Installation](#installation)
 - [Quick Start (CLI)](#quick-start-cli)
 - [LLM Setup](#llm-setup)
+- [Structured Extraction](#structured-extraction)
 - [CLI Reference](#cli-reference)
 - [Python API](#python-api)
 - [Supported File Types](#supported-file-types)
@@ -182,6 +189,54 @@ export QUERDEX_LLM_TIER2_MODEL=gpt-4o         # powerful
 
 ---
 
+## Structured Extraction
+
+Pull structured facts out of any indexed document — with **source grounding**: every extraction carries the exact section, page, and character span it came from, so nothing is silently hallucinated.
+
+The schema is defined **by example**, not by code. Describe what you want and (optionally) show one or two examples:
+
+```bash
+querdex extract --doc-id demo \
+  --prompt "Extract revenue figures and executive names" \
+  --examples examples.json \
+  --html review.html
+```
+
+`examples.json`:
+```json
+[
+  {
+    "text": "Alice Chen reported revenue of $5M in Q1.",
+    "extractions": [
+      {"extraction_class": "metric", "extraction_text": "revenue of $5M", "attributes": {"period": "Q1"}},
+      {"extraction_class": "person", "extraction_text": "Alice Chen"}
+    ]
+  }
+]
+```
+
+The classes and attribute keys in your examples define the output schema. Every result is aligned back to the source text (exact → fuzzy matching); model output that cannot be located is kept but flagged `unaligned` so you can review it instead of trusting it.
+
+`--html` writes a self-contained review page: the full document with color-coded highlights per extraction class, toggleable legend, attribute tooltips, and a click-to-jump list of all extractions.
+
+Long documents are chunked and processed in parallel; use `--passes 2` to trade extra LLM calls for higher recall. Without an LLM configured, extraction degrades to literal matching of your example texts.
+
+Python API:
+
+```python
+from querdex.extraction import ExtractionTask, ExtractionExample, ExampleExtraction
+
+task = ExtractionTask(
+    description="Extract revenue figures and executive names",
+    examples=[...],
+)
+run = engine.extract_document("demo", task, passes=1)
+for e in run.extractions:
+    print(e.extraction_class, repr(e.extraction_text), e.section_id, e.char_start, e.alignment)
+```
+
+---
+
 ## CLI Reference
 
 ```
@@ -192,6 +247,7 @@ querdex [--db PATH] <command> [options]
 |---------|-------------|
 | `index <file>` | Index a document. Auto-detects format from extension. |
 | `query` | Query an indexed document. |
+| `extract` | Run schema-by-example structured extraction over an indexed document. |
 | `delete` | Remove a document and all its data from the store. |
 
 ### `index`
@@ -217,6 +273,20 @@ querdex query --doc-id ID --query TEXT [--session-id ID]
 | `--query` | required | Natural language question |
 | `--session-id` | none | Enables multi-turn context (pass same ID across turns) |
 
+### `extract`
+
+```
+querdex extract --doc-id ID --prompt TEXT [--examples FILE] [--passes N] [--html FILE]
+```
+
+| Argument | Default | Description |
+|----------|---------|-------------|
+| `--doc-id` | required | Document to extract from |
+| `--prompt` | required | Natural language description of what to extract |
+| `--examples` | none | JSON file with few-shot examples (defines the output schema) |
+| `--passes` | `1` | Extraction passes; more passes improve recall |
+| `--html` | none | Write an interactive HTML review page to this path |
+
 ### `delete`
 
 ```

{querdex-0.1.0 → querdex-0.2.0}/pyproject.toml

@@ -1,11 +1,11 @@
 [project]
 name = "querdex"
-version = "0.1.0"
+version = "0.2.0"
 description = "Reasoning-first document intelligence system"
 readme = "README.md"
 requires-python = ">=3.11"
 license = { text = "MIT" }
-authors = [{ name = "Your Name", email = "you@example.com" }]
+authors = [{ name = "Animay Tiwari", email = "animaytiwari@outlook.com" }]
 keywords = ["document", "indexing", "retrieval", "llm", "rag"]
 classifiers = [
   "Development Status :: 3 - Alpha",

{querdex-0.1.0 → querdex-0.2.0}/scripts/run_release_gate.py

@@ -5,12 +5,12 @@ import asyncio
 import json
 import sys
 
-from hyperindex.evaluation import EvaluationHarness, KPIDashboardBuilder, RetrievalMetricsHarness
-from hyperindex.services import build_engine
+from querdex.evaluation import EvaluationHarness, KPIDashboardBuilder, RetrievalMetricsHarness
+from querdex.services import build_engine
 
 
 def _build_parser() -> argparse.ArgumentParser:
-    parser = argparse.ArgumentParser(description="Run HyperIndex evaluation and release gate checks.")
+    parser = argparse.ArgumentParser(description="Run querdex evaluation and release gate checks.")
     parser.add_argument("--db", required=True, help="SQLite database path")
     parser.add_argument("--doc-path", required=True, help="Path to document to index for the benchmark run")
     parser.add_argument("--doc-id", required=True, help="Document id used for benchmark queries")

{querdex-0.1.0 → querdex-0.2.0}/src/querdex/__init__.py

@@ -2,4 +2,4 @@
 
 __all__ = ["__version__"]
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"

{querdex-0.1.0 → querdex-0.2.0}/src/querdex/cli.py

@@ -2,8 +2,10 @@ from __future__ import annotations
 
 import argparse
 import asyncio
+import json
 from pathlib import Path
 
+from querdex.extraction import ExtractionExample, ExtractionTask, render_extraction_html
 from querdex.services import build_engine
 
 
@@ -25,6 +27,13 @@ def _build_parser() -> argparse.ArgumentParser:
     delete_parser = subparsers.add_parser("delete", help="Delete an indexed document")
     delete_parser.add_argument("--doc-id", required=True, help="Document id to delete")
 
+    extract_parser = subparsers.add_parser("extract", help="Run structured extraction over an indexed document")
+    extract_parser.add_argument("--doc-id", required=True, help="Document id to extract from")
+    extract_parser.add_argument("--prompt", required=True, help="Description of what to extract")
+    extract_parser.add_argument("--examples", default=None, help="Path to JSON file with few-shot examples")
+    extract_parser.add_argument("--passes", type=int, default=1, help="Extraction passes (more passes, more recall)")
+    extract_parser.add_argument("--html", default=None, help="Write an interactive HTML review page to this path")
+
     return parser
 
 
@@ -52,6 +61,33 @@ def main() -> None:
             engine.store.delete_document(args.doc_id)
             print(f"Deleted doc_id={args.doc_id}")
 
+        elif args.command == "extract":
+            examples = []
+            if args.examples:
+                raw_examples = json.loads(Path(args.examples).read_text(encoding="utf-8"))
+                examples = [ExtractionExample.model_validate(item) for item in raw_examples]
+            task = ExtractionTask(description=args.prompt, examples=examples)
+            run = engine.extract_document(args.doc_id, task, passes=args.passes)
+            stats = run.stats
+            print(
+                f"Run {run.run_id}: {len(run.extractions)} extractions "
+                f"(exact={stats.exact_count} fuzzy={stats.fuzzy_count} unaligned={stats.unaligned_count}) "
+                f"in {stats.latency_ms}ms across {stats.chunk_count} chunks"
+            )
+            for extraction in run.extractions[:20]:
+                location = (
+                    f"p{extraction.page_number} {extraction.section_id}"
+                    if extraction.section_id
+                    else "ungrounded"
+                )
+                print(f"  [{extraction.extraction_class}] {extraction.extraction_text!r} ({location})")
+            if len(run.extractions) > 20:
+                print(f"  ... and {len(run.extractions) - 20} more")
+            if args.html:
+                sections = engine.store.sections_for_doc(args.doc_id)
+                Path(args.html).write_text(render_extraction_html(run, sections), encoding="utf-8")
+                print(f"HTML review written to {args.html}")
+
     finally:
         engine.store.close()
 

querdex-0.2.0/src/querdex/extraction/__init__.py

@@ -0,0 +1,29 @@
+from querdex.extraction.aligner import align
+from querdex.extraction.chunker import ChunkPiece, TextChunk, chunk_sections
+from querdex.extraction.extractor import StructuredExtractor
+from querdex.extraction.models import (
+    AlignmentStatus,
+    ExampleExtraction,
+    Extraction,
+    ExtractionExample,
+    ExtractionRun,
+    ExtractionStats,
+    ExtractionTask,
+)
+from querdex.extraction.visualize import render_extraction_html
+
+__all__ = [
+    "AlignmentStatus",
+    "ChunkPiece",
+    "ExampleExtraction",
+    "Extraction",
+    "ExtractionExample",
+    "ExtractionRun",
+    "ExtractionStats",
+    "ExtractionTask",
+    "StructuredExtractor",
+    "TextChunk",
+    "align",
+    "chunk_sections",
+    "render_extraction_html",
+]

querdex-0.2.0/src/querdex/extraction/aligner.py

@@ -0,0 +1,81 @@
+from __future__ import annotations
+
+from difflib import SequenceMatcher
+
+from querdex.extraction.models import AlignmentStatus
+
+_MIN_FUZZY_RATIO = 0.75
+
+
+def _normalize(text: str) -> tuple[str, list[int]]:
+    """Lowercase and collapse whitespace, keeping a map back to original indices.
+
+    ``index_map[i]`` is the index in ``text`` of normalized character ``i``,
+    which lets fuzzy matches report offsets in the original string.
+    """
+    normalized_chars: list[str] = []
+    index_map: list[int] = []
+    prev_space = True  # drop leading whitespace
+    for i, ch in enumerate(text):
+        if ch.isspace():
+            if prev_space:
+                continue
+            normalized_chars.append(" ")
+            index_map.append(i)
+            prev_space = True
+        else:
+            normalized_chars.append(ch.lower())
+            index_map.append(i)
+            prev_space = False
+    if normalized_chars and normalized_chars[-1] == " ":
+        normalized_chars.pop()
+        index_map.pop()
+    return "".join(normalized_chars), index_map
+
+
+def align(needle: str, haystack: str) -> tuple[int, int, AlignmentStatus] | None:
+    """Locate needle inside haystack, returning (start, end, status) in haystack coordinates.
+
+    Tries progressively looser strategies: exact, case-insensitive,
+    whitespace-normalized, then fuzzy via difflib. Returns None when no
+    sufficiently similar region exists - callers should treat that
+    extraction as unverified model output.
+    """
+    needle = needle.strip()
+    if not needle:
+        return None
+
+    idx = haystack.find(needle)
+    if idx >= 0:
+        return idx, idx + len(needle), "exact"
+
+    idx = haystack.lower().find(needle.lower())
+    if idx >= 0:
+        return idx, idx + len(needle), "exact"
+
+    norm_haystack, index_map = _normalize(haystack)
+    norm_needle, _ = _normalize(needle)
+    if not norm_needle or not norm_haystack:
+        return None
+
+    idx = norm_haystack.find(norm_needle)
+    if idx >= 0:
+        start = index_map[idx]
+        end = index_map[idx + len(norm_needle) - 1] + 1
+        return start, end, "fuzzy"
+
+    matcher = SequenceMatcher(a=norm_haystack, b=norm_needle, autojunk=False)
+    match = matcher.find_longest_match(0, len(norm_haystack), 0, len(norm_needle))
+    if match.size == 0:
+        return None
+    window_start = max(0, match.a - match.b)
+    window_end = min(len(norm_haystack), window_start + len(norm_needle))
+    if window_end <= window_start:
+        return None
+    window = norm_haystack[window_start:window_end]
+    ratio = SequenceMatcher(a=window, b=norm_needle, autojunk=False).ratio()
+    if ratio < _MIN_FUZZY_RATIO:
+        return None
+    start = index_map[window_start]
+    end = index_map[window_end - 1] + 1
+    return start, end, "fuzzy"

querdex-0.2.0/src/querdex/extraction/chunker.py

@@ -0,0 +1,108 @@
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+
+from querdex.schemas import Section
+
+_SENTENCE_END_RE = re.compile(r"[.!?]\s")
+_PIECE_SEPARATOR = "\n\n"
+
+
+@dataclass
+class ChunkPiece:
+    """A contiguous region of a chunk mapped back to its source section."""
+
+    section_id: str
+    page_number: int
+    chunk_offset: int
+    section_offset: int
+    length: int
+
+
+@dataclass
+class TextChunk:
+    chunk_id: str
+    text: str
+    pieces: list[ChunkPiece] = field(default_factory=list)
+
+    def locate(self, char_start: int) -> ChunkPiece | None:
+        """Return the piece containing char_start (chunk coordinates)."""
+        for piece in self.pieces:
+            if piece.chunk_offset <= char_start < piece.chunk_offset + piece.length:
+                return piece
+        return None
+
+
+def _split_long_text(text: str, max_chars: int) -> list[tuple[int, str]]:
+    """Split text into (section_offset, piece) tuples of at most max_chars.
+
+    Cuts prefer sentence boundaries, then word boundaries, so extractions
+    are unlikely to straddle a chunk edge.
+    """
+    pieces: list[tuple[int, str]] = []
+    start = 0
+    while start < len(text):
+        if len(text) - start <= max_chars:
+            pieces.append((start, text[start:]))
+            break
+        window = text[start : start + max_chars]
+        cut = max((m.end() for m in _SENTENCE_END_RE.finditer(window)), default=0)
+        if cut < max_chars // 2:
+            space = window.rfind(" ")
+            cut = space + 1 if space > max_chars // 2 else max_chars
+        pieces.append((start, text[start : start + cut]))
+        start += cut
+    return pieces
+
+
+def chunk_sections(sections: list[Section], *, max_chars: int = 4000) -> list[TextChunk]:
+    """Group section contents into chunks of at most max_chars.
+
+    Consecutive small sections are packed together; oversized sections are
+    split at sentence boundaries. Every region of every chunk maps back to
+    its source section and offset so extractions can be grounded precisely.
+    """
+    chunks: list[TextChunk] = []
+    current_parts: list[str] = []
+    current_pieces: list[ChunkPiece] = []
+    current_len = 0
+
+    def flush() -> None:
+        nonlocal current_parts, current_pieces, current_len
+        if current_parts:
+            chunks.append(
+                TextChunk(
+                    chunk_id=f"chunk_{len(chunks) + 1:04d}",
+                    text="".join(current_parts),
+                    pieces=current_pieces,
+                )
+            )
+        current_parts = []
+        current_pieces = []
+        current_len = 0
+
+    def add_piece(section: Section, section_offset: int, piece_text: str) -> None:
+        nonlocal current_len
+        if current_parts:
+            current_parts.append(_PIECE_SEPARATOR)
+            current_len += len(_PIECE_SEPARATOR)
+        current_pieces.append(
+            ChunkPiece(
+                section_id=section.section_id,
+                page_number=section.page_number,
+                chunk_offset=current_len,
+                section_offset=section_offset,
+                length=len(piece_text),
+            )
+        )
+        current_parts.append(piece_text)
+        current_len += len(piece_text)
+
+    for section in sections:
+        for section_offset, piece_text in _split_long_text(section.content, max_chars):
+            if current_len and current_len + len(_PIECE_SEPARATOR) + len(piece_text) > max_chars:
+                flush()
+            add_piece(section, section_offset, piece_text)
+    flush()
+    return chunks