whowhen-rag 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- whowhen_rag-0.1.0/.github/ISSUE_TEMPLATE/bug_report.yml +34 -0
- whowhen_rag-0.1.0/.github/ISSUE_TEMPLATE/config.yml +5 -0
- whowhen_rag-0.1.0/.github/ISSUE_TEMPLATE/feature_request.yml +16 -0
- whowhen_rag-0.1.0/.github/workflows/publish.yml +44 -0
- whowhen_rag-0.1.0/.github/workflows/tests.yml +14 -0
- whowhen_rag-0.1.0/.gitignore +9 -0
- whowhen_rag-0.1.0/CONTRIBUTING.md +32 -0
- whowhen_rag-0.1.0/LICENSE +21 -0
- whowhen_rag-0.1.0/PKG-INFO +220 -0
- whowhen_rag-0.1.0/README.md +201 -0
- whowhen_rag-0.1.0/SECURITY.md +7 -0
- whowhen_rag-0.1.0/benchmarks/RESULTS.md +34 -0
- whowhen_rag-0.1.0/benchmarks/extraction.py +131 -0
- whowhen_rag-0.1.0/benchmarks/extraction_cases.py +95 -0
- whowhen_rag-0.1.0/benchmarks/gates.py +49 -0
- whowhen_rag-0.1.0/benchmarks/results/2026-11-20_extraction_claude-haiku-4.5_main_prompt-v1.json +263 -0
- whowhen_rag-0.1.0/benchmarks/results/2026-11-21_extraction_claude-haiku-4.5_holdout_prompt-v1.json +145 -0
- whowhen_rag-0.1.0/benchmarks/results/2026-11-21_extraction_claude-haiku-4.5_holdout_prompt-v2.json +137 -0
- whowhen_rag-0.1.0/benchmarks/results/2026-11-21_extraction_claude-haiku-4.5_main_prompt-v2.json +257 -0
- whowhen_rag-0.1.0/benchmarks/results/2026-11-22_extraction_claude-haiku-4.5_fresh_prompt-v2.json +142 -0
- whowhen_rag-0.1.0/docs/prompts.md +68 -0
- whowhen_rag-0.1.0/examples/quickstart.py +52 -0
- whowhen_rag-0.1.0/pyproject.toml +35 -0
- whowhen_rag-0.1.0/src/whowhen_rag/__init__.py +33 -0
- whowhen_rag-0.1.0/src/whowhen_rag/classifier.py +93 -0
- whowhen_rag-0.1.0/src/whowhen_rag/detector.py +164 -0
- whowhen_rag-0.1.0/src/whowhen_rag/extract.py +142 -0
- whowhen_rag-0.1.0/src/whowhen_rag/fields.py +76 -0
- whowhen_rag-0.1.0/src/whowhen_rag/filters.py +123 -0
- whowhen_rag-0.1.0/src/whowhen_rag/gates.py +67 -0
- whowhen_rag-0.1.0/src/whowhen_rag/prompts/v1.txt +31 -0
- whowhen_rag-0.1.0/src/whowhen_rag/prompts/v2.txt +38 -0
- whowhen_rag-0.1.0/tests/conftest.py +58 -0
- whowhen_rag-0.1.0/tests/queries.py +76 -0
- whowhen_rag-0.1.0/tests/test_classifier.py +44 -0
- whowhen_rag-0.1.0/tests/test_detector.py +203 -0
- whowhen_rag-0.1.0/tests/test_filters.py +46 -0
- whowhen_rag-0.1.0/tests/test_gates.py +25 -0
- whowhen_rag-0.1.0/tests/test_prompts.py +44 -0
- whowhen_rag-0.1.0/uv.lock +549 -0
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
name: Bug report
|
|
2
|
+
description: Wrong filters, a crash, or behaviour that does not match the docs
|
|
3
|
+
labels: [bug]
|
|
4
|
+
body:
|
|
5
|
+
- type: textarea
|
|
6
|
+
id: query
|
|
7
|
+
attributes:
|
|
8
|
+
label: Query and result
|
|
9
|
+
description: The query, the filters you got (`f.to_dict()`), and the filters you expected. Please replace any private names or data.
|
|
10
|
+
placeholder: |
|
|
11
|
+
Query: "docs Priya edited last month"
|
|
12
|
+
Got: ...
|
|
13
|
+
Expected: ...
|
|
14
|
+
validations:
|
|
15
|
+
required: true
|
|
16
|
+
- type: textarea
|
|
17
|
+
id: setup
|
|
18
|
+
attributes:
|
|
19
|
+
label: Setup
|
|
20
|
+
description: Your Field declarations, the LLM and model you pass as `llm=`, the prompt version, and whether the classifier is on.
|
|
21
|
+
validations:
|
|
22
|
+
required: true
|
|
23
|
+
- type: input
|
|
24
|
+
id: versions
|
|
25
|
+
attributes:
|
|
26
|
+
label: Versions
|
|
27
|
+
description: Output of `python -c "import whowhen_rag, sys; print(whowhen.__version__, sys.version)"`
|
|
28
|
+
validations:
|
|
29
|
+
required: true
|
|
30
|
+
- type: textarea
|
|
31
|
+
id: trace
|
|
32
|
+
attributes:
|
|
33
|
+
label: Trace or traceback
|
|
34
|
+
description: "`f.trace`, and any warning logged by the `whowhen_rag` logger or a full traceback."
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
name: Feature request
|
|
2
|
+
description: A new option, store adapter, role, or language
|
|
3
|
+
labels: [enhancement]
|
|
4
|
+
body:
|
|
5
|
+
- type: textarea
|
|
6
|
+
id: problem
|
|
7
|
+
attributes:
|
|
8
|
+
label: What are you trying to do?
|
|
9
|
+
description: The kind of query or setup that whowhen-rag does not handle today, with example queries.
|
|
10
|
+
validations:
|
|
11
|
+
required: true
|
|
12
|
+
- type: textarea
|
|
13
|
+
id: proposal
|
|
14
|
+
attributes:
|
|
15
|
+
label: What would help?
|
|
16
|
+
description: The option or behaviour you have in mind, if any.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
name: publish
|
|
2
|
+
# Release: bump version in pyproject.toml, commit, then `git tag vX.Y.Z && git push origin vX.Y.Z`.
|
|
3
|
+
# Uploads to PyPI with trusted publishing (no API token stored anywhere). One-time setup on
|
|
4
|
+
# pypi.org: Account > Publishing > add a pending publisher for owner rahulbansalc6414,
|
|
5
|
+
# repository whowhen-rag, workflow publish.yml, environment pypi.
|
|
6
|
+
on:
|
|
7
|
+
push:
|
|
8
|
+
tags: ["v*"]
|
|
9
|
+
|
|
10
|
+
jobs:
|
|
11
|
+
build:
|
|
12
|
+
runs-on: ubuntu-latest
|
|
13
|
+
steps:
|
|
14
|
+
- uses: actions/checkout@v4
|
|
15
|
+
- uses: astral-sh/setup-uv@v5
|
|
16
|
+
- name: Tag matches pyproject version
|
|
17
|
+
run: |
|
|
18
|
+
version=$(grep -m1 '^version = ' pyproject.toml | cut -d'"' -f2)
|
|
19
|
+
if [ "v$version" != "$GITHUB_REF_NAME" ]; then
|
|
20
|
+
echo "Tag $GITHUB_REF_NAME does not match pyproject version $version"; exit 1
|
|
21
|
+
fi
|
|
22
|
+
- run: uv sync
|
|
23
|
+
- run: uv run ruff check .
|
|
24
|
+
- run: uv run pytest -q
|
|
25
|
+
- run: uv build
|
|
26
|
+
- uses: actions/upload-artifact@v4
|
|
27
|
+
with:
|
|
28
|
+
name: dist
|
|
29
|
+
path: dist/
|
|
30
|
+
|
|
31
|
+
publish:
|
|
32
|
+
needs: build
|
|
33
|
+
runs-on: ubuntu-latest
|
|
34
|
+
environment:
|
|
35
|
+
name: pypi
|
|
36
|
+
url: https://pypi.org/project/whowhen-rag/
|
|
37
|
+
permissions:
|
|
38
|
+
id-token: write # trusted publishing
|
|
39
|
+
steps:
|
|
40
|
+
- uses: actions/download-artifact@v4
|
|
41
|
+
with:
|
|
42
|
+
name: dist
|
|
43
|
+
path: dist/
|
|
44
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
name: tests
|
|
2
|
+
on: [push, pull_request]
|
|
3
|
+
jobs:
|
|
4
|
+
test:
|
|
5
|
+
runs-on: ubuntu-latest
|
|
6
|
+
strategy:
|
|
7
|
+
matrix:
|
|
8
|
+
python-version: ["3.10", "3.12", "3.13"]
|
|
9
|
+
steps:
|
|
10
|
+
- uses: actions/checkout@v4
|
|
11
|
+
- uses: astral-sh/setup-uv@v5
|
|
12
|
+
- run: uv sync --python ${{ matrix.python-version }}
|
|
13
|
+
- run: uv run ruff check .
|
|
14
|
+
- run: uv run pytest -q
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Bug reports, example queries that whowhen-rag gets wrong, and pull requests are all welcome.
|
|
4
|
+
|
|
5
|
+
## Set up
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
git clone https://github.com/rahulbansalc6414/whowhen-rag.git
|
|
9
|
+
cd whowhen-rag
|
|
10
|
+
uv sync
|
|
11
|
+
uv run pytest # no network or API keys needed
|
|
12
|
+
uv run ruff check .
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Ground rules
|
|
16
|
+
|
|
17
|
+
- **Tests need no network or keys.** Use the fake LLM and classifier in `tests/conftest.py`.
|
|
18
|
+
- **Failures never break search.** A classifier or LLM problem returns "no filter" and logs a warning; keep it that way.
|
|
19
|
+
- **Synthetic data only.** Test and benchmark queries use made-up names and topics. Do not add real company, customer or personal data.
|
|
20
|
+
- **No em dashes** in docs or messages; use colons, commas or parentheses.
|
|
21
|
+
|
|
22
|
+
## Changing the gates or the prompt
|
|
23
|
+
|
|
24
|
+
These change what the package returns, so they need numbers:
|
|
25
|
+
|
|
26
|
+
- **Regex gate** (`src/whowhen_rag/gates.py`): add the queries it should now catch to `tests/queries.py`, and check the plain questions still miss. Run `uv run python benchmarks/gates.py` (set `TYPESAFE_API_KEY` to include the classifier).
|
|
27
|
+
- **Prompt**: add a new file (`src/whowhen_rag/prompts/v3.txt`) rather than editing an existing version, and score it: `uv run python benchmarks/extraction.py --prompt v3` and `--set holdout`. See [docs/prompts.md](docs/prompts.md).
|
|
28
|
+
- Append the results to `benchmarks/RESULTS.md` with the date and commit. Do not overwrite earlier rows.
|
|
29
|
+
|
|
30
|
+
## Releases
|
|
31
|
+
|
|
32
|
+
Maintainers release by bumping `version` in `pyproject.toml` (the only place it is set) and pushing a matching tag `vX.Y.Z`. The publish workflow builds and uploads to PyPI.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Rahul Bansal
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,220 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: whowhen-rag
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Detect time and author filters in RAG queries, cheaply: a free regex gate, a yes/no classifier, and your LLM only when a filter is likely.
|
|
5
|
+
Project-URL: Homepage, https://github.com/rahulbansalc6414/whowhen-rag
|
|
6
|
+
Project-URL: Issues, https://github.com/rahulbansalc6414/whowhen-rag/issues
|
|
7
|
+
Author: Rahul Bansal
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: llm,metadata-filtering,query-understanding,rag,retrieval,vector-search
|
|
11
|
+
Classifier: Development Status :: 3 - Alpha
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
15
|
+
Classifier: Topic :: Text Processing :: Linguistic
|
|
16
|
+
Requires-Python: >=3.10
|
|
17
|
+
Requires-Dist: typesafe-sdk>=0.7
|
|
18
|
+
Description-Content-Type: text/markdown
|
|
19
|
+
|
|
20
|
+
# whowhen-rag
|
|
21
|
+
|
|
22
|
+
**Find the "who" and "when" in a search query, and only pay for an LLM when there is one.**
|
|
23
|
+
|
|
24
|
+
`whowhen-rag` turns questions like *"what did Priya change in the pricing docs last week?"* into metadata filters for your RAG system:
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
f = detector.detect("what did Priya change in the pricing docs last week?")
|
|
28
|
+
f.conditions # [Condition('modified_on', '>=', '2026-11-09'),
|
|
29
|
+
# Condition('modified_on', '<', '2026-11-16'),
|
|
30
|
+
# Condition('last_editor', '=', 'priya@example.com')]
|
|
31
|
+
f.cleaned_query # 'pricing docs' (use this for the semantic search)
|
|
32
|
+
f.to_sql() # ('"modified_on" >= %s AND "modified_on" < %s AND "last_editor" = %s', [...])
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
It works with any vector store or database, any LLM, and any column names.
|
|
36
|
+
|
|
37
|
+
## Why it is cheap
|
|
38
|
+
|
|
39
|
+
Most questions sent to a knowledge base have no time or author condition at all. Framework tools for this (LangChain's `SelfQueryRetriever`, LlamaIndex auto-retrieval) send **every** query to an LLM to find out. `whowhen-rag` puts two cheap gates in front of the LLM:
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
query
|
|
43
|
+
│
|
|
44
|
+
├─ 1. Regex gate ......... free. "last week", "written by", "latest" ... ───────┐
|
|
45
|
+
│ miss │ hit
|
|
46
|
+
├─ 2. Jev classifier ..... a yes/no answer, no text generation. ─────────────────┤ yes
|
|
47
|
+
│ no: return "no filter", no LLM call │
|
|
48
|
+
│ ▼
|
|
49
|
+
└─ 3. Your LLM ........... extracts dates, names, sort order. Only for likely filters.
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
- **Gate 1** catches the common wording for free.
|
|
53
|
+
- **Gate 2** ([Jev](https://pypi.org/project/typesafe-sdk/)) catches unusual wording ("anything Priya put together on onboarding?"). It answers yes or no with a probability. It cannot pull out the dates or names, which is why step 3 exists.
|
|
54
|
+
- **Step 3** runs only when a gate says yes, so plain questions ("how does caching work?") never reach a paid model.
|
|
55
|
+
|
|
56
|
+
Every result carries a `trace` (`gate`, `classifier_called`, `llm_called`), so you can measure the savings on your own traffic.
|
|
57
|
+
|
|
58
|
+
### Measured
|
|
59
|
+
|
|
60
|
+
On the synthetic benchmark sets (details and history in [`benchmarks/RESULTS.md`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/benchmarks/RESULTS.md)):
|
|
61
|
+
|
|
62
|
+
- **Plain questions:** 20 of 20 stopped by the gates. No LLM call.
|
|
63
|
+
- **Unusual wording the regex misses:** 8 of 8 caught by the Jev classifier.
|
|
64
|
+
- **Extraction** (Claude Haiku 4.5, prompt v2): **13 of 15** exactly right (dates, people, sort) on a fresh set written after the prompt was final, and 45 of 45 on the sets used while writing it. Misses on the fresh set: "since September" read as last year, and one arguable person filter.
|
|
65
|
+
- **Cost of an extraction:** about 750 input and 75 output tokens, about **$0.0011** at Haiku 4.5 list price (about $1.10 per 1,000 queries that reach the LLM), median 1.6 s.
|
|
66
|
+
|
|
67
|
+
These are small, hand-written sets. Measure on your own queries before relying on the numbers.
|
|
68
|
+
|
|
69
|
+
## Install
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
pip install whowhen-rag
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Quick start
|
|
76
|
+
|
|
77
|
+
```python
|
|
78
|
+
from whowhen_rag import Detector, Field
|
|
79
|
+
|
|
80
|
+
detector = Detector(
|
|
81
|
+
fields=[
|
|
82
|
+
Field("created_on", role="created_at"),
|
|
83
|
+
Field("modified_on", role="updated_at"),
|
|
84
|
+
Field("owner_email", role="created_by", resolve=lookup_email), # name -> your stored value
|
|
85
|
+
],
|
|
86
|
+
llm=my_llm, # any function: prompt string in, answer string out
|
|
87
|
+
)
|
|
88
|
+
|
|
89
|
+
f = detector.detect("pages Priya wrote this month about onboarding")
|
|
90
|
+
if f: # same as f.needs_filter
|
|
91
|
+
where, params = f.to_sql()
|
|
92
|
+
search_text = f.cleaned_query # "onboarding"
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Set `TYPESAFE_API_KEY` for the Jev classifier. A runnable example with sqlite is in [`examples/quickstart.py`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/examples/quickstart.py).
|
|
96
|
+
|
|
97
|
+
## Your columns, your names
|
|
98
|
+
|
|
99
|
+
You declare your columns once. The LLM never sees them: it answers in terms of four **roles**, and `whowhen-rag` maps each role to your column.
|
|
100
|
+
|
|
101
|
+
| Role | Meaning |
|
|
102
|
+
|---|---|
|
|
103
|
+
| `created_at` | when the document was created |
|
|
104
|
+
| `updated_at` | when the document was last edited |
|
|
105
|
+
| `created_by` | who created or wrote it |
|
|
106
|
+
| `updated_by` | who last edited it |
|
|
107
|
+
|
|
108
|
+
Declare only the roles you have. If you have one date column, "changed last week" and "created last week" both use it.
|
|
109
|
+
|
|
110
|
+
```python
|
|
111
|
+
Field(
|
|
112
|
+
"author_id", # your column or metadata key
|
|
113
|
+
role="created_by",
|
|
114
|
+
resolve=lambda name: users.find_id(name), # str, int, list of IDs, or None if unknown
|
|
115
|
+
)
|
|
116
|
+
Field(
|
|
117
|
+
"updated_ts",
|
|
118
|
+
role="updated_at",
|
|
119
|
+
value_format="epoch", # "date" (default), "datetime", "epoch", "epoch_ms"
|
|
120
|
+
description="when the ticket was last touched", # optional wording for the LLM
|
|
121
|
+
)
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
- **`resolve`** turns a name from the query into what your column stores. Without it, the name is used as written. If it returns `None`, that condition is dropped and the name is listed in `f.trace.unresolved_people`.
|
|
125
|
+
- **`value_format`** matches how your dates are stored. Dates are midnight UTC.
|
|
126
|
+
|
|
127
|
+
## Bring your own LLM
|
|
128
|
+
|
|
129
|
+
`llm` is any function that takes the prompt and returns the model's text. `whowhen-rag` writes the prompt and validates the answer. Tested with Claude Haiku 4.5; any model that follows JSON instructions should work.
|
|
130
|
+
|
|
131
|
+
**Anthropic**
|
|
132
|
+
|
|
133
|
+
```python
|
|
134
|
+
import anthropic
|
|
135
|
+
|
|
136
|
+
client = anthropic.Anthropic()
|
|
137
|
+
|
|
138
|
+
def llm(prompt: str) -> str:
|
|
139
|
+
msg = client.messages.create(model="claude-haiku-4-5", max_tokens=512,
|
|
140
|
+
messages=[{"role": "user", "content": prompt}])
|
|
141
|
+
return msg.content[0].text
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
**OpenAI or OpenRouter**
|
|
145
|
+
|
|
146
|
+
```python
|
|
147
|
+
from openai import OpenAI
|
|
148
|
+
|
|
149
|
+
client = OpenAI() # OpenRouter: OpenAI(base_url="https://openrouter.ai/api/v1", api_key=...)
|
|
150
|
+
|
|
151
|
+
def llm(prompt: str) -> str:
|
|
152
|
+
r = client.chat.completions.create(model="gpt-4o-mini", temperature=0,
|
|
153
|
+
messages=[{"role": "user", "content": prompt}])
|
|
154
|
+
return r.choices[0].message.content or ""
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
## Prompts
|
|
158
|
+
|
|
159
|
+
The extraction prompt is a plain text file, versioned and measured. `v2` is the default; `v1` is kept so you can see what changed and why.
|
|
160
|
+
|
|
161
|
+
```python
|
|
162
|
+
Detector(fields, llm, prompt="v1") # an older built-in version
|
|
163
|
+
Detector(fields, llm, prompt="my_prompt.txt") # your own template
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
v2 fixed five failure patterns found by the benchmark ("in October" read as last year, a year in a topic name read as a filter, and others), going from 37 to 45 of 45 correct on the sets used while writing it, and 13 of 15 on a fresh set. See [`docs/prompts.md`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/docs/prompts.md) for the before and after, the template placeholders, and how to score your own prompt.
|
|
167
|
+
|
|
168
|
+
## The classifier (Jev)
|
|
169
|
+
|
|
170
|
+
On by default. It reads `TYPESAFE_API_KEY`, or you pass the key:
|
|
171
|
+
|
|
172
|
+
```python
|
|
173
|
+
from whowhen_rag import Detector, JevClassifier
|
|
174
|
+
|
|
175
|
+
Detector(fields, llm, classifier=JevClassifier(api_key="...", threshold=0.5))
|
|
176
|
+
Detector(fields, llm, use_classifier=False) # regex gate only
|
|
177
|
+
Detector(fields, llm, classifier=my_classifier) # any function: query -> bool
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
Without a key, `whowhen-rag` logs one warning and carries on with the regex gate only; queries the regex misses then get no filter.
|
|
181
|
+
|
|
182
|
+
## Using the result
|
|
183
|
+
|
|
184
|
+
| | |
|
|
185
|
+
|---|---|
|
|
186
|
+
| `f.needs_filter`, `bool(f)` | any condition, sort or list request found |
|
|
187
|
+
| `f.conditions` | `Condition(column, op, value)`; op is `>=`, `<`, `=` or `in`; combine with AND |
|
|
188
|
+
| `f.sort` | `Sort(column, descending)` for "latest" or "oldest", else `None` |
|
|
189
|
+
| `f.cleaned_query` | the query without its time and person words, for semantic search |
|
|
190
|
+
| `f.list_only` | the query only asks to list documents ("pages Priya created this week"); you can skip semantic search |
|
|
191
|
+
| `f.to_sql(placeholder="%s")` | `(where_clause, params)`: columns quoted, values parameterised; `"?"` for sqlite |
|
|
192
|
+
| `f.order_by_sql()` | `ORDER BY "col" DESC`, or `""` |
|
|
193
|
+
| `f.to_chroma()` | a Chroma `where` dict (Chroma ranges need numbers: use `value_format="epoch"`) |
|
|
194
|
+
| `f.to_qdrant()` | a Qdrant filter dict: `models.Filter(**f.to_qdrant())` |
|
|
195
|
+
| `f.to_dict()` | plain dict, JSON-serialisable |
|
|
196
|
+
| `f.trace` | which gate fired, whether the classifier and LLM ran, errors |
|
|
197
|
+
|
|
198
|
+
Only need the yes or no? `detector.has_signal(query)` runs the cheap gates and never calls the LLM.
|
|
199
|
+
|
|
200
|
+
## Failure behaviour
|
|
201
|
+
|
|
202
|
+
Search should never break because of filtering. If the classifier or LLM fails, times out or returns something unusable, `detect()` returns "no filter" with the original query and logs a warning (logger `whowhen_rag`). The LLM's answer is validated: unknown roles, invalid dates and empty ranges are dropped, never guessed.
|
|
203
|
+
|
|
204
|
+
## Limitations
|
|
205
|
+
|
|
206
|
+
- English queries only (the regex gate and the prompt are English).
|
|
207
|
+
- Synchronous. Use `asyncio.to_thread` from async code.
|
|
208
|
+
- Four roles (created and updated, time and person). "My documents" needs the current user, which `whowhen-rag` does not know.
|
|
209
|
+
|
|
210
|
+
## Development
|
|
211
|
+
|
|
212
|
+
```bash
|
|
213
|
+
uv sync
|
|
214
|
+
uv run pytest # no network or API keys needed
|
|
215
|
+
uv run python benchmarks/gates.py # set TYPESAFE_API_KEY to include the classifier
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
## License
|
|
219
|
+
|
|
220
|
+
MIT
|
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
# whowhen-rag
|
|
2
|
+
|
|
3
|
+
**Find the "who" and "when" in a search query, and only pay for an LLM when there is one.**
|
|
4
|
+
|
|
5
|
+
`whowhen-rag` turns questions like *"what did Priya change in the pricing docs last week?"* into metadata filters for your RAG system:
|
|
6
|
+
|
|
7
|
+
```python
|
|
8
|
+
f = detector.detect("what did Priya change in the pricing docs last week?")
|
|
9
|
+
f.conditions # [Condition('modified_on', '>=', '2026-11-09'),
|
|
10
|
+
# Condition('modified_on', '<', '2026-11-16'),
|
|
11
|
+
# Condition('last_editor', '=', 'priya@example.com')]
|
|
12
|
+
f.cleaned_query # 'pricing docs' (use this for the semantic search)
|
|
13
|
+
f.to_sql() # ('"modified_on" >= %s AND "modified_on" < %s AND "last_editor" = %s', [...])
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
It works with any vector store or database, any LLM, and any column names.
|
|
17
|
+
|
|
18
|
+
## Why it is cheap
|
|
19
|
+
|
|
20
|
+
Most questions sent to a knowledge base have no time or author condition at all. Framework tools for this (LangChain's `SelfQueryRetriever`, LlamaIndex auto-retrieval) send **every** query to an LLM to find out. `whowhen-rag` puts two cheap gates in front of the LLM:
|
|
21
|
+
|
|
22
|
+
```
|
|
23
|
+
query
|
|
24
|
+
│
|
|
25
|
+
├─ 1. Regex gate ......... free. "last week", "written by", "latest" ... ───────┐
|
|
26
|
+
│ miss │ hit
|
|
27
|
+
├─ 2. Jev classifier ..... a yes/no answer, no text generation. ─────────────────┤ yes
|
|
28
|
+
│ no: return "no filter", no LLM call │
|
|
29
|
+
│ ▼
|
|
30
|
+
└─ 3. Your LLM ........... extracts dates, names, sort order. Only for likely filters.
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
- **Gate 1** catches the common wording for free.
|
|
34
|
+
- **Gate 2** ([Jev](https://pypi.org/project/typesafe-sdk/)) catches unusual wording ("anything Priya put together on onboarding?"). It answers yes or no with a probability. It cannot pull out the dates or names, which is why step 3 exists.
|
|
35
|
+
- **Step 3** runs only when a gate says yes, so plain questions ("how does caching work?") never reach a paid model.
|
|
36
|
+
|
|
37
|
+
Every result carries a `trace` (`gate`, `classifier_called`, `llm_called`), so you can measure the savings on your own traffic.
|
|
38
|
+
|
|
39
|
+
### Measured
|
|
40
|
+
|
|
41
|
+
On the synthetic benchmark sets (details and history in [`benchmarks/RESULTS.md`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/benchmarks/RESULTS.md)):
|
|
42
|
+
|
|
43
|
+
- **Plain questions:** 20 of 20 stopped by the gates. No LLM call.
|
|
44
|
+
- **Unusual wording the regex misses:** 8 of 8 caught by the Jev classifier.
|
|
45
|
+
- **Extraction** (Claude Haiku 4.5, prompt v2): **13 of 15** exactly right (dates, people, sort) on a fresh set written after the prompt was final, and 45 of 45 on the sets used while writing it. Misses on the fresh set: "since September" read as last year, and one arguable person filter.
|
|
46
|
+
- **Cost of an extraction:** about 750 input and 75 output tokens, about **$0.0011** at Haiku 4.5 list price (about $1.10 per 1,000 queries that reach the LLM), median 1.6 s.
|
|
47
|
+
|
|
48
|
+
These are small, hand-written sets. Measure on your own queries before relying on the numbers.
|
|
49
|
+
|
|
50
|
+
## Install
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
pip install whowhen-rag
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Quick start
|
|
57
|
+
|
|
58
|
+
```python
|
|
59
|
+
from whowhen_rag import Detector, Field
|
|
60
|
+
|
|
61
|
+
detector = Detector(
|
|
62
|
+
fields=[
|
|
63
|
+
Field("created_on", role="created_at"),
|
|
64
|
+
Field("modified_on", role="updated_at"),
|
|
65
|
+
Field("owner_email", role="created_by", resolve=lookup_email), # name -> your stored value
|
|
66
|
+
],
|
|
67
|
+
llm=my_llm, # any function: prompt string in, answer string out
|
|
68
|
+
)
|
|
69
|
+
|
|
70
|
+
f = detector.detect("pages Priya wrote this month about onboarding")
|
|
71
|
+
if f: # same as f.needs_filter
|
|
72
|
+
where, params = f.to_sql()
|
|
73
|
+
search_text = f.cleaned_query # "onboarding"
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Set `TYPESAFE_API_KEY` for the Jev classifier. A runnable example with sqlite is in [`examples/quickstart.py`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/examples/quickstart.py).
|
|
77
|
+
|
|
78
|
+
## Your columns, your names
|
|
79
|
+
|
|
80
|
+
You declare your columns once. The LLM never sees them: it answers in terms of four **roles**, and `whowhen-rag` maps each role to your column.
|
|
81
|
+
|
|
82
|
+
| Role | Meaning |
|
|
83
|
+
|---|---|
|
|
84
|
+
| `created_at` | when the document was created |
|
|
85
|
+
| `updated_at` | when the document was last edited |
|
|
86
|
+
| `created_by` | who created or wrote it |
|
|
87
|
+
| `updated_by` | who last edited it |
|
|
88
|
+
|
|
89
|
+
Declare only the roles you have. If you have one date column, "changed last week" and "created last week" both use it.
|
|
90
|
+
|
|
91
|
+
```python
|
|
92
|
+
Field(
|
|
93
|
+
"author_id", # your column or metadata key
|
|
94
|
+
role="created_by",
|
|
95
|
+
resolve=lambda name: users.find_id(name), # str, int, list of IDs, or None if unknown
|
|
96
|
+
)
|
|
97
|
+
Field(
|
|
98
|
+
"updated_ts",
|
|
99
|
+
role="updated_at",
|
|
100
|
+
value_format="epoch", # "date" (default), "datetime", "epoch", "epoch_ms"
|
|
101
|
+
description="when the ticket was last touched", # optional wording for the LLM
|
|
102
|
+
)
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
- **`resolve`** turns a name from the query into what your column stores. Without it, the name is used as written. If it returns `None`, that condition is dropped and the name is listed in `f.trace.unresolved_people`.
|
|
106
|
+
- **`value_format`** matches how your dates are stored. Dates are midnight UTC.
|
|
107
|
+
|
|
108
|
+
## Bring your own LLM
|
|
109
|
+
|
|
110
|
+
`llm` is any function that takes the prompt and returns the model's text. `whowhen-rag` writes the prompt and validates the answer. Tested with Claude Haiku 4.5; any model that follows JSON instructions should work.
|
|
111
|
+
|
|
112
|
+
**Anthropic**
|
|
113
|
+
|
|
114
|
+
```python
|
|
115
|
+
import anthropic
|
|
116
|
+
|
|
117
|
+
client = anthropic.Anthropic()
|
|
118
|
+
|
|
119
|
+
def llm(prompt: str) -> str:
|
|
120
|
+
msg = client.messages.create(model="claude-haiku-4-5", max_tokens=512,
|
|
121
|
+
messages=[{"role": "user", "content": prompt}])
|
|
122
|
+
return msg.content[0].text
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
**OpenAI or OpenRouter**
|
|
126
|
+
|
|
127
|
+
```python
|
|
128
|
+
from openai import OpenAI
|
|
129
|
+
|
|
130
|
+
client = OpenAI() # OpenRouter: OpenAI(base_url="https://openrouter.ai/api/v1", api_key=...)
|
|
131
|
+
|
|
132
|
+
def llm(prompt: str) -> str:
|
|
133
|
+
r = client.chat.completions.create(model="gpt-4o-mini", temperature=0,
|
|
134
|
+
messages=[{"role": "user", "content": prompt}])
|
|
135
|
+
return r.choices[0].message.content or ""
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
## Prompts
|
|
139
|
+
|
|
140
|
+
The extraction prompt is a plain text file, versioned and measured. `v2` is the default; `v1` is kept so you can see what changed and why.
|
|
141
|
+
|
|
142
|
+
```python
|
|
143
|
+
Detector(fields, llm, prompt="v1") # an older built-in version
|
|
144
|
+
Detector(fields, llm, prompt="my_prompt.txt") # your own template
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
v2 fixed five failure patterns found by the benchmark ("in October" read as last year, a year in a topic name read as a filter, and others), going from 37 to 45 of 45 correct on the sets used while writing it, and 13 of 15 on a fresh set. See [`docs/prompts.md`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/docs/prompts.md) for the before and after, the template placeholders, and how to score your own prompt.
|
|
148
|
+
|
|
149
|
+
## The classifier (Jev)
|
|
150
|
+
|
|
151
|
+
On by default. It reads `TYPESAFE_API_KEY`, or you pass the key:
|
|
152
|
+
|
|
153
|
+
```python
|
|
154
|
+
from whowhen_rag import Detector, JevClassifier
|
|
155
|
+
|
|
156
|
+
Detector(fields, llm, classifier=JevClassifier(api_key="...", threshold=0.5))
|
|
157
|
+
Detector(fields, llm, use_classifier=False) # regex gate only
|
|
158
|
+
Detector(fields, llm, classifier=my_classifier) # any function: query -> bool
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
Without a key, `whowhen-rag` logs one warning and carries on with the regex gate only; queries the regex misses then get no filter.
|
|
162
|
+
|
|
163
|
+
## Using the result
|
|
164
|
+
|
|
165
|
+
| | |
|
|
166
|
+
|---|---|
|
|
167
|
+
| `f.needs_filter`, `bool(f)` | any condition, sort or list request found |
|
|
168
|
+
| `f.conditions` | `Condition(column, op, value)`; op is `>=`, `<`, `=` or `in`; combine with AND |
|
|
169
|
+
| `f.sort` | `Sort(column, descending)` for "latest" or "oldest", else `None` |
|
|
170
|
+
| `f.cleaned_query` | the query without its time and person words, for semantic search |
|
|
171
|
+
| `f.list_only` | the query only asks to list documents ("pages Priya created this week"); you can skip semantic search |
|
|
172
|
+
| `f.to_sql(placeholder="%s")` | `(where_clause, params)`: columns quoted, values parameterised; `"?"` for sqlite |
|
|
173
|
+
| `f.order_by_sql()` | `ORDER BY "col" DESC`, or `""` |
|
|
174
|
+
| `f.to_chroma()` | a Chroma `where` dict (Chroma ranges need numbers: use `value_format="epoch"`) |
|
|
175
|
+
| `f.to_qdrant()` | a Qdrant filter dict: `models.Filter(**f.to_qdrant())` |
|
|
176
|
+
| `f.to_dict()` | plain dict, JSON-serialisable |
|
|
177
|
+
| `f.trace` | which gate fired, whether the classifier and LLM ran, errors |
|
|
178
|
+
|
|
179
|
+
Only need the yes or no? `detector.has_signal(query)` runs the cheap gates and never calls the LLM.
|
|
180
|
+
|
|
181
|
+
## Failure behaviour
|
|
182
|
+
|
|
183
|
+
Search should never break because of filtering. If the classifier or LLM fails, times out or returns something unusable, `detect()` returns "no filter" with the original query and logs a warning (logger `whowhen_rag`). The LLM's answer is validated: unknown roles, invalid dates and empty ranges are dropped, never guessed.
|
|
184
|
+
|
|
185
|
+
## Limitations
|
|
186
|
+
|
|
187
|
+
- English queries only (the regex gate and the prompt are English).
|
|
188
|
+
- Synchronous. Use `asyncio.to_thread` from async code.
|
|
189
|
+
- Four roles (created and updated, time and person). "My documents" needs the current user, which `whowhen-rag` does not know.
|
|
190
|
+
|
|
191
|
+
## Development
|
|
192
|
+
|
|
193
|
+
```bash
|
|
194
|
+
uv sync
|
|
195
|
+
uv run pytest # no network or API keys needed
|
|
196
|
+
uv run python benchmarks/gates.py # set TYPESAFE_API_KEY to include the classifier
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
## License
|
|
200
|
+
|
|
201
|
+
MIT
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# Security
|
|
2
|
+
|
|
3
|
+
Please report security problems privately through GitHub: **Security > Report a vulnerability** on this repository ([direct link](https://github.com/rahulbansalc6414/whowhen-rag/security/advisories/new)). Do not open a public issue.
|
|
4
|
+
|
|
5
|
+
Relevant areas include SQL produced by `Filters.to_sql()` (column names are quoted and values are always parameters), prompt injection through user queries that changes the extracted filters, and handling of API keys.
|
|
6
|
+
|
|
7
|
+
Only the latest release receives fixes.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Benchmark results
|
|
2
|
+
|
|
3
|
+
Append new runs; do not overwrite. All queries are synthetic (`tests/queries.py`, `benchmarks/extraction_cases.py`).
|
|
4
|
+
Raw per-query output for extraction runs is in `results/`.
|
|
5
|
+
|
|
6
|
+
## Gates (`benchmarks/gates.py`)
|
|
7
|
+
|
|
8
|
+
| Date | Commit | Classifier | Time (20) | Author (12) | Unusual wording (8) | Plain, no filter (20) |
|
|
9
|
+
|---|---|---|---|---|---|---|
|
|
10
|
+
| 2026-11-20 | cb236e2 | off | 20/20 regex | 12/12 regex | 0/8 | 20/20 stopped |
|
|
11
|
+
| 2026-11-20 | cb236e2 | Jev (threshold 0.5) | 20/20 regex | 12/12 regex | 8/8 Jev yes | 20/20 Jev no |
|
|
12
|
+
|
|
13
|
+
The regex patterns and these queries were written together, so the regex rows show coverage, not accuracy on unseen wording. Jev is the independent check: it never saw the query set.
|
|
14
|
+
|
|
15
|
+
## Extraction (`benchmarks/extraction.py`, Claude Haiku 4.5 via OpenRouter, temperature 0)
|
|
16
|
+
|
|
17
|
+
"Exact" = time ranges, people and sort all correct. An end date after today counts as open-ended.
|
|
18
|
+
|
|
19
|
+
| Date | Commit | Prompt | Set | Exact | list_only | Tokens in / out per call | Median latency |
|
|
20
|
+
|---|---|---|---|---|---|---|---|
|
|
21
|
+
| 2026-11-20 | cb236e2 | v1 | main (30) | 26/30 | 27/30 | 544 / 76 | 1.23 s |
|
|
22
|
+
| 2026-11-21 | cb236e2 | v1 | holdout (15) | 11/15 | 13/15 | 545 / 80 | 1.78 s |
|
|
23
|
+
| 2026-11-21 | a7d1ba0 | v2 | main (30) | 30/30 | 29/30 | 748 / 74 | 1.56 s |
|
|
24
|
+
| 2026-11-21 | a7d1ba0 | v2 | holdout (15) | 15/15 | 15/15 | 749 / 77 | 1.63 s |
|
|
25
|
+
| 2026-11-22 | b465710 | v2 | **fresh (15)** | **13/15** | 15/15 | 749 / 77 | 1.58 s |
|
|
26
|
+
|
|
27
|
+
Prompt v1 failures: a month without a year read as last year ("in October" -> 2025), "last month" read as the last 30 days, a year in a topic name read as a filter ("the 2025 pricing model", "2026 hiring plan"), a sort added when none was asked, and "past two weeks" ending at today (which drops today's documents).
|
|
28
|
+
Prompt v2 adds a rule for each of these. The holdout set was written before v2, but its v1 failures were seen while writing v2, so the v2 holdout score is not a clean held-out measurement.
|
|
29
|
+
|
|
30
|
+
The **fresh** set (15 queries) was written and labelled after v2 was final and run once, with no prompt change afterwards: the clean measurement for v2 is **13/15**. Failures: "since September" read as September 2025 (the month-without-a-year rule does not hold for "since"), and "what is Marco's opinion on the caching proposal" given a filter on documents by Marco (labelled no filter; arguable). Candidates for a v3, to be scored on another fresh set.
|
|
31
|
+
|
|
32
|
+
On 2026-11-21 the prompts moved from code into `src/whowhen_rag/prompts/v1.txt` and `v2.txt`. A check confirmed both produce byte-for-byte the same text as the versions measured above, so these numbers apply to the files. Run `benchmarks/extraction.py --prompt v1|v2|<file>` to reproduce.
|
|
33
|
+
|
|
34
|
+
Cost at list price (Haiku 4.5: $1 / MTok input, $5 / MTok output): about $0.0011 per extraction, about $1.10 per 1,000 queries that reach the LLM.
|