whowhen-rag 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. whowhen_rag-0.1.0/.github/ISSUE_TEMPLATE/bug_report.yml +34 -0
  2. whowhen_rag-0.1.0/.github/ISSUE_TEMPLATE/config.yml +5 -0
  3. whowhen_rag-0.1.0/.github/ISSUE_TEMPLATE/feature_request.yml +16 -0
  4. whowhen_rag-0.1.0/.github/workflows/publish.yml +44 -0
  5. whowhen_rag-0.1.0/.github/workflows/tests.yml +14 -0
  6. whowhen_rag-0.1.0/.gitignore +9 -0
  7. whowhen_rag-0.1.0/CONTRIBUTING.md +32 -0
  8. whowhen_rag-0.1.0/LICENSE +21 -0
  9. whowhen_rag-0.1.0/PKG-INFO +220 -0
  10. whowhen_rag-0.1.0/README.md +201 -0
  11. whowhen_rag-0.1.0/SECURITY.md +7 -0
  12. whowhen_rag-0.1.0/benchmarks/RESULTS.md +34 -0
  13. whowhen_rag-0.1.0/benchmarks/extraction.py +131 -0
  14. whowhen_rag-0.1.0/benchmarks/extraction_cases.py +95 -0
  15. whowhen_rag-0.1.0/benchmarks/gates.py +49 -0
  16. whowhen_rag-0.1.0/benchmarks/results/2026-11-20_extraction_claude-haiku-4.5_main_prompt-v1.json +263 -0
  17. whowhen_rag-0.1.0/benchmarks/results/2026-11-21_extraction_claude-haiku-4.5_holdout_prompt-v1.json +145 -0
  18. whowhen_rag-0.1.0/benchmarks/results/2026-11-21_extraction_claude-haiku-4.5_holdout_prompt-v2.json +137 -0
  19. whowhen_rag-0.1.0/benchmarks/results/2026-11-21_extraction_claude-haiku-4.5_main_prompt-v2.json +257 -0
  20. whowhen_rag-0.1.0/benchmarks/results/2026-11-22_extraction_claude-haiku-4.5_fresh_prompt-v2.json +142 -0
  21. whowhen_rag-0.1.0/docs/prompts.md +68 -0
  22. whowhen_rag-0.1.0/examples/quickstart.py +52 -0
  23. whowhen_rag-0.1.0/pyproject.toml +35 -0
  24. whowhen_rag-0.1.0/src/whowhen_rag/__init__.py +33 -0
  25. whowhen_rag-0.1.0/src/whowhen_rag/classifier.py +93 -0
  26. whowhen_rag-0.1.0/src/whowhen_rag/detector.py +164 -0
  27. whowhen_rag-0.1.0/src/whowhen_rag/extract.py +142 -0
  28. whowhen_rag-0.1.0/src/whowhen_rag/fields.py +76 -0
  29. whowhen_rag-0.1.0/src/whowhen_rag/filters.py +123 -0
  30. whowhen_rag-0.1.0/src/whowhen_rag/gates.py +67 -0
  31. whowhen_rag-0.1.0/src/whowhen_rag/prompts/v1.txt +31 -0
  32. whowhen_rag-0.1.0/src/whowhen_rag/prompts/v2.txt +38 -0
  33. whowhen_rag-0.1.0/tests/conftest.py +58 -0
  34. whowhen_rag-0.1.0/tests/queries.py +76 -0
  35. whowhen_rag-0.1.0/tests/test_classifier.py +44 -0
  36. whowhen_rag-0.1.0/tests/test_detector.py +203 -0
  37. whowhen_rag-0.1.0/tests/test_filters.py +46 -0
  38. whowhen_rag-0.1.0/tests/test_gates.py +25 -0
  39. whowhen_rag-0.1.0/tests/test_prompts.py +44 -0
  40. whowhen_rag-0.1.0/uv.lock +549 -0
@@ -0,0 +1,34 @@
1
+ name: Bug report
2
+ description: Wrong filters, a crash, or behaviour that does not match the docs
3
+ labels: [bug]
4
+ body:
5
+ - type: textarea
6
+ id: query
7
+ attributes:
8
+ label: Query and result
9
+ description: The query, the filters you got (`f.to_dict()`), and the filters you expected. Please replace any private names or data.
10
+ placeholder: |
11
+ Query: "docs Priya edited last month"
12
+ Got: ...
13
+ Expected: ...
14
+ validations:
15
+ required: true
16
+ - type: textarea
17
+ id: setup
18
+ attributes:
19
+ label: Setup
20
+ description: Your Field declarations, the LLM and model you pass as `llm=`, the prompt version, and whether the classifier is on.
21
+ validations:
22
+ required: true
23
+ - type: input
24
+ id: versions
25
+ attributes:
26
+ label: Versions
27
+ description: Output of `python -c "import whowhen_rag, sys; print(whowhen.__version__, sys.version)"`
28
+ validations:
29
+ required: true
30
+ - type: textarea
31
+ id: trace
32
+ attributes:
33
+ label: Trace or traceback
34
+ description: "`f.trace`, and any warning logged by the `whowhen_rag` logger or a full traceback."
@@ -0,0 +1,5 @@
1
+ blank_issues_enabled: true
2
+ contact_links:
3
+ - name: Report a security issue privately
4
+ url: https://github.com/rahulbansalc6414/whowhen-rag/security/advisories/new
5
+ about: Please do not open a public issue for security problems.
@@ -0,0 +1,16 @@
1
+ name: Feature request
2
+ description: A new option, store adapter, role, or language
3
+ labels: [enhancement]
4
+ body:
5
+ - type: textarea
6
+ id: problem
7
+ attributes:
8
+ label: What are you trying to do?
9
+ description: The kind of query or setup that whowhen-rag does not handle today, with example queries.
10
+ validations:
11
+ required: true
12
+ - type: textarea
13
+ id: proposal
14
+ attributes:
15
+ label: What would help?
16
+ description: The option or behaviour you have in mind, if any.
@@ -0,0 +1,44 @@
1
+ name: publish
2
+ # Release: bump version in pyproject.toml, commit, then `git tag vX.Y.Z && git push origin vX.Y.Z`.
3
+ # Uploads to PyPI with trusted publishing (no API token stored anywhere). One-time setup on
4
+ # pypi.org: Account > Publishing > add a pending publisher for owner rahulbansalc6414,
5
+ # repository whowhen-rag, workflow publish.yml, environment pypi.
6
+ on:
7
+ push:
8
+ tags: ["v*"]
9
+
10
+ jobs:
11
+ build:
12
+ runs-on: ubuntu-latest
13
+ steps:
14
+ - uses: actions/checkout@v4
15
+ - uses: astral-sh/setup-uv@v5
16
+ - name: Tag matches pyproject version
17
+ run: |
18
+ version=$(grep -m1 '^version = ' pyproject.toml | cut -d'"' -f2)
19
+ if [ "v$version" != "$GITHUB_REF_NAME" ]; then
20
+ echo "Tag $GITHUB_REF_NAME does not match pyproject version $version"; exit 1
21
+ fi
22
+ - run: uv sync
23
+ - run: uv run ruff check .
24
+ - run: uv run pytest -q
25
+ - run: uv build
26
+ - uses: actions/upload-artifact@v4
27
+ with:
28
+ name: dist
29
+ path: dist/
30
+
31
+ publish:
32
+ needs: build
33
+ runs-on: ubuntu-latest
34
+ environment:
35
+ name: pypi
36
+ url: https://pypi.org/project/whowhen-rag/
37
+ permissions:
38
+ id-token: write # trusted publishing
39
+ steps:
40
+ - uses: actions/download-artifact@v4
41
+ with:
42
+ name: dist
43
+ path: dist/
44
+ - uses: pypa/gh-action-pypi-publish@release/v1
@@ -0,0 +1,14 @@
1
+ name: tests
2
+ on: [push, pull_request]
3
+ jobs:
4
+ test:
5
+ runs-on: ubuntu-latest
6
+ strategy:
7
+ matrix:
8
+ python-version: ["3.10", "3.12", "3.13"]
9
+ steps:
10
+ - uses: actions/checkout@v4
11
+ - uses: astral-sh/setup-uv@v5
12
+ - run: uv sync --python ${{ matrix.python-version }}
13
+ - run: uv run ruff check .
14
+ - run: uv run pytest -q
@@ -0,0 +1,9 @@
1
+ .venv/
2
+ __pycache__/
3
+ *.pyc
4
+ dist/
5
+ build/
6
+ *.egg-info/
7
+ .pytest_cache/
8
+ .ruff_cache/
9
+ .env
@@ -0,0 +1,32 @@
1
+ # Contributing
2
+
3
+ Bug reports, example queries that whowhen-rag gets wrong, and pull requests are all welcome.
4
+
5
+ ## Set up
6
+
7
+ ```bash
8
+ git clone https://github.com/rahulbansalc6414/whowhen-rag.git
9
+ cd whowhen-rag
10
+ uv sync
11
+ uv run pytest # no network or API keys needed
12
+ uv run ruff check .
13
+ ```
14
+
15
+ ## Ground rules
16
+
17
+ - **Tests need no network or keys.** Use the fake LLM and classifier in `tests/conftest.py`.
18
+ - **Failures never break search.** A classifier or LLM problem returns "no filter" and logs a warning; keep it that way.
19
+ - **Synthetic data only.** Test and benchmark queries use made-up names and topics. Do not add real company, customer or personal data.
20
+ - **No em dashes** in docs or messages; use colons, commas or parentheses.
21
+
22
+ ## Changing the gates or the prompt
23
+
24
+ These change what the package returns, so they need numbers:
25
+
26
+ - **Regex gate** (`src/whowhen_rag/gates.py`): add the queries it should now catch to `tests/queries.py`, and check the plain questions still miss. Run `uv run python benchmarks/gates.py` (set `TYPESAFE_API_KEY` to include the classifier).
27
+ - **Prompt**: add a new file (`src/whowhen_rag/prompts/v3.txt`) rather than editing an existing version, and score it: `uv run python benchmarks/extraction.py --prompt v3` and `--set holdout`. See [docs/prompts.md](docs/prompts.md).
28
+ - Append the results to `benchmarks/RESULTS.md` with the date and commit. Do not overwrite earlier rows.
29
+
30
+ ## Releases
31
+
32
+ Maintainers release by bumping `version` in `pyproject.toml` (the only place it is set) and pushing a matching tag `vX.Y.Z`. The publish workflow builds and uploads to PyPI.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Rahul Bansal
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,220 @@
1
+ Metadata-Version: 2.5
2
+ Name: whowhen-rag
3
+ Version: 0.1.0
4
+ Summary: Detect time and author filters in RAG queries, cheaply: a free regex gate, a yes/no classifier, and your LLM only when a filter is likely.
5
+ Project-URL: Homepage, https://github.com/rahulbansalc6414/whowhen-rag
6
+ Project-URL: Issues, https://github.com/rahulbansalc6414/whowhen-rag/issues
7
+ Author: Rahul Bansal
8
+ License-Expression: MIT
9
+ License-File: LICENSE
10
+ Keywords: llm,metadata-filtering,query-understanding,rag,retrieval,vector-search
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Programming Language :: Python :: 3
14
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
15
+ Classifier: Topic :: Text Processing :: Linguistic
16
+ Requires-Python: >=3.10
17
+ Requires-Dist: typesafe-sdk>=0.7
18
+ Description-Content-Type: text/markdown
19
+
20
+ # whowhen-rag
21
+
22
+ **Find the "who" and "when" in a search query, and only pay for an LLM when there is one.**
23
+
24
+ `whowhen-rag` turns questions like *"what did Priya change in the pricing docs last week?"* into metadata filters for your RAG system:
25
+
26
+ ```python
27
+ f = detector.detect("what did Priya change in the pricing docs last week?")
28
+ f.conditions # [Condition('modified_on', '>=', '2026-11-09'),
29
+ # Condition('modified_on', '<', '2026-11-16'),
30
+ # Condition('last_editor', '=', 'priya@example.com')]
31
+ f.cleaned_query # 'pricing docs' (use this for the semantic search)
32
+ f.to_sql() # ('"modified_on" >= %s AND "modified_on" < %s AND "last_editor" = %s', [...])
33
+ ```
34
+
35
+ It works with any vector store or database, any LLM, and any column names.
36
+
37
+ ## Why it is cheap
38
+
39
+ Most questions sent to a knowledge base have no time or author condition at all. Framework tools for this (LangChain's `SelfQueryRetriever`, LlamaIndex auto-retrieval) send **every** query to an LLM to find out. `whowhen-rag` puts two cheap gates in front of the LLM:
40
+
41
+ ```
42
+ query
43
+ │
44
+ ├─ 1. Regex gate ......... free. "last week", "written by", "latest" ... ───────┐
45
+ │ miss │ hit
46
+ ├─ 2. Jev classifier ..... a yes/no answer, no text generation. ─────────────────┤ yes
47
+ │ no: return "no filter", no LLM call │
48
+ │ ▼
49
+ └─ 3. Your LLM ........... extracts dates, names, sort order. Only for likely filters.
50
+ ```
51
+
52
+ - **Gate 1** catches the common wording for free.
53
+ - **Gate 2** ([Jev](https://pypi.org/project/typesafe-sdk/)) catches unusual wording ("anything Priya put together on onboarding?"). It answers yes or no with a probability. It cannot pull out the dates or names, which is why step 3 exists.
54
+ - **Step 3** runs only when a gate says yes, so plain questions ("how does caching work?") never reach a paid model.
55
+
56
+ Every result carries a `trace` (`gate`, `classifier_called`, `llm_called`), so you can measure the savings on your own traffic.
57
+
58
+ ### Measured
59
+
60
+ On the synthetic benchmark sets (details and history in [`benchmarks/RESULTS.md`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/benchmarks/RESULTS.md)):
61
+
62
+ - **Plain questions:** 20 of 20 stopped by the gates. No LLM call.
63
+ - **Unusual wording the regex misses:** 8 of 8 caught by the Jev classifier.
64
+ - **Extraction** (Claude Haiku 4.5, prompt v2): **13 of 15** exactly right (dates, people, sort) on a fresh set written after the prompt was final, and 45 of 45 on the sets used while writing it. Misses on the fresh set: "since September" read as last year, and one arguable person filter.
65
+ - **Cost of an extraction:** about 750 input and 75 output tokens, about **$0.0011** at Haiku 4.5 list price (about $1.10 per 1,000 queries that reach the LLM), median 1.6 s.
66
+
67
+ These are small, hand-written sets. Measure on your own queries before relying on the numbers.
68
+
69
+ ## Install
70
+
71
+ ```bash
72
+ pip install whowhen-rag
73
+ ```
74
+
75
+ ## Quick start
76
+
77
+ ```python
78
+ from whowhen_rag import Detector, Field
79
+
80
+ detector = Detector(
81
+ fields=[
82
+ Field("created_on", role="created_at"),
83
+ Field("modified_on", role="updated_at"),
84
+ Field("owner_email", role="created_by", resolve=lookup_email), # name -> your stored value
85
+ ],
86
+ llm=my_llm, # any function: prompt string in, answer string out
87
+ )
88
+
89
+ f = detector.detect("pages Priya wrote this month about onboarding")
90
+ if f: # same as f.needs_filter
91
+ where, params = f.to_sql()
92
+ search_text = f.cleaned_query # "onboarding"
93
+ ```
94
+
95
+ Set `TYPESAFE_API_KEY` for the Jev classifier. A runnable example with sqlite is in [`examples/quickstart.py`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/examples/quickstart.py).
96
+
97
+ ## Your columns, your names
98
+
99
+ You declare your columns once. The LLM never sees them: it answers in terms of four **roles**, and `whowhen-rag` maps each role to your column.
100
+
101
+ | Role | Meaning |
102
+ |---|---|
103
+ | `created_at` | when the document was created |
104
+ | `updated_at` | when the document was last edited |
105
+ | `created_by` | who created or wrote it |
106
+ | `updated_by` | who last edited it |
107
+
108
+ Declare only the roles you have. If you have one date column, "changed last week" and "created last week" both use it.
109
+
110
+ ```python
111
+ Field(
112
+ "author_id", # your column or metadata key
113
+ role="created_by",
114
+ resolve=lambda name: users.find_id(name), # str, int, list of IDs, or None if unknown
115
+ )
116
+ Field(
117
+ "updated_ts",
118
+ role="updated_at",
119
+ value_format="epoch", # "date" (default), "datetime", "epoch", "epoch_ms"
120
+ description="when the ticket was last touched", # optional wording for the LLM
121
+ )
122
+ ```
123
+
124
+ - **`resolve`** turns a name from the query into what your column stores. Without it, the name is used as written. If it returns `None`, that condition is dropped and the name is listed in `f.trace.unresolved_people`.
125
+ - **`value_format`** matches how your dates are stored. Dates are midnight UTC.
126
+
127
+ ## Bring your own LLM
128
+
129
+ `llm` is any function that takes the prompt and returns the model's text. `whowhen-rag` writes the prompt and validates the answer. Tested with Claude Haiku 4.5; any model that follows JSON instructions should work.
130
+
131
+ **Anthropic**
132
+
133
+ ```python
134
+ import anthropic
135
+
136
+ client = anthropic.Anthropic()
137
+
138
+ def llm(prompt: str) -> str:
139
+ msg = client.messages.create(model="claude-haiku-4-5", max_tokens=512,
140
+ messages=[{"role": "user", "content": prompt}])
141
+ return msg.content[0].text
142
+ ```
143
+
144
+ **OpenAI or OpenRouter**
145
+
146
+ ```python
147
+ from openai import OpenAI
148
+
149
+ client = OpenAI() # OpenRouter: OpenAI(base_url="https://openrouter.ai/api/v1", api_key=...)
150
+
151
+ def llm(prompt: str) -> str:
152
+ r = client.chat.completions.create(model="gpt-4o-mini", temperature=0,
153
+ messages=[{"role": "user", "content": prompt}])
154
+ return r.choices[0].message.content or ""
155
+ ```
156
+
157
+ ## Prompts
158
+
159
+ The extraction prompt is a plain text file, versioned and measured. `v2` is the default; `v1` is kept so you can see what changed and why.
160
+
161
+ ```python
162
+ Detector(fields, llm, prompt="v1") # an older built-in version
163
+ Detector(fields, llm, prompt="my_prompt.txt") # your own template
164
+ ```
165
+
166
+ v2 fixed five failure patterns found by the benchmark ("in October" read as last year, a year in a topic name read as a filter, and others), going from 37 to 45 of 45 correct on the sets used while writing it, and 13 of 15 on a fresh set. See [`docs/prompts.md`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/docs/prompts.md) for the before and after, the template placeholders, and how to score your own prompt.
167
+
168
+ ## The classifier (Jev)
169
+
170
+ On by default. It reads `TYPESAFE_API_KEY`, or you pass the key:
171
+
172
+ ```python
173
+ from whowhen_rag import Detector, JevClassifier
174
+
175
+ Detector(fields, llm, classifier=JevClassifier(api_key="...", threshold=0.5))
176
+ Detector(fields, llm, use_classifier=False) # regex gate only
177
+ Detector(fields, llm, classifier=my_classifier) # any function: query -> bool
178
+ ```
179
+
180
+ Without a key, `whowhen-rag` logs one warning and carries on with the regex gate only; queries the regex misses then get no filter.
181
+
182
+ ## Using the result
183
+
184
+ | | |
185
+ |---|---|
186
+ | `f.needs_filter`, `bool(f)` | any condition, sort or list request found |
187
+ | `f.conditions` | `Condition(column, op, value)`; op is `>=`, `<`, `=` or `in`; combine with AND |
188
+ | `f.sort` | `Sort(column, descending)` for "latest" or "oldest", else `None` |
189
+ | `f.cleaned_query` | the query without its time and person words, for semantic search |
190
+ | `f.list_only` | the query only asks to list documents ("pages Priya created this week"); you can skip semantic search |
191
+ | `f.to_sql(placeholder="%s")` | `(where_clause, params)`: columns quoted, values parameterised; `"?"` for sqlite |
192
+ | `f.order_by_sql()` | `ORDER BY "col" DESC`, or `""` |
193
+ | `f.to_chroma()` | a Chroma `where` dict (Chroma ranges need numbers: use `value_format="epoch"`) |
194
+ | `f.to_qdrant()` | a Qdrant filter dict: `models.Filter(**f.to_qdrant())` |
195
+ | `f.to_dict()` | plain dict, JSON-serialisable |
196
+ | `f.trace` | which gate fired, whether the classifier and LLM ran, errors |
197
+
198
+ Only need the yes or no? `detector.has_signal(query)` runs the cheap gates and never calls the LLM.
199
+
200
+ ## Failure behaviour
201
+
202
+ Search should never break because of filtering. If the classifier or LLM fails, times out or returns something unusable, `detect()` returns "no filter" with the original query and logs a warning (logger `whowhen_rag`). The LLM's answer is validated: unknown roles, invalid dates and empty ranges are dropped, never guessed.
203
+
204
+ ## Limitations
205
+
206
+ - English queries only (the regex gate and the prompt are English).
207
+ - Synchronous. Use `asyncio.to_thread` from async code.
208
+ - Four roles (created and updated, time and person). "My documents" needs the current user, which `whowhen-rag` does not know.
209
+
210
+ ## Development
211
+
212
+ ```bash
213
+ uv sync
214
+ uv run pytest # no network or API keys needed
215
+ uv run python benchmarks/gates.py # set TYPESAFE_API_KEY to include the classifier
216
+ ```
217
+
218
+ ## License
219
+
220
+ MIT
@@ -0,0 +1,201 @@
1
+ # whowhen-rag
2
+
3
+ **Find the "who" and "when" in a search query, and only pay for an LLM when there is one.**
4
+
5
+ `whowhen-rag` turns questions like *"what did Priya change in the pricing docs last week?"* into metadata filters for your RAG system:
6
+
7
+ ```python
8
+ f = detector.detect("what did Priya change in the pricing docs last week?")
9
+ f.conditions # [Condition('modified_on', '>=', '2026-11-09'),
10
+ # Condition('modified_on', '<', '2026-11-16'),
11
+ # Condition('last_editor', '=', 'priya@example.com')]
12
+ f.cleaned_query # 'pricing docs' (use this for the semantic search)
13
+ f.to_sql() # ('"modified_on" >= %s AND "modified_on" < %s AND "last_editor" = %s', [...])
14
+ ```
15
+
16
+ It works with any vector store or database, any LLM, and any column names.
17
+
18
+ ## Why it is cheap
19
+
20
+ Most questions sent to a knowledge base have no time or author condition at all. Framework tools for this (LangChain's `SelfQueryRetriever`, LlamaIndex auto-retrieval) send **every** query to an LLM to find out. `whowhen-rag` puts two cheap gates in front of the LLM:
21
+
22
+ ```
23
+ query
24
+ │
25
+ ├─ 1. Regex gate ......... free. "last week", "written by", "latest" ... ───────┐
26
+ │ miss │ hit
27
+ ├─ 2. Jev classifier ..... a yes/no answer, no text generation. ─────────────────┤ yes
28
+ │ no: return "no filter", no LLM call │
29
+ │ ▼
30
+ └─ 3. Your LLM ........... extracts dates, names, sort order. Only for likely filters.
31
+ ```
32
+
33
+ - **Gate 1** catches the common wording for free.
34
+ - **Gate 2** ([Jev](https://pypi.org/project/typesafe-sdk/)) catches unusual wording ("anything Priya put together on onboarding?"). It answers yes or no with a probability. It cannot pull out the dates or names, which is why step 3 exists.
35
+ - **Step 3** runs only when a gate says yes, so plain questions ("how does caching work?") never reach a paid model.
36
+
37
+ Every result carries a `trace` (`gate`, `classifier_called`, `llm_called`), so you can measure the savings on your own traffic.
38
+
39
+ ### Measured
40
+
41
+ On the synthetic benchmark sets (details and history in [`benchmarks/RESULTS.md`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/benchmarks/RESULTS.md)):
42
+
43
+ - **Plain questions:** 20 of 20 stopped by the gates. No LLM call.
44
+ - **Unusual wording the regex misses:** 8 of 8 caught by the Jev classifier.
45
+ - **Extraction** (Claude Haiku 4.5, prompt v2): **13 of 15** exactly right (dates, people, sort) on a fresh set written after the prompt was final, and 45 of 45 on the sets used while writing it. Misses on the fresh set: "since September" read as last year, and one arguable person filter.
46
+ - **Cost of an extraction:** about 750 input and 75 output tokens, about **$0.0011** at Haiku 4.5 list price (about $1.10 per 1,000 queries that reach the LLM), median 1.6 s.
47
+
48
+ These are small, hand-written sets. Measure on your own queries before relying on the numbers.
49
+
50
+ ## Install
51
+
52
+ ```bash
53
+ pip install whowhen-rag
54
+ ```
55
+
56
+ ## Quick start
57
+
58
+ ```python
59
+ from whowhen_rag import Detector, Field
60
+
61
+ detector = Detector(
62
+ fields=[
63
+ Field("created_on", role="created_at"),
64
+ Field("modified_on", role="updated_at"),
65
+ Field("owner_email", role="created_by", resolve=lookup_email), # name -> your stored value
66
+ ],
67
+ llm=my_llm, # any function: prompt string in, answer string out
68
+ )
69
+
70
+ f = detector.detect("pages Priya wrote this month about onboarding")
71
+ if f: # same as f.needs_filter
72
+ where, params = f.to_sql()
73
+ search_text = f.cleaned_query # "onboarding"
74
+ ```
75
+
76
+ Set `TYPESAFE_API_KEY` for the Jev classifier. A runnable example with sqlite is in [`examples/quickstart.py`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/examples/quickstart.py).
77
+
78
+ ## Your columns, your names
79
+
80
+ You declare your columns once. The LLM never sees them: it answers in terms of four **roles**, and `whowhen-rag` maps each role to your column.
81
+
82
+ | Role | Meaning |
83
+ |---|---|
84
+ | `created_at` | when the document was created |
85
+ | `updated_at` | when the document was last edited |
86
+ | `created_by` | who created or wrote it |
87
+ | `updated_by` | who last edited it |
88
+
89
+ Declare only the roles you have. If you have one date column, "changed last week" and "created last week" both use it.
90
+
91
+ ```python
92
+ Field(
93
+ "author_id", # your column or metadata key
94
+ role="created_by",
95
+ resolve=lambda name: users.find_id(name), # str, int, list of IDs, or None if unknown
96
+ )
97
+ Field(
98
+ "updated_ts",
99
+ role="updated_at",
100
+ value_format="epoch", # "date" (default), "datetime", "epoch", "epoch_ms"
101
+ description="when the ticket was last touched", # optional wording for the LLM
102
+ )
103
+ ```
104
+
105
+ - **`resolve`** turns a name from the query into what your column stores. Without it, the name is used as written. If it returns `None`, that condition is dropped and the name is listed in `f.trace.unresolved_people`.
106
+ - **`value_format`** matches how your dates are stored. Dates are midnight UTC.
107
+
108
+ ## Bring your own LLM
109
+
110
+ `llm` is any function that takes the prompt and returns the model's text. `whowhen-rag` writes the prompt and validates the answer. Tested with Claude Haiku 4.5; any model that follows JSON instructions should work.
111
+
112
+ **Anthropic**
113
+
114
+ ```python
115
+ import anthropic
116
+
117
+ client = anthropic.Anthropic()
118
+
119
+ def llm(prompt: str) -> str:
120
+ msg = client.messages.create(model="claude-haiku-4-5", max_tokens=512,
121
+ messages=[{"role": "user", "content": prompt}])
122
+ return msg.content[0].text
123
+ ```
124
+
125
+ **OpenAI or OpenRouter**
126
+
127
+ ```python
128
+ from openai import OpenAI
129
+
130
+ client = OpenAI() # OpenRouter: OpenAI(base_url="https://openrouter.ai/api/v1", api_key=...)
131
+
132
+ def llm(prompt: str) -> str:
133
+ r = client.chat.completions.create(model="gpt-4o-mini", temperature=0,
134
+ messages=[{"role": "user", "content": prompt}])
135
+ return r.choices[0].message.content or ""
136
+ ```
137
+
138
+ ## Prompts
139
+
140
+ The extraction prompt is a plain text file, versioned and measured. `v2` is the default; `v1` is kept so you can see what changed and why.
141
+
142
+ ```python
143
+ Detector(fields, llm, prompt="v1") # an older built-in version
144
+ Detector(fields, llm, prompt="my_prompt.txt") # your own template
145
+ ```
146
+
147
+ v2 fixed five failure patterns found by the benchmark ("in October" read as last year, a year in a topic name read as a filter, and others), going from 37 to 45 of 45 correct on the sets used while writing it, and 13 of 15 on a fresh set. See [`docs/prompts.md`](https://github.com/rahulbansalc6414/whowhen-rag/blob/main/docs/prompts.md) for the before and after, the template placeholders, and how to score your own prompt.
148
+
149
+ ## The classifier (Jev)
150
+
151
+ On by default. It reads `TYPESAFE_API_KEY`, or you pass the key:
152
+
153
+ ```python
154
+ from whowhen_rag import Detector, JevClassifier
155
+
156
+ Detector(fields, llm, classifier=JevClassifier(api_key="...", threshold=0.5))
157
+ Detector(fields, llm, use_classifier=False) # regex gate only
158
+ Detector(fields, llm, classifier=my_classifier) # any function: query -> bool
159
+ ```
160
+
161
+ Without a key, `whowhen-rag` logs one warning and carries on with the regex gate only; queries the regex misses then get no filter.
162
+
163
+ ## Using the result
164
+
165
+ | | |
166
+ |---|---|
167
+ | `f.needs_filter`, `bool(f)` | any condition, sort or list request found |
168
+ | `f.conditions` | `Condition(column, op, value)`; op is `>=`, `<`, `=` or `in`; combine with AND |
169
+ | `f.sort` | `Sort(column, descending)` for "latest" or "oldest", else `None` |
170
+ | `f.cleaned_query` | the query without its time and person words, for semantic search |
171
+ | `f.list_only` | the query only asks to list documents ("pages Priya created this week"); you can skip semantic search |
172
+ | `f.to_sql(placeholder="%s")` | `(where_clause, params)`: columns quoted, values parameterised; `"?"` for sqlite |
173
+ | `f.order_by_sql()` | `ORDER BY "col" DESC`, or `""` |
174
+ | `f.to_chroma()` | a Chroma `where` dict (Chroma ranges need numbers: use `value_format="epoch"`) |
175
+ | `f.to_qdrant()` | a Qdrant filter dict: `models.Filter(**f.to_qdrant())` |
176
+ | `f.to_dict()` | plain dict, JSON-serialisable |
177
+ | `f.trace` | which gate fired, whether the classifier and LLM ran, errors |
178
+
179
+ Only need the yes or no? `detector.has_signal(query)` runs the cheap gates and never calls the LLM.
180
+
181
+ ## Failure behaviour
182
+
183
+ Search should never break because of filtering. If the classifier or LLM fails, times out or returns something unusable, `detect()` returns "no filter" with the original query and logs a warning (logger `whowhen_rag`). The LLM's answer is validated: unknown roles, invalid dates and empty ranges are dropped, never guessed.
184
+
185
+ ## Limitations
186
+
187
+ - English queries only (the regex gate and the prompt are English).
188
+ - Synchronous. Use `asyncio.to_thread` from async code.
189
+ - Four roles (created and updated, time and person). "My documents" needs the current user, which `whowhen-rag` does not know.
190
+
191
+ ## Development
192
+
193
+ ```bash
194
+ uv sync
195
+ uv run pytest # no network or API keys needed
196
+ uv run python benchmarks/gates.py # set TYPESAFE_API_KEY to include the classifier
197
+ ```
198
+
199
+ ## License
200
+
201
+ MIT
@@ -0,0 +1,7 @@
1
+ # Security
2
+
3
+ Please report security problems privately through GitHub: **Security > Report a vulnerability** on this repository ([direct link](https://github.com/rahulbansalc6414/whowhen-rag/security/advisories/new)). Do not open a public issue.
4
+
5
+ Relevant areas include SQL produced by `Filters.to_sql()` (column names are quoted and values are always parameters), prompt injection through user queries that changes the extracted filters, and handling of API keys.
6
+
7
+ Only the latest release receives fixes.
@@ -0,0 +1,34 @@
1
+ # Benchmark results
2
+
3
+ Append new runs; do not overwrite. All queries are synthetic (`tests/queries.py`, `benchmarks/extraction_cases.py`).
4
+ Raw per-query output for extraction runs is in `results/`.
5
+
6
+ ## Gates (`benchmarks/gates.py`)
7
+
8
+ | Date | Commit | Classifier | Time (20) | Author (12) | Unusual wording (8) | Plain, no filter (20) |
9
+ |---|---|---|---|---|---|---|
10
+ | 2026-11-20 | cb236e2 | off | 20/20 regex | 12/12 regex | 0/8 | 20/20 stopped |
11
+ | 2026-11-20 | cb236e2 | Jev (threshold 0.5) | 20/20 regex | 12/12 regex | 8/8 Jev yes | 20/20 Jev no |
12
+
13
+ The regex patterns and these queries were written together, so the regex rows show coverage, not accuracy on unseen wording. Jev is the independent check: it never saw the query set.
14
+
15
+ ## Extraction (`benchmarks/extraction.py`, Claude Haiku 4.5 via OpenRouter, temperature 0)
16
+
17
+ "Exact" = time ranges, people and sort all correct. An end date after today counts as open-ended.
18
+
19
+ | Date | Commit | Prompt | Set | Exact | list_only | Tokens in / out per call | Median latency |
20
+ |---|---|---|---|---|---|---|---|
21
+ | 2026-11-20 | cb236e2 | v1 | main (30) | 26/30 | 27/30 | 544 / 76 | 1.23 s |
22
+ | 2026-11-21 | cb236e2 | v1 | holdout (15) | 11/15 | 13/15 | 545 / 80 | 1.78 s |
23
+ | 2026-11-21 | a7d1ba0 | v2 | main (30) | 30/30 | 29/30 | 748 / 74 | 1.56 s |
24
+ | 2026-11-21 | a7d1ba0 | v2 | holdout (15) | 15/15 | 15/15 | 749 / 77 | 1.63 s |
25
+ | 2026-11-22 | b465710 | v2 | **fresh (15)** | **13/15** | 15/15 | 749 / 77 | 1.58 s |
26
+
27
+ Prompt v1 failures: a month without a year read as last year ("in October" -> 2025), "last month" read as the last 30 days, a year in a topic name read as a filter ("the 2025 pricing model", "2026 hiring plan"), a sort added when none was asked, and "past two weeks" ending at today (which drops today's documents).
28
+ Prompt v2 adds a rule for each of these. The holdout set was written before v2, but its v1 failures were seen while writing v2, so the v2 holdout score is not a clean held-out measurement.
29
+
30
+ The **fresh** set (15 queries) was written and labelled after v2 was final and run once, with no prompt change afterwards: the clean measurement for v2 is **13/15**. Failures: "since September" read as September 2025 (the month-without-a-year rule does not hold for "since"), and "what is Marco's opinion on the caching proposal" given a filter on documents by Marco (labelled no filter; arguable). Candidates for a v3, to be scored on another fresh set.
31
+
32
+ On 2026-11-21 the prompts moved from code into `src/whowhen_rag/prompts/v1.txt` and `v2.txt`. A check confirmed both produce byte-for-byte the same text as the versions measured above, so these numbers apply to the files. Run `benchmarks/extraction.py --prompt v1|v2|<file>` to reproduce.
33
+
34
+ Cost at list price (Haiku 4.5: $1 / MTok input, $5 / MTok output): about $0.0011 per extraction, about $1.10 per 1,000 queries that reach the LLM.