tunarag-python 0.3.0__tar.gz → 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/.gitignore +1 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/CHANGELOG.md +52 -0
- tunarag_python-0.4.0/LICENSE +21 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/PKG-INFO +153 -88
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/README.md +149 -86
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/pyproject.toml +11 -3
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/__init__.py +1 -1
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/dataset.py +19 -3
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/domain.py +5 -2
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/engine.py +134 -151
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/evaluators.py +18 -9
- tunarag_python-0.4.0/src/tunarag/integrations/google_adk.py +233 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/integrations/mlflow.py +24 -7
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/integrations/runnables.py +162 -31
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/objective.py +12 -2
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/result.py +23 -7
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/retry.py +50 -5
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/search.py +3 -1
- tunarag_python-0.4.0/src/tunarag/serialization.py +177 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/stopping.py +9 -2
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/store.py +3 -2
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/synthetic.py +36 -3
- tunarag_python-0.3.0/src/tunarag/integrations/google_adk.py +0 -144
- tunarag_python-0.3.0/src/tunarag/serialization.py +0 -78
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/cache.py +0 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/config.py +0 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/contracts.py +0 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/errors.py +0 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/integrations/__init__.py +0 -0
- {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/py.typed +0 -0
|
@@ -1,5 +1,57 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.4.0 - 2026-11-21
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
|
|
7
|
+
- Code audit fixes (`docs/audit-2026-11-21/`), each covered by `tests/test_audit_regressions.py`:
|
|
8
|
+
- A NaN or infinite metric or cost fails only its trial instead of crashing the study.
|
|
9
|
+
- Ctrl-C/cancelled and crashed (failed) studies can be resumed; interrupted trials re-run.
|
|
10
|
+
- Token usage is captured from every chat-model call (LangChain usage callback), including
|
|
11
|
+
string-output chains such as `create_retrieval_chain`; new `token_prices` option on all
|
|
12
|
+
adapters gives an estimated cost so `CostBudget` works.
|
|
13
|
+
- Unknown cost is reported as `unavailable`, not an exact $0.
|
|
14
|
+
- Usage from failed, timed-out, and cancelled attempts is recorded.
|
|
15
|
+
- Provider timeouts, connection errors, rate limits (429), and 408/5xx errors are retryable,
|
|
16
|
+
including OpenAI, httpx, LiteLLM, and Google GenAI exceptions and wrapped causes.
|
|
17
|
+
- Without an explicit `identity`, evaluator and adapter identities include digests of
|
|
18
|
+
scorer, factory, mapper, wrapped function, graph node, and ADK agent code, and adapters
|
|
19
|
+
warn. An explicit `identity` (now also accepted by `MetricEvaluator`) is the declared
|
|
20
|
+
version and replaces the code digests.
|
|
21
|
+
- RAGAS evaluations use per-call metric copies, fixing a race that silently dropped samples.
|
|
22
|
+
- Objectives mark partial-coverage metrics `estimated`; new `ObjectiveTerm.min_coverage`.
|
|
23
|
+
- Google ADK output includes `tool_responses` (retrieval evidence); new `events_mapper`;
|
|
24
|
+
error events keep their error code.
|
|
25
|
+
- MLflow reuses one run per study across processes and reopens it on resume.
|
|
26
|
+
- Custom search-space parameters (sample/accepts/describe) work with the optimizer.
|
|
27
|
+
- `TimeBudget` counts active time across resumes; consecutive `MaxFailures` ignores
|
|
28
|
+
cancelled trials; LLMSearch replay matches a live run's history.
|
|
29
|
+
- Dataset versions ignore row order; duplicate rows report their record numbers.
|
|
30
|
+
- Synthetic generation reports every failed document; `allow_partial=True` keeps the rest.
|
|
31
|
+
- Floats and numeric strings no longer share a hash.
|
|
32
|
+
- Failed attempts log their underlying cause to the `tunarag` logger.
|
|
33
|
+
- Documentation builds require `mkdocstrings` 1.x; 0.x fails with `pymdown-extensions` 13
|
|
34
|
+
(`Highlight.__init__() missing 'md'`).
|
|
35
|
+
|
|
36
|
+
### Migration
|
|
37
|
+
|
|
38
|
+
- The audit fixes change identity hashes (dataset versions, float encoding, adapter and evaluator
|
|
39
|
+
fingerprints). Studies started with earlier versions cannot be resumed (resume raises
|
|
40
|
+
`ResumeError`), and earlier evaluation-cache entries are not reused. Completed studies and
|
|
41
|
+
reports remain readable.
|
|
42
|
+
- Pass `identity=` to every adapter to silence the new warning and version your pipeline.
|
|
43
|
+
|
|
44
|
+
## 0.3.1 - 2026-10-09
|
|
45
|
+
|
|
46
|
+
- Replace both raw Mermaid README blocks with pre-rendered PNG diagrams compatible with PyPI.
|
|
47
|
+
Use public absolute documentation/image URLs and add the documentation metadata link.
|
|
48
|
+
|
|
49
|
+
- Add the complete MIT license with warranty and liability disclaimers to the repository,
|
|
50
|
+
wheel and source archive. Require the license file in distribution-content checks.
|
|
51
|
+
- Audit all six README illustrations: remove the redundant loop, replace misleading
|
|
52
|
+
adapter/dataset visuals with two-way framework control and untouched holdout illustrations,
|
|
53
|
+
and move the quality/cost/latency illustration to the objective explanation.
|
|
54
|
+
|
|
3
55
|
All notable changes to TunaRAG follow semantic versioning.
|
|
4
56
|
|
|
5
57
|
## 0.3.0 - 2026-10-09
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Shivam Shinde
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -1,12 +1,14 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: tunarag-python
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.4.0
|
|
4
4
|
Summary: Tune LangChain, LangGraph, and Google ADK RAG pipelines
|
|
5
|
+
Project-URL: Documentation, https://shivamshinde123.github.io/tunarag-python/
|
|
5
6
|
Project-URL: Repository, https://github.com/shivamshinde123/tunarag-python
|
|
6
7
|
Project-URL: Issues, https://github.com/shivamshinde123/tunarag-python/issues
|
|
7
8
|
Project-URL: Changelog, https://github.com/shivamshinde123/tunarag-python/blob/main/CHANGELOG.md
|
|
8
9
|
Author: Shivam Shinde
|
|
9
10
|
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
10
12
|
Keywords: evaluation,llm,optimization,rag,retrieval
|
|
11
13
|
Classifier: Development Status :: 3 - Alpha
|
|
12
14
|
Classifier: Intended Audience :: Developers
|
|
@@ -35,7 +37,7 @@ Provides-Extra: docs
|
|
|
35
37
|
Requires-Dist: mkdocs-include-markdown-plugin<8,>=7; extra == 'docs'
|
|
36
38
|
Requires-Dist: mkdocs-material<10,>=9.5; extra == 'docs'
|
|
37
39
|
Requires-Dist: mkdocs<2,>=1.6; extra == 'docs'
|
|
38
|
-
Requires-Dist: mkdocstrings[python]<1
|
|
40
|
+
Requires-Dist: mkdocstrings[python]<2,>=1; extra == 'docs'
|
|
39
41
|
Provides-Extra: frameworks
|
|
40
42
|
Requires-Dist: google-adk<3,>=2; extra == 'frameworks'
|
|
41
43
|
Requires-Dist: langchain<2,>=1; extra == 'frameworks'
|
|
@@ -100,24 +102,19 @@ indexes, provider clients, and credentials; TunaRAG owns the optimization workfl
|
|
|
100
102
|
|
|
101
103
|
**Documentation:** [shivamshinde123.github.io/tunarag-python](https://shivamshinde123.github.io/tunarag-python/)
|
|
102
104
|
|
|
105
|
+
**Usage handbook:** [Download the editable DOCX handbook](https://shivamshinde123.github.io/tunarag-python/TunaRAG_Python_Usage_Handbook.docx).
|
|
106
|
+
Start with RAG basics, then follow practical code examples for integrating your framework pipeline
|
|
107
|
+
and using advanced tuning features.
|
|
108
|
+
|
|
103
109
|
**Lightweight distribution:** pip installs runtime code and metadata, not tests, datasets,
|
|
104
110
|
or documentation images. Framework dependencies are optional extras. See
|
|
105
|
-
[package contents and release checks](
|
|
106
|
-
|
|
107
|
-
**Live integration checks:** [OpenRouter, RAGAS, and MLflow test plan](docs/live-integration-test-plan.md).
|
|
108
|
-
|
|
109
|
-
**End-to-end verification:** [Full-PDF campaigns, held-out quality checks, crash recovery,
|
|
110
|
-
and clean-wheel testing](docs/comprehensive-testing.md). Paid tests are explicit opt-ins;
|
|
111
|
-
ordinary CI never makes paid provider calls.
|
|
112
|
-
These opt-in tests include spending safeguards; normal tests never make paid calls.
|
|
111
|
+
[package contents and release checks](https://shivamshinde123.github.io/tunarag-python/package-distribution/).
|
|
113
112
|
|
|
114
|
-
**
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
See [mixed-topic validation](docs/multicorpus-validation.md) for cross-model QA,
|
|
120
|
-
simulated faults/load, and the unresolved adversarial-prompt failure.
|
|
113
|
+
**Contributor verification:** Test plans and campaign evidence remain in the
|
|
114
|
+
[repository documentation](https://github.com/shivamshinde123/tunarag-python/tree/main/docs).
|
|
115
|
+
Paid tests are explicit opt-ins; ordinary CI never makes paid provider calls.
|
|
116
|
+
The [mixed-topic validation report](https://github.com/shivamshinde123/tunarag-python/blob/main/docs/multicorpus-validation.md)
|
|
117
|
+
records known adversarial-prompt failures; tuning does not guarantee document safety.
|
|
121
118
|
|
|
122
119
|
## Comprehensive Usage Guide
|
|
123
120
|
|
|
@@ -154,27 +151,12 @@ TunaRAG compares configurations of an **existing** RAG pipeline. You provide the
|
|
|
154
151
|
evaluation dataset, a search space, evaluators, and an objective. TunaRAG schedules trials, isolates
|
|
155
152
|
failures, persists state, and returns an inspectable result.
|
|
156
153
|
|
|
157
|
-
|
|
158
|
-
flowchart LR
|
|
159
|
-
D[Versioned dataset] --> O[Optimizer]
|
|
160
|
-
S[Search strategy] -->|Candidate| O
|
|
161
|
-
O -->|Candidate + example| A[RAG adapter]
|
|
162
|
-
A -->|Output + usage| E[Evaluators]
|
|
163
|
-
E -->|Metrics| J[Objective]
|
|
164
|
-
J -->|Benefit score| O
|
|
165
|
-
O -->|Observe| S
|
|
166
|
-
O --> DB[(SQLite store)]
|
|
167
|
-
O <--> C[(Evaluation cache)]
|
|
168
|
-
DB --> CB[Post-commit callbacks]
|
|
169
|
-
O --> R[OptimizationResult]
|
|
170
|
-
```
|
|
154
|
+

|
|
171
155
|
|
|
172
156
|
A **study** contains ordered **trials**. Each trial evaluates one complete candidate configuration
|
|
173
157
|
against every dataset example. Same-named sample metrics are aggregated into trial metrics, then the
|
|
174
158
|
objective converts those metrics into one larger-is-better score.
|
|
175
159
|
|
|
176
|
-

|
|
177
|
-
|
|
178
160
|
TunaRAG does not create or host a RAG application, provision indexes, manage provider credentials,
|
|
179
161
|
or expose a hosted API. Your application keeps ownership of those concerns.
|
|
180
162
|
|
|
@@ -254,7 +236,9 @@ dataset = EvaluationDataset.from_records(
|
|
|
254
236
|
)
|
|
255
237
|
optimizer = Optimizer(
|
|
256
238
|
adapter=LangChainAdapter(
|
|
257
|
-
factory=build_chain,
|
|
239
|
+
factory=build_chain,
|
|
240
|
+
input_mapper=lambda example, candidate: example.query,
|
|
241
|
+
identity={"quickstart": "offline-langchain", "revision": 1},
|
|
258
242
|
),
|
|
259
243
|
strategy=RandomSearch(SearchSpace({"top_k": IntegerRange(1, 2)}), seed=7),
|
|
260
244
|
evaluators=(
|
|
@@ -295,10 +279,13 @@ from langchain_core.runnables import RunnableLambda
|
|
|
295
279
|
from tunarag import LangChainAdapter
|
|
296
280
|
|
|
297
281
|
chain = RunnableLambda(lambda inputs: {"answer": inputs["question"]})
|
|
298
|
-
adapter = LangChainAdapter(chain)
|
|
282
|
+
adapter = LangChainAdapter(chain, identity={"chain": "echo", "revision": 1})
|
|
299
283
|
```
|
|
300
284
|
|
|
301
|
-

|
|
286
|
+
|
|
287
|
+
The adapter connects two responsibilities; it does not move your RAG into TunaRAG.
|
|
288
|
+
Your framework must consume the candidate settings for tuning to change its behavior.
|
|
302
289
|
|
|
303
290
|
Candidate parameters go into framework configuration or initial ADK session state. A parameter
|
|
304
291
|
only changes behavior if the framework consumes it. For model, prompt, retriever, or index choices
|
|
@@ -316,8 +303,14 @@ adapter = LangChainAdapter(
|
|
|
316
303
|
```
|
|
317
304
|
|
|
318
305
|
Include stable corpus, index, model, prompt, and mapper revisions in `identity` for cache and resume
|
|
319
|
-
compatibility. Never include credentials.
|
|
320
|
-
|
|
306
|
+
compatibility. Never include credentials. When you pass `identity`, it is the pipeline's
|
|
307
|
+
declared version: TunaRAG uses it (plus the names of your factory and mappers) and does not
|
|
308
|
+
fingerprint code, so bump it whenever behavior changes. An adapter without `identity` emits a
|
|
309
|
+
`UserWarning` and is identified by code structure instead (factory and mapper code, wrapped
|
|
310
|
+
`RunnableLambda` functions, LangGraph node functions, chain steps, ADK agent instruction, model and
|
|
311
|
+
tools), which cannot see prompt-text, index, or model-setting changes. Framework factories and
|
|
312
|
+
mapping callbacks remain normal Python configuration code; retrieval and generation execute through
|
|
313
|
+
the framework.
|
|
321
314
|
|
|
322
315
|
## LangChain integration
|
|
323
316
|
|
|
@@ -388,14 +381,26 @@ adapter = LangChainAdapter(
|
|
|
388
381
|
output_tokens=output["usage"]["output_tokens"],
|
|
389
382
|
),
|
|
390
383
|
),
|
|
384
|
+
identity={"chain": "support-rag", "version": 2},
|
|
391
385
|
)
|
|
392
386
|
```
|
|
393
387
|
|
|
394
|
-
|
|
395
|
-
|
|
388
|
+
By default the adapter attaches LangChain's `UsageMetadataCallbackHandler` to the run, so token
|
|
389
|
+
usage is captured from every chat-model call inside the runnable, including chains such as
|
|
390
|
+
`create_retrieval_chain` whose answer is a plain string. If no callback usage is seen (for example
|
|
391
|
+
nested calls on Python 3.10 without an explicit `config`), it falls back to discovering standard
|
|
392
|
+
`usage_metadata` in mappings, sequences, and message objects. Token records keep the component
|
|
393
|
+
name `langchain.message` (`langgraph.message` for graphs); the reported model name is used to look
|
|
394
|
+
up `token_prices`. The adapter always adds `langchain.runnable` latency. Set
|
|
396
395
|
`capture_message_usage=False` to disable automatic token extraction. A custom `usage_mapper` takes
|
|
397
396
|
precedence.
|
|
398
397
|
|
|
398
|
+
Providers do not report dollar cost to LangChain, so cost is **unavailable** unless you price it.
|
|
399
|
+
Pass `token_prices={"<model name>": (usd_per_1m_input, usd_per_1m_output)}` (keys match the
|
|
400
|
+
model name the provider reports, e.g. `"openai/gpt-4.1-mini"` through OpenRouter); priced records
|
|
401
|
+
get an `estimated` cost, which `CostBudget` counts. The same option exists on
|
|
402
|
+
`LangGraphAdapter` and `GoogleADKAdapter`.
|
|
403
|
+
|
|
399
404
|
For a complete provider-backed pipeline—from documents and embeddings through optimization and
|
|
400
405
|
reports—run
|
|
401
406
|
[`examples/langchain_full_rag.py`](https://github.com/shivamshinde123/tunarag-python/blob/main/examples/langchain_full_rag.py).
|
|
@@ -447,6 +452,7 @@ adapter = LangGraphAdapter(
|
|
|
447
452
|
},
|
|
448
453
|
config_mapper=lambda candidate: None,
|
|
449
454
|
output_mapper=lambda state: state["answer"],
|
|
455
|
+
identity={"graph": "qa", "revision": 2},
|
|
450
456
|
)
|
|
451
457
|
```
|
|
452
458
|
|
|
@@ -462,18 +468,20 @@ agent tools, corpus retrieval, and runner ownership remain with the application.
|
|
|
462
468
|
|
|
463
469
|
Use `factory=lambda candidate: build_runner(candidate)` when tuning settings such as a model or
|
|
464
470
|
retrieval tool that require constructing an agent. `state_mapper` can map candidates to agent state;
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
471
|
+
By default the output is `{"answer": final_text, "tool_responses": [{"name": ..., "response": ...}]}`:
|
|
472
|
+
tool results such as retrieved passages are collected from the whole run, so retrieval metrics and
|
|
473
|
+
RAGAS can see the evidence. `output_mapper` receives only the final ADK event; `events_mapper`
|
|
474
|
+
receives every event of the run (use one or the other) for custom outputs. Usage captures reported
|
|
475
|
+
model token counts and runner latency; pass `token_prices` (keyed by the event's `model_version`)
|
|
476
|
+
for an estimated cost. Missing final responses and ADK error events fail the trial rather than
|
|
477
|
+
becoming empty answers; the trial records the ADK error code, and the provider's error message is
|
|
478
|
+
logged to the console (not persisted).
|
|
469
479
|
|
|
470
480
|
See the executable [ADK example](https://github.com/shivamshinde123/tunarag-python/blob/main/examples/google_adk_tuning.py)
|
|
471
481
|
and [integration reference](https://github.com/shivamshinde123/tunarag-python/blob/main/docs/reference/integrations.md).
|
|
472
482
|
|
|
473
483
|
## Evaluation datasets
|
|
474
484
|
|
|
475
|
-

|
|
476
|
-
|
|
477
485
|
### Canonical model
|
|
478
486
|
|
|
479
487
|
```python
|
|
@@ -523,8 +531,10 @@ csv_dataset = EvaluationDataset.from_csv("evaluation.csv")
|
|
|
523
531
|
```
|
|
524
532
|
|
|
525
533
|
JSON contains an array; JSONL contains one object per nonblank line. CSV collection and metadata
|
|
526
|
-
cells are JSON-encoded. Missing IDs are deterministically derived
|
|
527
|
-
|
|
534
|
+
cells are JSON-encoded. Missing IDs are deterministically derived from the record's content, so two
|
|
535
|
+
identical rows without IDs are reported as a duplicate (with both record numbers). Every
|
|
536
|
+
`EvaluationDataset` has a content-derived `version` used by caches and resume validation; it does
|
|
537
|
+
not depend on row order.
|
|
528
538
|
|
|
529
539
|
### Deterministic splitting
|
|
530
540
|
|
|
@@ -542,6 +552,13 @@ optimization_data = split_dataset.select(DatasetSplit.OPTIMIZE)
|
|
|
542
552
|
Splits are hash-based and stable across input order. Group splitting keeps related examples
|
|
543
553
|
together. `select()` rejects an empty split.
|
|
544
554
|
|
|
555
|
+

|
|
556
|
+
|
|
557
|
+
Pass only the selected optimization subset to the tuning study. Use validation data for
|
|
558
|
+
development decisions, and evaluate the frozen winner on the untouched test subset.
|
|
559
|
+
TunaRAG does not automatically enforce this separation or run the final held-out evaluation;
|
|
560
|
+
calling `optimize()` with the test set would use it for parameter selection and leak test data.
|
|
561
|
+
|
|
545
562
|
For custom examples, use a user-managed version:
|
|
546
563
|
|
|
547
564
|
```python
|
|
@@ -604,11 +621,14 @@ generation_usage = generated.usage
|
|
|
604
621
|
|
|
605
622
|
`SourceSpan` is a zero-based half-open range. TunaRAG validates spans, confidence, cardinality,
|
|
606
623
|
source IDs, and usage. Generated examples carry provenance and `synthetic=True`. Review them before
|
|
607
|
-
using them as gold data. Generation usage is separate from optimization usage.
|
|
624
|
+
using them as gold data. Generation usage is separate from optimization usage. Every document is
|
|
625
|
+
attempted; if some fail, the error lists each failed document and its cause. Pass
|
|
626
|
+
`allow_partial=True` to keep the successful documents instead (failed IDs are in
|
|
627
|
+
`generated.failed_document_ids`).
|
|
608
628
|
|
|
609
629
|
## Search spaces and strategies
|
|
610
630
|
|
|
611
|
-

|
|
612
632
|
|
|
613
633
|
### Typed space
|
|
614
634
|
|
|
@@ -719,7 +739,13 @@ class RetrievalEvaluator:
|
|
|
719
739
|
|
|
720
740
|
`MetricEvaluator` accepts sync or async scorers. `None`, NaN, and infinity become unavailable;
|
|
721
741
|
nonnumeric values raise `EvaluationError`. Coverage must be `[0, 1]`. `tunarag.objective` is a
|
|
722
|
-
reserved metric name.
|
|
742
|
+
reserved metric name. `MetricValue` rejects NaN and infinity, so a custom evaluator that produces
|
|
743
|
+
one fails that trial (the study continues); return `None` for "no value".
|
|
744
|
+
|
|
745
|
+
Without `identity=`, a `MetricEvaluator`'s identity includes a digest of its scorer's code, so
|
|
746
|
+
editing a scorer, or swapping one lambda for another, invalidates cached scores. Pass `identity=`
|
|
747
|
+
to declare the scorer's version yourself, for example when its behavior depends on outside state
|
|
748
|
+
(files, models, thresholds read at call time); the code digest is then not used.
|
|
723
749
|
|
|
724
750
|
### RAGAS
|
|
725
751
|
|
|
@@ -742,12 +768,20 @@ ragas_evaluator = RagasEvaluator(
|
|
|
742
768
|
)
|
|
743
769
|
```
|
|
744
770
|
|
|
745
|
-
TunaRAG constructs one RAGAS `SingleTurnSample` and calls the async evaluation API.
|
|
746
|
-
the
|
|
747
|
-
|
|
771
|
+
TunaRAG constructs one RAGAS `SingleTurnSample` and calls the async evaluation API. Each call uses
|
|
772
|
+
its own copies of the metric objects, so concurrent samples cannot detach each other's judge LLM.
|
|
773
|
+
Names receive the configured prefix. Mapping and RAGAS failures become `EvaluationError`; with
|
|
774
|
+
`raise_exceptions=False`, a sample RAGAS could not score becomes unavailable and lowers the
|
|
775
|
+
metric's coverage. Use `ObjectiveTerm(..., min_coverage=1.0)` for judge metrics that must cover
|
|
776
|
+
every sample. Credentials and provider configuration remain owned by your RAGAS components.
|
|
748
777
|
|
|
749
778
|
## Objectives
|
|
750
779
|
|
|
780
|
+

|
|
781
|
+
|
|
782
|
+
Your objective weights define the tradeoff; the winning configuration is best among the
|
|
783
|
+
evaluated trials under that objective, not a guarantee of the globally best RAG.
|
|
784
|
+
|
|
751
785
|
```python
|
|
752
786
|
from tunarag import Objective, ObjectiveDirection, ObjectiveTerm
|
|
753
787
|
|
|
@@ -775,7 +809,10 @@ objective = Objective(
|
|
|
775
809
|
Bounded metrics are normalized and clamped to `[0, 1]`; bounded minimize terms become
|
|
776
810
|
`1 - normalized`. Unbounded maximize terms use raw values and unbounded minimize terms use negative
|
|
777
811
|
values. Weights are contributions and are not automatically normalized. A missing or unavailable
|
|
778
|
-
required metric makes the objective unavailable
|
|
812
|
+
required metric makes the objective unavailable, and so does a metric whose coverage (the fraction
|
|
813
|
+
of samples that produced it) is below the term's `min_coverage` (default `0.0`). A metric measured
|
|
814
|
+
on only some samples makes the objective `estimated`. An unavailable objective fails the trial, not
|
|
815
|
+
the study.
|
|
779
816
|
|
|
780
817
|
Usage does not automatically become an objective metric. Emit `cost_usd` or `latency_seconds` from
|
|
781
818
|
an evaluator when you want those values in the objective.
|
|
@@ -826,6 +863,12 @@ cache_policy = "read_write"
|
|
|
826
863
|
|
|
827
864
|
Cache policies: `off`, `read_only`, `write_only`, `read_write`. Settings contain no provider keys.
|
|
828
865
|
|
|
866
|
+
`Optimizer` reads the limits, concurrency, retry, timeout, and cache-policy settings. The path
|
|
867
|
+
settings and `seed` are resolved values for your application to use: TunaRAG does not open
|
|
868
|
+
`db_path` or `cache_dir` by itself (pass `SQLiteStore(settings.db_path)` and a `SQLiteCache`, as
|
|
869
|
+
below), does not write to `artifact_dir`, and does not seed strategies from `seed` (pass
|
|
870
|
+
`RandomSearch(space, seed=settings.seed)`).
|
|
871
|
+
|
|
829
872
|
## Running optimization
|
|
830
873
|
|
|
831
874
|
Use async in async programs:
|
|
@@ -912,14 +955,16 @@ Stop conditions are OR-composed:
|
|
|
912
955
|
|---|---|
|
|
913
956
|
| `MaxTrials(limit)` | Completed trial count reaches the limit |
|
|
914
957
|
| `CostBudget(limit, count_estimated=True)` | Exact plus optional estimated cost reaches limit |
|
|
915
|
-
| `TimeBudget(seconds)` |
|
|
958
|
+
| `TimeBudget(seconds)` | Active run time (summed across resumes) reaches limit |
|
|
916
959
|
| `TargetScore(target)` | Any benefit-oriented objective reaches target |
|
|
917
960
|
| `MaxFailures(limit, consecutive=False)` | Total or trailing consecutive failures reach limit |
|
|
918
961
|
| `NoImprovement(patience, min_delta=0, warmup_trials=1)` | Best score stops improving |
|
|
919
962
|
|
|
920
963
|
Always include a hard bound. Conditions are checked between trial batches, so active trials can
|
|
921
964
|
finish after a cost, time, score, failure, or stagnation threshold is reached. Use `concurrency=1`
|
|
922
|
-
for strict between-trial stopping.
|
|
965
|
+
for strict between-trial stopping. `CostBudget` only sees priced usage: give your adapter
|
|
966
|
+
`token_prices` (or a `usage_mapper` that sets `cost`), otherwise cost is unavailable and the budget
|
|
967
|
+
never triggers. Usage from attempts that fail or time out still counts.
|
|
923
968
|
|
|
924
969
|
```python
|
|
925
970
|
from tunarag import RetryPolicy
|
|
@@ -933,9 +978,14 @@ policy = RetryPolicy(
|
|
|
933
978
|
)
|
|
934
979
|
```
|
|
935
980
|
|
|
936
|
-
Three retries means four total attempts. Timeouts
|
|
937
|
-
|
|
938
|
-
|
|
981
|
+
Three retries means four total attempts. Timeouts, connection errors, rate limits (HTTP 429), and
|
|
982
|
+
transient server errors (408 and 5xx) are retryable, including provider SDK exceptions (OpenAI,
|
|
983
|
+
httpx, LiteLLM, Google GenAI, ...) recognized by type name or HTTP status without importing those
|
|
984
|
+
SDKs, and wrapped causes. A `TunaRAGError` retries when `retryable=True` or when it wraps such a
|
|
985
|
+
transient cause; other exceptions fail that trial. Backoff is deterministic and capped. Attempt
|
|
986
|
+
usage is append-only, including failed, timed-out, and cancelled attempts. Each failed attempt is
|
|
987
|
+
logged to the `tunarag` logger with its underlying cause; persisted errors keep only the stage and
|
|
988
|
+
exception type (plus TunaRAG's own message), so third-party text never lands in SQLite or reports.
|
|
939
989
|
|
|
940
990
|
`concurrency` bounds simultaneous trials; `sample_concurrency` bounds samples inside each trial.
|
|
941
991
|
Their product approximates maximum concurrent adapter calls. Completion may be concurrent, but
|
|
@@ -961,8 +1011,9 @@ optimizer = Optimizer(
|
|
|
961
1011
|
)
|
|
962
1012
|
```
|
|
963
1013
|
|
|
964
|
-
Identity includes dataset version, candidate, adapter, and evaluators
|
|
965
|
-
|
|
1014
|
+
Identity includes dataset version, candidate, adapter, and evaluators (including digests of
|
|
1015
|
+
factory, mapper, and scorer code; see `identity` above). A hit skips adapter and evaluator execution
|
|
1016
|
+
but still calculates the objective and records the trial.
|
|
966
1017
|
|
|
967
1018
|
Direct API:
|
|
968
1019
|
|
|
@@ -985,7 +1036,7 @@ labels are metadata, not encryption.
|
|
|
985
1036
|
|
|
986
1037
|
## SQLite persistence and resume
|
|
987
1038
|
|
|
988
|
-

|
|
989
1040
|
|
|
990
1041
|
```python
|
|
991
1042
|
from pathlib import Path
|
|
@@ -1005,8 +1056,11 @@ resumed = await resumed_optimizer.aresume(dataset, study_id=result.study_id)
|
|
|
1005
1056
|
|
|
1006
1057
|
Resume validates semantic configuration and dataset identity, returns completed studies without
|
|
1007
1058
|
provider execution, abandons expired leases, replays committed strategy history, continues
|
|
1008
|
-
unfinished work, and schedules new candidates.
|
|
1009
|
-
|
|
1059
|
+
unfinished work, and schedules new candidates. Interrupted (Ctrl-C/cancelled) and crashed (failed)
|
|
1060
|
+
studies resume too: their in-flight trials are re-run. A cancellation releases its attempts at
|
|
1061
|
+
once; after a hard process kill, attempts stay leased until `trial_timeout_seconds + 60` passes,
|
|
1062
|
+
and resuming earlier raises a retryable `ResumeError`. Unknown or incompatible studies raise
|
|
1063
|
+
`ResumeError`.
|
|
1010
1064
|
|
|
1011
1065
|
Read persisted state:
|
|
1012
1066
|
|
|
@@ -1044,18 +1098,7 @@ Events cover study, trial, and attempt lifecycles plus callback failures. Handle
|
|
|
1044
1098
|
types defensively. Callback failures never roll back committed work; TunaRAG records them as durable
|
|
1045
1099
|
events. Make callbacks idempotent.
|
|
1046
1100
|
|
|
1047
|
-
|
|
1048
|
-
sequenceDiagram
|
|
1049
|
-
participant O as Optimizer
|
|
1050
|
-
participant S as SQLite
|
|
1051
|
-
participant C as Callback
|
|
1052
|
-
O->>S: Commit terminal trial and event
|
|
1053
|
-
S-->>O: Committed
|
|
1054
|
-
O->>C: on_event(event)
|
|
1055
|
-
alt callback fails
|
|
1056
|
-
O->>S: Append callback.failed
|
|
1057
|
-
end
|
|
1058
|
-
```
|
|
1101
|
+

|
|
1059
1102
|
|
|
1060
1103
|
MLflow is an optional post-commit projection; SQLite remains authoritative:
|
|
1061
1104
|
|
|
@@ -1078,14 +1121,13 @@ optimizer = Optimizer(
|
|
|
1078
1121
|
)
|
|
1079
1122
|
```
|
|
1080
1123
|
|
|
1081
|
-
One MLflow run is created per study
|
|
1124
|
+
One MLflow run is created per study, also across processes: a resumed study finds its run by the
|
|
1125
|
+
`tunarag.study_id` tag and reopens it. Candidate parameters, committed metrics, statuses, and event
|
|
1082
1126
|
position are mirrored. An MLflow outage does not invalidate local results. Automatic external
|
|
1083
|
-
reconciliation is not included
|
|
1127
|
+
reconciliation of missed events is not included.
|
|
1084
1128
|
|
|
1085
1129
|
## Results and reports
|
|
1086
1130
|
|
|
1087
|
-

|
|
1088
|
-
|
|
1089
1131
|
`OptimizationResult` exposes:
|
|
1090
1132
|
|
|
1091
1133
|
```python
|
|
@@ -1129,8 +1171,11 @@ result.write_json("artifacts/result.json")
|
|
|
1129
1171
|
result.write_csv("artifacts/trials.csv")
|
|
1130
1172
|
```
|
|
1131
1173
|
|
|
1132
|
-
Reports are deterministic and redacted. Usage separates exact and estimated cost.
|
|
1133
|
-
|
|
1174
|
+
Reports are deterministic and redacted. Usage separates exact and estimated cost; `usage.status`
|
|
1175
|
+
says how well cost is known: `exact` (every model call has an exact cost), `estimated` (some are
|
|
1176
|
+
estimated or unpriced), or `unavailable` (no cost reported, never shown as an exact $0). A
|
|
1177
|
+
completed result may have no best candidate if every trial failed or every objective was
|
|
1178
|
+
unavailable.
|
|
1134
1179
|
|
|
1135
1180
|
## Errors, privacy, and secrets
|
|
1136
1181
|
|
|
@@ -1206,7 +1251,9 @@ class EvenInteger:
|
|
|
1206
1251
|
}
|
|
1207
1252
|
```
|
|
1208
1253
|
|
|
1209
|
-
|
|
1254
|
+
Custom parameters are identified for cache and resume by their type and `describe()` output, so
|
|
1255
|
+
keep `describe()` complete and deterministic. Provider descriptions and component fingerprints must
|
|
1256
|
+
not expose secrets.
|
|
1210
1257
|
|
|
1211
1258
|
## Production checklist
|
|
1212
1259
|
|
|
@@ -1225,7 +1272,7 @@ Provider descriptions and component fingerprints must not expose secrets.
|
|
|
1225
1272
|
|
|
1226
1273
|
## Limitations and glossary
|
|
1227
1274
|
|
|
1228
|
-
V0.
|
|
1275
|
+
V0.4 does not provide a hosted service/dashboard, automatic RAG construction, vector-database
|
|
1229
1276
|
provisioning, distributed workers, Pareto optimization, Bayesian/evolutionary search, encrypted
|
|
1230
1277
|
payloads, MLflow reconciliation, provider pricing management, or a TypeScript implementation.
|
|
1231
1278
|
LangChain, LangGraph, and Google ADK 2.x are official integrations. Additional frameworks require a future first-party integration; custom RAG adapters are unsupported.
|
|
@@ -1245,6 +1292,24 @@ LangChain, LangGraph, and Google ADK 2.x are official integrations. Additional f
|
|
|
1245
1292
|
| Trial | One candidate evaluated over the whole dataset |
|
|
1246
1293
|
| Usage | Tokens, cost, latency, confidence, and pricing provenance |
|
|
1247
1294
|
|
|
1248
|
-
For deeper contracts and architecture, see [
|
|
1249
|
-
[
|
|
1250
|
-
[
|
|
1295
|
+
For deeper contracts and architecture, see the [API specification](https://shivamshinde123.github.io/tunarag-python/api-specification/),
|
|
1296
|
+
[system architecture](https://shivamshinde123.github.io/tunarag-python/system-architecture-document/), and
|
|
1297
|
+
[technical design](https://shivamshinde123.github.io/tunarag-python/technical-design-document/).
|
|
1298
|
+
|
|
1299
|
+
## License and responsibility
|
|
1300
|
+
|
|
1301
|
+
TunaRAG is distributed under the [MIT License](https://opensource.org/license/mit).
|
|
1302
|
+
The full text and copyright notice are in the repository and distribution `LICENSE` file.
|
|
1303
|
+
It permits use, modification,
|
|
1304
|
+
and redistribution subject to retaining the copyright and permission notice. The software
|
|
1305
|
+
is provided **as is, without warranty**, and the license includes a limitation of authors'
|
|
1306
|
+
and copyright holders' liability. The complete license text, not this summary, governs.
|
|
1307
|
+
|
|
1308
|
+
You are responsible for testing fitness for your use case, validating generated answers,
|
|
1309
|
+
protecting data and credentials, setting provider-side spending limits, and monitoring costs.
|
|
1310
|
+
TunaRAG does not guarantee accuracy, security, optimization improvements, or provider bills.
|
|
1311
|
+
See the [Security Policy](https://github.com/shivamshinde123/tunarag-python/blob/main/SECURITY.md)
|
|
1312
|
+
for boundaries and known responsibilities.
|
|
1313
|
+
|
|
1314
|
+
No license can guarantee protection from every claim or override applicable mandatory law.
|
|
1315
|
+
Seek qualified legal advice for liability-sensitive use or distribution.
|