tunarag-python 0.3.0__tar.gz → 0.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/.gitignore +1 -0
  2. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/CHANGELOG.md +52 -0
  3. tunarag_python-0.4.0/LICENSE +21 -0
  4. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/PKG-INFO +153 -88
  5. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/README.md +149 -86
  6. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/pyproject.toml +11 -3
  7. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/__init__.py +1 -1
  8. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/dataset.py +19 -3
  9. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/domain.py +5 -2
  10. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/engine.py +134 -151
  11. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/evaluators.py +18 -9
  12. tunarag_python-0.4.0/src/tunarag/integrations/google_adk.py +233 -0
  13. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/integrations/mlflow.py +24 -7
  14. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/integrations/runnables.py +162 -31
  15. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/objective.py +12 -2
  16. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/result.py +23 -7
  17. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/retry.py +50 -5
  18. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/search.py +3 -1
  19. tunarag_python-0.4.0/src/tunarag/serialization.py +177 -0
  20. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/stopping.py +9 -2
  21. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/store.py +3 -2
  22. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/synthetic.py +36 -3
  23. tunarag_python-0.3.0/src/tunarag/integrations/google_adk.py +0 -144
  24. tunarag_python-0.3.0/src/tunarag/serialization.py +0 -78
  25. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/cache.py +0 -0
  26. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/config.py +0 -0
  27. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/contracts.py +0 -0
  28. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/errors.py +0 -0
  29. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/integrations/__init__.py +0 -0
  30. {tunarag_python-0.3.0 → tunarag_python-0.4.0}/src/tunarag/py.typed +0 -0
@@ -9,6 +9,7 @@ __pycache__/
9
9
  .coverage
10
10
  htmlcov/
11
11
  site/
12
+ .playwright-cli/
12
13
 
13
14
  # Local environments
14
15
  .venv/
@@ -1,5 +1,57 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.4.0 - 2026-11-21
4
+
5
+ ### Fixed
6
+
7
+ - Code audit fixes (`docs/audit-2026-11-21/`), each covered by `tests/test_audit_regressions.py`:
8
+ - A NaN or infinite metric or cost fails only its trial instead of crashing the study.
9
+ - Ctrl-C/cancelled and crashed (failed) studies can be resumed; interrupted trials re-run.
10
+ - Token usage is captured from every chat-model call (LangChain usage callback), including
11
+ string-output chains such as `create_retrieval_chain`; new `token_prices` option on all
12
+ adapters gives an estimated cost so `CostBudget` works.
13
+ - Unknown cost is reported as `unavailable`, not an exact $0.
14
+ - Usage from failed, timed-out, and cancelled attempts is recorded.
15
+ - Provider timeouts, connection errors, rate limits (429), and 408/5xx errors are retryable,
16
+ including OpenAI, httpx, LiteLLM, and Google GenAI exceptions and wrapped causes.
17
+ - Without an explicit `identity`, evaluator and adapter identities include digests of
18
+ scorer, factory, mapper, wrapped function, graph node, and ADK agent code, and adapters
19
+ warn. An explicit `identity` (now also accepted by `MetricEvaluator`) is the declared
20
+ version and replaces the code digests.
21
+ - RAGAS evaluations use per-call metric copies, fixing a race that silently dropped samples.
22
+ - Objectives mark partial-coverage metrics `estimated`; new `ObjectiveTerm.min_coverage`.
23
+ - Google ADK output includes `tool_responses` (retrieval evidence); new `events_mapper`;
24
+ error events keep their error code.
25
+ - MLflow reuses one run per study across processes and reopens it on resume.
26
+ - Custom search-space parameters (sample/accepts/describe) work with the optimizer.
27
+ - `TimeBudget` counts active time across resumes; consecutive `MaxFailures` ignores
28
+ cancelled trials; LLMSearch replay matches a live run's history.
29
+ - Dataset versions ignore row order; duplicate rows report their record numbers.
30
+ - Synthetic generation reports every failed document; `allow_partial=True` keeps the rest.
31
+ - Floats and numeric strings no longer share a hash.
32
+ - Failed attempts log their underlying cause to the `tunarag` logger.
33
+ - Documentation builds require `mkdocstrings` 1.x; 0.x fails with `pymdown-extensions` 13
34
+ (`Highlight.__init__() missing 'md'`).
35
+
36
+ ### Migration
37
+
38
+ - The audit fixes change identity hashes (dataset versions, float encoding, adapter and evaluator
39
+ fingerprints). Studies started with earlier versions cannot be resumed (resume raises
40
+ `ResumeError`), and earlier evaluation-cache entries are not reused. Completed studies and
41
+ reports remain readable.
42
+ - Pass `identity=` to every adapter to silence the new warning and version your pipeline.
43
+
44
+ ## 0.3.1 - 2026-10-09
45
+
46
+ - Replace both raw Mermaid README blocks with pre-rendered PNG diagrams compatible with PyPI.
47
+ Use public absolute documentation/image URLs and add the documentation metadata link.
48
+
49
+ - Add the complete MIT license with warranty and liability disclaimers to the repository,
50
+ wheel and source archive. Require the license file in distribution-content checks.
51
+ - Audit all six README illustrations: remove the redundant loop, replace misleading
52
+ adapter/dataset visuals with two-way framework control and untouched holdout illustrations,
53
+ and move the quality/cost/latency illustration to the objective explanation.
54
+
3
55
  All notable changes to TunaRAG follow semantic versioning.
4
56
 
5
57
  ## 0.3.0 - 2026-10-09
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Shivam Shinde
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -1,12 +1,14 @@
1
1
  Metadata-Version: 2.5
2
2
  Name: tunarag-python
3
- Version: 0.3.0
3
+ Version: 0.4.0
4
4
  Summary: Tune LangChain, LangGraph, and Google ADK RAG pipelines
5
+ Project-URL: Documentation, https://shivamshinde123.github.io/tunarag-python/
5
6
  Project-URL: Repository, https://github.com/shivamshinde123/tunarag-python
6
7
  Project-URL: Issues, https://github.com/shivamshinde123/tunarag-python/issues
7
8
  Project-URL: Changelog, https://github.com/shivamshinde123/tunarag-python/blob/main/CHANGELOG.md
8
9
  Author: Shivam Shinde
9
10
  License-Expression: MIT
11
+ License-File: LICENSE
10
12
  Keywords: evaluation,llm,optimization,rag,retrieval
11
13
  Classifier: Development Status :: 3 - Alpha
12
14
  Classifier: Intended Audience :: Developers
@@ -35,7 +37,7 @@ Provides-Extra: docs
35
37
  Requires-Dist: mkdocs-include-markdown-plugin<8,>=7; extra == 'docs'
36
38
  Requires-Dist: mkdocs-material<10,>=9.5; extra == 'docs'
37
39
  Requires-Dist: mkdocs<2,>=1.6; extra == 'docs'
38
- Requires-Dist: mkdocstrings[python]<1,>=0.27; extra == 'docs'
40
+ Requires-Dist: mkdocstrings[python]<2,>=1; extra == 'docs'
39
41
  Provides-Extra: frameworks
40
42
  Requires-Dist: google-adk<3,>=2; extra == 'frameworks'
41
43
  Requires-Dist: langchain<2,>=1; extra == 'frameworks'
@@ -100,24 +102,19 @@ indexes, provider clients, and credentials; TunaRAG owns the optimization workfl
100
102
 
101
103
  **Documentation:** [shivamshinde123.github.io/tunarag-python](https://shivamshinde123.github.io/tunarag-python/)
102
104
 
105
+ **Usage handbook:** [Download the editable DOCX handbook](https://shivamshinde123.github.io/tunarag-python/TunaRAG_Python_Usage_Handbook.docx).
106
+ Start with RAG basics, then follow practical code examples for integrating your framework pipeline
107
+ and using advanced tuning features.
108
+
103
109
  **Lightweight distribution:** pip installs runtime code and metadata, not tests, datasets,
104
110
  or documentation images. Framework dependencies are optional extras. See
105
- [package contents and release checks](docs/package-distribution.md).
106
-
107
- **Live integration checks:** [OpenRouter, RAGAS, and MLflow test plan](docs/live-integration-test-plan.md).
108
-
109
- **End-to-end verification:** [Full-PDF campaigns, held-out quality checks, crash recovery,
110
- and clean-wheel testing](docs/comprehensive-testing.md). Paid tests are explicit opt-ins;
111
- ordinary CI never makes paid provider calls.
112
- These opt-in tests include spending safeguards; normal tests never make paid calls.
111
+ [package contents and release checks](https://shivamshinde123.github.io/tunarag-python/package-distribution/).
113
112
 
114
- **Google ADK validation:** [Real OpenRouter RAG tuning, RAGAS scoring, SQLite resume,
115
- and MLflow checks](docs/live-google-adk-testing.md).
116
-
117
- **Quality and reliability:** [Ordered validation plan](docs/quality-validation-plan.md)
118
- and [bounded repeated tuning, ADK tools, billing and recovery checks](docs/reliability-validation.md).
119
- See [mixed-topic validation](docs/multicorpus-validation.md) for cross-model QA,
120
- simulated faults/load, and the unresolved adversarial-prompt failure.
113
+ **Contributor verification:** Test plans and campaign evidence remain in the
114
+ [repository documentation](https://github.com/shivamshinde123/tunarag-python/tree/main/docs).
115
+ Paid tests are explicit opt-ins; ordinary CI never makes paid provider calls.
116
+ The [mixed-topic validation report](https://github.com/shivamshinde123/tunarag-python/blob/main/docs/multicorpus-validation.md)
117
+ records known adversarial-prompt failures; tuning does not guarantee document safety.
121
118
 
122
119
  ## Comprehensive Usage Guide
123
120
 
@@ -154,27 +151,12 @@ TunaRAG compares configurations of an **existing** RAG pipeline. You provide the
154
151
  evaluation dataset, a search space, evaluators, and an objective. TunaRAG schedules trials, isolates
155
152
  failures, persists state, and returns an inspectable result.
156
153
 
157
- ```mermaid
158
- flowchart LR
159
- D[Versioned dataset] --> O[Optimizer]
160
- S[Search strategy] -->|Candidate| O
161
- O -->|Candidate + example| A[RAG adapter]
162
- A -->|Output + usage| E[Evaluators]
163
- E -->|Metrics| J[Objective]
164
- J -->|Benefit score| O
165
- O -->|Observe| S
166
- O --> DB[(SQLite store)]
167
- O <--> C[(Evaluation cache)]
168
- DB --> CB[Post-commit callbacks]
169
- O --> R[OptimizationResult]
170
- ```
154
+ ![Optimization loop: a strategy proposes candidates, the framework adapter runs each example, evaluators produce metrics, and the objective scores trials; SQLite persists results before callbacks](https://shivamshinde123.github.io/tunarag-python/assets/diagrams/optimization-loop.png)
171
155
 
172
156
  A **study** contains ordered **trials**. Each trial evaluates one complete candidate configuration
173
157
  against every dataset example. Same-named sample metrics are aggregated into trial metrics, then the
174
158
  objective converts those metrics into one larger-is-better score.
175
159
 
176
- ![A hand-drawn Xiaohei operating TunaRAG's iterative optimization loop](docs/assets/tunarag-python-readme-illustrations/01-optimization-loop.png)
177
-
178
160
  TunaRAG does not create or host a RAG application, provision indexes, manage provider credentials,
179
161
  or expose a hosted API. Your application keeps ownership of those concerns.
180
162
 
@@ -254,7 +236,9 @@ dataset = EvaluationDataset.from_records(
254
236
  )
255
237
  optimizer = Optimizer(
256
238
  adapter=LangChainAdapter(
257
- factory=build_chain, input_mapper=lambda example, candidate: example.query
239
+ factory=build_chain,
240
+ input_mapper=lambda example, candidate: example.query,
241
+ identity={"quickstart": "offline-langchain", "revision": 1},
258
242
  ),
259
243
  strategy=RandomSearch(SearchSpace({"top_k": IntegerRange(1, 2)}), seed=7),
260
244
  evaluators=(
@@ -295,10 +279,13 @@ from langchain_core.runnables import RunnableLambda
295
279
  from tunarag import LangChainAdapter
296
280
 
297
281
  chain = RunnableLambda(lambda inputs: {"answer": inputs["question"]})
298
- adapter = LangChainAdapter(chain)
282
+ adapter = LangChainAdapter(chain, identity={"chain": "echo", "revision": 1})
299
283
  ```
300
284
 
301
- ![A hand-drawn Xiaohei connecting a framework RAG pipeline through TunaRAG](docs/assets/tunarag-python-readme-illustrations/02-rag-adapter-boundary.png)
285
+ ![TunaRAG sends settings and a query through an official adapter to a separate framework RAG; answers and usage return in the opposite direction](https://shivamshinde123.github.io/tunarag-python/assets/tunarag-python-readme-illustrations/07-framework-adapter-control.png)
286
+
287
+ The adapter connects two responsibilities; it does not move your RAG into TunaRAG.
288
+ Your framework must consume the candidate settings for tuning to change its behavior.
302
289
 
303
290
  Candidate parameters go into framework configuration or initial ADK session state. A parameter
304
291
  only changes behavior if the framework consumes it. For model, prompt, retriever, or index choices
@@ -316,8 +303,14 @@ adapter = LangChainAdapter(
316
303
  ```
317
304
 
318
305
  Include stable corpus, index, model, prompt, and mapper revisions in `identity` for cache and resume
319
- compatibility. Never include credentials. Framework factories and mapping callbacks remain normal
320
- Python configuration code; retrieval and generation execute through the framework.
306
+ compatibility. Never include credentials. When you pass `identity`, it is the pipeline's
307
+ declared version: TunaRAG uses it (plus the names of your factory and mappers) and does not
308
+ fingerprint code, so bump it whenever behavior changes. An adapter without `identity` emits a
309
+ `UserWarning` and is identified by code structure instead (factory and mapper code, wrapped
310
+ `RunnableLambda` functions, LangGraph node functions, chain steps, ADK agent instruction, model and
311
+ tools), which cannot see prompt-text, index, or model-setting changes. Framework factories and
312
+ mapping callbacks remain normal Python configuration code; retrieval and generation execute through
313
+ the framework.
321
314
 
322
315
  ## LangChain integration
323
316
 
@@ -388,14 +381,26 @@ adapter = LangChainAdapter(
388
381
  output_tokens=output["usage"]["output_tokens"],
389
382
  ),
390
383
  ),
384
+ identity={"chain": "support-rag", "version": 2},
391
385
  )
392
386
  ```
393
387
 
394
- The default usage mapper recursively discovers standard `usage_metadata` in mappings, sequences,
395
- and message objects. The adapter always adds `langchain.runnable` latency. Set
388
+ By default the adapter attaches LangChain's `UsageMetadataCallbackHandler` to the run, so token
389
+ usage is captured from every chat-model call inside the runnable, including chains such as
390
+ `create_retrieval_chain` whose answer is a plain string. If no callback usage is seen (for example
391
+ nested calls on Python 3.10 without an explicit `config`), it falls back to discovering standard
392
+ `usage_metadata` in mappings, sequences, and message objects. Token records keep the component
393
+ name `langchain.message` (`langgraph.message` for graphs); the reported model name is used to look
394
+ up `token_prices`. The adapter always adds `langchain.runnable` latency. Set
396
395
  `capture_message_usage=False` to disable automatic token extraction. A custom `usage_mapper` takes
397
396
  precedence.
398
397
 
398
+ Providers do not report dollar cost to LangChain, so cost is **unavailable** unless you price it.
399
+ Pass `token_prices={"<model name>": (usd_per_1m_input, usd_per_1m_output)}` (keys match the
400
+ model name the provider reports, e.g. `"openai/gpt-4.1-mini"` through OpenRouter); priced records
401
+ get an `estimated` cost, which `CostBudget` counts. The same option exists on
402
+ `LangGraphAdapter` and `GoogleADKAdapter`.
403
+
399
404
  For a complete provider-backed pipeline—from documents and embeddings through optimization and
400
405
  reports—run
401
406
  [`examples/langchain_full_rag.py`](https://github.com/shivamshinde123/tunarag-python/blob/main/examples/langchain_full_rag.py).
@@ -447,6 +452,7 @@ adapter = LangGraphAdapter(
447
452
  },
448
453
  config_mapper=lambda candidate: None,
449
454
  output_mapper=lambda state: state["answer"],
455
+ identity={"graph": "qa", "revision": 2},
450
456
  )
451
457
  ```
452
458
 
@@ -462,18 +468,20 @@ agent tools, corpus retrieval, and runner ownership remain with the application.
462
468
 
463
469
  Use `factory=lambda candidate: build_runner(candidate)` when tuning settings such as a model or
464
470
  retrieval tool that require constructing an agent. `state_mapper` can map candidates to agent state;
465
- `output_mapper` receives the final ADK event, allowing retrieval evidence to be exposed for RAGAS.
466
- By default the output is `{"answer": final_text}`. Usage captures reported model token counts and
467
- runner latency; dollar cost is unavailable unless your evaluator supplies a priced metric.
468
- Missing final responses and ADK error events fail the trial rather than becoming empty answers.
471
+ By default the output is `{"answer": final_text, "tool_responses": [{"name": ..., "response": ...}]}`:
472
+ tool results such as retrieved passages are collected from the whole run, so retrieval metrics and
473
+ RAGAS can see the evidence. `output_mapper` receives only the final ADK event; `events_mapper`
474
+ receives every event of the run (use one or the other) for custom outputs. Usage captures reported
475
+ model token counts and runner latency; pass `token_prices` (keyed by the event's `model_version`)
476
+ for an estimated cost. Missing final responses and ADK error events fail the trial rather than
477
+ becoming empty answers; the trial records the ADK error code, and the provider's error message is
478
+ logged to the console (not persisted).
469
479
 
470
480
  See the executable [ADK example](https://github.com/shivamshinde123/tunarag-python/blob/main/examples/google_adk_tuning.py)
471
481
  and [integration reference](https://github.com/shivamshinde123/tunarag-python/blob/main/docs/reference/integrations.md).
472
482
 
473
483
  ## Evaluation datasets
474
484
 
475
- ![A hand-drawn Xiaohei normalizing, validating, hashing, and splitting evaluation records](docs/assets/tunarag-python-readme-illustrations/03-dataset-preparation.png)
476
-
477
485
  ### Canonical model
478
486
 
479
487
  ```python
@@ -523,8 +531,10 @@ csv_dataset = EvaluationDataset.from_csv("evaluation.csv")
523
531
  ```
524
532
 
525
533
  JSON contains an array; JSONL contains one object per nonblank line. CSV collection and metadata
526
- cells are JSON-encoded. Missing IDs are deterministically derived. Every `EvaluationDataset` has a
527
- content-derived `version` used by caches and resume validation.
534
+ cells are JSON-encoded. Missing IDs are deterministically derived from the record's content, so two
535
+ identical rows without IDs are reported as a duplicate (with both record numbers). Every
536
+ `EvaluationDataset` has a content-derived `version` used by caches and resume validation; it does
537
+ not depend on row order.
528
538
 
529
539
  ### Deterministic splitting
530
540
 
@@ -542,6 +552,13 @@ optimization_data = split_dataset.select(DatasetSplit.OPTIMIZE)
542
552
  Splits are hash-based and stable across input order. Group splitting keeps related examples
543
553
  together. `select()` rejects an empty split.
544
554
 
555
+ ![Tune and choose a configuration using optimization data while keeping the test set untouched for a final check](https://shivamshinde123.github.io/tunarag-python/assets/tunarag-python-readme-illustrations/08-held-out-evaluation.png)
556
+
557
+ Pass only the selected optimization subset to the tuning study. Use validation data for
558
+ development decisions, and evaluate the frozen winner on the untouched test subset.
559
+ TunaRAG does not automatically enforce this separation or run the final held-out evaluation;
560
+ calling `optimize()` with the test set would use it for parameter selection and leak test data.
561
+
545
562
  For custom examples, use a user-managed version:
546
563
 
547
564
  ```python
@@ -604,11 +621,14 @@ generation_usage = generated.usage
604
621
 
605
622
  `SourceSpan` is a zero-based half-open range. TunaRAG validates spans, confidence, cardinality,
606
623
  source IDs, and usage. Generated examples carry provenance and `synthetic=True`. Review them before
607
- using them as gold data. Generation usage is separate from optimization usage.
624
+ using them as gold data. Generation usage is separate from optimization usage. Every document is
625
+ attempted; if some fail, the error lists each failed document and its cause. Pass
626
+ `allow_partial=True` to keep the successful documents instead (failed IDs are in
627
+ `generated.failed_document_ids`).
608
628
 
609
629
  ## Search spaces and strategies
610
630
 
611
- ![A hand-drawn Xiaohei validating candidates from RandomSearch and LLMSearch](docs/assets/tunarag-python-readme-illustrations/04-search-strategies.png)
631
+ ![A hand-drawn Xiaohei validating candidates from RandomSearch and LLMSearch](https://shivamshinde123.github.io/tunarag-python/assets/tunarag-python-readme-illustrations/04-search-strategies.png)
612
632
 
613
633
  ### Typed space
614
634
 
@@ -719,7 +739,13 @@ class RetrievalEvaluator:
719
739
 
720
740
  `MetricEvaluator` accepts sync or async scorers. `None`, NaN, and infinity become unavailable;
721
741
  nonnumeric values raise `EvaluationError`. Coverage must be `[0, 1]`. `tunarag.objective` is a
722
- reserved metric name.
742
+ reserved metric name. `MetricValue` rejects NaN and infinity, so a custom evaluator that produces
743
+ one fails that trial (the study continues); return `None` for "no value".
744
+
745
+ Without `identity=`, a `MetricEvaluator`'s identity includes a digest of its scorer's code, so
746
+ editing a scorer, or swapping one lambda for another, invalidates cached scores. Pass `identity=`
747
+ to declare the scorer's version yourself, for example when its behavior depends on outside state
748
+ (files, models, thresholds read at call time); the code digest is then not used.
723
749
 
724
750
  ### RAGAS
725
751
 
@@ -742,12 +768,20 @@ ragas_evaluator = RagasEvaluator(
742
768
  )
743
769
  ```
744
770
 
745
- TunaRAG constructs one RAGAS `SingleTurnSample` and calls the async evaluation API. Names receive
746
- the configured prefix. Mapping and RAGAS failures become `EvaluationError`. Credentials and provider
747
- configuration remain owned by your RAGAS components.
771
+ TunaRAG constructs one RAGAS `SingleTurnSample` and calls the async evaluation API. Each call uses
772
+ its own copies of the metric objects, so concurrent samples cannot detach each other's judge LLM.
773
+ Names receive the configured prefix. Mapping and RAGAS failures become `EvaluationError`; with
774
+ `raise_exceptions=False`, a sample RAGAS could not score becomes unavailable and lowers the
775
+ metric's coverage. Use `ObjectiveTerm(..., min_coverage=1.0)` for judge metrics that must cover
776
+ every sample. Credentials and provider configuration remain owned by your RAGAS components.
748
777
 
749
778
  ## Objectives
750
779
 
780
+ ![A hand-drawn Xiaohei adjusting quality, cost, and latency tradeoffs to choose a configuration](https://shivamshinde123.github.io/tunarag-python/assets/tunarag-python-readme-illustrations/06-results-and-reports.png)
781
+
782
+ Your objective weights define the tradeoff; the winning configuration is best among the
783
+ evaluated trials under that objective, not a guarantee of the globally best RAG.
784
+
751
785
  ```python
752
786
  from tunarag import Objective, ObjectiveDirection, ObjectiveTerm
753
787
 
@@ -775,7 +809,10 @@ objective = Objective(
775
809
  Bounded metrics are normalized and clamped to `[0, 1]`; bounded minimize terms become
776
810
  `1 - normalized`. Unbounded maximize terms use raw values and unbounded minimize terms use negative
777
811
  values. Weights are contributions and are not automatically normalized. A missing or unavailable
778
- required metric makes the objective unavailable.
812
+ required metric makes the objective unavailable, and so does a metric whose coverage (the fraction
813
+ of samples that produced it) is below the term's `min_coverage` (default `0.0`). A metric measured
814
+ on only some samples makes the objective `estimated`. An unavailable objective fails the trial, not
815
+ the study.
779
816
 
780
817
  Usage does not automatically become an objective metric. Emit `cost_usd` or `latency_seconds` from
781
818
  an evaluator when you want those values in the objective.
@@ -826,6 +863,12 @@ cache_policy = "read_write"
826
863
 
827
864
  Cache policies: `off`, `read_only`, `write_only`, `read_write`. Settings contain no provider keys.
828
865
 
866
+ `Optimizer` reads the limits, concurrency, retry, timeout, and cache-policy settings. The path
867
+ settings and `seed` are resolved values for your application to use: TunaRAG does not open
868
+ `db_path` or `cache_dir` by itself (pass `SQLiteStore(settings.db_path)` and a `SQLiteCache`, as
869
+ below), does not write to `artifact_dir`, and does not seed strategies from `seed` (pass
870
+ `RandomSearch(space, seed=settings.seed)`).
871
+
829
872
  ## Running optimization
830
873
 
831
874
  Use async in async programs:
@@ -912,14 +955,16 @@ Stop conditions are OR-composed:
912
955
  |---|---|
913
956
  | `MaxTrials(limit)` | Completed trial count reaches the limit |
914
957
  | `CostBudget(limit, count_estimated=True)` | Exact plus optional estimated cost reaches limit |
915
- | `TimeBudget(seconds)` | Wall-clock duration reaches limit |
958
+ | `TimeBudget(seconds)` | Active run time (summed across resumes) reaches limit |
916
959
  | `TargetScore(target)` | Any benefit-oriented objective reaches target |
917
960
  | `MaxFailures(limit, consecutive=False)` | Total or trailing consecutive failures reach limit |
918
961
  | `NoImprovement(patience, min_delta=0, warmup_trials=1)` | Best score stops improving |
919
962
 
920
963
  Always include a hard bound. Conditions are checked between trial batches, so active trials can
921
964
  finish after a cost, time, score, failure, or stagnation threshold is reached. Use `concurrency=1`
922
- for strict between-trial stopping.
965
+ for strict between-trial stopping. `CostBudget` only sees priced usage: give your adapter
966
+ `token_prices` (or a `usage_mapper` that sets `cost`), otherwise cost is unavailable and the budget
967
+ never triggers. Usage from attempts that fail or time out still counts.
923
968
 
924
969
  ```python
925
970
  from tunarag import RetryPolicy
@@ -933,9 +978,14 @@ policy = RetryPolicy(
933
978
  )
934
979
  ```
935
980
 
936
- Three retries means four total attempts. Timeouts and connection errors are retryable. A
937
- `TunaRAGError` retries only when `retryable=True`; other exceptions fail that trial. Backoff is
938
- deterministic and capped. Attempt usage is append-only, including failed attempts.
981
+ Three retries means four total attempts. Timeouts, connection errors, rate limits (HTTP 429), and
982
+ transient server errors (408 and 5xx) are retryable, including provider SDK exceptions (OpenAI,
983
+ httpx, LiteLLM, Google GenAI, ...) recognized by type name or HTTP status without importing those
984
+ SDKs, and wrapped causes. A `TunaRAGError` retries when `retryable=True` or when it wraps such a
985
+ transient cause; other exceptions fail that trial. Backoff is deterministic and capped. Attempt
986
+ usage is append-only, including failed, timed-out, and cancelled attempts. Each failed attempt is
987
+ logged to the `tunarag` logger with its underlying cause; persisted errors keep only the stage and
988
+ exception type (plus TunaRAG's own message), so third-party text never lands in SQLite or reports.
939
989
 
940
990
  `concurrency` bounds simultaneous trials; `sample_concurrency` bounds samples inside each trial.
941
991
  Their product approximates maximum concurrent adapter calls. Completion may be concurrent, but
@@ -961,8 +1011,9 @@ optimizer = Optimizer(
961
1011
  )
962
1012
  ```
963
1013
 
964
- Identity includes dataset version, candidate, adapter, and evaluators. A hit skips adapter and
965
- evaluator execution but still calculates the objective and records the trial.
1014
+ Identity includes dataset version, candidate, adapter, and evaluators (including digests of
1015
+ factory, mapper, and scorer code; see `identity` above). A hit skips adapter and evaluator execution
1016
+ but still calculates the objective and records the trial.
966
1017
 
967
1018
  Direct API:
968
1019
 
@@ -985,7 +1036,7 @@ labels are metadata, not encryption.
985
1036
 
986
1037
  ## SQLite persistence and resume
987
1038
 
988
- ![A hand-drawn Xiaohei recovering and resuming a committed SQLite study after interruption](docs/assets/tunarag-python-readme-illustrations/05-persistence-and-resume.png)
1039
+ ![A hand-drawn Xiaohei recovering and resuming a committed SQLite study after interruption](https://shivamshinde123.github.io/tunarag-python/assets/tunarag-python-readme-illustrations/05-persistence-and-resume.png)
989
1040
 
990
1041
  ```python
991
1042
  from pathlib import Path
@@ -1005,8 +1056,11 @@ resumed = await resumed_optimizer.aresume(dataset, study_id=result.study_id)
1005
1056
 
1006
1057
  Resume validates semantic configuration and dataset identity, returns completed studies without
1007
1058
  provider execution, abandons expired leases, replays committed strategy history, continues
1008
- unfinished work, and schedules new candidates. Unknown, failed, cancelled, or incompatible studies
1009
- raise `ResumeError`. An active lease produces a retryable error; retry later.
1059
+ unfinished work, and schedules new candidates. Interrupted (Ctrl-C/cancelled) and crashed (failed)
1060
+ studies resume too: their in-flight trials are re-run. A cancellation releases its attempts at
1061
+ once; after a hard process kill, attempts stay leased until `trial_timeout_seconds + 60` passes,
1062
+ and resuming earlier raises a retryable `ResumeError`. Unknown or incompatible studies raise
1063
+ `ResumeError`.
1010
1064
 
1011
1065
  Read persisted state:
1012
1066
 
@@ -1044,18 +1098,7 @@ Events cover study, trial, and attempt lifecycles plus callback failures. Handle
1044
1098
  types defensively. Callback failures never roll back committed work; TunaRAG records them as durable
1045
1099
  events. Make callbacks idempotent.
1046
1100
 
1047
- ```mermaid
1048
- sequenceDiagram
1049
- participant O as Optimizer
1050
- participant S as SQLite
1051
- participant C as Callback
1052
- O->>S: Commit terminal trial and event
1053
- S-->>O: Committed
1054
- O->>C: on_event(event)
1055
- alt callback fails
1056
- O->>S: Append callback.failed
1057
- end
1058
- ```
1101
+ ![SQLite commits the terminal trial and event before the callback runs; a failing callback appends a callback.failed event without rolling back the trial](https://shivamshinde123.github.io/tunarag-python/assets/diagrams/post-commit-callbacks.png)
1059
1102
 
1060
1103
  MLflow is an optional post-commit projection; SQLite remains authoritative:
1061
1104
 
@@ -1078,14 +1121,13 @@ optimizer = Optimizer(
1078
1121
  )
1079
1122
  ```
1080
1123
 
1081
- One MLflow run is created per study. Candidate parameters, committed metrics, statuses, and event
1124
+ One MLflow run is created per study, also across processes: a resumed study finds its run by the
1125
+ `tunarag.study_id` tag and reopens it. Candidate parameters, committed metrics, statuses, and event
1082
1126
  position are mirrored. An MLflow outage does not invalidate local results. Automatic external
1083
- reconciliation is not included in V0.2.
1127
+ reconciliation of missed events is not included.
1084
1128
 
1085
1129
  ## Results and reports
1086
1130
 
1087
- ![A hand-drawn Xiaohei balancing quality, cost, and latency before exporting the winning result](docs/assets/tunarag-python-readme-illustrations/06-results-and-reports.png)
1088
-
1089
1131
  `OptimizationResult` exposes:
1090
1132
 
1091
1133
  ```python
@@ -1129,8 +1171,11 @@ result.write_json("artifacts/result.json")
1129
1171
  result.write_csv("artifacts/trials.csv")
1130
1172
  ```
1131
1173
 
1132
- Reports are deterministic and redacted. Usage separates exact and estimated cost. A completed result
1133
- may have no best candidate if every trial failed or every objective was unavailable.
1174
+ Reports are deterministic and redacted. Usage separates exact and estimated cost; `usage.status`
1175
+ says how well cost is known: `exact` (every model call has an exact cost), `estimated` (some are
1176
+ estimated or unpriced), or `unavailable` (no cost reported, never shown as an exact $0). A
1177
+ completed result may have no best candidate if every trial failed or every objective was
1178
+ unavailable.
1134
1179
 
1135
1180
  ## Errors, privacy, and secrets
1136
1181
 
@@ -1206,7 +1251,9 @@ class EvenInteger:
1206
1251
  }
1207
1252
  ```
1208
1253
 
1209
- Provider descriptions and component fingerprints must not expose secrets.
1254
+ Custom parameters are identified for cache and resume by their type and `describe()` output, so
1255
+ keep `describe()` complete and deterministic. Provider descriptions and component fingerprints must
1256
+ not expose secrets.
1210
1257
 
1211
1258
  ## Production checklist
1212
1259
 
@@ -1225,7 +1272,7 @@ Provider descriptions and component fingerprints must not expose secrets.
1225
1272
 
1226
1273
  ## Limitations and glossary
1227
1274
 
1228
- V0.2 does not provide a hosted service/dashboard, automatic RAG construction, vector-database
1275
+ V0.4 does not provide a hosted service/dashboard, automatic RAG construction, vector-database
1229
1276
  provisioning, distributed workers, Pareto optimization, Bayesian/evolutionary search, encrypted
1230
1277
  payloads, MLflow reconciliation, provider pricing management, or a TypeScript implementation.
1231
1278
  LangChain, LangGraph, and Google ADK 2.x are official integrations. Additional frameworks require a future first-party integration; custom RAG adapters are unsupported.
@@ -1245,6 +1292,24 @@ LangChain, LangGraph, and Google ADK 2.x are official integrations. Additional f
1245
1292
  | Trial | One candidate evaluated over the whole dataset |
1246
1293
  | Usage | Tokens, cost, latency, confidence, and pricing provenance |
1247
1294
 
1248
- For deeper contracts and architecture, see [`docs/api-specification.md`](docs/api-specification.md),
1249
- [`docs/system-architecture-document.md`](docs/system-architecture-document.md), and
1250
- [`docs/technical-design-document.md`](docs/technical-design-document.md).
1295
+ For deeper contracts and architecture, see the [API specification](https://shivamshinde123.github.io/tunarag-python/api-specification/),
1296
+ [system architecture](https://shivamshinde123.github.io/tunarag-python/system-architecture-document/), and
1297
+ [technical design](https://shivamshinde123.github.io/tunarag-python/technical-design-document/).
1298
+
1299
+ ## License and responsibility
1300
+
1301
+ TunaRAG is distributed under the [MIT License](https://opensource.org/license/mit).
1302
+ The full text and copyright notice are in the repository and distribution `LICENSE` file.
1303
+ It permits use, modification,
1304
+ and redistribution subject to retaining the copyright and permission notice. The software
1305
+ is provided **as is, without warranty**, and the license includes a limitation of authors'
1306
+ and copyright holders' liability. The complete license text, not this summary, governs.
1307
+
1308
+ You are responsible for testing fitness for your use case, validating generated answers,
1309
+ protecting data and credentials, setting provider-side spending limits, and monitoring costs.
1310
+ TunaRAG does not guarantee accuracy, security, optimization improvements, or provider bills.
1311
+ See the [Security Policy](https://github.com/shivamshinde123/tunarag-python/blob/main/SECURITY.md)
1312
+ for boundaries and known responsibilities.
1313
+
1314
+ No license can guarantee protection from every claim or override applicable mandatory law.
1315
+ Seek qualified legal advice for liability-sensitive use or distribution.