tunarag-python 0.3.0__py3-none-any.whl → 0.3.1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- tunarag/__init__.py +1 -1
- {tunarag_python-0.3.0.dist-info → tunarag_python-0.3.1.dist-info}/METADATA +51 -46
- {tunarag_python-0.3.0.dist-info → tunarag_python-0.3.1.dist-info}/RECORD +5 -4
- tunarag_python-0.3.1.dist-info/licenses/LICENSE +21 -0
- {tunarag_python-0.3.0.dist-info → tunarag_python-0.3.1.dist-info}/WHEEL +0 -0
tunarag/__init__.py
CHANGED
|
@@ -1,12 +1,14 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: tunarag-python
|
|
3
|
-
Version: 0.3.
|
|
3
|
+
Version: 0.3.1
|
|
4
4
|
Summary: Tune LangChain, LangGraph, and Google ADK RAG pipelines
|
|
5
|
+
Project-URL: Documentation, https://shivamshinde123.github.io/tunarag-python/
|
|
5
6
|
Project-URL: Repository, https://github.com/shivamshinde123/tunarag-python
|
|
6
7
|
Project-URL: Issues, https://github.com/shivamshinde123/tunarag-python/issues
|
|
7
8
|
Project-URL: Changelog, https://github.com/shivamshinde123/tunarag-python/blob/main/CHANGELOG.md
|
|
8
9
|
Author: Shivam Shinde
|
|
9
10
|
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
10
12
|
Keywords: evaluation,llm,optimization,rag,retrieval
|
|
11
13
|
Classifier: Development Status :: 3 - Alpha
|
|
12
14
|
Classifier: Intended Audience :: Developers
|
|
@@ -102,21 +104,21 @@ indexes, provider clients, and credentials; TunaRAG owns the optimization workfl
|
|
|
102
104
|
|
|
103
105
|
**Lightweight distribution:** pip installs runtime code and metadata, not tests, datasets,
|
|
104
106
|
or documentation images. Framework dependencies are optional extras. See
|
|
105
|
-
[package contents and release checks](
|
|
107
|
+
[package contents and release checks](https://shivamshinde123.github.io/tunarag-python/package-distribution/).
|
|
106
108
|
|
|
107
|
-
**Live integration checks:** [OpenRouter, RAGAS, and MLflow test plan](
|
|
109
|
+
**Live integration checks:** [OpenRouter, RAGAS, and MLflow test plan](https://shivamshinde123.github.io/tunarag-python/live-integration-test-plan/).
|
|
108
110
|
|
|
109
111
|
**End-to-end verification:** [Full-PDF campaigns, held-out quality checks, crash recovery,
|
|
110
|
-
and clean-wheel testing](
|
|
112
|
+
and clean-wheel testing](https://shivamshinde123.github.io/tunarag-python/comprehensive-testing/). Paid tests are explicit opt-ins;
|
|
111
113
|
ordinary CI never makes paid provider calls.
|
|
112
114
|
These opt-in tests include spending safeguards; normal tests never make paid calls.
|
|
113
115
|
|
|
114
116
|
**Google ADK validation:** [Real OpenRouter RAG tuning, RAGAS scoring, SQLite resume,
|
|
115
|
-
and MLflow checks](
|
|
117
|
+
and MLflow checks](https://shivamshinde123.github.io/tunarag-python/live-google-adk-testing/).
|
|
116
118
|
|
|
117
|
-
**Quality and reliability:** [Ordered validation plan](
|
|
118
|
-
and [bounded repeated tuning, ADK tools, billing and recovery checks](
|
|
119
|
-
See [mixed-topic validation](
|
|
119
|
+
**Quality and reliability:** [Ordered validation plan](https://shivamshinde123.github.io/tunarag-python/quality-validation-plan/)
|
|
120
|
+
and [bounded repeated tuning, ADK tools, billing and recovery checks](https://shivamshinde123.github.io/tunarag-python/reliability-validation/).
|
|
121
|
+
See [mixed-topic validation](https://shivamshinde123.github.io/tunarag-python/multicorpus-validation/) for cross-model QA,
|
|
120
122
|
simulated faults/load, and the unresolved adversarial-prompt failure.
|
|
121
123
|
|
|
122
124
|
## Comprehensive Usage Guide
|
|
@@ -154,27 +156,12 @@ TunaRAG compares configurations of an **existing** RAG pipeline. You provide the
|
|
|
154
156
|
evaluation dataset, a search space, evaluators, and an objective. TunaRAG schedules trials, isolates
|
|
155
157
|
failures, persists state, and returns an inspectable result.
|
|
156
158
|
|
|
157
|
-
|
|
158
|
-
flowchart LR
|
|
159
|
-
D[Versioned dataset] --> O[Optimizer]
|
|
160
|
-
S[Search strategy] -->|Candidate| O
|
|
161
|
-
O -->|Candidate + example| A[RAG adapter]
|
|
162
|
-
A -->|Output + usage| E[Evaluators]
|
|
163
|
-
E -->|Metrics| J[Objective]
|
|
164
|
-
J -->|Benefit score| O
|
|
165
|
-
O -->|Observe| S
|
|
166
|
-
O --> DB[(SQLite store)]
|
|
167
|
-
O <--> C[(Evaluation cache)]
|
|
168
|
-
DB --> CB[Post-commit callbacks]
|
|
169
|
-
O --> R[OptimizationResult]
|
|
170
|
-
```
|
|
159
|
+

|
|
171
160
|
|
|
172
161
|
A **study** contains ordered **trials**. Each trial evaluates one complete candidate configuration
|
|
173
162
|
against every dataset example. Same-named sample metrics are aggregated into trial metrics, then the
|
|
174
163
|
objective converts those metrics into one larger-is-better score.
|
|
175
164
|
|
|
176
|
-

|
|
177
|
-
|
|
178
165
|
TunaRAG does not create or host a RAG application, provision indexes, manage provider credentials,
|
|
179
166
|
or expose a hosted API. Your application keeps ownership of those concerns.
|
|
180
167
|
|
|
@@ -298,7 +285,10 @@ chain = RunnableLambda(lambda inputs: {"answer": inputs["question"]})
|
|
|
298
285
|
adapter = LangChainAdapter(chain)
|
|
299
286
|
```
|
|
300
287
|
|
|
301
|
-

|
|
289
|
+
|
|
290
|
+
The adapter connects two responsibilities; it does not move your RAG into TunaRAG.
|
|
291
|
+
Your framework must consume the candidate settings for tuning to change its behavior.
|
|
302
292
|
|
|
303
293
|
Candidate parameters go into framework configuration or initial ADK session state. A parameter
|
|
304
294
|
only changes behavior if the framework consumes it. For model, prompt, retriever, or index choices
|
|
@@ -472,8 +462,6 @@ and [integration reference](https://github.com/shivamshinde123/tunarag-python/bl
|
|
|
472
462
|
|
|
473
463
|
## Evaluation datasets
|
|
474
464
|
|
|
475
|
-

|
|
476
|
-
|
|
477
465
|
### Canonical model
|
|
478
466
|
|
|
479
467
|
```python
|
|
@@ -542,6 +530,13 @@ optimization_data = split_dataset.select(DatasetSplit.OPTIMIZE)
|
|
|
542
530
|
Splits are hash-based and stable across input order. Group splitting keeps related examples
|
|
543
531
|
together. `select()` rejects an empty split.
|
|
544
532
|
|
|
533
|
+

|
|
534
|
+
|
|
535
|
+
Pass only the selected optimization subset to the tuning study. Use validation data for
|
|
536
|
+
development decisions, and evaluate the frozen winner on the untouched test subset.
|
|
537
|
+
TunaRAG does not automatically enforce this separation or run the final held-out evaluation;
|
|
538
|
+
calling `optimize()` with the test set would use it for parameter selection and leak test data.
|
|
539
|
+
|
|
545
540
|
For custom examples, use a user-managed version:
|
|
546
541
|
|
|
547
542
|
```python
|
|
@@ -608,7 +603,7 @@ using them as gold data. Generation usage is separate from optimization usage.
|
|
|
608
603
|
|
|
609
604
|
## Search spaces and strategies
|
|
610
605
|
|
|
611
|
-

|
|
612
607
|
|
|
613
608
|
### Typed space
|
|
614
609
|
|
|
@@ -748,6 +743,11 @@ configuration remain owned by your RAGAS components.
|
|
|
748
743
|
|
|
749
744
|
## Objectives
|
|
750
745
|
|
|
746
|
+

|
|
747
|
+
|
|
748
|
+
Your objective weights define the tradeoff; the winning configuration is best among the
|
|
749
|
+
evaluated trials under that objective, not a guarantee of the globally best RAG.
|
|
750
|
+
|
|
751
751
|
```python
|
|
752
752
|
from tunarag import Objective, ObjectiveDirection, ObjectiveTerm
|
|
753
753
|
|
|
@@ -985,7 +985,7 @@ labels are metadata, not encryption.
|
|
|
985
985
|
|
|
986
986
|
## SQLite persistence and resume
|
|
987
987
|
|
|
988
|
-

|
|
989
989
|
|
|
990
990
|
```python
|
|
991
991
|
from pathlib import Path
|
|
@@ -1044,18 +1044,7 @@ Events cover study, trial, and attempt lifecycles plus callback failures. Handle
|
|
|
1044
1044
|
types defensively. Callback failures never roll back committed work; TunaRAG records them as durable
|
|
1045
1045
|
events. Make callbacks idempotent.
|
|
1046
1046
|
|
|
1047
|
-
|
|
1048
|
-
sequenceDiagram
|
|
1049
|
-
participant O as Optimizer
|
|
1050
|
-
participant S as SQLite
|
|
1051
|
-
participant C as Callback
|
|
1052
|
-
O->>S: Commit terminal trial and event
|
|
1053
|
-
S-->>O: Committed
|
|
1054
|
-
O->>C: on_event(event)
|
|
1055
|
-
alt callback fails
|
|
1056
|
-
O->>S: Append callback.failed
|
|
1057
|
-
end
|
|
1058
|
-
```
|
|
1047
|
+

|
|
1059
1048
|
|
|
1060
1049
|
MLflow is an optional post-commit projection; SQLite remains authoritative:
|
|
1061
1050
|
|
|
@@ -1084,8 +1073,6 @@ reconciliation is not included in V0.2.
|
|
|
1084
1073
|
|
|
1085
1074
|
## Results and reports
|
|
1086
1075
|
|
|
1087
|
-

|
|
1088
|
-
|
|
1089
1076
|
`OptimizationResult` exposes:
|
|
1090
1077
|
|
|
1091
1078
|
```python
|
|
@@ -1245,6 +1232,24 @@ LangChain, LangGraph, and Google ADK 2.x are official integrations. Additional f
|
|
|
1245
1232
|
| Trial | One candidate evaluated over the whole dataset |
|
|
1246
1233
|
| Usage | Tokens, cost, latency, confidence, and pricing provenance |
|
|
1247
1234
|
|
|
1248
|
-
For deeper contracts and architecture, see [
|
|
1249
|
-
[
|
|
1250
|
-
[
|
|
1235
|
+
For deeper contracts and architecture, see the [API specification](https://shivamshinde123.github.io/tunarag-python/api-specification/),
|
|
1236
|
+
[system architecture](https://shivamshinde123.github.io/tunarag-python/system-architecture-document/), and
|
|
1237
|
+
[technical design](https://shivamshinde123.github.io/tunarag-python/technical-design-document/).
|
|
1238
|
+
|
|
1239
|
+
## License and responsibility
|
|
1240
|
+
|
|
1241
|
+
TunaRAG is distributed under the [MIT License](https://opensource.org/license/mit).
|
|
1242
|
+
The full text and copyright notice are in the repository and distribution `LICENSE` file.
|
|
1243
|
+
It permits use, modification,
|
|
1244
|
+
and redistribution subject to retaining the copyright and permission notice. The software
|
|
1245
|
+
is provided **as is, without warranty**, and the license includes a limitation of authors'
|
|
1246
|
+
and copyright holders' liability. The complete license text, not this summary, governs.
|
|
1247
|
+
|
|
1248
|
+
You are responsible for testing fitness for your use case, validating generated answers,
|
|
1249
|
+
protecting data and credentials, setting provider-side spending limits, and monitoring costs.
|
|
1250
|
+
TunaRAG does not guarantee accuracy, security, optimization improvements, or provider bills.
|
|
1251
|
+
See the [Security Policy](https://github.com/shivamshinde123/tunarag-python/blob/main/SECURITY.md)
|
|
1252
|
+
for boundaries and known responsibilities.
|
|
1253
|
+
|
|
1254
|
+
No license can guarantee protection from every claim or override applicable mandatory law.
|
|
1255
|
+
Seek qualified legal advice for liability-sensitive use or distribution.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
tunarag/__init__.py,sha256=
|
|
1
|
+
tunarag/__init__.py,sha256=1ugGAQZJbEWCvjzENV0mizMhgQKpAIIvfGt_gpc3ojk,3695
|
|
2
2
|
tunarag/cache.py,sha256=prra0uAjW1JYVez6nA9xXFnRwANODHo1rj6o1EiGLI8,11556
|
|
3
3
|
tunarag/config.py,sha256=EzoRWhViL4OQRMFadhiNwEwvne64p2XYMgigaV4tDqg,9325
|
|
4
4
|
tunarag/contracts.py,sha256=H3rpzKJjN9F34t6ukJj4q7iBrOjFa9PzOwLk-s8A_Pk,2073
|
|
@@ -20,6 +20,7 @@ tunarag/integrations/__init__.py,sha256=7eE5ScMAhleHTFwqglB5ZXR4oyjblovIIosIYYcO
|
|
|
20
20
|
tunarag/integrations/google_adk.py,sha256=P_fL0g60Ok7dirwQlGxBq6BBMxNrP8OSEroSk9ciB8Q,5942
|
|
21
21
|
tunarag/integrations/mlflow.py,sha256=Ig96h8CItgEnm-OX41Sp_MQvW9cbQureneZR2xCdikM,5807
|
|
22
22
|
tunarag/integrations/runnables.py,sha256=B9RLnrpn8nZnYnOqkbDQmMmul2SfxOG8oOE-fiPv948,10268
|
|
23
|
-
tunarag_python-0.3.
|
|
24
|
-
tunarag_python-0.3.
|
|
25
|
-
tunarag_python-0.3.
|
|
23
|
+
tunarag_python-0.3.1.dist-info/METADATA,sha256=oC8AqMNj6aJWy1hKa5kss90R4q6EYW7V93TlBYnxI3U,46536
|
|
24
|
+
tunarag_python-0.3.1.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
|
|
25
|
+
tunarag_python-0.3.1.dist-info/licenses/LICENSE,sha256=ve5VOCp_tHEW3KWjdX4JuIMa0TKdOyAXyaSIS9okbUA,1070
|
|
26
|
+
tunarag_python-0.3.1.dist-info/RECORD,,
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Shivam Shinde
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
File without changes
|