traceground 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- traceground-0.1.0/.gitignore +25 -0
- traceground-0.1.0/CHANGELOG.md +271 -0
- traceground-0.1.0/LICENSE +21 -0
- traceground-0.1.0/PKG-INFO +271 -0
- traceground-0.1.0/README.md +242 -0
- traceground-0.1.0/pyproject.toml +52 -0
- traceground-0.1.0/src/traceground/__init__.py +26 -0
- traceground-0.1.0/src/traceground/__main__.py +6 -0
- traceground-0.1.0/src/traceground/api.py +187 -0
- traceground-0.1.0/src/traceground/binding.py +700 -0
- traceground-0.1.0/src/traceground/citations.py +39 -0
- traceground-0.1.0/src/traceground/cli.py +93 -0
- traceground-0.1.0/src/traceground/numbers.py +596 -0
- traceground-0.1.0/src/traceground/py.typed +0 -0
- traceground-0.1.0/src/traceground/quotes.py +238 -0
- traceground-0.1.0/src/traceground/result.py +55 -0
- traceground-0.1.0/src/traceground/selftalk.py +124 -0
- traceground-0.1.0/src/traceground/textnorm.py +139 -0
- traceground-0.1.0/src/traceground/toolchecks.py +130 -0
- traceground-0.1.0/src/traceground/traces.py +252 -0
- traceground-0.1.0/tests/data/real_answers_2026-11-17.json +1210 -0
- traceground-0.1.0/tests/test_api.py +82 -0
- traceground-0.1.0/tests/test_binding.py +162 -0
- traceground-0.1.0/tests/test_cli.py +116 -0
- traceground-0.1.0/tests/test_examples.py +21 -0
- traceground-0.1.0/tests/test_message_formats.py +89 -0
- traceground-0.1.0/tests/test_numbers.py +167 -0
- traceground-0.1.0/tests/test_quotes.py +133 -0
- traceground-0.1.0/tests/test_real_answers.py +328 -0
- traceground-0.1.0/tests/test_rules_dev12.py +112 -0
- traceground-0.1.0/tests/test_smoke.py +58 -0
- traceground-0.1.0/tests/test_survey_cases.py +64 -0
- traceground-0.1.0/tests/test_textnorm.py +29 -0
- traceground-0.1.0/tests/test_traces.py +98 -0
- traceground-0.1.0/tests/test_wording_and_tools.py +149 -0
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# secrets: keys go in .env locally, never in git
|
|
2
|
+
.env
|
|
3
|
+
.env.*
|
|
4
|
+
!.env.example
|
|
5
|
+
# RAGTruth (eval/ragtruth.py): downloaded data and review files with its texts stay local
|
|
6
|
+
eval/data/
|
|
7
|
+
# survey logs hold local paths; the results in survey/results/ are kept
|
|
8
|
+
survey/survey_log*.txt
|
|
9
|
+
logs/
|
|
10
|
+
# Python and build output
|
|
11
|
+
__pycache__/
|
|
12
|
+
*.pyc
|
|
13
|
+
.pytest_cache/
|
|
14
|
+
.ruff_cache/
|
|
15
|
+
.mypy_cache/
|
|
16
|
+
build/
|
|
17
|
+
dist/
|
|
18
|
+
*.egg-info/
|
|
19
|
+
.venv/
|
|
20
|
+
venv/
|
|
21
|
+
*.zip
|
|
22
|
+
# editors
|
|
23
|
+
.idea/
|
|
24
|
+
.vscode/
|
|
25
|
+
.ipynb_checkpoints/
|
|
@@ -0,0 +1,271 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to traceground are listed here. Versions follow [Semantic Versioning](https://semver.org/).
|
|
4
|
+
|
|
5
|
+
## 0.1.0 (first release)
|
|
6
|
+
|
|
7
|
+
The first version published on PyPI. Its checks are those of 0.1.0.dev12 below, unchanged; the version
|
|
8
|
+
number, the README and the demo's requirements are what changed.
|
|
9
|
+
|
|
10
|
+
## 0.1.0.dev12 (not published)
|
|
11
|
+
|
|
12
|
+
Rules found by measuring on the RAGTruth train split (eval/ragtruth.py); each removes a kind of false
|
|
13
|
+
alarm or makes a finding more precise. The dev and held-out evaluations and the survey give the same
|
|
14
|
+
numbers as with 0.1.0.dev11.
|
|
15
|
+
|
|
16
|
+
### Added
|
|
17
|
+
|
|
18
|
+
- Clock times: "2:30 pm" is checked against the times in the evidence, which may be in 24-hour form
|
|
19
|
+
("14:30", or opening hours such as "11:0-2:30", which close at 2:30 at night, so "2:30 pm" is reported).
|
|
20
|
+
- A conversion in brackets, "400°F (200°C)" or "5 km (3.1 miles)", is reported as minor (worked out in
|
|
21
|
+
the answer) when it is right (within 3%, or 6 degrees); a wrong one is reported as unsupported.
|
|
22
|
+
- The day of a date written with an ordinal ("April 19th", "the 19th of April") is checked; other
|
|
23
|
+
ordinals ("1st place") are not.
|
|
24
|
+
- Quotes: a quote that differs from the source only in small words (a, the, and, of, my, those ...),
|
|
25
|
+
punctuation, the spelling of one word or singular and plural is reported as minor ("The quote differs
|
|
26
|
+
slightly from S1"), with a new count, quotes_changed_slightly. A changed word that changes the meaning
|
|
27
|
+
("can" for "cannot") is still reported.
|
|
28
|
+
- `eval/ragtruth.py` (repository only): measures traceground on RAGTruth. The data is downloaded by
|
|
29
|
+
the user into eval/data/ (gitignored); the results hold numbers and ids only. Test split, measured once
|
|
30
|
+
with this version: 7 of 172 high findings were false alarms by hand review; 85 of the 306 marked spans
|
|
31
|
+
that contain a digit were flagged (eval/README.md).
|
|
32
|
+
|
|
33
|
+
### Changed (repository only)
|
|
34
|
+
|
|
35
|
+
- Demo: a verdict banner; the answer with each finding marked and numbered, a legend of only the kinds
|
|
36
|
+
that occur, and the sentences with no number or quote in grey (with how many were checked: claims in
|
|
37
|
+
words are not); each finding as a card with its severity and "Show the evidence", which opens the source
|
|
38
|
+
or tool result with the value marked; "Use this check in your code" (the Python that gives the same
|
|
39
|
+
report); the report as JSON or Markdown to download. The answer box comes first, and the outputs appear
|
|
40
|
+
after the first check. Labels use the words of the summary ("quote not found", "value attached to the
|
|
41
|
+
wrong field").
|
|
42
|
+
- Demo: five real examples, unedited chats from the evaluation (gpt-oss-120b and gpt-oss-20b on held-out
|
|
43
|
+
questions): a correct answer with a sentence that is not checked, a departure time no tool gave, US
|
|
44
|
+
dollars reported as euros, a growth rate worked out wrong (14.4% for 14.2%), and true numbers from the
|
|
45
|
+
model's own knowledge. The made-up currency example they replace is gone. Markdown bold marks (**) are
|
|
46
|
+
not shown on the marked answer.
|
|
47
|
+
- Demo as an API and an MCP tool: one endpoint, `traceground_check` (answer, sources, tool calls or a
|
|
48
|
+
whole chat, question; returns the report as JSON). `python space/app.py` starts the MCP server too when
|
|
49
|
+
gradio[mcp] is installed (http://127.0.0.1:7860/gradio_api/mcp/); `space/mcp_check.py` connects to it
|
|
50
|
+
as an MCP client and checks two answers. The page's own endpoint is no longer public.
|
|
51
|
+
|
|
52
|
+
### Fixed
|
|
53
|
+
|
|
54
|
+
- Not checked any more: the answer counting its own length ("in 150 words", "a 98-word summary"), a
|
|
55
|
+
rating scale ("out of 5", "4/5"), "24/7", labels such as "Passage 2" and "Review 3".
|
|
56
|
+
- How many items a list in a tool result has counts as given ("based on 3 reviews" for a list of three
|
|
57
|
+
reviews), and is not read as a field value; years in date strings are not field values either.
|
|
58
|
+
- Value-to-field: a number that is part of a name or an address in a result ("4 Eggs & Pizza", "223
|
|
59
|
+
Anacapa St") is not a field value; after "is", a word inside a phrase that starts with a preposition
|
|
60
|
+
("the rating ... according to customer reviews is 3.5") does not name the field; a more specific
|
|
61
|
+
field named next to the number ("one review gave it 1 star": review_stars) is the one meant.
|
|
62
|
+
- Quotes: HTML entities (&) and JSON escapes (\") are ignored; text between two quotations is not a
|
|
63
|
+
quote; example text ('a subject line such as "..."') and the question restated are not quotes.
|
|
64
|
+
- Self-contradiction: "Passage 3 does not mention X" followed by X from another passage is not a
|
|
65
|
+
contradiction; a question or a heading ("follow these steps:") does not count as giving the thing.
|
|
66
|
+
|
|
67
|
+
## 0.1.0.dev11 (not published)
|
|
68
|
+
|
|
69
|
+
### Added
|
|
70
|
+
|
|
71
|
+
- `check_messages()` also reads Anthropic Messages API chats (tool_use and tool_result blocks, is_error)
|
|
72
|
+
and LangChain messages (the objects, or the dicts of messages_to_dict; a ToolMessage with status
|
|
73
|
+
"error" is a failed call). No new dependency: the formats are read as they are.
|
|
74
|
+
|
|
75
|
+
### Fixed
|
|
76
|
+
|
|
77
|
+
- The summary counted a number twice when a quote or a value-to-field finding replaced the number
|
|
78
|
+
finding ("1 number not supported; 1 value attached to the wrong field" for one finding).
|
|
79
|
+
- A changed quote whose nearest wording is far off (similarity 0.4 to 0.6, three words or more in
|
|
80
|
+
common) now shows that wording ("says it differently: ...") instead of "nothing in them is close".
|
|
81
|
+
|
|
82
|
+
### Changed (repository only)
|
|
83
|
+
|
|
84
|
+
- Demo: one form with optional sources and optional tool calls (or a chat), so an agent that uses both
|
|
85
|
+
can be checked; one endpoint, check. A "documents and a tool together" example; a how-to section; a
|
|
86
|
+
currency sign or quote marks next to a finding are marked with it; the label "wrong value for its
|
|
87
|
+
field"; a wider Severity column.
|
|
88
|
+
|
|
89
|
+
## 0.1.0.dev10 (not published)
|
|
90
|
+
|
|
91
|
+
### Added (repository only)
|
|
92
|
+
|
|
93
|
+
- `space/`: a Gradio demo (Gradio 6.32), run locally with `python space/app.py`. Two tabs: an answer with
|
|
94
|
+
sources, and an answer with tool calls or a chat. Marks each finding on the answer, lists the findings,
|
|
95
|
+
and shows the full JSON report. Two named endpoints, check_sources and check_trace. Made-up examples,
|
|
96
|
+
one per check. Not published.
|
|
97
|
+
|
|
98
|
+
### Fixed
|
|
99
|
+
|
|
100
|
+
- Argument check: when a question's words fit two arguments, the one they name best is reported ("5-year"
|
|
101
|
+
is years=3, not annual_rate, although "annual" means yearly).
|
|
102
|
+
|
|
103
|
+
## 0.1.0.dev9 (not published)
|
|
104
|
+
|
|
105
|
+
### Fixed (found on the held-out set; from here on, held-out numbers are no longer blind)
|
|
106
|
+
|
|
107
|
+
- Numbers: a duration in two units whose total is a source value ("535 minutes, 8 hours 55 minutes").
|
|
108
|
+
- Numbers and value-to-field: a scale in a key name (revenue_musd = 1840.5 supports "$1.84 billion").
|
|
109
|
+
- Value-to-field: the right value in another currency with no converting call is reported ("€221.4
|
|
110
|
+
million" for net_income_musd), instead of passing as "another unit".
|
|
111
|
+
- Value-to-field: unit words inside keys (co2_ppm, pressure_hpa, kcal) no longer name a field on their
|
|
112
|
+
own ("typical levels of 400-600 ppm" is a number from outside the tools, not a wrong co2_ppm).
|
|
113
|
+
- Value-to-field: years ("in 2024", or a year the calls give) and numbers followed by a comparative
|
|
114
|
+
("3.3 hPa higher", "7 °C warmer") are not compared with fields; result fields named year, date or
|
|
115
|
+
timestamp are not compared with numbers ("year-over-year").
|
|
116
|
+
|
|
117
|
+
## 0.1.0.dev8 (not published)
|
|
118
|
+
|
|
119
|
+
### Changed: value-to-field finds more wrong fields (developed on the dev set only)
|
|
120
|
+
|
|
121
|
+
- A lone word that only one multi-word field name has binds the number ("€789.52 in interest" for
|
|
122
|
+
total_interest); measure nouns ("years", "hours") and words several fields share ("total") do not.
|
|
123
|
+
- The subject of "... is N" binds N ("The total interest you'll pay on the loan is $3,034.15").
|
|
124
|
+
- The two ends of a range bind to a low/high or min/max pair ("between $69.80 and $72.90", "6-10 days").
|
|
125
|
+
- Time units name fields too ("249 hours" against battery_hours), like kg and $ in dev7.
|
|
126
|
+
- "min" and "max" next to a time noun in a key mean minimum and maximum (days_min), not minutes.
|
|
127
|
+
- Words ending in -ed are stemmed ("converted_amount" matches "converts"; "interest_earned", "earn").
|
|
128
|
+
- "more points (31)": a bracket that opens on the number no longer hides the field before it.
|
|
129
|
+
- Not compared: a value another call worked out from the field (a calculator given price_usd), and a
|
|
130
|
+
number whose clause says it is worked out ("the difference in monthly payment is $166.03").
|
|
131
|
+
- Argument check: a percentage is not taken for an argument whose name has no rate word ("4%
|
|
132
|
+
compounded" is not compounds_per_year).
|
|
133
|
+
|
|
134
|
+
### Added (repository only)
|
|
135
|
+
|
|
136
|
+
- A held-out set: `eval/questions_heldout.json` (20 questions) and seven new tools in `eval/tools.py`
|
|
137
|
+
(flights, invoices, company figures, nutrition, sensors, shipping, cities). `--set heldout` in
|
|
138
|
+
run_eval.py, score.py and mutate.py; runs go to `eval/runs/heldout/`.
|
|
139
|
+
- mutate.py leaves numbers inside dates alone (changing 2026-11-16 is not a wrong field value).
|
|
140
|
+
|
|
141
|
+
## 0.1.0.dev7 (not published)
|
|
142
|
+
|
|
143
|
+
### Changed (found with errors planted in the real answers; see eval/results/review_2026-11-17.md)
|
|
144
|
+
|
|
145
|
+
- Numbers: a value computed from numbers the answer states is accepted only when its sentence says so
|
|
146
|
+
("total", "together", "extra", "warmer" ... for sums and differences; "each", "per", "times" for a
|
|
147
|
+
count from the question). Products and ratios of two stated numbers are no longer tried. Before,
|
|
148
|
+
chance sums hid errors: a date "2026-11-61" passed as 27 + 34.
|
|
149
|
+
- Numbers: a percentage is not supported by a bare fraction below 1 ("0.075%" by annual_rate 0.075),
|
|
150
|
+
unless the evidence writes it as a percentage or under a key such as change_pct. Worked-out
|
|
151
|
+
percentages are compared as fractions ("0.15% of 240 is 36" is reported).
|
|
152
|
+
- Value-to-field: a unit names a field when the results have only one field with that unit ("weighs
|
|
153
|
+
249 kg" against weight_kg); it is reported when the value is another field of the same call.
|
|
154
|
+
|
|
155
|
+
### Added (repository only)
|
|
156
|
+
|
|
157
|
+
- `eval/mutate.py`: plants five kinds of grounding error in the real answers that are correct by hand
|
|
158
|
+
and counts how many traceground reports; `eval/hand_labels.json` lists the answers found wrong.
|
|
159
|
+
|
|
160
|
+
## 0.1.0.dev6 (not published)
|
|
161
|
+
|
|
162
|
+
### Fixed (from the second real-answer run, Groq openai/gpt-oss-20b, 2026-11-17: 4 false alarms on 44 correct answers, now 0)
|
|
163
|
+
|
|
164
|
+
- Value-to-field: "on" also ends the search for a field name ("the monthly payment on a $20,000 loan").
|
|
165
|
+
- Numbers: a thin, narrow no-break or no-break space between groups of three digits is a thousands
|
|
166
|
+
separator ("40 983.61"); a plain space is not.
|
|
167
|
+
- Numbers: a source number times or divided by a whole count from the question ("3 lamps" at 39.9 is
|
|
168
|
+
119.70) is a minor finding, "worked out in the answer, not by a tool", instead of an unsupported
|
|
169
|
+
number. New stat `numbers_computed_in_answer`.
|
|
170
|
+
|
|
171
|
+
### Added (repository only)
|
|
172
|
+
|
|
173
|
+
- Six gpt-oss-20b answers in `tests/data/real_answers_2026-11-17.json`, with wrong-number versions that
|
|
174
|
+
must still be caught.
|
|
175
|
+
|
|
176
|
+
## 0.1.0.dev5 (not published)
|
|
177
|
+
|
|
178
|
+
### Fixed (from the first real-answer run, Groq openai/gpt-oss-120b, 2026-11-17: 3 false alarms on 43 correct answers, now 0)
|
|
179
|
+
|
|
180
|
+
- Value-to-field: a preposition before the number ("for", "from", "per", ...) ends the search for a
|
|
181
|
+
field name, so "the monthly payment for a $30,000 loan" no longer ties 30,000 to monthly_payment.
|
|
182
|
+
- Value-to-field: a number written in another unit of the same kind as the field's unit suffix
|
|
183
|
+
("91.4 °F" for high_c, "€165" for price_usd) is not compared with that field.
|
|
184
|
+
- Numbers: a number the answer computes from numbers it states and the evidence supports is accepted
|
|
185
|
+
(sum, difference, product, ratio, a stated number times a question number, the sum of all), at the
|
|
186
|
+
precision it is written in. Wrong arithmetic is still reported. New stat `numbers_derived`.
|
|
187
|
+
|
|
188
|
+
### Added (repository only)
|
|
189
|
+
|
|
190
|
+
- `tests/test_real_answers.py` and `tests/data/real_answers_2026-11-17.json`: seven real answers checked
|
|
191
|
+
by hand, with wrong-number versions that must still be caught.
|
|
192
|
+
|
|
193
|
+
|
|
194
|
+
### Added (repository only, 0.1.0.dev4)
|
|
195
|
+
|
|
196
|
+
- `eval/`: the real-answer evaluation. Deterministic local tools, 44 questions with the calls and values
|
|
197
|
+
a correct answer needs, a standard-library client for OpenAI-compatible endpoints (Groq, Hugging Face,
|
|
198
|
+
Gemini; OpenAI only with ALLOW_OPENAI=1) with a request cap, an agent loop that records each chat, and
|
|
199
|
+
a scorer that labels answers and lists every disagreement with traceground for review by hand.
|
|
200
|
+
|
|
201
|
+
### Added
|
|
202
|
+
|
|
203
|
+
- Failed calls (`toolchecks.py`): a failed call that was not called again successfully, nor replaced by
|
|
204
|
+
another tool called about the same thing, and that the answer reports as if it had worked.
|
|
205
|
+
- Tools never called: claims of a web search, of running code, of a database query, or of using a named
|
|
206
|
+
tool, with no such call in the trace.
|
|
207
|
+
- Thinking aloud (`selftalk.py`, ported from the Quantum Information Agent): "Wait, let me check:",
|
|
208
|
+
"Let me report the tool's numbers:", retractions, "X? No -"; quoted text and ordinary openers are not
|
|
209
|
+
flagged. Deleting a self-check is offered as a certain fix.
|
|
210
|
+
- Self-contradiction (ported from "Ask Your Quantum Physics Research Papers"): "the sources do not say
|
|
211
|
+
X" followed by X from the sources.
|
|
212
|
+
- `check_answer(..., check_wording=True)`, `--no-wording`. Thinking aloud is a problem even without
|
|
213
|
+
evidence.
|
|
214
|
+
|
|
215
|
+
## 0.1.0.dev3
|
|
216
|
+
|
|
217
|
+
### Added
|
|
218
|
+
|
|
219
|
+
- Tool traces (`traces.py`): a list of `{name, args, result}` (other common key names too) or
|
|
220
|
+
OpenAI-style chat messages; failed calls recognised; `check_answer(..., trace=...)` and
|
|
221
|
+
`check_messages(messages)`; the command line's `--trace FILE`. Tool results and arguments are
|
|
222
|
+
evidence for the number and quote checks.
|
|
223
|
+
- Value-to-field binding (`binding.py`): a number attached to a field name ("concurrence = 0.7",
|
|
224
|
+
"7 rebounds", "4% chance of rain") must be the value a tool result gives for that field;
|
|
225
|
+
`field_synonyms` adds names.
|
|
226
|
+
- Argument binding: a question value that no call used while a call used the same digits shifted by a
|
|
227
|
+
power of ten; a question value whose neighbouring word names an argument called with another value;
|
|
228
|
+
two values swapped between two calls.
|
|
229
|
+
- The survey gives traceground the trace cases as traces.
|
|
230
|
+
|
|
231
|
+
## 0.1.0.dev2
|
|
232
|
+
|
|
233
|
+
### Added
|
|
234
|
+
|
|
235
|
+
- The quote check: quotations of three words or more must occur in the source their sentence cites, or
|
|
236
|
+
in any source; ellipses and [editorial insertions] allowed; the closest real wording, or the source
|
|
237
|
+
the quote is really in, is reported. Titles, quotes in code and single-quoted text are not checked.
|
|
238
|
+
- `citations.py`: which sources a sentence's markers name ([S1], [doc2], [report.txt], [2]). A sentence
|
|
239
|
+
that is only markers ("... night. [S1]") belongs to the sentence before.
|
|
240
|
+
- `check_answer(..., check_quotes=True, min_quote_words=3)`; summaries name what was checked and found.
|
|
241
|
+
|
|
242
|
+
### Fixed
|
|
243
|
+
|
|
244
|
+
- A percentage read as a fraction (70% as 0.70) is no longer offered "1" as its closest value.
|
|
245
|
+
- The source id "question" is refused for every shape of sources, as it names the question.
|
|
246
|
+
|
|
247
|
+
## 0.1.0.dev1
|
|
248
|
+
|
|
249
|
+
### Added
|
|
250
|
+
|
|
251
|
+
- `check_answer(answer, sources, question)` and the `Result` / `Finding` model (verdict, findings with
|
|
252
|
+
span, sentence, plain message, evidence and suggestion; `to_dict`, `to_json`, `to_text`).
|
|
253
|
+
- The number check over plain sources: rounding to the precision written, thousands separators
|
|
254
|
+
(Indian grouping too), scientific notation, scale words, percent and fraction, approximate round
|
|
255
|
+
numbers, number words and fractions in the evidence; citation markers, reference labels, list
|
|
256
|
+
numbers, cited years, URLs, e-mail addresses, code, digits glued to words and plain integers 0 to 10
|
|
257
|
+
are not checked. A number cut short (1.97 for
|
|
258
|
+
1.9799) is a minor finding with the certain fix.
|
|
259
|
+
- `textnorm`, ported from "Ask Your Quantum Physics Research Papers" and generalized.
|
|
260
|
+
- The command line: `traceground --answer FILE --sources FILE ... [--question FILE] [--json]`, with exit
|
|
261
|
+
codes 0 ok, 1 problems, 2 usage error, 3 unsure.
|
|
262
|
+
- The survey runs traceground too, and never overwrites an earlier run's files.
|
|
263
|
+
|
|
264
|
+
## [0.0.1]
|
|
265
|
+
|
|
266
|
+
### Added
|
|
267
|
+
|
|
268
|
+
- The package skeleton: `pyproject.toml` (src layout, no dependencies), the `traceground` command with
|
|
269
|
+
`--version`, smoke tests, and CI on Python 3.10 to 3.14 (Linux) and 3.11 and 3.14 (Windows).
|
|
270
|
+
- Publishing to TestPyPI and PyPI with Trusted Publishing.
|
|
271
|
+
- `survey/` (repository only, not in the package): the Phase-0 survey of existing checkers on 20 cases.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Animesh Sinha Roy
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,271 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: traceground
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Checks an AI answer against its sources or its tool-call trace: every number, quote, value and argument the evidence doesn't support. No model needed.
|
|
5
|
+
Project-URL: Homepage, https://github.com/animeshQKD/traceground
|
|
6
|
+
Project-URL: Issues, https://github.com/animeshQKD/traceground/issues
|
|
7
|
+
Project-URL: Changelog, https://github.com/animeshQKD/traceground/blob/main/CHANGELOG.md
|
|
8
|
+
Author: Animesh Sinha Roy
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: agents,evaluation,grounding,guardrails,hallucination,llm,rag,tool-calling
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
22
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
23
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
24
|
+
Classifier: Typing :: Typed
|
|
25
|
+
Requires-Python: >=3.10
|
|
26
|
+
Provides-Extra: test
|
|
27
|
+
Requires-Dist: pytest>=8; extra == 'test'
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
30
|
+
# traceground
|
|
31
|
+
|
|
32
|
+
[](https://github.com/animeshQKD/traceground/actions/workflows/tests.yml)
|
|
33
|
+
[](https://pypi.org/project/traceground/)
|
|
34
|
+
[](https://github.com/animeshQKD/traceground/blob/main/LICENSE)
|
|
35
|
+
|
|
36
|
+
**Status: alpha (0.1.0).** It has the number and quote checks over sources; the tool-trace checks
|
|
37
|
+
(value-to-field and argument binding, failed calls, tools never called); the wording checks (thinking
|
|
38
|
+
aloud, self-contradiction); `check_answer()`, `check_messages()`, the command line, and a Gradio demo
|
|
39
|
+
that is also an MCP tool (`space/`). Rules and messages may still change between 0.x versions.
|
|
40
|
+
|
|
41
|
+
## What it is for
|
|
42
|
+
|
|
43
|
+
traceground takes an AI answer together with its sources or its tool-call trace, and lists every
|
|
44
|
+
number, quote, value or argument that the evidence doesn't support. Each finding names the exact span,
|
|
45
|
+
the reason, and what the evidence actually says. No language model is involved, so the same input
|
|
46
|
+
always gives the same verdict.
|
|
47
|
+
|
|
48
|
+
An example of the error it is built for:
|
|
49
|
+
|
|
50
|
+
* **Question:** "What is the 5-year cost at 7% interest?"
|
|
51
|
+
* **Tool call:** `loan_cost(rate=0.07, years=3)` returned `{"total": 12250.43, "monthly": 340.29}`
|
|
52
|
+
* **Answer:** "Over 5 years at 7% you pay 12,250 in total, 340.29 a month."
|
|
53
|
+
|
|
54
|
+
Every number in the answer matches the tool result or the question (12,250 is 12,250.43 rounded), so
|
|
55
|
+
a check of numbers alone passes it. The error is that the tool was called with `years=3` when the question asks for 5 years.
|
|
56
|
+
|
|
57
|
+
## Use
|
|
58
|
+
|
|
59
|
+
```python
|
|
60
|
+
from traceground import check_answer
|
|
61
|
+
|
|
62
|
+
result = check_answer(
|
|
63
|
+
"Acme's Q3 revenue was $42.1 million, up 8% from Q2.",
|
|
64
|
+
sources=["Acme's revenue for the third quarter was $41.2 million, up 8% from the second quarter."],
|
|
65
|
+
)
|
|
66
|
+
print(result.to_text())
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
PROBLEMS: 2 numbers checked against 1 source: 1 number not supported.
|
|
71
|
+
1. [number] "42.1 million" in: Acme's Q3 revenue was $42.1 million, up 8% from Q2.
|
|
72
|
+
"42.1 million" is not in the sources or the question. The closest values they give: 41.2 million (S1).
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
From the command line (exit code 0 ok, 1 problems, 2 usage error, 3 unsure, for use in CI):
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
traceground --answer answer.txt --sources report.txt --question question.txt
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## The number check (implemented)
|
|
82
|
+
|
|
83
|
+
A number in the answer is supported when the sources or the question contain it at the precision the
|
|
84
|
+
answer wrote it with: "1.98" is supported by 1.9799 and "about 24" by 23.7. Thousands separators,
|
|
85
|
+
scientific notation, scale words ("41.2 million"), 7% for 0.07, "about 12,000" for 12,250, and number
|
|
86
|
+
words and fractions in the sources ("twenty", "1/2") are understood. Citation markers, labels such as
|
|
87
|
+
"Eq. (3)" or "Passage 2", list numbers, cited years, URLs, code, digits glued to words (Q3, CO2, MX-200,
|
|
88
|
+
1st), the answer counting its own length ("in 150 words", "a 98-word summary"), a rating scale ("out of
|
|
89
|
+
5"), "24/7" and plain integers from 0 to 10 are not checked: when unsure, it says nothing. A number cut
|
|
90
|
+
short (1.97 for 1.9799) is reported as minor.
|
|
91
|
+
|
|
92
|
+
A scale in a key name is understood (revenue_musd = 1840.5 supports "$1.84 billion"), and so is a duration
|
|
93
|
+
written in two units ("535 minutes, about 8 hours 55 minutes"). Clock times are compared as times ("2:30
|
|
94
|
+
pm" against "14:30"), and the day of a date is checked even with an ordinal ("April 19th"). A conversion
|
|
95
|
+
in brackets, "400°F (200°C)", is reported as minor when it is right (within 3%, or 6 degrees) and as
|
|
96
|
+
unsupported when it is wrong. A tool result's list counts as giving its length ("based on 3 reviews").
|
|
97
|
+
|
|
98
|
+
A percentage is not supported by a bare fraction below 1: "0.075%" is not 0.075 (that is 7.5%),
|
|
99
|
+
unless the source writes it as a percentage or under a key such as change_pct.
|
|
100
|
+
|
|
101
|
+
A number the answer computes from numbers it states, which the sources support, is accepted when its
|
|
102
|
+
sentence says it is computed: "315 EUR and 234 EUR, a total of 549 EUR" (a sum or difference of two
|
|
103
|
+
of them, with a word such as "total", "together", "extra", "warmer"), or "3 lamps at 39.90 each cost
|
|
104
|
+
119.70" (a stated number times a count from the question, with "each", "per", "times"). It must round
|
|
105
|
+
to the written number at the precision it is written in: "a total of 550 EUR" is reported. A source number times a count from the question
|
|
106
|
+
("3 lamps" at 39.90 each is 119.70) is reported as minor: right, but worked out in the answer rather
|
|
107
|
+
than by a tool. Thin and narrow no-break spaces work as thousands separators ("40 983.61").
|
|
108
|
+
|
|
109
|
+
Not yet: signs are not compared, units are converted only in a pair written together ("5 km (3.1
|
|
110
|
+
miles)"), and a value computed from numbers the answer does not state (a sum of two source values that
|
|
111
|
+
the answer leaves out) is reported as unsupported.
|
|
112
|
+
|
|
113
|
+
## The quote check (implemented)
|
|
114
|
+
|
|
115
|
+
Every quotation of three words or more must occur in the source its sentence cites ([S1], [doc2],
|
|
116
|
+
[report.txt], or [2] for the second source), or in any source if the sentence cites none. Case,
|
|
117
|
+
accents, dashes, quote marks, spacing and punctuation do not matter; the words do. Ellipses and
|
|
118
|
+
editorial insertions in square brackets are allowed. A quote that differs only in small words (a, the,
|
|
119
|
+
of, my ...), punctuation, the spelling of one word, or singular and plural is reported as minor. A quote
|
|
120
|
+
that is not found is reported with the closest wording the sources contain; a quote found in another
|
|
121
|
+
source than the one cited is reported with the source it is really in.
|
|
122
|
+
|
|
123
|
+
```
|
|
124
|
+
The quote is not in the cited source. The closest wording, in S1: "does not guarantee that the
|
|
125
|
+
reconstructed matrix is positive".
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
Not checked: quotes of one or two words, titles (every main word capitalized), quotes in code, quotes in
|
|
129
|
+
single quote marks, example text ('a subject line such as "..."') and the question restated.
|
|
130
|
+
|
|
131
|
+
## Tool traces (implemented)
|
|
132
|
+
|
|
133
|
+
Give the tool calls as a list of `{"name", "args", "result"}`, or as a chat (then the question and the
|
|
134
|
+
final answer are taken from it: `check_messages(messages)`). A chat can be OpenAI-style messages,
|
|
135
|
+
Anthropic Messages API messages (`tool_use` and `tool_result` blocks), or LangChain messages (the
|
|
136
|
+
objects, or the dicts of `messages_to_dict`). Sources can be given at the same time, for an agent that
|
|
137
|
+
reads documents and calls tools. The tool results and arguments become evidence for the number and quote
|
|
138
|
+
checks, and two more checks run.
|
|
139
|
+
|
|
140
|
+
**Argument binding**: the values in the question must be the ones the tools were called with. Every
|
|
141
|
+
number below is in the tool result or the question, so a number check passes this answer; the error is
|
|
142
|
+
that the tool computed 3 years, not 5.
|
|
143
|
+
|
|
144
|
+
```python
|
|
145
|
+
result = check_answer(
|
|
146
|
+
"Over 5 years at 7% you pay 12,250 in total, 340.29 a month.",
|
|
147
|
+
question="What is the 5-year cost at 7% interest?",
|
|
148
|
+
trace=[{"name": "loan_cost", "args": {"rate": 0.07, "years": 3},
|
|
149
|
+
"result": {"total": 12250.43, "monthly": 340.29}}],
|
|
150
|
+
)
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
```
|
|
154
|
+
PROBLEMS: 3 numbers and 2 value-to-field bindings checked against 1 tool call and the question: 1 tool argument different from the question.
|
|
155
|
+
1. [argument] "5" in: Over 5 years at 7% you pay 12,250 in total, 340.29 a month.
|
|
156
|
+
The question asks for 5-year, but loan_cost was called with years=3.
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
It also reports a question value whose digits the call shifted by a power of ten (asked p = 0.3,
|
|
160
|
+
called with 0.03) and two values swapped between two calls (asked 120 USD and 80 GBP, called with
|
|
161
|
+
amount=80 for USD and amount=120 for GBP).
|
|
162
|
+
|
|
163
|
+
**Value-to-field binding**: a number the answer attaches to a field name must be the value the tool
|
|
164
|
+
gave for that field. Field names come from the JSON keys (`rain_chance_pct` is "rain chance";
|
|
165
|
+
`field_synonyms` adds names).
|
|
166
|
+
|
|
167
|
+
```python
|
|
168
|
+
result = check_answer(
|
|
169
|
+
"Jordan Reyes scored 27 points with 7 rebounds and 11 assists.",
|
|
170
|
+
trace=[{"name": "player_game_stats", "args": {"player": "Jordan Reyes"},
|
|
171
|
+
"result": {"points": 27, "rebounds": 11, "assists": 7}}],
|
|
172
|
+
)
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
```
|
|
176
|
+
PROBLEMS: 2 numbers and 3 value-to-field bindings checked against 1 tool call: 2 values attached to the wrong field.
|
|
177
|
+
1. [value-to-field] "7 rebounds" in: Jordan Reyes scored 27 points with 7 rebounds and 11 assists.
|
|
178
|
+
"7 rebounds": the tool results give rebounds = 11 (player_game_stats). 7 is the assists (player_game_stats).
|
|
179
|
+
2. [value-to-field] "11 assists" in: Jordan Reyes scored 27 points with 7 rebounds and 11 assists.
|
|
180
|
+
"11 assists": the tool results give assists = 7 (player_game_stats). 11 is the rebounds (player_game_stats).
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
A field can also be named in other ways: by a unit (in "weighs 249 kg" the 249 is compared with the
|
|
184
|
+
results' only kg field, and reported when it is the value of another field of the same call), by a
|
|
185
|
+
word only one multi-word field name has ("€789.52 in interest" for total_interest), by the subject of
|
|
186
|
+
"... is N" ("The total interest you'll pay on the loan is $3,034.15"), and by a range ("between $69.80
|
|
187
|
+
and $72.90" for day_low and day_high). The right value in another currency is reported when no call
|
|
188
|
+
converted it ("€221.4 million" when the tool gives net_income_musd = 221.4).
|
|
189
|
+
|
|
190
|
+
Both checks stay silent when a binding is unsure: no field word right next to the number (a
|
|
191
|
+
preposition such as "for" or "on" starts a new phrase: "the monthly payment for a $30,000 loan"), a number in
|
|
192
|
+
another unit than the field ("91.4 °F" for high_c), a clause that says the number is worked out ("the
|
|
193
|
+
difference in monthly payment is $166") or a comparative follows it ("3.3 hPa higher"), a year ("in
|
|
194
|
+
2024"), a sentence with "respectively", a question number
|
|
195
|
+
with no argument that names it or resembles it, a number that is part of a name or an address in the
|
|
196
|
+
results ("4 Eggs & Pizza", "223 Anacapa St").
|
|
197
|
+
|
|
198
|
+
From the command line: `traceground --trace chat.json`, or `traceground --trace calls.json --answer
|
|
199
|
+
answer.txt --question question.txt`.
|
|
200
|
+
|
|
201
|
+
## What the answer says about its tools, and its own wording (implemented)
|
|
202
|
+
|
|
203
|
+
* **Failed calls**: a call failed, was not called again successfully (nor replaced by another tool
|
|
204
|
+
called about the same thing), and the answer reads as if it had worked.
|
|
205
|
+
* **Tools never called**: "I searched the web" with no search tool, "I ran the code", "I queried the
|
|
206
|
+
database", "I used the X tool". Offers ("I can search the web") and advice are not claims.
|
|
207
|
+
* **Thinking aloud** left in the final answer: "Wait, let me check:", "Let me report the tool's
|
|
208
|
+
numbers:", a paragraph that takes back what came before. Ordinary openers ("Let me explain") are fine.
|
|
209
|
+
* **Self-contradiction**: "The passages do not say how long the warranty lasts." followed by a sentence
|
|
210
|
+
that gives it from the sources. A statement about one passage ("Passage 3 does not mention it") is not
|
|
211
|
+
a contradiction when another passage gives it.
|
|
212
|
+
|
|
213
|
+
```python
|
|
214
|
+
result = check_answer(
|
|
215
|
+
"I searched the web and found that the MX-200 warranty is 2 years. Wait, let me check: yes, 2 years.",
|
|
216
|
+
trace=[{"name": "lookup_product", "args": {"sku": "MX-200"}, "result": {"warranty_years": 2}}],
|
|
217
|
+
)
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
```
|
|
221
|
+
PROBLEMS: 2 value-to-field bindings checked against 1 tool call: 1 claim about a tool that was not called; 1 passage of the model thinking aloud.
|
|
222
|
+
1. [tool-not-called] "I searched the web" in: I searched the web and found that the MX-200 warranty is 2 years.
|
|
223
|
+
The answer says "I searched the web", but no tool for a web search was called (calls: lookup_product).
|
|
224
|
+
2. [self-talk] "Wait, let me check:" in: Wait, let me check: yes, 2 years.
|
|
225
|
+
This is the model thinking aloud, left in the final answer: "Wait, let me check:".
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
## Measured on RAGTruth
|
|
229
|
+
|
|
230
|
+
On the test split of [RAGTruth](https://github.com/ParticleMedia/RAGTruth), 2,675 real answers by six models
|
|
231
|
+
with hallucinations marked by people, measured once with 0.1.0.dev12 (whose checks 0.1.0 keeps unchanged)
|
|
232
|
+
after development on the train split:
|
|
233
|
+
7 of traceground's 172 high-severity findings (4%) were false alarms, by hand review. It flagged 85 of the
|
|
234
|
+
306 marked spans that contain a digit, and 8% to 16% of the answers with any marked span: most marked
|
|
235
|
+
hallucinations are claims in words, which it does not check. Details, and how to rerun it, in
|
|
236
|
+
[eval/README.md](https://github.com/animeshQKD/traceground/blob/main/eval/README.md).
|
|
237
|
+
|
|
238
|
+
## Demo
|
|
239
|
+
|
|
240
|
+
`space/app.py` is a Gradio page for trying traceground in a browser: paste an answer with its sources, its
|
|
241
|
+
tool calls (or a whole chat), or both, and the page marks every finding on the answer, shows the evidence
|
|
242
|
+
for each, and greys out the sentences it has nothing to check in. It runs on your own computer and needs no
|
|
243
|
+
API key. From a copy of this repository:
|
|
244
|
+
|
|
245
|
+
```
|
|
246
|
+
pip install traceground "gradio[mcp]==6.32.0"
|
|
247
|
+
python space/app.py
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
then open http://127.0.0.1:7860. The same check is an MCP tool, `traceground_check`, at
|
|
251
|
+
http://127.0.0.1:7860/gradio_api/mcp/ while the page runs ([space/README.md](https://github.com/animeshQKD/traceground/blob/main/space/README.md)).
|
|
252
|
+
|
|
253
|
+
## Planned next
|
|
254
|
+
|
|
255
|
+
* Citations mode: marker styles, every marker checked, every factual sentence cited (optional).
|
|
256
|
+
* An opt-in automatic fix where a fix is certain (a number cut short, thinking aloud removed).
|
|
257
|
+
* The demo published as a Hugging Face Space, and LangGraph / LangChain helpers.
|
|
258
|
+
* Report-only stays the default; when a check is unsure, it says nothing.
|
|
259
|
+
|
|
260
|
+
## Install
|
|
261
|
+
|
|
262
|
+
```
|
|
263
|
+
pip install traceground
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
Python 3.10 to 3.14. The package depends on the standard library only. To work on it, from a copy of this
|
|
267
|
+
repository: `pip install -e ".[test]"`, then `python -m pytest`.
|
|
268
|
+
|
|
269
|
+
## License
|
|
270
|
+
|
|
271
|
+
MIT
|