traceground 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. traceground-0.1.0/.gitignore +25 -0
  2. traceground-0.1.0/CHANGELOG.md +271 -0
  3. traceground-0.1.0/LICENSE +21 -0
  4. traceground-0.1.0/PKG-INFO +271 -0
  5. traceground-0.1.0/README.md +242 -0
  6. traceground-0.1.0/pyproject.toml +52 -0
  7. traceground-0.1.0/src/traceground/__init__.py +26 -0
  8. traceground-0.1.0/src/traceground/__main__.py +6 -0
  9. traceground-0.1.0/src/traceground/api.py +187 -0
  10. traceground-0.1.0/src/traceground/binding.py +700 -0
  11. traceground-0.1.0/src/traceground/citations.py +39 -0
  12. traceground-0.1.0/src/traceground/cli.py +93 -0
  13. traceground-0.1.0/src/traceground/numbers.py +596 -0
  14. traceground-0.1.0/src/traceground/py.typed +0 -0
  15. traceground-0.1.0/src/traceground/quotes.py +238 -0
  16. traceground-0.1.0/src/traceground/result.py +55 -0
  17. traceground-0.1.0/src/traceground/selftalk.py +124 -0
  18. traceground-0.1.0/src/traceground/textnorm.py +139 -0
  19. traceground-0.1.0/src/traceground/toolchecks.py +130 -0
  20. traceground-0.1.0/src/traceground/traces.py +252 -0
  21. traceground-0.1.0/tests/data/real_answers_2026-11-17.json +1210 -0
  22. traceground-0.1.0/tests/test_api.py +82 -0
  23. traceground-0.1.0/tests/test_binding.py +162 -0
  24. traceground-0.1.0/tests/test_cli.py +116 -0
  25. traceground-0.1.0/tests/test_examples.py +21 -0
  26. traceground-0.1.0/tests/test_message_formats.py +89 -0
  27. traceground-0.1.0/tests/test_numbers.py +167 -0
  28. traceground-0.1.0/tests/test_quotes.py +133 -0
  29. traceground-0.1.0/tests/test_real_answers.py +328 -0
  30. traceground-0.1.0/tests/test_rules_dev12.py +112 -0
  31. traceground-0.1.0/tests/test_smoke.py +58 -0
  32. traceground-0.1.0/tests/test_survey_cases.py +64 -0
  33. traceground-0.1.0/tests/test_textnorm.py +29 -0
  34. traceground-0.1.0/tests/test_traces.py +98 -0
  35. traceground-0.1.0/tests/test_wording_and_tools.py +149 -0
@@ -0,0 +1,25 @@
1
+ # secrets: keys go in .env locally, never in git
2
+ .env
3
+ .env.*
4
+ !.env.example
5
+ # RAGTruth (eval/ragtruth.py): downloaded data and review files with its texts stay local
6
+ eval/data/
7
+ # survey logs hold local paths; the results in survey/results/ are kept
8
+ survey/survey_log*.txt
9
+ logs/
10
+ # Python and build output
11
+ __pycache__/
12
+ *.pyc
13
+ .pytest_cache/
14
+ .ruff_cache/
15
+ .mypy_cache/
16
+ build/
17
+ dist/
18
+ *.egg-info/
19
+ .venv/
20
+ venv/
21
+ *.zip
22
+ # editors
23
+ .idea/
24
+ .vscode/
25
+ .ipynb_checkpoints/
@@ -0,0 +1,271 @@
1
+ # Changelog
2
+
3
+ All notable changes to traceground are listed here. Versions follow [Semantic Versioning](https://semver.org/).
4
+
5
+ ## 0.1.0 (first release)
6
+
7
+ The first version published on PyPI. Its checks are those of 0.1.0.dev12 below, unchanged; the version
8
+ number, the README and the demo's requirements are what changed.
9
+
10
+ ## 0.1.0.dev12 (not published)
11
+
12
+ Rules found by measuring on the RAGTruth train split (eval/ragtruth.py); each removes a kind of false
13
+ alarm or makes a finding more precise. The dev and held-out evaluations and the survey give the same
14
+ numbers as with 0.1.0.dev11.
15
+
16
+ ### Added
17
+
18
+ - Clock times: "2:30 pm" is checked against the times in the evidence, which may be in 24-hour form
19
+ ("14:30", or opening hours such as "11:0-2:30", which close at 2:30 at night, so "2:30 pm" is reported).
20
+ - A conversion in brackets, "400°F (200°C)" or "5 km (3.1 miles)", is reported as minor (worked out in
21
+ the answer) when it is right (within 3%, or 6 degrees); a wrong one is reported as unsupported.
22
+ - The day of a date written with an ordinal ("April 19th", "the 19th of April") is checked; other
23
+ ordinals ("1st place") are not.
24
+ - Quotes: a quote that differs from the source only in small words (a, the, and, of, my, those ...),
25
+ punctuation, the spelling of one word or singular and plural is reported as minor ("The quote differs
26
+ slightly from S1"), with a new count, quotes_changed_slightly. A changed word that changes the meaning
27
+ ("can" for "cannot") is still reported.
28
+ - `eval/ragtruth.py` (repository only): measures traceground on RAGTruth. The data is downloaded by
29
+ the user into eval/data/ (gitignored); the results hold numbers and ids only. Test split, measured once
30
+ with this version: 7 of 172 high findings were false alarms by hand review; 85 of the 306 marked spans
31
+ that contain a digit were flagged (eval/README.md).
32
+
33
+ ### Changed (repository only)
34
+
35
+ - Demo: a verdict banner; the answer with each finding marked and numbered, a legend of only the kinds
36
+ that occur, and the sentences with no number or quote in grey (with how many were checked: claims in
37
+ words are not); each finding as a card with its severity and "Show the evidence", which opens the source
38
+ or tool result with the value marked; "Use this check in your code" (the Python that gives the same
39
+ report); the report as JSON or Markdown to download. The answer box comes first, and the outputs appear
40
+ after the first check. Labels use the words of the summary ("quote not found", "value attached to the
41
+ wrong field").
42
+ - Demo: five real examples, unedited chats from the evaluation (gpt-oss-120b and gpt-oss-20b on held-out
43
+ questions): a correct answer with a sentence that is not checked, a departure time no tool gave, US
44
+ dollars reported as euros, a growth rate worked out wrong (14.4% for 14.2%), and true numbers from the
45
+ model's own knowledge. The made-up currency example they replace is gone. Markdown bold marks (**) are
46
+ not shown on the marked answer.
47
+ - Demo as an API and an MCP tool: one endpoint, `traceground_check` (answer, sources, tool calls or a
48
+ whole chat, question; returns the report as JSON). `python space/app.py` starts the MCP server too when
49
+ gradio[mcp] is installed (http://127.0.0.1:7860/gradio_api/mcp/); `space/mcp_check.py` connects to it
50
+ as an MCP client and checks two answers. The page's own endpoint is no longer public.
51
+
52
+ ### Fixed
53
+
54
+ - Not checked any more: the answer counting its own length ("in 150 words", "a 98-word summary"), a
55
+ rating scale ("out of 5", "4/5"), "24/7", labels such as "Passage 2" and "Review 3".
56
+ - How many items a list in a tool result has counts as given ("based on 3 reviews" for a list of three
57
+ reviews), and is not read as a field value; years in date strings are not field values either.
58
+ - Value-to-field: a number that is part of a name or an address in a result ("4 Eggs & Pizza", "223
59
+ Anacapa St") is not a field value; after "is", a word inside a phrase that starts with a preposition
60
+ ("the rating ... according to customer reviews is 3.5") does not name the field; a more specific
61
+ field named next to the number ("one review gave it 1 star": review_stars) is the one meant.
62
+ - Quotes: HTML entities (&) and JSON escapes (\") are ignored; text between two quotations is not a
63
+ quote; example text ('a subject line such as "..."') and the question restated are not quotes.
64
+ - Self-contradiction: "Passage 3 does not mention X" followed by X from another passage is not a
65
+ contradiction; a question or a heading ("follow these steps:") does not count as giving the thing.
66
+
67
+ ## 0.1.0.dev11 (not published)
68
+
69
+ ### Added
70
+
71
+ - `check_messages()` also reads Anthropic Messages API chats (tool_use and tool_result blocks, is_error)
72
+ and LangChain messages (the objects, or the dicts of messages_to_dict; a ToolMessage with status
73
+ "error" is a failed call). No new dependency: the formats are read as they are.
74
+
75
+ ### Fixed
76
+
77
+ - The summary counted a number twice when a quote or a value-to-field finding replaced the number
78
+ finding ("1 number not supported; 1 value attached to the wrong field" for one finding).
79
+ - A changed quote whose nearest wording is far off (similarity 0.4 to 0.6, three words or more in
80
+ common) now shows that wording ("says it differently: ...") instead of "nothing in them is close".
81
+
82
+ ### Changed (repository only)
83
+
84
+ - Demo: one form with optional sources and optional tool calls (or a chat), so an agent that uses both
85
+ can be checked; one endpoint, check. A "documents and a tool together" example; a how-to section; a
86
+ currency sign or quote marks next to a finding are marked with it; the label "wrong value for its
87
+ field"; a wider Severity column.
88
+
89
+ ## 0.1.0.dev10 (not published)
90
+
91
+ ### Added (repository only)
92
+
93
+ - `space/`: a Gradio demo (Gradio 6.32), run locally with `python space/app.py`. Two tabs: an answer with
94
+ sources, and an answer with tool calls or a chat. Marks each finding on the answer, lists the findings,
95
+ and shows the full JSON report. Two named endpoints, check_sources and check_trace. Made-up examples,
96
+ one per check. Not published.
97
+
98
+ ### Fixed
99
+
100
+ - Argument check: when a question's words fit two arguments, the one they name best is reported ("5-year"
101
+ is years=3, not annual_rate, although "annual" means yearly).
102
+
103
+ ## 0.1.0.dev9 (not published)
104
+
105
+ ### Fixed (found on the held-out set; from here on, held-out numbers are no longer blind)
106
+
107
+ - Numbers: a duration in two units whose total is a source value ("535 minutes, 8 hours 55 minutes").
108
+ - Numbers and value-to-field: a scale in a key name (revenue_musd = 1840.5 supports "$1.84 billion").
109
+ - Value-to-field: the right value in another currency with no converting call is reported ("€221.4
110
+ million" for net_income_musd), instead of passing as "another unit".
111
+ - Value-to-field: unit words inside keys (co2_ppm, pressure_hpa, kcal) no longer name a field on their
112
+ own ("typical levels of 400-600 ppm" is a number from outside the tools, not a wrong co2_ppm).
113
+ - Value-to-field: years ("in 2024", or a year the calls give) and numbers followed by a comparative
114
+ ("3.3 hPa higher", "7 °C warmer") are not compared with fields; result fields named year, date or
115
+ timestamp are not compared with numbers ("year-over-year").
116
+
117
+ ## 0.1.0.dev8 (not published)
118
+
119
+ ### Changed: value-to-field finds more wrong fields (developed on the dev set only)
120
+
121
+ - A lone word that only one multi-word field name has binds the number ("€789.52 in interest" for
122
+ total_interest); measure nouns ("years", "hours") and words several fields share ("total") do not.
123
+ - The subject of "... is N" binds N ("The total interest you'll pay on the loan is $3,034.15").
124
+ - The two ends of a range bind to a low/high or min/max pair ("between $69.80 and $72.90", "6-10 days").
125
+ - Time units name fields too ("249 hours" against battery_hours), like kg and $ in dev7.
126
+ - "min" and "max" next to a time noun in a key mean minimum and maximum (days_min), not minutes.
127
+ - Words ending in -ed are stemmed ("converted_amount" matches "converts"; "interest_earned", "earn").
128
+ - "more points (31)": a bracket that opens on the number no longer hides the field before it.
129
+ - Not compared: a value another call worked out from the field (a calculator given price_usd), and a
130
+ number whose clause says it is worked out ("the difference in monthly payment is $166.03").
131
+ - Argument check: a percentage is not taken for an argument whose name has no rate word ("4%
132
+ compounded" is not compounds_per_year).
133
+
134
+ ### Added (repository only)
135
+
136
+ - A held-out set: `eval/questions_heldout.json` (20 questions) and seven new tools in `eval/tools.py`
137
+ (flights, invoices, company figures, nutrition, sensors, shipping, cities). `--set heldout` in
138
+ run_eval.py, score.py and mutate.py; runs go to `eval/runs/heldout/`.
139
+ - mutate.py leaves numbers inside dates alone (changing 2026-11-16 is not a wrong field value).
140
+
141
+ ## 0.1.0.dev7 (not published)
142
+
143
+ ### Changed (found with errors planted in the real answers; see eval/results/review_2026-11-17.md)
144
+
145
+ - Numbers: a value computed from numbers the answer states is accepted only when its sentence says so
146
+ ("total", "together", "extra", "warmer" ... for sums and differences; "each", "per", "times" for a
147
+ count from the question). Products and ratios of two stated numbers are no longer tried. Before,
148
+ chance sums hid errors: a date "2026-11-61" passed as 27 + 34.
149
+ - Numbers: a percentage is not supported by a bare fraction below 1 ("0.075%" by annual_rate 0.075),
150
+ unless the evidence writes it as a percentage or under a key such as change_pct. Worked-out
151
+ percentages are compared as fractions ("0.15% of 240 is 36" is reported).
152
+ - Value-to-field: a unit names a field when the results have only one field with that unit ("weighs
153
+ 249 kg" against weight_kg); it is reported when the value is another field of the same call.
154
+
155
+ ### Added (repository only)
156
+
157
+ - `eval/mutate.py`: plants five kinds of grounding error in the real answers that are correct by hand
158
+ and counts how many traceground reports; `eval/hand_labels.json` lists the answers found wrong.
159
+
160
+ ## 0.1.0.dev6 (not published)
161
+
162
+ ### Fixed (from the second real-answer run, Groq openai/gpt-oss-20b, 2026-11-17: 4 false alarms on 44 correct answers, now 0)
163
+
164
+ - Value-to-field: "on" also ends the search for a field name ("the monthly payment on a $20,000 loan").
165
+ - Numbers: a thin, narrow no-break or no-break space between groups of three digits is a thousands
166
+ separator ("40 983.61"); a plain space is not.
167
+ - Numbers: a source number times or divided by a whole count from the question ("3 lamps" at 39.9 is
168
+ 119.70) is a minor finding, "worked out in the answer, not by a tool", instead of an unsupported
169
+ number. New stat `numbers_computed_in_answer`.
170
+
171
+ ### Added (repository only)
172
+
173
+ - Six gpt-oss-20b answers in `tests/data/real_answers_2026-11-17.json`, with wrong-number versions that
174
+ must still be caught.
175
+
176
+ ## 0.1.0.dev5 (not published)
177
+
178
+ ### Fixed (from the first real-answer run, Groq openai/gpt-oss-120b, 2026-11-17: 3 false alarms on 43 correct answers, now 0)
179
+
180
+ - Value-to-field: a preposition before the number ("for", "from", "per", ...) ends the search for a
181
+ field name, so "the monthly payment for a $30,000 loan" no longer ties 30,000 to monthly_payment.
182
+ - Value-to-field: a number written in another unit of the same kind as the field's unit suffix
183
+ ("91.4 °F" for high_c, "€165" for price_usd) is not compared with that field.
184
+ - Numbers: a number the answer computes from numbers it states and the evidence supports is accepted
185
+ (sum, difference, product, ratio, a stated number times a question number, the sum of all), at the
186
+ precision it is written in. Wrong arithmetic is still reported. New stat `numbers_derived`.
187
+
188
+ ### Added (repository only)
189
+
190
+ - `tests/test_real_answers.py` and `tests/data/real_answers_2026-11-17.json`: seven real answers checked
191
+ by hand, with wrong-number versions that must still be caught.
192
+
193
+
194
+ ### Added (repository only, 0.1.0.dev4)
195
+
196
+ - `eval/`: the real-answer evaluation. Deterministic local tools, 44 questions with the calls and values
197
+ a correct answer needs, a standard-library client for OpenAI-compatible endpoints (Groq, Hugging Face,
198
+ Gemini; OpenAI only with ALLOW_OPENAI=1) with a request cap, an agent loop that records each chat, and
199
+ a scorer that labels answers and lists every disagreement with traceground for review by hand.
200
+
201
+ ### Added
202
+
203
+ - Failed calls (`toolchecks.py`): a failed call that was not called again successfully, nor replaced by
204
+ another tool called about the same thing, and that the answer reports as if it had worked.
205
+ - Tools never called: claims of a web search, of running code, of a database query, or of using a named
206
+ tool, with no such call in the trace.
207
+ - Thinking aloud (`selftalk.py`, ported from the Quantum Information Agent): "Wait, let me check:",
208
+ "Let me report the tool's numbers:", retractions, "X? No -"; quoted text and ordinary openers are not
209
+ flagged. Deleting a self-check is offered as a certain fix.
210
+ - Self-contradiction (ported from "Ask Your Quantum Physics Research Papers"): "the sources do not say
211
+ X" followed by X from the sources.
212
+ - `check_answer(..., check_wording=True)`, `--no-wording`. Thinking aloud is a problem even without
213
+ evidence.
214
+
215
+ ## 0.1.0.dev3
216
+
217
+ ### Added
218
+
219
+ - Tool traces (`traces.py`): a list of `{name, args, result}` (other common key names too) or
220
+ OpenAI-style chat messages; failed calls recognised; `check_answer(..., trace=...)` and
221
+ `check_messages(messages)`; the command line's `--trace FILE`. Tool results and arguments are
222
+ evidence for the number and quote checks.
223
+ - Value-to-field binding (`binding.py`): a number attached to a field name ("concurrence = 0.7",
224
+ "7 rebounds", "4% chance of rain") must be the value a tool result gives for that field;
225
+ `field_synonyms` adds names.
226
+ - Argument binding: a question value that no call used while a call used the same digits shifted by a
227
+ power of ten; a question value whose neighbouring word names an argument called with another value;
228
+ two values swapped between two calls.
229
+ - The survey gives traceground the trace cases as traces.
230
+
231
+ ## 0.1.0.dev2
232
+
233
+ ### Added
234
+
235
+ - The quote check: quotations of three words or more must occur in the source their sentence cites, or
236
+ in any source; ellipses and [editorial insertions] allowed; the closest real wording, or the source
237
+ the quote is really in, is reported. Titles, quotes in code and single-quoted text are not checked.
238
+ - `citations.py`: which sources a sentence's markers name ([S1], [doc2], [report.txt], [2]). A sentence
239
+ that is only markers ("... night. [S1]") belongs to the sentence before.
240
+ - `check_answer(..., check_quotes=True, min_quote_words=3)`; summaries name what was checked and found.
241
+
242
+ ### Fixed
243
+
244
+ - A percentage read as a fraction (70% as 0.70) is no longer offered "1" as its closest value.
245
+ - The source id "question" is refused for every shape of sources, as it names the question.
246
+
247
+ ## 0.1.0.dev1
248
+
249
+ ### Added
250
+
251
+ - `check_answer(answer, sources, question)` and the `Result` / `Finding` model (verdict, findings with
252
+ span, sentence, plain message, evidence and suggestion; `to_dict`, `to_json`, `to_text`).
253
+ - The number check over plain sources: rounding to the precision written, thousands separators
254
+ (Indian grouping too), scientific notation, scale words, percent and fraction, approximate round
255
+ numbers, number words and fractions in the evidence; citation markers, reference labels, list
256
+ numbers, cited years, URLs, e-mail addresses, code, digits glued to words and plain integers 0 to 10
257
+ are not checked. A number cut short (1.97 for
258
+ 1.9799) is a minor finding with the certain fix.
259
+ - `textnorm`, ported from "Ask Your Quantum Physics Research Papers" and generalized.
260
+ - The command line: `traceground --answer FILE --sources FILE ... [--question FILE] [--json]`, with exit
261
+ codes 0 ok, 1 problems, 2 usage error, 3 unsure.
262
+ - The survey runs traceground too, and never overwrites an earlier run's files.
263
+
264
+ ## [0.0.1]
265
+
266
+ ### Added
267
+
268
+ - The package skeleton: `pyproject.toml` (src layout, no dependencies), the `traceground` command with
269
+ `--version`, smoke tests, and CI on Python 3.10 to 3.14 (Linux) and 3.11 and 3.14 (Windows).
270
+ - Publishing to TestPyPI and PyPI with Trusted Publishing.
271
+ - `survey/` (repository only, not in the package): the Phase-0 survey of existing checkers on 20 cases.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Animesh Sinha Roy
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,271 @@
1
+ Metadata-Version: 2.5
2
+ Name: traceground
3
+ Version: 0.1.0
4
+ Summary: Checks an AI answer against its sources or its tool-call trace: every number, quote, value and argument the evidence doesn't support. No model needed.
5
+ Project-URL: Homepage, https://github.com/animeshQKD/traceground
6
+ Project-URL: Issues, https://github.com/animeshQKD/traceground/issues
7
+ Project-URL: Changelog, https://github.com/animeshQKD/traceground/blob/main/CHANGELOG.md
8
+ Author: Animesh Sinha Roy
9
+ License-Expression: MIT
10
+ License-File: LICENSE
11
+ Keywords: agents,evaluation,grounding,guardrails,hallucination,llm,rag,tool-calling
12
+ Classifier: Development Status :: 3 - Alpha
13
+ Classifier: Intended Audience :: Developers
14
+ Classifier: Operating System :: OS Independent
15
+ Classifier: Programming Language :: Python :: 3
16
+ Classifier: Programming Language :: Python :: 3 :: Only
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Programming Language :: Python :: 3.14
22
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
23
+ Classifier: Topic :: Software Development :: Quality Assurance
24
+ Classifier: Typing :: Typed
25
+ Requires-Python: >=3.10
26
+ Provides-Extra: test
27
+ Requires-Dist: pytest>=8; extra == 'test'
28
+ Description-Content-Type: text/markdown
29
+
30
+ # traceground
31
+
32
+ [![tests](https://github.com/animeshQKD/traceground/actions/workflows/tests.yml/badge.svg)](https://github.com/animeshQKD/traceground/actions/workflows/tests.yml)
33
+ [![PyPI](https://img.shields.io/pypi/v/traceground)](https://pypi.org/project/traceground/)
34
+ [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](https://github.com/animeshQKD/traceground/blob/main/LICENSE)
35
+
36
+ **Status: alpha (0.1.0).** It has the number and quote checks over sources; the tool-trace checks
37
+ (value-to-field and argument binding, failed calls, tools never called); the wording checks (thinking
38
+ aloud, self-contradiction); `check_answer()`, `check_messages()`, the command line, and a Gradio demo
39
+ that is also an MCP tool (`space/`). Rules and messages may still change between 0.x versions.
40
+
41
+ ## What it is for
42
+
43
+ traceground takes an AI answer together with its sources or its tool-call trace, and lists every
44
+ number, quote, value or argument that the evidence doesn't support. Each finding names the exact span,
45
+ the reason, and what the evidence actually says. No language model is involved, so the same input
46
+ always gives the same verdict.
47
+
48
+ An example of the error it is built for:
49
+
50
+ * **Question:** "What is the 5-year cost at 7% interest?"
51
+ * **Tool call:** `loan_cost(rate=0.07, years=3)` returned `{"total": 12250.43, "monthly": 340.29}`
52
+ * **Answer:** "Over 5 years at 7% you pay 12,250 in total, 340.29 a month."
53
+
54
+ Every number in the answer matches the tool result or the question (12,250 is 12,250.43 rounded), so
55
+ a check of numbers alone passes it. The error is that the tool was called with `years=3` when the question asks for 5 years.
56
+
57
+ ## Use
58
+
59
+ ```python
60
+ from traceground import check_answer
61
+
62
+ result = check_answer(
63
+ "Acme's Q3 revenue was $42.1 million, up 8% from Q2.",
64
+ sources=["Acme's revenue for the third quarter was $41.2 million, up 8% from the second quarter."],
65
+ )
66
+ print(result.to_text())
67
+ ```
68
+
69
+ ```
70
+ PROBLEMS: 2 numbers checked against 1 source: 1 number not supported.
71
+ 1. [number] "42.1 million" in: Acme's Q3 revenue was $42.1 million, up 8% from Q2.
72
+ "42.1 million" is not in the sources or the question. The closest values they give: 41.2 million (S1).
73
+ ```
74
+
75
+ From the command line (exit code 0 ok, 1 problems, 2 usage error, 3 unsure, for use in CI):
76
+
77
+ ```
78
+ traceground --answer answer.txt --sources report.txt --question question.txt
79
+ ```
80
+
81
+ ## The number check (implemented)
82
+
83
+ A number in the answer is supported when the sources or the question contain it at the precision the
84
+ answer wrote it with: "1.98" is supported by 1.9799 and "about 24" by 23.7. Thousands separators,
85
+ scientific notation, scale words ("41.2 million"), 7% for 0.07, "about 12,000" for 12,250, and number
86
+ words and fractions in the sources ("twenty", "1/2") are understood. Citation markers, labels such as
87
+ "Eq. (3)" or "Passage 2", list numbers, cited years, URLs, code, digits glued to words (Q3, CO2, MX-200,
88
+ 1st), the answer counting its own length ("in 150 words", "a 98-word summary"), a rating scale ("out of
89
+ 5"), "24/7" and plain integers from 0 to 10 are not checked: when unsure, it says nothing. A number cut
90
+ short (1.97 for 1.9799) is reported as minor.
91
+
92
+ A scale in a key name is understood (revenue_musd = 1840.5 supports "$1.84 billion"), and so is a duration
93
+ written in two units ("535 minutes, about 8 hours 55 minutes"). Clock times are compared as times ("2:30
94
+ pm" against "14:30"), and the day of a date is checked even with an ordinal ("April 19th"). A conversion
95
+ in brackets, "400°F (200°C)", is reported as minor when it is right (within 3%, or 6 degrees) and as
96
+ unsupported when it is wrong. A tool result's list counts as giving its length ("based on 3 reviews").
97
+
98
+ A percentage is not supported by a bare fraction below 1: "0.075%" is not 0.075 (that is 7.5%),
99
+ unless the source writes it as a percentage or under a key such as change_pct.
100
+
101
+ A number the answer computes from numbers it states, which the sources support, is accepted when its
102
+ sentence says it is computed: "315 EUR and 234 EUR, a total of 549 EUR" (a sum or difference of two
103
+ of them, with a word such as "total", "together", "extra", "warmer"), or "3 lamps at 39.90 each cost
104
+ 119.70" (a stated number times a count from the question, with "each", "per", "times"). It must round
105
+ to the written number at the precision it is written in: "a total of 550 EUR" is reported. A source number times a count from the question
106
+ ("3 lamps" at 39.90 each is 119.70) is reported as minor: right, but worked out in the answer rather
107
+ than by a tool. Thin and narrow no-break spaces work as thousands separators ("40 983.61").
108
+
109
+ Not yet: signs are not compared, units are converted only in a pair written together ("5 km (3.1
110
+ miles)"), and a value computed from numbers the answer does not state (a sum of two source values that
111
+ the answer leaves out) is reported as unsupported.
112
+
113
+ ## The quote check (implemented)
114
+
115
+ Every quotation of three words or more must occur in the source its sentence cites ([S1], [doc2],
116
+ [report.txt], or [2] for the second source), or in any source if the sentence cites none. Case,
117
+ accents, dashes, quote marks, spacing and punctuation do not matter; the words do. Ellipses and
118
+ editorial insertions in square brackets are allowed. A quote that differs only in small words (a, the,
119
+ of, my ...), punctuation, the spelling of one word, or singular and plural is reported as minor. A quote
120
+ that is not found is reported with the closest wording the sources contain; a quote found in another
121
+ source than the one cited is reported with the source it is really in.
122
+
123
+ ```
124
+ The quote is not in the cited source. The closest wording, in S1: "does not guarantee that the
125
+ reconstructed matrix is positive".
126
+ ```
127
+
128
+ Not checked: quotes of one or two words, titles (every main word capitalized), quotes in code, quotes in
129
+ single quote marks, example text ('a subject line such as "..."') and the question restated.
130
+
131
+ ## Tool traces (implemented)
132
+
133
+ Give the tool calls as a list of `{"name", "args", "result"}`, or as a chat (then the question and the
134
+ final answer are taken from it: `check_messages(messages)`). A chat can be OpenAI-style messages,
135
+ Anthropic Messages API messages (`tool_use` and `tool_result` blocks), or LangChain messages (the
136
+ objects, or the dicts of `messages_to_dict`). Sources can be given at the same time, for an agent that
137
+ reads documents and calls tools. The tool results and arguments become evidence for the number and quote
138
+ checks, and two more checks run.
139
+
140
+ **Argument binding**: the values in the question must be the ones the tools were called with. Every
141
+ number below is in the tool result or the question, so a number check passes this answer; the error is
142
+ that the tool computed 3 years, not 5.
143
+
144
+ ```python
145
+ result = check_answer(
146
+ "Over 5 years at 7% you pay 12,250 in total, 340.29 a month.",
147
+ question="What is the 5-year cost at 7% interest?",
148
+ trace=[{"name": "loan_cost", "args": {"rate": 0.07, "years": 3},
149
+ "result": {"total": 12250.43, "monthly": 340.29}}],
150
+ )
151
+ ```
152
+
153
+ ```
154
+ PROBLEMS: 3 numbers and 2 value-to-field bindings checked against 1 tool call and the question: 1 tool argument different from the question.
155
+ 1. [argument] "5" in: Over 5 years at 7% you pay 12,250 in total, 340.29 a month.
156
+ The question asks for 5-year, but loan_cost was called with years=3.
157
+ ```
158
+
159
+ It also reports a question value whose digits the call shifted by a power of ten (asked p = 0.3,
160
+ called with 0.03) and two values swapped between two calls (asked 120 USD and 80 GBP, called with
161
+ amount=80 for USD and amount=120 for GBP).
162
+
163
+ **Value-to-field binding**: a number the answer attaches to a field name must be the value the tool
164
+ gave for that field. Field names come from the JSON keys (`rain_chance_pct` is "rain chance";
165
+ `field_synonyms` adds names).
166
+
167
+ ```python
168
+ result = check_answer(
169
+ "Jordan Reyes scored 27 points with 7 rebounds and 11 assists.",
170
+ trace=[{"name": "player_game_stats", "args": {"player": "Jordan Reyes"},
171
+ "result": {"points": 27, "rebounds": 11, "assists": 7}}],
172
+ )
173
+ ```
174
+
175
+ ```
176
+ PROBLEMS: 2 numbers and 3 value-to-field bindings checked against 1 tool call: 2 values attached to the wrong field.
177
+ 1. [value-to-field] "7 rebounds" in: Jordan Reyes scored 27 points with 7 rebounds and 11 assists.
178
+ "7 rebounds": the tool results give rebounds = 11 (player_game_stats). 7 is the assists (player_game_stats).
179
+ 2. [value-to-field] "11 assists" in: Jordan Reyes scored 27 points with 7 rebounds and 11 assists.
180
+ "11 assists": the tool results give assists = 7 (player_game_stats). 11 is the rebounds (player_game_stats).
181
+ ```
182
+
183
+ A field can also be named in other ways: by a unit (in "weighs 249 kg" the 249 is compared with the
184
+ results' only kg field, and reported when it is the value of another field of the same call), by a
185
+ word only one multi-word field name has ("€789.52 in interest" for total_interest), by the subject of
186
+ "... is N" ("The total interest you'll pay on the loan is $3,034.15"), and by a range ("between $69.80
187
+ and $72.90" for day_low and day_high). The right value in another currency is reported when no call
188
+ converted it ("€221.4 million" when the tool gives net_income_musd = 221.4).
189
+
190
+ Both checks stay silent when a binding is unsure: no field word right next to the number (a
191
+ preposition such as "for" or "on" starts a new phrase: "the monthly payment for a $30,000 loan"), a number in
192
+ another unit than the field ("91.4 °F" for high_c), a clause that says the number is worked out ("the
193
+ difference in monthly payment is $166") or a comparative follows it ("3.3 hPa higher"), a year ("in
194
+ 2024"), a sentence with "respectively", a question number
195
+ with no argument that names it or resembles it, a number that is part of a name or an address in the
196
+ results ("4 Eggs & Pizza", "223 Anacapa St").
197
+
198
+ From the command line: `traceground --trace chat.json`, or `traceground --trace calls.json --answer
199
+ answer.txt --question question.txt`.
200
+
201
+ ## What the answer says about its tools, and its own wording (implemented)
202
+
203
+ * **Failed calls**: a call failed, was not called again successfully (nor replaced by another tool
204
+ called about the same thing), and the answer reads as if it had worked.
205
+ * **Tools never called**: "I searched the web" with no search tool, "I ran the code", "I queried the
206
+ database", "I used the X tool". Offers ("I can search the web") and advice are not claims.
207
+ * **Thinking aloud** left in the final answer: "Wait, let me check:", "Let me report the tool's
208
+ numbers:", a paragraph that takes back what came before. Ordinary openers ("Let me explain") are fine.
209
+ * **Self-contradiction**: "The passages do not say how long the warranty lasts." followed by a sentence
210
+ that gives it from the sources. A statement about one passage ("Passage 3 does not mention it") is not
211
+ a contradiction when another passage gives it.
212
+
213
+ ```python
214
+ result = check_answer(
215
+ "I searched the web and found that the MX-200 warranty is 2 years. Wait, let me check: yes, 2 years.",
216
+ trace=[{"name": "lookup_product", "args": {"sku": "MX-200"}, "result": {"warranty_years": 2}}],
217
+ )
218
+ ```
219
+
220
+ ```
221
+ PROBLEMS: 2 value-to-field bindings checked against 1 tool call: 1 claim about a tool that was not called; 1 passage of the model thinking aloud.
222
+ 1. [tool-not-called] "I searched the web" in: I searched the web and found that the MX-200 warranty is 2 years.
223
+ The answer says "I searched the web", but no tool for a web search was called (calls: lookup_product).
224
+ 2. [self-talk] "Wait, let me check:" in: Wait, let me check: yes, 2 years.
225
+ This is the model thinking aloud, left in the final answer: "Wait, let me check:".
226
+ ```
227
+
228
+ ## Measured on RAGTruth
229
+
230
+ On the test split of [RAGTruth](https://github.com/ParticleMedia/RAGTruth), 2,675 real answers by six models
231
+ with hallucinations marked by people, measured once with 0.1.0.dev12 (whose checks 0.1.0 keeps unchanged)
232
+ after development on the train split:
233
+ 7 of traceground's 172 high-severity findings (4%) were false alarms, by hand review. It flagged 85 of the
234
+ 306 marked spans that contain a digit, and 8% to 16% of the answers with any marked span: most marked
235
+ hallucinations are claims in words, which it does not check. Details, and how to rerun it, in
236
+ [eval/README.md](https://github.com/animeshQKD/traceground/blob/main/eval/README.md).
237
+
238
+ ## Demo
239
+
240
+ `space/app.py` is a Gradio page for trying traceground in a browser: paste an answer with its sources, its
241
+ tool calls (or a whole chat), or both, and the page marks every finding on the answer, shows the evidence
242
+ for each, and greys out the sentences it has nothing to check in. It runs on your own computer and needs no
243
+ API key. From a copy of this repository:
244
+
245
+ ```
246
+ pip install traceground "gradio[mcp]==6.32.0"
247
+ python space/app.py
248
+ ```
249
+
250
+ then open http://127.0.0.1:7860. The same check is an MCP tool, `traceground_check`, at
251
+ http://127.0.0.1:7860/gradio_api/mcp/ while the page runs ([space/README.md](https://github.com/animeshQKD/traceground/blob/main/space/README.md)).
252
+
253
+ ## Planned next
254
+
255
+ * Citations mode: marker styles, every marker checked, every factual sentence cited (optional).
256
+ * An opt-in automatic fix where a fix is certain (a number cut short, thinking aloud removed).
257
+ * The demo published as a Hugging Face Space, and LangGraph / LangChain helpers.
258
+ * Report-only stays the default; when a check is unsure, it says nothing.
259
+
260
+ ## Install
261
+
262
+ ```
263
+ pip install traceground
264
+ ```
265
+
266
+ Python 3.10 to 3.14. The package depends on the standard library only. To work on it, from a copy of this
267
+ repository: `pip install -e ".[test]"`, then `python -m pytest`.
268
+
269
+ ## License
270
+
271
+ MIT