attestql 0.2.1__tar.gz

attestql-0.2.1/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

attestql-0.2.1/NOTICE

@@ -0,0 +1,44 @@
+AttestQL
+Copyright 2026 Hoang Le
+
+This repository is licensed under the Apache License, Version 2.0 (see LICENSE), with the
+exceptions listed here.
+
+The following files contain material excerpted from BIRD Mini-Dev
+(https://github.com/bird-bench/mini_dev), which is licensed under the Creative Commons
+Attribution-ShareAlike 4.0 International licence (CC BY-SA 4.0,
+https://creativecommons.org/licenses/by-sa/4.0/). That material is not covered by the
+Apache License and keeps its own licence and attribution:
+
+  tools/audit-sandbox/questions.json
+      the question text, evidence text, gold SQL and difficulty of Mini-Dev questions 1029, 879
+      and 207, reproduced because the defect the sandbox replays is in that shipped gold; the rows
+      those statements run against are this repository's own synthetic fixture.
+  tools/audit-sandbox-sqlite/questions.json
+      the question text, evidence text, gold SQL and difficulty of the same three questions as
+      they are written in Mini-Dev's SQLite copy, reproduced for the same reason; the rows those
+      statements run against are this repository's own synthetic fixture.
+  plans/reports/spike-260902-three-gold-defects/
+      evidence records and counterexamples for the same three questions, holding their gold SQL
+      and a few result rows from the Mini-Dev PostgreSQL databases.
+  plans/reports/sweep-260902-gold-only-probes/fired.json
+      the question text, evidence text and gold SQL of the 52 Mini-Dev questions on which a
+      gold-only probe fired, with the rows the probe read.
+  plans/reports/parser-260903-postgast-against-pglast/current-parser.json and previous-parser.json
+      the table names and the ordering expressions of the 498 Mini-Dev gold statements, as two
+      parsers read them.
+  plans/reports/minidev-260903-zip-against-hf/
+      the two gold statements that differ between the GitHub zip and the Hugging Face file, in
+      both forms.
+  plans/reports/prediction-mode-260904-real-predictions/examples/
+      four counterexamples (Mini-Dev questions 565, 1035, 1473 and 1057): the question text,
+      evidence text and gold SQL, a prediction from BIRD's own llm/exp_result/sql_output_kg
+      files, and the rows both returned from the Mini-Dev PostgreSQL databases.
+  plans/reports/minidev-sqlite-260907/examples/
+      three counterexamples (Mini-Dev questions 846, 45 and 1035): the question text, evidence
+      text and gold SQL as Mini-Dev's SQLite copy writes them, a prediction from BIRD's own
+      llm/exp_result/sql_output_kg files, and the rows both returned from the Mini-Dev SQLite
+      databases.
+
+Every other file, including the run outputs that carry only question ids, verdicts, hashes and
+counts, is this repository's own work under the Apache License.

attestql-0.2.1/PKG-INFO

@@ -0,0 +1,187 @@
+Metadata-Version: 2.4
+Name: attestql
+Version: 0.2.1
+Summary: Audit text-to-SQL gold and predictions on PostgreSQL with typed replay evidence.
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/ivermin1123/attestql
+Project-URL: Repository, https://github.com/ivermin1123/attestql
+Project-URL: Issues, https://github.com/ivermin1123/attestql/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: psycopg[binary]<4,>=3.3.4
+Requires-Dist: postgast==0.1.0
+Requires-Dist: sqlglot==30.18.0
+Dynamic: license-file
+
+# AttestQL
+
+You have a text-to-SQL prediction file and a benchmark that scored it. Run one command and get,
+per question, whether the prediction and the gold disagree on the shipped data, the rows that
+differ, and what the benchmark's own scorer would have said: for a question scored 0, the rows
+that decided it, and for one scored 1, whether a comparison that keeps duplicates, order and types
+agrees with the set comparison that credited it.
+
+```text
+attestql audit --engine sqlite --dsn sandbox/fixture.sqlite \
+    --questions tools/audit-sandbox-sqlite/questions.json \
+    --predictions tools/audit-sandbox-sqlite/predictions.json --out audit
+```
+
+That run is this repository's own sandbox: the three BIRD Mini-Dev golds that upstream reports
+found wrong (q1029, q879, q207) on a fixture of a few rows, each paired with a corrected statement
+as its prediction, plus three synthetic questions. Its output, unedited:
+
+```text
+q1029 european_football_2 R-ORD  NOT_EQUAL  smells=none  audit/q1029/
+q879  formula_1   R-ORD  NOT_EQUAL  smells=ordering-over-numeric-text  audit/q879/
+q207  toxicology  R-SET  NOT_EQUAL  smells=none  audit/q207/
+q900001 synthetic   R-ORD  GOLD-ONLY  smells=not-a-function-of-the-data  audit/q900001/
+q900002 synthetic   R-ORD  GOLD-ONLY  smells=arbitrary-cut,not-a-function-of-the-data  audit/q900002/
+q900005 synthetic   R-ORD  EQUAL      smells=ordering-over-numeric-text  audit/q900005/
+6 questions: 3 NOT_EQUAL, 5 smells fired, 0 credited by BIRD but NOT_EQUAL (0 multiplicity, 0 type, 0 order, 0 truncation), 0 timed out (0 gold, 0 prediction)
+```
+
+Every `NOT_EQUAL` line has a directory. `audit/q879/counterexample.json` says why that one
+disagrees, trimmed here to the fields a reader opens first:
+
+```text
+"gold":   "... ORDER BY T2.fastestLapSpeed DESC LIMIT 1"                -> "Norwegian"
+"second": "... ORDER BY CAST(T2.fastestLapSpeed AS REAL) DESC LIMIT 1"  -> "Peruvian"
+"bird_ex": {"value": 0, "method": "set(second_rows) == set(gold_rows), ..."}
+"differing_rows": {"in_gold_not_in_second": [["Norwegian"]], "in_second_not_in_gold": [["Peruvian"]]}
+```
+
+and `audit/q879/smells.json` names the mechanism: `fastestLapSpeed` is a text column holding only
+numbers, so the gold sorts `9.5` above `10` and the fastest lap is not the one it returns. The
+same directory holds the two evidence records, each with the statement, the role, the engine,
+the session settings, the result and its hash, and how to run it again.
+
+**NOT_EQUAL never means the gold is wrong.** It means these two statements disagree on this data
+under this rule; here are the rows; decide. The benchmark's own reading,
+`set(predicted) == set(gold)`, is computed beside every verdict as `bird_ex`, and the typed
+comparison beside it keeps duplicate rows, keeps order where the gold orders, and compares values
+by declared type, so a prediction the benchmark credits and this tool calls `NOT_EQUAL` is listed
+on the summary line by what makes the two readings differ.
+
+## Install
+
+Python 3.11 or later. From PyPI:
+
+```text
+uv tool install attestql        # or: pip install attestql
+```
+
+Or from a checkout, which is also how the sandbox above is built:
+
+```text
+git clone https://github.com/ivermin1123/attestql && cd attestql
+uv run python tools/audit-sandbox-sqlite/build.py sandbox
+uv run attestql audit --engine sqlite --dsn sandbox/fixture.sqlite \
+    --questions tools/audit-sandbox-sqlite/questions.json \
+    --predictions tools/audit-sandbox-sqlite/predictions.json --out audit
+```
+
+On your own files the command is the same. On SQLite, `--dsn` is the path to one database file and
+`--ids` picks the questions that database answers, so a benchmark of eleven databases is eleven
+runs; BIRD Mini-Dev and BIRD dev ship as SQLite files and need nothing else installed. On
+PostgreSQL 16 one server holds every database of the benchmark, so one run covers the whole
+question file:
+
+```text
+attestql audit --dsn "host=localhost dbname=bird" \
+    --questions minidev/MINIDEV/mini_dev_postgresql.json --predictions preds.json --out audit
+```
+
+A predictions file written for this tool is keyed by question id (`{"879": "SELECT ..."}`); BIRD's
+own `predict_dev.json` files are keyed by position and are read with
+`--predictions-keyed-by position`. Without `--predictions` the run audits the golds alone and runs
+the mechanical probes that need no second statement: an ordering key that is text holding numbers,
+an arbitrary or null-first cut that changes the answer, and a result that changes when the
+referenced tables are copied in another row order. Exit status is 0 with no disagreement, 1 with at least one, 2 when
+the tool could not run. Every flag, every line of the output and every key of `summary.json` is
+described in [docs/audit-command.md](docs/audit-command.md).
+
+## What it found
+
+Three measurements, each on BIRD's own published files, each with its report, its artifact and
+BIRD's own evaluator run beside the tool as the check.
+
+**BIRD Mini-Dev on PostgreSQL.** Of the 1,239 predictions in BIRD's nine published Mini-Dev
+prediction files that BIRD's evaluator scores 1, 164 (13.2 %) are `NOT_EQUAL` under the typed
+comparison, and read by hand 69 of those (42.1 %; 5.6 % of everything BIRD credits) are wrong
+answers the benchmark credited, 74 are duplicated rows a reader would forgive, and 21 are the
+typed rule alone. The two readings of EX agree on 4,476 of 4,482 predictions, and every
+disagreement is one float sum whose last digits depend on the order its parts are added in
+([report](plans/reports/measurement-260904-0046-prediction-mode-on-real-predictions.md)).
+
+**BIRD Mini-Dev on SQLite.** The same nine files on the eleven Mini-Dev database files, with no
+server: the tool's reading of BIRD's EX and BIRD's own evaluator agree on 4,481 of 4,482, and of
+the 1,650 predictions BIRD credits, 237 (14.4 %) are `NOT_EQUAL` under the typed comparison. A
+sample of 50 of those 237, read by hand, is 27 wrong answers the benchmark credited, 22 duplicated
+rows a reader would forgive and 1 the typed rule itself
+([report](plans/reports/measurement-260907-1106-minidev-sqlite.md)).
+
+**BIRD dev, both copies of the golds.** BIRD's 2025-11-06 quality pass rewrote 399 of the 1,534
+dev golds; the gold-only probes, run over the older copy, fire on 31 of those 399 (7.8 %) against
+25 of the 963 golds BIRD left alone (2.6 %), and 29 of the 31 go quiet once BIRD's own rewrite
+replaces the old gold. The 25 fires on golds BIRD did not touch were read by hand: 23 are golds
+that do not answer their question on this data, one is harmless, one is the tool's own rule
+([report](plans/reports/measurement-260907-1435-bird-dev-sqlite.md)).
+
+## What was reported upstream
+
+Six reports, filed under the owner's own GitHub and Hugging Face identity, each with its
+counterexample. The replies, as of 2026-09-07:
+
+| # | Where | What | Reply |
+|---|---|---|---|
+| 1 | [mini_dev issue 38](https://github.com/bird-bench/mini_dev/issues/38#issuecomment-5529732756) | q1029 orders `ASC NULLS FIRST` for "highest" | BIRD team, 2026-09-05: "We will review and correct this issue in the next patch." Issue closed |
+| 2 | [mini_dev issue 39](https://github.com/bird-bench/mini_dev/issues/39) | q207 joins `bond` on `molecule_id`, 13 elements instead of 5 | BIRD team, 2026-09-05: "We will review and correct this issue in the next patch." Issue closed |
+| 3 | [mini_dev issue 40](https://github.com/bird-bench/mini_dev/issues/40) | the GitHub zip and the Hugging Face dataset are different question sets | BIRD team, 2026-09-05: README changed to name the Hugging Face dataset as the one to download. Issue closed |
+| 4 | [SpotIt-plus issue 1](https://github.com/ai-ar-research/SpotIt-plus/issues/1) | asked which licence applies | Withdrawn by the owner on 2026-09-05: the LICENSE does grant use after its first sentence, so the report rested on a misreading |
+| 5 | [mini_dev issue 48](https://github.com/bird-bench/mini_dev/issues/48) | the PostgreSQL evaluator scores q1473 differently on two runs of the same database | none yet |
+| 6 | [mini_dev issue 49](https://github.com/bird-bench/mini_dev/issues/49), [dataset discussion 3](https://huggingface.co/datasets/birdsql/bird_sql_dev_20251106/discussions/3) | 23 BIRD dev golds the 2025-11-06 pass left unchanged that do not answer their question on the shipped data | none yet |
+
+## What it does not do
+
+- It does not decide which statement is right. A second statement is supplied by you: a model's
+  prediction, an upstream correction, a human's fix.
+- It does not generate differentiating data. Two statements that agree on the shipped rows but
+  differ semantically are found only by the shuffle probe, not by search.
+- The probes are heuristics and say so in their own evidence; the direction probe alone is 17 %
+  precise on Mini-Dev and is off by default.
+- It proves nothing about correctness, security, or production use. It runs as the role you give
+  it; give it a read-only one.
+
+## How to check rather than believe
+
+Every number above has an owning artifact in [the claims register](docs/claims-register.md),
+which also lists the claims this project deliberately does not make and dates every negative
+claim, because negative claims decay. The three shipped-gold defects reproduce in the merge gate
+on both engines (`tools/audit-sandbox/`, `tools/audit-sandbox-sqlite/`) every time `just check`
+runs. [ADR-0013](docs/adr/0013-audit-text-to-sql-gold-with-typed-replay-evidence.md) records the
+decision this tool implements and the date by which it is reconsidered if nobody uses it.
+
+The history is short and stated: this repository was developed privately from 2026-08-25 under a
+different product direction; it was reoriented on 2026-09-02 by ADR-0013 to the problem above, and
+the public history starts after that. The code was written with AI assistance under the owner's
+review, and the gate, not the author, is what vouches for it.
+
+## Licence
+
+Apache-2.0 for this repository (`LICENSE`). The material excerpted from BIRD Mini-Dev, listed in
+`NOTICE`, keeps its own CC BY-SA 4.0 licence. The PostgreSQL parser, `postgast`, is a BSD binding
+to `libpg_query`, PostgreSQL's own grammar as a library, and the PostgreSQL driver, `psycopg`, is
+LGPL. The SQLite parser, `sqlglot`, is MIT, and the SQLite driver is the standard library's.

attestql-0.2.1/README.md

@@ -0,0 +1,160 @@
+# AttestQL
+
+You have a text-to-SQL prediction file and a benchmark that scored it. Run one command and get,
+per question, whether the prediction and the gold disagree on the shipped data, the rows that
+differ, and what the benchmark's own scorer would have said: for a question scored 0, the rows
+that decided it, and for one scored 1, whether a comparison that keeps duplicates, order and types
+agrees with the set comparison that credited it.
+
+```text
+attestql audit --engine sqlite --dsn sandbox/fixture.sqlite \
+    --questions tools/audit-sandbox-sqlite/questions.json \
+    --predictions tools/audit-sandbox-sqlite/predictions.json --out audit
+```
+
+That run is this repository's own sandbox: the three BIRD Mini-Dev golds that upstream reports
+found wrong (q1029, q879, q207) on a fixture of a few rows, each paired with a corrected statement
+as its prediction, plus three synthetic questions. Its output, unedited:
+
+```text
+q1029 european_football_2 R-ORD  NOT_EQUAL  smells=none  audit/q1029/
+q879  formula_1   R-ORD  NOT_EQUAL  smells=ordering-over-numeric-text  audit/q879/
+q207  toxicology  R-SET  NOT_EQUAL  smells=none  audit/q207/
+q900001 synthetic   R-ORD  GOLD-ONLY  smells=not-a-function-of-the-data  audit/q900001/
+q900002 synthetic   R-ORD  GOLD-ONLY  smells=arbitrary-cut,not-a-function-of-the-data  audit/q900002/
+q900005 synthetic   R-ORD  EQUAL      smells=ordering-over-numeric-text  audit/q900005/
+6 questions: 3 NOT_EQUAL, 5 smells fired, 0 credited by BIRD but NOT_EQUAL (0 multiplicity, 0 type, 0 order, 0 truncation), 0 timed out (0 gold, 0 prediction)
+```
+
+Every `NOT_EQUAL` line has a directory. `audit/q879/counterexample.json` says why that one
+disagrees, trimmed here to the fields a reader opens first:
+
+```text
+"gold":   "... ORDER BY T2.fastestLapSpeed DESC LIMIT 1"                -> "Norwegian"
+"second": "... ORDER BY CAST(T2.fastestLapSpeed AS REAL) DESC LIMIT 1"  -> "Peruvian"
+"bird_ex": {"value": 0, "method": "set(second_rows) == set(gold_rows), ..."}
+"differing_rows": {"in_gold_not_in_second": [["Norwegian"]], "in_second_not_in_gold": [["Peruvian"]]}
+```
+
+and `audit/q879/smells.json` names the mechanism: `fastestLapSpeed` is a text column holding only
+numbers, so the gold sorts `9.5` above `10` and the fastest lap is not the one it returns. The
+same directory holds the two evidence records, each with the statement, the role, the engine,
+the session settings, the result and its hash, and how to run it again.
+
+**NOT_EQUAL never means the gold is wrong.** It means these two statements disagree on this data
+under this rule; here are the rows; decide. The benchmark's own reading,
+`set(predicted) == set(gold)`, is computed beside every verdict as `bird_ex`, and the typed
+comparison beside it keeps duplicate rows, keeps order where the gold orders, and compares values
+by declared type, so a prediction the benchmark credits and this tool calls `NOT_EQUAL` is listed
+on the summary line by what makes the two readings differ.
+
+## Install
+
+Python 3.11 or later. From PyPI:
+
+```text
+uv tool install attestql        # or: pip install attestql
+```
+
+Or from a checkout, which is also how the sandbox above is built:
+
+```text
+git clone https://github.com/ivermin1123/attestql && cd attestql
+uv run python tools/audit-sandbox-sqlite/build.py sandbox
+uv run attestql audit --engine sqlite --dsn sandbox/fixture.sqlite \
+    --questions tools/audit-sandbox-sqlite/questions.json \
+    --predictions tools/audit-sandbox-sqlite/predictions.json --out audit
+```
+
+On your own files the command is the same. On SQLite, `--dsn` is the path to one database file and
+`--ids` picks the questions that database answers, so a benchmark of eleven databases is eleven
+runs; BIRD Mini-Dev and BIRD dev ship as SQLite files and need nothing else installed. On
+PostgreSQL 16 one server holds every database of the benchmark, so one run covers the whole
+question file:
+
+```text
+attestql audit --dsn "host=localhost dbname=bird" \
+    --questions minidev/MINIDEV/mini_dev_postgresql.json --predictions preds.json --out audit
+```
+
+A predictions file written for this tool is keyed by question id (`{"879": "SELECT ..."}`); BIRD's
+own `predict_dev.json` files are keyed by position and are read with
+`--predictions-keyed-by position`. Without `--predictions` the run audits the golds alone and runs
+the mechanical probes that need no second statement: an ordering key that is text holding numbers,
+an arbitrary or null-first cut that changes the answer, and a result that changes when the
+referenced tables are copied in another row order. Exit status is 0 with no disagreement, 1 with at least one, 2 when
+the tool could not run. Every flag, every line of the output and every key of `summary.json` is
+described in [docs/audit-command.md](docs/audit-command.md).
+
+## What it found
+
+Three measurements, each on BIRD's own published files, each with its report, its artifact and
+BIRD's own evaluator run beside the tool as the check.
+
+**BIRD Mini-Dev on PostgreSQL.** Of the 1,239 predictions in BIRD's nine published Mini-Dev
+prediction files that BIRD's evaluator scores 1, 164 (13.2 %) are `NOT_EQUAL` under the typed
+comparison, and read by hand 69 of those (42.1 %; 5.6 % of everything BIRD credits) are wrong
+answers the benchmark credited, 74 are duplicated rows a reader would forgive, and 21 are the
+typed rule alone. The two readings of EX agree on 4,476 of 4,482 predictions, and every
+disagreement is one float sum whose last digits depend on the order its parts are added in
+([report](plans/reports/measurement-260904-0046-prediction-mode-on-real-predictions.md)).
+
+**BIRD Mini-Dev on SQLite.** The same nine files on the eleven Mini-Dev database files, with no
+server: the tool's reading of BIRD's EX and BIRD's own evaluator agree on 4,481 of 4,482, and of
+the 1,650 predictions BIRD credits, 237 (14.4 %) are `NOT_EQUAL` under the typed comparison. A
+sample of 50 of those 237, read by hand, is 27 wrong answers the benchmark credited, 22 duplicated
+rows a reader would forgive and 1 the typed rule itself
+([report](plans/reports/measurement-260907-1106-minidev-sqlite.md)).
+
+**BIRD dev, both copies of the golds.** BIRD's 2025-11-06 quality pass rewrote 399 of the 1,534
+dev golds; the gold-only probes, run over the older copy, fire on 31 of those 399 (7.8 %) against
+25 of the 963 golds BIRD left alone (2.6 %), and 29 of the 31 go quiet once BIRD's own rewrite
+replaces the old gold. The 25 fires on golds BIRD did not touch were read by hand: 23 are golds
+that do not answer their question on this data, one is harmless, one is the tool's own rule
+([report](plans/reports/measurement-260907-1435-bird-dev-sqlite.md)).
+
+## What was reported upstream
+
+Six reports, filed under the owner's own GitHub and Hugging Face identity, each with its
+counterexample. The replies, as of 2026-09-07:
+
+| # | Where | What | Reply |
+|---|---|---|---|
+| 1 | [mini_dev issue 38](https://github.com/bird-bench/mini_dev/issues/38#issuecomment-5529732756) | q1029 orders `ASC NULLS FIRST` for "highest" | BIRD team, 2026-09-05: "We will review and correct this issue in the next patch." Issue closed |
+| 2 | [mini_dev issue 39](https://github.com/bird-bench/mini_dev/issues/39) | q207 joins `bond` on `molecule_id`, 13 elements instead of 5 | BIRD team, 2026-09-05: "We will review and correct this issue in the next patch." Issue closed |
+| 3 | [mini_dev issue 40](https://github.com/bird-bench/mini_dev/issues/40) | the GitHub zip and the Hugging Face dataset are different question sets | BIRD team, 2026-09-05: README changed to name the Hugging Face dataset as the one to download. Issue closed |
+| 4 | [SpotIt-plus issue 1](https://github.com/ai-ar-research/SpotIt-plus/issues/1) | asked which licence applies | Withdrawn by the owner on 2026-09-05: the LICENSE does grant use after its first sentence, so the report rested on a misreading |
+| 5 | [mini_dev issue 48](https://github.com/bird-bench/mini_dev/issues/48) | the PostgreSQL evaluator scores q1473 differently on two runs of the same database | none yet |
+| 6 | [mini_dev issue 49](https://github.com/bird-bench/mini_dev/issues/49), [dataset discussion 3](https://huggingface.co/datasets/birdsql/bird_sql_dev_20251106/discussions/3) | 23 BIRD dev golds the 2025-11-06 pass left unchanged that do not answer their question on the shipped data | none yet |
+
+## What it does not do
+
+- It does not decide which statement is right. A second statement is supplied by you: a model's
+  prediction, an upstream correction, a human's fix.
+- It does not generate differentiating data. Two statements that agree on the shipped rows but
+  differ semantically are found only by the shuffle probe, not by search.
+- The probes are heuristics and say so in their own evidence; the direction probe alone is 17 %
+  precise on Mini-Dev and is off by default.
+- It proves nothing about correctness, security, or production use. It runs as the role you give
+  it; give it a read-only one.
+
+## How to check rather than believe
+
+Every number above has an owning artifact in [the claims register](docs/claims-register.md),
+which also lists the claims this project deliberately does not make and dates every negative
+claim, because negative claims decay. The three shipped-gold defects reproduce in the merge gate
+on both engines (`tools/audit-sandbox/`, `tools/audit-sandbox-sqlite/`) every time `just check`
+runs. [ADR-0013](docs/adr/0013-audit-text-to-sql-gold-with-typed-replay-evidence.md) records the
+decision this tool implements and the date by which it is reconsidered if nobody uses it.
+
+The history is short and stated: this repository was developed privately from 2026-08-25 under a
+different product direction; it was reoriented on 2026-09-02 by ADR-0013 to the problem above, and
+the public history starts after that. The code was written with AI assistance under the owner's
+review, and the gate, not the author, is what vouches for it.
+
+## Licence
+
+Apache-2.0 for this repository (`LICENSE`). The material excerpted from BIRD Mini-Dev, listed in
+`NOTICE`, keeps its own CC BY-SA 4.0 licence. The PostgreSQL parser, `postgast`, is a BSD binding
+to `libpg_query`, PostgreSQL's own grammar as a library, and the PostgreSQL driver, `psycopg`, is
+LGPL. The SQLite parser, `sqlglot`, is MIT, and the SQLite driver is the standard library's.