sqlverdict 0.1.0__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/AGENTS.md +2 -1
- sqlverdict-0.1.2/PKG-INFO +304 -0
- sqlverdict-0.1.0/README.md → sqlverdict-0.1.2/README.ko.md +14 -12
- sqlverdict-0.1.2/README.md +281 -0
- sqlverdict-0.1.2/src/sqlverdict/_version.py +1 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/compare.py +18 -11
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/hints.py +18 -6
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/output/html_out.py +2 -2
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/output/table.py +10 -1
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/score.py +6 -3
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/templates/AGENTS.md +1 -1
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_cli.py +1 -1
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_compare.py +51 -2
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_hints.py +19 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_output.py +9 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_score.py +28 -3
- sqlverdict-0.1.0/PKG-INFO +0 -298
- sqlverdict-0.1.0/src/sqlverdict/_version.py +0 -1
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/.github/workflows/ci.yml +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/.github/workflows/release.yml +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/.gitignore +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/LICENSE +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/SECURITY.md +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/pyproject.toml +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/__init__.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/check.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/cli.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/loader.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/model.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/normalize.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/output/__init__.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/output/json_out.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/templates/golden.yaml +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/src/sqlverdict/templates/predictions.yaml +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/db/docker-compose.yml +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/db/probe.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/db/test_real_drivers.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_check.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_html.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_loader.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/tests/test_normalize.py +0 -0
- {sqlverdict-0.1.0 → sqlverdict-0.1.2}/uv.lock +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# AGENTS.md — sqlverdict 저장소 작업 지침
|
|
2
2
|
|
|
3
|
-
Text-to-SQL 결과를 규칙으로 채점하는 라이브러리. 사용법은 README.md.
|
|
3
|
+
Text-to-SQL 결과를 규칙으로 채점하는 라이브러리. 사용법은 README.md (한국어 README.ko.md).
|
|
4
4
|
(`sqlverdict init`이 사용자 프로젝트에 만드는 `evals/AGENTS.md`는 `src/sqlverdict/templates/AGENTS.md`이며 이 파일과 별개다.)
|
|
5
5
|
|
|
6
6
|
## 명령
|
|
@@ -28,4 +28,5 @@ Text-to-SQL 결과를 규칙으로 채점하는 라이브러리. 사용법은 RE
|
|
|
28
28
|
- normalize 규칙을 바꾸면 `tests/test_normalize.py`와 `tests/db/test_real_drivers.py`의 `SAME`/`PER_DB` 표를 함께 고치고, README "DB별 값 처리 규칙"·"알려진 한계"도 맞춘다.
|
|
29
29
|
- 템플릿을 바꾸면 `tests/test_cli.py`의 데모 판정 개수 테스트를 확인한다.
|
|
30
30
|
- 판정·힌트 문구를 바꾸면 README 예시 출력도 다시 뽑는다.
|
|
31
|
+
- README는 영어(`README.md`)와 한국어(`README.ko.md`) 두 벌이다. 한쪽을 고치면 다른 쪽도 같이 고친다.
|
|
31
32
|
- 커밋 메시지는 `모듈: 무엇을 (왜)` 형식, 한국어.
|
|
@@ -0,0 +1,304 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: sqlverdict
|
|
3
|
+
Version: 0.1.2
|
|
4
|
+
Summary: Rule-based scorer for Text-to-SQL results that catches silent_wrong SQL
|
|
5
|
+
Project-URL: Repository, https://github.com/songwookun/sqlverdict
|
|
6
|
+
Author: songwookun
|
|
7
|
+
License-Expression: MIT
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Keywords: evaluation,execution-accuracy,llm,sql,text-to-sql
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
18
|
+
Classifier: Topic :: Database
|
|
19
|
+
Classifier: Topic :: Software Development :: Testing
|
|
20
|
+
Requires-Python: >=3.10
|
|
21
|
+
Requires-Dist: pyyaml>=6
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
|
|
24
|
+
# sqlverdict
|
|
25
|
+
|
|
26
|
+
**English** · [한국어](https://github.com/songwookun/sqlverdict/blob/main/README.ko.md)
|
|
27
|
+
|
|
28
|
+
[](https://github.com/songwookun/sqlverdict/actions/workflows/ci.yml)
|
|
29
|
+
[](https://pypi.org/project/sqlverdict/)
|
|
30
|
+
[](https://pypi.org/project/sqlverdict/)
|
|
31
|
+
[](https://github.com/songwookun/sqlverdict/blob/main/LICENSE)
|
|
32
|
+
|
|
33
|
+
An open-source library that scores Text-to-SQL results with rules and tells you where to fix them.
|
|
34
|
+
Its goal is to catch SQL that runs without error but returns the wrong result (**silent_wrong**).
|
|
35
|
+
|
|
36
|
+
- **Free**: `pip install sqlverdict` and go. No server, no API costs.
|
|
37
|
+
- **No LLM in the loop**: verdicts are rule-based, so the same results always get the same verdict.
|
|
38
|
+
- **Faster with AI**: an AI can draft the golden set and read the report to find what to fix. A human confirms the answers.
|
|
39
|
+
|
|
40
|
+
> CLI messages, hints and templates are currently in Korean. Verdict names, JSON keys and options are English.
|
|
41
|
+
|
|
42
|
+
## What is silent_wrong?
|
|
43
|
+
|
|
44
|
+
Question: "How many customers placed an order in 2024?"
|
|
45
|
+
|
|
46
|
+
| | SQL | Result |
|
|
47
|
+
|---|---|---|
|
|
48
|
+
| Golden | `SELECT COUNT(DISTINCT customer_id) FROM orders WHERE …` | 312 |
|
|
49
|
+
| Prediction | `SELECT COUNT(o.customer_id) FROM orders o JOIN order_items i ON … WHERE …` | 936 |
|
|
50
|
+
|
|
51
|
+
No error, and the number looks plausible. If you only measure execution success, it counts as a pass.
|
|
52
|
+
sqlverdict marks it `silent_wrong` and adds a hint: the value is exactly 3× — suspect a duplicate join or a missing filter.
|
|
53
|
+
|
|
54
|
+
## Install
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
pip install sqlverdict
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Python 3.10+. The only runtime dependency is PyYAML.
|
|
61
|
+
**sqlverdict never connects to your database.** You run the SQL; sqlverdict only scores the results.
|
|
62
|
+
|
|
63
|
+
## Quick start
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
sqlverdict init # creates example golden.yaml · predictions.yaml · AGENTS.md in evals/
|
|
67
|
+
sqlverdict check evals/golden.yaml evals/predictions.yaml
|
|
68
|
+
sqlverdict score evals/golden.yaml evals/predictions.yaml
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Output (messages are in Korean — 정확도 = accuracy, 판정 = verdicts, 제외 = excluded, 안내 = hint):
|
|
72
|
+
|
|
73
|
+
```
|
|
74
|
+
정확도 40.0% (correct 2 / 채점 5 / 전체 6)
|
|
75
|
+
판정 correct 2 · correct_approx 1 · extra_columns 0 · silent_wrong 1 · error 1
|
|
76
|
+
제외 skipped_unverified 1 · unscorable 0 · missing_prediction 0
|
|
77
|
+
…
|
|
78
|
+
id 판정 level trap 질문 안내
|
|
79
|
+
Q002 silent_wrong L2 dup_join 2024년에 주문한 고객은 몇 명? 값이 정확히 3배: 중복 조인·필터 누락 의심
|
|
80
|
+
Q003 error L3 unique_id 2024년 매출 상위 3명 고객의 i… 실행 실패: no such column: c.nm
|
|
81
|
+
Q004 correct_approx L3 biz_term 2024년 활성 고객의 평균 주문 … 허용 오차 안에서만 같음: 반올림·자료형(FLOAT/DECIMAL) 차이 의심
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
The example numbers are made up. To use it for real:
|
|
85
|
+
|
|
86
|
+
1. Rewrite the questions in `golden.yaml` for your database, run each `gold_sql` to fill in `expected`, and set `snapshot` to that day's date.
|
|
87
|
+
2. The example items are marked `verified: true`. Keep `true` only for items a human has checked; set the rest to `false`.
|
|
88
|
+
3. Run the SQL your Text-to-SQL system produced and write the results to `predictions.yaml` — or compare directly with the [Python API](#python-api).
|
|
89
|
+
4. `check` → `score`
|
|
90
|
+
|
|
91
|
+
## Verdicts
|
|
92
|
+
|
|
93
|
+
| Verdict | Meaning | Example | Counts as correct |
|
|
94
|
+
|---|---|---|---|
|
|
95
|
+
| `correct` | Equal up to floating-point noise (1e-9) | Different WHERE wording, same count | Yes |
|
|
96
|
+
| `correct_approx` | Equal only within tolerance (default 0.1%) | Average 152340.3333 vs 152340.33 | Reported separately |
|
|
97
|
+
| `extra_columns` | Required columns match, but extra columns were returned | 20 columns from `SELECT *` | No |
|
|
98
|
+
| `silent_wrong` | Ran fine, result differs | Count tripled by a duplicate join | No |
|
|
99
|
+
| `error` | Execution failed on your side | Unknown column name | No |
|
|
100
|
+
|
|
101
|
+
Accuracy = `correct` ÷ scored items. `accuracy_with_approx` = (`correct` + `correct_approx`) ÷ scored items — use it to compare with tools that round before comparing (the table shows it when any `correct_approx` exists). In `--baseline` comparisons both `correct` and `correct_approx` count as passing, so moving between them is not a regression.
|
|
102
|
+
|
|
103
|
+
Items that were not scored get a **status** instead of a verdict:
|
|
104
|
+
|
|
105
|
+
| Status | Meaning |
|
|
106
|
+
|---|---|
|
|
107
|
+
| `skipped_unverified` | `verified: false` — not yet checked by a human (include with `--include-unverified`) |
|
|
108
|
+
| `unscorable` | Golden or prediction entry is malformed (the reason is shown) |
|
|
109
|
+
| `missing_prediction` | No prediction for this item |
|
|
110
|
+
|
|
111
|
+
### Comparison rules
|
|
112
|
+
- **Column matching**: same name (case-insensitive) first; golden columns left without a name match are paired, in order, with the prediction columns not used by name. It never hunts for a column with similar values — so a coincidental match can't pass.
|
|
113
|
+
- **Row order**: rows are sorted before comparing. Only `ordered: true` items compare order.
|
|
114
|
+
- **Column count**: more columns than needed → `extra_columns` (not counted as correct, as in Spider/BIRD). A required column missing → `silent_wrong`.
|
|
115
|
+
|
|
116
|
+
## File formats
|
|
117
|
+
|
|
118
|
+
Keep the golden set fixed and store predictions separately per run (prompt version, model, …). They are paired by `id`.
|
|
119
|
+
|
|
120
|
+
### Golden set `golden.yaml`
|
|
121
|
+
|
|
122
|
+
```yaml
|
|
123
|
+
dialect: mysql # sqlite | mysql | postgresql | oracle
|
|
124
|
+
snapshot: 2026-11-13 # date the expected results were pulled (warning after 30 days)
|
|
125
|
+
items:
|
|
126
|
+
- id: q007
|
|
127
|
+
question: Top 5 customers by number of orders in 2018?
|
|
128
|
+
level: L4
|
|
129
|
+
trap: dup_join
|
|
130
|
+
why: joining orders with order_items multiplies each order by its item count
|
|
131
|
+
verified: true
|
|
132
|
+
ordered: true
|
|
133
|
+
gold_sql: SELECT customer_id, COUNT(*) AS order_count FROM orders WHERE …
|
|
134
|
+
expected:
|
|
135
|
+
columns: [customer_id, order_count]
|
|
136
|
+
rows: [[A, 12], [B, 9], [C, 7], [D, 6], [E, 5]]
|
|
137
|
+
- id: q008
|
|
138
|
+
question: Which day had the most orders in 2018?
|
|
139
|
+
level: L2
|
|
140
|
+
trap: none
|
|
141
|
+
verified: true
|
|
142
|
+
expected: {columns: [order_date], rows: [[2018-11-23]]}
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
| Field | Required | Description |
|
|
146
|
+
|---|---|---|
|
|
147
|
+
| `id`, `question`, `level`, `trap`, `expected` | Yes | `level` is L1–L4; `trap` is dup_join / unique_id / biz_term / empty_ok / none |
|
|
148
|
+
| `why` | If trap ≠ none | What the item tests and why the answer is right. Reviewers verify from this alone |
|
|
149
|
+
| `verified` | | Default false (excluded from scoring) |
|
|
150
|
+
| `ordered` | | Compare row order too |
|
|
151
|
+
| `compare_columns` | | Compare only these columns |
|
|
152
|
+
| `allow_extra_columns` | | Allow extra columns in the prediction |
|
|
153
|
+
| `tolerance` | | Per-item tolerance (relative; 0 = strict only) |
|
|
154
|
+
| `empty_ok` | | An empty result is the correct answer. If golden and prediction both have 0 rows, it is correct as long as every golden column can be matched (by name or position); extra columns only produce a hint |
|
|
155
|
+
| `gold_sql` | | The golden SQL. Never executed; used by `check` and for re-pulling answers |
|
|
156
|
+
|
|
157
|
+
Authoring rules (mix in trap items, ask only about closed past periods, …) are in the `evals/AGENTS.md` that `init` creates; `check` enforces what can be checked mechanically.
|
|
158
|
+
|
|
159
|
+
### Predictions `predictions.yaml`
|
|
160
|
+
|
|
161
|
+
```yaml
|
|
162
|
+
run_name: prompt-v2
|
|
163
|
+
dialect: postgresql # optional: defaults to the golden dialect (set it only for DB migration checks)
|
|
164
|
+
items:
|
|
165
|
+
- id: q007
|
|
166
|
+
pred_sql: SELECT … # optional: used for hints such as SELECT *
|
|
167
|
+
result:
|
|
168
|
+
columns: [customer_id, cnt]
|
|
169
|
+
rows: [[A, 24], [B, 18], [C, 14], [D, 12], [E, 10]]
|
|
170
|
+
- id: q008
|
|
171
|
+
error: "Unknown column 'order_dt'" # instead of result when execution failed
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
> **Write dates without quotes.** `2024-01-01` is read as a date and goes through the per-database rules; `"2024-01-01"` is just a string.
|
|
175
|
+
> Quote values that contain `: `, such as error messages.
|
|
176
|
+
> YAML aliases (`*name`, `<<: *name`) are rejected, to stop a tiny file from expanding into huge data.
|
|
177
|
+
|
|
178
|
+
## CLI
|
|
179
|
+
|
|
180
|
+
```bash
|
|
181
|
+
sqlverdict init [dir] # default evals/. Stops if files exist (--force to overwrite)
|
|
182
|
+
sqlverdict check golden.yaml [predictions.yaml] [--format table|json]
|
|
183
|
+
sqlverdict score golden.yaml predictions.yaml [options]
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
| score option | Description |
|
|
187
|
+
|---|---|
|
|
188
|
+
| `--format table\|json\|html` | Default table (terminal) |
|
|
189
|
+
| `-o path` | Write to a file instead of stdout |
|
|
190
|
+
| `--tolerance 0.001` | Global tolerance (relative; 0.001 = 0.1%) |
|
|
191
|
+
| `--baseline prev.json` | Compare with a previous run: newly failing / newly passing items |
|
|
192
|
+
| `--fail-on VERDICT` | Exit code 1 if any item has this verdict or status. Repeatable |
|
|
193
|
+
| `--head N` | Include the first N rows of each prediction (default 0) |
|
|
194
|
+
| `--include-expected` | Include golden result values (for debugging) |
|
|
195
|
+
| `--include-unverified` | Also score `verified: false` items |
|
|
196
|
+
|
|
197
|
+
**Exit codes**: `0` OK · `1` `--fail-on` matched or `check` found errors · `2` input error (missing file, bad YAML, bad option)
|
|
198
|
+
|
|
199
|
+
### Compare with a previous run
|
|
200
|
+
|
|
201
|
+
```bash
|
|
202
|
+
sqlverdict score golden.yaml preds_v1.yaml --format json -o v1.json
|
|
203
|
+
sqlverdict score golden.yaml preds_v2.yaml --baseline v1.json
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
If many previously correct items fail at once, you get a warning that the golden answers may be stale because the data changed — not the SQL.
|
|
207
|
+
|
|
208
|
+
### Block a deploy in CI (GitHub Actions example)
|
|
209
|
+
|
|
210
|
+
```yaml
|
|
211
|
+
- run: pip install sqlverdict
|
|
212
|
+
- run: python scripts/run_text2sql.py --out evals/predictions.yaml # your Text-to-SQL run (your code)
|
|
213
|
+
- run: sqlverdict check evals/golden.yaml evals/predictions.yaml
|
|
214
|
+
- run: sqlverdict score evals/golden.yaml evals/predictions.yaml --fail-on silent_wrong --fail-on error --format html -o report.html
|
|
215
|
+
- uses: actions/upload-artifact@v7
|
|
216
|
+
if: always()
|
|
217
|
+
with: {name: sqlverdict-report, path: report.html}
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
The report is written before the exit code is decided, so `report.html` exists even when the run fails.
|
|
221
|
+
|
|
222
|
+
## Python API
|
|
223
|
+
|
|
224
|
+
Passing the values your DB driver returned (Decimal, date objects, …) gives the most accurate normalization.
|
|
225
|
+
|
|
226
|
+
```python
|
|
227
|
+
from sqlverdict import compare, to_result
|
|
228
|
+
|
|
229
|
+
cur.execute(gold_sql)
|
|
230
|
+
expected = to_result(cur) # DB-API cursor → Result (does not execute; reads the remaining rows)
|
|
231
|
+
cur.execute(pred_sql)
|
|
232
|
+
got = to_result(cur)
|
|
233
|
+
|
|
234
|
+
v = compare(expected, got, dialect="mysql", pred_sql=pred_sql)
|
|
235
|
+
v.label # 'silent_wrong'
|
|
236
|
+
v.detail # {'expected_rows': 2, 'got_rows': 2, 'matched_by': 'position',
|
|
237
|
+
# 'column_map': {'order_count': 'cnt'}, 'differing_columns': ['order_count'], 'ratio': 2.0}
|
|
238
|
+
v.hints # ['값이 정확히 2배: 중복 조인·필터 누락 의심', (value exactly 2×: duplicate join / missing filter?)
|
|
239
|
+
# '이름이 달라 위치로 비교함: 컬럼 이름을 맞추거나 compare_columns 지정 권장'] (names differ, compared by position)
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
- A dict of the form `{"columns": [...], "rows": [[...]]}` works too.
|
|
243
|
+
- When the golden and prediction databases differ (e.g. migration checks): `expected_dialect="mysql", got_dialect="postgresql"`.
|
|
244
|
+
- Other arguments: `ordered`, `compare_columns`, `allow_extra_columns`, `tolerance`, `empty_ok` (same meaning as the golden fields).
|
|
245
|
+
|
|
246
|
+
## Per-database value rules
|
|
247
|
+
|
|
248
|
+
Values from both sides are converted to one canonical form per database before comparing.
|
|
249
|
+
These rules were verified on MySQL 8.4 · PostgreSQL 17 · Oracle 23ai Free with pymysql · psycopg 3 · python-oracledb (thin).
|
|
250
|
+
|
|
251
|
+
| Value | Canonical form |
|
|
252
|
+
|---|---|
|
|
253
|
+
| Numbers (Decimal · float · bool) | int if integral, otherwise float. true/false → 1/0 |
|
|
254
|
+
| Date | `2024-01-01`. A timezone-naive midnight datetime (Oracle DATE) also becomes a date |
|
|
255
|
+
| Date + time | `2024-01-01 12:00:00.5` (fractional seconds kept when non-zero) |
|
|
256
|
+
| Timezone-aware datetime | Converted to UTC: `2024-01-01 03:00:00+00:00`. Never equal to a naive value |
|
|
257
|
+
| Time (PG TIME · MySQL TIME) | `12:30:00` |
|
|
258
|
+
| Interval | Python timedelta notation (`1 day, 0:00:00`) |
|
|
259
|
+
| UUID · JSON · arrays | Strings. JSON keys are sorted |
|
|
260
|
+
| NaN | `'NaN'` (NaN equals NaN, but not NULL) |
|
|
261
|
+
| Oracle empty string | NULL (Oracle stores `''` as NULL) |
|
|
262
|
+
|
|
263
|
+
## Known limitations
|
|
264
|
+
|
|
265
|
+
False passes (wrong answers judged correct) are blocked. What remains mostly goes the other way — **a correct answer reported as wrong** — so it shows up in the verdict and you can fix it in SQL. The last two items are outside the verdict.
|
|
266
|
+
|
|
267
|
+
- **CHAR(n) padding**: PostgreSQL and Oracle return `'A '` → use `RTRIM`
|
|
268
|
+
- **Oracle TIMESTAMP WITH TIME ZONE**: python-oracledb thin mode drops the timezone → select `SYS_EXTRACT_UTC(col)`
|
|
269
|
+
- **Single-precision floats across databases**: MySQL FLOAT or Oracle BINARY_FLOAT vs another database gives `correct_approx`
|
|
270
|
+
- **JSON across databases**: MySQL emits JSON keys in its own order, so the string may differ from PostgreSQL
|
|
271
|
+
- **Values that arrive as strings stay as-is**: uppercase UUID strings, quoted dates in YAML
|
|
272
|
+
- **Time with timezone (timetz)**: the timezone is ignored
|
|
273
|
+
- **Values within tolerance** can swap sort order, so different rows may be compared (unordered items with many approximate values)
|
|
274
|
+
- The `SELECT *` hint is a regex and also matches inside comments (no effect on verdicts)
|
|
275
|
+
- Left to humans: ties at the boundary of top-N items, ambiguous questions, resetting `verified` to false after editing `gold_sql`
|
|
276
|
+
|
|
277
|
+
## Data and security
|
|
278
|
+
|
|
279
|
+
- **Result values are not included in reports by default.** Golden values let an AI memorize answers (test leakage), and prediction values can contain production data such as customer details. Be careful sharing reports made with `--head` or `--include-expected`.
|
|
280
|
+
- **Prediction `error` messages are included (up to 300 characters).** Database error messages can contain values.
|
|
281
|
+
- **Don't let the system that generates predictions read `golden.yaml`.** It contains `expected` and `gold_sql` verbatim.
|
|
282
|
+
- The HTML report is a single file; every value is escaped, and a CSP blocks scripts and external resources.
|
|
283
|
+
- sqlverdict makes no database or network connections.
|
|
284
|
+
|
|
285
|
+
Report vulnerabilities privately — see [SECURITY.md](https://github.com/songwookun/sqlverdict/blob/main/SECURITY.md).
|
|
286
|
+
|
|
287
|
+
## Development
|
|
288
|
+
|
|
289
|
+
```bash
|
|
290
|
+
uv sync
|
|
291
|
+
uv run pytest # default (no database)
|
|
292
|
+
|
|
293
|
+
docker compose -f tests/db/docker-compose.yml up -d --wait # MySQL + PostgreSQL
|
|
294
|
+
uv sync --group db
|
|
295
|
+
uv run pytest -m db # real driver value tests
|
|
296
|
+
uv run python tests/db/probe.py # show the types each driver returns
|
|
297
|
+
docker compose -f tests/db/docker-compose.yml --profile oracle down
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
Oracle runs separately because of memory: `docker compose -f tests/db/docker-compose.yml --profile oracle up -d --wait oracle`. `pytest -m db` skips databases it cannot reach.
|
|
301
|
+
|
|
302
|
+
## License
|
|
303
|
+
|
|
304
|
+
MIT
|
|
@@ -1,5 +1,12 @@
|
|
|
1
1
|
# sqlverdict
|
|
2
2
|
|
|
3
|
+
[English](README.md) · **한국어**
|
|
4
|
+
|
|
5
|
+
[](https://github.com/songwookun/sqlverdict/actions/workflows/ci.yml)
|
|
6
|
+
[](https://pypi.org/project/sqlverdict/)
|
|
7
|
+
[](https://pypi.org/project/sqlverdict/)
|
|
8
|
+
[](https://github.com/songwookun/sqlverdict/blob/main/LICENSE)
|
|
9
|
+
|
|
3
10
|
Text-to-SQL 결과를 규칙으로 채점하고, 어디를 고쳐야 하는지 알려주는 오픈소스 라이브러리.
|
|
4
11
|
에러 없이 실행되지만 결과가 틀린 SQL(silent_wrong)을 잡는 것이 목표다.
|
|
5
12
|
|
|
@@ -7,13 +14,6 @@ Text-to-SQL 결과를 규칙으로 채점하고, 어디를 고쳐야 하는지
|
|
|
7
14
|
- **LLM 없이 채점**: 판정은 규칙이라 같은 결과를 넣으면 항상 같은 판정이 나온다.
|
|
8
15
|
- **AI와 쓰면 더 빠르게**: AI가 골든셋 초안을 만들고 결과를 읽어 고칠 곳을 찾는다. 정답 확정은 사람이 한다.
|
|
9
16
|
|
|
10
|
-
> **English summary** — sqlverdict is a rule-based scorer for Text-to-SQL results.
|
|
11
|
-
> It compares the result set produced by your model's SQL with a verified golden result
|
|
12
|
-
> (SQLite / MySQL / PostgreSQL / Oracle) and flags *silent_wrong* answers:
|
|
13
|
-
> queries that run without error but return wrong numbers, such as a duplicate join that triples a count.
|
|
14
|
-
> No LLM and no server: verdicts are deterministic and come with fix hints.
|
|
15
|
-
> CLI messages and docs are currently in Korean.
|
|
16
|
-
|
|
17
17
|
## silent_wrong이란
|
|
18
18
|
|
|
19
19
|
질문: "2024년에 주문한 고객은 몇 명?"
|
|
@@ -71,7 +71,7 @@ Q004 correct_approx L3 biz_term 2024년 활성 고객의 평균 주문
|
|
|
71
71
|
| `silent_wrong` | 실행됐는데 결과가 다름 | 중복 조인으로 건수가 3배 | X |
|
|
72
72
|
| `error` | 사용자 쪽 실행이 실패함 | 없는 컬럼 이름 | X |
|
|
73
73
|
|
|
74
|
-
정확도 = `correct` ÷ 채점한 문항. 반올림 비교를 쓰는 다른 도구와 견줄
|
|
74
|
+
정확도 = `correct` ÷ 채점한 문항. `accuracy_with_approx` = (`correct` + `correct_approx`) ÷ 채점한 문항 — 반올림 비교를 쓰는 다른 도구와 견줄 때 쓴다(근사 판정이 있으면 표에도 함께 나온다). `--baseline` 비교에서는 `correct`와 `correct_approx`를 모두 맞음으로 보므로, 둘 사이를 오간 것은 "새로 틀림"이 아니다.
|
|
75
75
|
|
|
76
76
|
채점하지 않은 문항은 판정이 아니라 **상태**로 따로 센다.
|
|
77
77
|
|
|
@@ -82,7 +82,7 @@ Q004 correct_approx L3 biz_term 2024년 활성 고객의 평균 주문
|
|
|
82
82
|
| `missing_prediction` | 이 문항의 예측이 없음 |
|
|
83
83
|
|
|
84
84
|
### 비교 규칙
|
|
85
|
-
- **컬럼 매칭**: 이름이 같으면(대소문자 무시) 그 컬럼끼리,
|
|
85
|
+
- **컬럼 매칭**: 이름이 같으면(대소문자 무시) 그 컬럼끼리, 이름을 못 찾은 정답 컬럼은 이름 매칭에 안 쓰인 예측 컬럼과 순서대로. 값이 비슷한 컬럼을 찾아다니지 않는다 — 우연히 같은 값이 맞은 것으로 잡히지 않게.
|
|
86
86
|
- **행 순서**: 기본은 정렬한 뒤 비교. `ordered: true` 문항만 순서까지 본다.
|
|
87
87
|
- **컬럼 수**: 예측이 더 많으면 `extra_columns`(정답으로 치지 않음, Spider·BIRD 관례). 필요한 컬럼이 없으면 `silent_wrong`.
|
|
88
88
|
|
|
@@ -124,7 +124,7 @@ items:
|
|
|
124
124
|
| `compare_columns` | | 이 컬럼만 비교 |
|
|
125
125
|
| `allow_extra_columns` | | 예측의 추가 컬럼 허용 |
|
|
126
126
|
| `tolerance` | | 이 문항의 허용 오차 (상대값, 0이면 엄격 비교만) |
|
|
127
|
-
| `empty_ok` | | 빈 결과가 정답 |
|
|
127
|
+
| `empty_ok` | | 빈 결과가 정답. 정답·예측 모두 0행이면, 정답 컬럼이 (이름 또는 위치로) 모두 대응될 때 correct (추가 컬럼은 힌트로만) |
|
|
128
128
|
| `gold_sql` | | 정답 SQL. 실행하지 않고 `check`의 검사·다시 뽑기용 |
|
|
129
129
|
|
|
130
130
|
작성 규칙(함정 섞기, 닫힌 과거 기간만 묻기 등)은 `init`이 만드는 `evals/AGENTS.md`에 있고, 기계로 확인 가능한 것은 `check`가 검사한다.
|
|
@@ -146,7 +146,7 @@ items:
|
|
|
146
146
|
|
|
147
147
|
> **날짜는 따옴표 없이** 쓴다. `2024-01-01`은 날짜로 읽혀 DB별 규칙을 타지만, `"2024-01-01"`은 그냥 문자열이다.
|
|
148
148
|
> 에러 메시지처럼 `: `가 들어간 값은 따옴표로 감싼다.
|
|
149
|
-
> YAML
|
|
149
|
+
> YAML 별칭(`*이름`, `<<: *이름`)은 쓸 수 없다. 작은 파일이 거대한 데이터로 펼쳐지는 공격을 막기 위해서다.
|
|
150
150
|
|
|
151
151
|
## CLI
|
|
152
152
|
|
|
@@ -214,7 +214,7 @@ v.hints # ['값이 정확히 2배: 중복 조인·필터 누락 의심',
|
|
|
214
214
|
|
|
215
215
|
- `{"columns": [...], "rows": [[...]]}` 형태의 dict도 그대로 넘길 수 있다.
|
|
216
216
|
- DB 마이그레이션 검증처럼 정답과 예측의 DB가 다르면 `expected_dialect="mysql", got_dialect="postgresql"`.
|
|
217
|
-
- 그 밖의 인자: `ordered`, `compare_columns`, `allow_extra_columns`, `tolerance` (골든셋 필드와 같은 뜻).
|
|
217
|
+
- 그 밖의 인자: `ordered`, `compare_columns`, `allow_extra_columns`, `tolerance`, `empty_ok` (골든셋 필드와 같은 뜻).
|
|
218
218
|
|
|
219
219
|
## DB별 값 처리 규칙
|
|
220
220
|
|
|
@@ -255,6 +255,8 @@ v.hints # ['값이 정확히 2배: 중복 조인·필터 누락 의심',
|
|
|
255
255
|
- HTML 보고서는 단일 파일이고, 모든 값을 이스케이프하며 CSP로 스크립트·외부 자원을 막는다.
|
|
256
256
|
- sqlverdict는 DB나 네트워크에 접속하지 않는다.
|
|
257
257
|
|
|
258
|
+
취약점은 비공개로 제보해 주세요 — [SECURITY.md](SECURITY.md).
|
|
259
|
+
|
|
258
260
|
## 개발
|
|
259
261
|
|
|
260
262
|
```bash
|