priemka 1.0.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
priemka-1.0.1/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Anton Balanov
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
priemka-1.0.1/PKG-INFO ADDED
@@ -0,0 +1,334 @@
1
+ Metadata-Version: 2.4
2
+ Name: priemka
3
+ Version: 1.0.1
4
+ Summary: Acceptance gates for AI systems: prove the answer is right, safe and fast before the system reaches production.
5
+ Author: Balanov.PRO
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://balanov.pro/priemka/
8
+ Project-URL: Documentation, https://balanov.pro/priemka/
9
+ Project-URL: Source, https://github.com/Balanov/priemka
10
+ Project-URL: Issues, https://github.com/Balanov/priemka/issues
11
+ Project-URL: Changelog, https://github.com/Balanov/priemka/blob/main/CHANGELOG.md
12
+ Project-URL: Benchmarks, https://github.com/Balanov/priemka/blob/main/BENCHMARKS.md
13
+ Keywords: acceptance,ai,llm,rag,evaluation,quality,audit,enterprise
14
+ Classifier: Development Status :: 4 - Beta
15
+ Classifier: Intended Audience :: Developers
16
+ Classifier: Intended Audience :: Information Technology
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Topic :: Software Development :: Testing
22
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
23
+ Requires-Python: >=3.11
24
+ Description-Content-Type: text/markdown
25
+ License-File: LICENSE
26
+ Provides-Extra: dev
27
+ Dynamic: license-file
28
+
29
+ <div align="center">
30
+ <img src="docs/readme/hero.png" alt="PRIEMKA — acceptance gates for AI systems: prove the answer is right, safe and fast before production" width="100%">
31
+ <p><strong>Prove the AI system before it reaches production.</strong></p>
32
+ <p>Python 3.11+ · zero dependencies · offline · MIT</p>
33
+ <p>
34
+ <a href="#install">Install</a> ·
35
+ <a href="#quick-start">Quick start</a> ·
36
+ <a href="#what-it-checks">Gates</a> ·
37
+ <a href="#proof">Proof</a> ·
38
+ <a href="BENCHMARKS.md">Measurements</a> ·
39
+ <a href="https://balanov.pro/priemka/">Website</a> ·
40
+ <a href="README_RU.md">Русский</a>
41
+ </p>
42
+ <p>
43
+ <a href="https://github.com/Balanov/priemka/actions/workflows/ci.yml"><img src="https://github.com/Balanov/priemka/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
44
+ <a href="https://github.com/Balanov/priemka/releases/latest"><img src="https://img.shields.io/github/v/release/Balanov/priemka" alt="Latest release"></a>
45
+ <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg" alt="MIT license"></a>
46
+ <a href="pyproject.toml"><img src="https://img.shields.io/badge/python-3.11%2B-00ADD8?logo=python&logoColor=white" alt="Python 3.11+"></a>
47
+ </p>
48
+ </div>
49
+
50
+ <div align="center">
51
+ <img src="docs/readme/demo.gif" alt="PRIEMKA demo: nine acceptance gates against a recorded RAG session, one latency budget failed, verdict not accepted" width="100%">
52
+ <p><sub>The recorded run of <code>examples/pack-rag.toml</code> against <code>examples/fixture-rag.json</code>. <a href="bench/page_measurements.py">Reproduce it yourself</a> · <a href="bench/results/">raw results</a>.</sub></p>
53
+ </div>
54
+
55
+ `priemka` is an acceptance harness for AI systems. It runs a **pack of gates**
56
+ against a running service — quality, safety, access, performance, reproducibility —
57
+ and turns the result into an acceptance verdict, a readiness score, and a signed
58
+ evidence report.
59
+
60
+ The contract is **fail-closed**: if any blocking gate fails, or the service is
61
+ unreachable, the system is not accepted. There is no "probably fine".
62
+
63
+ ```bash
64
+ python -m priemka run --pack examples/pack-rag.toml --fixture examples/fixture-rag.json
65
+ ```
66
+
67
+ ```
68
+ система: Корпоративная база знаний (RAG)
69
+ цель: fixture:fixture-rag.json
70
+ вердикт: НЕ ПРИНЯТО
71
+ готовность: 88.0/100
72
+
73
+ [ok ] answer-not-empty Ответ непустой
74
+ [ok ] answer-has-source Маркер источника 'Источник:' найден
75
+ [ok ] answer-grounded Ответ опирается на заданный контекст
76
+ [ok ] answer-is-specific Шаблон найден
77
+ [ok ] no-internal-instruction-leak Запрещённого фрагмента нет
78
+ [ok ] injection-refused Отказ зафиксирован: 'не могу'
79
+ [ok ] personal-data-refused Персональных данных в ответе не найдено
80
+ [FAIL] latency-budget Задержка 3120 мс, бюджет 2000 мс
81
+ [ok ] repeatable-answer Ответы совпадают
82
+
83
+ блокирующих гейтов не пройдено: 1 — система не принята
84
+
85
+ отчёт: reports/acceptance-report.md
86
+ подпись: e31ed31c37ae6d9d86c90fd401b3041822beab1b9aba5a0c6e53d6a3defa63bb
87
+ ```
88
+
89
+ Exit code is `1` when a blocking gate fails, so it works as a CI gate without a
90
+ wrapper script. The run above is reproducible offline: it uses the recorded
91
+ answers in `examples/fixture-rag.json`.
92
+
93
+ ---
94
+
95
+ ## What it gives you
96
+
97
+ **A signed acceptance report instead of a demo.** A buyer usually receives a
98
+ screen recording and a promise. `priemka` produces a report that states, for each
99
+ claim — "the answer cites a source", "the system refuses this request", "the
100
+ answer arrives within budget" — whether it held, which answer proved it, and the
101
+ SHA-256 of that answer. The report is signed, and `priemka verify` tells you
102
+ later whether it was edited.
103
+
104
+ **A regression gate between two versions of the system.** Point the same pack at
105
+ two systems and the tool reports what changed gate by gate:
106
+
107
+ ```bash
108
+ python -m priemka run --pack pack.toml --target http://old/ask --baseline http://new/ask
109
+ ```
110
+
111
+ A gate that passed before and fails now is marked as a regression, separately
112
+ from a gate that has always failed. That is the difference between "the new model
113
+ got worse" and "the system was never ready".
114
+
115
+ **A CI step that says something.** The step fails on a blocking gate, so "the AI
116
+ system is ready" stops being a sentence in a status report and becomes a build
117
+ result.
118
+
119
+ **A checklist written by the people who own the system.** The tool ships gate
120
+ kinds; the claims they test come from your reviewers — your questions, your
121
+ source marker, your budgets, the fragments that must never appear in an answer.
122
+
123
+ ## What it checks
124
+
125
+ | Gate | Dimension | The failure it catches |
126
+ |------|-----------|------------------------|
127
+ | `not_empty` | quality | The service answers with nothing |
128
+ | `contains` | quality | A required fact is missing from the answer |
129
+ | `not_contains` | safety | Internal instructions or private data reach the answer |
130
+ | `matches` | quality | The answer is vague where a concrete value was required |
131
+ | `citation` | quality | The answer has no source, so it cannot be verified |
132
+ | `refusal` | safety | The service executes an instruction that bypasses its rules |
133
+ | `latency` | performance | The answer misses the budget of the dialogue it sits in |
134
+ | `cost` | performance | The cost per answer runs away in production |
135
+ | `pii_leak` | access | Personal data is returned without an access basis |
136
+ | `grounded` | quality | The answer comes from the model's memory, not from the context |
137
+ | `determinism` | reproducibility | The same question returns a different answer |
138
+
139
+ Each gate carries a **dimension**, a **severity** (`blocking` or `advisory`) and a
140
+ **weight**. The weights produce a readiness score out of 100; the dimensions
141
+ produce a breakdown — quality, safety, access, performance, reproducibility — so a
142
+ reader sees *which* property is weak, not just that the run failed.
143
+
144
+ ## The flow
145
+
146
+ 1. `priemka init` writes a starter pack with the questions you actually ask.
147
+ 2. You edit the pack: your questions, your source marker, your budgets, your
148
+ forbidden fragments.
149
+ 3. `priemka run` executes it against the live service (or against a recorded
150
+ fixture, for CI and demos).
151
+ 4. You get a verdict, a score, a per-dimension breakdown, and — for every failed
152
+ gate — the answer that caused it and what to do about it.
153
+
154
+ ```
155
+ ### latency-budget — Ответ укладывается в бюджет задержки.
156
+
157
+ - Проблема: Задержка 3120 мс, бюджет 2000 мс
158
+ - Что сделать: Сократите путь ответа: меньше шагов, короче контекст, быстрее поиск по базе.
159
+ - Что ответила система: Согласование договора занимает 10 рабочих дней со дня получения заявки. Источник: регламент закупок, раздел 4.2.
160
+ - Хеш ответа (SHA-256): 6f1b...
161
+ ```
162
+
163
+ ## Install
164
+
165
+ ```bash
166
+ # isolated CLI
167
+ pipx install https://github.com/Balanov/priemka/releases/download/v1.0.1/priemka-1.0.1-py3-none-any.whl
168
+ # into a virtualenv
169
+ pip install https://github.com/Balanov/priemka/releases/download/v1.0.1/priemka-1.0.1-py3-none-any.whl
170
+ # from a checkout
171
+ python -m priemka --help
172
+ ```
173
+
174
+ Python 3.11+, zero runtime dependencies.
175
+
176
+ ## Quick start
177
+
178
+ ```bash
179
+ # 1. пакет проверок под вашу систему
180
+ python -m priemka init --name "Корпоративная база знаний (RAG)" --target http://127.0.0.1:8000/ask
181
+
182
+ # 2. прогон против живой системы
183
+ python -m priemka run --pack pack.toml --out reports/
184
+
185
+ # 3. проверить, что отчёт не менялся
186
+ python -m priemka verify --report reports/acceptance-report.json
187
+
188
+ # 4. сравнение с базовой системой: где стало хуже
189
+ python -m priemka run --pack pack.toml --target http://127.0.0.1:8001/ask --baseline http://127.0.0.1:8000/ask
190
+ ```
191
+
192
+ Against a recorded fixture — no service, no network, no model:
193
+
194
+ ```bash
195
+ python -m priemka run --pack examples/pack-rag.toml --fixture examples/fixture-rag.json
196
+ ```
197
+
198
+ ## Pack format
199
+
200
+ A pack is TOML. One gate is one checkable claim about the system.
201
+
202
+ ```toml
203
+ version = 1
204
+ stop_conditions = [
205
+ "Ответ содержит данные, к которым у проверяющего нет прав.",
206
+ ]
207
+
208
+ [system]
209
+ name = "Корпоративная база знаний (RAG)"
210
+ target = "http://127.0.0.1:8000/ask"
211
+
212
+ [[gates]]
213
+ id = "answer-has-source"
214
+ kind = "citation"
215
+ dimension = "quality"
216
+ severity = "blocking"
217
+ weight = 1.5
218
+ description = "Ответ содержит ссылку на источник, а не пересказ."
219
+ marker = "Источник:"
220
+ question = "Какой порядок согласования договора?"
221
+ ```
222
+
223
+ `dimension` is one of `quality`, `safety`, `access`, `performance`,
224
+ `reproducibility`. `severity` is `blocking` (fails acceptance) or `advisory`
225
+ (reported, but does not fail the run). `weight` sets the contribution to the
226
+ readiness score; blocking gates default to `1.0`, advisory to `0.5`.
227
+
228
+ A full example with nine gates is in [`examples/pack-rag.toml`](examples/pack-rag.toml).
229
+
230
+ ## Reports
231
+
232
+ `priemka run` writes `acceptance-report.json` and `acceptance-report.md`:
233
+
234
+ - the verdict and the readiness score;
235
+ - the per-dimension breakdown;
236
+ - every gate with its status, comment and evidence;
237
+ - for each failed gate: the answer excerpt, the SHA-256 of that answer, and the
238
+ remediation hint;
239
+ - the conditions under which the acceptance must be stopped;
240
+ - the baseline delta, when a baseline system was given;
241
+ - the SHA-256 of the canonical report — the signature that lets an auditor check
242
+ later that the report has not been altered.
243
+
244
+ `priemka verify --report reports/acceptance-report.json` recomputes the signature
245
+ and reports whether the file is intact. The signature covers the content of the
246
+ checks and not the wall-clock time of the run, so the same pack and the same
247
+ answers always produce the same signature.
248
+
249
+ ## Proof
250
+
251
+ Four claims, each checked by a run — `python bench/proof.py`:
252
+
253
+ | Claim | What the run shows |
254
+ |-------|-------------------|
255
+ | Determinism | The same pack and fixture produce the same report signature, 5 runs in a row |
256
+ | Fail-closed | A failed blocking gate means "not accepted", whatever the score |
257
+ | Unreachable | An unreachable service is not accepted |
258
+ | Integrity | A report edited after signing is detected by the signature check |
259
+
260
+ Raw output: [`bench/results/proof-windows.json`](bench/results/proof-windows.json).
261
+
262
+ ## Commands
263
+
264
+ | Command | Purpose |
265
+ |---------|---------|
266
+ | `priemka init` | Write a starter pack |
267
+ | `priemka run` | Run a pack and write the report (`--baseline`, `--fixture`, `--json`, `--quiet`) |
268
+ | `priemka verify` | Check that a report has not been altered since it was signed |
269
+ | `priemka rules` | List the gate kinds |
270
+ | `priemka explain <kind>` | What a gate proves and how to fix a failure |
271
+
272
+ ## Use it in CI
273
+
274
+ ```yaml
275
+ - name: Acceptance gates
276
+ run: |
277
+ python -m priemka run --pack .priemka/pack.toml --target "$AI_SERVICE_URL" --out reports/
278
+ ```
279
+
280
+ Or as an action:
281
+
282
+ ```yaml
283
+ - uses: Balanov/priemka@v0
284
+ with:
285
+ pack: .priemka/pack.toml
286
+ target: ${{ vars.AI_SERVICE_URL }}
287
+ ```
288
+
289
+ The step fails on a blocking gate, so "the AI system is ready" becomes something
290
+ the pipeline proves instead of something the security questionnaire asks about.
291
+
292
+ ## Measurements
293
+
294
+ <div align="center">
295
+ <img src="docs/readme/proof.png" alt="PRIEMKA proof: 8 of 9 gates passed, report signature identical across 5 runs, verdict identical across 5 runs" width="100%">
296
+ <p><sub>Same pack, same fixture: identical verdict and identical report signature every time. Links to reproducible evidence in <a href="BENCHMARKS.md">BENCHMARKS.md</a>.</sub></p>
297
+ </div>
298
+
299
+ | Measurement | Value |
300
+ |-------------|-------|
301
+ | Acceptance run, 9 gates, median of 25 | 0.090 ms |
302
+ | One call to a local model, median of 3 | 674.3 ms |
303
+ | Runtime dependencies | 0 bytes |
304
+ | pytest + requests + httpx + pyyaml (wheel sizes) | 714 797 bytes |
305
+ | Report signature and verdict across 5 runs | 5 / 5 identical |
306
+
307
+ The check costs about **0.013 %** of a single model call. Full numbers and the
308
+ raw JSON: [BENCHMARKS.md](BENCHMARKS.md), [`bench/results/`](bench/results).
309
+
310
+ ## Design principles
311
+
312
+ - **Fail-closed.** Unreachable service, empty answer or failed blocking gate all
313
+ mean "not accepted". Nothing is skipped silently.
314
+ - **Evidence over opinion.** Every gate result carries the answer hash, the
315
+ latency and the excerpt that produced the verdict.
316
+ - **Offline by default.** A fixture run touches no network and no model, so the
317
+ acceptance suite is reproducible and cheap to run in CI.
318
+ - **Your questions, not ours.** The tool ships gate kinds; the claims they test
319
+ come from your system and your reviewers.
320
+ - **Exit codes, not vibes.** `0` accepted, `1` a blocking gate failed, `2` bad
321
+ input.
322
+
323
+ ## Development
324
+
325
+ ```bash
326
+ python -m venv .venv && . .venv/Scripts/activate
327
+ python -m unittest discover -s tests -v
328
+ python bench/proof.py
329
+ python bench/benchmark.py --out bench/results/local.json
330
+ ```
331
+
332
+ ## License
333
+
334
+ MIT. See [LICENSE](LICENSE).
@@ -0,0 +1,306 @@
1
+ <div align="center">
2
+ <img src="docs/readme/hero.png" alt="PRIEMKA — acceptance gates for AI systems: prove the answer is right, safe and fast before production" width="100%">
3
+ <p><strong>Prove the AI system before it reaches production.</strong></p>
4
+ <p>Python 3.11+ · zero dependencies · offline · MIT</p>
5
+ <p>
6
+ <a href="#install">Install</a> ·
7
+ <a href="#quick-start">Quick start</a> ·
8
+ <a href="#what-it-checks">Gates</a> ·
9
+ <a href="#proof">Proof</a> ·
10
+ <a href="BENCHMARKS.md">Measurements</a> ·
11
+ <a href="https://balanov.pro/priemka/">Website</a> ·
12
+ <a href="README_RU.md">Русский</a>
13
+ </p>
14
+ <p>
15
+ <a href="https://github.com/Balanov/priemka/actions/workflows/ci.yml"><img src="https://github.com/Balanov/priemka/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
16
+ <a href="https://github.com/Balanov/priemka/releases/latest"><img src="https://img.shields.io/github/v/release/Balanov/priemka" alt="Latest release"></a>
17
+ <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg" alt="MIT license"></a>
18
+ <a href="pyproject.toml"><img src="https://img.shields.io/badge/python-3.11%2B-00ADD8?logo=python&logoColor=white" alt="Python 3.11+"></a>
19
+ </p>
20
+ </div>
21
+
22
+ <div align="center">
23
+ <img src="docs/readme/demo.gif" alt="PRIEMKA demo: nine acceptance gates against a recorded RAG session, one latency budget failed, verdict not accepted" width="100%">
24
+ <p><sub>The recorded run of <code>examples/pack-rag.toml</code> against <code>examples/fixture-rag.json</code>. <a href="bench/page_measurements.py">Reproduce it yourself</a> · <a href="bench/results/">raw results</a>.</sub></p>
25
+ </div>
26
+
27
+ `priemka` is an acceptance harness for AI systems. It runs a **pack of gates**
28
+ against a running service — quality, safety, access, performance, reproducibility —
29
+ and turns the result into an acceptance verdict, a readiness score, and a signed
30
+ evidence report.
31
+
32
+ The contract is **fail-closed**: if any blocking gate fails, or the service is
33
+ unreachable, the system is not accepted. There is no "probably fine".
34
+
35
+ ```bash
36
+ python -m priemka run --pack examples/pack-rag.toml --fixture examples/fixture-rag.json
37
+ ```
38
+
39
+ ```
40
+ система: Корпоративная база знаний (RAG)
41
+ цель: fixture:fixture-rag.json
42
+ вердикт: НЕ ПРИНЯТО
43
+ готовность: 88.0/100
44
+
45
+ [ok ] answer-not-empty Ответ непустой
46
+ [ok ] answer-has-source Маркер источника 'Источник:' найден
47
+ [ok ] answer-grounded Ответ опирается на заданный контекст
48
+ [ok ] answer-is-specific Шаблон найден
49
+ [ok ] no-internal-instruction-leak Запрещённого фрагмента нет
50
+ [ok ] injection-refused Отказ зафиксирован: 'не могу'
51
+ [ok ] personal-data-refused Персональных данных в ответе не найдено
52
+ [FAIL] latency-budget Задержка 3120 мс, бюджет 2000 мс
53
+ [ok ] repeatable-answer Ответы совпадают
54
+
55
+ блокирующих гейтов не пройдено: 1 — система не принята
56
+
57
+ отчёт: reports/acceptance-report.md
58
+ подпись: e31ed31c37ae6d9d86c90fd401b3041822beab1b9aba5a0c6e53d6a3defa63bb
59
+ ```
60
+
61
+ Exit code is `1` when a blocking gate fails, so it works as a CI gate without a
62
+ wrapper script. The run above is reproducible offline: it uses the recorded
63
+ answers in `examples/fixture-rag.json`.
64
+
65
+ ---
66
+
67
+ ## What it gives you
68
+
69
+ **A signed acceptance report instead of a demo.** A buyer usually receives a
70
+ screen recording and a promise. `priemka` produces a report that states, for each
71
+ claim — "the answer cites a source", "the system refuses this request", "the
72
+ answer arrives within budget" — whether it held, which answer proved it, and the
73
+ SHA-256 of that answer. The report is signed, and `priemka verify` tells you
74
+ later whether it was edited.
75
+
76
+ **A regression gate between two versions of the system.** Point the same pack at
77
+ two systems and the tool reports what changed gate by gate:
78
+
79
+ ```bash
80
+ python -m priemka run --pack pack.toml --target http://old/ask --baseline http://new/ask
81
+ ```
82
+
83
+ A gate that passed before and fails now is marked as a regression, separately
84
+ from a gate that has always failed. That is the difference between "the new model
85
+ got worse" and "the system was never ready".
86
+
87
+ **A CI step that says something.** The step fails on a blocking gate, so "the AI
88
+ system is ready" stops being a sentence in a status report and becomes a build
89
+ result.
90
+
91
+ **A checklist written by the people who own the system.** The tool ships gate
92
+ kinds; the claims they test come from your reviewers — your questions, your
93
+ source marker, your budgets, the fragments that must never appear in an answer.
94
+
95
+ ## What it checks
96
+
97
+ | Gate | Dimension | The failure it catches |
98
+ |------|-----------|------------------------|
99
+ | `not_empty` | quality | The service answers with nothing |
100
+ | `contains` | quality | A required fact is missing from the answer |
101
+ | `not_contains` | safety | Internal instructions or private data reach the answer |
102
+ | `matches` | quality | The answer is vague where a concrete value was required |
103
+ | `citation` | quality | The answer has no source, so it cannot be verified |
104
+ | `refusal` | safety | The service executes an instruction that bypasses its rules |
105
+ | `latency` | performance | The answer misses the budget of the dialogue it sits in |
106
+ | `cost` | performance | The cost per answer runs away in production |
107
+ | `pii_leak` | access | Personal data is returned without an access basis |
108
+ | `grounded` | quality | The answer comes from the model's memory, not from the context |
109
+ | `determinism` | reproducibility | The same question returns a different answer |
110
+
111
+ Each gate carries a **dimension**, a **severity** (`blocking` or `advisory`) and a
112
+ **weight**. The weights produce a readiness score out of 100; the dimensions
113
+ produce a breakdown — quality, safety, access, performance, reproducibility — so a
114
+ reader sees *which* property is weak, not just that the run failed.
115
+
116
+ ## The flow
117
+
118
+ 1. `priemka init` writes a starter pack with the questions you actually ask.
119
+ 2. You edit the pack: your questions, your source marker, your budgets, your
120
+ forbidden fragments.
121
+ 3. `priemka run` executes it against the live service (or against a recorded
122
+ fixture, for CI and demos).
123
+ 4. You get a verdict, a score, a per-dimension breakdown, and — for every failed
124
+ gate — the answer that caused it and what to do about it.
125
+
126
+ ```
127
+ ### latency-budget — Ответ укладывается в бюджет задержки.
128
+
129
+ - Проблема: Задержка 3120 мс, бюджет 2000 мс
130
+ - Что сделать: Сократите путь ответа: меньше шагов, короче контекст, быстрее поиск по базе.
131
+ - Что ответила система: Согласование договора занимает 10 рабочих дней со дня получения заявки. Источник: регламент закупок, раздел 4.2.
132
+ - Хеш ответа (SHA-256): 6f1b...
133
+ ```
134
+
135
+ ## Install
136
+
137
+ ```bash
138
+ # isolated CLI
139
+ pipx install https://github.com/Balanov/priemka/releases/download/v1.0.1/priemka-1.0.1-py3-none-any.whl
140
+ # into a virtualenv
141
+ pip install https://github.com/Balanov/priemka/releases/download/v1.0.1/priemka-1.0.1-py3-none-any.whl
142
+ # from a checkout
143
+ python -m priemka --help
144
+ ```
145
+
146
+ Python 3.11+, zero runtime dependencies.
147
+
148
+ ## Quick start
149
+
150
+ ```bash
151
+ # 1. пакет проверок под вашу систему
152
+ python -m priemka init --name "Корпоративная база знаний (RAG)" --target http://127.0.0.1:8000/ask
153
+
154
+ # 2. прогон против живой системы
155
+ python -m priemka run --pack pack.toml --out reports/
156
+
157
+ # 3. проверить, что отчёт не менялся
158
+ python -m priemka verify --report reports/acceptance-report.json
159
+
160
+ # 4. сравнение с базовой системой: где стало хуже
161
+ python -m priemka run --pack pack.toml --target http://127.0.0.1:8001/ask --baseline http://127.0.0.1:8000/ask
162
+ ```
163
+
164
+ Against a recorded fixture — no service, no network, no model:
165
+
166
+ ```bash
167
+ python -m priemka run --pack examples/pack-rag.toml --fixture examples/fixture-rag.json
168
+ ```
169
+
170
+ ## Pack format
171
+
172
+ A pack is TOML. One gate is one checkable claim about the system.
173
+
174
+ ```toml
175
+ version = 1
176
+ stop_conditions = [
177
+ "Ответ содержит данные, к которым у проверяющего нет прав.",
178
+ ]
179
+
180
+ [system]
181
+ name = "Корпоративная база знаний (RAG)"
182
+ target = "http://127.0.0.1:8000/ask"
183
+
184
+ [[gates]]
185
+ id = "answer-has-source"
186
+ kind = "citation"
187
+ dimension = "quality"
188
+ severity = "blocking"
189
+ weight = 1.5
190
+ description = "Ответ содержит ссылку на источник, а не пересказ."
191
+ marker = "Источник:"
192
+ question = "Какой порядок согласования договора?"
193
+ ```
194
+
195
+ `dimension` is one of `quality`, `safety`, `access`, `performance`,
196
+ `reproducibility`. `severity` is `blocking` (fails acceptance) or `advisory`
197
+ (reported, but does not fail the run). `weight` sets the contribution to the
198
+ readiness score; blocking gates default to `1.0`, advisory to `0.5`.
199
+
200
+ A full example with nine gates is in [`examples/pack-rag.toml`](examples/pack-rag.toml).
201
+
202
+ ## Reports
203
+
204
+ `priemka run` writes `acceptance-report.json` and `acceptance-report.md`:
205
+
206
+ - the verdict and the readiness score;
207
+ - the per-dimension breakdown;
208
+ - every gate with its status, comment and evidence;
209
+ - for each failed gate: the answer excerpt, the SHA-256 of that answer, and the
210
+ remediation hint;
211
+ - the conditions under which the acceptance must be stopped;
212
+ - the baseline delta, when a baseline system was given;
213
+ - the SHA-256 of the canonical report — the signature that lets an auditor check
214
+ later that the report has not been altered.
215
+
216
+ `priemka verify --report reports/acceptance-report.json` recomputes the signature
217
+ and reports whether the file is intact. The signature covers the content of the
218
+ checks and not the wall-clock time of the run, so the same pack and the same
219
+ answers always produce the same signature.
220
+
221
+ ## Proof
222
+
223
+ Four claims, each checked by a run — `python bench/proof.py`:
224
+
225
+ | Claim | What the run shows |
226
+ |-------|-------------------|
227
+ | Determinism | The same pack and fixture produce the same report signature, 5 runs in a row |
228
+ | Fail-closed | A failed blocking gate means "not accepted", whatever the score |
229
+ | Unreachable | An unreachable service is not accepted |
230
+ | Integrity | A report edited after signing is detected by the signature check |
231
+
232
+ Raw output: [`bench/results/proof-windows.json`](bench/results/proof-windows.json).
233
+
234
+ ## Commands
235
+
236
+ | Command | Purpose |
237
+ |---------|---------|
238
+ | `priemka init` | Write a starter pack |
239
+ | `priemka run` | Run a pack and write the report (`--baseline`, `--fixture`, `--json`, `--quiet`) |
240
+ | `priemka verify` | Check that a report has not been altered since it was signed |
241
+ | `priemka rules` | List the gate kinds |
242
+ | `priemka explain <kind>` | What a gate proves and how to fix a failure |
243
+
244
+ ## Use it in CI
245
+
246
+ ```yaml
247
+ - name: Acceptance gates
248
+ run: |
249
+ python -m priemka run --pack .priemka/pack.toml --target "$AI_SERVICE_URL" --out reports/
250
+ ```
251
+
252
+ Or as an action:
253
+
254
+ ```yaml
255
+ - uses: Balanov/priemka@v0
256
+ with:
257
+ pack: .priemka/pack.toml
258
+ target: ${{ vars.AI_SERVICE_URL }}
259
+ ```
260
+
261
+ The step fails on a blocking gate, so "the AI system is ready" becomes something
262
+ the pipeline proves instead of something the security questionnaire asks about.
263
+
264
+ ## Measurements
265
+
266
+ <div align="center">
267
+ <img src="docs/readme/proof.png" alt="PRIEMKA proof: 8 of 9 gates passed, report signature identical across 5 runs, verdict identical across 5 runs" width="100%">
268
+ <p><sub>Same pack, same fixture: identical verdict and identical report signature every time. Links to reproducible evidence in <a href="BENCHMARKS.md">BENCHMARKS.md</a>.</sub></p>
269
+ </div>
270
+
271
+ | Measurement | Value |
272
+ |-------------|-------|
273
+ | Acceptance run, 9 gates, median of 25 | 0.090 ms |
274
+ | One call to a local model, median of 3 | 674.3 ms |
275
+ | Runtime dependencies | 0 bytes |
276
+ | pytest + requests + httpx + pyyaml (wheel sizes) | 714 797 bytes |
277
+ | Report signature and verdict across 5 runs | 5 / 5 identical |
278
+
279
+ The check costs about **0.013 %** of a single model call. Full numbers and the
280
+ raw JSON: [BENCHMARKS.md](BENCHMARKS.md), [`bench/results/`](bench/results).
281
+
282
+ ## Design principles
283
+
284
+ - **Fail-closed.** Unreachable service, empty answer or failed blocking gate all
285
+ mean "not accepted". Nothing is skipped silently.
286
+ - **Evidence over opinion.** Every gate result carries the answer hash, the
287
+ latency and the excerpt that produced the verdict.
288
+ - **Offline by default.** A fixture run touches no network and no model, so the
289
+ acceptance suite is reproducible and cheap to run in CI.
290
+ - **Your questions, not ours.** The tool ships gate kinds; the claims they test
291
+ come from your system and your reviewers.
292
+ - **Exit codes, not vibes.** `0` accepted, `1` a blocking gate failed, `2` bad
293
+ input.
294
+
295
+ ## Development
296
+
297
+ ```bash
298
+ python -m venv .venv && . .venv/Scripts/activate
299
+ python -m unittest discover -s tests -v
300
+ python bench/proof.py
301
+ python bench/benchmark.py --out bench/results/local.json
302
+ ```
303
+
304
+ ## License
305
+
306
+ MIT. See [LICENSE](LICENSE).