ci-guard-check 0.5.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ci_guard_check-0.5.0/.gitignore +8 -0
- ci_guard_check-0.5.0/LICENSE +21 -0
- ci_guard_check-0.5.0/PKG-INFO +644 -0
- ci_guard_check-0.5.0/README.md +619 -0
- ci_guard_check-0.5.0/SKILL.md +269 -0
- ci_guard_check-0.5.0/pyproject.toml +43 -0
- ci_guard_check-0.5.0/src/ci_guard/__init__.py +3 -0
- ci_guard_check-0.5.0/src/ci_guard/__main__.py +6 -0
- ci_guard_check-0.5.0/src/ci_guard/_providers/__init__.py +0 -0
- ci_guard_check-0.5.0/src/ci_guard/_providers/github.py +105 -0
- ci_guard_check-0.5.0/src/ci_guard/actions.py +219 -0
- ci_guard_check-0.5.0/src/ci_guard/classify.py +156 -0
- ci_guard_check-0.5.0/src/ci_guard/cli.py +200 -0
- ci_guard_check-0.5.0/src/ci_guard/config.py +91 -0
- ci_guard_check-0.5.0/src/ci_guard/gate.py +128 -0
- ci_guard_check-0.5.0/src/ci_guard/init.py +145 -0
- ci_guard_check-0.5.0/src/ci_guard/ledger.py +320 -0
- ci_guard_check-0.5.0/src/ci_guard/watch.py +551 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Vinicius Carvalho
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,644 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: ci-guard-check
|
|
3
|
+
Version: 0.5.0
|
|
4
|
+
Summary: Guard CI from wasted minutes by classifying failures and maintaining a per-repo flaky-test ledger.
|
|
5
|
+
Project-URL: Homepage, https://github.com/Viniciuscarvalho/ci-guard
|
|
6
|
+
Project-URL: Issues, https://github.com/Viniciuscarvalho/ci-guard/issues
|
|
7
|
+
License: MIT
|
|
8
|
+
License-File: LICENSE
|
|
9
|
+
Keywords: ci,devops,flaky-tests,github-actions
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Environment :: Console
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Software Development :: Build Tools
|
|
22
|
+
Classifier: Topic :: Software Development :: Testing
|
|
23
|
+
Requires-Python: >=3.9
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
|
|
26
|
+
<p align="center">
|
|
27
|
+
<img src="assets/banner.svg" alt="ci-guard — classify before you retry" width="900">
|
|
28
|
+
</p>
|
|
29
|
+
|
|
30
|
+
<p align="center">
|
|
31
|
+
<strong>Stop burning CI minutes on blind retries. Classify every failure before touching rerun.</strong>
|
|
32
|
+
</p>
|
|
33
|
+
|
|
34
|
+
<p align="center">
|
|
35
|
+
<a href="https://pypi.org/project/ci-guard-check/">
|
|
36
|
+
<img src="https://img.shields.io/pypi/v/ci-guard-check.svg" alt="PyPI version">
|
|
37
|
+
</a>
|
|
38
|
+
<a href="https://github.com/marketplace/actions/ci-guard">
|
|
39
|
+
<img src="https://img.shields.io/badge/GitHub%20Action-1--Click-blue.svg?logo=githubactions" alt="GitHub Action: 1-Click">
|
|
40
|
+
</a>
|
|
41
|
+
<a href="https://github.com/Viniciuscarvalho/ci-guard/actions/workflows/ci.yml">
|
|
42
|
+
<img src="https://github.com/Viniciuscarvalho/ci-guard/actions/workflows/ci.yml/badge.svg" alt="CI">
|
|
43
|
+
</a>
|
|
44
|
+
<a href="https://github.com/Viniciuscarvalho/ci-guard/releases">
|
|
45
|
+
<img src="https://img.shields.io/github/v/release/Viniciuscarvalho/ci-guard?include_prereleases" alt="Release">
|
|
46
|
+
</a>
|
|
47
|
+
<a href="https://github.com/Viniciuscarvalho/ci-guard/blob/main/LICENSE">
|
|
48
|
+
<img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License: MIT">
|
|
49
|
+
</a>
|
|
50
|
+
<a href="https://github.com/Viniciuscarvalho/ci-guard">
|
|
51
|
+
<img src="https://img.shields.io/badge/python-3.9%2B-green.svg" alt="Python 3.9+">
|
|
52
|
+
</a>
|
|
53
|
+
<a href="https://github.com/Viniciuscarvalho/ci-guard">
|
|
54
|
+
<img src="https://img.shields.io/badge/works%20with-Claude%20Code-8b5cf6.svg" alt="Works with Claude Code">
|
|
55
|
+
</a>
|
|
56
|
+
<a href="https://github.com/Viniciuscarvalho/ci-guard">
|
|
57
|
+
<img src="https://img.shields.io/badge/CI-GitHub%20Actions-black.svg" alt="GitHub Actions">
|
|
58
|
+
</a>
|
|
59
|
+
<a href="https://github.com/sponsors/Viniciuscarvalho">
|
|
60
|
+
<img src="https://img.shields.io/badge/sponsor-♥-ea4aaa.svg" alt="Sponsor">
|
|
61
|
+
</a>
|
|
62
|
+
<a href="https://skills.sh/openai/codex/ci-guard">
|
|
63
|
+
<img src="https://img.shields.io/badge/Try%20on-skills.sh-orange.svg" alt="Try on skills.sh">
|
|
64
|
+
</a>
|
|
65
|
+
</p>
|
|
66
|
+
|
|
67
|
+
<p align="center">
|
|
68
|
+
<code>failure classification</code> · <code>flaky ledger</code> · <code>cost guard</code> · <code>verify greens</code> · <code>quarantine</code> · <code>GitHub Actions</code>
|
|
69
|
+
</p>
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
Protects your CI from two failure modes that silently drain money and hide bugs:
|
|
74
|
+
|
|
75
|
+
1. **Blind retries** — a check fails, someone hits "rerun failed jobs", it passes, the PR merges. The flaky test never gets fixed and burns minutes on every future PR.
|
|
76
|
+
2. **Trusting a single green** — a known-flaky test passes once after several failures. The PR merges. Production breaks.
|
|
77
|
+
|
|
78
|
+
ci-guard makes your coding agent refuse to retry until a failure is _classified_, and refuse to trust a green on a known-flaky test until it is _verified_.
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
> **Compatibility** — ci-guard's Python scripts call no LLM and require no API key.
|
|
83
|
+
> Your coding agent reads `SKILL.md` and drives the scripts via your shell. It works
|
|
84
|
+
> with Claude Code, OpenAI Codex, opencode, skills.sh runtimes, and any agent that
|
|
85
|
+
> can paste `SKILL.md` into its context (Gemini CLI, Cursor, Copilot, etc.).
|
|
86
|
+
>
|
|
87
|
+
> → See [`docs/agents.md`](docs/agents.md) for the full agent matrix and install paths.
|
|
88
|
+
|
|
89
|
+
## Quick start
|
|
90
|
+
|
|
91
|
+
### ⚡ 1-Click GitHub Action (Zero setup)
|
|
92
|
+
|
|
93
|
+
Add `ci-guard` directly into any GitHub Actions PR workflow:
|
|
94
|
+
|
|
95
|
+
```yaml
|
|
96
|
+
- name: CI Guard Gate
|
|
97
|
+
uses: Viniciuscarvalho/ci-guard@v0.5.0
|
|
98
|
+
with:
|
|
99
|
+
mode: gate
|
|
100
|
+
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Or run as an automated delivery watcher that retries flakes, halts on branch failures, and comments the savings report:
|
|
104
|
+
|
|
105
|
+
```yaml
|
|
106
|
+
- name: CI Guard Delivery Loop
|
|
107
|
+
uses: Viniciuscarvalho/ci-guard@v0.5.0
|
|
108
|
+
with:
|
|
109
|
+
mode: guard
|
|
110
|
+
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
### 💻 Local / Agent Quick Start
|
|
114
|
+
|
|
115
|
+
**Step 1 — install (once per machine)**
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
pip install ci-guard-check
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
**Step 2 — register the skill for your agent (once per machine)**
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
# Claude Code
|
|
125
|
+
ln -s /path/to/ci-guard ~/.claude/skills/ci-guard
|
|
126
|
+
|
|
127
|
+
# Codex
|
|
128
|
+
ln -s /path/to/ci-guard ~/.codex/skills/ci-guard
|
|
129
|
+
|
|
130
|
+
# opencode
|
|
131
|
+
ln -s /path/to/ci-guard ~/.opencode/skills/ci-guard
|
|
132
|
+
|
|
133
|
+
# Any skills.sh-compatible runtime (auto-detects the right path)
|
|
134
|
+
npx skills add https://github.com/Viniciuscarvalho/ci-guard --skill ci-guard
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
> In v0.7+: `ci-guard install-skill --agent claude` (or `codex`, `opencode`, `all`).
|
|
138
|
+
|
|
139
|
+
**Step 3 — bootstrap a repo (once per project, from repo root)**
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
ci-guard init
|
|
143
|
+
git add .ci-guard .gitignore && git commit -m "ci: bootstrap ci-guard"
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
**Done. Core commands:**
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
# Snapshot a failing PR — always start here
|
|
150
|
+
ci-guard watch --pr auto --once
|
|
151
|
+
|
|
152
|
+
# Generate / post PR savings report
|
|
153
|
+
ci-guard report --pr auto --comment
|
|
154
|
+
|
|
155
|
+
# Quality-gate check
|
|
156
|
+
ci-guard gate --pr auto
|
|
157
|
+
|
|
158
|
+
# Budget-safe retry (refuses on branch_failure or exhausted budget)
|
|
159
|
+
ci-guard watch --pr auto --retry-failed-now
|
|
160
|
+
|
|
161
|
+
# Verify a flaky green before merging
|
|
162
|
+
ci-guard watch --pr auto --verify-flaky-green
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
→ Full install details and agent matrix: [Installation](#installation)
|
|
166
|
+
→ All commands with output examples: [Usage](#usage)
|
|
167
|
+
|
|
168
|
+
---
|
|
169
|
+
|
|
170
|
+
## 💰 PR Savings & Efficiency Report
|
|
171
|
+
|
|
172
|
+
Every time `ci-guard` analyzes or guards a PR, it surfaces real-time metrics showing how many blind retries were prevented and how many compute minutes were saved:
|
|
173
|
+
|
|
174
|
+
```markdown
|
|
175
|
+
<!-- ci-guard-report -->
|
|
176
|
+
## 🛡️ **ci-guard** — PR #42 | `abc1234`
|
|
177
|
+
|
|
178
|
+
### 💰 Savings & Efficiency Summary
|
|
179
|
+
| Metric | Result | Context |
|
|
180
|
+
|:---|:---:|:---|
|
|
181
|
+
| ⏱️ **CI Minutes Saved** | **~24 min** | Avoided running doomed reruns |
|
|
182
|
+
| 🚫 **Blind Retries Avoided** | **2** | Blocked retry on code/branch errors |
|
|
183
|
+
| 🔄 **Flaky Retries Executed** | **1/5** | Safe, isolated reruns |
|
|
184
|
+
| ⏳ **CI Minutes Consumed** | **15/90 min** | Run compute budget |
|
|
185
|
+
| 🎯 **PR Terminal Status** | ✅ `pr_merged` | Gate conclusion |
|
|
186
|
+
|
|
187
|
+
### 🔍 Failure Classifications
|
|
188
|
+
| Check | Classification | Confidence |
|
|
189
|
+
|---|---|---|
|
|
190
|
+
| `test/unit` | `branch_failure` | high |
|
|
191
|
+
|
|
192
|
+
Budget: 1/5 retries · 15/90 min
|
|
193
|
+
|
|
194
|
+
Terminal: ✅ `pr_merged`
|
|
195
|
+
---
|
|
196
|
+
<sub>🛡️ Guarded by ci-guard — stop burning CI minutes on blind retries.</sub>
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
## How it works
|
|
202
|
+
|
|
203
|
+
Every retry decision goes through a strict gate sequence:
|
|
204
|
+
|
|
205
|
+
```
|
|
206
|
+
Failure detected
|
|
207
|
+
│
|
|
208
|
+
▼
|
|
209
|
+
Gate 1 — Snapshot classify all failing checks
|
|
210
|
+
│
|
|
211
|
+
▼
|
|
212
|
+
Gate 2 — Classify branch_failure? → never retry, patch code
|
|
213
|
+
infra_flake? → retry once if budget allows
|
|
214
|
+
test_flake? → retry once + verify the green
|
|
215
|
+
dependency? → retry once; second time = branch_failure
|
|
216
|
+
unknown? → read logs manually, never auto-retry
|
|
217
|
+
│
|
|
218
|
+
▼
|
|
219
|
+
Gate 3 — Cost guard retries_per_job / retries_per_pr / minutes_per_pr
|
|
220
|
+
budget exceeded? → stop, surface to user
|
|
221
|
+
│
|
|
222
|
+
▼
|
|
223
|
+
Gate 4 — Verify green test_flake turned green? → require one verification rerun
|
|
224
|
+
two consecutive greens on same SHA → trust it
|
|
225
|
+
│
|
|
226
|
+
▼
|
|
227
|
+
Gate 5 — Quarantine failure_count_30d ≥ 3 AND flake_rate ≥ 5%?
|
|
228
|
+
→ recommend quarantine (human decides)
|
|
229
|
+
│
|
|
230
|
+
▼
|
|
231
|
+
Gate 6 — Loop --watch streams JSONL until terminal state
|
|
232
|
+
pr_merged / pr_closed → exit 0
|
|
233
|
+
needs_help → exit 2 (intervene)
|
|
234
|
+
budget_exhausted → exit 3 (investigate)
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
### Failure classifications
|
|
238
|
+
|
|
239
|
+
| Class | What it means | Action |
|
|
240
|
+
| -------------------- | -------------------------------------------------------------------- | -------------------------------------------------- |
|
|
241
|
+
| `branch_failure` | Compile error, type error, lint fail, test in a file this PR touched | **Never retry.** Fix the code. |
|
|
242
|
+
| `infra_flake` | Runner shutdown, DNS timeout, registry 502, disk pressure | Retry once within budget. |
|
|
243
|
+
| `test_flake` | Test appears in the flaky ledger, or log shows race/timeout language | Retry once + verify the resulting green. |
|
|
244
|
+
| `dependency_failure` | npm/pip/cargo registry 5xx, lockfile drift | Retry once. Recurs on same SHA → `branch_failure`. |
|
|
245
|
+
| `unknown` | Heuristics couldn't classify | Read the log manually. Never default to retry. |
|
|
246
|
+
|
|
247
|
+
### Retry budgets (default, configurable)
|
|
248
|
+
|
|
249
|
+
| Budget | Default | Meaning |
|
|
250
|
+
| ----------------- | ------- | ---------------------------------------- |
|
|
251
|
+
| `retries_per_job` | 2 | Max reruns of one job per PR |
|
|
252
|
+
| `retries_per_pr` | 5 | Max reruns across all jobs per PR |
|
|
253
|
+
| `minutes_per_pr` | 90 | Cumulative CI minutes consumed by reruns |
|
|
254
|
+
|
|
255
|
+
Exceeding any budget stops all retries and surfaces the situation. There are no warnings or grace periods — friction is the point.
|
|
256
|
+
|
|
257
|
+
### Flaky ledger
|
|
258
|
+
|
|
259
|
+
`.ci-guard/flaky-ledger.json` is committed to the repo. It records every test failure and pass event with a rolling 30-day window. Each entry tracks:
|
|
260
|
+
|
|
261
|
+
- `failure_count_30d` / `pass_count_30d` — rolling counts
|
|
262
|
+
- `flake_rate` — `failures / (failures + passes)` over 30 days
|
|
263
|
+
- `status` — `watched`, `quarantined`, or `fixed`
|
|
264
|
+
- `events` — append-only log (source of truth for all derived fields)
|
|
265
|
+
|
|
266
|
+
A test crosses the quarantine threshold at `failure_count_30d ≥ 3` AND `flake_rate ≥ 5%`. ci-guard recommends quarantine; a human confirms it.
|
|
267
|
+
|
|
268
|
+
### Using ci-guard as an engine
|
|
269
|
+
|
|
270
|
+
`--stream` emits JSONL until a `terminal` value is set. Each snapshot's `actions` list tells the caller what to do next; ci-guard never executes mutations.
|
|
271
|
+
|
|
272
|
+
```python
|
|
273
|
+
#!/usr/bin/env python3
|
|
274
|
+
"""Minimal wrapper: consume ci-guard watch --stream JSONL and execute actions."""
|
|
275
|
+
import json, subprocess, sys
|
|
276
|
+
|
|
277
|
+
pr = sys.argv[1] if len(sys.argv) > 1 else "auto"
|
|
278
|
+
proc = subprocess.Popen(
|
|
279
|
+
["ci-guard", "watch", "--pr", pr, "--stream"],
|
|
280
|
+
stdout=subprocess.PIPE, text=True,
|
|
281
|
+
)
|
|
282
|
+
for line in proc.stdout:
|
|
283
|
+
snap = json.loads(line)
|
|
284
|
+
for action in snap.get("actions", []):
|
|
285
|
+
a = action["action"]
|
|
286
|
+
if a == "retry_failed_now":
|
|
287
|
+
subprocess.run(["ci-guard", "watch", "--pr", pr, "--retry-failed-now"])
|
|
288
|
+
elif a == "verify_flaky_green":
|
|
289
|
+
subprocess.run(["ci-guard", "watch", "--pr", pr, "--verify-flaky-green"])
|
|
290
|
+
elif a == "stop":
|
|
291
|
+
print(f"ci-guard: terminal={snap['terminal']}", flush=True)
|
|
292
|
+
proc.terminate()
|
|
293
|
+
sys.exit(0 if snap["terminal"] in {"pr_merged", "pr_closed"} else 1)
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
The complete JSON contract — every snapshot field, action shape, terminal value, and exit code — is in `references/wrapper-contract.md`.
|
|
297
|
+
|
|
298
|
+
---
|
|
299
|
+
|
|
300
|
+
## Project structure
|
|
301
|
+
|
|
302
|
+
```
|
|
303
|
+
ci-guard/
|
|
304
|
+
├── pyproject.toml # Package metadata; ci-guard console_script entry point
|
|
305
|
+
├── SKILL.md # Playbook any agent reads at runtime
|
|
306
|
+
├── agents/
|
|
307
|
+
│ └── openai.yaml # Codex / skills.sh interface manifest
|
|
308
|
+
├── src/ci_guard/ # Installable Python package (pip install ci-guard-check)
|
|
309
|
+
│ ├── cli.py # Unified CLI — dispatches init/watch/classify/ledger/…
|
|
310
|
+
│ ├── watch.py # Snapshot CI state, classify, gate retries
|
|
311
|
+
│ ├── classify.py # Heuristic log analysis
|
|
312
|
+
│ ├── ledger.py # Persistent flaky-test ledger CLI + Python API
|
|
313
|
+
│ ├── actions.py # GitHub CI consumer — posts PR comments / annotations
|
|
314
|
+
│ ├── init.py # Per-project bootstrap (called by ci-guard init)
|
|
315
|
+
│ ├── config.py # Shared paths, budget defaults, config loader
|
|
316
|
+
│ └── _providers/
|
|
317
|
+
│ └── github.py # gh CLI wrapper (isolated for future multi-provider)
|
|
318
|
+
├── scripts/ # Backward-compat shims (deprecated; use ci-guard <sub>)
|
|
319
|
+
│ ├── bootstrap.py # → ci-guard init
|
|
320
|
+
│ ├── ci_watch.py # → ci-guard watch
|
|
321
|
+
│ ├── classify_failure.py # → ci-guard classify
|
|
322
|
+
│ ├── flaky_ledger.py # → ci-guard ledger
|
|
323
|
+
│ ├── action_runner.py # → ci-guard run-actions
|
|
324
|
+
│ └── config.py # (imported by shims; not a standalone entry point)
|
|
325
|
+
├── docs/
|
|
326
|
+
│ ├── agents.md # Supported agents and LLM compatibility
|
|
327
|
+
│ ├── architecture.md # Component diagram and data flow
|
|
328
|
+
│ ├── github-actions.md # deliver.yml + actions.py walkthrough
|
|
329
|
+
│ ├── updating.md # Keeping ci-guard current
|
|
330
|
+
│ └── troubleshooting.md # Common issues and fixes
|
|
331
|
+
├── references/
|
|
332
|
+
│ ├── heuristics.md # Failure classification decision tree
|
|
333
|
+
│ ├── cost-controls.md # Retry budgets and rationale
|
|
334
|
+
│ ├── flaky-detection.md # Ledger schema and verification protocol
|
|
335
|
+
│ ├── setup.md # Per-project setup steps
|
|
336
|
+
│ ├── ci-providers.md # Adapting to GitLab / CircleCI / Buildkite
|
|
337
|
+
│ └── wrapper-contract.md # --stream JSON contract for wrapper authors
|
|
338
|
+
├── .github/workflows/
|
|
339
|
+
│ ├── ci.yml # Matrix tests (Python 3.9 / 3.11 / 3.13)
|
|
340
|
+
│ └── deliver.yml # workflow_run → ci-guard watch | ci-guard run-actions
|
|
341
|
+
└── assets/
|
|
342
|
+
└── flaky-quarantine-template.md # Issue body template
|
|
343
|
+
```
|
|
344
|
+
|
|
345
|
+
---
|
|
346
|
+
|
|
347
|
+
## Installation
|
|
348
|
+
|
|
349
|
+
### Step 1 — Install the package (once per machine)
|
|
350
|
+
|
|
351
|
+
```bash
|
|
352
|
+
pip install ci-guard-check
|
|
353
|
+
```
|
|
354
|
+
|
|
355
|
+
This installs the `ci-guard` command and all six subcommands: `init`, `watch`, `classify`, `ledger`, `gate`, `run-actions`.
|
|
356
|
+
|
|
357
|
+
### Step 2 — Register the skill for your agent (once per machine)
|
|
358
|
+
|
|
359
|
+
| Agent | Command |
|
|
360
|
+
| --------------------------- | ------------------------------------------------------------------------------ |
|
|
361
|
+
| Claude Code | `ln -s /path/to/ci-guard ~/.claude/skills/ci-guard` |
|
|
362
|
+
| Codex | `ln -s /path/to/ci-guard ~/.codex/skills/ci-guard` |
|
|
363
|
+
| opencode | `ln -s /path/to/ci-guard ~/.opencode/skills/ci-guard` |
|
|
364
|
+
| skills.sh (auto) | `npx skills add https://github.com/Viniciuscarvalho/ci-guard --skill ci-guard` |
|
|
365
|
+
| Gemini CLI / Cursor / other | no native path — see below |
|
|
366
|
+
|
|
367
|
+
> In v0.7+: `ci-guard install-skill --agent claude` (or `codex`, `opencode`, `all`).
|
|
368
|
+
|
|
369
|
+
**Gemini CLI, Cursor, Copilot, or any other agent with no native skill path:**
|
|
370
|
+
paste the contents of `SKILL.md` into your agent context file
|
|
371
|
+
(`.gemini/GEMINI.md`, `.cursorrules`, `AGENTS.md`, etc.).
|
|
372
|
+
|
|
373
|
+
See [`docs/agents.md`](docs/agents.md) for the full agent compatibility matrix.
|
|
374
|
+
|
|
375
|
+
### Step 3 — Bootstrap the repo (once per project)
|
|
376
|
+
|
|
377
|
+
From the project root:
|
|
378
|
+
|
|
379
|
+
```bash
|
|
380
|
+
ci-guard init
|
|
381
|
+
git add .ci-guard .gitignore && git commit -m "ci: bootstrap ci-guard"
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
`ci-guard init` is idempotent — re-running on an already-bootstrapped repo is safe.
|
|
385
|
+
Pass `--dry-run` to preview what would be written without touching any files.
|
|
386
|
+
|
|
387
|
+
### Prerequisites
|
|
388
|
+
|
|
389
|
+
- Python 3.9+
|
|
390
|
+
- `gh` CLI authenticated against the repo's GitHub host (`gh auth status`)
|
|
391
|
+
|
|
392
|
+
### Updating
|
|
393
|
+
|
|
394
|
+
→ Full update guide: [`docs/updating.md`](docs/updating.md)
|
|
395
|
+
|
|
396
|
+
```bash
|
|
397
|
+
pip install --upgrade ci-guard-check
|
|
398
|
+
```
|
|
399
|
+
|
|
400
|
+
That's it. Because scripts now live in the installed package (not in `.ci-guard/scripts/`),
|
|
401
|
+
upgrading the package is the only step needed — no per-project re-bootstrap required.
|
|
402
|
+
|
|
403
|
+
---
|
|
404
|
+
|
|
405
|
+
## Usage
|
|
406
|
+
|
|
407
|
+
All commands run from the project root. `ci-guard` uses `gh` internally — no tokens to manage.
|
|
408
|
+
|
|
409
|
+
### Snapshot a failing PR
|
|
410
|
+
|
|
411
|
+
```bash
|
|
412
|
+
# Infer PR from current branch
|
|
413
|
+
ci-guard watch --pr auto --once
|
|
414
|
+
|
|
415
|
+
# Specific PR number
|
|
416
|
+
ci-guard watch --pr 42 --once
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
Output:
|
|
420
|
+
|
|
421
|
+
```
|
|
422
|
+
PR #42 SHA a1b2c3d
|
|
423
|
+
─────────────────────────────────────────
|
|
424
|
+
Failing checks:
|
|
425
|
+
• test-unit [branch_failure] never retry — fix the code
|
|
426
|
+
• test-e2e [test_flake] retry allowed (1/2 job, 1/5 PR)
|
|
427
|
+
|
|
428
|
+
Greens needing verification:
|
|
429
|
+
• test-integration (in flaky ledger; 8.7% over 30d)
|
|
430
|
+
|
|
431
|
+
Budget: 1/5 retries, 23/90 min spent
|
|
432
|
+
Quarantine candidates: 0
|
|
433
|
+
|
|
434
|
+
Recommended next action: patch the branch_failure in test-unit before retrying test-e2e
|
|
435
|
+
```
|
|
436
|
+
|
|
437
|
+
### Budget-safe retry
|
|
438
|
+
|
|
439
|
+
The only sanctioned way to retry. Refuses if any failure is `branch_failure` or budgets are exceeded.
|
|
440
|
+
|
|
441
|
+
```bash
|
|
442
|
+
ci-guard watch --pr auto --retry-failed-now
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
### Verify a flaky green before merging
|
|
446
|
+
|
|
447
|
+
```bash
|
|
448
|
+
ci-guard watch --pr auto --verify-flaky-green
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
Requires two consecutive greens on the same SHA before the check is trusted.
|
|
452
|
+
|
|
453
|
+
### Continuous stream mode
|
|
454
|
+
|
|
455
|
+
```bash
|
|
456
|
+
ci-guard watch --pr auto --stream
|
|
457
|
+
```
|
|
458
|
+
|
|
459
|
+
Emits JSONL snapshots until a terminal state is reached. Use only when explicitly monitoring a PR; for most diagnosis, `--once` is correct.
|
|
460
|
+
|
|
461
|
+
### Classify an arbitrary run log
|
|
462
|
+
|
|
463
|
+
```bash
|
|
464
|
+
ci-guard classify --run-id <run-id>
|
|
465
|
+
ci-guard classify --log-file <path>
|
|
466
|
+
ci-guard classify --stdin # pipe raw log text
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
### Ledger operations
|
|
470
|
+
|
|
471
|
+
```bash
|
|
472
|
+
# Query a specific test
|
|
473
|
+
ci-guard ledger query --test "tests/auth/test_login.py::test_oauth"
|
|
474
|
+
|
|
475
|
+
# List quarantine candidates
|
|
476
|
+
ci-guard ledger quarantine-candidates
|
|
477
|
+
|
|
478
|
+
# Mark a test quarantined after filing the issue
|
|
479
|
+
ci-guard ledger set-status \
|
|
480
|
+
--test "tests/auth/test_login.py::test_oauth" \
|
|
481
|
+
--status quarantined \
|
|
482
|
+
--issue-url "https://github.com/org/repo/issues/42"
|
|
483
|
+
|
|
484
|
+
# Remove stale entries (no failures in 60 days, not quarantined)
|
|
485
|
+
ci-guard ledger prune --older-than 60
|
|
486
|
+
|
|
487
|
+
# Record a failure manually (e.g. from CI, see below)
|
|
488
|
+
ci-guard ledger record-failure \
|
|
489
|
+
--test "tests/auth/test_login.py::test_oauth" --sha abc1234 --run-id 9876543
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
---
|
|
493
|
+
|
|
494
|
+
## GitHub Actions integration
|
|
495
|
+
|
|
496
|
+
The skill works on-demand from your machine. Two optional CI hooks sharpen the ledger automatically.
|
|
497
|
+
|
|
498
|
+
### Hook 1 — Record failures from the test runner
|
|
499
|
+
|
|
500
|
+
Add to your test workflow's after-tests step:
|
|
501
|
+
|
|
502
|
+
```yaml
|
|
503
|
+
- name: Update flaky ledger
|
|
504
|
+
if: always()
|
|
505
|
+
run: |
|
|
506
|
+
pip install ci-guard-check --quiet
|
|
507
|
+
ci-guard ledger record-failure \
|
|
508
|
+
--test "${TEST_ID}" \
|
|
509
|
+
--sha "${{ github.sha }}" \
|
|
510
|
+
--run-id "${{ github.run_id }}" || true
|
|
511
|
+
```
|
|
512
|
+
|
|
513
|
+
Adapt `TEST_ID` to your framework. Most runners produce JUnit XML; a short loop over failed cases is enough.
|
|
514
|
+
|
|
515
|
+
### Hook 2 — Warn on quarantine candidates
|
|
516
|
+
|
|
517
|
+
Add as a required status check or a standalone workflow step:
|
|
518
|
+
|
|
519
|
+
```yaml
|
|
520
|
+
- name: Quarantine guard
|
|
521
|
+
run: |
|
|
522
|
+
pip install ci-guard-check --quiet
|
|
523
|
+
candidates=$(ci-guard ledger quarantine-candidates)
|
|
524
|
+
if [ -n "$(echo "$candidates" | jq -r '.[]' 2>/dev/null)" ]; then
|
|
525
|
+
echo "::warning::Quarantine candidates exist. Review before merging."
|
|
526
|
+
echo "$candidates"
|
|
527
|
+
fi
|
|
528
|
+
```
|
|
529
|
+
|
|
530
|
+
### Hook 3 — Auto-retry flakes & surface savings report via `deliver.yml`
|
|
531
|
+
|
|
532
|
+
The `deliver.yml` workflow wires `ci-guard` into a `workflow_run` trigger so classifications and savings reports appear on the PR page automatically:
|
|
533
|
+
|
|
534
|
+
```yaml
|
|
535
|
+
# .github/workflows/deliver.yml
|
|
536
|
+
on:
|
|
537
|
+
workflow_run:
|
|
538
|
+
workflows: [ci] # replace with your workflow name(s)
|
|
539
|
+
types: [completed]
|
|
540
|
+
|
|
541
|
+
jobs:
|
|
542
|
+
guard:
|
|
543
|
+
if: github.event.workflow_run.event == 'pull_request'
|
|
544
|
+
runs-on: ubuntu-latest
|
|
545
|
+
permissions:
|
|
546
|
+
actions: write # trigger reruns via gh run rerun
|
|
547
|
+
checks: read
|
|
548
|
+
contents: read
|
|
549
|
+
pull-requests: write # post comments and annotations
|
|
550
|
+
steps:
|
|
551
|
+
- uses: actions/checkout@v4
|
|
552
|
+
with:
|
|
553
|
+
ref: ${{ github.event.workflow_run.head_sha }}
|
|
554
|
+
- name: Watch and guard
|
|
555
|
+
uses: Viniciuscarvalho/ci-guard@v0.5.0
|
|
556
|
+
with:
|
|
557
|
+
mode: guard
|
|
558
|
+
github-token: ${{ secrets.GITHUB_TOKEN }}
|
|
559
|
+
```
|
|
560
|
+
|
|
561
|
+
`ci-guard` watches the PR, handles flaky reruns, and posts:
|
|
562
|
+
|
|
563
|
+
- `::error::` / `::warning::` **workflow annotations** visible in the job log and Files tab.
|
|
564
|
+
- **PR comments** with the complete **Savings & Efficiency Summary** and failure classifications.
|
|
565
|
+
- A **markdown report** in `$GITHUB_STEP_SUMMARY` (visible on the workflow run summary page).
|
|
566
|
+
|
|
567
|
+
→ See [`docs/github-actions.md`](docs/github-actions.md) for the full walkthrough, opt-in instructions, and troubleshooting.
|
|
568
|
+
|
|
569
|
+
### Recommended: enable concurrency cancellation
|
|
570
|
+
|
|
571
|
+
Most "wasted CI" comes from running stale workflows on superseded SHAs, not from retries. Add this to every workflow that runs on PRs:
|
|
572
|
+
|
|
573
|
+
```yaml
|
|
574
|
+
concurrency:
|
|
575
|
+
group: ${{ github.workflow }}-${{ github.ref }}
|
|
576
|
+
cancel-in-progress: true
|
|
577
|
+
```
|
|
578
|
+
|
|
579
|
+
A repo without this typically wastes 10–30% of CI minutes on superseded runs.
|
|
580
|
+
|
|
581
|
+
---
|
|
582
|
+
|
|
583
|
+
## Quarantine workflow
|
|
584
|
+
|
|
585
|
+
When a test crosses the threshold (`failure_count_30d ≥ 3` AND `flake_rate ≥ 5%`):
|
|
586
|
+
|
|
587
|
+
1. ci-guard surfaces a recommendation with the test ID, flake rate, and a skip-pragma snippet.
|
|
588
|
+
2. A human reviews and confirms.
|
|
589
|
+
3. Add the skip pragma to the test file:
|
|
590
|
+
|
|
591
|
+
| Framework | Pragma |
|
|
592
|
+
| ------------- | -------------------------------------------------------------- |
|
|
593
|
+
| pytest | `@pytest.mark.skip(reason="ci-guard quarantine: <issue-url>")` |
|
|
594
|
+
| Jest / Vitest | `it.skip("name", ...)` with a comment linking the issue |
|
|
595
|
+
| Go | `t.Skip("ci-guard quarantine: <issue-url>")` |
|
|
596
|
+
| RSpec | `it "...", skip: "ci-guard quarantine: <issue-url>"` |
|
|
597
|
+
| JUnit 5 | `@Disabled("ci-guard quarantine: <issue-url>")` |
|
|
598
|
+
| Cargo | `#[ignore = "ci-guard quarantine: <issue-url>"]` |
|
|
599
|
+
|
|
600
|
+
4. File a GitHub issue using `assets/flaky-quarantine-template.md`.
|
|
601
|
+
5. Mark the ledger entry quarantined:
|
|
602
|
+
|
|
603
|
+
```bash
|
|
604
|
+
ci-guard ledger set-status \
|
|
605
|
+
--test "<test-id>" --status quarantined \
|
|
606
|
+
--issue-url "https://github.com/org/repo/issues/<n>"
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
Quarantine skips the test in CI — it does not delete it. The fix happens in the tracking issue.
|
|
610
|
+
|
|
611
|
+
---
|
|
612
|
+
|
|
613
|
+
## Other CI providers
|
|
614
|
+
|
|
615
|
+
The scripts target GitHub Actions by default. The provider interface is a single `gh()` wrapper in `ci_watch.py` — only four functions need replacing to adapt to another provider. See `references/ci-providers.md` for GitLab CI (`glab`), CircleCI (REST API), and Buildkite.
|
|
616
|
+
|
|
617
|
+
---
|
|
618
|
+
|
|
619
|
+
## Dependencies
|
|
620
|
+
|
|
621
|
+
- Python 3.9+ (`pip install ci-guard-check` — stdlib only, no transitive deps)
|
|
622
|
+
- `gh` CLI, authenticated against the repo's GitHub host
|
|
623
|
+
|
|
624
|
+
No extra services. Portable across any environment that runs Python and has `gh`.
|
|
625
|
+
|
|
626
|
+
---
|
|
627
|
+
|
|
628
|
+
## Relationship to babysit-pr
|
|
629
|
+
|
|
630
|
+
`babysit-pr` monitors a PR end-to-end through merge. `ci-guard` is the diagnostic and cost layer underneath that decision. They compose: `babysit-pr` can shell out to `ci_watch.py` to classify failures before deciding to retry, and ci-guard's verification protocol catches single-pass-by-luck greens before `babysit-pr` declares the PR ready.
|
|
631
|
+
|
|
632
|
+
---
|
|
633
|
+
|
|
634
|
+
## Documentation
|
|
635
|
+
|
|
636
|
+
| File | Contents |
|
|
637
|
+
| ---------------------------------------------------------- | ------------------------------------------------------------------------- |
|
|
638
|
+
| [`docs/agents.md`](docs/agents.md) | Supported agents, LLM compatibility, full install matrix |
|
|
639
|
+
| [`docs/architecture.md`](docs/architecture.md) | How the pieces fit together (skill → scripts → ledger → PR page) |
|
|
640
|
+
| [`docs/github-actions.md`](docs/github-actions.md) | PR-page surfacing via `deliver.yml` + `action_runner.py` |
|
|
641
|
+
| [`docs/updating.md`](docs/updating.md) | Keeping the skill and per-project scripts current |
|
|
642
|
+
| [`docs/troubleshooting.md`](docs/troubleshooting.md) | Common issues and fixes |
|
|
643
|
+
| [`docs/testing-boss-audit.md`](docs/testing-boss-audit.md) | testing-boss doctrine audit: pass/gap verdicts for tests and skill |
|
|
644
|
+
| [`references/`](references/) | Heuristics, cost rationale, ledger schema, wrapper contract, CI providers |
|