verifaied 0.1.0.dev22__tar.gz → 0.1.0.dev24__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/.gitignore +3 -0
- verifaied-0.1.0.dev24/PKG-INFO +200 -0
- verifaied-0.1.0.dev24/README.md +182 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/pyproject.toml +1 -1
- verifaied-0.1.0.dev22/PKG-INFO +0 -108
- verifaied-0.1.0.dev22/README.md +0 -90
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/src/verifaied/__init__.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/src/verifaied/__main__.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/src/verifaied/cli.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/src/verifaied/client.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/src/verifaied/config.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/src/verifaied/uploader.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/tests/__init__.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/tests/test_cli.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/tests/test_client.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/tests/test_uploader.py +0 -0
- {verifaied-0.1.0.dev22 → verifaied-0.1.0.dev24}/uv.lock +0 -0
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: verifaied
|
|
3
|
+
Version: 0.1.0.dev24
|
|
4
|
+
Summary: Upload local pytest coverage to verifAIed for instant feedback
|
|
5
|
+
Project-URL: Homepage, https://pypi.org/project/verifaied/
|
|
6
|
+
Author: Kyle Richards
|
|
7
|
+
License: MIT
|
|
8
|
+
Keywords: ai,coverage,llm,pytest,testing
|
|
9
|
+
Classifier: Development Status :: 3 - Alpha
|
|
10
|
+
Classifier: Environment :: Console
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Topic :: Software Development :: Testing
|
|
13
|
+
Requires-Python: >=3.10
|
|
14
|
+
Requires-Dist: httpx>=0.27.0
|
|
15
|
+
Requires-Dist: rich>=13.7.0
|
|
16
|
+
Requires-Dist: typer>=0.12.0
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
# verifaied
|
|
20
|
+
|
|
21
|
+
CLI for uploading local pytest coverage to verifAIed so an LLM (or you) can see what's untested without waiting for CI.
|
|
22
|
+
|
|
23
|
+
## Install
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
uv tool install verifaied # or
|
|
27
|
+
pipx install verifaied
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## Configure
|
|
31
|
+
|
|
32
|
+
Set the API token (mint one from the verifAIed app) and, for self-hosted
|
|
33
|
+
or local backends, the API URL:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
export VERIFAIED_API_TOKEN=vr_live_...
|
|
37
|
+
export VERIFAIED_API_URL=http://localhost:8000 # default
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Use
|
|
41
|
+
|
|
42
|
+
From inside any git repo you've connected to verifAIed:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
pytest --cov --cov-report=json --junitxml=junit.xml
|
|
46
|
+
verifaied upload
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
That's it — `--repo` is optional. The CLI parses `git remote get-url origin`,
|
|
50
|
+
looks the repo up under your account via `/installations/me`, and uses the
|
|
51
|
+
resulting UUID. You can also be explicit:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
verifaied upload --repo kyle/verifaied # owner/name slug
|
|
55
|
+
verifaied upload --repo <UUID> # raw UUID, no lookup
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The CLI reads `coverage.json`, pulls the source for every file it
|
|
59
|
+
references from your working tree, and posts everything to
|
|
60
|
+
`/repositories/<id>/local-coverage`. The response prints a summary of
|
|
61
|
+
untested / partial / failing functions so you (or your LLM) can fix
|
|
62
|
+
them on the next iteration.
|
|
63
|
+
|
|
64
|
+
### What gets uploaded
|
|
65
|
+
|
|
66
|
+
The CLI only sends files that appear in `coverage.json`'s `"files"`
|
|
67
|
+
map — i.e. exactly the files `pytest --cov` instrumented. There is no
|
|
68
|
+
directory walk and no glob:
|
|
69
|
+
|
|
70
|
+
- `.env`, build artefacts, vendored libraries, and anything outside
|
|
71
|
+
your coverage scope are never read.
|
|
72
|
+
- Test files themselves are only uploaded if your coverage config
|
|
73
|
+
includes them (e.g. `--cov=tests`).
|
|
74
|
+
|
|
75
|
+
To audit the exact file list (and total bytes) before anything leaves
|
|
76
|
+
your machine, run with `--dry-run`:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
verifaied upload --dry-run
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
This prints the resolved branch / commit / file table and exits without
|
|
83
|
+
contacting the backend. Useful when you're about to upload from an
|
|
84
|
+
unfamiliar repo, or when narrowing down where an unexpected file is
|
|
85
|
+
coming from.
|
|
86
|
+
|
|
87
|
+
If a sensitive value did make it into a covered source file (a baked-in
|
|
88
|
+
API key in a fixture, etc.), open the repo in the verifAIed web UI,
|
|
89
|
+
expand **Recently deleted** under the branches grid, and use
|
|
90
|
+
**Permanently delete**. That hard-deletes the analysis row and cascades
|
|
91
|
+
to the uploaded source — the per-card **Delete** is a soft-delete that
|
|
92
|
+
keeps the data around for accidental-deletion recovery.
|
|
93
|
+
|
|
94
|
+
### Flags
|
|
95
|
+
|
|
96
|
+
- `--repo / -r <UUID|owner/name>` — repository to upload to. If omitted,
|
|
97
|
+
the CLI auto-detects from the GitHub `origin` remote.
|
|
98
|
+
- `--branch / -b <name>` — branch to attach the upload to (default:
|
|
99
|
+
`git branch --show-current`, then `local`)
|
|
100
|
+
- `--coverage <path>` — path to coverage.json (default: `./coverage.json`)
|
|
101
|
+
- `--junit <path>` — optional JUnit XML for failing-test detail
|
|
102
|
+
- `--commit-sha <sha>` — commit sha for display (default: `git rev-parse HEAD`)
|
|
103
|
+
- `--root <path>` — root the coverage paths are relative to (default: cwd)
|
|
104
|
+
- `--api-url <url>` — backend base URL (overrides `VERIFAIED_API_URL`)
|
|
105
|
+
- `--token <token>` — API token (overrides `VERIFAIED_API_TOKEN`)
|
|
106
|
+
- `--dry-run` — print the file list and total bytes that would be
|
|
107
|
+
uploaded, then exit without contacting the backend. Skips the token
|
|
108
|
+
requirement so you can audit without configuring auth.
|
|
109
|
+
|
|
110
|
+
## Agent loop
|
|
111
|
+
|
|
112
|
+
We've found the most fruitful way to use verifAIed is to put a short
|
|
113
|
+
loop into your coding agent's instructions file (`CLAUDE.md`,
|
|
114
|
+
`AGENTS.md`, `.cursorrules`, `GEMINI.md`,
|
|
115
|
+
`.github/copilot-instructions.md`) so it self-corrects on every change.
|
|
116
|
+
|
|
117
|
+
The easiest way is to let your agent write that loop into the rules
|
|
118
|
+
file for you. Paste the prompt below into your agent — it will inspect
|
|
119
|
+
the repo, find the project's real test command, confirm the CLI is
|
|
120
|
+
installed and `VERIFAIED_API_TOKEN` is set, and write the loop section
|
|
121
|
+
into the matching rules file, tailored to your repo:
|
|
122
|
+
|
|
123
|
+
```markdown
|
|
124
|
+
We've found that adding a short test-coverage loop to your agent's instructions is the most fruitful way to use verifAIed. I want you to write that loop into this repo's agent-instructions file, tailored to how this repo actually runs its tests.
|
|
125
|
+
|
|
126
|
+
verifAIed is a coverage analysis tool. Its CLI (`verifaied upload`) pushes a local `coverage.json` + `junit.xml` to the server; its MCP server exposes a `fix_branch` tool that returns a single prompt describing every untested function, partial branch, and failing test on the current branch. The agent loop is: run tests → `verifaied upload` → call `fix_branch` → apply the returned prompt → repeat until `fix_branch` returns no findings.
|
|
127
|
+
|
|
128
|
+
Do the following, in order:
|
|
129
|
+
|
|
130
|
+
# 1. Detect the project setup
|
|
131
|
+
|
|
132
|
+
- **Test runner & command**: look at `pyproject.toml` (`[tool.pytest.ini_options]`, `[project.scripts]`), `pytest.ini`, `tox.ini`, `setup.cfg`, `Makefile` targets (`test`, `check`), `justfile`, or a top-level `scripts/` directory. Use the test command the project already uses — don't invent a new one.
|
|
133
|
+
- **Coverage flags**: pytest must produce `coverage.json` with branch coverage AND per-test contexts, plus `junit.xml`. If the existing command already does that, reuse it. Otherwise build the command:
|
|
134
|
+
```
|
|
135
|
+
pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml
|
|
136
|
+
coverage json --show-contexts -o coverage.json
|
|
137
|
+
```
|
|
138
|
+
The `coverage json --show-contexts` step is non-negotiable — pytest-cov's `--cov-report=json` alone drops the per-test contexts, which makes verifAIed mark the upload "Needs attention".
|
|
139
|
+
- **Agent-instructions file**: pick the one that matches the agent you (the model) are. If unsure, fall back to the project's existing convention.
|
|
140
|
+
- Claude Code: `CLAUDE.md`
|
|
141
|
+
- Codex CLI / OpenAI: `AGENTS.md`
|
|
142
|
+
- Cursor: `.cursorrules` or `.cursor/rules/*.mdc`
|
|
143
|
+
- Gemini CLI: `GEMINI.md`
|
|
144
|
+
- GitHub Copilot: `.github/copilot-instructions.md`
|
|
145
|
+
- If none exists in the repo, create the one matching your own agent.
|
|
146
|
+
|
|
147
|
+
# 2. Verify the CLI is installed and configured
|
|
148
|
+
|
|
149
|
+
- Run `verifaied --help`. If it exits non-zero, install it: `uv tool install verifaied` (preferred) or `pipx install verifaied`.
|
|
150
|
+
- Check `echo $VERIFAIED_API_TOKEN`. If empty, stop and tell the user to mint one from the verifAIed app (Settings → Tokens) and export it: `export VERIFAIED_API_TOKEN=vr_live_...`. Do NOT invent a placeholder and do NOT proceed without it.
|
|
151
|
+
|
|
152
|
+
# 3. Write the loop into the agent-instructions file
|
|
153
|
+
|
|
154
|
+
Append (or create) a section titled `## Test coverage loop with verifAIed` containing:
|
|
155
|
+
|
|
156
|
+
- The exact test command(s) you built in step 1 — so future runs of the agent don't have to re-derive them.
|
|
157
|
+
- The exact upload command (`verifaied upload`, plus any flags this repo needs — e.g. `--coverage path/to/coverage.json` if your test command lands the JSON outside the repo root).
|
|
158
|
+
- An instruction to call the `fix_branch` tool on the verifAIed MCP server with `scope: everything`, then apply the returned prompt.
|
|
159
|
+
- An instruction to repeat from step 1 until `fix_branch` returns no findings.
|
|
160
|
+
- One line: "If `VERIFAIED_API_TOKEN` is unset, stop and ask the user — do not skip this loop."
|
|
161
|
+
|
|
162
|
+
# 4. Confirm with the user
|
|
163
|
+
|
|
164
|
+
Show the user the diff of the file you edited (or created) and the test command you wrote in. Do NOT run the loop yourself yet — let the user review first.
|
|
165
|
+
|
|
166
|
+
# Hard rules
|
|
167
|
+
|
|
168
|
+
- Do NOT change the test command's behavior beyond adding coverage / junit flags.
|
|
169
|
+
- Do NOT add new dependencies if pytest-cov and coverage are already transitive deps.
|
|
170
|
+
- Do NOT write a fallback that skips the loop when the token is missing — the agent must stop and ask.
|
|
171
|
+
- Do NOT enumerate functions yourself or write tests; `fix_branch` returns the prompt that does that.
|
|
172
|
+
- Do NOT replace an existing agent-instructions file — append a new section.
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
### Or paste the instructions in by hand
|
|
176
|
+
|
|
177
|
+
If you'd rather skip the meta-prompt, drop this section into your
|
|
178
|
+
agent's rules file as-is. Swap in your repo's actual test command if
|
|
179
|
+
it differs from the pytest default:
|
|
180
|
+
|
|
181
|
+
```markdown
|
|
182
|
+
## Test coverage loop with verifAIed
|
|
183
|
+
|
|
184
|
+
After implementing or modifying any code in this repo, run this loop until verifAIed reports clean:
|
|
185
|
+
|
|
186
|
+
1. Run the test suite with branch coverage and per-test contexts:
|
|
187
|
+
```
|
|
188
|
+
pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml
|
|
189
|
+
coverage json --show-contexts -o coverage.json
|
|
190
|
+
```
|
|
191
|
+
2. Upload the results to verifAIed:
|
|
192
|
+
```
|
|
193
|
+
verifaied upload
|
|
194
|
+
```
|
|
195
|
+
3. Call the `fix_branch` tool on the verifAIed MCP server with `scope: everything`. It returns a single prompt describing every untested function, partial branch, and failing test on this branch.
|
|
196
|
+
4. Apply the fixes the prompt describes.
|
|
197
|
+
5. Go back to step 1.
|
|
198
|
+
|
|
199
|
+
Stop when `fix_branch` returns no findings. If `VERIFAIED_API_TOKEN` is not set, stop and ask the user — do not skip this loop.
|
|
200
|
+
```
|
|
@@ -0,0 +1,182 @@
|
|
|
1
|
+
# verifaied
|
|
2
|
+
|
|
3
|
+
CLI for uploading local pytest coverage to verifAIed so an LLM (or you) can see what's untested without waiting for CI.
|
|
4
|
+
|
|
5
|
+
## Install
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
uv tool install verifaied # or
|
|
9
|
+
pipx install verifaied
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
## Configure
|
|
13
|
+
|
|
14
|
+
Set the API token (mint one from the verifAIed app) and, for self-hosted
|
|
15
|
+
or local backends, the API URL:
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
export VERIFAIED_API_TOKEN=vr_live_...
|
|
19
|
+
export VERIFAIED_API_URL=http://localhost:8000 # default
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Use
|
|
23
|
+
|
|
24
|
+
From inside any git repo you've connected to verifAIed:
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
pytest --cov --cov-report=json --junitxml=junit.xml
|
|
28
|
+
verifaied upload
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
That's it — `--repo` is optional. The CLI parses `git remote get-url origin`,
|
|
32
|
+
looks the repo up under your account via `/installations/me`, and uses the
|
|
33
|
+
resulting UUID. You can also be explicit:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
verifaied upload --repo kyle/verifaied # owner/name slug
|
|
37
|
+
verifaied upload --repo <UUID> # raw UUID, no lookup
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The CLI reads `coverage.json`, pulls the source for every file it
|
|
41
|
+
references from your working tree, and posts everything to
|
|
42
|
+
`/repositories/<id>/local-coverage`. The response prints a summary of
|
|
43
|
+
untested / partial / failing functions so you (or your LLM) can fix
|
|
44
|
+
them on the next iteration.
|
|
45
|
+
|
|
46
|
+
### What gets uploaded
|
|
47
|
+
|
|
48
|
+
The CLI only sends files that appear in `coverage.json`'s `"files"`
|
|
49
|
+
map — i.e. exactly the files `pytest --cov` instrumented. There is no
|
|
50
|
+
directory walk and no glob:
|
|
51
|
+
|
|
52
|
+
- `.env`, build artefacts, vendored libraries, and anything outside
|
|
53
|
+
your coverage scope are never read.
|
|
54
|
+
- Test files themselves are only uploaded if your coverage config
|
|
55
|
+
includes them (e.g. `--cov=tests`).
|
|
56
|
+
|
|
57
|
+
To audit the exact file list (and total bytes) before anything leaves
|
|
58
|
+
your machine, run with `--dry-run`:
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
verifaied upload --dry-run
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
This prints the resolved branch / commit / file table and exits without
|
|
65
|
+
contacting the backend. Useful when you're about to upload from an
|
|
66
|
+
unfamiliar repo, or when narrowing down where an unexpected file is
|
|
67
|
+
coming from.
|
|
68
|
+
|
|
69
|
+
If a sensitive value did make it into a covered source file (a baked-in
|
|
70
|
+
API key in a fixture, etc.), open the repo in the verifAIed web UI,
|
|
71
|
+
expand **Recently deleted** under the branches grid, and use
|
|
72
|
+
**Permanently delete**. That hard-deletes the analysis row and cascades
|
|
73
|
+
to the uploaded source — the per-card **Delete** is a soft-delete that
|
|
74
|
+
keeps the data around for accidental-deletion recovery.
|
|
75
|
+
|
|
76
|
+
### Flags
|
|
77
|
+
|
|
78
|
+
- `--repo / -r <UUID|owner/name>` — repository to upload to. If omitted,
|
|
79
|
+
the CLI auto-detects from the GitHub `origin` remote.
|
|
80
|
+
- `--branch / -b <name>` — branch to attach the upload to (default:
|
|
81
|
+
`git branch --show-current`, then `local`)
|
|
82
|
+
- `--coverage <path>` — path to coverage.json (default: `./coverage.json`)
|
|
83
|
+
- `--junit <path>` — optional JUnit XML for failing-test detail
|
|
84
|
+
- `--commit-sha <sha>` — commit sha for display (default: `git rev-parse HEAD`)
|
|
85
|
+
- `--root <path>` — root the coverage paths are relative to (default: cwd)
|
|
86
|
+
- `--api-url <url>` — backend base URL (overrides `VERIFAIED_API_URL`)
|
|
87
|
+
- `--token <token>` — API token (overrides `VERIFAIED_API_TOKEN`)
|
|
88
|
+
- `--dry-run` — print the file list and total bytes that would be
|
|
89
|
+
uploaded, then exit without contacting the backend. Skips the token
|
|
90
|
+
requirement so you can audit without configuring auth.
|
|
91
|
+
|
|
92
|
+
## Agent loop
|
|
93
|
+
|
|
94
|
+
We've found the most fruitful way to use verifAIed is to put a short
|
|
95
|
+
loop into your coding agent's instructions file (`CLAUDE.md`,
|
|
96
|
+
`AGENTS.md`, `.cursorrules`, `GEMINI.md`,
|
|
97
|
+
`.github/copilot-instructions.md`) so it self-corrects on every change.
|
|
98
|
+
|
|
99
|
+
The easiest way is to let your agent write that loop into the rules
|
|
100
|
+
file for you. Paste the prompt below into your agent — it will inspect
|
|
101
|
+
the repo, find the project's real test command, confirm the CLI is
|
|
102
|
+
installed and `VERIFAIED_API_TOKEN` is set, and write the loop section
|
|
103
|
+
into the matching rules file, tailored to your repo:
|
|
104
|
+
|
|
105
|
+
```markdown
|
|
106
|
+
We've found that adding a short test-coverage loop to your agent's instructions is the most fruitful way to use verifAIed. I want you to write that loop into this repo's agent-instructions file, tailored to how this repo actually runs its tests.
|
|
107
|
+
|
|
108
|
+
verifAIed is a coverage analysis tool. Its CLI (`verifaied upload`) pushes a local `coverage.json` + `junit.xml` to the server; its MCP server exposes a `fix_branch` tool that returns a single prompt describing every untested function, partial branch, and failing test on the current branch. The agent loop is: run tests → `verifaied upload` → call `fix_branch` → apply the returned prompt → repeat until `fix_branch` returns no findings.
|
|
109
|
+
|
|
110
|
+
Do the following, in order:
|
|
111
|
+
|
|
112
|
+
# 1. Detect the project setup
|
|
113
|
+
|
|
114
|
+
- **Test runner & command**: look at `pyproject.toml` (`[tool.pytest.ini_options]`, `[project.scripts]`), `pytest.ini`, `tox.ini`, `setup.cfg`, `Makefile` targets (`test`, `check`), `justfile`, or a top-level `scripts/` directory. Use the test command the project already uses — don't invent a new one.
|
|
115
|
+
- **Coverage flags**: pytest must produce `coverage.json` with branch coverage AND per-test contexts, plus `junit.xml`. If the existing command already does that, reuse it. Otherwise build the command:
|
|
116
|
+
```
|
|
117
|
+
pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml
|
|
118
|
+
coverage json --show-contexts -o coverage.json
|
|
119
|
+
```
|
|
120
|
+
The `coverage json --show-contexts` step is non-negotiable — pytest-cov's `--cov-report=json` alone drops the per-test contexts, which makes verifAIed mark the upload "Needs attention".
|
|
121
|
+
- **Agent-instructions file**: pick the one that matches the agent you (the model) are. If unsure, fall back to the project's existing convention.
|
|
122
|
+
- Claude Code: `CLAUDE.md`
|
|
123
|
+
- Codex CLI / OpenAI: `AGENTS.md`
|
|
124
|
+
- Cursor: `.cursorrules` or `.cursor/rules/*.mdc`
|
|
125
|
+
- Gemini CLI: `GEMINI.md`
|
|
126
|
+
- GitHub Copilot: `.github/copilot-instructions.md`
|
|
127
|
+
- If none exists in the repo, create the one matching your own agent.
|
|
128
|
+
|
|
129
|
+
# 2. Verify the CLI is installed and configured
|
|
130
|
+
|
|
131
|
+
- Run `verifaied --help`. If it exits non-zero, install it: `uv tool install verifaied` (preferred) or `pipx install verifaied`.
|
|
132
|
+
- Check `echo $VERIFAIED_API_TOKEN`. If empty, stop and tell the user to mint one from the verifAIed app (Settings → Tokens) and export it: `export VERIFAIED_API_TOKEN=vr_live_...`. Do NOT invent a placeholder and do NOT proceed without it.
|
|
133
|
+
|
|
134
|
+
# 3. Write the loop into the agent-instructions file
|
|
135
|
+
|
|
136
|
+
Append (or create) a section titled `## Test coverage loop with verifAIed` containing:
|
|
137
|
+
|
|
138
|
+
- The exact test command(s) you built in step 1 — so future runs of the agent don't have to re-derive them.
|
|
139
|
+
- The exact upload command (`verifaied upload`, plus any flags this repo needs — e.g. `--coverage path/to/coverage.json` if your test command lands the JSON outside the repo root).
|
|
140
|
+
- An instruction to call the `fix_branch` tool on the verifAIed MCP server with `scope: everything`, then apply the returned prompt.
|
|
141
|
+
- An instruction to repeat from step 1 until `fix_branch` returns no findings.
|
|
142
|
+
- One line: "If `VERIFAIED_API_TOKEN` is unset, stop and ask the user — do not skip this loop."
|
|
143
|
+
|
|
144
|
+
# 4. Confirm with the user
|
|
145
|
+
|
|
146
|
+
Show the user the diff of the file you edited (or created) and the test command you wrote in. Do NOT run the loop yourself yet — let the user review first.
|
|
147
|
+
|
|
148
|
+
# Hard rules
|
|
149
|
+
|
|
150
|
+
- Do NOT change the test command's behavior beyond adding coverage / junit flags.
|
|
151
|
+
- Do NOT add new dependencies if pytest-cov and coverage are already transitive deps.
|
|
152
|
+
- Do NOT write a fallback that skips the loop when the token is missing — the agent must stop and ask.
|
|
153
|
+
- Do NOT enumerate functions yourself or write tests; `fix_branch` returns the prompt that does that.
|
|
154
|
+
- Do NOT replace an existing agent-instructions file — append a new section.
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
### Or paste the instructions in by hand
|
|
158
|
+
|
|
159
|
+
If you'd rather skip the meta-prompt, drop this section into your
|
|
160
|
+
agent's rules file as-is. Swap in your repo's actual test command if
|
|
161
|
+
it differs from the pytest default:
|
|
162
|
+
|
|
163
|
+
```markdown
|
|
164
|
+
## Test coverage loop with verifAIed
|
|
165
|
+
|
|
166
|
+
After implementing or modifying any code in this repo, run this loop until verifAIed reports clean:
|
|
167
|
+
|
|
168
|
+
1. Run the test suite with branch coverage and per-test contexts:
|
|
169
|
+
```
|
|
170
|
+
pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml
|
|
171
|
+
coverage json --show-contexts -o coverage.json
|
|
172
|
+
```
|
|
173
|
+
2. Upload the results to verifAIed:
|
|
174
|
+
```
|
|
175
|
+
verifaied upload
|
|
176
|
+
```
|
|
177
|
+
3. Call the `fix_branch` tool on the verifAIed MCP server with `scope: everything`. It returns a single prompt describing every untested function, partial branch, and failing test on this branch.
|
|
178
|
+
4. Apply the fixes the prompt describes.
|
|
179
|
+
5. Go back to step 1.
|
|
180
|
+
|
|
181
|
+
Stop when `fix_branch` returns no findings. If `VERIFAIED_API_TOKEN` is not set, stop and ask the user — do not skip this loop.
|
|
182
|
+
```
|
verifaied-0.1.0.dev22/PKG-INFO
DELETED
|
@@ -1,108 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: verifaied
|
|
3
|
-
Version: 0.1.0.dev22
|
|
4
|
-
Summary: Upload local pytest coverage to verifAIed for instant feedback
|
|
5
|
-
Project-URL: Homepage, https://pypi.org/project/verifaied/
|
|
6
|
-
Author: Kyle Richards
|
|
7
|
-
License: MIT
|
|
8
|
-
Keywords: ai,coverage,llm,pytest,testing
|
|
9
|
-
Classifier: Development Status :: 3 - Alpha
|
|
10
|
-
Classifier: Environment :: Console
|
|
11
|
-
Classifier: Intended Audience :: Developers
|
|
12
|
-
Classifier: Topic :: Software Development :: Testing
|
|
13
|
-
Requires-Python: >=3.10
|
|
14
|
-
Requires-Dist: httpx>=0.27.0
|
|
15
|
-
Requires-Dist: rich>=13.7.0
|
|
16
|
-
Requires-Dist: typer>=0.12.0
|
|
17
|
-
Description-Content-Type: text/markdown
|
|
18
|
-
|
|
19
|
-
# verifaied
|
|
20
|
-
|
|
21
|
-
CLI for uploading local pytest coverage to verifAIed so an LLM (or you) can see what's untested without waiting for CI.
|
|
22
|
-
|
|
23
|
-
## Install
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
uv tool install verifaied # or
|
|
27
|
-
pipx install verifaied
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
## Configure
|
|
31
|
-
|
|
32
|
-
Set the API token (mint one from the verifAIed app) and, for self-hosted
|
|
33
|
-
or local backends, the API URL:
|
|
34
|
-
|
|
35
|
-
```bash
|
|
36
|
-
export VERIFAIED_API_TOKEN=vr_live_...
|
|
37
|
-
export VERIFAIED_API_URL=http://localhost:8000 # default
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
## Use
|
|
41
|
-
|
|
42
|
-
From inside any git repo you've connected to verifAIed:
|
|
43
|
-
|
|
44
|
-
```bash
|
|
45
|
-
pytest --cov --cov-report=json --junitxml=junit.xml
|
|
46
|
-
verifaied upload
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
That's it — `--repo` is optional. The CLI parses `git remote get-url origin`,
|
|
50
|
-
looks the repo up under your account via `/installations/me`, and uses the
|
|
51
|
-
resulting UUID. You can also be explicit:
|
|
52
|
-
|
|
53
|
-
```bash
|
|
54
|
-
verifaied upload --repo kyle/verifaied # owner/name slug
|
|
55
|
-
verifaied upload --repo <UUID> # raw UUID, no lookup
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
The CLI reads `coverage.json`, pulls the source for every file it
|
|
59
|
-
references from your working tree, and posts everything to
|
|
60
|
-
`/repositories/<id>/local-coverage`. The response prints a summary of
|
|
61
|
-
untested / partial / failing functions so you (or your LLM) can fix
|
|
62
|
-
them on the next iteration.
|
|
63
|
-
|
|
64
|
-
### What gets uploaded
|
|
65
|
-
|
|
66
|
-
The CLI only sends files that appear in `coverage.json`'s `"files"`
|
|
67
|
-
map — i.e. exactly the files `pytest --cov` instrumented. There is no
|
|
68
|
-
directory walk and no glob:
|
|
69
|
-
|
|
70
|
-
- `.env`, build artefacts, vendored libraries, and anything outside
|
|
71
|
-
your coverage scope are never read.
|
|
72
|
-
- Test files themselves are only uploaded if your coverage config
|
|
73
|
-
includes them (e.g. `--cov=tests`).
|
|
74
|
-
|
|
75
|
-
To audit the exact file list (and total bytes) before anything leaves
|
|
76
|
-
your machine, run with `--dry-run`:
|
|
77
|
-
|
|
78
|
-
```bash
|
|
79
|
-
verifaied upload --dry-run
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
This prints the resolved branch / commit / file table and exits without
|
|
83
|
-
contacting the backend. Useful when you're about to upload from an
|
|
84
|
-
unfamiliar repo, or when narrowing down where an unexpected file is
|
|
85
|
-
coming from.
|
|
86
|
-
|
|
87
|
-
If a sensitive value did make it into a covered source file (a baked-in
|
|
88
|
-
API key in a fixture, etc.), open the repo in the verifAIed web UI,
|
|
89
|
-
expand **Recently deleted** under the branches grid, and use
|
|
90
|
-
**Permanently delete**. That hard-deletes the analysis row and cascades
|
|
91
|
-
to the uploaded source — the per-card **Delete** is a soft-delete that
|
|
92
|
-
keeps the data around for accidental-deletion recovery.
|
|
93
|
-
|
|
94
|
-
### Flags
|
|
95
|
-
|
|
96
|
-
- `--repo / -r <UUID|owner/name>` — repository to upload to. If omitted,
|
|
97
|
-
the CLI auto-detects from the GitHub `origin` remote.
|
|
98
|
-
- `--branch / -b <name>` — branch to attach the upload to (default:
|
|
99
|
-
`git branch --show-current`, then `local`)
|
|
100
|
-
- `--coverage <path>` — path to coverage.json (default: `./coverage.json`)
|
|
101
|
-
- `--junit <path>` — optional JUnit XML for failing-test detail
|
|
102
|
-
- `--commit-sha <sha>` — commit sha for display (default: `git rev-parse HEAD`)
|
|
103
|
-
- `--root <path>` — root the coverage paths are relative to (default: cwd)
|
|
104
|
-
- `--api-url <url>` — backend base URL (overrides `VERIFAIED_API_URL`)
|
|
105
|
-
- `--token <token>` — API token (overrides `VERIFAIED_API_TOKEN`)
|
|
106
|
-
- `--dry-run` — print the file list and total bytes that would be
|
|
107
|
-
uploaded, then exit without contacting the backend. Skips the token
|
|
108
|
-
requirement so you can audit without configuring auth.
|
verifaied-0.1.0.dev22/README.md
DELETED
|
@@ -1,90 +0,0 @@
|
|
|
1
|
-
# verifaied
|
|
2
|
-
|
|
3
|
-
CLI for uploading local pytest coverage to verifAIed so an LLM (or you) can see what's untested without waiting for CI.
|
|
4
|
-
|
|
5
|
-
## Install
|
|
6
|
-
|
|
7
|
-
```bash
|
|
8
|
-
uv tool install verifaied # or
|
|
9
|
-
pipx install verifaied
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
## Configure
|
|
13
|
-
|
|
14
|
-
Set the API token (mint one from the verifAIed app) and, for self-hosted
|
|
15
|
-
or local backends, the API URL:
|
|
16
|
-
|
|
17
|
-
```bash
|
|
18
|
-
export VERIFAIED_API_TOKEN=vr_live_...
|
|
19
|
-
export VERIFAIED_API_URL=http://localhost:8000 # default
|
|
20
|
-
```
|
|
21
|
-
|
|
22
|
-
## Use
|
|
23
|
-
|
|
24
|
-
From inside any git repo you've connected to verifAIed:
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
pytest --cov --cov-report=json --junitxml=junit.xml
|
|
28
|
-
verifaied upload
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
That's it — `--repo` is optional. The CLI parses `git remote get-url origin`,
|
|
32
|
-
looks the repo up under your account via `/installations/me`, and uses the
|
|
33
|
-
resulting UUID. You can also be explicit:
|
|
34
|
-
|
|
35
|
-
```bash
|
|
36
|
-
verifaied upload --repo kyle/verifaied # owner/name slug
|
|
37
|
-
verifaied upload --repo <UUID> # raw UUID, no lookup
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
The CLI reads `coverage.json`, pulls the source for every file it
|
|
41
|
-
references from your working tree, and posts everything to
|
|
42
|
-
`/repositories/<id>/local-coverage`. The response prints a summary of
|
|
43
|
-
untested / partial / failing functions so you (or your LLM) can fix
|
|
44
|
-
them on the next iteration.
|
|
45
|
-
|
|
46
|
-
### What gets uploaded
|
|
47
|
-
|
|
48
|
-
The CLI only sends files that appear in `coverage.json`'s `"files"`
|
|
49
|
-
map — i.e. exactly the files `pytest --cov` instrumented. There is no
|
|
50
|
-
directory walk and no glob:
|
|
51
|
-
|
|
52
|
-
- `.env`, build artefacts, vendored libraries, and anything outside
|
|
53
|
-
your coverage scope are never read.
|
|
54
|
-
- Test files themselves are only uploaded if your coverage config
|
|
55
|
-
includes them (e.g. `--cov=tests`).
|
|
56
|
-
|
|
57
|
-
To audit the exact file list (and total bytes) before anything leaves
|
|
58
|
-
your machine, run with `--dry-run`:
|
|
59
|
-
|
|
60
|
-
```bash
|
|
61
|
-
verifaied upload --dry-run
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
This prints the resolved branch / commit / file table and exits without
|
|
65
|
-
contacting the backend. Useful when you're about to upload from an
|
|
66
|
-
unfamiliar repo, or when narrowing down where an unexpected file is
|
|
67
|
-
coming from.
|
|
68
|
-
|
|
69
|
-
If a sensitive value did make it into a covered source file (a baked-in
|
|
70
|
-
API key in a fixture, etc.), open the repo in the verifAIed web UI,
|
|
71
|
-
expand **Recently deleted** under the branches grid, and use
|
|
72
|
-
**Permanently delete**. That hard-deletes the analysis row and cascades
|
|
73
|
-
to the uploaded source — the per-card **Delete** is a soft-delete that
|
|
74
|
-
keeps the data around for accidental-deletion recovery.
|
|
75
|
-
|
|
76
|
-
### Flags
|
|
77
|
-
|
|
78
|
-
- `--repo / -r <UUID|owner/name>` — repository to upload to. If omitted,
|
|
79
|
-
the CLI auto-detects from the GitHub `origin` remote.
|
|
80
|
-
- `--branch / -b <name>` — branch to attach the upload to (default:
|
|
81
|
-
`git branch --show-current`, then `local`)
|
|
82
|
-
- `--coverage <path>` — path to coverage.json (default: `./coverage.json`)
|
|
83
|
-
- `--junit <path>` — optional JUnit XML for failing-test detail
|
|
84
|
-
- `--commit-sha <sha>` — commit sha for display (default: `git rev-parse HEAD`)
|
|
85
|
-
- `--root <path>` — root the coverage paths are relative to (default: cwd)
|
|
86
|
-
- `--api-url <url>` — backend base URL (overrides `VERIFAIED_API_URL`)
|
|
87
|
-
- `--token <token>` — API token (overrides `VERIFAIED_API_TOKEN`)
|
|
88
|
-
- `--dry-run` — print the file list and total bytes that would be
|
|
89
|
-
uploaded, then exit without contacting the backend. Skips the token
|
|
90
|
-
requirement so you can audit without configuring auth.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|