agent-council 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/DESIGN.md +378 -0
- data/INSTALL.md +98 -0
- data/LICENSE.txt +21 -0
- data/README.md +852 -0
- data/RELEASE_NOTES.md +25 -0
- data/code-review-agents.md +28 -0
- data/examples/configs/consolidated.yml +22 -0
- data/examples/configs/single.yml +10 -0
- data/examples/prompts/dnsruby-pull-221-prompt.md +4 -0
- data/examples/prompts/project-name-prompt.md +3 -0
- data/examples/prompts/recommend-model-ensemble-prompt.md +62 -0
- data/examples/prompts/review-docs-prompt.md +101 -0
- data/examples/prompts/review-latest-commit-prompt.md +46 -0
- data/examples/prompts/review-pull-request-prompt.md +91 -0
- data/examples/prompts/smoke-test-prompt.md +1 -0
- data/examples/prompts/state-of-the-code-base-prompt.md +172 -0
- data/examples/prompts/test-coverage-analysis-prompt.md +346 -0
- data/exe/agent-council +12 -0
- data/exe/pi-models +39 -0
- data/lib/agent_council/harnesses/pi_harness.rb +169 -0
- data/lib/agent_council/pi_models.rb +185 -0
- data/lib/agent_council/platform.rb +27 -0
- data/lib/agent_council/templates/analyst_format.md +12 -0
- data/lib/agent_council/templates/consolidate_prompt.md +72 -0
- data/lib/agent_council/version.rb +5 -0
- data/lib/agent_council/worker_pool.rb +217 -0
- data/lib/agent_council.rb +1220 -0
- metadata +84 -0
|
@@ -0,0 +1,346 @@
|
|
|
1
|
+
# Test Coverage Analysis by Logical Area Prompt
|
|
2
|
+
|
|
3
|
+
Give this prompt to an AI coding agent from the root of your Ruby project. It
|
|
4
|
+
uses cov-loupe to produce a report of line coverage organized by the logical
|
|
5
|
+
parts of your system, down to individual classes and methods, and ranks the
|
|
6
|
+
gaps by risk. It never asks for input, so it also works headless (e.g.
|
|
7
|
+
`claude -p`, `codex exec`, CI).
|
|
8
|
+
|
|
9
|
+
**Requirement:** run your full test suite (every suite that feeds SimpleCov)
|
|
10
|
+
first, make sure it passes, and don't change source or test files until the
|
|
11
|
+
analysis finishes. The prompt does not run tests; it reads the existing
|
|
12
|
+
`coverage.json` and refuses to start if that file looks out of date. This lets
|
|
13
|
+
several agents run it concurrently against the same coverage data.
|
|
14
|
+
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
### Unattended Operation
|
|
18
|
+
|
|
19
|
+
This prompt never asks the user anything, so it runs the same interactively or
|
|
20
|
+
headless. Never stop to wait for input: when a precondition fails, stop with an
|
|
21
|
+
error message instead. The invocation may name a different report location
|
|
22
|
+
(e.g. "write the report to `tmp/reports`") or coverage file location.
|
|
23
|
+
|
|
24
|
+
### Preconditions
|
|
25
|
+
|
|
26
|
+
Before you begin the analysis:
|
|
27
|
+
|
|
28
|
+
1. If your current working directory is not the project root (the output of
|
|
29
|
+
`git rev-parse --show-toplevel`), stop with an error that states your
|
|
30
|
+
current working directory and the expected root. Do not run the analysis.
|
|
31
|
+
2. Always open the report by citing the most recent git commit at the time you
|
|
32
|
+
begin writing.
|
|
33
|
+
3. If `git status` shows uncommitted changes, include them in the analysis:
|
|
34
|
+
analyze the working tree as it is, and list the `git status` details
|
|
35
|
+
immediately after the commit citation.
|
|
36
|
+
4. Analyze the files git would consider part of the project: tracked files
|
|
37
|
+
plus new files not yet added, excluding anything git ignores
|
|
38
|
+
(`git ls-files --cached --others --exclude-standard`). Mark the untracked
|
|
39
|
+
files in the `git status` details.
|
|
40
|
+
|
|
41
|
+
----
|
|
42
|
+
|
|
43
|
+
- You are a senior engineer assessing the test suite of this project.
|
|
44
|
+
- Your task is to produce a report of line coverage **organized by the logical
|
|
45
|
+
parts of the system**, not just a per-file list, and to judge where the gaps
|
|
46
|
+
actually matter.
|
|
47
|
+
- Get every coverage number from cov-loupe. Do not read `coverage.json`
|
|
48
|
+
directly, and do not estimate coverage by reading tests. Aggregating
|
|
49
|
+
cov-loupe's per-line output into area totals is expected (see Step 4).
|
|
50
|
+
- Prefer the cov-loupe MCP server. If it is unavailable, use the CLI
|
|
51
|
+
(`cov-loupe`, or `bundle exec cov-loupe` if it is in the Gemfile; run
|
|
52
|
+
`cov-loupe --help` for usage), adding `-m cli -fJ` for JSON output. Say in
|
|
53
|
+
the report which one you used.
|
|
54
|
+
- Use source and test files only to define the logical areas and to judge the
|
|
55
|
+
importance of uncovered code.
|
|
56
|
+
- Do not run the test suite, and do not modify source files, test files, or
|
|
57
|
+
coverage output. Other agents may be running this analysis concurrently
|
|
58
|
+
against the same data. This is a read-only analysis; the only file you
|
|
59
|
+
write is the report.
|
|
60
|
+
- Before recommending tests for uncovered code, check whether it is
|
|
61
|
+
intentionally excluded or untested: SimpleCov filters and `# :nocov:`
|
|
62
|
+
markers, and any project documentation or code comments explaining why.
|
|
63
|
+
|
|
64
|
+
## Step 1: Verify the coverage data is current
|
|
65
|
+
|
|
66
|
+
The user must run the full, passing test suite before this analysis. Verify
|
|
67
|
+
what you can, and if any check below fails, stop with an error that names the
|
|
68
|
+
failed check, gives the coverage file's path, size, and modification time (if
|
|
69
|
+
it exists), and tells the user to rerun the full suite. Do not write a report.
|
|
70
|
+
|
|
71
|
+
1. Find the SimpleCov configuration (`.simplecov`, `spec/spec_helper.rb`,
|
|
72
|
+
`test/test_helper.rb`, or wherever the project starts SimpleCov) and the
|
|
73
|
+
coverage file (`coverage/coverage.json` unless the configuration or the
|
|
74
|
+
invocation says otherwise). The file must exist and be written by
|
|
75
|
+
SimpleCov 1.x. Record its path, size in bytes, and modification time (in
|
|
76
|
+
UTC) from the file system (e.g. `ls -l` or `stat`); this metadata is not
|
|
77
|
+
coverage data, so reading it is allowed.
|
|
78
|
+
2. Run the project totals with the tracked globs from Step 2
|
|
79
|
+
(`project_coverage_totals` / `cov-loupe -m cli -fJ -g '<globs>' totals`)
|
|
80
|
+
and the file list with the same globs (`project_coverage` /
|
|
81
|
+
`cov-loupe -m cli -fJ -g '<globs>' list`). `timestamp_status` must be `ok`,
|
|
82
|
+
and `newer_files`, `deleted_files`, `length_mismatch_files`,
|
|
83
|
+
`unreadable_files`, and `skipped_files` must all be empty. These show
|
|
84
|
+
whether source files changed after the run.
|
|
85
|
+
3. No test, fixture, or configuration file may be newer than the coverage
|
|
86
|
+
file, since cov-loupe checks only source files. For example:
|
|
87
|
+
`find spec test features Gemfile.lock .rspec .simplecov -newer coverage/coverage.json`
|
|
88
|
+
(adjusted to the paths that exist). Include directories in the search,
|
|
89
|
+
not just files, so that deleted test files show up too.
|
|
90
|
+
4. If the project records test results, they must show no failures: for
|
|
91
|
+
RSpec, the file named by `example_status_persistence_file_path`
|
|
92
|
+
(commonly `.rspec_status`) must have no `failed` entries.
|
|
93
|
+
|
|
94
|
+
These checks cannot prove that the last run was the complete suite. Say in
|
|
95
|
+
the report that the analysis relies on the user's assurance that it was.
|
|
96
|
+
|
|
97
|
+
`missing_tracked_files` (tracked files the suite never loaded) is not a
|
|
98
|
+
staleness failure; it is a finding for Step 2.
|
|
99
|
+
|
|
100
|
+
## Step 2: Establish the project baseline
|
|
101
|
+
|
|
102
|
+
1. Get overall totals with no globs, then with tracked globs matching the
|
|
103
|
+
code SimpleCov is meant to measure (from its `track_files` and filter
|
|
104
|
+
settings; typically `lib/**/*.rb`, or `app/**/*.rb,lib/**/*.rb` for Rails).
|
|
105
|
+
Passing `tracked_globs` makes cov-loupe report tracked files that have
|
|
106
|
+
**no** coverage data at all (`missing_tracked_files`, `without_coverage`);
|
|
107
|
+
these are files never loaded by the suite and are often the biggest real
|
|
108
|
+
gap.
|
|
109
|
+
2. Get the full file list as JSON, sorted ascending (worst first).
|
|
110
|
+
3. Note what the numbers cover: cov-loupe reports **line** coverage only.
|
|
111
|
+
If SimpleCov also collects branch coverage, say that branch coverage is out
|
|
112
|
+
of scope for this report.
|
|
113
|
+
|
|
114
|
+
## Step 3: Identify the logical areas
|
|
115
|
+
|
|
116
|
+
Before computing any rollups, decide how to carve up the code. The goal is the
|
|
117
|
+
most precise, targeted analysis, so draw boundaries by what the code does, not
|
|
118
|
+
by where it lives. File and directory boundaries are not area boundaries: one
|
|
119
|
+
file may hold a class or module with a critical mission next to code that
|
|
120
|
+
matters little, and those belong in different areas. This takes more reading
|
|
121
|
+
and more tokens than a file-level rollup; that is the point.
|
|
122
|
+
|
|
123
|
+
1. Read the code to understand its structure: entry points, `require` and
|
|
124
|
+
autoload structure, namespaces, classes, modules, and methods, and how
|
|
125
|
+
responsibilities flow between them. Identify the kind of project (Rails
|
|
126
|
+
app, gem, CLI tool, service) and its conventions.
|
|
127
|
+
2. Define **logical areas at whatever granularity is meaningful**. An area is
|
|
128
|
+
a set of code units (classes, modules, methods, or line ranges) wherever
|
|
129
|
+
they live. It can be:
|
|
130
|
+
- a broad grouping spanning several directories (e.g. "everything in the
|
|
131
|
+
checkout flow", "the public API"),
|
|
132
|
+
- a single namespace or class,
|
|
133
|
+
- a single method or a few methods within a class (e.g. the price
|
|
134
|
+
calculation inside an order model, or the argument parsing in a CLI
|
|
135
|
+
command),
|
|
136
|
+
- or a cross-cutting concern spanning many places (e.g. authorization,
|
|
137
|
+
error handling, input validation).
|
|
138
|
+
3. Arrange the areas in a hierarchy where that helps (a broad area containing
|
|
139
|
+
narrower ones), and nest as deep as it is useful. Cross-cutting areas may
|
|
140
|
+
overlap others; say so where they do.
|
|
141
|
+
4. Record each area's definition precisely as file paths with line ranges, and
|
|
142
|
+
name the classes, modules, or methods those ranges cover so a reader can
|
|
143
|
+
follow them. When an area is made of whole files, a glob is a fine
|
|
144
|
+
shorthand.
|
|
145
|
+
5. Make sure at least one level of the hierarchy is a **complete partition**:
|
|
146
|
+
every coverable line in the source files under analysis belongs to exactly
|
|
147
|
+
one area at that level, with an explicit "Other" area for code that fits
|
|
148
|
+
nowhere. This level is what the totals are reconciled against.
|
|
149
|
+
6. Give each area a **criticality** judgment with a short reason. Treat it as a
|
|
150
|
+
continuum, not a fixed set of buckets: place each area relative to the
|
|
151
|
+
others and explain what drives the placement. Weigh user-visible behavior,
|
|
152
|
+
data correctness and integrity, money and security (authentication,
|
|
153
|
+
authorization, code evaluation, file system access), error paths, public
|
|
154
|
+
API, and how often the code runs. Use the framework's usual danger spots as
|
|
155
|
+
prompts, not as fixed categories: in a Rails app, validations, callbacks,
|
|
156
|
+
and scopes that guard data; authorization in controllers; background-job
|
|
157
|
+
retries and failure handling; mailers; CSRF and parameter filtering. In
|
|
158
|
+
other projects, look at domain rules and calculations, state transitions,
|
|
159
|
+
external API clients, file I/O and parsing, serialization, and concurrency.
|
|
160
|
+
|
|
161
|
+
## Step 4: Compute coverage per area
|
|
162
|
+
|
|
163
|
+
1. For each file touched by an area, get per-line data with
|
|
164
|
+
`file_coverage_detailed` / `cov-loupe -m cli -fJ detailed <path>`. Each row
|
|
165
|
+
gives a line number, its hit count, and whether it is covered; lines that
|
|
166
|
+
are not coverable (comments, blank lines, `end`) do not appear.
|
|
167
|
+
2. For each area, sum the covered and coverable lines that fall in its line
|
|
168
|
+
ranges, and compute the percentage. Also record the number of code units
|
|
169
|
+
in the area and its weakest units (e.g. the least-covered methods).
|
|
170
|
+
3. Get a broader area's numbers by summing its children's line counts, never
|
|
171
|
+
by averaging child percentages.
|
|
172
|
+
4. For an area made of whole files, cross-check your sums against
|
|
173
|
+
`project_coverage_totals` with `tracked_globs` /
|
|
174
|
+
`cov-loupe -m cli -fJ -g '<globs>' totals`; they must match exactly.
|
|
175
|
+
5. Confirm that the line totals for the partition level from Step 3 add up to
|
|
176
|
+
the project's covered and total lines for the same overall globs. If they
|
|
177
|
+
don't, find the overlap or orphaned lines and fix the map; state any
|
|
178
|
+
remaining discrepancy.
|
|
179
|
+
6. Keep the aggregation mechanical and reproducible. If you write a helper
|
|
180
|
+
script to sum the per-line rows, put it in a temporary directory of your
|
|
181
|
+
own outside the project (e.g. from `mktemp -d`, so concurrent agents don't
|
|
182
|
+
collide), feed it cov-loupe output only, and include it in the appendix.
|
|
183
|
+
|
|
184
|
+
## Step 5: Examine the gaps
|
|
185
|
+
|
|
186
|
+
Decide for yourself which files and areas deserve a close look. Do not apply a
|
|
187
|
+
fixed percentage cutoff: weigh each area's coverage against its criticality,
|
|
188
|
+
so a critical area at a high percentage can still warrant scrutiny while a
|
|
189
|
+
low-stakes area at a lower percentage may not. Always include every tracked
|
|
190
|
+
file with no coverage data. State the reasoning behind your selection in the
|
|
191
|
+
report.
|
|
192
|
+
|
|
193
|
+
For each selected file or area:
|
|
194
|
+
|
|
195
|
+
1. Get the uncovered lines (`file_uncovered_lines` /
|
|
196
|
+
`cov-loupe -m cli -fJ uncovered <path>`); use `file_coverage_detailed` or
|
|
197
|
+
`cov-loupe -m cli -s uncovered -n 2 uncovered <path>` to see the source in
|
|
198
|
+
context.
|
|
199
|
+
2. Characterize each cluster of uncovered lines: error/rescue path, edge-case
|
|
200
|
+
branch, defensive guard, platform-specific code, dead code, debug/logging
|
|
201
|
+
code, or core logic.
|
|
202
|
+
3. Find the tests for that code, following the project's conventions (e.g.
|
|
203
|
+
`spec/` mirroring `lib/` or `app/`, or `test/`). Note whether the gap is
|
|
204
|
+
"no tests at all", "tests exist but miss these paths", or "covered only
|
|
205
|
+
indirectly by integration tests".
|
|
206
|
+
4. Judge the **risk** of each gap: what goes wrong for a user if this code is
|
|
207
|
+
broken and no test notices? Be concrete.
|
|
208
|
+
5. Judge, separately from risk, how **hard the gap is to test**: dependencies
|
|
209
|
+
to set up, I/O, external services, time, concurrency, or missing seams
|
|
210
|
+
(code that can't be exercised without refactoring).
|
|
211
|
+
6. Flag likely dead code separately: it should be deleted, not tested.
|
|
212
|
+
|
|
213
|
+
Do not pad the report: a high percentage with an untested critical error path
|
|
214
|
+
is worth more attention than modest coverage of formatting helpers.
|
|
215
|
+
|
|
216
|
+
## Step 6: Look beyond line percentages
|
|
217
|
+
|
|
218
|
+
Briefly assess, per major area, using the tests themselves:
|
|
219
|
+
|
|
220
|
+
- Whether coverage comes from focused unit tests or only from broad
|
|
221
|
+
integration/end-to-end tests (high coverage via integration alone can hide
|
|
222
|
+
weak assertions).
|
|
223
|
+
- Whether tests assert on behavior or merely execute code (spot-check a few
|
|
224
|
+
tests per area; cite them).
|
|
225
|
+
- Consistency across entry points: behavior reachable through more than one
|
|
226
|
+
interface (e.g. web UI and API, CLI and library) should be tested through
|
|
227
|
+
each.
|
|
228
|
+
- Testing anti-patterns. Report only those you actually observe, such as:
|
|
229
|
+
over-mocking that makes tests brittle or hides integration bugs;
|
|
230
|
+
near-duplicate test code that should use shared examples, helpers, or
|
|
231
|
+
iteration over input arrays; untested error handling and failure
|
|
232
|
+
scenarios; missing integration tests for critical user flows; untested
|
|
233
|
+
authorization logic; controller tests where request specs would serve
|
|
234
|
+
better (Rails); missing tests for jobs, mailers, external integrations,
|
|
235
|
+
CLI argument parsing, or data transformation pipelines.
|
|
236
|
+
- Testing infrastructure problems: inadequate fixtures or factories, missing
|
|
237
|
+
helpers, or code structure that makes tests hard to write.
|
|
238
|
+
- Well-tested areas: where coverage and test quality are both strong, and
|
|
239
|
+
the testing patterns there that are worth repeating elsewhere.
|
|
240
|
+
|
|
241
|
+
Keep this qualitative and evidence-based; cite test files and line numbers.
|
|
242
|
+
|
|
243
|
+
## Report
|
|
244
|
+
|
|
245
|
+
Write your analysis in the project root (or in the directory the invocation
|
|
246
|
+
names) as a Markdown file whose name is:
|
|
247
|
+
- today's date in UTC `%Y-%m-%d-%H-%M` format +
|
|
248
|
+
- '-test-coverage-analysis-' +
|
|
249
|
+
- your name (e.g. 'codex', 'claude', 'gemini') +
|
|
250
|
+
- the `.md` extension.
|
|
251
|
+
|
|
252
|
+
If a file with that name already exists (another agent finished in the same
|
|
253
|
+
minute), append `-2`, `-3`, and so on before the extension. Never overwrite an
|
|
254
|
+
existing report.
|
|
255
|
+
|
|
256
|
+
The file should have the following structure:
|
|
257
|
+
|
|
258
|
+
---
|
|
259
|
+
|
|
260
|
+
### Executive Summary
|
|
261
|
+
- Commit cited, cov-loupe interface used (MCP or CLI), and the results of the
|
|
262
|
+
Step 1 checks.
|
|
263
|
+
- The coverage file's path (relative to the project root), size, and
|
|
264
|
+
modification time (UTC), so readers can tell which test run the report
|
|
265
|
+
describes.
|
|
266
|
+
- Overall line coverage (with and without tracked globs).
|
|
267
|
+
- The best- and worst-covered areas, and the top risk areas.
|
|
268
|
+
- A **one-line verdict** (e.g. *"Strong domain coverage; payment error
|
|
269
|
+
handling is the main gap"*).
|
|
270
|
+
|
|
271
|
+
### Area Map
|
|
272
|
+
The hierarchy of logical areas from Step 3, showing for each area its parent,
|
|
273
|
+
its definition (file paths with line ranges and the code units they cover, or
|
|
274
|
+
globs for whole files), and criticality with reason. Mark the partition level
|
|
275
|
+
used for reconciliation and any overlapping cross-cutting areas. Explain why
|
|
276
|
+
you carved the code this way.
|
|
277
|
+
|
|
278
|
+
### Coverage by Area
|
|
279
|
+
One table per hierarchy level that is worth showing, each sorted ascending by
|
|
280
|
+
percentage:
|
|
281
|
+
|
|
282
|
+
| Area | Criticality | Code units | Covered | Total | % | Weakest unit (%) |
|
|
283
|
+
|------|-------------|------------|---------|-------|---|------------------|
|
|
284
|
+
|
|
285
|
+
List tracked files with no coverage data separately, with the areas they
|
|
286
|
+
belong to.
|
|
287
|
+
|
|
288
|
+
Follow with a short interpretation of each area that stands out, positively or
|
|
289
|
+
negatively, relative to its criticality.
|
|
290
|
+
|
|
291
|
+
### Gap Analysis
|
|
292
|
+
State how you chose what to examine. Then, for each gap from Step 5, grouped by
|
|
293
|
+
area and ordered by risk:
|
|
294
|
+
- File and uncovered line numbers (as reported by cov-loupe).
|
|
295
|
+
- What the code does and the gap's category.
|
|
296
|
+
- Test status (none / partial / indirect only).
|
|
297
|
+
- Risk, with a concrete failure scenario.
|
|
298
|
+
- Difficulty to test, with what makes it hard.
|
|
299
|
+
- Recommendation: test to add (describe the scenario, not the code), code to
|
|
300
|
+
delete, or "acceptable as is" with the reason.
|
|
301
|
+
|
|
302
|
+
### Well-Tested Areas
|
|
303
|
+
Areas where coverage and test quality are both strong, and the patterns
|
|
304
|
+
worth repeating, with citations.
|
|
305
|
+
|
|
306
|
+
### Test Quality Observations
|
|
307
|
+
The remaining findings from Step 6 (unit vs. integration balance, assertion
|
|
308
|
+
strength, entry-point consistency, anti-patterns, infrastructure), with
|
|
309
|
+
citations.
|
|
310
|
+
|
|
311
|
+
### Data Caveats
|
|
312
|
+
Reliance on the user's assurance that the last run was the complete, passing
|
|
313
|
+
suite; tracked files missing from the coverage data; branch coverage being out
|
|
314
|
+
of scope; ambiguous line assignments at area boundaries; anything else that
|
|
315
|
+
limits confidence in the numbers.
|
|
316
|
+
|
|
317
|
+
### Prioritized Recommendations
|
|
318
|
+
A table of tasks, sorted by priority (highest first):
|
|
319
|
+
|
|
320
|
+
| # | Task | Area | Importance (1–10) | Effort (S/M/L) | Priority | Expected improvement |
|
|
321
|
+
|---|------|------|-------------------|----------------|----------|----------------------|
|
|
322
|
+
|
|
323
|
+
- **Task**: one concrete action (tests to add, code to delete, or an
|
|
324
|
+
infrastructure fix).
|
|
325
|
+
- **Importance**: the risk the task removes, from the area's criticality and
|
|
326
|
+
the gap's risk.
|
|
327
|
+
- **Effort**: from the gap's difficulty to test (Step 5).
|
|
328
|
+
- **Priority**: rank by importance relative to effort. Among tasks of similar
|
|
329
|
+
importance, put the cheaper one first: quick wins are easy to clear, and each
|
|
330
|
+
one cleared drops out of the next report, so later reports concentrate on
|
|
331
|
+
what remains. Place infrastructure fixes ahead of the tasks they unblock.
|
|
332
|
+
|
|
333
|
+
Briefly state the rule you used to combine importance and effort. After the
|
|
334
|
+
table, give each task a ready-to-use prompt the user can give an AI assistant
|
|
335
|
+
to implement it, keyed by its number.
|
|
336
|
+
|
|
337
|
+
### Appendix: Commands Run
|
|
338
|
+
Every cov-loupe call (MCP tool name and arguments, or CLI command) and
|
|
339
|
+
freshness check used to produce the numbers, plus any aggregation script, so the
|
|
340
|
+
report can be reproduced.
|
|
341
|
+
|
|
342
|
+
---
|
|
343
|
+
|
|
344
|
+
End your final message with the report's path and the one-line verdict, or,
|
|
345
|
+
if you stopped early, with the error. A headless caller can then find the
|
|
346
|
+
outcome without parsing the rest of your output.
|
data/exe/agent-council
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
# agent-council — executable entry point. All code lives in lib/:
|
|
5
|
+
# lib/agent_council.rb orchestration (AgentCouncil)
|
|
6
|
+
# lib/agent_council/harnesses/pi_harness.rb pi-specific behavior (AgentCouncil::Harnesses::PiHarness)
|
|
7
|
+
#
|
|
8
|
+
# Usage, options, and AGENT_COUNCIL_OPTS: agent-council -h, and the README.
|
|
9
|
+
|
|
10
|
+
require_relative '../lib/agent_council'
|
|
11
|
+
|
|
12
|
+
AgentCouncil.main(ARGV)
|
data/exe/pi-models
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
require_relative '../lib/agent_council/platform'
|
|
5
|
+
|
|
6
|
+
AgentCouncil.abort_if_native_windows('pi-models')
|
|
7
|
+
|
|
8
|
+
require_relative '../lib/agent_council/pi_models'
|
|
9
|
+
|
|
10
|
+
help_output = ARGV.delete('-h') || ARGV.delete('--help')
|
|
11
|
+
json_output = ARGV.delete('--json') || ARGV.delete('-j')
|
|
12
|
+
ids_output = ARGV.delete('--ids') || ARGV.delete('-i')
|
|
13
|
+
ARGV.delete('--table') || ARGV.delete('-t') # table is the default format; accepted for explicitness
|
|
14
|
+
|
|
15
|
+
if help_output
|
|
16
|
+
puts AgentCouncil::PiModels.help_text
|
|
17
|
+
exit
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
unless ARGV.empty?
|
|
21
|
+
warn "pi-models: unknown option(s): #{ARGV.join(' ')}"
|
|
22
|
+
warn "Try 'pi-models --help' for usage."
|
|
23
|
+
exit 1
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
begin
|
|
27
|
+
models = AgentCouncil::PiModels.fetch_models
|
|
28
|
+
rescue AgentCouncil::PiModels::Error => e
|
|
29
|
+
warn "pi-models: #{e.message}"
|
|
30
|
+
exit 1
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
if json_output
|
|
34
|
+
puts AgentCouncil::PiModels.format_as_json(models)
|
|
35
|
+
elsif ids_output
|
|
36
|
+
print AgentCouncil::PiModels.format_as_ids(models)
|
|
37
|
+
else
|
|
38
|
+
puts AgentCouncil::PiModels.format_as_table(models)
|
|
39
|
+
end
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# AgentCouncil::Harnesses::PiHarness — all knowledge specific to the pi
|
|
4
|
+
# coding-agent harness: how to invoke the pi CLI for one model run, the
|
|
5
|
+
# tool allowlist, and read-only scaffolding delivered with it, and how to
|
|
6
|
+
# read pi's JSONL event stream afterward.
|
|
7
|
+
#
|
|
8
|
+
# AgentCouncil is harness-agnostic and refers to this module only via
|
|
9
|
+
# AgentCouncil::HARNESS. A second harness (claude, codex, ...) would be
|
|
10
|
+
# another module under AgentCouncil::Harnesses shaped like this one —
|
|
11
|
+
# this is deliberately a plain namespace, not an adapter framework.
|
|
12
|
+
|
|
13
|
+
require 'json'
|
|
14
|
+
|
|
15
|
+
module AgentCouncil
|
|
16
|
+
module Harnesses
|
|
17
|
+
module PiHarness
|
|
18
|
+
NAME = 'pi'
|
|
19
|
+
|
|
20
|
+
# Analysis toolset (pi's built-in tool names): edit/write (and the
|
|
21
|
+
# Windows-only powershell) are withheld. bash, a full shell, stays for
|
|
22
|
+
# investigation (git log/status, dependency inspection, ...); only the
|
|
23
|
+
# scaffolding text asks the model to use it read-only.
|
|
24
|
+
TOOLS = 'read,grep,find,ls,bash'
|
|
25
|
+
|
|
26
|
+
# Invocation for one model run. The substantive prompt is not part of argv;
|
|
27
|
+
# it is piped to stdin (byte-identical for every analyst; the
|
|
28
|
+
# consolidator gets its own).
|
|
29
|
+
module_function def argv(provider, model, scratch_dir)
|
|
30
|
+
[
|
|
31
|
+
NAME,
|
|
32
|
+
'-p', # non-interactive: process prompt and exit
|
|
33
|
+
'--mode', 'json', # JSONL event stream on stdout
|
|
34
|
+
'--no-session', # ephemeral: no session state between runs
|
|
35
|
+
'--tools', TOOLS,
|
|
36
|
+
'--provider', provider,
|
|
37
|
+
'--model', model,
|
|
38
|
+
'--append-system-prompt', scaffold_text(scratch_dir)
|
|
39
|
+
]
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
# Scaffolding appended to pi's system prompt. Identical for every agent
|
|
43
|
+
# except the scratch dir path.
|
|
44
|
+
module_function def scaffold_text(scratch_dir)
|
|
45
|
+
<<~TEXT
|
|
46
|
+
You are one of the independent agents agent-council runs on this
|
|
47
|
+
repository. This is a strictly read-only analysis task.
|
|
48
|
+
|
|
49
|
+
Rules:
|
|
50
|
+
- Do not modify, create, rename, or delete any project source or
|
|
51
|
+
configuration file. Do not change git state, install or remove
|
|
52
|
+
software, modify user or system configuration, or start persistent
|
|
53
|
+
services.
|
|
54
|
+
- Do not run test suites, builds, coverage tools, application servers,
|
|
55
|
+
or database tasks unless the user's prompt explicitly permits it:
|
|
56
|
+
other agents may be running concurrently and such commands can collide
|
|
57
|
+
through shared state (databases, ports, Docker, fixed temp paths).
|
|
58
|
+
Where the prompt permits them, direct their temporary output to the
|
|
59
|
+
scratch directory below, and still skip any that need shared
|
|
60
|
+
services (databases, servers, fixed ports, Docker). Otherwise, if
|
|
61
|
+
running such a command would materially improve your analysis,
|
|
62
|
+
describe what you would run and why instead of running it.
|
|
63
|
+
- You may run clearly read-only commands: reading, searching, and
|
|
64
|
+
listing files; inspecting git history and status; inspecting
|
|
65
|
+
dependency and configuration information.
|
|
66
|
+
- If you need scratch space for temporary artifacts, use this
|
|
67
|
+
disposable directory that is unique to your run: #{scratch_dir}
|
|
68
|
+
- Deliver your complete analysis as your final message text. Do not
|
|
69
|
+
write the report to a file; the orchestrator captures your final
|
|
70
|
+
message directly.
|
|
71
|
+
TEXT
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# --- event stream reading ---
|
|
75
|
+
|
|
76
|
+
# The assistant message from a single pi JSONL line, or nil if the line
|
|
77
|
+
# is not an assistant message_end event.
|
|
78
|
+
module_function def assistant_message_from_line(line)
|
|
79
|
+
event =
|
|
80
|
+
begin
|
|
81
|
+
JSON.parse(line)
|
|
82
|
+
rescue JSON::ParserError
|
|
83
|
+
nil
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
return nil unless event.is_a?(Hash)
|
|
87
|
+
return nil unless event['type'] == 'message_end'
|
|
88
|
+
|
|
89
|
+
message = event['message']
|
|
90
|
+
return nil unless message && message['role'] == 'assistant'
|
|
91
|
+
|
|
92
|
+
message
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
# Assistant messages from all message_end events in a pi JSONL stream,
|
|
96
|
+
# in order. Unparseable lines are skipped.
|
|
97
|
+
module_function def assistant_messages(jsonl)
|
|
98
|
+
jsonl.each_line.filter_map { |line| assistant_message_from_line(line) }
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
module_function def response_text(message)
|
|
102
|
+
Array(message['content'])
|
|
103
|
+
.select { |block| block['type'] == 'text' }
|
|
104
|
+
.map { |block| block['text'] }
|
|
105
|
+
.join("\n")
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
# The final assistant response: per pi's docs, message_end holds the
|
|
109
|
+
# authoritative final message of each turn; the last assistant message_end
|
|
110
|
+
# with actual text is the report (earlier ones belong to tool-use turns).
|
|
111
|
+
module_function def final_response(jsonl)
|
|
112
|
+
message = saved_message(assistant_messages(jsonl))
|
|
113
|
+
message && response_text(message)
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
# The message final_response saves: the last one with text.
|
|
117
|
+
module_function def saved_message(messages)
|
|
118
|
+
messages.reverse_each.detect { |m| !response_text(m).empty? }
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
# Our usage keys, each mapped to pi's name for it. pi normalizes usage
|
|
122
|
+
# across providers; its input excludes cache reads and writes.
|
|
123
|
+
USAGE_KEYS = { 'input' => 'input', 'output' => 'output',
|
|
124
|
+
'cache_read' => 'cacheRead', 'cache_write' => 'cacheWrite' }.freeze
|
|
125
|
+
|
|
126
|
+
# Token usage summed over all assistant messages (each is a separate
|
|
127
|
+
# model request), as AgentCouncil expects it: integer 'input', 'output',
|
|
128
|
+
# 'cache_read', and 'cache_write', plus 'cost' in US dollars when pi
|
|
129
|
+
# priced the run. nil when the stream reports no usage, as some
|
|
130
|
+
# providers don't.
|
|
131
|
+
module_function def usage(jsonl)
|
|
132
|
+
usages = assistant_messages(jsonl).map { |m| m['usage'] }.grep(Hash)
|
|
133
|
+
totals = USAGE_KEYS.transform_values { |key| usages.sum { |u| count(u[key]) } }
|
|
134
|
+
return nil if totals.values.all?(&:zero?)
|
|
135
|
+
|
|
136
|
+
cost = usages.sum { |u| u['cost'].is_a?(Hash) ? count(u['cost']['total']) : 0 }
|
|
137
|
+
cost.positive? ? totals.merge('cost' => cost.round(6)) : totals
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
module_function def count(value)
|
|
141
|
+
value.is_a?(Numeric) && value.positive? ? value : 0
|
|
142
|
+
end
|
|
143
|
+
|
|
144
|
+
# Stop reasons meaning a message was cut off, cancelled, or failed, not
|
|
145
|
+
# completed. error normally comes with an errorMessage, which is reported
|
|
146
|
+
# instead; this catches one that is missing or blank.
|
|
147
|
+
UNUSABLE_STOP_REASONS = %w[length aborted error].freeze
|
|
148
|
+
|
|
149
|
+
# Why the run produced no usable report, or nil. Pi exits 0 even when the
|
|
150
|
+
# provider rejects the request or the output hits its token limit, so this
|
|
151
|
+
# is how such failures are surfaced. Two messages are checked: the final
|
|
152
|
+
# one (an error in an earlier turn that a later turn recovered from is not
|
|
153
|
+
# a failure, while an error on the last turn means any earlier text is a
|
|
154
|
+
# partial fragment) and the one final_response would save (a cut-off
|
|
155
|
+
# message must not be saved just because a text-less message followed it).
|
|
156
|
+
# A non-blank string errorMessage on either wins over a cut-off stopReason on
|
|
157
|
+
# the other, since it says more about what went wrong.
|
|
158
|
+
module_function def error_message(jsonl)
|
|
159
|
+
messages = assistant_messages(jsonl)
|
|
160
|
+
candidates = [messages.last, saved_message(messages)].compact.uniq
|
|
161
|
+
message = candidates.detect { |m| m['errorMessage'].is_a?(String) && !m['errorMessage'].strip.empty? }
|
|
162
|
+
return message['errorMessage'] if message
|
|
163
|
+
|
|
164
|
+
message = candidates.detect { |m| UNUSABLE_STOP_REASONS.include?(m['stopReason']) }
|
|
165
|
+
message && "stopped early (stopReason: #{message['stopReason']})"
|
|
166
|
+
end
|
|
167
|
+
end
|
|
168
|
+
end
|
|
169
|
+
end
|