claude-dev-env 8.96.2 → 8.96.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/build-eval/reference/build-evaluation.md +4 -4
- package/.agents/skills/pr-lifecycle/SKILL.md +3 -3
- package/.agents/skills/pull-request/SKILL.md +4 -3
- package/.agents/skills/pull-request/scripts/github_pr_command_constants/config/constants.py +14 -0
- package/.agents/skills/pull-request/scripts/pull_request.py +56 -0
- package/.agents/skills/pull-request/scripts/test_pull_request.py +82 -0
- package/docs/rule-guides/session-title.md +1 -1
- package/hooks/blocking/session_title_turn_gate.py +54 -11
- package/hooks/blocking/test_session_title_turn_gate.py +49 -0
- package/hooks/features/conduct-gates.md +1 -1
- package/package.json +1 -1
|
@@ -6,14 +6,14 @@ Use this process when an AI workflow needs measured output quality. Keep the exi
|
|
|
6
6
|
2. Inspect existing cases, saved runs and graders. Record which checks execute a model, replay stored outputs, exercise code with mocked providers or verify an environment end state. Reuse trustworthy components. A test name containing "eval" is not execution evidence.
|
|
7
7
|
3. Gather representative inputs. Prefer retained, sanitized application failures and accepted outputs. Record source IDs, label authors and label provenance. Minimize sensitive inputs. If only synthetic cases are feasible, anchor them to documented contracts, name that limitation and keep human review outstanding.
|
|
8
8
|
4. Cover positive, negative and near-miss cases. Include confusing valid inputs and malicious instructions inside task data. Group related cases before splitting so paraphrases and paired examples do not cross development and held-out sets. Save a dataset hash. Keep labels and scoring code outside the model's task context.
|
|
9
|
-
5. Choose grading that measures the outcome. Use exact labels, schemas, resource constraints, rendered pixels, hidden tests or saved environment state when possible. Transcript checks measure process. They do not prove file contents, API actions or rendered outputs. For open-ended judgments, define checkable dimensions, calibrate on human-labeled good/bad/borderline outputs, blind model identity and candidate order, allow ties, and preserve disagreement. A judge must treat all candidate text as untrusted data.
|
|
9
|
+
5. Choose grading that measures the outcome. Use exact labels, schemas, resource constraints, rendered pixels, hidden tests or saved environment state when possible. Transcript checks measure process. They do not prove file contents, API actions or rendered outputs. For open-ended judgments, define checkable dimensions, calibrate on human-labeled good/bad/borderline outputs, blind model identity and candidate order, allow ties, and preserve disagreement. A judge must treat all candidate text as untrusted data. A grader never calls the production code that made the output. Use an independent check. A stretch grader that reused the production corridor function passed every key while the designer saw corner smear.
|
|
10
10
|
6. Prove the grader catches the intended mistakes. Score known-good, known-bad and borderline controls. Deliberately break an important output or end state and show the expected failure. Report these as harness validation with zero task-model performance claims.
|
|
11
11
|
7. Prepare a bounded fresh smoke. State call count, token/cost estimate where available and total runtime cap. Use existing authorized credentials. Run in an isolated disposable workspace, enforce tool/side-effect boundaries, preserve traces and stop on infrastructure failure. Do not retry indefinitely or start a service to make the evaluation work.
|
|
12
12
|
8. Record each completed attempt immediately. Include case/group/split, repeat, dataset and prompt hashes, source revision, model and effort, dependency versions, raw output, trace, latency, usage and cost if available. Preserve one authoritative output per attempt. Keep infrastructure, parsing, grader and task failures separate. A zero-exit launcher alone is not evidence of completed inference.
|
|
13
|
-
9. Inspect pilot outputs beside grades. For a classification task, show confusion counts, precision, recall, specificity and false-positive rate. For tool workflows, score the isolated final state. Report task failures even when format adherence passes. Include missing/failed attempts; do not compare variants on unmatched survivors.
|
|
14
|
-
10. Freeze the baseline. Run candidates on matched inputs and settings, measure repeated-run variation and keep held-out cases out of tuning. Review cases and grading with the owner before using scores as a release gate. When a held-out set repeatedly influences changes, retire it into development data and create a new reviewed holdout.
|
|
13
|
+
9. Inspect pilot outputs beside grades. For a classification task, show confusion counts, precision, recall, specificity and false-positive rate. For tool workflows, score the isolated final state. Report task failures even when format adherence passes. Include missing/failed attempts; do not compare variants on unmatched survivors. A metric with the same value in every arm or cell is a grader fault. Fix it before you read any comparison.
|
|
14
|
+
10. Freeze the baseline. Run candidates on matched inputs and settings, measure repeated-run variation and keep held-out cases out of tuning. A grader change re-scores every candidate, the shipped one included, before a ship decision. Review cases and grading with the owner before using scores as a release gate. When a held-out set repeatedly influences changes, retire it into development data and create a new reviewed holdout.
|
|
15
15
|
|
|
16
|
-
Deliver the readable baseline report, cases, grading contract, one reproducible command and the next coverage gap. Name the execution tier in every headline result. A two-case smoke validates integration; it cannot establish a broad improvement.
|
|
16
|
+
Deliver the readable baseline report, cases, grading contract, one reproducible command and the next coverage gap. Name the execution tier in every headline result. The headline gives passed over graded, overall and per cohort, before any narrative. A two-case smoke validates integration; it cannot establish a broad improvement.
|
|
17
17
|
|
|
18
18
|
## Direct and plugin execution
|
|
19
19
|
|
|
@@ -24,7 +24,7 @@ User-level rule: applies to **every** git repo that uses GitHub with `gh`. Small
|
|
|
24
24
|
|
|
25
25
|
**Prove the change where it runs before opening a PR.** Read `~/.claude/docs/rule-guides/proof-before-pull-request.md` for the body and hook requirements. A `feat` PR also needs an "Eval" section and a session that invoked `/claude-api build-eval`; read `~/.claude/docs/rule-guides/features-start-with-an-eval.md` before you build the feature.
|
|
26
26
|
|
|
27
|
-
**Open PRs ready for review; use `--draft` when the owner requests it.**
|
|
27
|
+
**Open PRs ready for review; use `--draft` when the owner requests it.** Before you commit copied files or open a PR, search open PRs for the same paths with `gh pr list --search "<path>"`, so you do not duplicate a peer's PR. Each pre-existing failure or remaining defect the PR body names links a tracked issue.
|
|
28
28
|
|
|
29
29
|
**A release bot's PR body is machine input. Leave it alone.** Release automation reads
|
|
30
30
|
back the body of its own merged pull request to decide it owns that merge. Rewriting the
|
|
@@ -78,7 +78,7 @@ errors, so it neither clears nor convicts a change.
|
|
|
78
78
|
|
|
79
79
|
A checks listing that reports nothing on the branch is a finding.
|
|
80
80
|
Find out whether the workflow's event filters exclude the branch, or whether
|
|
81
|
-
the check simply never ran, before you treat that branch as clean.
|
|
81
|
+
the check simply never ran, before you treat that branch as clean. `gh pr checks` exits 8 while a check is still pending. Exit 8 reports a pending state, so wait and read the checks again after they report.
|
|
82
82
|
|
|
83
83
|
### Each stack level stands on its own
|
|
84
84
|
|
|
@@ -458,7 +458,7 @@ slurped array and take the last element. For one page, cap the request with a
|
|
|
458
458
|
|
|
459
459
|
Stage the files you edited this session right before you commit them. A plain `git commit` records only the staged snapshot; a tracked file this session changed but left unstaged stays behind in the working tree.
|
|
460
460
|
|
|
461
|
-
No hook denies a commit that would drop tracked session edits. Run `git status` before you commit, then stage what you changed with `git add <paths>` or commit with `git commit -a`.
|
|
461
|
+
No hook denies a commit that would drop tracked session edits. Run `git status` before you commit, then stage what you changed with `git add <paths>` or commit with `git commit -a`. Before you say nothing is left to commit, run `git status` in every worktree the session wrote to, as `git worktree list` lists them.
|
|
462
462
|
|
|
463
463
|
Staging covers tracked files you edited. Do not commit untracked files unless the user explicitly instructs it. An untracked file in the working tree is outside the change until they say otherwise.
|
|
464
464
|
|
|
@@ -81,9 +81,10 @@ draft. Record the target before any write.
|
|
|
81
81
|
|
|
82
82
|
### 2. Author the title and body
|
|
83
83
|
|
|
84
|
-
For create
|
|
85
|
-
|
|
86
|
-
|
|
84
|
+
For create, run `pull_request.py template --body-file <path> --title <title>`
|
|
85
|
+
first. It appends each heading the linter and proof gate require that the file
|
|
86
|
+
lacks. Then fill the body, or invoke `pstack:poteto-agent` with the
|
|
87
|
+
diff-specific writing instructions to draft it. A comment or
|
|
87
88
|
review may use a supplied body file when it does not rewrite the pull request
|
|
88
89
|
description.
|
|
89
90
|
|
|
@@ -6,6 +6,20 @@ ACTION_COMMENT = "comment"
|
|
|
6
6
|
ACTION_CREATE = "create"
|
|
7
7
|
ACTION_EDIT = "edit"
|
|
8
8
|
ACTION_REVIEW = "review"
|
|
9
|
+
ACTION_TEMPLATE = "template"
|
|
10
|
+
|
|
11
|
+
ALL_BODY_TEMPLATE_HEADINGS: tuple[str, ...] = (
|
|
12
|
+
"Why",
|
|
13
|
+
"Existing work",
|
|
14
|
+
"Proof in practice",
|
|
15
|
+
"Verification",
|
|
16
|
+
)
|
|
17
|
+
FEATURE_BODY_TEMPLATE_HEADING = "Eval"
|
|
18
|
+
FEATURE_TITLE_PATTERN: re.Pattern[str] = re.compile(r"^\s*feat(\([^)]*\))?!?:")
|
|
19
|
+
BODY_TEMPLATE_HEADING_FORMAT = "## {heading}"
|
|
20
|
+
BODY_TEMPLATE_ENCODING = "utf-8"
|
|
21
|
+
BODY_TEMPLATE_LINE_END = "\n"
|
|
22
|
+
BODY_TEMPLATE_BLOCK_SEPARATOR = "\n\n"
|
|
9
23
|
|
|
10
24
|
LINTER_ACTION_COMMENT = "pr-comment"
|
|
11
25
|
LINTER_ACTION_CREATE = "pr-create"
|
|
@@ -18,7 +18,15 @@ from github_pr_command_constants.config.constants import (
|
|
|
18
18
|
ACTION_EDIT,
|
|
19
19
|
ACTION_FAILED_MESSAGE,
|
|
20
20
|
ACTION_REVIEW,
|
|
21
|
+
ACTION_TEMPLATE,
|
|
22
|
+
ALL_BODY_TEMPLATE_HEADINGS,
|
|
21
23
|
ALL_DURABLE_POST_LINTER_RELATIVE_PATH_PARTS,
|
|
24
|
+
BODY_TEMPLATE_BLOCK_SEPARATOR,
|
|
25
|
+
BODY_TEMPLATE_ENCODING,
|
|
26
|
+
BODY_TEMPLATE_HEADING_FORMAT,
|
|
27
|
+
BODY_TEMPLATE_LINE_END,
|
|
28
|
+
FEATURE_BODY_TEMPLATE_HEADING,
|
|
29
|
+
FEATURE_TITLE_PATTERN,
|
|
22
30
|
ALL_GITHUB_AUTHORIZATION_ENVIRONMENT_KEYS,
|
|
23
31
|
ALL_LINTER_ACTIONS_BY_COMMAND,
|
|
24
32
|
ALL_REVIEW_FLAGS_BY_EVENT,
|
|
@@ -62,9 +70,53 @@ def _build_parser() -> argparse.ArgumentParser:
|
|
|
62
70
|
choices=tuple(ALL_REVIEW_FLAGS_BY_EVENT),
|
|
63
71
|
default=REVIEW_EVENT_COMMENT,
|
|
64
72
|
)
|
|
73
|
+
template = all_actions.add_parser(ACTION_TEMPLATE, allow_abbrev=False)
|
|
74
|
+
template.add_argument("--body-file", required=True, type=Path)
|
|
75
|
+
template.add_argument("--title", default="")
|
|
65
76
|
return parser
|
|
66
77
|
|
|
67
78
|
|
|
79
|
+
def _template_headings(title: str) -> list[str]:
|
|
80
|
+
all_headings = [
|
|
81
|
+
BODY_TEMPLATE_HEADING_FORMAT.format(heading=each_heading)
|
|
82
|
+
for each_heading in ALL_BODY_TEMPLATE_HEADINGS
|
|
83
|
+
]
|
|
84
|
+
if FEATURE_TITLE_PATTERN.match(title):
|
|
85
|
+
all_headings.append(BODY_TEMPLATE_HEADING_FORMAT.format(heading=FEATURE_BODY_TEMPLATE_HEADING))
|
|
86
|
+
return all_headings
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
def write_body_template(body_file: Path, title: str) -> list[str]:
|
|
90
|
+
"""Append each required heading the body file lacks and return the added headings.
|
|
91
|
+
|
|
92
|
+
::
|
|
93
|
+
|
|
94
|
+
new file, title "fix: x" -> ["## Why", "## Existing work", "## Proof in practice", "## Verification"]
|
|
95
|
+
new file, title "feat: x" -> the same plus "## Eval"
|
|
96
|
+
file already has them all -> [] and the file is unchanged
|
|
97
|
+
|
|
98
|
+
Args:
|
|
99
|
+
body_file: The pull request body file, created when absent.
|
|
100
|
+
title: The pull request title, which decides whether the Eval heading is required.
|
|
101
|
+
|
|
102
|
+
Returns:
|
|
103
|
+
The heading lines appended to the file.
|
|
104
|
+
"""
|
|
105
|
+
body_text = body_file.read_text(encoding=BODY_TEMPLATE_ENCODING) if body_file.is_file() else ""
|
|
106
|
+
all_present_lines = {each_line.strip() for each_line in body_text.splitlines()}
|
|
107
|
+
all_missing_headings = [
|
|
108
|
+
each_heading for each_heading in _template_headings(title) if each_heading not in all_present_lines
|
|
109
|
+
]
|
|
110
|
+
if not all_missing_headings:
|
|
111
|
+
return []
|
|
112
|
+
all_body_blocks = [body_text.rstrip(), *all_missing_headings] if body_text.strip() else all_missing_headings
|
|
113
|
+
body_file.write_text(
|
|
114
|
+
BODY_TEMPLATE_BLOCK_SEPARATOR.join(all_body_blocks) + BODY_TEMPLATE_LINE_END,
|
|
115
|
+
encoding=BODY_TEMPLATE_ENCODING,
|
|
116
|
+
)
|
|
117
|
+
return all_missing_headings
|
|
118
|
+
|
|
119
|
+
|
|
68
120
|
def _configured_managed_root(all_environment: Mapping[str, str]) -> Path:
|
|
69
121
|
configured_root_text = all_environment.get(CONFIG_DIR_OVERRIDE_ENVIRONMENT_KEY, "").strip()
|
|
70
122
|
if not configured_root_text:
|
|
@@ -253,6 +305,10 @@ def main(
|
|
|
253
305
|
) -> int:
|
|
254
306
|
"""Run one validated pull request action."""
|
|
255
307
|
arguments = _build_parser().parse_args(list(all_arguments))
|
|
308
|
+
if arguments.action == ACTION_TEMPLATE:
|
|
309
|
+
for each_heading in write_body_template(arguments.body_file, arguments.title):
|
|
310
|
+
sys.stdout.write(each_heading + "\n")
|
|
311
|
+
return 0
|
|
256
312
|
completed_lint = command_runner(
|
|
257
313
|
_linter_arguments(arguments, all_environment), check=False, shell=False
|
|
258
314
|
)
|
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
from __future__ import annotations
|
|
4
4
|
|
|
5
5
|
import os
|
|
6
|
+
import re
|
|
6
7
|
import subprocess
|
|
7
8
|
import sys
|
|
8
9
|
import threading
|
|
@@ -462,3 +463,84 @@ def should_hand_the_head_branch_to_the_linter_on_create(body_file: Path) -> None
|
|
|
462
463
|
linter_command[linter_command.index("--head-branch") + 1]
|
|
463
464
|
== "release-please--branches--main--components--package"
|
|
464
465
|
)
|
|
466
|
+
|
|
467
|
+
|
|
468
|
+
PROOF_CONSTANTS_PATH = (
|
|
469
|
+
_SCRIPTS_DIRECTORY.parents[3] / "hooks/hooks_constants/pull_request_proof_constants.py"
|
|
470
|
+
)
|
|
471
|
+
|
|
472
|
+
|
|
473
|
+
def _proof_heading_patterns() -> dict[str, str]:
|
|
474
|
+
constants_text = PROOF_CONSTANTS_PATH.read_text(encoding="utf-8")
|
|
475
|
+
all_names = ("PROOF_HEADING_PATTERN", "EXISTING_WORK_HEADING_PATTERN", "EVAL_HEADING_PATTERN")
|
|
476
|
+
all_patterns: dict[str, str] = {}
|
|
477
|
+
for each_line in constants_text.splitlines():
|
|
478
|
+
each_name, _, each_value = each_line.partition(" = ")
|
|
479
|
+
if each_name in all_names:
|
|
480
|
+
all_patterns[each_name] = each_value.strip()[2:-1]
|
|
481
|
+
return all_patterns
|
|
482
|
+
|
|
483
|
+
|
|
484
|
+
def _run_template(body_path: Path, title: str) -> tuple[int, Mock]:
|
|
485
|
+
command_runner = Mock()
|
|
486
|
+
exit_code = pull_request.main(
|
|
487
|
+
["template", "--body-file", str(body_path), "--title", title],
|
|
488
|
+
all_environment={},
|
|
489
|
+
command_runner=command_runner,
|
|
490
|
+
)
|
|
491
|
+
return exit_code, command_runner
|
|
492
|
+
|
|
493
|
+
|
|
494
|
+
def _body_has_heading(body_text: str, heading_pattern: str) -> bool:
|
|
495
|
+
return any(
|
|
496
|
+
re.match(heading_pattern, each_line.strip(), re.IGNORECASE)
|
|
497
|
+
for each_line in body_text.splitlines()
|
|
498
|
+
)
|
|
499
|
+
|
|
500
|
+
|
|
501
|
+
def should_write_a_template_that_clears_the_linter_and_proof_headings(tmp_path: Path) -> None:
|
|
502
|
+
body_path = tmp_path / BODY_FILENAME
|
|
503
|
+
exit_code, command_runner = _run_template(body_path, PR_TITLE)
|
|
504
|
+
body_text = body_path.read_text(encoding="utf-8")
|
|
505
|
+
lint = subprocess.run(
|
|
506
|
+
[
|
|
507
|
+
sys.executable,
|
|
508
|
+
str(LINTER_PATH),
|
|
509
|
+
"--action",
|
|
510
|
+
"pr-create",
|
|
511
|
+
"--title",
|
|
512
|
+
PR_TITLE,
|
|
513
|
+
"--body-file",
|
|
514
|
+
str(body_path),
|
|
515
|
+
"--repository",
|
|
516
|
+
REPOSITORY,
|
|
517
|
+
],
|
|
518
|
+
capture_output=True,
|
|
519
|
+
check=False,
|
|
520
|
+
text=True,
|
|
521
|
+
)
|
|
522
|
+
assert exit_code == 0
|
|
523
|
+
assert command_runner.call_count == 0
|
|
524
|
+
assert "missing-pr-description-heading" not in lint.stdout + lint.stderr
|
|
525
|
+
assert all(
|
|
526
|
+
_body_has_heading(body_text, each_pattern)
|
|
527
|
+
for each_pattern in _proof_heading_patterns().values()
|
|
528
|
+
)
|
|
529
|
+
|
|
530
|
+
|
|
531
|
+
def should_add_the_eval_heading_only_for_a_feature_title(tmp_path: Path) -> None:
|
|
532
|
+
fix_body_path = tmp_path / "fix.md"
|
|
533
|
+
_run_template(fix_body_path, "fix(hooks): narrow the gate")
|
|
534
|
+
eval_pattern = _proof_heading_patterns()["EVAL_HEADING_PATTERN"]
|
|
535
|
+
assert not _body_has_heading(fix_body_path.read_text(encoding="utf-8"), eval_pattern)
|
|
536
|
+
|
|
537
|
+
|
|
538
|
+
def should_keep_written_text_and_add_only_missing_headings(tmp_path: Path) -> None:
|
|
539
|
+
body_path = tmp_path / BODY_FILENAME
|
|
540
|
+
body_path.write_text("## Why\nThe gate bounced twice.\n", encoding="utf-8")
|
|
541
|
+
_run_template(body_path, PR_TITLE)
|
|
542
|
+
first_text = body_path.read_text(encoding="utf-8")
|
|
543
|
+
_run_template(body_path, PR_TITLE)
|
|
544
|
+
assert first_text.startswith("## Why\nThe gate bounced twice.\n")
|
|
545
|
+
assert first_text.count("## Why") == 1
|
|
546
|
+
assert body_path.read_text(encoding="utf-8") == first_text
|
|
@@ -31,4 +31,4 @@ The name has 25 characters or fewer and names the main change or outcome in conc
|
|
|
31
31
|
|
|
32
32
|
## Enforcement
|
|
33
33
|
|
|
34
|
-
`hooks/blocking/session_title_format_gate.py` denies a malformed title. `hooks/blocking/session_title_turn_gate.py` is a Stop hook. It holds the end of a turn until a title call in that turn comes back without an error. It applies when the transcript shows the title tool. A held turn whose title call failed may end, so a failing title tool cannot hold it forever.
|
|
34
|
+
`hooks/blocking/session_title_format_gate.py` denies a malformed title. `hooks/blocking/session_title_turn_gate.py` is a Stop hook. It holds the end of a turn until a title call in that turn comes back without an error. A turn that called no tool may end when an earlier turn already set the title. It applies when the transcript shows the title tool. A held turn whose title call failed may end, so a failing title tool cannot hold it forever.
|
|
@@ -7,7 +7,9 @@ a Stop hook block lands as an attachment, so a held turn stays the same turn.
|
|
|
7
7
|
A title call on a sidechain entry belongs to a subagent and does not count.
|
|
8
8
|
The turn may end once a tool whose name ends in __set_session_title ran in it
|
|
9
9
|
and its result came back without an error. A title the format gate denied
|
|
10
|
-
does not count. A
|
|
10
|
+
does not count. A turn that called no tool may also end once an earlier
|
|
11
|
+
main-chain turn set the title, since that title still shows and a text-only
|
|
12
|
+
turn starts no new task. A held turn (``stop_hook_active`` true) whose title call
|
|
11
13
|
failed may end, so a title tool that keeps failing cannot hold it forever. A
|
|
12
14
|
held turn with no title call stays held, whichever Stop gate held it.
|
|
13
15
|
|
|
@@ -153,8 +155,8 @@ def _apply_tool_listing(
|
|
|
153
155
|
all_available_names.pop(each_removed_name, None)
|
|
154
156
|
|
|
155
157
|
|
|
156
|
-
def
|
|
157
|
-
|
|
158
|
+
def _last_prompt_index(all_entries: list[dict[str, object]]) -> int:
|
|
159
|
+
return max(
|
|
158
160
|
(
|
|
159
161
|
each_index
|
|
160
162
|
for each_index, each_entry in enumerate(all_entries)
|
|
@@ -162,11 +164,18 @@ def _turn_entries(all_entries: list[dict[str, object]]) -> list[dict[str, object
|
|
|
162
164
|
),
|
|
163
165
|
default=-1,
|
|
164
166
|
)
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
167
|
+
|
|
168
|
+
|
|
169
|
+
def _main_chain(all_entries: list[dict[str, object]]) -> list[dict[str, object]]:
|
|
170
|
+
return [each_entry for each_entry in all_entries if not each_entry.get(SIDECHAIN_FLAG_KEY)]
|
|
171
|
+
|
|
172
|
+
|
|
173
|
+
def _turn_entries(all_entries: list[dict[str, object]]) -> list[dict[str, object]]:
|
|
174
|
+
return _main_chain(all_entries[_last_prompt_index(all_entries) + 1 :])
|
|
175
|
+
|
|
176
|
+
|
|
177
|
+
def _earlier_entries(all_entries: list[dict[str, object]]) -> list[dict[str, object]]:
|
|
178
|
+
return _main_chain(all_entries[: max(_last_prompt_index(all_entries), 0)])
|
|
170
179
|
|
|
171
180
|
|
|
172
181
|
def _title_call_ids(all_turn_entries: list[dict[str, object]]) -> set[object]:
|
|
@@ -205,13 +214,44 @@ def title_set_this_turn(all_entries: list[dict[str, object]]) -> bool:
|
|
|
205
214
|
Returns:
|
|
206
215
|
True when this turn set the title.
|
|
207
216
|
"""
|
|
208
|
-
|
|
209
|
-
|
|
217
|
+
return _title_succeeded_in(_turn_entries(all_entries))
|
|
218
|
+
|
|
219
|
+
|
|
220
|
+
def title_set_before_this_turn(all_entries: list[dict[str, object]]) -> bool:
|
|
221
|
+
"""Return True when a main-chain title call before the latest prompt came back without an error.
|
|
222
|
+
|
|
223
|
+
Args:
|
|
224
|
+
all_entries: The transcript entries, oldest first.
|
|
225
|
+
|
|
226
|
+
Returns:
|
|
227
|
+
True when an earlier turn set the title the session still shows.
|
|
228
|
+
"""
|
|
229
|
+
return _title_succeeded_in(_earlier_entries(all_entries))
|
|
230
|
+
|
|
231
|
+
|
|
232
|
+
def turn_ran_a_tool(all_entries: list[dict[str, object]]) -> bool:
|
|
233
|
+
"""Return True when this turn called any tool other than the title tool.
|
|
234
|
+
|
|
235
|
+
Args:
|
|
236
|
+
all_entries: The transcript entries, oldest first.
|
|
237
|
+
|
|
238
|
+
Returns:
|
|
239
|
+
True when the turn did work beyond answering in text.
|
|
240
|
+
"""
|
|
241
|
+
return any(
|
|
242
|
+
not TITLE_TOOL_NAME_PATTERN.match(str(each_block.get(BLOCK_NAME_KEY)))
|
|
243
|
+
for each_entry in _turn_entries(all_entries)
|
|
244
|
+
for each_block in _tool_use_blocks(each_entry)
|
|
245
|
+
)
|
|
246
|
+
|
|
247
|
+
|
|
248
|
+
def _title_succeeded_in(all_scoped_entries: list[dict[str, object]]) -> bool:
|
|
249
|
+
all_title_call_ids = _title_call_ids(all_scoped_entries)
|
|
210
250
|
return any(
|
|
211
251
|
each_block.get(ENTRY_TYPE_KEY) == TOOL_RESULT_BLOCK_TYPE
|
|
212
252
|
and each_block.get(TOOL_USE_ID_KEY) in all_title_call_ids
|
|
213
253
|
and not each_block.get(IS_ERROR_KEY)
|
|
214
|
-
for each_entry in
|
|
254
|
+
for each_entry in all_scoped_entries
|
|
215
255
|
if each_entry.get(ENTRY_TYPE_KEY) == USER_ENTRY_TYPE
|
|
216
256
|
for each_block in _content_blocks(each_entry)
|
|
217
257
|
)
|
|
@@ -226,6 +266,7 @@ def turn_block_reason(
|
|
|
226
266
|
|
|
227
267
|
no title tool in the transcript -> None
|
|
228
268
|
title set this turn -> None
|
|
269
|
+
tool-free turn after an earlier turn set the title -> None
|
|
229
270
|
held turn whose title call failed (stop_hook_active) -> None
|
|
230
271
|
no title call this turn -> reason
|
|
231
272
|
|
|
@@ -239,6 +280,8 @@ def turn_block_reason(
|
|
|
239
280
|
all_title_tools = available_title_tools(all_entries)
|
|
240
281
|
if not all_title_tools or title_set_this_turn(all_entries):
|
|
241
282
|
return None
|
|
283
|
+
if title_set_before_this_turn(all_entries) and not turn_ran_a_tool(all_entries):
|
|
284
|
+
return None
|
|
242
285
|
if is_stop_hook_active and title_attempted_this_turn(all_entries):
|
|
243
286
|
return None
|
|
244
287
|
return TURN_BLOCK_REASON.format(tool_name=TITLE_TOOL_DISPLAY_SEPARATOR.join(all_title_tools))
|
|
@@ -167,6 +167,53 @@ def test_should_hold_when_the_title_was_set_only_in_an_earlier_turn(
|
|
|
167
167
|
assert block_output["decision"] == "block"
|
|
168
168
|
|
|
169
169
|
|
|
170
|
+
@pytest.mark.parametrize(
|
|
171
|
+
"all_turn_tail",
|
|
172
|
+
[
|
|
173
|
+
[REPLY],
|
|
174
|
+
[title_call("t2"), title_result("t2", is_error=True), REPLY],
|
|
175
|
+
],
|
|
176
|
+
)
|
|
177
|
+
def test_should_pass_a_tool_free_turn_after_an_earlier_title(
|
|
178
|
+
monkeypatch: pytest.MonkeyPatch,
|
|
179
|
+
capsys: pytest.CaptureFixture[str],
|
|
180
|
+
tmp_path: Path,
|
|
181
|
+
all_turn_tail: list[dict[str, object]],
|
|
182
|
+
) -> None:
|
|
183
|
+
all_entries = [
|
|
184
|
+
TOOL_LISTING,
|
|
185
|
+
PROMPT,
|
|
186
|
+
title_call("t1"),
|
|
187
|
+
title_result("t1"),
|
|
188
|
+
REPLY,
|
|
189
|
+
PROMPT,
|
|
190
|
+
*all_turn_tail,
|
|
191
|
+
]
|
|
192
|
+
assert run_gate(monkeypatch, capsys, tmp_path, all_entries) == (0, "")
|
|
193
|
+
|
|
194
|
+
|
|
195
|
+
@pytest.mark.parametrize(
|
|
196
|
+
"all_earlier_title_entries",
|
|
197
|
+
[
|
|
198
|
+
[],
|
|
199
|
+
[title_call("t1"), title_result("t1", is_error=True)],
|
|
200
|
+
[
|
|
201
|
+
{**title_call("s1"), "isSidechain": True},
|
|
202
|
+
{**title_result("s1"), "isSidechain": True},
|
|
203
|
+
],
|
|
204
|
+
],
|
|
205
|
+
)
|
|
206
|
+
def test_should_hold_a_tool_free_turn_without_an_earlier_title(
|
|
207
|
+
monkeypatch: pytest.MonkeyPatch,
|
|
208
|
+
capsys: pytest.CaptureFixture[str],
|
|
209
|
+
tmp_path: Path,
|
|
210
|
+
all_earlier_title_entries: list[dict[str, object]],
|
|
211
|
+
) -> None:
|
|
212
|
+
all_entries = [TOOL_LISTING, PROMPT, *all_earlier_title_entries, REPLY, PROMPT, REPLY]
|
|
213
|
+
stdout_text = run_gate(monkeypatch, capsys, tmp_path, all_entries)[1]
|
|
214
|
+
assert json.loads(stdout_text)["decision"] == "block"
|
|
215
|
+
|
|
216
|
+
|
|
170
217
|
def test_should_start_a_turn_at_an_image_only_prompt(
|
|
171
218
|
monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], tmp_path: Path
|
|
172
219
|
) -> None:
|
|
@@ -181,6 +228,8 @@ def test_should_start_a_turn_at_an_image_only_prompt(
|
|
|
181
228
|
title_result("t1"),
|
|
182
229
|
REPLY,
|
|
183
230
|
image_prompt,
|
|
231
|
+
WORK_CALL,
|
|
232
|
+
WORK_RESULT,
|
|
184
233
|
REPLY,
|
|
185
234
|
]
|
|
186
235
|
stdout_text = run_gate(monkeypatch, capsys, tmp_path, all_entries)[1]
|
|
@@ -12,7 +12,7 @@ This family stops a chat reply, edited message, or tool call when its payload br
|
|
|
12
12
|
- `blocking/edit_marker_gate.py` rejects strikethrough and edit-note markers in replacement chat text or cards.
|
|
13
13
|
- `blocking/issue_close_handoff_gate.py` denies an issue comment or issue write that closes the issue and routes a found defect to another issue or epic.
|
|
14
14
|
- `blocking/session_title_format_gate.py` denies a session title that breaks the `<emoji> <name>` shape or has a name longer than 25 characters.
|
|
15
|
-
- `blocking/session_title_turn_gate.py` holds the end of a turn in a session that has a `__set_session_title` tool until a title call in that turn comes back without an error. A held turn whose title call failed may end.
|
|
15
|
+
- `blocking/session_title_turn_gate.py` holds the end of a turn in a session that has a `__set_session_title` tool until a title call in that turn comes back without an error. A turn that called no tool may end when an earlier turn already set the title. A held turn whose title call failed may end.
|
|
16
16
|
- `blocking/step_note_gate.py` requires a status line before a tool call while its opt-in flag is on.
|
|
17
17
|
- `blocking/verify_before_acting.py` blocks a completed mutating call when the agent's reasoning contains an unchecked claim.
|
|
18
18
|
- `blocking/pr_lifecycle_skill_gate.py` denies a commit, push, pull request action, or merge until the `pr-lifecycle` skill appears in the transcript since the last compaction.
|