@torus-engineering/tas-kit 1.5.1 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/README.md +83 -0
- package/.claude/agents/architect.md +53 -0
- package/.claude/agents/aws-reviewer.md +71 -0
- package/.claude/agents/build-resolver.md +59 -0
- package/.claude/agents/code-architect.md +62 -0
- package/.claude/agents/code-explorer.md +63 -0
- package/.claude/agents/code-simplifier.md +53 -0
- package/.claude/agents/comment-analyzer.md +59 -0
- package/.claude/agents/conversation-analyzer.md +57 -0
- package/.claude/agents/csharp-reviewer.md +62 -0
- package/.claude/agents/database-reviewer.md +73 -0
- package/.claude/agents/doc-updater.md +66 -0
- package/.claude/agents/docs-lookup.md +55 -0
- package/.claude/agents/e2e-runner.md +61 -0
- package/.claude/agents/harness-optimizer.md +62 -0
- package/.claude/agents/loop-operator.md +56 -0
- package/.claude/agents/performance-optimizer.md +78 -0
- package/.claude/agents/planner.md +82 -0
- package/.claude/agents/pr-test-analyzer.md +68 -0
- package/.claude/agents/python-reviewer.md +67 -0
- package/.claude/agents/pytorch-build-resolver.md +76 -0
- package/.claude/agents/refactor-cleaner.md +70 -0
- package/.claude/agents/security-reviewer.md +79 -0
- package/.claude/agents/seo-specialist.md +75 -0
- package/.claude/agents/silent-failure-hunter.md +69 -0
- package/.claude/agents/tdd-guide.md +84 -0
- package/.claude/agents/type-design-analyzer.md +75 -0
- package/.claude/agents/typescript-reviewer.md +65 -0
- package/.claude/commands/ado-create.md +2 -1
- package/.claude/commands/ado-delete.md +3 -2
- package/.claude/commands/ado-get.md +2 -1
- package/.claude/commands/ado-status.md +2 -1
- package/.claude/commands/ado-update.md +2 -1
- package/.claude/commands/tas-adr.md +13 -12
- package/.claude/commands/tas-bug.md +97 -50
- package/.claude/commands/tas-design.md +3 -1
- package/.claude/commands/tas-dev.md +115 -0
- package/.claude/commands/tas-epic.md +4 -2
- package/.claude/commands/tas-feature.md +5 -3
- package/.claude/commands/tas-fix.md +47 -0
- package/.claude/commands/tas-plan.md +184 -0
- package/.claude/commands/tas-prd.md +3 -1
- package/.claude/commands/tas-review.md +104 -0
- package/.claude/commands/tas-sad.md +3 -1
- package/.claude/commands/tas-security.md +80 -0
- package/.claude/commands/tas-spec.md +50 -0
- package/.claude/commands/tas-story.md +77 -40
- package/.claude/commands/tas-verify.md +8 -0
- package/.claude/hooks/code-quality.js +127 -0
- package/.claude/hooks/session-end.js +116 -0
- package/.claude/rules/.gitkeep +0 -0
- package/.claude/rules/common/agents.md +65 -0
- package/.claude/rules/common/code-review.md +124 -0
- package/.claude/rules/common/coding-style.md +90 -0
- package/.claude/rules/common/development-workflow.md +44 -0
- package/.claude/rules/common/git-workflow.md +24 -0
- package/.claude/rules/common/hooks.md +30 -0
- package/.claude/rules/common/patterns.md +31 -0
- package/.claude/rules/common/performance.md +55 -0
- package/.claude/rules/common/post-review-agent.md +39 -0
- package/.claude/rules/common/project-status.md +80 -0
- package/.claude/rules/common/security.md +29 -0
- package/.claude/rules/common/stack-detection.md +29 -0
- package/.claude/rules/common/testing.md +57 -0
- package/.claude/rules/csharp/coding-style.md +72 -0
- package/.claude/rules/csharp/hooks.md +25 -0
- package/.claude/rules/csharp/patterns.md +50 -0
- package/.claude/rules/csharp/security.md +58 -0
- package/.claude/rules/csharp/testing.md +46 -0
- package/.claude/rules/python/coding-style.md +42 -0
- package/.claude/rules/python/hooks.md +19 -0
- package/.claude/rules/python/patterns.md +39 -0
- package/.claude/rules/python/security.md +30 -0
- package/.claude/rules/python/testing.md +38 -0
- package/.claude/rules/typescript/coding-style.md +199 -0
- package/.claude/rules/typescript/hooks.md +22 -0
- package/.claude/rules/typescript/patterns.md +52 -0
- package/.claude/rules/typescript/security.md +28 -0
- package/.claude/rules/typescript/testing.md +18 -0
- package/.claude/rules/web/coding-style.md +96 -0
- package/.claude/rules/web/design-quality.md +63 -0
- package/.claude/rules/web/hooks.md +120 -0
- package/.claude/rules/web/patterns.md +79 -0
- package/.claude/rules/web/performance.md +64 -0
- package/.claude/rules/web/security.md +57 -0
- package/.claude/rules/web/testing.md +55 -0
- package/.claude/settings.json +37 -0
- package/.claude/settings.local.json +38 -0
- package/.claude/skills/ado-integration/SKILL.md +44 -1
- package/.claude/skills/agent-harness-construction/SKILL.md +77 -0
- package/.claude/skills/agent-introspection-debugging/SKILL.md +157 -0
- package/.claude/skills/ai-regression-testing/SKILL.md +364 -0
- package/.claude/skills/api-design/SKILL.md +528 -0
- package/.claude/skills/architecture-decision-records/SKILL.md +184 -0
- package/.claude/skills/backend-patterns/SKILL.md +602 -0
- package/.claude/skills/benchmark/SKILL.md +98 -0
- package/.claude/skills/browser-qa/SKILL.md +92 -0
- package/.claude/skills/canary-watch/SKILL.md +104 -0
- package/.claude/skills/tas-conventions/SKILL.md +51 -3
- package/.claude/skills/tas-implementation-complete/SKILL.md +97 -0
- package/.claude/skills/tas-tdd/SKILL.md +72 -16
- package/.tas/README.md +29 -24
- package/.tas/tas-example.yaml +2 -1
- package/.tas/templates/Story.md +18 -18
- package/CLAUDE-Example.md +1 -1
- package/README.md +20 -5
- package/package.json +1 -1
- package/.claude/commands/tas-dev-story.md +0 -61
- package/.claude/commands/tas-review-code.md +0 -42
- package/.claude/commands/tas-security-check.md +0 -30
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
{
|
|
2
|
+
"permissions": {
|
|
3
|
+
"allow": [
|
|
4
|
+
"Skill(*)",
|
|
5
|
+
"Bash(*)",
|
|
6
|
+
"Edit(*)",
|
|
7
|
+
"Write(*)",
|
|
8
|
+
"Read(*)",
|
|
9
|
+
"Glob(*)",
|
|
10
|
+
"Grep(*)",
|
|
11
|
+
"Agent(*)",
|
|
12
|
+
"ExitPlanMode",
|
|
13
|
+
"EnterWorktree",
|
|
14
|
+
"ExitWorktree",
|
|
15
|
+
"TodoWrite",
|
|
16
|
+
"NotebookEdit",
|
|
17
|
+
"CronCreate",
|
|
18
|
+
"CronDelete",
|
|
19
|
+
"CronList",
|
|
20
|
+
"RemoteTrigger(*)",
|
|
21
|
+
"TaskOutput",
|
|
22
|
+
"TaskStop",
|
|
23
|
+
"AskUserQuestion"
|
|
24
|
+
],
|
|
25
|
+
"deny": [
|
|
26
|
+
"Read:env:*",
|
|
27
|
+
"Bash:sudo:*",
|
|
28
|
+
"WebSearch",
|
|
29
|
+
"WebFetch",
|
|
30
|
+
"Skill(mcp__web_reader__webReader)",
|
|
31
|
+
"Skill(mcp__4_5v_mcp__analyze_image)"
|
|
32
|
+
]
|
|
33
|
+
},
|
|
34
|
+
"enabledMcpjsonServers": [
|
|
35
|
+
"figma"
|
|
36
|
+
],
|
|
37
|
+
"enableAllProjectMcpServers": true
|
|
38
|
+
}
|
|
@@ -10,13 +10,42 @@ allowed-tools: Read, Write, Edit, Bash, Grep
|
|
|
10
10
|
# ADO Integration Skill
|
|
11
11
|
|
|
12
12
|
Cho phép đồng bộ hai chiều giữa file .md trong repo và work items trên Azure DevOps.
|
|
13
|
+
ADO sync là **thao tác có chủ đích** — không tự động sau mỗi lần edit file.
|
|
14
|
+
|
|
15
|
+
## When to Use
|
|
16
|
+
|
|
17
|
+
- User yêu cầu sync, push, pull work item lên/từ ADO
|
|
18
|
+
- User chạy `/ado-create`, `/ado-update`, `/ado-status`, `/ado-get`, `/ado-delete`
|
|
19
|
+
- KHÔNG invoke khi: user chỉ edit file .md thông thường mà không nhắc đến ADO
|
|
20
|
+
|
|
21
|
+
## Always / Ask / Never
|
|
22
|
+
|
|
23
|
+
| | Hành động |
|
|
24
|
+
|---|---|
|
|
25
|
+
| **Always** | Đọc `tas.yaml` và kiểm tra `ado.enabled` trước bất kỳ thao tác nào |
|
|
26
|
+
| **Always** | Hiển thị ADO ID và URL sau mỗi create/update thành công |
|
|
27
|
+
| **Always** | Cập nhật frontmatter `ado_id`, `ado_state`, `last_ado_sync` trong file .md sau sync |
|
|
28
|
+
| **Ask** | Khi sync bulk nhiều items cùng lúc — confirm list trước khi chạy |
|
|
29
|
+
| **Ask** | Khi phát hiện conflict giữa file .md và ADO item (ai là source of truth?) |
|
|
30
|
+
| **Ask** | Khi delete work item — đây là thao tác không thể undo |
|
|
31
|
+
| **Never** | Auto-sync mỗi khi file .md được edit (quá aggressive, dễ gây noise) |
|
|
32
|
+
| **Never** | Xóa ADO item mà không có xác nhận rõ ràng từ user |
|
|
33
|
+
| **Never** | Tạo duplicate work item nếu đã có `ado_id` trong frontmatter |
|
|
34
|
+
|
|
35
|
+
## Bước đầu tiên — Kiểm tra ADO enabled
|
|
36
|
+
|
|
37
|
+
Trước khi thực hiện bất kỳ thao tác nào, đọc `tas.yaml` tại root và kiểm tra `ado.enabled`:
|
|
38
|
+
- Nếu `ado.enabled: false` hoặc field không tồn tại: thông báo "ADO integration bị tắt trong tas.yaml (`ado.enabled: false`). Bật lên nếu project dùng ADO." rồi dừng lại.
|
|
39
|
+
- Nếu `ado.enabled: true`: tiếp tục bình thường.
|
|
13
40
|
|
|
14
41
|
## Prerequisites
|
|
42
|
+
|
|
15
43
|
- Azure CLI + extension azure-devops: `az extension add --name azure-devops --upgrade`
|
|
16
44
|
- Python 3.8+ với pyyaml: `pip install pyyaml`
|
|
17
45
|
- PAT trong file .env: `AzureDevops_Personal_AccessToken=your-pat-here`
|
|
18
46
|
|
|
19
47
|
## Commands
|
|
48
|
+
|
|
20
49
|
Tất cả ADO commands chạy qua: `python .tas/tools/tas-ado.py <command> [args]`
|
|
21
50
|
|
|
22
51
|
Hoặc dùng slash commands:
|
|
@@ -27,6 +56,20 @@ Hoặc dùng slash commands:
|
|
|
27
56
|
- `/ado-delete <type> <ado-id>`
|
|
28
57
|
|
|
29
58
|
## File Convention
|
|
59
|
+
|
|
30
60
|
- Tên file: `{type}-{ado_id}-{slug-title}.md`
|
|
31
|
-
- Mỗi file có frontmatter YAML: ado_id
|
|
61
|
+
- Mỗi file có frontmatter YAML: `ado_id`, `ado_type`, `ado_state`, `last_ado_sync`
|
|
32
62
|
- File .md là single source of truth, sync lên ADO khi cần
|
|
63
|
+
|
|
64
|
+
## Red Flags
|
|
65
|
+
|
|
66
|
+
- File có `ado_id` nhưng state trong file khác ADO → confirm với user trước khi ghi đè
|
|
67
|
+
- PAT hết hạn → hướng dẫn rotate, không log token ra stdout
|
|
68
|
+
- `ado.enabled: true` nhưng project chưa setup Azure CLI → check prerequisites trước
|
|
69
|
+
|
|
70
|
+
## Anti-Rationalization
|
|
71
|
+
|
|
72
|
+
| Rationalization | Counter |
|
|
73
|
+
|---|---|
|
|
74
|
+
| "Auto-sync tiện hơn, không cần nhớ" | Hook auto-sync gây unintended pushes khi edit draft — sync phải có chủ đích |
|
|
75
|
+
| "Delete chắc OK, mình biết mình đang làm gì" | ADO delete không có undo — luôn confirm, dù user trông có vẻ chắc chắn |
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agent-harness-construction
|
|
3
|
+
description: |
|
|
4
|
+
Auto-invoke when designing or optimizing agent tool definitions, action spaces,
|
|
5
|
+
or observation formats. Also when an agent is looping on tools without progress,
|
|
6
|
+
failing to converge, or producing poor-quality completions due to harness design.
|
|
7
|
+
origin: ECC
|
|
8
|
+
allowed-tools: Read, Grep, Glob
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Agent Harness Construction
|
|
12
|
+
|
|
13
|
+
Use this skill when you are improving how an agent plans, calls tools, recovers from errors, and converges on completion.
|
|
14
|
+
|
|
15
|
+
## Core Model
|
|
16
|
+
|
|
17
|
+
Agent output quality is constrained by:
|
|
18
|
+
1. Action space quality
|
|
19
|
+
2. Observation quality
|
|
20
|
+
3. Recovery quality
|
|
21
|
+
4. Context budget quality
|
|
22
|
+
|
|
23
|
+
## Action Space Design
|
|
24
|
+
|
|
25
|
+
1. Use stable, explicit tool names.
|
|
26
|
+
2. Keep inputs schema-first and narrow.
|
|
27
|
+
3. Return deterministic output shapes.
|
|
28
|
+
4. Avoid catch-all tools unless isolation is impossible.
|
|
29
|
+
|
|
30
|
+
## Granularity Rules
|
|
31
|
+
|
|
32
|
+
- Use micro-tools for high-risk operations (deploy, migration, permissions).
|
|
33
|
+
- Use medium tools for common edit/read/search loops.
|
|
34
|
+
- Use macro-tools only when round-trip overhead is the dominant cost.
|
|
35
|
+
|
|
36
|
+
## Observation Design
|
|
37
|
+
|
|
38
|
+
Every tool response should include:
|
|
39
|
+
- `status`: success|warning|error
|
|
40
|
+
- `summary`: one-line result
|
|
41
|
+
- `next_actions`: actionable follow-ups
|
|
42
|
+
- `artifacts`: file paths / IDs
|
|
43
|
+
|
|
44
|
+
## Error Recovery Contract
|
|
45
|
+
|
|
46
|
+
For every error path, include:
|
|
47
|
+
- root cause hint
|
|
48
|
+
- safe retry instruction
|
|
49
|
+
- explicit stop condition
|
|
50
|
+
|
|
51
|
+
## Context Budgeting
|
|
52
|
+
|
|
53
|
+
1. Keep system prompt minimal and invariant.
|
|
54
|
+
2. Move large guidance into skills loaded on demand.
|
|
55
|
+
3. Prefer references to files over inlining long documents.
|
|
56
|
+
4. Compact at phase boundaries, not arbitrary token thresholds.
|
|
57
|
+
|
|
58
|
+
## Architecture Pattern Guidance
|
|
59
|
+
|
|
60
|
+
- ReAct: best for exploratory tasks with uncertain path.
|
|
61
|
+
- Function-calling: best for structured deterministic flows.
|
|
62
|
+
- Hybrid (recommended): ReAct planning + typed tool execution.
|
|
63
|
+
|
|
64
|
+
## Benchmarking
|
|
65
|
+
|
|
66
|
+
Track:
|
|
67
|
+
- completion rate
|
|
68
|
+
- retries per task
|
|
69
|
+
- pass@1 and pass@3
|
|
70
|
+
- cost per successful task
|
|
71
|
+
|
|
72
|
+
## Anti-Patterns
|
|
73
|
+
|
|
74
|
+
- Too many tools with overlapping semantics.
|
|
75
|
+
- Opaque tool output with no recovery hints.
|
|
76
|
+
- Error-only output without next steps.
|
|
77
|
+
- Context overloading with irrelevant references.
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: agent-introspection-debugging
|
|
3
|
+
description: |
|
|
4
|
+
Auto-invoke when an agent run fails repeatedly, hits max tool call limits,
|
|
5
|
+
loops on the same tools without forward progress, or drifts from its intended task.
|
|
6
|
+
Use for structured self-debugging (capture → diagnose → recover) before escalating to human.
|
|
7
|
+
origin: ECC
|
|
8
|
+
allowed-tools: Read, Bash, Grep, Glob
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
# Agent Introspection Debugging
|
|
12
|
+
|
|
13
|
+
Use this skill when an agent run is failing repeatedly, consuming tokens without progress, looping on the same tools, or drifting away from the intended task.
|
|
14
|
+
|
|
15
|
+
This is a workflow skill, not a hidden runtime. It teaches the agent to debug itself systematically before escalating to a human.
|
|
16
|
+
|
|
17
|
+
## When to Activate
|
|
18
|
+
|
|
19
|
+
- Maximum tool call / loop-limit failures
|
|
20
|
+
- Repeated retries with no forward progress
|
|
21
|
+
- Context growth or prompt drift that starts degrading output quality
|
|
22
|
+
- File-system or environment state mismatch between expectation and reality
|
|
23
|
+
- Tool failures that are likely recoverable with diagnosis and a smaller corrective action
|
|
24
|
+
|
|
25
|
+
## Scope Boundaries
|
|
26
|
+
|
|
27
|
+
Activate this skill for:
|
|
28
|
+
- capturing failure state before retrying blindly
|
|
29
|
+
- diagnosing common agent-specific failure patterns
|
|
30
|
+
- applying contained recovery actions
|
|
31
|
+
- producing a structured human-readable debug report
|
|
32
|
+
|
|
33
|
+
Do not use this skill as the primary source for:
|
|
34
|
+
- feature verification after code changes; use `/tas-verify` instead
|
|
35
|
+
- framework-specific debugging when a narrower skill or agent already covers it
|
|
36
|
+
- runtime promises the current harness cannot enforce automatically
|
|
37
|
+
|
|
38
|
+
## Four-Phase Loop
|
|
39
|
+
|
|
40
|
+
### Phase 1: Failure Capture
|
|
41
|
+
|
|
42
|
+
Before trying to recover, record the failure precisely.
|
|
43
|
+
|
|
44
|
+
Capture:
|
|
45
|
+
- error type, message, and stack trace when available
|
|
46
|
+
- last meaningful tool call sequence
|
|
47
|
+
- what the agent was trying to do
|
|
48
|
+
- current context pressure: repeated prompts, oversized pasted logs, duplicated plans, or runaway notes
|
|
49
|
+
- current environment assumptions: cwd, branch, relevant service state, expected files
|
|
50
|
+
|
|
51
|
+
Minimum capture template:
|
|
52
|
+
|
|
53
|
+
```markdown
|
|
54
|
+
## Failure Capture
|
|
55
|
+
- Session / task:
|
|
56
|
+
- Goal in progress:
|
|
57
|
+
- Error:
|
|
58
|
+
- Last successful step:
|
|
59
|
+
- Last failed tool / command:
|
|
60
|
+
- Repeated pattern seen:
|
|
61
|
+
- Environment assumptions to verify:
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
### Phase 2: Root-Cause Diagnosis
|
|
65
|
+
|
|
66
|
+
Match the failure to a known pattern before changing anything.
|
|
67
|
+
|
|
68
|
+
| Pattern | Likely Cause | Check |
|
|
69
|
+
| --- | --- | --- |
|
|
70
|
+
| Maximum tool calls / repeated same command | loop or no-exit observer path | inspect the last N tool calls for repetition |
|
|
71
|
+
| Context overflow / degraded reasoning | unbounded notes, repeated plans, oversized logs | inspect recent context for duplication and low-signal bulk |
|
|
72
|
+
| `ECONNREFUSED` / timeout | service unavailable or wrong port | verify service health, URL, and port assumptions |
|
|
73
|
+
| `429` / quota exhaustion | retry storm or missing backoff | count repeated calls and inspect retry spacing |
|
|
74
|
+
| file missing after write / stale diff | race, wrong cwd, or branch drift | re-check path, cwd, git status, and actual file existence |
|
|
75
|
+
| tests still failing after “fix” | wrong hypothesis | isolate the exact failing test and re-derive the bug |
|
|
76
|
+
|
|
77
|
+
Diagnosis questions:
|
|
78
|
+
- is this a logic failure, state failure, environment failure, or policy failure?
|
|
79
|
+
- did the agent lose the real objective and start optimizing the wrong subtask?
|
|
80
|
+
- is the failure deterministic or transient?
|
|
81
|
+
- what is the smallest reversible action that would validate the diagnosis?
|
|
82
|
+
|
|
83
|
+
### Phase 3: Contained Recovery
|
|
84
|
+
|
|
85
|
+
Recover with the smallest action that changes the diagnosis surface.
|
|
86
|
+
|
|
87
|
+
Safe recovery actions:
|
|
88
|
+
- stop repeated retries and restate the hypothesis
|
|
89
|
+
- trim low-signal context and keep only the active goal, blockers, and evidence
|
|
90
|
+
- re-check the actual filesystem / branch / process state
|
|
91
|
+
- narrow the task to one failing command, one file, or one test
|
|
92
|
+
- switch from speculative reasoning to direct observation
|
|
93
|
+
- escalate to a human when the failure is high-risk or externally blocked
|
|
94
|
+
|
|
95
|
+
Do not claim unsupported auto-healing actions like “reset agent state” or “update harness config” unless you are actually doing them through real tools in the current environment.
|
|
96
|
+
|
|
97
|
+
Contained recovery checklist:
|
|
98
|
+
|
|
99
|
+
```markdown
|
|
100
|
+
## Recovery Action
|
|
101
|
+
- Diagnosis chosen:
|
|
102
|
+
- Smallest action taken:
|
|
103
|
+
- Why this is safe:
|
|
104
|
+
- What evidence would prove the fix worked:
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
### Phase 4: Introspection Report
|
|
108
|
+
|
|
109
|
+
End with a report that makes the recovery legible to the next agent or human.
|
|
110
|
+
|
|
111
|
+
```markdown
|
|
112
|
+
## Agent Self-Debug Report
|
|
113
|
+
- Session / task:
|
|
114
|
+
- Failure:
|
|
115
|
+
- Root cause:
|
|
116
|
+
- Recovery action:
|
|
117
|
+
- Result: success | partial | blocked
|
|
118
|
+
- Token / time burn risk:
|
|
119
|
+
- Follow-up needed:
|
|
120
|
+
- Preventive change to encode later:
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
## Recovery Heuristics
|
|
124
|
+
|
|
125
|
+
Prefer these interventions in order:
|
|
126
|
+
|
|
127
|
+
1. Restate the real objective in one sentence.
|
|
128
|
+
2. Verify the world state instead of trusting memory.
|
|
129
|
+
3. Shrink the failing scope.
|
|
130
|
+
4. Run one discriminating check.
|
|
131
|
+
5. Only then retry.
|
|
132
|
+
|
|
133
|
+
Bad pattern:
|
|
134
|
+
- retrying the same action three times with slightly different wording
|
|
135
|
+
|
|
136
|
+
Good pattern:
|
|
137
|
+
- capture failure
|
|
138
|
+
- classify the pattern
|
|
139
|
+
- run one direct check
|
|
140
|
+
- change the plan only if the check supports it
|
|
141
|
+
|
|
142
|
+
## Integration with TAS Kit
|
|
143
|
+
|
|
144
|
+
- Run `/tas-verify` after recovery if code was changed — confirms implementation meets acceptance criteria.
|
|
145
|
+
- Use `/tas-bug` when the failure pattern reveals a reproducible bug worth tracking.
|
|
146
|
+
- Escalate to human via `AskUserQuestion` when the issue is not technical failure but decision ambiguity.
|
|
147
|
+
- Run `git status` + `git diff` if the failure came from conflicting local state or repo drift.
|
|
148
|
+
|
|
149
|
+
## Output Standard
|
|
150
|
+
|
|
151
|
+
When this skill is active, do not end with “I fixed it” alone.
|
|
152
|
+
|
|
153
|
+
Always provide:
|
|
154
|
+
- the failure pattern
|
|
155
|
+
- the root-cause hypothesis
|
|
156
|
+
- the recovery action
|
|
157
|
+
- the evidence that the situation is now better or still blocked
|
|
@@ -0,0 +1,364 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ai-regression-testing
|
|
3
|
+
description: |
|
|
4
|
+
Auto-invoke when an AI agent has modified API routes or backend logic, when a bug
|
|
5
|
+
is found and needs a regression test written, or when running bug-check workflows
|
|
6
|
+
on AI-generated code. Especially valuable when a sandbox/mock mode exists —
|
|
7
|
+
enables fast, DB-free API testing to catch sandbox/production path mismatches.
|
|
8
|
+
origin: ECC
|
|
9
|
+
allowed-tools: Read, Write, Edit, Bash, Grep, Glob
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# AI Regression Testing
|
|
13
|
+
|
|
14
|
+
Testing patterns specifically designed for AI-assisted development, where the same model writes code and reviews it — creating systematic blind spots that only automated tests can catch.
|
|
15
|
+
|
|
16
|
+
## When to Activate
|
|
17
|
+
|
|
18
|
+
- AI agent (Claude Code, Cursor, Codex) has modified API routes or backend logic
|
|
19
|
+
- A bug was found and fixed — need to prevent re-introduction
|
|
20
|
+
- Project has a sandbox/mock mode that can be leveraged for DB-free testing
|
|
21
|
+
- Running `/tas-verify` or post-fix review workflows after code changes
|
|
22
|
+
- Multiple code paths exist (sandbox vs production, feature flags, etc.)
|
|
23
|
+
|
|
24
|
+
## The Core Problem
|
|
25
|
+
|
|
26
|
+
When an AI writes code and then reviews its own work, it carries the same assumptions into both steps. This creates a predictable failure pattern:
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
AI writes fix → AI reviews fix → AI says "looks correct" → Bug still exists
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
**Real-world example** (observed in production):
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
Fix 1: Added notification_settings to API response
|
|
36
|
+
→ Forgot to add it to the SELECT query
|
|
37
|
+
→ AI reviewed and missed it (same blind spot)
|
|
38
|
+
|
|
39
|
+
Fix 2: Added it to SELECT query
|
|
40
|
+
→ TypeScript build error (column not in generated types)
|
|
41
|
+
→ AI reviewed Fix 1 but didn't catch the SELECT issue
|
|
42
|
+
|
|
43
|
+
Fix 3: Changed to SELECT *
|
|
44
|
+
→ Fixed production path, forgot sandbox path
|
|
45
|
+
→ AI reviewed and missed it AGAIN (4th occurrence)
|
|
46
|
+
|
|
47
|
+
Fix 4: Test caught it instantly on first run PASS:
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
The pattern: **sandbox/production path inconsistency** is the #1 AI-introduced regression.
|
|
51
|
+
|
|
52
|
+
## Sandbox-Mode API Testing
|
|
53
|
+
|
|
54
|
+
Most projects with AI-friendly architecture have a sandbox/mock mode. This is the key to fast, DB-free API testing.
|
|
55
|
+
|
|
56
|
+
### Setup (Vitest + Next.js App Router)
|
|
57
|
+
|
|
58
|
+
```typescript
|
|
59
|
+
// vitest.config.ts
|
|
60
|
+
import { defineConfig } from "vitest/config";
|
|
61
|
+
import path from "path";
|
|
62
|
+
|
|
63
|
+
export default defineConfig({
|
|
64
|
+
test: {
|
|
65
|
+
environment: "node",
|
|
66
|
+
globals: true,
|
|
67
|
+
include: ["__tests__/**/*.test.ts"],
|
|
68
|
+
setupFiles: ["__tests__/setup.ts"],
|
|
69
|
+
},
|
|
70
|
+
resolve: {
|
|
71
|
+
alias: {
|
|
72
|
+
"@": path.resolve(__dirname, "."),
|
|
73
|
+
},
|
|
74
|
+
},
|
|
75
|
+
});
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
```typescript
|
|
79
|
+
// __tests__/setup.ts
|
|
80
|
+
// Force sandbox mode — no database needed
|
|
81
|
+
process.env.SANDBOX_MODE = "true";
|
|
82
|
+
process.env.NEXT_PUBLIC_SUPABASE_URL = "";
|
|
83
|
+
process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY = "";
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
### Test Helper for Next.js API Routes
|
|
87
|
+
|
|
88
|
+
```typescript
|
|
89
|
+
// __tests__/helpers.ts
|
|
90
|
+
import { NextRequest } from "next/server";
|
|
91
|
+
|
|
92
|
+
export function createTestRequest(
|
|
93
|
+
url: string,
|
|
94
|
+
options?: {
|
|
95
|
+
method?: string;
|
|
96
|
+
body?: Record<string, unknown>;
|
|
97
|
+
headers?: Record<string, string>;
|
|
98
|
+
sandboxUserId?: string;
|
|
99
|
+
},
|
|
100
|
+
): NextRequest {
|
|
101
|
+
const { method = "GET", body, headers = {}, sandboxUserId } = options || {};
|
|
102
|
+
const fullUrl = url.startsWith("http") ? url : `http://localhost:3000${url}`;
|
|
103
|
+
const reqHeaders: Record<string, string> = { ...headers };
|
|
104
|
+
|
|
105
|
+
if (sandboxUserId) {
|
|
106
|
+
reqHeaders["x-sandbox-user-id"] = sandboxUserId;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
const init: { method: string; headers: Record<string, string>; body?: string } = {
|
|
110
|
+
method,
|
|
111
|
+
headers: reqHeaders,
|
|
112
|
+
};
|
|
113
|
+
|
|
114
|
+
if (body) {
|
|
115
|
+
init.body = JSON.stringify(body);
|
|
116
|
+
reqHeaders["content-type"] = "application/json";
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
return new NextRequest(fullUrl, init);
|
|
120
|
+
}
|
|
121
|
+
|
|
122
|
+
export async function parseResponse(response: Response) {
|
|
123
|
+
const json = await response.json();
|
|
124
|
+
return { status: response.status, json };
|
|
125
|
+
}
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
### Writing Regression Tests
|
|
129
|
+
|
|
130
|
+
The key principle: **write tests for bugs that were found, not for code that works**.
|
|
131
|
+
|
|
132
|
+
```typescript
|
|
133
|
+
// __tests__/api/user/profile.test.ts
|
|
134
|
+
import { describe, it, expect } from "vitest";
|
|
135
|
+
import { createTestRequest, parseResponse } from "../../helpers";
|
|
136
|
+
import { GET, PATCH } from "@/app/api/user/profile/route";
|
|
137
|
+
|
|
138
|
+
// Define the contract — what fields MUST be in the response
|
|
139
|
+
const REQUIRED_FIELDS = [
|
|
140
|
+
"id",
|
|
141
|
+
"email",
|
|
142
|
+
"full_name",
|
|
143
|
+
"phone",
|
|
144
|
+
"role",
|
|
145
|
+
"created_at",
|
|
146
|
+
"avatar_url",
|
|
147
|
+
"notification_settings", // ← Added after bug found it missing
|
|
148
|
+
];
|
|
149
|
+
|
|
150
|
+
describe("GET /api/user/profile", () => {
|
|
151
|
+
it("returns all required fields", async () => {
|
|
152
|
+
const req = createTestRequest("/api/user/profile");
|
|
153
|
+
const res = await GET(req);
|
|
154
|
+
const { status, json } = await parseResponse(res);
|
|
155
|
+
|
|
156
|
+
expect(status).toBe(200);
|
|
157
|
+
for (const field of REQUIRED_FIELDS) {
|
|
158
|
+
expect(json.data).toHaveProperty(field);
|
|
159
|
+
}
|
|
160
|
+
});
|
|
161
|
+
|
|
162
|
+
// Regression test — this exact bug was introduced by AI 4 times
|
|
163
|
+
it("notification_settings is not undefined (BUG-R1 regression)", async () => {
|
|
164
|
+
const req = createTestRequest("/api/user/profile");
|
|
165
|
+
const res = await GET(req);
|
|
166
|
+
const { json } = await parseResponse(res);
|
|
167
|
+
|
|
168
|
+
expect("notification_settings" in json.data).toBe(true);
|
|
169
|
+
const ns = json.data.notification_settings;
|
|
170
|
+
expect(ns === null || typeof ns === "object").toBe(true);
|
|
171
|
+
});
|
|
172
|
+
});
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
### Testing Sandbox/Production Parity
|
|
176
|
+
|
|
177
|
+
The most common AI regression: fixing production path but forgetting sandbox path (or vice versa).
|
|
178
|
+
|
|
179
|
+
```typescript
|
|
180
|
+
// Test that sandbox responses match the expected contract
|
|
181
|
+
describe("GET /api/user/messages (conversation list)", () => {
|
|
182
|
+
it("includes partner_name in sandbox mode", async () => {
|
|
183
|
+
const req = createTestRequest("/api/user/messages", {
|
|
184
|
+
sandboxUserId: "user-001",
|
|
185
|
+
});
|
|
186
|
+
const res = await GET(req);
|
|
187
|
+
const { json } = await parseResponse(res);
|
|
188
|
+
|
|
189
|
+
// This caught a bug where partner_name was added
|
|
190
|
+
// to production path but not sandbox path
|
|
191
|
+
if (json.data.length > 0) {
|
|
192
|
+
for (const conv of json.data) {
|
|
193
|
+
expect("partner_name" in conv).toBe(true);
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
});
|
|
197
|
+
});
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
## Integrating Tests into Bug-Check Workflow
|
|
201
|
+
|
|
202
|
+
### Workflow Integration with TAS Kit
|
|
203
|
+
|
|
204
|
+
Pair with `/tas-bug` for bug tracking and `/tas-verify` for post-fix verification.
|
|
205
|
+
|
|
206
|
+
```
|
|
207
|
+
User: "/tas-bug" or reports a bug
|
|
208
|
+
│
|
|
209
|
+
├─ Step 1: npm run test
|
|
210
|
+
│ ├─ FAIL → Bug found mechanically (no AI judgment needed)
|
|
211
|
+
│ └─ PASS → Continue
|
|
212
|
+
│
|
|
213
|
+
├─ Step 2: npm run build
|
|
214
|
+
│ ├─ FAIL → Type error found mechanically
|
|
215
|
+
│ └─ PASS → Continue
|
|
216
|
+
│
|
|
217
|
+
├─ Step 3: AI code review (with known blind spots in mind)
|
|
218
|
+
│ └─ Findings reported
|
|
219
|
+
│
|
|
220
|
+
└─ Step 4: For each fix, write a regression test
|
|
221
|
+
└─ Next bug-check catches if fix breaks
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
## Common AI Regression Patterns
|
|
225
|
+
|
|
226
|
+
### Pattern 1: Sandbox/Production Path Mismatch
|
|
227
|
+
|
|
228
|
+
**Frequency**: Most common (observed in 3 out of 4 regressions)
|
|
229
|
+
|
|
230
|
+
```typescript
|
|
231
|
+
// FAIL: AI adds field to production path only
|
|
232
|
+
if (isSandboxMode()) {
|
|
233
|
+
return { data: { id, email, name } }; // Missing new field
|
|
234
|
+
}
|
|
235
|
+
// Production path
|
|
236
|
+
return { data: { id, email, name, notification_settings } };
|
|
237
|
+
|
|
238
|
+
// PASS: Both paths must return the same shape
|
|
239
|
+
if (isSandboxMode()) {
|
|
240
|
+
return { data: { id, email, name, notification_settings: null } };
|
|
241
|
+
}
|
|
242
|
+
return { data: { id, email, name, notification_settings } };
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
**Test to catch it**:
|
|
246
|
+
|
|
247
|
+
```typescript
|
|
248
|
+
it("sandbox and production return same fields", async () => {
|
|
249
|
+
// In test env, sandbox mode is forced ON
|
|
250
|
+
const res = await GET(createTestRequest("/api/user/profile"));
|
|
251
|
+
const { json } = await parseResponse(res);
|
|
252
|
+
|
|
253
|
+
for (const field of REQUIRED_FIELDS) {
|
|
254
|
+
expect(json.data).toHaveProperty(field);
|
|
255
|
+
}
|
|
256
|
+
});
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
### Pattern 2: SELECT Clause Omission
|
|
260
|
+
|
|
261
|
+
**Frequency**: Common with Supabase/Prisma when adding new columns
|
|
262
|
+
|
|
263
|
+
```typescript
|
|
264
|
+
// FAIL: New column added to response but not to SELECT
|
|
265
|
+
const { data } = await supabase
|
|
266
|
+
.from("users")
|
|
267
|
+
.select("id, email, name") // notification_settings not here
|
|
268
|
+
.single();
|
|
269
|
+
|
|
270
|
+
return { data: { ...data, notification_settings: data.notification_settings } };
|
|
271
|
+
// → notification_settings is always undefined
|
|
272
|
+
|
|
273
|
+
// PASS: Use SELECT * or explicitly include new columns
|
|
274
|
+
const { data } = await supabase
|
|
275
|
+
.from("users")
|
|
276
|
+
.select("*")
|
|
277
|
+
.single();
|
|
278
|
+
```
|
|
279
|
+
|
|
280
|
+
### Pattern 3: Error State Leakage
|
|
281
|
+
|
|
282
|
+
**Frequency**: Moderate — when adding error handling to existing components
|
|
283
|
+
|
|
284
|
+
```typescript
|
|
285
|
+
// FAIL: Error state set but old data not cleared
|
|
286
|
+
catch (err) {
|
|
287
|
+
setError("Failed to load");
|
|
288
|
+
// reservations still shows data from previous tab!
|
|
289
|
+
}
|
|
290
|
+
|
|
291
|
+
// PASS: Clear related state on error
|
|
292
|
+
catch (err) {
|
|
293
|
+
setReservations([]); // Clear stale data
|
|
294
|
+
setError("Failed to load");
|
|
295
|
+
}
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
### Pattern 4: Optimistic Update Without Proper Rollback
|
|
299
|
+
|
|
300
|
+
```typescript
|
|
301
|
+
// FAIL: No rollback on failure
|
|
302
|
+
const handleRemove = async (id: string) => {
|
|
303
|
+
setItems(prev => prev.filter(i => i.id !== id));
|
|
304
|
+
await fetch(`/api/items/${id}`, { method: "DELETE" });
|
|
305
|
+
// If API fails, item is gone from UI but still in DB
|
|
306
|
+
};
|
|
307
|
+
|
|
308
|
+
// PASS: Capture previous state and rollback on failure
|
|
309
|
+
const handleRemove = async (id: string) => {
|
|
310
|
+
const prevItems = [...items];
|
|
311
|
+
setItems(prev => prev.filter(i => i.id !== id));
|
|
312
|
+
try {
|
|
313
|
+
const res = await fetch(`/api/items/${id}`, { method: "DELETE" });
|
|
314
|
+
if (!res.ok) throw new Error("API error");
|
|
315
|
+
} catch {
|
|
316
|
+
setItems(prevItems); // Rollback
|
|
317
|
+
alert("削除に失敗しました");
|
|
318
|
+
}
|
|
319
|
+
};
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
## Strategy: Test Where Bugs Were Found
|
|
323
|
+
|
|
324
|
+
Don't aim for 100% coverage. Instead:
|
|
325
|
+
|
|
326
|
+
```
|
|
327
|
+
Bug found in /api/user/profile → Write test for profile API
|
|
328
|
+
Bug found in /api/user/messages → Write test for messages API
|
|
329
|
+
Bug found in /api/user/favorites → Write test for favorites API
|
|
330
|
+
No bug in /api/user/notifications → Don't write test (yet)
|
|
331
|
+
```
|
|
332
|
+
|
|
333
|
+
**Why this works with AI development:**
|
|
334
|
+
|
|
335
|
+
1. AI tends to make the **same category of mistake** repeatedly
|
|
336
|
+
2. Bugs cluster in complex areas (auth, multi-path logic, state management)
|
|
337
|
+
3. Once tested, that exact regression **cannot happen again**
|
|
338
|
+
4. Test count grows organically with bug fixes — no wasted effort
|
|
339
|
+
|
|
340
|
+
## Quick Reference
|
|
341
|
+
|
|
342
|
+
| AI Regression Pattern | Test Strategy | Priority |
|
|
343
|
+
|---|---|---|
|
|
344
|
+
| Sandbox/production mismatch | Assert same response shape in sandbox mode | High |
|
|
345
|
+
| SELECT clause omission | Assert all required fields in response | High |
|
|
346
|
+
| Error state leakage | Assert state cleanup on error | Medium |
|
|
347
|
+
| Missing rollback | Assert state restored on API failure | Medium |
|
|
348
|
+
| Type cast masking null | Assert field is not undefined | Medium |
|
|
349
|
+
|
|
350
|
+
## DO / DON'T
|
|
351
|
+
|
|
352
|
+
**DO:**
|
|
353
|
+
- Write tests immediately after finding a bug (before fixing it if possible)
|
|
354
|
+
- Test the API response shape, not the implementation
|
|
355
|
+
- Run tests as the first step of every bug-check
|
|
356
|
+
- Keep tests fast (< 1 second total with sandbox mode)
|
|
357
|
+
- Name tests after the bug they prevent (e.g., "BUG-R1 regression")
|
|
358
|
+
|
|
359
|
+
**DON'T:**
|
|
360
|
+
- Write tests for code that has never had a bug
|
|
361
|
+
- Trust AI self-review as a substitute for automated tests
|
|
362
|
+
- Skip sandbox path testing because "it's just mock data"
|
|
363
|
+
- Write integration tests when unit tests suffice
|
|
364
|
+
- Aim for coverage percentage — aim for regression prevention
|