test-genie-mcp 2.0.2 → 3.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +224 -0
- package/LICENSE +21 -0
- package/README.md +334 -169
- package/SECURITY.md +150 -0
- package/dist/analyzers/performanceAnalyzer.d.ts.map +1 -1
- package/dist/analyzers/performanceAnalyzer.js +0 -3
- package/dist/analyzers/performanceAnalyzer.js.map +1 -1
- package/dist/analyzers/raceConditionAnalyzer.d.ts +39 -0
- package/dist/analyzers/raceConditionAnalyzer.d.ts.map +1 -0
- package/dist/analyzers/raceConditionAnalyzer.js +412 -0
- package/dist/analyzers/raceConditionAnalyzer.js.map +1 -0
- package/dist/analyzers/securityAnalyzer.d.ts +45 -0
- package/dist/analyzers/securityAnalyzer.d.ts.map +1 -0
- package/dist/analyzers/securityAnalyzer.js +457 -0
- package/dist/analyzers/securityAnalyzer.js.map +1 -0
- package/dist/core/mcpServerFactory.d.ts +20 -0
- package/dist/core/mcpServerFactory.d.ts.map +1 -0
- package/dist/core/mcpServerFactory.js +133 -0
- package/dist/core/mcpServerFactory.js.map +1 -0
- package/dist/core/prompts.d.ts +38 -0
- package/dist/core/prompts.d.ts.map +1 -0
- package/dist/core/prompts.js +158 -0
- package/dist/core/prompts.js.map +1 -0
- package/dist/core/resources.d.ts +47 -0
- package/dist/core/resources.d.ts.map +1 -0
- package/dist/core/resources.js +116 -0
- package/dist/core/resources.js.map +1 -0
- package/dist/core/subprocess.d.ts +66 -0
- package/dist/core/subprocess.d.ts.map +1 -0
- package/dist/core/subprocess.js +147 -0
- package/dist/core/subprocess.js.map +1 -0
- package/dist/core/syntaxValidator.d.ts +50 -0
- package/dist/core/syntaxValidator.d.ts.map +1 -0
- package/dist/core/syntaxValidator.js +296 -0
- package/dist/core/syntaxValidator.js.map +1 -0
- package/dist/core/toolRegistry.d.ts +30 -0
- package/dist/core/toolRegistry.d.ts.map +1 -0
- package/dist/core/toolRegistry.js +655 -0
- package/dist/core/toolRegistry.js.map +1 -0
- package/dist/index.d.ts +10 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +17 -670
- package/dist/index.js.map +1 -1
- package/dist/llm/index.d.ts +54 -0
- package/dist/llm/index.d.ts.map +1 -0
- package/dist/llm/index.js +190 -0
- package/dist/llm/index.js.map +1 -0
- package/dist/platforms/android/index.d.ts +2 -2
- package/dist/platforms/android/index.d.ts.map +1 -1
- package/dist/platforms/android/index.js +189 -164
- package/dist/platforms/android/index.js.map +1 -1
- package/dist/platforms/flutter/index.d.ts +1 -1
- package/dist/platforms/flutter/index.d.ts.map +1 -1
- package/dist/platforms/flutter/index.js +133 -193
- package/dist/platforms/flutter/index.js.map +1 -1
- package/dist/platforms/ios/index.d.ts.map +1 -1
- package/dist/platforms/ios/index.js +101 -102
- package/dist/platforms/ios/index.js.map +1 -1
- package/dist/platforms/react-native/index.d.ts +1 -1
- package/dist/platforms/react-native/index.d.ts.map +1 -1
- package/dist/platforms/react-native/index.js +122 -240
- package/dist/platforms/react-native/index.js.map +1 -1
- package/dist/platforms/web/index.d.ts.map +1 -1
- package/dist/platforms/web/index.js +45 -28
- package/dist/platforms/web/index.js.map +1 -1
- package/dist/security.d.ts +81 -0
- package/dist/security.d.ts.map +1 -0
- package/dist/security.js +197 -0
- package/dist/security.js.map +1 -0
- package/dist/storage/index.d.ts +35 -0
- package/dist/storage/index.d.ts.map +1 -1
- package/dist/storage/index.js +33 -1
- package/dist/storage/index.js.map +1 -1
- package/dist/tools/automation/diagnoseAutoFix.d.ts +131 -0
- package/dist/tools/automation/diagnoseAutoFix.d.ts.map +1 -0
- package/dist/tools/automation/diagnoseAutoFix.js +366 -0
- package/dist/tools/automation/diagnoseAutoFix.js.map +1 -0
- package/dist/tools/automation/diagnoseProject.d.ts +57 -0
- package/dist/tools/automation/diagnoseProject.d.ts.map +1 -0
- package/dist/tools/automation/diagnoseProject.js +399 -0
- package/dist/tools/automation/diagnoseProject.js.map +1 -0
- package/dist/tools/automation/runIterativeFixLoop.d.ts +92 -0
- package/dist/tools/automation/runIterativeFixLoop.d.ts.map +1 -0
- package/dist/tools/automation/runIterativeFixLoop.js +474 -0
- package/dist/tools/automation/runIterativeFixLoop.js.map +1 -0
- package/dist/tools/fixing/applyFix.d.ts.map +1 -1
- package/dist/tools/fixing/applyFix.js +11 -37
- package/dist/tools/fixing/applyFix.js.map +1 -1
- package/dist/tools/fixing/llmFixSuggester.d.ts +29 -0
- package/dist/tools/fixing/llmFixSuggester.d.ts.map +1 -0
- package/dist/tools/fixing/llmFixSuggester.js +128 -0
- package/dist/tools/fixing/llmFixSuggester.js.map +1 -0
- package/package.json +29 -17
package/README.md
CHANGED
|
@@ -1,219 +1,384 @@
|
|
|
1
|
-
#
|
|
1
|
+
# test-genie-mcp
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**Built for vibe coders: one command, get a prioritized list of what's actually broken about your project.**
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Self-healing test automation for iOS, Android, Flutter, React Native and Web apps — as an MCP server.
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
[](https://www.npmjs.com/package/test-genie-mcp)
|
|
8
|
+
[](https://github.com/MUSE-CODE-SPACE/test-genie-mcp/actions)
|
|
9
|
+
[](https://opensource.org/licenses/MIT)
|
|
10
|
+
[](https://modelcontextprotocol.io)
|
|
8
11
|
|
|
9
|
-
|
|
10
|
-
- **analyze_app_structure**: 앱 코드베이스 분석 (화면, 컴포넌트, API, 상태관리)
|
|
11
|
-
- **generate_scenarios**: AI 기반 테스트 시나리오 자동 생성
|
|
12
|
-
- **create_test_plan**: 테스트 계획 수립 및 스케줄링
|
|
12
|
+
> **v3.1.1 — vibe-check + honest auto-fix.** One MCP call, ~30 seconds: race conditions + security issues + memory leaks + logic errors + perf smells, prioritized. Stays on your machine, no telemetry. Pass `autoFix: true` for the small, safe mechanical fixes (weak-hash, simple `Math.random` assignment) — backup + syntax-validate + rollback-on-syntax-fail. For test-verified application of harder fixes, use [v3.0.0](#how-the-iterate-fix-loop-works)'s iterate-fix loop.
|
|
13
13
|
|
|
14
|
-
|
|
15
|
-
- **run_scenario_test**: 개별 시나리오 테스트 실행
|
|
16
|
-
- **run_simulation**: 사용자 행동 시뮬레이션 (랜덤/패턴 기반)
|
|
17
|
-
- **run_stress_test**: 스트레스/부하 테스트
|
|
14
|
+
---
|
|
18
15
|
|
|
19
|
-
|
|
20
|
-
- **detect_memory_leaks**: 메모리 릭 감지 (힙 분석, 순환 참조)
|
|
21
|
-
- **detect_logic_errors**: 논리적 오류 검출 (레이스 컨디션, 상태 불일치)
|
|
16
|
+
## Vibe coders quickstart
|
|
22
17
|
|
|
23
|
-
|
|
24
|
-
- **suggest_fixes**: AI 기반 수정 방안 제안
|
|
25
|
-
- **confirm_fix**: 수정 사항 사용자 확인
|
|
26
|
-
- **apply_fix**: 확인된 수정 사항 적용
|
|
27
|
-
- **rollback_fix**: 적용된 수정 롤백
|
|
18
|
+
You don't read the docs. You open the project, talk to Claude, and want a verdict. Here it is:
|
|
28
19
|
|
|
29
|
-
|
|
30
|
-
- **run_full_automation**: 전체 파이프라인 자동 실행
|
|
31
|
-
- **generate_report**: 상세 보고서 생성 (Markdown, HTML, JSON)
|
|
20
|
+
In Claude (with test-genie-mcp installed — [setup](#5-minute-quickstart)):
|
|
32
21
|
|
|
33
|
-
|
|
34
|
-
-
|
|
35
|
-
|
|
36
|
-
- **generate_cicd_config**: CI/CD 설정 자동 생성 (GitHub Actions, Jenkins, GitLab CI)
|
|
22
|
+
```
|
|
23
|
+
/vibe-check /Users/me/my-app
|
|
24
|
+
```
|
|
37
25
|
|
|
38
|
-
|
|
26
|
+
Claude calls `diagnose_project` under the hood. ~30 seconds later you see:
|
|
39
27
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
| iOS | Swift, Objective-C | XCTest, XCUITest |
|
|
43
|
-
| Android | Kotlin, Java | Espresso, UI Automator |
|
|
44
|
-
| Flutter | Dart | flutter_test |
|
|
45
|
-
| React Native | TypeScript, JavaScript | Detox, Jest |
|
|
46
|
-
| Web | TypeScript, JavaScript | Playwright, Puppeteer |
|
|
28
|
+
```text
|
|
29
|
+
# vibe-check report
|
|
47
30
|
|
|
48
|
-
|
|
31
|
+
- Project: /Users/me/my-app
|
|
32
|
+
- Platform: web
|
|
33
|
+
- Findings: 11 total — 4 critical, 4 high, 1 medium, 1 low
|
|
34
|
+
- Estimated fix time: ~85 min
|
|
49
35
|
|
|
50
|
-
|
|
51
|
-
|
|
36
|
+
## Top 5 issues
|
|
37
|
+
|
|
38
|
+
### 1. [CRIT] Hardcoded AWS access key id found in source
|
|
39
|
+
- File: `server.js:7`
|
|
40
|
+
- Category: security / secret (CWE-798)
|
|
41
|
+
- Confidence: 95%
|
|
42
|
+
- Fix: Move the value to an env var, gitignore the config, rotate the leaked key.
|
|
43
|
+
|
|
44
|
+
### 2. [CRIT] SQL string built by concatenating user input
|
|
45
|
+
- File: `server.js:21`
|
|
46
|
+
- Category: security / injection (CWE-89)
|
|
47
|
+
- Fix: Use parameterized queries (`db.query("... WHERE id = ?", [id])`).
|
|
48
|
+
|
|
49
|
+
### 3. [HIGH] useState setter called after await without mount guard
|
|
50
|
+
- File: `UserProfile.tsx:16`
|
|
51
|
+
- Category: race-condition / react-setstate-after-await (CWE-362)
|
|
52
|
+
- Confidence: 78%
|
|
53
|
+
- Fix: Use AbortController and check signal.aborted before calling setters.
|
|
54
|
+
|
|
55
|
+
… (top 5 shown — full list at output: "detailed")
|
|
56
|
+
|
|
57
|
+
## Next steps
|
|
58
|
+
1. Address the critical / high findings above.
|
|
59
|
+
2. Re-run diagnose_project after fixing to confirm convergence.
|
|
60
|
+
3. Use run_iterative_fix_loop for test-driven verification of each fix.
|
|
52
61
|
```
|
|
53
62
|
|
|
54
|
-
|
|
63
|
+
If any finding is `autoFixable: true` and is at `high`/`critical` severity, the `diagnose_project` call accepts `autoFix: true` to apply the mechanical replacement directly (with backup + syntax validation — see [SAFETY.md](SAFETY.md) for the exact guards). The v3.1.1 honest scope is narrow: weak hash (`createHash('md5'|'sha1')` → `createHash('sha256')`) and standalone `Math.random()` in security-sensitive files. For broader/structural fixes (race conditions, eval, exec injection) run `run_iterative_fix_loop` separately — it re-runs tests and auto-rolls-back on regression.
|
|
55
64
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## Why test-genie?
|
|
68
|
+
|
|
69
|
+
The bottleneck in mobile + cross-platform test automation isn't writing tests — it's the loop *between* a failing test and a passing test. test-genie closes that loop:
|
|
70
|
+
|
|
71
|
+
```
|
|
72
|
+
failing test → analyzer flags issue → fix proposed → dry-run + syntax check →
|
|
73
|
+
applied with backup → affected tests re-run → regression check → loop or stop
|
|
61
74
|
```
|
|
62
75
|
|
|
63
|
-
|
|
76
|
+
This full loop is the `run_iterative_fix_loop` tool. The `diagnose_project autoFix: true` path in v3.1.1 covers a strict subset — backup + dry-run + syntax-validate + apply, **without** re-running tests (so no test-regression rollback in that path). Use the right tool for the job — and see [SAFETY.md](SAFETY.md) for the exact guards on each.
|
|
64
77
|
|
|
65
|
-
|
|
78
|
+
Other tools (Detox, Maestro, Playwright, `xcodebuild test`) run tests. test-genie **runs tests *and* drives the fix until the bar is met or it can no longer make progress** — without you scrubbing through stack traces.
|
|
66
79
|
|
|
67
|
-
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## 5-minute Quickstart
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
# 1. Install
|
|
86
|
+
npm install -g test-genie-mcp
|
|
87
|
+
|
|
88
|
+
# 2. Add to Claude Desktop config (~/.config/claude/claude_desktop_config.json)
|
|
68
89
|
{
|
|
69
90
|
"mcpServers": {
|
|
70
91
|
"test-genie": {
|
|
71
92
|
"command": "npx",
|
|
72
|
-
"args": ["test-genie-mcp"]
|
|
93
|
+
"args": ["test-genie-mcp"],
|
|
94
|
+
"env": {
|
|
95
|
+
"TEST_GENIE_ALLOWED_ROOT": "/path/to/your/project"
|
|
96
|
+
}
|
|
73
97
|
}
|
|
74
98
|
}
|
|
75
99
|
}
|
|
100
|
+
|
|
101
|
+
# 3. Restart Claude Desktop. From a chat:
|
|
102
|
+
# "Run the iterate-fix loop on /Users/me/my-rn-app with autoApply=false"
|
|
76
103
|
```
|
|
77
104
|
|
|
78
|
-
|
|
105
|
+
Expected output (truncated):
|
|
79
106
|
|
|
80
|
-
### Full Automation
|
|
81
|
-
```
|
|
82
|
-
User: "자동화 테스트해줘"
|
|
83
|
-
|
|
84
|
-
Claude will:
|
|
85
|
-
1. Analyze your app structure
|
|
86
|
-
2. Generate test scenarios
|
|
87
|
-
3. Execute tests
|
|
88
|
-
4. Detect issues (memory leaks, logic errors)
|
|
89
|
-
5. Suggest fixes
|
|
90
|
-
6. Wait for your confirmation
|
|
91
|
-
7. Apply approved fixes
|
|
92
|
-
8. Generate report
|
|
93
107
|
```
|
|
108
|
+
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
109
|
+
Iterative fix loop f8b3… — PAUSED-FOR-CONFIRMATION
|
|
110
|
+
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
|
94
111
|
|
|
95
|
-
|
|
112
|
+
Iterations completed: 1
|
|
113
|
+
Fixes applied: 0
|
|
114
|
+
Regressions rolled back: 0
|
|
115
|
+
Final tests: 7/10 passing (3 failing)
|
|
96
116
|
|
|
117
|
+
Pending confirmations (3):
|
|
118
|
+
- 71fbe…: Fix: useEffect missing cleanup for setInterval (confidence: 85)
|
|
119
|
+
- 92ad1…: Fix: Force-unwrap on possibly-undefined name (confidence: 85)
|
|
120
|
+
- …
|
|
121
|
+
|
|
122
|
+
Resume token: f8b3…
|
|
97
123
|
```
|
|
98
|
-
# Analyze app
|
|
99
|
-
analyze_app_structure(projectPath: "/path/to/app")
|
|
100
124
|
|
|
101
|
-
|
|
102
|
-
generate_scenarios(projectPath: "/path/to/app", testTypes: ["e2e", "unit"])
|
|
125
|
+
Re-call with `autoApply: true` (or `resumeToken: "f8b3…"`) to actually patch the files.
|
|
103
126
|
|
|
104
|
-
|
|
105
|
-
detect_memory_leaks(projectPath: "/path/to/app")
|
|
127
|
+
---
|
|
106
128
|
|
|
107
|
-
|
|
108
|
-
suggest_fixes(projectPath: "/path/to/app")
|
|
129
|
+
## Real use cases
|
|
109
130
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
131
|
+
> The flows below describe the **`run_iterative_fix_loop` path** (v3.0
|
|
132
|
+
> headline) — full detect → propose → dry-run → apply-with-backup →
|
|
133
|
+
> re-run-tests → rollback-on-regression. The `diagnose_project autoFix`
|
|
134
|
+
> path in v3.1.1 is the narrower mechanical-replacement-only path; see
|
|
135
|
+
> [SAFETY.md](SAFETY.md) §4 for what that one actually touches.
|
|
136
|
+
|
|
137
|
+
### 1. React Native memory-leak self-healing
|
|
138
|
+
|
|
139
|
+
A team adds `setInterval(...)` in a `useEffect` and forgets cleanup. test-genie's `detect_memory_leaks` flags it, `suggest_fixes` proposes `return () => clearInterval(id)` (`src/tools/fixing/suggestFixes.ts:169-179`), the loop dry-runs the patch through the TS compiler, applies with backup, re-runs only the affected snapshot test, confirms 100% pass, stops. **Before:** 1 failing snapshot. **After:** 0 failing, 1 fix applied, 1 backup at `.test-genie-backups/`.
|
|
140
|
+
|
|
141
|
+
### 2. Flutter widget `dispose()` automation
|
|
142
|
+
|
|
143
|
+
`AnimationController` left undisposed. test-genie sees the missing `dispose()` override, generates a Dart `@override dispose() { controller.dispose(); super.dispose(); }` block (`suggestFixes.ts:214-217`), runs `dart analyze` on the patched file, applies, re-runs `flutter test`, converges.
|
|
114
144
|
|
|
115
|
-
|
|
145
|
+
### 3. iOS retain-cycle (closure capture)
|
|
116
146
|
|
|
117
|
-
|
|
147
|
+
`self.timer = Timer.scheduledTimer(...) { _ in self.tick() }` — rule-based detector flags closure self-capture, fixer rewrites to `[weak self] _ in guard let self = self else { return }; self.tick()` (`suggestFixes.ts:239-242`). If `swiftc` is on PATH the syntax check is real; otherwise test-genie reports "downgraded validation" so you know.
|
|
118
148
|
|
|
119
|
-
|
|
120
|
-
2. **Show Diff**: Display exactly what will change
|
|
121
|
-
3. **Await Confirmation**: Wait for user approval
|
|
122
|
-
4. **Apply Fix**: Only apply after explicit approval
|
|
123
|
-
5. **Backup**: Automatically create backups before applying
|
|
149
|
+
---
|
|
124
150
|
|
|
125
|
-
|
|
151
|
+
## How the iterate-fix loop works
|
|
126
152
|
|
|
127
153
|
```
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
│
|
|
139
|
-
|
|
140
|
-
│
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
│
|
|
147
|
-
|
|
148
|
-
│
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
154
|
+
┌────────────────────┐
|
|
155
|
+
│ collect tests │ (run_scenario_test / supplied list)
|
|
156
|
+
└─────────┬──────────┘
|
|
157
|
+
│
|
|
158
|
+
pass-rate ≥ threshold? ── yes ──▶ SUCCESS
|
|
159
|
+
│ no
|
|
160
|
+
▼
|
|
161
|
+
┌────────────────────┐
|
|
162
|
+
│ detect issues │ memory + logic analyzers
|
|
163
|
+
└─────────┬──────────┘
|
|
164
|
+
│
|
|
165
|
+
┌────────────────────┐
|
|
166
|
+
│ suggest fixes │ rule-based (default) → LLM (hybrid, optional)
|
|
167
|
+
└─────────┬──────────┘
|
|
168
|
+
│
|
|
169
|
+
┌────────────────────┐
|
|
170
|
+
│ dry-run + syntax │ TS compiler API / platform compiler / brace check
|
|
171
|
+
└─────────┬──────────┘
|
|
172
|
+
│
|
|
173
|
+
┌────────────────────┐
|
|
174
|
+
│ apply with backup │ per-file `.test-genie-backups/`
|
|
175
|
+
└─────────┬──────────┘
|
|
176
|
+
│
|
|
177
|
+
┌────────────────────┐
|
|
178
|
+
│ re-run tests │ regression? yes → auto-rollback
|
|
179
|
+
└─────────┬──────────┘
|
|
180
|
+
│
|
|
181
|
+
▼
|
|
182
|
+
loop (≤ maxIterations, ≤ totalTimeout)
|
|
153
183
|
```
|
|
154
184
|
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
|
162
|
-
|
|
163
|
-
| `
|
|
164
|
-
| `
|
|
165
|
-
| `
|
|
166
|
-
| `
|
|
167
|
-
| `
|
|
168
|
-
| `
|
|
169
|
-
| `
|
|
170
|
-
| `
|
|
171
|
-
| `
|
|
172
|
-
| `
|
|
173
|
-
| `
|
|
174
|
-
| `
|
|
175
|
-
| `
|
|
176
|
-
| `
|
|
177
|
-
| `
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
-
|
|
185
|
+
See **[docs/ITERATE_FIX_LOOP.md](docs/ITERATE_FIX_LOOP.md)** for a sequence diagram and the full safety-guard list.
|
|
186
|
+
|
|
187
|
+
---
|
|
188
|
+
|
|
189
|
+
## Tools (23)
|
|
190
|
+
|
|
191
|
+
| # | Tool | Mode |
|
|
192
|
+
|---|------|------|
|
|
193
|
+
| 1 | `analyze_app_structure` | real |
|
|
194
|
+
| 2 | `generate_scenarios` | real |
|
|
195
|
+
| 3 | `create_test_plan` | real |
|
|
196
|
+
| 4 | `run_scenario_test` | hybrid |
|
|
197
|
+
| 5 | `run_simulation` | simulated |
|
|
198
|
+
| 6 | `run_stress_test` | hybrid |
|
|
199
|
+
| 7 | `detect_memory_leaks` | real |
|
|
200
|
+
| 8 | `detect_logic_errors` | real |
|
|
201
|
+
| 9 | `suggest_fixes` | real |
|
|
202
|
+
| 10 | `confirm_fix` | real |
|
|
203
|
+
| 11 | `apply_fix` | real |
|
|
204
|
+
| 12 | `rollback_fix` | real |
|
|
205
|
+
| 13 | `run_full_automation` | hybrid |
|
|
206
|
+
| 14 | `run_iterative_fix_loop` (v3.0 headline) | hybrid |
|
|
207
|
+
| 15 | `generate_report` | real |
|
|
208
|
+
| 16 | `get_pending_fixes` | real |
|
|
209
|
+
| 17 | `get_test_history` | real |
|
|
210
|
+
| 18 | `analyze_performance` | real |
|
|
211
|
+
| 19 | `analyze_code_deep` | real |
|
|
212
|
+
| 20 | `generate_cicd_config` | real |
|
|
213
|
+
| 21 | **`diagnose_project`** (v3.1 headline — vibe-check) | real |
|
|
214
|
+
| 22 | `detect_race_conditions` | real |
|
|
215
|
+
| 23 | `detect_security_issues` | real |
|
|
216
|
+
|
|
217
|
+
`mode` legend in **[docs/SIMULATION_VS_REAL.md](docs/SIMULATION_VS_REAL.md)**.
|
|
218
|
+
|
|
219
|
+
Plus 4 resources (`test-genie://iteration-logs`, `…/test-history/{path}`, `…/iteration-logs/{loopId}`, `…/applied-fixes/{path}`) and 3 prompts (`full-test-pipeline`, `diagnose-failure`, `vibe-check`).
|
|
220
|
+
|
|
221
|
+
---
|
|
222
|
+
|
|
223
|
+
## What vibe-check catches
|
|
224
|
+
|
|
225
|
+
Race conditions (`detect_race_conditions` / `diagnose_project`):
|
|
226
|
+
|
|
227
|
+
| Pattern | Language | Severity | Auto-fixable (v3.1.1) |
|
|
228
|
+
|---|---|---|---|
|
|
229
|
+
| `useState` setter called after `await` without mount guard | TS/JS/React | high | no (structural) |
|
|
230
|
+
| `useEffect` with async fetch, no AbortController/cleanup | TS/JS/React | high | no (structural) |
|
|
231
|
+
| `arr.forEach(async ...)` (silent fire-and-forget) | TS/JS | medium | no (ordering-sensitive) |
|
|
232
|
+
| Adjacent fetches without `Promise.all` / sequencing | TS/JS | medium | no |
|
|
233
|
+
| TOCTOU: `existsSync` then `readFileSync` without lock | TS/JS Node | medium | no |
|
|
234
|
+
| Non-atomic counter increment in async context | TS/JS | low | no |
|
|
235
|
+
| `@Published` mutation outside `@MainActor` | Swift | medium | no |
|
|
236
|
+
| Concurrent `DispatchQueue` writes without `.barrier` | Swift | medium | no |
|
|
237
|
+
| `MutableStateFlow` mutated off `Dispatchers.Main` | Kotlin | medium | no |
|
|
238
|
+
| `Flow` collected without `flowOn` | Kotlin | low | no |
|
|
239
|
+
| Goroutine + shared map without `sync.Mutex` | Go | high | no |
|
|
240
|
+
|
|
241
|
+
> v3.1.1 honesty audit: `useEffect-no-abort` and `forEach-await` were
|
|
242
|
+
> previously advertised as auto-fixable. They are not — wrapping with
|
|
243
|
+
> `AbortController` or rewriting to `Promise.all(arr.map(...))` changes
|
|
244
|
+
> behavior we can't verify statically. They are now report-only. See
|
|
245
|
+
> [SAFETY.md](SAFETY.md).
|
|
246
|
+
|
|
247
|
+
Security (`detect_security_issues` / `diagnose_project`):
|
|
248
|
+
|
|
249
|
+
| Pattern | Severity | CWE | Auto-fixable (v3.1.1) |
|
|
250
|
+
|---|---|---|---|
|
|
251
|
+
| Hardcoded AWS / Stripe / GitHub / Google / Slack token | critical / high | CWE-798 | no (rotate) |
|
|
252
|
+
| Hardcoded JWT secret literal | high | CWE-798 | no |
|
|
253
|
+
| API token in URL query string | high | CWE-200 | no |
|
|
254
|
+
| `.env` file present but not gitignored | high | CWE-538 | no (rotation must follow) |
|
|
255
|
+
| SQL string concat with `req.params` / `req.body` | critical | CWE-89 | no |
|
|
256
|
+
| `innerHTML` / `dangerouslySetInnerHTML` with dynamic value | high | CWE-79 | no |
|
|
257
|
+
| `eval()` / `new Function()` with non-literal | critical | CWE-95 | no |
|
|
258
|
+
| `Math.random()` in security-sensitive file, **standalone assignment** | high | CWE-338 | **yes** (`crypto.randomInt`) |
|
|
259
|
+
| `Math.random()` mixed into arithmetic | high | CWE-338 | no (semantic) |
|
|
260
|
+
| `createHash('md5'\|'sha1')` in security-keyword file | high | CWE-327 | **yes** (`'sha256'`) |
|
|
261
|
+
| `createHash('md5'\|'sha1')` elsewhere | medium | CWE-327 | no (below severity floor) |
|
|
262
|
+
| `child_process.exec` with user-input template literal | critical | CWE-78 | no |
|
|
263
|
+
| `fetch(req.query.url)` (SSRF) | high | CWE-918 | no |
|
|
264
|
+
| CORS `*` origin + `Allow-Credentials: true` | high | CWE-942 | no |
|
|
265
|
+
| Cookie set without `httpOnly` / `secure` / `sameSite` | low | CWE-1004 | no |
|
|
266
|
+
| `yaml.load` without safe schema | medium | CWE-502 | no |
|
|
267
|
+
|
|
268
|
+
> v3.1.1 honesty audit: `.env`/`Math.random` (general)/`yaml.load` were
|
|
269
|
+
> previously advertised as auto-fixable. They were either too risky to
|
|
270
|
+
> rewrite blindly or no strategy shipped — flipped to report-only. See
|
|
271
|
+
> [SAFETY.md](SAFETY.md) §5.
|
|
272
|
+
|
|
273
|
+
---
|
|
274
|
+
|
|
275
|
+
## What vibe-check misses (honest list)
|
|
276
|
+
|
|
277
|
+
This is a "catch the obvious stuff in 30s" filter, not Snyk / Semgrep / a full SAST tool. We don't catch:
|
|
278
|
+
|
|
279
|
+
- **Cross-file data-flow.** If user input flows through three files before reaching a `db.query`, the regex won't connect the dots. A real SAST traces taint across the call graph. Roadmap: ts-morph reference walking for top-N entry points.
|
|
280
|
+
- **Vulnerable transitive deps.** We don't query npm advisories — that's `npm audit`'s job, and bundling a stale advisory list would lie. Run `npm audit --json` in parallel if you want dep-CVE coverage.
|
|
281
|
+
- **Race conditions across processes.** We catch in-process JS / Swift / Kotlin / Go races. Distributed races (lock ordering across services, DB transactions) need different tooling.
|
|
282
|
+
- **Type-correct but logic-broken code.** The analyzer is syntactic, not semantic. A `Math.random()` named `getNonce` won't fool us; a properly-named `crypto.randomBytes` used with a tiny entropy budget will.
|
|
283
|
+
- **Custom secret formats.** Internal company tokens with unique prefixes need a regex you can add to `securityAnalyzer.SECRET_PATTERNS`. PR welcome.
|
|
284
|
+
- **Real-time / dynamic issues.** Memory leaks under load, network timeouts, slow renders mid-interaction — those need `run_stress_test` / `run_simulation`, not static analysis.
|
|
285
|
+
|
|
286
|
+
If you want deeper coverage on top of vibe-check: feed the findings into `run_iterative_fix_loop` for test-verified application, or escalate to Snyk / Semgrep / GitHub Advanced Security for compliance use cases.
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## vibe-check vs alternatives
|
|
291
|
+
|
|
292
|
+
| | vibe-check (test-genie) | Snyk | Semgrep | GitHub Advanced Security |
|
|
293
|
+
|------------------------|-------------------------|------|---------|--------------------------|
|
|
294
|
+
| Runs locally | yes | hybrid (cloud) | yes | no (cloud) |
|
|
295
|
+
| Telemetry-free | yes (zero network calls) | no | partial | no |
|
|
296
|
+
| Fix loop integration | yes (`run_iterative_fix_loop`) | no | no | no |
|
|
297
|
+
| Race-condition detection | yes (JS/Swift/Kotlin/Go) | no | partial | partial |
|
|
298
|
+
| Cross-file taint flow | no (roadmap) | yes | yes | yes |
|
|
299
|
+
| Setup time | none (already installed if test-genie is installed) | account + auth | install + ruleset | repo-level enable |
|
|
212
300
|
|
|
213
|
-
|
|
301
|
+
If your goal is "before I commit, what's broken?", vibe-check wins on latency. If your goal is "compliance + supply chain audit", use the dedicated tools.
|
|
302
|
+
|
|
303
|
+
---
|
|
304
|
+
|
|
305
|
+
## When NOT to use test-genie
|
|
306
|
+
|
|
307
|
+
- **Production-gate test runs.** test-genie is built for the *development* feedback loop. For shipping decisions, use a proper CI that you control end-to-end.
|
|
308
|
+
- **Code your team must hand-review every line of.** The loop's job is to *propose and apply* fixes; if every fix needs a human eye, leave `autoApply: false` (the default) and use it as a fix-proposal generator only.
|
|
309
|
+
- **No backup / no version control situations.** test-genie's auto-rollback is best-effort and requires the per-file backup to exist. Always run inside a git working tree.
|
|
310
|
+
|
|
311
|
+
---
|
|
312
|
+
|
|
313
|
+
## Comparison
|
|
314
|
+
|
|
315
|
+
| | test-genie | Detox | Maestro | xcodebuild test |
|
|
316
|
+
|---|---|---|---|---|
|
|
317
|
+
| Runs E2E / unit tests | ✅ (via Jest/Detox/etc.) | ✅ | ✅ | ✅ |
|
|
318
|
+
| Detects code issues | ✅ rule + LLM | ❌ | ❌ | ❌ |
|
|
319
|
+
| **Iterative fix loop** | **✅** (`run_iterative_fix_loop`) | ❌ | ❌ | ❌ |
|
|
320
|
+
| Auto-rollback on test regression | ✅ inside `run_iterative_fix_loop` only | ❌ | ❌ | ❌ |
|
|
321
|
+
| Auto-rollback on syntax failure | ✅ all apply paths | ❌ | ❌ | ❌ |
|
|
322
|
+
| MCP-native (talks to Claude / agents) | ✅ | ❌ | ❌ | ❌ |
|
|
323
|
+
| Multi-platform | iOS+Android+Web+Flutter+RN | iOS+Android | iOS+Android | iOS only |
|
|
324
|
+
|
|
325
|
+
> Scope note: `diagnose_project autoFix: true` rolls back on syntax-validate
|
|
326
|
+
> failure (`applyFix.ts:185-202`) but does **not** re-run tests, so it
|
|
327
|
+
> cannot detect test regressions. For test-driven rollback use
|
|
328
|
+
> `run_iterative_fix_loop`. See [SAFETY.md](SAFETY.md) §2.4.
|
|
329
|
+
|
|
330
|
+
test-genie *uses* tools like Jest, Detox, and `xcodebuild test` under the hood — it sits at the orchestration layer, not the test-runner layer.
|
|
331
|
+
|
|
332
|
+
---
|
|
333
|
+
|
|
334
|
+
## Known limitations
|
|
335
|
+
|
|
336
|
+
- **Platform syntax check downgrade.** For Swift/Kotlin/Java/Dart we try the platform compiler in `-typecheck` mode. If the compiler isn't on PATH, we fall back to brace-balance validation and surface `downgraded: true` in the result. Install `swiftc` / `kotlinc` / `javac` / `dart` for real validation.
|
|
337
|
+
- **LLM is optional and gated.** `strategy: 'hybrid'` only kicks LLM in when rule-based confidence is below threshold. Without an API key the loop is rule-based-only — no failure.
|
|
338
|
+
- **Storage is per-machine.** Test history / iteration logs live under `$TEST_GENIE_STORAGE_DIR` (defaults to `~/.test-genie-mcp`). Not synced across machines.
|
|
339
|
+
- **Simulated mode is "simulation," not magic.** `run_simulation` returns *plausible* anomalies, not real ones. Use `run_scenario_test` (hybrid) for real-device runs.
|
|
340
|
+
|
|
341
|
+
---
|
|
214
342
|
|
|
215
|
-
|
|
343
|
+
## Configuration
|
|
216
344
|
|
|
217
|
-
|
|
345
|
+
| Env var | Default | Purpose |
|
|
346
|
+
|---|---|---|
|
|
347
|
+
| `TEST_GENIE_ALLOWED_ROOT` | `cwd` | Capability-based path safety — server refuses to read/write outside this root. |
|
|
348
|
+
| `TEST_GENIE_STORAGE_DIR` | `~/.test-genie-mcp` | Where scenarios / results / iteration logs live. |
|
|
349
|
+
| `TEST_GENIE_LLM_PROVIDER` | auto-detect | `anthropic` / `openai` / `none`. |
|
|
350
|
+
| `ANTHROPIC_API_KEY` | — | Used when provider = `anthropic`. |
|
|
351
|
+
| `OPENAI_API_KEY` | — | Used when provider = `openai`. |
|
|
352
|
+
| `TEST_GENIE_ANTHROPIC_MODEL` | `claude-haiku-4-5` | Override Anthropic model. |
|
|
353
|
+
| `TEST_GENIE_OPENAI_MODEL` | `gpt-4o-mini` | Override OpenAI model. |
|
|
354
|
+
|
|
355
|
+
---
|
|
356
|
+
|
|
357
|
+
## Migrating from v2.x
|
|
358
|
+
|
|
359
|
+
- `run_full_automation` still works. The `confirmMode` / `autoFix` options are kept for compatibility but **`autoApply: boolean` is the new way** — `autoApply: true` is equivalent to `confirmMode: 'auto'`.
|
|
360
|
+
- Subprocess hardening means platform tools now reject scheme / device / package-name arguments that contain shell metacharacters. If your CI was passing weird-looking values, sanitize them first.
|
|
361
|
+
- See **[CHANGELOG.md](CHANGELOG.md)** for the full breaking-change list + migration recipes.
|
|
362
|
+
|
|
363
|
+
---
|
|
364
|
+
|
|
365
|
+
## Roadmap
|
|
366
|
+
|
|
367
|
+
- LLM-based fix-proposal **voting** (multiple proposals → pick the best by syntax + retest delta)
|
|
368
|
+
- Multi-repo sync (run the loop across N repos in parallel from one MCP call)
|
|
369
|
+
- A "watch mode" that runs the loop on file save
|
|
370
|
+
- Better Detox / Maestro artifact ingestion (link videos into iteration logs)
|
|
371
|
+
|
|
372
|
+
---
|
|
373
|
+
|
|
374
|
+
## Contributing
|
|
375
|
+
|
|
376
|
+
Issues, PRs, and ideas welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)** (TODO). Code lives under `src/`, tests under `tests/`. Run `npm test` before sending a PR.
|
|
377
|
+
|
|
378
|
+
## Maintainer
|
|
379
|
+
|
|
380
|
+
[@MUSE-CODE-SPACE](https://github.com/MUSE-CODE-SPACE) — Yoonkyoung Gong.
|
|
381
|
+
|
|
382
|
+
## License
|
|
218
383
|
|
|
219
|
-
|
|
384
|
+
MIT — see [LICENSE](LICENSE).
|