agentic-loop 3.6.2 → 3.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/prd.md +95 -37
- package/README.md +0 -18
- package/package.json +1 -1
- package/ralph/init.sh +2 -0
- package/ralph/loop.sh +37 -1
- package/ralph/utils.sh +41 -1
- package/ralph/verify/api.sh +279 -0
- package/ralph/verify/lint.sh +43 -0
- package/ralph/verify/tests.sh +2 -0
- package/ralph/verify.sh +22 -3
- package/templates/config/elixir.json +84 -0
- package/templates/config/fullstack.json +5 -0
- package/templates/config/node.json +6 -0
- package/templates/config/python.json +1 -0
package/.claude/commands/prd.md
CHANGED
|
@@ -99,31 +99,83 @@ Break the idea into small, executable stories:
|
|
|
99
99
|
|
|
100
100
|
- Each story completable in one Claude session (~10-15 min)
|
|
101
101
|
- Max 3-4 acceptance criteria per story
|
|
102
|
-
- Order by dependency
|
|
103
102
|
- Max 10 stories (suggest phases if more needed)
|
|
104
103
|
- If appending, start IDs from the next available number
|
|
105
104
|
|
|
106
|
-
### Step 5: Write PRD
|
|
105
|
+
### Step 5: Write Draft PRD
|
|
107
106
|
|
|
108
|
-
|
|
107
|
+
Write the initial PRD to `.ralph/prd.json`:
|
|
108
|
+
|
|
109
|
+
1. Ensure .ralph directory exists:
|
|
109
110
|
```bash
|
|
110
111
|
mkdir -p .ralph && touch .ralph/.prd-edit-allowed
|
|
111
112
|
```
|
|
112
113
|
|
|
113
|
-
2. Write to `.ralph/prd.json
|
|
114
|
-
- If **
|
|
115
|
-
|
|
114
|
+
2. Write all stories to `.ralph/prd.json`
|
|
115
|
+
- If **appending**: Read existing JSON, add new stories, update count
|
|
116
|
+
|
|
117
|
+
**Do not present to user yet - validation comes next.**
|
|
118
|
+
|
|
119
|
+
### Step 6: Validate and Fix (MANDATORY)
|
|
120
|
+
|
|
121
|
+
**Read back the PRD you just wrote and validate EVERY story.**
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
cat .ralph/prd.json
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
For EACH story, ask yourself:
|
|
128
|
+
|
|
129
|
+
1. **"Is this testable?"** - Can the testSteps actually run?
|
|
130
|
+
- ❌ `grep -q 'function' file.py` → Only checks code exists, not behavior
|
|
131
|
+
- ❌ `test -f src/component.tsx` → Only checks file exists
|
|
132
|
+
- ❌ "Visit the page and verify" → Not executable
|
|
133
|
+
- ✅ `curl ... | jq -e` → Tests actual API response
|
|
134
|
+
- ✅ `npm test` / `pytest` → Runs real tests
|
|
135
|
+
- ✅ `npx playwright test` → Runs real tests
|
|
136
|
+
|
|
137
|
+
2. **"Is this passable?"** - Given prior stories completed, can this story's tests pass?
|
|
138
|
+
- If TASK-003 needs a user to exist, does TASK-001 or TASK-002 create one?
|
|
139
|
+
- If TASK-004 tests a login flow, does a prior story create the auth endpoint?
|
|
140
|
+
|
|
141
|
+
**Fix any issues you find:**
|
|
142
|
+
|
|
143
|
+
| Problem | Fix |
|
|
144
|
+
|---------|-----|
|
|
145
|
+
| testSteps use grep/test only | Replace with curl, pytest, npm test, playwright |
|
|
146
|
+
| Story depends on something not yet created | Reorder stories or add missing dependency story |
|
|
147
|
+
| testSteps would pass on current code | Strengthen tests to verify NEW behavior |
|
|
148
|
+
| No testSteps for backend story | Add `curl -s {config.urls.backend}/endpoint \| jq -e '.field'` |
|
|
149
|
+
| No testSteps for frontend story | Add `npx tsc --noEmit` + `npm test` |
|
|
116
150
|
|
|
117
|
-
|
|
151
|
+
### Step 7: Reorder if Needed
|
|
118
152
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
153
|
+
If validation found dependency issues, reorder stories:
|
|
154
|
+
|
|
155
|
+
1. Stories that create foundations (DB schemas, base components) come first
|
|
156
|
+
2. Stories that depend on others come after their dependencies
|
|
157
|
+
3. Update `dependsOn` arrays to reflect the order
|
|
158
|
+
4. Re-number story IDs if needed (TASK-001, TASK-002, etc.)
|
|
159
|
+
|
|
160
|
+
**After reordering, re-run Step 6 validation to confirm the new order works.**
|
|
161
|
+
|
|
162
|
+
### Step 8: Present Final PRD
|
|
163
|
+
|
|
164
|
+
Open the PRD for review:
|
|
165
|
+
```bash
|
|
166
|
+
open -a TextEdit .ralph/prd.json
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Say: "I've {created|updated} the PRD with {N} stories and opened it in TextEdit.
|
|
170
|
+
|
|
171
|
+
Review the PRD and let me know:
|
|
172
|
+
- **'approved'** - Ready for `ralph run`
|
|
173
|
+
- **'edit [changes]'** - Tell me what to change
|
|
174
|
+
- Or edit the JSON directly and say **'done'**"
|
|
123
175
|
|
|
124
176
|
**STOP and wait for user response.**
|
|
125
177
|
|
|
126
|
-
### Step
|
|
178
|
+
### Step 9: Final Instructions
|
|
127
179
|
|
|
128
180
|
Once approved, say:
|
|
129
181
|
|
|
@@ -232,7 +284,8 @@ Ralph will work through each story, running tests and committing as it goes."
|
|
|
232
284
|
},
|
|
233
285
|
|
|
234
286
|
"testSteps": [
|
|
235
|
-
"
|
|
287
|
+
"curl -s {config.urls.backend}/endpoint | jq -e '.expected == true'",
|
|
288
|
+
"npx playwright test tests/e2e/feature.spec.ts"
|
|
236
289
|
],
|
|
237
290
|
|
|
238
291
|
"testUrl": "{config.urls.frontend}/feature-page",
|
|
@@ -412,19 +465,6 @@ Example for a Dashboard component:
|
|
|
412
465
|
|
|
413
466
|
### Testing Anti-Patterns (AVOID THESE)
|
|
414
467
|
|
|
415
|
-
**The "grep for code" trap:**
|
|
416
|
-
```json
|
|
417
|
-
// ❌ BAD - verifies code exists, not that it works
|
|
418
|
-
"testSteps": [
|
|
419
|
-
"grep -q 'astream_events' app/domains/chat/agent/graph.py"
|
|
420
|
-
]
|
|
421
|
-
|
|
422
|
-
// ✅ GOOD - verifies actual behavior
|
|
423
|
-
"testSteps": [
|
|
424
|
-
"curl -N {config.urls.backend}/chat -d '{\"message\":\"test\"}' | grep -q 'progress'"
|
|
425
|
-
]
|
|
426
|
-
```
|
|
427
|
-
|
|
428
468
|
**Missing integration points:**
|
|
429
469
|
```json
|
|
430
470
|
// ❌ BAD - creates function but doesn't verify callers use it
|
|
@@ -443,6 +483,8 @@ Example for a Dashboard component:
|
|
|
443
483
|
}
|
|
444
484
|
```
|
|
445
485
|
|
|
486
|
+
**(See "The Grep for Code Trap" section above for the #1 anti-pattern)**
|
|
487
|
+
|
|
446
488
|
### Removing/Modifying UI - Update Tests!
|
|
447
489
|
|
|
448
490
|
**CRITICAL: When a story removes or modifies UI elements, it MUST update related tests.**
|
|
@@ -572,8 +614,28 @@ Example:
|
|
|
572
614
|
|
|
573
615
|
## Test Steps - CRITICAL
|
|
574
616
|
|
|
617
|
+
⚠️ **THE #1 CAUSE OF FALSE PASSES: grep-only test steps that verify code exists but not behavior.**
|
|
618
|
+
|
|
575
619
|
**Test steps MUST be executable shell commands.** Ralph runs them with bash.
|
|
576
620
|
|
|
621
|
+
### The "Grep for Code" Trap - NEVER DO THIS
|
|
622
|
+
|
|
623
|
+
```json
|
|
624
|
+
// ❌ BAD - This will PASS even when the feature is completely broken!
|
|
625
|
+
"testSteps": [
|
|
626
|
+
"grep -q 'astream_events' app/domains/chat/agent/graph.py",
|
|
627
|
+
"grep -q 'export function' src/api/users.ts"
|
|
628
|
+
]
|
|
629
|
+
|
|
630
|
+
// ✅ GOOD - This actually tests if the feature works
|
|
631
|
+
"testSteps": [
|
|
632
|
+
"curl -N {config.urls.backend}/chat -d '{\"message\":\"test\"}' | grep -q 'progress'",
|
|
633
|
+
"curl -s {config.urls.backend}/users | jq -e '.data | length >= 0'"
|
|
634
|
+
]
|
|
635
|
+
```
|
|
636
|
+
|
|
637
|
+
**Why is grep bad?** Ralph runs `grep -q 'function' file.py` → returns 0 → marks story as PASSED. But the function could be completely broken, have wrong parameters, or never get called. The test passed but the feature doesn't work.
|
|
638
|
+
|
|
577
639
|
### Backend Stories MUST Have Curl Tests
|
|
578
640
|
|
|
579
641
|
**CRITICAL: Every backend story MUST include curl commands that verify actual API behavior.**
|
|
@@ -591,15 +653,7 @@ Use `{config.urls.backend}` - Ralph expands this from `.ralph/config.json`:
|
|
|
591
653
|
|
|
592
654
|
Ralph reads `.ralph/config.json` and expands `{config.urls.backend}` before running.
|
|
593
655
|
|
|
594
|
-
**Why?** Grep tests verify code exists. Curl tests verify the feature works.
|
|
595
|
-
|
|
596
|
-
```json
|
|
597
|
-
// ❌ NEVER DO THIS for backend stories
|
|
598
|
-
"testSteps": [
|
|
599
|
-
"grep -q 'astream_events' app/domains/chat/agent/graph.py"
|
|
600
|
-
]
|
|
601
|
-
// This passed but the feature was broken!
|
|
602
|
-
```
|
|
656
|
+
**Why?** Grep tests verify code exists. Curl tests verify the feature works. (See "The Grep for Code Trap" above.)
|
|
603
657
|
|
|
604
658
|
### Test Steps by Story Type
|
|
605
659
|
|
|
@@ -640,15 +694,19 @@ Ralph reads `.ralph/config.json` and expands `{config.urls.backend}` before runn
|
|
|
640
694
|
]
|
|
641
695
|
```
|
|
642
696
|
|
|
643
|
-
### Bad Test Steps (will
|
|
697
|
+
### Bad Test Steps (will PASS but miss bugs)
|
|
644
698
|
```json
|
|
645
699
|
"testSteps": [
|
|
646
|
-
"grep -q 'function createUser' app/services/user.py", // ❌
|
|
700
|
+
"grep -q 'function createUser' app/services/user.py", // ❌ PASSES if code exists, even if broken
|
|
701
|
+
"grep -q 'export default' src/components/Dashboard.tsx", // ❌ PASSES even if component crashes
|
|
702
|
+
"test -f src/api/users.ts", // ❌ PASSES if file exists, even if empty
|
|
647
703
|
"Visit http://localhost:3000/dashboard", // ❌ Not executable
|
|
648
704
|
"User can see the dashboard" // ❌ Not executable
|
|
649
705
|
]
|
|
650
706
|
```
|
|
651
707
|
|
|
708
|
+
**NEVER use grep/test to verify behavior.** These will mark stories as PASSED when the feature is broken.
|
|
709
|
+
|
|
652
710
|
**If a step can't be automated**, put it in `acceptanceCriteria` instead. Claude will verify it visually using MCP tools.
|
|
653
711
|
|
|
654
712
|
---
|
package/README.md
CHANGED
|
@@ -8,24 +8,6 @@ You describe what you want to build. Claude Code writes a PRD (Product Requireme
|
|
|
8
8
|
|
|
9
9
|
---
|
|
10
10
|
|
|
11
|
-
## Supported Project Types
|
|
12
|
-
|
|
13
|
-
Ralph auto-detects your project type and configures itself accordingly:
|
|
14
|
-
|
|
15
|
-
| Type | Detection | Auto-Configured |
|
|
16
|
-
|------|-----------|-----------------|
|
|
17
|
-
| **FastMCP** | `fastmcp` in pyproject.toml | Server module, MCP port, transport, subprojects |
|
|
18
|
-
| **FastAPI** | `fastapi` in pyproject.toml | uvicorn dev server, pytest, ruff |
|
|
19
|
-
| **Django** | `django` in pyproject.toml or manage.py | migrations, pytest, ruff |
|
|
20
|
-
| **Python** | pyproject.toml or requirements.txt | pytest, ruff, uv/poetry detection |
|
|
21
|
-
| **Node.js** | package.json | npm/yarn/pnpm, vitest/jest, eslint |
|
|
22
|
-
| **React** | `react` in package.json | Vite/Next.js, TypeScript, Tailwind |
|
|
23
|
-
| **Go/Hugo** | go.mod or hugo.toml | Hugo server, Go build |
|
|
24
|
-
| **Rust** | Cargo.toml | cargo build/test/clippy |
|
|
25
|
-
| **Fullstack** | frontend + backend directories | Monorepo support, separate configs |
|
|
26
|
-
|
|
27
|
-
---
|
|
28
|
-
|
|
29
11
|
## What It Does
|
|
30
12
|
|
|
31
13
|
**Brainstorm ideas with `/idea`**
|
package/package.json
CHANGED
package/ralph/init.sh
CHANGED
|
@@ -145,6 +145,8 @@ detect_project_type() {
|
|
|
145
145
|
project_type="rust"
|
|
146
146
|
elif [[ -f "go.mod" ]]; then
|
|
147
147
|
project_type="go"
|
|
148
|
+
elif [[ -f "mix.exs" ]]; then
|
|
149
|
+
project_type="elixir"
|
|
148
150
|
# Check for Python framework variants (more specific first)
|
|
149
151
|
elif [[ -f "pyproject.toml" ]]; then
|
|
150
152
|
# FastMCP detection (check for fastmcp in any quote style)
|
package/ralph/loop.sh
CHANGED
|
@@ -336,7 +336,43 @@ run_loop() {
|
|
|
336
336
|
claude_args=(--continue "${claude_args[@]}")
|
|
337
337
|
fi
|
|
338
338
|
|
|
339
|
-
|
|
339
|
+
# Run Claude with crash detection and retry logic
|
|
340
|
+
local claude_output_log claude_exit_code max_crash_retries=3 crash_attempt=0
|
|
341
|
+
claude_output_log=$(create_temp_file ".log") || { rm -f "$prompt_file"; return 1; }
|
|
342
|
+
|
|
343
|
+
while [[ $crash_attempt -lt $max_crash_retries ]]; do
|
|
344
|
+
claude_exit_code=0
|
|
345
|
+
# Use pipefail to capture Claude's exit code, not tee's
|
|
346
|
+
set -o pipefail
|
|
347
|
+
cat "$prompt_file" | run_with_timeout "$timeout_seconds" claude "${claude_args[@]}" 2>&1 | tee "$claude_output_log" || claude_exit_code=$?
|
|
348
|
+
set +o pipefail
|
|
349
|
+
|
|
350
|
+
# Check for recoverable CLI crashes
|
|
351
|
+
if grep -qE "(No messages returned|unhandled.*promise.*rejection)" "$claude_output_log" 2>/dev/null; then
|
|
352
|
+
((crash_attempt++))
|
|
353
|
+
print_warning "Claude CLI crashed (attempt $crash_attempt/$max_crash_retries) - retrying..."
|
|
354
|
+
log_progress "$story" "CLI_CRASH" "Claude crashed, retry $crash_attempt"
|
|
355
|
+
session_started=false # Reset session on crash
|
|
356
|
+
sleep 2 # Brief pause before retry
|
|
357
|
+
continue
|
|
358
|
+
fi
|
|
359
|
+
|
|
360
|
+
# Not a crash - exit retry loop
|
|
361
|
+
break
|
|
362
|
+
done
|
|
363
|
+
|
|
364
|
+
rm -f "$claude_output_log"
|
|
365
|
+
|
|
366
|
+
if [[ $crash_attempt -ge $max_crash_retries ]]; then
|
|
367
|
+
print_error "Claude CLI crashed $max_crash_retries times - stopping loop"
|
|
368
|
+
log_progress "$story" "CLI_CRASH" "Gave up after $max_crash_retries crashes"
|
|
369
|
+
rm -f "$prompt_file"
|
|
370
|
+
echo ""
|
|
371
|
+
echo "Claude CLI is unstable. Try again with: ralph run $story"
|
|
372
|
+
return 1
|
|
373
|
+
fi
|
|
374
|
+
|
|
375
|
+
if [[ $claude_exit_code -ne 0 ]]; then
|
|
340
376
|
print_warning "Claude session ended (timeout or error)"
|
|
341
377
|
log_progress "$story" "TIMEOUT" "Claude session ended after ${timeout_seconds}s"
|
|
342
378
|
rm -f "$prompt_file"
|
package/ralph/utils.sh
CHANGED
|
@@ -181,6 +181,7 @@ run_with_timeout() {
|
|
|
181
181
|
fi
|
|
182
182
|
}
|
|
183
183
|
|
|
184
|
+
|
|
184
185
|
# Safely update JSON file atomically
|
|
185
186
|
# Usage: update_json <file> [jq args...] <filter>
|
|
186
187
|
# Example: update_json file.json --arg id "TASK-001" '.stories[] | select(.id==$id)'
|
|
@@ -191,12 +192,25 @@ update_json() {
|
|
|
191
192
|
tmpfile=$(mktemp)
|
|
192
193
|
lockdir="${file}.lock"
|
|
193
194
|
|
|
195
|
+
# Remove stale locks (from crashed processes)
|
|
196
|
+
if [[ -d "$lockdir" ]]; then
|
|
197
|
+
local lock_age=0
|
|
198
|
+
local now=$(date +%s)
|
|
199
|
+
# Cross-platform: macOS uses -f %m, Linux uses -c %Y
|
|
200
|
+
local lock_mtime=$(stat -f %m "$lockdir" 2>/dev/null || stat -c %Y "$lockdir" 2>/dev/null || echo "$now")
|
|
201
|
+
lock_age=$((now - lock_mtime))
|
|
202
|
+
if [[ $lock_age -gt 30 ]]; then
|
|
203
|
+
print_warning "Removing stale lock (${lock_age}s old): $lockdir"
|
|
204
|
+
rm -rf "$lockdir"
|
|
205
|
+
fi
|
|
206
|
+
fi
|
|
207
|
+
|
|
194
208
|
# Acquire lock (mkdir is atomic)
|
|
195
209
|
local attempts=0
|
|
196
210
|
while ! mkdir "$lockdir" 2>/dev/null; do
|
|
197
211
|
((attempts++))
|
|
198
212
|
if [[ $attempts -gt 50 ]]; then
|
|
199
|
-
print_error "Could not acquire lock on $file"
|
|
213
|
+
print_error "Could not acquire lock on $file (locked for 5s+)"
|
|
200
214
|
rm -f "$tmpfile"
|
|
201
215
|
return 1
|
|
202
216
|
fi
|
|
@@ -495,6 +509,26 @@ validate_prd() {
|
|
|
495
509
|
print_warning "PRD is missing feature name (will show as 'unnamed')"
|
|
496
510
|
fi
|
|
497
511
|
|
|
512
|
+
# Check for grep-only testSteps (the #1 cause of false passes)
|
|
513
|
+
# Matches: grep, test -f/-e/-d, [ -f file ], [[ -f file ]]
|
|
514
|
+
local grep_only_stories
|
|
515
|
+
grep_only_stories=$(jq -r '
|
|
516
|
+
.stories[] |
|
|
517
|
+
select(.testSteps != null and (.testSteps | length > 0)) |
|
|
518
|
+
select(.testSteps | all(test("^(grep|test\\s+-[fed]|\\[\\[?\\s+-[fed])"; "x"))) |
|
|
519
|
+
.id
|
|
520
|
+
' "$prd_file" 2>/dev/null)
|
|
521
|
+
|
|
522
|
+
if [[ -n "$grep_only_stories" ]]; then
|
|
523
|
+
print_warning "These stories have grep-only testSteps (may cause false passes):"
|
|
524
|
+
echo "$grep_only_stories" | while read -r story_id; do
|
|
525
|
+
[[ -n "$story_id" ]] && echo " - $story_id"
|
|
526
|
+
done
|
|
527
|
+
echo ""
|
|
528
|
+
echo "Grep verifies code exists, not that it works. Add curl/playwright tests."
|
|
529
|
+
echo ""
|
|
530
|
+
fi
|
|
531
|
+
|
|
498
532
|
return 0
|
|
499
533
|
}
|
|
500
534
|
|
|
@@ -537,6 +571,12 @@ detect_migration_tool() {
|
|
|
537
571
|
return 0
|
|
538
572
|
fi
|
|
539
573
|
|
|
574
|
+
# Ecto (Elixir/Phoenix)
|
|
575
|
+
if [[ -f "$search_dir/mix.exs" ]] && [[ -d "$search_dir/priv/repo/migrations" ]]; then
|
|
576
|
+
echo "cd $search_dir && mix ecto.migrate"
|
|
577
|
+
return 0
|
|
578
|
+
fi
|
|
579
|
+
|
|
540
580
|
# Prisma (Node.js)
|
|
541
581
|
if [[ -d "$search_dir/prisma/migrations" ]] || [[ -f "$search_dir/prisma/schema.prisma" ]]; then
|
|
542
582
|
echo "cd $search_dir && npx prisma migrate deploy"
|
|
@@ -0,0 +1,279 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# shellcheck shell=bash
|
|
3
|
+
# api.sh - API and frontend smoke test verification module for ralph
|
|
4
|
+
#
|
|
5
|
+
# Catches broken APIs/frontends that unit tests miss (because they mock everything).
|
|
6
|
+
# Uses config.json for endpoints - no project-specific hardcoding.
|
|
7
|
+
|
|
8
|
+
# Run API smoke test against configured endpoints
|
|
9
|
+
# Config options:
|
|
10
|
+
# api.baseUrl - Base URL (e.g., http://localhost:8001)
|
|
11
|
+
# api.healthEndpoint - Health check path (e.g., /health)
|
|
12
|
+
# api.smokeEndpoints - Array of paths to test (e.g., ["/api/v1/users", "/api/v1/items"])
|
|
13
|
+
#
|
|
14
|
+
# Also tests story-specific apiEndpoints from PRD if defined.
|
|
15
|
+
run_api_smoke_test() {
|
|
16
|
+
local story="$1"
|
|
17
|
+
|
|
18
|
+
# Check if API smoke tests are enabled (default: true if baseUrl configured)
|
|
19
|
+
local base_url
|
|
20
|
+
base_url=$(get_config '.api.baseUrl' "")
|
|
21
|
+
|
|
22
|
+
# No API configured, skip silently
|
|
23
|
+
[[ -z "$base_url" ]] && return 0
|
|
24
|
+
|
|
25
|
+
echo ""
|
|
26
|
+
echo " [4/4] Running API smoke tests..."
|
|
27
|
+
|
|
28
|
+
local failed=0
|
|
29
|
+
local endpoints_tested=0
|
|
30
|
+
|
|
31
|
+
# 1. Health endpoint (most important)
|
|
32
|
+
local health_endpoint
|
|
33
|
+
health_endpoint=$(get_config '.api.healthEndpoint' "/health")
|
|
34
|
+
if [[ -n "$health_endpoint" ]]; then
|
|
35
|
+
if ! _smoke_test_endpoint "$base_url" "$health_endpoint" "health"; then
|
|
36
|
+
failed=1
|
|
37
|
+
fi
|
|
38
|
+
((endpoints_tested++))
|
|
39
|
+
fi
|
|
40
|
+
|
|
41
|
+
# 2. Configured smoke endpoints
|
|
42
|
+
local smoke_endpoints
|
|
43
|
+
smoke_endpoints=$(get_config '.api.smokeEndpoints' "[]")
|
|
44
|
+
if [[ "$smoke_endpoints" != "[]" && "$smoke_endpoints" != "null" ]]; then
|
|
45
|
+
while IFS= read -r endpoint; do
|
|
46
|
+
[[ -z "$endpoint" ]] && continue
|
|
47
|
+
if ! _smoke_test_endpoint "$base_url" "$endpoint" "smoke"; then
|
|
48
|
+
failed=1
|
|
49
|
+
fi
|
|
50
|
+
((endpoints_tested++))
|
|
51
|
+
done < <(echo "$smoke_endpoints" | jq -r '.[]' 2>/dev/null)
|
|
52
|
+
fi
|
|
53
|
+
|
|
54
|
+
# 3. Story-specific apiEndpoints from PRD
|
|
55
|
+
local story_endpoints
|
|
56
|
+
story_endpoints=$(jq -r --arg id "$story" '.stories[] | select(.id==$id) | .apiEndpoints[]?' "$RALPH_DIR/prd.json" 2>/dev/null)
|
|
57
|
+
if [[ -n "$story_endpoints" ]]; then
|
|
58
|
+
while IFS= read -r endpoint_spec; do
|
|
59
|
+
[[ -z "$endpoint_spec" ]] && continue
|
|
60
|
+
# Format: "GET /api/v1/users" or "POST /api/v1/items" or just "/api/v1/users"
|
|
61
|
+
local method="GET"
|
|
62
|
+
local path="$endpoint_spec"
|
|
63
|
+
if [[ "$endpoint_spec" =~ ^(GET|POST|PUT|DELETE|PATCH)[[:space:]]+(.*) ]]; then
|
|
64
|
+
method="${BASH_REMATCH[1]}"
|
|
65
|
+
path="${BASH_REMATCH[2]}"
|
|
66
|
+
fi
|
|
67
|
+
if ! _smoke_test_endpoint "$base_url" "$path" "story" "$method"; then
|
|
68
|
+
failed=1
|
|
69
|
+
fi
|
|
70
|
+
((endpoints_tested++))
|
|
71
|
+
done <<< "$story_endpoints"
|
|
72
|
+
fi
|
|
73
|
+
|
|
74
|
+
if [[ $endpoints_tested -eq 0 ]]; then
|
|
75
|
+
echo " (no endpoints configured, skipping)"
|
|
76
|
+
return 0
|
|
77
|
+
fi
|
|
78
|
+
|
|
79
|
+
return $failed
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
# Run frontend smoke test
|
|
83
|
+
# Config options:
|
|
84
|
+
# urls.frontend - Frontend URL (e.g., http://localhost:3000)
|
|
85
|
+
# frontend.smokePages - Array of paths to test (e.g., ["/", "/login", "/dashboard"])
|
|
86
|
+
run_frontend_smoke_test() {
|
|
87
|
+
local story="$1"
|
|
88
|
+
local story_type="${RALPH_STORY_TYPE:-general}"
|
|
89
|
+
|
|
90
|
+
# Get frontend URL from config (try multiple locations)
|
|
91
|
+
local frontend_url
|
|
92
|
+
frontend_url=$(get_config '.urls.frontend' "")
|
|
93
|
+
[[ -z "$frontend_url" ]] && frontend_url=$(get_config '.playwright.baseUrl' "")
|
|
94
|
+
|
|
95
|
+
# No frontend configured, skip silently
|
|
96
|
+
[[ -z "$frontend_url" ]] && return 0
|
|
97
|
+
|
|
98
|
+
# Skip for backend-only stories (optional optimization)
|
|
99
|
+
# [[ "$story_type" == "backend" ]] && return 0
|
|
100
|
+
|
|
101
|
+
echo ""
|
|
102
|
+
echo " [5/5] Running frontend smoke tests..."
|
|
103
|
+
|
|
104
|
+
local failed=0
|
|
105
|
+
local pages_tested=0
|
|
106
|
+
|
|
107
|
+
# 1. Test root page (most important)
|
|
108
|
+
if ! _smoke_test_page "$frontend_url" "/" "root"; then
|
|
109
|
+
failed=1
|
|
110
|
+
fi
|
|
111
|
+
((pages_tested++))
|
|
112
|
+
|
|
113
|
+
# 2. Configured smoke pages
|
|
114
|
+
local smoke_pages
|
|
115
|
+
smoke_pages=$(get_config '.frontend.smokePages' "[]")
|
|
116
|
+
if [[ "$smoke_pages" != "[]" && "$smoke_pages" != "null" ]]; then
|
|
117
|
+
while IFS= read -r page; do
|
|
118
|
+
[[ -z "$page" ]] && continue
|
|
119
|
+
[[ "$page" == "/" ]] && continue # Already tested root
|
|
120
|
+
if ! _smoke_test_page "$frontend_url" "$page" "smoke"; then
|
|
121
|
+
failed=1
|
|
122
|
+
fi
|
|
123
|
+
((pages_tested++))
|
|
124
|
+
done < <(echo "$smoke_pages" | jq -r '.[]' 2>/dev/null)
|
|
125
|
+
fi
|
|
126
|
+
|
|
127
|
+
# 3. Story-specific testUrl from PRD
|
|
128
|
+
local test_url
|
|
129
|
+
test_url=$(jq -r --arg id "$story" '.stories[] | select(.id==$id) | .testUrl // empty' "$RALPH_DIR/prd.json" 2>/dev/null)
|
|
130
|
+
if [[ -n "$test_url" ]]; then
|
|
131
|
+
# testUrl can be full URL or just path
|
|
132
|
+
if [[ "$test_url" =~ ^https?:// ]]; then
|
|
133
|
+
if ! _smoke_test_page "" "$test_url" "story"; then
|
|
134
|
+
failed=1
|
|
135
|
+
fi
|
|
136
|
+
else
|
|
137
|
+
if ! _smoke_test_page "$frontend_url" "$test_url" "story"; then
|
|
138
|
+
failed=1
|
|
139
|
+
fi
|
|
140
|
+
fi
|
|
141
|
+
((pages_tested++))
|
|
142
|
+
fi
|
|
143
|
+
|
|
144
|
+
return $failed
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
# Test a single page
|
|
148
|
+
# Usage: _smoke_test_page <base_url> <path> <type>
|
|
149
|
+
_smoke_test_page() {
|
|
150
|
+
local base_url="$1"
|
|
151
|
+
local path="$2"
|
|
152
|
+
local test_type="$3"
|
|
153
|
+
|
|
154
|
+
local url
|
|
155
|
+
if [[ -z "$base_url" ]]; then
|
|
156
|
+
url="$path"
|
|
157
|
+
else
|
|
158
|
+
url="${base_url}${path}"
|
|
159
|
+
fi
|
|
160
|
+
|
|
161
|
+
echo -n " GET $path... "
|
|
162
|
+
|
|
163
|
+
local response_file
|
|
164
|
+
response_file=$(mktemp)
|
|
165
|
+
local http_code
|
|
166
|
+
|
|
167
|
+
http_code=$(curl -s -o "$response_file" -w "%{http_code}" \
|
|
168
|
+
--max-time "$CURL_TIMEOUT_SECONDS" \
|
|
169
|
+
"$url" 2>/dev/null) || http_code="000"
|
|
170
|
+
|
|
171
|
+
# Check response
|
|
172
|
+
if [[ "$http_code" == "000" ]]; then
|
|
173
|
+
print_error "connection failed (is frontend running?)"
|
|
174
|
+
rm -f "$response_file"
|
|
175
|
+
return 1
|
|
176
|
+
elif [[ "$http_code" =~ ^5 ]]; then
|
|
177
|
+
print_error "HTTP $http_code - Server Error"
|
|
178
|
+
|
|
179
|
+
# Save for failure context
|
|
180
|
+
{
|
|
181
|
+
echo "Frontend smoke test failed: $url"
|
|
182
|
+
echo "HTTP Status: $http_code"
|
|
183
|
+
echo "Response (first 50 lines):"
|
|
184
|
+
head -50 "$response_file"
|
|
185
|
+
} >> "$RALPH_DIR/last_frontend_failure.log"
|
|
186
|
+
|
|
187
|
+
rm -f "$response_file"
|
|
188
|
+
return 1
|
|
189
|
+
elif [[ "$http_code" =~ ^4 ]]; then
|
|
190
|
+
# 404 on a specific page is a real error (unlike API auth)
|
|
191
|
+
if [[ "$http_code" == "404" ]]; then
|
|
192
|
+
print_error "HTTP 404 - Page not found"
|
|
193
|
+
rm -f "$response_file"
|
|
194
|
+
return 1
|
|
195
|
+
fi
|
|
196
|
+
# Other 4xx (401, 403) might be OK - auth required
|
|
197
|
+
print_warning "HTTP $http_code (may need auth)"
|
|
198
|
+
rm -f "$response_file"
|
|
199
|
+
return 0
|
|
200
|
+
else
|
|
201
|
+
# Check for React/Next.js error boundary or crash indicators
|
|
202
|
+
if grep -qi "application error\|something went wrong\|error boundary\|chunk load error" "$response_file" 2>/dev/null; then
|
|
203
|
+
print_error "HTTP $http_code but page shows error"
|
|
204
|
+
{
|
|
205
|
+
echo "Frontend smoke test failed: $url"
|
|
206
|
+
echo "Page loaded but contains error indicators"
|
|
207
|
+
head -50 "$response_file"
|
|
208
|
+
} >> "$RALPH_DIR/last_frontend_failure.log"
|
|
209
|
+
rm -f "$response_file"
|
|
210
|
+
return 1
|
|
211
|
+
fi
|
|
212
|
+
print_success "HTTP $http_code"
|
|
213
|
+
rm -f "$response_file"
|
|
214
|
+
return 0
|
|
215
|
+
fi
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
# Test a single endpoint
|
|
219
|
+
# Usage: _smoke_test_endpoint <base_url> <path> <type> [method]
|
|
220
|
+
_smoke_test_endpoint() {
|
|
221
|
+
local base_url="$1"
|
|
222
|
+
local path="$2"
|
|
223
|
+
local test_type="$3"
|
|
224
|
+
local method="${4:-GET}"
|
|
225
|
+
|
|
226
|
+
local url="${base_url}${path}"
|
|
227
|
+
echo -n " $method $path... "
|
|
228
|
+
|
|
229
|
+
local response_file
|
|
230
|
+
response_file=$(mktemp)
|
|
231
|
+
local http_code
|
|
232
|
+
|
|
233
|
+
# Make request with timeout, capture status code
|
|
234
|
+
if [[ "$method" == "GET" ]]; then
|
|
235
|
+
http_code=$(curl -s -o "$response_file" -w "%{http_code}" \
|
|
236
|
+
--max-time "$CURL_TIMEOUT_SECONDS" \
|
|
237
|
+
"$url" 2>/dev/null) || http_code="000"
|
|
238
|
+
else
|
|
239
|
+
# For non-GET, just check endpoint exists (OPTIONS or empty body)
|
|
240
|
+
http_code=$(curl -s -o "$response_file" -w "%{http_code}" \
|
|
241
|
+
--max-time "$CURL_TIMEOUT_SECONDS" \
|
|
242
|
+
-X "$method" \
|
|
243
|
+
-H "Content-Type: application/json" \
|
|
244
|
+
-d '{}' \
|
|
245
|
+
"$url" 2>/dev/null) || http_code="000"
|
|
246
|
+
fi
|
|
247
|
+
|
|
248
|
+
# Check response
|
|
249
|
+
if [[ "$http_code" == "000" ]]; then
|
|
250
|
+
print_error "connection failed (is server running?)"
|
|
251
|
+
rm -f "$response_file"
|
|
252
|
+
return 1
|
|
253
|
+
elif [[ "$http_code" =~ ^5 ]]; then
|
|
254
|
+
print_error "HTTP $http_code - Internal Server Error"
|
|
255
|
+
echo ""
|
|
256
|
+
echo " Response body:"
|
|
257
|
+
head -20 "$response_file" | sed 's/^/ /'
|
|
258
|
+
|
|
259
|
+
# Save for failure context
|
|
260
|
+
{
|
|
261
|
+
echo "API smoke test failed: $method $url"
|
|
262
|
+
echo "HTTP Status: $http_code"
|
|
263
|
+
echo "Response:"
|
|
264
|
+
cat "$response_file"
|
|
265
|
+
} >> "$RALPH_DIR/last_api_failure.log"
|
|
266
|
+
|
|
267
|
+
rm -f "$response_file"
|
|
268
|
+
return 1
|
|
269
|
+
elif [[ "$http_code" =~ ^4 ]]; then
|
|
270
|
+
# 4xx might be OK (auth required, etc.) - warn but don't fail
|
|
271
|
+
print_warning "HTTP $http_code (may need auth)"
|
|
272
|
+
rm -f "$response_file"
|
|
273
|
+
return 0
|
|
274
|
+
else
|
|
275
|
+
print_success "HTTP $http_code"
|
|
276
|
+
rm -f "$response_file"
|
|
277
|
+
return 0
|
|
278
|
+
fi
|
|
279
|
+
}
|
package/ralph/verify/lint.sh
CHANGED
|
@@ -355,6 +355,46 @@ verify_go() {
|
|
|
355
355
|
return 0
|
|
356
356
|
}
|
|
357
357
|
|
|
358
|
+
# Verify Elixir code with mix credo
|
|
359
|
+
verify_elixir() {
|
|
360
|
+
local elixir_log="$RALPH_DIR/last_elixir_failure.log"
|
|
361
|
+
|
|
362
|
+
# Skip if not an Elixir project
|
|
363
|
+
[[ ! -f "mix.exs" ]] && return 0
|
|
364
|
+
command -v mix &>/dev/null || return 0
|
|
365
|
+
|
|
366
|
+
# Clear previous failure log
|
|
367
|
+
rm -f "$elixir_log"
|
|
368
|
+
|
|
369
|
+
# Mix credo (Elixir's static analysis tool)
|
|
370
|
+
echo -n " Mix credo... "
|
|
371
|
+
local credo_output
|
|
372
|
+
if credo_output=$(mix credo --strict 2>&1); then
|
|
373
|
+
print_success "passed"
|
|
374
|
+
return 0
|
|
375
|
+
fi
|
|
376
|
+
|
|
377
|
+
# Check if credo is installed
|
|
378
|
+
if echo "$credo_output" | grep -qi "could not find.*credo\|mix credo.*not found"; then
|
|
379
|
+
echo -n "not installed, trying mix compile... "
|
|
380
|
+
if credo_output=$(mix compile --warnings-as-errors 2>&1); then
|
|
381
|
+
print_success "passed"
|
|
382
|
+
return 0
|
|
383
|
+
fi
|
|
384
|
+
fi
|
|
385
|
+
|
|
386
|
+
# Failed
|
|
387
|
+
print_error "failed"
|
|
388
|
+
echo ""
|
|
389
|
+
echo " Elixir errors:"
|
|
390
|
+
echo "$credo_output" | head -"$MAX_LINT_ERROR_LINES" | sed 's/^/ /'
|
|
391
|
+
{
|
|
392
|
+
echo "Elixir errors:"
|
|
393
|
+
echo "$credo_output"
|
|
394
|
+
} >> "$elixir_log"
|
|
395
|
+
return 1
|
|
396
|
+
}
|
|
397
|
+
|
|
358
398
|
# Verify Rust code with clippy
|
|
359
399
|
verify_rust() {
|
|
360
400
|
local rust_log="$RALPH_DIR/last_rust_failure.log"
|
|
@@ -505,6 +545,9 @@ run_configured_checks() {
|
|
|
505
545
|
if ! verify_rust; then
|
|
506
546
|
return 1
|
|
507
547
|
fi
|
|
548
|
+
if ! verify_elixir; then
|
|
549
|
+
return 1
|
|
550
|
+
fi
|
|
508
551
|
fi
|
|
509
552
|
|
|
510
553
|
# FastAPI response model check
|
package/ralph/verify/tests.sh
CHANGED
package/ralph/verify.sh
CHANGED
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
VERIFY_DIR="${RALPH_LIB:-$(dirname "${BASH_SOURCE[0]}")}"
|
|
10
10
|
source "$VERIFY_DIR/verify/lint.sh"
|
|
11
11
|
source "$VERIFY_DIR/verify/tests.sh"
|
|
12
|
+
source "$VERIFY_DIR/verify/api.sh"
|
|
12
13
|
|
|
13
14
|
run_verification() {
|
|
14
15
|
local story="$1"
|
|
@@ -27,7 +28,7 @@ run_verification() {
|
|
|
27
28
|
# ========================================
|
|
28
29
|
# STEP 1: Run lint checks
|
|
29
30
|
# ========================================
|
|
30
|
-
echo " [1/
|
|
31
|
+
echo " [1/5] Running lint checks..."
|
|
31
32
|
if ! run_configured_checks "$story_type"; then
|
|
32
33
|
failed=1
|
|
33
34
|
fi
|
|
@@ -37,7 +38,7 @@ run_verification() {
|
|
|
37
38
|
# ========================================
|
|
38
39
|
if [[ $failed -eq 0 ]]; then
|
|
39
40
|
echo ""
|
|
40
|
-
echo " [2/
|
|
41
|
+
echo " [2/5] Running tests..."
|
|
41
42
|
# First check that test files exist for new code
|
|
42
43
|
if ! verify_test_files_exist; then
|
|
43
44
|
failed=1
|
|
@@ -51,12 +52,30 @@ run_verification() {
|
|
|
51
52
|
# ========================================
|
|
52
53
|
if [[ $failed -eq 0 ]]; then
|
|
53
54
|
echo ""
|
|
54
|
-
echo " [3/
|
|
55
|
+
echo " [3/5] Running PRD test steps..."
|
|
55
56
|
if ! verify_prd_criteria "$story"; then
|
|
56
57
|
failed=1
|
|
57
58
|
fi
|
|
58
59
|
fi
|
|
59
60
|
|
|
61
|
+
# ========================================
|
|
62
|
+
# STEP 4: API smoke test (if configured)
|
|
63
|
+
# ========================================
|
|
64
|
+
if [[ $failed -eq 0 ]]; then
|
|
65
|
+
if ! run_api_smoke_test "$story"; then
|
|
66
|
+
failed=1
|
|
67
|
+
fi
|
|
68
|
+
fi
|
|
69
|
+
|
|
70
|
+
# ========================================
|
|
71
|
+
# STEP 5: Frontend smoke test (if configured)
|
|
72
|
+
# ========================================
|
|
73
|
+
if [[ $failed -eq 0 ]]; then
|
|
74
|
+
if ! run_frontend_smoke_test "$story"; then
|
|
75
|
+
failed=1
|
|
76
|
+
fi
|
|
77
|
+
fi
|
|
78
|
+
|
|
60
79
|
# ========================================
|
|
61
80
|
# Final result
|
|
62
81
|
# ========================================
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
{
|
|
2
|
+
"auth": {
|
|
3
|
+
"testUser": "",
|
|
4
|
+
"testPassword": "",
|
|
5
|
+
"loginEndpoint": "/api/auth/login",
|
|
6
|
+
"loginMethod": "POST",
|
|
7
|
+
"tokenType": "jwt",
|
|
8
|
+
"tokenHeader": "Authorization",
|
|
9
|
+
"tokenPrefix": "Bearer"
|
|
10
|
+
},
|
|
11
|
+
|
|
12
|
+
"docker": {
|
|
13
|
+
"enabled": false,
|
|
14
|
+
"composeFile": "docker-compose.yml",
|
|
15
|
+
"serviceName": "app",
|
|
16
|
+
"execPrefix": "docker compose exec -T"
|
|
17
|
+
},
|
|
18
|
+
|
|
19
|
+
"paths": {
|
|
20
|
+
"src": "lib",
|
|
21
|
+
"tests": "test",
|
|
22
|
+
"e2e": "test/e2e"
|
|
23
|
+
},
|
|
24
|
+
|
|
25
|
+
"commands": {
|
|
26
|
+
"dev": "mix phx.server",
|
|
27
|
+
"install": "mix deps.get",
|
|
28
|
+
"seed": "mix run priv/repo/seeds.exs",
|
|
29
|
+
"resetDb": "mix ecto.reset"
|
|
30
|
+
},
|
|
31
|
+
|
|
32
|
+
"migrations": {
|
|
33
|
+
"command": "mix ecto.migrate",
|
|
34
|
+
"pattern": "priv/repo/migrations/.*\\.exs$"
|
|
35
|
+
},
|
|
36
|
+
|
|
37
|
+
"checks": {
|
|
38
|
+
"lint": true,
|
|
39
|
+
"typecheck": false,
|
|
40
|
+
"build": true,
|
|
41
|
+
"test": true,
|
|
42
|
+
"fastapi": false
|
|
43
|
+
},
|
|
44
|
+
|
|
45
|
+
"api": {
|
|
46
|
+
"baseUrl": "http://localhost:4000",
|
|
47
|
+
"healthEndpoint": "/api/health",
|
|
48
|
+
"smokeEndpoints": [],
|
|
49
|
+
"timeout": 30
|
|
50
|
+
},
|
|
51
|
+
|
|
52
|
+
"playwright": {
|
|
53
|
+
"enabled": false,
|
|
54
|
+
"testDir": "test/e2e",
|
|
55
|
+
"projects": ["chromium"],
|
|
56
|
+
"baseUrl": "http://localhost:4000"
|
|
57
|
+
},
|
|
58
|
+
|
|
59
|
+
"verification": {
|
|
60
|
+
"codeReviewEnabled": true,
|
|
61
|
+
"browserEnabled": true,
|
|
62
|
+
"a11yEnabled": false,
|
|
63
|
+
"mobileViewport": 375,
|
|
64
|
+
"screenshotOnFailure": true
|
|
65
|
+
},
|
|
66
|
+
|
|
67
|
+
"urls": {
|
|
68
|
+
"app": "http://localhost:4000",
|
|
69
|
+
"docs": "http://localhost:4000/dev/dashboard"
|
|
70
|
+
},
|
|
71
|
+
|
|
72
|
+
"env": {
|
|
73
|
+
"required": ["DATABASE_URL"],
|
|
74
|
+
"optional": ["SECRET_KEY_BASE", "PHX_HOST"]
|
|
75
|
+
},
|
|
76
|
+
|
|
77
|
+
"maxIterations": 20,
|
|
78
|
+
"maxSessionSeconds": 600,
|
|
79
|
+
|
|
80
|
+
"contextRotThreshold": {
|
|
81
|
+
"maxStories": 10,
|
|
82
|
+
"maxFilesChanged": 20
|
|
83
|
+
}
|
|
84
|
+
}
|
|
@@ -47,6 +47,7 @@
|
|
|
47
47
|
"api": {
|
|
48
48
|
"baseUrl": "http://localhost:8000",
|
|
49
49
|
"healthEndpoint": "/api/health",
|
|
50
|
+
"smokeEndpoints": [],
|
|
50
51
|
"timeout": 30
|
|
51
52
|
},
|
|
52
53
|
|
|
@@ -71,6 +72,10 @@
|
|
|
71
72
|
"docs": "http://localhost:8000/api/docs"
|
|
72
73
|
},
|
|
73
74
|
|
|
75
|
+
"frontend": {
|
|
76
|
+
"smokePages": ["/", "/login"]
|
|
77
|
+
},
|
|
78
|
+
|
|
74
79
|
"env": {
|
|
75
80
|
"required": ["DATABASE_URL", "SECRET_KEY"],
|
|
76
81
|
"optional": ["REDIS_URL", "SENTRY_DSN"]
|
|
@@ -45,6 +45,7 @@
|
|
|
45
45
|
"api": {
|
|
46
46
|
"baseUrl": "http://localhost:3000",
|
|
47
47
|
"healthEndpoint": "/api/health",
|
|
48
|
+
"smokeEndpoints": [],
|
|
48
49
|
"timeout": 30
|
|
49
50
|
},
|
|
50
51
|
|
|
@@ -65,9 +66,14 @@
|
|
|
65
66
|
|
|
66
67
|
"urls": {
|
|
67
68
|
"app": "http://localhost:3000",
|
|
69
|
+
"frontend": "http://localhost:3000",
|
|
68
70
|
"docs": "http://localhost:3000/api/docs"
|
|
69
71
|
},
|
|
70
72
|
|
|
73
|
+
"frontend": {
|
|
74
|
+
"smokePages": ["/"]
|
|
75
|
+
},
|
|
76
|
+
|
|
71
77
|
"env": {
|
|
72
78
|
"required": ["DATABASE_URL"],
|
|
73
79
|
"optional": ["REDIS_URL", "SENTRY_DSN"]
|