agentic-loop 3.6.2 → 3.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/commands/prd.md +55 -27
- package/README.md +0 -18
- package/package.json +1 -1
- package/ralph/init.sh +2 -0
- package/ralph/loop.sh +37 -1
- package/ralph/utils.sh +41 -1
- package/ralph/verify/api.sh +279 -0
- package/ralph/verify/lint.sh +43 -0
- package/ralph/verify/tests.sh +2 -0
- package/ralph/verify.sh +22 -3
- package/templates/config/elixir.json +84 -0
- package/templates/config/fullstack.json +5 -0
- package/templates/config/node.json +6 -0
- package/templates/config/python.json +1 -0
package/.claude/commands/prd.md
CHANGED
|
@@ -103,7 +103,29 @@ Break the idea into small, executable stories:
|
|
|
103
103
|
- Max 10 stories (suggest phases if more needed)
|
|
104
104
|
- If appending, start IDs from the next available number
|
|
105
105
|
|
|
106
|
-
### Step 5:
|
|
106
|
+
### Step 5: Validate Test Steps (CRITICAL)
|
|
107
|
+
|
|
108
|
+
**Before writing the PRD, validate EVERY story's testSteps:**
|
|
109
|
+
|
|
110
|
+
For each story, check:
|
|
111
|
+
- ❌ **REJECT** if testSteps only use `grep` to check code exists
|
|
112
|
+
- ❌ **REJECT** if testSteps are human instructions ("Visit the page", "User can see")
|
|
113
|
+
- ✅ **REQUIRE** curl commands for backend stories that verify actual API behavior
|
|
114
|
+
- ✅ **REQUIRE** `npx tsc --noEmit` for TypeScript frontend stories
|
|
115
|
+
- ✅ **REQUIRE** playwright/test commands for UI stories
|
|
116
|
+
|
|
117
|
+
**Common mistake to catch:**
|
|
118
|
+
```json
|
|
119
|
+
// ❌ This will pass but the feature is broken!
|
|
120
|
+
"testSteps": ["grep -q 'myFunction' src/api/users.ts"]
|
|
121
|
+
|
|
122
|
+
// ✅ This actually verifies behavior
|
|
123
|
+
"testSteps": ["curl -s {config.urls.backend}/users | jq -e '.data | length >= 0'"]
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
If a story's testSteps won't catch a broken implementation, FIX THEM before proceeding.
|
|
127
|
+
|
|
128
|
+
### Step 6: Write PRD
|
|
107
129
|
|
|
108
130
|
1. Ensure .ralph directory exists and allow PRD edit:
|
|
109
131
|
```bash
|
|
@@ -123,7 +145,7 @@ Break the idea into small, executable stories:
|
|
|
123
145
|
|
|
124
146
|
**STOP and wait for user response.**
|
|
125
147
|
|
|
126
|
-
### Step
|
|
148
|
+
### Step 7: Final Instructions
|
|
127
149
|
|
|
128
150
|
Once approved, say:
|
|
129
151
|
|
|
@@ -232,7 +254,8 @@ Ralph will work through each story, running tests and committing as it goes."
|
|
|
232
254
|
},
|
|
233
255
|
|
|
234
256
|
"testSteps": [
|
|
235
|
-
"
|
|
257
|
+
"curl -s {config.urls.backend}/endpoint | jq -e '.expected == true'",
|
|
258
|
+
"npx playwright test tests/e2e/feature.spec.ts"
|
|
236
259
|
],
|
|
237
260
|
|
|
238
261
|
"testUrl": "{config.urls.frontend}/feature-page",
|
|
@@ -412,19 +435,6 @@ Example for a Dashboard component:
|
|
|
412
435
|
|
|
413
436
|
### Testing Anti-Patterns (AVOID THESE)
|
|
414
437
|
|
|
415
|
-
**The "grep for code" trap:**
|
|
416
|
-
```json
|
|
417
|
-
// ❌ BAD - verifies code exists, not that it works
|
|
418
|
-
"testSteps": [
|
|
419
|
-
"grep -q 'astream_events' app/domains/chat/agent/graph.py"
|
|
420
|
-
]
|
|
421
|
-
|
|
422
|
-
// ✅ GOOD - verifies actual behavior
|
|
423
|
-
"testSteps": [
|
|
424
|
-
"curl -N {config.urls.backend}/chat -d '{\"message\":\"test\"}' | grep -q 'progress'"
|
|
425
|
-
]
|
|
426
|
-
```
|
|
427
|
-
|
|
428
438
|
**Missing integration points:**
|
|
429
439
|
```json
|
|
430
440
|
// ❌ BAD - creates function but doesn't verify callers use it
|
|
@@ -443,6 +453,8 @@ Example for a Dashboard component:
|
|
|
443
453
|
}
|
|
444
454
|
```
|
|
445
455
|
|
|
456
|
+
**(See "The Grep for Code Trap" section above for the #1 anti-pattern)**
|
|
457
|
+
|
|
446
458
|
### Removing/Modifying UI - Update Tests!
|
|
447
459
|
|
|
448
460
|
**CRITICAL: When a story removes or modifies UI elements, it MUST update related tests.**
|
|
@@ -572,8 +584,28 @@ Example:
|
|
|
572
584
|
|
|
573
585
|
## Test Steps - CRITICAL
|
|
574
586
|
|
|
587
|
+
⚠️ **THE #1 CAUSE OF FALSE PASSES: grep-only test steps that verify code exists but not behavior.**
|
|
588
|
+
|
|
575
589
|
**Test steps MUST be executable shell commands.** Ralph runs them with bash.
|
|
576
590
|
|
|
591
|
+
### The "Grep for Code" Trap - NEVER DO THIS
|
|
592
|
+
|
|
593
|
+
```json
|
|
594
|
+
// ❌ BAD - This will PASS even when the feature is completely broken!
|
|
595
|
+
"testSteps": [
|
|
596
|
+
"grep -q 'astream_events' app/domains/chat/agent/graph.py",
|
|
597
|
+
"grep -q 'export function' src/api/users.ts"
|
|
598
|
+
]
|
|
599
|
+
|
|
600
|
+
// ✅ GOOD - This actually tests if the feature works
|
|
601
|
+
"testSteps": [
|
|
602
|
+
"curl -N {config.urls.backend}/chat -d '{\"message\":\"test\"}' | grep -q 'progress'",
|
|
603
|
+
"curl -s {config.urls.backend}/users | jq -e '.data | length >= 0'"
|
|
604
|
+
]
|
|
605
|
+
```
|
|
606
|
+
|
|
607
|
+
**Why is grep bad?** Ralph runs `grep -q 'function' file.py` → returns 0 → marks story as PASSED. But the function could be completely broken, have wrong parameters, or never get called. The test passed but the feature doesn't work.
|
|
608
|
+
|
|
577
609
|
### Backend Stories MUST Have Curl Tests
|
|
578
610
|
|
|
579
611
|
**CRITICAL: Every backend story MUST include curl commands that verify actual API behavior.**
|
|
@@ -591,15 +623,7 @@ Use `{config.urls.backend}` - Ralph expands this from `.ralph/config.json`:
|
|
|
591
623
|
|
|
592
624
|
Ralph reads `.ralph/config.json` and expands `{config.urls.backend}` before running.
|
|
593
625
|
|
|
594
|
-
**Why?** Grep tests verify code exists. Curl tests verify the feature works.
|
|
595
|
-
|
|
596
|
-
```json
|
|
597
|
-
// ❌ NEVER DO THIS for backend stories
|
|
598
|
-
"testSteps": [
|
|
599
|
-
"grep -q 'astream_events' app/domains/chat/agent/graph.py"
|
|
600
|
-
]
|
|
601
|
-
// This passed but the feature was broken!
|
|
602
|
-
```
|
|
626
|
+
**Why?** Grep tests verify code exists. Curl tests verify the feature works. (See "The Grep for Code Trap" above.)
|
|
603
627
|
|
|
604
628
|
### Test Steps by Story Type
|
|
605
629
|
|
|
@@ -640,15 +664,19 @@ Ralph reads `.ralph/config.json` and expands `{config.urls.backend}` before runn
|
|
|
640
664
|
]
|
|
641
665
|
```
|
|
642
666
|
|
|
643
|
-
### Bad Test Steps (will
|
|
667
|
+
### Bad Test Steps (will PASS but miss bugs)
|
|
644
668
|
```json
|
|
645
669
|
"testSteps": [
|
|
646
|
-
"grep -q 'function createUser' app/services/user.py", // ❌
|
|
670
|
+
"grep -q 'function createUser' app/services/user.py", // ❌ PASSES if code exists, even if broken
|
|
671
|
+
"grep -q 'export default' src/components/Dashboard.tsx", // ❌ PASSES even if component crashes
|
|
672
|
+
"test -f src/api/users.ts", // ❌ PASSES if file exists, even if empty
|
|
647
673
|
"Visit http://localhost:3000/dashboard", // ❌ Not executable
|
|
648
674
|
"User can see the dashboard" // ❌ Not executable
|
|
649
675
|
]
|
|
650
676
|
```
|
|
651
677
|
|
|
678
|
+
**NEVER use grep/test to verify behavior.** These will mark stories as PASSED when the feature is broken.
|
|
679
|
+
|
|
652
680
|
**If a step can't be automated**, put it in `acceptanceCriteria` instead. Claude will verify it visually using MCP tools.
|
|
653
681
|
|
|
654
682
|
---
|
package/README.md
CHANGED
|
@@ -8,24 +8,6 @@ You describe what you want to build. Claude Code writes a PRD (Product Requireme
|
|
|
8
8
|
|
|
9
9
|
---
|
|
10
10
|
|
|
11
|
-
## Supported Project Types
|
|
12
|
-
|
|
13
|
-
Ralph auto-detects your project type and configures itself accordingly:
|
|
14
|
-
|
|
15
|
-
| Type | Detection | Auto-Configured |
|
|
16
|
-
|------|-----------|-----------------|
|
|
17
|
-
| **FastMCP** | `fastmcp` in pyproject.toml | Server module, MCP port, transport, subprojects |
|
|
18
|
-
| **FastAPI** | `fastapi` in pyproject.toml | uvicorn dev server, pytest, ruff |
|
|
19
|
-
| **Django** | `django` in pyproject.toml or manage.py | migrations, pytest, ruff |
|
|
20
|
-
| **Python** | pyproject.toml or requirements.txt | pytest, ruff, uv/poetry detection |
|
|
21
|
-
| **Node.js** | package.json | npm/yarn/pnpm, vitest/jest, eslint |
|
|
22
|
-
| **React** | `react` in package.json | Vite/Next.js, TypeScript, Tailwind |
|
|
23
|
-
| **Go/Hugo** | go.mod or hugo.toml | Hugo server, Go build |
|
|
24
|
-
| **Rust** | Cargo.toml | cargo build/test/clippy |
|
|
25
|
-
| **Fullstack** | frontend + backend directories | Monorepo support, separate configs |
|
|
26
|
-
|
|
27
|
-
---
|
|
28
|
-
|
|
29
11
|
## What It Does
|
|
30
12
|
|
|
31
13
|
**Brainstorm ideas with `/idea`**
|
package/package.json
CHANGED
package/ralph/init.sh
CHANGED
|
@@ -145,6 +145,8 @@ detect_project_type() {
|
|
|
145
145
|
project_type="rust"
|
|
146
146
|
elif [[ -f "go.mod" ]]; then
|
|
147
147
|
project_type="go"
|
|
148
|
+
elif [[ -f "mix.exs" ]]; then
|
|
149
|
+
project_type="elixir"
|
|
148
150
|
# Check for Python framework variants (more specific first)
|
|
149
151
|
elif [[ -f "pyproject.toml" ]]; then
|
|
150
152
|
# FastMCP detection (check for fastmcp in any quote style)
|
package/ralph/loop.sh
CHANGED
|
@@ -336,7 +336,43 @@ run_loop() {
|
|
|
336
336
|
claude_args=(--continue "${claude_args[@]}")
|
|
337
337
|
fi
|
|
338
338
|
|
|
339
|
-
|
|
339
|
+
# Run Claude with crash detection and retry logic
|
|
340
|
+
local claude_output_log claude_exit_code max_crash_retries=3 crash_attempt=0
|
|
341
|
+
claude_output_log=$(create_temp_file ".log") || { rm -f "$prompt_file"; return 1; }
|
|
342
|
+
|
|
343
|
+
while [[ $crash_attempt -lt $max_crash_retries ]]; do
|
|
344
|
+
claude_exit_code=0
|
|
345
|
+
# Use pipefail to capture Claude's exit code, not tee's
|
|
346
|
+
set -o pipefail
|
|
347
|
+
cat "$prompt_file" | run_with_timeout "$timeout_seconds" claude "${claude_args[@]}" 2>&1 | tee "$claude_output_log" || claude_exit_code=$?
|
|
348
|
+
set +o pipefail
|
|
349
|
+
|
|
350
|
+
# Check for recoverable CLI crashes
|
|
351
|
+
if grep -qE "(No messages returned|unhandled.*promise.*rejection)" "$claude_output_log" 2>/dev/null; then
|
|
352
|
+
((crash_attempt++))
|
|
353
|
+
print_warning "Claude CLI crashed (attempt $crash_attempt/$max_crash_retries) - retrying..."
|
|
354
|
+
log_progress "$story" "CLI_CRASH" "Claude crashed, retry $crash_attempt"
|
|
355
|
+
session_started=false # Reset session on crash
|
|
356
|
+
sleep 2 # Brief pause before retry
|
|
357
|
+
continue
|
|
358
|
+
fi
|
|
359
|
+
|
|
360
|
+
# Not a crash - exit retry loop
|
|
361
|
+
break
|
|
362
|
+
done
|
|
363
|
+
|
|
364
|
+
rm -f "$claude_output_log"
|
|
365
|
+
|
|
366
|
+
if [[ $crash_attempt -ge $max_crash_retries ]]; then
|
|
367
|
+
print_error "Claude CLI crashed $max_crash_retries times - stopping loop"
|
|
368
|
+
log_progress "$story" "CLI_CRASH" "Gave up after $max_crash_retries crashes"
|
|
369
|
+
rm -f "$prompt_file"
|
|
370
|
+
echo ""
|
|
371
|
+
echo "Claude CLI is unstable. Try again with: ralph run $story"
|
|
372
|
+
return 1
|
|
373
|
+
fi
|
|
374
|
+
|
|
375
|
+
if [[ $claude_exit_code -ne 0 ]]; then
|
|
340
376
|
print_warning "Claude session ended (timeout or error)"
|
|
341
377
|
log_progress "$story" "TIMEOUT" "Claude session ended after ${timeout_seconds}s"
|
|
342
378
|
rm -f "$prompt_file"
|
package/ralph/utils.sh
CHANGED
|
@@ -181,6 +181,7 @@ run_with_timeout() {
|
|
|
181
181
|
fi
|
|
182
182
|
}
|
|
183
183
|
|
|
184
|
+
|
|
184
185
|
# Safely update JSON file atomically
|
|
185
186
|
# Usage: update_json <file> [jq args...] <filter>
|
|
186
187
|
# Example: update_json file.json --arg id "TASK-001" '.stories[] | select(.id==$id)'
|
|
@@ -191,12 +192,25 @@ update_json() {
|
|
|
191
192
|
tmpfile=$(mktemp)
|
|
192
193
|
lockdir="${file}.lock"
|
|
193
194
|
|
|
195
|
+
# Remove stale locks (from crashed processes)
|
|
196
|
+
if [[ -d "$lockdir" ]]; then
|
|
197
|
+
local lock_age=0
|
|
198
|
+
local now=$(date +%s)
|
|
199
|
+
# Cross-platform: macOS uses -f %m, Linux uses -c %Y
|
|
200
|
+
local lock_mtime=$(stat -f %m "$lockdir" 2>/dev/null || stat -c %Y "$lockdir" 2>/dev/null || echo "$now")
|
|
201
|
+
lock_age=$((now - lock_mtime))
|
|
202
|
+
if [[ $lock_age -gt 30 ]]; then
|
|
203
|
+
print_warning "Removing stale lock (${lock_age}s old): $lockdir"
|
|
204
|
+
rm -rf "$lockdir"
|
|
205
|
+
fi
|
|
206
|
+
fi
|
|
207
|
+
|
|
194
208
|
# Acquire lock (mkdir is atomic)
|
|
195
209
|
local attempts=0
|
|
196
210
|
while ! mkdir "$lockdir" 2>/dev/null; do
|
|
197
211
|
((attempts++))
|
|
198
212
|
if [[ $attempts -gt 50 ]]; then
|
|
199
|
-
print_error "Could not acquire lock on $file"
|
|
213
|
+
print_error "Could not acquire lock on $file (locked for 5s+)"
|
|
200
214
|
rm -f "$tmpfile"
|
|
201
215
|
return 1
|
|
202
216
|
fi
|
|
@@ -495,6 +509,26 @@ validate_prd() {
|
|
|
495
509
|
print_warning "PRD is missing feature name (will show as 'unnamed')"
|
|
496
510
|
fi
|
|
497
511
|
|
|
512
|
+
# Check for grep-only testSteps (the #1 cause of false passes)
|
|
513
|
+
# Matches: grep, test -f/-e/-d, [ -f file ], [[ -f file ]]
|
|
514
|
+
local grep_only_stories
|
|
515
|
+
grep_only_stories=$(jq -r '
|
|
516
|
+
.stories[] |
|
|
517
|
+
select(.testSteps != null and (.testSteps | length > 0)) |
|
|
518
|
+
select(.testSteps | all(test("^(grep|test\\s+-[fed]|\\[\\[?\\s+-[fed])"; "x"))) |
|
|
519
|
+
.id
|
|
520
|
+
' "$prd_file" 2>/dev/null)
|
|
521
|
+
|
|
522
|
+
if [[ -n "$grep_only_stories" ]]; then
|
|
523
|
+
print_warning "These stories have grep-only testSteps (may cause false passes):"
|
|
524
|
+
echo "$grep_only_stories" | while read -r story_id; do
|
|
525
|
+
[[ -n "$story_id" ]] && echo " - $story_id"
|
|
526
|
+
done
|
|
527
|
+
echo ""
|
|
528
|
+
echo "Grep verifies code exists, not that it works. Add curl/playwright tests."
|
|
529
|
+
echo ""
|
|
530
|
+
fi
|
|
531
|
+
|
|
498
532
|
return 0
|
|
499
533
|
}
|
|
500
534
|
|
|
@@ -537,6 +571,12 @@ detect_migration_tool() {
|
|
|
537
571
|
return 0
|
|
538
572
|
fi
|
|
539
573
|
|
|
574
|
+
# Ecto (Elixir/Phoenix)
|
|
575
|
+
if [[ -f "$search_dir/mix.exs" ]] && [[ -d "$search_dir/priv/repo/migrations" ]]; then
|
|
576
|
+
echo "cd $search_dir && mix ecto.migrate"
|
|
577
|
+
return 0
|
|
578
|
+
fi
|
|
579
|
+
|
|
540
580
|
# Prisma (Node.js)
|
|
541
581
|
if [[ -d "$search_dir/prisma/migrations" ]] || [[ -f "$search_dir/prisma/schema.prisma" ]]; then
|
|
542
582
|
echo "cd $search_dir && npx prisma migrate deploy"
|
|
@@ -0,0 +1,279 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# shellcheck shell=bash
|
|
3
|
+
# api.sh - API and frontend smoke test verification module for ralph
|
|
4
|
+
#
|
|
5
|
+
# Catches broken APIs/frontends that unit tests miss (because they mock everything).
|
|
6
|
+
# Uses config.json for endpoints - no project-specific hardcoding.
|
|
7
|
+
|
|
8
|
+
# Run API smoke test against configured endpoints
|
|
9
|
+
# Config options:
|
|
10
|
+
# api.baseUrl - Base URL (e.g., http://localhost:8001)
|
|
11
|
+
# api.healthEndpoint - Health check path (e.g., /health)
|
|
12
|
+
# api.smokeEndpoints - Array of paths to test (e.g., ["/api/v1/users", "/api/v1/items"])
|
|
13
|
+
#
|
|
14
|
+
# Also tests story-specific apiEndpoints from PRD if defined.
|
|
15
|
+
run_api_smoke_test() {
|
|
16
|
+
local story="$1"
|
|
17
|
+
|
|
18
|
+
# Check if API smoke tests are enabled (default: true if baseUrl configured)
|
|
19
|
+
local base_url
|
|
20
|
+
base_url=$(get_config '.api.baseUrl' "")
|
|
21
|
+
|
|
22
|
+
# No API configured, skip silently
|
|
23
|
+
[[ -z "$base_url" ]] && return 0
|
|
24
|
+
|
|
25
|
+
echo ""
|
|
26
|
+
echo " [4/4] Running API smoke tests..."
|
|
27
|
+
|
|
28
|
+
local failed=0
|
|
29
|
+
local endpoints_tested=0
|
|
30
|
+
|
|
31
|
+
# 1. Health endpoint (most important)
|
|
32
|
+
local health_endpoint
|
|
33
|
+
health_endpoint=$(get_config '.api.healthEndpoint' "/health")
|
|
34
|
+
if [[ -n "$health_endpoint" ]]; then
|
|
35
|
+
if ! _smoke_test_endpoint "$base_url" "$health_endpoint" "health"; then
|
|
36
|
+
failed=1
|
|
37
|
+
fi
|
|
38
|
+
((endpoints_tested++))
|
|
39
|
+
fi
|
|
40
|
+
|
|
41
|
+
# 2. Configured smoke endpoints
|
|
42
|
+
local smoke_endpoints
|
|
43
|
+
smoke_endpoints=$(get_config '.api.smokeEndpoints' "[]")
|
|
44
|
+
if [[ "$smoke_endpoints" != "[]" && "$smoke_endpoints" != "null" ]]; then
|
|
45
|
+
while IFS= read -r endpoint; do
|
|
46
|
+
[[ -z "$endpoint" ]] && continue
|
|
47
|
+
if ! _smoke_test_endpoint "$base_url" "$endpoint" "smoke"; then
|
|
48
|
+
failed=1
|
|
49
|
+
fi
|
|
50
|
+
((endpoints_tested++))
|
|
51
|
+
done < <(echo "$smoke_endpoints" | jq -r '.[]' 2>/dev/null)
|
|
52
|
+
fi
|
|
53
|
+
|
|
54
|
+
# 3. Story-specific apiEndpoints from PRD
|
|
55
|
+
local story_endpoints
|
|
56
|
+
story_endpoints=$(jq -r --arg id "$story" '.stories[] | select(.id==$id) | .apiEndpoints[]?' "$RALPH_DIR/prd.json" 2>/dev/null)
|
|
57
|
+
if [[ -n "$story_endpoints" ]]; then
|
|
58
|
+
while IFS= read -r endpoint_spec; do
|
|
59
|
+
[[ -z "$endpoint_spec" ]] && continue
|
|
60
|
+
# Format: "GET /api/v1/users" or "POST /api/v1/items" or just "/api/v1/users"
|
|
61
|
+
local method="GET"
|
|
62
|
+
local path="$endpoint_spec"
|
|
63
|
+
if [[ "$endpoint_spec" =~ ^(GET|POST|PUT|DELETE|PATCH)[[:space:]]+(.*) ]]; then
|
|
64
|
+
method="${BASH_REMATCH[1]}"
|
|
65
|
+
path="${BASH_REMATCH[2]}"
|
|
66
|
+
fi
|
|
67
|
+
if ! _smoke_test_endpoint "$base_url" "$path" "story" "$method"; then
|
|
68
|
+
failed=1
|
|
69
|
+
fi
|
|
70
|
+
((endpoints_tested++))
|
|
71
|
+
done <<< "$story_endpoints"
|
|
72
|
+
fi
|
|
73
|
+
|
|
74
|
+
if [[ $endpoints_tested -eq 0 ]]; then
|
|
75
|
+
echo " (no endpoints configured, skipping)"
|
|
76
|
+
return 0
|
|
77
|
+
fi
|
|
78
|
+
|
|
79
|
+
return $failed
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
# Run frontend smoke test
|
|
83
|
+
# Config options:
|
|
84
|
+
# urls.frontend - Frontend URL (e.g., http://localhost:3000)
|
|
85
|
+
# frontend.smokePages - Array of paths to test (e.g., ["/", "/login", "/dashboard"])
|
|
86
|
+
run_frontend_smoke_test() {
|
|
87
|
+
local story="$1"
|
|
88
|
+
local story_type="${RALPH_STORY_TYPE:-general}"
|
|
89
|
+
|
|
90
|
+
# Get frontend URL from config (try multiple locations)
|
|
91
|
+
local frontend_url
|
|
92
|
+
frontend_url=$(get_config '.urls.frontend' "")
|
|
93
|
+
[[ -z "$frontend_url" ]] && frontend_url=$(get_config '.playwright.baseUrl' "")
|
|
94
|
+
|
|
95
|
+
# No frontend configured, skip silently
|
|
96
|
+
[[ -z "$frontend_url" ]] && return 0
|
|
97
|
+
|
|
98
|
+
# Skip for backend-only stories (optional optimization)
|
|
99
|
+
# [[ "$story_type" == "backend" ]] && return 0
|
|
100
|
+
|
|
101
|
+
echo ""
|
|
102
|
+
echo " [5/5] Running frontend smoke tests..."
|
|
103
|
+
|
|
104
|
+
local failed=0
|
|
105
|
+
local pages_tested=0
|
|
106
|
+
|
|
107
|
+
# 1. Test root page (most important)
|
|
108
|
+
if ! _smoke_test_page "$frontend_url" "/" "root"; then
|
|
109
|
+
failed=1
|
|
110
|
+
fi
|
|
111
|
+
((pages_tested++))
|
|
112
|
+
|
|
113
|
+
# 2. Configured smoke pages
|
|
114
|
+
local smoke_pages
|
|
115
|
+
smoke_pages=$(get_config '.frontend.smokePages' "[]")
|
|
116
|
+
if [[ "$smoke_pages" != "[]" && "$smoke_pages" != "null" ]]; then
|
|
117
|
+
while IFS= read -r page; do
|
|
118
|
+
[[ -z "$page" ]] && continue
|
|
119
|
+
[[ "$page" == "/" ]] && continue # Already tested root
|
|
120
|
+
if ! _smoke_test_page "$frontend_url" "$page" "smoke"; then
|
|
121
|
+
failed=1
|
|
122
|
+
fi
|
|
123
|
+
((pages_tested++))
|
|
124
|
+
done < <(echo "$smoke_pages" | jq -r '.[]' 2>/dev/null)
|
|
125
|
+
fi
|
|
126
|
+
|
|
127
|
+
# 3. Story-specific testUrl from PRD
|
|
128
|
+
local test_url
|
|
129
|
+
test_url=$(jq -r --arg id "$story" '.stories[] | select(.id==$id) | .testUrl // empty' "$RALPH_DIR/prd.json" 2>/dev/null)
|
|
130
|
+
if [[ -n "$test_url" ]]; then
|
|
131
|
+
# testUrl can be full URL or just path
|
|
132
|
+
if [[ "$test_url" =~ ^https?:// ]]; then
|
|
133
|
+
if ! _smoke_test_page "" "$test_url" "story"; then
|
|
134
|
+
failed=1
|
|
135
|
+
fi
|
|
136
|
+
else
|
|
137
|
+
if ! _smoke_test_page "$frontend_url" "$test_url" "story"; then
|
|
138
|
+
failed=1
|
|
139
|
+
fi
|
|
140
|
+
fi
|
|
141
|
+
((pages_tested++))
|
|
142
|
+
fi
|
|
143
|
+
|
|
144
|
+
return $failed
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
# Test a single page
|
|
148
|
+
# Usage: _smoke_test_page <base_url> <path> <type>
|
|
149
|
+
_smoke_test_page() {
|
|
150
|
+
local base_url="$1"
|
|
151
|
+
local path="$2"
|
|
152
|
+
local test_type="$3"
|
|
153
|
+
|
|
154
|
+
local url
|
|
155
|
+
if [[ -z "$base_url" ]]; then
|
|
156
|
+
url="$path"
|
|
157
|
+
else
|
|
158
|
+
url="${base_url}${path}"
|
|
159
|
+
fi
|
|
160
|
+
|
|
161
|
+
echo -n " GET $path... "
|
|
162
|
+
|
|
163
|
+
local response_file
|
|
164
|
+
response_file=$(mktemp)
|
|
165
|
+
local http_code
|
|
166
|
+
|
|
167
|
+
http_code=$(curl -s -o "$response_file" -w "%{http_code}" \
|
|
168
|
+
--max-time "$CURL_TIMEOUT_SECONDS" \
|
|
169
|
+
"$url" 2>/dev/null) || http_code="000"
|
|
170
|
+
|
|
171
|
+
# Check response
|
|
172
|
+
if [[ "$http_code" == "000" ]]; then
|
|
173
|
+
print_error "connection failed (is frontend running?)"
|
|
174
|
+
rm -f "$response_file"
|
|
175
|
+
return 1
|
|
176
|
+
elif [[ "$http_code" =~ ^5 ]]; then
|
|
177
|
+
print_error "HTTP $http_code - Server Error"
|
|
178
|
+
|
|
179
|
+
# Save for failure context
|
|
180
|
+
{
|
|
181
|
+
echo "Frontend smoke test failed: $url"
|
|
182
|
+
echo "HTTP Status: $http_code"
|
|
183
|
+
echo "Response (first 50 lines):"
|
|
184
|
+
head -50 "$response_file"
|
|
185
|
+
} >> "$RALPH_DIR/last_frontend_failure.log"
|
|
186
|
+
|
|
187
|
+
rm -f "$response_file"
|
|
188
|
+
return 1
|
|
189
|
+
elif [[ "$http_code" =~ ^4 ]]; then
|
|
190
|
+
# 404 on a specific page is a real error (unlike API auth)
|
|
191
|
+
if [[ "$http_code" == "404" ]]; then
|
|
192
|
+
print_error "HTTP 404 - Page not found"
|
|
193
|
+
rm -f "$response_file"
|
|
194
|
+
return 1
|
|
195
|
+
fi
|
|
196
|
+
# Other 4xx (401, 403) might be OK - auth required
|
|
197
|
+
print_warning "HTTP $http_code (may need auth)"
|
|
198
|
+
rm -f "$response_file"
|
|
199
|
+
return 0
|
|
200
|
+
else
|
|
201
|
+
# Check for React/Next.js error boundary or crash indicators
|
|
202
|
+
if grep -qi "application error\|something went wrong\|error boundary\|chunk load error" "$response_file" 2>/dev/null; then
|
|
203
|
+
print_error "HTTP $http_code but page shows error"
|
|
204
|
+
{
|
|
205
|
+
echo "Frontend smoke test failed: $url"
|
|
206
|
+
echo "Page loaded but contains error indicators"
|
|
207
|
+
head -50 "$response_file"
|
|
208
|
+
} >> "$RALPH_DIR/last_frontend_failure.log"
|
|
209
|
+
rm -f "$response_file"
|
|
210
|
+
return 1
|
|
211
|
+
fi
|
|
212
|
+
print_success "HTTP $http_code"
|
|
213
|
+
rm -f "$response_file"
|
|
214
|
+
return 0
|
|
215
|
+
fi
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
# Test a single endpoint
|
|
219
|
+
# Usage: _smoke_test_endpoint <base_url> <path> <type> [method]
|
|
220
|
+
_smoke_test_endpoint() {
|
|
221
|
+
local base_url="$1"
|
|
222
|
+
local path="$2"
|
|
223
|
+
local test_type="$3"
|
|
224
|
+
local method="${4:-GET}"
|
|
225
|
+
|
|
226
|
+
local url="${base_url}${path}"
|
|
227
|
+
echo -n " $method $path... "
|
|
228
|
+
|
|
229
|
+
local response_file
|
|
230
|
+
response_file=$(mktemp)
|
|
231
|
+
local http_code
|
|
232
|
+
|
|
233
|
+
# Make request with timeout, capture status code
|
|
234
|
+
if [[ "$method" == "GET" ]]; then
|
|
235
|
+
http_code=$(curl -s -o "$response_file" -w "%{http_code}" \
|
|
236
|
+
--max-time "$CURL_TIMEOUT_SECONDS" \
|
|
237
|
+
"$url" 2>/dev/null) || http_code="000"
|
|
238
|
+
else
|
|
239
|
+
# For non-GET, just check endpoint exists (OPTIONS or empty body)
|
|
240
|
+
http_code=$(curl -s -o "$response_file" -w "%{http_code}" \
|
|
241
|
+
--max-time "$CURL_TIMEOUT_SECONDS" \
|
|
242
|
+
-X "$method" \
|
|
243
|
+
-H "Content-Type: application/json" \
|
|
244
|
+
-d '{}' \
|
|
245
|
+
"$url" 2>/dev/null) || http_code="000"
|
|
246
|
+
fi
|
|
247
|
+
|
|
248
|
+
# Check response
|
|
249
|
+
if [[ "$http_code" == "000" ]]; then
|
|
250
|
+
print_error "connection failed (is server running?)"
|
|
251
|
+
rm -f "$response_file"
|
|
252
|
+
return 1
|
|
253
|
+
elif [[ "$http_code" =~ ^5 ]]; then
|
|
254
|
+
print_error "HTTP $http_code - Internal Server Error"
|
|
255
|
+
echo ""
|
|
256
|
+
echo " Response body:"
|
|
257
|
+
head -20 "$response_file" | sed 's/^/ /'
|
|
258
|
+
|
|
259
|
+
# Save for failure context
|
|
260
|
+
{
|
|
261
|
+
echo "API smoke test failed: $method $url"
|
|
262
|
+
echo "HTTP Status: $http_code"
|
|
263
|
+
echo "Response:"
|
|
264
|
+
cat "$response_file"
|
|
265
|
+
} >> "$RALPH_DIR/last_api_failure.log"
|
|
266
|
+
|
|
267
|
+
rm -f "$response_file"
|
|
268
|
+
return 1
|
|
269
|
+
elif [[ "$http_code" =~ ^4 ]]; then
|
|
270
|
+
# 4xx might be OK (auth required, etc.) - warn but don't fail
|
|
271
|
+
print_warning "HTTP $http_code (may need auth)"
|
|
272
|
+
rm -f "$response_file"
|
|
273
|
+
return 0
|
|
274
|
+
else
|
|
275
|
+
print_success "HTTP $http_code"
|
|
276
|
+
rm -f "$response_file"
|
|
277
|
+
return 0
|
|
278
|
+
fi
|
|
279
|
+
}
|
package/ralph/verify/lint.sh
CHANGED
|
@@ -355,6 +355,46 @@ verify_go() {
|
|
|
355
355
|
return 0
|
|
356
356
|
}
|
|
357
357
|
|
|
358
|
+
# Verify Elixir code with mix credo
|
|
359
|
+
verify_elixir() {
|
|
360
|
+
local elixir_log="$RALPH_DIR/last_elixir_failure.log"
|
|
361
|
+
|
|
362
|
+
# Skip if not an Elixir project
|
|
363
|
+
[[ ! -f "mix.exs" ]] && return 0
|
|
364
|
+
command -v mix &>/dev/null || return 0
|
|
365
|
+
|
|
366
|
+
# Clear previous failure log
|
|
367
|
+
rm -f "$elixir_log"
|
|
368
|
+
|
|
369
|
+
# Mix credo (Elixir's static analysis tool)
|
|
370
|
+
echo -n " Mix credo... "
|
|
371
|
+
local credo_output
|
|
372
|
+
if credo_output=$(mix credo --strict 2>&1); then
|
|
373
|
+
print_success "passed"
|
|
374
|
+
return 0
|
|
375
|
+
fi
|
|
376
|
+
|
|
377
|
+
# Check if credo is installed
|
|
378
|
+
if echo "$credo_output" | grep -qi "could not find.*credo\|mix credo.*not found"; then
|
|
379
|
+
echo -n "not installed, trying mix compile... "
|
|
380
|
+
if credo_output=$(mix compile --warnings-as-errors 2>&1); then
|
|
381
|
+
print_success "passed"
|
|
382
|
+
return 0
|
|
383
|
+
fi
|
|
384
|
+
fi
|
|
385
|
+
|
|
386
|
+
# Failed
|
|
387
|
+
print_error "failed"
|
|
388
|
+
echo ""
|
|
389
|
+
echo " Elixir errors:"
|
|
390
|
+
echo "$credo_output" | head -"$MAX_LINT_ERROR_LINES" | sed 's/^/ /'
|
|
391
|
+
{
|
|
392
|
+
echo "Elixir errors:"
|
|
393
|
+
echo "$credo_output"
|
|
394
|
+
} >> "$elixir_log"
|
|
395
|
+
return 1
|
|
396
|
+
}
|
|
397
|
+
|
|
358
398
|
# Verify Rust code with clippy
|
|
359
399
|
verify_rust() {
|
|
360
400
|
local rust_log="$RALPH_DIR/last_rust_failure.log"
|
|
@@ -505,6 +545,9 @@ run_configured_checks() {
|
|
|
505
545
|
if ! verify_rust; then
|
|
506
546
|
return 1
|
|
507
547
|
fi
|
|
548
|
+
if ! verify_elixir; then
|
|
549
|
+
return 1
|
|
550
|
+
fi
|
|
508
551
|
fi
|
|
509
552
|
|
|
510
553
|
# FastAPI response model check
|
package/ralph/verify/tests.sh
CHANGED
package/ralph/verify.sh
CHANGED
|
@@ -9,6 +9,7 @@
|
|
|
9
9
|
VERIFY_DIR="${RALPH_LIB:-$(dirname "${BASH_SOURCE[0]}")}"
|
|
10
10
|
source "$VERIFY_DIR/verify/lint.sh"
|
|
11
11
|
source "$VERIFY_DIR/verify/tests.sh"
|
|
12
|
+
source "$VERIFY_DIR/verify/api.sh"
|
|
12
13
|
|
|
13
14
|
run_verification() {
|
|
14
15
|
local story="$1"
|
|
@@ -27,7 +28,7 @@ run_verification() {
|
|
|
27
28
|
# ========================================
|
|
28
29
|
# STEP 1: Run lint checks
|
|
29
30
|
# ========================================
|
|
30
|
-
echo " [1/
|
|
31
|
+
echo " [1/5] Running lint checks..."
|
|
31
32
|
if ! run_configured_checks "$story_type"; then
|
|
32
33
|
failed=1
|
|
33
34
|
fi
|
|
@@ -37,7 +38,7 @@ run_verification() {
|
|
|
37
38
|
# ========================================
|
|
38
39
|
if [[ $failed -eq 0 ]]; then
|
|
39
40
|
echo ""
|
|
40
|
-
echo " [2/
|
|
41
|
+
echo " [2/5] Running tests..."
|
|
41
42
|
# First check that test files exist for new code
|
|
42
43
|
if ! verify_test_files_exist; then
|
|
43
44
|
failed=1
|
|
@@ -51,12 +52,30 @@ run_verification() {
|
|
|
51
52
|
# ========================================
|
|
52
53
|
if [[ $failed -eq 0 ]]; then
|
|
53
54
|
echo ""
|
|
54
|
-
echo " [3/
|
|
55
|
+
echo " [3/5] Running PRD test steps..."
|
|
55
56
|
if ! verify_prd_criteria "$story"; then
|
|
56
57
|
failed=1
|
|
57
58
|
fi
|
|
58
59
|
fi
|
|
59
60
|
|
|
61
|
+
# ========================================
|
|
62
|
+
# STEP 4: API smoke test (if configured)
|
|
63
|
+
# ========================================
|
|
64
|
+
if [[ $failed -eq 0 ]]; then
|
|
65
|
+
if ! run_api_smoke_test "$story"; then
|
|
66
|
+
failed=1
|
|
67
|
+
fi
|
|
68
|
+
fi
|
|
69
|
+
|
|
70
|
+
# ========================================
|
|
71
|
+
# STEP 5: Frontend smoke test (if configured)
|
|
72
|
+
# ========================================
|
|
73
|
+
if [[ $failed -eq 0 ]]; then
|
|
74
|
+
if ! run_frontend_smoke_test "$story"; then
|
|
75
|
+
failed=1
|
|
76
|
+
fi
|
|
77
|
+
fi
|
|
78
|
+
|
|
60
79
|
# ========================================
|
|
61
80
|
# Final result
|
|
62
81
|
# ========================================
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
{
|
|
2
|
+
"auth": {
|
|
3
|
+
"testUser": "",
|
|
4
|
+
"testPassword": "",
|
|
5
|
+
"loginEndpoint": "/api/auth/login",
|
|
6
|
+
"loginMethod": "POST",
|
|
7
|
+
"tokenType": "jwt",
|
|
8
|
+
"tokenHeader": "Authorization",
|
|
9
|
+
"tokenPrefix": "Bearer"
|
|
10
|
+
},
|
|
11
|
+
|
|
12
|
+
"docker": {
|
|
13
|
+
"enabled": false,
|
|
14
|
+
"composeFile": "docker-compose.yml",
|
|
15
|
+
"serviceName": "app",
|
|
16
|
+
"execPrefix": "docker compose exec -T"
|
|
17
|
+
},
|
|
18
|
+
|
|
19
|
+
"paths": {
|
|
20
|
+
"src": "lib",
|
|
21
|
+
"tests": "test",
|
|
22
|
+
"e2e": "test/e2e"
|
|
23
|
+
},
|
|
24
|
+
|
|
25
|
+
"commands": {
|
|
26
|
+
"dev": "mix phx.server",
|
|
27
|
+
"install": "mix deps.get",
|
|
28
|
+
"seed": "mix run priv/repo/seeds.exs",
|
|
29
|
+
"resetDb": "mix ecto.reset"
|
|
30
|
+
},
|
|
31
|
+
|
|
32
|
+
"migrations": {
|
|
33
|
+
"command": "mix ecto.migrate",
|
|
34
|
+
"pattern": "priv/repo/migrations/.*\\.exs$"
|
|
35
|
+
},
|
|
36
|
+
|
|
37
|
+
"checks": {
|
|
38
|
+
"lint": true,
|
|
39
|
+
"typecheck": false,
|
|
40
|
+
"build": true,
|
|
41
|
+
"test": true,
|
|
42
|
+
"fastapi": false
|
|
43
|
+
},
|
|
44
|
+
|
|
45
|
+
"api": {
|
|
46
|
+
"baseUrl": "http://localhost:4000",
|
|
47
|
+
"healthEndpoint": "/api/health",
|
|
48
|
+
"smokeEndpoints": [],
|
|
49
|
+
"timeout": 30
|
|
50
|
+
},
|
|
51
|
+
|
|
52
|
+
"playwright": {
|
|
53
|
+
"enabled": false,
|
|
54
|
+
"testDir": "test/e2e",
|
|
55
|
+
"projects": ["chromium"],
|
|
56
|
+
"baseUrl": "http://localhost:4000"
|
|
57
|
+
},
|
|
58
|
+
|
|
59
|
+
"verification": {
|
|
60
|
+
"codeReviewEnabled": true,
|
|
61
|
+
"browserEnabled": true,
|
|
62
|
+
"a11yEnabled": false,
|
|
63
|
+
"mobileViewport": 375,
|
|
64
|
+
"screenshotOnFailure": true
|
|
65
|
+
},
|
|
66
|
+
|
|
67
|
+
"urls": {
|
|
68
|
+
"app": "http://localhost:4000",
|
|
69
|
+
"docs": "http://localhost:4000/dev/dashboard"
|
|
70
|
+
},
|
|
71
|
+
|
|
72
|
+
"env": {
|
|
73
|
+
"required": ["DATABASE_URL"],
|
|
74
|
+
"optional": ["SECRET_KEY_BASE", "PHX_HOST"]
|
|
75
|
+
},
|
|
76
|
+
|
|
77
|
+
"maxIterations": 20,
|
|
78
|
+
"maxSessionSeconds": 600,
|
|
79
|
+
|
|
80
|
+
"contextRotThreshold": {
|
|
81
|
+
"maxStories": 10,
|
|
82
|
+
"maxFilesChanged": 20
|
|
83
|
+
}
|
|
84
|
+
}
|
|
@@ -47,6 +47,7 @@
|
|
|
47
47
|
"api": {
|
|
48
48
|
"baseUrl": "http://localhost:8000",
|
|
49
49
|
"healthEndpoint": "/api/health",
|
|
50
|
+
"smokeEndpoints": [],
|
|
50
51
|
"timeout": 30
|
|
51
52
|
},
|
|
52
53
|
|
|
@@ -71,6 +72,10 @@
|
|
|
71
72
|
"docs": "http://localhost:8000/api/docs"
|
|
72
73
|
},
|
|
73
74
|
|
|
75
|
+
"frontend": {
|
|
76
|
+
"smokePages": ["/", "/login"]
|
|
77
|
+
},
|
|
78
|
+
|
|
74
79
|
"env": {
|
|
75
80
|
"required": ["DATABASE_URL", "SECRET_KEY"],
|
|
76
81
|
"optional": ["REDIS_URL", "SENTRY_DSN"]
|
|
@@ -45,6 +45,7 @@
|
|
|
45
45
|
"api": {
|
|
46
46
|
"baseUrl": "http://localhost:3000",
|
|
47
47
|
"healthEndpoint": "/api/health",
|
|
48
|
+
"smokeEndpoints": [],
|
|
48
49
|
"timeout": 30
|
|
49
50
|
},
|
|
50
51
|
|
|
@@ -65,9 +66,14 @@
|
|
|
65
66
|
|
|
66
67
|
"urls": {
|
|
67
68
|
"app": "http://localhost:3000",
|
|
69
|
+
"frontend": "http://localhost:3000",
|
|
68
70
|
"docs": "http://localhost:3000/api/docs"
|
|
69
71
|
},
|
|
70
72
|
|
|
73
|
+
"frontend": {
|
|
74
|
+
"smokePages": ["/"]
|
|
75
|
+
},
|
|
76
|
+
|
|
71
77
|
"env": {
|
|
72
78
|
"required": ["DATABASE_URL"],
|
|
73
79
|
"optional": ["REDIS_URL", "SENTRY_DSN"]
|