ace-demo 0.23.2 → 0.25.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +76 -0
- data/README.md +21 -0
- data/handbook/skills/as-demo-analyze-cast/SKILL.md +27 -0
- data/handbook/workflow-instructions/demo/analyze-cast.wf.md +53 -0
- data/handbook/workflow-instructions/demo/create.wf.md +32 -6
- data/handbook/workflow-instructions/demo/record.wf.md +64 -2
- data/lib/ace/demo/atoms/demo_yaml_parser.rb +36 -1
- data/lib/ace/demo/cli/commands/record.rb +34 -2
- data/lib/ace/demo/cli/commands/verify.rb +70 -0
- data/lib/ace/demo/cli.rb +4 -0
- data/lib/ace/demo/models/recording_result.rb +4 -2
- data/lib/ace/demo/models/verification_result.rb +12 -2
- data/lib/ace/demo/molecules/cast_verifier.rb +144 -6
- data/lib/ace/demo/molecules/recording_manifest_writer.rb +47 -0
- data/lib/ace/demo/molecules/verification_report_writer.rb +118 -0
- data/lib/ace/demo/molecules/vhs_executor.rb +34 -1
- data/lib/ace/demo/organisms/demo_recorder.rb +25 -6
- data/lib/ace/demo/version.rb +1 -1
- data/lib/ace/demo.rb +2 -0
- metadata +7 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ae9f0aced46022fd683e573a13e899c0e0e5d119e1d50cd65903f0e1188983ec
|
|
4
|
+
data.tar.gz: 769e24a54a5c0162e30fa691ad378161c917f3774f87e1ed066f3a39a68a58ca
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a1d750f6d1c26789d9ed615352536aece06fd42dbf2b9b4619aa8db5bb6d889cbb629034f2217ba81c4630c59b77360bc1a76cf6adb638a5e6a8e828fb0bc606
|
|
7
|
+
data.tar.gz: '0368a6c97f32dcccd9a49e429e65b8b90ba45f0590d6dbc9bc3acd6ff9560ed04f4d7c630d2ea3dedb83de0356bf718f3aef0b9090baa157326a446ee9ce509f'
|
data/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,82 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.25.1] - 2026-04-16
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
- Exported resolved browser executable variables for VHS runs and aligned non-browser E2E verification with explicit constrained-environment failure evidence.
|
|
14
|
+
|
|
15
|
+
## [0.25.0] - 2026-04-14
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
- Rewrote `TS-DEMO-001` smoke E2E coverage around public-surface goal contracts by adding non-dry-run preset recording success coverage and stabilizing dry-run and missing-`--pr` verification assertions.
|
|
19
|
+
|
|
20
|
+
### Technical
|
|
21
|
+
- Added `TC-005-record-preset-success-artifact` runner/verifier assets and updated scenario/aggregate E2E manifests to execute and verify 5 goals.
|
|
22
|
+
- Updated scenario setup to use `${ACE_E2E_SOURCE_ROOT:-$PROJECT_ROOT_PATH}` for resilient `mise.toml` bootstrap in package sandbox runs.
|
|
23
|
+
|
|
24
|
+
## [0.24.6] - 2026-04-15
|
|
25
|
+
|
|
26
|
+
### Fixed
|
|
27
|
+
- Made `ace-demo record` print verification summaries safely when optional failure detail arrays are absent, so reruns surface the real verification outcome instead of crashing in CLI output formatting.
|
|
28
|
+
|
|
29
|
+
## [0.24.5] - 2026-04-15
|
|
30
|
+
|
|
31
|
+
### Added
|
|
32
|
+
- Added `ace-demo verify` plus a dedicated cast-analysis workflow so failed asciinema recordings can be re-verified and routed to scenario, product, or verifier fixes before re-recording.
|
|
33
|
+
|
|
34
|
+
### Changed
|
|
35
|
+
- Expanded YAML demo verification with required visible output and ordered output-transition checks, renamed scenario-side failures to `scenario_defect`, and preserved failed recording sandboxes/manifests for follow-up triage.
|
|
36
|
+
|
|
37
|
+
## [0.24.4] - 2026-04-15
|
|
38
|
+
|
|
39
|
+
### Changed
|
|
40
|
+
- Tightened demo planning and recording workflows so tmux/UI demos define the visible transition up front and validate the operator viewpoint before recording.
|
|
41
|
+
|
|
42
|
+
### Technical
|
|
43
|
+
- Refreshed the handbook demo workflow instructions to reject recordings that miss the visible `before -> trigger -> effect -> after` contract for state-transition demos.
|
|
44
|
+
|
|
45
|
+
## [0.24.3] - 2026-04-13
|
|
46
|
+
|
|
47
|
+
### Changed
|
|
48
|
+
- Completed the batch i05 migration follow-through for this package and aligned it with the restarted `fast` / `feat` / `e2e` verification model.
|
|
49
|
+
|
|
50
|
+
### Technical
|
|
51
|
+
- Included in the coordinated assignment-driven patch release for batch i05 package updates.
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
## [0.24.2] - 2026-04-12
|
|
55
|
+
|
|
56
|
+
### Changed
|
|
57
|
+
- Migrated deterministic `ace-demo` coverage to `test/fast`, kept scenario assets in `test/e2e`, and aligned E2E metadata references to the new fast test paths.
|
|
58
|
+
- Added package-level testing contract guidance for `ace-test ace-demo`, `ace-test ace-demo all`, and `ace-test-e2e ace-demo`.
|
|
59
|
+
|
|
60
|
+
## [0.24.1] - 2026-04-07
|
|
61
|
+
|
|
62
|
+
### Added
|
|
63
|
+
- Added fail-closed verification behavior for recording workflows with richer report classification when verification fails.
|
|
64
|
+
|
|
65
|
+
### Changed
|
|
66
|
+
- Updated demo CLI/recording paths and parser logic to preserve typed failure semantics through verification and recorder execution.
|
|
67
|
+
|
|
68
|
+
## [0.24.0] - 2026-04-07
|
|
69
|
+
|
|
70
|
+
### Added
|
|
71
|
+
- Added semantic `verify:` support for YAML/asciinema demo tapes, including required exported variables, forbidden output signatures, and final-state assertion commands.
|
|
72
|
+
- Added structured demo verification reports under `.ace-local/demo/` so failed recordings preserve actionable evidence for retry or bug triage.
|
|
73
|
+
|
|
74
|
+
### Changed
|
|
75
|
+
- Made `ace-demo record` fail closed on verification errors instead of treating cast mismatches as warning-only output.
|
|
76
|
+
- Classified recording failures as `instruction_defect`, `product_bug`, or `verification_error`, and blocked PR upload/comment when verification does not pass.
|
|
77
|
+
|
|
78
|
+
### Technical
|
|
79
|
+
- Expanded parser, verifier, recorder, and CLI coverage for semantic verification, report writing, and fail-closed record behavior.
|
|
80
|
+
|
|
81
|
+
## [0.23.3] - 2026-03-29
|
|
82
|
+
|
|
83
|
+
### Technical
|
|
84
|
+
- Normalized published gem metadata so RubyGems and Ruby Toolbox use current release information instead of the 1980 fallback date.
|
|
85
|
+
|
|
10
86
|
## [0.23.2] - 2026-03-29
|
|
11
87
|
|
|
12
88
|
### Technical
|
data/README.md
CHANGED
|
@@ -65,5 +65,26 @@ Legacy `.tape` files use raw VHS syntax directly. See the [Usage Guide](docs/usa
|
|
|
65
65
|
|
|
66
66
|
*Future: web interaction recording is planned alongside terminal capture.*
|
|
67
67
|
|
|
68
|
+
## Testing Contract
|
|
69
|
+
|
|
70
|
+
Run deterministic package coverage with:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
ace-test ace-demo
|
|
74
|
+
ace-test ace-demo all
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Run deterministic feature coverage only when `test/feat/` exists:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
ace-test ace-demo feat
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Run retained workflow scenarios in E2E:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
ace-test-e2e ace-demo
|
|
87
|
+
```
|
|
88
|
+
|
|
68
89
|
---
|
|
69
90
|
[Getting Started](docs/getting-started.md) | [Usage Guide](docs/usage.md) | [Handbook - Skills, Agents, Templates](docs/handbook.md) | Part of [ACE](https://github.com/cs3b/ace)
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: as-demo-analyze-cast
|
|
3
|
+
description: Analyze failed demo casts and route them to scenario, product, or verifier fixes
|
|
4
|
+
# bundle: wfi://demo/analyze-cast
|
|
5
|
+
# agent: general-purpose
|
|
6
|
+
user-invocable: true
|
|
7
|
+
allowed-tools:
|
|
8
|
+
- Bash(ace-demo:*)
|
|
9
|
+
- Bash(ace-bundle:*)
|
|
10
|
+
- Read
|
|
11
|
+
argument-hint: "<cast-file> --tape <tape-ref-or-path> [--sandbox-path <path>]"
|
|
12
|
+
last_modified: 2026-04-15
|
|
13
|
+
source: ace-demo
|
|
14
|
+
integration:
|
|
15
|
+
targets:
|
|
16
|
+
- claude
|
|
17
|
+
- codex
|
|
18
|
+
- gemini
|
|
19
|
+
- opencode
|
|
20
|
+
- pi
|
|
21
|
+
skill:
|
|
22
|
+
kind: workflow
|
|
23
|
+
execution:
|
|
24
|
+
workflow: wfi://demo/analyze-cast
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
Load and run `ace-bundle wfi://demo/analyze-cast` in the current project, then follow the loaded workflow as the source of truth and execute it end-to-end instead of only summarizing it.
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
---
|
|
2
|
+
doc-type: workflow
|
|
3
|
+
title: Analyze Demo Cast Workflow
|
|
4
|
+
purpose: cast verification triage workflow instruction
|
|
5
|
+
ace-docs:
|
|
6
|
+
last-updated: 2026-04-15
|
|
7
|
+
last-checked: 2026-04-15
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# Analyze Demo Cast Workflow
|
|
11
|
+
|
|
12
|
+
## Purpose
|
|
13
|
+
|
|
14
|
+
Analyze a recorded `.cast` after demo verification fails, decide whether the problem is the recording scenario, the product, or the verification tooling, and route to the correct next workflow before re-recording.
|
|
15
|
+
|
|
16
|
+
## Instructions
|
|
17
|
+
|
|
18
|
+
1. Run cast verification:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
ace-demo verify <cast-file> --tape <tape-ref-or-path>
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
If the failed recording preserved a sandbox and the tape uses `assert_commands`, include it:
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
ace-demo verify <cast-file> --tape <tape-ref-or-path> --sandbox-path <sandbox-path>
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
2. Read the generated report under `.ace-local/demo/` or the selected `--report-dir`.
|
|
31
|
+
|
|
32
|
+
3. Route by classification:
|
|
33
|
+
|
|
34
|
+
- `scenario_defect`:
|
|
35
|
+
- fix the tape, camera/viewpoint, setup, or `verify:` contract
|
|
36
|
+
- keep the product code unchanged
|
|
37
|
+
- re-record after updating the demo scenario
|
|
38
|
+
- `product_bug`:
|
|
39
|
+
- treat the cast as a real product/runtime failure
|
|
40
|
+
- run the normal bug analysis/fix workflow for the affected package
|
|
41
|
+
- re-record only after the product fix lands
|
|
42
|
+
- `verification_error`:
|
|
43
|
+
- fix the verifier, cast parser, tape metadata, or recorder/tooling problem
|
|
44
|
+
- re-run verification
|
|
45
|
+
- re-record only if the original cast is unusable or no longer representative
|
|
46
|
+
|
|
47
|
+
4. Do not upload or comment on a PR until `ace-demo verify` returns `pass`.
|
|
48
|
+
|
|
49
|
+
## Success Criteria
|
|
50
|
+
|
|
51
|
+
- Every failed demo recording is triaged through `ace-demo verify`
|
|
52
|
+
- The next action is unambiguous: scenario fix, product bug fix, or verifier/tooling fix
|
|
53
|
+
- Re-recording happens only after the correct upstream issue is fixed
|
|
@@ -29,7 +29,31 @@ Tapes are VHS script files that define terminal sessions: commands to type, timi
|
|
|
29
29
|
|
|
30
30
|
## Instructions
|
|
31
31
|
|
|
32
|
-
1. **
|
|
32
|
+
1. **Define the demo contract before scripting anything**:
|
|
33
|
+
|
|
34
|
+
Capture these decisions in notes or the task/PR before you create a tape:
|
|
35
|
+
|
|
36
|
+
- **Viewpoint**: what exact terminal or tmux client is being recorded
|
|
37
|
+
- **Starting state**: what must already be visible on screen before the trigger action
|
|
38
|
+
- **Trigger action**: the concrete command or interaction that causes the behavior
|
|
39
|
+
- **Visible reaction**: the system state change that must be seen in the recording
|
|
40
|
+
- **End state**: what the reviewer should understand after the recording finishes
|
|
41
|
+
|
|
42
|
+
For demos about tmux, panes, windows, or session routing, the contract must explicitly say which tmux client view is being recorded. A shell transcript alone is not sufficient if the feature is about visible tmux behavior.
|
|
43
|
+
|
|
44
|
+
2. **Keep setup off-camera unless setup is the feature**:
|
|
45
|
+
|
|
46
|
+
Pre-stage long or distracting setup before recording whenever possible:
|
|
47
|
+
|
|
48
|
+
- create fixtures, assignments, and helper scripts before the recording starts
|
|
49
|
+
- start from the state the viewer must recognize
|
|
50
|
+
- only leave setup on camera when the setup step itself is what the demo is proving
|
|
51
|
+
|
|
52
|
+
For tmux fork/window demos, prefer starting in the operator's normal work window and let the recording capture the visible transition into the fork window. Do not begin already attached to the fork window unless the feature is specifically about that attach flow.
|
|
53
|
+
|
|
54
|
+
If the demo is intended to be durable evidence for docs, review, or PR proof, prefer a committed `.tape.yml` / asciinema flow over ad-hoc inline/VHS capture so the resulting `.cast` can be re-verified later.
|
|
55
|
+
|
|
56
|
+
3. **Preview the tape** before writing:
|
|
33
57
|
|
|
34
58
|
```bash
|
|
35
59
|
ace-demo create <name> --dry-run -- "cmd1" "cmd2"
|
|
@@ -37,7 +61,7 @@ Tapes are VHS script files that define terminal sessions: commands to type, timi
|
|
|
37
61
|
|
|
38
62
|
This prints the generated tape content without writing any file.
|
|
39
63
|
|
|
40
|
-
|
|
64
|
+
4. **Create the tape**:
|
|
41
65
|
|
|
42
66
|
```bash
|
|
43
67
|
ace-demo create <name> -- "cmd1" "cmd2"
|
|
@@ -48,13 +72,13 @@ Tapes are VHS script files that define terminal sessions: commands to type, timi
|
|
|
48
72
|
ace-demo create <name> --desc "What this demo shows" --tags "feature,setup" -- "cmd1" "cmd2"
|
|
49
73
|
```
|
|
50
74
|
|
|
51
|
-
|
|
75
|
+
5. **Update an existing tape** (overwrite):
|
|
52
76
|
|
|
53
77
|
```bash
|
|
54
78
|
ace-demo create <name> --force -- "cmd1" "cmd2"
|
|
55
79
|
```
|
|
56
80
|
|
|
57
|
-
|
|
81
|
+
6. **Verify** the created tape:
|
|
58
82
|
|
|
59
83
|
```bash
|
|
60
84
|
ace-demo show <name>
|
|
@@ -62,7 +86,7 @@ Tapes are VHS script files that define terminal sessions: commands to type, timi
|
|
|
62
86
|
|
|
63
87
|
This displays metadata and full tape contents.
|
|
64
88
|
|
|
65
|
-
|
|
89
|
+
7. **List all available tapes** to confirm visibility:
|
|
66
90
|
|
|
67
91
|
```bash
|
|
68
92
|
ace-demo list
|
|
@@ -86,4 +110,6 @@ Tapes are VHS script files that define terminal sessions: commands to type, timi
|
|
|
86
110
|
|
|
87
111
|
- Tape file created at `.ace/demo/tapes/<name>.tape`
|
|
88
112
|
- `ace-demo show <name>` displays correct metadata and commands
|
|
89
|
-
- `ace-demo list` shows the new tape
|
|
113
|
+
- `ace-demo list` shows the new tape
|
|
114
|
+
- The tape starts from the intended viewer-recognizable state instead of rebuilding irrelevant setup on camera
|
|
115
|
+
- For state-transition demos, the tape visibly shows `before -> trigger -> visible effect -> after`
|
|
@@ -29,10 +29,27 @@ Record terminal demos using `ace-demo record`. Supports two modes: tape-based re
|
|
|
29
29
|
|
|
30
30
|
## Instructions
|
|
31
31
|
|
|
32
|
+
### Validate the Recording Contract First
|
|
33
|
+
|
|
34
|
+
Before choosing tape mode vs inline mode, lock these points:
|
|
35
|
+
|
|
36
|
+
- **What is the user-visible behavior?**
|
|
37
|
+
- **What screen or tmux client must the reviewer be looking at to see it?**
|
|
38
|
+
- **What setup can happen off-camera so the recording starts at the meaningful baseline?**
|
|
39
|
+
|
|
40
|
+
If the feature is about visible tmux behavior such as opening a pane, switching windows, or showing a running fork in the current session:
|
|
41
|
+
|
|
42
|
+
- record the tmux client view that will actually show that transition
|
|
43
|
+
- do not treat a shell transcript as sufficient proof
|
|
44
|
+
- start from the operator's normal work window unless the feature is specifically about attach/reattach behavior
|
|
45
|
+
- prefer showing the transition directly over showing commands that reconstruct the state later
|
|
46
|
+
|
|
32
47
|
### Record from Existing Tape
|
|
33
48
|
|
|
34
49
|
The tape argument accepts a **preset name** (from `ace-demo list`) or a **direct file path** to a `.tape` or `.tape.yml` file.
|
|
35
50
|
|
|
51
|
+
For demos that serve as durable evidence, prefer a committed `.tape.yml` so the generated `.cast` can be verified again with `ace-demo verify`.
|
|
52
|
+
|
|
36
53
|
1. **Find available tapes**:
|
|
37
54
|
|
|
38
55
|
```bash
|
|
@@ -62,6 +79,15 @@ The tape argument accepts a **preset name** (from `ace-demo list`) or a **direct
|
|
|
62
79
|
ace-demo record path/to/tape.tape.yml --output path/to/output.gif
|
|
63
80
|
```
|
|
64
81
|
|
|
82
|
+
5. **Preflight the camera contract for state-transition demos**:
|
|
83
|
+
|
|
84
|
+
Ask these questions before recording:
|
|
85
|
+
|
|
86
|
+
- Does the first part of the recording show the baseline state the reviewer needs?
|
|
87
|
+
- Will the trigger action happen on camera?
|
|
88
|
+
- Will the visible system reaction happen on camera?
|
|
89
|
+
- If the feature is tmux-related, will the recording clearly show the tmux window/pane/session change rather than only a later shell prompt?
|
|
90
|
+
|
|
65
91
|
### Record Inline (Ad-Hoc Commands)
|
|
66
92
|
|
|
67
93
|
1. **Preview** generated tape content:
|
|
@@ -139,8 +165,44 @@ TEST_PATH=ace-bundle ace-demo record test
|
|
|
139
165
|
| `--font-size <n>` | Font size — inline mode (default: 16) |
|
|
140
166
|
| `--playback-speed <speed>` | Postprocess speed: `1x`, `2x`, `4x`, `8x` |
|
|
141
167
|
|
|
168
|
+
## Verification and Recovery
|
|
169
|
+
|
|
170
|
+
For YAML/asciinema demos, recording is not complete when the GIF exists. The cast must pass semantic verification.
|
|
171
|
+
|
|
172
|
+
Use tape `verify:` rules to express:
|
|
173
|
+
- required exported variables (`require_vars`)
|
|
174
|
+
- forbidden output/error signatures (`forbid_output`)
|
|
175
|
+
- final-state assertions (`assert_commands`)
|
|
176
|
+
|
|
177
|
+
After recording:
|
|
178
|
+
|
|
179
|
+
1. If verification passes, continue normally.
|
|
180
|
+
2. If verification fails with `scenario_defect`:
|
|
181
|
+
- inspect the generated report in `.ace-local/demo/`
|
|
182
|
+
- run `ace-demo verify <cast> --tape <tape>` again if needed
|
|
183
|
+
- fix the tape instructions, viewpoint, or setup contract
|
|
184
|
+
- retry recording once
|
|
185
|
+
3. If verification fails with `product_bug`:
|
|
186
|
+
- stop
|
|
187
|
+
- keep the generated report in `.ace-local/demo/`
|
|
188
|
+
- run the cast-analysis workflow and treat it as a real code/runtime bug
|
|
189
|
+
4. Never upload or comment on a PR when verification is not a true pass.
|
|
190
|
+
5. For any non-pass result, branch through `wfi://demo/analyze-cast` before re-recording.
|
|
191
|
+
|
|
192
|
+
### Additional Validation for UX / Tmux Demos
|
|
193
|
+
|
|
194
|
+
Even when the recorder succeeds technically, reject the demo and re-record if any of these are true:
|
|
195
|
+
|
|
196
|
+
- the recording starts after the important state change already happened
|
|
197
|
+
- the recording shows only setup and end state, not the visible transition
|
|
198
|
+
- a tmux/window/pane feature is represented only by status files or textual explanation
|
|
199
|
+
- the viewer cannot identify the operator window, trigger action, and resulting fork/window/pane state from the recording alone
|
|
200
|
+
|
|
142
201
|
## Success Criteria
|
|
143
202
|
|
|
144
203
|
- Recording file produced in `.ace-local/demo/` (plus optional retimed artifact)
|
|
145
|
-
-
|
|
146
|
-
- If `--
|
|
204
|
+
- YAML/asciinema recordings pass semantic verification
|
|
205
|
+
- If `--pr` used: demo uploaded to `demo-assets` release and comment posted on PR only after verification passes
|
|
206
|
+
- If verification fails: error report written to `.ace-local/demo/`
|
|
207
|
+
- If `--dry-run`: preview printed, no side effects
|
|
208
|
+
- For UI/state-transition demos, the recording visibly shows `before -> trigger -> visible effect -> after` from the correct viewer perspective
|
|
@@ -6,7 +6,7 @@ module Ace
|
|
|
6
6
|
module Demo
|
|
7
7
|
module Atoms
|
|
8
8
|
module DemoYamlParser
|
|
9
|
-
ALLOWED_ROOT_KEYS = %w[description tags settings setup scenes teardown].freeze
|
|
9
|
+
ALLOWED_ROOT_KEYS = %w[description tags settings setup scenes verify teardown].freeze
|
|
10
10
|
|
|
11
11
|
module_function
|
|
12
12
|
|
|
@@ -35,6 +35,7 @@ module Ace
|
|
|
35
35
|
"settings" => normalize_settings(data["settings"], source_path: source_path),
|
|
36
36
|
"setup" => normalize_directives(data["setup"], "setup", source_path: source_path),
|
|
37
37
|
"scenes" => normalize_scenes(data["scenes"], source_path: source_path),
|
|
38
|
+
"verify" => normalize_verify(data["verify"], source_path: source_path),
|
|
38
39
|
"teardown" => normalize_directives(data["teardown"], "teardown", source_path: source_path)
|
|
39
40
|
}
|
|
40
41
|
|
|
@@ -164,6 +165,40 @@ module Ace
|
|
|
164
165
|
end
|
|
165
166
|
private_class_method :normalize_scenes
|
|
166
167
|
|
|
168
|
+
def normalize_verify(verify, source_path:)
|
|
169
|
+
return {} if verify.nil?
|
|
170
|
+
raise DemoYamlParseError, "verify must be a map in #{source_path}" unless verify.is_a?(Hash)
|
|
171
|
+
|
|
172
|
+
normalized = {}
|
|
173
|
+
normalized["require_vars"] = normalize_string_list(verify["require_vars"], "verify.require_vars", source_path) if verify.key?("require_vars")
|
|
174
|
+
normalized["require_output"] = normalize_string_list(verify["require_output"], "verify.require_output", source_path) if verify.key?("require_output")
|
|
175
|
+
if verify.key?("require_output_sequence")
|
|
176
|
+
normalized["require_output_sequence"] = normalize_string_list(
|
|
177
|
+
verify["require_output_sequence"],
|
|
178
|
+
"verify.require_output_sequence",
|
|
179
|
+
source_path
|
|
180
|
+
)
|
|
181
|
+
end
|
|
182
|
+
normalized["forbid_output"] = normalize_string_list(verify["forbid_output"], "verify.forbid_output", source_path) if verify.key?("forbid_output")
|
|
183
|
+
normalized["assert_commands"] = normalize_string_list(verify["assert_commands"], "verify.assert_commands", source_path) if verify.key?("assert_commands")
|
|
184
|
+
normalized
|
|
185
|
+
end
|
|
186
|
+
private_class_method :normalize_verify
|
|
187
|
+
|
|
188
|
+
def normalize_string_list(value, field, source_path)
|
|
189
|
+
raise DemoYamlParseError, "#{field} must be an array in #{source_path}" unless value.is_a?(Array)
|
|
190
|
+
|
|
191
|
+
value.map.with_index do |item, index|
|
|
192
|
+
text = item&.to_s
|
|
193
|
+
if text.nil? || text.strip.empty?
|
|
194
|
+
raise DemoYamlParseError, "#{field}[#{index}] must be a non-empty string in #{source_path}"
|
|
195
|
+
end
|
|
196
|
+
|
|
197
|
+
text
|
|
198
|
+
end
|
|
199
|
+
end
|
|
200
|
+
private_class_method :normalize_string_list
|
|
201
|
+
|
|
167
202
|
def normalize_command(command, scene_index, command_index, source_path:)
|
|
168
203
|
unless command.is_a?(Hash)
|
|
169
204
|
raise DemoYamlParseError,
|
|
@@ -156,7 +156,9 @@ module Ace
|
|
|
156
156
|
recording = normalize_recording_result(recorder.record(**record_kwargs))
|
|
157
157
|
puts "Recorded backend: #{recording.backend}"
|
|
158
158
|
puts "Cast: #{recording.cast_path}" if recording.cast_path
|
|
159
|
-
|
|
159
|
+
manifest_path = Molecules::RecordingManifestWriter.new.write(recording: recording)
|
|
160
|
+
ensure_successful_verification!(recording, manifest_path: manifest_path)
|
|
161
|
+
puts "Manifest: #{manifest_path}"
|
|
160
162
|
puts "Recorded: #{recording.visual_path}"
|
|
161
163
|
|
|
162
164
|
attach_path = recording.visual_path
|
|
@@ -310,8 +312,38 @@ module Ace
|
|
|
310
312
|
puts "Verification: #{verification.status}"
|
|
311
313
|
return if verification.success?
|
|
312
314
|
|
|
315
|
+
puts "Classification: #{verification.classification}" if verification.classification
|
|
316
|
+
puts "Summary: #{verification.summary}" if verification.summary
|
|
313
317
|
missing = verification.commands_missing
|
|
314
|
-
puts "
|
|
318
|
+
puts "Missing commands: #{missing.join(', ')}" unless missing.empty?
|
|
319
|
+
missing_vars = verification.details&.fetch(:missing_vars, []) || []
|
|
320
|
+
puts "Missing vars: #{missing_vars.join(', ')}" unless missing_vars.empty?
|
|
321
|
+
missing_output = verification.details&.fetch(:missing_output, []) || []
|
|
322
|
+
puts "Missing output: #{missing_output.join(', ')}" unless missing_output.empty?
|
|
323
|
+
missing_sequence = verification.details&.fetch(:missing_output_sequence, []) || []
|
|
324
|
+
puts "Missing output sequence: #{missing_sequence.join(' -> ')}" unless missing_sequence.empty?
|
|
325
|
+
puts "Assertions replay: skipped" if verification.details&.fetch(:assertions_skipped, false)
|
|
326
|
+
end
|
|
327
|
+
|
|
328
|
+
def ensure_successful_verification!(recording, manifest_path: nil)
|
|
329
|
+
verification = recording.verification
|
|
330
|
+
return unless verification
|
|
331
|
+
|
|
332
|
+
print_verification(verification)
|
|
333
|
+
return if verification.success?
|
|
334
|
+
|
|
335
|
+
report_path = Molecules::VerificationReportWriter.new.write(
|
|
336
|
+
demo_name: verification_demo_name(recording),
|
|
337
|
+
verification: verification
|
|
338
|
+
)
|
|
339
|
+
Molecules::RecordingManifestWriter.new.write(recording: recording) if manifest_path
|
|
340
|
+
puts "Verification report: #{report_path}"
|
|
341
|
+
raise Ace::Support::Cli::Error, "Demo verification failed (#{verification.classification}). Report: #{report_path}"
|
|
342
|
+
end
|
|
343
|
+
|
|
344
|
+
def verification_demo_name(recording)
|
|
345
|
+
source = recording.cast_path || recording.visual_path || "demo"
|
|
346
|
+
File.basename(source, File.extname(source))
|
|
315
347
|
end
|
|
316
348
|
end
|
|
317
349
|
end
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "ace/support/cli"
|
|
4
|
+
require "ace/core"
|
|
5
|
+
|
|
6
|
+
module Ace
|
|
7
|
+
module Demo
|
|
8
|
+
module CLI
|
|
9
|
+
module Commands
|
|
10
|
+
class Verify < Ace::Support::Cli::Command
|
|
11
|
+
include Ace::Support::Cli::Base
|
|
12
|
+
|
|
13
|
+
desc "Verify an existing asciinema cast against a YAML demo tape"
|
|
14
|
+
|
|
15
|
+
argument :cast, required: true, desc: "Cast file path"
|
|
16
|
+
|
|
17
|
+
option :tape, type: :string, required: true, desc: "Tape preset name or .tape.yml path"
|
|
18
|
+
option :sandbox_path, type: :string, desc: "Optional sandbox path for assertion replay"
|
|
19
|
+
option :report_dir, type: :string, desc: "Directory for non-pass verification reports"
|
|
20
|
+
|
|
21
|
+
def call(cast:, tape:, **options)
|
|
22
|
+
resolved_tape = Molecules::TapeResolver.new.resolve(tape)
|
|
23
|
+
unless resolved_tape.end_with?(".tape.yml", ".tape.yaml")
|
|
24
|
+
raise Ace::Support::Cli::Error, "ace-demo verify requires a .tape.yml tape"
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
spec = Atoms::DemoYamlParser.parse_file(resolved_tape)
|
|
28
|
+
verification = Molecules::CastVerifier.new.verify(
|
|
29
|
+
cast_path: File.expand_path(cast),
|
|
30
|
+
tape_spec: spec,
|
|
31
|
+
sandbox_path: options[:sandbox_path],
|
|
32
|
+
env: {}
|
|
33
|
+
)
|
|
34
|
+
|
|
35
|
+
print_verification(verification)
|
|
36
|
+
return if verification.success?
|
|
37
|
+
|
|
38
|
+
report_path = Molecules::VerificationReportWriter.new(
|
|
39
|
+
base_dir: options[:report_dir] || File.join(Dir.pwd, ".ace-local/demo")
|
|
40
|
+
).write(
|
|
41
|
+
demo_name: File.basename(cast, File.extname(cast)),
|
|
42
|
+
verification: verification
|
|
43
|
+
)
|
|
44
|
+
puts "Verification report: #{report_path}"
|
|
45
|
+
raise Ace::Support::Cli::Error, "Demo verification failed (#{verification.classification}). Report: #{report_path}"
|
|
46
|
+
rescue TapeNotFoundError, DemoYamlParseError, CastParseError, ArgumentError => e
|
|
47
|
+
raise Ace::Support::Cli::Error, e.message
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
private
|
|
51
|
+
|
|
52
|
+
def print_verification(verification)
|
|
53
|
+
puts "Verification: #{verification.status}"
|
|
54
|
+
puts "Classification: #{verification.classification}" if verification.classification
|
|
55
|
+
puts "Summary: #{verification.summary}" if verification.summary
|
|
56
|
+
missing = verification.commands_missing
|
|
57
|
+
puts "Missing commands: #{missing.join(', ')}" unless missing.empty?
|
|
58
|
+
missing_vars = verification.details&.fetch(:missing_vars, [])
|
|
59
|
+
puts "Missing vars: #{missing_vars.join(', ')}" unless missing_vars.empty?
|
|
60
|
+
missing_output = verification.details&.fetch(:missing_output, [])
|
|
61
|
+
puts "Missing output: #{missing_output.join(', ')}" unless missing_output.empty?
|
|
62
|
+
missing_sequence = verification.details&.fetch(:missing_output_sequence, [])
|
|
63
|
+
puts "Missing output sequence: #{missing_sequence.join(' -> ')}" unless missing_sequence.empty?
|
|
64
|
+
puts "Assertions replay: skipped" if verification.details&.fetch(:assertions_skipped, false)
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
data/lib/ace/demo/cli.rb
CHANGED
|
@@ -8,6 +8,7 @@ require_relative "cli/commands/list"
|
|
|
8
8
|
require_relative "cli/commands/show"
|
|
9
9
|
require_relative "cli/commands/create"
|
|
10
10
|
require_relative "cli/commands/retime"
|
|
11
|
+
require_relative "cli/commands/verify"
|
|
11
12
|
require_relative "version"
|
|
12
13
|
|
|
13
14
|
module Ace
|
|
@@ -21,6 +22,7 @@ module Ace
|
|
|
21
22
|
["list", "List available demo tapes"],
|
|
22
23
|
["show", "Show metadata and contents for a demo tape"],
|
|
23
24
|
["record", "Record a VHS tape to gif/mp4/webm"],
|
|
25
|
+
["verify", "Verify an existing asciinema cast against a YAML tape"],
|
|
24
26
|
["retime", "Post-process recording speed for gif/mp4/webm"],
|
|
25
27
|
["attach", "Attach an existing demo GIF to a PR"],
|
|
26
28
|
["create", "Create a new demo tape from shell commands"]
|
|
@@ -34,6 +36,7 @@ module Ace
|
|
|
34
36
|
"ace-demo record hello --output /tmp/demo.gif",
|
|
35
37
|
"ace-demo record my-demo -- \"git status\" \"make deploy\"",
|
|
36
38
|
"ace-demo record my-demo --timeout 3s --width 1200 -- \"git status\"",
|
|
39
|
+
"ace-demo verify .ace-local/demo/hello.cast --tape ./hello.tape.yml",
|
|
37
40
|
"ace-demo attach .ace-local/demo/hello.gif --pr 123",
|
|
38
41
|
"ace-demo record hello --pr 123 --dry-run",
|
|
39
42
|
"ace-demo retime .ace-local/demo/hello.gif --playback-speed 4x",
|
|
@@ -45,6 +48,7 @@ module Ace
|
|
|
45
48
|
register "list", Commands::List
|
|
46
49
|
register "show", Commands::Show
|
|
47
50
|
register "record", Commands::Record
|
|
51
|
+
register "verify", Commands::Verify
|
|
48
52
|
register "retime", Commands::Retime
|
|
49
53
|
register "attach", Commands::Attach
|
|
50
54
|
register "create", Commands::Create
|
|
@@ -4,13 +4,15 @@ module Ace
|
|
|
4
4
|
module Demo
|
|
5
5
|
module Models
|
|
6
6
|
class RecordingResult
|
|
7
|
-
attr_reader :backend, :visual_path, :cast_path, :verification
|
|
7
|
+
attr_reader :backend, :visual_path, :cast_path, :verification, :tape_path, :sandbox_path
|
|
8
8
|
|
|
9
|
-
def initialize(backend:, visual_path:, cast_path: nil, verification: nil)
|
|
9
|
+
def initialize(backend:, visual_path:, cast_path: nil, verification: nil, tape_path: nil, sandbox_path: nil)
|
|
10
10
|
@backend = backend
|
|
11
11
|
@visual_path = visual_path
|
|
12
12
|
@cast_path = cast_path
|
|
13
13
|
@verification = verification
|
|
14
|
+
@tape_path = tape_path
|
|
15
|
+
@sandbox_path = sandbox_path
|
|
14
16
|
end
|
|
15
17
|
end
|
|
16
18
|
end
|
|
@@ -4,19 +4,29 @@ module Ace
|
|
|
4
4
|
module Demo
|
|
5
5
|
module Models
|
|
6
6
|
class VerificationResult
|
|
7
|
-
attr_reader :status, :commands_found, :commands_missing, :details
|
|
7
|
+
attr_reader :status, :commands_found, :commands_missing, :details, :classification, :summary, :retryable
|
|
8
|
+
attr_accessor :report_path
|
|
8
9
|
|
|
9
|
-
def initialize(success:, status:, commands_found:, commands_missing:, details: nil
|
|
10
|
+
def initialize(success:, status:, commands_found:, commands_missing:, details: nil,
|
|
11
|
+
classification: nil, summary: nil, retryable: false, report_path: nil)
|
|
10
12
|
@success = success
|
|
11
13
|
@status = status
|
|
12
14
|
@commands_found = commands_found
|
|
13
15
|
@commands_missing = commands_missing
|
|
14
16
|
@details = details
|
|
17
|
+
@classification = classification
|
|
18
|
+
@summary = summary
|
|
19
|
+
@retryable = retryable
|
|
20
|
+
@report_path = report_path
|
|
15
21
|
end
|
|
16
22
|
|
|
17
23
|
def success?
|
|
18
24
|
@success
|
|
19
25
|
end
|
|
26
|
+
|
|
27
|
+
def retryable?
|
|
28
|
+
@retryable
|
|
29
|
+
end
|
|
20
30
|
end
|
|
21
31
|
end
|
|
22
32
|
end
|
|
@@ -1,16 +1,18 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "open3"
|
|
3
4
|
require "shellwords"
|
|
4
5
|
|
|
5
6
|
module Ace
|
|
6
7
|
module Demo
|
|
7
8
|
module Molecules
|
|
8
9
|
class CastVerifier
|
|
9
|
-
def initialize(parser: Atoms::CastFileParser)
|
|
10
|
+
def initialize(parser: Atoms::CastFileParser, command_runner: nil)
|
|
10
11
|
@parser = parser
|
|
12
|
+
@command_runner = command_runner || method(:default_command_runner)
|
|
11
13
|
end
|
|
12
14
|
|
|
13
|
-
def verify(cast_path:, tape_spec:)
|
|
15
|
+
def verify(cast_path:, tape_spec:, sandbox_path: nil, env: {})
|
|
14
16
|
recording = @parser.parse(cast_path)
|
|
15
17
|
expected = expected_commands(tape_spec)
|
|
16
18
|
recorded_inputs = recording.events
|
|
@@ -22,32 +24,78 @@ module Ace
|
|
|
22
24
|
.flat_map { |event| echoed_output_lines(event.data) }
|
|
23
25
|
script_commands = script_commands(recording: recording)
|
|
24
26
|
recorded_commands = (recorded_inputs + echoed_commands + script_commands).uniq
|
|
27
|
+
verification_spec = tape_spec.fetch("verify", {})
|
|
28
|
+
captured_vars = extract_output_vars(echoed_commands)
|
|
25
29
|
|
|
26
30
|
commands_found = expected.select { |command| include_command?(recorded_commands, command) }
|
|
27
31
|
commands_missing = expected - commands_found
|
|
32
|
+
required_vars = verification_spec.fetch("require_vars", [])
|
|
33
|
+
missing_vars = required_vars.reject { |name| present?(captured_vars[name]) || present?(env[name]) }
|
|
34
|
+
required_output = verification_spec.fetch("require_output", [])
|
|
35
|
+
missing_output = missing_output_patterns(echoed_commands, required_output)
|
|
36
|
+
required_output_sequence = verification_spec.fetch("require_output_sequence", [])
|
|
37
|
+
missing_output_sequence = missing_output_sequence_patterns(echoed_commands, required_output_sequence)
|
|
38
|
+
forbidden_hits = forbidden_output_hits(echoed_commands, verification_spec.fetch("forbid_output", []))
|
|
39
|
+
assertion_failures, assertions_skipped = run_assertions(
|
|
40
|
+
verification_spec.fetch("assert_commands", []),
|
|
41
|
+
sandbox_path: sandbox_path,
|
|
42
|
+
env: env.merge(captured_vars)
|
|
43
|
+
)
|
|
44
|
+
|
|
45
|
+
success = commands_missing.empty? && missing_vars.empty? && missing_output.empty? &&
|
|
46
|
+
missing_output_sequence.empty? && forbidden_hits.empty? && assertion_failures.empty?
|
|
47
|
+
classification, status, summary, retryable = classify(
|
|
48
|
+
commands_missing: commands_missing,
|
|
49
|
+
missing_vars: missing_vars,
|
|
50
|
+
missing_output: missing_output,
|
|
51
|
+
missing_output_sequence: missing_output_sequence,
|
|
52
|
+
forbidden_hits: forbidden_hits,
|
|
53
|
+
assertion_failures: assertion_failures
|
|
54
|
+
)
|
|
28
55
|
|
|
29
|
-
status = commands_missing.empty? ? "pass" : "warn"
|
|
30
56
|
Models::VerificationResult.new(
|
|
31
|
-
success:
|
|
57
|
+
success: success,
|
|
32
58
|
status: status,
|
|
33
59
|
commands_found: commands_found,
|
|
34
60
|
commands_missing: commands_missing,
|
|
61
|
+
classification: classification,
|
|
62
|
+
summary: summary,
|
|
63
|
+
retryable: retryable,
|
|
35
64
|
details: {
|
|
36
65
|
cast_path: cast_path,
|
|
37
66
|
inputs_recorded: recorded_inputs.length,
|
|
38
67
|
echoed_commands_recorded: echoed_commands.length,
|
|
39
68
|
script_commands_recorded: script_commands.length,
|
|
40
|
-
commands_expected: expected.length
|
|
69
|
+
commands_expected: expected.length,
|
|
70
|
+
captured_vars: captured_vars,
|
|
71
|
+
missing_vars: missing_vars,
|
|
72
|
+
missing_output: missing_output,
|
|
73
|
+
missing_output_sequence: missing_output_sequence,
|
|
74
|
+
forbidden_hits: forbidden_hits,
|
|
75
|
+
assertion_failures: assertion_failures,
|
|
76
|
+
assertions_skipped: assertions_skipped
|
|
41
77
|
}
|
|
42
78
|
)
|
|
43
79
|
rescue CastParseError => e
|
|
44
80
|
Models::VerificationResult.new(
|
|
45
81
|
success: false,
|
|
46
|
-
status: "
|
|
82
|
+
status: "verification-error",
|
|
47
83
|
commands_found: [],
|
|
48
84
|
commands_missing: expected_commands(tape_spec),
|
|
85
|
+
classification: "verification_error",
|
|
86
|
+
summary: "Failed to parse the recorded cast",
|
|
49
87
|
details: {error: e.message, cast_path: cast_path}
|
|
50
88
|
)
|
|
89
|
+
rescue StandardError => e
|
|
90
|
+
Models::VerificationResult.new(
|
|
91
|
+
success: false,
|
|
92
|
+
status: "verification-error",
|
|
93
|
+
commands_found: [],
|
|
94
|
+
commands_missing: expected_commands(tape_spec),
|
|
95
|
+
classification: "verification_error",
|
|
96
|
+
summary: "Demo verification failed unexpectedly",
|
|
97
|
+
details: {error: "#{e.class}: #{e.message}", cast_path: cast_path}
|
|
98
|
+
)
|
|
51
99
|
end
|
|
52
100
|
|
|
53
101
|
private
|
|
@@ -69,10 +117,100 @@ module Ace
|
|
|
69
117
|
end
|
|
70
118
|
end
|
|
71
119
|
|
|
120
|
+
def classify(commands_missing:, missing_vars:, missing_output:, missing_output_sequence:, forbidden_hits:, assertion_failures:)
|
|
121
|
+
return ["pass", "pass", "Verification passed", false] if commands_missing.empty? && missing_vars.empty? &&
|
|
122
|
+
missing_output.empty? && missing_output_sequence.empty? && forbidden_hits.empty? && assertion_failures.empty?
|
|
123
|
+
unless commands_missing.empty? && missing_vars.empty? && missing_output.empty? && missing_output_sequence.empty?
|
|
124
|
+
return ["scenario_defect", "scenario-defect", "Recording scenario failed verification", true]
|
|
125
|
+
end
|
|
126
|
+
|
|
127
|
+
["product_bug", "product-bug", "Recorded product behavior failed verification", false]
|
|
128
|
+
end
|
|
129
|
+
|
|
72
130
|
def normalized(command)
|
|
73
131
|
command.to_s.strip.gsub(/\s+/, " ")
|
|
74
132
|
end
|
|
75
133
|
|
|
134
|
+
def missing_output_patterns(lines, patterns)
|
|
135
|
+
patterns.reject do |pattern|
|
|
136
|
+
regexp = pattern_to_regexp(pattern)
|
|
137
|
+
lines.any? { |line| line.match?(regexp) }
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
def missing_output_sequence_patterns(lines, patterns)
|
|
142
|
+
return [] if patterns.empty?
|
|
143
|
+
|
|
144
|
+
sequence_index = 0
|
|
145
|
+
patterns.each do |pattern|
|
|
146
|
+
regexp = pattern_to_regexp(pattern)
|
|
147
|
+
found_index = lines.each_index.find do |index|
|
|
148
|
+
index >= sequence_index && lines[index].match?(regexp)
|
|
149
|
+
end
|
|
150
|
+
return patterns[sequence_index..] unless found_index
|
|
151
|
+
|
|
152
|
+
sequence_index = found_index + 1
|
|
153
|
+
end
|
|
154
|
+
|
|
155
|
+
[]
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
def extract_output_vars(lines)
|
|
159
|
+
lines.each_with_object({}) do |line, vars|
|
|
160
|
+
match = line.match(/\A([A-Z][A-Z0-9_]*)=(.+)\z/)
|
|
161
|
+
next unless match
|
|
162
|
+
|
|
163
|
+
vars[match[1]] = match[2].strip
|
|
164
|
+
end
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
def present?(value)
|
|
168
|
+
!value.to_s.strip.empty?
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
def forbidden_output_hits(lines, patterns)
|
|
172
|
+
patterns.flat_map do |pattern|
|
|
173
|
+
regexp = pattern_to_regexp(pattern)
|
|
174
|
+
lines.filter_map do |line|
|
|
175
|
+
next unless line.match?(regexp)
|
|
176
|
+
|
|
177
|
+
{pattern: pattern, line: line}
|
|
178
|
+
end
|
|
179
|
+
end
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
def pattern_to_regexp(pattern)
|
|
183
|
+
text = pattern.to_s
|
|
184
|
+
if text.start_with?("/") && text.end_with?("/") && text.length > 2
|
|
185
|
+
Regexp.new(text[1..-2])
|
|
186
|
+
else
|
|
187
|
+
/#{Regexp.escape(text)}/
|
|
188
|
+
end
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
def run_assertions(commands, sandbox_path:, env:)
|
|
192
|
+
return [[], false] if commands.empty?
|
|
193
|
+
return [[], true] if sandbox_path.to_s.strip.empty?
|
|
194
|
+
|
|
195
|
+
failures = commands.filter_map do |command|
|
|
196
|
+
stdout, stderr, status = @command_runner.call(command, sandbox_path, env)
|
|
197
|
+
next if status.to_i.zero?
|
|
198
|
+
|
|
199
|
+
{
|
|
200
|
+
command: command,
|
|
201
|
+
exit_code: status.to_i,
|
|
202
|
+
stdout: stdout.to_s.strip,
|
|
203
|
+
stderr: stderr.to_s.strip
|
|
204
|
+
}
|
|
205
|
+
end
|
|
206
|
+
[failures, false]
|
|
207
|
+
end
|
|
208
|
+
|
|
209
|
+
def default_command_runner(command, sandbox_path, env)
|
|
210
|
+
stdout, stderr, status = Open3.capture3(env, "bash", "-lc", command, chdir: sandbox_path)
|
|
211
|
+
[stdout, stderr, status.exitstatus]
|
|
212
|
+
end
|
|
213
|
+
|
|
76
214
|
def script_commands(recording:)
|
|
77
215
|
script_path = script_path_from_header(recording.header)
|
|
78
216
|
return [] unless script_path && File.file?(script_path)
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "fileutils"
|
|
4
|
+
require "json"
|
|
5
|
+
|
|
6
|
+
module Ace
|
|
7
|
+
module Demo
|
|
8
|
+
module Molecules
|
|
9
|
+
class RecordingManifestWriter
|
|
10
|
+
def write(recording:)
|
|
11
|
+
target_path = manifest_path_for(recording)
|
|
12
|
+
FileUtils.mkdir_p(File.dirname(target_path))
|
|
13
|
+
File.write(target_path, JSON.pretty_generate(payload(recording)))
|
|
14
|
+
target_path
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
private
|
|
18
|
+
|
|
19
|
+
def manifest_path_for(recording)
|
|
20
|
+
source_path = recording.cast_path || recording.visual_path || "demo"
|
|
21
|
+
ext = File.extname(source_path)
|
|
22
|
+
base = source_path.sub(/#{Regexp.escape(ext)}\z/, "")
|
|
23
|
+
"#{base}.recording.json"
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def payload(recording)
|
|
27
|
+
verification = recording.verification
|
|
28
|
+
{
|
|
29
|
+
backend: recording.backend,
|
|
30
|
+
tape_path: recording.tape_path,
|
|
31
|
+
cast_path: recording.cast_path,
|
|
32
|
+
visual_path: recording.visual_path,
|
|
33
|
+
sandbox_path: recording.sandbox_path,
|
|
34
|
+
verification: verification && {
|
|
35
|
+
status: verification.status,
|
|
36
|
+
classification: verification.classification,
|
|
37
|
+
summary: verification.summary,
|
|
38
|
+
retryable: verification.retryable?,
|
|
39
|
+
report_path: verification.report_path,
|
|
40
|
+
details: verification.details
|
|
41
|
+
}
|
|
42
|
+
}.compact
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "fileutils"
|
|
4
|
+
require "json"
|
|
5
|
+
|
|
6
|
+
module Ace
|
|
7
|
+
module Demo
|
|
8
|
+
module Molecules
|
|
9
|
+
class VerificationReportWriter
|
|
10
|
+
def initialize(base_dir: ".ace-local/demo")
|
|
11
|
+
@base_dir = base_dir
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
def write(demo_name:, verification:)
|
|
15
|
+
FileUtils.mkdir_p(@base_dir)
|
|
16
|
+
basename = demo_name.to_s.strip.empty? ? "demo" : demo_name.to_s.strip.gsub(/[^A-Za-z0-9._-]+/, "-")
|
|
17
|
+
markdown_path = File.expand_path(File.join(@base_dir, "#{basename}-error-report.md"), Dir.pwd)
|
|
18
|
+
json_path = markdown_path.sub(/\.md\z/, ".json")
|
|
19
|
+
|
|
20
|
+
File.write(markdown_path, markdown_content(verification))
|
|
21
|
+
File.write(json_path, JSON.pretty_generate(json_payload(verification)))
|
|
22
|
+
verification.report_path = markdown_path
|
|
23
|
+
markdown_path
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
private
|
|
27
|
+
|
|
28
|
+
def markdown_content(verification)
|
|
29
|
+
details = verification.details || {}
|
|
30
|
+
lines = []
|
|
31
|
+
lines << "# Demo Verification Failure"
|
|
32
|
+
lines << ""
|
|
33
|
+
lines << "- Status: `#{verification.status}`"
|
|
34
|
+
lines << "- Classification: `#{verification.classification}`"
|
|
35
|
+
lines << "- Retryable: `#{verification.retryable?}`"
|
|
36
|
+
lines << "- Summary: #{verification.summary}"
|
|
37
|
+
lines << "- Cast: `#{details[:cast_path]}`" if details[:cast_path]
|
|
38
|
+
lines << ""
|
|
39
|
+
|
|
40
|
+
append_list(lines, "Missing Commands", verification.commands_missing)
|
|
41
|
+
append_list(lines, "Missing Vars", details[:missing_vars])
|
|
42
|
+
append_list(lines, "Missing Output", details[:missing_output])
|
|
43
|
+
append_list(lines, "Missing Output Sequence", details[:missing_output_sequence])
|
|
44
|
+
append_hits(lines, "Forbidden Output Hits", details[:forbidden_hits])
|
|
45
|
+
append_assertion_skip(lines, details[:assertions_skipped])
|
|
46
|
+
append_assertions(lines, details[:assertion_failures])
|
|
47
|
+
|
|
48
|
+
if details[:error]
|
|
49
|
+
lines << "## Error"
|
|
50
|
+
lines << ""
|
|
51
|
+
lines << "```text"
|
|
52
|
+
lines << details[:error].to_s
|
|
53
|
+
lines << "```"
|
|
54
|
+
lines << ""
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
lines.join("\n")
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
def json_payload(verification)
|
|
61
|
+
{
|
|
62
|
+
status: verification.status,
|
|
63
|
+
classification: verification.classification,
|
|
64
|
+
retryable: verification.retryable?,
|
|
65
|
+
summary: verification.summary,
|
|
66
|
+
commands_found: verification.commands_found,
|
|
67
|
+
commands_missing: verification.commands_missing,
|
|
68
|
+
details: verification.details
|
|
69
|
+
}
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
def append_list(lines, title, items)
|
|
73
|
+
return if items.nil? || items.empty?
|
|
74
|
+
|
|
75
|
+
lines << "## #{title}"
|
|
76
|
+
lines << ""
|
|
77
|
+
items.each { |item| lines << "- `#{item}`" }
|
|
78
|
+
lines << ""
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
def append_hits(lines, title, hits)
|
|
82
|
+
return if hits.nil? || hits.empty?
|
|
83
|
+
|
|
84
|
+
lines << "## #{title}"
|
|
85
|
+
lines << ""
|
|
86
|
+
hits.each do |hit|
|
|
87
|
+
lines << "- Pattern: `#{hit[:pattern]}`"
|
|
88
|
+
lines << " Line: `#{hit[:line]}`"
|
|
89
|
+
end
|
|
90
|
+
lines << ""
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def append_assertions(lines, assertions)
|
|
94
|
+
return if assertions.nil? || assertions.empty?
|
|
95
|
+
|
|
96
|
+
lines << "## Assertion Failures"
|
|
97
|
+
lines << ""
|
|
98
|
+
assertions.each do |failure|
|
|
99
|
+
lines << "- Command: `#{failure[:command]}`"
|
|
100
|
+
lines << " Exit: `#{failure[:exit_code]}`"
|
|
101
|
+
lines << " Stdout: `#{failure[:stdout]}`" unless failure[:stdout].to_s.empty?
|
|
102
|
+
lines << " Stderr: `#{failure[:stderr]}`" unless failure[:stderr].to_s.empty?
|
|
103
|
+
end
|
|
104
|
+
lines << ""
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def append_assertion_skip(lines, assertions_skipped)
|
|
108
|
+
return unless assertions_skipped
|
|
109
|
+
|
|
110
|
+
lines << "## Assertion Replay"
|
|
111
|
+
lines << ""
|
|
112
|
+
lines << "- Assertion commands were skipped because no sandbox path was available."
|
|
113
|
+
lines << ""
|
|
114
|
+
end
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
end
|
|
118
|
+
end
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require "open3"
|
|
4
|
+
require "shellwords"
|
|
4
5
|
|
|
5
6
|
module Ace
|
|
6
7
|
module Demo
|
|
@@ -18,7 +19,7 @@ module Ace
|
|
|
18
19
|
def run(cmd, vhs_bin: "vhs", chdir: nil)
|
|
19
20
|
options = {}
|
|
20
21
|
options[:chdir] = chdir if chdir
|
|
21
|
-
stdout, stderr, status = Open3.capture3(*cmd, **options)
|
|
22
|
+
stdout, stderr, status = Open3.capture3(browser_environment, *cmd, **options)
|
|
22
23
|
result = Models::ExecutionResult.new(
|
|
23
24
|
stdout: stdout.strip,
|
|
24
25
|
stderr: stderr.strip,
|
|
@@ -32,6 +33,38 @@ module Ace
|
|
|
32
33
|
rescue Errno::ENOENT
|
|
33
34
|
raise VhsNotFoundError, "VHS not found. Install: #{INSTALL_URL}"
|
|
34
35
|
end
|
|
36
|
+
|
|
37
|
+
private
|
|
38
|
+
|
|
39
|
+
def browser_environment
|
|
40
|
+
browser = resolve_browser_path
|
|
41
|
+
return {} unless browser
|
|
42
|
+
|
|
43
|
+
{
|
|
44
|
+
"BROWSER" => browser,
|
|
45
|
+
"CHROME_BIN" => browser,
|
|
46
|
+
"CHROMIUM_BIN" => browser
|
|
47
|
+
}
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def resolve_browser_path
|
|
51
|
+
%w[chromium google-chrome].each do |candidate|
|
|
52
|
+
path = which(candidate)
|
|
53
|
+
return path if path
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
nil
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
def which(command)
|
|
60
|
+
stdout, _stderr, status = Open3.capture3("bash", "-lc", "command -v #{Shellwords.escape(command)}")
|
|
61
|
+
return nil unless status.success?
|
|
62
|
+
|
|
63
|
+
resolved = stdout.strip
|
|
64
|
+
resolved.empty? ? nil : resolved
|
|
65
|
+
rescue Errno::ENOENT
|
|
66
|
+
nil
|
|
67
|
+
end
|
|
35
68
|
end
|
|
36
69
|
end
|
|
37
70
|
end
|
|
@@ -79,7 +79,8 @@ module Ace
|
|
|
79
79
|
@executor.run(cmd)
|
|
80
80
|
Models::RecordingResult.new(
|
|
81
81
|
backend: "vhs",
|
|
82
|
-
visual_path: output_path
|
|
82
|
+
visual_path: output_path,
|
|
83
|
+
tape_path: tape_path
|
|
83
84
|
)
|
|
84
85
|
end
|
|
85
86
|
|
|
@@ -128,14 +129,22 @@ module Ace
|
|
|
128
129
|
vhs_bin: @vhs_bin
|
|
129
130
|
)
|
|
130
131
|
@executor.run(cmd, chdir: sandbox[:path])
|
|
131
|
-
return Models::RecordingResult.new(
|
|
132
|
+
return Models::RecordingResult.new(
|
|
133
|
+
backend: "vhs",
|
|
134
|
+
visual_path: raw_output_path,
|
|
135
|
+
tape_path: tape_path
|
|
136
|
+
) unless selected_speed
|
|
132
137
|
|
|
133
138
|
retimed = @media_retimer.retime(
|
|
134
139
|
input_path: raw_output_path,
|
|
135
140
|
speed: selected_speed[:label],
|
|
136
141
|
output_path: retime_output_path
|
|
137
142
|
)
|
|
138
|
-
Models::RecordingResult.new(
|
|
143
|
+
Models::RecordingResult.new(
|
|
144
|
+
backend: "vhs",
|
|
145
|
+
visual_path: retimed[:output_path],
|
|
146
|
+
tape_path: tape_path
|
|
147
|
+
)
|
|
139
148
|
ensure
|
|
140
149
|
@teardown_executor.execute(steps: spec["teardown"] || [], sandbox_path: sandbox[:path]) if sandbox
|
|
141
150
|
end
|
|
@@ -154,6 +163,7 @@ module Ace
|
|
|
154
163
|
FileUtils.mkdir_p(File.dirname(retime_output_path)) if retime_output_path
|
|
155
164
|
|
|
156
165
|
sandbox = @sandbox_builder.build(source_tape_path: tape_path, setup_steps: spec["setup"] || [])
|
|
166
|
+
verification = nil
|
|
157
167
|
begin
|
|
158
168
|
inject_sandbox_env(spec, sandbox[:path])
|
|
159
169
|
env = (settings["env"] || {}).transform_values(&:to_s)
|
|
@@ -177,7 +187,12 @@ module Ace
|
|
|
177
187
|
agg_bin: @agg_bin
|
|
178
188
|
)
|
|
179
189
|
@agg_executor.run(convert_cmd, chdir: sandbox[:path])
|
|
180
|
-
verification = @cast_verifier.verify(
|
|
190
|
+
verification = @cast_verifier.verify(
|
|
191
|
+
cast_path: cast_output_path,
|
|
192
|
+
tape_spec: spec,
|
|
193
|
+
sandbox_path: sandbox[:path],
|
|
194
|
+
env: env.merge("PROJECT_ROOT_PATH" => sandbox[:path])
|
|
195
|
+
)
|
|
181
196
|
|
|
182
197
|
visual_path =
|
|
183
198
|
if selected_speed
|
|
@@ -195,10 +210,14 @@ module Ace
|
|
|
195
210
|
backend: "asciinema",
|
|
196
211
|
cast_path: cast_output_path,
|
|
197
212
|
visual_path: visual_path,
|
|
198
|
-
verification: verification
|
|
213
|
+
verification: verification,
|
|
214
|
+
tape_path: tape_path,
|
|
215
|
+
sandbox_path: sandbox[:path]
|
|
199
216
|
)
|
|
200
217
|
ensure
|
|
201
|
-
|
|
218
|
+
if sandbox && (verification.nil? || verification.success?)
|
|
219
|
+
@teardown_executor.execute(steps: spec["teardown"] || [], sandbox_path: sandbox[:path])
|
|
220
|
+
end
|
|
202
221
|
end
|
|
203
222
|
end
|
|
204
223
|
|
data/lib/ace/demo/version.rb
CHANGED
data/lib/ace/demo.rb
CHANGED
|
@@ -38,6 +38,8 @@ require_relative "demo/molecules/gh_asset_uploader"
|
|
|
38
38
|
require_relative "demo/molecules/inline_recorder"
|
|
39
39
|
require_relative "demo/molecules/media_retimer"
|
|
40
40
|
require_relative "demo/molecules/demo_comment_poster"
|
|
41
|
+
require_relative "demo/molecules/verification_report_writer"
|
|
42
|
+
require_relative "demo/molecules/recording_manifest_writer"
|
|
41
43
|
require_relative "demo/organisms/demo_recorder"
|
|
42
44
|
require_relative "demo/organisms/demo_attacher"
|
|
43
45
|
require_relative "demo/organisms/tape_creator"
|
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: ace-demo
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.25.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Michal Czyz
|
|
8
8
|
bindir: exe
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date:
|
|
10
|
+
date: 2026-04-20 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
12
|
- !ruby/object:Gem::Dependency
|
|
13
13
|
name: ace-support-cli
|
|
@@ -140,8 +140,10 @@ files:
|
|
|
140
140
|
- README.md
|
|
141
141
|
- Rakefile
|
|
142
142
|
- exe/ace-demo
|
|
143
|
+
- handbook/skills/as-demo-analyze-cast/SKILL.md
|
|
143
144
|
- handbook/skills/as-demo-create/SKILL.md
|
|
144
145
|
- handbook/skills/as-demo-record/SKILL.md
|
|
146
|
+
- handbook/workflow-instructions/demo/analyze-cast.wf.md
|
|
145
147
|
- handbook/workflow-instructions/demo/create.wf.md
|
|
146
148
|
- handbook/workflow-instructions/demo/record.wf.md
|
|
147
149
|
- lib/ace/demo.rb
|
|
@@ -168,6 +170,7 @@ files:
|
|
|
168
170
|
- lib/ace/demo/cli/commands/record.rb
|
|
169
171
|
- lib/ace/demo/cli/commands/retime.rb
|
|
170
172
|
- lib/ace/demo/cli/commands/show.rb
|
|
173
|
+
- lib/ace/demo/cli/commands/verify.rb
|
|
171
174
|
- lib/ace/demo/models/cast_event.rb
|
|
172
175
|
- lib/ace/demo/models/cast_recording.rb
|
|
173
176
|
- lib/ace/demo/models/execution_result.rb
|
|
@@ -182,9 +185,11 @@ files:
|
|
|
182
185
|
- lib/ace/demo/molecules/gh_asset_uploader.rb
|
|
183
186
|
- lib/ace/demo/molecules/inline_recorder.rb
|
|
184
187
|
- lib/ace/demo/molecules/media_retimer.rb
|
|
188
|
+
- lib/ace/demo/molecules/recording_manifest_writer.rb
|
|
185
189
|
- lib/ace/demo/molecules/tape_resolver.rb
|
|
186
190
|
- lib/ace/demo/molecules/tape_scanner.rb
|
|
187
191
|
- lib/ace/demo/molecules/tape_writer.rb
|
|
192
|
+
- lib/ace/demo/molecules/verification_report_writer.rb
|
|
188
193
|
- lib/ace/demo/molecules/vhs_executor.rb
|
|
189
194
|
- lib/ace/demo/organisms/demo_attacher.rb
|
|
190
195
|
- lib/ace/demo/organisms/demo_recorder.rb
|