@zibby/cli 0.2.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +117 -16
- package/dist/bin/zibby.js +3 -3
- package/dist/commands/chat-agents.js +1 -1
- package/dist/commands/chat.js +62 -62
- package/dist/commands/credentials-api.js +1 -1
- package/dist/commands/credentials-file.js +1 -1
- package/dist/commands/creds.js +1 -1
- package/dist/commands/init.js +180 -82
- package/dist/commands/memory.js +48 -20
- package/dist/commands/run.js +52 -55
- package/dist/commands/template.js +9 -0
- package/dist/commands/uninstall.js +15 -14
- package/dist/commands/workflows/agent-helpers.js +7 -7
- package/dist/commands/workflows/generate.js +54 -32
- package/dist/commands/workflows/run-local.js +19 -19
- package/dist/commands/workflows/run.js +5 -5
- package/dist/commands/workflows/start.js +12 -12
- package/dist/package.json +2 -2
- package/dist/templates/zibby-workflow-claude/agents-md-block.md +65 -5
- package/dist/templates/zibby-workflow-claude/claude/agents/zibby-test-author.md +16 -1
- package/dist/templates/zibby-workflow-claude/claude/agents/zibby-workflow-builder.md +22 -2
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-add-node.md +1 -1
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-debug.md +1 -1
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-delete.md +1 -1
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-deploy.md +24 -14
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-list.md +2 -2
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-memory-cost.md +39 -0
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-memory-pull.md +47 -0
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-memory-remote-use-hosted.md +61 -0
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-memory-stats.md +38 -0
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-static-ip.md +8 -6
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-tail.md +1 -1
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-test-debug.md +2 -2
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-test-generate.md +1 -1
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-test-run.md +3 -2
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-test-write.md +1 -1
- package/dist/templates/zibby-workflow-claude/claude/commands/zibby-trigger.md +10 -6
- package/dist/templates/zibby-workflow-claude/cursor/rules/zibby-workflows.mdc +76 -13
- package/dist/templates/zibby-workflow-claude/manifest.json +5 -1
- package/dist/utils/agent-credentials.js +4 -3
- package/dist/utils/apply-memory-sync-config.js +1 -0
- package/dist/utils/credentials-loader.js +1 -1
- package/dist/utils/hosted-memory-sync.js +1 -0
- package/package.json +2 -2
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- BEGIN zibby-workflows zibby-template-version:
|
|
1
|
+
<!-- BEGIN zibby-workflows zibby-template-version: 4 -->
|
|
2
2
|
## Zibby
|
|
3
3
|
|
|
4
4
|
This project uses **Zibby** — there are two surfaces:
|
|
@@ -16,7 +16,7 @@ Both share `.zibby.config.mjs` at the project root.
|
|
|
16
16
|
Files:
|
|
17
17
|
```
|
|
18
18
|
<paths.workflows or .zibby/workflows>/<name>/
|
|
19
|
-
├── workflow.json name, entryClass, triggers, schemas
|
|
19
|
+
├── workflow.json name, entryClass, triggers, schemas (manifest)
|
|
20
20
|
├── graph.mjs nodes + edges from START to END
|
|
21
21
|
├── nodes/
|
|
22
22
|
│ ├── index.mjs barrel export
|
|
@@ -33,19 +33,42 @@ Each node has `async run(ctx)` where `ctx` provides:
|
|
|
33
33
|
Common dev loop:
|
|
34
34
|
```
|
|
35
35
|
zibby workflow new <name> # scaffold
|
|
36
|
-
zibby workflow run <name> # one-shot local run (
|
|
36
|
+
zibby workflow run <name> # one-shot local run (preferred for the dev loop)
|
|
37
37
|
zibby workflow run <name> -p k=v # with input
|
|
38
38
|
zibby workflow deploy <name> # build + push to Zibby Cloud
|
|
39
39
|
zibby workflow trigger <uuid> # invoke the cloud workflow
|
|
40
40
|
zibby workflow logs <uuid> -t # tail live logs (docker-compose-style)
|
|
41
|
-
zibby workflow list # find UUIDs and statuses
|
|
41
|
+
zibby workflow list # find UUIDs and statuses (local + cloud)
|
|
42
|
+
zibby workflow download <uuid> # pull the cloud workflow source back to .zibby/workflows/
|
|
42
43
|
zibby workflow delete <uuid> # remove a deployed workflow
|
|
43
44
|
```
|
|
44
45
|
|
|
45
|
-
|
|
46
|
+
**`run` vs `start`.** `workflow run` is the one-shot CLI iteration command — load the graph, execute it once, print the result, exit. That's the right primitive for the dev loop and for CI/CD. `workflow start` is a *long-lived* local dev server (default port 3848) used by Studio for replay/debug; for plain CLI iteration always prefer `run`.
|
|
47
|
+
|
|
48
|
+
`run` and `trigger` accept the same input flag surface — flip the verb to switch between local and cloud:
|
|
49
|
+
- `-p key=value` (repeatable) — highest precedence
|
|
50
|
+
- `--input '<json>'` — JSON string
|
|
51
|
+
- `--input-file path.json` — JSON file, lowest precedence
|
|
46
52
|
|
|
47
53
|
Static outbound IPs (for customers behind firewalls): see `--dedicated-ip` flag on `deploy`.
|
|
48
54
|
|
|
55
|
+
#### Per-workflow env vars
|
|
56
|
+
|
|
57
|
+
Each deployed workflow has its own encrypted env-var bag (KMS-backed). Vars get injected into the Fargate task at trigger time, and **workflow env wins over project secrets on conflict**. Use this for per-pipeline credentials (different `ANTHROPIC_API_KEY` per workflow, a workflow-only `DATABASE_URL`, etc.).
|
|
58
|
+
|
|
59
|
+
```
|
|
60
|
+
zibby workflow env list <uuid> # show key names (values never returned)
|
|
61
|
+
zibby workflow env set <uuid> ANTHROPIC_API_KEY=sk-… # add or rotate one key
|
|
62
|
+
zibby workflow env unset <uuid> OLD_KEY # remove one key
|
|
63
|
+
zibby workflow env push <uuid> --file .env [--file .env.prod] # bulk replace from .env files
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Fast path on first deploy — sync a `.env` in one shot:
|
|
67
|
+
```
|
|
68
|
+
zibby workflow deploy my-pipeline --env .env [--env .env.prod]
|
|
69
|
+
```
|
|
70
|
+
The CLI deploys, then runs `push` against the freshly-minted UUID.
|
|
71
|
+
|
|
49
72
|
---
|
|
50
73
|
|
|
51
74
|
### Tests
|
|
@@ -55,6 +78,7 @@ Files:
|
|
|
55
78
|
test-specs/ source `.txt` specs (paths.specs)
|
|
56
79
|
tests/ generated `.spec.js` (paths.generated; regenerated each run)
|
|
57
80
|
test-results/ videos, traces, JSON results per run
|
|
81
|
+
.zibby/memory/.dolt/ local test memory DB (selectors, page model, history)
|
|
58
82
|
playwright.config.js
|
|
59
83
|
```
|
|
60
84
|
|
|
@@ -64,6 +88,8 @@ Common dev loop:
|
|
|
64
88
|
```
|
|
65
89
|
zibby test test-specs/<name>.txt # run a spec
|
|
66
90
|
zibby test "go to example.com and ..." # inline, no file
|
|
91
|
+
zibby test <spec> --agent claude # override the configured agent (claude|cursor|codex|gemini)
|
|
92
|
+
zibby test --sources <ids> --execution <id> # cloud test cases (run from a stored execution)
|
|
67
93
|
zibby generate -t ENG-1234 # generate specs from a Jira ticket
|
|
68
94
|
zibby video # organize videos next to spec files
|
|
69
95
|
zibby upload <spec-path> # upload existing artifacts to cloud
|
|
@@ -71,6 +97,39 @@ zibby upload <spec-path> # upload existing artifacts to cloud
|
|
|
71
97
|
|
|
72
98
|
When debugging a failed test, watch the video at `test-results/<spec>/video.webm` — that's almost always faster than reading logs.
|
|
73
99
|
|
|
100
|
+
#### Test memory
|
|
101
|
+
|
|
102
|
+
`.zibby/memory/.dolt/` is a **local-first Dolt SQL database** (Git-for-data) that learns from every test run — selectors that worked, page-element fingerprints, navigation transitions, timing quirks, recorded insights. The runner auto-pulls before a run and auto-pushes after a passing run. Keying is **per-domain** (not per-spec), so any spec that hits `myapp.com` benefits from selectors learned by every other spec on the same domain.
|
|
103
|
+
|
|
104
|
+
When `zibby test` runs and `.zibby/memory/.dolt/` exists, the agent gets 5 MCP tools auto-exposed:
|
|
105
|
+
|
|
106
|
+
- `memory_get_test_history` — query recent runs (pass/fail/timing); filter by spec-path substring
|
|
107
|
+
- `memory_get_selectors` — query known selectors per page with stability metrics (success/fail counts)
|
|
108
|
+
- `memory_get_page_model` — query page structure (elements, roles, accessible names, best selector)
|
|
109
|
+
- `memory_get_navigation` — known page-to-page transitions (what click/submit produced what URL)
|
|
110
|
+
- `memory_save_insight` — save an observation. Categories: `selector_tip | timing | navigation | workaround | flaky | general`
|
|
111
|
+
|
|
112
|
+
> **AFTER completing the test, you MUST call `memory_save_insight` at least once.** Save any useful finding: reliable selectors, timing quirks, navigation patterns, workarounds. Be specific — future runs will read your insights. (Lifted from the memory skill's `promptFragment`.)
|
|
113
|
+
|
|
114
|
+
Local CLI:
|
|
115
|
+
```
|
|
116
|
+
zibby memory stats # row counts, last commit, per-spec breakdown
|
|
117
|
+
zibby memory cost # real LLM token spend per spec / per domain
|
|
118
|
+
zibby memory compact # prune old runs + Dolt GC (--max-runs 50, --max-age 90d)
|
|
119
|
+
zibby memory reset -f # wipe the DB
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
**Team sync.** Memory is local-first; opt into a shared remote so teammates' learnings flow back to you:
|
|
123
|
+
|
|
124
|
+
```
|
|
125
|
+
zibby memory remote add aws://my-bucket/team/proj/main # BYO S3 / GCS / DoltHub / file:///
|
|
126
|
+
zibby memory remote use --hosted # OR: Zibby-managed S3 (signed-in only)
|
|
127
|
+
zibby memory pull # manual override (auto on test start)
|
|
128
|
+
zibby memory push # manual override (auto on passing test)
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
Set `memorySync.remote` in `.zibby.config.mjs` (`'hosted'` or an `aws://...` URL) and `zibby init` auto-wires the remote — teammates clone the repo, run `zibby init`, and they're plugged into the same memory.
|
|
132
|
+
|
|
74
133
|
---
|
|
75
134
|
|
|
76
135
|
### How to invoke the CLI
|
|
@@ -97,6 +156,7 @@ Don't waste time on `npx @zibby/cli` — not always published.
|
|
|
97
156
|
- Deploying & bundling: https://docs.zibby.app/workflows/deploying
|
|
98
157
|
- Triggering & inputs: https://docs.zibby.app/workflows/triggers
|
|
99
158
|
- Live log streaming: https://docs.zibby.app/workflows/logs
|
|
159
|
+
- Per-workflow env vars: https://docs.zibby.app/cloud/env-vars
|
|
100
160
|
- Egress proxy / static IPs: https://docs.zibby.app/workflows/egress
|
|
101
161
|
- Security & secrets: https://docs.zibby.app/workflows/security
|
|
102
162
|
- Debugging: https://docs.zibby.app/workflows/debugging
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
---
|
|
3
3
|
name: zibby-test-author
|
|
4
4
|
description: Sub-agent that helps the user design and author Zibby test specs end-to-end. Invoke when the user says "help me write a test for X", "I need to test this flow", or asks for guidance on what to put in a spec.
|
|
@@ -53,12 +53,27 @@ A spec is unambiguous English with one action per line. See `/zibby-test-write`
|
|
|
53
53
|
|
|
54
54
|
5. **Stop when the spec exercises the goal end-to-end.** Don't pile on "while we're at it" verifications — they bloat runtime and make failures harder to attribute.
|
|
55
55
|
|
|
56
|
+
## Test memory (`.zibby/memory/.dolt/`)
|
|
57
|
+
|
|
58
|
+
When `zibby test` runs and `.zibby/memory/.dolt/` exists (initialized by `zibby memory init` or auto-created on first run with `-m` / a `memorySync.remote` config), the agent gets 5 MCP tools auto-exposed. They read from a local-first Dolt SQL DB that learns selectors, page model, navigation, and history **per-domain** across every spec hitting the same site:
|
|
59
|
+
|
|
60
|
+
- `memory_get_test_history` — recent runs (filter by spec-path substring) — pass/fail and timing
|
|
61
|
+
- `memory_get_selectors` — known selectors per page with stability metrics (success/fail counts)
|
|
62
|
+
- `memory_get_page_model` — page elements, ARIA roles, accessible names, best-known selector
|
|
63
|
+
- `memory_get_navigation` — known page-to-page transitions (what click/submit produced what URL)
|
|
64
|
+
- `memory_save_insight` — save observations: `selector_tip | timing | navigation | workaround | flaky | general`
|
|
65
|
+
|
|
66
|
+
> **Hard rule: after every test run, the agent MUST call `memory_save_insight` at least once.** Save reliable selectors, timing quirks, navigation patterns, workarounds — be specific. Future runs read these. (This is in the memory skill's prompt fragment; surface it to the user if they ask why their tests keep getting smarter.)
|
|
67
|
+
|
|
68
|
+
Team sync (optional): a project may have `memorySync.remote: 'hosted'` (Zibby-managed S3, signed-in only) or `'aws://...' / 'gs://...'` (BYO) configured in `.zibby.config.mjs`. If set, the runner auto-pulls before each run and auto-pushes after passing runs. Manual override: `zibby memory pull` / `zibby memory push`.
|
|
69
|
+
|
|
56
70
|
## Hard rules
|
|
57
71
|
|
|
58
72
|
- **Never recommend `--headless` for first runs.** Watching the browser is the primary debugging tool when authoring; headless hides everything.
|
|
59
73
|
- **Never recommend disabling video.** Videos are 99% of post-mortem signal; they're cheap.
|
|
60
74
|
- **Don't write CSS selectors into specs.** Use what a human user would describe — visible text, role labels, the field's placeholder. Selectors belong in generated `.spec.js`, not the source.
|
|
61
75
|
- **Don't suggest `npx playwright test` directly** to bypass Zibby for "speed". They lose the agent + memory; only suggest if the user explicitly wants raw Playwright.
|
|
76
|
+
- **Always call `memory_save_insight` at the end of a test run.** This is non-negotiable — without it, memory degrades to the seeded baseline and stops compounding.
|
|
62
77
|
|
|
63
78
|
## Reference
|
|
64
79
|
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
---
|
|
3
3
|
name: zibby-workflow-builder
|
|
4
4
|
description: Sub-agent that walks the user through building, testing, and deploying a Zibby agent workflow end-to-end. Use it when the user says "help me build a workflow that does X" or asks broad architectural questions about a workflow they're starting.
|
|
@@ -49,7 +49,7 @@ The return value of `run()` is the node's output, available to downstream nodes
|
|
|
49
49
|
|
|
50
50
|
4. **Generate the scaffold** if they don't have one yet:
|
|
51
51
|
```
|
|
52
|
-
zibby workflow
|
|
52
|
+
zibby workflow new <slug>
|
|
53
53
|
```
|
|
54
54
|
Then add nodes one at a time using the `/zibby-add-node` command.
|
|
55
55
|
|
|
@@ -64,6 +64,26 @@ The return value of `run()` is the node's output, available to downstream nodes
|
|
|
64
64
|
|
|
65
65
|
6. **Stop when the workflow does the goal end-to-end.** Don't pile on speculative nodes.
|
|
66
66
|
|
|
67
|
+
## Per-workflow env vars
|
|
68
|
+
|
|
69
|
+
Each deployed workflow has its own encrypted env-var bag (KMS-backed). Workflow env wins over project secrets on conflict.
|
|
70
|
+
|
|
71
|
+
- `zibby workflow env list <uuid>` — show key names (values never returned)
|
|
72
|
+
- `zibby workflow env set <uuid> ANTHROPIC_API_KEY=sk-…` — add or rotate one key
|
|
73
|
+
- `zibby workflow env unset <uuid> OLD_KEY` — remove one key
|
|
74
|
+
- `zibby workflow env push <uuid> --file .env [--file .env.prod]` — bulk replace from .env files (later files override)
|
|
75
|
+
- `zibby workflow deploy <slug> --env .env` — fast path: deploy + auto-`push` of .env to the new UUID
|
|
76
|
+
|
|
77
|
+
Use this for credentials specific to one workflow (per-pipeline `ANTHROPIC_API_KEY`, a workflow-only `DATABASE_URL`, an external webhook secret). Project-wide secrets stay on the project record.
|
|
78
|
+
|
|
79
|
+
## Pulling a deployed workflow back to local
|
|
80
|
+
|
|
81
|
+
```
|
|
82
|
+
zibby workflow download <uuid>
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Pulls the cloud workflow's source back into `.zibby/workflows/<name>/`. Useful when collaborators need the source from cloud (e.g. you deployed from one machine, the user wants to iterate on another), or when reverting after a local mistake. UUIDs come from `zibby workflow list`.
|
|
86
|
+
|
|
67
87
|
## Hard rules
|
|
68
88
|
|
|
69
89
|
- **Never recommend `--force` flags or skipping checks** to make a deploy go faster. Build problems are signal.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
# /zibby-deploy — deploy a Zibby workflow to the cloud
|
|
3
3
|
|
|
4
4
|
You are helping the user deploy a workflow they've been building locally.
|
|
@@ -47,31 +47,41 @@ Canonical docs: **https://docs.zibby.app/workflows/deploying**
|
|
|
47
47
|
|
|
48
48
|
## Optional flags worth knowing
|
|
49
49
|
|
|
50
|
+
`zibby workflow deploy` accepts:
|
|
50
51
|
- `--project <id>` — skip the interactive project picker
|
|
51
52
|
- `--api-key <key>` — use a PAT instead of the session token (for CI)
|
|
53
|
+
- `--env <path>` — sync a `.env` file into per-workflow env vars after deploy. Repeatable; later files override.
|
|
52
54
|
- `--verbose` — print raw CodeBuild output during the build (helpful for debugging build failures)
|
|
53
|
-
- `--dedicated-ip <action>` — opt this workflow into the dedicated egress addon (static outbound IP). See `/zibby-static-ip` for setup.
|
|
54
55
|
|
|
55
|
-
|
|
56
|
+
### Seeding per-workflow env on first deploy
|
|
56
57
|
|
|
57
|
-
If the
|
|
58
|
+
If the workflow needs its own `ANTHROPIC_API_KEY`, `DATABASE_URL`, etc., put them in a `.env` and pass `--env`:
|
|
58
59
|
|
|
59
|
-
|
|
60
|
+
```
|
|
61
|
+
zibby workflow deploy <name> --env .env
|
|
62
|
+
zibby workflow deploy <name> --env .env --env .env.prod # later files win
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
After deploy, manage them surgically with `zibby workflow env set/unset/list/push <uuid>`. See `/zibby-list` to recover the UUID; full guide at https://docs.zibby.app/cloud/env-vars.
|
|
66
|
+
|
|
67
|
+
## Static outbound IP (dedicated egress)
|
|
68
|
+
|
|
69
|
+
If the user's workflow needs to call APIs that require IP allowlisting (corporate GitHub, GitLab Enterprise, paranoid SaaS firewalls), the workflow needs the **dedicated egress IP** addon. The flag lives on the legacy alias `zibby deploy` (NOT `zibby workflow deploy`):
|
|
60
70
|
|
|
61
71
|
| Flag | What it does |
|
|
62
72
|
|------|-------------|
|
|
63
|
-
|
|
|
64
|
-
|
|
|
65
|
-
|
|
|
66
|
-
|
|
|
67
|
-
|
|
|
73
|
+
| `zibby deploy <name> --dedicated-ip status` | Show current addon state for the account (active / inactive / billing) |
|
|
74
|
+
| `zibby deploy <name> --dedicated-ip enable` | Enable the addon on the account (Pro subscription required, ~$50/mo). One-time per account. |
|
|
75
|
+
| `zibby deploy <name> --dedicated-ip use` | Mark THIS workflow as using the static egress IP (per-workflow opt-in, after `enable`) |
|
|
76
|
+
| `zibby deploy <name> --dedicated-ip unuse` | Stop routing this workflow through the static IP |
|
|
77
|
+
| `zibby deploy <name> --dedicated-ip disable` | Disable the addon for the whole account |
|
|
68
78
|
|
|
69
79
|
Typical first-time flow when the user says "I need a static outbound IP":
|
|
70
|
-
1. `zibby
|
|
71
|
-
2. If inactive → `zibby
|
|
72
|
-
3. `zibby
|
|
80
|
+
1. `zibby deploy <name> --dedicated-ip status` — check whether they have it
|
|
81
|
+
2. If inactive → `zibby deploy <name> --dedicated-ip enable` — enables the account-wide addon (interactive billing prompt; prerequisite Pro subscription)
|
|
82
|
+
3. `zibby deploy <name> --dedicated-ip use` — opts this specific workflow in
|
|
73
83
|
4. Regular `zibby workflow deploy <name>` from now on uses the static IP
|
|
74
84
|
|
|
75
85
|
After `--dedicated-ip use`, every node in this workflow gets its outbound HTTP routed through the egress proxy, and `process.env.HTTP_PROXY` / `HTTPS_PROXY` are set in the sandbox automatically. Their static IPs are visible to customers via `https://docs.zibby.app/workflows/egress`.
|
|
76
86
|
|
|
77
|
-
**Don't** run `--dedicated-ip enable` without confirming with the user — it has billing impact ($50/mo addon). Always confirm.
|
|
87
|
+
**Don't** run `--dedicated-ip enable` without confirming with the user — it has billing impact ($50/mo addon). Always confirm. See `/zibby-static-ip` for the deeper walkthrough.
|
|
@@ -1,9 +1,9 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
# /zibby-list — list workflows (local + cloud) with their UUIDs and statuses
|
|
3
3
|
|
|
4
4
|
You are helping the user see what workflows exist — locally scaffolded and remotely deployed.
|
|
5
5
|
|
|
6
|
-
Canonical docs: **https://docs.zibby.app/
|
|
6
|
+
Canonical docs: **https://docs.zibby.app/cli-reference#workflow-list**
|
|
7
7
|
|
|
8
8
|
## Steps
|
|
9
9
|
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
|
+
# /zibby-memory-cost — show real LLM token spend across past test runs
|
|
3
|
+
|
|
4
|
+
You are helping the user see how many input/output/cache tokens their tests have actually burned, broken down per spec and per domain. This is real measured spend (read off run records in `.zibby/memory/.dolt/`), not an estimate.
|
|
5
|
+
|
|
6
|
+
Canonical docs: **https://docs.zibby.app/tests/memory**
|
|
7
|
+
|
|
8
|
+
## What the command shows
|
|
9
|
+
|
|
10
|
+
```
|
|
11
|
+
Bash(zibby memory cost)
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Per-spec and per-domain rollup of:
|
|
15
|
+
- Input tokens
|
|
16
|
+
- Output tokens
|
|
17
|
+
- Cache hit / cache write tokens (when the agent supports prompt caching)
|
|
18
|
+
- Estimated $ cost (uses current public model pricing)
|
|
19
|
+
- Recent-runs trend, so you can see if a spec is getting cheaper or more expensive over time
|
|
20
|
+
|
|
21
|
+
The numbers are pulled from `test_runs` rows in the Dolt DB — every test run records the agent's actual usage on completion.
|
|
22
|
+
|
|
23
|
+
## When to invoke
|
|
24
|
+
|
|
25
|
+
- User asks "how much are my tests costing me?" or "which spec is the expensive one?"
|
|
26
|
+
- After enabling prompt caching to confirm cache hits are landing
|
|
27
|
+
- When deciding whether to swap to a cheaper agent on hot specs (`--agent` per run)
|
|
28
|
+
- When triaging a regression in test runtime — high token counts often correlate with the agent retrying
|
|
29
|
+
|
|
30
|
+
## Caveats
|
|
31
|
+
|
|
32
|
+
- **Only counts what's in local memory.** Runs on machines that haven't pulled from the team remote won't appear. Run `/zibby-memory-pull` first if you want the full team picture.
|
|
33
|
+
- **Pricing is informational.** Public API pricing changes; treat the $ column as a guide, not a bill. The token counts themselves are exact.
|
|
34
|
+
- **Empty if you've never run a test with memory enabled.** Confirm the runs are in there with `/zibby-memory-stats` first.
|
|
35
|
+
|
|
36
|
+
## Related
|
|
37
|
+
|
|
38
|
+
- `/zibby-memory-stats` — what's in the DB at all
|
|
39
|
+
- `/zibby-memory-pull` — refresh from team remote before reading cost
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
|
+
# /zibby-memory-pull — pull the team's latest test memory from the configured remote
|
|
3
|
+
|
|
4
|
+
You are helping the user fetch the team's latest learnings (selectors, page model, insights, run history) from the project's configured memory remote into local `.zibby/memory/.dolt/`.
|
|
5
|
+
|
|
6
|
+
Canonical docs: **https://docs.zibby.app/tests/memory**
|
|
7
|
+
|
|
8
|
+
## When this is needed (vs. just runs automatically)
|
|
9
|
+
|
|
10
|
+
`zibby test` auto-pulls before every run when a remote is configured, and auto-pushes after every passing run. So most of the time the user doesn't need to invoke pull manually. Manual pull is for:
|
|
11
|
+
|
|
12
|
+
- Fresh clone of the repo — first sync to seed `.zibby/memory/.dolt/` from the remote
|
|
13
|
+
- After a teammate landed a big batch of new learnings and the user wants them before running anything
|
|
14
|
+
- Inspecting team memory (`/zibby-memory-stats`, `/zibby-memory-cost`) without running a test
|
|
15
|
+
- Reconciling after a manual conflict in the Dolt DB
|
|
16
|
+
|
|
17
|
+
## How to run
|
|
18
|
+
|
|
19
|
+
```
|
|
20
|
+
Bash(zibby memory pull)
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
The CLI fetches from whatever remote `zibby memory remote info` reports — BYO S3/GCS/DoltHub URL or the Zibby-hosted backend. No flags.
|
|
24
|
+
|
|
25
|
+
## Pre-flight: is a remote configured?
|
|
26
|
+
|
|
27
|
+
Before suggesting `pull`, check:
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
Bash(zibby memory remote info)
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
- **No remote configured** → pull errors out. Tell the user to either:
|
|
34
|
+
- Add their own: `zibby memory remote add aws://my-bucket/team/proj/main`
|
|
35
|
+
- Use the hosted one: `zibby memory remote use --hosted` (requires `zibby login`)
|
|
36
|
+
- See `/zibby-memory-remote-use-hosted` for the hosted path.
|
|
37
|
+
- **Hosted remote, signed out** → `zibby login` first.
|
|
38
|
+
|
|
39
|
+
## After pulling
|
|
40
|
+
|
|
41
|
+
Confirm the pull landed with `/zibby-memory-stats` — row counts should jump (selectors, runs, insights) compared to before.
|
|
42
|
+
|
|
43
|
+
## Related
|
|
44
|
+
|
|
45
|
+
- `zibby memory push` — manual push (auto on passing test, but sometimes you want to share now)
|
|
46
|
+
- `/zibby-memory-stats` — verify what came in
|
|
47
|
+
- `/zibby-memory-remote-use-hosted` — switch to the Zibby-managed S3 backend
|
package/dist/templates/zibby-workflow-claude/claude/commands/zibby-memory-remote-use-hosted.md
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
|
+
# /zibby-memory-remote-use-hosted — switch this project's memory remote to Zibby-managed S3
|
|
3
|
+
|
|
4
|
+
You are helping the user point their `.zibby/memory/.dolt/` at Zibby's hosted S3 backend, instead of running their own S3 bucket / GCS / DoltHub repo.
|
|
5
|
+
|
|
6
|
+
Canonical docs: **https://docs.zibby.app/tests/memory**
|
|
7
|
+
|
|
8
|
+
## What this does
|
|
9
|
+
|
|
10
|
+
```
|
|
11
|
+
Bash(zibby memory remote use --hosted)
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Allocates a tenant-scoped prefix on Zibby-managed S3 for this project (keyed on the projectId in `.zibby.config.mjs`) and writes that as the local Dolt remote. After this, every `zibby test` run auto-pulls before and auto-pushes after — same as a BYO remote, just without the bucket plumbing.
|
|
15
|
+
|
|
16
|
+
## Prerequisite: signed in
|
|
17
|
+
|
|
18
|
+
Hosted remote is **signed-in users only**. Verify:
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
Bash(zibby status)
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
If not signed in, run `zibby login` first. The CLI uses the saved session to derive the tenant prefix; it won't fall back to anonymous.
|
|
25
|
+
|
|
26
|
+
## When to use hosted vs BYO
|
|
27
|
+
|
|
28
|
+
| | Hosted (`--hosted`) | BYO (`zibby memory remote add aws://...`) |
|
|
29
|
+
|---|---|---|
|
|
30
|
+
| Setup time | Zero — `--hosted` and you're done | Provision an S3 bucket, IAM, optional KMS |
|
|
31
|
+
| Who can read | Everyone with project access on Zibby | Whoever you grant in IAM |
|
|
32
|
+
| Where data lives | Zibby-managed AWS account | Your account |
|
|
33
|
+
| Compliance / data-residency | Limited regions | Wherever you want |
|
|
34
|
+
| Cost | Included in plan | Your S3 bill |
|
|
35
|
+
|
|
36
|
+
If the user has any data-residency requirement or a regulated workload, prefer BYO. Otherwise hosted is the path of least resistance.
|
|
37
|
+
|
|
38
|
+
## Switching from BYO to hosted
|
|
39
|
+
|
|
40
|
+
`zibby memory remote use --hosted` overwrites the existing remote. If they had a BYO remote and might want to keep its history, run `zibby memory push` against the old remote first so nothing's lost — then switch.
|
|
41
|
+
|
|
42
|
+
## After switching
|
|
43
|
+
|
|
44
|
+
1. `zibby memory pull` — seed `.zibby/memory/.dolt/` from the hosted prefix (no-op the very first time per project)
|
|
45
|
+
2. `/zibby-memory-stats` — confirm
|
|
46
|
+
3. Commit `.zibby.config.mjs` if you set `memorySync.remote: 'hosted'` so teammates auto-wire on next `zibby init`
|
|
47
|
+
|
|
48
|
+
## Reverting
|
|
49
|
+
|
|
50
|
+
```
|
|
51
|
+
Bash(zibby memory remote remove)
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Drops the remote — memory becomes local-only again. The data on Zibby's S3 isn't deleted (it's still tenant-scoped), but nothing pushes or pulls until a new remote is configured.
|
|
55
|
+
|
|
56
|
+
## Related
|
|
57
|
+
|
|
58
|
+
- `/zibby-memory-pull` — manual pull (auto on test start)
|
|
59
|
+
- `/zibby-memory-stats` — verify what's in the local DB
|
|
60
|
+
- `zibby memory remote info` — show current remote config
|
|
61
|
+
- `zibby memory remote add <url>` — BYO remote (S3/GCS/DoltHub/file:///)
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
|
+
# /zibby-memory-stats — inspect the local test memory database
|
|
3
|
+
|
|
4
|
+
You are helping the user see what's in their `.zibby/memory/.dolt/` test-memory DB — row counts per table, last commit, and per-spec breakdown.
|
|
5
|
+
|
|
6
|
+
Canonical docs: **https://docs.zibby.app/tests/memory**
|
|
7
|
+
|
|
8
|
+
## What the command shows
|
|
9
|
+
|
|
10
|
+
```
|
|
11
|
+
Bash(zibby memory stats)
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Prints a summary of the local Dolt database:
|
|
15
|
+
- **Test runs** — total runs recorded, pass/fail split, last run timestamp
|
|
16
|
+
- **Selectors** — total cached selectors, top pages by selector count
|
|
17
|
+
- **Page model** — pages mapped, total elements
|
|
18
|
+
- **Navigation** — known transitions
|
|
19
|
+
- **Insights** — count by category (`selector_tip | timing | navigation | workaround | flaky | general`)
|
|
20
|
+
- **Dolt status** — current branch, last commit hash, uncommitted changes
|
|
21
|
+
|
|
22
|
+
## When to invoke
|
|
23
|
+
|
|
24
|
+
- User asks "what does Zibby know about my app?" or "show me what's in test memory"
|
|
25
|
+
- After running a few tests, to confirm the agent is actually persisting learnings
|
|
26
|
+
- Before a `zibby memory compact` to see how much there is to prune
|
|
27
|
+
- Before a `zibby memory remote add` to know what's about to ship to the team
|
|
28
|
+
|
|
29
|
+
## Empty database?
|
|
30
|
+
|
|
31
|
+
If the user just ran `zibby memory init` (or it auto-initialized on first `zibby test`), most counts will be 0. That's fine — selectors and page model populate after the first successful run. Suggest running a test first.
|
|
32
|
+
|
|
33
|
+
## Related commands
|
|
34
|
+
|
|
35
|
+
- `/zibby-memory-cost` — real LLM token spend per spec / per domain
|
|
36
|
+
- `/zibby-memory-pull` — pull team's latest learnings from the configured remote
|
|
37
|
+
- `zibby memory compact` — prune old runs (`--max-runs N`, `--max-age <days>`)
|
|
38
|
+
- `zibby memory reset -f` — wipe the DB (destructive — confirm first)
|
|
@@ -1,10 +1,12 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
# /zibby-static-ip — set up dedicated outbound static IP for a workflow
|
|
3
3
|
|
|
4
4
|
You are helping the user route a workflow's outbound traffic through a static IP address — needed when the workflow calls APIs that require IP allowlisting (corporate GitLab/GitHub Enterprise, internal SaaS, firewalls).
|
|
5
5
|
|
|
6
6
|
Canonical docs: **https://docs.zibby.app/workflows/egress**
|
|
7
7
|
|
|
8
|
+
> Note: the `--dedicated-ip` flag lives on the legacy alias `zibby deploy <name>`, NOT on `zibby workflow deploy <name>`. The two share a handler, but only `zibby deploy` exposes this flag in `--help`.
|
|
9
|
+
|
|
8
10
|
## What "static IP" means here
|
|
9
11
|
|
|
10
12
|
By default, workflow tasks run on Fargate and their outbound traffic exits via AWS-managed IPs that rotate. With the **dedicated egress** addon enabled, the workflow's outbound traffic is routed through a Zibby-managed proxy whose IP is pinned and customer-allowlistable.
|
|
@@ -23,19 +25,19 @@ Two pieces:
|
|
|
23
25
|
|
|
24
26
|
2. **Check current state:**
|
|
25
27
|
```
|
|
26
|
-
Bash(zibby
|
|
28
|
+
Bash(zibby deploy <name> --dedicated-ip status)
|
|
27
29
|
```
|
|
28
30
|
Output tells you: addon active or inactive, this workflow currently using it or not, and the assigned IPs to publish to customers.
|
|
29
31
|
|
|
30
32
|
3. **If addon is inactive — enable it** (only after explicit user confirmation):
|
|
31
33
|
```
|
|
32
|
-
Bash(zibby
|
|
34
|
+
Bash(zibby deploy <name> --dedicated-ip enable)
|
|
33
35
|
```
|
|
34
36
|
This is one-time per account. After this, the addon is active for ALL workflows in the account that opt in.
|
|
35
37
|
|
|
36
38
|
4. **Opt this workflow in:**
|
|
37
39
|
```
|
|
38
|
-
Bash(zibby
|
|
40
|
+
Bash(zibby deploy <name> --dedicated-ip use)
|
|
39
41
|
```
|
|
40
42
|
From now on, every deploy of this workflow + every triggered execution routes outbound through the static IP.
|
|
41
43
|
|
|
@@ -52,8 +54,8 @@ Two pieces:
|
|
|
52
54
|
|
|
53
55
|
## Reverting
|
|
54
56
|
|
|
55
|
-
- `Bash(zibby
|
|
56
|
-
- `Bash(zibby
|
|
57
|
+
- `Bash(zibby deploy <name> --dedicated-ip unuse)` — stop routing this workflow's egress through the static IP. Other opted-in workflows are unaffected.
|
|
58
|
+
- `Bash(zibby deploy <name> --dedicated-ip disable)` — disable the addon entirely (also stops billing).
|
|
57
59
|
|
|
58
60
|
## Tell the user the IPs
|
|
59
61
|
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
# /zibby-test-debug — diagnose a failing Zibby test
|
|
3
3
|
|
|
4
4
|
You are helping the user figure out why a test failed.
|
|
@@ -34,7 +34,7 @@ Spec ambiguity is the most common cause. If the spec says "click the button" and
|
|
|
34
34
|
```
|
|
35
35
|
Bash(zibby test test-specs/<name>.txt --verbose) # info-level logs
|
|
36
36
|
Bash(zibby test test-specs/<name>.txt --debug) # all logs, lots
|
|
37
|
-
Bash(zibby test test-specs/<name>.txt
|
|
37
|
+
Bash(zibby test test-specs/<name>.txt) # default is headed — drop --headless to watch the browser
|
|
38
38
|
```
|
|
39
39
|
|
|
40
40
|
### 5. Re-execute one node from a prior session
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
# /zibby-test-generate — generate test specs from a Jira ticket / requirements
|
|
3
3
|
|
|
4
4
|
You are helping the user auto-generate test specs from a ticket description (Jira) or a free-text requirements doc. Zibby's `generate` command runs the configured AI agent against the codebase + ticket and produces `.txt` specs in `test-specs/`.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
# /zibby-test-run — execute a Zibby test spec
|
|
3
3
|
|
|
4
4
|
You are helping the user run an existing test spec through Zibby. A spec is a `.txt` file describing what to test in plain language; Zibby's runner turns it into a Playwright execution and produces a video + JSON results.
|
|
@@ -40,9 +40,10 @@ Canonical docs: **https://docs.zibby.app/tests/running**
|
|
|
40
40
|
- `--verbose` / `--debug` — escalate log levels
|
|
41
41
|
- `-m, --mem` — enable test memory (Dolt-backed knowledge from prior runs)
|
|
42
42
|
- `--sync` / `--no-sync` — force / skip cloud upload regardless of config
|
|
43
|
+
- `--sources <ids> --execution <id>` — run cloud-stored test cases from a specific execution (comma-separated IDs)
|
|
43
44
|
|
|
44
45
|
## Common failure modes
|
|
45
46
|
|
|
46
47
|
- **"No spec found"** — path is relative to project root, not cwd. Check `paths.specs` in `.zibby.config.mjs`.
|
|
47
|
-
- **"Browser crashed"** — usually the playwright browser cache is stale.
|
|
48
|
+
- **"Browser crashed"** — usually the playwright browser cache is stale. Drop `--headless` once (default is headed) so you can see what's happening, then re-add `--headless` once it's healthy.
|
|
48
49
|
- **MCP errors during `execute_live`** — the agent's MCP tool config may need refreshing. See `/zibby-test-debug`.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
# /zibby-test-write — author a new Zibby test spec
|
|
3
3
|
|
|
4
4
|
You are helping the user write a new test spec. Specs are plain-language `.txt` files in `test-specs/` (configurable via `.zibby.config.mjs` `paths.specs`). Zibby's runner converts them to Playwright at execution time.
|
|
@@ -1,11 +1,11 @@
|
|
|
1
|
-
<!-- zibby-template-version:
|
|
1
|
+
<!-- zibby-template-version: 4 -->
|
|
2
2
|
# /zibby-trigger — run a deployed Zibby workflow
|
|
3
3
|
|
|
4
4
|
You are helping the user trigger a deployed workflow execution.
|
|
5
5
|
|
|
6
6
|
A trigger creates a new ECS Fargate task that loads the workflow's bundle, runs the graph, and writes status + logs as it goes.
|
|
7
7
|
|
|
8
|
-
Canonical docs: **https://docs.zibby.app/
|
|
8
|
+
Canonical docs: **https://docs.zibby.app/cloud/triggering**
|
|
9
9
|
|
|
10
10
|
## Steps
|
|
11
11
|
|
|
@@ -13,15 +13,19 @@ Canonical docs: **https://docs.zibby.app/workflows/triggering**
|
|
|
13
13
|
|
|
14
14
|
2. **Construct the input.** Workflows take a JSON input that nodes can read via `ctx.input`. Ask the user what input the workflow expects (or read `workflow.json`'s `inputSchema` if present).
|
|
15
15
|
|
|
16
|
-
3. **Run the trigger
|
|
16
|
+
3. **Run the trigger.** Three ways to pass input — they merge with this **precedence (highest → lowest)**:
|
|
17
|
+
1. `-p key=value` (repeatable) — wins over everything; great for shell-friendly tweaks on top of a base payload
|
|
18
|
+
2. `--input '<json>'` — full JSON payload as a string
|
|
19
|
+
3. `--input-file path.json` — full JSON/YAML payload from a file (lowest precedence; `-p` and `--input` override individual keys)
|
|
20
|
+
|
|
17
21
|
```
|
|
18
22
|
zibby workflow trigger <uuid> --input '{"key":"value"}'
|
|
19
|
-
```
|
|
20
|
-
Or use `-p key=value` for individual params (more shell-friendly than embedded JSON):
|
|
21
|
-
```
|
|
22
23
|
zibby workflow trigger <uuid> -p ticket=ENG-1234 -p priority=high
|
|
24
|
+
zibby workflow trigger <uuid> --input-file payload.json -p priority=urgent # mix
|
|
23
25
|
```
|
|
24
26
|
|
|
27
|
+
Same flag surface as `zibby workflow run` (local) — flip the verb and the same call shape goes from local to remote.
|
|
28
|
+
|
|
25
29
|
4. **Tail the logs immediately:**
|
|
26
30
|
```
|
|
27
31
|
zibby workflow logs <uuid> -t
|