@ishlabs/cli 0.8.3 → 0.8.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -0
- package/dist/auth.d.ts +1 -0
- package/dist/auth.js +12 -3
- package/dist/commands/ask.js +59 -16
- package/dist/commands/iteration.js +45 -11
- package/dist/commands/profile.js +65 -12
- package/dist/commands/study-run.js +49 -0
- package/dist/commands/study-tester.js +5 -2
- package/dist/commands/study.js +71 -16
- package/dist/connect.js +7 -7
- package/dist/index.js +119 -2
- package/dist/lib/api-client.js +29 -7
- package/dist/lib/command-helpers.d.ts +14 -0
- package/dist/lib/command-helpers.js +40 -0
- package/dist/lib/docs.js +430 -13
- package/dist/lib/output.js +437 -63
- package/dist/lib/skill-content.js +102 -9
- package/dist/lib/types.d.ts +3 -1
- package/dist/upgrade.js +3 -3
- package/package.json +1 -1
package/dist/lib/docs.js
CHANGED
|
@@ -43,6 +43,8 @@ Two top-level run verbs:
|
|
|
43
43
|
## Where to look next
|
|
44
44
|
|
|
45
45
|
- New here? \`ish docs get-page concepts/workspace\`, then \`concepts/study\`.
|
|
46
|
+
- **Cold start?** Run \`ish status\` (alias \`ish whoami\`) — confirms login
|
|
47
|
+
and prints active workspace/study/ask. See \`concepts/active-context\`.
|
|
46
48
|
- Running your first study? \`ish docs get-page guides/first-study\`.
|
|
47
49
|
- Comparing study vs ask? \`ish docs get-page concepts/run-verbs\`.
|
|
48
50
|
- Need machine-readable output? \`ish docs get-page reference/json-mode\`.
|
|
@@ -81,6 +83,10 @@ ish workspace use w-6ec # set as active
|
|
|
81
83
|
ish workspace get # show the active workspace
|
|
82
84
|
ish workspace site-access status
|
|
83
85
|
\`\`\`
|
|
86
|
+
|
|
87
|
+
## Related
|
|
88
|
+
|
|
89
|
+
- \`reference/billing-limits\` — \`maxProducts\` cap on workspace creation.
|
|
84
90
|
`;
|
|
85
91
|
const CONCEPT_STUDY = `# concept: study
|
|
86
92
|
|
|
@@ -101,17 +107,63 @@ its iterations. Think: a study is the recipe; an iteration is one batch.
|
|
|
101
107
|
|
|
102
108
|
## Lifecycle
|
|
103
109
|
|
|
104
|
-
1. \`ish study create --name "Onboarding UX" --modality interactive --assignment "Sign up:Complete the signup flow" --question "How easy was it?"\`
|
|
105
|
-
2. \`ish iteration create --url https://example.com\`
|
|
106
|
-
3. \`ish study run --sample 5 --country SE\`
|
|
110
|
+
1. \`ish study create --name "Onboarding UX" --modality interactive --assignment "Sign up:Complete the signup flow" --question "How easy was it?"\` — creates the recipe with **zero iterations**.
|
|
111
|
+
2. \`ish iteration create --url https://example.com\` — first iteration becomes label \`A\`.
|
|
112
|
+
3. \`ish study run --sample 5 --country SE\` — dispatches simulations.
|
|
107
113
|
4. \`ish study results\` or \`ish study wait\` to gather outputs.
|
|
108
114
|
|
|
115
|
+
### One-shot variant
|
|
116
|
+
|
|
117
|
+
\`study create\` now accepts \`--content-text\` (text modality) or
|
|
118
|
+
\`--url\` (interactive modality) inline; iteration A is created in the
|
|
119
|
+
same call. Useful when you have a single test artifact and don't need
|
|
120
|
+
to A/B iterations:
|
|
121
|
+
|
|
122
|
+
\`\`\`
|
|
123
|
+
ish study create --modality text --content-type email \\
|
|
124
|
+
--name "Daily Brief concept" \\
|
|
125
|
+
--assignment "Read:Read the email and react" \\
|
|
126
|
+
--question "What stood out?" \\
|
|
127
|
+
--content-text @./brief.md
|
|
128
|
+
# → study + iteration A in one call, ready for \`study run\`.
|
|
129
|
+
\`\`\`
|
|
130
|
+
|
|
131
|
+
Without those flags no iteration is created — agents can no longer
|
|
132
|
+
trip the old "empty A" footgun where \`study run\` silently targeted a
|
|
133
|
+
placeholder.
|
|
134
|
+
|
|
135
|
+
## Status fields (read \`runtime_status\`, not \`status\`)
|
|
136
|
+
|
|
137
|
+
Every study response carries two status-shaped fields:
|
|
138
|
+
|
|
139
|
+
- \`status\` — the raw lifecycle column on the row, values
|
|
140
|
+
\`draft | running | completed | cancelled\`. Updated lazily; can
|
|
141
|
+
disagree with what the testers actually did.
|
|
142
|
+
- \`runtime_status\` — derived by aggregating the iteration testers'
|
|
143
|
+
states. Values: \`draft | running | completed |
|
|
144
|
+
completed_with_errors | cancelled\`. **Never reports \`failed\` while
|
|
145
|
+
completed runs exist** (the Bk2 invariant). Prefer this for any
|
|
146
|
+
agent decision.
|
|
147
|
+
|
|
148
|
+
The CLI also surfaces a \`status_inferred\` field + stderr warning when
|
|
149
|
+
it detects raw-vs-derived inconsistencies. See \`reference/json-mode\`.
|
|
150
|
+
|
|
151
|
+
## Generate vs create
|
|
152
|
+
|
|
153
|
+
\`ish study generate --problem "..."\` runs an LLM-backed flow that
|
|
154
|
+
picks a sensible modality from your brief and returns a
|
|
155
|
+
\`modality_rationale\` field (≤30 words) explaining the choice.
|
|
156
|
+
Override before adding iterations via
|
|
157
|
+
\`ish study update <id> --modality text\` if the rationale shows the
|
|
158
|
+
pick was wrong.
|
|
159
|
+
|
|
109
160
|
## Related
|
|
110
161
|
|
|
111
162
|
- \`concepts/iteration\` — the unit of execution within a study.
|
|
112
163
|
- \`concepts/assignment\` — task definition syntax.
|
|
113
164
|
- \`concepts/questionnaire\` — question types and timing.
|
|
114
165
|
- \`concepts/run-verbs\` — when to use \`study run\` vs \`ask run\`.
|
|
166
|
+
- \`reference/billing-limits\` — \`maxStudiesPerProduct\` cap on study creation.
|
|
115
167
|
`;
|
|
116
168
|
const CONCEPT_ITERATION = `# concept: iteration
|
|
117
169
|
|
|
@@ -157,11 +209,42 @@ ish iteration list --study s-b2c
|
|
|
157
209
|
ish iteration get i-d4e
|
|
158
210
|
\`\`\`
|
|
159
211
|
|
|
212
|
+
## No more auto-empty iteration A
|
|
213
|
+
|
|
214
|
+
\`ish study create\` and \`ish study generate\` **do not auto-create
|
|
215
|
+
iteration A** anymore (Pattern E remediation, ish-cli v0.8.x). The
|
|
216
|
+
first explicit \`ish iteration create\` becomes label A, second is B,
|
|
217
|
+
etc. Running \`ish study run\` on a study with zero iterations exits
|
|
218
|
+
2 with a clear error pointing you to \`ish iteration create\`.
|
|
219
|
+
|
|
220
|
+
If you do somehow run against an interactive iteration without a URL
|
|
221
|
+
(or a media iteration without content), \`study run\` exits 2 with:
|
|
222
|
+
|
|
223
|
+
\`\`\`
|
|
224
|
+
Iteration "A" (i-...) has no URL configured yet. Add a URL with
|
|
225
|
+
\`ish iteration create --study s-... --url <url>\` (or update the
|
|
226
|
+
existing iteration via \`ish iteration update i-... --details-json '{...}'\`),
|
|
227
|
+
then retry.
|
|
228
|
+
\`\`\`
|
|
229
|
+
|
|
230
|
+
Treat this as actionable, not transient — re-running won't change anything.
|
|
231
|
+
|
|
232
|
+
## Default segmentation for text/image iterations
|
|
233
|
+
|
|
234
|
+
For text-modality iterations created with just \`--content-text\` (and
|
|
235
|
+
similarly \`--image-urls\` for image), the worker now synthesises a
|
|
236
|
+
single whole-content section if no \`segmentation\` was supplied. This
|
|
237
|
+
means a minimal \`ish iteration create --study s-XYZ --content-text
|
|
238
|
+
"..."\` actually runs end-to-end without you needing to author a
|
|
239
|
+
SegmentationConfig manually. Author your own segmentation when you
|
|
240
|
+
want section-level reactions; otherwise the default just works.
|
|
241
|
+
|
|
160
242
|
## Related
|
|
161
243
|
|
|
162
244
|
- \`concepts/study\` — the parent artifact.
|
|
163
245
|
- \`concepts/run-verbs\` — how \`ish study run\` selects the iteration.
|
|
164
246
|
- \`concepts/audience\` — how testers are picked for a run.
|
|
247
|
+
- \`reference/billing-limits\` — \`maxIterationsPerStudy\` cap on iteration creation.
|
|
165
248
|
`;
|
|
166
249
|
const CONCEPT_ASSIGNMENT = `# concept: assignment
|
|
167
250
|
|
|
@@ -213,7 +296,7 @@ replaces the full assignment list — additive editing is not supported.
|
|
|
213
296
|
const CONCEPT_QUESTIONNAIRE = `# concept: questionnaire
|
|
214
297
|
|
|
215
298
|
The **questionnaire** is the list of \`interview_questions\` a tester
|
|
216
|
-
answers before
|
|
299
|
+
answers before or after their assignments. A study has 0..N
|
|
217
300
|
questions, each with a type and a timing.
|
|
218
301
|
|
|
219
302
|
## Question shape
|
|
@@ -221,12 +304,12 @@ questions, each with a type and a timing.
|
|
|
221
304
|
\`\`\`json
|
|
222
305
|
{
|
|
223
306
|
"question": "How easy was checkout?",
|
|
224
|
-
"type": "slider", // text | slider | likert |
|
|
225
|
-
//
|
|
226
|
-
"timing": "after", // before |
|
|
307
|
+
"type": "slider", // text | slider | likert |
|
|
308
|
+
// single-choice | multiple-choice | number
|
|
309
|
+
"timing": "after", // before | after
|
|
227
310
|
"min": 1, "max": 7, "step": 1,
|
|
228
311
|
"labels": ["Hard", "Easy"],
|
|
229
|
-
"options": ["A", "B", "C"] // only for
|
|
312
|
+
"options": ["A", "B", "C"] // only for single-choice / multiple-choice
|
|
230
313
|
}
|
|
231
314
|
\`\`\`
|
|
232
315
|
|
|
@@ -289,8 +372,53 @@ ish ask run --prompt "And now which?" \\
|
|
|
289
372
|
ish ask list
|
|
290
373
|
ish ask get a-6ec --round 2
|
|
291
374
|
ish ask results a-6ec
|
|
375
|
+
ish ask results a-6ec --json | jq '.rounds[0].aggregates'
|
|
376
|
+
\`\`\`
|
|
377
|
+
|
|
378
|
+
## Reading the verdict
|
|
379
|
+
|
|
380
|
+
For \`--wants-pick\` / \`--wants-ratings\` rounds, \`ask results --json\`
|
|
381
|
+
includes an \`aggregates\` field per round so you don't have to parse
|
|
382
|
+
prose:
|
|
383
|
+
|
|
384
|
+
\`\`\`json
|
|
385
|
+
{
|
|
386
|
+
"picks": { "A": 3, "B": 0 },
|
|
387
|
+
"ratings": { "A": { "mean": 4.667, "n": 3 },
|
|
388
|
+
"B": { "mean": 2.000, "n": 3 } },
|
|
389
|
+
"winner": { "letter": "A", "count": 3, "tied": false }
|
|
390
|
+
}
|
|
292
391
|
\`\`\`
|
|
293
392
|
|
|
393
|
+
When the ask has 2+ rounds, \`ask results\` also includes a top-level
|
|
394
|
+
\`cross_round_summary\` block with per-round picks/winner and a
|
|
395
|
+
\`picks_delta\` (R1 → last round). Skip the manual diffing of two
|
|
396
|
+
\`ask results\` calls.
|
|
397
|
+
|
|
398
|
+
\`\`\`json
|
|
399
|
+
"cross_round_summary": {
|
|
400
|
+
"rounds": [
|
|
401
|
+
{ "round_number": 1, "picks": {"A": 1, "B": 2}, "winner": {"letter": "B", "count": 2, "tied": false } },
|
|
402
|
+
{ "round_number": 2, "picks": {"A": 3, "B": 0}, "winner": {"letter": "A", "count": 3, "tied": false } }
|
|
403
|
+
],
|
|
404
|
+
"picks_delta": { "A": +2, "B": -2 }
|
|
405
|
+
}
|
|
406
|
+
\`\`\`
|
|
407
|
+
|
|
408
|
+
## Adding follow-up questions to a round
|
|
409
|
+
|
|
410
|
+
\`ish ask add-questions --round N --questions ./qs.json\` is **additive
|
|
411
|
+
by default**: prior phase-1 outputs (comment, pick, ratings) are
|
|
412
|
+
preserved on every non-errored response, and the worker only answers
|
|
413
|
+
the newly-added questions for each tester. Existing picks stay stable.
|
|
414
|
+
|
|
415
|
+
Pass \`--redispatch-all\` for the legacy reset behavior — useful when a
|
|
416
|
+
question is sufficiently different that you want fresh first
|
|
417
|
+
impressions, not augmentation. Without that flag, agents iterating on
|
|
418
|
+
copy can safely append questions without losing prior round results.
|
|
419
|
+
|
|
420
|
+
See \`reference/json-mode\` for the full shape.
|
|
421
|
+
|
|
294
422
|
## Variant syntax
|
|
295
423
|
|
|
296
424
|
\`--variant <type>:<value>[::label=<label>]\`
|
|
@@ -327,6 +455,15 @@ ish ask wait a-6ec --round 2 --timeout 600
|
|
|
327
455
|
ish ask results a-6ec --round 1
|
|
328
456
|
\`\`\`
|
|
329
457
|
|
|
458
|
+
## \`add-questions\` is additive
|
|
459
|
+
|
|
460
|
+
Appending questions to a completed round preserves prior data — variant
|
|
461
|
+
comments, picks, ratings, and earlier-question answers all stay. Only
|
|
462
|
+
the new question(s) get dispatched to the existing testers. Cost is
|
|
463
|
+
roughly N phase-2 LLM calls instead of 2N (no phase-1 re-run). Errored
|
|
464
|
+
responses are skipped entirely; completed responses flip to PENDING and
|
|
465
|
+
re-finalize after the new question is answered.
|
|
466
|
+
|
|
330
467
|
## Related
|
|
331
468
|
|
|
332
469
|
- \`concepts/ask\` — the parent artifact.
|
|
@@ -384,6 +521,7 @@ Expected JSON: \`{ "name": "...", "type": "ai", "gender": "female",
|
|
|
384
521
|
|
|
385
522
|
- \`concepts/source\` — the inputs to \`profile generate\`.
|
|
386
523
|
- \`concepts/audience\` — how profiles get selected into a run.
|
|
524
|
+
- \`reference/billing-limits\` — \`maxCustomTesterProfiles\` cap on profile creation.
|
|
387
525
|
`;
|
|
388
526
|
const CONCEPT_SOURCE = `# concept: source
|
|
389
527
|
|
|
@@ -613,8 +751,8 @@ mode is **auto-enabled when stdout is piped**, so an agent rarely needs
|
|
|
613
751
|
- \`--json\` — force JSON output even on a TTY.
|
|
614
752
|
- \`--fields a,b,c\` — keep only these fields in JSON output (e.g.
|
|
615
753
|
\`alias,name,status\`). Filters per item only;
|
|
616
|
-
|
|
617
|
-
offset}\`) keep their shape.
|
|
754
|
+
list wrappers (\`{items, total, returned,
|
|
755
|
+
limit, offset, has_more}\`) keep their shape.
|
|
618
756
|
- \`--verbose\` — include full UUIDs, timestamps, and (on
|
|
619
757
|
write paths) the full server payload instead
|
|
620
758
|
of the compact response.
|
|
@@ -625,9 +763,26 @@ mode is **auto-enabled when stdout is piped**, so an agent rarely needs
|
|
|
625
763
|
|
|
626
764
|
The CLI guarantees these contracts so agents can chain safely:
|
|
627
765
|
|
|
628
|
-
- **
|
|
629
|
-
|
|
630
|
-
|
|
766
|
+
- **Every list response is a six-key envelope.** All
|
|
767
|
+
\`<entity> list --json\` responses (workspace, study, iteration, ask,
|
|
768
|
+
profile, config) return:
|
|
769
|
+
|
|
770
|
+
\`\`\`json
|
|
771
|
+
{
|
|
772
|
+
"items": [...],
|
|
773
|
+
"total": 121, // server-provided when paginated; else items.length
|
|
774
|
+
"returned": 50, // items.length, always present
|
|
775
|
+
"limit": 50,
|
|
776
|
+
"offset": 0,
|
|
777
|
+
"has_more": true // total > offset + returned
|
|
778
|
+
}
|
|
779
|
+
\`\`\`
|
|
780
|
+
|
|
781
|
+
When the server doesn't paginate, \`total = returned = limit\`,
|
|
782
|
+
\`offset = 0\`, \`has_more = false\` (synthesized client-side).
|
|
783
|
+
\`--fields\` strips per-item, never the envelope — those six keys are
|
|
784
|
+
always present. Use \`has_more\` to detect truncation rather than
|
|
785
|
+
counting items yourself.
|
|
631
786
|
- **Write paths always include \`id\` AND \`alias\`.** Even with
|
|
632
787
|
\`--fields\` set, you can identify the affected resource. Default
|
|
633
788
|
write-path JSON is compact (\`{id, alias, name, updated_at,
|
|
@@ -635,9 +790,99 @@ The CLI guarantees these contracts so agents can chain safely:
|
|
|
635
790
|
- **\`profile generate\` trims \`simulation_config\` by default** (~9×
|
|
636
791
|
smaller than the raw response). Pass \`--include-simulation-config\`
|
|
637
792
|
if you need it.
|
|
793
|
+
- **\`<entity> get\` accepts multiple IDs.** \`profile get\`, \`study get\`,
|
|
794
|
+
\`iteration get\`, and \`ask get\` all take \`<ids...>\` — pass two or
|
|
795
|
+
more aliases (space- or comma-separated) and the response is a
|
|
796
|
+
\`{items:[...], total:N}\` envelope. Use this instead of piping
|
|
797
|
+
\`list --json\` to \`jq\`/\`python\` to filter by alias.
|
|
798
|
+
- **Ask detail JSON includes denormalized counts** so agents don't
|
|
799
|
+
have to count nested arrays. \`ask get\`, \`ask create --wait\`,
|
|
800
|
+
\`ask run --wait\`, and \`ask wait --verbose\` all add:
|
|
801
|
+
|
|
802
|
+
\`\`\`json
|
|
803
|
+
{
|
|
804
|
+
"testers_count": 3,
|
|
805
|
+
"responses_total": 9,
|
|
806
|
+
"responses_complete": 9,
|
|
807
|
+
"rounds": [
|
|
808
|
+
{ "responses_total": 3, "responses_complete": 3, "...": "..." }
|
|
809
|
+
]
|
|
810
|
+
}
|
|
811
|
+
\`\`\`
|
|
812
|
+
|
|
813
|
+
\`responses_errored\` only appears when at least one response errored.
|
|
814
|
+
Use these instead of \`jq '.testers | length'\` /
|
|
815
|
+
\`jq '.rounds[0].responses | length'\`.
|
|
638
816
|
- **\`study run --json\` exposes tester handles.** The top-level
|
|
639
817
|
\`tester_ids[]\` and \`tester_aliases[]\` arrays are the canonical
|
|
640
818
|
inputs to \`ish study poll/wait/cancel\`.
|
|
819
|
+
- **Study responses carry a derived \`runtime_status\` field**
|
|
820
|
+
(\`draft | running | completed | completed_with_errors | cancelled\`).
|
|
821
|
+
Prefer this over the raw \`status\` field — \`runtime_status\` is
|
|
822
|
+
computed from the iteration testers' actual run state and never
|
|
823
|
+
reports \`failed\` while completed runs exist. Available on
|
|
824
|
+
\`study get\`, \`study results\`, and the response from
|
|
825
|
+
\`study generate\`. The CLI also surfaces a \`status_inferred\` field
|
|
826
|
+
alongside the raw \`status\` when it detects a partial-failure
|
|
827
|
+
inconsistency, plus a stderr warning ("Warning: study reports
|
|
828
|
+
status='failed' but N/M testers completed…").
|
|
829
|
+
- **\`study generate --json\` includes a \`modality_rationale\`** —
|
|
830
|
+
one short sentence explaining why the LLM picked that modality. Use
|
|
831
|
+
it to detect mis-classifications (e.g. brief was a static concept doc
|
|
832
|
+
but rationale says "live UI flow") and override via
|
|
833
|
+
\`study update <id> --modality text\` before adding iterations.
|
|
834
|
+
- **\`ask add-questions\` is additive by default.** Appending questions
|
|
835
|
+
preserves variant comments / picks / ratings / prior-question
|
|
836
|
+
answers; only the new question(s) get dispatched. Cost: roughly N
|
|
837
|
+
phase-2 LLM calls instead of 2N. Pass \`--redispatch-all\` for the
|
|
838
|
+
legacy reset behavior when you want fresh first impressions.
|
|
839
|
+
- **\`ask results --json\` includes \`cross_round_summary\` for 2+
|
|
840
|
+
rounds.** Top-level field with per-round picks/winner snapshots and
|
|
841
|
+
a \`picks_delta\` (R1 → last round). Replaces hand-rolled diffing of
|
|
842
|
+
two \`ask results\` calls.
|
|
843
|
+
- **No more auto-empty iteration A.** \`study create\` and
|
|
844
|
+
\`study generate\` no longer produce a placeholder iteration A. The
|
|
845
|
+
first explicit \`ish iteration create\` becomes label A.
|
|
846
|
+
\`study create\` now accepts \`--content-text\` (text) or \`--url\`
|
|
847
|
+
(interactive) inline so a single call yields a runnable study.
|
|
848
|
+
Running \`study run\` on a study with zero iterations exits 2 with
|
|
849
|
+
a suggestion to run \`ish iteration create\` first.
|
|
850
|
+
- **Tester responses include \`error_message\`.** When a tester is
|
|
851
|
+
\`status: failed\`, the JSON exposes \`error_message: "<reason>"\` so
|
|
852
|
+
agents can act without drilling into logs. \`study results\` rolls
|
|
853
|
+
this up: top-level \`failed_count\`, plus per-tester \`error_message\`
|
|
854
|
+
in the \`testers[]\` array, and a "Failed testers" subsection in
|
|
855
|
+
human output. Empty when the tester succeeded.
|
|
856
|
+
- **\`profile list\` emits a stderr pagination hint** when
|
|
857
|
+
\`has_more=true\` and stdout is human (TTY, not piped, not \`--quiet\`).
|
|
858
|
+
Format: "showing N–M of TOTAL; pass --offset M --limit N for more."
|
|
859
|
+
JSON consumers read \`has_more\` directly off the envelope.
|
|
860
|
+
- **\`ask results --json\` adds an \`aggregates\` field per round.** For
|
|
861
|
+
rounds with \`wants_pick\`/\`wants_ratings\`, the CLI computes the
|
|
862
|
+
verdict locally so agents don't have to parse comment prose:
|
|
863
|
+
|
|
864
|
+
\`\`\`json
|
|
865
|
+
{
|
|
866
|
+
"aggregates": {
|
|
867
|
+
"picks": { "A": 3, "B": 0 },
|
|
868
|
+
"ratings": { "A": { "mean": 4.667, "n": 3 },
|
|
869
|
+
"B": { "mean": 2.000, "n": 3 } },
|
|
870
|
+
"winner": { "letter": "A", "count": 3, "tied": false }
|
|
871
|
+
}
|
|
872
|
+
}
|
|
873
|
+
\`\`\`
|
|
874
|
+
|
|
875
|
+
\`picks\` is present iff \`wants_pick\`; \`ratings\` is present iff
|
|
876
|
+
\`wants_ratings\` and ≥ 1 rating was submitted; \`winner\` is the
|
|
877
|
+
highest pick count (\`tied: true\` if multiple variants share the
|
|
878
|
+
top). \`mean\` is rounded to 3 decimal places; \`n\` is the rating
|
|
879
|
+
count for that variant.
|
|
880
|
+
- **\`ask results --json\` deduplicates tester profile snapshots.** When
|
|
881
|
+
\`tester_profile\` and \`tester_profile_snapshot\` share all
|
|
882
|
+
overlapping fields (the common case — they only diverge if the
|
|
883
|
+
profile was edited after dispatch), the snapshot is collapsed to
|
|
884
|
+
\`{snapshotted_at, snapshot_version, _matches_tester_profile: true}\`.
|
|
885
|
+
Use \`--verbose\` to keep both copies in full.
|
|
641
886
|
|
|
642
887
|
## Exit codes
|
|
643
888
|
|
|
@@ -774,6 +1019,166 @@ ish study results --json | jq .
|
|
|
774
1019
|
- Want a quick reaction test instead of an interactive study? Skip to
|
|
775
1020
|
\`ish docs get-page concepts/ask\`.
|
|
776
1021
|
`;
|
|
1022
|
+
const CONCEPT_ACTIVE_CONTEXT = `# concept: active context
|
|
1023
|
+
|
|
1024
|
+
The CLI keeps a small amount of session state in \`~/.ish/config.json\`
|
|
1025
|
+
(or wherever \`ISH_HOME\` points) so commands don't need to repeat IDs:
|
|
1026
|
+
|
|
1027
|
+
- \`access_token\` / \`refresh_token\` — the OAuth pair from \`ish login\`.
|
|
1028
|
+
- \`workspace\` — set by \`ish workspace use <id>\`.
|
|
1029
|
+
- \`study\` — set by \`ish study use <id>\`.
|
|
1030
|
+
- \`ask\` — set by \`ish ask use <id>\`.
|
|
1031
|
+
|
|
1032
|
+
Most commands fall back to these when their corresponding flag is
|
|
1033
|
+
omitted (\`--workspace\`, \`--study\`, \`--ask\`).
|
|
1034
|
+
|
|
1035
|
+
## Inspecting active context
|
|
1036
|
+
|
|
1037
|
+
\`ish status\` (alias: \`ish whoami\`) is the canonical way to see what's
|
|
1038
|
+
configured. **Run it as the first command on a cold start** — it
|
|
1039
|
+
confirms login, prints the active workspace/study/ask handles, and
|
|
1040
|
+
shows how long the token has left.
|
|
1041
|
+
|
|
1042
|
+
\`\`\`bash
|
|
1043
|
+
ish status
|
|
1044
|
+
# User: you@example.com (token valid, expires in 47m)
|
|
1045
|
+
# Workspace: Onboarding revamp (w-6ec)
|
|
1046
|
+
# Study: —
|
|
1047
|
+
# Ask: a-6ec "tagline AB"
|
|
1048
|
+
# Home: /home/you/.ish
|
|
1049
|
+
# API: https://api.ishlabs.io
|
|
1050
|
+
\`\`\`
|
|
1051
|
+
|
|
1052
|
+
JSON shape (\`ish status --json\` or piped):
|
|
1053
|
+
|
|
1054
|
+
\`\`\`json
|
|
1055
|
+
{
|
|
1056
|
+
"user": { "email": "...", "token_valid": true, "expires_in_seconds": 2820 },
|
|
1057
|
+
"workspace": { "id": "...", "alias": "w-6ec", "name": "Onboarding revamp" },
|
|
1058
|
+
"study": null,
|
|
1059
|
+
"ask": { "id": "...", "alias": "a-6ec", "name": "tagline AB" },
|
|
1060
|
+
"api_url": "https://api.ishlabs.io",
|
|
1061
|
+
"home": "/home/you/.ish"
|
|
1062
|
+
}
|
|
1063
|
+
\`\`\`
|
|
1064
|
+
|
|
1065
|
+
\`status\` does not error when the user is logged out — it returns
|
|
1066
|
+
\`user: null\` plus a \`hint\` field telling the caller to run
|
|
1067
|
+
\`ish login\`. Safe to run unconditionally at the start of any
|
|
1068
|
+
script or agent session.
|
|
1069
|
+
|
|
1070
|
+
## Setting / clearing active context
|
|
1071
|
+
|
|
1072
|
+
\`\`\`bash
|
|
1073
|
+
ish workspace use w-6ec # set
|
|
1074
|
+
ish workspace use --clear # clear
|
|
1075
|
+
|
|
1076
|
+
ish study use s-b2c
|
|
1077
|
+
ish study use --clear
|
|
1078
|
+
|
|
1079
|
+
ish ask use a-6ec
|
|
1080
|
+
ish ask use --clear
|
|
1081
|
+
\`\`\`
|
|
1082
|
+
|
|
1083
|
+
## Overriding without persisting
|
|
1084
|
+
|
|
1085
|
+
Every read command accepts \`--workspace <id>\`, \`--study <id>\`, or
|
|
1086
|
+
\`--ask <id>\` to override the saved active value for one invocation
|
|
1087
|
+
without touching the config. Useful for one-off pokes at another
|
|
1088
|
+
resource.
|
|
1089
|
+
|
|
1090
|
+
\`--workspace\` is accepted on **every workspace-scoped subcommand**
|
|
1091
|
+
(\`ask\`, \`study\`, \`iteration\`, \`profile\`, \`source\` and their
|
|
1092
|
+
descendants). When workspace is inferable from the subject ID alias
|
|
1093
|
+
(e.g. \`ish ask delete a-6ec\`) the value is silently ignored — agents
|
|
1094
|
+
can pass it reflexively without tripping "unknown option" errors. Out
|
|
1095
|
+
of scope: \`workspace\`, \`config\`, \`docs\`, \`init\`, \`login\`,
|
|
1096
|
+
\`logout\`, \`whoami\`, \`upgrade\` (none of these need a workspace).
|
|
1097
|
+
|
|
1098
|
+
## Related
|
|
1099
|
+
|
|
1100
|
+
- \`reference/aliases\` — the prefix scheme used by every entity.
|
|
1101
|
+
- \`reference/json-mode\` — output contracts for piping \`ish status\`.
|
|
1102
|
+
`;
|
|
1103
|
+
const REFERENCE_BILLING_LIMITS = `# reference: billing tier limits
|
|
1104
|
+
|
|
1105
|
+
Some create operations are gated by your account's billing tier. The
|
|
1106
|
+
backend enforces these. The CLI just renders the structured rejection.
|
|
1107
|
+
There is no way to bypass enforcement from the CLI; running the same
|
|
1108
|
+
\`POST\` with \`curl\` will hit the same gate.
|
|
1109
|
+
|
|
1110
|
+
The web UI reads these caps at runtime from
|
|
1111
|
+
\`GET /api/v1/billing/limits\` (cached for one hour) and falls back to
|
|
1112
|
+
its build-time snapshot if the endpoint is unreachable. The table below
|
|
1113
|
+
is the CLI's own snapshot, intentionally release-pinned for offline
|
|
1114
|
+
use; re-pull it after each \`ish-cli\` release. The source of truth at
|
|
1115
|
+
request time, for any client, is the backend's \`TIER_LIMITS\` dict in
|
|
1116
|
+
\`tier_limits.py\`.
|
|
1117
|
+
|
|
1118
|
+
## Limits enforced
|
|
1119
|
+
|
|
1120
|
+
| Limit | Free | Media | Starter | Pro | Enterprise |
|
|
1121
|
+
|-----------------------------|------|-------|---------|-----|------------|
|
|
1122
|
+
| \`maxProducts\` | 1 | 1 | ∞ | ∞ | ∞ |
|
|
1123
|
+
| \`maxStudiesPerProduct\` | 3 | ∞ | ∞ | ∞ | ∞ |
|
|
1124
|
+
| \`maxIterationsPerStudy\` | 2 | ∞ | ∞ | ∞ | ∞ |
|
|
1125
|
+
| \`maxCustomTesterProfiles\` | 3 | 10 | 10 | ∞ | ∞ |
|
|
1126
|
+
|
|
1127
|
+
Commands that may hit a limit: \`ish workspace create\`,
|
|
1128
|
+
\`ish study create\`, \`ish study generate\`, \`ish iteration create\`,
|
|
1129
|
+
\`ish profile create\`, \`ish profile generate\`.
|
|
1130
|
+
|
|
1131
|
+
## What you see when a limit is hit
|
|
1132
|
+
|
|
1133
|
+
Human output (stderr):
|
|
1134
|
+
|
|
1135
|
+
\`\`\`
|
|
1136
|
+
Error: Free plan allows 3 studies per workspace. Upgrade to add more.
|
|
1137
|
+
→ Upgrade your plan at https://app.ishlabs.io/billing
|
|
1138
|
+
→ Run \`ish docs get-page reference/billing-limits\` for the tier table
|
|
1139
|
+
\`\`\`
|
|
1140
|
+
|
|
1141
|
+
JSON output (stdout — \`--json\` or piped):
|
|
1142
|
+
|
|
1143
|
+
\`\`\`json
|
|
1144
|
+
{
|
|
1145
|
+
"error": "Free plan allows 3 studies per workspace. Upgrade to add more.",
|
|
1146
|
+
"error_code": "usage_limit_reached",
|
|
1147
|
+
"status": 403,
|
|
1148
|
+
"retryable": false,
|
|
1149
|
+
"tier": "free",
|
|
1150
|
+
"limit": "maxStudiesPerProduct",
|
|
1151
|
+
"current": 3,
|
|
1152
|
+
"max": 3,
|
|
1153
|
+
"upgrade_url": "https://app.ishlabs.io/billing",
|
|
1154
|
+
"suggestions": ["Upgrade your plan at https://app.ishlabs.io/billing", "..."]
|
|
1155
|
+
}
|
|
1156
|
+
\`\`\`
|
|
1157
|
+
|
|
1158
|
+
Exit code: \`1\` (general — non-retryable). Don't retry; the user has to
|
|
1159
|
+
upgrade or delete an existing resource to free up headroom.
|
|
1160
|
+
|
|
1161
|
+
## Agent-side handling
|
|
1162
|
+
|
|
1163
|
+
- Branch on \`error_code === "usage_limit_reached"\` (preferred) or
|
|
1164
|
+
\`status === 403\` with that error_code in the body. \`forbidden\`
|
|
1165
|
+
errors that are *not* tier-related keep \`error_code: "forbidden"\`.
|
|
1166
|
+
- Use \`limit\`, \`current\`, \`max\`, \`tier\` to construct your own
|
|
1167
|
+
recovery message. The \`limit\` value matches the table above and is
|
|
1168
|
+
stable.
|
|
1169
|
+
- The \`generate\` endpoints (\`study generate\`, \`profile generate\`)
|
|
1170
|
+
refuse the entire batch when the post-generation count would exceed
|
|
1171
|
+
the cap, rather than partially fulfilling — re-issue with a smaller
|
|
1172
|
+
\`--count\` after upgrading or pruning.
|
|
1173
|
+
|
|
1174
|
+
## Related
|
|
1175
|
+
|
|
1176
|
+
- \`concepts/workspace\` — \`maxProducts\` is per-account.
|
|
1177
|
+
- \`concepts/study\` — \`maxStudiesPerProduct\` gates study creation.
|
|
1178
|
+
- \`concepts/iteration\` — \`maxIterationsPerStudy\` gates iteration creation.
|
|
1179
|
+
- \`concepts/profile\` — \`maxCustomTesterProfiles\` gates profile creation.
|
|
1180
|
+
- \`reference/json-mode\` — full error envelope shape and exit codes.
|
|
1181
|
+
`;
|
|
777
1182
|
const PAGES = [
|
|
778
1183
|
{
|
|
779
1184
|
slug: "overview",
|
|
@@ -853,6 +1258,12 @@ const PAGES = [
|
|
|
853
1258
|
description: "Side-by-side; decision rule for choosing one over the other.",
|
|
854
1259
|
body: CONCEPT_RUN_VERBS,
|
|
855
1260
|
},
|
|
1261
|
+
{
|
|
1262
|
+
slug: "concepts/active-context",
|
|
1263
|
+
title: "concept: active context",
|
|
1264
|
+
description: "Saved workspace/study/ask state and how to inspect it (ish status).",
|
|
1265
|
+
body: CONCEPT_ACTIVE_CONTEXT,
|
|
1266
|
+
},
|
|
856
1267
|
{
|
|
857
1268
|
slug: "reference/aliases",
|
|
858
1269
|
title: "reference: aliases",
|
|
@@ -865,6 +1276,12 @@ const PAGES = [
|
|
|
865
1276
|
description: "JSON, --fields, --verbose, exit codes, pipe behaviour.",
|
|
866
1277
|
body: REFERENCE_JSON_MODE,
|
|
867
1278
|
},
|
|
1279
|
+
{
|
|
1280
|
+
slug: "reference/billing-limits",
|
|
1281
|
+
title: "reference: billing tier limits",
|
|
1282
|
+
description: "Per-tier caps on workspaces/studies/iterations/profiles; usage_limit_reached error shape.",
|
|
1283
|
+
body: REFERENCE_BILLING_LIMITS,
|
|
1284
|
+
},
|
|
868
1285
|
{
|
|
869
1286
|
slug: "guides/first-study",
|
|
870
1287
|
title: "guide: your first study, end to end",
|