@lookover-ai/cli 0.0.0-stage → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/README.md +159 -2
  2. package/dist/webagent.cjs +38031 -0
  3. package/package.json +47 -4
package/README.md CHANGED
@@ -1,3 +1,160 @@
1
- # Temporary Holding Version
1
+ # @lookover-ai/cli
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Command `lookover` (also installed as `webagent`, the old name). Two commands: `lookover test` (below) and `lookover vault add` (save a site login so your AI agent can sign in; see
4
+ "Save a login" further down).
5
+
6
+ `lookover test` runs your `.webagent/tests.yaml` against a preview URL and sets the exit code, so a CI job can
7
+ gate a deploy. The file format is `readTestsFile` in `@lookover-ai/webagent-contracts`
8
+ (`src/public/tests-file.ts`). In the API a "workflow" is a saved set of cases; a run of it gives one verdict.
9
+
10
+ ```sh
11
+ export WEBAGENT_API_KEY=wa_... # a CI secret (API keys page)
12
+ lookover test --url "$PREVIEW_URL" --wait-for-url --junit out/junit.xml --markdown out/report.md
13
+ ```
14
+
15
+ | Option | |
16
+ |---|---|
17
+ | `--file <path>` | default `.webagent/tests.yaml` |
18
+ | `--url <url>` | the site to test; wins over the file's target |
19
+ | `--target <name>` | a target of the file (default: its `defaultTarget`) |
20
+ | `--tags a,b` / `--case-ids a,b` | run a subset |
21
+ | `--workflow <key>` | the workflow key (default: `key` in the file); with no file, runs the saved workflow with that key |
22
+ | `--sync` | upsert the workflow by key first, then run it (use on **default-branch pushes**). Without it the run is ad hoc: nothing is saved, so a PR's edited file never redefines the workflow |
23
+ | `--retries 0..2` | extra tries for a failed case (the API defaults to 1 for CI runs) |
24
+ | `--timeout-minutes 15` | on timeout or Ctrl-C the run is cancelled |
25
+ | `--wait-for-url`, `--wait-for-url-timeout <s>` | poll the URL until it answers 2xx/3xx (max 180 s, default 120) before starting |
26
+ | `--on-error block\|neutral` | "could not test" exits 2 (default) or 0 |
27
+ | `--junit <path>`, `--markdown <path>`, `--json` | reports; `--json` prints the run on stdout, messages go to stderr |
28
+ | `--ci-context <file.json>` | pull request context `{"title","body","changedFiles":[...]}`; wins over the variables below |
29
+ | `--stop-after <n>` | stop early once n required cases failed; the rest show "Not run (stopped early)" (off by default; 3 suits PRs) |
30
+ | `--failed-only` | only the cases that did not pass in the last finished run (same PR in CI); passes are carried over, not re-run, not billed |
31
+ | `--rerun-of <wfr_...>` | re-run that earlier run; with `--failed-only`, only what did not pass in it |
32
+ | `--environment <name>` | record the run against this environment (for example `Production`); sent as `environment` on the run |
33
+ | `--ci-pr <n>` | pull request number of the run; wins over the detected one (a `deployment_status` job names no PR) |
34
+
35
+ Environment: `WEBAGENT_API_KEY` (required), `WEBAGENT_API_URL` (default: the staging host in
36
+ `src/constants.ts`, one constant, until production exists). Inside GitHub Actions the run carries
37
+ `trigger: ci` and `ci {provider, repo, ref, sha, pr, buildUrl}` from `GITHUB_*` and the event file, automatically
38
+ (`GITHUB_ACTIONS=true`): `sha` is the PR head on a pull request (not the merge commit), `ref` is `GITHUB_HEAD_REF` or
39
+ `GITHUB_REF_NAME`, `buildUrl` the Actions run page; a missing or unreadable event file only leaves `pr` out. On GitLab CI
40
+ (`GITLAB_CI`: `CI_COMMIT_SHA`, `CI_COMMIT_REF_NAME` or the merge request source branch, `CI_MERGE_REQUEST_IID`,
41
+ `CI_MERGE_REQUEST_TITLE`, `CI_PROJECT_PATH`, `CI_PIPELINE_URL`) and Jenkins (`JENKINS_URL`: `GIT_COMMIT`, `GIT_BRANCH`,
42
+ `CHANGE_ID`, `CHANGE_TITLE`, `BUILD_URL`) the same fields are filled with `provider: "other"` (the schema has no
43
+ gitlab or jenkins provider). `--sync` still needs GitHub (it sets the repo marker from `GITHUB_*`).
44
+
45
+ Pull request context for failure triage: `WEBAGENT_CI_TITLE`, `WEBAGENT_CI_BODY`, `WEBAGENT_CI_CHANGED_FILES` (one path
46
+ per line) or `--ci-context` (a file is the better door for a long body: the OS caps the environment). They become
47
+ `ci.title/body/changedFiles`, cut to the schema limits (200 / 2000 / 300 paths of 300). The text is untrusted: the CLI
48
+ only forwards it, and it is never printed in the log, the table, the JUnit file or the report text.
49
+
50
+ **Report words:** `Lookover QA · Failed · 7 passed · 1 failed` (Passed, Failed, Couldn't test), then one
51
+ `Failed: <title> · Expected … · Saw …` line per failed case and a `Took … · Cost: $…` line when the run has them.
52
+ **JUnit:** `classname` is the workflow key (a slug), `name` the case title, and each testcase has
53
+ `<property name="case_id" value="<caseId>"/>`, so a rename is not a new test in TestRail, Xray or Qase.
54
+
55
+ `--sync` sets
56
+ `managedBy {kind: "repo", repo, sha, path}`, always (a PUT without it clears the marker), so `--sync` needs `GITHUB_REPOSITORY` and exits 3 without it. The Markdown report is the api's `report.md` written as it comes (the final PR-comment layout), never re-rendered.
57
+
58
+ **Exit codes:** `0` pass (flaky passes and failed optional cases included) · `1` a required case failed ·
59
+ `2` could not test (platform error, timeout, preview never answered; not an app failure) · `3` setup problem
60
+ (bad file or flags, missing or rejected key, API unreachable). A busy platform at start is retried twice first.
61
+
62
+ **Steps and expectations are opaque (0.3.0).** The CLI reads only what it needs itself from a case: `id` (or one made from
63
+ `title`), `title`, `path`, `tags`, `priority`, `required`, `signedIn`, and the file shape (targets, duplicate ids). A case's
64
+ `steps`, `expect`, `checks` and every other key (new step kinds, expectation lists, assertions, vars, `use`/`before`/`after`,
65
+ `lane`...) are sent to the API unchanged, and the **API validates them**. When it refuses a case, the CLI prints the API's own
66
+ sentence (with the failing paths when it sends them) and exits `3`; it never retries a refusal. So an older CLI keeps working
67
+ when the test format grows. The same holds for top-level keys of the file the CLI does not know (a future `vars`, `before`,
68
+ `after`, shared steps): they go to the API unchanged, in the ad-hoc run and in the `--sync` upsert (never `trigger`, `ci`,
69
+ `managedBy`, `workflowKey`, `baselineRunId`, `engineId`, `session`: those are the CLI's).
70
+
71
+ **Replies are read tolerantly (0.3.0).** The CLI checks only the fields it uses (run id, verdict, per-case verdict, counts,
72
+ failure text); a new field, step kind or count in a reply is ignored, and `--json` prints the reply as the API sent it. A
73
+ verdict the CLI does not know, on the run or on a required case, is "could not test" (exit `2`, or `0` with
74
+ `--on-error neutral`) with a sentence that names the value: never a pass. A 2xx reply with no run id is the same.
75
+
76
+ ## Save a login: `lookover vault add` (0.4.0)
77
+
78
+ Your AI agent (through the MCP tool `add_login`) gets you a single-use code. You type the password in your own terminal, so
79
+ it never passes through the chat or the model. The code can create exactly one login for one site; it cannot read or list
80
+ anything, and it expires after 10 minutes.
81
+
82
+ ```sh
83
+ npx @lookover-ai/cli vault add https://app.acme.com --code ABCD-EFGH
84
+ ```
85
+
86
+ It first shows what the code will create ("Adding a login for https://app.acme.com in Acme / Web app", the username if the
87
+ code carries one, the expiry) and refuses if the site you typed is not the code's site. Then, in a normal terminal window, it
88
+ asks for `Username:` (only if the code has none and `--username` is not given) and `Password:` with **hidden input**. It saves
89
+ the login and prints `Saved login "<label>" for <origin>. Your AI agent can use it now.`
90
+
91
+ | Option | |
92
+ |---|---|
93
+ | `<origin>` | the site, as the agent showed it (case, trailing slash and default port do not matter) |
94
+ | `--code <ABCD-EFGH>` | the code from the agent (required) |
95
+ | `--username <name>` | the login name; the code or a prompt supplies it otherwise |
96
+ | `--password-stdin` | read the password (first line of stdin) instead of asking; never prompts, so the username must come from `--username` or the code |
97
+ | `--totp` | also ask for the authenticator (TOTP) seed, hidden, empty for none (with `--password-stdin`: second line of stdin) |
98
+ | `--api <base url>` | API to talk to; wins over `WEBAGENT_API_URL`, which wins over the built-in host. `https://` only (plain `http://` just for localhost) |
99
+
100
+ There is **no `--password` flag** on purpose: a password in the command line lands in your shell history and the process
101
+ list. The CLI never prints the password or the TOTP seed. No API key is needed (the code is the only auth).
102
+
103
+ **Commands started with `!` in an AI chat tool (Claude Code and similar) have no terminal**, so there is nothing to type a
104
+ hidden password into. The CLI then exits `2` and says so. Run it in a normal terminal window, or put the password on the
105
+ clipboard and pipe it in (the password is not echoed anywhere):
106
+
107
+ ```sh
108
+ pbpaste | npx @lookover-ai/cli vault add https://app.acme.com --code ABCD-EFGH --password-stdin
109
+ # Linux: xclip -o | npx @lookover-ai/cli vault add ...
110
+ # Windows: Get-Clipboard | npx @lookover-ai/cli vault add ...
111
+ ```
112
+
113
+ **Exit codes of `vault add`:** `0` saved · `1` refused or failed (code expired or already used, rate limit, wrong site,
114
+ API unreachable) · `2` no terminal and no `--password-stdin` · `3` bad flags · `130` Ctrl-C at a prompt (nothing saved).
115
+ It calls `POST /v1/credentials/add-codes/peek` and `.../claim` (contracts `src/public/credential-add-codes.ts`).
116
+
117
+ ## Job summary and CI recipes
118
+
119
+ `--markdown <file>` writes the report as Markdown (the PR-comment layout), also when the run could not be tested, so it is the
120
+ CI job summary: on GitHub Actions, `cat report.md >> "$GITHUB_STEP_SUMMARY"`. `--junit <file>` feeds the CI's test report.
121
+
122
+ **GitLab CI** (`.gitlab-ci.yml`). Store the key as a masked CI/CD variable named `WEBAGENT_API_KEY` (Settings, CI/CD,
123
+ Variables). The job installs the CLI from npm, runs your `.webagent/tests.yaml` against the preview of the merge request and
124
+ publishes the JUnit file (the Tests tab) and the Markdown report (job artifact). Exit `1` fails the job; exit `2` (could not
125
+ test: not an app failure) only warns. `--sync` is for GitHub (it needs `GITHUB_REPOSITORY`), so GitLab runs stay ad hoc.
126
+
127
+ ```yaml
128
+ webagent-qa:
129
+ stage: test
130
+ image: node:22
131
+ rules:
132
+ - if: $CI_PIPELINE_SOURCE == "merge_request_event"
133
+ variables:
134
+ PREVIEW_URL: "https://mr-$CI_MERGE_REQUEST_IID.preview.example.com" # your preview host
135
+ script:
136
+ - npm install --global @lookover-ai/cli
137
+ - lookover test --url "$PREVIEW_URL" --wait-for-url --junit reports/junit.xml --markdown reports/report.md
138
+ after_script:
139
+ - cat reports/report.md || true
140
+ allow_failure:
141
+ exit_codes: 2
142
+ artifacts:
143
+ when: always
144
+ paths:
145
+ - reports/report.md
146
+ reports:
147
+ junit: reports/junit.xml
148
+ ```
149
+
150
+ Optional: `WEBAGENT_API_URL` (or `--api` for `vault add`) points the CLI at another API host. Pin a version (`@lookover-ai/cli@0.3`) once you
151
+ want repeatable runs. A test in `cli/src/readme-recipe.test.ts` checks that every flag in this recipe exists.
152
+
153
+ `pnpm build` bundles everything into one file, `dist/webagent.cjs` (no `node_modules` needed; Node 22), which the
154
+ GitHub Action vendors. It contains contracts, the SDK, zod and yaml, so the npm package has no dependencies (its tarball is
155
+ `dist/webagent.cjs`, this README and `package.json`).
156
+
157
+ **Publishing** (public npm, `@lookover-ai` scope; by hand, with an npm token that may publish to it, exported as
158
+ `NODE_AUTH_TOKEN`): `pnpm -C cli check && cd cli && pnpm publish --no-git-checks` (`publishConfig` sets the registry
159
+ `https://registry.npmjs.org` and `access: public`; `npm pack --dry-run` first must list exactly those three files).
160
+ The GitLab recipe works once 0.3.0 is published there.