@lookover-ai/cli 0.0.0-stage → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +159 -2
- package/dist/webagent.cjs +38031 -0
- package/package.json +47 -4
package/README.md
CHANGED
|
@@ -1,3 +1,160 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @lookover-ai/cli
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Command `lookover` (also installed as `webagent`, the old name). Two commands: `lookover test` (below) and `lookover vault add` (save a site login so your AI agent can sign in; see
|
|
4
|
+
"Save a login" further down).
|
|
5
|
+
|
|
6
|
+
`lookover test` runs your `.webagent/tests.yaml` against a preview URL and sets the exit code, so a CI job can
|
|
7
|
+
gate a deploy. The file format is `readTestsFile` in `@lookover-ai/webagent-contracts`
|
|
8
|
+
(`src/public/tests-file.ts`). In the API a "workflow" is a saved set of cases; a run of it gives one verdict.
|
|
9
|
+
|
|
10
|
+
```sh
|
|
11
|
+
export WEBAGENT_API_KEY=wa_... # a CI secret (API keys page)
|
|
12
|
+
lookover test --url "$PREVIEW_URL" --wait-for-url --junit out/junit.xml --markdown out/report.md
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
| Option | |
|
|
16
|
+
|---|---|
|
|
17
|
+
| `--file <path>` | default `.webagent/tests.yaml` |
|
|
18
|
+
| `--url <url>` | the site to test; wins over the file's target |
|
|
19
|
+
| `--target <name>` | a target of the file (default: its `defaultTarget`) |
|
|
20
|
+
| `--tags a,b` / `--case-ids a,b` | run a subset |
|
|
21
|
+
| `--workflow <key>` | the workflow key (default: `key` in the file); with no file, runs the saved workflow with that key |
|
|
22
|
+
| `--sync` | upsert the workflow by key first, then run it (use on **default-branch pushes**). Without it the run is ad hoc: nothing is saved, so a PR's edited file never redefines the workflow |
|
|
23
|
+
| `--retries 0..2` | extra tries for a failed case (the API defaults to 1 for CI runs) |
|
|
24
|
+
| `--timeout-minutes 15` | on timeout or Ctrl-C the run is cancelled |
|
|
25
|
+
| `--wait-for-url`, `--wait-for-url-timeout <s>` | poll the URL until it answers 2xx/3xx (max 180 s, default 120) before starting |
|
|
26
|
+
| `--on-error block\|neutral` | "could not test" exits 2 (default) or 0 |
|
|
27
|
+
| `--junit <path>`, `--markdown <path>`, `--json` | reports; `--json` prints the run on stdout, messages go to stderr |
|
|
28
|
+
| `--ci-context <file.json>` | pull request context `{"title","body","changedFiles":[...]}`; wins over the variables below |
|
|
29
|
+
| `--stop-after <n>` | stop early once n required cases failed; the rest show "Not run (stopped early)" (off by default; 3 suits PRs) |
|
|
30
|
+
| `--failed-only` | only the cases that did not pass in the last finished run (same PR in CI); passes are carried over, not re-run, not billed |
|
|
31
|
+
| `--rerun-of <wfr_...>` | re-run that earlier run; with `--failed-only`, only what did not pass in it |
|
|
32
|
+
| `--environment <name>` | record the run against this environment (for example `Production`); sent as `environment` on the run |
|
|
33
|
+
| `--ci-pr <n>` | pull request number of the run; wins over the detected one (a `deployment_status` job names no PR) |
|
|
34
|
+
|
|
35
|
+
Environment: `WEBAGENT_API_KEY` (required), `WEBAGENT_API_URL` (default: the staging host in
|
|
36
|
+
`src/constants.ts`, one constant, until production exists). Inside GitHub Actions the run carries
|
|
37
|
+
`trigger: ci` and `ci {provider, repo, ref, sha, pr, buildUrl}` from `GITHUB_*` and the event file, automatically
|
|
38
|
+
(`GITHUB_ACTIONS=true`): `sha` is the PR head on a pull request (not the merge commit), `ref` is `GITHUB_HEAD_REF` or
|
|
39
|
+
`GITHUB_REF_NAME`, `buildUrl` the Actions run page; a missing or unreadable event file only leaves `pr` out. On GitLab CI
|
|
40
|
+
(`GITLAB_CI`: `CI_COMMIT_SHA`, `CI_COMMIT_REF_NAME` or the merge request source branch, `CI_MERGE_REQUEST_IID`,
|
|
41
|
+
`CI_MERGE_REQUEST_TITLE`, `CI_PROJECT_PATH`, `CI_PIPELINE_URL`) and Jenkins (`JENKINS_URL`: `GIT_COMMIT`, `GIT_BRANCH`,
|
|
42
|
+
`CHANGE_ID`, `CHANGE_TITLE`, `BUILD_URL`) the same fields are filled with `provider: "other"` (the schema has no
|
|
43
|
+
gitlab or jenkins provider). `--sync` still needs GitHub (it sets the repo marker from `GITHUB_*`).
|
|
44
|
+
|
|
45
|
+
Pull request context for failure triage: `WEBAGENT_CI_TITLE`, `WEBAGENT_CI_BODY`, `WEBAGENT_CI_CHANGED_FILES` (one path
|
|
46
|
+
per line) or `--ci-context` (a file is the better door for a long body: the OS caps the environment). They become
|
|
47
|
+
`ci.title/body/changedFiles`, cut to the schema limits (200 / 2000 / 300 paths of 300). The text is untrusted: the CLI
|
|
48
|
+
only forwards it, and it is never printed in the log, the table, the JUnit file or the report text.
|
|
49
|
+
|
|
50
|
+
**Report words:** `Lookover QA · Failed · 7 passed · 1 failed` (Passed, Failed, Couldn't test), then one
|
|
51
|
+
`Failed: <title> · Expected … · Saw …` line per failed case and a `Took … · Cost: $…` line when the run has them.
|
|
52
|
+
**JUnit:** `classname` is the workflow key (a slug), `name` the case title, and each testcase has
|
|
53
|
+
`<property name="case_id" value="<caseId>"/>`, so a rename is not a new test in TestRail, Xray or Qase.
|
|
54
|
+
|
|
55
|
+
`--sync` sets
|
|
56
|
+
`managedBy {kind: "repo", repo, sha, path}`, always (a PUT without it clears the marker), so `--sync` needs `GITHUB_REPOSITORY` and exits 3 without it. The Markdown report is the api's `report.md` written as it comes (the final PR-comment layout), never re-rendered.
|
|
57
|
+
|
|
58
|
+
**Exit codes:** `0` pass (flaky passes and failed optional cases included) · `1` a required case failed ·
|
|
59
|
+
`2` could not test (platform error, timeout, preview never answered; not an app failure) · `3` setup problem
|
|
60
|
+
(bad file or flags, missing or rejected key, API unreachable). A busy platform at start is retried twice first.
|
|
61
|
+
|
|
62
|
+
**Steps and expectations are opaque (0.3.0).** The CLI reads only what it needs itself from a case: `id` (or one made from
|
|
63
|
+
`title`), `title`, `path`, `tags`, `priority`, `required`, `signedIn`, and the file shape (targets, duplicate ids). A case's
|
|
64
|
+
`steps`, `expect`, `checks` and every other key (new step kinds, expectation lists, assertions, vars, `use`/`before`/`after`,
|
|
65
|
+
`lane`...) are sent to the API unchanged, and the **API validates them**. When it refuses a case, the CLI prints the API's own
|
|
66
|
+
sentence (with the failing paths when it sends them) and exits `3`; it never retries a refusal. So an older CLI keeps working
|
|
67
|
+
when the test format grows. The same holds for top-level keys of the file the CLI does not know (a future `vars`, `before`,
|
|
68
|
+
`after`, shared steps): they go to the API unchanged, in the ad-hoc run and in the `--sync` upsert (never `trigger`, `ci`,
|
|
69
|
+
`managedBy`, `workflowKey`, `baselineRunId`, `engineId`, `session`: those are the CLI's).
|
|
70
|
+
|
|
71
|
+
**Replies are read tolerantly (0.3.0).** The CLI checks only the fields it uses (run id, verdict, per-case verdict, counts,
|
|
72
|
+
failure text); a new field, step kind or count in a reply is ignored, and `--json` prints the reply as the API sent it. A
|
|
73
|
+
verdict the CLI does not know, on the run or on a required case, is "could not test" (exit `2`, or `0` with
|
|
74
|
+
`--on-error neutral`) with a sentence that names the value: never a pass. A 2xx reply with no run id is the same.
|
|
75
|
+
|
|
76
|
+
## Save a login: `lookover vault add` (0.4.0)
|
|
77
|
+
|
|
78
|
+
Your AI agent (through the MCP tool `add_login`) gets you a single-use code. You type the password in your own terminal, so
|
|
79
|
+
it never passes through the chat or the model. The code can create exactly one login for one site; it cannot read or list
|
|
80
|
+
anything, and it expires after 10 minutes.
|
|
81
|
+
|
|
82
|
+
```sh
|
|
83
|
+
npx @lookover-ai/cli vault add https://app.acme.com --code ABCD-EFGH
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
It first shows what the code will create ("Adding a login for https://app.acme.com in Acme / Web app", the username if the
|
|
87
|
+
code carries one, the expiry) and refuses if the site you typed is not the code's site. Then, in a normal terminal window, it
|
|
88
|
+
asks for `Username:` (only if the code has none and `--username` is not given) and `Password:` with **hidden input**. It saves
|
|
89
|
+
the login and prints `Saved login "<label>" for <origin>. Your AI agent can use it now.`
|
|
90
|
+
|
|
91
|
+
| Option | |
|
|
92
|
+
|---|---|
|
|
93
|
+
| `<origin>` | the site, as the agent showed it (case, trailing slash and default port do not matter) |
|
|
94
|
+
| `--code <ABCD-EFGH>` | the code from the agent (required) |
|
|
95
|
+
| `--username <name>` | the login name; the code or a prompt supplies it otherwise |
|
|
96
|
+
| `--password-stdin` | read the password (first line of stdin) instead of asking; never prompts, so the username must come from `--username` or the code |
|
|
97
|
+
| `--totp` | also ask for the authenticator (TOTP) seed, hidden, empty for none (with `--password-stdin`: second line of stdin) |
|
|
98
|
+
| `--api <base url>` | API to talk to; wins over `WEBAGENT_API_URL`, which wins over the built-in host. `https://` only (plain `http://` just for localhost) |
|
|
99
|
+
|
|
100
|
+
There is **no `--password` flag** on purpose: a password in the command line lands in your shell history and the process
|
|
101
|
+
list. The CLI never prints the password or the TOTP seed. No API key is needed (the code is the only auth).
|
|
102
|
+
|
|
103
|
+
**Commands started with `!` in an AI chat tool (Claude Code and similar) have no terminal**, so there is nothing to type a
|
|
104
|
+
hidden password into. The CLI then exits `2` and says so. Run it in a normal terminal window, or put the password on the
|
|
105
|
+
clipboard and pipe it in (the password is not echoed anywhere):
|
|
106
|
+
|
|
107
|
+
```sh
|
|
108
|
+
pbpaste | npx @lookover-ai/cli vault add https://app.acme.com --code ABCD-EFGH --password-stdin
|
|
109
|
+
# Linux: xclip -o | npx @lookover-ai/cli vault add ...
|
|
110
|
+
# Windows: Get-Clipboard | npx @lookover-ai/cli vault add ...
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
**Exit codes of `vault add`:** `0` saved · `1` refused or failed (code expired or already used, rate limit, wrong site,
|
|
114
|
+
API unreachable) · `2` no terminal and no `--password-stdin` · `3` bad flags · `130` Ctrl-C at a prompt (nothing saved).
|
|
115
|
+
It calls `POST /v1/credentials/add-codes/peek` and `.../claim` (contracts `src/public/credential-add-codes.ts`).
|
|
116
|
+
|
|
117
|
+
## Job summary and CI recipes
|
|
118
|
+
|
|
119
|
+
`--markdown <file>` writes the report as Markdown (the PR-comment layout), also when the run could not be tested, so it is the
|
|
120
|
+
CI job summary: on GitHub Actions, `cat report.md >> "$GITHUB_STEP_SUMMARY"`. `--junit <file>` feeds the CI's test report.
|
|
121
|
+
|
|
122
|
+
**GitLab CI** (`.gitlab-ci.yml`). Store the key as a masked CI/CD variable named `WEBAGENT_API_KEY` (Settings, CI/CD,
|
|
123
|
+
Variables). The job installs the CLI from npm, runs your `.webagent/tests.yaml` against the preview of the merge request and
|
|
124
|
+
publishes the JUnit file (the Tests tab) and the Markdown report (job artifact). Exit `1` fails the job; exit `2` (could not
|
|
125
|
+
test: not an app failure) only warns. `--sync` is for GitHub (it needs `GITHUB_REPOSITORY`), so GitLab runs stay ad hoc.
|
|
126
|
+
|
|
127
|
+
```yaml
|
|
128
|
+
webagent-qa:
|
|
129
|
+
stage: test
|
|
130
|
+
image: node:22
|
|
131
|
+
rules:
|
|
132
|
+
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
|
|
133
|
+
variables:
|
|
134
|
+
PREVIEW_URL: "https://mr-$CI_MERGE_REQUEST_IID.preview.example.com" # your preview host
|
|
135
|
+
script:
|
|
136
|
+
- npm install --global @lookover-ai/cli
|
|
137
|
+
- lookover test --url "$PREVIEW_URL" --wait-for-url --junit reports/junit.xml --markdown reports/report.md
|
|
138
|
+
after_script:
|
|
139
|
+
- cat reports/report.md || true
|
|
140
|
+
allow_failure:
|
|
141
|
+
exit_codes: 2
|
|
142
|
+
artifacts:
|
|
143
|
+
when: always
|
|
144
|
+
paths:
|
|
145
|
+
- reports/report.md
|
|
146
|
+
reports:
|
|
147
|
+
junit: reports/junit.xml
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Optional: `WEBAGENT_API_URL` (or `--api` for `vault add`) points the CLI at another API host. Pin a version (`@lookover-ai/cli@0.3`) once you
|
|
151
|
+
want repeatable runs. A test in `cli/src/readme-recipe.test.ts` checks that every flag in this recipe exists.
|
|
152
|
+
|
|
153
|
+
`pnpm build` bundles everything into one file, `dist/webagent.cjs` (no `node_modules` needed; Node 22), which the
|
|
154
|
+
GitHub Action vendors. It contains contracts, the SDK, zod and yaml, so the npm package has no dependencies (its tarball is
|
|
155
|
+
`dist/webagent.cjs`, this README and `package.json`).
|
|
156
|
+
|
|
157
|
+
**Publishing** (public npm, `@lookover-ai` scope; by hand, with an npm token that may publish to it, exported as
|
|
158
|
+
`NODE_AUTH_TOKEN`): `pnpm -C cli check && cd cli && pnpm publish --no-git-checks` (`publishConfig` sets the registry
|
|
159
|
+
`https://registry.npmjs.org` and `access: public`; `npm pack --dry-run` first must list exactly those three files).
|
|
160
|
+
The GitLab recipe works once 0.3.0 is published there.
|