@cloudbeat/atlas-cli 0.2.0 → 0.4.0

package/dist/docs/agent-tasks.md

@@ -0,0 +1,169 @@
+# What an Agent Can Do
+
+The whole lifecycle, from an empty project to a report you can send someone.
+Each section below is a real thing to ask for, what the agent does about it, and
+what it should ask you along the way.
+
+## Find its bearings
+
+> What can I load test here, and how much capacity do I have?
+
+Before anything else, a well-behaved assistant establishes what it is looking
+at: who it is signed in as, which account is active, which projects it can
+reach, which test plans and data files those hold, what ran recently, and how
+much capacity and usage allowance you have left.
+
+That last part matters more than it sounds. Sizing a test without knowing the
+fleet is how people ask for 50 machines on a 20-machine account, or blow a
+monthly allowance on a warm-up run.
+
+## Record a scenario
+
+> Open our staging site, sign in as the demo user, add two items to the basket
+> and check out. Turn that into a test plan.
+
+The agent opens a **real, visible browser**. It reads each page as an
+accessibility tree — the same structure a screen reader sees — so it clicks
+"Add to basket" because that button exists, not because a selector was guessed.
+It names each step *before* performing it, because those names become the rows
+in your results report, and `Rome` is a useless row title.
+
+You can watch the whole thing and take over whenever you like. For a password,
+an MFA prompt, or a decision only you can make, the agent hands the window back,
+you do that step, and it carries on recording.
+
+Prefer to drive yourself? Say so, and it opens a plain recording session for you
+to click through while it waits.
+
+Export your cookies, or reuse a saved browser profile, and the recording starts
+already signed in — see [Recording a Scenario](/recorder/). Never give an agent
+a password to type.
+
+## Turn the recording into a real test
+
+This is the step people skip, and it is the difference between a load test and
+an expensive replay.
+
+A recording is a draft: it contains exactly what *you* did. The agent reads the
+capture and works through it with you, stage by stage, asking rather than
+deciding:
+
+1. **Is the flow right?** The steps, their names, anything missing.
+2. **Which recorded values should become test data?** The username and password
+   you typed. The search term. The product id that appears in three different
+   requests. If every simulated user sends the same ones, they collide on the
+   same session and hit the same cache — the application looks fast because it
+   is barely working. The agent proposes a CSV, the columns, and how many rows;
+   you say whether to generate them or supply real data. Atlas then divides
+   those rows between machines at run time, so no two users get the same row.
+3. **What changes per session?** Session ids, CSRF tokens, cart ids — values the
+   *server* issued, which must be captured from an earlier response instead of
+   replayed. The recorder detects these with evidence, and the agent pins
+   anything it missed. See [Dynamic values](/recorder/#dynamic-values-are-handled).
+4. **Do you need checks?** A `200` that returns a login page is a pass as far as
+   JMeter is concerned. Assertions on the responses that matter turn that into a
+   visible failure.
+5. **Upload and smoke-run it.**
+
+## Edit an existing plan
+
+> Add a "search by postcode" step to our existing plan, and point it at staging.
+
+The agent downloads the stored plan first — uploading replaces a file and there
+is no version history — then makes the narrowest change that does the job:
+recording a new scenario straight into the plan as an additional thread group,
+or editing specific values in place.
+
+For structural surgery it will tell you plainly that the JMeter GUI is the right
+tool, rather than rewriting a plan nobody has the source for.
+
+## Debug it locally, in seconds
+
+If JMeter is installed on your machine, the agent can run a single user through
+the plan locally before spending any Atlas capacity — the fastest way to confirm
+a new step, an assertion or a data file actually works.
+
+It knows the trick that makes this practical: recorded plans contain **think
+time**, typically ten to thirty seconds between steps, which is correct for a
+real load test and absurd for a debug run. The agent switches it off in a
+throwaway copy, so a check that would have taken two minutes takes five seconds,
+then restores it — your plan keeps its pauses, and nothing gets uploaded with
+the think time missing.
+
+It can also prove an assertion *fails* when it should, which is the only way to
+know a check is doing anything at all.
+
+## Check before you spend
+
+> Will this plan actually run?
+
+`atlas plan check` catches what otherwise surfaces as an empty dashboard an hour
+later:
+
+- **A data file the plan reads that the project does not have.** Every user in
+  that thread group dies before sending a request, so the group disappears from
+  your results with *no errors at all* while the rest look healthy.
+- **A plan Atlas cannot attach its reporting to** — it runs, and reports nothing.
+- **A broken structure, or no enabled thread group.**
+- **Fixed user counts**, meaning more machines is the only way to scale it.
+
+## Size and run it
+
+> Run it with 500 users for ten minutes, ramped up over five.
+
+Sizing is where load tests are most often wrong, in both directions. Each
+machine runs the **whole** plan, so 100 users in the plan across 5 machines is
+500 users, not 100. The agent does that arithmetic against your real fleet,
+explains it, and confirms with you before starting anything larger than a smoke
+test.
+
+It smoke-runs first — one machine, a couple of users — and checks the result
+contains actual data. A smoke run that finishes in seconds with nothing in it is
+a broken plan, not a fast application, and catching that costs one minute
+instead of twenty machine-minutes.
+
+While the run is live it can report progress, and stop the test if you ask.
+
+## Read the results
+
+> How did it go? Where did it start to break?
+
+Numbers are the easy part. The agent reports which steps degraded and at what
+user count, what the errors actually say — with the real request and response,
+not just a code — and how the run behaved over time rather than as one average.
+
+It is also taught to distrust a clean-looking result. A test that measured
+nothing looks *better* than one that worked: a missing thread group with zero
+errors, a run that ended in seconds, or fast `200`s returning login pages. Those
+are reported as failures of the test, not as performance.
+
+## Wire it into CI
+
+> Add a nightly load test to our pipeline and fail the build if p95 goes over
+> 800ms.
+
+The agent writes the pipeline step, using thresholds and exit codes rather than
+a human reading a dashboard:
+
+```sh
+atlas run -p SHOP --jmx checkout.jmx --servers 4 \
+  --gate "error_rate<1,p95<800" --junit results.xml
+```
+
+`0` passed · `1` a threshold was breached · `2` a usage or sign-in problem ·
+`3` the run failed or Atlas was unreachable. JUnit output drops into whatever
+already reports your test results. See [Running in CI](/cli/ci).
+
+## What it should always ask
+
+- Which project, and which plan when there is more than one
+- How many users, for how long, over what ramp-up
+- Fixed machines or a cluster
+- Confirmation before the full run
+- Whether to generate test data or use yours
+
+And what it should never do without you: start a large run, stop someone else's
+test, delete a file it did not just upload, or upload a plan it has not checked.
+
+[The skills behind this →](/agents/skills) ·
+[Build your own integration →](/agents/automation)

package/dist/docs/ai-agents.md

@@ -0,0 +1,90 @@
+# Atlas for AI Agents
+
+**Describe the test. Let the agent build it.**
+
+Load testing has always had a gap between wanting a test and having one. Someone
+has to click through the flow, capture the requests, work out which values are
+per-session tokens, wire up the data files, size the run, and then interpret a
+report full of percentiles. It is a day of work for a thirty-second question:
+*will this hold up on launch day?*
+
+Atlas closes that gap. Every part of it — recording, test plans, data files,
+machines, runs, results — is reachable through one command-line tool that speaks
+JSON, and it ships with the guidance an AI coding assistant needs to use it
+properly. Point your assistant at Atlas and ask in plain language:
+
+> Record a checkout on our staging site, then run it with 500 users for ten
+> minutes and tell me where it breaks.
+
+It records the flow in a real browser, turns the recording into a genuine load
+test, checks the plan will actually report, smoke-runs it, asks you to confirm
+the real run, and comes back with what broke and at what point.
+
+<div class="tip custom-block" style="padding-top: 8px">
+
+New here? [Set it up](/agents/setup) in two commands, then see
+[what it can do](/agents/capabilities).
+
+</div>
+
+## Why this works with Atlas
+
+**One tool, every surface.** The agent does not need infrastructure credentials,
+a database, or your cloud account. Everything goes through the same CLI you use,
+so your permissions, your account's quota and your machine limits apply exactly
+as they do in the browser.
+
+**Machine-readable by design.** Every command takes `--json`. Failures print
+JSON on standard error, so clean output stays clean. `atlas schema --json`
+describes the entire command tree, so an agent can discover the surface instead
+of guessing at flags.
+
+**It knows the traps.** Load testing fails quietly. A data file that was never
+uploaded silently removes an entire thread group — with *zero* errors to explain
+it. A plan whose structure defeats reporting runs perfectly and tells you
+nothing. Fast responses that are really login pages look like excellent
+performance. The bundled guidance teaches an assistant to recognise all of these
+before you act on a number that means nothing.
+
+**The documentation travels with it.** `atlas docs` prints this documentation
+offline, so an agent in a sandbox or a CI container still has the manual.
+
+## What you get
+
+| | |
+|---|---|
+| **Recording without clicking** | The agent opens a real browser, reads the page as an accessibility tree, clicks through your flow and names each step — or hands the window to you for the parts only you can do, like a password or an MFA prompt. |
+| **A real test, not a replay** | It spots the values every simulated user would otherwise repeat — credentials, search terms, product ids — and moves them into a data file, so you are testing your application rather than your cache. |
+| **Plans that actually report** | Before anything runs, `atlas plan check` catches the three failures that produce an empty dashboard an hour later. |
+| **Sizing you can defend** | It reads your real capacity and explains what 500 users means in machines, because each machine runs the whole plan. |
+| **Results in sentences** | Not just p95 numbers: which step degraded, at what user count, and what the errors actually say. |
+| **CI that fails honestly** | Pass/fail thresholds, JUnit output and exit codes, written into your pipeline by the same assistant. |
+
+## Guardrails
+
+An agent that can start a load test can spend money and put load on production.
+The bundled guidance is explicit about this, and a good assistant will:
+
+- **Confirm before any real run.** Machines are powered on for a test and billed
+  to your account. Smoke tests are cheap and encouraged; a 20-machine run is not
+  something to start on an assumption.
+- **Never stop a run it did not start.** Someone else's test is someone else's.
+- **Ask rather than guess** about the things that change the result: which
+  project, how many users, how long, ramp-up, fixed machines or a cluster.
+- **Report what happened, not what should have.** Including failed thresholds, a
+  thread group that vanished, and its own uncertainty.
+- **Leave your files as it found them.** It downloads before it edits, and it
+  restores anything it changed for debugging — think time included.
+
+Everything an agent can do, you can do by hand with the same commands. Nothing
+here is required, and nothing here is hidden from you.
+
+## Which assistants
+
+| Assistant | How it reads the guidance |
+|---|---|
+| **Claude Code** | Skills — loaded automatically when your request is about load testing |
+| **GitHub Copilot, Cursor, Codex, Gemini CLI, Windsurf** | `AGENTS.md` in your project root |
+| **Anything else** | `atlas schema --json`, `atlas docs`, and `--json` on every command |
+
+[Set it up →](/agents/setup)

package/dist/docs/auth.md

@@ -0,0 +1,74 @@
+# Signing In
+
+## For everyday use
+
+```sh
+atlas login --url https://atlas.cloudbeat.io -u you@company.com
+```
+
+You'll be prompted for your password. The session lasts **8 hours**.
+
+Credentials are stored in `~/.atlas/config.json`, readable only by you.
+
+```sh
+atlas whoami     # who am I, and in which account
+atlas logout     # forget the credential
+```
+
+## For CI: API tokens
+
+A password prompt is no use in a pipeline, and storing a human's password is
+worse. Create an **API token** instead.
+
+On your Atlas instance, go to **Profile → API tokens** and create one. The token
+is shown **once** — copy it then.
+
+```sh
+atlas login --token atlas_pat_...
+```
+
+Or, better in CI, pass it as an environment variable and don't store it at all:
+
+```sh
+export ATLAS_TOKEN=atlas_pat_...
+```
+
+### What a token can and can't do
+
+A token acts as **the person who created it**, with the same permissions and in
+one fixed account. If that person's access is reduced or removed, the token
+follows immediately.
+
+Two things it deliberately **cannot** do:
+
+- create more tokens
+- change a password
+
+So a leaked pipeline token can't mint replacements for itself or lock you out.
+Revoking it on the Profile page stops it working at once.
+
+If you belong to several accounts, switch to the one the pipeline should use
+*before* creating the token. The token is fixed to whichever account was active
+when it was made.
+
+A token is a credential. Put it in your CI system's secret store, not in the
+repository, and not in a build log.
+
+## Settings and precedence
+
+Command-line options win over environment variables, which win over the saved
+configuration file.
+
+| Variable | Meaning |
+|---|---|
+| `ATLAS_URL` | your Atlas instance |
+| `ATLAS_TOKEN` | API token |
+| `ATLAS_PROJECT` | project to use when `--project` is omitted |
+| `ATLAS_DEBUG` | set to any value to print full error details |
+
+To avoid repeating yourself:
+
+```sh
+atlas config set url https://atlas.cloudbeat.io
+atlas config set project SHOP
+```

package/dist/docs/automation.md

@@ -0,0 +1,150 @@
+# Building Your Own Integration
+
+The same surfaces the bundled skills use are there for your own scripts, bots
+and agents. No SDK, no separate API client: the CLI is the integration point,
+and everything it does is scriptable.
+
+## JSON everywhere
+
+```sh
+atlas projects --json
+atlas capacity --json
+atlas results SHOP-20260917-101500 --json
+```
+
+Every command takes `--json`. On failure it prints the error as JSON on
+**standard error**, so standard out stays parseable:
+
+```json
+{"error":{"message":"Project 'SHOPP' not found","code":"not_found"}}
+```
+
+Exit codes are the contract, and they do not change with `--json`:
+
+| Code | Meaning |
+|---|---|
+| `0` | Success, and any thresholds passed |
+| `1` | A `--gate` threshold was breached — the test ran, the numbers failed |
+| `2` | A usage or authentication problem: bad flags, unknown project, not signed in |
+| `3` | The run failed or timed out, or Atlas could not be reached |
+
+`1` and `3` are deliberately distinct: a breached threshold is a result, an
+unreachable server is not.
+
+## Discovering the surface
+
+Rather than hard-coding flags, read them:
+
+```sh
+atlas schema --json
+```
+
+This returns every command, argument and option, with its description. Two
+fields worth knowing: `takesValue` says whether the flag expects a value, and
+`valueOptional` whether that value may be omitted. Neither means the flag itself
+is mandatory.
+
+## Documentation, offline
+
+```sh
+atlas docs                  # list the topics
+atlas docs troubleshooting  # print one
+```
+
+The documentation is bundled inside the package, so it works with no network —
+in CI, in a container, in a sandboxed agent.
+
+## Checking a plan without running it
+
+```sh
+atlas plan check ./checkout.jmx -p SHOP --json
+```
+
+Read-only and advisory: it never rewrites your plan. Useful as a pre-commit
+hook or a pipeline step, because it exits non-zero when the plan would produce
+no usable results.
+
+## Driving the recorder from code
+
+The recorder can be controlled step by step, which is what lets an agent click
+through a flow. It is also a perfectly good scripting interface:
+
+```sh
+atlas-rec start https://shop.example.com -o flow.jmx -n "Checkout"
+atlas-rec snapshot                              # the page, as an accessibility tree
+atlas-rec tx new "Search"                       # name the step that follows
+atlas-rec click --role button --name "Search"
+atlas-rec fill "laptop" --selector "#q"
+atlas-rec wait --selector ".results"
+atlas-rec state --full                          # everything captured so far
+atlas-rec stop                                  # writes the plan and the capture
+```
+
+The window stays visible and a person can click in the same session — the two
+mix freely, so a script can pause for a human to sign in and then carry on.
+
+The control channel is **local to your machine**: it listens on the loopback
+interface, on an ephemeral port, and requires a per-session token that is kept
+out of the process list. It only accepts commands while a recording you started
+is open.
+
+Afterwards, a capture can be re-read and rebuilt without recording again:
+
+```sh
+atlas-rec inspect flow.capture.json    # transactions, hosts, response codes
+atlas-rec suggest flow.capture.json    # per-session values, with evidence
+atlas-rec generate flow.capture.json -o flow.jmx --correlations rules.json
+```
+
+The same capture always produces the same plan, so regenerating is safe to
+repeat and easy to review.
+
+## Authentication for automation
+
+Use an API token, never a password:
+
+```sh
+export ATLAS_TOKEN=atlas_pat_...
+export ATLAS_URL=https://atlas.cloudbeat.io
+export ATLAS_PROJECT=SHOP
+atlas run --servers 4 --gate "error_rate<1,p95<800" --junit results.xml
+```
+
+A token acts as its creating user, in one account. Revoking it takes effect
+immediately, and a token cannot mint further tokens or change a password — so a
+leaked CI credential cannot entrench itself. See
+[Signing In](/cli/authentication).
+
+## A worked example
+
+Everything together — gate a deployment on a load test:
+
+```sh
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Refuse to run a plan that could not report
+atlas plan check -p SHOP --jmx checkout.jmx --json >/dev/null
+
+# `|| rc=$?` keeps `set -e` from aborting on a breached threshold, which is a
+# result we want to act on rather than a crash
+rc=0
+atlas run -p SHOP --jmx checkout.jmx --servers 4 \
+  --args "-Jthreads=125 -Jrampup=300" \
+  --gate "error_rate<1,p95<800" \
+  --junit results.xml --json > run.json || rc=$?
+
+case $rc in
+  0) echo "passed: $(jq -r .testId run.json)" ;;
+  1) echo "thresholds breached"; jq '.gates[] | select(.passed | not)' run.json; exit 1 ;;
+  *) echo "the test did not run (exit $rc)"; exit "$rc" ;;
+esac
+```
+
+With `--wait` — which `--gate` and `--junit` imply — the JSON on standard out
+carries `testId`, `state`, `durationSec`, the overall figures and every gate
+result. It is written **before** a breached threshold sets the exit code, so
+`run.json` is complete either way.
+
+[Running in CI](/cli/ci) has the pipeline-ready version, including GitHub
+Actions.

package/dist/docs/ci.md

@@ -0,0 +1,105 @@
+# Running in CI
+
+The point of running Atlas from CI is to catch a performance regression the same
+way you catch a failing unit test — automatically, on the build that caused it.
+
+## A complete example
+
+```sh
+export ATLAS_URL=https://atlas.cloudbeat.io
+export ATLAS_TOKEN=atlas_pat_...
+
+# Only if the plan or its data live in this repository and may have changed.
+atlas files upload -p SHOP ./checkout.jmx ./data/users.csv
+
+atlas run -p SHOP --jmx checkout.jmx --servers 5 \
+          --gate "error_rate<1,p95<2000" \
+          --junit results.xml
+```
+
+If your test plan is maintained in Atlas rather than in the repository, drop the
+upload step — the run command works on its own.
+
+## Pass/fail thresholds
+
+`--gate` fails the command when a threshold is breached, checked against the
+run's overall totals.
+
+| Metric | Meaning |
+|---|---|
+| `error_rate` | percentage of failed requests |
+| `avg` | average response time, ms |
+| `p90`, `p95`, `p99` | percentile response times, ms |
+| `max` | slowest response, ms |
+| `throughput` | requests per second |
+| `requests` | total requests |
+
+Use `<`, `<=`, `>` or `>=`. Separate several with commas, or repeat `--gate`.
+
+```sh
+--gate "error_rate<1,p95<2000,throughput>50"
+```
+
+Setting a gate implies `--wait` — there is nothing to check until the test has
+finished.
+
+`error_rate` and a percentile. Averages hide the tail: a p95 catches the slow
+requests a handful of users actually felt, which an average smooths away.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | the test ran and every threshold passed |
+| `1` | the test ran but a `--gate` threshold was breached |
+| `2` | something was wrong with the request — bad option, unknown project, not signed in |
+| `3` | the test failed or timed out, or the server could not be reached |
+
+The distinction that matters: **`1` means your system was too slow, `2` and `3`
+mean the test never produced a verdict.** Treating them the same makes a broken
+pipeline look like a performance regression.
+
+## JUnit report
+
+```sh
+atlas run -p SHOP --servers 5 --junit results.xml
+```
+
+Writes a standard JUnit file with one test case per threshold, so CI shows each
+one as a separate pass or fail. The run's headline numbers are included as
+properties. Point your CI's test reporter at it and gates appear alongside your
+other tests.
+
+## Example: GitHub Actions
+
+```yaml
+- name: Load test
+  env:
+    ATLAS_URL: https://atlas.cloudbeat.io
+    ATLAS_TOKEN: ${{ secrets.ATLAS_TOKEN }}
+  run: |
+    npm install -g @cloudbeat/atlas-cli
+    atlas run -p SHOP --servers 5 \
+              --gate "error_rate<1,p95<2000" \
+              --junit results.xml
+
+- name: Publish results
+  if: always()
+  uses: actions/upload-artifact@v4
+  with:
+    name: load-test
+    path: results.xml
+```
+
+## Things worth knowing
+
+**Load tests take minutes, not seconds.** Give the job a generous timeout, and
+consider running it on a schedule or on merges to main rather than on every
+push.
+
+**Machines may be busy.** If another test holds the generators you asked for,
+your run is queued and starts when they free up. Set `--timeout` to bound how
+long you're willing to wait.
+
+**A queued or failed run is not a regression.** That's the exit-code distinction
+above — don't let a `3` open a performance bug.

package/dist/docs/clusters.md

@@ -0,0 +1,48 @@
+# Clusters
+
+A **cluster** is elastic capacity: instead of fixed machines that sit there
+between tests, units are created when a test starts and destroyed when it ends.
+
+You pay for what a test uses, and you're not limited by how many machines you
+happened to set up in advance.
+
+## When to use one
+
+Use a cluster when your load varies a lot between tests, or when you want a
+large run occasionally without keeping that capacity around. Use fixed
+[servers](/generators/adding) when you want predictable, always-available
+machines — or when your test needs to come from a specific network location.
+
+## Setting one up
+
+A cluster runs on a Docker host you provide. Atlas talks to it through a small
+**controller** you install on that host, which connects outbound — so the host
+needs no inbound access, exactly like a load generator.
+
+Go to **Fleet → Clusters**, create a cluster, and run the install command it
+gives you on your Docker host. When the controller connects, the cluster shows
+as ready.
+
+## Running on a cluster
+
+On the **Run Test** tab choose **Cluster** instead of Servers, and say how many
+units you want:
+
+```bash
+atlas run -p SHOP --cluster my-cluster --units 10
+```
+
+Each unit is one JMeter instance, equivalent to one load generator. Ten units
+running a 100-thread plan is 1,000 concurrent users.
+
+## While it runs
+
+Units register themselves as they come up, then the test deploys and starts as
+usual. When the test finishes, every unit is destroyed.
+
+If a run ends unexpectedly, check the Clusters tab to confirm nothing was left
+behind.
+
+Clusters are newer than fixed servers. If a run needs to be reliable — a
+scheduled regression test, a customer demo — fixed machines are the safer
+choice today.