@cloudbeat/atlas-cli 0.2.0 → 0.4.0

package/dist/skills/AGENTS.md

@@ -0,0 +1,70 @@
+# Atlas load testing
+
+This project uses [Atlas](https://atlas.cloudbeat.io) for load testing. The
+`atlas` CLI drives it; `atlas-rec` records scenarios from a browser.
+
+Every command takes `--json`, and failures print JSON on standard error with a
+meaningful exit code (`0` pass, `1` a threshold was breached, `2` a usage or
+authentication problem, `3` the run failed or the server was unreachable).
+
+```bash
+atlas schema --json          # every command, argument and option
+atlas docs                   # the documentation, offline
+atlas docs troubleshooting   # one topic
+```
+
+## The flow
+
+```bash
+atlas whoami --json                       # who am I, which account
+atlas projects --json                     # what can I see
+atlas files list -p <project> --json      # test plans and data files
+atlas files download -p <project> <file> -o ./<file>   # before editing anything
+atlas plan check ./<file> -p <project>    # will it run, and report?
+atlas files upload -p <project> ./<file>  # same name replaces, no version history
+atlas capacity --json                     # machines, clusters, usage allowance
+atlas run -p <project> --jmx <plan> --servers <n> --wait --gate "error_rate<1,p95<2000"
+atlas results <testId> --json             # aggregation report
+atlas timeseries <testId> --json          # the shape of the run
+atlas errors <testId> --full              # what failed, with request and response
+```
+
+Recording a scenario, driven from the command line:
+
+```bash
+atlas-rec start <url> -o flow.jmx    # opens a browser, returns a session
+atlas-rec snapshot                   # what is on the page
+atlas-rec click --text "Sign in"     # act on it
+atlas-rec tx new "Checkout"          # name the step that comes next
+atlas-rec stop                       # write the plan
+atlas-rec suggest flow.capture.json  # values that change per session
+```
+
+## Rules that prevent real damage
+
+- **A run costs machine time on someone's account.** Confirm before starting one
+  that is not a smoke test, and never stop a run you did not start.
+- **Smoke-run first**: one machine, two users, a minute. Then check the results
+  actually contain data before scaling up.
+- **Load multiplies by machines.** Each machine runs the whole plan, so total
+  users = users-in-the-plan x machines.
+- **A test that measured nothing looks better than one that worked.** A missing
+  thread group with zero errors, a run that ended in seconds, or fast 200s that
+  return the login page all mean the test failed, not that the application is
+  fast.
+- **Download before you edit or replace a stored file.** Uploading replaces it
+  and nothing keeps the old version.
+- **Check a plan before uploading or running it.** A data file the project does
+  not have takes out a whole thread group with zero errors to explain it.
+- **Think time is a setting, not a property.** Recorded plans pause between steps
+  using a `Settings - Sleeps` variables element; `-J` does not change it. Before
+  running a plan locally to debug it, check that element and disable it in your
+  working copy, or one user takes minutes instead of seconds. Put it back exactly
+  as it was before uploading.
+- **A recording is a draft.** After stopping, report what was captured and ask
+  what to do next rather than uploading it. Offer to move the values every user
+  would otherwise repeat — credentials, search terms, ids — into a data file.
+- **Report what happened**, including failed gates and uncertainty.
+
+Full guidance is in the Atlas skills (`atlas skills install`) and at
+https://atlas.cloudbeat.io/docs.

package/dist/skills/atlas-analyze/SKILL.md

@@ -0,0 +1,104 @@
+---
+name: atlas-analyze
+description: Read and explain the results of an Atlas load test - the aggregation report, the shape of the run over time, the errors behind a failure - and recognise the failure modes where a broken test looks like a healthy one. Use when someone asks how a test went, what the numbers mean, why a test failed, or why results look wrong or missing.
+---
+
+# Reading results
+
+```bash
+atlas results <testId>                 # aggregation report, TOTAL first
+atlas timeseries <testId>              # users, response time, error rate over time
+atlas errors <testId>                  # distinct errors, most common first
+atlas errors <testId> --full           # with the request and response behind each
+```
+
+Add `--json` to any of them. `--label "<name>"` narrows timeseries and errors to
+one transaction or request.
+
+## Before interpreting anything, check the test was valid
+
+A load test that measured nothing usually looks *better* than one that worked.
+Rule these out first:
+
+**Is every thread group present?** Compare the report against the plan. A missing
+thread group with **zero errors** is the classic symptom of a data file the plan
+could not open: every user in that group fails before sending a request, so there
+is nothing to report as an error and the group simply vanishes while the others
+look healthy. The cause is almost always a data file that was never uploaded to
+the project. `atlas plan check -p <project> --jmx <plan>` names it in one line.
+(A path from another machine, `C:\data\users.csv`, is not itself the problem —
+Atlas resolves that to the uploaded file. Nothing uploaded under that name is.)
+
+**Did it run for a plausible length of time?** A run that ends in a few seconds
+with an empty dashboard did not run: the plan could not be started at all.
+
+**Is the dashboard empty despite a full-length run?** Atlas attaches its
+reporting to the plan when it deploys, and a file with an unusual structure can
+defeat that — the test runs and reports nothing. `atlas plan check` says so
+before you spend the time.
+
+**Are the response codes real?** A wall of fast 200s that return the login page
+is a failed test that passes every threshold. Check `atlas errors --full`, and be
+suspicious of response times that are far better than the application's normal
+behaviour.
+
+**Did every machine report?** Users should add up to roughly what you asked for
+(users-per-machine x machines). If the peak is a fraction of that, a machine was
+not running.
+
+## Then read the numbers
+
+- **Percentiles, not averages.** p95 and p99 are what users feel; an average
+  hides the tail completely.
+- **Error rate is errors over requests for the whole test**, not an average of
+  per-interval rates. The TOTAL row is authoritative.
+- **Throughput is the result, not the goal.** If throughput flattens while users
+  keep climbing, you have found the saturation point — that is usually the
+  headline of the whole test.
+- **Look at the shape.** `timeseries` shows where response time started to climb
+  and whether errors began at a particular user count. "It broke at about 300
+  users" is a more useful finding than any single number.
+- **Per-transaction matters more than TOTAL.** One slow step in an otherwise fast
+  plan is invisible in the total and obvious per label.
+
+## Explaining errors
+
+`atlas errors --full` gives the actual request and response for each distinct
+error. Sort out which kind you have:
+
+- **4xx** — usually the test, not the application: a stale correlated value, an
+  expired session, data that does not exist in the target environment.
+- **5xx** — usually the application, and worth reporting with the response body.
+- **Timeouts / connection errors** — the application or something in front of it
+  giving up. Check when they started against the user count.
+- **Assertion failures** — the request succeeded but returned the wrong thing,
+  such as a login page instead of the page you asked for.
+
+A failure that appears only above a certain load is the interesting one. Report
+the level it appeared at.
+
+## Comparing runs
+
+```bash
+atlas tests -p <project> --json    # recent runs with peak users, requests, error %
+atlas results <previous> --json
+```
+
+Compare like with like: the same plan, the same number of machines, the same
+user count. A regression is only a regression if the shape of the test was the
+same.
+
+## Writing it up
+
+A good report says, in this order:
+
+1. Whether it passed, against the thresholds that were set.
+2. What load was actually applied — users, ramp, duration, machines.
+3. Response time under that load: p95 per key transaction, not just the total.
+4. Where it started to degrade, from the time series.
+5. What failed, with real examples from the error detail.
+6. What to do next.
+
+State the uncertainty plainly where it exists. "The test never reached the target
+load because one machine dropped out" is a far more useful sentence than a
+confident summary of numbers that do not mean what they appear to.

package/dist/skills/atlas-correlate/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: atlas-correlate
+description: Handle dynamic values in a recorded load-test scenario - CSRF tokens, session ids, anything the server issues per session - by reviewing what the Atlas recorder detected and pinning the rest. Use when a recorded test plan fails on replay, returns login pages or 403s under load, or when someone asks about correlation or parameterisation.
+---
+
+# Dynamic values
+
+A recording captures literals. If the server issued a value for that one session
+— a CSRF token, a session id, an order reference — replaying the literal fails,
+usually as a 403, a redirect back to the login page, or an application error that
+only appears when the test runs.
+
+The recorder already handles this on evidence: a value a request sends is
+correlated only if that exact value appeared in an **earlier response**. That rule
+keeps it from rewriting things that merely look dynamic, but it also means it
+cannot catch a value the browser computed on its own.
+
+## Review what it found
+
+```bash
+atlas-rec suggest flow.capture.json
+```
+
+Each suggestion shows the variable, the value, **where it came from** (the
+response that defined it, and how it will be extracted) and **who sends it**.
+Judge them:
+
+- Source is the application, extraction looks stable, consumers make sense →
+  keep it.
+- The value is a fixed application constant that merely appeared in a response
+  (a country code, a page size) → correlating it is harmless but pointless.
+- A value that is sent but appears in **no** earlier response → the recorder
+  cannot correlate it, and it is exactly the kind that breaks. Find out what it
+  is: generated in the browser (a timestamp, a random id, a fingerprint), or
+  supposed to vary per user (a login, an account number).
+
+## Pin the rest yourself
+
+Write a rules file mapping parameter names to what should be sent instead:
+
+```json
+{
+  "authToken": "${authToken}",
+  "username": "${user}",
+  "requestId": "${__UUID()}"
+}
+```
+
+- `${user}` — a variable the plan reads from a data file. Per-user data belongs
+  in a CSV, not in the plan.
+- `${__UUID()}`, `${__time()}` — JMeter functions, for values that just have to
+  be fresh and unique.
+- A literal — when the value genuinely is a constant.
+
+Timestamps are already replaced with `${__time()}` automatically; `--no-timestamps`
+turns that off.
+
+Then rebuild the plan from the same capture — no re-recording:
+
+```bash
+atlas-rec generate flow.capture.json -o flow.jmx --correlations rules.json
+```
+
+Preset rules win over auto-correlation for the same parameter name, so a rule is
+how you override a detection you disagree with. `--no-correlate` turns detection
+off entirely and replays everything literally; that is a debugging tool, rarely
+what you want.
+
+## Prove it before scaling
+
+Correlation is not verified until the plan runs against the application. Upload
+it and smoke-run it (see `atlas-load-test`), then:
+
+```bash
+atlas errors <testId> --full
+```
+
+Read the actual responses. The tells that correlation is still wrong:
+
+- 403 / 419 / "invalid token" / "session expired" on requests that worked while
+  recording.
+- A 200 that returns the **login page** — technically a success, functionally a
+  failed test. Response-time numbers from a run like that are meaningless, and
+  they look good, which is the dangerous part.
+- The first iteration passes and later ones fail: something is being reused that
+  should be re-fetched per iteration.
+
+Run it with two users and two iterations rather than one of each — a single pass
+hides exactly this class of bug.

package/dist/skills/atlas-load-test/SKILL.md

@@ -0,0 +1,154 @@
+---
+name: atlas-load-test
+description: Run a load test with Atlas end to end - record or pick a test plan, upload test data, size the run, do a smoke test, run it, and report the results. Use when someone asks to load-test an application, create or run a performance test, check whether a site handles N users, or interpret a test that has already run.
+---
+
+# Load testing with Atlas
+
+This is the ordered flow. Work through it in order, stop at each **Ask** and wait
+for an answer, and never skip the smoke run. Other skills cover the detail:
+`atlas-record` (capturing a scenario), `atlas-test-plan` (downloading, editing
+and uploading plans and data files), `atlas-correlate` (dynamic values),
+`atlas-run` (sizing and gates), `atlas-analyze` (reading results).
+
+A load test costs real machine time on someone's account. Treat starting one the
+way you would treat a deploy: confirm before you spend.
+
+## 0. Check the ground
+
+```bash
+atlas whoami --json        # who, which account, which instance
+atlas projects --json      # what this credential can see
+```
+
+If `whoami` fails with exit code 2, the person needs to authenticate — tell them
+to run `atlas login --token <token>` with a token from their Profile page. Do not
+ask for their password.
+
+If `atlas` itself is missing: `npm install -g @cloudbeat/atlas-cli`.
+
+## 1. Pick the project
+
+**Ask** which project, if it is not obvious from the request or already set
+(`atlas config get`). Projects are the unit that owns test plans, data files and
+run history. Do not create one — that is done in the web interface.
+
+## 2. Get a test plan
+
+```bash
+atlas files list -p <project> --json
+```
+
+Three cases, and **ask** which one applies if the request does not already say:
+
+- **Use an existing plan as it is.** When several exist, ask which — `--jmx`
+  chooses, and the project's default may not be the one they mean.
+- **Change an existing plan**, or add a new scenario to it. Follow
+  `atlas-test-plan`: download it first, because uploading replaces the stored
+  file and there is no version history.
+- **Record a new one.** Follow `atlas-record`, then `atlas-test-plan` to check
+  and upload it.
+
+## 3. Test data and a plan that will actually run
+
+Plans usually read a CSV or TXT file per user (logins, search terms, IDs). Check
+the plan against what the project holds, rather than assuming:
+
+```bash
+atlas plan check -p <project> --jmx <plan>
+```
+
+That reports a data file the plan reads but the project does not have — the
+failure that makes an entire thread group disappear from the results with **no
+errors at all**. Upload whatever is missing, and anything that has gone stale:
+
+```bash
+atlas files upload -p <project> ./users.csv ./search-terms.txt
+```
+
+Worth saying out loud, because it surprises people: data files are **divided
+between machines**, not copied to each. Ten thousand rows across five machines is
+two thousand rows each. If every machine must see every row, the run needs
+`--no-split`. The rest of the file lifecycle is in `atlas-test-plan`.
+
+## 4. Size the run
+
+```bash
+atlas capacity --json
+```
+
+This gives machines, clusters, how many are free, and how much usage allowance is
+left. Read it **before** asking, so the question is grounded in real numbers.
+
+**Ask** three things:
+
+1. How many concurrent users should the test reach?
+2. How long should it run, and over how long should users ramp up?
+3. Fixed machines or a cluster? (Only if both are available.)
+
+Then do the arithmetic and show your working: **each machine runs the whole plan
+independently**, so total users = users-in-the-plan x machines. A plan with 100
+users on 5 machines is 500 users. A plan recorded with the recorder exposes its
+size as a run-time setting, so you can size it without editing the file:
+
+```bash
+atlas run -p <project> --jmx flow.jmx --servers 5 --args "-Jthreads=100 -Jrampup=300"
+```
+
+If the plan was not written that way, the thread counts are fixed inside the
+file, and machines are the only dial you have. Say so rather than silently
+running the wrong size.
+
+## 5. Smoke run first
+
+Never go straight to the full run. One machine, a couple of users, a minute or
+two:
+
+```bash
+atlas run -p <project> --jmx <plan> --servers 1 --args "-Jthreads=2 -Jloops=2" --wait
+```
+
+Then check it actually worked, which the exit code alone does not tell you:
+
+```bash
+atlas results <testId> --json
+atlas errors <testId> --json
+```
+
+Look for: every thread group present, requests actually sent, errors understood.
+A smoke run that finishes in seconds with no data is a broken plan, not a fast
+application — `atlas-analyze` lists the usual causes. Fix that before scaling up.
+
+**Ask** for confirmation before the full run, showing what you are about to
+start: plan, machines, users, duration, and the smoke result.
+
+## 6. The full run
+
+```bash
+atlas run -p <project> --jmx <plan> --servers <n> \
+  --args "-Jthreads=<per machine> -Jrampup=<seconds>" \
+  --wait --gate "error_rate<1,p95<2000"
+```
+
+`--wait` blocks until it finishes; `--gate` turns the result into a pass or fail
+(exit code 1 when a threshold is breached). Tell the person the test id and the
+dashboard link as soon as the run starts, so they can watch it themselves.
+
+If they do not want to wait, drop `--wait` and poll `atlas status <testId>`.
+
+## 7. Report
+
+Follow `atlas-analyze`. A useful report answers: did it pass, what was the
+response time under load, where did it start to degrade, what failed and why, and
+what to do next. Percentiles matter more than averages, and the error detail
+matters more than the error rate.
+
+## Guardrails
+
+- **Confirm before starting a full run.** Machines get powered on and billed.
+- **Never stop someone else's run.** `atlas tests -p <project>` shows what is
+  running; if something is already going, ask before touching it.
+- **Report what happened, not what should have happened.** If a thread group is
+  missing or a gate failed, say so plainly with the numbers.
+- **`atlas stop` can come back unconfirmed.** If it reports machines it could not
+  reach, the test may still be running — say that rather than "stopped".

package/dist/skills/atlas-record/SKILL.md

@@ -0,0 +1,167 @@
+---
+name: atlas-record
+description: Record a browser scenario into a JMeter test plan with the Atlas recorder, including driving the browser yourself - open pages, click, fill forms, name transactions - and rebuilding the plan from a capture afterwards. Use when someone wants to create a load-test scenario from a real user flow, or asks you to click through an application and turn it into a test.
+---
+
+# Recording a scenario
+
+The recorder watches a real browser session and writes a JMeter plan from it.
+A person can click through the flow themselves, or you can do the clicking. Both
+are recorded the same way, and both can happen in the same session.
+
+Requires `@cloudbeat/atlas-recorder` (`npm install -g @cloudbeat/atlas-recorder`).
+It downloads its own browser on install.
+
+## Which mode
+
+**Ask** which the person wants, unless the request already says:
+
+- **They click** — `atlas-rec record <url> -o flow.jmx`. Blocks until they press
+  Stop. Use this when the flow needs a human: a password only they know, a 2FA
+  prompt, a decision about what to test.
+- **You click** — the commands below. Use this when the flow is mechanical and
+  they have told you what to do.
+
+Either way the window is visible and they can watch, or take over.
+
+## Driving it yourself
+
+```bash
+atlas-rec start https://shop.example.com -o flow.jmx -n "Checkout"
+```
+
+Returns JSON with a `sessionId`. The browser is now open and recording. Later
+commands find the session automatically when only one is running; pass
+`--session <id>` when more than one is.
+
+**Look before you act.** `snapshot` gives the page as an accessibility tree —
+what is there and what can be clicked:
+
+```bash
+atlas-rec snapshot
+```
+
+Then act. Prefer the target the snapshot actually showed you:
+
+```bash
+atlas-rec click --role button --name "Find Flights"
+atlas-rec click --text "Choose This Flight"       # by visible text
+atlas-rec fill "you@example.com" --selector "#email"
+atlas-rec select "Rome" --selector "select[name=toPort]"
+atlas-rec press Enter --selector "#search"
+atlas-rec wait --selector "table"                 # or --url, or --ms
+atlas-rec goto https://shop.example.com/cart
+```
+
+Add `--nth <n>` when several things match. Snapshot again after anything that
+changes the page — do not assume the click landed where you expected.
+
+## Name the steps as you go
+
+Each user action becomes a transaction in the plan, and transaction names are
+what the results are reported against. Auto-generated names like "Rome" are
+useless in a report. Name the step **before** you perform it:
+
+```bash
+atlas-rec tx new "Search flights"
+atlas-rec click --role button --name "Find Flights"
+```
+
+Check what has been captured at any point:
+
+```bash
+atlas-rec state          # one line per transaction
+atlas-rec state --full   # every captured request
+```
+
+Housekeeping while recording:
+
+```bash
+atlas-rec tx rename t3 "Add to cart"
+atlas-rec req exclude r12     # keep a request out of the plan
+atlas-rec req delete r12      # drop it from the recording entirely
+atlas-rec pause / atlas-rec resume
+atlas-rec scenario "Checkout"
+```
+
+## Finishing
+
+```bash
+atlas-rec stop
+```
+
+This writes the plan and a capture file beside it, and returns both paths. The
+plan is also written if the window is closed or the process is interrupted — a
+recording is not lost by closing the browser.
+
+## Signing in
+
+Three options, in order of preference:
+
+- `--cookies <file>` — exported cookies, so the recording starts already signed
+  in. Accepts Cookie-Editor, Netscape, a raw `Cookie:` header, or a storage-state
+  file.
+- `--profile <file>` — saves the signed-in session on stop and reuses it next
+  time. Good for repeated recordings against the same app.
+- Record the login itself — but then the plan contains one user's credentials and
+  every simulated user will send them. That is usually wrong: a real test needs a
+  data file of users. Say so.
+
+Never ask for a password to type into the browser yourself. If the flow needs
+credentials, hand the window back to the person for that step: they can type it
+while the recording is live, and it is captured like anything else.
+
+## After the recording
+
+```bash
+atlas-rec inspect flow.capture.json    # transactions, hosts, response codes
+atlas-rec suggest flow.capture.json    # values that change per session
+```
+
+Read `inspect` before declaring success. Things to check:
+
+- Every step you performed has a transaction, and the count of included requests
+  is not zero.
+- No 4xx/5xx in `statusCodes` that you did not expect — a failed request recorded
+  as-is becomes a failing request in the test.
+- Hosts are the application's, not a third party's. Analytics and CDN traffic is
+  filtered out already, but an embedded widget may not be.
+
+## Then stop and agree the next step
+
+A recording is a draft, not a test. When you have stopped, **report what you
+captured and ask what to do with it** — do not upload it, and do not start
+editing it, before the person has answered.
+
+Say what is there: the scenario name, each transaction with its request count,
+any unexpected status codes, and where the plan and capture files are. Then work
+through the stages with them — flow, test data, dynamic values, checks, upload —
+asking rather than deciding. Those stages are in `atlas-test-plan`, under
+"Turning a recording into a test"; **read it before suggesting what comes next**,
+because the question it insists on (which recorded values every simulated user
+would otherwise repeat) is what separates a load test from a replay.
+
+Rebuild the plan from the capture rather than re-recording whenever the change is
+in how the plan is generated:
+
+```bash
+atlas-rec generate flow.capture.json -o flow.jmx --correlations rules.json
+```
+
+The same capture always produces the same plan, so this is safe to repeat.
+
+## Things that will bite you
+
+- **Never trigger a native dialog** (`alert`, `confirm`, a file picker). They
+  block the browser and the session stops responding. Avoid clicking anything
+  that might delete or confirm.
+- **One click can record two requests** when a page redirects, so a step can hit
+  its destination twice per iteration. Check `inspect` and exclude the duplicate.
+- **Everything before the first action is one transaction** — redirects, an SSO
+  hop, a failed load. Rename it to something meaningful.
+- **The recording is one pass.** A cookie banner, an A/B variant or a retry that
+  did not happen while you clicked is not in the plan.
+- **Recorded values are literal.** A search term, a product id, a date you picked
+  will be sent by every simulated user unless it is parameterised from a data
+  file. Flag this to the person; it is usually the difference between a realistic
+  test and a cached one.