@cloudbeat/atlas-cli 0.3.0 → 0.4.0

package/dist/skills/atlas-correlate/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: atlas-correlate
+description: Handle dynamic values in a recorded load-test scenario - CSRF tokens, session ids, anything the server issues per session - by reviewing what the Atlas recorder detected and pinning the rest. Use when a recorded test plan fails on replay, returns login pages or 403s under load, or when someone asks about correlation or parameterisation.
+---
+
+# Dynamic values
+
+A recording captures literals. If the server issued a value for that one session
+— a CSRF token, a session id, an order reference — replaying the literal fails,
+usually as a 403, a redirect back to the login page, or an application error that
+only appears when the test runs.
+
+The recorder already handles this on evidence: a value a request sends is
+correlated only if that exact value appeared in an **earlier response**. That rule
+keeps it from rewriting things that merely look dynamic, but it also means it
+cannot catch a value the browser computed on its own.
+
+## Review what it found
+
+```bash
+atlas-rec suggest flow.capture.json
+```
+
+Each suggestion shows the variable, the value, **where it came from** (the
+response that defined it, and how it will be extracted) and **who sends it**.
+Judge them:
+
+- Source is the application, extraction looks stable, consumers make sense →
+  keep it.
+- The value is a fixed application constant that merely appeared in a response
+  (a country code, a page size) → correlating it is harmless but pointless.
+- A value that is sent but appears in **no** earlier response → the recorder
+  cannot correlate it, and it is exactly the kind that breaks. Find out what it
+  is: generated in the browser (a timestamp, a random id, a fingerprint), or
+  supposed to vary per user (a login, an account number).
+
+## Pin the rest yourself
+
+Write a rules file mapping parameter names to what should be sent instead:
+
+```json
+{
+  "authToken": "${authToken}",
+  "username": "${user}",
+  "requestId": "${__UUID()}"
+}
+```
+
+- `${user}` — a variable the plan reads from a data file. Per-user data belongs
+  in a CSV, not in the plan.
+- `${__UUID()}`, `${__time()}` — JMeter functions, for values that just have to
+  be fresh and unique.
+- A literal — when the value genuinely is a constant.
+
+Timestamps are already replaced with `${__time()}` automatically; `--no-timestamps`
+turns that off.
+
+Then rebuild the plan from the same capture — no re-recording:
+
+```bash
+atlas-rec generate flow.capture.json -o flow.jmx --correlations rules.json
+```
+
+Preset rules win over auto-correlation for the same parameter name, so a rule is
+how you override a detection you disagree with. `--no-correlate` turns detection
+off entirely and replays everything literally; that is a debugging tool, rarely
+what you want.
+
+## Prove it before scaling
+
+Correlation is not verified until the plan runs against the application. Upload
+it and smoke-run it (see `atlas-load-test`), then:
+
+```bash
+atlas errors <testId> --full
+```
+
+Read the actual responses. The tells that correlation is still wrong:
+
+- 403 / 419 / "invalid token" / "session expired" on requests that worked while
+  recording.
+- A 200 that returns the **login page** — technically a success, functionally a
+  failed test. Response-time numbers from a run like that are meaningless, and
+  they look good, which is the dangerous part.
+- The first iteration passes and later ones fail: something is being reused that
+  should be re-fetched per iteration.
+
+Run it with two users and two iterations rather than one of each — a single pass
+hides exactly this class of bug.

package/dist/skills/atlas-load-test/SKILL.md

@@ -0,0 +1,154 @@
+---
+name: atlas-load-test
+description: Run a load test with Atlas end to end - record or pick a test plan, upload test data, size the run, do a smoke test, run it, and report the results. Use when someone asks to load-test an application, create or run a performance test, check whether a site handles N users, or interpret a test that has already run.
+---
+
+# Load testing with Atlas
+
+This is the ordered flow. Work through it in order, stop at each **Ask** and wait
+for an answer, and never skip the smoke run. Other skills cover the detail:
+`atlas-record` (capturing a scenario), `atlas-test-plan` (downloading, editing
+and uploading plans and data files), `atlas-correlate` (dynamic values),
+`atlas-run` (sizing and gates), `atlas-analyze` (reading results).
+
+A load test costs real machine time on someone's account. Treat starting one the
+way you would treat a deploy: confirm before you spend.
+
+## 0. Check the ground
+
+```bash
+atlas whoami --json        # who, which account, which instance
+atlas projects --json      # what this credential can see
+```
+
+If `whoami` fails with exit code 2, the person needs to authenticate — tell them
+to run `atlas login --token <token>` with a token from their Profile page. Do not
+ask for their password.
+
+If `atlas` itself is missing: `npm install -g @cloudbeat/atlas-cli`.
+
+## 1. Pick the project
+
+**Ask** which project, if it is not obvious from the request or already set
+(`atlas config get`). Projects are the unit that owns test plans, data files and
+run history. Do not create one — that is done in the web interface.
+
+## 2. Get a test plan
+
+```bash
+atlas files list -p <project> --json
+```
+
+Three cases, and **ask** which one applies if the request does not already say:
+
+- **Use an existing plan as it is.** When several exist, ask which — `--jmx`
+  chooses, and the project's default may not be the one they mean.
+- **Change an existing plan**, or add a new scenario to it. Follow
+  `atlas-test-plan`: download it first, because uploading replaces the stored
+  file and there is no version history.
+- **Record a new one.** Follow `atlas-record`, then `atlas-test-plan` to check
+  and upload it.
+
+## 3. Test data and a plan that will actually run
+
+Plans usually read a CSV or TXT file per user (logins, search terms, IDs). Check
+the plan against what the project holds, rather than assuming:
+
+```bash
+atlas plan check -p <project> --jmx <plan>
+```
+
+That reports a data file the plan reads but the project does not have — the
+failure that makes an entire thread group disappear from the results with **no
+errors at all**. Upload whatever is missing, and anything that has gone stale:
+
+```bash
+atlas files upload -p <project> ./users.csv ./search-terms.txt
+```
+
+Worth saying out loud, because it surprises people: data files are **divided
+between machines**, not copied to each. Ten thousand rows across five machines is
+two thousand rows each. If every machine must see every row, the run needs
+`--no-split`. The rest of the file lifecycle is in `atlas-test-plan`.
+
+## 4. Size the run
+
+```bash
+atlas capacity --json
+```
+
+This gives machines, clusters, how many are free, and how much usage allowance is
+left. Read it **before** asking, so the question is grounded in real numbers.
+
+**Ask** three things:
+
+1. How many concurrent users should the test reach?
+2. How long should it run, and over how long should users ramp up?
+3. Fixed machines or a cluster? (Only if both are available.)
+
+Then do the arithmetic and show your working: **each machine runs the whole plan
+independently**, so total users = users-in-the-plan x machines. A plan with 100
+users on 5 machines is 500 users. A plan recorded with the recorder exposes its
+size as a run-time setting, so you can size it without editing the file:
+
+```bash
+atlas run -p <project> --jmx flow.jmx --servers 5 --args "-Jthreads=100 -Jrampup=300"
+```
+
+If the plan was not written that way, the thread counts are fixed inside the
+file, and machines are the only dial you have. Say so rather than silently
+running the wrong size.
+
+## 5. Smoke run first
+
+Never go straight to the full run. One machine, a couple of users, a minute or
+two:
+
+```bash
+atlas run -p <project> --jmx <plan> --servers 1 --args "-Jthreads=2 -Jloops=2" --wait
+```
+
+Then check it actually worked, which the exit code alone does not tell you:
+
+```bash
+atlas results <testId> --json
+atlas errors <testId> --json
+```
+
+Look for: every thread group present, requests actually sent, errors understood.
+A smoke run that finishes in seconds with no data is a broken plan, not a fast
+application — `atlas-analyze` lists the usual causes. Fix that before scaling up.
+
+**Ask** for confirmation before the full run, showing what you are about to
+start: plan, machines, users, duration, and the smoke result.
+
+## 6. The full run
+
+```bash
+atlas run -p <project> --jmx <plan> --servers <n> \
+  --args "-Jthreads=<per machine> -Jrampup=<seconds>" \
+  --wait --gate "error_rate<1,p95<2000"
+```
+
+`--wait` blocks until it finishes; `--gate` turns the result into a pass or fail
+(exit code 1 when a threshold is breached). Tell the person the test id and the
+dashboard link as soon as the run starts, so they can watch it themselves.
+
+If they do not want to wait, drop `--wait` and poll `atlas status <testId>`.
+
+## 7. Report
+
+Follow `atlas-analyze`. A useful report answers: did it pass, what was the
+response time under load, where did it start to degrade, what failed and why, and
+what to do next. Percentiles matter more than averages, and the error detail
+matters more than the error rate.
+
+## Guardrails
+
+- **Confirm before starting a full run.** Machines get powered on and billed.
+- **Never stop someone else's run.** `atlas tests -p <project>` shows what is
+  running; if something is already going, ask before touching it.
+- **Report what happened, not what should have happened.** If a thread group is
+  missing or a gate failed, say so plainly with the numbers.
+- **`atlas stop` can come back unconfirmed.** If it reports machines it could not
+  reach, the test may still be running — say that rather than "stopped".

package/dist/skills/atlas-record/SKILL.md

@@ -0,0 +1,167 @@
+---
+name: atlas-record
+description: Record a browser scenario into a JMeter test plan with the Atlas recorder, including driving the browser yourself - open pages, click, fill forms, name transactions - and rebuilding the plan from a capture afterwards. Use when someone wants to create a load-test scenario from a real user flow, or asks you to click through an application and turn it into a test.
+---
+
+# Recording a scenario
+
+The recorder watches a real browser session and writes a JMeter plan from it.
+A person can click through the flow themselves, or you can do the clicking. Both
+are recorded the same way, and both can happen in the same session.
+
+Requires `@cloudbeat/atlas-recorder` (`npm install -g @cloudbeat/atlas-recorder`).
+It downloads its own browser on install.
+
+## Which mode
+
+**Ask** which the person wants, unless the request already says:
+
+- **They click** — `atlas-rec record <url> -o flow.jmx`. Blocks until they press
+  Stop. Use this when the flow needs a human: a password only they know, a 2FA
+  prompt, a decision about what to test.
+- **You click** — the commands below. Use this when the flow is mechanical and
+  they have told you what to do.
+
+Either way the window is visible and they can watch, or take over.
+
+## Driving it yourself
+
+```bash
+atlas-rec start https://shop.example.com -o flow.jmx -n "Checkout"
+```
+
+Returns JSON with a `sessionId`. The browser is now open and recording. Later
+commands find the session automatically when only one is running; pass
+`--session <id>` when more than one is.
+
+**Look before you act.** `snapshot` gives the page as an accessibility tree —
+what is there and what can be clicked:
+
+```bash
+atlas-rec snapshot
+```
+
+Then act. Prefer the target the snapshot actually showed you:
+
+```bash
+atlas-rec click --role button --name "Find Flights"
+atlas-rec click --text "Choose This Flight"       # by visible text
+atlas-rec fill "you@example.com" --selector "#email"
+atlas-rec select "Rome" --selector "select[name=toPort]"
+atlas-rec press Enter --selector "#search"
+atlas-rec wait --selector "table"                 # or --url, or --ms
+atlas-rec goto https://shop.example.com/cart
+```
+
+Add `--nth <n>` when several things match. Snapshot again after anything that
+changes the page — do not assume the click landed where you expected.
+
+## Name the steps as you go
+
+Each user action becomes a transaction in the plan, and transaction names are
+what the results are reported against. Auto-generated names like "Rome" are
+useless in a report. Name the step **before** you perform it:
+
+```bash
+atlas-rec tx new "Search flights"
+atlas-rec click --role button --name "Find Flights"
+```
+
+Check what has been captured at any point:
+
+```bash
+atlas-rec state          # one line per transaction
+atlas-rec state --full   # every captured request
+```
+
+Housekeeping while recording:
+
+```bash
+atlas-rec tx rename t3 "Add to cart"
+atlas-rec req exclude r12     # keep a request out of the plan
+atlas-rec req delete r12      # drop it from the recording entirely
+atlas-rec pause / atlas-rec resume
+atlas-rec scenario "Checkout"
+```
+
+## Finishing
+
+```bash
+atlas-rec stop
+```
+
+This writes the plan and a capture file beside it, and returns both paths. The
+plan is also written if the window is closed or the process is interrupted — a
+recording is not lost by closing the browser.
+
+## Signing in
+
+Three options, in order of preference:
+
+- `--cookies <file>` — exported cookies, so the recording starts already signed
+  in. Accepts Cookie-Editor, Netscape, a raw `Cookie:` header, or a storage-state
+  file.
+- `--profile <file>` — saves the signed-in session on stop and reuses it next
+  time. Good for repeated recordings against the same app.
+- Record the login itself — but then the plan contains one user's credentials and
+  every simulated user will send them. That is usually wrong: a real test needs a
+  data file of users. Say so.
+
+Never ask for a password to type into the browser yourself. If the flow needs
+credentials, hand the window back to the person for that step: they can type it
+while the recording is live, and it is captured like anything else.
+
+## After the recording
+
+```bash
+atlas-rec inspect flow.capture.json    # transactions, hosts, response codes
+atlas-rec suggest flow.capture.json    # values that change per session
+```
+
+Read `inspect` before declaring success. Things to check:
+
+- Every step you performed has a transaction, and the count of included requests
+  is not zero.
+- No 4xx/5xx in `statusCodes` that you did not expect — a failed request recorded
+  as-is becomes a failing request in the test.
+- Hosts are the application's, not a third party's. Analytics and CDN traffic is
+  filtered out already, but an embedded widget may not be.
+
+## Then stop and agree the next step
+
+A recording is a draft, not a test. When you have stopped, **report what you
+captured and ask what to do with it** — do not upload it, and do not start
+editing it, before the person has answered.
+
+Say what is there: the scenario name, each transaction with its request count,
+any unexpected status codes, and where the plan and capture files are. Then work
+through the stages with them — flow, test data, dynamic values, checks, upload —
+asking rather than deciding. Those stages are in `atlas-test-plan`, under
+"Turning a recording into a test"; **read it before suggesting what comes next**,
+because the question it insists on (which recorded values every simulated user
+would otherwise repeat) is what separates a load test from a replay.
+
+Rebuild the plan from the capture rather than re-recording whenever the change is
+in how the plan is generated:
+
+```bash
+atlas-rec generate flow.capture.json -o flow.jmx --correlations rules.json
+```
+
+The same capture always produces the same plan, so this is safe to repeat.
+
+## Things that will bite you
+
+- **Never trigger a native dialog** (`alert`, `confirm`, a file picker). They
+  block the browser and the session stops responding. Avoid clicking anything
+  that might delete or confirm.
+- **One click can record two requests** when a page redirects, so a step can hit
+  its destination twice per iteration. Check `inspect` and exclude the duplicate.
+- **Everything before the first action is one transaction** — redirects, an SSO
+  hop, a failed load. Rename it to something meaningful.
+- **The recording is one pass.** A cookie banner, an A/B variant or a retry that
+  did not happen while you clicked is not in the plan.
+- **Recorded values are literal.** A search term, a product id, a date you picked
+  will be sent by every simulated user unless it is parameterised from a data
+  file. Flag this to the person; it is usually the difference between a realistic
+  test and a cached one.

package/dist/skills/atlas-run/SKILL.md

@@ -0,0 +1,109 @@
+---
+name: atlas-run
+description: Start and size an Atlas load test - how many machines or cluster units, how users map onto them, ramp-up, pass/fail gates, exit codes and CI usage. Use when someone asks to run a test, asks how many machines they need, wants a test to fail a build on a threshold, or is waiting on a queued run.
+---
+
+# Running and sizing
+
+```bash
+atlas run -p <project> --jmx <plan> --servers 5 --wait --gate "error_rate<1,p95<2000"
+```
+
+## How load adds up
+
+**Each machine runs the whole plan independently.** There is no controller
+splitting users between them. So:
+
+```
+total users = users in the plan x number of machines
+```
+
+A plan with 100 users on 5 machines is 500 concurrent users. This is the single
+most common thing to get wrong, and it is wrong by a factor of the fleet size.
+
+A plan produced by the recorder exposes its size as run-time settings, so you can
+scale it without editing the file:
+
+```bash
+atlas run -p SHOP --jmx flow.jmx --servers 5 --args "-Jthreads=100 -Jrampup=300 -Jloops=-1"
+```
+
+- `-Jthreads` — users **per machine**
+- `-Jrampup` — seconds to start them all; ramping 500 users instantly tests your
+  own patience, not the application
+- `-Jloops` — iterations per user; `-1` is continuous
+
+An older or hand-written plan usually has fixed numbers inside it. Then machines
+are the only dial, and a plan with 2 users cannot be made into a 500-user test by
+asking for 5 machines. Say so instead of running it anyway.
+
+## Where it runs
+
+```bash
+atlas capacity --json
+```
+
+- `--servers <n>` — n available machines
+- `--server <name...>` — specific machines by name
+- `--cluster <name> --units <n>` — elastic capacity: n containers created for the
+  test and destroyed afterwards
+
+Machines that are off are fine: they are powered on for the run and powered off
+afterwards. That is also why a run costs money — confirm before starting a large
+one.
+
+If every machine is busy, the run is **queued** rather than rejected, and starts
+when they free up. `atlas run` says so; with `--wait` it waits through the queue.
+
+## Data files
+
+Data files are **divided** between machines by default — 10,000 rows across 5
+machines is 2,000 each, so no two users get the same row. Use `--no-split` when
+every machine must see the whole file (a small lookup table, a list of products
+every user should hit).
+
+## Gates
+
+```bash
+--gate "error_rate<1,p95<2000" --gate "throughput>50"
+```
+
+Metrics: `error_rate`, `avg`, `p90`, `p95`, `p99`, `max`, `throughput`,
+`requests`, with `<`, `<=`, `>`, `>=`. They are evaluated against the whole test.
+`--gate` and `--junit` imply `--wait`, because there is nothing to judge until
+the run has finished.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | ran, every gate passed |
+| 1 | ran, a gate was breached |
+| 2 | usage error: bad flags, unknown project, not authenticated |
+| 3 | the run failed or timed out, or the server was unreachable |
+
+1 and 3 are different in the way that matters for CI: 1 is a result, 3 is a
+failure to get a result. Do not paper over either.
+
+## In CI
+
+```bash
+atlas run -p SHOP --jmx flow.jmx --servers 3 --wait \
+  --gate "error_rate<1,p95<2000" --junit results.xml
+```
+
+Authenticate with a token in `ATLAS_TOKEN` — never a password. A token acts as
+its owner in one account, and it stops working the moment that user is removed.
+
+## While it runs, and stopping
+
+```bash
+atlas status <testId>      # still going? headline numbers
+atlas tests -p <project>   # recent runs, including anything running now
+atlas stop <testId>
+```
+
+`atlas stop` only reports success once it has **confirmed** the process is gone
+on every machine. If it comes back with machines it could not reach, the test may
+well still be running and reporting: say that, and retry. Treating an unconfirmed
+stop as a stop is how a runaway test goes unnoticed for hours.