@cloudbeat/atlas-cli 0.3.0 → 0.4.1

package/dist/skills/atlas-run/SKILL.md

@@ -0,0 +1,109 @@
+---
+name: atlas-run
+description: Start and size an Atlas load test - how many machines or cluster units, how users map onto them, ramp-up, pass/fail gates, exit codes and CI usage. Use when someone asks to run a test, asks how many machines they need, wants a test to fail a build on a threshold, or is waiting on a queued run.
+---
+
+# Running and sizing
+
+```bash
+atlas run -p <project> --jmx <plan> --servers 5 --wait --gate "error_rate<1,p95<2000"
+```
+
+## How load adds up
+
+**Each machine runs the whole plan independently.** There is no controller
+splitting users between them. So:
+
+```
+total users = users in the plan x number of machines
+```
+
+A plan with 100 users on 5 machines is 500 concurrent users. This is the single
+most common thing to get wrong, and it is wrong by a factor of the fleet size.
+
+A plan produced by the recorder exposes its size as run-time settings, so you can
+scale it without editing the file:
+
+```bash
+atlas run -p SHOP --jmx flow.jmx --servers 5 --args "-Jthreads=100 -Jrampup=300 -Jloops=-1"
+```
+
+- `-Jthreads` — users **per machine**
+- `-Jrampup` — seconds to start them all; ramping 500 users instantly tests your
+  own patience, not the application
+- `-Jloops` — iterations per user; `-1` is continuous
+
+An older or hand-written plan usually has fixed numbers inside it. Then machines
+are the only dial, and a plan with 2 users cannot be made into a 500-user test by
+asking for 5 machines. Say so instead of running it anyway.
+
+## Where it runs
+
+```bash
+atlas capacity --json
+```
+
+- `--servers <n>` — n available machines
+- `--server <name...>` — specific machines by name
+- `--cluster <name> --units <n>` — elastic capacity: n containers created for the
+  test and destroyed afterwards
+
+Machines that are off are fine: they are powered on for the run and powered off
+afterwards. That is also why a run costs money — confirm before starting a large
+one.
+
+If every machine is busy, the run is **queued** rather than rejected, and starts
+when they free up. `atlas run` says so; with `--wait` it waits through the queue.
+
+## Data files
+
+Data files are **divided** between machines by default — 10,000 rows across 5
+machines is 2,000 each, so no two users get the same row. Use `--no-split` when
+every machine must see the whole file (a small lookup table, a list of products
+every user should hit).
+
+## Gates
+
+```bash
+--gate "error_rate<1,p95<2000" --gate "throughput>50"
+```
+
+Metrics: `error_rate`, `avg`, `p90`, `p95`, `p99`, `max`, `throughput`,
+`requests`, with `<`, `<=`, `>`, `>=`. They are evaluated against the whole test.
+`--gate` and `--junit` imply `--wait`, because there is nothing to judge until
+the run has finished.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | ran, every gate passed |
+| 1 | ran, a gate was breached |
+| 2 | usage error: bad flags, unknown project, not authenticated |
+| 3 | the run failed or timed out, or the server was unreachable |
+
+1 and 3 are different in the way that matters for CI: 1 is a result, 3 is a
+failure to get a result. Do not paper over either.
+
+## In CI
+
+```bash
+atlas run -p SHOP --jmx flow.jmx --servers 3 --wait \
+  --gate "error_rate<1,p95<2000" --junit results.xml
+```
+
+Authenticate with a token in `ATLAS_TOKEN` — never a password. A token acts as
+its owner in one account, and it stops working the moment that user is removed.
+
+## While it runs, and stopping
+
+```bash
+atlas status <testId>      # still going? headline numbers
+atlas tests -p <project>   # recent runs, including anything running now
+atlas stop <testId>
+```
+
+`atlas stop` only reports success once it has **confirmed** the process is gone
+on every machine. If it comes back with machines it could not reach, the test may
+well still be running and reporting: say that, and retry. Treating an unconfirmed
+stop as a stop is how a runaway test goes unnoticed for hours.

package/dist/skills/atlas-test-plan/SKILL.md

@@ -0,0 +1,281 @@
+---
+name: atlas-test-plan
+description: Get a test plan and its data files into an Atlas project - download an existing plan, edit it or add a scenario to it, turn a fresh recording into a real test, put repeated values into a data file, debug it locally, check it will actually run, and upload it back. Use when someone wants to change an existing test, add a flow to one, asks what to do next with a plan or a recording they just made, fix a plan that reports nothing, or manage a project's test data files.
+---
+
+# Test plans and data files
+
+Every test needs a plan (`.jmx`) and usually data files (`.csv`/`.txt`) in the
+project. There are two ways to arrive at one, and both end the same way: check
+it, upload it, smoke-run it.
+
+**Start by looking at what is already there:**
+
+```bash
+atlas files list -p <project> --json
+```
+
+## Path A — change an existing plan
+
+```bash
+atlas files download -p <project> checkout.jmx -o ./checkout.jmx
+```
+
+Downloading first is not optional: **uploading replaces a file of the same name**
+and there is no version history. The copy you downloaded is the only way back.
+
+Then edit. Three kinds of change, in order of preference:
+
+**Add a whole new scenario** — record it straight into the downloaded plan. An
+existing target file is merged into, not overwritten, so the new flow arrives as
+an additional thread group and a `.bak` is written first:
+
+```bash
+atlas-rec start https://shop.example.com -o ./checkout.jmx -n "New checkout"
+# ... drive it, see atlas-record ...
+atlas-rec stop
+```
+
+**Change values in place** — thread counts, a data-file name, a host. Edit the
+specific `<stringProp>` and nothing else:
+
+```xml
+<stringProp name="ThreadGroup.num_threads">50</stringProp>
+<stringProp name="filename">users.csv</stringProp>
+```
+
+Never reformat the file, and never add or remove an element by hand unless you
+are certain about its `<hashTree>`. Every element in a JMeter plan is followed by
+a `<hashTree>` holding its children; an unbalanced one means JMeter will not open
+the file at all. `atlas plan check` counts them for you.
+
+**Structural surgery** — new controllers, reordering, deleting a thread group.
+Say plainly that this is better done in the JMeter GUI, and offer the targeted
+alternative instead. Replacing a plan you do not fully understand is how someone
+loses a scenario nobody has the source for.
+
+## Path B — a new plan
+
+Record one (see `atlas-record`), which produces a plan whose thread counts can be
+set at run time. Then continue below.
+
+## Turning a recording into a test
+
+A freshly recorded plan is a draft. Someone arriving with one — "I recorded a
+flow, what next?" — needs these stages, in this order, **asking as you go rather
+than deciding for them**. Do not upload it, and do not start editing it, before
+they have answered.
+
+Start by saying what is there: the scenario name, each transaction with its
+request count, any unexpected status codes, and where the plan and capture files
+are.
+
+**1. Is the flow right?** Are the steps and their names the ones they want
+reported? A missing step means recording again; a bad name is a one-line fix.
+
+**2. Should the recorded values become test data?** This is the question that
+turns a recording into a load test, so raise it every time. Read the capture and
+list what each simulated user would otherwise send identically:
+
+```bash
+atlas-rec state --full            # while recording
+atlas-rec inspect flow.capture.json
+```
+
+Look through the request parameters and bodies for:
+
+- **Credentials** — a username and password typed during the recording. Every
+  user signing in as one person is the most common way a test measures nothing
+  real: sessions collide, caches hide the work, and the login step is not tested
+  at all.
+- **Search terms, product ids, categories, postcodes** — one value means one
+  cached answer, repeated.
+- **Quantities, dates, addresses** — anything a real user varies.
+
+Distinguish these from **values the server issued** — session ids, tokens,
+cart ids, anything that came back in an earlier response. Those are correlation,
+not test data, and belong to `atlas-correlate`. `atlas-rec suggest` finds them.
+
+Propose a data file for the rest: say which values you would parameterise, what
+you would call the columns, and how many rows they need. Ask whether they have
+real data to use or want you to generate it — realistic data is theirs to
+provide, and generated data is fine when they say so.
+
+```csv
+username,password,search_term
+user001,...,laptop
+user002,...,headphones
+```
+
+Then wire the plan to read it: a CSV data set at the top of the thread group, and
+`${username}`-style variables replacing the literals in the requests. That edit is
+"Change values in place" below, and Atlas divides the rows between machines at run time,
+so each user gets a different row.
+
+**3. Do the dynamic values need handling?** Run `atlas-rec suggest`, show what it
+found with its evidence, and pin anything it missed — see `atlas-correlate`.
+
+**4. Are checks needed?** A 200 that returns a login page or an empty result is a
+pass as far as the plan is concerned. Offer assertions on the responses that
+matter, and debug them locally before uploading (see below).
+
+**5. Upload and smoke-run it** — the rest of this skill, then `atlas-load-test`.
+
+## How these plans are put together
+
+A recorded plan — and most plans in these projects — has the same skeleton, and
+knowing it saves you from reading 700 lines of XML to find one value:
+
+- **`Settings - Sleeps`** — a variables element at the top of the plan holding
+  `round_sleep`, `short_sleep` and `long_sleep`. Every pause in the plan is a
+  *Think Time* step reading one of them: `round_sleep` once per iteration,
+  `short_sleep` between steps the person clicked through quickly, `long_sleep`
+  where they paused. The pauses themselves stay enabled; **the variables element
+  is what switches them on and off.** Disabled, the variables do not exist, every
+  pause resolves to 0 ms, and the plan runs flat out. Enabled, its values are the
+  wait in milliseconds — typically 10s and 30s, so a four-step scenario takes
+  well over a minute per user.
+- **`Settings - Host`** — variables for the target host, normally disabled so the
+  recorded URLs are used as-is. Enable and edit it to point the same plan at
+  another environment.
+- **Cache and Cookie Managers, and HTTP Request Defaults**, applying to the whole
+  plan.
+- **One thread group per scenario**, each holding numbered transactions
+  (`01 Open home page`) and numbered samplers inside them (`01.01 /`). The
+  numbering is what makes a results report readable — keep it when you add a
+  step.
+- **Sizing through run-time settings** — `${__P(threads,..)}`, `${__P(rampup,..)}`
+  and `${__P(loops,..)}`, so `--args "-Jthreads=50"` scales the plan without
+  editing it. Note the sleep variables are **not** settings of that kind: `-J`
+  will not change them.
+
+Merging a recorded scenario into an existing plan keeps **that plan's** settings,
+not the recording's. A new recording has its sleeps switched off; a plan someone
+has been running usually has them on, and the new scenario inherits that.
+
+## Debugging a plan locally
+
+Running the plan yourself with one user is the fastest way to confirm a new
+scenario, an assertion or a correlation actually works. It needs JMeter on the
+machine (`jmeter -v`); when it is missing, say so and use a smoke run on Atlas
+instead.
+
+**Work on a copy.** Never debug the file you are about to upload:
+
+```bash
+cp ./checkout.jmx ./debug.jmx
+```
+
+**Check the sleeps, do not assume.** This is the single biggest time sink: with
+the pauses on, a four-step scenario takes ~100 s per user instead of ~5 s, and
+every re-check costs another two minutes.
+
+```bash
+grep -n 'testname="Settings - Sleeps"' ./debug.jmx
+```
+
+If the element does not carry `enabled="false"`, the pauses are live. Disable it
+in the copy — the element only, not the pauses:
+
+```xml
+<Arguments ... testname="Settings - Sleeps" enabled="false">
+```
+
+**Cut the run down to what you are testing:** add `enabled="false"` to the thread
+groups you are not debugging, and run one user, one iteration.
+
+```bash
+JVM_ARGS="-XX:TieredStopAtLevel=1" HEAP="-Xms256m -Xmx512m" \
+  jmeter -n -t debug.jmx -l debug.jtl -Jthreads=1 -Jloops=1 \
+  -Jjmeter.save.saveservice.output_format=csv \
+  -Jjmeter.save.saveservice.assertion_results_failure_message=true >/dev/null 2>&1
+awk -F, '{print $3" | "$4" | "$8" | "$9}' debug.jtl   # label, code, ok, why not
+```
+
+**Read the results file, not the console.** JMeter ends every non-GUI run with a
+page of log4j shutdown lines that mean nothing; the `.jtl` is the answer. The
+`JVM_ARGS`/`HEAP` prefix roughly halves JMeter's startup for a one-user run. A
+four-step scenario like this takes about 5 seconds end to end — if it takes a
+minute or more, the sleeps are still on.
+
+**Restore anything you changed.** If you had to disable the sleeps in the real
+plan rather than a copy — say, because the person asked you to debug that file —
+note the original values and the enabled/disabled state first, put them back
+before uploading, and confirm with the same `grep`. A plan uploaded with its
+think time switched off hammers the target far harder than intended, and nothing
+downstream will warn anyone.
+
+**Prove an assertion can fail, cheaply.** An assertion that passes may simply
+never have been evaluated. Check it against the recording before spending a run:
+the capture file holds the real response bodies, so a JSON path or a text match
+can be tested in a second with `python3`/`jq` — including the failing case. When
+you do want JMeter to prove it, put the passing and failing variants in the same
+copy, or start both runs at once, rather than one run after another.
+
+## Check before uploading
+
+```bash
+atlas plan check ./checkout.jmx -p <project>
+```
+
+This answers the questions that otherwise only surface as an empty dashboard an
+hour later. With `-p` it also checks the plan's data files against what the
+project actually holds. It exits non-zero when the plan would not produce
+results, and reports:
+
+- **A data file the plan reads that is not in the project.** This is the one that
+  hurts: every user in that thread group fails before sending a request, so the
+  group vanishes from the results *with no errors at all* while the other groups
+  look healthy.
+- **A plan Atlas cannot attach its reporting to** — it would run and report
+  nothing.
+- **A broken structure**, or no enabled thread group.
+- **Fixed user counts**, meaning machines are the only way to scale it.
+- Paths from another machine and GUI listeners are reported for information:
+  Atlas rewrites data-file paths to the uploaded file and switches off View
+  Results Tree and Simple Data Writer in the copy it deploys. Your stored file is
+  never modified.
+
+## Upload
+
+```bash
+atlas files upload -p <project> ./checkout.jmx ./users.csv
+```
+
+Same name replaces. Different name adds — so `--jmx` chooses which plan a run
+uses, and a project can hold several. Uploading a plan does not start anything.
+
+Then check it from the project's point of view, which is what will actually run:
+
+```bash
+atlas plan check -p <project> --jmx checkout.jmx
+```
+
+## Data files
+
+- **Names must match** what the plan reads. The plan referring to
+  `C:\data\users.csv` is fine as long as `users.csv` is uploaded — Atlas resolves
+  it. Nothing named `users.csv` in the project is not fine.
+- **Rows are divided between machines**, not copied. 10,000 rows across 5
+  machines is 2,000 each, so no two users get the same row. Pass `--no-split` at
+  run time when every machine must see the whole file.
+- **The first line is a header only if the plan says so.** Atlas reads that from
+  the plan's data-set configuration, so leave the file exactly as the plan
+  expects it — do not add or strip a header row to be helpful.
+- Generating test data is fine and often useful; say what you generated and how
+  many rows, and make sure it matches the columns the plan expects.
+
+## Deleting
+
+```bash
+atlas files rm -p <project> old-data.csv --yes
+```
+
+Deletion is not undoable from here. Download first if there is any doubt, and
+never delete a file you did not just upload without asking.
+
+## Finish the job
+
+An uploaded plan is not a working plan. Smoke-run it — one machine, a couple of
+users — and check the results contain data before anyone sizes it up. That flow
+is in `atlas-load-test`, and reading the result is in `atlas-analyze`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cloudbeat/atlas-cli",
-  "version": "0.3.0",
+  "version": "0.4.1",
   "description": "Atlas CLI — run JMeter load tests from CI and record scenarios",
   "main": "dist/cli.js",
   "bin": {
@@ -24,11 +24,13 @@
   },
   "license": "UNLICENSED",
   "scripts": {
-    "build": "tsc",
+    "build": "tsc && node scripts/bundle-docs.js && node scripts/bundle-skills.js",
     "dev": "ts-node src/cli.ts",
     "start": "node dist/cli.js",
     "watch": "tsc --watch",
-    "prepublishOnly": "npm run build"
+    "prepublishOnly": "npm run build",
+    "test:plan": "ts-node --compiler-options '{\"module\":\"commonjs\"}' test/plan.test.ts",
+    "test": "npm run test:plan"
   },
   "dependencies": {
     "chalk": "^4.1.2",