@cloudbeat/atlas-cli 0.2.0 → 0.4.0

package/dist/docs/results.md

@@ -0,0 +1,79 @@
+# Reading Results
+
+Every run has a dashboard. Open it from a project's **Results** tab, or from the
+test list at `/<project>`.
+
+The dashboard refreshes itself while a test is running, so you can leave it open
+from start to finish.
+
+## Headline numbers
+
+| | |
+|---|---|
+| **Status** | Whether the test is still running |
+| **Test Started** | When it began |
+| **Duration** | How long it has been running, or ran for |
+| **Max Users** | The peak number of concurrent users, added up across every machine |
+| **Req/sec** | Throughput |
+| **Avg Response Time** | Average across the whole test |
+| **Error Rate** | Failed requests as a percentage of all requests, across the whole test |
+
+Each load generator reports its own thread count, and Atlas adds them up. If you
+ran a 100-thread plan on 5 machines, expect roughly 500 — if it shows 100,
+something stopped four of your generators from producing load.
+
+Average response time and error rate are **whole-test** figures, not the last
+few seconds, so they won't jump around as the test finishes.
+
+## The chart
+
+Three series on one timeline:
+
+- **Active Users** (green, right axis) — concurrency over time
+- **Avg. Response Time** (blue, left axis) — how the system responded as load grew
+- **Error Rate %** (red) — when things started failing
+
+Read together, these tell you where your system's limit is: the point where
+response time climbs and errors begin while users are still increasing.
+
+## The aggregation report
+
+One row per request, grouped by thread group, with count, response times
+(average, median, percentiles, min/max), throughput and error percentage.
+
+Every column sorts. Rows are colour-coded:
+
+| | |
+|---|---|
+| **TOTAL** | The whole test |
+| **Thread group** | One per thread group in your plan |
+| **Transaction** | A named step, if your plan uses Transaction Controllers |
+| **Request** | An individual HTTP request |
+
+The same sampler name in two different thread groups stays two separate rows,
+attributed to the right group — so a shared "login" step used by several groups
+doesn't get merged into one misleading number.
+
+### Drilling in
+
+- **Click an Error %** — opens a panel listing the actual errors for that
+  request: response codes and messages, with the full request and response for
+  each distinct failure.
+- **Click a Label** — opens the same chart filtered to that one request, so you
+  can see whether a single endpoint degraded while the rest stayed healthy.
+  There's an error-type dropdown to isolate one specific failure.
+
+## Exporting
+
+**↓ Export Excel** downloads the run as a spreadsheet with the chart and the full
+table, colour-coded to match. Useful for attaching to a report.
+
+## Dark mode
+
+Add `?theme=dark` to any dashboard URL.
+
+## Comparing runs
+
+There is no side-by-side comparison view yet. Name your runs (in the Results
+tab) so you can tell variations apart, and export to Excel to compare them
+outside Atlas.

package/dist/docs/running.md

@@ -0,0 +1,80 @@
+# Running a Test
+
+## From the web interface
+
+Open a project, go to **Run Test**, choose where to run, and press **Run**.
+
+There is no separate deploy step. Atlas reserves the machines, powers on any
+that are switched off, copies the files, and starts JMeter.
+
+## Choosing where it runs
+
+| | |
+|---|---|
+| **Servers** | Pick individual load generators by name. Switched-off machines can still be selected — they are powered on for you. |
+| **Cluster** | Run on a cluster and say how many units you want. Units are created for the test and destroyed afterwards. |
+
+The number of machines multiplies your load: a plan with 100 threads run on 5
+generators applies 500 concurrent users to your system.
+
+Atlas does not divide your thread count across machines — each one runs the plan
+as written. Five generators running a 100-thread plan is 500 users, not 100.
+
+## From the CLI
+
+```bash
+atlas run -p SHOP --jmx checkout.jmx --servers 5 --wait
+```
+
+| | |
+|---|---|
+| `--servers 5` | use any 5 available load generators |
+| `--server gen-01 --server gen-02` | pick specific ones by name |
+| `--cluster my-cluster --units 10` | run on 10 units in a cluster |
+| `--jmx <filename>` | which uploaded plan to run (needed only if the project holds more than one) |
+| `--args "<jmeter args>"` | extra JMeter arguments |
+| `--no-split` | send whole data files to every machine |
+| `--wait` | follow the run and print the report when it finishes |
+| `--timeout 30m` | how long `--wait` waits before giving up (default `2h`) |
+
+## While it runs
+
+The Run Test tab becomes a live view showing each stage — powering on,
+deploying, running — with a **Stop** button.
+
+**Watch** in the sidebar shows live per-machine statistics while a test is in
+progress, which is the quickest way to confirm every generator is actually
+producing load.
+
+From the CLI:
+
+```bash
+atlas status SHOP-20260818-194201
+```
+
+## If a machine is busy
+
+A load generator can only run one test at a time. If you ask for machines that
+are already in use, your run is **queued** and starts when they free up. You
+don't need to retry.
+
+## Stopping a test
+
+Press **Stop**, or:
+
+```bash
+atlas stop SHOP-20260818-194201
+```
+
+Atlas asks JMeter to shut down cleanly so the final results are recorded, and
+only reports the machine as stopped once the process has actually ended.
+
+If a machine can't be reached when you stop a test, Atlas tells you rather than
+pretending it worked, and the run stays open. Retry once the machine is
+reachable — this prevents a test carrying on unnoticed in the background.
+
+## When a test ends
+
+A test ends by itself when the plan says so — a thread group's duration or loop
+count running out. When it does, the machines are released and any that Atlas
+powered on are switched off.

package/dist/docs/skills.md

@@ -0,0 +1,88 @@
+# The Skills
+
+`atlas skills install` writes six skills plus a vendor-neutral `AGENTS.md`. Each
+skill is a short document covering one part of load testing: what to do, in what
+order, and which mistakes to avoid.
+
+They are plain Markdown. Read them, edit them for your project, commit them —
+they are yours once installed.
+
+## How they load
+
+Claude Code reads each skill's description and loads the one that fits your
+request, so you never name a skill yourself. Ask *"why did the test fail?"* and
+it loads the results skill; ask *"record a checkout"* and it loads the recording
+one. Several can load in one conversation as the work moves on.
+
+Assistants without skill support read the same guidance, condensed, from
+`AGENTS.md`.
+
+## What each one covers
+
+### `atlas-load-test`
+
+The end-to-end flow, and the one that usually loads first for *"load test our
+checkout"*. Picking or creating a plan, uploading data, sizing, the mandatory
+smoke run, then the real run and the report. Mostly it sequences the others.
+
+### `atlas-record`
+
+Recording a scenario from a real browser. Choosing who clicks — you or the
+assistant — driving the browser through the accessibility tree, naming
+transactions before performing them, signing in without handing over a password,
+and the traps: native dialogs that freeze a session, one click recording two
+requests, everything before the first action landing in one transaction.
+
+### `atlas-test-plan`
+
+Plans and data files, and the longest of the six. Downloading before editing,
+the three kinds of change in order of preference, and how these plans are laid
+out — think-time settings, host settings, the numbered transactions that make a
+report readable, and sizing through run-time properties.
+
+It also carries the two things that save the most time in practice: the stages
+for turning a fresh recording into a real test (flow → test data → dynamic
+values → checks → upload), and a local debugging recipe that switches think time
+off in a throwaway copy, runs one user, and restores the original — seconds
+instead of minutes per attempt.
+
+### `atlas-correlate`
+
+Values the server issues per session — CSRF tokens, session ids, cart ids — and
+why a replayed one produces a fast, clean, meaningless test. Reviewing what the
+recorder detected with its evidence, pinning what it missed, and telling those
+apart from values that belong in a data file instead.
+
+### `atlas-run`
+
+Starting and sizing a run: how users map onto machines, ramp-up, fixed machines
+versus a cluster, pass/fail thresholds, exit codes and CI usage, and what to do
+while a run is queued behind someone else's.
+
+### `atlas-analyze`
+
+Reading a result: the aggregation report, the shape of a run over time, the
+errors behind a failure — and recognising the failure modes where a broken test
+looks like a healthy one. If you only read one, read this one.
+
+## Editing them
+
+Installed skills are ordinary files in your repository. Adding your own
+conventions is normal and encouraged:
+
+- which project maps to which service
+- the thresholds your team gates on
+- environments that must never be load tested
+- who to ask before a run above a certain size
+
+`atlas skills install` will not overwrite a file you have edited unless you pass
+`--force`, so your changes survive an upgrade. When you do upgrade, diff first.
+
+## Listing what you have
+
+```sh
+atlas skills list              # what this CLI version carries
+atlas skills install --force   # refresh after upgrading the CLI
+```
+
+[Build your own integration →](/agents/automation)

package/dist/docs/test-plans.md

@@ -0,0 +1,92 @@
+# Test Plan Requirements
+
+Atlas runs standard JMeter test plans, but a plan written on a desktop needs a
+few changes before it runs correctly on a fleet of Linux load generators. These
+are the things that actually catch people out.
+
+## Data file paths
+
+Reference data files by a plain relative path:
+
+```
+./users.csv          ✅
+users.csv            ✅
+C:\Users\me\users.csv   ❌
+/home/me/data/users.csv ❌
+./data/users.csv        ❌
+```
+
+A project's files live in one flat folder — see [Test Files](/guide/test-files).
+
+If a **CSV Data Set Config** path can't be resolved, every thread in that thread
+group throws before issuing a single request. There are no failed samples: the
+summary reports `0.00%` errors and the dashboard **omits that thread group**
+entirely while the others look healthy.
+
+If a thread group is missing from your results, check its data file paths first.
+
+Every file the plan references must also actually be uploaded.
+
+## Disable GUI listeners
+
+Switch these off before uploading:
+
+- **View Results Tree**
+- **Aggregate Report**
+- **Simple Data Writer**
+
+They consume a lot of memory during a headless run — enough to destabilise a
+long test — and Atlas collects its own results regardless. A **Simple Data
+Writer** pointing at a `C:\...` path is doubly useless.
+
+## Thread counts are per machine
+
+Atlas does not divide your threads across machines. A 100-thread plan on 5
+generators is **500** concurrent users.
+
+Decide the number you want in total, then divide by how many generators you plan
+to use.
+
+## Header rows in CSV files
+
+Whether line 1 of a data file is a header is decided by your plan's **CSV Data
+Set Config**, not guessed:
+
+- If **Variable Names** is filled in and **Ignore first line** is true, line 1 is
+  a header.
+- If **Variable Names** is empty, JMeter reads line 1 as the names — so it counts
+  as a header too.
+
+Get this right and each machine's slice keeps the header while the data rows are
+divided. Get it wrong and you either inject a bogus row or lose a real one.
+
+## Timers and pacing
+
+Think time matters more in a distributed test, not less. Without it, every
+virtual user hammers as fast as the network allows, which produces a number that
+looks impressive and means nothing.
+
+## Parameterising the target
+
+If you test more than one environment, drive the host from a variable rather
+than hardcoding it, and pass it at run time:
+
+```sh
+atlas run -p SHOP --args "-Jhost=staging.example.com"
+```
+
+## A checklist
+
+Before your first run:
+
+- [ ] Data file paths are relative (`./file.csv`)
+- [ ] Every referenced file is uploaded
+- [ ] GUI listeners disabled
+- [ ] Thread count is per-machine, and you've multiplied by your generator count
+- [ ] Header handling matches your files
+- [ ] Timers present
+- [ ] The plan runs locally in headless mode first: `jmeter -n -t plan.jmx`
+
+That last one is the highest-value check. `jmeter -n -t plan.jmx` on your own
+machine catches most problems in seconds, without waiting for machines to power
+on.

package/dist/docs/troubleshooting.md

@@ -0,0 +1,79 @@
+# Troubleshooting
+
+## A thread group is missing from my results
+
+The usual cause is a **data file path that can't be resolved**. Every thread in
+that group fails before sending a request, so there are no errors to report — the
+group simply isn't there.
+
+Check the **CSV Data Set Config** entries in that thread group and make each one
+a relative path (`./data.csv`), then confirm the file is uploaded. See
+[Test Plan Requirements](/reference/test-plans).
+
+## The test finished in seconds with no results
+
+Usually the plan didn't start properly. Check:
+
+- The plan runs locally: `jmeter -n -t plan.jmx`
+- Every referenced file is uploaded
+- The machine is big enough — an undersized generator can fail to start JMeter
+  at all rather than running slowly
+
+## Max Users is lower than I expected
+
+Max Users adds up the concurrent users reported by every machine. If you ran on
+5 generators and it shows roughly one machine's worth, the others weren't
+producing load.
+
+Check the Fleet page: were all the machines connected? Open **Watch** during a
+run to see per-machine activity live.
+
+## A machine shows as on, but not connected
+
+The machine is running but its agent isn't reaching Atlas. On the machine:
+
+- Confirm the Atlas agent service is running
+- Confirm it can reach your Atlas instance over outbound HTTPS
+- Check for a proxy or firewall rule blocking outbound connections
+
+The agent reconnects on its own once the connection is restored.
+
+## My test is queued and hasn't started
+
+A load generator runs one test at a time. If the machines you asked for are
+busy, your run waits and starts automatically when they free up. Either wait, or
+choose different machines.
+
+## Stop said it couldn't reach a machine
+
+Atlas only marks a machine stopped once it has confirmed the process actually
+ended. If it can't reach one, it tells you and the run stays open — deliberately,
+so a test can't keep running unnoticed while the record says it stopped.
+
+Retry the stop when the machine is reachable again.
+
+## Error rate is high but I can't see why
+
+Click the **Error %** cell for a request in the aggregation report. That opens
+the actual errors — response codes, messages, and the full request and response
+for each distinct failure.
+
+If errors are concentrated in one endpoint, click its **Label** to chart that
+request on its own and see when it started failing.
+
+## Results look wrong for one request
+
+Check whether the same sampler name exists in more than one thread group. Atlas
+keeps them as separate rows attributed to the right group — if you expected them
+combined, that's why the numbers differ from a single-machine JMeter run.
+
+## A recorded plan fails on the second run
+
+Usually a dynamic value that wasn't correlated — a token that was valid when you
+recorded and isn't now. Re-record with correlation enabled (the default), and
+check that the step which issues the token is part of the recording.
+
+## Getting help
+
+When reporting a problem, include the **test ID** (`SHOP-20260818-194201`). It
+identifies the run and everything recorded about it.

package/dist/plan.js

@@ -0,0 +1,250 @@
+"use strict";
+/**
+ * Reading a JMeter test plan well enough to say whether it will work here.
+ *
+ * This is an advisory scan, not the deploy path: the control plane does its own
+ * work on the copy it sends out (flattening data-file paths it recognises,
+ * switching off GUI listeners, attaching Atlas reporting). What it CANNOT fix is
+ * a data file that was never uploaded, a plan with no enabled thread group, or a
+ * file whose structure it cannot attach reporting to — and every one of those
+ * fails quietly at run time, which is what this exists to catch beforehand.
+ *
+ * Regex over the XML, deliberately: the same shape the control plane uses, no
+ * parser dependency, and it never rewrites anything.
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.checkPlan = checkPlan;
+/** Lower-cased basename, tolerant of Windows separators — matches how the
+ *  control plane decides whether a reference means an uploaded file. */
+function basenameKey(filename) {
+    const parts = filename.replace(/\\/g, '/').split('/');
+    return (parts[parts.length - 1] ?? filename).trim().toLowerCase();
+}
+function basename(filename) {
+    const parts = filename.replace(/\\/g, '/').split('/');
+    return (parts[parts.length - 1] ?? filename).trim();
+}
+function stringProp(block, name) {
+    const m = block.match(new RegExp(`<stringProp name="${name}">([\\s\\S]*?)</stringProp>`));
+    return m ? m[1].trim() : '';
+}
+function intProp(block, name) {
+    const m = block.match(new RegExp(`<(?:string|int)Prop name="${name}">([\\s\\S]*?)</(?:string|int)Prop>`));
+    return m ? m[1].trim() : '';
+}
+function attr(tag, name) {
+    return new RegExp(`${name}="([^"]*)"`).exec(tag)?.[1] ?? '';
+}
+/** JMeter treats a missing `enabled` attribute as true. */
+function isEnabled(tag) {
+    return attr(tag, 'enabled') !== 'false';
+}
+const THREAD_GROUP_TAGS = [
+    'ThreadGroup',
+    'SetupThreadGroup',
+    'PostThreadGroup',
+    'com\\.blazemeter\\.jmeter\\.threads\\.concurrency\\.ConcurrencyThreadGroup',
+    'kg\\.apc\\.jmeter\\.threads\\.UltimateThreadGroup',
+];
+function readThreadGroups(jmx) {
+    const out = [];
+    for (const tag of THREAD_GROUP_TAGS) {
+        const re = new RegExp(`<(${tag})\\b([^>]*)>([\\s\\S]*?)</\\1>`, 'g');
+        let m;
+        while ((m = re.exec(jmx)) !== null) {
+            const [, kind, attrs, body] = m;
+            const threads = intProp(body, 'ThreadGroup.num_threads') || intProp(body, 'TargetLevel');
+            const property = /\$\{__P\(([^,)]+)/.exec(threads)?.[1]?.trim();
+            out.push({
+                name: attr(`<x ${attrs}>`, 'testname'),
+                kind,
+                enabled: isEnabled(`<x ${attrs}>`),
+                threads,
+                loops: intProp(body, 'LoopController.loops'),
+                ...(property ? { sizeProperty: property } : {}),
+            });
+        }
+    }
+    return out;
+}
+function readDataFiles(jmx, uploaded) {
+    const known = uploaded ? new Set(uploaded.map(basenameKey)) : undefined;
+    // Keyed by the file it resolves to: a plan commonly reads one file from
+    // several data sets, and reporting it four times helps nobody.
+    const byFile = new Map();
+    const re = /<CSVDataSet\b([^>]*)>([\s\S]*?)<\/CSVDataSet>/g;
+    let m;
+    while ((m = re.exec(jmx)) !== null) {
+        const [, attrs, body] = m;
+        const reference = stringProp(body, 'filename');
+        if (!reference)
+            continue;
+        // A path built at run time can't be judged now, and the control plane leaves
+        // it alone for the same reason.
+        if (reference.includes('${'))
+            continue;
+        const key = basenameKey(reference);
+        const enabled = isEnabled(`<x ${attrs}>`);
+        const hasPath = /[\\/]/.test(reference);
+        const existing = byFile.get(key);
+        if (existing) {
+            if (!existing.references.includes(reference))
+                existing.references.push(reference);
+            existing.enabled ||= enabled;
+            existing.hasPath ||= hasPath;
+            continue;
+        }
+        byFile.set(key, {
+            filename: basename(reference),
+            references: [reference],
+            enabled,
+            hasPath,
+            ...(known ? { uploaded: known.has(key) } : {}),
+        });
+    }
+    return [...byFile.values()];
+}
+/** Listeners the control plane switches off in the deployed copy. */
+const DISABLED_AT_DEPLOY = {
+    ViewResultsFullVisualizer: 'View Results Tree',
+    SimpleDataWriter: 'Simple Data Writer',
+};
+function readListeners(jmx) {
+    const out = [];
+    const re = /<ResultCollector\b([^>]*)>/g;
+    let m;
+    while ((m = re.exec(jmx)) !== null) {
+        const tag = m[0];
+        const guiclass = attr(tag, 'guiclass');
+        if (!(guiclass in DISABLED_AT_DEPLOY))
+            continue;
+        out.push({
+            guiclass,
+            name: attr(tag, 'testname') || DISABLED_AT_DEPLOY[guiclass],
+            enabled: isEnabled(tag),
+        });
+    }
+    return out;
+}
+/**
+ * Whether Atlas can attach its reporting.
+ *
+ * It is inserted as the last child of the Test Plan's hashTree, which needs the
+ * file to end in the standard way. A plan that fails this runs and produces
+ * nothing — no dashboard, no results — so it is an error, not a nicety.
+ */
+function reportingCanAttach(jmx) {
+    return /<\/hashTree>(\s*)<\/hashTree>(\s*)<\/jmeterTestPlan>(\s*)$/.test(jmx);
+}
+function checkPlan(jmx, uploaded) {
+    const issues = [];
+    const threadGroups = readThreadGroups(jmx);
+    const dataFiles = readDataFiles(jmx, uploaded);
+    const listeners = readListeners(jmx);
+    const canAttach = reportingCanAttach(jmx);
+    if (!/<jmeterTestPlan\b/.test(jmx)) {
+        issues.push({
+            severity: 'error',
+            code: 'not-a-test-plan',
+            message: 'This file is not a JMeter test plan.',
+        });
+    }
+    const opens = (jmx.match(/<hashTree>/g) ?? []).length;
+    const closes = (jmx.match(/<\/hashTree>/g) ?? []).length;
+    if (opens !== closes) {
+        issues.push({
+            severity: 'error',
+            code: 'unbalanced',
+            message: `The plan's structure is broken: ${opens} <hashTree> against ${closes} closing tags.`,
+            fix: 'Restore the file and redo the edit — JMeter will not open it in this state.',
+        });
+    }
+    if (!canAttach) {
+        issues.push({
+            severity: 'error',
+            code: 'reporting-cannot-attach',
+            message: 'Atlas cannot attach its reporting to this plan, so a run would produce no results.',
+            fix: 'Open and re-save the plan in JMeter so it ends with the standard structure.',
+        });
+    }
+    const enabledGroups = threadGroups.filter(g => g.enabled);
+    if (threadGroups.length === 0) {
+        issues.push({
+            severity: 'error',
+            code: 'no-thread-groups',
+            message: 'The plan has no thread groups, so it would run nothing.',
+        });
+    }
+    else if (enabledGroups.length === 0) {
+        issues.push({
+            severity: 'error',
+            code: 'all-thread-groups-disabled',
+            message: `All ${threadGroups.length} thread group(s) are disabled, so the run would do nothing.`,
+            fix: 'Enable the one you mean to run.',
+        });
+    }
+    for (const file of dataFiles.filter(f => f.enabled)) {
+        const shown = file.references[0];
+        if (file.uploaded === false) {
+            // The one that hurts: every thread in the group fails before sampling, so
+            // the group disappears from the results with no errors to explain it.
+            issues.push({
+                severity: 'error',
+                code: 'data-file-missing',
+                message: `"${file.filename}" is read by the plan but is not in the project.`,
+                fix: `Upload it: atlas files upload ${file.filename}`,
+            });
+        }
+        else if (file.uploaded && file.hasPath) {
+            issues.push({
+                severity: 'info',
+                code: 'data-file-path',
+                message: `"${shown}" is a path; Atlas will use the uploaded "${file.filename}".`,
+            });
+        }
+        else if (file.uploaded === undefined && file.hasPath) {
+            issues.push({
+                severity: 'warning',
+                code: 'data-file-path-unchecked',
+                message: `"${shown}" is a path from another machine.`,
+                fix: `Make sure "${file.filename}" is uploaded to the project; then Atlas resolves it for you.`,
+            });
+        }
+    }
+    const liveListeners = listeners.filter(l => l.enabled);
+    if (liveListeners.length) {
+        issues.push({
+            severity: 'info',
+            code: 'gui-listeners',
+            message: `${liveListeners.map(l => `"${l.name}"`).join(', ')} will be switched off for the run.`,
+        });
+    }
+    if (enabledGroups.length && enabledGroups.every(g => !g.sizeProperty)) {
+        issues.push({
+            severity: 'warning',
+            code: 'fixed-thread-count',
+            message: 'User counts are fixed inside the plan, so the only way to scale it is more machines.',
+            fix: 'Use ${__P(threads,N)} for the thread count to size the run with -Jthreads.',
+        });
+    }
+    else {
+        // Say which property, because it is whatever the plan's author chose.
+        const props = [...new Set(enabledGroups.map(g => g.sizeProperty).filter(Boolean))];
+        if (props.length) {
+            issues.push({
+                severity: 'info',
+                code: 'sizable',
+                message: `Size this run with ${props.map(p => `-J${p}=<n>`).join(' ')}.`,
+            });
+        }
+    }
+    return {
+        threadGroups,
+        dataFiles,
+        listeners,
+        reportingCanAttach: canAttach,
+        issues,
+        ok: !issues.some(i => i.severity === 'error'),
+    };
+}
+//# sourceMappingURL=plan.js.map