@xano/cli 1.1.0 → 1.2.0

package/README.md

@@ -487,7 +487,7 @@ Every `run_all` command takes `--concurrency` (default `1`):
 
 ```bash
 xano unit_test run_all --concurrency 4
-xano tenant unit_test run_all my-tenant --concurrency 8
+xano tenant unit_test run_all --tenant my-tenant --concurrency 8
 ```
 
 Output stays in test order regardless of concurrency, so a parallel run is still diffable against a sequential one, and the JSON `results` array keeps a stable order.
@@ -496,6 +496,48 @@ Output stays in test order regardless of concurrency, so a parallel run is still
 
 **The default is 1 on purpose.** These tests execute against a shared workspace database, so tests that touch the same tables or records can interfere with each other when run at the same time — producing failures that look like flakiness rather than a real regression. Raise `--concurrency` once you know your tests are independent.
 
+#### Exit codes
+
+Every `run` and `run_all` command — on all four surfaces (`unit_test`, `workflow_test`, `sandbox`, `tenant`) — uses the same exit codes:
+
+| Code | Meaning |
+|------|---------|
+| `0` | All tests passed, **or** there were no tests to run |
+| `1` | At least one test failed |
+| `2` | The command could not run the tests at all — bad workspace, auth failure, unreadable profile, or the test-list request failed |
+
+**This is identical for `-o json` and summary output.** Adding `-o json` to get machine-readable output does not change the exit code, so a CI job can rely on it in either mode.
+
+```bash
+xano workflow_test run_all -o json || echo "tests failed or the command errored"
+```
+
+**One asymmetry worth knowing about.** In `run`, a non-2xx response from the run endpoint is a CLI error and exits `2`. In `run_all`, a non-2xx response for an *individual* test is recorded as a failed test result with an `API error <status>` message and contributes to exit `1` — because `run_all`'s job is to finish the batch and report a roll-up rather than abort on one test's transport error. Exit `2` in `run_all` is reserved for failures that stop the batch from starting at all.
+
+If you need to tell a backend outage apart from a genuine assertion failure, the exit code alone will not do it — inspect `results[].message` for the `API error` prefix:
+
+```bash
+xano workflow_test run_all -o json | jq '[.results[] | select(.message // "" | startswith("API error"))] | length'
+```
+
+#### JSON output
+
+`run` returns the API's result object unchanged. Treat **any** `status` other than `"ok"` as a failure — several distinct non-`ok` values are returned in practice, so match on "not `ok`" rather than enumerating known failure values:
+
+```json
+{ "status": "exception", "timing": 0.02, "message": "Precondition failed." }
+```
+
+`run_all` returns a roll-up. A branch with no tests is a normal state — freshly created branches routinely have none — and still emits valid JSON rather than plain text, so `JSON.parse` is always safe:
+
+```json
+{ "passed": 0, "failed": 0, "results": [], "total_timing": 0 }
+```
+
+The `total_timing` field is currently emitted by `workflow_test run_all` only; the `unit_test`, `tenant`, and `sandbox` variants omit it. The empty-suite envelope always carries the same keys as that command's populated envelope.
+
+> **Changed in this release:** a failing `run` in summary mode previously exited `2` and printed a spurious `EEXIT: 1` line; it now exits `1` with clean output. A failing `run` with `-o json` previously exited `0`, which silently hid failures from CI. If you match on exit code `2` specifically to detect test failures, switch to a non-zero check.
+
 ### Tenants
 
 Manage tenants, their environment variables, backups, deployments, and clusters.

package/dist/commands/sandbox/unit_test/run/index.js

@@ -46,24 +46,30 @@ Result: PASS
                 this.error(message);
             }
             const result = (await response.json());
+            // Anything other than 'ok' is a failure -- the API has been observed
+            // returning several distinct non-ok statuses, so never enumerate them.
+            const passed = result.status === 'ok';
             if (flags.output === 'json') {
                 this.log(JSON.stringify(result, null, 2));
             }
+            else if (passed) {
+                this.log('Result: PASS');
+            }
             else {
-                if (result.status === 'ok') {
-                    this.log('Result: PASS');
-                }
-                else {
-                    this.log('Result: FAIL');
-                    const failedExpects = result.results?.filter((r) => r.status === 'fail') ?? [];
-                    for (const expect of failedExpects) {
-                        if (expect.message) {
-                            this.log(`  Error: ${expect.message}`);
-                        }
+                this.log('Result: FAIL');
+                const failedExpects = result.results?.filter((r) => r.status === 'fail') ?? [];
+                for (const expect of failedExpects) {
+                    if (expect.message) {
+                        this.log(`  Error: ${expect.message}`);
                     }
-                    this.exit(1);
                 }
             }
+            // Set the code rather than throwing via this.exit(): the exit must apply
+            // in both output modes, and process.exitCode cannot be swallowed by the
+            // surrounding catch. Mirrors what run_all already does.
+            if (!passed) {
+                process.exitCode = 1;
+            }
         }
         catch (error) {
             if (error instanceof Error && 'oclif' in error)

package/dist/commands/sandbox/unit_test/run_all/index.js

@@ -78,7 +78,15 @@ Results: 4 passed, 1 failed
                 },
             });
             if (tests.length === 0) {
-                this.log('No unit tests found');
+                // An empty suite is a normal state (fresh branches have no tests) and
+                // is a success, not a failure. Under -o json it must still be parseable,
+                // with the same keys the populated path emits below.
+                if (flags.output === 'json') {
+                    this.log(JSON.stringify({ failed: 0, passed: 0, results: [] }, null, 2));
+                }
+                else {
+                    this.log('No unit tests found');
+                }
                 return;
             }
             if (flags.output === 'summary') {

package/dist/commands/sandbox/workflow_test/run/index.js

@@ -46,22 +46,25 @@ Result: PASS (0.25s)
                 this.error(message);
             }
             const result = (await response.json());
+            // Anything other than 'ok' is a failure -- the API has been observed
+            // returning several distinct non-ok statuses, so never enumerate them.
+            const passed = result.status === 'ok';
             if (flags.output === 'json') {
                 this.log(JSON.stringify(result, null, 2));
             }
             else {
                 const timing = result.timing ? ` (${result.timing}s)` : '';
-                if (result.status === 'ok') {
-                    this.log(`Result: PASS${timing}`);
-                }
-                else {
-                    this.log(`Result: FAIL${timing}`);
-                    if (result.message) {
-                        this.log(`  Error: ${result.message}`);
-                    }
-                    this.exit(1);
+                this.log(`Result: ${passed ? 'PASS' : 'FAIL'}${timing}`);
+                if (!passed && result.message) {
+                    this.log(`  Error: ${result.message}`);
                 }
             }
+            // Set the code rather than throwing via this.exit(): the exit must apply
+            // in both output modes, and process.exitCode cannot be swallowed by the
+            // surrounding catch. Mirrors what run_all already does.
+            if (!passed) {
+                process.exitCode = 1;
+            }
         }
         catch (error) {
             if (error instanceof Error && 'oclif' in error)

package/dist/commands/sandbox/workflow_test/run_all/index.js

@@ -71,7 +71,15 @@ Results: 2 passed, 1 failed
                 },
             });
             if (tests.length === 0) {
-                this.log('No workflow tests found');
+                // An empty suite is a normal state (fresh branches have no tests) and
+                // is a success, not a failure. Under -o json it must still be parseable,
+                // with the same keys the populated path emits below.
+                if (flags.output === 'json') {
+                    this.log(JSON.stringify({ failed: 0, passed: 0, results: [] }, null, 2));
+                }
+                else {
+                    this.log('No workflow tests found');
+                }
                 return;
             }
             if (flags.output === 'summary') {

package/dist/commands/tenant/unit_test/run/index.js

@@ -68,26 +68,34 @@ Result: PASS
                 this.error(`API request failed with status ${response.status}: ${response.statusText}\n${errorText}`);
             }
             const result = (await response.json());
+            // Anything other than 'ok' is a failure -- the API has been observed
+            // returning several distinct non-ok statuses, so never enumerate them.
+            const passed = result.status === 'ok';
             if (flags.output === 'json') {
                 this.log(JSON.stringify(result, null, 2));
             }
+            else if (passed) {
+                this.log('Result: PASS');
+            }
             else {
-                if (result.status === 'ok') {
-                    this.log('Result: PASS');
-                }
-                else {
-                    this.log('Result: FAIL');
-                    const failedExpects = result.results?.filter((r) => r.status === 'fail') ?? [];
-                    for (const expect of failedExpects) {
-                        if (expect.message) {
-                            this.log(`  Error: ${expect.message}`);
-                        }
+                this.log('Result: FAIL');
+                const failedExpects = result.results?.filter((r) => r.status === 'fail') ?? [];
+                for (const expect of failedExpects) {
+                    if (expect.message) {
+                        this.log(`  Error: ${expect.message}`);
                     }
-                    this.exit(1);
                 }
             }
+            // Set the code rather than throwing via this.exit(): the exit must apply
+            // in both output modes, and process.exitCode cannot be swallowed by the
+            // surrounding catch. Mirrors what run_all already does.
+            if (!passed) {
+                process.exitCode = 1;
+            }
         }
         catch (error) {
+            if (error instanceof Error && 'oclif' in error)
+                throw error;
             if (error instanceof Error) {
                 this.error(`Failed to run unit test: ${error.message}`);
             }

package/dist/commands/tenant/unit_test/run_all/index.js

@@ -95,7 +95,15 @@ Results: 4 passed, 1 failed
                 },
             });
             if (tests.length === 0) {
-                this.log('No unit tests found');
+                // An empty suite is a normal state (fresh branches have no tests) and
+                // is a success, not a failure. Under -o json it must still be parseable,
+                // with the same keys the populated path emits below.
+                if (flags.output === 'json') {
+                    this.log(JSON.stringify({ failed: 0, passed: 0, results: [] }, null, 2));
+                }
+                else {
+                    this.log('No unit tests found');
+                }
                 return;
             }
             if (flags.output === 'summary') {
@@ -162,6 +170,8 @@ Results: 4 passed, 1 failed
                     }
                 }
                 catch (error) {
+                    if (error instanceof Error && 'oclif' in error)
+                        throw error;
                     const message = error instanceof Error ? error.message : String(error);
                     results.push({
                         message,
@@ -202,6 +212,8 @@ Results: 4 passed, 1 failed
             }
         }
         catch (error) {
+            if (error instanceof Error && 'oclif' in error)
+                throw error;
             if (error instanceof Error) {
                 this.error(`Failed to run unit tests: ${error.message}`);
             }

package/dist/commands/tenant/workflow_test/run/index.js

@@ -61,24 +61,29 @@ Result: PASS (0.25s)
                 this.error(`API request failed with status ${response.status}: ${response.statusText}\n${errorText}`);
             }
             const result = (await response.json());
+            // Anything other than 'ok' is a failure -- the API has been observed
+            // returning several distinct non-ok statuses, so never enumerate them.
+            const passed = result.status === 'ok';
             if (flags.output === 'json') {
                 this.log(JSON.stringify(result, null, 2));
             }
             else {
                 const timing = result.timing ? ` (${result.timing}s)` : '';
-                if (result.status === 'ok') {
-                    this.log(`Result: PASS${timing}`);
-                }
-                else {
-                    this.log(`Result: FAIL${timing}`);
-                    if (result.message) {
-                        this.log(`  Error: ${result.message}`);
-                    }
-                    this.exit(1);
+                this.log(`Result: ${passed ? 'PASS' : 'FAIL'}${timing}`);
+                if (!passed && result.message) {
+                    this.log(`  Error: ${result.message}`);
                 }
             }
+            // Set the code rather than throwing via this.exit(): the exit must apply
+            // in both output modes, and process.exitCode cannot be swallowed by the
+            // surrounding catch. Mirrors what run_all already does.
+            if (!passed) {
+                process.exitCode = 1;
+            }
         }
         catch (error) {
+            if (error instanceof Error && 'oclif' in error)
+                throw error;
             if (error instanceof Error) {
                 this.error(`Failed to run workflow test: ${error.message}`);
             }

package/dist/commands/tenant/workflow_test/run_all/index.js

@@ -85,7 +85,15 @@ Results: 2 passed, 1 failed
                 },
             });
             if (tests.length === 0) {
-                this.log('No workflow tests found');
+                // An empty suite is a normal state (fresh branches have no tests) and
+                // is a success, not a failure. Under -o json it must still be parseable,
+                // with the same keys the populated path emits below.
+                if (flags.output === 'json') {
+                    this.log(JSON.stringify({ failed: 0, passed: 0, results: [] }, null, 2));
+                }
+                else {
+                    this.log('No workflow tests found');
+                }
                 return;
             }
             if (flags.output === 'summary') {
@@ -146,6 +154,8 @@ Results: 2 passed, 1 failed
                     }
                 }
                 catch (error) {
+                    if (error instanceof Error && 'oclif' in error)
+                        throw error;
                     const message = error instanceof Error ? error.message : String(error);
                     results.push({
                         message,
@@ -184,6 +194,8 @@ Results: 2 passed, 1 failed
             }
         }
         catch (error) {
+            if (error instanceof Error && 'oclif' in error)
+                throw error;
             if (error instanceof Error) {
                 this.error(`Failed to run workflow tests: ${error.message}`);
             }

package/dist/commands/unit_test/run/index.js

@@ -63,26 +63,34 @@ Result: PASS
                 this.error(`API request failed with status ${response.status}: ${response.statusText}\n${errorText}`);
             }
             const result = await response.json();
+            // Anything other than 'ok' is a failure -- the API has been observed
+            // returning several distinct non-ok statuses, so never enumerate them.
+            const passed = result.status === 'ok';
             if (flags.output === 'json') {
                 this.log(JSON.stringify(result, null, 2));
             }
+            else if (passed) {
+                this.log('Result: PASS');
+            }
             else {
-                if (result.status === 'ok') {
-                    this.log('Result: PASS');
-                }
-                else {
-                    this.log('Result: FAIL');
-                    const failedExpects = result.results?.filter(r => r.status === 'fail') ?? [];
-                    for (const expect of failedExpects) {
-                        if (expect.message) {
-                            this.log(`  Error: ${expect.message}`);
-                        }
+                this.log('Result: FAIL');
+                const failedExpects = result.results?.filter(r => r.status === 'fail') ?? [];
+                for (const expect of failedExpects) {
+                    if (expect.message) {
+                        this.log(`  Error: ${expect.message}`);
                     }
-                    this.exit(1);
                 }
             }
+            // Set the code rather than throwing via this.exit(): the exit must apply
+            // in both output modes, and process.exitCode cannot be swallowed by the
+            // surrounding catch. Mirrors what run_all already does.
+            if (!passed) {
+                process.exitCode = 1;
+            }
         }
         catch (error) {
+            if (error instanceof Error && 'oclif' in error)
+                throw error;
             if (error instanceof Error) {
                 this.error(`Failed to run unit test: ${error.message}`);
             }

package/dist/commands/unit_test/run_all/index.js

@@ -91,7 +91,15 @@ Results: 2 passed, 1 failed
                 },
             });
             if (tests.length === 0) {
-                this.log('No unit tests found');
+                // An empty suite is a normal state (fresh branches have no tests) and
+                // is a success, not a failure. Under -o json it must still be parseable,
+                // with the same keys the populated path emits below.
+                if (flags.output === 'json') {
+                    this.log(JSON.stringify({ failed: 0, passed: 0, results: [] }, null, 2));
+                }
+                else {
+                    this.log('No unit tests found');
+                }
                 return;
             }
             if (flags.output === 'summary') {
@@ -161,6 +169,8 @@ Results: 2 passed, 1 failed
                     }
                 }
                 catch (error) {
+                    if (error instanceof Error && 'oclif' in error)
+                        throw error;
                     const message = error instanceof Error ? error.message : String(error);
                     results.push({
                         message,
@@ -202,6 +212,8 @@ Results: 2 passed, 1 failed
             }
         }
         catch (error) {
+            if (error instanceof Error && 'oclif' in error)
+                throw error;
             if (error instanceof Error) {
                 this.error(`Failed to run unit tests: ${error.message}`);
             }

package/dist/commands/workflow_test/run/index.js

@@ -55,24 +55,29 @@ Result: PASS (1.234s)
                 this.error(`API request failed with status ${response.status}: ${response.statusText}\n${errorText}`);
             }
             const result = await response.json();
+            // Anything other than 'ok' is a failure -- the API has been observed
+            // returning several distinct non-ok statuses, so never enumerate them.
+            const passed = result.status === 'ok';
             if (flags.output === 'json') {
                 this.log(JSON.stringify(result, null, 2));
             }
             else {
-                const timing = `(${result.timing.toFixed(3)}s)`;
-                if (result.status === 'ok') {
-                    this.log(`Result: PASS ${timing}`);
-                }
-                else {
-                    this.log(`Result: FAIL ${timing}`);
-                    if (result.message) {
-                        this.log(`  Error: ${result.message}`);
-                    }
-                    this.exit(1);
+                const timing = typeof result.timing === 'number' ? ` (${result.timing.toFixed(3)}s)` : '';
+                this.log(`Result: ${passed ? 'PASS' : 'FAIL'}${timing}`);
+                if (!passed && result.message) {
+                    this.log(`  Error: ${result.message}`);
                 }
             }
+            // Set the code rather than throwing via this.exit(): the exit must apply
+            // in both output modes, and process.exitCode cannot be swallowed by the
+            // surrounding catch. Mirrors what run_all already does.
+            if (!passed) {
+                process.exitCode = 1;
+            }
         }
         catch (error) {
+            if (error instanceof Error && 'oclif' in error)
+                throw error;
             if (error instanceof Error) {
                 this.error(`Failed to run workflow test: ${error.message}`);
             }

package/dist/commands/workflow_test/run_all/index.js

@@ -82,7 +82,15 @@ Results: 2 passed, 1 failed (2.691s total)
                 },
             });
             if (tests.length === 0) {
-                this.log('No workflow tests found');
+                // An empty suite is a normal state (fresh branches have no tests) and
+                // is a success, not a failure. Under -o json it must still be parseable,
+                // with the same keys the populated path emits below.
+                if (flags.output === 'json') {
+                    this.log(JSON.stringify({ failed: 0, passed: 0, results: [], total_timing: 0 }, null, 2));
+                }
+                else {
+                    this.log('No workflow tests found');
+                }
                 return;
             }
             if (flags.output === 'summary') {
@@ -134,9 +142,9 @@ Results: 2 passed, 1 failed (2.691s total)
                         timing: runResult.timing,
                     };
                     results.push(result);
-                    totalTiming += runResult.timing;
+                    totalTiming += runResult.timing ?? 0;
                     if (flags.output === 'summary') {
-                        const timing = `(${runResult.timing.toFixed(3)}s)`;
+                        const timing = `(${(runResult.timing ?? 0).toFixed(3)}s)`;
                         if (passed) {
                             log(`PASS  ${test.name} ${timing}`);
                         }
@@ -149,6 +157,8 @@ Results: 2 passed, 1 failed (2.691s total)
                     }
                 }
                 catch (error) {
+                    if (error instanceof Error && 'oclif' in error)
+                        throw error;
                     const message = error instanceof Error ? error.message : String(error);
                     results.push({
                         message,
@@ -189,6 +199,8 @@ Results: 2 passed, 1 failed (2.691s total)
             }
         }
         catch (error) {
+            if (error instanceof Error && 'oclif' in error)
+                throw error;
             if (error instanceof Error) {
                 this.error(`Failed to run workflow tests: ${error.message}`);
             }