@memberjunction/cli 5.37.0 → 5.39.0

package/dist/init-templates/generic-web/README.md

@@ -0,0 +1,115 @@
+# Generic Web Example — Mode C Reference
+
+A working **Mode C** (non-MJ web app) scaffold that drives the MJ regression
+runner against [MDN Web Docs](https://developer.mozilla.org). Three Computer
+Use tests plus one **custom IOracle** plugged in via `--oracles-module`.
+
+Use this directory as the starting point when authoring tests for your own web
+app: copy it, edit `target.json` to point at your URL, replace the three test
+JSONs, and tweak (or remove) the custom oracle.
+
+## What's in here
+
+```
+generic-web/
+├── README.md                                  # this file
+├── target.json                                # Mode C target profile (kind: generic-web)
+├── metadata/
+│   ├── .mj-sync.json                          # directoryOrder = [tests, test-suites]
+│   ├── tests/
+│   │   ├── .mj-sync.json                      # entity = "MJ: Tests"
+│   │   ├── .GW01-home-loads.json              # MDN home page smoke
+│   │   ├── .GW02-search-array-map.json        # search-flow test
+│   │   └── .GW03-navigate-references.json     # top-nav navigation test
+│   └── test-suites/
+│       ├── .mj-sync.json                      # entity = "MJ: Test Suites"
+│       └── .generic-web-suite.json            # binds the 3 tests
+└── oracles/
+    └── mdn-oracles.cjs                        # custom IOracle exports
+```
+
+## The custom oracle module
+
+[`oracles/mdn-oracles.cjs`](oracles/mdn-oracles.cjs) exports two oracles in two
+different styles to show both supported patterns:
+
+| Type | Export style | Purpose |
+|---|---|---|
+| `final-url-host` | `class FinalUrlHostOracle implements IOracle` | Verifies the test ended on a specific hostname (catches off-domain wanders). Used by GW02 + GW03. |
+| `min-step-count` | object instance `{ type, evaluate }` | Fails when the agent finished too fast (likely a silent navigation error). Available to any test but not currently referenced — feel free to wire it in. |
+
+The loader (`packages/TestingFramework/CLI/src/utils/oracle-module-loader.ts`)
+duck-types each export: if it has `type: string` + async `evaluate()`, it
+gets registered. Anything else is silently skipped.
+
+## Running it
+
+### Inside Docker (Mode C via `mj test regression remote`)
+
+From the MJ monorepo root:
+
+```bash
+mj test regression remote --target=docker/regression/examples/generic-web/target.json
+```
+
+What happens:
+
+1. The CLI loads [`target.json`](target.json), resolves any `env:` refs (none
+   in this example), and exports `MJ_TEST_VAR_baseUrl=https://developer.mozilla.org`,
+   `MJ_TEST_VAR_allowedDomains=["developer.mozilla.org","*.mozilla.org"]`,
+   `TEST_SUITE_NAME="Generic Web Example Suite"`,
+   `EXTRA_METADATA_DIRS=/app/docker/regression/examples/generic-web/metadata`,
+   `ORACLES_MODULE=/app/docker/regression/examples/generic-web/oracles/mdn-oracles.cjs`,
+   plus the entrypoint dispatch (`TEST_RUNNER_ENTRYPOINT=test-runner-remote-entrypoint.sh`).
+2. Compose brings up the full profile so the test-runner has a local
+   ephemeral DB for `TestRun` / `TestRunOutput` rows (the remote URL itself
+   isn't MJ — the DB is just for results recording).
+3. The remote entrypoint pushes the example's metadata to the DB, then
+   invokes `mj test suite --name "Generic Web Example Suite"
+   --oracles-module=/app/.../mdn-oracles.cjs --parallel --max-parallel 4`.
+4. Reports land in `docker/regression/test-results/run-{TIMESTAMP}/`.
+
+### Locally (against your dev DB) for fast iteration
+
+If you're authoring tests and don't want the Docker round-trip every time,
+push the metadata to your local MJ DB and run the suite directly:
+
+```bash
+# 1. Push the example's test + suite metadata
+npx mj sync push --dir=docker/regression/examples/generic-web/metadata
+
+# 2. Run the suite with the custom oracle module + variable substitutions
+mj test suite \
+    --name "Generic Web Example Suite" \
+    --oracles-module=docker/regression/examples/generic-web/oracles/mdn-oracles.cjs \
+    --var baseUrl=https://developer.mozilla.org \
+    --var 'allowedDomains=["developer.mozilla.org","*.mozilla.org"]' \
+    --parallel --max-parallel 3
+```
+
+The `--var` flags drive the `{{baseUrl}}` / `{{allowedDomains}}` substitution
+inside the test JSONs. JSON-shaped values (the array) work because the
+substitution layer auto-parses them.
+
+## Copying this example for your own app
+
+1. **Copy the directory** to `docker/regression/examples/my-app/` (or anywhere
+   outside the monorepo, then mount it).
+2. **Edit `target.json`**:
+   - Set `baseUrl` to your app's URL.
+   - Set `allowedDomains` to the hosts the agent is allowed to touch.
+   - Add an `auth` block if your app requires login (see
+     [`docker/regression/targets/staging-mj.example.target.json`](../../../../../docker/regression/targets/staging-mj.example.target.json)
+     for the auth shape).
+   - Rename `suite` to match your suite, update `extraMetadataDirs` to your
+     metadata path, point `oraclesModule` at your oracles file.
+3. **Replace the tests** in `metadata/tests/` with goals specific to your app.
+   Reference `{{baseUrl}}` (and any custom vars you add) inside `startUrl`,
+   oracle `pattern` configs, etc.
+4. **Edit the suite** in `metadata/test-suites/.generic-web-suite.json` to
+   list your tests by name.
+5. **Write your oracles** in `oracles/*.cjs`. The duck-type contract is:
+   `{ type: string, async evaluate(input, config) → OracleResult }`.
+
+See [`../bring-your-own-app/`](../bring-your-own-app/) for an example that
+also boots its own app inside the regression network (Mode D).

package/dist/init-templates/generic-web/metadata/.mj-sync.json

@@ -0,0 +1,10 @@
+{
+  "version": "1.0.0",
+  "push": {
+    "autoCreateMissingRecords": true
+  },
+  "directoryOrder": [
+    "tests",
+    "test-suites"
+  ]
+}

package/dist/init-templates/generic-web/metadata/test-suites/.generic-web-suite.json

@@ -0,0 +1,48 @@
+{
+  "fields": {
+    "Name": "Generic Web Example Suite",
+    "Description": "Mode C reference suite — runs 3 tests against MDN Web Docs to demonstrate variable templating ({{baseUrl}}, {{allowedDomains}}) and a custom IOracle plugged in via `mj test suite --oracles-module=…`. Used by the docker/regression/examples/generic-web/ scaffold.",
+    "Status": "Active"
+  },
+  "relatedEntities": {
+    "MJ: Test Suite Tests": [
+      {
+        "fields": {
+          "SuiteID": "@parent:ID",
+          "TestID": "@lookup:MJ: Tests.Name=GW01 - MDN Home Loads",
+          "Sequence": 1,
+          "Status": "Active"
+        },
+        "primaryKey": {
+          "ID": "EE53002B-DCD5-49AC-AE16-211254BCE6FE"
+        },
+        "sync": {
+          "lastModified": "2026-05-17T23:05:53.394Z",
+          "checksum": "09664d1aa1dba6101156b1ad5cb56893aa025fee4b4569ecd1f3e86325617d6f"
+        }
+      },
+      {
+        "fields": {
+          "SuiteID": "@parent:ID",
+          "TestID": "@lookup:MJ: Tests.Name=GW02 - Search Array.prototype.map",
+          "Sequence": 2,
+          "Status": "Active"
+        },
+        "primaryKey": {
+          "ID": "DB858A9C-BA30-40E1-9FAA-1452A48611F2"
+        },
+        "sync": {
+          "lastModified": "2026-05-17T23:05:53.408Z",
+          "checksum": "db7b7b0e7106adcae9bcbf9d1c06c2d4175b70fad2dd6d2282ea44d9d93e3cd4"
+        }
+      }
+    ]
+  },
+  "primaryKey": {
+    "ID": "3C13E3D6-F022-409A-A6FE-B5857BCE7FA4"
+  },
+  "sync": {
+    "lastModified": "2026-05-17T23:05:53.359Z",
+    "checksum": "b6ac99fec8087fb090387e0869228705c4ed7e36dcae2ef667e729ede0101bba"
+  }
+}

package/dist/init-templates/generic-web/metadata/test-suites/.mj-sync.json

@@ -0,0 +1,12 @@
+{
+  "entity": "MJ: Test Suites",
+  "filePattern": "**/.*.json",
+  "pull": {
+    "relatedEntities": {
+      "MJ: Test Suite Tests": {
+        "entity": "MJ: Test Suite Tests",
+        "foreignKey": "TestSuiteID"
+      }
+    }
+  }
+}

package/dist/init-templates/generic-web/metadata/tests/.GW01-home-loads.json

@@ -0,0 +1,74 @@
+{
+  "fields": {
+    "TypeID": "@lookup:MJ: Test Types.Name=Computer Use",
+    "Name": "GW01 - MDN Home Loads",
+    "Description": "Navigates to the MDN Web Docs home page and verifies the landing page renders with its main navigation. Demonstrates Mode C variable templating ({{baseUrl}}) — no MJ assumptions, no auth.",
+    "InputDefinition": {
+      "goal": "Verify the home page has loaded successfully. Look for the main navigation header (References, Guides, Plus) and the prominent 'MDN Web Docs' branding. Do NOT click anything — just confirm the page rendered without errors and the nav elements are visible.",
+      "startUrl": "{{baseUrl}}",
+      "allowedDomains": "{{allowedDomains}}",
+      "auth": {
+        "bindings": []
+      }
+    },
+    "ExpectedOutcomes": {
+      "goalCompleted": true,
+      "finalUrlPattern": "^https?://(www\\.)?developer\\.mozilla\\.org/?",
+      "minConfidence": 0.7,
+      "maxSteps": 10,
+      "judgeValidationCriteria": [
+        "Page rendered without errors",
+        "Main navigation visible (HTML, CSS, Learn, Tools, etc)", 
+        "MDN branding is present"
+      ]
+    },
+    "Configuration": {
+      "headless": true,
+      "maxSteps": 10,
+      "maxExecutionTime": 120000,
+      "screenshotHistoryDepth": 3,
+      "viewportWidth": 1280,
+      "viewportHeight": 720,
+      "controllerPromptName": "Computer Use - Controller",
+      "judgePromptName": "Computer Use - Judge",
+      "judgeFrequency": "EveryNSteps:3",
+      "oracles": [
+        {
+          "type": "goal-completion",
+          "weight": 0.5,
+          "config": {
+            "minConfidence": 0.7
+          }
+        },
+        {
+          "type": "url-match",
+          "weight": 0.3,
+          "config": {
+            "pattern": "^https?://(www\\.)?developer\\.mozilla\\.org/?"
+          }
+        },
+        {
+          "type": "step-count",
+          "weight": 0.2,
+          "config": {
+            "maxSteps": 10
+          }
+        }
+      ],
+      "scoringWeights": {
+        "goal-completion": 0.5,
+        "url-match": 0.3,
+        "step-count": 0.2
+      }
+    },
+    "Status": "Active",
+    "Tags": "P0,generic-web,smoke"
+  },
+  "primaryKey": {
+    "ID": "C21C91DF-8EC8-4741-9251-EB14187A2532"
+  },
+  "sync": {
+    "lastModified": "2026-05-17T23:05:53.299Z",
+    "checksum": "cd52d0df6d51b2d14f5559cd9c2bb8894b29f7343733673fd618687db8506876"
+  }
+}

package/dist/init-templates/generic-web/metadata/tests/.GW02-search-array-map.json

@@ -0,0 +1,82 @@
+{
+  "fields": {
+    "TypeID": "@lookup:MJ: Test Types.Name=Computer Use",
+    "Name": " GW02 - Search Array.prototype.map",
+    "Description": "Uses the search box to find documentation for Array.prototype.map and verifies a relevant result page loads. Exercises text input + click + page navigation against a public site.",
+    "InputDefinition": {
+      "goal": "Open the search box (top-right magnifying-glass icon or 'Search MDN' input). Type 'Array.prototype.map' and submit. Follow the most relevant result. Verify a page about Array.prototype.map (the JavaScript method) loads",
+      "startUrl": "{{baseUrl}}",
+      "allowedDomains": "{{allowedDomains}}",
+      "auth": {
+        "bindings": []
+      }
+    },
+    "ExpectedOutcomes": {
+      "goalCompleted": true,
+      "finalUrlPattern": "(?i)/array(/prototype)?/map",
+      "minConfidence": 0.7,
+      "maxSteps": 20,
+      "judgeValidationCriteria": [
+        "Final page is about Array.prototype.map",
+        "Page contains a code example",
+        "Page is a reference doc (not an unrelated search result)"
+      ]
+    },
+    "Configuration": {
+      "headless": true,
+      "maxSteps": 20,
+      "maxExecutionTime": 240000,
+      "screenshotHistoryDepth": 3,
+      "viewportWidth": 1280,
+      "viewportHeight": 720,
+      "controllerPromptName": "Computer Use - Controller",
+      "judgePromptName": "Computer Use - Judge",
+      "judgeFrequency": "EveryNSteps:3",
+      "oracles": [
+        {
+          "type": "goal-completion",
+          "weight": 0.4,
+          "config": {
+            "minConfidence": 0.7
+          }
+        },
+        {
+          "type": "url-match",
+          "weight": 0.3,
+          "config": {
+            "pattern": "(?i)/array(/prototype)?/map"
+          }
+        },
+        {
+          "type": "step-count",
+          "weight": 0.1,
+          "config": {
+            "maxSteps": 20
+          }
+        },
+        {
+          "type": "final-url-host",
+          "weight": 0.2,
+          "config": {
+            "host": "developer.mozilla.org"
+          }
+        }
+      ],
+      "scoringWeights": {
+        "goal-completion": 0.4,
+        "url-match": 0.3,
+        "step-count": 0.1,
+        "final-url-host": 0.2
+      }
+    },
+    "Status": "Active",
+    "Tags": "P1,generic-web,search"
+  },
+  "primaryKey": {
+    "ID": "2E8DE64D-B817-4E3D-B471-FF1142FABFDC"
+  },
+  "sync": {
+    "lastModified": "2026-05-17T23:05:53.319Z",
+    "checksum": "eeec8bd96e8ff50437a3a20c2a821dc991fcece9277c23510b635e5b6501b88b"
+  }
+}

package/dist/init-templates/generic-web/metadata/tests/.mj-sync.json

@@ -0,0 +1,12 @@
+{
+  "entity": "MJ: Tests",
+  "filePattern": "**/.*.json",
+  "pull": {
+    "lookupFields": {
+      "TypeID": {
+        "entity": "MJ: Test Types",
+        "field": "Name"
+      }
+    }
+  }
+}

package/dist/init-templates/generic-web/oracles/mdn-oracles.cjs

@@ -0,0 +1,121 @@
+/**
+ * Example custom-oracle module for the generic-web example.
+ *
+ * Loaded at suite startup via:
+ *   mj test suite --oracles-module=./oracles/mdn-oracles.cjs
+ *
+ * Each export that looks like an `IOracle` (has `type: string` + async
+ * `evaluate(input, config)`) gets registered on the TestEngine and becomes
+ * available to any test that references its `type` in `Configuration.oracles`.
+ *
+ * Supported export shapes (see packages/TestingFramework/CLI/src/utils/
+ * oracle-module-loader.ts for the duck-type check):
+ *   - A class implementing IOracle (instantiated with no args)
+ *   - An object instance with `{ type: '...', async evaluate() { } }`
+ *
+ * This module demonstrates both shapes so adopters can copy whichever style
+ * they prefer.
+ */
+'use strict';
+
+/**
+ * FinalUrlHostOracle — class-export style.
+ *
+ * Validates that the test's final URL is on a specific host. Useful as a
+ * defensive check: if the agent wandered off-domain (e.g., chased an external
+ * link), the URL hostname won't match and this oracle fails the test.
+ *
+ * Config:
+ *   - host: string — the expected hostname (e.g., "developer.mozilla.org").
+ *   - allowSubdomains: boolean (default true) — accept "*.host" matches too.
+ */
+class FinalUrlHostOracle {
+    constructor() {
+        this.type = 'final-url-host';
+    }
+
+    async evaluate(input, config) {
+        const actual = (input && input.actualOutput) || {};
+        const finalUrl = actual.finalUrl || '';
+        const expectedHost = (config && config.host) || '';
+        const allowSubdomains = config && config.allowSubdomains !== false;
+
+        if (!expectedHost) {
+            return {
+                oracleType: this.type,
+                passed: false,
+                score: 0,
+                message: 'No host configured for final-url-host oracle',
+            };
+        }
+        if (!finalUrl) {
+            return {
+                oracleType: this.type,
+                passed: false,
+                score: 0,
+                message: 'No final URL recorded for the test',
+            };
+        }
+
+        let actualHost;
+        try {
+            actualHost = new URL(finalUrl).hostname.toLowerCase();
+        } catch (err) {
+            return {
+                oracleType: this.type,
+                passed: false,
+                score: 0,
+                message: `Final URL is not parseable: ${finalUrl}`,
+            };
+        }
+
+        const expected = expectedHost.toLowerCase();
+        const matches = allowSubdomains
+            ? actualHost === expected || actualHost.endsWith('.' + expected)
+            : actualHost === expected;
+
+        return {
+            oracleType: this.type,
+            passed: matches,
+            score: matches ? 1.0 : 0,
+            message: matches
+                ? `Final URL host matches: ${actualHost}`
+                : `Final URL host mismatch — expected ${expected}, got ${actualHost}`,
+            details: { actualHost, expectedHost: expected, finalUrl },
+        };
+    }
+}
+
+/**
+ * minStepCountOracle — instance-export style.
+ *
+ * Inverse of the built-in step-count oracle: fails when the agent finished too
+ * fast (e.g., immediately hit an error page and gave up). Useful for catching
+ * silent navigation failures.
+ *
+ * Config:
+ *   - minSteps: number — minimum acceptable step count (default 2).
+ */
+const minStepCountOracle = {
+    type: 'min-step-count',
+    async evaluate(input, config) {
+        const totalSteps =
+            (input && input.actualOutput && input.actualOutput.totalSteps) || 0;
+        const minSteps = (config && typeof config.minSteps === 'number') ? config.minSteps : 2;
+        const passed = totalSteps >= minSteps;
+        return {
+            oracleType: this.type,
+            passed,
+            score: passed ? 1.0 : 0,
+            message: passed
+                ? `Test took ${totalSteps} step(s) (≥ ${minSteps})`
+                : `Test finished too quickly: ${totalSteps} step(s) < min ${minSteps}`,
+            details: { totalSteps, minSteps },
+        };
+    },
+};
+
+module.exports = {
+    FinalUrlHostOracle,
+    minStepCountOracle,
+};

package/dist/init-templates/generic-web/target.json

@@ -0,0 +1,25 @@
+{
+  "schemaVersion": "1.0",
+  "name": "generic-web-mdn",
+  "kind": "generic-web",
+  "description": "Mode C reference target: drives the MJ regression runner against MDN Web Docs (a stable public site) with 3 example tests and one custom IOracle. Used to verify the generic-web flow end-to-end.",
+
+  "baseUrl": "https://developer.mozilla.org",
+  "allowedDomains": ["developer.mozilla.org", "*.mozilla.org"],
+
+  "auth": {
+    "bindings": []
+  },
+
+  "suite": "Generic Web Example Suite",
+  "extraMetadataDirs": [
+    "/app/docker/regression/examples/generic-web/metadata"
+  ],
+
+  "oraclesModule": "/app/docker/regression/examples/generic-web/oracles/mdn-oracles.cjs",
+
+  "archive": {
+    "tag": "generic-web-mdn",
+    "source": "mode-c-mdn"
+  }
+}

package/dist/init-templates/github-actions/README.md

@@ -0,0 +1,26 @@
+# GitHub Actions — run the regression suite from your own repo
+
+[`regression.yml`](regression.yml) is a copy-paste workflow that runs the suite
+against your MJ instance using the published `memberjunction/agentic-test-runner`
+image — **no MJ monorepo checkout**.
+
+## Use it
+
+1. Copy `regression.yml` to `.github/workflows/regression.yml` in your repo.
+2. Add your suite next to it: a `my-suite/target.json` (baseUrl + auth `env:`
+   refs + suite name) and `my-suite/metadata/` (tests + test-suites). Scaffold a
+   starting point with `mj test regression init remote-mj`.
+3. Add repo **secrets** (names match the root `.env` convention):
+   - `AI_VENDOR_API_KEY__GeminiLLM` (primary model) — and/or `AI_VENDOR_API_KEY__AnthropicLLM`
+   - `DB_HOST` / `DB_DATABASE` / `DB_USERNAME` / `DB_PASSWORD` — a reachable MJ
+     provider DB (your instance under test, or a throwaway) where test
+     definitions/prompts are pushed and results recorded.
+   - any auth vars your `target.json` references (e.g. `STAGING_TEST_USER`).
+4. Pin `IMAGE` to the MJ version your instance runs.
+
+Results (incl. `report.html` + screenshots) upload as the `regression-results`
+artifact. Diff locally with `mj test regression compare`.
+
+> This is Mode B/C (drive a URL). For Mode D (boot your app in CI alongside the
+> runner) add your app via `docker compose` and `mj test regression remote
+> --overlay`, or a services: block. See [REGRESSION_TESTING.md](../../../../../docker/regression/REGRESSION_TESTING.md).

package/dist/init-templates/github-actions/regression.yml

@@ -0,0 +1,52 @@
+# Copy-paste GitHub Actions workflow to run the MJ regression suite against
+# YOUR app/instance using the published agentic-test-runner image — no MJ
+# monorepo checkout required.
+#
+# Setup:
+#   1. Drop this file in your repo at .github/workflows/regression.yml
+#   2. Add a target.json + metadata/ (tests + test-suites) to your repo (see
+#      `mj test regression init remote-mj` or the examples/ directory).
+#   3. Add repo secrets: AI_VENDOR_API_KEY__GeminiLLM (primary; and/or
+#      __AnthropicLLM), DB_HOST/DB_DATABASE/DB_USERNAME/DB_PASSWORD (a reachable
+#      MJ provider DB), and any auth env: refs your target.json uses
+#      (e.g. STAGING_TEST_USER / STAGING_TEST_PASSWORD).
+#
+# Pin IMAGE to the MJ version your instance runs.
+name: regression
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 7 * * *'   # nightly
+
+env:
+  IMAGE: memberjunction/agentic-test-runner:latest   # pin to :v<MJ-version>
+
+jobs:
+  regression:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Run regression suite
+        run: |
+          docker run --rm \
+            -e DB_HOST="${{ secrets.DB_HOST }}" \
+            -e DB_DATABASE="${{ secrets.DB_DATABASE }}" \
+            -e DB_USERNAME="${{ secrets.DB_USERNAME }}" \
+            -e DB_PASSWORD="${{ secrets.DB_PASSWORD }}" \
+            -e DB_TRUST_SERVER_CERTIFICATE=true \
+            -e AI_VENDOR_API_KEY__GeminiLLM="${{ secrets.AI_VENDOR_API_KEY__GeminiLLM }}" \
+            -e AI_VENDOR_API_KEY__AnthropicLLM="${{ secrets.AI_VENDOR_API_KEY__AnthropicLLM }}" \
+            -e STAGING_TEST_USER="${{ secrets.STAGING_TEST_USER }}" \
+            -e STAGING_TEST_PASSWORD="${{ secrets.STAGING_TEST_PASSWORD }}" \
+            -v "${{ github.workspace }}/my-suite:/work:ro" \
+            -v "${{ github.workspace }}/test-results:/app/test-results" \
+            "$IMAGE" run --target=/work/target.json
+
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: regression-results
+          path: test-results/