@memberjunction/testing-cli 4.0.0 → 4.1.0

package/README.md

@@ -1,502 +1,186 @@
-# MemberJunction Testing CLI
+# @memberjunction/testing-cli
 
-Command-line interface for the MemberJunction Testing Framework. Provides a thin CLI layer on top of the Testing Engine, enabling test execution, management, and reporting from the command line.
+Command-line interface for the MemberJunction Testing Framework. Provides commands for running tests, managing suites, viewing history, comparing runs, generating reports, and validating configurations.
 
 ## Architecture
 
-The CLI is designed as a thin wrapper around the Testing Engine, ensuring feature parity between CLI and API:
+```mermaid
+graph TD
+    subgraph "@memberjunction/testing-cli"
+        A[CLI Entry Point] --> B[Commands]
+        B --> C[run]
+        B --> D[suite]
+        B --> E[list]
+        B --> F[history]
+        B --> G[compare]
+        B --> H[report]
+        B --> I[validate]
+
+        A --> J[Utilities]
+        J --> K[Config Loader]
+        J --> L[Output Formatter]
+        J --> M[Spinner Manager]
+        J --> N[Variable Parser]
+
+        A --> O[MJ Provider]
+    end
+
+    subgraph "Engine Layer"
+        P["@memberjunction/testing-engine<br/>(Execution)"]
+        Q["@memberjunction/testing-engine-base<br/>(Metadata)"]
+    end
+
+    C --> P
+    D --> P
+    E --> Q
+    F --> Q
+    G --> Q
+    I --> P
+
+    style A fill:#2d6a9f,stroke:#1a4971,color:#fff
+    style C fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style D fill:#2d8659,stroke:#1a5c3a,color:#fff
+    style E fill:#7c5295,stroke:#563a6b,color:#fff
+    style F fill:#7c5295,stroke:#563a6b,color:#fff
+    style G fill:#b8762f,stroke:#8a5722,color:#fff
+    style H fill:#b8762f,stroke:#8a5722,color:#fff
+    style I fill:#7c5295,stroke:#563a6b,color:#fff
+    style O fill:#2d6a9f,stroke:#1a4971,color:#fff
+```
+
+## Overview
+
+This package provides the CLI layer on top of the MemberJunction Testing Engine. It is designed to be used both standalone and as an integrated part of the `mj test` command group in the MemberJunction CLI.
+
+**Available commands:**
+
+| Command | Description |
+|---------|-------------|
+| `run` | Execute a single test by ID or name |
+| `suite` | Execute a test suite with parallel/sequential control |
+| `list` | List available tests, suites, and types |
+| `history` | View execution history for tests or suites |
+| `compare` | Compare results across multiple test runs |
+| `report` | Generate execution reports |
+| `validate` | Validate test or suite configurations without executing |
+
+## Installation
 
-```
-MJCLI (mj command)
-    ↓
-Testing CLI Commands (mj test *)
-    ↓
-Testing Engine (actual logic)
-    ↓
-Database / Entities
-```
-
-This architecture allows:
-- **CLI and API parity** - Same engine powers both interfaces
-- **Thin CLI layer** - Commands focus on user interaction, not business logic
-- **Testability** - Engine can be tested independently
-- **Flexibility** - Easy to add GraphQL/REST APIs later
-
-## Integration with MJCLI
-
-The Testing CLI integrates with the main MJCLI package following the same pattern as MetadataSync:
-
-**In MJCLI package** (`packages/MJCLI/src/commands/test/`):
-```typescript
-// Thin command that delegates to Testing CLI
-import { RunCommand } from '@memberjunction/testing-cli';
-
-export default class TestRun extends Command {
-  async run() {
-    // Minimal setup, delegate to testing CLI
-    await RunCommand.execute(this.parse);
-  }
-}
-```
-
-**In Testing CLI package** (`packages/TestingFramework/CLI/src/commands/`):
-```typescript
-// Full implementation
-export class RunCommand {
-  async execute(args, flags) {
-    // Use Testing Engine for actual work
-    const engine = new TestEngine();
-    const result = await engine.runTest(...);
-    // Format output for CLI
-    this.displayResults(result);
-  }
-}
+```bash
+npm install @memberjunction/testing-cli
 ```
 
 ## Commands
 
-### `mj test run`
-
-Run a single test or suite of tests.
+### run -- Execute a Test
 
 ```bash
-# Run specific test by ID
-mj test run <test-id>
-
-# Run test by name
-mj test run --name="Active Members Count"
-
-# Run test suite
-mj test run --suite=<suite-id>
-
-# Run with filtering
-mj test run --tag=smoke
-mj test run --category=agent-eval
-mj test run --difficulty=easy
-
-# Run all tests
-mj test run --all
+# Run by test name
+mj test run --name "Agent Summarization Test"
 
-# Dry run (validate without executing)
-mj test run <test-id> --dry-run
+# Run by test ID
+mj test run --id "test-guid"
 
-# Specify environment
-mj test run <test-id> --environment=staging
+# With variables
+mj test run --name "My Test" --var AIConfiguration=gpt-4o --var Temperature=0.3
 
-# Output formats
-mj test run <test-id> --format=json
-mj test run <test-id> --format=markdown
-mj test run <test-id> --format=console (default)
+# Verbose output
+mj test run --name "My Test" --verbose
 
-# Save results to file
-mj test run <test-id> --output=results.json
+# Dry run (validate only)
+mj test run --name "My Test" --dry-run
 ```
 
-**Example Output:**
-```
-[TEST_START] Active Members Count
-[TYPE] Agent Eval
-[CONTEXT] {"testId": "abc-123", "environment": "staging"}
-
-[STEP 1/3] Execute Agent
-[INPUT] "How many active members do we have?"
-[AGENT] Skip Analytics Agent
-[DURATION] 1.8s
-✓ Agent executed successfully
-
-[STEP 2/3] Run Oracles
-[ORACLE] trace-no-errors: PASSED ✓
-[ORACLE] sql-validate: COUNT(*) = 402 (expected 380-420) ✓
-[ORACLE] schema-validate: Output matches schema ✓
-
-[STEP 3/3] Calculate Score
-[PASSED_CHECKS] 3/3
-[SCORE] 1.0000 (100%)
-[COST] $0.0042 USD
-
-[TEST_PASS] Active Members Count
-[DURATION] 1.8s
-[TRACE_ID] Agent Run: abc-xyz-123
-```
-
-### `mj test suite`
-
-Run a complete test suite.
+### suite -- Execute a Suite
 
 ```bash
-# Run suite by ID
-mj test suite <suite-id>
-
-# Run suite by name
-mj test suite --name="Agent Quality Suite"
+# Run a suite
+mj test suite --name "Regression Suite"
 
 # Parallel execution
-mj test suite <suite-id> --parallel
+mj test suite --name "Suite" --parallel --max-parallel 5
 
-# Fail fast (stop on first failure)
-mj test suite <suite-id> --fail-fast
+# Stop on first failure
+mj test suite --name "Suite" --fail-fast
 
-# Specific sequence
-mj test suite <suite-id> --sequence=1,3,5
+# Run specific sequence range
+mj test suite --name "Suite" --sequence-start 3 --sequence-end 7
 ```
 
-**Example Output:**
-```
-[SUITE_START] Agent Quality Suite
-[TESTS] 15 tests queued
-
-[1/15] Active Members Count
-✓ PASSED (1.8s, score: 1.0000)
-
-[2/15] Revenue Year-to-Date
-✓ PASSED (2.1s, score: 0.9500)
-
-[3/15] Complex Aggregation
-✗ FAILED (3.2s, score: 0.6000)
-  - Oracle 'llm-judge' failed: Component quality below threshold
-
-...
-
-[SUITE_COMPLETE] Agent Quality Suite
-[SUMMARY] 13/15 passed (86.7%)
-[DURATION] 28.4s
-[COST] $0.12 USD
-[FAILURES] 2 tests failed - see details above
-```
-
-### `mj test list`
-
-List available tests, suites, and types.
+### list -- List Tests and Suites
 
 ```bash
 # List all tests
 mj test list
 
-# List by type
-mj test list --type=agent-eval
-
 # List test suites
 mj test list --suites
 
 # List test types
 mj test list --types
-
-# Verbose output (show configuration)
-mj test list --verbose
-
-# Filter by tag
-mj test list --tag=smoke
-
-# Filter by status
-mj test list --status=active
-```
-
-**Example Output:**
 ```
-Available Tests (42):
-
-Agent Evals (28):
-  - active-members-count     [easy]      Tags: membership, kpi
-  - revenue-ytd              [easy]      Tags: financial, kpi
-  - trend-analysis           [medium]    Tags: analytics, trends
-  - complex-aggregation      [hard]      Tags: advanced, performance
-
-Workflow Scenarios (8):
-  - customer-onboarding      [medium]    Tags: workflow, customer
-  - invoice-processing       [hard]      Tags: financial, workflow
-
-Test Suites (5):
-  - smoke-tests              15 tests
-  - regression-suite         42 tests
-  - nightly-full            87 tests
-```
-
-### `mj test validate`
 
-Validate test definitions without executing.
+### history -- View History
 
 ```bash
-# Validate specific test
-mj test validate <test-id>
-
-# Validate all tests
-mj test validate --all
-
-# Validate by type
-mj test validate --type=agent-eval
-
-# Save validation report
-mj test validate --all --save-report
+# View test run history
+mj test history --name "My Test"
 
-# Specify output file
-mj test validate --all --output=validation-report.md
+# View suite run history
+mj test history --suite "My Suite"
 ```
 
-**Example Output:**
-```
-Validating Tests...
-
-✓ active-members-count
-  - Configuration valid
-  - Oracles registered: trace-no-errors, sql-validate, schema-validate
-  - Expected outcomes defined
-  - No issues found
-
-✗ complex-aggregation
-  - Configuration valid
-  - Oracles registered: trace-no-errors, llm-judge
-  ⚠ Warning: Missing schema validation
-  ⚠ Warning: LLM judge rubric not found: 'component-quality-v2'
-
-✓ revenue-ytd
-  - Configuration valid
-  - Oracles registered: trace-no-errors, sql-validate
-  - Expected outcomes defined
-  - No issues found
-
-[SUMMARY] 2/3 tests valid, 1 with warnings
-```
-
-### `mj test report`
-
-Generate test run reports.
+### compare -- Compare Runs
 
 ```bash
-# Generate report for recent runs
-mj test report --suite=<suite-id>
-
-# Date range
-mj test report --from=2025-01-01 --to=2025-01-31
-
-# Specific tests
-mj test report --test=<test-id>
-
-# Output formats
-mj test report --format=markdown
-mj test report --format=json
-mj test report --format=html
-
-# Save to file
-mj test report --suite=<suite-id> --output=report.md
-
-# Include cost analysis
-mj test report --suite=<suite-id> --include-costs
-
-# Include trends
-mj test report --suite=<suite-id> --include-trends
+# Compare two test runs
+mj test compare --run1 "run-id-1" --run2 "run-id-2"
 ```
 
-**Example Markdown Output:**
-```markdown
-# Test Report: Agent Quality Suite
-Generated: 2025-01-09 14:30:00
-
-## Summary
-- **Suite:** Agent Quality Suite
-- **Period:** Last 30 days
-- **Total Runs:** 42
-- **Success Rate:** 95.2%
-- **Total Cost:** $5.28 USD
-- **Avg Duration:** 28.4s
-
-## Test Results
-| Test Name              | Runs | Pass Rate | Avg Score | Avg Cost | Trend |
-|------------------------|------|-----------|-----------|----------|-------|
-| Active Members Count   | 42   | 100%      | 1.0000    | $0.004   | →     |
-| Revenue YTD            | 42   | 97.6%     | 0.9850    | $0.005   | ↗     |
-| Complex Aggregation    | 42   | 85.7%     | 0.8200    | $0.012   | ↘     |
-
-## Failures
-2 failures in the last 30 days:
-
-### Complex Aggregation (2025-01-08)
-- **Score:** 0.6000
-- **Reason:** LLM judge failed - component quality below threshold
-- **Cost:** $0.012
-- **Trace:** [View Agent Run](link)
-
-## Cost Analysis
-- **Total Cost:** $5.28
-- **Most Expensive Test:** Complex Aggregation ($0.50)
-- **Cost Trend:** +12% vs previous month
-
-## Recommendations
-- Complex Aggregation: Review LLM judge rubric threshold
-- Consider caching results for Active Members Count (no failures)
-```
-
-### `mj test history`
-
-View test execution history.
+### validate -- Validate Configuration
 
 ```bash
-# History for specific test
-mj test history <test-id>
-
-# Recent runs
-mj test history --recent=10
-
-# By date range
-mj test history --from=2025-01-01
+# Validate a test
+mj test validate --name "My Test"
 
-# Filter by status
-mj test history --status=failed
-
-# Show details
-mj test history <test-id> --verbose
-```
-
-### `mj test compare`
-
-Compare test runs to detect regressions.
-
-```bash
-# Compare two specific runs
-mj test compare <run-id-1> <run-id-2>
-
-# Compare versions
-mj test compare --version=2.1.0 --version=2.2.0
-
-# Compare git commits
-mj test compare --commit=abc123 --commit=def456
-
-# Show only differences
-mj test compare <run-id-1> <run-id-2> --diff-only
-```
-
-**Example Output:**
-```
-Comparing Test Runs:
-  Run 1: 2025-01-01 (v2.1.0, commit abc123)
-  Run 2: 2025-01-09 (v2.2.0, commit def456)
-
-Test Results:
-  Active Members Count:       SAME    (1.0000 → 1.0000)
-  Revenue YTD:                SAME    (0.9850 → 0.9850)
-  Complex Aggregation:        WORSE   (0.9200 → 0.6000) ⚠
-  Trend Analysis:             BETTER  (0.7500 → 0.9100) ↗
-
-Performance:
-  Avg Duration:               BETTER  (28.4s → 24.1s) ↗
-  Total Cost:                 WORSE   ($0.12 → $0.18) ⚠
-
-Summary: 1 regression detected in Complex Aggregation
-```
-
-## Output Formats
-
-### Console (Default)
-Human-readable, color-coded output with progress indicators and emojis.
-
-### JSON
-Machine-readable for CI/CD integration:
-```json
-{
-  "testId": "abc-123",
-  "status": "passed",
-  "score": 1.0,
-  "duration": 1.8,
-  "cost": 0.0042,
-  "oracleResults": [
-    {
-      "type": "trace-no-errors",
-      "passed": true,
-      "score": 1.0,
-      "message": "All 3 steps completed without errors"
-    }
-  ]
-}
-```
-
-### Markdown
-Formatted reports for documentation:
-```markdown
-# Test Run: Active Members Count
-**Status:** PASSED ✓
-**Score:** 1.0000 (100%)
-**Duration:** 1.8s
-**Cost:** $0.0042
-
-## Oracle Results
-- ✓ trace-no-errors: All 3 steps completed without errors
-- ✓ sql-validate: COUNT(*) = 402 (expected 380-420)
-- ✓ schema-validate: Output matches schema
-```
-
-## CI/CD Integration
-
-### GitHub Actions Example
-```yaml
-name: Test Suite
-on: [push, pull_request]
-
-jobs:
-  test-suite:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-
-      - name: Run Smoke Tests
-        run: mj test suite smoke-tests --fail-fast --format=json > results.json
-
-      - name: Check Results
-        run: |
-          if ! jq -e '.passRate >= 0.95' results.json; then
-            echo "Test pass rate below 95%"
-            exit 1
-          fi
-
-      - name: Upload Results
-        uses: actions/upload-artifact@v3
-        with:
-          name: test-results
-          path: results.json
+# Validate a suite
+mj test validate --suite "My Suite"
 ```
 
 ## Configuration
 
-The CLI uses the standard MJ configuration file (`mj.config.cjs`):
-
-```javascript
-module.exports = {
-  // Standard MJ config
-  databaseSettings: { ... },
-
-  // Testing framework config
-  testing: {
-    defaultEnvironment: 'dev',
-    defaultFormat: 'console',
-    failFast: false,
-    parallel: false,
-    maxParallelTests: 5,
-    timeout: 300000  // 5 minutes
-  }
-};
-```
+The CLI uses `cosmiconfig` to load configuration from:
 
-## Error Handling
+- `.mjtestrc`
+- `.mjtestrc.json`
+- `mj.config.cjs` (in the `testing` section)
 
-The CLI provides clear error messages with actionable guidance:
+Configuration includes database connection settings for the MJ Provider.
 
-```
-✗ Error: Test 'complex-aggregation' failed validation
+## Utilities
 
-Issues found:
-  - LLM judge rubric not found: 'component-quality-v2'
-  - Missing required oracle: 'schema-validate'
+| Utility | Description |
+|---------|-------------|
+| `ConfigLoader` | Loads CLI configuration via cosmiconfig |
+| `OutputFormatter` | Formats test results for terminal display with chalk |
+| `SpinnerManager` | Manages ora spinners for long-running operations |
+| `VariableParser` | Parses `--var key=value` CLI arguments into typed variables |
+| `MJProvider` | Initializes the MemberJunction SQL Server data provider |
 
-Suggestions:
-  1. Create rubric 'component-quality-v2' or update test config
-  2. Add schema-validate oracle to test configuration
+## Dependencies
 
-Run 'mj test validate complex-aggregation --verbose' for details
-```
+| Package | Purpose |
+|---------|---------|
+| `@memberjunction/testing-engine` | Test execution engine |
+| `@memberjunction/testing-engine-base` | Metadata and types |
+| `@memberjunction/core` | Core MJ functionality |
+| `@memberjunction/core-entities` | Entity types |
+| `@memberjunction/sqlserver-dataprovider` | Database connectivity |
+| `@oclif/core` | CLI framework |
+| `chalk` | Terminal styling |
+| `ora-classic` | Terminal spinners |
+| `cosmiconfig` | Configuration loading |
 
-## Future Enhancements
+## License
 
-- [ ] Interactive mode for test creation
-- [ ] Watch mode (re-run on file changes)
-- [ ] Test coverage reports
-- [ ] Parallel suite execution
-- [ ] Result streaming for long-running tests
-- [ ] Integration with MJ UI for clickable trace links
+ISC

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@memberjunction/testing-cli",
   "type": "module",
-  "version": "4.0.0",
+  "version": "4.1.0",
   "description": "MemberJunction Testing Framework CLI - Command-line interface for test execution and management",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
@@ -21,12 +21,12 @@
     "@types/debug": "^4.1.12"
   },
   "dependencies": {
-    "@memberjunction/global": "4.0.0",
-    "@memberjunction/core": "4.0.0",
-    "@memberjunction/core-entities": "4.0.0",
-    "@memberjunction/testing-engine": "4.0.0",
-    "@memberjunction/testing-engine-base": "4.0.0",
-    "@memberjunction/sqlserver-dataprovider": "4.0.0",
+    "@memberjunction/global": "4.1.0",
+    "@memberjunction/core": "4.1.0",
+    "@memberjunction/core-entities": "4.1.0",
+    "@memberjunction/testing-engine": "4.1.0",
+    "@memberjunction/testing-engine-base": "4.1.0",
+    "@memberjunction/sqlserver-dataprovider": "4.1.0",
     "@oclif/core": "^3.27.0",
     "ora-classic": "^5.4.2",
     "chalk": "^5.6.2",