@telnyx/voice-agent-tester 0.4.7 → 1.0.0

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## [1.0.0](https://github.com/team-telnyx/voice-agent-tester/compare/v0.4.7...v1.0.0) (2026-03-20)
+
+### ⚠ BREAKING CHANGES
+
+* The tool is now focused exclusively on **Telnyx vs Vapi** comparisons. ElevenLabs, Retell, and Livetok provider support has been removed.
+* The `--branch-id` CLI option has been removed (was ElevenLabs-specific).
+* The `--provider` flag now only accepts `vapi`.
+* Application configs for `elevenlabs.yaml` and `livetok.yaml` have been removed.
+
+### Features
+
+* Focused Telnyx vs Vapi comparison tool for v1.0.0 release ([#33](https://github.com/team-telnyx/voice-agent-tester/pull/33))
+* Streamlined CLI — fewer flags, simpler setup for Vapi-to-Telnyx benchmarking
+* Rewritten README centered on the Vapi vs Telnyx comparison workflow
+
 ## [0.4.7](https://github.com/team-telnyx/voice-agent-tester/compare/v0.4.6...v0.4.7) (2026-03-19)
 
 ## [0.4.6](https://github.com/team-telnyx/voice-agent-tester/compare/v0.4.5...v0.4.6) (2026-03-18)

package/README.md

@@ -3,33 +3,9 @@
 [![CI](https://github.com/team-telnyx/voice-agent-tester/actions/workflows/ci.yml/badge.svg)](https://github.com/team-telnyx/voice-agent-tester/actions/workflows/ci.yml)
 [![npm version](https://img.shields.io/npm/v/@telnyx/voice-agent-tester.svg)](https://www.npmjs.com/package/@telnyx/voice-agent-tester)
 
-Automated benchmarking CLI for voice AI agents. Import your assistant from any provider, run identical test scenarios on both platforms, and get a side-by-side latency comparison.
+Automated benchmarking CLI that compares **Vapi** voice agents against **Telnyx**. Import your Vapi assistant into Telnyx, run the same test scenario on both platforms, and get a side-by-side latency report.
 
-Supports **Telnyx**, **ElevenLabs**, **Vapi**, and **Retell**.
-
-## Compare Your Voice Agent Against Telnyx
-
-The tool imports your assistant from an external provider into Telnyx, then runs the **same scenario** on both platforms and produces a head-to-head latency report:
-
-```
-📈 Latency Comparison (elapsed_time):
---------------------------------------------------------------------------------
-Metric                                  vapi        Telnyx      Delta            Winner
---------------------------------------------------------------------------------
-Response #1 (wait_for_voice_elapsed_time) 2849ms    1552ms      -1297ms (-45.5%) 🏆 Telnyx
-Response #2 (wait_for_voice_elapsed_time) 3307ms    704ms       -2603ms (-78.7%) 🏆 Telnyx
---------------------------------------------------------------------------------
-
-📊 Overall Summary:
-   Compared 2 matched response latencies
-   vapi total latency: 6156ms
-   Telnyx total latency: 2256ms
-   Difference: -3900ms (-63.3%)
-
-   🏆 Result: Telnyx is faster overall
-```
-
-### Vapi vs Telnyx
+## Quick Start
 
 ```bash
 npx @telnyx/voice-agent-tester@latest \
@@ -42,49 +18,40 @@ npx @telnyx/voice-agent-tester@latest \
   --provider-import-id <VAPI_ASSISTANT_ID>
 ```
 
-### ElevenLabs vs Telnyx
+## How It Works
 
-```bash
-npx @telnyx/voice-agent-tester@latest \
-  -a applications/telnyx.yaml \
-  -s scenarios/appointment.yaml \
-  --provider elevenlabs \
-  --api-key <TELNYX_API_KEY> \
-  --provider-api-key <ELEVENLABS_API_KEY> \
-  --provider-import-id <ELEVENLABS_AGENT_ID>
-```
-
-### Retell vs Telnyx
+1. **Import** — Your Vapi assistant is imported into Telnyx via the Import API
+2. **Phase 1: Vapi Direct** — Runs the test scenario on Vapi's native widget
+3. **Phase 2: Telnyx Import** — Runs the same scenario on the Telnyx-imported assistant
+4. **Report** — Produces a side-by-side comparison with latency deltas and a winner per response
 
-```bash
-npx @telnyx/voice-agent-tester@latest \
-  -a applications/telnyx.yaml \
-  -s scenarios/appointment.yaml \
-  --provider retell \
-  --api-key <TELNYX_API_KEY> \
-  --provider-api-key <RETELL_API_KEY> \
-  --provider-import-id <RETELL_AGENT_ID>
 ```
+📊 COMPARISON: VAPI vs TELNYX
+================================================================================
 
-### How Comparison Works
+   Average response latency (2 matched responses):
 
-1. **Import** — The assistant is imported from the external provider into Telnyx
-2. **Phase 1: Provider Direct** — Runs the scenario on the provider's native widget
-3. **Phase 2: Telnyx Import** — Runs the same scenario on the Telnyx-imported assistant
-4. **Report** — Produces a side-by-side comparison with latency delta and winner per response
+   vapi             2849ms
+   Telnyx           1552ms
+   Difference       -1297ms (-45.5%)
 
-### Provider-Specific Keys
+   🏆 Telnyx is 45.5% faster
 
-Some providers need an extra key to load their demo widget. If not passed via CLI, the tool prompts with instructions.
+================================================================================
+```
 
-| Provider | Flag | Required? | How to find it |
-|----------|------|-----------|----------------|
-| Vapi | `--share-key` | Yes | Dashboard → select assistant → click 🔗 link icon next to the assistant ID |
-| ElevenLabs | `--branch-id` | No | Dashboard → Agents → select agent → Publish dropdown → "Copy shareable link" |
+## Where to Find Your Keys
 
-### Import Only (Skip Comparison)
+| Key | Where to find it |
+|-----|------------------|
+| `--api-key` | [Telnyx Portal → API Keys](https://portal.telnyx.com/#/app/api-keys) |
+| `--provider-api-key` | [Vapi Dashboard → Organization Settings](https://dashboard.vapi.ai/org/api-keys) |
+| `--provider-import-id` | Vapi Dashboard → select your assistant → copy the assistant ID |
+| `--share-key` | Vapi Dashboard → select assistant → click 🔗 link icon next to the assistant ID |
 
-To import without running the provider benchmark:
+## Import + Benchmark (Skip Comparison)
+
+Import your Vapi assistant into Telnyx and benchmark the imported Telnyx assistant only (skips the Vapi direct phase):
 
 ```bash
 npx @telnyx/voice-agent-tester@latest \
@@ -97,51 +64,17 @@ npx @telnyx/voice-agent-tester@latest \
   --provider-import-id <VAPI_ASSISTANT_ID>
 ```
 
-## Quick Start
+> **Note:** `--no-compare` still runs the benchmark against the imported Telnyx assistant. It only skips the Vapi direct benchmark phase and the side-by-side comparison report.
 
-Run directly with npx (no installation required):
+## Test Telnyx Directly
 
-```bash
-npx @telnyx/voice-agent-tester@latest \
-  -a applications/telnyx.yaml \
-  -s scenarios/appointment.yaml \
-  --assistant-id <YOUR_ASSISTANT_ID>
-```
-
-Or install globally:
-
-```bash
-npm install -g @telnyx/voice-agent-tester
-voice-agent-tester -a applications/telnyx.yaml -s scenarios/appointment.yaml --assistant-id <YOUR_ASSISTANT_ID>
-```
-
-## Provider Examples
-
-### Telnyx
+Benchmark a Telnyx assistant without comparison:
 
 ```bash
 npx @telnyx/voice-agent-tester@latest \
   -a applications/telnyx.yaml \
   -s scenarios/appointment.yaml \
-  --assistant-id <ASSISTANT_ID>
-```
-
-### ElevenLabs
-
-```bash
-npx @telnyx/voice-agent-tester@latest \
-  -a applications/elevenlabs.yaml \
-  -s scenarios/appointment.yaml \
-  --assistant-id <AGENT_ID>
-```
-
-### Vapi
-
-```bash
-npx @telnyx/voice-agent-tester@latest \
-  -a applications/vapi.yaml \
-  -s scenarios/appointment.yaml \
-  --assistant-id <ASSISTANT_ID>
+  --assistant-id <TELNYX_ASSISTANT_ID>
 ```
 
 ## CLI Reference
@@ -150,87 +83,28 @@ npx @telnyx/voice-agent-tester@latest \
 |--------|---------|-------------|
 | `-a, --applications` | required | Application config path(s) or folder |
 | `-s, --scenarios` | required | Scenario config path(s) or folder |
-| `--assistant-id` | | Telnyx or provider assistant ID |
+| `--provider` | | Set to `vapi` for comparison mode |
 | `--api-key` | | Telnyx API key |
-| `--provider` | | Import from provider (`vapi`, `elevenlabs`, `retell`) |
-| `--provider-api-key` | | External provider API key |
-| `--provider-import-id` | | Provider assistant/agent ID to import |
+| `--provider-api-key` | | Vapi API key |
+| `--provider-import-id` | | Vapi assistant ID to import |
 | `--share-key` | | Vapi share key for comparison mode |
-| `--branch-id` | | ElevenLabs branch ID (optional) |
-| `--compare` | `true` | Run provider direct + Telnyx import benchmarks |
-| `--no-compare` | | Skip provider direct benchmark |
+| `--assistant-id` | | Telnyx assistant ID (direct mode) |
+| `--compare` | `true` | Run both Vapi + Telnyx benchmarks |
+| `--no-compare` | | Skip Vapi direct benchmark and comparison (import + Telnyx benchmark only) |
 | `-d, --debug` | `false` | Detailed timeout diagnostics |
 | `-v, --verbose` | `false` | Show browser console logs |
 | `--headless` | `true` | Run browser in headless mode |
-| `--repeat` | `1` | Repetitions per app+scenario combination |
+| `--repeat` | `1` | Repetitions per test combination |
 | `-c, --concurrency` | `1` | Parallel test runs |
 | `-r, --report` | | CSV report output path |
 | `-p, --params` | | URL template params (`key=value,key2=value2`) |
 | `--retries` | `0` | Retry failed runs |
-| `--application-tags` | | Filter applications by tags |
-| `--scenario-tags` | | Filter scenarios by tags |
 | `--record` | `false` | Record video+audio (webm) |
-| `--audio-url` | | URL to audio file played as input during run |
+| `--audio-url` | | URL to audio file played as mic input |
 | `--audio-volume` | `1.0` | Audio input volume (0.0–1.0) |
 | `--assets-server` | `http://localhost:3333` | Assets server URL |
-
-## Bundled Configs
-
-**Applications:**
-
-| Config | Provider |
-|--------|----------|
-| `applications/telnyx.yaml` | Telnyx AI Widget |
-| `applications/elevenlabs.yaml` | ElevenLabs |
-| `applications/vapi.yaml` | Vapi |
-| `applications/retell.yaml` | Retell |
-
-**Scenarios:**
-
-| Config | Description |
-|--------|-------------|
-| `scenarios/appointment.yaml` | Appointment booking test |
-| `scenarios/appointment_with_noise.yaml` | Appointment with background crowd noise |
-
-## Background Noise Testing
-
-Test how voice agents perform with ambient noise by using pre-mixed audio files:
-
-```bash
-# With background noise
-npx @telnyx/voice-agent-tester@latest \
-  -a applications/telnyx.yaml \
-  -s scenarios/appointment_with_noise.yaml \
-  --assistant-id <ASSISTANT_ID>
-
-# Without noise (same assistant, compare results)
-npx @telnyx/voice-agent-tester@latest \
-  -a applications/telnyx.yaml \
-  -s scenarios/appointment.yaml \
-  --assistant-id <ASSISTANT_ID>
-```
-
-### Custom Audio Input
-
-Play any audio file from a URL as microphone input throughout the benchmark:
-
-```bash
-npx @telnyx/voice-agent-tester@latest \
-  -a applications/telnyx.yaml \
-  -s scenarios/appointment.yaml \
-  --assistant-id <ASSISTANT_ID> \
-  --audio-url "https://example.com/test-audio.mp3" \
-  --audio-volume 0.8
-```
-
-### Audio Assets
-
-| File | Description |
-|------|-------------|
-| `hello_make_an_appointment.mp3` | Clean appointment request |
-| `hello_make_an_appointment_with_noise.mp3` | Appointment request + crowd noise |
-| `appointment_data.mp3` | Clean appointment details |
-| `appointment_data_with_noise.mp3` | Appointment details + crowd noise |
+| `--application-tags` | | Filter applications by tags |
+| `--scenario-tags` | | Filter scenarios by tags |
 
 ## Scenario Configuration
 
@@ -269,15 +143,56 @@ steps:
 | `screenshot` | Capture a screenshot |
 | `listen` | Record agent audio, transcribe, and evaluate |
 
-## Debugging
+## Background Noise Testing
+
+Test how voice agents perform with ambient noise:
+
+```bash
+npx @telnyx/voice-agent-tester@latest \
+  -a applications/telnyx.yaml \
+  -s scenarios/appointment_with_noise.yaml \
+  --provider vapi \
+  --share-key <VAPI_SHARE_KEY> \
+  --api-key <TELNYX_API_KEY> \
+  --provider-api-key <VAPI_API_KEY> \
+  --provider-import-id <VAPI_ASSISTANT_ID>
+```
+
+### Custom Audio Input
 
-If benchmarks fail or time out, use `--debug` for detailed diagnostics including audio monitor state, WebRTC connection info, and RTP stats:
+Play any audio file from a URL as microphone input throughout the benchmark:
 
 ```bash
 npx @telnyx/voice-agent-tester@latest \
   -a applications/telnyx.yaml \
   -s scenarios/appointment.yaml \
   --assistant-id <ASSISTANT_ID> \
+  --audio-url "https://example.com/test-audio.mp3" \
+  --audio-volume 0.8
+```
+
+### Audio Assets
+
+| File | Description |
+|------|-------------|
+| `hello_make_an_appointment.mp3` | Clean appointment request |
+| `hello_make_an_appointment_with_noise.mp3` | Appointment request + crowd noise |
+| `appointment_data.mp3` | Clean appointment details |
+| `appointment_data_with_noise.mp3` | Appointment details + crowd noise |
+
+## Debugging
+
+Use `--debug` for detailed diagnostics including audio monitor state, WebRTC connections, and RTP stats:
+
+```bash
+npx @telnyx/voice-agent-tester@latest \
+  -a applications/telnyx.yaml \
+  -s scenarios/appointment.yaml \
+  --provider vapi \
+  --share-key <VAPI_SHARE_KEY> \
+  --api-key <TELNYX_API_KEY> \
+  --provider-api-key <VAPI_API_KEY> \
+  --provider-import-id <VAPI_ASSISTANT_ID> \
   --debug
 ```
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@telnyx/voice-agent-tester",
-  "version": "0.4.7",
+  "version": "1.0.0",
   "description": "A command-line tool to test voice agents using Puppeteer",
   "main": "src/index.js",
   "type": "module",

package/src/index.js

@@ -89,7 +89,6 @@ function substituteUrlParams(url, params) {
 
 /**
  * Get the list of missing provider-specific parameters required for comparison mode.
- * Each provider has its own set of required params for the direct widget benchmark.
  *
  * @param {Object} argv - Parsed CLI arguments
  * @returns {Array<{key: string, flag: string, description: string}>} Missing params
@@ -97,21 +96,13 @@ function substituteUrlParams(url, params) {
 function getCompareRequiredParams(argv) {
   const missing = [];
 
-  switch (argv.provider) {
-    case 'vapi':
-      if (!argv.shareKey) {
-        missing.push({
-          key: 'shareKey',
-          flag: '--share-key',
-          description: 'Vapi share key',
-          hint: 'In the Vapi Dashboard, select your assistant, then click the link icon (🔗) next to the assistant ID at the top. This copies the demo link containing your share key.'
-        });
-      }
-      break;
-    case 'elevenlabs':
-      // branchId is optional — the talk-to URL works with just agent_id
-      break;
-    // retell and others: no extra params needed yet
+  if (!argv.shareKey) {
+    missing.push({
+      key: 'shareKey',
+      flag: '--share-key',
+      description: 'Vapi share key',
+      hint: 'In the Vapi Dashboard, select your assistant, then click the link icon (🔗) next to the assistant ID at the top. This copies the demo link containing your share key.'
+    });
   }
 
   return missing;
@@ -124,28 +115,7 @@ function getCompareRequiredParams(argv) {
  * @returns {Object} Template params to merge into provider params
  */
 function getCompareTemplateParams(argv) {
-  switch (argv.provider) {
-    case 'vapi':
-      return { shareKey: argv.shareKey };
-    default:
-      return {};
-  }
-}
-
-/**
- * Get provider-specific extra query parameters to append to the comparison URL.
- * Unlike template params, these are appended as-is (not substituted into {{...}} placeholders).
- *
- * @param {Object} argv - Parsed CLI arguments
- * @returns {Object} Key-value pairs to append as query parameters
- */
-function getCompareExtraQueryParams(argv) {
-  switch (argv.provider) {
-    case 'elevenlabs':
-      return argv.branchId ? { branch_id: argv.branchId } : {};
-    default:
-      return {};
-  }
+  return { shareKey: argv.shareKey };
 }
 
 // Helper function to load and validate application config
@@ -298,15 +268,11 @@ const argv = yargs(hideBin(process.argv))
   })
   .option('share-key', {
     type: 'string',
-    description: 'Vapi share key for direct widget testing (required for comparison mode with --provider vapi)'
-  })
-  .option('branch-id', {
-    type: 'string',
-    description: 'ElevenLabs branch ID for direct widget testing (optional, appended to demo URL when provided)'
+    description: 'Vapi share key for direct widget testing (required for comparison mode)'
   })
   .option('assistant-id', {
     type: 'string',
-    description: 'Assistant/agent ID for direct benchmarking (works with all providers)'
+    description: 'Telnyx assistant ID for direct benchmarking (without comparison)'
   })
   .option('debug', {
     alias: 'd',
@@ -718,17 +684,6 @@ async function main() {
       }
 
       const providerApp = loadApplicationConfig(providerAppPath, providerParams);
-
-      // Append optional extra query parameters (e.g. branch_id for ElevenLabs)
-      const extraQueryParams = getCompareExtraQueryParams(argv);
-      if (providerApp.url && Object.keys(extraQueryParams).length > 0) {
-        const url = new URL(providerApp.url);
-        for (const [key, value] of Object.entries(extraQueryParams)) {
-          url.searchParams.set(key, value);
-        }
-        providerApp.url = url.toString();
-      }
-
       const providerApplications = [providerApp];
       
       const providerResults = await runBenchmark({

package/src/provider-import.js

@@ -15,7 +15,7 @@ const TELNYX_IMPORT_ENDPOINT = `${TELNYX_BASE_URL}/ai/assistants/import`;
 const TELNYX_ASSISTANTS_ENDPOINT = `${TELNYX_BASE_URL}/ai/assistants`;
 
 // Supported providers
-const SUPPORTED_PROVIDERS = ['vapi', 'elevenlabs', 'retell'];
+const SUPPORTED_PROVIDERS = ['vapi'];
 
 // Default widget settings for benchmarking
 const DEFAULT_WIDGET_SETTINGS = {

package/applications/elevenlabs.yaml

@@ -1,18 +0,0 @@
-url: "https://elevenlabs.io/app/talk-to?agent_id={{assistantId}}"
-tags:
-  - provider
-  - elevenlabs
-steps:
-  - action: wait_for_element
-    selector: "text=Call AI agent"
-  - action: sleep
-    time: 3000
-  - action: click
-    selector: "text=Call AI agent"
-  # ElevenLabs shows a terms consent dialog — click "Agree" to proceed
-  - action: wait_for_element
-    selector: "text=Agree"
-  - action: click
-    selector: "text=Agree"
-  - action: sleep
-    time: 2000

package/applications/livetok.yaml

@@ -1,16 +0,0 @@
-url: "https://rti.livetok.io/demo/index.html"
-tags:
-  - default
-  - basic
-steps:
-  - action: fill
-    selector: "input[type='password']"
-    text: "GOOGLE_API_KEY HERE"
-  # - action: select
-  #   selector: "#model"
-  #   value: "gemini-2.5-flash-preview-native-audio-dialog"
-  # - action: fill
-  #   selector: "#tools"
-  #   text: "[]"
-  - action: click
-    selector: "#start"