@altairalabs/promptarena 0.0.1

package/README.md

@@ -0,0 +1,202 @@
+# @altairalabs/promptarena
+
+> PromptKit Arena - Multi-turn conversation simulation and testing tool for LLM applications
+
+## Installation
+
+### npx (No Installation Required)
+
+```bash
+npx @altairalabs/promptarena run -c ./examples/customer-support
+```
+
+### Global Installation
+
+```bash
+npm install -g @altairalabs/promptarena
+
+# Use directly
+promptarena --version
+promptarena run -c ./config
+```
+
+### Project Dev Dependency
+
+```bash
+npm install --save-dev @altairalabs/promptarena
+
+# Use via npm scripts
+# Add to package.json:
+{
+  "scripts": {
+    "test:prompts": "promptarena run -c ./tests/arena-config"
+  }
+}
+```
+
+## What is PromptKit Arena?
+
+PromptKit Arena is a comprehensive testing framework for LLM-based applications. It allows you to:
+
+- 🎯 **Test conversations** across multiple LLM providers (OpenAI, Anthropic, Google, Azure)
+- 🔄 **Run multi-turn simulations** with automated agent interactions
+- ✅ **Validate outputs** using assertions and quality metrics
+- 📊 **Generate reports** with detailed analysis and comparisons
+- 🛡️ **Test guardrails** and safety measures
+- 🔧 **Validate tool usage** and function calling
+
+## Quick Start
+
+1. Create a test configuration:
+
+```yaml
+# arena.yaml
+name: Customer Support Test
+prompts:
+  - name: support-agent
+    system_prompt: |
+      You are a helpful customer support agent.
+      Be professional and empathetic.
+
+conversations:
+  - name: refund-request
+    turns:
+      - role: user
+        content: "I'd like a refund for order #12345"
+      - role: assistant
+        expected_topics: ["refund", "order"]
+
+providers:
+  - type: openai
+    model: gpt-4
+    api_key: ${OPENAI_API_KEY}
+```
+
+2. Run the test:
+
+```bash
+promptarena run -c arena.yaml
+```
+
+3. View the HTML report:
+
+```bash
+open out/report.html
+```
+
+## Features
+
+### Multi-Provider Testing
+
+Test the same prompts across different LLM providers:
+
+```yaml
+providers:
+  - type: openai
+    model: gpt-4
+  - type: anthropic
+    model: claude-3-5-sonnet-20241022
+  - type: google
+    model: gemini-1.5-pro
+```
+
+### Automated Assertions
+
+Validate LLM responses automatically:
+
+```yaml
+turns:
+  - role: assistant
+    assertions:
+      - type: contains
+        value: "refund"
+      - type: tone
+        expected: professional
+      - type: length
+        min: 50
+        max: 500
+```
+
+### Self-Play Mode
+
+Let AI agents interact with each other:
+
+```yaml
+self_play:
+  enabled: true
+  rounds: 5
+  agents:
+    - role: customer
+      prompt: "Act as a frustrated customer"
+    - role: support
+      prompt: "Act as a patient support agent"
+```
+
+## How It Works
+
+This npm package downloads pre-built Go binaries from [GitHub Releases](https://github.com/AltairaLabs/PromptKit/releases) during installation. The binaries are:
+
+1. Downloaded for your specific OS and architecture
+2. Extracted from the release archive
+3. Made executable (Unix-like systems)
+4. Invoked through a thin Node.js wrapper
+
+No Go toolchain is required on your machine.
+
+## Supported Platforms
+
+- macOS (Intel and Apple Silicon)
+- Linux (x86_64 and arm64)
+- Windows (x86_64 and arm64)
+
+## Documentation
+
+- [Full Documentation](https://github.com/AltairaLabs/PromptKit#readme)
+- [Examples](https://github.com/AltairaLabs/PromptKit/tree/main/examples)
+- [Configuration Reference](https://github.com/AltairaLabs/PromptKit/tree/main/docs)
+
+## Troubleshooting
+
+### Binary Download Fails
+
+If the postinstall script fails:
+
+1. Check your internet connection
+2. Verify the version exists in [GitHub Releases](https://github.com/AltairaLabs/PromptKit/releases)
+3. Check npm proxy/registry settings
+4. Try manual installation:
+
+```bash
+# Download binary directly
+curl -L https://github.com/AltairaLabs/PromptKit/releases/download/v0.0.1/PromptKit_v0.0.1_Darwin_arm64.tar.gz -o promptarena.tar.gz
+tar -xzf promptarena.tar.gz promptarena
+chmod +x promptarena
+```
+
+### Permission Denied
+
+On Unix-like systems:
+
+```bash
+chmod +x node_modules/@altairalabs/promptarena/promptarena
+```
+
+## Alternative Installation Methods
+
+- **Homebrew**: `brew install altairalabs/tap/promptkit`
+- **Go Install**: `go install github.com/AltairaLabs/PromptKit/tools/arena/cmd/promptarena@latest`
+- **Direct Download**: [GitHub Releases](https://github.com/AltairaLabs/PromptKit/releases)
+- **Build from Source**: Clone repo and run `make install-tools`
+
+## License
+
+Apache-2.0 - see [LICENSE](https://github.com/AltairaLabs/PromptKit/blob/main/LICENSE)
+
+## Contributing
+
+Contributions welcome! See [CONTRIBUTING.md](https://github.com/AltairaLabs/PromptKit/blob/main/CONTRIBUTING.md)
+
+## Support
+
+- [GitHub Issues](https://github.com/AltairaLabs/PromptKit/issues)
+- [Discussions](https://github.com/AltairaLabs/PromptKit/discussions)

package/bin/promptarena.js

@@ -0,0 +1,34 @@
+#!/usr/bin/env node
+
+const { spawn } = require('child_process');
+const path = require('path');
+const fs = require('fs');
+
+const binaryName = process.platform === 'win32' ? 'promptarena.exe' : 'promptarena';
+const binaryPath = path.join(__dirname, '..', binaryName);
+
+// Check if binary exists
+if (!fs.existsSync(binaryPath)) {
+  console.error('Error: promptarena binary not found.');
+  console.error('Please try reinstalling: npm install @altairalabs/promptarena');
+  process.exit(1);
+}
+
+// Spawn the Go binary with all arguments
+const child = spawn(binaryPath, process.argv.slice(2), {
+  stdio: 'inherit',
+  windowsHide: false
+});
+
+child.on('error', (err) => {
+  console.error('Failed to start promptarena:', err.message);
+  process.exit(1);
+});
+
+child.on('exit', (code, signal) => {
+  if (signal) {
+    process.kill(process.pid, signal);
+  } else {
+    process.exit(code || 0);
+  }
+});

package/package.json

@@ -0,0 +1,54 @@
+{
+  "name": "@altairalabs/promptarena",
+  "version": "0.0.1",
+  "description": "PromptKit Arena - Multi-turn conversation simulation and testing tool for LLM applications",
+  "bin": {
+    "promptarena": "./bin/promptarena.js"
+  },
+  "scripts": {
+    "postinstall": "node postinstall.js",
+    "test": "node bin/promptarena.js --version"
+  },
+  "keywords": [
+    "llm",
+    "testing",
+    "prompt",
+    "ai",
+    "conversation",
+    "evaluation",
+    "chatbot",
+    "openai",
+    "anthropic",
+    "gemini",
+    "quality-assurance",
+    "test-automation"
+  ],
+  "author": "Altaira Labs",
+  "license": "Apache-2.0",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/AltairaLabs/PromptKit.git",
+    "directory": "npm/promptarena"
+  },
+  "bugs": {
+    "url": "https://github.com/AltairaLabs/PromptKit/issues"
+  },
+  "homepage": "https://github.com/AltairaLabs/PromptKit#readme",
+  "engines": {
+    "node": ">=18.0.0"
+  },
+  "os": [
+    "darwin",
+    "linux",
+    "win32"
+  ],
+  "cpu": [
+    "x64",
+    "arm64"
+  ],
+  "files": [
+    "bin/",
+    "postinstall.js",
+    "README.md"
+  ]
+}

package/postinstall.js

@@ -0,0 +1,134 @@
+#!/usr/bin/env node
+
+const https = require('node:https');
+const http = require('node:http');
+const fs = require('node:fs');
+const path = require('node:path');
+const { execSync } = require('node:child_process');
+const { pipeline } = require('node:stream');
+const { promisify } = require('node:util');
+
+const streamPipeline = promisify(pipeline);
+
+const BINARY_NAME = 'promptarena';
+const GITHUB_REPO = 'AltairaLabs/PromptKit';
+const VERSION = require('./package.json').version;
+
+// Platform mapping to match GoReleaser output
+const PLATFORM_MAP = {
+  darwin: 'Darwin',
+  linux: 'Linux',
+  win32: 'Windows'
+};
+
+const ARCH_MAP = {
+  x64: 'x86_64',
+  arm64: 'arm64'
+};
+
+function getPlatformInfo() {
+  const platform = PLATFORM_MAP[process.platform];
+  const arch = ARCH_MAP[process.arch];
+  
+  if (!platform || !arch) {
+    throw new Error(`Unsupported platform: ${process.platform}-${process.arch}`);
+  }
+  
+  return { platform, arch };
+}
+
+function getDownloadUrl(platform, arch) {
+  const archiveExt = platform === 'Windows' ? 'zip' : 'tar.gz';
+  const archiveName = `PromptKit_v${VERSION}_${platform}_${arch}.${archiveExt}`;
+  return `https://github.com/${GITHUB_REPO}/releases/download/v${VERSION}/${archiveName}`;
+}
+
+async function downloadFile(url, destPath) {
+  return new Promise((resolve, reject) => {
+    const client = url.startsWith('https:') ? https : http;
+    
+    client.get(url, (response) => {
+      // Follow redirects
+      if (response.statusCode === 302 || response.statusCode === 301) {
+        downloadFile(response.headers.location, destPath)
+          .then(resolve)
+          .catch(reject);
+        return;
+      }
+      
+      if (response.statusCode !== 200) {
+        reject(new Error(`Failed to download: HTTP ${response.statusCode}`));
+        return;
+      }
+      
+      const fileStream = fs.createWriteStream(destPath);
+      streamPipeline(response, fileStream)
+        .then(resolve)
+        .catch(reject);
+    }).on('error', reject);
+  });
+}
+
+function extractBinary(archivePath, platform, binaryName) {
+  const binaryWithExt = platform === 'Windows' ? `${binaryName}.exe` : binaryName;
+  const destPath = path.join(__dirname, binaryWithExt);
+  
+  try {
+    if (platform === 'Windows') {
+      // Extract from zip - the binary should be in the archive root
+      execSync(`unzip -j "${archivePath}" "${binaryWithExt}" -d "${__dirname}"`, {
+        stdio: 'inherit'
+      });
+    } else {
+      // Extract from tar.gz - the binary should be in the archive root
+      execSync(`tar -xzf "${archivePath}" -C "${__dirname}" "${binaryWithExt}"`, {
+        stdio: 'inherit'
+      });
+    }
+    
+    // Make executable on Unix-like systems
+    if (platform !== 'Windows') {
+      fs.chmodSync(destPath, 0o755);
+    }
+    
+    console.log(`✓ Extracted ${binaryWithExt}`);
+    return destPath;
+  } catch (error) {
+    throw new Error(`Failed to extract binary: ${error.message}`);
+  }
+}
+
+async function install() {
+  console.log(`Installing ${BINARY_NAME} v${VERSION}...`);
+  
+  const { platform, arch } = getPlatformInfo();
+  console.log(`Platform: ${platform} ${arch}`);
+  
+  const url = getDownloadUrl(platform, arch);
+  const archiveExt = platform === 'Windows' ? 'zip' : 'tar.gz';
+  const archivePath = path.join(__dirname, `archive.${archiveExt}`);
+  
+  try {
+    console.log('Downloading binary from GitHub Releases...');
+    await downloadFile(url, archivePath);
+    console.log('✓ Download complete');
+    
+    console.log('Extracting binary...');
+    extractBinary(archivePath, platform, BINARY_NAME);
+    
+    // Clean up archive
+    fs.unlinkSync(archivePath);
+    
+    console.log(`✓ ${BINARY_NAME} installed successfully!`);
+  } catch (error) {
+    console.error(`\n❌ Installation failed: ${error.message}`);
+    console.error('\nTroubleshooting:');
+    console.error('1. Verify version exists: https://github.com/AltairaLabs/PromptKit/releases');
+    console.error('2. Check your internet connection');
+    console.error('3. Try downloading manually from GitHub Releases');
+    console.error(`   URL: ${url}`);
+    process.exit(1);
+  }
+}
+
+await install();