evals 1.0.7 → 2.1.0

package/README.md

@@ -1,103 +1,53 @@
-# Umbrage Evals JavaScript CLI
+# evals
 
-## Overview
+An npm package for Arize evals functionality.
 
-This is the Javascript CLI for Umbrage Evals. It also supports TypeScript.
+## About Arize AI
 
-## Installation
+Arize AI is a leading company in AI observability and evaluation, dedicated to ensuring that artificial intelligence systems operate reliably in real-world applications. Founded in 2019, Arize provides tools that help machine learning teams monitor, troubleshoot, and improve model performance across various domains, including structured data, computer vision, and large language models (LLMs).
 
-To install the package, run the following command:
+## Arize Phoenix
 
-```bash
-npm install evals
-```
+**Arize Phoenix** is an open-source library designed for LLM tracing and evaluation. It enables developers to evaluate, experiment, and optimize AI products in real time. Key features include:
 
-## Dependencies
+- **Application Tracing**: Collects LLM application data with automatic instrumentation, providing comprehensive visibility into model operations
+- **Interactive Prompt Playground**: Offers a flexible environment for prompt and model iteration, allowing users to compare prompts, visualize outputs, and debug failures within their workflow
+- **Streamlined Evaluations and Annotations**: Facilitates efficient assessment and documentation of model performance
 
--   Node.js (or another JavaScript runtime environment like Bun)
+Phoenix is built on OpenTelemetry, ensuring seamless setup, full transparency, and no vendor lock-in. It's perfect for teams who want to get started with LLM observability and evaluation in a fully local, open-source environment.
 
-## Environment
+## Arize AX
 
-You will need to set the following environment variables:
+**Arize AX** is the enterprise AI engineering platform that extends the capabilities of Phoenix, offering a comprehensive suite for development, evaluation, and observability. Key features include:
 
-```bash
-UMBRAGE_EVALS_API_KEY=<your api key for the umbrage evals project>
-OPENAI_API_KEY=<your openai api key>
-```
-
-These can be set in your `~/.bashrc` or `~/.zshrc` file on Mac or Linux, or in your `~/.bash_profile` file on Windows.
+- **Prompts**: Tools for prompt engineering, including a prompt playground, prompt hub for management and versioning, and a prompt builder powered by AI
+- **Experiments**: Systematic A/B testing of prompts against large datasets to optimize model performance
+- **Tracing**: Detailed tracing capabilities to monitor and debug AI applications effectively
+- **Evaluation**: Comprehensive evaluation metrics and tools to assess model outputs and performance
+- **Alyx**: An AI engineering agent embedded within the platform to assist with various tasks
 
-It would look like:
-
-```bash
-export UMBRAGE_EVALS_API_KEY=<your api key for the umbrage evals project>
-export OPENAI_API_KEY=<your openai api key>
-```
+Arize AX is designed to support teams and organizations with larger data needs, providing robust support, collaboration features, and multiple deployment options including SaaS, Virtual Private Cloud (VPC), and Arize Private Connect.
 
-To save the file, press `Ctrl + X`, then `Y`, then `Enter`. Then, run `source ~/.bashrc` or `source ~/.zshrc` on Mac or Linux, or `source ~/.bash_profile` on Windows.
 
-## Example usage
+## Installation
 
 ```bash
-npx evals@latest fetch-evals
-
-npx evals@latest run-evals
+npm install evals
 ```
 
-## Example Prompt BaseModel
-
-```typescript
-export interface ModelSettings {
-  temperature: number;
-  [key: string]: any;
-}
-
-export interface Prompt {
-  prompt_type: string;
-  prompt_text: string;
-}
-
-export interface Config {
-  promptName: string;
-  modelName: string;
-  modelSettings: ModelSettings;
-  environment: string;
-}
-
-class BaseModel {
-  promptName: string;
-  modelName: string;
-  modelSettings: ModelSettings;
-  environment: string;
-
-  constructor(config: Config) {
-    this.promptName = config.promptName;
-    this.modelName = config.modelName;
-    this.modelSettings = config.modelSettings;
-    this.environment = config.environment || 'development';
-}
-
-  /**
-   * This method should be overridden in subclasses.
-   * It should return an object containing the response and the prompts array.
-   * @param userMessage The user message to be processed by the model, optional.
-   * @returns Promise<{ response: string, prompts: Prompt[] }>
-   */
-  async callModel(userMessage: string = ''): Promise<{ response: string; prompts: Prompt[] }> {
-      throw new Error('callModel method needs to be implemented');
-  }
-}
-
-export default BaseModel;
-```
+The post-install script will automatically launch the CLI interface.
 
+## Usage
 
-## LICENSE
+Run the CLI manually:
 
-Copyright (c) 2023 Umbrage Studios, LLC
+```bash
+npm start
+```
 
-Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to use the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+Or use the binary:
 
-The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+```bash
+npx evals
+```
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/cli.js

@@ -0,0 +1,209 @@
+#!/usr/bin/env node
+
+import React, { useState, useEffect } from 'react';
+import { render, Box, Text, useInput, useApp, Static } from 'ink';
+import Gradient from 'ink-gradient';
+import { exec } from 'child_process';
+
+const e = React.createElement;
+
+// "ARIZE EVALS" - EXACTLY matched width (both 44 chars)
+const largeLogo = `
+ █████╗   ██████╗    ██╗  ███████╗  ███████╗
+██╔══██╗  ██╔══██╗   ██║  ╚══███╔╝  ██╔════╝
+███████║  ██████╔╝   ██║    ███╔╝   █████╗
+██╔══██║  ██╔══██╗   ██║   ███╔╝    ██╔══╝
+██║  ██║  ██║  ██║   ██║  ███████╗  ███████╗
+╚═╝  ╚═╝  ╚═╝  ╚═╝   ╚═╝  ╚══════╝  ╚══════╝
+███████╗ ██╗   ██╗  █████╗  ██╗     ███████╗
+██╔════╝ ██║   ██║ ██╔══██╗ ██║     ██╔════╝
+█████╗   ██║   ██║ ███████║ ██║     ███████╗
+██╔══╝   ╚██╗ ██╔╝ ██╔══██║ ██║     ╚════██║
+███████╗  ╚████╔╝  ██║  ██║ ███████╗███████║
+╚══════╝   ╚═══╝   ╚═╝  ╚═╝ ╚══════╝╚══════╝`;
+
+const mediumLogo = `
+ █████╗   ██████╗    ██╗  ███████╗  ███████╗
+██╔══██╗  ██╔══██╗   ██║  ╚══███╔╝  ██╔════╝
+███████║  ██████╔╝   ██║    ███╔╝   █████╗
+██╔══██║  ██╔══██╗   ██║   ███╔╝    ██╔══╝
+██║  ██║  ██║  ██║   ██║  ███████╗  ███████╗
+╚═╝  ╚═╝  ╚═╝  ╚═╝   ╚═╝  ╚══════╝  ╚══════╝
+█▀▀ █ █ ▄▀█ █   █▀
+██▄ ▀▄▀ █▀█ █▄▄ ▄█`;
+
+const smallLogo = `ARIZE
+EVALS`;
+
+const OPTIONS = [
+  {
+    name: 'Evals Primer',
+    subtext: 'Learn the Craft',
+    url: 'https://arize.com/llm-evaluation/?utm_source=npmevals'
+  },
+  {
+    name: 'Arize AX SaaS',
+    subtext: 'Enterprise Agent Evaluation, Best in Class',
+    url: 'https://arize.com/docs/ax?utm_source=npmevals'
+  },
+  {
+    name: 'Arize Phoenix',
+    subtext: 'OSS Agent Evals & Traces, Fully Local',
+    url: 'https://arize.com/docs/phoenix?utm_source=npmevals'
+  },
+  {
+    name: 'Book an Eval Assessment',
+    subtext: 'Improve Your Agent Today',
+    url: 'https://arize.com/get-an-eval-assessment?utm_source=npmevals'
+  }
+];
+
+function openUrlInBrowser(url) {
+  const command = process.platform === 'win32'
+    ? `start ${url}`
+    : process.platform === 'darwin'
+      ? `open "${url}"`
+      : `xdg-open "${url}"`;
+
+  exec(command, (error) => {
+    if (error) {
+      console.error(`Failed to open browser: ${error.message}`);
+    }
+    process.exit(0);
+  });
+}
+
+// Header with hot pink to dark blue gradient
+function Header() {
+  const terminalWidth = process.stdout.columns || 80;
+
+  let logo;
+  if (terminalWidth >= 45) {
+    logo = largeLogo;
+  } else if (terminalWidth >= 35) {
+    logo = mediumLogo;
+  } else {
+    logo = smallLogo;
+  }
+
+  return e(Box, { flexDirection: 'column', marginBottom: 1 },
+    e(Gradient, { colors: ['#FF008C', '#FF1493', '#8B5CF6', '#1E3A8A'] },
+      e(Text, null, logo)
+    ),
+    e(Text, { color: 'gray', italic: true }, 'Evals and Observability for Agentic AI')
+  );
+}
+
+// Character-by-character shimmer component
+function ShimmerText({ text, shimmerPos }) {
+  const baseColor = '#FF1493';
+  const shimmerColors = ['#FF5AA7', '#FF85C0', '#FFB8D9', '#FFE0EE', '#FFB8D9', '#FF85C0', '#FF5AA7'];
+  const shimmerWidth = shimmerColors.length;
+
+  const chars = text.split('').map((char, i) => {
+    const distanceFromShimmer = i - shimmerPos;
+
+    let color = baseColor;
+    if (distanceFromShimmer >= 0 && distanceFromShimmer < shimmerWidth) {
+      color = shimmerColors[distanceFromShimmer];
+    }
+
+    return e(Text, { key: i, color, bold: true }, char);
+  });
+
+  return e(Box, null, ...chars);
+}
+
+// Menu item component - single line, hot pink selected with character shimmer
+function MenuItem({ name, subtext, isSelected, shimmerPos }) {
+  if (isSelected) {
+    return e(Box, null,
+      e(Text, { bold: true, color: '#FF1493' }, '❯ '),
+      e(ShimmerText, { text: name, shimmerPos }),
+      e(Text, { color: 'gray' }, ' — ' + subtext)
+    );
+  }
+
+  return e(Box, null,
+    e(Text, { color: 'white' }, '  ' + name),
+    e(Text, { color: 'gray' }, ' — ' + subtext)
+  );
+}
+
+// Main App component
+function App() {
+  const [selectedIndex, setSelectedIndex] = useState(0);
+  const [shimmerPos, setShimmerPos] = useState(-7); // Start off-screen
+  const { exit } = useApp();
+
+  const selectedName = OPTIONS[selectedIndex].name;
+
+  // Character-by-character shimmer animation
+  useEffect(() => {
+    const textLength = selectedName.length;
+    const shimmerWidth = 7;
+    const totalPositions = textLength + shimmerWidth + 5; // Extra padding for smooth loop
+
+    const timer = setInterval(() => {
+      setShimmerPos(prev => {
+        const next = prev + 1;
+        return next > totalPositions ? -shimmerWidth : next;
+      });
+    }, 120); // Speed of shimmer movement
+
+    return () => clearInterval(timer);
+  }, [selectedName]);
+
+  // Reset shimmer position when selection changes
+  useEffect(() => {
+    setShimmerPos(-7);
+  }, [selectedIndex]);
+
+  useInput((input, key) => {
+    if (input === 'q' || key.escape) {
+      exit();
+      return;
+    }
+
+    if (key.upArrow || input === 'k') {
+      setSelectedIndex(prev => (prev > 0 ? prev - 1 : OPTIONS.length - 1));
+    }
+
+    if (key.downArrow || input === 'j') {
+      setSelectedIndex(prev => (prev < OPTIONS.length - 1 ? prev + 1 : 0));
+    }
+
+    if (key.return) {
+      const selected = OPTIONS[selectedIndex];
+      exit();
+      setTimeout(() => {
+        console.log(`\nOpening ${selected.url} in your browser...\n`);
+        openUrlInBrowser(selected.url);
+      }, 100);
+    }
+  });
+
+  return e(Box, { flexDirection: 'column', padding: 1 },
+    e(Static, { items: ['header'] }, () => e(Header)),
+    e(Box, { marginBottom: 1 },
+      e(Text, { bold: true, color: 'white' }, 'Select an option:')
+    ),
+    e(Box, { flexDirection: 'column', paddingX: 1 },
+      ...OPTIONS.map((option, index) =>
+        e(MenuItem, {
+          key: index.toString(),
+          name: option.name,
+          subtext: option.subtext,
+          isSelected: index === selectedIndex,
+          shimmerPos: shimmerPos
+        })
+      )
+    ),
+    e(Box, { marginTop: 1 },
+      e(Text, { dimColor: true }, 'Enter to select · ↑↓ to navigate · Esc to cancel')
+    )
+  );
+}
+
+// Run the app
+render(e(App), { patchConsole: false });

package/package.json

@@ -1,40 +1,26 @@
 {
   "name": "evals",
-  "module": "index.ts",
-  "main": "dist/index.js",
-  "types": "dist/index.d.ts",
+  "version": "2.1.0",
+  "description": "Arize evals package",
   "type": "module",
-  "version": "1.0.7",
-  "description": "Umbrage Evals CLI",
-  "bin": {
-    "evals": "./src/index.js"
-  },
+  "main": "cli.js",
   "scripts": {
-    "build": "bun build --target=node ./src/index.js --outfile=dist/index.js"
+    "postinstall": "node cli.js",
+    "start": "node cli.js"
   },
-  "devDependencies": {
-    "bun-types": "latest",
-    "prettier": "^3.0.3",
-    "rimraf": "^5.0.5",
-    "typescript": "^5.2.2"
+  "dependencies": {
+    "@arizeai/phoenix-otel": "*",
+    "ink": "^6.0.0",
+    "ink-gradient": "^3.0.0",
+    "react": "^19.0.0"
   },
-  "peerDependencies": {
-    "typescript": "^5.0.0"
+  "bin": {
+    "evals": "./cli.js"
   },
   "keywords": [
-    "evals",
-    "umbrage",
-    "ai",
-    "cli"
-  ],
-  "author": "umbrage.ai",
-  "license": "UNLICENSED",
-  "files": [
-    "dist/*.js",
-    "dist/*.d.ts"
+    "arize",
+    "evals"
   ],
-  "dependencies": {
-    "commander": "^11.1.0",
-    "openai": "^4.24.1"
-  }
+  "author": "",
+  "license": "ISC"
 }

package/LICENSE

@@ -1,7 +0,0 @@
-Copyright (c) 2023 Umbrage Studios, LLC
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to use the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.