@tyvm/knowhow 0.0.33 → 0.0.35

package/autodoc/plugins/downloader/downloader.mdx

@@ -25,7 +25,7 @@ The Downloader Plugin is a comprehensive tool designed to facilitate the downloa
 
 To ensure the plugin functions correctly, the following environment variables must be set:
 
-- **OPENAI_API_KEY**: This variable is required to authenticate requests to the OpenAI API for transcription services.
+- **OPENAI_KEY**: This variable is required to authenticate requests to the OpenAI API for transcription services.
 
 ## Usage Scenarios
 
@@ -35,4 +35,4 @@ To ensure the plugin functions correctly, the following environment variables mu
 
 ## Conclusion
 
-The Downloader Plugin is a versatile tool that simplifies the process of downloading, chunking, and transcribing media content. By leveraging advanced transcription services and providing detailed media information, it adds significant value to chat applications and other systems requiring rich, contextual data. Ensure the necessary environment variables are set to fully utilize the plugin's capabilities.
+The Downloader Plugin is a versatile tool that simplifies the process of downloading, chunking, and transcribing media content. By leveraging advanced transcription services and providing detailed media information, it adds significant value to chat applications and other systems requiring rich, contextual data. Ensure the necessary environment variables are set to fully utilize the plugin's capabilities.

package/benchmarks/.dockerignore

@@ -0,0 +1,7 @@
+node_modules
+dist
+results
+.git
+*.log
+.DS_Store
+.env*

package/benchmarks/README.md

@@ -0,0 +1,166 @@
+# Knowhow Benchmarks
+
+A benchmarking framework for testing the Knowhow terminal agent against coding exercises from Exercism.
+
+## Overview
+
+This package provides tools to:
+- Clone and setup Exercism coding exercises
+- Run the Knowhow agent against these exercises in a controlled environment
+- Collect metrics (turns, time, cost) and success rates
+- Generate detailed reports
+
+## Quick Start
+
+### 1. Build the Docker Container
+
+From the main Knowhow repository root:
+
+```bash
+docker build -f benchmarks/docker/Dockerfile -t knowhow-bench .
+```
+
+### 2. Run Benchmarks
+
+```bash
+# Run 5 JavaScript exercises with GPT-4o-mini
+docker run --rm -v $(pwd)/benchmarks/results:/app/knowhow/benchmarks/results \
+  knowhow-bench run --language javascript --count 5 --model gpt-4o-mini
+
+# Setup exercises only (without running)
+docker run --rm knowhow-bench setup --language javascript --count 10
+```
+
+## Configuration Options
+
+### Command Line Arguments
+
+- `--language <lang>`: Programming language to test (default: javascript)
+- `--count <num>`: Maximum number of exercises to run (default: 10)
+- `--model <model>`: AI model to use (default: gpt-4o-mini)
+- `--provider <provider>`: AI provider (default: openai)
+- `--max-turns <num>`: Maximum turns per exercise (default: 20)
+- `--max-time <seconds>`: Maximum time per exercise (default: 300)
+- `--max-cost <dollars>`: Maximum cost per exercise (default: 1.0)
+- `--output <file>`: Output file for results (default: results.json)
+
+### Example Commands
+
+```bash
+# Run Python exercises with custom limits
+docker run --rm knowhow-bench run \
+  --language python \
+  --count 15 \
+  --model gpt-4 \
+  --max-turns 30 \
+  --max-time 600 \
+  --output python-results.json
+
+# Run with Claude
+docker run --rm knowhow-bench run \
+  --provider anthropic \
+  --model claude-3-sonnet-20240229 \
+  --count 10
+```
+
+## Results Format
+
+The benchmark generates a JSON file with detailed results:
+
+```json
+{
+  "config": {
+    "language": "javascript",
+    "maxExercises": 5,
+    "model": "gpt-4o-mini",
+    "provider": "openai"
+  },
+  "exercises": [
+    {
+      "exerciseName": "hello-world",
+      "status": "success",
+      "turns": 3,
+      "timeElapsed": 45.2,
+      "cost": 0.025,
+      "startTime": "2024-01-15T10:00:00Z",
+      "endTime": "2024-01-15T10:00:45Z"
+    }
+  ],
+  "summary": {
+    "totalExercises": 5,
+    "successCount": 4,
+    "failureCount": 1,
+    "successRate": 0.8,
+    "averageTurns": 4.2,
+    "averageTime": 62.5,
+    "totalCost": 0.15
+  }
+}
+```
+
+## Supported Languages
+
+Currently supports any language available in Exercism. Start with one language for initial testing:
+
+- `javascript` (recommended for initial testing)
+- `python`
+- `java`
+- `typescript`
+- `go`
+- `rust`
+- And many more...
+
+## Development
+
+### Local Development
+
+```bash
+cd benchmarks
+npm install
+npm run dev setup --language javascript --count 5
+```
+
+### Building
+
+```bash
+npm run build
+```
+
+### Testing
+
+```bash
+npm test
+```
+
+## Architecture
+
+- **Docker Container**: Isolated environment with Node.js, Git, and all dependencies
+- **Exercise Cloning**: Based on Aider's approach, clones from Exercism repositories
+- **Agent Integration**: Instantiates Knowhow agents programmatically
+- **Metrics Collection**: Tracks turns, time, cost, and success rates
+- **Result Recording**: Outputs detailed JSON reports
+
+## Limitations (MVP)
+
+This is an MVP implementation with the following limitations:
+- Single language support per run
+- Basic metrics collection
+- Simple failure detection
+- Minimal configuration options
+
+Future versions will expand these capabilities based on initial results.
+
+## Troubleshooting
+
+### Container Build Issues
+- Ensure Docker has enough memory allocated
+- Check that the Knowhow codebase is properly copied into the container
+
+### Exercise Setup Issues
+- Verify internet connectivity for cloning repositories
+- Check that the specified language track exists in Exercism
+
+### Agent Execution Issues
+- Review the output logs for specific error messages
+- Verify model and provider configuration
+- Check API key availability in the container environment

package/benchmarks/docker/Dockerfile

@@ -0,0 +1,68 @@
+# Use Node.js 18 as base image
+FROM node:18-bullseye
+
+# Install system dependencies (cached layer - rarely changes)
+RUN apt-get update && apt-get install -y \
+    git \
+    curl \
+    wget \
+    python3 \
+    python3-pip \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Set working directory
+WORKDIR /app
+
+# Copy package files for dependency caching
+COPY package*.json ./
+# Install main dependencies (cached until package.json changes)
+RUN npm install
+
+# Copy build configuration (cached until config changes)
+COPY tsconfig.json ./
+
+# Copy source code (this layer will be invalidated on source changes)
+COPY src/ ./src/
+
+# Build the main project (will rebuild when source changes)
+RUN npm run compile
+
+COPY . .
+
+# Now set up benchmarks in a subdirectory
+WORKDIR /app/benchmarks
+
+# Copy benchmarks package.json for dependency caching
+COPY benchmarks/package*.json ./
+
+# Create a symlink to make @tyvm/knowhow available
+RUN mkdir -p node_modules/@tyvm && \
+    ln -sf /app/dist node_modules/@tyvm/knowhow
+
+# Install benchmarks dependencies (cached until benchmarks package.json changes)
+RUN npm install
+
+# Copy benchmarks source and build configuration
+COPY benchmarks/src/ ./src/
+COPY benchmarks/tsconfig.json ./
+
+# Build benchmarks (will rebuild when benchmarks source changes)
+RUN npm run build
+
+# Make CLI executable
+RUN chmod +x dist/cli.js
+
+# Environment variables for AI providers
+ENV OPENAI_API_KEY=""
+ENV ANTHROPIC_API_KEY=""
+ENV GEMINI_API_KEY=""
+ENV XAI_API_KEY=""
+ENV CONTAINER=true
+
+# Create exercises directory and set working directory
+RUN mkdir -p /app/exercises
+WORKDIR /app/exercises
+
+# Default command
+CMD ["node", "/app/benchmarks/dist/cli.js"]

package/benchmarks/example-config.yml

@@ -0,0 +1,27 @@
+# Example configuration for Knowhow benchmarks
+# This shows the available options - CLI arguments override these values
+
+language: javascript
+maxExercises: 10
+model: gpt-4o-mini
+provider: openai
+
+limits:
+  maxTurns: 20
+  maxTime: 300    # seconds
+  maxCost: 1.0    # dollars
+
+output:
+  file: results.json
+  format: json
+
+# Future configuration options (not yet implemented)
+# filters:
+#   difficulty: ["easy", "medium"]
+#   topics: ["arrays", "strings"]
+# 
+# agents:
+#   - name: "developer"
+#     config: {}
+#   - name: "patcher" 
+#     config: {}

package/benchmarks/jest.config.js

@@ -0,0 +1,13 @@
+module.exports = {
+  preset: 'ts-jest',
+  testEnvironment: 'node',
+  roots: ['<rootDir>/src'],
+  testMatch: ['**/__tests__/**/*.ts', '**/?(*.)+(spec|test).ts'],
+  transform: {
+    '^.+\\.ts$': 'ts-jest',
+  },
+  collectCoverageFrom: [
+    'src/**/*.ts',
+    '!src/**/*.d.ts',
+  ],
+};