@mastra/longmemeval 0.1.1-alpha.0

package/CHANGELOG.md

@@ -0,0 +1,13 @@
+# @mastra/longmemeval
+
+## 0.1.1-alpha.0
+
+### Patch Changes
+
+- Updated dependencies [0b56518]
+- Updated dependencies [2ba5b76]
+- Updated dependencies [c3a30de]
+- Updated dependencies [cf3a184]
+- Updated dependencies [d6bfd60]
+  - @mastra/core@0.10.15-alpha.1
+  - @mastra/memory@0.11.3-alpha.1

package/DATA_DOWNLOAD_GUIDE.md

@@ -0,0 +1,117 @@
+# LongMemEval Dataset Download Guide
+
+The LongMemEval datasets are large files (several GB) hosted on HuggingFace with Git LFS. Here are all the ways to download them:
+
+## Option 1: JavaScript/Node.js Download
+
+### 1. Get your HuggingFace token
+
+- Go to https://huggingface.co/settings/tokens
+- Create a new token with read permissions
+- Copy the token
+
+### 2. Set the token as environment variable
+
+```bash
+export HF_TOKEN=your_token_here
+# or
+export HUGGINGFACE_TOKEN=your_token_here
+```
+
+### 3. Install dependencies and download
+
+```bash
+pnpm install
+pnpm download:hf
+```
+
+## Option 2: Git LFS
+
+### 1. Install Git LFS
+
+```bash
+# macOS
+brew install git-lfs
+
+# Ubuntu/Debian
+sudo apt-get install git-lfs
+
+# Initialize Git LFS
+git lfs install
+```
+
+### 2. Clone with Git LFS
+
+```bash
+git clone https://huggingface.co/datasets/xiaowu0162/longmemeval
+cd longmemeval
+cp *.json ../data/
+```
+
+## Option 3: Manual Download from Google Drive
+
+### 1. Download the archive
+
+Go to: https://drive.google.com/file/d/1zJgtYRFhOh5zDQzzatiddfjYhFSnyQ80/view
+
+### 2. Extract the files
+
+```bash
+cd packages/longmemeval/data
+tar -xzvf ~/Downloads/longmemeval_data.tar.gz
+```
+
+### 3. Verify the files
+
+```bash
+ls -lh *.json
+# You should see:
+# - longmemeval_s.json (~40MB)
+# - longmemeval_m.json (~200MB)
+# - longmemeval_oracle.json (~2MB)
+```
+
+## Option 4: Direct Browser Download
+
+If you have a HuggingFace account:
+
+1. Go to https://huggingface.co/datasets/xiaowu0162/longmemeval
+2. Click on "Files and versions"
+3. Download each JSON file directly
+4. Move them to `packages/longmemeval/data/`
+
+## Troubleshooting
+
+### "Entry not found" or small files (15 bytes)
+
+This means the download failed due to authentication. Use one of the authenticated methods above.
+
+### Git LFS bandwidth exceeded
+
+HuggingFace has bandwidth limits. Try:
+
+- Using the Google Drive link instead
+- Waiting until the next day when bandwidth resets
+- Using a different download method
+
+### Permission denied
+
+Make sure you're logged in to HuggingFace and have accepted any dataset terms of use.
+
+## Verification
+
+After downloading, verify the files:
+
+```bash
+# Check file sizes
+ls -lh data/*.json
+
+# Check file content (should be valid JSON)
+head -n 5 data/longmemeval_s.json
+```
+
+Expected sizes:
+
+- `longmemeval_oracle.json`: ~2MB
+- `longmemeval_s.json`: ~40MB
+- `longmemeval_m.json`: ~200MB

package/LICENSE.md

@@ -0,0 +1,15 @@
+# Apache License 2.0
+
+Copyright (c) 2025 Kepler Software, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

package/README.md

@@ -0,0 +1,173 @@
+# LongMemEval Benchmark for Mastra
+
+This package implements the [LongMemEval benchmark](https://arxiv.org/abs/2410.10813) ([+Github](https://github.com/xiaowu0162/LongMemEval)) for testing Mastra's long-term memory capabilities.
+
+## About LongMemEval
+
+LongMemEval is a comprehensive benchmark designed by researchers to evaluate the long-term memory capabilities of chat assistants. It was introduced in the paper:
+
+**"LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory"**  
+_Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu (ICLR 2025)_  
+📄 [Paper](https://arxiv.org/abs/2410.10813) | 🌐 [Website](https://xiaowu0162.github.io/long-mem-eval/) | 🤗 [Dataset](https://huggingface.co/datasets/xiaowu0162/longmemeval)
+
+### What LongMemEval Tests
+
+The benchmark evaluates five core long-term memory abilities through 500 meticulously curated questions:
+
+1. **Information Extraction** - Recalling specific information from extensive interactive histories
+2. **Multi-Session Reasoning** - Synthesizing information across multiple history sessions
+3. **Knowledge Updates** - Handling information that changes over time
+4. **Temporal Reasoning** - Understanding time-based relationships in conversations
+5. **Abstention** - Recognizing when information is not available
+
+### Why This Matters
+
+Current LLMs show a 30-60% performance drop when tested on LongMemEval, revealing significant challenges in maintaining coherent long-term memory. This benchmark helps identify and improve these limitations.
+
+## Quick Start
+
+```bash
+# From packages/longmemeval directory
+
+# 1. Set your API keys
+export OPENAI_API_KEY=your_openai_key_here
+export HF_TOKEN=your_huggingface_token_here  # For automatic dataset download
+
+# 2. Run a benchmark (downloads datasets automatically if needed)
+pnpm bench:s          # Run small dataset (10 parallel requests)
+pnpm bench:m          # Run medium dataset (10 parallel requests)
+pnpm bench:oracle     # Run oracle dataset (10 parallel requests)
+
+# Or run quick 10-question tests
+pnpm bench:s:quick    # Test with 10 questions from small dataset
+pnpm bench:m:quick    # Test with 10 questions from medium dataset
+pnpm bench:oracle:quick # Test with 10 questions from oracle dataset
+```
+
+**Note:** The benchmark will automatically download datasets on first run. Get your HuggingFace token from https://huggingface.co/settings/tokens
+
+## Manual Setup
+
+### 1. Install Dependencies
+
+```bash
+# From the monorepo root
+pnpm install
+pnpm build
+```
+
+### 2. Download Dataset
+
+```bash
+# Set your HuggingFace token
+export HF_TOKEN=your_token_here
+
+# Download datasets (no Python or Git LFS required)
+pnpm download
+```
+
+If automatic download fails, see [DOWNLOAD_GUIDE.md](./DOWNLOAD_GUIDE.md) for manual download instructions.
+
+## Usage
+
+### Run Benchmark
+
+```bash
+# From packages/longmemeval directory
+
+# Quick commands for each dataset (10 parallel requests)
+pnpm bench:s          # Small dataset (full run)
+pnpm bench:m          # Medium dataset (full run)
+pnpm bench:oracle     # Oracle dataset (full run)
+
+# Quick test runs (10 questions only, 5 parallel)
+pnpm bench:s:quick    # Small dataset (quick test)
+pnpm bench:m:quick    # Medium dataset (quick test)
+pnpm bench:oracle:quick # Oracle dataset (quick test)
+
+# Advanced: Use full CLI with custom options
+pnpm cli run --dataset longmemeval_s --model gpt-4o
+
+# Adjust parallelization (default: 5)
+pnpm cli run --dataset longmemeval_s --model gpt-4o --concurrency 20
+
+# Graceful shutdown: Press Ctrl+C to stop and save progress
+
+# Run with specific memory configuration
+pnpm cli run --dataset longmemeval_s --memory-config last-k --model gpt-4o
+pnpm cli run --dataset longmemeval_s --memory-config semantic-recall --model gpt-4o
+pnpm cli run --dataset longmemeval_s --memory-config working-memory --model gpt-4o
+
+# Custom subset size
+pnpm cli run --dataset longmemeval_oracle --model gpt-4o --subset 25
+```
+
+### View Dataset Statistics
+
+```bash
+pnpm cli stats --dataset longmemeval_s
+```
+
+### Evaluate Existing Results
+
+```bash
+pnpm cli evaluate --results ./results/run_12345/results.jsonl --dataset longmemeval_s
+```
+
+### Generate Report
+
+```bash
+pnpm cli report --results ./results/
+```
+
+## Memory Configurations
+
+- **full-history**: Provide complete chat history (baseline)
+- **last-k**: Use Mastra's lastMessages configuration (last 20 messages)
+- **semantic-recall**: Use Mastra's semantic recall feature (requires vector store)
+- **working-memory**: Use Mastra's working memory with template
+- **combined**: Combination of last-k and semantic recall
+
+## Output
+
+Results are saved in the `results/` directory with:
+
+- `results.jsonl`: Individual question results
+- `hypotheses.json`: Model responses
+- `questions.json`: Questions for reference
+- `metrics.json`: Aggregated metrics and configuration
+
+## Benchmark Datasets
+
+LongMemEval provides three dataset variants:
+
+- **longmemeval_s (Small)**: ~115k tokens per question (30-40 sessions)
+  - Designed to fit within 128k context windows
+  - Tests memory across dozens of conversation sessions
+- **longmemeval_m (Medium)**: ~1.5M tokens per question (500 sessions)
+  - Challenges even the largest context windows
+  - Tests memory across hundreds of sessions
+- **longmemeval_oracle**: Only evidence sessions included
+  - Used as a control to verify models can answer when given only relevant context
+  - Helps isolate memory retrieval issues from comprehension issues
+
+## Citation
+
+If you use this benchmark in your research, please cite the original paper:
+
+```bibtex
+@article{wu2024longmemeval,
+  title={LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory},
+  author={Wu, Di and Wang, Hongwei and Yu, Wenhao and Zhang, Yuwei and Chang, Kai-Wei and Yu, Dong},
+  journal={arXiv preprint arXiv:2410.10813},
+  year={2024}
+}
+```
+
+## Extending the Benchmark
+
+To add custom memory configurations:
+
+1. Edit `src/benchmark/runner.ts` and add your configuration to `getMemoryConfig()`
+2. Update the `MemoryConfigType` in `src/data/types.ts`
+3. Implement the configuration logic in `src/memory-adapters/mastra-adapter.ts`

package/USAGE.md

@@ -0,0 +1,105 @@
+# LongMemEval Usage Guide
+
+## Quick Start
+
+### 1. Prepare Data (Required First)
+
+The prepare step processes the dataset through mock agents to populate the storage:
+
+```bash
+# Quick test with 5 questions
+pnpm prepare:quick
+
+# Full dataset with different memory configs
+pnpm prepare:s              # Small dataset, semantic-recall (default)
+pnpm prepare:s:lastk        # Small dataset, last-k messages
+pnpm prepare:s:working      # Small dataset, working-memory
+pnpm prepare:s:combined     # Small dataset, combined (semantic + working memory)
+pnpm prepare:m              # Medium dataset, semantic-recall
+```
+
+### 2. Run Benchmark
+
+After preparing data, run the benchmark:
+
+```bash
+# Quick test
+pnpm run:quick
+
+# Full runs
+pnpm run:s              # Small dataset with semantic-recall (default)
+pnpm run:s:lastk        # Small dataset with last-k
+pnpm run:s:working      # Small dataset with working-memory
+pnpm run:s:combined     # Small dataset with combined
+```
+
+## Full CLI Options
+
+### Prepare Command
+
+```bash
+pnpm cli prepare \
+  -d <dataset>              # longmemeval_s, longmemeval_m, longmemeval_oracle
+  -c <memory-config>        # full-history, last-k, semantic-recall, working-memory, combined
+  [--subset <n>]            # Process only n questions
+  [--output <dir>]          # Output directory (default: ./prepared-data)
+```
+
+### Run Command
+
+```bash
+pnpm cli run \
+  -d <dataset>              # longmemeval_s, longmemeval_m, longmemeval_oracle
+  -m <model>                # Model name (e.g., gpt-4o)
+  -c <memory-config>        # full-history, last-k, semantic-recall, working-memory, combined
+  [--subset <n>]            # Run only n questions
+  [--concurrency <n>]       # Parallel requests (default: 5)
+  [--prepared-data <dir>]   # Prepared data directory
+  [--output <dir>]          # Results directory
+```
+
+## Memory Configurations
+
+- **semantic-recall** (default): Uses embeddings to find relevant messages (requires OPENAI_API_KEY)
+- **last-k**: Loads last 50 messages only
+- **working-memory**: Maintains a summary of user context
+- **combined**: Semantic recall + working memory
+
+Note: `full-history` is available but not recommended for testing memory systems as it defeats the purpose by loading everything into context.
+
+## Environment Variables
+
+```bash
+# Required for running benchmarks
+export OPENAI_API_KEY=your-key-here
+
+# Optional for downloading datasets
+export HF_TOKEN=your-huggingface-token
+```
+
+## Example Workflow
+
+```bash
+# 1. Test with small subset first
+pnpm prepare:quick
+pnpm run:quick
+
+# 2. Run full benchmark with semantic recall
+pnpm prepare:s
+pnpm run:s
+```
+
+## Viewing Results
+
+Results are saved in `./results/run_<timestamp>/`:
+
+- `results.jsonl`: Raw evaluation results
+- `metrics.json`: Aggregated metrics
+
+```bash
+# View all runs
+pnpm cli report -r ./results
+
+# Check specific metrics
+cat results/run_*/metrics.json | jq '.overall_accuracy'
+```

package/package.json

@@ -0,0 +1,58 @@
+{
+  "name": "@mastra/longmemeval",
+  "version": "0.1.1-alpha.0",
+  "description": "LongMemEval benchmark implementation for Mastra Memory",
+  "dependencies": {
+    "@ai-sdk/openai": "^1.3.23",
+    "@ai-sdk/provider": "^1.1.3",
+    "@huggingface/hub": "^0.15.1",
+    "@mastra/rag": "^1.0.2",
+    "@node-rs/xxhash": "^1.7.6",
+    "ai": "^4.3.17",
+    "async-mutex": "^0.5.0",
+    "chalk": "^5.3.0",
+    "commander": "^12.1.0",
+    "fastq": "^1.19.1",
+    "imvectordb": "^0.0.6",
+    "openai": "^4.73.1",
+    "ora": "^8.1.1",
+    "zod": "^3.23.8",
+    "@mastra/core": "0.10.15-alpha.1",
+    "@mastra/fastembed": "0.10.1",
+    "@mastra/libsql": "0.11.0",
+    "@mastra/memory": "0.11.3-alpha.1"
+  },
+  "devDependencies": {
+    "@ai-sdk/google": "^1.2.19",
+    "@types/node": "^22.10.2",
+    "tsx": "^4.19.2",
+    "typescript": "^5.7.2",
+    "vitest": "^2.1.8"
+  },
+  "engines": {
+    "node": ">=20"
+  },
+  "scripts": {
+    "test": "vitest",
+    "typecheck": "tsc --noEmit",
+    "generate-embeddings": "tsx scripts/generate-embeddings.ts",
+    "setup": "tsx scripts/setup.ts",
+    "download": "tsx scripts/download.ts",
+    "find-failed": "tsx scripts/find-failed.ts",
+    "clean-failed": "tsx scripts/find-failed.ts --delete",
+    "generate-wm-templates": "tsx scripts/generate-wm-templates.ts",
+    "prepare:s:semantic": "tsx src/cli.ts prepare -d longmemeval_s -c semantic-recall --concurrency 5",
+    "bench:s:semantic": "tsx src/cli.ts run --dataset longmemeval_s --model gpt-4o --memory-config semantic-recall --concurrency 30",
+    "prepare:s:working": "tsx src/cli.ts prepare -d longmemeval_s -c working-memory --concurrency 35",
+    "bench:s:working": "tsx src/cli.ts run --dataset longmemeval_s --model gpt-4o --memory-config working-memory --concurrency 10",
+    "prepare:s:combined": "tsx src/cli.ts prepare -d longmemeval_s -c combined --concurrency 10",
+    "bench:s:combined": "tsx src/cli.ts run -d longmemeval_s -m gpt-4o -c combined --concurrency 20",
+    "prepare:s:combined-tailored": "tsx src/cli.ts prepare -d longmemeval_s -c combined-tailored --concurrency 20",
+    "bench:s:combined-tailored": "tsx src/cli.ts run -d longmemeval_s -m gpt-4o -c combined-tailored --concurrency 30",
+    "prepare:s:working-tailored": "tsx src/cli.ts prepare -d longmemeval_s -c working-memory-tailored --concurrency 20",
+    "bench:s:working-tailored": "tsx src/cli.ts run -d longmemeval_s -m gpt-4o -c working-memory-tailored --concurrency 30",
+    "results": "tsx src/cli.ts results",
+    "results:all": "tsx src/cli.ts results --all",
+    "results:s": "tsx src/cli.ts results -d longmemeval_s"
+  }
+}

package/scripts/download.ts

@@ -0,0 +1,180 @@
+#!/usr/bin/env tsx
+
+import { downloadFile } from '@huggingface/hub';
+import { createWriteStream, existsSync, mkdirSync, statSync } from 'fs';
+import { join } from 'path';
+import ora from 'ora';
+import chalk from 'chalk';
+import { pipeline } from 'stream/promises';
+
+const REPO_ID = 'xiaowu0162/longmemeval';
+const DATA_DIR = join(process.cwd(), 'data');
+
+const FILES = [
+  { filename: 'longmemeval_oracle.json', repoPath: 'longmemeval_oracle' },
+  { filename: 'longmemeval_s.json', repoPath: 'longmemeval_s' },
+  { filename: 'longmemeval_m.json', repoPath: 'longmemeval_m' },
+];
+
+function formatFileSize(bytes: number): string {
+  if (bytes < 1024) return bytes + ' B';
+  const kb = bytes / 1024;
+  if (kb < 1024) return kb.toFixed(1) + ' KB';
+  const mb = kb / 1024;
+  if (mb < 1024) return mb.toFixed(1) + ' MB';
+  const gb = mb / 1024;
+  return gb.toFixed(1) + ' GB';
+}
+
+async function getFileSize(filePath: string): Promise<number> {
+  try {
+    const stats = statSync(filePath);
+    return stats.size;
+  } catch {
+    return 0;
+  }
+}
+
+async function downloadWithFetch(url: string, outputPath: string, token: string): Promise<void> {
+  const response = await fetch(url, {
+    headers: {
+      Authorization: `Bearer ${token}`,
+      'User-Agent': 'longmemeval-downloader/1.0',
+    },
+  });
+
+  if (!response.ok) {
+    throw new Error(`HTTP ${response.status}: ${response.statusText}`);
+  }
+
+  const buffer = await response.arrayBuffer();
+  const { writeFile } = await import('fs/promises');
+  await writeFile(outputPath, Buffer.from(buffer));
+}
+
+async function main() {
+  console.log(chalk.blue('\n📥 LongMemEval Dataset Downloader\n'));
+
+  // Create data directory if it doesn't exist
+  if (!existsSync(DATA_DIR)) {
+    mkdirSync(DATA_DIR, { recursive: true });
+  }
+
+  // Check if all files already exist
+  let existingCount = 0;
+  for (const fileInfo of FILES) {
+    const outputPath = join(DATA_DIR, fileInfo.filename);
+    const size = await getFileSize(outputPath);
+    if (size > 1000000) {
+      // > 1MB
+      console.log(chalk.green(`✓ ${fileInfo.filename} already exists (${formatFileSize(size)})`));
+      existingCount++;
+    }
+  }
+
+  if (existingCount === FILES.length) {
+    console.log(chalk.green('\n✅ All datasets already downloaded!\n'));
+    console.log(chalk.gray('You can now run the benchmark:'));
+    console.log(chalk.cyan('  pnpm cli run --dataset longmemeval_s --model gpt-4o'));
+    return;
+  }
+
+  // Check for HuggingFace token
+  const token = process.env.HF_TOKEN || process.env.HUGGINGFACE_TOKEN;
+
+  if (!token) {
+    console.log(chalk.yellow('⚠️  No HuggingFace token found!\n'));
+    console.log(chalk.blue('The LongMemEval datasets require authentication to download.\n'));
+
+    console.log(chalk.gray('1. Get your token from:'));
+    console.log(chalk.cyan('   https://huggingface.co/settings/tokens\n'));
+
+    console.log(chalk.gray('2. Set it as an environment variable:'));
+    console.log(chalk.cyan('   export HF_TOKEN=your_token_here\n'));
+
+    console.log(chalk.gray('3. Run this script again:'));
+    console.log(chalk.cyan('   pnpm download\n'));
+
+    console.log(chalk.blue('Alternative: Download manually from Google Drive'));
+    console.log(chalk.gray('See DOWNLOAD_GUIDE.md for instructions'));
+
+    process.exit(1);
+  }
+
+  // Download missing files
+  console.log(chalk.blue('Downloading missing datasets...\n'));
+  let successCount = existingCount;
+
+  for (const fileInfo of FILES) {
+    const { filename, repoPath } = fileInfo;
+    const outputPath = join(DATA_DIR, filename);
+
+    // Skip if already exists
+    const existingSize = await getFileSize(outputPath);
+    if (existingSize > 1000000) {
+      continue;
+    }
+
+    const spinner = ora(`Downloading ${filename}...`).start();
+
+    try {
+      // Try HuggingFace Hub API first
+      try {
+        const response = await downloadFile({
+          repo: REPO_ID,
+          path: repoPath,
+          credentials: { accessToken: token },
+        });
+
+        if (response && response.body) {
+          const fileStream = createWriteStream(outputPath);
+          await pipeline(response.body as any, fileStream);
+        } else {
+          throw new Error('Empty response');
+        }
+      } catch (hubError: any) {
+        // Fallback to direct HTTPS download
+        const directUrl = `https://huggingface.co/datasets/${REPO_ID}/resolve/main/${repoPath}?download=true`;
+        await downloadWithFetch(directUrl, outputPath, token);
+      }
+
+      // Verify file size
+      const downloadedSize = await getFileSize(outputPath);
+      if (downloadedSize > 1000000) {
+        spinner.succeed(`Downloaded ${filename} (${formatFileSize(downloadedSize)})`);
+        successCount++;
+      } else {
+        spinner.fail(`Downloaded ${filename} but file seems too small (${formatFileSize(downloadedSize)})`);
+        // Remove invalid file
+        const { unlink } = await import('fs/promises');
+        await unlink(outputPath).catch(() => {});
+      }
+    } catch (error: any) {
+      spinner.fail(`Failed to download ${filename}`);
+      console.error(chalk.red(`  Error: ${error.message}`));
+
+      if (error.message.includes('401') || error.message.includes('403')) {
+        console.log(chalk.yellow('\n  Authentication issue. Please check:'));
+        console.log(chalk.gray('  - Your token is valid'));
+        console.log(chalk.gray('  - You have accepted the dataset terms of use'));
+        console.log(chalk.cyan(`  - Visit: https://huggingface.co/datasets/${REPO_ID}`));
+      }
+    }
+  }
+
+  // Final summary
+  console.log('');
+  if (successCount === FILES.length) {
+    console.log(chalk.green('✅ All datasets downloaded successfully!\n'));
+    console.log(chalk.gray('You can now run the benchmark:'));
+    console.log(chalk.cyan('  pnpm cli run --dataset longmemeval_s --model gpt-4o'));
+  } else {
+    console.log(chalk.yellow(`⚠️  Downloaded ${successCount}/${FILES.length} files\n`));
+    console.log(chalk.blue('If downloads failed, please check:'));
+    console.log(chalk.gray('- Your HuggingFace token is valid'));
+    console.log(chalk.gray('- You have accepted the dataset terms (if any)'));
+    console.log(chalk.gray('\nAlternatively, see DOWNLOAD_GUIDE.md for manual download instructions'));
+  }
+}
+
+main().catch(console.error);