@ruvector/attention-wasm 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 rUv
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,193 @@
+# ruvector-attention-wasm
+
+WebAssembly bindings for the ruvector-attention package, providing high-performance attention mechanisms for browser and Node.js environments.
+
+## Features
+
+- **Multiple Attention Mechanisms**:
+  - Scaled Dot-Product Attention
+  - Multi-Head Attention
+  - Hyperbolic Attention (for hierarchical data)
+  - Linear Attention (Performer-style)
+  - Flash Attention (memory-efficient)
+  - Local-Global Attention
+  - Mixture of Experts (MoE) Attention
+
+- **Training Utilities**:
+  - InfoNCE contrastive loss
+  - Adam optimizer
+  - AdamW optimizer (with decoupled weight decay)
+  - Learning rate scheduler (warmup + cosine decay)
+
+- **TypeScript Support**: Full type definitions and modern API
+
+## Installation
+
+```bash
+npm install ruvector-attention-wasm
+```
+
+## Usage
+
+### TypeScript/JavaScript
+
+```typescript
+import { initialize, MultiHeadAttention, utils } from 'ruvector-attention-wasm';
+
+// Initialize WASM module
+await initialize();
+
+// Create multi-head attention
+const attention = new MultiHeadAttention({ dim: 64, numHeads: 8 });
+
+// Prepare inputs
+const query = new Float32Array(64);
+const keys = [new Float32Array(64), new Float32Array(64)];
+const values = [new Float32Array(64), new Float32Array(64)];
+
+// Compute attention
+const output = attention.compute(query, keys, values);
+
+// Use utilities
+const similarity = utils.cosineSimilarity(query, keys[0]);
+```
+
+### Advanced Examples
+
+#### Hyperbolic Attention
+
+```typescript
+import { HyperbolicAttention } from 'ruvector-attention-wasm';
+
+const hyperbolic = new HyperbolicAttention({
+  dim: 128,
+  curvature: 1.0
+});
+
+const output = hyperbolic.compute(query, keys, values);
+```
+
+#### MoE Attention with Expert Stats
+
+```typescript
+import { MoEAttention } from 'ruvector-attention-wasm';
+
+const moe = new MoEAttention({
+  dim: 64,
+  numExperts: 4,
+  topK: 2
+});
+
+const output = moe.compute(query, keys, values);
+
+// Get expert utilization
+const stats = moe.getExpertStats();
+console.log('Load balance:', stats.loadBalance);
+```
+
+#### Training with InfoNCE Loss
+
+```typescript
+import { InfoNCELoss, Adam } from 'ruvector-attention-wasm';
+
+const loss = new InfoNCELoss(0.07);
+const optimizer = new Adam(paramCount, {
+  learningRate: 0.001,
+  beta1: 0.9,
+  beta2: 0.999,
+});
+
+// Training loop
+const lossValue = loss.compute(anchor, positive, negatives);
+optimizer.step(params, gradients);
+```
+
+#### Learning Rate Scheduling
+
+```typescript
+import { LRScheduler, AdamW } from 'ruvector-attention-wasm';
+
+const scheduler = new LRScheduler({
+  initialLR: 0.001,
+  warmupSteps: 1000,
+  totalSteps: 10000,
+});
+
+const optimizer = new AdamW(paramCount, {
+  learningRate: scheduler.getLR(),
+  weightDecay: 0.01,
+});
+
+// Training loop
+for (let step = 0; step < 10000; step++) {
+  optimizer.learningRate = scheduler.getLR();
+  optimizer.step(params, gradients);
+  scheduler.step();
+}
+```
+
+## Building from Source
+
+### Prerequisites
+
+- Rust 1.70+
+- wasm-pack
+
+### Build Commands
+
+```bash
+# Build for web (ES modules)
+wasm-pack build --target web --out-dir pkg
+
+# Build for Node.js
+wasm-pack build --target nodejs --out-dir pkg-node
+
+# Build for bundlers (webpack, vite, etc.)
+wasm-pack build --target bundler --out-dir pkg-bundler
+
+# Run tests
+wasm-pack test --headless --firefox
+```
+
+## API Reference
+
+### Attention Mechanisms
+
+- `MultiHeadAttention` - Standard multi-head attention
+- `HyperbolicAttention` - Attention in hyperbolic space
+- `LinearAttention` - Linear complexity attention (Performer)
+- `FlashAttention` - Memory-efficient attention
+- `LocalGlobalAttention` - Combined local and global attention
+- `MoEAttention` - Mixture of Experts attention
+- `scaledDotAttention()` - Functional API for basic attention
+
+### Training
+
+- `InfoNCELoss` - Contrastive loss function
+- `Adam` - Adam optimizer
+- `AdamW` - AdamW optimizer with weight decay
+- `LRScheduler` - Learning rate scheduler
+
+### Utilities
+
+- `utils.cosineSimilarity()` - Cosine similarity between vectors
+- `utils.l2Norm()` - L2 norm of a vector
+- `utils.normalize()` - Normalize vector to unit length
+- `utils.softmax()` - Apply softmax transformation
+- `utils.attentionWeights()` - Compute attention weights from scores
+- `utils.batchNormalize()` - Batch normalization
+- `utils.randomOrthogonalMatrix()` - Generate random orthogonal matrix
+- `utils.pairwiseDistances()` - Compute pairwise distances
+
+## Performance
+
+The WASM bindings provide near-native performance for attention computations:
+
+- Optimized with `opt-level = "s"` and LTO
+- SIMD acceleration where available
+- Efficient memory management
+- Zero-copy data transfer where possible
+
+## License
+
+MIT OR Apache-2.0

package/js/index.ts

@@ -0,0 +1,412 @@
+/**
+ * TypeScript wrapper for ruvector-attention-wasm
+ * Provides a clean, type-safe API for attention mechanisms
+ */
+
+import init, * as wasm from '../pkg/ruvector_attention_wasm';
+import type {
+  AttentionConfig,
+  MultiHeadConfig,
+  HyperbolicConfig,
+  LinearAttentionConfig,
+  FlashAttentionConfig,
+  LocalGlobalConfig,
+  MoEConfig,
+  TrainingConfig,
+  SchedulerConfig,
+  ExpertStats,
+  AttentionType,
+} from './types';
+
+export * from './types';
+
+let initialized = false;
+
+/**
+ * Initialize the WASM module
+ * Must be called before using any attention mechanisms
+ */
+export async function initialize(): Promise<void> {
+  if (!initialized) {
+    await init();
+    initialized = true;
+  }
+}
+
+/**
+ * Get the version of the ruvector-attention-wasm package
+ */
+export function version(): string {
+  return wasm.version();
+}
+
+/**
+ * Get list of available attention mechanisms
+ */
+export function availableMechanisms(): AttentionType[] {
+  return wasm.available_mechanisms() as AttentionType[];
+}
+
+/**
+ * Multi-head attention mechanism
+ */
+export class MultiHeadAttention {
+  private inner: wasm.WasmMultiHeadAttention;
+
+  constructor(config: MultiHeadConfig) {
+    this.inner = new wasm.WasmMultiHeadAttention(config.dim, config.numHeads);
+  }
+
+  /**
+   * Compute multi-head attention
+   */
+  compute(query: Float32Array, keys: Float32Array[], values: Float32Array[]): Float32Array {
+    const result = this.inner.compute(query, keys, values);
+    return new Float32Array(result);
+  }
+
+  get numHeads(): number {
+    return this.inner.num_heads;
+  }
+
+  get dim(): number {
+    return this.inner.dim;
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * Hyperbolic attention mechanism
+ */
+export class HyperbolicAttention {
+  private inner: wasm.WasmHyperbolicAttention;
+
+  constructor(config: HyperbolicConfig) {
+    this.inner = new wasm.WasmHyperbolicAttention(config.dim, config.curvature);
+  }
+
+  compute(query: Float32Array, keys: Float32Array[], values: Float32Array[]): Float32Array {
+    const result = this.inner.compute(query, keys, values);
+    return new Float32Array(result);
+  }
+
+  get curvature(): number {
+    return this.inner.curvature;
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * Linear attention (Performer-style)
+ */
+export class LinearAttention {
+  private inner: wasm.WasmLinearAttention;
+
+  constructor(config: LinearAttentionConfig) {
+    this.inner = new wasm.WasmLinearAttention(config.dim, config.numFeatures);
+  }
+
+  compute(query: Float32Array, keys: Float32Array[], values: Float32Array[]): Float32Array {
+    const result = this.inner.compute(query, keys, values);
+    return new Float32Array(result);
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * Flash attention mechanism
+ */
+export class FlashAttention {
+  private inner: wasm.WasmFlashAttention;
+
+  constructor(config: FlashAttentionConfig) {
+    this.inner = new wasm.WasmFlashAttention(config.dim, config.blockSize);
+  }
+
+  compute(query: Float32Array, keys: Float32Array[], values: Float32Array[]): Float32Array {
+    const result = this.inner.compute(query, keys, values);
+    return new Float32Array(result);
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * Local-global attention mechanism
+ */
+export class LocalGlobalAttention {
+  private inner: wasm.WasmLocalGlobalAttention;
+
+  constructor(config: LocalGlobalConfig) {
+    this.inner = new wasm.WasmLocalGlobalAttention(
+      config.dim,
+      config.localWindow,
+      config.globalTokens
+    );
+  }
+
+  compute(query: Float32Array, keys: Float32Array[], values: Float32Array[]): Float32Array {
+    const result = this.inner.compute(query, keys, values);
+    return new Float32Array(result);
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * Mixture of Experts attention
+ */
+export class MoEAttention {
+  private inner: wasm.WasmMoEAttention;
+
+  constructor(config: MoEConfig) {
+    this.inner = new wasm.WasmMoEAttention(config.dim, config.numExperts, config.topK);
+  }
+
+  compute(query: Float32Array, keys: Float32Array[], values: Float32Array[]): Float32Array {
+    const result = this.inner.compute(query, keys, values);
+    return new Float32Array(result);
+  }
+
+  getExpertStats(): ExpertStats {
+    return this.inner.expert_stats() as ExpertStats;
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * InfoNCE contrastive loss
+ */
+export class InfoNCELoss {
+  private inner: wasm.WasmInfoNCELoss;
+
+  constructor(temperature: number = 0.07) {
+    this.inner = new wasm.WasmInfoNCELoss(temperature);
+  }
+
+  compute(anchor: Float32Array, positive: Float32Array, negatives: Float32Array[]): number {
+    return this.inner.compute(anchor, positive, negatives);
+  }
+
+  computeMultiPositive(
+    anchor: Float32Array,
+    positives: Float32Array[],
+    negatives: Float32Array[]
+  ): number {
+    return this.inner.compute_multi_positive(anchor, positives, negatives);
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * Adam optimizer
+ */
+export class Adam {
+  private inner: wasm.WasmAdam;
+
+  constructor(paramCount: number, config: TrainingConfig) {
+    this.inner = new wasm.WasmAdam(
+      paramCount,
+      config.learningRate,
+      config.beta1,
+      config.beta2,
+      config.epsilon
+    );
+  }
+
+  step(params: Float32Array, gradients: Float32Array): void {
+    this.inner.step(params, gradients);
+  }
+
+  reset(): void {
+    this.inner.reset();
+  }
+
+  get learningRate(): number {
+    return this.inner.learning_rate;
+  }
+
+  set learningRate(lr: number) {
+    this.inner.learning_rate = lr;
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * AdamW optimizer (Adam with decoupled weight decay)
+ */
+export class AdamW {
+  private inner: wasm.WasmAdamW;
+
+  constructor(paramCount: number, config: TrainingConfig) {
+    if (!config.weightDecay) {
+      throw new Error('AdamW requires weightDecay parameter');
+    }
+
+    this.inner = new wasm.WasmAdamW(
+      paramCount,
+      config.learningRate,
+      config.weightDecay,
+      config.beta1,
+      config.beta2,
+      config.epsilon
+    );
+  }
+
+  step(params: Float32Array, gradients: Float32Array): void {
+    this.inner.step(params, gradients);
+  }
+
+  reset(): void {
+    this.inner.reset();
+  }
+
+  get learningRate(): number {
+    return this.inner.learning_rate;
+  }
+
+  set learningRate(lr: number) {
+    this.inner.learning_rate = lr;
+  }
+
+  get weightDecay(): number {
+    return this.inner.weight_decay;
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * Learning rate scheduler with warmup and cosine decay
+ */
+export class LRScheduler {
+  private inner: wasm.WasmLRScheduler;
+
+  constructor(config: SchedulerConfig) {
+    this.inner = new wasm.WasmLRScheduler(
+      config.initialLR,
+      config.warmupSteps,
+      config.totalSteps
+    );
+  }
+
+  getLR(): number {
+    return this.inner.get_lr();
+  }
+
+  step(): void {
+    this.inner.step();
+  }
+
+  reset(): void {
+    this.inner.reset();
+  }
+
+  free(): void {
+    this.inner.free();
+  }
+}
+
+/**
+ * Utility functions
+ */
+export const utils = {
+  /**
+   * Compute cosine similarity between two vectors
+   */
+  cosineSimilarity(a: Float32Array, b: Float32Array): number {
+    return wasm.cosine_similarity(a, b);
+  },
+
+  /**
+   * Compute L2 norm of a vector
+   */
+  l2Norm(vec: Float32Array): number {
+    return wasm.l2_norm(vec);
+  },
+
+  /**
+   * Normalize a vector to unit length (in-place)
+   */
+  normalize(vec: Float32Array): void {
+    wasm.normalize(vec);
+  },
+
+  /**
+   * Apply softmax to a vector (in-place)
+   */
+  softmax(vec: Float32Array): void {
+    wasm.softmax(vec);
+  },
+
+  /**
+   * Compute attention weights from scores (in-place)
+   */
+  attentionWeights(scores: Float32Array, temperature?: number): void {
+    wasm.attention_weights(scores, temperature);
+  },
+
+  /**
+   * Batch normalize vectors
+   */
+  batchNormalize(vectors: Float32Array[], epsilon?: number): Float32Array {
+    const result = wasm.batch_normalize(vectors, epsilon);
+    return new Float32Array(result);
+  },
+
+  /**
+   * Generate random orthogonal matrix
+   */
+  randomOrthogonalMatrix(dim: number): Float32Array {
+    const result = wasm.random_orthogonal_matrix(dim);
+    return new Float32Array(result);
+  },
+
+  /**
+   * Compute pairwise distances between vectors
+   */
+  pairwiseDistances(vectors: Float32Array[]): Float32Array {
+    const result = wasm.pairwise_distances(vectors);
+    return new Float32Array(result);
+  },
+};
+
+/**
+ * Simple scaled dot-product attention (functional API)
+ */
+export function scaledDotAttention(
+  query: Float32Array,
+  keys: Float32Array[],
+  values: Float32Array[],
+  scale?: number
+): Float32Array {
+  const result = wasm.scaled_dot_attention(query, keys, values, scale);
+  return new Float32Array(result);
+}
+
+// Re-export WASM module for advanced usage
+export { wasm };

package/js/types.ts

@@ -0,0 +1,108 @@
+/**
+ * TypeScript type definitions for ruvector-attention-wasm
+ */
+
+export interface AttentionConfig {
+  /** Embedding dimension */
+  dim: number;
+  /** Number of attention heads (for multi-head attention) */
+  numHeads?: number;
+  /** Dropout probability */
+  dropout?: number;
+  /** Scaling factor for attention scores */
+  scale?: number;
+  /** Whether to use causal masking */
+  causal?: boolean;
+}
+
+export interface MultiHeadConfig extends AttentionConfig {
+  numHeads: number;
+}
+
+export interface HyperbolicConfig extends AttentionConfig {
+  /** Hyperbolic space curvature */
+  curvature: number;
+}
+
+export interface LinearAttentionConfig extends AttentionConfig {
+  /** Number of random features for kernel approximation */
+  numFeatures: number;
+}
+
+export interface FlashAttentionConfig extends AttentionConfig {
+  /** Block size for tiling */
+  blockSize: number;
+}
+
+export interface LocalGlobalConfig extends AttentionConfig {
+  /** Size of local attention window */
+  localWindow: number;
+  /** Number of global attention tokens */
+  globalTokens: number;
+}
+
+export interface MoEConfig extends AttentionConfig {
+  /** Number of expert attention mechanisms */
+  numExperts: number;
+  /** Number of experts to use per query */
+  topK: number;
+  /** Maximum capacity per expert */
+  expertCapacity?: number;
+  /** Load balancing coefficient */
+  balanceCoeff?: number;
+}
+
+export interface TrainingConfig {
+  /** Learning rate for optimizer */
+  learningRate: number;
+  /** Temperature parameter for contrastive loss */
+  temperature?: number;
+  /** First moment decay rate (Adam/AdamW) */
+  beta1?: number;
+  /** Second moment decay rate (Adam/AdamW) */
+  beta2?: number;
+  /** Weight decay coefficient (AdamW) */
+  weightDecay?: number;
+  /** Numerical stability constant */
+  epsilon?: number;
+}
+
+export interface SchedulerConfig {
+  /** Initial learning rate */
+  initialLR: number;
+  /** Number of warmup steps */
+  warmupSteps: number;
+  /** Total training steps */
+  totalSteps: number;
+}
+
+export interface ExpertStats {
+  /** Number of times each expert was selected */
+  selectionCounts: number[];
+  /** Average load per expert */
+  averageLoad: number[];
+  /** Load balance factor (lower is better) */
+  loadBalance: number;
+}
+
+/**
+ * Attention mechanism types
+ */
+export type AttentionType =
+  | 'scaled_dot_product'
+  | 'multi_head'
+  | 'hyperbolic'
+  | 'linear'
+  | 'flash'
+  | 'local_global'
+  | 'moe';
+
+/**
+ * Optimizer types
+ */
+export type OptimizerType = 'adam' | 'adamw';
+
+/**
+ * Loss function types
+ */
+export type LossType = 'info_nce';

package/package.json

@@ -0,0 +1,45 @@
+{
+  "name": "@ruvector/attention-wasm",
+  "version": "0.1.0",
+  "description": "WebAssembly bindings for ruvector-attention - high-performance attention mechanisms",
+  "main": "pkg/ruvector_attention_wasm.js",
+  "types": "js/index.ts",
+  "files": [
+    "pkg/",
+    "js/"
+  ],
+  "scripts": {
+    "build": "wasm-pack build --target web --out-dir pkg",
+    "build:node": "wasm-pack build --target nodejs --out-dir pkg-node",
+    "build:bundler": "wasm-pack build --target bundler --out-dir pkg-bundler",
+    "build:all": "npm run build && npm run build:node && npm run build:bundler",
+    "test": "wasm-pack test --headless --firefox",
+    "test:chrome": "wasm-pack test --headless --chrome",
+    "clean": "rm -rf pkg pkg-node pkg-bundler target"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/ruvnet/ruvector"
+  },
+  "keywords": [
+    "wasm",
+    "webassembly",
+    "attention",
+    "transformer",
+    "machine-learning",
+    "neural-networks",
+    "hyperbolic",
+    "moe",
+    "flash-attention"
+  ],
+  "author": "rUv",
+  "license": "MIT OR Apache-2.0",
+  "bugs": {
+    "url": "https://github.com/ruvnet/ruvector/issues"
+  },
+  "homepage": "https://ruv.io/ruvector",
+  "devDependencies": {
+    "@types/node": "^20.0.0",
+    "typescript": "^5.0.0"
+  }
+}

package/pkg/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 rUv
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/pkg/README.md

@@ -0,0 +1,193 @@
+# ruvector-attention-wasm
+
+WebAssembly bindings for the ruvector-attention package, providing high-performance attention mechanisms for browser and Node.js environments.
+
+## Features
+
+- **Multiple Attention Mechanisms**:
+  - Scaled Dot-Product Attention
+  - Multi-Head Attention
+  - Hyperbolic Attention (for hierarchical data)
+  - Linear Attention (Performer-style)
+  - Flash Attention (memory-efficient)
+  - Local-Global Attention
+  - Mixture of Experts (MoE) Attention
+
+- **Training Utilities**:
+  - InfoNCE contrastive loss
+  - Adam optimizer
+  - AdamW optimizer (with decoupled weight decay)
+  - Learning rate scheduler (warmup + cosine decay)
+
+- **TypeScript Support**: Full type definitions and modern API
+
+## Installation
+
+```bash
+npm install ruvector-attention-wasm
+```
+
+## Usage
+
+### TypeScript/JavaScript
+
+```typescript
+import { initialize, MultiHeadAttention, utils } from 'ruvector-attention-wasm';
+
+// Initialize WASM module
+await initialize();
+
+// Create multi-head attention
+const attention = new MultiHeadAttention({ dim: 64, numHeads: 8 });
+
+// Prepare inputs
+const query = new Float32Array(64);
+const keys = [new Float32Array(64), new Float32Array(64)];
+const values = [new Float32Array(64), new Float32Array(64)];
+
+// Compute attention
+const output = attention.compute(query, keys, values);
+
+// Use utilities
+const similarity = utils.cosineSimilarity(query, keys[0]);
+```
+
+### Advanced Examples
+
+#### Hyperbolic Attention
+
+```typescript
+import { HyperbolicAttention } from 'ruvector-attention-wasm';
+
+const hyperbolic = new HyperbolicAttention({
+  dim: 128,
+  curvature: 1.0
+});
+
+const output = hyperbolic.compute(query, keys, values);
+```
+
+#### MoE Attention with Expert Stats
+
+```typescript
+import { MoEAttention } from 'ruvector-attention-wasm';
+
+const moe = new MoEAttention({
+  dim: 64,
+  numExperts: 4,
+  topK: 2
+});
+
+const output = moe.compute(query, keys, values);
+
+// Get expert utilization
+const stats = moe.getExpertStats();
+console.log('Load balance:', stats.loadBalance);
+```
+
+#### Training with InfoNCE Loss
+
+```typescript
+import { InfoNCELoss, Adam } from 'ruvector-attention-wasm';
+
+const loss = new InfoNCELoss(0.07);
+const optimizer = new Adam(paramCount, {
+  learningRate: 0.001,
+  beta1: 0.9,
+  beta2: 0.999,
+});
+
+// Training loop
+const lossValue = loss.compute(anchor, positive, negatives);
+optimizer.step(params, gradients);
+```
+
+#### Learning Rate Scheduling
+
+```typescript
+import { LRScheduler, AdamW } from 'ruvector-attention-wasm';
+
+const scheduler = new LRScheduler({
+  initialLR: 0.001,
+  warmupSteps: 1000,
+  totalSteps: 10000,
+});
+
+const optimizer = new AdamW(paramCount, {
+  learningRate: scheduler.getLR(),
+  weightDecay: 0.01,
+});
+
+// Training loop
+for (let step = 0; step < 10000; step++) {
+  optimizer.learningRate = scheduler.getLR();
+  optimizer.step(params, gradients);
+  scheduler.step();
+}
+```
+
+## Building from Source
+
+### Prerequisites
+
+- Rust 1.70+
+- wasm-pack
+
+### Build Commands
+
+```bash
+# Build for web (ES modules)
+wasm-pack build --target web --out-dir pkg
+
+# Build for Node.js
+wasm-pack build --target nodejs --out-dir pkg-node
+
+# Build for bundlers (webpack, vite, etc.)
+wasm-pack build --target bundler --out-dir pkg-bundler
+
+# Run tests
+wasm-pack test --headless --firefox
+```
+
+## API Reference
+
+### Attention Mechanisms
+
+- `MultiHeadAttention` - Standard multi-head attention
+- `HyperbolicAttention` - Attention in hyperbolic space
+- `LinearAttention` - Linear complexity attention (Performer)
+- `FlashAttention` - Memory-efficient attention
+- `LocalGlobalAttention` - Combined local and global attention
+- `MoEAttention` - Mixture of Experts attention
+- `scaledDotAttention()` - Functional API for basic attention
+
+### Training
+
+- `InfoNCELoss` - Contrastive loss function
+- `Adam` - Adam optimizer
+- `AdamW` - AdamW optimizer with weight decay
+- `LRScheduler` - Learning rate scheduler
+
+### Utilities
+
+- `utils.cosineSimilarity()` - Cosine similarity between vectors
+- `utils.l2Norm()` - L2 norm of a vector
+- `utils.normalize()` - Normalize vector to unit length
+- `utils.softmax()` - Apply softmax transformation
+- `utils.attentionWeights()` - Compute attention weights from scores
+- `utils.batchNormalize()` - Batch normalization
+- `utils.randomOrthogonalMatrix()` - Generate random orthogonal matrix
+- `utils.pairwiseDistances()` - Compute pairwise distances
+
+## Performance
+
+The WASM bindings provide near-native performance for attention computations:
+
+- Optimized with `opt-level = "s"` and LTO
+- SIMD acceleration where available
+- Efficient memory management
+- Zero-copy data transfer where possible
+
+## License
+
+MIT OR Apache-2.0