@amitdeshmukh/ax-crew 6.0.0 → 8.0.0

package/.claude/settings.local.json

@@ -0,0 +1,7 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(npm test:*)"
+    ]
+  }
+}

package/CHANGELOG.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## [8.0.0] - 2025-01-17
+
+### Added
+- **ACE (Agentic Context Engineering) Support** - Per-agent learning from human feedback
+  - Agents can learn from real-time feedback and update their behavior
+  - Playbook persistence to JSON files or custom storage via callbacks
+  - Teacher/Student model architecture for distilling feedback into rules
+  - Feedback routing across agent dependency chains (`applyTaskFeedback`)
+  - New `ACEConfig` interface with `teacher`, `persistence`, `options`, and `metric` fields
+- New type exports: `ACEConfig`, `ACETeacherConfig`, `ACEPersistenceConfig`, `ACEOptionsConfig`, `ACEMetricConfig`
+- Export `AxCrewOptions` type for consumer type safety
+- ACE examples: `ace-customer-support.ts` and `ace-flight-finder.ts`
+- ACE documentation section in README with configuration options and usage examples
+
+### Changed
+- ACE configuration is presence-based (no redundant `enabled` boolean needed)
+
+## [7.0.0] - 2025-12-27
+
+### Breaking
+- Change to `AxCrewConfig` interface
+
+### Added
+- Optional OpenTelemetry telemetry support for distributed tracing and metrics collection
+- New `AxCrewOptions` interface with `telemetry` field accepting `tracer` and `meter` instances
+- Automatic instrumentation of agent execution, function calls, and token metrics when telemetry is enabled
+- Support for multiple OpenTelemetry exporters (console, Jaeger, Prometheus, etc.)
+- Complete telemetry example in `examples/telemetry-demo.ts` demonstrating setup with console and Jaeger exporters
+- Comprehensive telemetry documentation in README with setup instructions, best practices, and examples
+- Test coverage for telemetry functionality in `tests/telemetry.test.ts`
+
+### Changed
+- AxCrew constructor now accepts optional fourth parameter `options` of type `AxCrewOptions` for telemetry configuration
+- Enhanced agent initialization to pass telemetry context to underlying agents
+
 ## [6.0.0] - 2025-10-22
 
 ### Breaking

package/README.md

@@ -1,5 +1,7 @@
 ![image](axcrew.png)
 
+[![npm version](https://img.shields.io/npm/v/@amitdeshmukh/ax-crew.svg)](https://www.npmjs.com/package/@amitdeshmukh/ax-crew) [![npm downloads](https://img.shields.io/npm/dm/@amitdeshmukh/ax-crew.svg)](https://www.npmjs.com/package/@amitdeshmukh/ax-crew)
+
 ### AxCrew — build a crew of AI agents with shared state (powered by AxLLM)
 
 AxCrew lets you define a team of AI agents in config and run them together with shared state, tools, streaming, MCP, and built‑in metrics/cost tracking. Bring your own functions or use the provided registry.
@@ -715,6 +717,244 @@ Notes:
 - Legacy cost APIs (`getLastUsageCost`, `getAccumulatedCosts`, `getAggregatedCosts`) are superseded by metrics methods.
 - Estimated cost values are numbers rounded to 5 decimal places.
 
+### Telemetry Support (OpenTelemetry)
+
+AxCrew provides optional OpenTelemetry integration for comprehensive observability. You can pass custom tracer and meter instances to monitor agent operations, track performance, and analyze behavior across your crew.
+
+#### Features
+
+- **Distributed Tracing**: Track agent execution flows, function calls, and dependencies
+- **Metrics Collection**: Monitor token usage, costs, latency, and error rates
+- **Multiple Exporters**: Support for console, Jaeger, Prometheus, and other OpenTelemetry backends
+
+#### Setup
+
+Install OpenTelemetry dependencies:
+
+```bash
+npm install @opentelemetry/api @opentelemetry/sdk-trace-node @opentelemetry/sdk-metrics
+```
+
+Optional: Install exporters for enhanced visualization:
+
+```bash
+# For Jaeger tracing UI
+npm install @opentelemetry/exporter-jaeger
+
+# For Prometheus metrics
+npm install @opentelemetry/exporter-prometheus
+```
+
+#### Basic Configuration
+
+```typescript
+import { AxCrew } from '@amitdeshmukh/ax-crew';
+import { trace, metrics } from '@opentelemetry/api';
+import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
+import { MeterProvider } from '@opentelemetry/sdk-metrics';
+import { ConsoleSpanExporter, SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
+
+// Initialize OpenTelemetry
+const tracerProvider = new NodeTracerProvider({
+  spanProcessors: [new SimpleSpanProcessor(new ConsoleSpanExporter())]
+});
+tracerProvider.register();
+
+const meterProvider = new MeterProvider();
+metrics.setGlobalMeterProvider(meterProvider);
+
+// Get tracer and meter instances
+const tracer = trace.getTracer('my-app');
+const meter = metrics.getMeter('my-app');
+
+// Pass to AxCrew
+const crew = new AxCrew(
+  config,
+  AxCrewFunctions,
+  undefined,
+  {
+    telemetry: {
+      tracer,
+      meter
+    }
+  }
+);
+```
+
+#### What Gets Traced
+
+When telemetry is enabled, AxCrew automatically instruments:
+
+- **Agent Execution**: Each agent's `forward()` call creates a span with timing and metadata
+- **Function Calls**: Tool/function invocations are traced with parameters and results
+- **Provider Information**: Model name, provider, and configuration details
+- **Token Metrics**: Input/output tokens and estimated costs
+- **Errors**: Exceptions and failures are captured with full context
+
+#### Advanced Configuration with Jaeger
+
+For enhanced visualization, you can export traces to Jaeger:
+
+```typescript
+import { JaegerExporter } from '@opentelemetry/exporter-jaeger';
+
+const tracerProvider = new NodeTracerProvider({
+  spanProcessors: [
+    new SimpleSpanProcessor(new ConsoleSpanExporter()),
+    new SimpleSpanProcessor(new JaegerExporter({
+      endpoint: 'http://localhost:14268/api/traces'
+    }))
+  ]
+});
+tracerProvider.register();
+
+// ... rest of setup
+```
+
+Start Jaeger with Docker:
+
+```bash
+docker run -d --name jaeger \
+  -p 16686:16686 \
+  -p 14268:14268 \
+  jaegertracing/all-in-one:latest
+```
+
+View traces at: http://localhost:16686
+
+#### Complete Example
+
+See [examples/telemetry-demo.ts](examples/telemetry-demo.ts) for a full working example that demonstrates:
+
+- Setting up OpenTelemetry with console and Jaeger exporters
+- Configuring multiple agents with different providers
+- Running a multi-step workflow with telemetry tracking
+- Viewing traces and metrics in the console and Jaeger UI
+
+Run the example:
+
+```bash
+# With console output only
+npm run dev examples/telemetry-demo.ts
+
+# With Jaeger (start Jaeger first)
+docker run -d --name jaeger -p 16686:16686 -p 14268:14268 jaegertracing/all-in-one:latest
+npm run dev examples/telemetry-demo.ts
+# Open http://localhost:16686 to view traces
+```
+
+#### Best Practices
+
+1. **Production Setup**: Use appropriate exporters for your infrastructure (Jaeger, Zipkin, Cloud providers)
+2. **Sampling**: Configure sampling strategies to control trace volume in production
+3. **Context Propagation**: OpenTelemetry automatically propagates trace context across agent calls
+4. **Custom Attributes**: Extend traces with custom attributes specific to your use case
+5. **Performance**: Telemetry adds minimal overhead when properly configured
+
+### ACE Support (Agentic Context Engineering)
+
+AxCrew integrates [Agentic Context Engineering (ACE)](https://www.youtube.com/watch?v=elgYgPo_vY4) from the Ax framework, enabling agents to learn and improve from human feedback. ACE maintains a "playbook" of learned rules that guide agent behavior, which can be persisted across sessions.
+
+#### Key Features
+
+- **Online Learning**: Agents learn from real-time feedback during conversations
+- **Playbook Persistence**: Save learned rules to JSON files or custom storage
+- **Teacher/Student Model**: Use a separate "teacher" LLM to distill feedback into actionable rules
+- **Feedback Routing**: Distribute feedback across agent dependency chains automatically
+
+#### Configuration
+
+Add the `ace` field to any agent configuration:
+
+```typescript
+{
+  name: "SupportAgent",
+  description: "Customer support agent",
+  signature: "ticket:string -> supportResponse:string, decision:string",
+  provider: "google-gemini",
+  providerKeyName: "GEMINI_API_KEY",
+  ai: { model: "gemini-flash-latest", temperature: 0.7 },
+  ace: {
+    teacher: {
+      provider: "google-gemini",
+      providerKeyName: "GEMINI_API_KEY",
+      ai: { model: "gemini-flash-latest" }
+    },
+    options: {
+      maxEpochs: 1,
+      allowDynamicSections: true
+    },
+    persistence: {
+      playbookPath: "playbooks/support-agent.json",
+      autoPersist: true
+    },
+    metric: {
+      primaryOutputField: "supportResponse"
+    }
+  }
+}
+```
+
+#### ACE Configuration Options
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `teacher` | object | Teacher model config (provider, model, apiURL) |
+| `persistence.playbookPath` | string | File path to save/load playbook |
+| `persistence.autoPersist` | boolean | Auto-save playbook after updates |
+| `persistence.onPersist` | function | Custom callback for saving playbook |
+| `persistence.onLoad` | function | Custom callback for loading playbook |
+| `options.maxEpochs` | number | Training epochs for offline compile |
+| `options.allowDynamicSections` | boolean | Allow playbook to create new sections |
+| `metric.primaryOutputField` | string | Output field to evaluate for quality |
+| `compileOnStart` | boolean | Run offline compile on agent init |
+
+#### Usage: Applying Feedback
+
+```typescript
+import { AxCrew, AxCrewFunctions } from '@amitdeshmukh/ax-crew';
+
+const crew = new AxCrew(config, AxCrewFunctions);
+await crew.addAgentsToCrew(['SupportAgent']);
+
+const agent = crew.agents.get('SupportAgent');
+
+// Run the agent
+const result = await agent.forward({ ticket: "Customer wants refund after 45 days" });
+const taskId = result._taskId;
+
+// Apply feedback to teach the agent
+await crew.applyTaskFeedback({
+  taskId,
+  feedback: "For loyal customers (5+ years), extend return window to 60 days",
+  strategy: "all"  // Apply to all agents involved in this task
+});
+
+// View the learned playbook
+const playbook = agent.getPlaybook?.();
+console.log(playbook);
+```
+
+#### Feedback Strategies
+
+| Strategy | Description |
+|----------|-------------|
+| `"all"` | Apply feedback to all agents involved in the task |
+| `"primary"` | Apply only to the primary (entry) agent |
+| `"leaf"` | Apply only to leaf agents (no sub-agents) |
+
+#### Examples
+
+See the ACE examples for complete demonstrations:
+
+- [`ace-customer-support.ts`](examples/ace-customer-support.ts) - Learn edge-case handling beyond standard policies
+- [`ace-feedback-routing.ts`](examples/ace-feedback-routing.ts) - Flight assistant with preference learning
+
+```bash
+# Run the customer support demo
+npx tsx examples/ace-customer-support.ts
+```
+
 ## Changelog
 
 See [CHANGELOG.md](CHANGELOG.md) for a list of changes and version updates.

package/dist/agents/ace.d.ts

@@ -0,0 +1,134 @@
+/**
+ * ACE (Agentic Context Engineering) integration for AxCrew
+ *
+ * This module provides helpers to build and manage AxACE optimizers for agents,
+ * enabling offline compilation and online learning from feedback.
+ *
+ * Reference: https://axllm.dev/ace/
+ */
+import { AxACE, type AxMetricFn } from "@ax-llm/ax";
+import type { AxAI } from "@ax-llm/ax";
+import type { ACEConfig, ACEPersistenceConfig, ACEMetricConfig, FunctionRegistryType } from "../types.js";
+export type { AxACE, AxMetricFn };
+/**
+ * Create an empty playbook structure
+ */
+export declare const createEmptyPlaybook: () => ACEPlaybook;
+/**
+ * Playbook types (mirroring AxACEPlaybook structure)
+ */
+export interface ACEBullet {
+    id: string;
+    section: string;
+    content: string;
+    helpfulCount: number;
+    harmfulCount: number;
+    createdAt: string;
+    updatedAt: string;
+    metadata?: Record<string, unknown>;
+}
+export interface ACEPlaybook {
+    version: number;
+    sections: Record<string, ACEBullet[]>;
+    stats: {
+        bulletCount: number;
+        helpfulCount: number;
+        harmfulCount: number;
+        tokenEstimate: number;
+    };
+    updatedAt: string;
+    description?: string;
+}
+/**
+ * Render a playbook into markdown instruction block for injection into prompts.
+ * Mirrors the AxACE renderPlaybook function.
+ */
+export declare const renderPlaybook: (playbook: Readonly<ACEPlaybook>) => string;
+/**
+ * Build an AxACE optimizer for an agent
+ *
+ * @param studentAI - The agent's AI instance (used as student)
+ * @param cfg - ACE configuration
+ * @returns Configured AxACE optimizer
+ */
+export declare const buildACEOptimizer: (studentAI: AxAI, cfg: ACEConfig) => AxACE;
+/**
+ * Load initial playbook from file, callback, or inline config
+ *
+ * @param cfg - Persistence configuration
+ * @returns Loaded playbook or undefined
+ */
+export declare const loadInitialPlaybook: (cfg?: ACEPersistenceConfig) => Promise<any | undefined>;
+/**
+ * Persist playbook to file or via callback
+ *
+ * @param pb - Playbook to persist
+ * @param cfg - Persistence configuration
+ */
+export declare const persistPlaybook: (pb: any, cfg?: ACEPersistenceConfig) => Promise<void>;
+/**
+ * Resolve metric function from registry or create equality-based metric
+ *
+ * @param cfg - Metric configuration
+ * @param registry - Function registry to search
+ * @returns Metric function or undefined
+ */
+export declare const resolveMetric: (cfg: ACEMetricConfig | undefined, registry: FunctionRegistryType) => AxMetricFn | undefined;
+/**
+ * Run offline ACE compilation
+ *
+ * @param args - Compilation arguments
+ * @returns Compilation result with optimized program
+ */
+export declare const runOfflineCompile: (args: {
+    program: any;
+    optimizer: AxACE;
+    metric: AxMetricFn;
+    examples: any[];
+    persistence?: ACEPersistenceConfig;
+}) => Promise<any>;
+/**
+ * Apply online update with feedback
+ *
+ * @param args - Update arguments
+ * @returns Curator delta (operations applied)
+ */
+export declare const runOnlineUpdate: (args: {
+    optimizer: AxACE;
+    example: any;
+    prediction: any;
+    feedback?: string;
+    persistence?: ACEPersistenceConfig;
+    tokenBudget?: number;
+    debug?: boolean;
+}) => Promise<any>;
+/**
+ * Use LLM to analyze feedback and generate playbook operations.
+ *
+ * This leverages AxGen with a proper signature (like AxACE's reflector/curator)
+ * to properly categorize feedback and extract actionable insights.
+ *
+ * IMPORTANT: The prompt explicitly tells the LLM to preserve specificity.
+ *
+ * @param ai - The AI instance to use for analysis
+ * @param feedback - User feedback string
+ * @param debug - Whether to log debug info
+ * @returns Promise of curator operations
+ */
+export declare const analyzeAndCategorizeFeedback: (ai: AxAI, feedback: string, debug?: boolean) => Promise<Array<{
+    type: "ADD" | "UPDATE" | "REMOVE";
+    section: string;
+    content: string;
+}>>;
+/**
+ * Add feedback to playbook using LLM analysis.
+ *
+ * Uses the AI to properly understand and categorize the feedback,
+ * then applies it as a curator operation.
+ *
+ * @param playbook - The playbook to update (mutated in place)
+ * @param feedback - User feedback string to add
+ * @param ai - AI instance for smart categorization
+ * @param debug - Whether to log debug info
+ */
+export declare const addFeedbackToPlaybook: (playbook: ACEPlaybook, feedback: string, ai: AxAI, debug?: boolean) => Promise<void>;