@minded-ai/mindedjs 1.0.0-ec2-beta-1

package/docs/tooling/classifier.md

@@ -0,0 +1,217 @@
+# Classifier Tool
+
+The classifier tool is a powerful utility in the MindedJS SDK that allows you to categorize text content using AI. It's available both as a tool that can be used in flows and as standalone utility functions that can be imported and used anywhere in your code.
+
+## Overview
+
+The classifier tool provides:
+
+- **Single-label classification**: Categorize content into one class
+- **Multi-label classification**: Categorize content into multiple classes
+- **Conversation classification**: Specialized classification for chat conversations
+- **Configurable output formats**: JSON or plain text
+- **Error handling with fallback defaults**
+- **Confidence scoring**
+
+## Using as a Tool
+
+### In Flows
+
+You can use the classifier tool in your flows by referencing `minded-classifier`:
+
+```yaml
+- id: classifyUserIntent
+  type: tool
+  toolName: minded-classifier
+  prompt: |
+    Classify the user's message into one of these categories:
+    - Technical Issue: Problems with the system
+    - Billing: Payment or pricing questions
+    - Feature Request: Asking for new features
+
+    Content: "{{userMessage}}"
+```
+
+### Tool Parameters
+
+| Parameter     | Type             | Description                                            | Required |
+| ------------- | ---------------- | ------------------------------------------------------ | -------- |
+| content       | string           | The text content to classify                           | Yes      |
+| classes       | array            | Classes to classify into (strings, tuples, or objects) | No\*     |
+| systemPrompt  | string           | Custom system prompt for classification                | No       |
+| includeReason | boolean          | Whether to include reasoning (default: true)           | No       |
+| outputFormat  | 'json' \| 'text' | Output format (default: 'json')                        | No       |
+| defaultClass  | string           | Fallback class if classification fails                 | No       |
+| defaultReason | string           | Fallback reason if classification fails                | No       |
+
+\*Classes can be provided in the tool parameters or stored in agent memory
+
+## Using as a Utility
+
+### Import the Utilities
+
+```typescript
+import { toolsLibrary } from '@minded-ai/mindedjs';
+
+const { classify, createClassifier, createConversationClassifier, multiClassify } = toolsLibrary.tools['minded-classifier'];
+```
+
+### Direct Classification
+
+```typescript
+const result = await classify(
+  "I can't access my account",
+  {
+    classes: [
+      { name: 'Technical', description: 'Technical issues' },
+      { name: 'Account', description: 'Account access problems' },
+    ],
+    includeReason: true,
+  },
+  agent.llm,
+);
+
+console.log(result);
+// { class: 'Account', reason: 'User cannot access their account', confidence: 0.95 }
+```
+
+### Create Reusable Classifiers
+
+```typescript
+// Simple classifier with string classes
+const sentimentClassifier = createClassifier(['positive', 'negative', 'neutral']);
+
+// Classifier with descriptions
+const intentClassifier = createClassifier([
+  ['greeting', 'User is greeting or saying hello'],
+  ['question', 'User is asking a question'],
+  ['complaint', 'User is complaining or expressing dissatisfaction'],
+  ['goodbye', 'User is saying goodbye or ending conversation'],
+]);
+
+// Use the classifier
+const sentiment = await sentimentClassifier('This is amazing!', agent.llm);
+const intent = await intentClassifier('Hello there!', agent.llm);
+```
+
+### Conversation Classification
+
+For classifying entire conversations:
+
+```typescript
+const conversationClassifier = createConversationClassifier([
+  { name: 'Resolved', description: 'Issue was resolved successfully' },
+  { name: 'Escalated', description: 'Needs escalation to human agent' },
+  { name: 'Pending', description: 'Awaiting more information' },
+]);
+
+const result = await conversationClassifier(messages, agent.llm);
+```
+
+### Multi-Label Classification
+
+When content can belong to multiple categories:
+
+```typescript
+const topics = await multiClassify(
+  'The app crashes on login and the UI looks outdated',
+  {
+    classes: [
+      { name: 'Bug', description: 'Software bugs or crashes' },
+      { name: 'UI/UX', description: 'User interface issues' },
+      { name: 'Authentication', description: 'Login or auth problems' },
+    ],
+    maxClasses: 3,
+  },
+  agent.llm,
+);
+
+console.log(topics);
+// [
+//   { class: 'Bug', reason: 'App crashes', confidence: 0.9 },
+//   { class: 'Authentication', reason: 'Login issues', confidence: 0.85 },
+//   { class: 'UI/UX', reason: 'Outdated UI', confidence: 0.7 }
+// ]
+```
+
+## Configuration Options
+
+### ClassifierConfig Interface
+
+```typescript
+interface ClassifierConfig {
+  classes: ClassDefinition[];
+  systemPrompt?: string;
+  outputFormat?: 'json' | 'text';
+  includeReason?: boolean;
+  defaultClass?: string;
+  defaultReason?: string;
+}
+```
+
+### Class Definition Formats
+
+Classes can be defined in multiple ways:
+
+```typescript
+// Simple strings
+['positive', 'negative', 'neutral'][
+  // Tuples with descriptions
+  (['technical', 'Technical problems'], ['billing', 'Payment issues'])
+][
+  // Objects
+  ({ name: 'urgent', description: 'Requires immediate attention' }, { name: 'normal', description: 'Standard priority' })
+];
+```
+
+## Example: Customer Support Bot
+
+```typescript
+import { Agent, toolsLibrary } from '@minded-ai/mindedjs';
+import { z } from 'zod';
+
+const classifierTool = toolsLibrary.tools['minded-classifier'].default;
+
+const agent = new Agent({
+  memorySchema: z.object({
+    ticketPriority: z.string().optional(),
+    ticketCategory: z.string().optional(),
+  }),
+  config: {
+    flows: ['./flows'],
+    llm: { provider: 'openai', model: 'gpt-4' },
+  },
+  tools: [classifierTool],
+});
+
+// Create specialized classifiers
+const priorityClassifier = createClassifier([
+  ['urgent', 'Requires immediate attention'],
+  ['high', 'Important but not critical'],
+  ['normal', 'Standard priority'],
+  ['low', 'Can be handled later'],
+]);
+
+const categoryClassifier = createClassifier([
+  ['technical', 'Technical issues or bugs'],
+  ['billing', 'Payment and subscription issues'],
+  ['feature', 'Feature requests'],
+  ['other', 'General inquiries'],
+]);
+
+// Use in message handler
+agent.on('message', async (message) => {
+  const [priority, category] = await Promise.all([
+    priorityClassifier(message.content, agent.llm),
+    categoryClassifier(message.content, agent.llm),
+  ]);
+
+  // Update agent memory
+  await agent.updateMemory({
+    ticketPriority: priority.class,
+    ticketCategory: category.class,
+  });
+});
+```
+
+`

package/docs/tooling/document-processing.md

@@ -0,0 +1,389 @@
+# Document Processing
+
+MindedJS provides powerful document processing capabilities that allow you to extract data from various document types including images, PDFs, Word documents, spreadsheets, and more.
+
+## Overview
+
+The document processing system supports three modes of operation:
+
+1. **Structured Extraction with Schema**: Extract data into a predefined structure using AI
+2. **Unstructured Extraction with Prompt**: Extract information based on prompt instructions
+3. **Raw Text Extraction**: Extract plain text without AI processing
+
+Supported document types:
+
+- **Images**: JPG, PNG, GIF, BMP, WebP, TIFF
+- **Documents**: PDF, DOC, DOCX, TXT, RTF, ODT
+- **Spreadsheets**: XLS, XLSX, CSV, ODS
+- **Presentations**: PPT, PPTX, ODP
+- **Web formats**: HTML, HTM, MD, XML
+
+## Library Tool Integration
+
+The document processing functionality is available as a **library tool** called `minded-parse-documents` that can be added to your flows through the Minded platform. As a library tool, it provides several configuration advantages:
+
+### System Prompt Configuration
+
+The `systemPrompt` property can be configured directly in the Minded platform's flow editor when adding the parseDocument tool to your flow. This allows you to:
+
+- Set extraction instructions without code changes
+- Customize prompts for different use cases within the same agent
+- Update extraction behavior through the UI without redeployment
+
+**When system prompt is configured in the platform:**
+
+- The platform-configured prompt takes precedence and guides the AI extraction
+- The prompt is automatically applied to all document processing operations in that flow node
+
+**When system prompt is not set:**
+
+- The tool falls back to using any Zod schema provided for structured extraction
+- If no schema is provided, the tool performs basic text extraction with minimal AI guidance
+- Raw text extraction mode bypasses AI processing entirely regardless of prompt configuration
+
+### Parameter Configuration
+
+Other tool parameters that can be pre-configured in the platform include:
+
+- `loadFrom`: Source type (url, path, buffer, string)
+- `extractRaw`: Whether to extract raw text without AI processing
+- `schema`: Zod schema for structured data extraction
+
+Parameters not set in the platform can still be provided by the LLM during execution, allowing for flexible hybrid configuration.
+
+## Quick Start
+
+### Structured Data Extraction (with Schema)
+
+```typescript
+import { extractFromDocument } from '@minded-ai/mindedjs';
+import { z } from 'zod';
+
+// Define the data structure you want to extract
+const schema = z.object({
+  name: z.string(),
+  email: z.string(),
+  phoneNumber: z.string().optional(),
+});
+
+// Extract structured data using the agent's LLM
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentPath: './path/to/id-card.jpg',
+  schema,
+  systemPrompt: 'Extract personal information from this ID document',
+});
+
+console.log(result.data.name); // Typed data matching your schema
+console.log(result.metadata.processingTime); // Processing time in ms
+```
+
+### Unstructured Extraction (Prompt Only)
+
+```typescript
+// Extract information without a predefined schema
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentPath: './contract.pdf',
+  systemPrompt: `
+    Extract and summarize:
+    1. All parties involved
+    2. Key dates and deadlines
+    3. Payment terms
+    4. Termination clauses
+  `,
+});
+
+console.log(result.data); // String containing the extracted information
+```
+
+### Raw Text Extraction (No AI)
+
+```typescript
+// Extract raw text without AI processing
+const result = await extractFromDocument({
+  documentPath: './document.pdf',
+  config: {
+    llamaCloudApiKey: process.env.LLAMA_CLOUD_API_KEY,
+  },
+});
+
+console.log(result.data); // Raw text content from the document
+```
+
+### Using Document URLs
+
+```typescript
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentUrl: 'https://example.com/invoice.pdf',
+  schema: z.object({
+    invoiceNumber: z.string(),
+    amount: z.number(),
+    dueDate: z.string(),
+  }),
+});
+```
+
+## Configuration
+
+### Environment Variables
+
+Set these environment variables for optimal functionality:
+
+```bash
+# Required for advanced document parsing (PDFs, Word docs, etc.)
+LLAMA_CLOUD_API_KEY=your_llama_cloud_api_key
+```
+
+### Document Processor Configuration
+
+```typescript
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentPath: './image.jpg',
+  schema: yourSchema,
+  config: {
+    llamaCloudApiKey: 'your-key', // Override env variable
+    useBase64: true, // Return images as base64
+    maxImageWidth: 1600, // Max image width for processing
+    imageQuality: 90, // JPEG quality (0-100)
+  },
+});
+```
+
+### LLM Configuration
+
+```typescript
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentPath: './document.pdf',
+  schema: yourSchema,
+  llmConfig: {
+    model: 'gpt-4o', // Override default model
+    temperature: 0.1, // Lower temperature for consistent extraction
+  },
+});
+```
+
+## Advanced Usage
+
+### Custom System Prompts
+
+Tailor the AI's extraction behavior with custom prompts:
+
+```typescript
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentPath: './medical-record.pdf',
+  schema: z.object({
+    patientName: z.string(),
+    diagnosis: z.string(),
+    medications: z.array(z.string()),
+  }),
+  systemPrompt: `
+    You are a medical data extraction specialist.
+    Extract information with high accuracy and attention to medical terminology.
+    If any required field is unclear or missing, mark it as "Not clearly specified".
+  `,
+});
+```
+
+### Handling Multiple Document Sources
+
+```typescript
+const processDocuments = async (documents: string[]) => {
+  const results = await Promise.all(
+    documents.map((path) =>
+      extractFromDocument({
+        llm: agent.llm,
+        documentPath: path,
+        schema: invoiceSchema,
+      }),
+    ),
+  );
+
+  return results.map((r) => r.data);
+};
+```
+
+### Error Handling
+
+```typescript
+try {
+  const result = await extractFromDocument({
+    llm: agent.llm,
+    documentPath: './document.pdf',
+    schema: yourSchema,
+  });
+
+  logger.info({ message: 'Extracted', data: result.data });
+} catch (err) {
+  if (err.message.includes('LLAMA_CLOUD_API_KEY')) {
+    logger.error({ message: 'LlamaCloud API key required for PDF processing', err });
+  } else if (err.message.includes('Document not found')) {
+    logger.error({ message: 'File does not exist', err });
+  } else {
+    logger.error({ message: 'Processing failed', err });
+  }
+}
+```
+
+## Example Use Cases
+
+### Invoice Processing
+
+```typescript
+const invoiceSchema = z.object({
+  invoiceNumber: z.string(),
+  vendor: z.string(),
+  amount: z.number(),
+  dueDate: z.string(),
+  lineItems: z.array(
+    z.object({
+      description: z.string(),
+      quantity: z.number(),
+      unitPrice: z.number(),
+    }),
+  ),
+});
+
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentPath: './invoice.pdf',
+  schema: invoiceSchema,
+  systemPrompt: 'Extract all invoice details including line items',
+});
+```
+
+### Identity Document Verification
+
+```typescript
+const idSchema = z.object({
+  documentType: z.enum(['passport', 'driver_license', 'national_id']),
+  fullName: z.string(),
+  dateOfBirth: z.string(),
+  documentNumber: z.string(),
+  expiryDate: z.string().optional(),
+  issuingCountry: z.string().optional(),
+});
+
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentPath: './id-document.jpg',
+  schema: idSchema,
+});
+```
+
+### Contract Analysis
+
+```typescript
+const contractSchema = z.object({
+  parties: z.array(z.string()),
+  effectiveDate: z.string(),
+  expiryDate: z.string().optional(),
+  keyTerms: z.array(z.string()),
+  terminationClauses: z.array(z.string()),
+  governingLaw: z.string().optional(),
+});
+
+const result = await extractFromDocument({
+  llm: agent.llm,
+  documentPath: './contract.docx',
+  schema: contractSchema,
+  systemPrompt: 'Focus on legal terms, dates, and parties involved',
+});
+```
+
+## Tool Implementation
+
+### Using the Library Tool
+
+The easiest way to add document processing to your flows is using the **minded-parse-documents library tool** available in the Minded platform. Simply add it to your flow through the platform UI and configure the parameters as needed.
+
+## Standalone Usage
+
+You can also use the document processor independently of the agent:
+
+```typescript
+import { DocumentProcessor, extractFromDocument } from '@minded-ai/mindedjs';
+
+// Using the class directly
+const processor = new DocumentProcessor({
+  llamaCloudApiKey: 'your-key',
+  maxImageWidth: 1200,
+});
+
+const result = await processor.extractFromDocument({
+  documentPath: './document.pdf',
+  schema: yourSchema,
+});
+
+// Using the convenience function
+const result2 = await extractFromDocument({
+  documentPath: './document.pdf',
+  schema: yourSchema,
+  config: {
+    llamaCloudApiKey: 'your-key',
+  },
+});
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**PDF Processing Fails**
+
+```
+Error: PDF processing requires LLAMA_CLOUD_API_KEY
+```
+
+_Solution_: Set the `LLAMA_CLOUD_API_KEY` environment variable.
+
+**Image Too Large**
+
+```
+Error: Image processing failed: Input image exceeds pixel limit
+```
+
+_Solution_: Reduce `maxImageWidth` in configuration or resize images beforehand.
+
+**Schema Validation Errors**
+
+```
+Error: LLM extraction failed: Invalid schema
+```
+
+_Solution_: Check that your Zod schema matches expected data structure.
+
+**Network Timeouts**
+
+```
+Error: Failed to fetch document from URL: timeout
+```
+
+_Solution_: Implement retry logic or download files locally first.
+
+### Debugging
+
+Enable debug logging to troubleshoot issues:
+
+```typescript
+// Set LOG_LEVEL environment variable
+process.env.LOG_LEVEL = 'debug';
+
+// Or use the logger directly
+import { logger } from '@minded-ai/mindedjs';
+logger.debug('Processing document:', { path: documentPath });
+```
+
+## Supported File Types
+
+| Category      | Extensions                                  | Notes                                    |
+| ------------- | ------------------------------------------- | ---------------------------------------- |
+| Images        | .jpg, .jpeg, .png, .gif, .bmp, .webp, .tiff | Direct processing                        |
+| Documents     | .pdf, .doc, .docx, .txt, .rtf, .odt         | Requires LlamaCloud for advanced formats |
+| Spreadsheets  | .xls, .xlsx, .csv, .ods                     | Requires LlamaCloud for binary formats   |
+| Presentations | .ppt, .pptx, .odp                           | Requires LlamaCloud                      |
+| Web           | .html, .htm, .md, .xml                      | Basic text extraction                    |

package/docs/tooling/extraction.md

@@ -0,0 +1,78 @@
+# Data Extraction
+
+MindedJS provides a powerful AI-based extraction tool for extracting structured data from unstructured text. The extraction system uses LLM capabilities to parse content and return data in a predefined format.
+
+## Overview
+
+The extraction tool (`minded-extraction`) enables you to extract specific information from text using:
+
+- **Structured Extraction with Zod Schema**: Define exact data structure using Zod schemas
+- **Prompt-based Extraction**: Extract information using custom prompts
+- **Validation and Retries**: Automatic validation against schema with retry logic
+
+## Key Features
+
+- **LLM Structured Output Support**: When available, uses the LLM's native structured output capabilities for guaranteed schema compliance
+- **Fallback JSON Parsing**: Automatically falls back to JSON parsing with validation when structured output is unavailable
+- **Schema Validation**: Built-in Zod validation ensures extracted data matches expected structure
+- **Retry Logic**: Configurable retry attempts with error feedback for improved accuracy
+- **Strict and Non-strict Modes**: Choose between validated extraction (strict) or flexible extraction
+
+## Library Tool Integration
+
+The extraction functionality is available as a library tool called `minded-extraction` that can be added to your flows through the Minded platform.
+
+### Configuration Options
+
+- `content`: The text to extract information from
+- `schema`: Optional Zod-compatible schema defining the structure
+- `systemPrompt`: Custom instructions for extraction
+- `examples`: Input/output examples to guide extraction
+- `strictMode`: Enable/disable schema validation (default: true)
+- `maxRetries`: Number of retry attempts on validation failure (default: 3)
+- `defaultValue`: Fallback value if extraction fails
+
+## How It Works
+
+1. **With Structured Output Support** (when available):
+
+   - The tool uses the LLM's `withStructuredOutput` method for direct schema-compliant extraction
+   - No manual JSON parsing or validation needed
+   - Guaranteed to match the provided Zod schema
+
+2. **Fallback Mode** (JSON parsing):
+
+   - Generates a prompt with schema description
+   - Uses JSON output parser to extract structured data
+   - Validates against Zod schema
+   - Retries with error feedback if validation fails
+
+3. **Non-strict Mode**:
+   - Skips validation for more flexible extraction
+   - Useful when schema compliance is not critical
+
+## Standalone Usage
+
+The extraction utility can also be used programmatically:
+
+```typescript
+import { extract, createExtractor } from '@minded-ai/mindedjs';
+import { z } from 'zod';
+
+// Direct extraction
+const result = await extract(
+  content,
+  {
+    schema: z.object({
+      name: z.string(),
+      age: z.number(),
+    }),
+    systemPrompt: 'Extract person details',
+  },
+  agent.llm,
+);
+
+// Create reusable extractor
+const extractor = createExtractor(schema, { systemPrompt: 'Extract data' });
+const result = await extractor(content, agent.llm);
+```