@neural-tools/fine-tune 0.1.4 → 0.1.6

package/LICENSE.md

@@ -1,80 +1,21 @@
-# Neural Tools License
-
-Copyright (c) 2025 Luke Amy. All rights reserved.
-
-## License Agreement
-
-This software is provided under a dual-license model:
-
-### 1. Free Tier License (MIT)
-
-The following components are licensed under the MIT License:
-
-- Basic MCP generation functionality
-- Claude command generation
-- Core utilities and types
-- Basic templates
-- Documentation and examples
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of the free tier components to use, copy, modify, merge, publish, and distribute, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
-
-### 2. Pro/Enterprise License (Proprietary)
-
-The following features require a valid Pro or Enterprise license:
-
-**Pro Features:**
-- Vector database integration
-- Semantic caching
-- Fine-tuning workflows
-- Cloud deployment templates (AWS/GCP)
-- Premium templates and examples
-- GitHub automation features
-
-**Enterprise Features:**
-- White-label support
-- Custom integrations
-- Priority support
-- SLA guarantees
-- Team collaboration features
-
-These features are proprietary and may not be used without a valid license key purchased from neural-tools.dev.
-
-### License Terms
-
-1. **Free Tier**: You may use the free tier features for any purpose, including commercial use, under the MIT License terms.
-
-2. **Pro/Enterprise**: You must purchase a license to access Pro or Enterprise features. Each license is:
-   - Per-user for individual licenses
-   - Per-organization for team/enterprise licenses
-   - Non-transferable without written consent
-   - Subject to the terms at neural-tools.dev/terms
-
-3. **Source Code**: This repository is private. You may not:
-   - Redistribute the source code
-   - Create derivative works for redistribution
-   - Reverse engineer Pro/Enterprise features
-   - Remove or circumvent license checks
-
-4. **Support**: Support is provided based on your license tier:
-   - Free: Community support only
-   - Pro: Email support (48-hour response)
-   - Enterprise: Priority support with SLA
-
-### Warranty Disclaimer
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-
-IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-### Contact
-
-For licensing inquiries:
-- Email: licensing@neural-tools.dev
-- Website: https://neural-tools.dev/pricing
-- Support: support@neural-tools.dev
-
----
-
-**Last Updated:** January 2025
+# MIT License
+
+Copyright (c) 2025 Luke Amy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,459 @@
+# @neural-tools/fine-tune
+
+> Fine-tuning utilities for Neural Tools
+
+[![npm version](https://img.shields.io/npm/v/@neural-tools/fine-tune)](https://www.npmjs.com/package/@neural-tools/fine-tune)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](../../LICENSE.md)
+
+Utilities for preparing, validating, and managing fine-tuning datasets for LLMs. Currently supports OpenAI's fine-tuning format.
+
+## Installation
+
+```bash
+npm install @neural-tools/fine-tune
+```
+
+### With OpenAI
+
+```bash
+npm install @neural-tools/fine-tune openai
+```
+
+## Features
+
+- **Dataset Preparation** - Convert various formats to fine-tuning format
+- **Validation** - Ensure datasets meet LLM requirements
+- **Cost Estimation** - Calculate fine-tuning costs before running
+- **Quality Analysis** - Analyze dataset quality and balance
+- **Format Conversion** - Convert between different training formats
+- **Token Counting** - Accurate token counting for cost estimation
+
+## Quick Start
+
+```typescript
+import { FineTuneDataset } from '@neural-tools/fine-tune';
+
+// Create dataset
+const dataset = new FineTuneDataset();
+
+// Add training examples
+dataset.addExample({
+  messages: [
+    { role: 'system', content: 'You are a helpful assistant.' },
+    { role: 'user', content: 'What is the capital of France?' },
+    { role: 'assistant', content: 'The capital of France is Paris.' }
+  ]
+});
+
+dataset.addExample({
+  messages: [
+    { role: 'system', content: 'You are a helpful assistant.' },
+    { role: 'user', content: 'What is 2 + 2?' },
+    { role: 'assistant', content: '2 + 2 equals 4.' }
+  ]
+});
+
+// Validate dataset
+const validation = await dataset.validate();
+if (!validation.isValid) {
+  console.error('Validation errors:', validation.errors);
+}
+
+// Get cost estimate
+const estimate = await dataset.estimateCost('gpt-3.5-turbo');
+console.log(`Estimated cost: $${estimate.totalCost.toFixed(2)}`);
+
+// Export for OpenAI
+const jsonl = dataset.toJSONL();
+await fs.writeFile('training-data.jsonl', jsonl);
+```
+
+## API Reference
+
+### FineTuneDataset
+
+Main class for managing fine-tuning datasets.
+
+#### Constructor
+
+```typescript
+new FineTuneDataset(options?: DatasetOptions)
+
+interface DatasetOptions {
+  format?: 'openai' | 'anthropic';  // Default: 'openai'
+  validateOnAdd?: boolean;           // Default: true
+}
+```
+
+#### Methods
+
+##### `addExample(example)`
+
+Add a training example to the dataset.
+
+```typescript
+dataset.addExample({
+  messages: [
+    { role: 'system', content: 'System prompt' },
+    { role: 'user', content: 'User message' },
+    { role: 'assistant', content: 'Assistant response' }
+  ]
+});
+```
+
+##### `addExamples(examples)`
+
+Add multiple examples at once.
+
+```typescript
+dataset.addExamples([
+  { messages: [...] },
+  { messages: [...] },
+  { messages: [...] }
+]);
+```
+
+##### `validate()`
+
+Validate the dataset.
+
+```typescript
+const result = await dataset.validate();
+
+interface ValidationResult {
+  isValid: boolean;
+  errors: string[];
+  warnings: string[];
+  stats: {
+    totalExamples: number;
+    avgTokensPerExample: number;
+    minTokens: number;
+    maxTokens: number;
+  };
+}
+```
+
+##### `estimateCost(model)`
+
+Estimate fine-tuning cost.
+
+```typescript
+const estimate = await dataset.estimateCost('gpt-3.5-turbo');
+
+interface CostEstimate {
+  model: string;
+  totalTokens: number;
+  trainingCost: number;
+  totalCost: number;
+  estimatedTime: number;  // Minutes
+}
+```
+
+##### `analyze()`
+
+Get dataset quality metrics.
+
+```typescript
+const analysis = await dataset.analyze();
+
+interface DatasetAnalysis {
+  exampleCount: number;
+  avgTokensPerMessage: number;
+  tokenDistribution: {
+    min: number;
+    max: number;
+    mean: number;
+    median: number;
+  };
+  roleBalance: {
+    system: number;
+    user: number;
+    assistant: number;
+  };
+  qualityScore: number;  // 0-100
+}
+```
+
+##### `toJSONL()`
+
+Export dataset as JSONL string.
+
+```typescript
+const jsonl = dataset.toJSONL();
+```
+
+##### `toJSON()`
+
+Export dataset as JSON array.
+
+```typescript
+const json = dataset.toJSON();
+```
+
+##### `fromJSONL(jsonl)`
+
+Load dataset from JSONL string.
+
+```typescript
+dataset.fromJSONL(jsonlString);
+```
+
+##### `fromJSON(json)`
+
+Load dataset from JSON array.
+
+```typescript
+dataset.fromJSON(jsonArray);
+```
+
+## Usage Examples
+
+### Basic Chat Fine-Tuning
+
+```typescript
+import { FineTuneDataset } from '@neural-tools/fine-tune';
+
+const dataset = new FineTuneDataset();
+
+// Add customer support examples
+const supportExamples = [
+  {
+    messages: [
+      { role: 'system', content: 'You are a customer support agent.' },
+      { role: 'user', content: 'How do I reset my password?' },
+      { role: 'assistant', content: 'To reset your password: 1. Go to login page, 2. Click "Forgot Password", 3. Check your email for reset link.' }
+    ]
+  },
+  {
+    messages: [
+      { role: 'system', content: 'You are a customer support agent.' },
+      { role: 'user', content: 'My order hasn\'t arrived yet.' },
+      { role: 'assistant', content: 'I can help track your order. Can you provide your order number?' }
+    ]
+  }
+  // Add 50-100 more examples for good results
+];
+
+dataset.addExamples(supportExamples);
+
+// Validate
+const validation = await dataset.validate();
+if (validation.isValid) {
+  // Save for upload
+  const jsonl = dataset.toJSONL();
+  await fs.writeFile('support-training.jsonl', jsonl);
+}
+```
+
+### Code Generation Fine-Tuning
+
+```typescript
+const dataset = new FineTuneDataset();
+
+dataset.addExamples([
+  {
+    messages: [
+      { role: 'system', content: 'You are a Python code generator.' },
+      { role: 'user', content: 'Write a function to reverse a string' },
+      { role: 'assistant', content: 'def reverse_string(s):\n    return s[::-1]' }
+    ]
+  },
+  {
+    messages: [
+      { role: 'system', content: 'You are a Python code generator.' },
+      { role: 'user', content: 'Create a function to check if number is prime' },
+      { role: 'assistant', content: 'def is_prime(n):\n    if n < 2:\n        return False\n    for i in range(2, int(n**0.5) + 1):\n        if n % i == 0:\n            return False\n    return True' }
+    ]
+  }
+]);
+```
+
+### Cost Estimation
+
+```typescript
+const dataset = new FineTuneDataset();
+// ... add examples ...
+
+// Estimate cost before running
+const estimate = await dataset.estimateCost('gpt-3.5-turbo');
+
+console.log(`Training ${dataset.size()} examples`);
+console.log(`Total tokens: ${estimate.totalTokens}`);
+console.log(`Estimated cost: $${estimate.totalCost.toFixed(2)}`);
+console.log(`Estimated time: ${estimate.estimatedTime} minutes`);
+
+// Only proceed if cost is acceptable
+if (estimate.totalCost < 50) {
+  await uploadAndTrain(dataset);
+}
+```
+
+### Dataset Quality Analysis
+
+```typescript
+const analysis = await dataset.analyze();
+
+console.log('Dataset Quality Report:');
+console.log(`Examples: ${analysis.exampleCount}`);
+console.log(`Avg tokens per message: ${analysis.avgTokensPerMessage}`);
+console.log(`Quality score: ${analysis.qualityScore}/100`);
+
+if (analysis.qualityScore < 70) {
+  console.warn('Dataset quality is low. Add more diverse examples.');
+}
+
+if (analysis.exampleCount < 50) {
+  console.warn('Dataset is small. Recommend at least 50-100 examples.');
+}
+```
+
+### Format Conversion
+
+```typescript
+// Load from CSV
+import { csvToFineTune } from '@neural-tools/fine-tune';
+
+const csv = `
+question,answer
+"What is AI?","Artificial Intelligence is..."
+"What is ML?","Machine Learning is..."
+`;
+
+const dataset = csvToFineTune(csv, {
+  systemPrompt: 'You are a helpful AI tutor.',
+  questionColumn: 'question',
+  answerColumn: 'answer'
+});
+
+// Export to JSONL
+const jsonl = dataset.toJSONL();
+```
+
+### Validation and Error Handling
+
+```typescript
+const dataset = new FineTuneDataset();
+dataset.addExample({
+  messages: [
+    { role: 'user', content: 'Hello' },
+    { role: 'assistant', content: 'Hi there!' }
+  ]
+});
+
+const validation = await dataset.validate();
+
+if (!validation.isValid) {
+  console.error('Errors:');
+  validation.errors.forEach(error => console.error(`  - ${error}`));
+}
+
+if (validation.warnings.length > 0) {
+  console.warn('Warnings:');
+  validation.warnings.forEach(warning => console.warn(`  - ${warning}`));
+}
+
+console.log('Stats:', validation.stats);
+```
+
+## Fine-Tuning with OpenAI
+
+```typescript
+import OpenAI from 'openai';
+import { FineTuneDataset } from '@neural-tools/fine-tune';
+import fs from 'fs/promises';
+
+const openai = new OpenAI();
+const dataset = new FineTuneDataset();
+
+// 1. Prepare dataset
+dataset.addExamples([/* your examples */]);
+
+// 2. Validate
+const validation = await dataset.validate();
+if (!validation.isValid) {
+  throw new Error('Invalid dataset');
+}
+
+// 3. Save to file
+const jsonl = dataset.toJSONL();
+await fs.writeFile('training.jsonl', jsonl);
+
+// 4. Upload file
+const file = await openai.files.create({
+  file: await fs.readFile('training.jsonl'),
+  purpose: 'fine-tune'
+});
+
+// 5. Create fine-tuning job
+const fineTune = await openai.fineTuning.jobs.create({
+  training_file: file.id,
+  model: 'gpt-3.5-turbo'
+});
+
+console.log(`Fine-tune job created: ${fineTune.id}`);
+```
+
+## Best Practices
+
+### 1. Dataset Size
+
+- Minimum: 10 examples (for testing)
+- Recommended: 50-100 examples
+- Optimal: 500+ examples
+
+### 2. Example Quality
+
+- Clear, consistent formatting
+- Diverse scenarios
+- Accurate, high-quality responses
+- Balanced across use cases
+
+### 3. Token Count
+
+- Keep examples under 4096 tokens
+- Aim for consistent lengths
+- Monitor token distribution
+
+### 4. System Prompts
+
+```typescript
+// Good: Specific, consistent
+{ role: 'system', content: 'You are a Python expert who writes clean, documented code.' }
+
+// Bad: Generic, vague
+{ role: 'system', content: 'You are helpful.' }
+```
+
+## Pricing (as of 2024)
+
+OpenAI fine-tuning costs:
+
+- **GPT-3.5 Turbo**: ~$0.008 per 1K tokens
+- **GPT-4**: ~$0.030 per 1K tokens
+
+Example:
+- 100 examples × 200 tokens = 20K tokens
+- Cost: 20 × $0.008 = **$0.16** (GPT-3.5)
+
+## Dependencies
+
+- [@neural-tools/core](../core) - Core utilities
+
+### Peer Dependencies
+
+- `openai` - Optional, for OpenAI integration
+
+## Contributing
+
+Contributions are welcome! See the [main repository](https://github.com/MacLeanLuke/neural-tools) for guidelines.
+
+## License
+
+MIT - See [LICENSE.md](../../LICENSE.md) for details.
+
+## Links
+
+- [Documentation](https://neural-tools.com/docs/fine-tune.html)
+- [GitHub](https://github.com/MacLeanLuke/neural-tools)
+- [npm](https://www.npmjs.com/package/@neural-tools/fine-tune)
+- [OpenAI Fine-Tuning Guide](https://platform.openai.com/docs/guides/fine-tuning)

package/dist/index.d.mts

@@ -0,0 +1,71 @@
+interface FineTuneConfig {
+    provider: 'openai' | 'anthropic' | 'custom';
+    model: string;
+    datasetPath: string;
+    validationSplit?: number;
+    epochs?: number;
+    learningRate?: number;
+    batchSize?: number;
+}
+interface TrainingExample {
+    messages: Array<{
+        role: 'system' | 'user' | 'assistant';
+        content: string;
+    }>;
+}
+interface FineTuneJob {
+    id: string;
+    status: 'pending' | 'running' | 'completed' | 'failed';
+    model: string;
+    trainingFile?: string;
+    validationFile?: string;
+    createdAt: number;
+    finishedAt?: number;
+    error?: string;
+}
+/**
+ * Prepare training data for fine-tuning
+ */
+declare function prepareTrainingData(examples: TrainingExample[], options?: {
+    validationSplit?: number;
+    outputDir?: string;
+}): Promise<{
+    trainingFile: string;
+    validationFile?: string;
+}>;
+/**
+ * Validate training data format
+ */
+declare function validateTrainingData(examples: TrainingExample[]): {
+    valid: boolean;
+    errors: string[];
+};
+/**
+ * Create a fine-tune job (placeholder - actual implementation would call provider APIs)
+ */
+declare function createFineTuneJob(config: FineTuneConfig): Promise<FineTuneJob>;
+/**
+ * Get fine-tune job status (placeholder)
+ */
+declare function getFineTuneJob(jobId: string): Promise<FineTuneJob>;
+/**
+ * Convert conversations to training examples
+ */
+declare function conversationsToExamples(conversations: Array<{
+    system?: string;
+    messages: Array<{
+        role: 'user' | 'assistant';
+        content: string;
+    }>;
+}>): TrainingExample[];
+/**
+ * Calculate dataset statistics
+ */
+declare function analyzeDataset(examples: TrainingExample[]): {
+    totalExamples: number;
+    avgMessagesPerExample: number;
+    avgTokensPerMessage: number;
+    roleDistribution: Record<string, number>;
+};
+
+export { type FineTuneConfig, type FineTuneJob, type TrainingExample, analyzeDataset, conversationsToExamples, createFineTuneJob, getFineTuneJob, prepareTrainingData, validateTrainingData };

package/dist/index.d.ts

@@ -1,4 +1,4 @@
-export interface FineTuneConfig {
+interface FineTuneConfig {
     provider: 'openai' | 'anthropic' | 'custom';
     model: string;
     datasetPath: string;
@@ -7,13 +7,13 @@ export interface FineTuneConfig {
     learningRate?: number;
     batchSize?: number;
 }
-export interface TrainingExample {
+interface TrainingExample {
     messages: Array<{
         role: 'system' | 'user' | 'assistant';
         content: string;
     }>;
 }
-export interface FineTuneJob {
+interface FineTuneJob {
     id: string;
     status: 'pending' | 'running' | 'completed' | 'failed';
     model: string;
@@ -26,7 +26,7 @@ export interface FineTuneJob {
 /**
  * Prepare training data for fine-tuning
  */
-export declare function prepareTrainingData(examples: TrainingExample[], options?: {
+declare function prepareTrainingData(examples: TrainingExample[], options?: {
     validationSplit?: number;
     outputDir?: string;
 }): Promise<{
@@ -36,22 +36,22 @@ export declare function prepareTrainingData(examples: TrainingExample[], options
 /**
  * Validate training data format
  */
-export declare function validateTrainingData(examples: TrainingExample[]): {
+declare function validateTrainingData(examples: TrainingExample[]): {
     valid: boolean;
     errors: string[];
 };
 /**
  * Create a fine-tune job (placeholder - actual implementation would call provider APIs)
  */
-export declare function createFineTuneJob(config: FineTuneConfig): Promise<FineTuneJob>;
+declare function createFineTuneJob(config: FineTuneConfig): Promise<FineTuneJob>;
 /**
  * Get fine-tune job status (placeholder)
  */
-export declare function getFineTuneJob(jobId: string): Promise<FineTuneJob>;
+declare function getFineTuneJob(jobId: string): Promise<FineTuneJob>;
 /**
  * Convert conversations to training examples
  */
-export declare function conversationsToExamples(conversations: Array<{
+declare function conversationsToExamples(conversations: Array<{
     system?: string;
     messages: Array<{
         role: 'user' | 'assistant';
@@ -61,9 +61,11 @@ export declare function conversationsToExamples(conversations: Array<{
 /**
  * Calculate dataset statistics
  */
-export declare function analyzeDataset(examples: TrainingExample[]): {
+declare function analyzeDataset(examples: TrainingExample[]): {
     totalExamples: number;
     avgMessagesPerExample: number;
     avgTokensPerMessage: number;
     roleDistribution: Record<string, number>;
 };
+
+export { type FineTuneConfig, type FineTuneJob, type TrainingExample, analyzeDataset, conversationsToExamples, createFineTuneJob, getFineTuneJob, prepareTrainingData, validateTrainingData };

package/dist/index.js

@@ -1,150 +1,3 @@
-"use strict";
-var __importDefault = (this && this.__importDefault) || function (mod) {
-    return (mod && mod.__esModule) ? mod : { "default": mod };
-};
-Object.defineProperty(exports, "__esModule", { value: true });
-exports.prepareTrainingData = prepareTrainingData;
-exports.validateTrainingData = validateTrainingData;
-exports.createFineTuneJob = createFineTuneJob;
-exports.getFineTuneJob = getFineTuneJob;
-exports.conversationsToExamples = conversationsToExamples;
-exports.analyzeDataset = analyzeDataset;
-const core_1 = require("@neural-tools/core");
-const promises_1 = __importDefault(require("fs/promises"));
-const path_1 = __importDefault(require("path"));
-/**
- * Prepare training data for fine-tuning
- */
-async function prepareTrainingData(examples, options = {}) {
-    await (0, core_1.requireFeature)('fine-tuning', 'Fine-tuning');
-    const validationSplit = options.validationSplit || 0;
-    const outputDir = options.outputDir || './fine-tune-data';
-    // Create output directory
-    await promises_1.default.mkdir(outputDir, { recursive: true });
-    // Split data
-    const splitIndex = Math.floor(examples.length * (1 - validationSplit));
-    const trainingExamples = examples.slice(0, splitIndex);
-    const validationExamples = validationSplit > 0 ? examples.slice(splitIndex) : [];
-    // Write training file
-    const trainingFile = path_1.default.join(outputDir, 'training.jsonl');
-    const trainingContent = trainingExamples
-        .map(ex => JSON.stringify(ex))
-        .join('\n');
-    await promises_1.default.writeFile(trainingFile, trainingContent, 'utf-8');
-    // Write validation file if needed
-    let validationFile;
-    if (validationExamples.length > 0) {
-        validationFile = path_1.default.join(outputDir, 'validation.jsonl');
-        const validationContent = validationExamples
-            .map(ex => JSON.stringify(ex))
-            .join('\n');
-        await promises_1.default.writeFile(validationFile, validationContent, 'utf-8');
-    }
-    return { trainingFile, validationFile };
-}
-/**
- * Validate training data format
- */
-function validateTrainingData(examples) {
-    const errors = [];
-    if (examples.length === 0) {
-        errors.push('No training examples provided');
-        return { valid: false, errors };
-    }
-    examples.forEach((example, index) => {
-        if (!example.messages || !Array.isArray(example.messages)) {
-            errors.push(`Example ${index}: Missing or invalid messages array`);
-            return;
-        }
-        if (example.messages.length === 0) {
-            errors.push(`Example ${index}: Messages array is empty`);
-        }
-        example.messages.forEach((message, msgIndex) => {
-            if (!message.role || !['system', 'user', 'assistant'].includes(message.role)) {
-                errors.push(`Example ${index}, Message ${msgIndex}: Invalid role`);
-            }
-            if (!message.content || typeof message.content !== 'string') {
-                errors.push(`Example ${index}, Message ${msgIndex}: Missing or invalid content`);
-            }
-        });
-    });
-    return {
-        valid: errors.length === 0,
-        errors
-    };
-}
-/**
- * Create a fine-tune job (placeholder - actual implementation would call provider APIs)
- */
-async function createFineTuneJob(config) {
-    await (0, core_1.requireFeature)('fine-tuning', 'Fine-tuning');
-    // Validate dataset exists
-    try {
-        await promises_1.default.access(config.datasetPath);
-    }
-    catch {
-        throw new Error(`Dataset file not found: ${config.datasetPath}`);
-    }
-    // In production, this would:
-    // 1. Upload training data to provider
-    // 2. Start fine-tuning job
-    // 3. Return job details
-    const job = {
-        id: `ft-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`,
-        status: 'pending',
-        model: config.model,
-        trainingFile: config.datasetPath,
-        createdAt: Date.now()
-    };
-    return job;
-}
-/**
- * Get fine-tune job status (placeholder)
- */
-async function getFineTuneJob(jobId) {
-    await (0, core_1.requireFeature)('fine-tuning', 'Fine-tuning');
-    // In production, this would query the provider API
-    throw new Error('Fine-tune job tracking coming soon');
-}
-/**
- * Convert conversations to training examples
- */
-function conversationsToExamples(conversations) {
-    return conversations.map(conv => {
-        const messages = [];
-        if (conv.system) {
-            messages.push({
-                role: 'system',
-                content: conv.system
-            });
-        }
-        messages.push(...conv.messages);
-        return { messages };
-    });
-}
-/**
- * Calculate dataset statistics
- */
-function analyzeDataset(examples) {
-    const stats = {
-        totalExamples: examples.length,
-        avgMessagesPerExample: 0,
-        avgTokensPerMessage: 0,
-        roleDistribution: {}
-    };
-    let totalMessages = 0;
-    let totalTokens = 0;
-    examples.forEach(example => {
-        totalMessages += example.messages.length;
-        example.messages.forEach(message => {
-            // Simple token estimation (real implementation would use tiktoken)
-            const tokens = message.content.split(/\s+/).length;
-            totalTokens += tokens;
-            stats.roleDistribution[message.role] =
-                (stats.roleDistribution[message.role] || 0) + 1;
-        });
-    });
-    stats.avgMessagesPerExample = totalMessages / examples.length;
-    stats.avgTokensPerMessage = totalTokens / totalMessages;
-    return stats;
-}
+"use strict";var v=Object.create;var l=Object.defineProperty;var E=Object.getOwnPropertyDescriptor;var y=Object.getOwnPropertyNames;var F=Object.getPrototypeOf,T=Object.prototype.hasOwnProperty;var x=(n,t)=>{for(var e in t)l(n,e,{get:t[e],enumerable:!0})},d=(n,t,e,r)=>{if(t&&typeof t=="object"||typeof t=="function")for(let i of y(t))!T.call(n,i)&&i!==e&&l(n,i,{get:()=>t[i],enumerable:!(r=E(t,i))||r.enumerable});return n};var f=(n,t,e)=>(e=n!=null?v(F(n)):{},d(t||!n||!n.__esModule?l(e,"default",{value:n,enumerable:!0}):e,n)),w=n=>d(l({},"__esModule",{value:!0}),n);var J={};x(J,{analyzeDataset:()=>A,conversationsToExamples:()=>$,createFineTuneJob:()=>P,getFineTuneJob:()=>j,prepareTrainingData:()=>M,validateTrainingData:()=>D});module.exports=w(J);var g=require("@neural-tools/core"),s=f(require("fs/promises")),p=f(require("path"));async function M(n,t={}){await(0,g.requireFeature)("fine-tuning","Fine-tuning");let e=t.validationSplit||0,r=t.outputDir||"./fine-tune-data";await s.default.mkdir(r,{recursive:!0});let i=Math.floor(n.length*(1-e)),a=n.slice(0,i),o=e>0?n.slice(i):[],c=p.default.join(r,"training.jsonl"),h=a.map(m=>JSON.stringify(m)).join(`
+`);await s.default.writeFile(c,h,"utf-8");let u;if(o.length>0){u=p.default.join(r,"validation.jsonl");let m=o.map(b=>JSON.stringify(b)).join(`
+`);await s.default.writeFile(u,m,"utf-8")}return{trainingFile:c,validationFile:u}}function D(n){let t=[];return n.length===0?(t.push("No training examples provided"),{valid:!1,errors:t}):(n.forEach((e,r)=>{if(!e.messages||!Array.isArray(e.messages)){t.push(`Example ${r}: Missing or invalid messages array`);return}e.messages.length===0&&t.push(`Example ${r}: Messages array is empty`),e.messages.forEach((i,a)=>{(!i.role||!["system","user","assistant"].includes(i.role))&&t.push(`Example ${r}, Message ${a}: Invalid role`),(!i.content||typeof i.content!="string")&&t.push(`Example ${r}, Message ${a}: Missing or invalid content`)})}),{valid:t.length===0,errors:t})}async function P(n){await(0,g.requireFeature)("fine-tuning","Fine-tuning");try{await s.default.access(n.datasetPath)}catch{throw new Error(`Dataset file not found: ${n.datasetPath}`)}return{id:`ft-${Date.now()}-${Math.random().toString(36).substr(2,9)}`,status:"pending",model:n.model,trainingFile:n.datasetPath,createdAt:Date.now()}}async function j(n){throw await(0,g.requireFeature)("fine-tuning","Fine-tuning"),new Error("Fine-tune job tracking coming soon")}function $(n){return n.map(t=>{let e=[];return t.system&&e.push({role:"system",content:t.system}),e.push(...t.messages),{messages:e}})}function A(n){let t={totalExamples:n.length,avgMessagesPerExample:0,avgTokensPerMessage:0,roleDistribution:{}},e=0,r=0;return n.forEach(i=>{e+=i.messages.length,i.messages.forEach(a=>{let o=a.content.split(/\s+/).length;r+=o,t.roleDistribution[a.role]=(t.roleDistribution[a.role]||0)+1})}),t.avgMessagesPerExample=e/n.length,t.avgTokensPerMessage=r/e,t}0&&(module.exports={analyzeDataset,conversationsToExamples,createFineTuneJob,getFineTuneJob,prepareTrainingData,validateTrainingData});

package/dist/index.mjs

@@ -0,0 +1,3 @@
+import{requireFeature as u}from"@neural-tools/core";import o from"fs/promises";import p from"path";async function v(n,t={}){await u("fine-tuning","Fine-tuning");let e=t.validationSplit||0,i=t.outputDir||"./fine-tune-data";await o.mkdir(i,{recursive:!0});let r=Math.floor(n.length*(1-e)),a=n.slice(0,r),s=e>0?n.slice(r):[],m=p.join(i,"training.jsonl"),c=a.map(g=>JSON.stringify(g)).join(`
+`);await o.writeFile(m,c,"utf-8");let l;if(s.length>0){l=p.join(i,"validation.jsonl");let g=s.map(d=>JSON.stringify(d)).join(`
+`);await o.writeFile(l,g,"utf-8")}return{trainingFile:m,validationFile:l}}function E(n){let t=[];return n.length===0?(t.push("No training examples provided"),{valid:!1,errors:t}):(n.forEach((e,i)=>{if(!e.messages||!Array.isArray(e.messages)){t.push(`Example ${i}: Missing or invalid messages array`);return}e.messages.length===0&&t.push(`Example ${i}: Messages array is empty`),e.messages.forEach((r,a)=>{(!r.role||!["system","user","assistant"].includes(r.role))&&t.push(`Example ${i}, Message ${a}: Invalid role`),(!r.content||typeof r.content!="string")&&t.push(`Example ${i}, Message ${a}: Missing or invalid content`)})}),{valid:t.length===0,errors:t})}async function y(n){await u("fine-tuning","Fine-tuning");try{await o.access(n.datasetPath)}catch{throw new Error(`Dataset file not found: ${n.datasetPath}`)}return{id:`ft-${Date.now()}-${Math.random().toString(36).substr(2,9)}`,status:"pending",model:n.model,trainingFile:n.datasetPath,createdAt:Date.now()}}async function F(n){throw await u("fine-tuning","Fine-tuning"),new Error("Fine-tune job tracking coming soon")}function T(n){return n.map(t=>{let e=[];return t.system&&e.push({role:"system",content:t.system}),e.push(...t.messages),{messages:e}})}function x(n){let t={totalExamples:n.length,avgMessagesPerExample:0,avgTokensPerMessage:0,roleDistribution:{}},e=0,i=0;return n.forEach(r=>{e+=r.messages.length,r.messages.forEach(a=>{let s=a.content.split(/\s+/).length;i+=s,t.roleDistribution[a.role]=(t.roleDistribution[a.role]||0)+1})}),t.avgMessagesPerExample=e/n.length,t.avgTokensPerMessage=i/e,t}export{x as analyzeDataset,T as conversationsToExamples,y as createFineTuneJob,F as getFineTuneJob,v as prepareTrainingData,E as validateTrainingData};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@neural-tools/fine-tune",
-  "version": "0.1.4",
+  "version": "0.1.6",
   "description": "Fine-tuning utilities for Neural Tools",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
@@ -13,7 +13,7 @@
     "url": "https://github.com/MacLeanLuke/neural-tools.git",
     "directory": "packages/fine-tune"
   },
-  "homepage": "https://neural-tools.com",
+  "homepage": "https://neural-tools.com/docs/fine-tune.html",
   "bugs": {
     "url": "https://github.com/MacLeanLuke/neural-tools/issues"
   },
@@ -26,7 +26,7 @@
     "ai"
   ],
   "dependencies": {
-    "@neural-tools/core": "0.1.4"
+    "@neural-tools/core": "0.1.6"
   },
   "devDependencies": {
     "@types/node": "^20.11.5",
@@ -44,8 +44,8 @@
     "dist"
   ],
   "scripts": {
-    "build": "tsc",
-    "dev": "tsc --watch",
+    "build": "tsup",
+    "dev": "tsup --watch",
     "clean": "rm -rf dist",
     "test": "echo 'Tests coming soon'"
   }