@lov3kaizen/agentsea-ingest 0.5.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 lovekaizen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,278 @@
+# @lov3kaizen/agentsea-ingest
+
+TypeScript-native document processing pipeline for AI/RAG applications.
+
+## Features
+
+- **Multi-format Parsing**: PDF, DOCX, HTML, Markdown, CSV, Excel, JSON
+- **Intelligent Chunking**: Fixed, recursive, sentence, paragraph, semantic, hierarchical
+- **Table & Image Extraction**: Automatic extraction with metadata
+- **Text Cleaning**: Normalization, deduplication, PII removal
+- **Flexible Pipelines**: Configurable processing stages
+- **Streaming Support**: Process large documents efficiently
+
+## Installation
+
+```bash
+pnpm add @lov3kaizen/agentsea-ingest
+```
+
+## Quick Start
+
+```typescript
+import { createIngester, pipelines } from '@lov3kaizen/agentsea-ingest';
+
+// Simple ingestion
+const ingester = createIngester();
+const doc = await ingester.ingestFile('./document.pdf');
+console.log(`Extracted ${doc.chunks.length} chunks`);
+
+// RAG-optimized pipeline
+const pipeline = pipelines.rag().build();
+const result = await pipeline.process({ path: './document.md' });
+```
+
+## Parsing Documents
+
+### Supported Formats
+
+| Format   | Parser         | MIME Types                                                              |
+| -------- | -------------- | ----------------------------------------------------------------------- |
+| PDF      | PDFParser      | application/pdf                                                         |
+| DOCX     | DOCXParser     | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
+| HTML     | HTMLParser     | text/html                                                               |
+| Markdown | MarkdownParser | text/markdown                                                           |
+| Text     | TextParser     | text/plain                                                              |
+| CSV      | CSVParser      | text/csv                                                                |
+| Excel    | ExcelParser    | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet       |
+| JSON     | JSONParser     | application/json                                                        |
+
+### Direct Parser Usage
+
+```typescript
+import {
+  createPDFParser,
+  createMarkdownParser,
+} from '@lov3kaizen/agentsea-ingest';
+import { readFileSync } from 'fs';
+
+const pdfParser = createPDFParser();
+const buffer = readFileSync('./document.pdf');
+const result = await pdfParser.parse(buffer);
+
+console.log(result.text);
+console.log(result.elements);
+console.log(result.tables);
+```
+
+## Chunking Strategies
+
+### Fixed Size
+
+```typescript
+import { createFixedChunker } from '@lov3kaizen/agentsea-ingest';
+
+const chunker = createFixedChunker();
+const chunks = chunker.chunk(text, {
+  maxTokens: 512,
+  overlap: 50,
+  splitOnSentences: true,
+});
+```
+
+### Recursive
+
+```typescript
+import { createRecursiveChunker } from '@lov3kaizen/agentsea-ingest';
+
+const chunker = createRecursiveChunker();
+const chunks = chunker.chunk(text, {
+  maxTokens: 512,
+  separators: ['\n\n', '\n', '. ', ' '],
+  keepSeparator: true,
+});
+```
+
+### Semantic
+
+```typescript
+import { createSemanticChunker } from '@lov3kaizen/agentsea-ingest';
+
+const chunker = createSemanticChunker();
+const chunks = await chunker.chunk(text, {
+  maxTokens: 512,
+  similarityThreshold: 0.5,
+  embedFunction: async (text) => myEmbeddingModel(text),
+});
+```
+
+### Hierarchical
+
+```typescript
+import { createHierarchicalChunker } from '@lov3kaizen/agentsea-ingest';
+
+const chunker = createHierarchicalChunker();
+const chunks = chunker.chunk(markdownText, {
+  maxTokens: 512,
+  headingLevels: [1, 2, 3],
+  includeParentContext: true,
+});
+```
+
+## Pipeline Builder
+
+```typescript
+import { createPipelineBuilder } from '@lov3kaizen/agentsea-ingest';
+
+const pipeline = createPipelineBuilder()
+  .withName('my-pipeline')
+  .withStages(['load', 'parse', 'clean', 'chunk', 'embed'])
+  .withChunking({
+    strategy: 'semantic',
+    maxTokens: 512,
+    overlap: 50,
+  })
+  .withCleaning({
+    operations: ['normalize_whitespace', 'remove_urls', 'trim'],
+  })
+  .withCallbacks({
+    onDocumentComplete: (doc) => console.log(`Processed: ${doc.id}`),
+  })
+  .build();
+
+const result = await pipeline.process({ path: './document.pdf' });
+```
+
+## Pre-built Pipelines
+
+```typescript
+import { pipelines } from '@lov3kaizen/agentsea-ingest';
+
+// Simple text extraction
+const simple = pipelines.simple().build();
+
+// Full processing with all stages
+const full = pipelines.full().build();
+
+// RAG-optimized pipeline
+const rag = pipelines.rag().build();
+
+// Document analysis (no chunking)
+const analysis = pipelines.analysis().build();
+
+// OCR pipeline for scanned documents
+const ocr = pipelines.ocr().build();
+```
+
+## Ingester
+
+The `Ingester` class provides a high-level API for document ingestion:
+
+```typescript
+import { createIngester } from '@lov3kaizen/agentsea-ingest';
+
+const ingester = createIngester({
+  chunking: {
+    strategy: 'recursive',
+    maxTokens: 512,
+  },
+  concurrency: 4,
+  fileSizeLimit: 10 * 1024 * 1024, // 10MB
+});
+
+// Ingest single file
+const doc = await ingester.ingestFile('./document.pdf');
+
+// Ingest from URL
+const webDoc = await ingester.ingestUrl('https://example.com/page.html');
+
+// Ingest from buffer
+const bufferDoc = await ingester.ingestBuffer(buffer, 'document.pdf');
+
+// Ingest directory
+const results = await ingester.ingestDirectory('./documents', {
+  recursive: true,
+  include: ['*.pdf', '*.docx'],
+  exclude: ['draft-*'],
+});
+```
+
+## Watch Mode
+
+```typescript
+const ingester = createIngester({
+  watchMode: {
+    enabled: true,
+    paths: ['./documents'],
+    include: ['*.pdf', '*.md'],
+    debounceDelay: 1000,
+    processExisting: true,
+  },
+});
+
+ingester.startWatching();
+// Files added/modified in ./documents will be automatically processed
+```
+
+## Events
+
+```typescript
+const pipeline = createPipeline(config);
+const emitter = pipeline.getEventEmitter();
+
+emitter.on('document:loaded', (event) => {
+  console.log(`Loaded: ${event.documentId}`);
+});
+
+emitter.on('document:chunked', (event) => {
+  console.log(`Created ${event.chunkCount} chunks`);
+});
+
+emitter.on('document:completed', (event) => {
+  console.log(`Completed: ${event.document.id}`);
+});
+```
+
+## API Reference
+
+### Types
+
+- `ProcessedDocument` - Processed document with chunks and metadata
+- `Chunk` - Text chunk with metadata and optional embedding
+- `Element` - Document element (paragraph, heading, list, etc.)
+- `TableData` - Extracted table data
+- `ImageData` - Extracted image data
+- `PipelineConfig` - Pipeline configuration options
+- `ChunkingOptions` - Chunking configuration options
+
+### Core Classes
+
+- `Pipeline` - Document processing pipeline
+- `PipelineBuilder` - Fluent pipeline builder
+- `Ingester` - High-level document ingester
+- `ParserRegistry` - Parser management
+- `ChunkerRegistry` - Chunker management
+
+### Parsers
+
+- `PDFParser` - PDF document parsing
+- `DOCXParser` - Word document parsing
+- `HTMLParser` - HTML document parsing
+- `MarkdownParser` - Markdown parsing
+- `TextParser` - Plain text parsing
+- `CSVParser` - CSV file parsing
+- `ExcelParser` - Excel file parsing
+- `JSONParser` - JSON file parsing
+
+### Chunkers
+
+- `FixedChunker` - Fixed-size chunks
+- `RecursiveChunker` - Recursive splitting
+- `SentenceChunker` - Sentence-based chunks
+- `ParagraphChunker` - Paragraph-based chunks
+- `SemanticChunker` - Semantic similarity-based
+- `HierarchicalChunker` - Heading-based hierarchy
+
+## License
+
+MIT