@markitdownjs/chunking 0.1.0 → 0.1.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 MarkItDownJS Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,66 @@
+# @markitdownjs/chunking
+
+[![npm](https://img.shields.io/npm/v/@markitdownjs/chunking)](https://www.npmjs.com/package/@markitdownjs/chunking)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+First-class document chunking for RAG pipelines. Splits a `DocumentNode` AST into semantic chunks with rich metadata — heading path, page number, token count, and chunk ID.
+
+## Install
+
+```bash
+npm install @markitdownjs/chunking
+```
+
+## Usage
+
+```ts
+import { Chunker, HeadingChunkStrategy } from "@markitdownjs/chunking";
+
+const chunker = new Chunker({ strategy: new HeadingChunkStrategy({ maxTokens: 512 }) });
+const chunks = chunker.chunk(documentNode, { sourceFile: "report.pdf" });
+```
+
+## Chunk Schema
+
+```ts
+interface Chunk {
+  chunkId: string;
+  sourceFile: string;
+  pageNumber: number | null;
+  headingPath: string[];
+  tokenCount: number;
+  metadata: Record<string, unknown>;
+  content: string;
+}
+```
+
+## Strategies
+
+| Export | Splits on |
+|---|---|
+| `HeadingChunkStrategy` | Document headings (h1–h6), with `maxTokens` overflow splitting |
+| `PageChunkStrategy` | Page boundaries extracted from the AST |
+| `TokenChunkStrategy` | Fixed token window with configurable overlap |
+| `SemanticChunkStrategy` | Sentence-boundary aware splits using embedding similarity |
+
+```ts
+import { Chunker, TokenChunkStrategy } from "@markitdownjs/chunking";
+
+const chunker = new Chunker({
+  strategy: new TokenChunkStrategy({ maxTokens: 256, overlap: 32 }),
+});
+```
+
+## Strategy Options
+
+```ts
+new HeadingChunkStrategy({
+  maxTokens: 512,        // split oversized sections further
+  minTokens: 50,         // merge sections below threshold
+  includeHeadingPath: true,
+});
+```
+
+---
+
+Part of the [MarkItDownJS](https://github.com/markitdownjs/markitdownjs) monorepo.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@markitdownjs/chunking",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "description": "AI-ready document chunking strategies for MarkItDownJS",
   "license": "MIT",
   "type": "module",
@@ -17,7 +17,7 @@
     "dist"
   ],
   "dependencies": {
-    "@markitdownjs/shared": "0.1.0"
+    "@markitdownjs/shared": "0.1.1"
   },
   "devDependencies": {
     "typescript": "^5.5.0"