@mastra/rag 2.0.0-beta.5 → 2.0.0-beta.7

package/CHANGELOG.md

@@ -1,5 +1,162 @@
 # @mastra/rag
 
+## 2.0.0-beta.7
+
+### Major Changes
+
+- Refactor workflow and tool types to remove Zod-specific constraints ([#11814](https://github.com/mastra-ai/mastra/pull/11814))
+
+  Removed Zod-specific type constraints across all workflow implementations and tool types, replacing them with generic types. This ensures type consistency across default, evented, and inngest workflows while preparing for Zod v4 migration.
+
+  **Workflow Changes:**
+  - Removed `z.ZodObject<any>` and `z.ZodType<any>` constraints from all workflow generic types
+  - Updated method signatures to use `TInput` and `TState` directly instead of `z.infer<TInput>` and `z.infer<TState>`
+  - Aligned conditional types across all workflow implementations using `TInput extends unknown` pattern
+  - Fixed `TSteps` generic to properly use `TEngineType` instead of `any`
+
+  **Tool Changes:**
+  - Removed Zod schema constraints from `ToolExecutionContext` and related interfaces
+  - Simplified type parameters from `TSuspendSchema extends ZodLikeSchema` to `TSuspend` and `TResume`
+  - Updated tool execution context types to use generic types
+
+  **Type Utilities:**
+  - Refactored type helpers to work with generic schemas instead of Zod-specific types
+  - Updated type extraction utilities for better compatibility
+
+  This change maintains backward compatibility while improving type consistency and preparing for Zod v4 support across all affected packages.
+
+### Minor Changes
+
+- Add schema-driven metadata extraction with Zod support ([#11833](https://github.com/mastra-ai/mastra/pull/11833))
+
+  Introduces a new `SchemaExtractor` that enables extraction of custom structured metadata from document chunks using user-defined Zod schemas. This allows for domain-specific metadata structures (e.g., product details, legal entities, sentiment analysis) to be reliably extracted via LLM structured output.
+  - Extract domain-specific metadata using your own Zod schemas (e.g., product details, legal entities, sentiment)
+  - Customize extraction behavior with your own LLM model and instructions
+  - Organize extracted data by nesting it under custom metadata keys
+  - Existing extractors (title, summary, keywords, questions) remain unchanged and fully compatible
+
+  **Before** (limited to built-in extractors):
+
+  ```typescript
+  await document.extractMetadata({
+    extract: {
+      title: true,
+      summary: true,
+    },
+  });
+  ```
+
+  **After** (with custom Zod schema):
+
+  ```typescript
+  import { z } from 'zod';
+
+  const productSchema = z.object({
+    name: z.string(),
+    price: z.number(),
+    category: z.string(),
+  });
+
+  await document.extractMetadata({
+    extract: {
+      title: true,
+      schema: {
+        schema: productSchema,
+        instructions: 'Extract product details from the document',
+        metadataKey: 'product',
+      },
+    },
+  });
+  ```
+
+  With `metadataKey`, extracted data is nested under the key:
+
+  ```typescript
+  {
+    title: "Product Document",
+    summary: "A comprehensive guide",
+    product: {
+      name: "Wireless Headphones",
+      price: 149.99,
+      category: "Electronics"
+    }
+  }
+  ```
+
+  Without `metadataKey`, extracted data is returned inline:
+
+  ```typescript
+  {
+    title: "Product Document",
+    summary: "A comprehensive guide",
+    name: "Wireless Headphones",
+    price: 149.99,
+    category: "Electronics"
+  }
+  ```
+
+  Fixes #11799
+
+- Renamed `keepSeparator` parameter to `separatorPosition` with a cleaner type. ([#11802](https://github.com/mastra-ai/mastra/pull/11802))
+
+  The `keepSeparator` parameter had a confusing `boolean | 'start' | 'end'` type where `true` was secretly an alias for `'start'`. The new `separatorPosition` parameter uses explicit `'start' | 'end'` values, and omitting the parameter discards the separator (previous default behavior).
+
+  **Migration**
+
+  ```typescript
+  // Before
+  await doc.chunk({
+    strategy: 'character',
+    separator: '.',
+    keepSeparator: true, // or 'start'
+  });
+
+  await doc.chunk({
+    strategy: 'character',
+    separator: '.',
+    keepSeparator: 'end',
+  });
+
+  await doc.chunk({
+    strategy: 'character',
+    separator: '.',
+    keepSeparator: false, // or omit entirely
+  });
+
+  // After
+  await doc.chunk({
+    strategy: 'character',
+    separator: '.',
+    separatorPosition: 'start',
+  });
+
+  await doc.chunk({
+    strategy: 'character',
+    separator: '.',
+    separatorPosition: 'end',
+  });
+
+  await doc.chunk({
+    strategy: 'character',
+    separator: '.',
+    // omit separatorPosition to discard separator
+  });
+  ```
+
+### Patch Changes
+
+- Updated dependencies [[`ebae12a`](https://github.com/mastra-ai/mastra/commit/ebae12a2dd0212e75478981053b148a2c246962d), [`c61a0a5`](https://github.com/mastra-ai/mastra/commit/c61a0a5de4904c88fd8b3718bc26d1be1c2ec6e7), [`69136e7`](https://github.com/mastra-ai/mastra/commit/69136e748e32f57297728a4e0f9a75988462f1a7), [`449aed2`](https://github.com/mastra-ai/mastra/commit/449aed2ba9d507b75bf93d427646ea94f734dfd1), [`eb648a2`](https://github.com/mastra-ai/mastra/commit/eb648a2cc1728f7678768dd70cd77619b448dab9), [`0131105`](https://github.com/mastra-ai/mastra/commit/0131105532e83bdcbb73352fc7d0879eebf140dc), [`9d5059e`](https://github.com/mastra-ai/mastra/commit/9d5059eae810829935fb08e81a9bb7ecd5b144a7), [`ef756c6`](https://github.com/mastra-ai/mastra/commit/ef756c65f82d16531c43f49a27290a416611e526), [`b00ccd3`](https://github.com/mastra-ai/mastra/commit/b00ccd325ebd5d9e37e34dd0a105caae67eb568f), [`3bdfa75`](https://github.com/mastra-ai/mastra/commit/3bdfa7507a91db66f176ba8221aa28dd546e464a), [`e770de9`](https://github.com/mastra-ai/mastra/commit/e770de941a287a49b1964d44db5a5763d19890a6), [`52e2716`](https://github.com/mastra-ai/mastra/commit/52e2716b42df6eff443de72360ae83e86ec23993), [`27b4040`](https://github.com/mastra-ai/mastra/commit/27b4040bfa1a95d92546f420a02a626b1419a1d6), [`610a70b`](https://github.com/mastra-ai/mastra/commit/610a70bdad282079f0c630e0d7bb284578f20151), [`8dc7f55`](https://github.com/mastra-ai/mastra/commit/8dc7f55900395771da851dc7d78d53ae84fe34ec), [`8379099`](https://github.com/mastra-ai/mastra/commit/8379099fc467af6bef54dd7f80c9bd75bf8bbddf), [`8c0ec25`](https://github.com/mastra-ai/mastra/commit/8c0ec25646c8a7df253ed1e5ff4863a0d3f1316c), [`ff4d9a6`](https://github.com/mastra-ai/mastra/commit/ff4d9a6704fc87b31a380a76ed22736fdedbba5a), [`69821ef`](https://github.com/mastra-ai/mastra/commit/69821ef806482e2c44e2197ac0b050c3fe3a5285), [`1ed5716`](https://github.com/mastra-ai/mastra/commit/1ed5716830867b3774c4a1b43cc0d82935f32b96), [`4186bdd`](https://github.com/mastra-ai/mastra/commit/4186bdd00731305726fa06adba0b076a1d50b49f), [`7aaf973`](https://github.com/mastra-ai/mastra/commit/7aaf973f83fbbe9521f1f9e7a4fd99b8de464617)]:
+  - @mastra/core@1.0.0-beta.22
+
+## 2.0.0-beta.6
+
+### Patch Changes
+
+- Remove unnecessary `ai` package peer dependency to enable compatibility with AI SDK v6. The rag package doesn't directly use the ai package, so this peer dependency was unnecessarily constraining version compatibility. ([#11724](https://github.com/mastra-ai/mastra/pull/11724))
+
+- Updated dependencies [[`08766f1`](https://github.com/mastra-ai/mastra/commit/08766f15e13ac0692fde2a8bd366c2e16e4321df), [`ae8baf7`](https://github.com/mastra-ai/mastra/commit/ae8baf7d8adcb0ff9dac11880400452bc49b33ff), [`cfabdd4`](https://github.com/mastra-ai/mastra/commit/cfabdd4aae7a726b706942d6836eeca110fb6267), [`a0e437f`](https://github.com/mastra-ai/mastra/commit/a0e437fac561b28ee719e0302d72b2f9b4c138f0), [`bec5efd`](https://github.com/mastra-ai/mastra/commit/bec5efde96653ccae6604e68c696d1bc6c1a0bf5), [`9eedf7d`](https://github.com/mastra-ai/mastra/commit/9eedf7de1d6e0022a2f4e5e9e6fe1ec468f9b43c)]:
+  - @mastra/core@1.0.0-beta.21
+
 ## 2.0.0-beta.5
 
 ### Minor Changes

package/dist/docs/README.md

@@ -29,4 +29,4 @@ docs/
 ## Version
 
 Package: @mastra/rag
-Version: 2.0.0-beta.5
+Version: 2.0.0-beta.7

package/dist/docs/SKILL.md

@@ -5,7 +5,7 @@ description: Documentation for @mastra/rag. Includes links to type definitions a
 
 # @mastra/rag Documentation
 
-> **Version**: 2.0.0-beta.5
+> **Version**: 2.0.0-beta.7
 > **Package**: @mastra/rag
 
 ## Quick Navigation

package/dist/docs/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "2.0.0-beta.5",
+  "version": "2.0.0-beta.7",
   "package": "@mastra/rag",
   "exports": {},
   "modules": {}

package/dist/docs/rag/02-chunking-and-embedding.md

@@ -51,7 +51,6 @@ const chunks = await doc.chunk({
   minSize: 50,
   overlap: 0,
   sentenceEnders: ["."],
-  keepSeparator: true,
 });
 ```
 

package/dist/docs/rag/03-retrieval.md

@@ -171,7 +171,7 @@ The Vector Query Tool supports database-specific configurations that enable you
 > **Note:**
 These configurations are for **query-time options** like namespaces, performance tuning, and filtering—not for database connection setup. 
 
-Connection credentials (URLs, auth tokens) are configured when you instantiate the vector store class (e.g., `new LibSQLVector({ connectionUrl: '...' })`).
+Connection credentials (URLs, auth tokens) are configured when you instantiate the vector store class (e.g., `new LibSQLVector({ url: '...' })`).
 
 ```ts
 import { createVectorQueryTool } from "@mastra/rag";
@@ -258,11 +258,10 @@ requestContext.set("databaseConfig", {
   },
 });
 
-await pineconeQueryTool.execute({
-  context: { queryText: "search query" },
-  mastra,
-  requestContext,
-});
+await pineconeQueryTool.execute(
+  { queryText: "search query" },
+  { mastra, requestContext }
+);
 ```
 
 For detailed configuration options and advanced usage, see the [Vector Query Tool Reference](https://mastra.ai/reference/v1/tools/vector-query-tool).

package/dist/docs/rag/05-reference.md

@@ -97,7 +97,7 @@ const results = await graphRag.query({
 
 ---
 
-## Reference: Reference: .chunk()
+## Reference: .chunk()
 
 > Documentation for the chunk function in Mastra, which splits documents into smaller segments using various strategies.
 
@@ -171,7 +171,6 @@ const chunks = await doc.chunk({
   minSize: 50, // Sentence-specific option
   sentenceEnders: ["."], // Sentence-specific option
   fallbackToCharacters: false, // Sentence-specific option
-  keepSeparator: true, // general option
 });
 
 // HTML strategy example
@@ -373,11 +372,10 @@ whereDocument: { "$contains": "API documentation" }
       }
     });
 
-    await vectorTool.execute({
-      context: { queryText: 'search query' },
-      mastra,
-      requestContext
-    });
+    await vectorTool.execute(
+      { queryText: 'search query' },
+      { mastra, requestContext }
+    );
     ```
 
   
@@ -672,6 +670,8 @@ The `extract` parameter accepts the following fields:
 
 ### KeywordExtractArgs
 
+### SchemaExtractArgs
+
 ## Advanced Example
 
 ```typescript
@@ -705,6 +705,16 @@ const chunks = await doc.chunk({
       keywords: 5, // Extract 5 keywords
       promptTemplate: "Extract {maxKeywords} key terms from: {context}",
     },
+
+    // Schema extraction with Zod
+    schema: {
+        schema: z.object({
+            productName: z.string(),
+            category: z.enum(["electronics", "clothing"]),
+        }),
+        instructions: "Extract product information.",
+        metadataKey: "product",
+    },
   },
 });
 
@@ -713,7 +723,11 @@ const chunks = await doc.chunk({
 //   documentTitle: "AI in Modern Computing",
 //   sectionSummary: "Overview of AI concepts and their applications in computing",
 //   questionsThisExcerptCanAnswer: "1. What is machine learning?\n2. How do neural networks work?",
-//   excerptKeywords: "1. Machine learning\n2. Neural networks\n3. Training data"
+//   excerptKeywords: "1. Machine learning\n2. Neural networks\n3. Training data",
+//   product: {
+//     productName: "Neural Net 2000",
+//     category: "electronics"
+//   }
 // }
 ```
 

package/dist/document/document.d.ts

@@ -10,7 +10,7 @@ export declare class MDocument {
         }[];
         type: string;
     });
-    extractMetadata({ title, summary, questions, keywords }: ExtractParams): Promise<MDocument>;
+    extractMetadata({ title, summary, questions, keywords, schema }: ExtractParams): Promise<MDocument>;
     static fromText(text: string, metadata?: Record<string, any>): MDocument;
     static fromHTML(html: string, metadata?: Record<string, any>): MDocument;
     static fromMarkdown(markdown: string, metadata?: Record<string, any>): MDocument;

package/dist/document/document.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"document.d.ts","sourceRoot":"","sources":["../../src/document/document.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,QAAQ,IAAI,KAAK,EAAgC,MAAM,UAAU,CAAC;AAU3E,OAAO,KAAK,EACV,WAAW,EAEX,aAAa,EACb,gBAAgB,EAChB,qBAAqB,EACrB,qBAAqB,EACrB,iBAAiB,EACjB,oBAAoB,EACpB,4BAA4B,EAC5B,gBAAgB,EAChB,iBAAiB,EACjB,oBAAoB,EAErB,MAAM,SAAS,CAAC;AAGjB,qBAAa,SAAS;IACpB,OAAO,CAAC,MAAM,CAAU;IACxB,OAAO,CAAC,IAAI,CAAS;gBAET,EAAE,IAAI,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE;YAAE,IAAI,EAAE,MAAM,CAAC;YAAC,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;SAAE,EAAE,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE;IAOhG,eAAe,CAAC,EAAE,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,QAAQ,EAAE,EAAE,aAAa,GAAG,OAAO,CAAC,SAAS,CAAC;IAmDjG,MAAM,CAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,SAAS;IAYxE,MAAM,CAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,SAAS;IAYxE,MAAM,CAAC,YAAY,CAAC,QAAQ,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,SAAS;IAYhF,MAAM,CAAC,QAAQ,CAAC,UAAU,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,SAAS;IAY9E,OAAO,CAAC,eAAe;IAevB,OAAO,CAAC,YAAY,CAAC,CAA4E;IAEjG,OAAO,KAAK,WAAW,GAetB;YAEa,OAAO;IASf,cAAc,CAAC,OAAO,CAAC,EAAE,qBAAqB,GAAG,OAAO,CAAC,IAAI,CAAC;IAa9D,cAAc,CAAC,OAAO,CAAC,EAAE,qBAAqB,GAAG,OAAO,CAAC,IAAI,CAAC;IAU9D,SAAS,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;IA8CpD,SAAS,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;IAmBpD,UAAU,CAAC,OAAO,CAAC,EAAE,iBAAiB,GAAG,OAAO,CAAC,IAAI,CAAC;IAMtD,UAAU,CAAC,OAAO,CAAC,EAAE,iBAAiB,GAAG,OAAO,CAAC,IAAI,CAAC;IAUtD,aAAa,CAAC,OAAO,CAAC,EAAE,oBAAoB,GAAG,OAAO,CAAC,IAAI,CAAC;IAa5D,aAAa,CAAC,OAAO,CAAC,EAAE,oBAAoB,GAAG,OAAO,CAAC,IAAI,CAAC;IAuB5D,qBAAqB,CAAC,OAAO,CAAC,EAAE,4BAA4B,GAAG,OAAO,CAAC,IAAI,CAAC;IAU5E,KAAK,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,KAAK,EAAE,CAAC;IAiBnD,OAAO,IAAI,KAAK,EAAE;IAIlB,OAAO,IAAI,MAAM,EAAE;IAInB,WAAW,IAAI,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE;CAGrC"}
+{"version":3,"file":"document.d.ts","sourceRoot":"","sources":["../../src/document/document.ts"],"names":[],"mappings":"AAQA,OAAO,EAAE,QAAQ,IAAI,KAAK,EAAgC,MAAM,UAAU,CAAC;AAU3E,OAAO,KAAK,EACV,WAAW,EAEX,aAAa,EACb,gBAAgB,EAChB,qBAAqB,EACrB,qBAAqB,EACrB,iBAAiB,EACjB,oBAAoB,EACpB,4BAA4B,EAC5B,gBAAgB,EAChB,iBAAiB,EACjB,oBAAoB,EAErB,MAAM,SAAS,CAAC;AAGjB,qBAAa,SAAS;IACpB,OAAO,CAAC,MAAM,CAAU;IACxB,OAAO,CAAC,IAAI,CAAS;gBAET,EAAE,IAAI,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE;YAAE,IAAI,EAAE,MAAM,CAAC;YAAC,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAA;SAAE,EAAE,CAAC;QAAC,IAAI,EAAE,MAAM,CAAA;KAAE;IAOhG,eAAe,CAAC,EAAE,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,QAAQ,EAAE,MAAM,EAAE,EAAE,aAAa,GAAG,OAAO,CAAC,SAAS,CAAC;IAuDzG,MAAM,CAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,SAAS;IAYxE,MAAM,CAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,SAAS;IAYxE,MAAM,CAAC,YAAY,CAAC,QAAQ,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,SAAS;IAYhF,MAAM,CAAC,QAAQ,CAAC,UAAU,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,SAAS;IAY9E,OAAO,CAAC,eAAe;IAevB,OAAO,CAAC,YAAY,CAAC,CAA4E;IAEjG,OAAO,KAAK,WAAW,GAetB;YAEa,OAAO;IASf,cAAc,CAAC,OAAO,CAAC,EAAE,qBAAqB,GAAG,OAAO,CAAC,IAAI,CAAC;IAa9D,cAAc,CAAC,OAAO,CAAC,EAAE,qBAAqB,GAAG,OAAO,CAAC,IAAI,CAAC;IAU9D,SAAS,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;IA8CpD,SAAS,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;IAmBpD,UAAU,CAAC,OAAO,CAAC,EAAE,iBAAiB,GAAG,OAAO,CAAC,IAAI,CAAC;IAMtD,UAAU,CAAC,OAAO,CAAC,EAAE,iBAAiB,GAAG,OAAO,CAAC,IAAI,CAAC;IAUtD,aAAa,CAAC,OAAO,CAAC,EAAE,oBAAoB,GAAG,OAAO,CAAC,IAAI,CAAC;IAa5D,aAAa,CAAC,OAAO,CAAC,EAAE,oBAAoB,GAAG,OAAO,CAAC,IAAI,CAAC;IAuB5D,qBAAqB,CAAC,OAAO,CAAC,EAAE,4BAA4B,GAAG,OAAO,CAAC,IAAI,CAAC;IAU5E,KAAK,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,KAAK,EAAE,CAAC;IAiBnD,OAAO,IAAI,KAAK,EAAE;IAIlB,OAAO,IAAI,MAAM,EAAE;IAInB,WAAW,IAAI,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE;CAGrC"}

package/dist/document/extractors/index.d.ts

@@ -2,5 +2,6 @@ export { TitleExtractor } from './title.js';
 export { SummaryExtractor } from './summary.js';
 export { QuestionsAnsweredExtractor } from './questions.js';
 export { KeywordExtractor } from './keywords.js';
-export type { KeywordExtractArgs, QuestionAnswerExtractArgs, SummaryExtractArgs, TitleExtractorsArgs } from './types.js';
+export { SchemaExtractor } from './schema.js';
+export type { KeywordExtractArgs, QuestionAnswerExtractArgs, SummaryExtractArgs, TitleExtractorsArgs, SchemaExtractArgs, } from './types.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/document/extractors/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/document/extractors/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,EAAE,gBAAgB,EAAE,MAAM,WAAW,CAAC;AAC7C,OAAO,EAAE,0BAA0B,EAAE,MAAM,aAAa,CAAC;AACzD,OAAO,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9C,YAAY,EAAE,kBAAkB,EAAE,yBAAyB,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,MAAM,SAAS,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../../src/document/extractors/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,SAAS,CAAC;AACzC,OAAO,EAAE,gBAAgB,EAAE,MAAM,WAAW,CAAC;AAC7C,OAAO,EAAE,0BAA0B,EAAE,MAAM,aAAa,CAAC;AACzD,OAAO,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAC9C,OAAO,EAAE,eAAe,EAAE,MAAM,UAAU,CAAC;AAC3C,YAAY,EACV,kBAAkB,EAClB,yBAAyB,EACzB,kBAAkB,EAClB,mBAAmB,EACnB,iBAAiB,GAClB,MAAM,SAAS,CAAC"}

package/dist/document/extractors/schema.d.ts

@@ -0,0 +1,13 @@
+import type { z } from 'zod';
+import type { BaseNode } from '../schema/index.js';
+import { BaseExtractor } from './base.js';
+import type { SchemaExtractArgs } from './types.js';
+export declare class SchemaExtractor<T extends z.ZodType> extends BaseExtractor {
+    private schema;
+    private llm?;
+    private instructions?;
+    private metadataKey?;
+    constructor(options: SchemaExtractArgs<T>);
+    extract(nodes: BaseNode[]): Promise<Record<string, any>[]>;
+}
+//# sourceMappingURL=schema.d.ts.map

package/dist/document/extractors/schema.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"schema.d.ts","sourceRoot":"","sources":["../../../src/document/extractors/schema.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAE7B,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAE1C,OAAO,EAAE,aAAa,EAAE,MAAM,QAAQ,CAAC;AAEvC,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,SAAS,CAAC;AAEjD,qBAAa,eAAe,CAAC,CAAC,SAAS,CAAC,CAAC,OAAO,CAAE,SAAQ,aAAa;IACrE,OAAO,CAAC,MAAM,CAAI;IAClB,OAAO,CAAC,GAAG,CAAC,CAAkD;IAC9D,OAAO,CAAC,YAAY,CAAC,CAAS;IAC9B,OAAO,CAAC,WAAW,CAAC,CAAS;gBAEjB,OAAO,EAAE,iBAAiB,CAAC,CAAC,CAAC;IAQnC,OAAO,CAAC,KAAK,EAAE,QAAQ,EAAE,GAAG,OAAO,CAAC,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAAC;CA6BjE"}

package/dist/document/extractors/types.d.ts

@@ -1,4 +1,5 @@
 import type { MastraLanguageModel, MastraLegacyLanguageModel } from '@mastra/core/agent';
+import type { z } from 'zod';
 import type { KeywordExtractPrompt, QuestionExtractPrompt, SummaryPrompt, TitleExtractorPrompt, TitleCombinePrompt } from '../prompts/index.js';
 export type KeywordExtractArgs = {
     llm?: MastraLegacyLanguageModel | MastraLanguageModel;
@@ -22,6 +23,12 @@ export type TitleExtractorsArgs = {
     nodeTemplate?: TitleExtractorPrompt['template'];
     combineTemplate?: TitleCombinePrompt['template'];
 };
+export type SchemaExtractArgs<T extends z.ZodType = z.ZodType> = {
+    schema: T;
+    llm?: MastraLegacyLanguageModel | MastraLanguageModel;
+    instructions?: string;
+    metadataKey?: string;
+};
 export declare const STRIP_REGEX: RegExp;
 export declare const baseLLM: MastraLegacyLanguageModel | MastraLanguageModel;
 //# sourceMappingURL=types.d.ts.map

package/dist/document/extractors/types.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../../src/document/extractors/types.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,mBAAmB,EAAE,yBAAyB,EAAE,MAAM,oBAAoB,CAAC;AACzF,OAAO,KAAK,EACV,oBAAoB,EACpB,qBAAqB,EACrB,aAAa,EACb,oBAAoB,EACpB,kBAAkB,EACnB,MAAM,YAAY,CAAC;AAEpB,MAAM,MAAM,kBAAkB,GAAG;IAC/B,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,cAAc,CAAC,EAAE,oBAAoB,CAAC,UAAU,CAAC,CAAC;CACnD,CAAC;AAEF,MAAM,MAAM,yBAAyB,GAAG;IACtC,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,cAAc,CAAC,EAAE,qBAAqB,CAAC,UAAU,CAAC,CAAC;IACnD,aAAa,CAAC,EAAE,OAAO,CAAC;CACzB,CAAC;AAEF,MAAM,MAAM,kBAAkB,GAAG;IAC/B,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;IACrB,cAAc,CAAC,EAAE,aAAa,CAAC,UAAU,CAAC,CAAC;CAC5C,CAAC;AAEF,MAAM,MAAM,mBAAmB,GAAG;IAChC,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,YAAY,CAAC,EAAE,oBAAoB,CAAC,UAAU,CAAC,CAAC;IAChD,eAAe,CAAC,EAAE,kBAAkB,CAAC,UAAU,CAAC,CAAC;CAClD,CAAC;AAEF,eAAO,MAAM,WAAW,QAAmB,CAAC;AAG5C,eAAO,MAAM,OAAO,EAAE,yBAAyB,GAAG,mBAAsC,CAAC"}
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../../src/document/extractors/types.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,mBAAmB,EAAE,yBAAyB,EAAE,MAAM,oBAAoB,CAAC;AACzF,OAAO,KAAK,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAC7B,OAAO,KAAK,EACV,oBAAoB,EACpB,qBAAqB,EACrB,aAAa,EACb,oBAAoB,EACpB,kBAAkB,EACnB,MAAM,YAAY,CAAC;AAEpB,MAAM,MAAM,kBAAkB,GAAG;IAC/B,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,cAAc,CAAC,EAAE,oBAAoB,CAAC,UAAU,CAAC,CAAC;CACnD,CAAC;AAEF,MAAM,MAAM,yBAAyB,GAAG;IACtC,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,cAAc,CAAC,EAAE,qBAAqB,CAAC,UAAU,CAAC,CAAC;IACnD,aAAa,CAAC,EAAE,OAAO,CAAC;CACzB,CAAC;AAEF,MAAM,MAAM,kBAAkB,GAAG;IAC/B,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;IACrB,cAAc,CAAC,EAAE,aAAa,CAAC,UAAU,CAAC,CAAC;CAC5C,CAAC;AAEF,MAAM,MAAM,mBAAmB,GAAG;IAChC,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,YAAY,CAAC,EAAE,oBAAoB,CAAC,UAAU,CAAC,CAAC;IAChD,eAAe,CAAC,EAAE,kBAAkB,CAAC,UAAU,CAAC,CAAC;CAClD,CAAC;AAEF,MAAM,MAAM,iBAAiB,CAAC,CAAC,SAAS,CAAC,CAAC,OAAO,GAAG,CAAC,CAAC,OAAO,IAAI;IAC/D,MAAM,EAAE,CAAC,CAAC;IACV,GAAG,CAAC,EAAE,yBAAyB,GAAG,mBAAmB,CAAC;IACtD,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB,CAAC;AAEF,eAAO,MAAM,WAAW,QAAmB,CAAC;AAG5C,eAAO,MAAM,OAAO,EAAE,yBAAyB,GAAG,mBAAsC,CAAC"}

package/dist/document/transformers/sentence.d.ts

@@ -7,7 +7,6 @@ export declare class SentenceTransformer extends TextTransformer {
     protected sentenceEnders: string[];
     protected fallbackToWords: boolean;
     protected fallbackToCharacters: boolean;
-    protected keepSeparator: boolean | 'start' | 'end';
     constructor(options: SentenceChunkOptions);
     private detectSentenceBoundaries;
     private isRealSentenceBoundary;

package/dist/document/transformers/sentence.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"sentence.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/sentence.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,oBAAoB,EAAE,MAAM,UAAU,CAAC;AACrD,OAAO,EAAE,eAAe,EAAE,MAAM,QAAQ,CAAC;AAEzC,qBAAa,mBAAoB,SAAQ,eAAe;IACtD,SAAS,CAAC,OAAO,EAAE,MAAM,CAAC;IAC1B,SAAS,CAAC,OAAO,EAAE,MAAM,CAAC;IAC1B,SAAS,CAAC,UAAU,EAAE,MAAM,CAAC;IAC7B,SAAS,CAAC,cAAc,EAAE,MAAM,EAAE,CAAC;IACnC,SAAS,CAAC,eAAe,EAAE,OAAO,CAAC;IACnC,SAAS,CAAC,oBAAoB,EAAE,OAAO,CAAC;IACxC,SAAS,CAAC,aAAa,EAAE,OAAO,GAAG,OAAO,GAAG,KAAK,CAAC;gBAEvC,OAAO,EAAE,oBAAoB;IAuBzC,OAAO,CAAC,wBAAwB;IA+BhC,OAAO,CAAC,sBAAsB;IAqB9B,OAAO,CAAC,oBAAoB;IA8B5B;;OAEG;IACH,OAAO,CAAC,wBAAwB;IAsDhC;;OAEG;IACH,OAAO,CAAC,uBAAuB;IAqB/B,OAAO,CAAC,sBAAsB;IAmC9B,OAAO,CAAC,2BAA2B;IAsBnC,OAAO,CAAC,wBAAwB;IA4BhC,OAAO,CAAC,kBAAkB;IAqB1B,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,MAAM,EAAE;CAShD"}
+{"version":3,"file":"sentence.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/sentence.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,oBAAoB,EAAE,MAAM,UAAU,CAAC;AACrD,OAAO,EAAE,eAAe,EAAE,MAAM,QAAQ,CAAC;AAEzC,qBAAa,mBAAoB,SAAQ,eAAe;IACtD,SAAS,CAAC,OAAO,EAAE,MAAM,CAAC;IAC1B,SAAS,CAAC,OAAO,EAAE,MAAM,CAAC;IAC1B,SAAS,CAAC,UAAU,EAAE,MAAM,CAAC;IAC7B,SAAS,CAAC,cAAc,EAAE,MAAM,EAAE,CAAC;IACnC,SAAS,CAAC,eAAe,EAAE,OAAO,CAAC;IACnC,SAAS,CAAC,oBAAoB,EAAE,OAAO,CAAC;gBAE5B,OAAO,EAAE,oBAAoB;IAsBzC,OAAO,CAAC,wBAAwB;IA+BhC,OAAO,CAAC,sBAAsB;IAqB9B,OAAO,CAAC,oBAAoB;IA8B5B;;OAEG;IACH,OAAO,CAAC,wBAAwB;IAsDhC;;OAEG;IACH,OAAO,CAAC,uBAAuB;IAqB/B,OAAO,CAAC,sBAAsB;IAmC9B,OAAO,CAAC,2BAA2B;IAsBnC,OAAO,CAAC,wBAAwB;IA4BhC,OAAO,CAAC,kBAAkB;IAqB1B,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,MAAM,EAAE;CAShD"}

package/dist/document/transformers/text.d.ts

@@ -5,10 +5,10 @@ export declare abstract class TextTransformer implements Transformer {
     protected maxSize: number;
     protected overlap: number;
     protected lengthFunction: (text: string) => number;
-    protected keepSeparator: boolean | 'start' | 'end';
+    protected separatorPosition?: 'start' | 'end';
     protected addStartIndex: boolean;
     protected stripWhitespace: boolean;
-    constructor({ maxSize, overlap, lengthFunction, keepSeparator, addStartIndex, stripWhitespace, }: BaseChunkOptions);
+    constructor({ maxSize, overlap, lengthFunction, separatorPosition, addStartIndex, stripWhitespace, }: BaseChunkOptions);
     setAddStartIndex(value: boolean): void;
     abstract splitText({ text }: {
         text: string;

package/dist/document/transformers/text.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"text.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/text.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAErC,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,eAAe,CAAC;AAEjD,8BAAsB,eAAgB,YAAW,WAAW;IAC1D,SAAS,CAAC,OAAO,EAAE,MAAM,CAAC;IAC1B,SAAS,CAAC,OAAO,EAAE,MAAM,CAAC;IAC1B,SAAS,CAAC,cAAc,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,MAAM,CAAC;IACnD,SAAS,CAAC,aAAa,EAAE,OAAO,GAAG,OAAO,GAAG,KAAK,CAAC;IACnD,SAAS,CAAC,aAAa,EAAE,OAAO,CAAC;IACjC,SAAS,CAAC,eAAe,EAAE,OAAO,CAAC;gBAEvB,EACV,OAAc,EACd,OAAa,EACb,cAA8C,EAC9C,aAAqB,EACrB,aAAqB,EACrB,eAAsB,GACvB,EAAE,gBAAgB;IAYnB,gBAAgB,CAAC,KAAK,EAAE,OAAO,GAAG,IAAI;IAItC,QAAQ,CAAC,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,MAAM,EAAE;IAExD,eAAe,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,GAAG,QAAQ,EAAE;IA4B/E,cAAc,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,QAAQ,EAAE;IAUjD,kBAAkB,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,QAAQ,EAAE;IAYrD,SAAS,CAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,SAAS,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI;IAQpE,SAAS,CAAC,WAAW,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE,SAAS,EAAE,MAAM,GAAG,MAAM,EAAE;CA4DrE"}
+{"version":3,"file":"text.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/text.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAErC,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,eAAe,CAAC;AAEjD,8BAAsB,eAAgB,YAAW,WAAW;IAC1D,SAAS,CAAC,OAAO,EAAE,MAAM,CAAC;IAC1B,SAAS,CAAC,OAAO,EAAE,MAAM,CAAC;IAC1B,SAAS,CAAC,cAAc,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,MAAM,CAAC;IACnD,SAAS,CAAC,iBAAiB,CAAC,EAAE,OAAO,GAAG,KAAK,CAAC;IAC9C,SAAS,CAAC,aAAa,EAAE,OAAO,CAAC;IACjC,SAAS,CAAC,eAAe,EAAE,OAAO,CAAC;gBAEvB,EACV,OAAc,EACd,OAAa,EACb,cAA8C,EAC9C,iBAAiB,EACjB,aAAqB,EACrB,eAAsB,GACvB,EAAE,gBAAgB;IAYnB,gBAAgB,CAAC,KAAK,EAAE,OAAO,GAAG,IAAI;IAItC,QAAQ,CAAC,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,MAAM,EAAE;IAExD,eAAe,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,GAAG,QAAQ,EAAE;IA4B/E,cAAc,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,QAAQ,EAAE;IAUjD,kBAAkB,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,QAAQ,EAAE;IAYrD,SAAS,CAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,SAAS,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI;IAQpE,SAAS,CAAC,WAAW,CAAC,MAAM,EAAE,MAAM,EAAE,EAAE,SAAS,EAAE,MAAM,GAAG,MAAM,EAAE;CA4DrE"}

package/dist/document/types.d.ts

@@ -1,5 +1,5 @@
 import type { TiktokenEncoding, TiktokenModel } from 'js-tiktoken';
-import type { TitleExtractorsArgs, SummaryExtractArgs, QuestionAnswerExtractArgs, KeywordExtractArgs } from './extractors/index.js';
+import type { TitleExtractorsArgs, SummaryExtractArgs, QuestionAnswerExtractArgs, KeywordExtractArgs, SchemaExtractArgs } from './extractors/index.js';
 export declare enum Language {
     CPP = "cpp",
     GO = "go",
@@ -33,12 +33,13 @@ export type ExtractParams = {
     summary?: SummaryExtractArgs | boolean;
     questions?: QuestionAnswerExtractArgs | boolean;
     keywords?: KeywordExtractArgs | boolean;
+    schema?: SchemaExtractArgs;
 };
 export type BaseChunkOptions = {
     maxSize?: number;
     overlap?: number;
     lengthFunction?: (text: string) => number;
-    keepSeparator?: boolean | 'start' | 'end';
+    separatorPosition?: 'start' | 'end';
     addStartIndex?: boolean;
     stripWhitespace?: boolean;
 };

package/dist/document/types.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/document/types.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AACnE,OAAO,KAAK,EACV,mBAAmB,EACnB,kBAAkB,EAClB,yBAAyB,EACzB,kBAAkB,EACnB,MAAM,cAAc,CAAC;AAEtB,oBAAY,QAAQ;IAClB,GAAG,QAAQ;IACX,EAAE,OAAO;IACT,IAAI,SAAS;IACb,MAAM,WAAW;IACjB,EAAE,OAAO;IACT,EAAE,OAAO;IACT,GAAG,QAAQ;IACX,KAAK,UAAU;IACf,MAAM,WAAW;IACjB,GAAG,QAAQ;IACX,IAAI,SAAS;IACb,IAAI,SAAS;IACb,KAAK,UAAU;IACf,KAAK,UAAU;IACf,QAAQ,aAAa;IACrB,KAAK,UAAU;IACf,IAAI,SAAS;IACb,GAAG,QAAQ;IACX,MAAM,WAAW;IACjB,KAAK,UAAU;IACf,CAAC,MAAM;IACP,GAAG,QAAQ;IACX,IAAI,SAAS;IACb,OAAO,YAAY;IACnB,MAAM,WAAW;IACjB,UAAU,eAAe;CAC1B;AAED,MAAM,MAAM,aAAa,GAAG;IAC1B,KAAK,CAAC,EAAE,mBAAmB,GAAG,OAAO,CAAC;IACtC,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC;IACvC,SAAS,CAAC,EAAE,yBAAyB,GAAG,OAAO,CAAC;IAChD,QAAQ,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC;CACzC,CAAC;AAEF,MAAM,MAAM,gBAAgB,GAAG;IAC7B,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,cAAc,CAAC,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,MAAM,CAAC;IAC1C,aAAa,CAAC,EAAE,OAAO,GAAG,OAAO,GAAG,KAAK,CAAC;IAC1C,aAAa,CAAC,EAAE,OAAO,CAAC;IACxB,eAAe,CAAC,EAAE,OAAO,CAAC;CAC3B,CAAC;AAEF,MAAM,MAAM,qBAAqB,GAAG,gBAAgB,GAAG;IACrD,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,gBAAgB,CAAC,EAAE,OAAO,CAAC;CAC5B,CAAC;AAEF,MAAM,MAAM,qBAAqB,GAAG,gBAAgB,GAAG;IACrD,UAAU,CAAC,EAAE,MAAM,EAAE,CAAC;IACtB,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,QAAQ,CAAC,EAAE,QAAQ,CAAC;CACrB,CAAC;AAEF,MAAM,MAAM,iBAAiB,GAAG,gBAAgB,GAAG;IACjD,YAAY,CAAC,EAAE,gBAAgB,CAAC;IAChC,SAAS,CAAC,EAAE,aAAa,CAAC;IAC1B,cAAc,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;IACrC,iBAAiB,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;CACzC,CAAC;AAEF,MAAM,MAAM,oBAAoB,GAAG,gBAAgB,GAAG;IACpD,OAAO,CAAC,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC;IAC7B,cAAc,CAAC,EAAE,OAAO,CAAC;IACzB,YAAY,CAAC,EAAE,OAAO,CAAC;CACxB,CAAC;AAEF,MAAM,MAAM,4BAA4B,GAAG,gBAAgB,GAAG;IAC5D,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,YAAY,CAAC,EAAE,gBAAgB,CAAC;IAChC,SAAS,CAAC,EAAE,aAAa,CAAC;IAC1B,cAAc,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;IACrC,iBAAiB,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;CACzC,CAAC;AAEF,MAAM,MAAM,gBAAgB,GAAG,gBAAgB,GAC7C,CACI;IAAE,OAAO,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC;IAAC,QAAQ,CAAC,EAAE,KAAK,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAA;CAAE,GAC3E;IAAE,QAAQ,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC;IAAC,OAAO,CAAC,EAAE,KAAK,CAAA;CAAE,CACpD,GAAG;IAAE,cAAc,CAAC,EAAE,OAAO,CAAA;CAAE,CAAC;AAEnC,MAAM,MAAM,gBAAgB,GAAG,gBAAgB,GAAG;IAChD,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,YAAY,CAAC,EAAE,OAAO,CAAC;CACxB,CAAC;AAEF,MAAM,MAAM,iBAAiB,GAAG,gBAAgB,GAAG,EAAE,CAAC;AAEtD,MAAM,MAAM,oBAAoB,GAAG,gBAAgB,GAAG;IACpD,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,cAAc,CAAC,EAAE,MAAM,EAAE,CAAC;IAC1B,eAAe,CAAC,EAAE,OAAO,CAAC;IAC1B,oBAAoB,CAAC,EAAE,OAAO,CAAC;CAChC,CAAC;AAEF,MAAM,MAAM,eAAe,GAAG;IAC5B,SAAS,EAAE,qBAAqB,CAAC;IACjC,SAAS,EAAE,qBAAqB,CAAC;IACjC,KAAK,EAAE,iBAAiB,CAAC;IACzB,QAAQ,EAAE,oBAAoB,CAAC;IAC/B,IAAI,EAAE,gBAAgB,CAAC;IACvB,IAAI,EAAE,gBAAgB,CAAC;IACvB,KAAK,EAAE,iBAAiB,CAAC;IACzB,QAAQ,EAAE,oBAAoB,CAAC;IAC/B,mBAAmB,EAAE,4BAA4B,CAAC;CACnD,CAAC;AAEF,MAAM,MAAM,aAAa,GACrB,WAAW,GACX,WAAW,GACX,OAAO,GACP,UAAU,GACV,MAAM,GACN,MAAM,GACN,OAAO,GACP,UAAU,GACV,mBAAmB,CAAC;AAExB,MAAM,MAAM,WAAW,GACnB,CAAC;IAAE,QAAQ,CAAC,EAAE,WAAW,CAAA;CAAE,GAAG,qBAAqB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GAClF,CAAC;IAAE,QAAQ,EAAE,WAAW,CAAA;CAAE,GAAG,qBAAqB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACjF,CAAC;IAAE,QAAQ,EAAE,OAAO,CAAA;CAAE,GAAG,iBAAiB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACzE,CAAC;IAAE,QAAQ,EAAE,UAAU,CAAA;CAAE,GAAG,oBAAoB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GAC/E,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAA;CAAE,GAAG,gBAAgB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACvE,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAA;CAAE,GAAG,gBAAgB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACvE,CAAC;IAAE,QAAQ,EAAE,OAAO,CAAA;CAAE,GAAG,iBAAiB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACzE,CAAC;IAAE,QAAQ,EAAE,UAAU,CAAA;CAAE,GAAG,oBAAoB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GAC/E,CAAC;IAAE,QAAQ,EAAE,mBAAmB,CAAA;CAAE,GAAG,4BAA4B,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,CAAC"}
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/document/types.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AACnE,OAAO,KAAK,EACV,mBAAmB,EACnB,kBAAkB,EAClB,yBAAyB,EACzB,kBAAkB,EAClB,iBAAiB,EAClB,MAAM,cAAc,CAAC;AAEtB,oBAAY,QAAQ;IAClB,GAAG,QAAQ;IACX,EAAE,OAAO;IACT,IAAI,SAAS;IACb,MAAM,WAAW;IACjB,EAAE,OAAO;IACT,EAAE,OAAO;IACT,GAAG,QAAQ;IACX,KAAK,UAAU;IACf,MAAM,WAAW;IACjB,GAAG,QAAQ;IACX,IAAI,SAAS;IACb,IAAI,SAAS;IACb,KAAK,UAAU;IACf,KAAK,UAAU;IACf,QAAQ,aAAa;IACrB,KAAK,UAAU;IACf,IAAI,SAAS;IACb,GAAG,QAAQ;IACX,MAAM,WAAW;IACjB,KAAK,UAAU;IACf,CAAC,MAAM;IACP,GAAG,QAAQ;IACX,IAAI,SAAS;IACb,OAAO,YAAY;IACnB,MAAM,WAAW;IACjB,UAAU,eAAe;CAC1B;AAED,MAAM,MAAM,aAAa,GAAG;IAC1B,KAAK,CAAC,EAAE,mBAAmB,GAAG,OAAO,CAAC;IACtC,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC;IACvC,SAAS,CAAC,EAAE,yBAAyB,GAAG,OAAO,CAAC;IAChD,QAAQ,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC;IACxC,MAAM,CAAC,EAAE,iBAAiB,CAAC;CAC5B,CAAC;AAEF,MAAM,MAAM,gBAAgB,GAAG;IAC7B,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,cAAc,CAAC,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,MAAM,CAAC;IAC1C,iBAAiB,CAAC,EAAE,OAAO,GAAG,KAAK,CAAC;IACpC,aAAa,CAAC,EAAE,OAAO,CAAC;IACxB,eAAe,CAAC,EAAE,OAAO,CAAC;CAC3B,CAAC;AAEF,MAAM,MAAM,qBAAqB,GAAG,gBAAgB,GAAG;IACrD,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,gBAAgB,CAAC,EAAE,OAAO,CAAC;CAC5B,CAAC;AAEF,MAAM,MAAM,qBAAqB,GAAG,gBAAgB,GAAG;IACrD,UAAU,CAAC,EAAE,MAAM,EAAE,CAAC;IACtB,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,QAAQ,CAAC,EAAE,QAAQ,CAAC;CACrB,CAAC;AAEF,MAAM,MAAM,iBAAiB,GAAG,gBAAgB,GAAG;IACjD,YAAY,CAAC,EAAE,gBAAgB,CAAC;IAChC,SAAS,CAAC,EAAE,aAAa,CAAC;IAC1B,cAAc,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;IACrC,iBAAiB,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;CACzC,CAAC;AAEF,MAAM,MAAM,oBAAoB,GAAG,gBAAgB,GAAG;IACpD,OAAO,CAAC,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC;IAC7B,cAAc,CAAC,EAAE,OAAO,CAAC;IACzB,YAAY,CAAC,EAAE,OAAO,CAAC;CACxB,CAAC;AAEF,MAAM,MAAM,4BAA4B,GAAG,gBAAgB,GAAG;IAC5D,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,YAAY,CAAC,EAAE,gBAAgB,CAAC;IAChC,SAAS,CAAC,EAAE,aAAa,CAAC;IAC1B,cAAc,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;IACrC,iBAAiB,CAAC,EAAE,GAAG,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;CACzC,CAAC;AAEF,MAAM,MAAM,gBAAgB,GAAG,gBAAgB,GAC7C,CACI;IAAE,OAAO,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC;IAAC,QAAQ,CAAC,EAAE,KAAK,CAAC;IAAC,cAAc,CAAC,EAAE,OAAO,CAAA;CAAE,GAC3E;IAAE,QAAQ,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC;IAAC,OAAO,CAAC,EAAE,KAAK,CAAA;CAAE,CACpD,GAAG;IAAE,cAAc,CAAC,EAAE,OAAO,CAAA;CAAE,CAAC;AAEnC,MAAM,MAAM,gBAAgB,GAAG,gBAAgB,GAAG;IAChD,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,WAAW,CAAC,EAAE,OAAO,CAAC;IACtB,YAAY,CAAC,EAAE,OAAO,CAAC;CACxB,CAAC;AAEF,MAAM,MAAM,iBAAiB,GAAG,gBAAgB,GAAG,EAAE,CAAC;AAEtD,MAAM,MAAM,oBAAoB,GAAG,gBAAgB,GAAG;IACpD,OAAO,EAAE,MAAM,CAAC;IAChB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,cAAc,CAAC,EAAE,MAAM,EAAE,CAAC;IAC1B,eAAe,CAAC,EAAE,OAAO,CAAC;IAC1B,oBAAoB,CAAC,EAAE,OAAO,CAAC;CAChC,CAAC;AAEF,MAAM,MAAM,eAAe,GAAG;IAC5B,SAAS,EAAE,qBAAqB,CAAC;IACjC,SAAS,EAAE,qBAAqB,CAAC;IACjC,KAAK,EAAE,iBAAiB,CAAC;IACzB,QAAQ,EAAE,oBAAoB,CAAC;IAC/B,IAAI,EAAE,gBAAgB,CAAC;IACvB,IAAI,EAAE,gBAAgB,CAAC;IACvB,KAAK,EAAE,iBAAiB,CAAC;IACzB,QAAQ,EAAE,oBAAoB,CAAC;IAC/B,mBAAmB,EAAE,4BAA4B,CAAC;CACnD,CAAC;AAEF,MAAM,MAAM,aAAa,GACrB,WAAW,GACX,WAAW,GACX,OAAO,GACP,UAAU,GACV,MAAM,GACN,MAAM,GACN,OAAO,GACP,UAAU,GACV,mBAAmB,CAAC;AAExB,MAAM,MAAM,WAAW,GACnB,CAAC;IAAE,QAAQ,CAAC,EAAE,WAAW,CAAA;CAAE,GAAG,qBAAqB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GAClF,CAAC;IAAE,QAAQ,EAAE,WAAW,CAAA;CAAE,GAAG,qBAAqB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACjF,CAAC;IAAE,QAAQ,EAAE,OAAO,CAAA;CAAE,GAAG,iBAAiB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACzE,CAAC;IAAE,QAAQ,EAAE,UAAU,CAAA;CAAE,GAAG,oBAAoB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GAC/E,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAA;CAAE,GAAG,gBAAgB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACvE,CAAC;IAAE,QAAQ,EAAE,MAAM,CAAA;CAAE,GAAG,gBAAgB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACvE,CAAC;IAAE,QAAQ,EAAE,OAAO,CAAA;CAAE,GAAG,iBAAiB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GACzE,CAAC;IAAE,QAAQ,EAAE,UAAU,CAAA;CAAE,GAAG,oBAAoB,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,GAC/E,CAAC;IAAE,QAAQ,EAAE,mBAAmB,CAAA;CAAE,GAAG,4BAA4B,GAAG;IAAE,OAAO,CAAC,EAAE,aAAa,CAAA;CAAE,CAAC,CAAC"}

package/dist/index.cjs

@@ -4450,6 +4450,44 @@ var KeywordExtractor = class extends BaseExtractor {
     return results;
   }
 };
+var SchemaExtractor = class extends BaseExtractor {
+  schema;
+  llm;
+  instructions;
+  metadataKey;
+  constructor(options) {
+    super();
+    this.schema = options.schema;
+    this.llm = options.llm;
+    this.instructions = options.instructions;
+    this.metadataKey = options.metadataKey;
+  }
+  async extract(nodes) {
+    const agent$1 = new agent.Agent({
+      name: "schema-extractor",
+      id: "schema-extractor",
+      instructions: this.instructions ?? "Extract structured data from the provided text.",
+      model: this.llm ?? baseLLM
+    });
+    const results = await Promise.all(
+      nodes.map(async (node) => {
+        try {
+          const result = await agent$1.generate([{ role: "user", content: node.getContent() }], {
+            structuredOutput: { schema: this.schema }
+          });
+          if (this.metadataKey) {
+            return { [this.metadataKey]: result.object };
+          }
+          return result.object;
+        } catch (error) {
+          console.error("Schema extraction failed:", error);
+          return {};
+        }
+      })
+    );
+    return results;
+  }
+};
 
 // src/document/types.ts
 var Language = /* @__PURE__ */ ((Language2) => {
@@ -4487,14 +4525,14 @@ var TextTransformer = class {
   maxSize;
   overlap;
   lengthFunction;
-  keepSeparator;
+  separatorPosition;
   addStartIndex;
   stripWhitespace;
   constructor({
     maxSize = 4e3,
     overlap = 200,
     lengthFunction = (text) => text.length,
-    keepSeparator = false,
+    separatorPosition,
     addStartIndex = false,
     stripWhitespace = true
   }) {
@@ -4504,7 +4542,7 @@ var TextTransformer = class {
     this.maxSize = maxSize;
     this.overlap = overlap;
     this.lengthFunction = lengthFunction;
-    this.keepSeparator = keepSeparator;
+    this.separatorPosition = separatorPosition;
     this.addStartIndex = addStartIndex;
     this.stripWhitespace = stripWhitespace;
   }
@@ -4610,11 +4648,11 @@ var TextTransformer = class {
 };
 
 // src/document/transformers/character.ts
-function splitTextWithRegex(text, separator, keepSeparator) {
+function splitTextWithRegex(text, separator, separatorPosition) {
   if (!separator) {
     return text.split("");
   }
-  if (!keepSeparator) {
+  if (!separatorPosition) {
     return text.split(new RegExp(separator)).filter((s) => s !== "");
   }
   if (!text) {
@@ -4622,7 +4660,7 @@ function splitTextWithRegex(text, separator, keepSeparator) {
   }
   const splits = text.split(new RegExp(`(${separator})`));
   const result = [];
-  if (keepSeparator === "end") {
+  if (separatorPosition === "end") {
     for (let i = 0; i < splits.length - 1; i += 2) {
       if (i + 1 < splits.length) {
         const chunk = splits[i] + (splits[i + 1] || "");
@@ -4654,7 +4692,7 @@ var CharacterTransformer = class extends TextTransformer {
   }
   splitText({ text }) {
     const separator = this.isSeparatorRegex ? this.separator : this.separator.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
-    const initialSplits = splitTextWithRegex(text, separator, this.keepSeparator);
+    const initialSplits = splitTextWithRegex(text, separator, this.separatorPosition);
     const chunks = [];
     for (const split of initialSplits) {
       if (this.lengthFunction(split) <= this.maxSize) {
@@ -4709,9 +4747,9 @@ var RecursiveCharacterTransformer = class _RecursiveCharacterTransformer extends
       }
     }
     const _separator = this.isSeparatorRegex ? separator : separator?.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
-    const splits = splitTextWithRegex(text, _separator, this.keepSeparator);
+    const splits = splitTextWithRegex(text, _separator, this.separatorPosition);
     const goodSplits = [];
-    const mergeSeparator = this.keepSeparator ? "" : separator;
+    const mergeSeparator = this.separatorPosition ? "" : separator;
     for (const s of splits) {
       if (this.lengthFunction(s) < this.maxSize) {
         goodSplits.push(s);
@@ -5857,7 +5895,6 @@ var SentenceTransformer = class extends TextTransformer {
   sentenceEnders;
   fallbackToWords;
   fallbackToCharacters;
-  keepSeparator;
   constructor(options) {
     const parentOverlap = Math.min(options.overlap ?? 0, options.maxSize - 1);
     const baseOptions = {
@@ -5872,7 +5909,6 @@ var SentenceTransformer = class extends TextTransformer {
     this.sentenceEnders = options.sentenceEnders ?? [".", "!", "?"];
     this.fallbackToWords = options.fallbackToWords ?? true;
     this.fallbackToCharacters = options.fallbackToCharacters ?? true;
-    this.keepSeparator = options.keepSeparator ?? false;
     this.overlap = options.overlap ?? 0;
   }
   detectSentenceBoundaries(text) {
@@ -6173,8 +6209,8 @@ var TokenTransformer = class _TokenTransformer extends TextTransformer {
 var baseChunkOptionsSchema = zod.z.object({
   maxSize: zod.z.number().positive().optional(),
   overlap: zod.z.number().min(0).optional(),
-  lengthFunction: zod.z.function().optional(),
-  keepSeparator: zod.z.union([zod.z.boolean(), zod.z.literal("start"), zod.z.literal("end")]).optional(),
+  lengthFunction: zod.z.optional(zod.z.function()),
+  separatorPosition: zod.z.enum(["start", "end"]).optional(),
   addStartIndex: zod.z.boolean().optional(),
   stripWhitespace: zod.z.boolean().optional()
 });
@@ -6269,8 +6305,11 @@ var MDocument = class _MDocument {
     });
     this.type = type;
   }
-  async extractMetadata({ title, summary, questions, keywords }) {
+  async extractMetadata({ title, summary, questions, keywords, schema }) {
     const transformations = [];
+    if (schema) {
+      transformations.push(new SchemaExtractor(schema));
+    }
     if (typeof summary !== "undefined") {
       transformations.push(new SummaryExtractor(typeof summary === "boolean" ? {} : summary));
     }
@@ -6421,7 +6460,7 @@ var MDocument = class _MDocument {
         const textSplitter = new RecursiveCharacterTransformer({
           maxSize: options.maxSize,
           overlap: options.overlap,
-          keepSeparator: options.keepSeparator,
+          separatorPosition: options.separatorPosition,
           addStartIndex: options.addStartIndex,
           stripWhitespace: options.stripWhitespace
         });
@@ -6437,7 +6476,7 @@ var MDocument = class _MDocument {
         const textSplitter = new RecursiveCharacterTransformer({
           maxSize: options.maxSize,
           overlap: options.overlap,
-          keepSeparator: options.keepSeparator,
+          separatorPosition: options.separatorPosition,
           addStartIndex: options.addStartIndex,
           stripWhitespace: options.stripWhitespace
         });
@@ -6500,7 +6539,7 @@ var MDocument = class _MDocument {
       sentenceEnders: options?.sentenceEnders,
       fallbackToWords: options?.fallbackToWords,
       fallbackToCharacters: options?.fallbackToCharacters,
-      keepSeparator: options?.keepSeparator,
+      separatorPosition: options?.separatorPosition,
       lengthFunction: options?.lengthFunction,
       addStartIndex: options?.addStartIndex,
       stripWhitespace: options?.stripWhitespace