@mastra/rag 2.7.1 → 2.7.2

package/dist/docs/SKILL.md

@@ -3,7 +3,7 @@ name: mastra-rag
 description: Documentation for @mastra/rag. Use when working with @mastra/rag APIs, configuration, or implementation.
 metadata:
   package: "@mastra/rag"
-  version: "2.7.1"
+  version: "2.7.2"
 ---
 
 ## When to use
@@ -36,7 +36,6 @@ Read the individual reference documents for detailed explanations and code examp
 - [Reference: createDocumentChunkerTool()](references/reference-tools-document-chunker-tool.md) - The createDocumentChunkerTool() function creates a tool for splitting documents into smaller chunks for efficient processing and retrieval.
 - [Reference: createGraphRAGTool()](references/reference-tools-graph-rag-tool.md) - Use createGraphRAGTool() to build semantic relationships between document chunks and retrieve connected context for graph-based RAG queries.
 - [Reference: createVectorQueryTool()](references/reference-tools-vector-query-tool.md) - The createVectorQueryTool() function creates a tool for semantic search over vector stores.
-- [Reference: OracleDB vector store](references/reference-vectors-oracledb.md) - OracleVector stores embeddings in Oracle Database VECTOR columns and exposes them through Mastra's vector interface.
 
 
 Read [assets/SOURCE_MAP.json](assets/SOURCE_MAP.json) for source code references.

package/dist/docs/assets/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "2.7.1",
+  "version": "2.7.2",
   "package": "@mastra/rag",
   "exports": {},
   "modules": {}

package/dist/docs/references/docs-guides-context-engineering.md

@@ -145,8 +145,8 @@ export const workspace = new Workspace({
   autoIndexPaths: ['**/*.md'],
 })
 
-// Agents receive tools including read_file, list_files, grep,
-// mastra_workspace_search, and mastra_workspace_index.
+// Agents receive tools including mastra_workspace_read_file, mastra_workspace_list_files,
+// mastra_workspace_grep, mastra_workspace_search, and mastra_workspace_index.
 await workspace.init()
 ```
 

package/dist/docs/references/reference-rag-chunk.md

@@ -202,7 +202,7 @@ When using the `headers` option, the markdown strategy ignores all general optio
 
 **maxSize** (`number`): Maximum size of each chunk
 
-**minSize** (`number`): Minimum size of each chunk
+**minSize** (`number`): Accepted for compatibility. Has no effect on JSON chunking: chunks smaller than this are kept, so no data is discarded.
 
 **ensureAscii** (`boolean`): Whether to ensure ASCII encoding
 

package/dist/docs/references/reference-rag-retrieval.md

@@ -251,7 +251,11 @@ const results = await pgVector.query({
 })
 ```
 
-Postgres vector indexes (HNSW and IVFFlat) can't restrict that search to rows matching a condition. When a filter is present, pgVector instead compares the query vector against every matching row and returns the closest `topK`. Results are exact, but the work grows with the number of rows the filter matches. Indexing the metadata column speeds up row retrieval. The distance comparisons still happen per row.
+With an HNSW index on pgvector 0.8.0 or newer, PostgreSQL searches the index and skips rows that don't match the filter. When the index search can't fill `topK`, pgVector reruns the query as an exact search.
+
+With IVFFlat, or HNSW on older pgvector versions, pgVector compares the query vector against every matching row. Results are exact, but the work grows with the number of matching rows.
+
+Indexing the metadata fields you filter on speeds up both cases and gives PostgreSQL the row counts it needs to choose between them.
 
 ### Vector Query Tool
 

package/dist/document/transformers/character.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"character.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/character.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,KAAK,EAAE,gBAAgB,EAAE,qBAAqB,EAAE,qBAAqB,EAAE,MAAM,UAAU,CAAC;AAE/F,OAAO,EAAE,eAAe,EAAE,MAAM,QAAQ,CAAC;AA6EzC,qBAAa,oBAAqB,SAAQ,eAAe;IACvD,SAAS,CAAC,SAAS,EAAE,MAAM,CAAC;IAC5B,SAAS,CAAC,gBAAgB,EAAE,OAAO,CAAC;IAEpC,YAAY,EAAE,SAAkB,EAAE,gBAAwB,EAAE,GAAG,WAAW,EAAE,GAAE,qBAA0B,EAIvG;IAED,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,MAAM,EAAE,CAmB9C;IAED,OAAO,CAAC,YAAY;CAyDrB;AAED,qBAAa,6BAA8B,SAAQ,eAAe;IAChE,SAAS,CAAC,UAAU,EAAE,MAAM,EAAE,CAAC;IAC/B,SAAS,CAAC,gBAAgB,EAAE,OAAO,CAAC;IAEpC,YAAY,EAAE,UAAU,EAAE,gBAAwB,EAAE,QAAQ,EAAE,GAAG,WAAW,EAAE,GAAE,qBAA0B,EAIzG;IAED,OAAO,CAAC,UAAU;IA8DlB,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,MAAM,EAAE,CAE9C;IAED,MAAM,CAAC,YAAY,CAAC,QAAQ,EAAE,QAAQ,EAAE,OAAO,GAAE,gBAAqB,GAAG,6BAA6B,CAQrG;IAED,MAAM,CAAC,wBAAwB,CAAC,QAAQ,EAAE,QAAQ,GAAG,MAAM,EAAE,CAua5D;CACF"}
+{"version":3,"file":"character.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/character.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,UAAU,CAAC;AACpC,OAAO,KAAK,EAAE,gBAAgB,EAAE,qBAAqB,EAAE,qBAAqB,EAAE,MAAM,UAAU,CAAC;AAE/F,OAAO,EAAE,eAAe,EAAE,MAAM,QAAQ,CAAC;AAqEzC,qBAAa,oBAAqB,SAAQ,eAAe;IACvD,SAAS,CAAC,SAAS,EAAE,MAAM,CAAC;IAC5B,SAAS,CAAC,gBAAgB,EAAE,OAAO,CAAC;IAEpC,YAAY,EAAE,SAAkB,EAAE,gBAAwB,EAAE,GAAG,WAAW,EAAE,GAAE,qBAA0B,EAIvG;IAED,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,MAAM,EAAE,CAmB9C;IAED,OAAO,CAAC,YAAY;CAyDrB;AAED,qBAAa,6BAA8B,SAAQ,eAAe;IAChE,SAAS,CAAC,UAAU,EAAE,MAAM,EAAE,CAAC;IAC/B,SAAS,CAAC,gBAAgB,EAAE,OAAO,CAAC;IAEpC,YAAY,EAAE,UAAU,EAAE,gBAAwB,EAAE,QAAQ,EAAE,GAAG,WAAW,EAAE,GAAE,qBAA0B,EAIzG;IAED,OAAO,CAAC,UAAU;IA8DlB,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,MAAM,EAAE,CAE9C;IAED,MAAM,CAAC,YAAY,CAAC,QAAQ,EAAE,QAAQ,EAAE,OAAO,GAAE,gBAAqB,GAAG,6BAA6B,CAQrG;IAED,MAAM,CAAC,wBAAwB,CAAC,QAAQ,EAAE,QAAQ,GAAG,MAAM,EAAE,CAua5D;CACF"}

package/dist/document/transformers/json.d.ts

@@ -2,10 +2,9 @@ import { Document } from '../schema/index.js';
 import type { JsonChunkOptions } from '../types.js';
 export declare class RecursiveJsonTransformer {
     private maxSize;
-    private minSize;
     private ensureAscii;
     private convertLists;
-    constructor({ maxSize, minSize, ensureAscii, convertLists }: JsonChunkOptions);
+    constructor({ maxSize, ensureAscii, convertLists }: JsonChunkOptions);
     private static jsonSize;
     /**
      * Transform JSON data while handling circular references
@@ -60,11 +59,6 @@ export declare class RecursiveJsonTransformer {
         jsonData: Record<string, any>;
         convertLists?: boolean;
     }): Record<string, any>[];
-    /**
-     * Converts Unicode characters to their escaped ASCII representation
-     * e.g., 'café' becomes 'caf\u00e9'
-     */
-    private escapeNonAscii;
     /**
      * Splits JSON into a list of JSON formatted strings
      */

package/dist/document/transformers/json.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"json.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/json.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AACrC,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,qBAAa,wBAAwB;IACnC,OAAO,CAAC,OAAO,CAAS;IACxB,OAAO,CAAC,OAAO,CAAS;IACxB,OAAO,CAAC,WAAW,CAAU;IAC7B,OAAO,CAAC,YAAY,CAAU;IAE9B,YAAY,EAAE,OAAc,EAAE,OAAO,EAAE,WAAmB,EAAE,YAAmB,EAAE,EAAE,gBAAgB,EAKlG;IAED,OAAO,CAAC,MAAM,CAAC,QAAQ;IAoCvB;;OAEG;IACI,SAAS,CAAC,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAiC/D;IAED;;OAEG;IACH,OAAO,CAAC,MAAM,CAAC,aAAa;IAS5B;;OAEG;IACH,OAAO,CAAC,uBAAuB;IAU/B;;;OAGG;IACH,OAAO,CAAC,oBAAoB;IAyC5B;;;OAGG;IACH,OAAO,CAAC,WAAW;IAMnB;;OAEG;IACH,OAAO,CAAC,iBAAiB;IAOzB;;;OAGG;IACH,OAAO,CAAC,WAAW;IA8DnB;;;OAGG;IACH,OAAO,CAAC,kBAAkB;IAmE1B;;;OAGG;IACH,OAAO,CAAC,eAAe;IAuBvB;;;OAGG;IACH,OAAO,CAAC,SAAS;IAuDjB;;OAEG;IACH,SAAS,CAAC,EACR,QAAQ,EACR,YAAoB,GACrB,EAAE;QACD,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;QAC9B,YAAY,CAAC,EAAE,OAAO,CAAC;KACxB,GAAG,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAUxB;IAED;;;OAGG;IACH,OAAO,CAAC,cAAc;IAiBtB;;OAEG;IACH,SAAS,CAAC,EACR,QAAQ,EACR,YAAoB,EACpB,WAAkB,GACnB,EAAE;QACD,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;QAC9B,YAAY,CAAC,EAAE,OAAO,CAAC;QACvB,WAAW,CAAC,EAAE,OAAO,CAAC;KACvB,GAAG,MAAM,EAAE,CAkBX;IAED;;OAEG;IACH,eAAe,CAAC,EACd,KAAK,EACL,YAAoB,EACpB,WAAkB,EAClB,SAAS,GACV,EAAE;QACD,KAAK,EAAE,MAAM,EAAE,CAAC;QAChB,YAAY,CAAC,EAAE,OAAO,CAAC;QACvB,WAAW,CAAC,EAAE,OAAO,CAAC;QACtB,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAAC;KACnC,GAAG,QAAQ,EAAE,CAkBb;IAED,kBAAkB,CAAC,EACjB,WAAW,EACX,SAAS,EACT,YAAY,GACb,EAAE;QACD,WAAW,CAAC,EAAE,OAAO,CAAC;QACtB,YAAY,CAAC,EAAE,OAAO,CAAC;QACvB,SAAS,EAAE,QAAQ,EAAE,CAAC;KACvB,GAAG,QAAQ,EAAE,CAgBb;CACF"}
+{"version":3,"file":"json.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/json.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AACrC,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,qBAAa,wBAAwB;IACnC,OAAO,CAAC,OAAO,CAAS;IACxB,OAAO,CAAC,WAAW,CAAU;IAC7B,OAAO,CAAC,YAAY,CAAU;IAE9B,YAAY,EAAE,OAAc,EAAE,WAAmB,EAAE,YAAmB,EAAE,EAAE,gBAAgB,EAIzF;IAED,OAAO,CAAC,MAAM,CAAC,QAAQ;IAoCvB;;OAEG;IACI,SAAS,CAAC,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAiC/D;IAED;;OAEG;IACH,OAAO,CAAC,MAAM,CAAC,aAAa;IAS5B;;OAEG;IACH,OAAO,CAAC,uBAAuB;IAU/B;;;OAGG;IACH,OAAO,CAAC,oBAAoB;IAyC5B;;;OAGG;IACH,OAAO,CAAC,WAAW;IAMnB;;OAEG;IACH,OAAO,CAAC,iBAAiB;IAOzB;;;OAGG;IACH,OAAO,CAAC,WAAW;IA2DnB;;;OAGG;IACH,OAAO,CAAC,kBAAkB;IAgE1B;;;OAGG;IACH,OAAO,CAAC,eAAe;IAuBvB;;;OAGG;IACH,OAAO,CAAC,SAAS;IAuDjB;;OAEG;IACH,SAAS,CAAC,EACR,QAAQ,EACR,YAAoB,GACrB,EAAE;QACD,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;QAC9B,YAAY,CAAC,EAAE,OAAO,CAAC;KACxB,GAAG,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAUxB;IAED;;OAEG;IACH,SAAS,CAAC,EACR,QAAQ,EACR,YAAoB,EACpB,WAAkB,GACnB,EAAE;QACD,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;QAC9B,YAAY,CAAC,EAAE,OAAO,CAAC;QACvB,WAAW,CAAC,EAAE,OAAO,CAAC;KACvB,GAAG,MAAM,EAAE,CAcX;IAED;;OAEG;IACH,eAAe,CAAC,EACd,KAAK,EACL,YAAoB,EACpB,WAAkB,EAClB,SAAS,GACV,EAAE;QACD,KAAK,EAAE,MAAM,EAAE,CAAC;QAChB,YAAY,CAAC,EAAE,OAAO,CAAC;QACvB,WAAW,CAAC,EAAE,OAAO,CAAC;QACtB,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,CAAC;KACnC,GAAG,QAAQ,EAAE,CAkBb;IAED,kBAAkB,CAAC,EACjB,WAAW,EACX,SAAS,EACT,YAAY,GACb,EAAE;QACD,WAAW,CAAC,EAAE,OAAO,CAAC;QACtB,YAAY,CAAC,EAAE,OAAO,CAAC;QACvB,SAAS,EAAE,QAAQ,EAAE,CAAC;KACvB,GAAG,QAAQ,EAAE,CAgBb;CACF"}

package/dist/document/transformers/markdown.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"markdown.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/markdown.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAGrC,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,OAAO,EAAE,6BAA6B,EAAE,MAAM,aAAa,CAAC;AAa5D,qBAAa,mBAAoB,SAAQ,6BAA6B;IACpE,YAAY,OAAO,GAAE,gBAAqB,EAGzC;CACF;AAED,qBAAa,yBAAyB;IACpC,OAAO,CAAC,gBAAgB,CAAqB;IAC7C,OAAO,CAAC,cAAc,CAAU;IAChC,OAAO,CAAC,YAAY,CAAU;IAE9B,YAAY,gBAAgB,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,EAAE,cAAc,GAAE,OAAe,EAAE,YAAY,GAAE,OAAc,EAI9G;IAED,OAAO,CAAC,sBAAsB;IAuD9B,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,QAAQ,EAAE,CAkIhD;IAED,eAAe,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,GAAG,QAAQ,EAAE,CAiB9E;IAED,kBAAkB,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,QAAQ,EAAE,CAUpD;CACF"}
+{"version":3,"file":"markdown.d.ts","sourceRoot":"","sources":["../../../src/document/transformers/markdown.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAGrC,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,UAAU,CAAC;AAEjD,OAAO,EAAE,6BAA6B,EAAE,MAAM,aAAa,CAAC;AAa5D,qBAAa,mBAAoB,SAAQ,6BAA6B;IACpE,YAAY,OAAO,GAAE,gBAAqB,EAGzC;CACF;AAED,qBAAa,yBAAyB;IACpC,OAAO,CAAC,gBAAgB,CAAqB;IAC7C,OAAO,CAAC,cAAc,CAAU;IAChC,OAAO,CAAC,YAAY,CAAU;IAE9B,YAAY,gBAAgB,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,EAAE,EAAE,cAAc,GAAE,OAAe,EAAE,YAAY,GAAE,OAAc,EAI9G;IAED,OAAO,CAAC,sBAAsB;IAuD9B,SAAS,CAAC,EAAE,IAAI,EAAE,EAAE;QAAE,IAAI,EAAE,MAAM,CAAA;KAAE,GAAG,QAAQ,EAAE,CAsIhD;IAED,eAAe,CAAC,KAAK,EAAE,MAAM,EAAE,EAAE,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,EAAE,GAAG,QAAQ,EAAE,CAiB9E;IAED,kBAAkB,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,QAAQ,EAAE,CAUpD;CACF"}

package/dist/index.cjs

@@ -4009,24 +4009,22 @@ var TextTransformer = class {
 // src/document/transformers/character.ts
 function splitTextWithRegex(text, separator, separatorPosition) {
 	if (!separator) return text.split("");
-	if (!separatorPosition) return text.split(new RegExp(separator)).filter((s) => s !== "");
 	if (!text) return [];
-	const splits = text.split(new RegExp(`(${separator})`));
 	const result = [];
-	if (separatorPosition === "end") {
-		for (let i = 0; i < splits.length - 1; i += 2) if (i + 1 < splits.length) {
-			const chunk = splits[i] + (splits[i + 1] || "");
-			if (chunk) result.push(chunk);
-		}
-		if (splits.length % 2 === 1 && splits[splits.length - 1]) result.push(splits?.[splits.length - 1]);
-	} else {
-		if (splits[0]) result.push(splits[0]);
-		for (let i = 1; i < splits.length - 1; i += 2) {
-			const separator2 = splits[i];
-			const text2 = splits[i + 1] ?? "";
-			if (separator2) result.push(separator2 + text2);
-		}
-	}
+	let lastIndex = 0;
+	let pendingSeparator = "";
+	for (const match of text.matchAll(new RegExp(separator, "g"))) {
+		const matched = match[0];
+		const piece = text.slice(lastIndex, match.index);
+		if (separatorPosition === "end") result.push(piece + matched);
+		else if (separatorPosition === "start") {
+			result.push(pendingSeparator + piece);
+			pendingSeparator = matched;
+		} else result.push(piece);
+		lastIndex = match.index + matched.length;
+	}
+	const tail = text.slice(lastIndex);
+	result.push(separatorPosition === "start" ? pendingSeparator + tail : tail);
 	return result.filter((s) => s !== "");
 }
 function splitTextWithMatchedSeparators(text, separator) {
@@ -4848,12 +4846,10 @@ var HTMLSectionTransformer = class {
 // src/document/transformers/json.ts
 var RecursiveJsonTransformer = class RecursiveJsonTransformer {
 	maxSize;
-	minSize;
 	ensureAscii;
 	convertLists;
-	constructor({ maxSize = 2e3, minSize, ensureAscii = false, convertLists = true }) {
+	constructor({ maxSize = 2e3, ensureAscii = false, convertLists = true }) {
 		this.maxSize = maxSize;
-		this.minSize = minSize ?? Math.max(maxSize - 200, 50);
 		this.ensureAscii = ensureAscii;
 		this.convertLists = convertLists;
 	}
@@ -4980,14 +4976,11 @@ var RecursiveJsonTransformer = class RecursiveJsonTransformer {
 		let currentGroup = [];
 		const saveCurrentGroup = () => {
 			if (currentGroup.length > 0) {
-				const groupChunk = this.createChunk(currentGroup, path);
-				if (RecursiveJsonTransformer.jsonSize(groupChunk) >= this.minSize) {
-					chunks.push(groupChunk);
-					currentGroup = [];
-				}
+				chunks.push(this.createChunk(currentGroup, path));
+				currentGroup = [];
 			}
 		};
-		for (const item of value) {
+		for (const [itemIndex, item] of value.entries()) {
 			const testGroup = [...currentGroup, item];
 			const testChunk = this.createChunk(testGroup, path);
 			if (this.isWithinSizeLimit(testChunk)) {
@@ -5000,7 +4993,7 @@ var RecursiveJsonTransformer = class RecursiveJsonTransformer {
 				const singleItemChunk = this.createChunk(singleItemArray, path);
 				if (this.isWithinSizeLimit(singleItemChunk)) currentGroup = singleItemArray;
 				else {
-					const itemPath = [...path, String(chunks.length)];
+					const itemPath = [...path, String(itemIndex)];
 					const nestedChunks = this.handleNestedObject(item, itemPath, depth + 1, maxDepth);
 					chunks.push(...nestedChunks);
 				}
@@ -5025,11 +5018,8 @@ var RecursiveJsonTransformer = class RecursiveJsonTransformer {
 		let currentChunk = {};
 		const saveCurrentChunk = () => {
 			if (Object.keys(currentChunk).length > 0) {
-				const objChunk = this.createChunk(currentChunk, path);
-				if (RecursiveJsonTransformer.jsonSize(objChunk) >= this.minSize) {
-					chunks.push(objChunk);
-					currentChunk = {};
-				}
+				chunks.push(this.createChunk(currentChunk, path));
+				currentChunk = {};
 			}
 		};
 		for (const [key, val] of Object.entries(value)) {
@@ -5117,30 +5107,15 @@ var RecursiveJsonTransformer = class RecursiveJsonTransformer {
 		return chunks;
 	}
 	/**
-	* Converts Unicode characters to their escaped ASCII representation
-	* e.g., 'café' becomes 'caf\u00e9'
-	*/
-	escapeNonAscii(obj) {
-		if (typeof obj === "string") return obj.replace(/[\u0080-\uffff]/g, (char) => {
-			return `\\u${char.charCodeAt(0).toString(16).padStart(4, "0")}`;
-		});
-		if (Array.isArray(obj)) return obj.map((item) => this.escapeNonAscii(item));
-		if (typeof obj === "object" && obj !== null) return Object.fromEntries(Object.entries(obj).map(([key, value]) => [key, this.escapeNonAscii(value)]));
-		return obj;
-	}
-	/**
 	* Splits JSON into a list of JSON formatted strings
 	*/
 	splitText({ jsonData, convertLists = false, ensureAscii = true }) {
-		const chunks = this.splitJson({
+		const serialized = this.splitJson({
 			jsonData,
 			convertLists
-		});
-		if (ensureAscii) return chunks.map((chunk) => this.escapeNonAscii(chunk)).map((chunk) => JSON.stringify(chunk));
-		return chunks.map((chunk) => JSON.stringify(chunk, (key, value) => {
-			if (typeof value === "string") return value.replace(/\\u[\da-f]{4}/gi, (match) => String.fromCharCode(parseInt(match.slice(2), 16)));
-			return value;
-		}));
+		}).map((chunk) => JSON.stringify(chunk));
+		if (!ensureAscii) return serialized;
+		return serialized.map((text) => text.replace(/[\u0080-\uffff]/g, (char) => `\\u${char.charCodeAt(0).toString(16).padStart(4, "0")}`));
 	}
 	/**
 	* Create documents from a list of json objects
@@ -5252,11 +5227,12 @@ var MarkdownHeaderTransformer = class {
 			const line = lines[i];
 			const strippedLine = line.trim();
 			if (!inCodeBlock) {
-				if (strippedLine.startsWith("```") && strippedLine.split("```").length === 2 || strippedLine.startsWith("~~~")) {
+				const fence = strippedLine.match(/^(`{3,}|~{3,})/);
+				if (fence && (fence[1][0] !== "`" || !strippedLine.slice(fence[1].length).includes("`"))) {
 					inCodeBlock = true;
-					openingFence = strippedLine.startsWith("```") ? "```" : "~~~";
+					openingFence = fence[1];
 				}
-			} else if (strippedLine.startsWith(openingFence)) {
+			} else if (strippedLine.startsWith(openingFence) && new RegExp(`^${openingFence[0]}{${openingFence.length},}\\s*$`).test(strippedLine)) {
 				inCodeBlock = false;
 				openingFence = "";
 			}
@@ -5264,13 +5240,9 @@ var MarkdownHeaderTransformer = class {
 				currentContent.push(line);
 				continue;
 			}
-			if (strippedLine.includes("|") && strippedLine.length > 0) {
-				if (!inTable) inTable = true;
-				currentContent.push(line);
-				continue;
-			} else if (inTable) inTable = false;
+			const isIndentedCode = strippedLine.includes("|") && /^(?: {4}|\t)/.test(line);
 			let headerMatched = false;
-			for (const [sep, name] of this.headersToSplitOn) if (strippedLine.startsWith(sep) && (strippedLine.length === sep.length || strippedLine[sep.length] === " ")) {
+			for (const [sep, name] of this.headersToSplitOn) if (!isIndentedCode && strippedLine.startsWith(sep) && (strippedLine.length === sep.length || strippedLine[sep.length] === " ")) {
 				headerMatched = true;
 				if (currentContent.length > 0) {
 					linesWithMetadata.push({
@@ -5302,7 +5274,14 @@ var MarkdownHeaderTransformer = class {
 				});
 				break;
 			}
-			if (!headerMatched) {
+			if (headerMatched) inTable = false;
+			else {
+				if (strippedLine.includes("|")) {
+					inTable = true;
+					currentContent.push(line);
+					continue;
+				}
+				if (inTable) inTable = false;
 				if (strippedLine || this.returnEachLine) {
 					currentContent.push(line);
 					if (this.returnEachLine) {