@convex-dev/rag 0.3.1 → 0.3.3-alpha.0

package/README.md

@@ -1,12 +1,14 @@
 # Convex RAG Component
 
-[![npm version](https://badge.fury.io/js/@convex-dev%2Fmemory.svg)](https://badge.fury.io/js/@convex-dev%2Fmemory)
+[![npm version](https://badge.fury.io/js/@convex-dev%2Frag.svg)](https://badge.fury.io/js/@convex-dev%2Frag)
 
 <!-- START: Include on https://convex.dev/components -->
 
 A component for semantic search, usually used to look up context for LLMs.
 Use with an Agent for Retrieval-Augmented Generation (RAG).
 
+[![Use AI to search HUGE amounts of text with the RAG Component](https://thumbs.video-to-markdown.com/1ff18153.jpg)](https://youtu.be/dGmtAmdAaFs)
+
 ## ✨ Key Features
 
 - **Add Content**: Add or replace content with text chunks and embeddings.
@@ -57,23 +59,13 @@ import { RAG } from "@convex-dev/rag";
 // Any AI SDK model that supports embeddings will work.
 import { openai } from "@ai-sdk/openai";
 
-const rag = new RAG<FilterTypes>(components.rag, {
-  filterNames: ["category", "contentType", "categoryAndType"],
+const rag = new RAG(components.rag, {
   textEmbeddingModel: openai.embedding("text-embedding-3-small"),
-  embeddingDimension: 1536,
+  embeddingDimension: 1536, // Needs to match your embedding model
 });
-
-// Optional: Add type safety to your filters.
-type FilterTypes = {
-  category: string;
-  contentType: string;
-  categoryAndType: { category: string; contentType: string };
-};
 ```
 
-## Usage Examples
-
-### Add context to RAG
+## Add context to RAG
 
 Add content with text chunks. Each call to `add` will create a new **entry**.
 It will embed the chunks automatically if you don't provide them.
@@ -91,52 +83,9 @@ export const add = action({
 });
 ```
 
-See below for how to add content asynchronously, e.g. to handle large files.
-
-### Generate a response based on RAG context
-
-You can use the `generateText` function to generate a response based on RAG context. This will automatically search for relevant entries and use them as context for the LLM, using default formatting.
-
-The arguments to `generateText` are compatible with all arguments to `generateText` from the AI SDK.
-
-To have more control over the context and prompting, you can use the `search` function to get the context, and then use any model to generate a response.
-See below for more details.
-
-```ts
-export const askQuestion = action({
-  args: {
-    prompt: v.string(),
-  },
-  handler: async (ctx, args) => {
-    const userId = await getAuthUserId(ctx);
-    const { text, context } = await rag.generateText(ctx, {
-      search: { namespace: userId, limit: 10 },
-      prompt: args.prompt,
-      model: openai.chat("gpt-4o-mini"),
-    });
-    return { answer: text, context };
-  },
-```
-
-Note: You can specify any of the search options available on `rag.search`.
-See below for more details.
-
-### Using your own content splitter
-
-By default, the component uses the `defaultChunker` to split the content into chunks.
-You can pass in your own content chunks to the `add` or `addAsync` functions.
-
-```ts
-const chunks = await textSplitter.split(content);
-await rag.add(ctx, { namespace: "global", chunks });
-```
-
-Note: The `textSplitter` here could be LangChain, Mastra, or something custom.
-The simplest version makes an array of strings like `content.split("\n")`.
+See below for how to chunk the text yourself or add content asynchronously, e.g. to handle large files.
 
-Note: you can pass in an async iterator instead of an array to handle large content.
-
-### Semantic Search
+## Semantic Search
 
 Search across content with vector similarity
 
@@ -157,7 +106,7 @@ export const search = action({
     const { results, text, entries } = await rag.search(ctx, {
       namespace: "global",
       query: args.query,
-      limit: 10
+      limit: 10,
       vectorScoreThreshold: 0.5, // Only return results with a score >= 0.5
     });
 
@@ -166,40 +115,93 @@ export const search = action({
 });
 ```
 
-### Using keys to gracefully replace content
+## Generate a response based on RAG context
 
-When you add content to a namespace, you can provide a `key` to uniquely identify the content.
-If you add content with the same key, it will replace the existing content.
+Once you have searched for the context, you can use it with an LLM.
+
+Generally you'll already be using something to make LLM requests, e.g.
+the [Agent Component](https://www.convex.dev/components/agent),
+which tracks the message history for you.
+See the [Agent Component docs](https://docs.convex.dev/agents)
+for more details on doing RAG with the Agent Component.
+
+However, if you just want a one-off response, you can use the `generateText`
+function as a convenience.
+
+This will automatically search for relevant entries and use them as context
+for the LLM, using default formatting.
+
+The arguments to `generateText` are compatible with all arguments to
+`generateText` from the AI SDK.
 
 ```ts
-await rag.add(ctx, { namespace: userId, key: "my-file.txt", text });
+export const askQuestion = action({
+  args: {
+    prompt: v.string(),
+  },
+  handler: async (ctx, args) => {
+    const userId = await getAuthUserId(ctx);
+    const { text, context } = await rag.generateText(ctx, {
+      search: { namespace: userId, limit: 10 },
+      prompt: args.prompt,
+      model: openai.chat("gpt-4o-mini"),
+    });
+    return { answer: text, context };
+  },
 ```
 
-When a new document is added, it will start with a status of "pending" while
-it chunks, embeds, and inserts the data into the database.
-Once all data is inserted, it will iterate over the chunks and swap the old
-content embeddings with the new ones, and then update the status to "ready",
-marking the previous version as "replaced".
+Note: You can specify any of the search options available on `rag.search`.
 
-The old content is kept around by default, so in-flight searches will get
-results for old vector search results.
-See below for more details on deleting.
+## Filtered Search
 
-This means that if searches are happening while the document is being added,
-they will see the old content results
-This is useful if you want to add content to a namespace and then immediately
-search for it, or if you want to add content to a namespace and then immediately
-add more content to the same namespace.
+You can provide filters when adding content and use them to search.
+To do this, you'll need to give the RAG component a list of the filter names.
+You can optionally provide a type parameter for type safety (no runtime validation).
 
-### Filtered Search
+Note: these filters can be OR'd together when searching. In order to get an AND,
+you provide a filter with a more complex value, such as `categoryAndType` below.
+
+```ts
+// convex/example.ts
+import { components } from "./_generated/api";
+import { RAG } from "@convex-dev/rag";
+// Any AI SDK model that supports embeddings will work.
+import { openai } from "@ai-sdk/openai";
+
+// Optional: Add type safety to your filters.
+type FilterTypes = {
+  category: string;
+  contentType: string;
+  categoryAndType: { category: string; contentType: string };
+};
+
+const rag = new RAG<FilterTypes>(components.rag, {
+  textEmbeddingModel: openai.embedding("text-embedding-3-small"),
+  embeddingDimension: 1536, // Needs to match your embedding model
+  filterNames: ["category", "contentType", "categoryAndType"],
+});
+```
+
+Adding content with filters:
+
+```ts
+await rag.add(ctx, {
+  namespace: "global",
+  text,
+  filterValues: [
+    { name: "category", value: "news" },
+    { name: "contentType", value: "article" },
+    { name: "categoryAndType", value: { category: "news", contentType: "article" } },
+  ],
+});
+```
 
 Search with metadata filters:
 
 ```ts
-export const searchByCategory = action({
+export const searchForNewsOrSports = action({
   args: {
     query: v.string(),
-    category: v.string(),
   },
   handler: async (ctx, args) => {
     const userId = await getUserId(ctx);
@@ -208,7 +210,10 @@ export const searchByCategory = action({
     const results = await rag.search(ctx, {
       namespace: userId,
       query: args.query,
-      filters: [{ name: "category", value: args.category }],
+      filters: [
+        { name: "category", value: "news" },
+        { name: "category", value: "sports" },
+      ],
       limit: 10,
     });
 
@@ -257,14 +262,14 @@ export const searchWithContext = action({
 });
 ```
 
-### Formatting results
+## Formatting results
 
 Formatting the results for use in a prompt depends a bit on the use case.
 By default, the results will be sorted by score, not necessarily in the order
 they appear in the original text. You may want to sort them by the order they
 appear in the original text so they follow the flow of the original document.
 
-For convenienct, the `text` field of the search results is a string formatted
+For convenience, the `text` field of the search results is a string formatted
 with `...` separating non-sequential chunks, `---` separating entries, and
 `# Title:` at each entry boundary (if titles are available).
 
@@ -274,14 +279,18 @@ console.log(text);
 ```
 
 ```txt
-# Title 1:
+## Title 1:
 Chunk 1 contents
 Chunk 2 contents
+
 ...
+
 Chunk 8 contents
 Chunk 9 contents
+
 ---
-# Title 2:
+
+## Title 2:
 Chunk 4 contents
 Chunk 5 contents
 ```
@@ -330,7 +339,49 @@ await generateText({
 });
 ```
 
-### Providing custom embeddings per-chunk
+## Using keys to gracefully replace content
+
+When you add content to a namespace, you can provide a `key` to uniquely identify the content.
+If you add content with the same key, it will make a new entry to replace the old one.
+
+```ts
+await rag.add(ctx, { namespace: userId, key: "my-file.txt", text });
+```
+
+When a new document is added, it will start with a status of "pending" while
+it chunks, embeds, and inserts the data into the database.
+Once all data is inserted, it will iterate over the chunks and swap the old
+content embeddings with the new ones, and then update the status to "ready",
+marking the previous version as "replaced".
+
+The old content is kept around by default, so in-flight searches will get
+results for old vector search results.
+See below for more details on deleting.
+
+This means that if searches are happening while the document is being added,
+they will see the old content results
+This is useful if you want to add content to a namespace and then immediately
+search for it, or if you want to add content to a namespace and then immediately
+add more content to the same namespace.
+
+## Using your own content splitter
+
+By default, the component uses the `defaultChunker` to split the content into chunks.
+You can pass in your own content chunks to the `add` or `addAsync` functions.
+
+```ts
+const chunks = await textSplitter.split(content);
+await rag.add(ctx, { namespace: "global", chunks });
+```
+
+Note: The `textSplitter` here could be LangChain, Mastra, or something custom.
+The simplest version makes an array of strings like `content.split("\n")`.
+
+Note: you can pass in an async iterator instead of an array to handle large content.
+Or use the `addAsync` function (see below).
+
+
+## Providing custom embeddings per-chunk
 
 In addition to the text, you can provide your own embeddings for each chunk.
 
@@ -348,7 +399,7 @@ const chunksWithEmbeddings = await Promise.all(chunks.map(async chunk => {
 await rag.add(ctx, { namespace: "global", chunks });
 ```
 
-### Add Entries Asynchronously using File Storage
+## Add Entries Asynchronously using File Storage
 
 For large files, you can upload them to file storage, then provide a chunker
 action to split them into chunks.
@@ -462,18 +513,196 @@ Generally you'd do this:
 1. Periodically by querying:
 
 ```ts
-const toDelete = await rag.list(ctx, {
-  status: "replaced",
-  paginationOpts: { cursor: null, numItems: 100 }
+// in convex/crons.ts
+import { cronJobs } from "convex/server";
+import { internal } from "./_generated/api.js";
+import { internalMutation } from "./_generated/server.js";
+import { v } from "convex/values";
+import { rag } from "./example.js";
+import { assert } from "convex-helpers";
+
+const WEEK = 7 * 24 * 60 * 60 * 1000;
+
+export const deleteOldContent = internalMutation({
+  args: { cursor: v.optional(v.string()) },
+  handler: async (ctx, args) => {
+    const toDelete = await rag.list(ctx, {
+      status: "replaced",
+      paginationOpts: { cursor: args.cursor ?? null, numItems: 100 },
+    });
+
+    for (const entry of toDelete.page) {
+      assert(entry.status === "replaced");
+      if (entry.replacedAt >= Date.now() - WEEK) {
+        return; // we're done when we catch up to a week ago
+      }
+      await rag.delete(ctx, { entryId: entry.entryId });
+    }
+    if (!toDelete.isDone) {
+      await ctx.scheduler.runAfter(0, internal.example.deleteOldContent, {
+        cursor: toDelete.continueCursor,
+      });
+    }
+  },
 });
 
-for (const entry of toDelete) {
-  assert(entry.status === "replaced");
-  if (entry.replacedAt >= Date.now() - ONE_WEEK_MS) {
-    break;
-  }
-  await rag.delete(ctx, { entryId: entry.entryId });
-}
+// See example/convex/crons.ts for a complete example.
+const crons = cronJobs();
+crons.interval("deleteOldContent", { hours: 1 }, internal.crons.deleteOldContent, {});
+export default crons;
+```
+
+## Working with types
+
+You can use the provided types to validate and store data.
+`import { ... } from "@convex-dev/rag";`
+
+Types for the various elements:
+
+`Entry`, `EntryFilter`, `SearchEntry`, `SearchResult`
+
+- `SearchEntry` is an `Entry` with a `text` field including the combined search
+  results for that entry, whereas a `SearchResult` is a specific chunk result,
+  along with surrounding chunks.
+
+`EntryId`, `NamespaceId`
+
+- While the `EntryId` and `NamespaceId` are strings under the hood, they are
+  given more specific types to make it easier to use them correctly.
+
+Validators can be used in `args` and schema table definitions:
+`vEntry`, `vEntryId`, `vNamespaceId`, `vSearchEntry`, `vSearchResult`
+
+e.g. `defineTable({ myDocTitle: v.string(), entryId: vEntryId })`
+
+The validators for the branded IDs will only validate they are strings,
+but will have the more specific types, to provide type safety.
+
+## Utility Functions
+
+In addition to the function on the `rag` instance, there are other utilities
+provided:
+
+### `defaultChunker`
+
+This is the default chunker used by the `add` and `addAsync` functions.
+
+It is customizable, but by default:
+- It tries to break up the text into paragraphs between 100-1k characters.
+- It will combine paragraphs to meet the minimum character count (100).
+- It will break up paragraphs into separate lines to keep it under 1k.
+- It will not split up a single line unless it's longer than 10k characters.
+
+```ts
+import { defaultChunker } from "@convex-dev/rag";
+
+const chunks = defaultChunker(text, {
+  // these are the defaults
+  minLines: 1,
+  minCharsSoftLimit: 100,
+  maxCharsSoftLimit: 1000,
+  maxCharsHardLimit: 10000,
+  delimiter: "\n\n",
+});
+```
+
+### `hybridRank`
+
+This is an implementation of "Reciprocal Rank Fusion" for ranking search results
+based on multiple scoring arrays. The premise is that if both arrays of results
+are sorted by score, the best results show up near the top of both arrays and
+should be preferred over results higher in one but much lower in the other.
+
+```ts
+import { hybridRank } from "@convex-dev/rag";
+
+const textSearchResults = [id1, id2, id3];
+const vectorSearchResults = [id2, id3, id1];
+const results = hybridRank([
+  textSearchResults,
+  vectorSearchResults,
+]);
+// results = [id2, id1, id3]
+```
+
+It can take more than two arrays, and you can provide weights for each array.
+
+```ts
+
+const recentSearchResults = [id5, id4, id3];
+const results = hybridRank([
+  textSearchResults,
+  vectorSearchResults,
+  recentSearchResults,
+], {
+  weights: [2, 1, 3], // prefer recent results more than text or vector
+});
+// results = [ id3, id5, id1, id2, id4 ]
+```
+
+To have it more biased towards the top few results, you can set the `k` value
+to a lower number (10 by default).
+
+```ts
+const results = hybridRank([
+  textSearchResults,
+  vectorSearchResults,
+  recentSearchResults,
+], { k: 1 });
+// results = [ id5, id1, id3, id2, id4 ]
+```
+
+### `contentHashFromArrayBuffer`
+
+This generates the hash of a file's contents, which can be used to avoid
+adding the same file twice.
+
+Note: doing `blob.arrayBuffer()` will consume the blob's data, so you'll need
+to make a new blob to use it after calling this function.
+
+```ts
+import { contentHashFromArrayBuffer } from "@convex-dev/rag";
+
+export const addFile = action({
+  args: { bytes: v.bytes() },
+  handler: async (ctx, { bytes }) => {
+
+    const hash = await contentHashFromArrayBuffer(bytes);
+
+    const existing = await rag.findEntryByContentHash(ctx, {
+      namespace: "global",
+      key: "my-file.txt",
+      contentHash: hash,
+    });
+    if (existing) {
+      console.log("File contents are the same, skipping");
+      return;
+    }
+    const blob = new Blob([bytes], { type: "text/plain" });
+    //...
+  },
+});
+```
+
+### `guessMimeTypeFromExtension`
+
+This guesses the mime type of a file from its extension.
+
+```ts
+import { guessMimeTypeFromExtension } from "@convex-dev/rag";
+
+const mimeType = guessMimeTypeFromExtension("my-file.mjs");
+console.log(mimeType); // "text/javascript"
+```
+
+### `guessMimeTypeFromContents`
+
+This guesses the mime type of a file from the first few bytes of its contents.
+
+```ts
+import { guessMimeTypeFromContents } from "@convex-dev/rag";
+
+const mimeType = guessMimeTypeFromContents(await file.arrayBuffer());
 ```
 
 ### Example Usage
@@ -482,5 +711,5 @@ See more example usage in [example.ts](./example/convex/example.ts).
 
 ### Running the example
 
-Run the example with `npm i && npm run example`.
+Run the example with `npm i && npm run setup && npm run example`.
 <!-- END: Include on https://convex.dev/components -->

package/dist/client/defaultChunker.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"defaultChunker.d.ts","sourceRoot":"","sources":["../../src/client/defaultChunker.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AACH,wBAAgB,cAAc,CAC5B,IAAI,EAAE,MAAM,EACZ,EACE,QAAY,EACZ,iBAAuB,EACvB,iBAAwB,EACxB,iBAAyB,EACzB,SAAkB,GACnB,GAAE;IACD,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,SAAS,CAAC,EAAE,MAAM,CAAC;CACf,GACL,MAAM,EAAE,CA6FV;AAoED,eAAe,cAAc,CAAC"}
+{"version":3,"file":"defaultChunker.d.ts","sourceRoot":"","sources":["../../src/client/defaultChunker.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AACH,wBAAgB,cAAc,CAC5B,IAAI,EAAE,MAAM,EACZ,EACE,QAAY,EACZ,iBAAuB,EACvB,iBAAwB,EACxB,iBAAyB,EACzB,SAAkB,GACnB,GAAE;IACD,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,SAAS,CAAC,EAAE,MAAM,CAAC;CACf,GACL,MAAM,EAAE,CA6HV;AA4FD,eAAe,cAAc,CAAC"}

package/dist/client/defaultChunker.js

@@ -4,7 +4,7 @@
  * By default, it will chunk into paragraphs and target
  * 200-2000 characters per chunk (only less than 1 line if the hard limit is reached).
  */
-export function defaultChunker(text, { minLines = 1, minCharsSoftLimit = 200, maxCharsSoftLimit = 2000, maxCharsHardLimit = 10000, delimiter = "\n\n", } = {}) {
+export function defaultChunker(text, { minLines = 1, minCharsSoftLimit = 100, maxCharsSoftLimit = 1000, maxCharsHardLimit = 10000, delimiter = "\n\n", } = {}) {
     if (!text)
         return [];
     // Split text into individual lines
@@ -19,13 +19,17 @@ export function defaultChunker(text, { minLines = 1, minCharsSoftLimit = 200, ma
         const potentialChunk = [...currentChunk, line].join("\n");
         // If adding this line would exceed max chars, finalize current chunk first
         if (potentialChunk.length > maxCharsSoftLimit && currentChunk.length > 0) {
-            const trimmedChunk = removeTrailingEmptyLines(currentChunk);
-            chunks.push(trimmedChunk.join("\n"));
+            const processedChunk = processChunkForOutput(currentChunk, lines, i - currentChunk.length);
+            if (processedChunk.trim()) {
+                chunks.push(processedChunk);
+            }
             // Split the line if it exceeds hard limit
             const splitLines = maybeSplitLine(line, maxCharsHardLimit);
             // Add all but the last split piece as separate chunks
             for (let j = 0; j < splitLines.length - 1; j++) {
-                chunks.push(splitLines[j]);
+                if (splitLines[j].trim()) {
+                    chunks.push(splitLines[j]);
+                }
             }
             // Keep the last piece for potential combination with next lines
             currentChunk = [splitLines[splitLines.length - 1]];
@@ -37,8 +41,11 @@ export function defaultChunker(text, { minLines = 1, minCharsSoftLimit = 200, ma
             currentChunk.join("\n").length >= Math.min(minCharsSoftLimit * 0.8, 150)) {
             // Simple logic: only split if potential chunk would exceed the soft max limit
             if (potentialChunk.length > maxCharsSoftLimit) {
-                // When splitting at delimiter boundary, preserve natural empty lines (don't remove trailing empty lines)
-                chunks.push(currentChunk.join("\n"));
+                // When splitting at delimiter boundary, preserve natural empty lines and trailing newlines
+                const processedChunk = processChunkForOutput(currentChunk, lines, i - currentChunk.length);
+                if (processedChunk.trim()) {
+                    chunks.push(processedChunk);
+                }
                 currentChunk = [line];
                 continue;
             }
@@ -53,22 +60,28 @@ export function defaultChunker(text, { minLines = 1, minCharsSoftLimit = 200, ma
                 if (splitLines.length > 1) {
                     // Line was split - add all but the last piece as separate chunks
                     for (let j = 0; j < splitLines.length - 1; j++) {
-                        chunks.push(splitLines[j]);
+                        if (splitLines[j].trim()) {
+                            chunks.push(splitLines[j]);
+                        }
                     }
                     // Keep the last piece for potential combination with next lines
                     currentChunk = [splitLines[splitLines.length - 1]];
                 }
                 else {
                     // Line doesn't exceed hard limit, keep it as is
-                    chunks.push(line);
+                    if (line.trim()) {
+                        chunks.push(line);
+                    }
                     currentChunk = [];
                 }
             }
             else {
                 // Remove last line and finalize chunk
                 const lastLine = currentChunk.pop();
-                const trimmedChunk = removeTrailingEmptyLines(currentChunk);
-                chunks.push(trimmedChunk.join("\n"));
+                const processedChunk = processChunkForOutput(currentChunk, lines, i - currentChunk.length);
+                if (processedChunk.trim()) {
+                    chunks.push(processedChunk);
+                }
                 currentChunk = [lastLine];
             }
         }
@@ -79,14 +92,32 @@ export function defaultChunker(text, { minLines = 1, minCharsSoftLimit = 200, ma
         if (remainingText.length > maxCharsHardLimit) {
             // Split the remaining chunk if it exceeds hard limit
             const splitLines = maybeSplitLine(remainingText, maxCharsHardLimit);
-            chunks.push(...splitLines);
+            chunks.push(...splitLines.filter((chunk) => chunk.trim()));
         }
         else {
-            const trimmedChunk = removeTrailingEmptyLines(currentChunk);
-            chunks.push(trimmedChunk.join("\n"));
+            const processedChunk = processChunkForOutput(currentChunk, lines, lines.length - currentChunk.length);
+            if (processedChunk.trim()) {
+                chunks.push(processedChunk);
+            }
         }
     }
-    return chunks;
+    // Filter out any empty chunks that might have slipped through
+    return chunks.filter((chunk) => chunk.trim().length > 0);
+}
+function processChunkForOutput(chunkLines, allLines, startIndex) {
+    if (chunkLines.length === 0)
+        return "";
+    // Remove trailing empty lines but preserve meaningful structure
+    const trimmedLines = removeTrailingEmptyLines(chunkLines);
+    // Check if we should preserve some trailing newlines by looking at the original context
+    const endIndex = startIndex + chunkLines.length - 1;
+    const hasTrailingNewlines = endIndex < allLines.length - 1 && chunkLines.length > trimmedLines.length;
+    // If we removed empty lines but there are more lines after this chunk,
+    // preserve one trailing newline to maintain paragraph separation
+    if (hasTrailingNewlines && trimmedLines.length > 0) {
+        return trimmedLines.join("\n") + "\n";
+    }
+    return trimmedLines.join("\n");
 }
 function maybeSplitLine(line, maxCharsHardLimit) {
     const inputs = [line]; // in reverse order
@@ -141,8 +172,8 @@ function removeTrailingEmptyLines(lines) {
             return lines.slice(0, i + 1);
         }
     }
-    // If all lines are empty, keep at least one
-    return lines.length > 0 ? [lines[0]] : [];
+    // If all lines are empty, return empty array instead of keeping empty strings
+    return [];
 }
 export default defaultChunker;
 //# sourceMappingURL=defaultChunker.js.map