semchunk 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f00f8beee6b49e975d6b3fc6f31dbfea804325af49bdafd3e70958e74357e287
-  data.tar.gz: c7b813779cb517ea5a4df048d244848c9475b1fc9f0a6276c96394e929fa89fa
+  metadata.gz: ba78c1e8ca6f2d54a7345a1f2c33a4492224c121022db4fdc4dd46742e4891c8
+  data.tar.gz: f7c5e2d0a24964d97f59aa6b1b16649712108e5ce9a3253a178e36d2a3196ecd
 SHA512:
-  metadata.gz: f66735229364b68db50409b4e2d297217a31fb59ceff792cf0396a5ead3277fa5974fcc5c2066892e67dff224a2345b0b2d4fe8ff525a7c20990e233915ed4dd
-  data.tar.gz: 42ce444f5a50ba23c7203cf49d1fa0b7d1ccf3fd4c6621b0cc061f3fb41d873be2f797451a3c1560c483ebbdbe388a9d16d77cf9cdf1699d1cd866de83ab4a80
+  metadata.gz: e41759111d572eff3fe47d518f33524e2fac447a09992e96e76a488c27cb760cbe82352b7533680a8761f8360ddfc2613ac5c7b148e21c9800357dd9dcbbed94
+  data.tar.gz: e4b4acaf2229dcf85d16d1c3869a4978f17f69fe60b3c2b2a86c67354a4e90f9982691f88f8c6595189863109bb0df82be2232b5144ea036ff2db4e0ce0425df

data/README.md

@@ -1,29 +1,334 @@
-# semchunk
+# Semchunk
 
 [![Gem Version](https://img.shields.io/gem/v/semchunk)](https://rubygems.org/gems/semchunk)
 [![Gem Downloads](https://img.shields.io/gem/dt/semchunk)](https://www.ruby-toolbox.com/projects/semchunk)
 [![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/philip-zhan/semchunk.rb/ci.yml)](https://github.com/philip-zhan/semchunk.rb/actions/workflows/ci.yml)
 
-TODO: Description of this gem goes here.
+Split text into semantically meaningful chunks of a specified size as determined by a provided token counter.
+
+This is a Ruby port of the Python [semchunk](https://github.com/umarbutler/semchunk) package.
+
+## Features
+
+- **Semantic chunking**: Splits text at natural boundaries (sentences, paragraphs, etc.) rather than at arbitrary character positions
+- **Token-aware**: Respects token limits from any tokenizer you provide
+- **Overlap support**: Create overlapping chunks for better context preservation
+- **Offset tracking**: Get the original positions of each chunk in the source text
+- **Flexible**: Works with any token counter (word count, character count, or tokenizers)
+- **Memoization**: Optional caching of token counts for improved performance
 
 ---
 
+- [Installation](#installation)
 - [Quick start](#quick-start)
+- [API Reference](#api-reference)
+- [Examples](#examples)
 - [Support](#support)
 - [License](#license)
 - [Code of conduct](#code-of-conduct)
 - [Contribution guide](#contribution-guide)
 
-## Quick start
+## Installation
 
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'semchunk'
 ```
+
+Or install it directly:
+
+```bash
 gem install semchunk
 ```
 
+## Quick start
+
+```ruby
+require "semchunk"
+
+# Define a simple token counter (or use a real tokenizer)
+token_counter = ->(text) { text.split.length }
+
+# Chunk some text
+text = "This is the first sentence. This is the second sentence. And this is the third sentence."
+chunks = Semchunk.chunk(text, chunk_size: 5, token_counter: token_counter)
+
+puts chunks.inspect
+# => ["This is the first sentence.", "This is the second sentence.", "And this is the third sentence."]
+```
+
+## API Reference
+
+### `Semchunk.chunk`
+
+Split a text into semantically meaningful chunks.
+
+```ruby
+Semchunk.chunk(
+  text,
+  chunk_size:,
+  token_counter:,
+  memoize: true,
+  offsets: false,
+  overlap: nil,
+  cache_maxsize: nil
+)
+```
+
+**Parameters:**
+- `text` (String): The text to be chunked
+- `chunk_size` (Integer): The maximum number of tokens a chunk may contain
+- `token_counter` (Proc, Lambda, Method): A callable that takes a string and returns the number of tokens in it
+- `memoize` (Boolean, optional): Whether to memoize the token counter. Defaults to `true`
+- `offsets` (Boolean, optional): Whether to return the start and end offsets of each chunk. Defaults to `false`
+- `overlap` (Float, Integer, nil, optional): The proportion of the chunk size (if < 1), or the number of tokens (if >= 1), by which chunks should overlap. Defaults to `nil`
+- `cache_maxsize` (Integer, nil, optional): The maximum number of text-token count pairs to cache. Defaults to `nil` (unbounded)
+
+**Returns:**
+- `Array<String>` if `offsets: false`: List of text chunks
+- `[Array<String>, Array<Array<Integer>>]` if `offsets: true`: List of chunks and their `[start, end]` offsets
+
+### `Semchunk.chunkerify`
+
+Create a reusable chunker object.
+
+```ruby
+Semchunk.chunkerify(
+  tokenizer_or_token_counter,
+  chunk_size: nil,
+  max_token_chars: nil,
+  memoize: true,
+  cache_maxsize: nil
+)
+```
+
+**Parameters:**
+- `tokenizer_or_token_counter`: A tokenizer object with an `encode` method, or a callable token counter
+- `chunk_size` (Integer, nil): Maximum tokens per chunk. If `nil`, will attempt to use tokenizer's `model_max_length`
+- `max_token_chars` (Integer, nil): Maximum characters per token (optimization parameter)
+- `memoize` (Boolean): Whether to cache token counts. Defaults to `true`
+- `cache_maxsize` (Integer, nil): Cache size limit. Defaults to `nil` (unbounded)
+
+**Returns:**
+- `Semchunk::Chunker`: A chunker instance
+
+### `Chunker#call`
+
+Process text(s) with the chunker.
+
+```ruby
+chunker.call(
+  text_or_texts,
+  processes: 1,
+  progress: false,
+  offsets: false,
+  overlap: nil
+)
+```
+
+**Parameters:**
+- `text_or_texts` (String, Array<String>): Single text or array of texts to chunk
+- `processes` (Integer): Number of processes for parallel chunking (not yet implemented)
+- `progress` (Boolean): Show progress bar for multiple texts (not yet implemented)
+- `offsets` (Boolean): Return offset information
+- `overlap` (Float, Integer, nil): Overlap configuration
+
+**Returns:**
+- For single text: `Array<String>` or `[Array<String>, Array<Array<Integer>>]`
+- For multiple texts: `Array<Array<String>>` or `[Array<Array<String>>, Array<Array<Array<Integer>>>]`
+
+## Examples
+
+### Basic Chunking
+
 ```ruby
 require "semchunk"
+
+text = "Natural language processing is fascinating. It allows computers to understand human language. This enables many applications."
+
+# Use word count as token counter
+token_counter = ->(text) { text.split.length }
+
+chunks = Semchunk.chunk(text, chunk_size: 8, token_counter: token_counter)
+
+chunks.each_with_index do |chunk, i|
+  puts "Chunk #{i + 1}: #{chunk}"
+end
+# => Chunk 1: Natural language processing is fascinating. It allows computers
+# => Chunk 2: to understand human language. This enables many applications.
+```
+
+### With Offsets
+
+Track where each chunk came from in the original text:
+
+```ruby
+text = "First paragraph here. Second paragraph here. Third paragraph here."
+token_counter = ->(text) { text.split.length }
+
+chunks, offsets = Semchunk.chunk(
+  text,
+  chunk_size: 5,
+  token_counter: token_counter,
+  offsets: true
+)
+
+chunks.zip(offsets).each do |chunk, (start_pos, end_pos)|
+  puts "Chunk: '#{chunk}'"
+  puts "Position: #{start_pos}...#{end_pos}"
+  puts "Verification: '#{text[start_pos...end_pos]}'"
+  puts
+end
 ```
 
+### With Overlap
+
+Create overlapping chunks to maintain context:
+
+```ruby
+text = "One two three four five six seven eight nine ten."
+token_counter = ->(text) { text.split.length }
+
+# 50% overlap
+chunks = Semchunk.chunk(
+  text,
+  chunk_size: 4,
+  token_counter: token_counter,
+  overlap: 0.5
+)
+
+puts "Overlapping chunks:"
+chunks.each { |chunk| puts "- #{chunk}" }
+
+# Fixed overlap of 2 tokens
+chunks = Semchunk.chunk(
+  text,
+  chunk_size: 6,
+  token_counter: token_counter,
+  overlap: 2
+)
+
+puts "\nWith 2-token overlap:"
+chunks.each { |chunk| puts "- #{chunk}" }
+```
+
+### Using Chunkerify for Reusable Chunkers
+
+```ruby
+# Create a chunker once
+token_counter = ->(text) { text.split.length }
+chunker = Semchunk.chunkerify(token_counter, chunk_size: 10)
+
+# Use it multiple times
+texts = [
+  "First document to process.",
+  "Second document to process.",
+  "Third document to process."
+]
+
+all_chunks = chunker.call(texts)
+
+all_chunks.each_with_index do |chunks, i|
+  puts "Document #{i + 1} chunks: #{chunks.inspect}"
+end
+```
+
+### Character-Level Chunking
+
+```ruby
+text = "abcdefghijklmnopqrstuvwxyz"
+
+# Character count as token counter
+token_counter = ->(text) { text.length }
+
+chunks = Semchunk.chunk(text, chunk_size: 5, token_counter: token_counter)
+
+puts chunks.inspect
+# => ["abcde", "fghij", "klmno", "pqrst", "uvwxy", "z"]
+```
+
+### Custom Token Counter
+
+```ruby
+# Token counter that counts punctuation as separate tokens
+def custom_token_counter(text)
+  text.scan(/\w+|[^\w\s]/).length
+end
+
+text = "Hello, world! How are you?"
+
+chunks = Semchunk.chunk(
+  text,
+  chunk_size: 5,
+  token_counter: method(:custom_token_counter)
+)
+
+puts chunks.inspect
+```
+
+### Working with Real Tokenizers
+
+If you have a tokenizer that implements an `encode` method:
+
+```ruby
+# Example with a hypothetical tokenizer
+class MyTokenizer
+  def encode(text, add_special_tokens: true)
+    # Your tokenization logic here
+    text.split.map { |word| word.hash }
+  end
+  
+  def model_max_length
+    512
+  end
+end
+
+tokenizer = MyTokenizer.new
+
+# chunkerify will automatically extract the token counter
+chunker = Semchunk.chunkerify(tokenizer, chunk_size: 100)
+
+text = "Your long text here..."
+chunks = chunker.call(text)
+```
+
+## How It Works
+
+Semchunk uses a hierarchical splitting strategy:
+
+1. **Primary split**: Tries to split on paragraph breaks (newlines)
+2. **Secondary split**: Falls back to sentences (periods, question marks, etc.)
+3. **Tertiary split**: Uses clauses (commas, semicolons) if needed
+4. **Final split**: Character-level splitting as last resort
+
+This ensures that chunks are semantically meaningful while respecting your token limits.
+
+The algorithm uses binary search to efficiently find the optimal split points, making it fast even for large documents.
+
+## Running the Examples
+
+This gem includes example scripts that demonstrate various features:
+
+```bash
+# Basic usage examples
+ruby examples/basic_usage.rb
+
+# Advanced usage with longer documents
+ruby examples/advanced_usage.rb
+```
+
+## Differences from Python Version
+
+This Ruby port maintains feature parity with the Python version, with a few notes:
+
+- Multiprocessing support is not yet implemented (`processes` parameter)
+- Progress bar support is not yet implemented (`progress` parameter)  
+- String tokenizer names (like `"gpt-4"`) are not yet supported
+- Otherwise, the API and behavior match the Python version
+
+See [MIGRATION.md](MIGRATION.md) for a detailed guide on migrating from the Python version.
+
 ## Support
 
 If you want to report a bug, or have ideas, feedback or questions about the gem, [let me know via GitHub issues](https://github.com/philip-zhan/semchunk.rb/issues/new) and I will do my best to provide a helpful answer. Happy hacking!

data/lib/semchunk/chunker.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+require "set"
+
+module Semchunk
+  # A class for chunking one or more texts into semantically meaningful chunks
+  class Chunker
+    attr_reader :chunk_size, :token_counter
+
+    def initialize(chunk_size:, token_counter:)
+      @chunk_size = chunk_size
+      @token_counter = token_counter
+    end
+
+    # Split text or texts into semantically meaningful chunks
+    #
+    # @param text_or_texts [String, Array<String>] The text or texts to be chunked
+    # @param processes [Integer] The number of processes to use when chunking multiple texts (not yet implemented)
+    # @param progress [Boolean] Whether to display a progress bar when chunking multiple texts (not yet implemented)
+    # @param offsets [Boolean] Whether to return the start and end offsets of each chunk
+    # @param overlap [Float, Integer, nil] The proportion of the chunk size, or, if >=1, the number of tokens, by which chunks should overlap
+    #
+    # @return [Array<String>, Array<Array>, Hash] Depending on the input and options, returns chunks and optionally offsets
+    def call(text_or_texts, processes: 1, progress: false, offsets: false, overlap: nil)
+      chunk_function = make_chunk_function(offsets: offsets, overlap: overlap)
+
+      # Handle single text
+      if text_or_texts.is_a?(String)
+        return chunk_function.call(text_or_texts)
+      end
+
+      # Handle multiple texts
+      if processes == 1
+        # TODO: Add progress bar support
+        chunks_and_offsets = text_or_texts.map { |text| chunk_function.call(text) }
+      else
+        # TODO: Add parallel processing support
+        raise NotImplementedError, "Parallel processing not yet implemented. Please use processes: 1"
+      end
+
+      # Return results
+      if offsets
+        chunks, offsets_arr = chunks_and_offsets.transpose
+        return [chunks.to_a, offsets_arr.to_a]
+      end
+
+      chunks_and_offsets
+    end
+
+    private
+
+    def make_chunk_function(offsets:, overlap:)
+      lambda do |text|
+        Semchunk.chunk(
+          text,
+          chunk_size: chunk_size,
+          token_counter: token_counter,
+          memoize: false,
+          offsets: offsets,
+          overlap: overlap
+        )
+      end
+    end
+  end
+end

data/lib/semchunk/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Semchunk
-  VERSION = "0.1.0"
+  VERSION = "0.1.1"
 end

data/lib/semchunk.rb

@@ -1,5 +1,387 @@
 # frozen_string_literal: true
 
+require_relative "semchunk/version"
+require_relative "semchunk/chunker"
+
 module Semchunk
-  autoload :VERSION, "semchunk/version"
+  # A map of token counters to their memoized versions
+  @memoized_token_counters = {}
+
+  class << self
+    attr_reader :memoized_token_counters
+
+    # Split a text into semantically meaningful chunks of a specified size as determined by the provided token counter.
+    #
+    # @param text [String] The text to be chunked.
+    # @param chunk_size [Integer] The maximum number of tokens a chunk may contain.
+    # @param token_counter [Proc, Method, #call] A callable that takes a string and returns the number of tokens in it.
+    # @param memoize [Boolean] Whether to memoize the token counter. Defaults to true.
+    # @param offsets [Boolean] Whether to return the start and end offsets of each chunk. Defaults to false.
+    # @param overlap [Float, Integer, nil] The proportion of the chunk size, or, if >=1, the number of tokens, by which chunks should overlap. Defaults to nil.
+    # @param cache_maxsize [Integer, nil] The maximum number of text-token count pairs that can be stored in the token counter's cache. Defaults to nil (unbounded).
+    # @param recursion_depth [Integer] Internal parameter for tracking recursion depth.
+    # @param start [Integer] Internal parameter for tracking character offset.
+    #
+    # @return [Array<String>, Array<Array>] A list of chunks up to chunk_size-tokens-long, with any whitespace used to split the text removed, and, if offsets is true, a list of tuples [start, end].
+    def chunk(text, chunk_size:, token_counter:, memoize: true, offsets: false, overlap: nil, cache_maxsize: nil, recursion_depth: 0, start: 0)
+      # Rename variables for clarity
+      return_offsets = offsets
+      local_chunk_size = chunk_size
+
+      # If this is the first call, memoize the token counter if memoization is enabled and reduce the effective chunk size if overlapping chunks
+      is_first_call = recursion_depth.zero?
+
+      if is_first_call
+        if memoize
+          token_counter = memoize_token_counter(token_counter, cache_maxsize)
+        end
+
+        if overlap
+          # Make relative overlaps absolute and floor both relative and absolute overlaps
+          overlap = if overlap < 1
+                      (chunk_size * overlap).floor
+                    else
+                      [overlap, chunk_size - 1].min
+                    end
+
+          # If the overlap has not been zeroed, compute the effective chunk size
+          if overlap.positive?
+            unoverlapped_chunk_size = chunk_size - overlap
+            local_chunk_size = [overlap, unoverlapped_chunk_size].min
+          end
+        end
+      end
+
+      # Split the text using the most semantically meaningful splitter possible
+      splitter, splitter_is_whitespace, splits = split_text(text)
+
+      offsets_arr = []
+      splitter_len = splitter.length
+      split_lens = splits.map(&:length)
+      cum_lens = [0]
+      split_lens.each { |len| cum_lens << cum_lens.last + len }
+
+      split_starts = [0]
+      split_lens.each_with_index do |split_len, i|
+        split_starts << split_starts[i] + split_len + splitter_len
+      end
+      split_starts = split_starts.map { |s| s + start }
+
+      num_splits_plus_one = splits.length + 1
+
+      chunks = []
+      skips = Set.new
+
+      # Iterate through the splits
+      splits.each_with_index do |split, i|
+        # Skip the split if it has already been added to a chunk
+        next if skips.include?(i)
+
+        split_start = split_starts[i]
+
+        # If the split is over the chunk size, recursively chunk it
+        if token_counter.call(split) > local_chunk_size
+          new_chunks, new_offsets = chunk(
+            split,
+            chunk_size: local_chunk_size,
+            token_counter: token_counter,
+            offsets: true,
+            recursion_depth: recursion_depth + 1,
+            start: split_start
+          )
+
+          chunks.concat(new_chunks)
+          offsets_arr.concat(new_offsets)
+        else
+          # Merge the split with subsequent splits until the chunk size is reached
+          final_split_in_chunk_i, new_chunk = merge_splits(
+            splits: splits,
+            cum_lens: cum_lens,
+            chunk_size: local_chunk_size,
+            splitter: splitter,
+            token_counter: token_counter,
+            start: i,
+            high: num_splits_plus_one
+          )
+
+          # Mark any splits included in the new chunk for exclusion from future chunks
+          ((i + 1)...final_split_in_chunk_i).each { |j| skips.add(j) }
+
+          # Add the chunk
+          chunks << new_chunk
+
+          # Add the chunk's offsets
+          split_end = split_starts[final_split_in_chunk_i] - splitter_len
+          offsets_arr << [split_start, split_end]
+        end
+
+        # If the splitter is not whitespace and the split is not the last split, add the splitter to the end of the latest chunk
+        unless splitter_is_whitespace || (i == splits.length - 1 || ((i + 1)...splits.length).all? { |j| skips.include?(j) })
+          last_chunk_with_splitter = chunks[-1] + splitter
+          if token_counter.call(last_chunk_with_splitter) <= local_chunk_size
+            chunks[-1] = last_chunk_with_splitter
+            offset_start, offset_end = offsets_arr[-1]
+            offsets_arr[-1] = [offset_start, offset_end + splitter_len]
+          else
+            offset_start = offsets_arr.empty? ? split_start : offsets_arr[-1][1]
+            chunks << splitter
+            offsets_arr << [offset_start, offset_start + splitter_len]
+          end
+        end
+      end
+
+      # If this is the first call, remove empty chunks and overlap if desired
+      if is_first_call
+        # Remove empty chunks and chunks comprised entirely of whitespace
+        chunks_and_offsets = chunks.zip(offsets_arr).reject { |chunk, _| chunk.empty? || chunk.strip.empty? }
+
+        if chunks_and_offsets.any?
+          chunks, offsets_arr = chunks_and_offsets.transpose
+        else
+          chunks = []
+          offsets_arr = []
+        end
+
+        # Overlap chunks if desired and there are chunks to overlap
+        if overlap && overlap.positive? && chunks.any?
+          # Rename variables for clarity
+          subchunk_size = local_chunk_size
+          subchunks = chunks
+          suboffsets = offsets_arr
+          num_subchunks = subchunks.length
+
+          # Merge the subchunks into overlapping chunks
+          subchunks_per_chunk = (chunk_size.to_f / subchunk_size).floor
+          subchunk_stride = (unoverlapped_chunk_size.to_f / subchunk_size).floor
+
+          num_overlapping_chunks = [1, ((num_subchunks - subchunks_per_chunk).to_f / subchunk_stride).ceil + 1].max
+
+          offsets_arr = (0...num_overlapping_chunks).map do |i|
+            start_idx = i * subchunk_stride
+            end_idx = [start_idx + subchunks_per_chunk, num_subchunks].min - 1
+            [suboffsets[start_idx][0], suboffsets[end_idx][1]]
+          end
+
+          chunks = offsets_arr.map { |s, e| text[s...e] }
+        end
+
+        # Return offsets if desired
+        return [chunks, offsets_arr] if return_offsets
+
+        return chunks
+      end
+
+      # Always return chunks and offsets if this is a recursive call
+      [chunks, offsets_arr]
+    end
+
+    # Construct a chunker that splits one or more texts into semantically meaningful chunks
+    #
+    # @param tokenizer_or_token_counter [String, #encode, Proc, Method, #call] Either: the name of a tokenizer; a tokenizer that possesses an encode method; or a token counter.
+    # @param chunk_size [Integer, nil] The maximum number of tokens a chunk may contain. Defaults to nil.
+    # @param max_token_chars [Integer, nil] The maximum number of characters a token may contain. Defaults to nil.
+    # @param memoize [Boolean] Whether to memoize the token counter. Defaults to true.
+    # @param cache_maxsize [Integer, nil] The maximum number of text-token count pairs that can be stored in the token counter's cache.
+    #
+    # @return [Chunker] A chunker instance
+    def chunkerify(tokenizer_or_token_counter, chunk_size: nil, max_token_chars: nil, memoize: true, cache_maxsize: nil)
+      # Handle string tokenizer names (would require tiktoken/transformers Ruby equivalents)
+      if tokenizer_or_token_counter.is_a?(String)
+        raise NotImplementedError, "String tokenizer names not yet supported in Ruby. Please pass a tokenizer object or token counter proc."
+      end
+
+      # Determine max_token_chars if not provided
+      if max_token_chars.nil?
+        if tokenizer_or_token_counter.respond_to?(:token_byte_values)
+          vocab = tokenizer_or_token_counter.token_byte_values
+          max_token_chars = vocab.map(&:length).max if vocab.respond_to?(:map)
+        elsif tokenizer_or_token_counter.respond_to?(:get_vocab)
+          vocab = tokenizer_or_token_counter.get_vocab
+          max_token_chars = vocab.keys.map(&:length).max if vocab.respond_to?(:keys)
+        end
+      end
+
+      # Determine chunk_size if not provided
+      if chunk_size.nil?
+        if tokenizer_or_token_counter.respond_to?(:model_max_length) && tokenizer_or_token_counter.model_max_length.is_a?(Integer)
+          chunk_size = tokenizer_or_token_counter.model_max_length
+
+          # Attempt to reduce the chunk size by the number of special characters
+          if tokenizer_or_token_counter.respond_to?(:encode)
+            begin
+              chunk_size -= tokenizer_or_token_counter.encode("").length
+            rescue StandardError
+              # Ignore errors
+            end
+          end
+        else
+          raise ArgumentError, "chunk_size not provided and tokenizer lacks model_max_length attribute"
+        end
+      end
+
+      # Construct token counter from tokenizer if needed
+      if tokenizer_or_token_counter.respond_to?(:encode)
+        tokenizer = tokenizer_or_token_counter
+        # Check if encode accepts add_special_tokens parameter
+        encode_params = tokenizer.method(:encode).parameters rescue []
+        has_special_tokens = encode_params.any? { |type, name| name == :add_special_tokens }
+
+        token_counter = if has_special_tokens
+                          ->(text) { tokenizer.encode(text, add_special_tokens: false).length }
+                        else
+                          ->(text) { tokenizer.encode(text).length }
+                        end
+      else
+        token_counter = tokenizer_or_token_counter
+      end
+
+      # Add fast token counter optimization if max_token_chars is known
+      if max_token_chars
+        max_token_chars -= 1
+        original_token_counter = token_counter
+
+        token_counter = lambda do |text|
+          heuristic = chunk_size * 6
+          if text.length > heuristic && original_token_counter.call(text[0...(heuristic + max_token_chars)]) > chunk_size
+            chunk_size + 1
+          else
+            original_token_counter.call(text)
+          end
+        end
+      end
+
+      # Memoize the token counter if necessary
+      if memoize
+        token_counter = memoize_token_counter(token_counter, cache_maxsize)
+      end
+
+      # Construct and return the chunker
+      Chunker.new(chunk_size: chunk_size, token_counter: token_counter)
+    end
+
+    private
+
+    # A tuple of semantically meaningful non-whitespace splitters
+    NON_WHITESPACE_SEMANTIC_SPLITTERS = [
+      # Sentence terminators
+      ".", "?", "!", "*",
+      # Clause separators
+      ";", ",", "(", ")", "[", "]", """, """, "'", "'", "'", '"', "`",
+      # Sentence interrupters
+      ":", "—", "…",
+      # Word joiners
+      "/", "\\", "–", "&", "-"
+    ].freeze
+
+    def split_text(text)
+      splitter_is_whitespace = true
+
+      # Try splitting at various levels
+      if text.include?("\n") || text.include?("\r")
+        newline_matches = text.scan(/[\r\n]+/)
+        splitter = newline_matches.max_by(&:length)
+      elsif text.include?("\t")
+        tab_matches = text.scan(/\t+/)
+        splitter = tab_matches.max_by(&:length)
+      elsif text.match?(/\s/)
+        whitespace_matches = text.scan(/\s+/)
+        splitter = whitespace_matches.max_by(&:length)
+
+        # If the splitter is only a single character, see if we can target whitespace preceded by semantic splitters
+        if splitter.length == 1
+          NON_WHITESPACE_SEMANTIC_SPLITTERS.each do |preceder|
+            escaped_preceder = Regexp.escape(preceder)
+            if (match = text.match(/#{escaped_preceder}(\s)/))
+              splitter = match[1]
+              escaped_splitter = Regexp.escape(splitter)
+              return [splitter, splitter_is_whitespace, text.split(/(?<=#{escaped_preceder})#{escaped_splitter}/)]
+            end
+          end
+        end
+      else
+        # Find the most desirable semantically meaningful non-whitespace splitter
+        splitter = NON_WHITESPACE_SEMANTIC_SPLITTERS.find { |s| text.include?(s) }
+
+        if splitter
+          splitter_is_whitespace = false
+        else
+          # No semantic splitter found, return characters
+          return ["", splitter_is_whitespace, text.chars]
+        end
+      end
+
+      [splitter, splitter_is_whitespace, text.split(splitter)]
+    end
+
+    def bisect_left(sorted, target, low, high)
+      while low < high
+        mid = (low + high) / 2
+        if sorted[mid] < target
+          low = mid + 1
+        else
+          high = mid
+        end
+      end
+      low
+    end
+
+    def merge_splits(splits:, cum_lens:, chunk_size:, splitter:, token_counter:, start:, high:)
+      average = 0.2
+      low = start
+
+      offset = cum_lens[start]
+      target = offset + (chunk_size * average)
+
+      while low < high
+        i = bisect_left(cum_lens, target, low, high)
+        midpoint = [i, high - 1].min
+
+        tokens = token_counter.call(splits[start...midpoint].join(splitter))
+
+        local_cum = cum_lens[midpoint] - offset
+
+        if local_cum.positive? && tokens.positive?
+          average = local_cum.to_f / tokens
+          target = offset + (chunk_size * average)
+        end
+
+        if tokens > chunk_size
+          high = midpoint
+        else
+          low = midpoint + 1
+        end
+      end
+
+      last_split_index = low - 1
+      [last_split_index, splits[start...last_split_index].join(splitter)]
+    end
+
+    def memoize_token_counter(token_counter, maxsize = nil)
+      return @memoized_token_counters[token_counter] if @memoized_token_counters.key?(token_counter)
+
+      cache = {}
+      queue = []
+
+      memoized = lambda do |text|
+        if cache.key?(text)
+          cache[text]
+        else
+          result = token_counter.call(text)
+          cache[text] = result
+
+          if maxsize
+            queue << text
+            if queue.length > maxsize
+              oldest = queue.shift
+              cache.delete(oldest)
+            end
+          end
+
+          result
+        end
+      end
+
+      @memoized_token_counters[token_counter] = memoized
+    end
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: semchunk
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - Philip Zhan
@@ -18,6 +18,7 @@ files:
 - LICENSE.txt
 - README.md
 - lib/semchunk.rb
+- lib/semchunk/chunker.rb
 - lib/semchunk/version.rb
 homepage: https://github.com/philip-zhan/semchunk.rb
 licenses: