opaque_id 1.4.0 → 1.7.0

data/docs/algorithms.md

@@ -8,7 +8,7 @@ permalink: /algorithms/
 
 # Algorithms
 
-OpaqueId uses sophisticated algorithms to generate cryptographically secure, collision-free opaque IDs. This guide explains the technical details behind the generation process, optimization strategies, and mathematical foundations.
+OpaqueId builds on Ruby's built-in `SecureRandom` methods to generate cryptographically secure, collision-free opaque IDs. This guide explains the technical details behind the generation process, optimization strategies, and mathematical foundations.
 
 - TOC
   {:toc}
@@ -45,10 +45,10 @@ end
 
 ### Key Features
 
-- **Bitwise Operations**: Uses `byte & 63` instead of `byte % 64` for faster computation
+- **Bitwise Operations**: Uses `byte & 63` instead of `byte % 64` for efficient computation
 - **No Modulo Bias**: 64 is a power of 2, so bitwise AND provides uniform distribution
 - **Single Random Call**: One `SecureRandom.random_bytes(1)` call per character
-- **Maximum Performance**: Optimized for speed with 64-character alphabets
+- **Performance Optimized**: Designed for speed with 64-character alphabets
 
 ### Mathematical Foundation
 
@@ -60,7 +60,7 @@ For a 64-character alphabet:
 
 ### Performance Characteristics
 
-- **Optimized for speed**: Uses bitwise operations for maximum performance
+- **Optimized for speed**: Uses bitwise operations for efficient performance
 - **No rejection sampling**: All generated bytes are used efficiently
 - **Linear time complexity**: O(n) where n is the ID length
 

data/docs/api-reference.md

@@ -19,20 +19,20 @@ The main module for generating opaque IDs.
 
 ### Constants
 
-#### ALPHANUMERIC_ALPHABET
+#### SLUG_LIKE_ALPHABET
 
 Default alphabet for ID generation.
 
 ```ruby
-OpaqueId::ALPHANUMERIC_ALPHABET
-# => "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
+OpaqueId::SLUG_LIKE_ALPHABET
+# => "0123456789abcdefghijklmnopqrstuvwxyz"
 ```
 
 **Characteristics:**
 
-- **Length**: 62 characters
-- **Characters**: A-Z, a-z, 0-9
-- **Use case**: General purpose, URL-safe
+- **Length**: 36 characters
+- **Characters**: 0-9, a-z
+- **Use case**: URL-friendly, double-click selectable
 - **Performance**: Good
 
 #### STANDARD_ALPHABET

data/docs/benchmarks.md

@@ -0,0 +1,385 @@
+# OpaqueId Benchmarks
+
+This document provides benchmark scripts that you can run to test OpaqueId's performance and uniqueness characteristics on your own system.
+
+## Performance Benchmarks
+
+### SecureRandom Comparison Test
+
+```ruby
+#!/usr/bin/env ruby
+
+require 'opaque_id'
+require 'securerandom'
+
+puts "OpaqueId vs SecureRandom Comparison"
+puts "=" * 50
+
+# Test different Ruby standard library methods
+methods = {
+  'OpaqueId.generate' => -> { OpaqueId.generate },
+  'SecureRandom.urlsafe_base64' => -> { SecureRandom.urlsafe_base64 },
+  'SecureRandom.urlsafe_base64(16)' => -> { SecureRandom.urlsafe_base64(16) },
+  'SecureRandom.hex(9)' => -> { SecureRandom.hex(9) },
+  'SecureRandom.alphanumeric(18)' => -> { SecureRandom.alphanumeric(18) }
+}
+
+count = 10000
+
+puts "Performance comparison (#{count} IDs each):"
+puts "-" * 50
+
+methods.each do |name, method|
+  start_time = Time.now
+  ids = count.times.map { method.call }
+  end_time = Time.now
+  duration = end_time - start_time
+  rate = (count / duration).round(0)
+
+  # Check uniqueness
+  unique_count = ids.uniq.length
+  collisions = count - unique_count
+
+  # Check characteristics
+  sample_id = ids.first
+  length = sample_id.length
+  has_uppercase = sample_id.match?(/[A-Z]/)
+  has_lowercase = sample_id.match?(/[a-z]/)
+  has_numbers = sample_id.match?(/[0-9]/)
+  has_special = sample_id.match?(/[^A-Za-z0-9]/)
+
+  puts "#{name.ljust(30)}: #{duration.round(4)}s (#{rate} IDs/sec)"
+  puts "  Length: #{length}, Collisions: #{collisions}"
+  puts "  Sample: '#{sample_id}'"
+  puts "  Chars: #{has_uppercase ? 'A-Z' : ''}#{has_lowercase ? 'a-z' : ''}#{has_numbers ? '0-9' : ''}#{has_special ? 'special' : ''}"
+  puts
+end
+
+puts "Comparison completed."
+```
+
+### Basic Performance Test
+
+```ruby
+#!/usr/bin/env ruby
+
+require 'opaque_id'
+
+puts "OpaqueId Performance Benchmark"
+puts "=" * 40
+
+# Test different batch sizes
+[100, 1000, 10000, 100000].each do |count|
+  start_time = Time.now
+  count.times { OpaqueId.generate }
+  end_time = Time.now
+  duration = end_time - start_time
+  rate = (count / duration).round(0)
+
+  puts "#{count.to_s.rjust(6)} IDs: #{duration.round(4)}s (#{rate} IDs/sec)"
+end
+
+puts "\nPerformance test completed."
+```
+
+### Alphabet Performance Comparison
+
+```ruby
+#!/usr/bin/env ruby
+
+require 'opaque_id'
+
+puts "Alphabet Performance Comparison"
+puts "=" * 40
+
+alphabets = {
+  'SLUG_LIKE_ALPHABET' => OpaqueId::SLUG_LIKE_ALPHABET,
+  'ALPHANUMERIC_ALPHABET' => OpaqueId::ALPHANUMERIC_ALPHABET,
+  'STANDARD_ALPHABET' => OpaqueId::STANDARD_ALPHABET
+}
+
+count = 10000
+
+alphabets.each do |name, alphabet|
+  start_time = Time.now
+  count.times { OpaqueId.generate(alphabet: alphabet) }
+  end_time = Time.now
+  duration = end_time - start_time
+  rate = (count / duration).round(0)
+
+  puts "#{name.ljust(20)}: #{duration.round(4)}s (#{rate} IDs/sec)"
+end
+
+puts "\nAlphabet comparison completed."
+```
+
+### Size Performance Test
+
+```ruby
+#!/usr/bin/env ruby
+
+require 'opaque_id'
+
+puts "Size Performance Test"
+puts "=" * 40
+
+sizes = [8, 12, 18, 24, 32, 48, 64]
+count = 10000
+
+sizes.each do |size|
+  start_time = Time.now
+  count.times { OpaqueId.generate(size: size) }
+  end_time = Time.now
+  duration = end_time - start_time
+  rate = (count / duration).round(0)
+
+  puts "Size #{size.to_s.rjust(2)}: #{duration.round(4)}s (#{rate} IDs/sec)"
+end
+
+puts "\nSize performance test completed."
+```
+
+## Uniqueness Tests
+
+### Collision Probability Test
+
+```ruby
+#!/usr/bin/env ruby
+
+require 'opaque_id'
+
+puts "Collision Probability Test"
+puts "=" * 40
+
+# Test different sample sizes
+[1000, 10000, 100000].each do |count|
+  puts "\nTesting #{count} IDs..."
+
+  start_time = Time.now
+  ids = count.times.map { OpaqueId.generate }
+  end_time = Time.now
+
+  unique_ids = ids.uniq
+  collisions = count - unique_ids.length
+  collision_rate = (collisions.to_f / count * 100).round(6)
+
+  puts "  Generated: #{count} IDs in #{(end_time - start_time).round(4)}s"
+  puts "  Unique: #{unique_ids.length} IDs"
+  puts "  Collisions: #{collisions} (#{collision_rate}%)"
+  puts "  Uniqueness: #{collision_rate == 0 ? '✅ Perfect' : '⚠️  Collisions detected'}"
+end
+
+puts "\nCollision test completed."
+```
+
+### Birthday Paradox Test
+
+```ruby
+#!/usr/bin/env ruby
+
+require 'opaque_id'
+
+puts "Birthday Paradox Test"
+puts "=" * 40
+
+# Test the birthday paradox with different sample sizes
+# For 18-character slug-like alphabet (36 chars), we have 36^18 possible combinations
+# This is approximately 10^28, so collisions should be extremely rare
+
+sample_sizes = [1000, 10000, 50000, 100000]
+
+sample_sizes.each do |count|
+  puts "\nTesting #{count} IDs for birthday paradox..."
+
+  start_time = Time.now
+  ids = count.times.map { OpaqueId.generate }
+  end_time = Time.now
+
+  unique_ids = ids.uniq
+  collisions = count - unique_ids.length
+
+  # Calculate theoretical collision probability
+  # For 18-char slug-like alphabet: 36^18 ≈ 10^28 possible combinations
+  alphabet_size = 36
+  id_length = 18
+  total_possibilities = alphabet_size ** id_length
+
+  # Approximate birthday paradox probability
+  # P(collision) ≈ 1 - e^(-n(n-1)/(2*N)) where n=sample_size, N=total_possibilities
+  n = count
+  N = total_possibilities
+  theoretical_prob = 1 - Math.exp(-(n * (n - 1)) / (2.0 * N))
+
+  puts "  Sample size: #{count}"
+  puts "  Total possibilities: #{total_possibilities.to_s.reverse.gsub(/(\d{3})(?=.)/, '\1,').reverse}"
+  puts "  Theoretical collision probability: #{theoretical_prob.round(20)}"
+  puts "  Actual collisions: #{collisions}"
+  puts "  Result: #{collisions == 0 ? '✅ No collisions (as expected)' : '⚠️  Collisions detected'}"
+end
+
+puts "\nBirthday paradox test completed."
+```
+
+### Alphabet Distribution Test
+
+```ruby
+#!/usr/bin/env ruby
+
+require 'opaque_id'
+
+puts "Alphabet Distribution Test"
+puts "=" * 40
+
+# Test that all characters in the alphabet are used roughly equally
+alphabet = OpaqueId::SLUG_LIKE_ALPHABET
+count = 100000
+
+puts "Testing distribution for #{alphabet.length}-character alphabet..."
+puts "Sample size: #{count} IDs"
+
+start_time = Time.now
+ids = count.times.map { OpaqueId.generate }
+end_time = Time.now
+
+# Count character frequency
+char_counts = Hash.new(0)
+ids.each do |id|
+  id.each_char { |char| char_counts[char] += 1 }
+end
+
+total_chars = ids.join.length
+expected_per_char = total_chars.to_f / alphabet.length
+
+puts "\nCharacter distribution:"
+puts "Character | Count | Expected | Deviation"
+puts "-" * 45
+
+alphabet.each_char do |char|
+  count = char_counts[char]
+  deviation = ((count - expected_per_char) / expected_per_char * 100).round(2)
+  puts "#{char.ljust(8)} | #{count.to_s.rjust(5)} | #{expected_per_char.round(1).to_s.rjust(8)} | #{deviation.to_s.rjust(6)}%"
+end
+
+# Calculate chi-square test for uniform distribution
+chi_square = alphabet.each_char.sum do |char|
+  observed = char_counts[char]
+  expected = expected_per_char
+  ((observed - expected) ** 2) / expected
+end
+
+puts "\nChi-square statistic: #{chi_square.round(4)}"
+puts "Distribution: #{chi_square < 30 ? '✅ Appears uniform' : '⚠️  May not be uniform'}"
+
+puts "\nDistribution test completed."
+```
+
+## Running the Benchmarks
+
+### Quick Performance Test
+
+```bash
+# Run basic performance test
+ruby -e "
+require 'opaque_id'
+puts 'OpaqueId Performance Test'
+puts '=' * 30
+[100, 1000, 10000].each do |count|
+  start = Time.now
+  count.times { OpaqueId.generate }
+  duration = Time.now - start
+  rate = (count / duration).round(0)
+  puts \"#{count.to_s.rjust(5)} IDs: #{duration.round(4)}s (#{rate} IDs/sec)\"
+end
+"
+```
+
+### Quick Uniqueness Test
+
+```bash
+# Run basic uniqueness test
+ruby -e "
+require 'opaque_id'
+puts 'OpaqueId Uniqueness Test'
+puts '=' * 30
+count = 10000
+ids = count.times.map { OpaqueId.generate }
+unique = ids.uniq
+collisions = count - unique.length
+puts \"Generated: #{count} IDs\"
+puts \"Unique: #{unique.length} IDs\"
+puts \"Collisions: #{collisions}\"
+puts \"Result: #{collisions == 0 ? 'Perfect uniqueness' : 'Collisions detected'}\"
+"
+```
+
+## Expected Results
+
+### Performance Expectations
+
+On a modern system, you should expect:
+
+- **100 IDs**: < 0.001s (100,000+ IDs/sec)
+- **1,000 IDs**: < 0.01s (100,000+ IDs/sec)
+- **10,000 IDs**: < 0.1s (100,000+ IDs/sec)
+- **100,000 IDs**: < 1s (100,000+ IDs/sec)
+
+### Uniqueness Expectations
+
+- **1,000 IDs**: 0 collisions (100% unique)
+- **10,000 IDs**: 0 collisions (100% unique)
+- **100,000 IDs**: 0 collisions (100% unique)
+- **1,000,000 IDs**: 0 collisions (100% unique)
+
+The theoretical collision probability for 1 million IDs is approximately 10^-16, making collisions virtually impossible in practice.
+
+## System Requirements
+
+These benchmarks require:
+
+- Ruby 2.7+ (for optimal performance)
+- OpaqueId gem installed
+- Sufficient memory for large sample sizes
+
+For the largest tests (100,000+ IDs), ensure you have at least 100MB of available memory.
+
+## Why Not Just Use SecureRandom?
+
+Ruby's `SecureRandom` already provides secure random generation. Here's how OpaqueId compares:
+
+### SecureRandom.urlsafe_base64 vs OpaqueId
+
+| Feature                      | SecureRandom.urlsafe_base64     | OpaqueId.generate            |
+| ---------------------------- | ------------------------------- | ---------------------------- |
+| **Length**                   | 22 characters (fixed)           | 18 characters (configurable) |
+| **Alphabet**                 | A-Z, a-z, 0-9, -, \_ (64 chars) | 0-9, a-z (36 chars)          |
+| **URL Safety**               | ✅ Yes                          | ✅ Yes                       |
+| **Double-click selectable**  | ❌ No (contains special chars)  | ✅ Yes (no special chars)    |
+| **Configurable length**      | ❌ No                           | ✅ Yes                       |
+| **Configurable alphabet**    | ❌ No                           | ✅ Yes                       |
+| **ActiveRecord integration** | ❌ Manual                       | ✅ Built-in                  |
+| **Rails generator**          | ❌ No                           | ✅ Yes                       |
+
+### When to Use Each
+
+**Use SecureRandom.urlsafe_base64 when:**
+
+- You need maximum entropy (22 chars vs 18)
+- You don't mind special characters (-, \_)
+- You don't need double-click selection
+- You're building a simple solution
+
+**Use OpaqueId when:**
+
+- You want slug-like IDs (no special characters)
+- You need double-click selectable IDs
+- You want configurable length and alphabet
+- You're using ActiveRecord models
+- You want consistent ID length (default: 18 characters)
+
+### Performance Comparison
+
+Run the [SecureRandom Comparison Test](#securerandom-comparison-test) to see how OpaqueId compares to various SecureRandom methods on your system.
+
+### Migration from nanoid.rb
+
+The nanoid.rb gem is [considered obsolete](https://github.com/radeno/nanoid.rb/issues/67) for Ruby 2.7+ because SecureRandom provides similar functionality. OpaqueId provides an alternative with different defaults and Rails integration.

data/docs/configuration.md

@@ -26,10 +26,10 @@ class User < ApplicationRecord
   # Custom column name
   self.opaque_id_column = :public_id
 
-  # Custom length (default: 21)
+  # Custom length (default: 18)
   self.opaque_id_length = 15
 
-  # Custom alphabet (default: ALPHANUMERIC_ALPHABET)
+  # Custom alphabet (default: SLUG_LIKE_ALPHABET)
   self.opaque_id_alphabet = OpaqueId::STANDARD_ALPHABET
 
   # Require letter start (default: false)
@@ -45,14 +45,14 @@ end
 
 ### Configuration Options Reference
 
-| Option                           | Type    | Default                 | Description                                   |
-| -------------------------------- | ------- | ----------------------- | --------------------------------------------- |
-| `opaque_id_column`               | Symbol  | `:opaque_id`            | Column name for storing the opaque ID         |
-| `opaque_id_length`               | Integer | `21`                    | Length of generated IDs                       |
-| `opaque_id_alphabet`             | String  | `ALPHANUMERIC_ALPHABET` | Character set for ID generation               |
-| `opaque_id_require_letter_start` | Boolean | `false`                 | Require IDs to start with a letter            |
-| `opaque_id_max_retry`            | Integer | `3`                     | Maximum retry attempts for collision handling |
-| `opaque_id_purge_chars`          | Array   | `[]`                    | Characters to exclude from generated IDs      |
+| Option                           | Type    | Default              | Description                                   |
+| -------------------------------- | ------- | -------------------- | --------------------------------------------- |
+| `opaque_id_column`               | Symbol  | `:opaque_id`         | Column name for storing the opaque ID         |
+| `opaque_id_length`               | Integer | `18`                 | Length of generated IDs                       |
+| `opaque_id_alphabet`             | String  | `SLUG_LIKE_ALPHABET` | Character set for ID generation               |
+| `opaque_id_require_letter_start` | Boolean | `false`              | Require IDs to start with a letter            |
+| `opaque_id_max_retry`            | Integer | `3`                  | Maximum retry attempts for collision handling |
+| `opaque_id_purge_chars`          | Array   | `[]`                 | Characters to exclude from generated IDs      |
 
 ## Global Configuration
 
@@ -97,12 +97,25 @@ end
 
 ### Built-in Alphabets
 
-#### ALPHANUMERIC_ALPHABET (Default)
+#### SLUG_LIKE_ALPHABET (Default)
+
+```ruby
+# Characters: 0-9, a-z (36 characters)
+# Use case: URL-safe, double-click selectable, no confusing characters
+# Example output: "izkpm55j334u8x9y2a"
+
+class User < ApplicationRecord
+  include OpaqueId::Model
+  self.opaque_id_alphabet = OpaqueId::SLUG_LIKE_ALPHABET
+end
+```
+
+#### ALPHANUMERIC_ALPHABET
 
 ```ruby
 # Characters: A-Z, a-z, 0-9 (62 characters)
 # Use case: General purpose, URL-safe
-# Example output: "V1StGXR8_Z5jdHi6B-myT"
+# Example output: "V1StGXR8Z5jdHi6BmyT"
 
 class User < ApplicationRecord
   include OpaqueId::Model
@@ -240,9 +253,9 @@ end
 # Now the methods use the custom column name
 user = User.create!(name: "John Doe")
 puts user.public_id
-# => "V1StGXR8_Z5jdHi6B-myT"
+# => "izkpm55j334u8x9y2a"
 
-user = User.find_by_public_id("V1StGXR8_Z5jdHi6B-myT")
+user = User.find_by_public_id("izkpm55j334u8x9y2a")
 ```
 
 ### Multiple Column Names
@@ -279,7 +292,7 @@ end
 # This will retry until it generates an ID starting with a letter
 user = User.create!(name: "John Doe")
 puts user.opaque_id
-# => "V1StGXR8_Z5jdHi6B-myT" (starts with 'V')
+# => "izkpm55j334u8x9y2a" (starts with 'i')
 ```
 
 ### Character Purging
@@ -295,7 +308,7 @@ end
 # Generated IDs will not contain these characters
 user = User.create!(name: "John Doe")
 puts user.opaque_id
-# => "V1StGXR8_Z5jdHi6B-myT" (no '0', 'O', 'l', 'I')
+# => "izkpm55j334u8x9y2a" (no '0', 'O', 'l', 'I')
 ```
 
 ## Collision Handling Configuration
@@ -496,8 +509,8 @@ class UserTest < ActiveSupport::TestCase
     user = User.new(name: "Test User")
 
     assert user.valid?
-    assert_equal 21, user.class.opaque_id_length
-    assert_equal OpaqueId::ALPHANUMERIC_ALPHABET, user.class.opaque_id_alphabet
+    assert_equal 18, user.class.opaque_id_length
+    assert_equal OpaqueId::SLUG_LIKE_ALPHABET, user.class.opaque_id_alphabet
   end
 
   test "opaque_id generation works with custom configuration" do
@@ -523,6 +536,7 @@ end
 
 ### 2. Select Suitable Alphabets
 
+- **SLUG_LIKE_ALPHABET** (default): URL-safe, double-click selectable, no confusing characters
 - **ALPHANUMERIC_ALPHABET**: General purpose, URL-safe
 - **STANDARD_ALPHABET**: Fastest generation, URL-safe
 - **Custom alphabets**: Specific requirements (numeric, hexadecimal, etc.)

data/docs/index.md

@@ -12,7 +12,7 @@ permalink: /
 [![Ruby Style Guide](https://img.shields.io/badge/code_style-rubocop-brightgreen.svg)](https://github.com/rubocop/rubocop)
 [![Gem Downloads](https://img.shields.io/badge/gem%20downloads-opaque_id-blue)](https://rubygems.org/gems/opaque_id)
 
-A Ruby gem for generating cryptographically secure, collision-free opaque IDs for ActiveRecord models. OpaqueId provides a drop-in replacement for `nanoid.rb` using Ruby's built-in `SecureRandom` methods with optimized algorithms for unbiased distribution.
+A simple Ruby gem for generating secure, opaque IDs for ActiveRecord models. OpaqueId provides a drop-in replacement for `nanoid.rb` using Ruby's built-in `SecureRandom` methods, with slug-like IDs as the default for optimal URL safety and user experience.
 
 - TOC
   {:toc}
@@ -57,10 +57,10 @@ end
 
 # Create a user - opaque_id is automatically generated
 user = User.create!(name: "John Doe")
-puts user.opaque_id # => "V1StGXR8_Z5jdHi6B-myT"
+puts user.opaque_id # => "izkpm55j334u8x9y2a"
 
 # Find by opaque_id
-user = User.find_by_opaque_id("V1StGXR8_Z5jdHi6B-myT")
+user = User.find_by_opaque_id("izkpm55j334u8x9y2a")
 ```
 
 ## Why OpaqueId?
@@ -128,6 +128,24 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 
 This project follows an "open source, closed contribution" model. We welcome bug reports, feature requests, and documentation improvements through GitHub Issues.
 
+## Performance & Benchmarks
+
+You can run benchmarks to test OpaqueId's performance and uniqueness characteristics on your system.
+
+**Quick Test:**
+
+```bash
+# Test 10,000 ID generation
+ruby -e "require 'opaque_id'; start=Time.now; 10000.times{OpaqueId.generate}; puts \"Generated 10,000 IDs in #{(Time.now-start).round(4)}s\""
+```
+
+**Expected Results:**
+
+- **Performance**: 100,000+ IDs per second on modern hardware
+- **Uniqueness**: Zero collisions in practice (theoretical probability < 10^-16 for 1M IDs)
+
+For comprehensive benchmarks including collision tests, alphabet distribution analysis, and performance comparisons, see the [Benchmarks Guide](benchmarks.md).
+
 ## Acknowledgements
 
 - [nanoid.rb](https://github.com/radeno/nanoid.rb) - Original inspiration and reference implementation

data/docs/performance.md

@@ -8,7 +8,7 @@ permalink: /performance/
 
 # Performance
 
-OpaqueId is designed for high performance with optimized algorithms and efficient memory usage. This guide covers benchmarks, optimization strategies, and scalability considerations.
+OpaqueId is designed for efficient ID generation with optimized algorithms and memory usage. This guide covers performance characteristics, optimization strategies, and scalability considerations.
 
 - TOC
   {:toc}
@@ -17,11 +17,11 @@ OpaqueId is designed for high performance with optimized algorithms and efficien
 
 ### Generation Speed
 
-OpaqueId is designed for high performance with optimized algorithms for different alphabet sizes and ID lengths.
+OpaqueId is designed for efficient ID generation with optimized algorithms for different alphabet sizes and ID lengths.
 
 #### Algorithm Performance
 
-- **Fast Path (64-character alphabets)**: Uses bitwise operations for maximum speed with no rejection sampling overhead
+- **Fast Path (64-character alphabets)**: Uses bitwise operations for efficient generation with no rejection sampling overhead
 - **Unbiased Path (Other alphabets)**: Uses rejection sampling for unbiased distribution with slight performance overhead
 - **Performance scales linearly** with ID length and batch size