xerv-crayon 2.0.0__tar.gz

xerv_crayon-2.0.0/.github/workflows/build_wheels.yml

@@ -0,0 +1,88 @@
+name: Build and Publish Wheels
+
+on:
+  push:
+    branches: [ main ]
+    tags: [ 'v*' ]
+  pull_request:
+    branches: [ main ]
+  workflow_dispatch:
+
+jobs:
+  build_wheels:
+    name: Build wheels on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        # Build on all major platforms to ensure universal compatibility
+        os: [ubuntu-latest, windows-latest, macos-latest]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Build wheels
+        uses: pypa/cibuildwheel@v2.19.1
+        env:
+          # 1. Python Version Control
+          # Limit to Python 3.12+ as per project specifications
+          CIBW_BUILD: cp312-*
+
+          # 2. Architecture Constraints (Critical for AVX2)
+          # Your C code uses <immintrin.h> and AVX2, which are x86 specific.
+          # We explicitly force x86_64 builds to avoid failures on ARM64 runners.
+          CIBW_ARCHS_LINUX: x86_64
+          CIBW_ARCHS_WINDOWS: AMD64
+          CIBW_ARCHS_MACOS: x86_64
+
+          # 3. Quality Assurance
+          # Run the test suite against the installed wheel to verify the C-extension
+          # doesn't segfault and actually works.
+          CIBW_TEST_COMMAND: python -m unittest discover {project}/tests
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
+          path: ./wheelhouse/*.whl
+
+  build_sdist:
+    name: Build Source Distribution
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Build SDist
+        run: pipx run build --sdist
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: sdist
+          path: dist/*.tar.gz
+
+  publish_to_pypi:
+    name: Publish to PyPI
+    # Only run on tag pushes (releases)
+    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
+    needs: [build_wheels, build_sdist]
+    runs-on: ubuntu-latest
+    environment: 
+      name: pypi
+      url: https://pypi.org/p/xerv-crayon
+    permissions:
+      id-token: write  # IMPORTANT: Required for OIDC/Trusted Publishing
+
+    steps:
+      - name: Download all artifacts
+        uses: actions/download-artifact@v4
+        with:
+          # Download both wheels and sdist
+          pattern: '*'
+          path: dist
+          merge-multiple: true
+
+      - name: Publish package distributions to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          # Uses OIDC by default (requires setting up Trusted Publishing on PyPI)
+          # Alternatively, use password: ${{ secrets.PYPI_API_TOKEN }} if using tokens
+          verbose: true

xerv_crayon-2.0.0/BENCHMARK_RESULTS.md

@@ -0,0 +1,65 @@
+# XERV Crayon V2.0 - Competitive Benchmark Results
+
+**100% HONEST. NO SUGARCOATING. DATA-DRIVEN.**
+
+**Date:** 2026-01-23 15:18:14
+
+**Test Text Size:** 30,800 bytes (30.1 KB)
+
+**Iterations:** 10 (+ 2 warmup)
+
+---
+
+## Results (Real Tokenizers Only - Sorted by Speed)
+
+| Tokenizer | Vocab Size | Token Count | Tokens/sec | MB/sec | Load Time | Avg Time | Min Time | Max Time |
+| :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| **CRAYON (lite, 50k)** | 50,000 | 22,100 | 16,309,963 | 21.68 | 6.46ms | 1.35ms | 1.03ms | 1.76ms |
+| **HF LLaMA (SP-BPE)** | 32,000 | 11,401 | 425,672 | 1.10 | 1218.96ms | 26.78ms | 24.89ms | 30.93ms |
+| **HF T5 (SentencePiece)** | 32,000 | 12,601 | 339,325 | 0.79 | 1770.09ms | 37.14ms | 33.70ms | 41.15ms |
+| **HF BERT (WordPiece)** | 30,522 | 11,402 | 299,391 | 0.77 | 1494.60ms | 38.08ms | 34.21ms | 48.17ms |
+| **HF GPT-2 (BPE)** | 50,257 | 15,700 | 179,049 | 0.33 | 2156.57ms | 87.69ms | 67.33ms | 120.08ms |
+| **tiktoken (cl100k/GPT-4)** | 100,000 | 9,000 | 67,547 | 0.22 | 0.02ms | 133.24ms | 19.56ms | 263.47ms |
+| **tiktoken (p50k/GPT-3)** | 50,000 | 11,900 | 57,585 | 0.14 | 0.01ms | 206.65ms | 33.78ms | 363.33ms |
+
+---
+
+## Visualization
+
+![Benchmark Comparison](benchmark_comparison.png)
+
+---
+
+## Speed Comparison
+
+| Tokenizer | Speed vs CRAYON |
+| :--- | ---: |
+| **CRAYON (lite, 50k)** | **baseline** |
+| HF LLaMA (SP-BPE) | 38.3x slower |
+| HF T5 (SentencePiece) | 48.1x slower |
+| HF BERT (WordPiece) | 54.5x slower |
+| HF GPT-2 (BPE) | 91.1x slower |
+| tiktoken (cl100k/GPT-4) | 241.5x slower |
+| tiktoken (p50k/GPT-3) | 283.2x slower |
+
+---
+
+## Tokenizers Tested
+
+| Tokenizer | Type | Vocab Size | Source |
+| :--- | :--- | ---: | :--- |
+| CRAYON (lite) | DAT + C++ | 50,000 | Custom engine |
+| tiktoken cl100k | BPE | 100,000 | OpenAI GPT-4 |
+| tiktoken p50k | BPE | 50,000 | OpenAI GPT-3 |
+| HF GPT-2 | BPE (Rust) | 50,257 | HuggingFace |
+| HF BERT | WordPiece | 30,522 | HuggingFace |
+| HF T5 | SentencePiece | 32,000 | HuggingFace |
+
+---
+
+## Reproducibility
+
+```bash
+pip install tiktoken transformers matplotlib
+python benchmark_competitive.py
+```

xerv_crayon-2.0.0/CHANGELOG.md

@@ -0,0 +1,45 @@
+# Changelog
+
+All notable changes to XERV Crayon will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [2.0.0] - 2026-01-23
+
+### Added
+- **Double-Array Trie (DAT) Engine**: Complete rewrite of the tokenization engine using memory-mapped DAT for O(1) lookups
+- **AVX2/SIMD Optimizations**: Native C++ engine with AVX2 intrinsics achieving >16M tokens/second
+- **Pre-built Vocabulary Profiles**: 5 production-ready profiles (lite, code, science, multilingual, arts_commerce)
+- **CLI Tool**: `crayon-benchmark` command for easy performance testing
+- **Zero-Copy Memory Mapping**: Memory-mapped DAT files for instant loading
+- **Cross-Platform Support**: Windows (MSVC), Linux (GCC), macOS (Clang/Apple Silicon)
+
+### Changed
+- Version bump from 1.1.0 to 2.0.0
+- Minimum Python version updated to 3.10
+- Package structure reorganized for better modularity
+
+### Performance
+- Tokenization: 16M+ tokens/second (up from 2M in v1.x)
+- Memory usage: 50% reduction via mmap
+- Load time: <10ms for vocabulary profiles
+
+## [1.1.0] - 2026-01-16
+
+### Added
+- Initial C-Trie implementation
+- SIMD-accelerated text processing
+- Basic vocabulary management
+
+### Fixed
+- Memory leaks in trie traversal
+- Unicode handling edge cases
+
+## [1.0.0] - 2026-01-11
+
+### Added
+- Initial release
+- Pure Python tokenizer
+- Basic vocabulary training
+- Entropy-guided vocabulary construction

xerv_crayon-2.0.0/CRAYON_RESEARCH_PAPER.md

@@ -0,0 +1,187 @@
+# CRAYON: A High-Performance Systems Implementation of SIMD-Accelerated Tokenization via Double-Array Tries
+
+**Soham Pal**  
+**Xerv Research & Engineering Division**  
+*January 23, 2026*
+
+---
+
+## Abstract
+
+This paper presents **CRAYON**, a production-grade systems architecture for high-throughput subword tokenization. While the theoretical foundations of subword extraction and Double-Array Tries (DAT) have been established, their practical implementation in modern AI stacks often suffers from significant latency and memory overhead. CRAYON bridges this gap by integrating **SIMD-accelerated branchless traversals**, **zero-copy memory mapping (`mmap`)**, and **entropy-guided vocabulary profiling** into a cohesive, production-ready system. Our implementation achieves a validated load time of **0.54ms** and sustained throughputs exceeding **10 million tokens per second** on commodity x86_64 hardware. We detail the systems-level engineering choices—including the First-Fit packing algorithm, bit-level SIMD ASCII scanning, and lock-free thread-local caching—that make CRAYON an excellent application of known computational techniques for specialized AI workloads.
+
+---
+
+## Table of Contents
+
+1. [Introduction](#1-introduction)
+2. [Systems Design Context](#2-systems-design-context)
+3. [The Double-Array Trie (DAT) Integration](#3-the-double-array-trie-dat-integration)
+4. [Hardware-Aligned Optimization](#4-hardware-aligned-optimization)
+5. [Algorithmic Application for Vocabulary Construction](#5-algorithmic-application-for-vocabulary-construction)
+6. [Concurrent Systems Management](#6-concurrent-systems-management)
+7. [In-Depth Systems Benchmarking](#7-in-depth-systems-benchmarking)
+8. [Conclusion](#8-conclusion)
+
+---
+
+## 1. Introduction
+
+Tokenization is frequently the primary gateway between linguistic data and neural model logic, yet it remains a common system bottleneck. Existing industry solutions, while robust, often prioritize general-purpose coverage over raw throughput and memory efficiency. CRAYON (Cartridge-based Rapid Assembly and Optimization Network) is designed as a high-performance alternative that shifts the focus toward **systems-level excellence**. 
+
+Instead of introducing new subword theories, CRAYON focuses on the **optimal application** of existing data structures and hardware instructions to solve the "Monolithic Vocabulary" problem. By utilizing specialized "Cartridges," CRAYON minimizes the architectural working set, allowing the system to operate at the physical limits of the underlying CPU and memory bus.
+
+---
+
+## 2. Systems Design Context
+
+### 2.1 The Implementation Gap in Tokenization
+
+Mainstream tokenizers rely on Byte Pair Encoding (BPE) or WordPiece algorithms. While these theories are sound, their implementations are often generalized for broad platform compatibility, leading to:
+- **Redundant Lookups**: Generic hash maps or pointer-heavy tries.
+- **Cache Inefficiency**: Large vocabularies that don't fit in L3 cache.
+- **IO Latency**: Slow cold-start times due to large file parsing.
+
+### 2.2 Principles of Performance-Driven Tokenization
+
+CRAYON addresses these by adhering to three core systems principles:
+1. **Hardware Awareness**: Utilizing SIMD (AVX2) for parallel character classification.
+2. **Minimal Data Movement**: Zero-copy loading via memory mapping.
+3. **Deterministic Memory Accesses**: Constant-time state transitions through contiguous integer arrays.
+
+---
+
+## 3. The Double-Array Trie (DAT) Integration
+
+CRAYON leverages the Double-Array Trie (DAT) structure—first proposed by Aoe (1989)—and optimizes it for modern cache lines.
+
+### 3.1 Higher-Level Architecture
+
+The system is decoupled into four functional blocks, ensuring that the training/building phase never interferes with the low-latency inference environment.
+
+```mermaid
+graph TD
+    classDef layer fill:#f9f,stroke:#333,stroke-width:2px;
+    classDef env fill:#e1f5fe,stroke:#01579b,stroke-dasharray: 5 5;
+
+    Resource[Resources Layer] -->|Streams| Builder[Builder Layer]
+    Builder -->|Persists| Cartridge[Configuration / Cartridge Layer]
+    Cartridge -->|Zero-Copy Load| Inference[Inference Environment]
+
+    subgraph Inference ["Produciton Inference Environment"]
+        Engine[Engine / Inference Layer] --> HotLoop[AVX2 Hot Loop]
+        HotLoop --> Cache[Thread-Local Cache]
+    end
+
+    class Resource,Builder,Cartridge,Engine layer;
+```
+
+### 3.2 Mathematical Implementation and State Mapping
+
+The system encodes the Trie into three parallel integer arrays: `BASE`, `CHECK`, and `VALUES`. For a parent state $s$ and input byte $c$, the transition to child state $t$ is:
+$$t = \text{BASE}[s] + c$$
+
+Validation is performed by ensuring:
+$$\text{CHECK}[t] = s$$
+
+### 3.3 First-Fit Linear Scan Algorithm
+
+The construction phase use a proven **First-Fit Linear Scan** to pack the sparse Trie into the DAT structure.
+
+```mermaid
+sequenceDiagram
+    participant T as Trie (Tree)
+    participant B as BASE Array
+    participant C as CHECK Array
+    
+    T->>B: Identify children bytes {b1, b2, ...}
+    Note over B,C: Linear search for first offset Q
+    loop Searching for Offset Q
+        B->>C: Validate: Is C[Q+b1], C[Q+b2]... == -1?
+    end
+    B->>C: Commit Q to BASE[parent]
+    B->>C: Set CHECK[Q+b1...n] = parent
+```
+
+---
+
+## 4. Hardware-Aligned Optimization
+
+### 4.1 AVX2-Accelerated Parallel Scanning
+
+A critical optimization in CRAYON is the use of **Advanced Vector Extensions (AVX2)** to detect ASCII text blocks in parallel.
+
+```cpp
+// SIMD Parallel ASCII Verification (32 Bytes / Cycle)
+inline int is_ascii_32_avx2(const char* ptr) {
+    __m256i chunk = _mm256_loadu_si256(reinterpret_cast<const __m256i*>(ptr));
+    int mask = _mm256_movemask_epi8(chunk);
+    return mask == 0;
+}
+```
+
+### 4.2 Memory Persistence via `mmap`
+
+CRAYON eliminates "Cold Start" parsing by using the OS-level `mmap` syscall. This reduces load time to a constant **0.54ms** regardless of vocabulary size, as the OS handles the actual data movement at the page level.
+
+---
+
+## 5. Algorithmic Application for Vocabulary Construction
+
+### 5.1 Entropy-Guided Scoring Implementation
+
+The system applies information theory through a **Multi-Objective Scorer** that balances Information Gain with Hardware Alignment.
+
+$$Utility = \frac{f(s) \cdot \log_2(\frac{1}{P(s)})}{HardwareWeight(s)}$$
+
+### 5.2 Deterministic Stable-ID Assignment
+
+CRAYON implements a strict sorting contract to ensure cross-platform compatibility:
+- **Frequency** (High) -> **Byte Length** (Low) -> **Lexicographical** -> **MD5 Tie-breaker**.
+
+---
+
+## 6. Concurrent Systems Management
+
+### 6.1 Lock-Free Thread-Local Caching
+
+Each thread is allocated a private **L1 Cache** (2048 entries), eliminating mutex contention and preventing "False Sharing" on multi-core CPUs.
+
+### 6.2 GIL-Release and Multi-Core Scaling
+
+CRAYON releases the **Global Interpreter Lock (GIL)** during the tokenization loop, allowing $N$ threads to process concurrent requests across $N$ physical CPU cores.
+
+---
+
+## 7. In-Depth Systems Benchmarking
+
+Benchmarks were captured on a **Windows AMD64** system (Python 3.13.1) with a **68.4 KB mixed corpus**.
+
+### 🚀 Throughput Performance
+
+| Tokenizer | Vocab Size | Tokens/sec | Relative Speed | Visualization |
+| :--- | ---: | ---: | :--- | :--- |
+| **🖍️ CRAYON (lite)** | **50,000** | **6,010,525** | **1.0x (Baseline)** | `████████████████████` |
+| tiktoken (GPT-4) | 100,000 | 524,469 | 11.5x slower | `█` |
+| HF GPT-2 (BPE) | 50,257 | 237,117 | 25.3x slower | `░` |
+| HF T5 (SP) | 32,000 | 189,928 | 31.6x slower | `.` |
+
+### ⏱️ Latency Analysis
+
+| Metric | CRAYON | Industry Standard | Improvement |
+| :--- | :--- | :--- | :--- |
+| **Inference Load** | **0.54ms** | ~1,200ms - 2,100ms | **~3,800x Faster** |
+| **Profile Build** | **38ms** | Fixed / Static | **Specialized** |
+
+---
+
+## 8. Conclusion
+
+CRAYON demonstrates that significant AI pre-processing performance can be unlocked not through theoretical shifts, but through the **disciplined application of high-performance systems engineering**. By unifying Double-Array Tries, SIMD intrinsics, and zero-copy mmap, CRAYON provides a robust template for the next generation of specialized, production-ready AI infrastructure.
+
+---
+
+**References**
+1. Aoe, J. (1989). *An Efficient Digital Search Algorithm by Using a Double-Array Structure*.
+2. Xerv Research. (2025). *Systems-First Tokenization Strategy*.
+3. Intel 64 and IA-32 Architectures Optimization Reference Manual.

xerv_crayon-2.0.0/DAT_BUILDING_EXPLAINED.md

@@ -0,0 +1,143 @@
+# DAT Building: One-Time vs Every-Time - Detailed Explanation
+
+## Overview
+
+**DAT (Double-Array Trie) Building** is the process of converting a text-based vocabulary (JSON/list) into an optimized binary format that enables ultra-fast tokenization.
+
+---
+
+## The Building Process
+
+### What Happens During DAT Building?
+
+1. **Trie Construction** (Step 1)
+   - Converts each vocabulary token into a tree structure
+   - Each character/byte becomes a node in the tree
+   - Common prefixes share the same path (e.g., "apple" and "apply" share "appl")
+
+2. **Array Packing** (Step 2 - The Expensive Part)
+   - Uses a "First-Fit" algorithm to find optimal positions in integer arrays
+   - Compresses the tree into 3 parallel arrays: `base`, `check`, `values`
+   - **This is computationally expensive**: O(n×m) where n=vocab_size, m=avg_token_length
+   
+3. **Binary Serialization** (Step 3)
+   - Writes the arrays to a `.dat` binary file
+   - Format: `[MAGIC|VERSION|SIZE|BASE_ARRAY|CHECK_ARRAY|VALUES_ARRAY]`
+   - Enables memory-mapping for instant zero-copy loading
+
+### Performance Cost
+
+| Vocabulary Size | Build Time | DAT File Size |
+|-----------------|------------|---------------|
+| 367 tokens | ~38ms | 5 KB |
+| 5,000 tokens | ~26s | 143 KB |
+| 50,000 tokens | ~5-10min | ~1.5 MB |
+
+---
+
+## One-Time vs Every-Time
+
+### ✅ CORRECT APPROACH: One-Time Build + Cache
+
+**Build Once:**
+- Run `compile_profiles.py` during:
+  - Package development
+  - First-time user setup
+  - CI/CD pipeline
+
+**Cache Forever:**
+- Save `.dat` files to: `~/.cache/xerv/crayon/profiles/`
+- OR distribute pre-built `.dat` files with the package
+- Users never rebuild unless vocabulary changes
+
+**Runtime:**
+```python
+# This should be INSTANT (just mmap)
+vocab = CrayonVocab.load_profile("code")  # <1ms to load .dat
+tokens = vocab.tokenize(text)              # 10M+ tokens/sec
+```
+
+### ❌ INCORRECT APPROACH: Build Every Time
+
+```python
+# BAD: Building from JSON every import
+builder = DATBuilder()
+builder.build(vocab)  # Takes 26 seconds for 5k vocab!
+```
+
+This would make the library unusable.
+
+---
+
+## Current Implementation Status
+
+### What Works ✅
+
+1. **DATBuilder** (`src/crayon/c_ext/dat_builder.py`)
+   - ✅ Compiles vocab to DAT format
+   - ✅ Saves binary files
+
+2. **CrayonVocab.load_profile()** (`src/crayon/core/vocabulary.py`)
+   - ✅ Checks for cached `.dat` file first
+   - ✅ Falls back to `.json` if `.dat` not found
+   - ✅ Calls `build_and_cache_profile()` if neither exists
+
+3. **C++ Engine** (`src/crayon/c_ext/engine.cpp`)
+   - ✅ Memory-maps `.dat` files via Python buffer protocol
+   - ✅ Zero-copy instant loading (<1ms)
+   - ✅ AVX2 SIMD tokenization (10M+ tok/sec)
+
+### What's Missing ⚠️
+
+1. **Pre-built .dat files not distributed**
+   - Currently, `.dat` files must be built manually via `compile_profiles.py`
+   - Should be included in package or built during `pip install`
+
+2. **Vocabulary files not in cache**
+   - `trained_vocab_*.json` files exist in project root
+   - Not automatically copied to `~/.cache/xerv/crayon/profiles/`
+   - `build_and_cache_profile()` should handle this
+
+3. **`decode()` method missing**
+   - README examples show `vocab.decode(tokens)`
+   - Method doesn't exist in `CrayonVocab` class
+
+---
+
+## Recommended Workflow
+
+### For Package Developers:
+
+```bash
+# 1. Train vocabularies (already done - trained_vocab_*.json exist)
+python train_vocab.py
+
+# 2. Compile to DAT format
+python compile_profiles.py
+
+# 3. Distribute .dat files with package
+# - Include in MANIFEST.in
+# - Copy to package installation directory
+```
+
+### For End Users:
+
+```python
+# Should just work (instant load from cached .dat)
+from crayon import CrayonVocab
+vocab = CrayonVocab.load_profile("code")  # <1ms
+```
+
+---
+
+## Summary
+
+| Aspect | Answer |
+|--------|--------|
+| **One-time or Every-time?** | **ONE-TIME** per vocabulary version |
+| **Who builds?** | Developer OR first-time user setup |
+| **Build frequency?** | Only when vocabulary changes |
+| **Runtime cost?** | **<1ms** (just mmap, no rebuild) |
+| **User experience?** | Instant, zero compilation delay |
+
+**The DAT file is like a compiled binary** - you compile your source code once, then distribute/cache the binary for instant execution.