blusher 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d4bf0252e2f945302dbf33bdeb00d4f3dad51c25be02a8ff2996afbcec745cc0
+  data.tar.gz: 03e7eef5f6093112f7d4b37a5d1647017c83b72560740a53d8f5d6365492a504
+SHA512:
+  metadata.gz: 676f07b45c6dfe42b28cb5b476bb9d179b238de517106e73e2c47afbe87a7fe1c63abc84a31f9cd8401de673b0542491ec441879bfbde7a46bfcb85aea5bd1a1
+  data.tar.gz: c3a1cc14b37d479369af5ac790c76b93c2c233c482a3463d2eb29ce1920e46745f4190158a30c8d08261337566bdc03a110c6115428133e61a4d61adb03ccafc

data/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+## 0.1.0 (unreleased)
+
+Initial release.
+
+- Drop-in alternative lexing backend for rouge: `require "blusher"` routes
+  `Rouge::RegexLexer#lex` through the Rust `carmine` engine, with byte-identical
+  output or transparent fallback to rouge.
+- Native engine via an rb-sys/magnus extension (builds the `[Token, value]`
+  pair array directly as Ruby objects); a Fiddle-loaded `carmine-ffi` cdylib
+  is kept as a dependency-light fallback.
+- Callback-free **routability allowlist**: lexers whose tables contain rouge
+  `proc` rules carmine can't execute are skipped up front, so they never pay a
+  wasted native-lex attempt.
+- **Fused lex+format HTML path**: for an unadorned `Rouge::Formatters::HTML`
+  pipeline (`Rouge.highlight`/Jekyll/kramdown default), `lex` returns a deferred
+  token stream and the patched `HTML#format` lexes AND formats in Rust, returning
+  one String — crossing the Ruby boundary once instead of per token. Other
+  formatters, the block form, and direct token consumers fall back to rouge.
+- Thread-local cache of parsed/compiled tables (the ~0.5 ms build no longer
+  repeats per call).
+- Correctness: rouge v5.0.0 full lexer spec suite — 757 runs, 5130 assertions,
+  0 failures; fused HTML output byte-identical across all 126 routable visual
+  samples + rouge's HTML formatter specs.
+- Performance: ~1.7× faster highlighting to HTML on a mixed real corpus
+  (2.5–2.7× on individual files), parity for callback lexers / non-HTML output.
+  (A token stream alone is only ~1.0×: Ruby object allocation at the boundary
+  dominates and both engines pay it — fusing the HTML output is what wins.)

data/Cargo.lock

@@ -0,0 +1,373 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 4
+
+[[package]]
+name = "aho-corasick"
+version = "1.1.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "bindgen"
+version = "0.72.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895"
+dependencies = [
+ "bitflags",
+ "cexpr",
+ "clang-sys",
+ "itertools",
+ "proc-macro2",
+ "quote",
+ "regex",
+ "rustc-hash",
+ "shlex",
+ "syn",
+]
+
+[[package]]
+name = "bit-set"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3"
+dependencies = [
+ "bit-vec",
+]
+
+[[package]]
+name = "bit-vec"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7"
+
+[[package]]
+name = "bitflags"
+version = "2.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8"
+
+[[package]]
+name = "blusher"
+version = "0.1.0"
+dependencies = [
+ "carmine",
+ "magnus",
+]
+
+[[package]]
+name = "carmine"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "415476756c3ae81a94343572754fe6f11a5f5a7a1ead77956327e0afbc5e2934"
+dependencies = [
+ "fancy-regex",
+ "regex-automata",
+ "serde_json",
+]
+
+[[package]]
+name = "cexpr"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766"
+dependencies = [
+ "nom",
+]
+
+[[package]]
+name = "cfg-if"
+version = "1.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
+
+[[package]]
+name = "clang-sys"
+version = "1.8.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4"
+dependencies = [
+ "glob",
+ "libc",
+ "libloading",
+]
+
+[[package]]
+name = "either"
+version = "1.16.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e"
+
+[[package]]
+name = "fancy-regex"
+version = "0.18.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e1e1dacd0d2082dfcf1351c4bdd566bbe89a2b263235a2b50058f1e130a47277"
+dependencies = [
+ "bit-set",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "glob"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280"
+
+[[package]]
+name = "itertools"
+version = "0.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"
+dependencies = [
+ "either",
+]
+
+[[package]]
+name = "itoa"
+version = "1.0.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
+
+[[package]]
+name = "lazy_static"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
+
+[[package]]
+name = "libc"
+version = "0.2.186"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66"
+
+[[package]]
+name = "libloading"
+version = "0.8.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55"
+dependencies = [
+ "cfg-if",
+ "windows-link",
+]
+
+[[package]]
+name = "magnus"
+version = "0.7.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3d87ae53030f3a22e83879e666cb94e58a7bdf31706878a0ba48752994146dab"
+dependencies = [
+ "magnus-macros",
+ "rb-sys",
+ "rb-sys-env",
+ "seq-macro",
+]
+
+[[package]]
+name = "magnus-macros"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5968c820e2960565f647819f5928a42d6e874551cab9d88d75e3e0660d7f71e3"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
+[[package]]
+name = "memchr"
+version = "2.8.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
+
+[[package]]
+name = "minimal-lexical"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
+
+[[package]]
+name = "nom"
+version = "7.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
+dependencies = [
+ "memchr",
+ "minimal-lexical",
+]
+
+[[package]]
+name = "proc-macro2"
+version = "1.0.106"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
+dependencies = [
+ "unicode-ident",
+]
+
+[[package]]
+name = "quote"
+version = "1.0.45"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
+dependencies = [
+ "proc-macro2",
+]
+
+[[package]]
+name = "rb-sys"
+version = "0.9.128"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "45ca28513560e56cfb79a62b1fce363c73af170a182024ce880c77ee9429920a"
+dependencies = [
+ "rb-sys-build",
+]
+
+[[package]]
+name = "rb-sys-build"
+version = "0.9.128"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ce04b2c55eff3a21aaa623fcc655d94373238e72cac6b3e1a3641ff31649f99a"
+dependencies = [
+ "bindgen",
+ "lazy_static",
+ "proc-macro2",
+ "quote",
+ "regex",
+ "shell-words",
+ "syn",
+]
+
+[[package]]
+name = "rb-sys-env"
+version = "0.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a35802679f07360454b418a5d1735c89716bde01d35b1560fc953c1415a0b3bb"
+
+[[package]]
+name = "regex"
+version = "1.12.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.4.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-syntax"
+version = "0.8.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
+
+[[package]]
+name = "rustc-hash"
+version = "2.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
+
+[[package]]
+name = "seq-macro"
+version = "0.3.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc"
+
+[[package]]
+name = "serde"
+version = "1.0.228"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
+dependencies = [
+ "serde_core",
+]
+
+[[package]]
+name = "serde_core"
+version = "1.0.228"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
+dependencies = [
+ "serde_derive",
+]
+
+[[package]]
+name = "serde_derive"
+version = "1.0.228"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
+[[package]]
+name = "serde_json"
+version = "1.0.150"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9"
+dependencies = [
+ "itoa",
+ "memchr",
+ "serde",
+ "serde_core",
+ "zmij",
+]
+
+[[package]]
+name = "shell-words"
+version = "1.1.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "dc6fe69c597f9c37bfeeeeeb33da3530379845f10be461a66d16d03eca2ded77"
+
+[[package]]
+name = "shlex"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
+
+[[package]]
+name = "syn"
+version = "2.0.118"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "unicode-ident",
+]
+
+[[package]]
+name = "unicode-ident"
+version = "1.0.24"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
+
+[[package]]
+name = "windows-link"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
+
+[[package]]
+name = "zmij"
+version = "1.0.21"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"

data/Cargo.toml

@@ -0,0 +1,6 @@
+# Workspace root for the gem's native extension. rb-sys/rake-compiler runs
+# `cargo metadata` from here to discover the `blusher` crate in ext/blusher,
+# and a committed Cargo.lock makes install-time builds reproducible.
+[workspace]
+members = ["ext/blusher"]
+resolver = "2"

data/LICENSE

@@ -0,0 +1,25 @@
+MIT License
+
+Copyright (c) 2026 momiji-rs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+Lexer rule tables under lib/blusher/tables/ are derived from rouge
+(https://github.com/rouge-ruby/rouge), MIT-licensed, © Jeanine Adkisson and
+contributors.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,98 @@
+# blusher
+
+A **faster, drop-in, byte-for-byte-compatible** alternative backend for Ruby's
+[rouge](https://github.com/rouge-ruby/rouge), powered by the Rust
+[`carmine`](https://crates.io/crates/carmine) engine.
+
+`require "blusher"` routes rouge's lexing — and, for the common case, its HTML
+formatting — through carmine, which executes rule tables extracted from rouge's
+own lexers. carmine either produces **byte-identical** output or **declines**,
+in which case blusher falls back to rouge unchanged. Zero code change, zero
+divergence; **~1.7× faster** highlighting to HTML on a mixed corpus (more on
+large files), parity elsewhere.
+
+```ruby
+require "rouge"
+require "blusher"   # ← that's it
+
+# The hot path: lexing + HTML formatting fused in Rust, one String returned.
+html = Rouge.highlight(File.read("data.json"), "json", "html")
+```
+
+## Performance
+
+**For the HTML-highlighting path — what rouge is overwhelmingly used for —
+blusher is ~1.7× faster on a mixed real corpus and 2.5–2.7× on individual
+files**, with byte-identical output. Measured over rouge's own 126 routable
+visual sample files (690 KiB) rendered to HTML with `Rouge::Formatters::HTML`:
+
+```
+              ms/pass    MB/s
+  rouge        187.1      3.8
+  blusher      110.2      6.4     → 1.70×
+```
+
+The trick is **not** faster lexing in isolation. A Ruby lexer's cost is
+dominated by allocating Ruby objects at the boundary — one String + one Array
+per token — which both engines pay identically, so `lex` alone is only ~1.0×
+even though carmine's core is ~4.6× faster Rust-to-Rust. But a token stream is
+just an *intermediate*: the real output is an HTML string. So blusher **fuses
+lexing and HTML formatting in Rust and returns one String**, crossing the Ruby
+boundary once (O(1)) instead of once per token (O(n)). That is where carmine's
+speed finally shows up end-to-end. (Two supporting wins: a thread-local cache of
+the parsed/compiled table so the ~0.5 ms build doesn't repeat per call, and
+resolving CSS class names in Rust.)
+
+Scope and honesty:
+
+- The fast path applies when the whole pipeline is `format(lex(src))` with an
+  **unadorned `Rouge::Formatters::HTML`** (the `Rouge.highlight` / Jekyll /
+  kramdown default). Subclasses (HTMLInline, HTMLTable, Pygments, …), the
+  token-streaming block form, and direct token consumers transparently fall
+  back to rouge — same output, no speedup.
+- It applies to the **126 of 227 callback-free lexers** (JSON, SQL, YAML, CSS,
+  many config/markup formats). The other 101 use rouge `proc` rules carmine
+  can't execute; blusher detects this up front and runs rouge for them (parity,
+  no wasted work). So a JSON/SQL/config-heavy workload wins big; a
+  Ruby/Python-heavy one trends toward parity.
+- carmine's raw 4.6× is fully realized only Rust-to-Rust (e.g. embedded in
+  [rubyrs](https://github.com/linyiru/rubyrs), no Ruby boundary at all).
+
+## How it works
+
+- `require "blusher"` aliases the original `Rouge::RegexLexer#lex` to
+  `__blusher_rouge_lex` and replaces it. For a routable lexer (table exists and
+  is callback-free), `lex` without a block returns a deferred
+  `Blusher::Shim::TokenStream` holding `(lexer, source)` — nothing is lexed yet.
+- The patched `Rouge::Formatters::HTML#format` recognises that stream and calls
+  the **fused** `Blusher::Engine.format_html`, which lexes *and* formats in Rust
+  and returns one HTML String. Any other consumer (a different formatter, the
+  block form, `.to_a`) just iterates the stream, which lexes via carmine on
+  demand and yields the same `[Token, value]` pairs rouge would.
+- carmine **declines** anything it can't reproduce identically (callback rules,
+  recursion, …) and blusher falls back to rouge, so the output is always exactly
+  rouge's — verified against the full lexer spec suite.
+- The native backend is the **rb-sys/magnus extension** (`blusher.{bundle,so}`).
+  A `carmine-ffi` + Fiddle path is kept as a dependency-light fallback (it
+  marshals tokens through JSON, can't fuse, and is for correctness only).
+
+## Correctness
+
+Verified against rouge v5.0.0's **full lexer spec suite: 757 runs, 5130
+assertions, 0 failures** (`rake spec`). The spec suite is the correctness gate —
+any new divergence must be fixed in carmine or the rule forced to decline.
+
+## Build (dev, in the rubyrs monorepo)
+
+```sh
+rake compile      # build the magnus ext → lib/blusher.<dlext>
+rake compile_ffi  # (optional) build the carmine-ffi cdylib fallback → ext/
+rake tables       # regenerate lib/blusher/tables/<tag>.json from installed rouge
+ROUGE_SRC=/path/to/rouge rake spec
+```
+
+## Status
+
+Part of [momiji-rs](https://github.com/momiji-rs) — Rust-backed engines for the
+Ruby ecosystem. Tables are derived from rouge (MIT, © Jeanine Adkisson and
+contributors).

data/ext/blusher/Cargo.toml

@@ -0,0 +1,18 @@
+# The gem-internal native extension, compiled at `gem install` time by
+# rb-sys/rake-compiler (and cross-compiled in CI for precompiled platform
+# gems — see .github/workflows/release.yml). Mirrors crates/blusher-ext in the
+# rubyrs monorepo, but depends on the PUBLISHED carmine from crates.io so the
+# standalone momiji-rs/blusher repo builds without the workspace.
+[package]
+name = "blusher"
+version = "0.1.0"
+edition = "2021"
+license = "MIT"
+publish = false
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+carmine = "0.3"
+magnus = "0.7"

data/ext/blusher/extconf.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+# rb-sys driven build: rake-compiler invokes this at `gem install`, and
+# `create_rust_makefile` emits a Makefile that runs `cargo build` and installs
+# the cdylib as the gem's loadable object. The argument names the output so it
+# lands at `lib/blusher/blusher.<dlext>` and is required as "blusher/blusher".
+require "mkmf"
+require "rb_sys/mkmf"
+
+create_rust_makefile("blusher/blusher")