antares 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a1968f47a0c8ea17a5e98fa2a3b6d4c339d2dcda58d4ae5032c7799d75ed17a6
-  data.tar.gz: 70e780331521d82f0dcbbe1ca6aa22de7f8ff7c2736eaca03a3e458ddb0e71f7
+  metadata.gz: 0243ed5a88d91e34cf4654f9e3437d590e9079caea1976fde034213dbf28ab8c
+  data.tar.gz: 77b2751092cc3f1fb5dd9aaf7dc0615c724512d958f1bfe7a198bc7cad70b463
 SHA512:
-  metadata.gz: 1ccfedd6fce3a1095aa81da6bd5f2285c7757a8dba9138eef211fdac309d5b5309e1b514c82e7fd593e93a6a851d9edded3d80c165176357cc7b13bfa43c7040
-  data.tar.gz: 5c18ad134cbf9fb791f625b3354a592b80d1052374fdf8f17e5c01b7917dd0cf73e4c2be863265d6372476776f769d31a3b2090757fce5eb9629eef0fb5ea7be
+  metadata.gz: b24e5c58ad94c655378e327344a9404a73a5bf57185d44854f84ab831bd29857381c5117ee3de4ac00a5de044e2c23560c564cf0a16bd7aa62a369546dc805f1
+  data.tar.gz: b988ef69cdf0395af22403e427677ebef54b1fe84cbbeae4b4a2127e0b4c78a243ce8fde131acf6cd0fc79aa3828c078b7dcb6057cb376bde3abcd1ee646c907

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.2.0 — 2026-09-15
+
+- Add edit-aware bracket matching and depth information.
+- Add fold regions for brackets, indentation, consecutive comments, and region markers.
+- Add sticky contexts and expanding selection ranges.
+- Add language-specific structure provider registration without a parser dependency.
+- Add bounded JSON and XML plist tmLanguage loading with Rouge-compatible output.
+
 ## 0.1.0 — 2026-09-10
 
 - Initial release.

data/README.md

@@ -1,8 +1,69 @@
-# Antares
+<h1 align="center">Antares</h1>
 
-Incremental syntax highlighting for Rouge, with measured per-lexer compatibility.
-The only runtime dependency is `rouge ~> 5.0`; document storage and colors belong
-to the caller. Ruby 3.1 or later is supported.
+<p align="center">
+  <strong>Incremental syntax highlighting for Rouge, with measured per-lexer compatibility</strong>
+</p>
+
+<p align="center">
+  <a href="https://rubygems.org/gems/antares"><img src="https://img.shields.io/gem/v/antares.svg?colorB=319e8c" alt="Gem Version"></a>
+  <a href="https://rubygems.org/gems/antares"><img src="https://img.shields.io/gem/dt/antares.svg" alt="Downloads"></a>
+  <img src="https://img.shields.io/badge/ruby-%3E%3D%203.1-ruby.svg" alt="Ruby Version">
+  <img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License">
+</p>
+
+<p align="center">
+  <a href="#features">Features</a> ·
+  <a href="#installation">Installation</a> ·
+  <a href="#quick-start">Quick Start</a> ·
+  <a href="#configuration">Configuration</a> ·
+  <a href="#how-it-works">How It Works</a> ·
+  <a href="#compatibility">Compatibility</a>
+</p>
+
+---
+
+Antares adds edit-aware, line-oriented syntax highlighting to [Rouge](https://github.com/rouge-ruby/rouge). It reuses lexer state where compatibility tests show that incremental highlighting is safe, and falls back to bounded re-highlighting for other lexers. Document storage and colors remain with the caller.
+
+## Features
+
+<a name="features"></a>
+
+- Lazy tokenization for individual lines or ranges
+- Incremental re-highlighting with sparse state checkpoints
+- Cached suffix reuse after edits converge
+- Measured compatibility for every Rouge lexer
+- Windowed and full-document fallback strategies
+- Configurable time and memory limits
+- Bracket pairs, fold regions, sticky contexts, and expanding selections
+- JSON and XML plist tmLanguage grammars for languages Rouge does not cover
+- Rouge and REXML are the only runtime dependencies
+
+## Installation
+
+<a name="installation"></a>
+
+Add to your Gemfile:
+
+```ruby
+gem "antares"
+```
+
+Then install:
+
+```bash
+bundle install
+```
+
+### Requirements
+
+<a name="requirements"></a>
+
+- Ruby 3.1+
+- Rouge 5.x
+
+## Quick Start
+
+<a name="quick-start"></a>
 
 ```ruby
 require "antares"
@@ -15,97 +76,160 @@ highlighter = Antares::Highlighter.new(
 )
 
 highlighter.tokens_for(0) # [[Rouge token class, UTF-8 text], ...]
+
 lines[0] = "value = 2\n"
 highlighter.edit(from_line: 0, removed: 1, inserted: 1)
 highlighter.tokens_in(0..1) # one token array per line
+
+structure = highlighter.structure # built lazily from the same Rouge tokens
+structure.bracket_at(0, 7)        # matching Antares::Bracket, or nil
+structure.fold_regions            # bracket, indentation, comment, and #region folds
+structure.context_at(1)           # outer-to-inner regions for sticky scroll
+structure.selection_ranges(1, 3)  # inner-to-outer Antares::Region values
 ```
 
-For a local checkout, `gem build antares.gemspec` creates an installable gem;
-install that file with `gem install ./antares-0.1.0.gem`. Publication is separate
-from building and is not performed by tests or CI.
+Update the provider before calling `edit`. Line indices are zero based;
+`removed` is the old line count and `inserted` is the new line count.
 
-## API and strategies
+The `lines` provider returns one valid UTF-8 logical line for each index,
+preferably including its newline. Tokens contain text rather than offsets: sum
+`text.bytesize` for byte offsets or `text.length` for character offsets.
+Returned rows, token pairs, and text are frozen.
 
-Line indices are zero based. Providers return one valid UTF-8 logical line each,
-preferably including its newline. Missing separators are added to non-final
-lines. A trailing empty logical line returns an empty token array. Tokens contain
-text, not offsets: sum `text.bytesize` for byte offsets, or `text.length` for
-character offsets. Returned rows/pairs/text are frozen.
+Bracket and selection columns are zero-based character offsets. A selection
+region's `end_column` is exclusive. Fold regions use inclusive, zero-based line
+indices; their column fields are `nil`. Brackets inside string and comment
+tokens are ignored. `Highlighter#edit` updates the existing `Structure`
+instance and reuses unchanged line data after token state converges.
 
-Call `edit` **after** updating the provider, with the old removed line count and
-new inserted line count. `frontier` is the first line not yet proven current;
-`advance(until_line: 200)` lets an event loop schedule incremental progress.
-`last_scanned_lines` reports actual lexical work, and `checkpoint_bytes` reports
+`frontier` is the first line not yet proven current.
+`advance(until_line: 200)` lets an event loop schedule incremental work.
+`last_scanned_lines` reports lexical work and `checkpoint_bytes` reports
 normalized checkpoint payload bytes. Highlighter instances are not thread safe.
 
+Language-specific structure providers can replace the generic derivation for
+future highlighters with the matching Rouge lexer tag:
+
+```ruby
+Antares::Structure.register(:ruby, RubyStructureProvider)
+# RubyStructureProvider.new receives the same five callbacks as Structure.new.
+Antares::Structure.unregister(:ruby)
+```
+
+A provider returns an object implementing `fold_regions`, `brackets`,
+`bracket_at`, `context_at`, `selection_ranges`, and `edit`. Registration is
+thread safe, case insensitive, and affects only structures built afterwards.
+Providers must reflect line changes before `Highlighter#edit`, just like the
+line provider. Antares does not depend on Prism.
+
+Load a UTF-8 JSON or XML plist tmLanguage file when Rouge has no lexer for the
+language, then pass the result to `Highlighter` normally:
+
+```ruby
+lexer = Antares::Grammar.load_tmlanguage("syntaxes/example.tmLanguage.json")
+highlighter = Antares::Highlighter.new(lexer: lexer, lines: lines, line_count: count)
+```
+
+The loader supports repository references, `$self`/`$base`, `match`, nested
+patterns, `begin`/`end` (including `applyEndPatternLast`) with numeric
+begin-capture references, and named captures. TextMate comment, string,
+numeric, keyword, storage, name, variable,
+punctuation, markup, and invalid scopes map to their nearest Rouge token types.
+Files are limited to 2 MiB, 64 levels, 10,000 rules, and 16 KiB per regular
+expression; lexing also has byte and time bounds. External grammar includes,
+`while`, nested repositories, capture subgrammars, and injection grammars raise
+`Antares::GrammarError` instead of being partially interpreted.
+The fixed Apple plist 1.0 document type is accepted without external lookup;
+custom document types and entity declarations are rejected.
+
+## Configuration
+
+<a name="configuration"></a>
+
+Pass options to `Antares::Highlighter.new`:
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `strategy` | Symbol | `:auto` | Highlighting strategy |
+| `checkpoint_interval` | Integer | `64` | Completed lines between checkpoints |
+| `window_context` | Integer | `200` | Lines of context for window mode |
+| `max_lines` | Integer | `100_000` | Maximum document lines |
+| `max_bytes` | Integer | `8 MiB` | Maximum source snapshot |
+| `max_line_bytes` | Integer | `16 KiB` | Maximum bytes per line |
+| `max_checkpoint_bytes` | Integer | `256 KiB` | Maximum checkpoint payload |
+| `max_seconds` | Numeric | `0.25` | Maximum time per scan |
+
+### Strategies
+
 | Strategy | Behavior |
-|---|---|
-| `:auto` | Uses the bundled, exact-Rouge-version compatibility verdict. Unknown versions and lexers use `:window`. |
-| `:incremental` | Forces state snapshots and convergence. Use for certified lexers; forcing a failed lexer can return incorrect classifications. |
-| `:window` | Restarts from root up to 200 lines before the requested range. Explicitly approximate for long-running constructs. |
-| `:full` | Runs ordinary Rouge over the complete bounded document after every edit; use for exact results on small files with failed incremental lexers. |
-
-`Antares.compatible?(Rouge::Lexers::Python)` returns `:incremental` or `:window`.
-Every registered lexer has an entry in `COMPATIBILITY` and a reproducible result
-in `COMPATIBILITY_DETAILS`; see the [complete compatibility table](docs/compatibility.md).
-Do not infer support merely because a lexer subclasses `Rouge::RegexLexer`.
-
-## State, limits, and correctness
-
-Antares reuses Rouge's rule interpreter and deep-copies persistent instance
-state, including nested arrays/hashes, their subclasses, delegate lexers, heredoc
-queues, sets and structs. Immutable rule templates are shared. Per-call scanners
-and callbacks are excluded. Dynamic rule closures compare conservatively by
-identity, and opaque state is rejected rather than shallow-copied.
-
-Checkpoints are placed approximately every 64 completed lines, **after entire
-rule callbacks**. State fingerprints between checkpoints allow early convergence;
-edits shift cached suffixes and invalidate stale boundaries. Calling
-`continue_lex` separately for each line is insufficient: Rouge regexes may consume
-several lines or inspect surrounding text. The driver therefore materializes a
-bounded source String on demand, uses a fixed-anchor scanner, and resumes lexical
-work at a checkpoint. It retains no array of provider lines. Source materialization
-is O(document bytes) after an edit and is included in reported timings.
-
-Some grammars are inherently nonlocal: for example, adding a distant `=end` makes
-Rouge classify an earlier unmatched Ruby `=begin` differently. These lexers fail
-the delayed-closure probes and select `:window`; the fallback is intentional.
-Other failures are classified as token mismatch (lookahead/state dependencies) or
-unsupported custom stream driver. The matrix is an empirical corpus guarantee,
-not a proof for every possible input or non-default lexer option.
-
-Default limits are 100,000 lines, an 8 MiB source snapshot, 16 KiB per line,
-256 KiB per checkpoint, and 250 ms per scan. Configure them with `max_lines`,
-`max_bytes`, `max_line_bytes`, `max_checkpoint_bytes`, and `max_seconds`.
-Exceeding a source limit switches to window mode; oversized individual lines and
-timed-out ranges return plain-text tokens without truncating content.
-`fallback_reason` explains the transition. These limits also apply to `:full`.
-
-## Development and verification
-
-```sh
+|----------|----------|
+| `:auto` | Uses the verdict bundled for the exact Rouge version. Unknown versions and lexers use `:window`. |
+| `:incremental` | Forces state snapshots and convergence. Use only for certified lexers. |
+| `:window` | Restarts from the root state up to `window_context` lines before the requested range. |
+| `:full` | Runs Rouge over the complete bounded document after every edit. |
+
+Exceeding a source limit switches to window mode. Oversized lines and timed-out
+ranges return complete plain-text tokens. `fallback_reason` explains the
+transition. These limits also apply to `:full`.
+
+## How It Works
+
+<a name="how-it-works"></a>
+
+1. Source Materialization builds a bounded source string from the line provider
+2. Checkpoint Restore resumes the nearest reusable lexer state
+3. Incremental Lexing records tokens and sparse state checkpoints
+4. State Convergence reuses the untouched cached suffix after an edit
+5. Fallback Selection uses window mode when incremental lexing is not certified
+
+Antares copies persistent Rouge lexer state, including nested collections,
+delegate lexers, and heredoc queues. Checkpoints are created after complete rule
+callbacks because Rouge expressions can consume multiple lines or inspect
+surrounding text.
+
+Some grammars are inherently nonlocal. For example, adding a distant `=end` can
+change how Rouge classifies an earlier unmatched Ruby `=begin`. Such lexers use
+window mode instead of unsafe incremental highlighting.
+
+## Compatibility
+
+<a name="compatibility"></a>
+
+`Antares.compatible?(Rouge::Lexers::Python)` returns `:incremental` or
+`:window`. Every registered lexer has a bundled verdict; see the
+[complete compatibility table](docs/compatibility.md).
+
+The matrix uses three source variants from Rouge's bundled MIT-licensed demos,
+applies 50 deterministic edits to each, and compares every line against a fresh
+full lex. Three distant-closure probes detect backward reclassification. The
+result is an empirical corpus guarantee, not a proof for every input or
+non-default lexer option.
+
+## Development
+
+<a name="development"></a>
+
+```bash
 bundle install
 bundle exec rake test
 bundle exec ruby script/compatibility --write
-BUDGET=1 bundle exec ruby --yjit bench/highlighting.rb
+BUDGET=1 bundle exec rake bench
 ```
 
-The matrix enumerates every Rouge lexer, uses three source variants based on
-Rouge's bundled MIT-licensed demos, applies 50 deterministic insert/delete/replace
-mutations to each, and compares all per-line tokens against a fresh full lex.
-Three extra distant-closure probes catch backward reclassification. The test suite
-reruns the complete matrix. `MUTATIONS=5 ruby script/compatibility python rust`
-is a short diagnostic run; `DETAILS=1` prints the full JSON failure details.
-Only full runs with 50 or more mutations may replace the bundled matrix.
-
-Scheduled CI updates Rouge, regenerates the matrix, reruns tests, and uploads the
-generated source/table for review. It does not silently publish a gem or modify a
-release. Regular CI tests Ruby 3.1–4.0 on Linux, macOS and Windows.
-
-On the development Mac with Ruby 4.0.0/YJIT, the 5,000-line Python benchmark
-measured 10.771 ms for the first 60 lines and 1.325 ms median for a one-line edit
-including source materialization and convergence. One line was rescanned, with
-80 checkpoints and 10,560 normalized payload bytes. Run the benchmark for current
-timings and retained-object memory; budgets are 15 ms / 5 ms / 2 MiB.
-
-Released under the [MIT license](LICENSE.txt).
+Use `MUTATIONS=5 ruby script/compatibility python rust` for a short compatibility
+run. Only full runs with at least 50 mutations may replace the bundled matrix.
+CI tests Ruby 3.1 through 4.0 on Linux, macOS, and Windows.
+
+See the [changelog](CHANGELOG.md) for release history.
+
+## Contributing
+
+<a name="contributing"></a>
+
+Bug reports and pull requests are welcome at https://github.com/noxdea/antares.
+
+## License
+
+<a name="license"></a>
+
+Released under the [MIT License](LICENSE.txt).