scilex 2026.6.0__tar.gz

scilex-2026.6.0/BENCHMARKS.md

@@ -0,0 +1,91 @@
+# SciLex — performance baseline
+
+A reproducible wall-time baseline for the Python binding, comparing SciLex against
+Python's standard-library `re`. Its purpose is twofold: a **regression tripwire**
+between versions on the same machine, and an honest statement of **where SciLex wins
+and where it does not**.
+
+Run it with **`make bench`** (or `python benchmarks/bench.py`). It is informational
+only — it prints a table, is never invoked by `full-local-gate`, and never fails a
+build.
+
+## The honest headline
+
+SciLex is **not** built to beat `re` on raw throughput, and it does not. `re` is a
+mature C backtracking engine; SciLex runs REAL's linear-time NFA at every position,
+through the abi3 binding, and builds rich `Token` objects. On benign input that costs
+a few times more per byte.
+
+What SciLex guarantees instead is **linear time, ReDoS-safe by construction**: no rule
+can make the scanner backtrack catastrophically. On an adversarial (or simply
+unlucky) pattern, `re` degrades exponentially while SciLex stays flat — and *that* is
+the difference that matters for a lexer fed untrusted or machine-generated input.
+
+### Gains and losses at a glance
+
+| input | winner | why |
+| --- | --- | --- |
+| benign token soup | `re` (~5×, see B1) | a mature C backtracking engine; SciLex runs REAL's NFA per position, through the binding, and builds rich `Token`s |
+| adversarial / ReDoS (B2) | **SciLex** (linear vs exponential) | REAL is linear-time and ReDoS-safe; `re` backtracks catastrophically |
+| untrusted / machine-generated | **SciLex** | the linear bound holds on *every* input — no pathological cliff |
+
+## Conditions of this baseline
+
+| | |
+| --- | --- |
+| Machine | Apple Silicon (`arm64`), Darwin 23.6.0 |
+| Binding | abi3 CPython extension as built by `setup.py` (`Py_LIMITED_API` 3.10) |
+| Method | best-of-5 timed runs, **minimum** reported |
+| As of | 2026-06-19 — the Python-binding maturation reserve |
+
+## Baseline
+
+### B1. Benign tokenization (the everyday case — `re` wins)
+
+Tokenizing ~10 KB of ordinary `ident = ident + number * ident - number ;` soup into
+4000 tokens (numbers, identifiers, operators; whitespace skipped). SciLex compiles the
+rule set once (a reused `Lexer`); the `re` baseline is the standard "master pattern"
+tokenizer (`(?P<NUM>…)|(?P<ID>…)|…` + `finditer`).
+
+| tokenizer | time | vs `re` |
+| --- | ---: | ---: |
+| `scilex.Lexer.tokenize` | ~7.4 ms | ~5.2× |
+| `re.finditer` (master pattern) | ~1.4 ms | 1.0× (baseline) |
+
+**Reading.** `re` is ~5× faster here. That is expected and reported plainly: the cost
+buys SciLex's linear guarantee and its ordered maximal-munch semantics, not a speed
+record on benign input. Tightening the per-position scan is a known, deliberately
+deferred lever — to be pursued only if a real workload makes it the bottleneck.
+
+### B2. Pathological input (the linearity guarantee — SciLex wins decisively)
+
+The classic ReDoS trigger `(a+)+b` over a run of `n` `a`s with no terminating `b`. A
+backtracking engine explores `O(2ⁿ)` partitions; REAL (and therefore SciLex) is linear.
+
+| n | `scilex` (linear) | `re.match` (backtracking) |
+| ---: | ---: | ---: |
+| 16 | ~2.1 µs | ~2.2 ms |
+| 18 | ~2.2 µs | ~9.1 ms |
+| 20 | ~2.4 µs | ~35.9 ms |
+| 22 | ~2.6 µs | ~142.7 ms |
+| 24 | ~2.8 µs | ~574 ms |
+| 26 | ~2.9 µs | ~2.30 s |
+| 1000 | ~78 µs | would not finish |
+
+**Reading.** `re`'s time roughly **quadruples every +2** in `n` (exponential); SciLex
+grows **linearly** and is still ~78 µs at `n = 1000`, where `re` would not finish in any
+practical time. This is the case SciLex exists for.
+
+## Methodology & reproduction
+
+- **Goal:** a regression tripwire plus an honest win/lose map — not a throughput
+  contest. Compare a fresh `make bench` to this table **on the same machine**; a clear,
+  repeatable change is the signal.
+- **Reproduce:** `make bench` builds the extension in place and runs
+  `benchmarks/bench.py`. The pathological sweep stops `re` once a single match passes
+  one second (its curve is already established); SciLex is measured well past that.
+- **Not gated.** `make bench` is excluded from `full-local-gate` on purpose — a noisy
+  wall-time measurement must never turn a clean build red.
+- **Deferred:** a compile-time `static_lexer` (REAL's `static_regex`) and a faster
+  per-position scan (e.g. first-byte / trie dispatch) are known levers, left until a
+  measured workload justifies them. No phantom numbers here for paths not yet built.

scilex-2026.6.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 René Chenard
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

scilex-2026.6.0/MANIFEST.in

@@ -0,0 +1,7 @@
+# Make the sdist self-contained: carry the C++ headers the binding compiles
+# against and the binding source, plus the project docs. (The headers are also
+# REAL's at build time, pulled from the real-regex build dependency.)
+graft include
+recursive-include python/src *.cpp
+recursive-include python/tests *.py
+include README.md LICENSE BENCHMARKS.md

scilex-2026.6.0/PKG-INFO

@@ -0,0 +1,227 @@
+Metadata-Version: 2.4
+Name: scilex
+Version: 2026.6.0
+Summary: SciLex — a generic, linear-time maximal-munch lexer built on REAL
+Author-email: René Chenard <rene.chenard.1@ulaval.ca>
+License-Expression: MIT
+Project-URL: Repository, https://github.com/RECHE23/scilex
+Project-URL: Issues, https://github.com/RECHE23/scilex/issues
+Keywords: lexer,tokenizer,maximal-munch,scanner,linear-time
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: C++
+Classifier: Topic :: Software Development :: Compilers
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# SciLex
+
+A small, header-only C++20 lexer built on [REAL](https://github.com/RECHE23/real-regex).
+
+Define an ordered set of token rules — each a kind paired with a REAL regular
+expression — and SciLex tokenizes source text by **maximal munch** (longest
+match wins, rule order breaks ties). Because REAL is a linear-time engine,
+tokenization is linear and ReDoS-safe by construction: no token rule can make
+the scanner backtrack catastrophically.
+
+This is a deliberate fresh start under the same premises as REAL — purity,
+simplicity, and measured optimality. SciLex is a thin layer over REAL, not a
+re-implementation of pattern matching.
+
+## Scope (v1)
+
+**Included:**
+
+- Token rules: a kind, a `real::regex`, and a `skip` flag (whitespace, comments).
+- Maximal-munch tokenization with rule-order priority on equal-length ties.
+- Source positions: byte `offset`, 1-based `line` and byte `column`.
+- Non-owning tokens: each `lexeme` views into the source.
+- Two ways to consume tokens: `tokenize` (eager, into a vector) and `scan`
+  (a lazy single-pass range that produces one token at a time — no token
+  vector is allocated; the parser-friendly access pattern).
+- Optional synthetic end-of-input token (`eof_policy::append`): emits a final
+  `end_of_input` token at the real end position, so a parser always has a
+  current token to match.
+- Optional indentation layout (`scilex::layout`, an opt-in header): rewrites a
+  token stream with synthetic `newline` / `indent` / `dedent` tokens for
+  indentation-significant languages; throws `layout_error` on an inconsistent
+  dedent.
+- Lexical errors as exceptions carrying the failing position (`lex_error`).
+- Linear-time / ReDoS-safe tokenization, inherited from REAL.
+
+**Not in v1 (and honestly excluded — no phantom features):**
+
+- Modes / context-sensitive lexing.
+- Compile-time `static_lexer` (built on REAL's `static_regex`).
+- Codepoint columns (columns are byte-based, matching REAL's UTF-8 model).
+- Command-line tool, JSON specification language.
+
+These may come later, each only if it earns its place — measured, tested, and
+kept minimal.
+
+## Dependencies
+
+SciLex is header-only and depends only on REAL's headers. By default the build
+looks for them in a sibling checkout:
+
+```
+~/Projects/
+├── real-v1/      # REAL
+└── scilex-v1/    # SciLex  (uses ../real-v1/include by default)
+```
+
+Point the build elsewhere with `REAL_INCLUDE` (Makefile) or
+`-DSCILEX_REAL_INCLUDE=...` (CMake) — for instance at the path printed by
+`python -c "import real; print(real.get_include())"` when REAL is installed via
+pip.
+
+For CI or a reproducible build — where no on-disk layout can be assumed — fetch
+REAL with CMake FetchContent instead (`make build FETCH=1`, or
+`-DSCILEX_FETCH_DEPS=ON`); point it at a remote and pin a tag with
+`-DSCILEX_REAL_REPO=https://… -DSCILEX_REAL_TAG=v2026.6.6`.
+
+## Build
+
+```bash
+make test        # build and run the test suite
+make coverage    # line-coverage summary + HTML report
+make sanitize    # tests under AddressSanitizer + UndefinedBehaviorSanitizer
+make lint        # clang-tidy
+make format      # uncrustify, in place
+make doc         # API reference (Doxygen) with embedded coverage
+```
+
+Override the compiler with `make test CXX=g++-14`.
+
+`scilex::scilex` is the CMake target — `add_subdirectory`, `FetchContent`, or an
+installed config package. The config calls `find_dependency(real)`, so installing
+REAL's config package alongside (on the same prefix) makes the whole chain
+resolve from one `find_package`:
+
+```cmake
+# With REAL and SciLex installed under <prefix>:
+find_package(scilex CONFIG REQUIRED)   # pulls in real:: transitively
+target_link_libraries(app PRIVATE scilex::scilex)
+```
+
+## Releasing
+
+`make release` computes the next calendar version `YYYY.M.PATCH` — the patch
+resets each month, the first release of a month is `.0` (PEP 440 drops leading
+zeros, so `2026.6.1`, never `2026.06.001`) — bumps it in `pyproject.toml` and
+`python/scilex/__init__.py`, then commits, tags and pushes from a clean `main`.
+The pushed tag drives `.github/workflows/release.yml`,
+which checks the tag matches the version, builds abi3 wheels (`cibuildwheel`,
+Linux/macOS/Windows) and the self-contained sdist, and publishes to PyPI via
+Trusted Publishing (OIDC, no stored secret). It then populates the GitHub
+`/releases` page with auto-generated notes and the built artifacts. The pushed tag
+is the single thing that triggers a publish; SciLex remains consumable as source
+too (sibling checkout / FetchContent / `get_include()`).
+
+**One-time PyPI setup.** Publishing needs a PyPI
+[Trusted Publisher](https://docs.pypi.org/trusted-publishers/) configured once for
+the project (publisher `RECHE23/scilex`, workflow `release.yml`, environment
+`pypi`) and a matching `pypi` GitHub environment — no API token is stored.
+
+## Python binding
+
+SciLex ships an abi3 CPython extension (use the C++ lexer from Python).
+`pip install scilex` installs one `cp310-abi3` wheel per platform (CPython 3.10+;
+the self-contained sdist compiles where no wheel matches, pulling REAL's headers
+from the `real-regex` build dependency). For a source checkout:
+
+```bash
+make python        # build the extension in place
+make python-test   # run the binding test suite
+```
+
+```python
+import scilex
+lx = scilex.Lexer([
+    (0, r"\s+", True),                       # (kind, pattern, skip) — skipped
+    (1, r"[0-9]+", False),                   # number
+    (2, r"[A-Za-z_][A-Za-z0-9_]*", False),   # identifier
+])
+
+# Eager: a list of rich Token objects (kind, lexeme, structured position).
+[(t.kind, t.lexeme) for t in lx.tokenize("foo 42")]      # [(2, 'foo'), (1, '42')]
+
+# Lazy: a generator yielding one Token at a time — nothing else is held.
+for tok in lx.scan("foo 42"):
+    tok.kind, tok.lexeme, tok.position.line, tok.position.column
+
+# A lexical error carries the failing position.
+try:
+    lx.tokenize("foo @")
+except scilex.error as e:
+    e.position                                            # Position(line=1, column=5, offset=4)
+
+# eof=True appends a terminal END_OF_INPUT token (a parser always has a token).
+lx.tokenize("42", eof=True)[-1].kind == scilex.END_OF_INPUT
+```
+
+For indentation-significant languages, `Layout` rewrites an `eof=True` token
+stream with `NEWLINE` / `INDENT` / `DEDENT` tokens read from each line's leading
+column:
+
+```python
+lx = scilex.Lexer([(0, r"\s+", True), (1, r"\w+", False), (2, r":", False)])
+laid = scilex.Layout().apply(lx.tokenize("if x:\n    a\nb", eof=True))
+[t.kind for t in laid]
+# [1, 1, 2, NEWLINE, INDENT, 1, NEWLINE, DEDENT, 1, NEWLINE, END_OF_INPUT]
+```
+
+`scilex.get_include()` returns the header directory so a C++ project can compile
+against SciLex located through its Python install (add `real.get_include()` too,
+since SciLex's headers include REAL's).
+
+## Example
+
+```cpp
+#include <scilex/scilex.hpp>
+#include <vector>
+
+enum kind { WS, KW_IF, ID, NUM, PLUS };
+
+std::vector<scilex::rule> rules;
+rules.push_back({WS,    real::regex("\\s+"),  true}); // skipped
+rules.push_back({KW_IF, real::regex("if")});          // before ID: wins ties
+rules.push_back({ID,    real::regex("[a-z]+")});
+rules.push_back({NUM,   real::regex("[0-9]+")});
+rules.push_back({PLUS,  real::regex("\\+")});
+
+const scilex::lexer lexer(std::move(rules));
+
+// Eager: all tokens in a vector.
+for (const scilex::token& t : lexer.tokenize("if x + 42")) {
+    // t.kind, t.lexeme, t.start.{offset,line,column}
+}
+
+// Lazy: one token at a time, nothing else materialized.
+for (const scilex::token& t : lexer.scan("if x + 42")) {
+    // ...
+}
+```
+
+## Performance
+
+See [BENCHMARKS.md](BENCHMARKS.md) for a reproducible, honest baseline (`make
+bench`). The short of it: on benign input Python's `re` is faster (a mature C
+backtracking engine), but SciLex is **linear-time and ReDoS-safe by construction**
+— on a pathological pattern like `(a+)+b` it stays flat (~78 µs at 1000 chars)
+where `re` explodes exponentially (seconds, then never finishing). SciLex trades
+raw throughput on easy inputs for a guarantee that holds on every input.
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+## Author
+
+René Chenard — rene.chenard@gmail.com

scilex-2026.6.0/README.md

@@ -0,0 +1,205 @@
+# SciLex
+
+A small, header-only C++20 lexer built on [REAL](https://github.com/RECHE23/real-regex).
+
+Define an ordered set of token rules — each a kind paired with a REAL regular
+expression — and SciLex tokenizes source text by **maximal munch** (longest
+match wins, rule order breaks ties). Because REAL is a linear-time engine,
+tokenization is linear and ReDoS-safe by construction: no token rule can make
+the scanner backtrack catastrophically.
+
+This is a deliberate fresh start under the same premises as REAL — purity,
+simplicity, and measured optimality. SciLex is a thin layer over REAL, not a
+re-implementation of pattern matching.
+
+## Scope (v1)
+
+**Included:**
+
+- Token rules: a kind, a `real::regex`, and a `skip` flag (whitespace, comments).
+- Maximal-munch tokenization with rule-order priority on equal-length ties.
+- Source positions: byte `offset`, 1-based `line` and byte `column`.
+- Non-owning tokens: each `lexeme` views into the source.
+- Two ways to consume tokens: `tokenize` (eager, into a vector) and `scan`
+  (a lazy single-pass range that produces one token at a time — no token
+  vector is allocated; the parser-friendly access pattern).
+- Optional synthetic end-of-input token (`eof_policy::append`): emits a final
+  `end_of_input` token at the real end position, so a parser always has a
+  current token to match.
+- Optional indentation layout (`scilex::layout`, an opt-in header): rewrites a
+  token stream with synthetic `newline` / `indent` / `dedent` tokens for
+  indentation-significant languages; throws `layout_error` on an inconsistent
+  dedent.
+- Lexical errors as exceptions carrying the failing position (`lex_error`).
+- Linear-time / ReDoS-safe tokenization, inherited from REAL.
+
+**Not in v1 (and honestly excluded — no phantom features):**
+
+- Modes / context-sensitive lexing.
+- Compile-time `static_lexer` (built on REAL's `static_regex`).
+- Codepoint columns (columns are byte-based, matching REAL's UTF-8 model).
+- Command-line tool, JSON specification language.
+
+These may come later, each only if it earns its place — measured, tested, and
+kept minimal.
+
+## Dependencies
+
+SciLex is header-only and depends only on REAL's headers. By default the build
+looks for them in a sibling checkout:
+
+```
+~/Projects/
+├── real-v1/      # REAL
+└── scilex-v1/    # SciLex  (uses ../real-v1/include by default)
+```
+
+Point the build elsewhere with `REAL_INCLUDE` (Makefile) or
+`-DSCILEX_REAL_INCLUDE=...` (CMake) — for instance at the path printed by
+`python -c "import real; print(real.get_include())"` when REAL is installed via
+pip.
+
+For CI or a reproducible build — where no on-disk layout can be assumed — fetch
+REAL with CMake FetchContent instead (`make build FETCH=1`, or
+`-DSCILEX_FETCH_DEPS=ON`); point it at a remote and pin a tag with
+`-DSCILEX_REAL_REPO=https://… -DSCILEX_REAL_TAG=v2026.6.6`.
+
+## Build
+
+```bash
+make test        # build and run the test suite
+make coverage    # line-coverage summary + HTML report
+make sanitize    # tests under AddressSanitizer + UndefinedBehaviorSanitizer
+make lint        # clang-tidy
+make format      # uncrustify, in place
+make doc         # API reference (Doxygen) with embedded coverage
+```
+
+Override the compiler with `make test CXX=g++-14`.
+
+`scilex::scilex` is the CMake target — `add_subdirectory`, `FetchContent`, or an
+installed config package. The config calls `find_dependency(real)`, so installing
+REAL's config package alongside (on the same prefix) makes the whole chain
+resolve from one `find_package`:
+
+```cmake
+# With REAL and SciLex installed under <prefix>:
+find_package(scilex CONFIG REQUIRED)   # pulls in real:: transitively
+target_link_libraries(app PRIVATE scilex::scilex)
+```
+
+## Releasing
+
+`make release` computes the next calendar version `YYYY.M.PATCH` — the patch
+resets each month, the first release of a month is `.0` (PEP 440 drops leading
+zeros, so `2026.6.1`, never `2026.06.001`) — bumps it in `pyproject.toml` and
+`python/scilex/__init__.py`, then commits, tags and pushes from a clean `main`.
+The pushed tag drives `.github/workflows/release.yml`,
+which checks the tag matches the version, builds abi3 wheels (`cibuildwheel`,
+Linux/macOS/Windows) and the self-contained sdist, and publishes to PyPI via
+Trusted Publishing (OIDC, no stored secret). It then populates the GitHub
+`/releases` page with auto-generated notes and the built artifacts. The pushed tag
+is the single thing that triggers a publish; SciLex remains consumable as source
+too (sibling checkout / FetchContent / `get_include()`).
+
+**One-time PyPI setup.** Publishing needs a PyPI
+[Trusted Publisher](https://docs.pypi.org/trusted-publishers/) configured once for
+the project (publisher `RECHE23/scilex`, workflow `release.yml`, environment
+`pypi`) and a matching `pypi` GitHub environment — no API token is stored.
+
+## Python binding
+
+SciLex ships an abi3 CPython extension (use the C++ lexer from Python).
+`pip install scilex` installs one `cp310-abi3` wheel per platform (CPython 3.10+;
+the self-contained sdist compiles where no wheel matches, pulling REAL's headers
+from the `real-regex` build dependency). For a source checkout:
+
+```bash
+make python        # build the extension in place
+make python-test   # run the binding test suite
+```
+
+```python
+import scilex
+lx = scilex.Lexer([
+    (0, r"\s+", True),                       # (kind, pattern, skip) — skipped
+    (1, r"[0-9]+", False),                   # number
+    (2, r"[A-Za-z_][A-Za-z0-9_]*", False),   # identifier
+])
+
+# Eager: a list of rich Token objects (kind, lexeme, structured position).
+[(t.kind, t.lexeme) for t in lx.tokenize("foo 42")]      # [(2, 'foo'), (1, '42')]
+
+# Lazy: a generator yielding one Token at a time — nothing else is held.
+for tok in lx.scan("foo 42"):
+    tok.kind, tok.lexeme, tok.position.line, tok.position.column
+
+# A lexical error carries the failing position.
+try:
+    lx.tokenize("foo @")
+except scilex.error as e:
+    e.position                                            # Position(line=1, column=5, offset=4)
+
+# eof=True appends a terminal END_OF_INPUT token (a parser always has a token).
+lx.tokenize("42", eof=True)[-1].kind == scilex.END_OF_INPUT
+```
+
+For indentation-significant languages, `Layout` rewrites an `eof=True` token
+stream with `NEWLINE` / `INDENT` / `DEDENT` tokens read from each line's leading
+column:
+
+```python
+lx = scilex.Lexer([(0, r"\s+", True), (1, r"\w+", False), (2, r":", False)])
+laid = scilex.Layout().apply(lx.tokenize("if x:\n    a\nb", eof=True))
+[t.kind for t in laid]
+# [1, 1, 2, NEWLINE, INDENT, 1, NEWLINE, DEDENT, 1, NEWLINE, END_OF_INPUT]
+```
+
+`scilex.get_include()` returns the header directory so a C++ project can compile
+against SciLex located through its Python install (add `real.get_include()` too,
+since SciLex's headers include REAL's).
+
+## Example
+
+```cpp
+#include <scilex/scilex.hpp>
+#include <vector>
+
+enum kind { WS, KW_IF, ID, NUM, PLUS };
+
+std::vector<scilex::rule> rules;
+rules.push_back({WS,    real::regex("\\s+"),  true}); // skipped
+rules.push_back({KW_IF, real::regex("if")});          // before ID: wins ties
+rules.push_back({ID,    real::regex("[a-z]+")});
+rules.push_back({NUM,   real::regex("[0-9]+")});
+rules.push_back({PLUS,  real::regex("\\+")});
+
+const scilex::lexer lexer(std::move(rules));
+
+// Eager: all tokens in a vector.
+for (const scilex::token& t : lexer.tokenize("if x + 42")) {
+    // t.kind, t.lexeme, t.start.{offset,line,column}
+}
+
+// Lazy: one token at a time, nothing else materialized.
+for (const scilex::token& t : lexer.scan("if x + 42")) {
+    // ...
+}
+```
+
+## Performance
+
+See [BENCHMARKS.md](BENCHMARKS.md) for a reproducible, honest baseline (`make
+bench`). The short of it: on benign input Python's `re` is faster (a mature C
+backtracking engine), but SciLex is **linear-time and ReDoS-safe by construction**
+— on a pathological pattern like `(a+)+b` it stays flat (~78 µs at 1000 chars)
+where `re` explodes exponentially (seconds, then never finishing). SciLex trades
+raw throughput on easy inputs for a guarantee that holds on every input.
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+## Author
+
+René Chenard — rene.chenard@gmail.com

scilex-2026.6.0/include/scilex/layout.hpp

@@ -0,0 +1,125 @@
+/*!
+ * \file layout.hpp
+ * \brief Optional indentation layout: insert NEWLINE / INDENT / DEDENT tokens.
+ *
+ * Some languages (Python-like, e.g. SciLang) make indentation significant. This
+ * opt-in pass turns a flat token stream into a layout-aware one: it inserts a
+ * \ref scilex::newline at each logical line end, and \ref scilex::indent /
+ * \ref scilex::dedent where the leading indentation changes.
+ *
+ * It works purely from token **positions** — every \ref scilex::token already
+ * carries its source line and (byte) column — so the base lexer needs no change
+ * and may keep skipping whitespace. Lines with no token (blank or
+ * comment-only) carry no structure and are naturally ignored.
+ *
+ * Indentation width is the byte column of a line's first token (tabs and spaces
+ * each count as one column; v1 does not police mixed tabs/spaces, and there is
+ * no implicit line continuation inside brackets).
+ *
+ * Input must be an end-of-input-terminated token sequence (the lexer's
+ * `eof_policy::append`); the terminal \ref scilex::end_of_input is preserved.
+ */
+#ifndef SCILEX_LAYOUT_HPP
+#define SCILEX_LAYOUT_HPP
+
+#include <cstddef>
+#include <limits>
+#include <span>
+#include <stdexcept>
+#include <string>
+#include <vector>
+
+#include "token.hpp"
+
+namespace scilex {
+
+  //! \brief Reserved kind: end of a logical line.
+  inline constexpr int newline {std::numeric_limits<int>::min() + 1};
+  //! \brief Reserved kind: indentation increased (start of a deeper block).
+  inline constexpr int indent  {std::numeric_limits<int>::min() + 2};
+  //! \brief Reserved kind: indentation decreased (end of a block).
+  inline constexpr int dedent  {std::numeric_limits<int>::min() + 3};
+
+  /*!
+   * \brief Thrown when a line's indentation matches no enclosing level.
+   */
+  class layout_error : public std::runtime_error
+  {
+  public:
+
+    //! \brief Builds the error. \param[in] message Cause. \param[in] where Position.
+    layout_error(const std::string& message,
+                 position           where)
+      : std::runtime_error(message),
+        where_(where)
+    {}
+
+    //! \brief Returns the position of the offending line.
+    [[nodiscard]] position where() const noexcept
+    {
+      return where_;
+    }
+
+  private:
+
+    position where_; //!< Where the indentation was inconsistent.
+  };
+
+  /*!
+   * \brief Rewrites \p tokens with NEWLINE / INDENT / DEDENT inserted.
+   *
+   * \param[in] tokens An end-of-input-terminated token sequence.
+   * \return The layout-aware token sequence (still end-of-input-terminated).
+   * \throws layout_error If a line dedents to an indentation that no open block
+   *         used.
+   */
+  [[nodiscard]] inline std::vector<token> layout(std::span<const token> tokens)
+  {
+    std::vector<token>       out;
+    std::vector<std::size_t> levels        {0};
+    bool                     started       {false};
+    std::size_t              previous_line {0};
+    position                 end_position  {0, 1, 1};
+
+    for (const token& current : tokens) {
+      if (current.kind == end_of_input) {
+        end_position = current.start; // remember; emit our own terminal at the end
+        continue;
+      }
+      if (!started || current.start.line != previous_line) {
+        if (started) {
+          out.push_back(token {newline, {}, current.start});
+        }
+        const std::size_t width {current.start.column - 1};
+        if (width > levels.back()) {
+          levels.push_back(width);
+          out.push_back(token {indent, {}, current.start});
+        }
+        else {
+          while (width < levels.back()) {
+            levels.pop_back();
+            out.push_back(token {dedent, {}, current.start});
+          }
+          if (width != levels.back()) {
+            throw layout_error("inconsistent indentation", current.start);
+          }
+        }
+        started = true;
+      }
+      out.push_back(current);
+      previous_line = current.start.line;
+    }
+
+    if (started) {
+      out.push_back(token {newline, {}, end_position});
+    }
+    while (levels.back() > 0) {
+      levels.pop_back();
+      out.push_back(token {dedent, {}, end_position});
+    }
+    out.push_back(token {end_of_input, {}, end_position});
+    return out;
+  }
+} // namespace scilex
+
+#endif // SCILEX_LAYOUT_HPP