lilt 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 27310c339a7875bd8f1e6705459689c05dc978c0c5dc6e1d59534b650493528e
+  data.tar.gz: 34cb476a1f8e961d55af835aeb5e490b2fbb7dde94d3b58fe3647c90181b40b2
+SHA512:
+  metadata.gz: 7159ca1d750e56ee6c0bbb8c958db5a22188dd8784895a8c9765a5f9038ec8612258914ca0cd6786e003d1f7883f99c3d0de020aa72fc7594f9fdf3324ca0ed4
+  data.tar.gz: 4fbb86254219e96138bf7d8409067791dcf3e4d5a43e0e1bc3b595969cfc263b6f2e60ff7a9aff9947cbee5e508dba8311ea44bc632eba5137fc80556daac37a

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Phil Crissman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,265 @@
+# Lilt
+
+A **Li**ttle **L**anguage **T**oolkit.
+
+I've been writing a lot of small languages, mostly lambda calculi and similar experiments, 
+from [TAPL](https://www.cis.upenn.edu/~bcpierce/tapl/) or other sources. I started to notice that the lexer and parser were all generally
+the same, and were not the most interesting part of the process, so I thought I would try 
+to create a library to make it easier to get a lexer and parser going for small languages.
+
+It has:
+
+- **A table-driven lexer.** You write the rules for recognizing tokens, and get back the 
+  list of tokens with line and column numbers.
+- **A Pratt parser engine.** You write tables of handlers, and get back whatever AST
+  your handlers build.
+- **An s-expression reader and printer.** It gives you a second front end for free, and
+  a readable way to print any AST.
+
+Grammars are plain data (arrays and hashes of procs), and the AST is yours: Lilt
+doesn't define node types. I've been using `Data.define` for nodes, which gives you
+structural equality and pattern matching for free.
+
+## Installation
+
+Add it to your Gemfile:
+
+```ruby
+gem "lilt"
+```
+
+or install it directly:
+
+```sh
+gem install lilt
+```
+
+It requires Ruby 3.2 or newer (for `Data`).
+
+## Lexing
+
+A lexer is a list of rules, each a pattern and a token type. Rules are tried in order,
+and the first match wins. A string matches literally; a regex matches a pattern; a
+`nil` type means "skip this".
+
+```ruby
+require "lilt"
+
+RULES = [
+  [/\s+/,      nil],      # skip whitespace
+  ["+",        :plus],
+  ["*",        :star],
+  ["(",        :lparen],
+  [")",        :rparen],
+  [/\d+/,      :int],
+  [/[a-z]\w*/, :ident],
+]
+
+Lilt::Lexer.lex("let x", RULES, keywords: %w[let])
+# => [#<data Lilt::Token type=:let, value="let", line=1, col=1>,
+#     #<data Lilt::Token type=:ident, value="x", line=1, col=5>,
+#     #<data Lilt::Token type=:eof, value=nil, line=1, col=6>]
+```
+
+`keywords:` promotes any token whose whole text is a keyword to its own type, so `let`
+becomes `:let` while `letter` stays an `:ident`. Every token list ends with `:eof`.
+Columns count characters, not bytes, so `λ` is fine. Input that no rule matches raises
+`Lilt::LexError` with its position.
+
+## Parsing
+
+A grammar table has `prefix:` handlers (for tokens that start an expression) and
+`infix:` entries (for tokens that continue one). An infix entry is a binding power and a
+handler; `Lilt.binary` builds the common case for you.
+
+```ruby
+Num = Data.define(:value)
+Add = Data.define(:left, :right)
+Mul = Data.define(:left, :right)
+
+ARITH = {
+  prefix: {
+    int:    proc { |tok| Num.new(tok.value.to_i) },
+    lparen: Lilt.group(:rparen),
+  },
+  infix: {
+    plus: Lilt.binary(10) { |l, r| Add.new(l, r) },
+    star: Lilt.binary(20) { |l, r| Mul.new(l, r) },
+  },
+}
+
+ast = Lilt.parse(ARITH, Lilt::Lexer.lex("1 + 2 * 3", RULES))
+# => #<data Add left=#<data Num value=1>, right=#<data Mul left=#<data Num value=2>, right=#<data Num value=3>>>
+
+Lilt::Sexp.print(ast)
+# => "(add (num 1) (mul (num 2) (num 3)))"
+```
+
+A higher binding power binds more tightly. `Lilt.binary(bp)` is left-associative;
+`Lilt.binary(bp, :right)` is right-associative. `Lilt.group(:rparen)` parses one
+expression and then expects the closing token.
+
+For prefix operators like unary minus, `Lilt.prefix(bp) { |operand| Neg.new(operand) }`
+parses its operand at `bp`. With a high `bp` (say 70), `-a * b` means `(-a) * b`; with
+one between `*` and `^`, `-a ^ b` means `-(a ^ b)`. A token can have both a prefix and
+an infix handler, so `-` can mean negation and subtraction.
+
+Since the AST is just data, doing something with it is a `case`/`in`:
+
+```ruby
+def calc(e)
+  case e
+  in Num[n]    then n
+  in Add[l, r] then calc(l) + calc(r)
+  in Mul[l, r] then calc(l) * calc(r)
+  end
+end
+
+calc(ast) # => 7
+```
+
+### Writing handlers
+
+When a helper doesn't fit, a handler is just a proc:
+
+- A **prefix** handler gets `(token, parser, table)`.
+- An **infix** handler gets `(left, token, parser, table)`.
+- An infix entry is `[binding_power, handler]`.
+
+Inside a handler, `parser.parse(table, min_bp)` parses a sub-expression,
+`parser.expect(:type)` consumes a token or raises, and `parser.peek` and
+`parser.advance` look at and consume tokens. `parser.error!(token, message)` raises a
+`Lilt::ParseError` with the token's position.
+
+Procs ignore arguments they don't ask for, so simple handlers stay short.
+
+### Application by juxtaposition
+
+For functional languages, `f x y` means `(f x) y`. Add a `juxtapose:` entry and any
+token that can *start* an expression, appearing where an operator could go, is treated
+as an invisible, left-associative application operator:
+
+```ruby
+Var = Data.define(:name)
+App = Data.define(:fn, :arg)
+
+LC = {
+  prefix: {
+    ident:  proc { |tok| Var.new(tok.value.to_sym) },
+    lparen: Lilt.group(:rparen),
+  },
+  infix: {
+    plus: Lilt.binary(10) { |l, r| Add.new(l, r) },
+  },
+  juxtapose: [100, proc { |fn, arg| App.new(fn, arg) }],
+}
+
+Lilt::Sexp.print(Lilt.parse(LC, Lilt::Lexer.lex("f x y + g y", RULES)))
+# => "(add (app (app (var f) (var x)) (var y)) (app (var g) (var y)))"
+```
+
+A real infix operator always wins over juxtaposition, and tokens with no prefix handler
+(like `)`) end an application.
+
+### Several tables
+
+A grammar can have more than one table (terms and types, say), and a handler switches
+between them just by parsing with a different one. For example, a lambda handler that
+parses `λx:T. body` does `parser.parse(TYPES)` for the annotation and
+`parser.parse(TERMS)` for the body. See `examples/stlc.rb`.
+
+You need a separate table for each syntactic category: a part of the grammar where the
+same tokens mean different things, or different operators apply. Many small languages,
+like the untyped lambda calculus, have just one category, so one table is all they need.
+
+### Errors and positions
+
+Parse errors say where they happened:
+
+```ruby
+Lilt.parse(ARITH, Lilt::Lexer.lex("1 +", RULES))
+# raises Lilt::ParseError: unexpected eof at 1:4
+```
+
+For errors found *after* parsing (type errors, say), `Lilt.parse_located` also
+returns a table from each node to the token where it starts. The table is compared by
+identity, so two equal nodes at different places in the source are told apart:
+
+```ruby
+ast, positions = Lilt.parse_located(ARITH, Lilt::Lexer.lex("1 + 2", RULES))
+positions[ast.right] # => #<data Lilt::Token type=:int, value="2", line=1, col=5>
+```
+
+This keeps positions out of the AST, so structural equality still works. The STLC
+example uses this to report type errors with a line and column, without its type checker
+knowing anything about positions.
+
+## S-expressions
+
+`Lilt::Sexp.read` turns s-expression text into nested arrays of symbols and integers,
+and `Lilt::Sexp.print` goes the other way. It also prints any `Data` node, using the
+snake-cased class name as the tag:
+
+```ruby
+Lilt::Sexp.read("(lambda (x Bool) (f x 42))")
+# => [:lambda, [:x, :Bool], [:f, :x, 42]]
+```
+
+This is handy for prototyping a language before its syntax has settled: write a small
+function that turns the arrays into your AST, and get to the semantics without writing
+a grammar at all. Later, the Pratt front end can produce the same AST.
+
+## What is Pratt parsing?
+
+The core loop is the one from Vaughan Pratt's 1973 paper *Top Down Operator
+Precedence*, as popularized by Douglas Crockford. Prefix handlers are Pratt's *nud*s,
+infix handlers are his *led*s, and parsing continues while the next token's binding power
+is greater than the current minimum.
+
+Lilt adds two things that aren't in the original:
+
+- **Juxtaposition**, for function application.
+- **Multiple tables** per grammar.
+
+Within any one table, it's the standard algorithm. The lexer and the s-expression reader
+aren't Pratt parsing; the reader is ordinary recursive descent.
+
+## The STLC example
+
+[`examples/stlc.rb`](examples/stlc.rb) is a complete simply typed lambda calculus with
+booleans, built on Lilt in about 200 lines. It has:
+
+- a Pratt grammar with two tables, terms and types
+- an s-expression front end that produces the same AST, and a test checking that the two agree
+- a type checker whose errors report line and column
+- an environment-based evaluator with closures
+- a De Bruijn index pass, with an evaluator for nameless terms
+
+```ruby
+STLC.interpret("(λx:Bool. if x then false else true) true")
+# => #<data STLC::Bool value=false>
+
+STLC.interpret("true false")
+# raises STLC::TypeError: expected a function, got (t_bool) at 1:1
+```
+
+From a clone, try it with:
+
+```sh
+ruby -Ilib -r./examples/stlc -e 'p STLC.interpret("(λx:Bool. x) true")'
+```
+
+Its tests are in [`examples/stlc_test.rb`](examples/stlc_test.rb).
+
+## Development
+
+```sh
+bundle install
+bundle exec rake
+```
+
+`rake` runs the tests for the gem and the examples.
+
+## License
+
+MIT. See [LICENSE.txt](LICENSE.txt).

data/lib/lilt/lexer.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require "strscan"
+
+module Lilt
+  Token = Data.define(:type, :value, :line, :col)
+
+  class LexError < StandardError; end
+
+  module Lexer
+    extend self
+
+    def lex(source, rules, keywords: [])
+      scanner = StringScanner.new(source)
+      tokens = []
+      line, line_start = 1, 0
+
+      until scanner.eos?
+        start = scanner.charpos
+        col = start - line_start + 1
+        rule = rules.find { |pattern, _| scanner.scan(pattern) }
+        raise LexError, "unexpected '#{source[start]}' at #{line}:#{col}" unless rule
+
+        text = scanner.matched
+        tokens << Token.new(token_type(rule.last, text, keywords), text, line, col) if rule.last
+        line, line_start = advance(line, line_start, start, text)
+      end
+
+      tokens << Token.new(:eof, nil, line, scanner.charpos - line_start + 1)
+    end
+
+    private
+
+    def token_type(type, text, keywords) = keywords.include?(text) ? text.to_sym : type
+
+    # Returns the [line, line_start] in effect after consuming +text+ from +start+.
+    def advance(line, line_start, start, text)
+      nl = text.rindex("\n") or return [line, line_start]
+      [line + text.count("\n"), start + nl + 1]
+    end
+  end
+end

data/lib/lilt/parser.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+module Lilt
+  extend self
+
+  class ParseError < StandardError; end
+
+  def parse(table, tokens) = parse_located(table, tokens).first
+
+  # Like parse, but also returns a Hash (compared by identity) from each node
+  # a handler built to the token where that node's expression starts.
+  def parse_located(table, tokens)
+    parser = Parser.new(tokens)
+    ast = parser.parse(table).tap { parser.expect(:eof) }
+    [ast, parser.positions]
+  end
+
+  # Builds an infix entry for a binary operator. The block receives the
+  # left and right operands (and the operator token) and returns the node.
+  def binary(bp, assoc = :left, &build)
+    right_bp = assoc == :right ? bp - 1 : bp
+    [bp, proc { |left, tok, p, table| build.call(left, p.parse(table, right_bp), tok) }]
+  end
+
+  # Builds a prefix handler for a prefix operator. The operand is parsed at
+  # +bp+, so it takes in exactly the infix operators that bind tighter than
+  # +bp+. The block receives the operand (and the operator token).
+  def prefix(bp, &build)
+    proc { |tok, p, table| build.call(p.parse(table, bp), tok) }
+  end
+
+  # Builds a prefix handler for a bracketing token: parses one expression in
+  # the current table, then expects +close+. Returns the inner expression.
+  def group(close)
+    proc { |_tok, p, table| p.parse(table).tap { p.expect(close) } }
+  end
+
+  # A cursor over a token stream, which always ends with an :eof token.
+  # Invariant: @pos always indexes a token; advance stops at :eof.
+  class Parser
+    attr_reader :positions
+
+    def initialize(tokens)
+      @tokens = tokens
+      @pos = 0
+      @positions = {}.compare_by_identity
+    end
+
+    def parse(table, min_bp = 0)
+      start = advance
+      nud = table.fetch(:prefix)[start.type] or error!(start, "unexpected #{start.type}")
+      left = locate(nud.call(start, self, table), start)
+
+      loop do
+        bp, led = led_for(table, peek)
+        break if bp.nil? || bp <= min_bp
+        left = locate(led.call(left), start)
+      end
+
+      left
+    end
+
+    def peek = @tokens[@pos]
+
+    def expect(type)
+      tok = peek
+      error!(tok, "expected #{type}, got #{tok.type}") unless tok.type == type
+      advance
+    end
+
+    def advance
+      tok = @tokens[@pos]
+      @pos += 1 unless tok.type == :eof
+      tok
+    end
+
+    def error!(tok, message) = raise(ParseError, "#{message} at #{tok.line}:#{tok.col}")
+
+    private
+
+    # Records where +node+ starts, unless an inner parse already did (as when
+    # a group handler returns its inner expression). Returns +node+.
+    def locate(node, tok)
+      @positions[node] ||= tok
+      node
+    end
+
+    # If +tok+ can continue an expression, returns its binding power and a
+    # callable that extends +left+. An infix entry consumes the operator
+    # token; juxtaposition applies when +tok+ can instead *start* an
+    # expression, and consumes nothing before parsing the right side.
+    def led_for(table, tok)
+      if (infix = table.fetch(:infix, {})[tok.type])
+        bp, handler = infix
+        return [bp, ->(left) { handler.call(left, advance, self, table) }]
+      end
+
+      bp, build = table[:juxtapose]
+      return unless bp && table.fetch(:prefix).key?(tok.type)
+
+      [bp, ->(left) { build.call(left, parse(table, bp)) }]
+    end
+  end
+end

data/lib/lilt/sexp.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require_relative "lexer"
+require_relative "parser"
+
+module Lilt
+  module Sexp
+    extend self
+
+    RULES = [[/\s+/, nil], ["(", :lparen], [")", :rparen], [/[^\s()]+/, :atom]]
+
+    def read(source)
+      parser = Parser.new(Lexer.lex(source, RULES))
+      datum(parser).tap { parser.expect(:eof) }
+    end
+
+    def print(node)
+      case node
+      in Data  then list([tag(node), *node.deconstruct])
+      in Array then list(node)
+      else node.to_s
+      end
+    end
+
+    private
+
+    def list(items) = "(#{items.map { print(_1) }.join(" ")})"
+
+    def datum(parser)
+      tok = parser.advance
+      case tok.type
+      when :lparen then items(parser)
+      when :atom   then atom(tok.value)
+      else parser.error!(tok, "unexpected #{tok.type}")
+      end
+    end
+
+    def atom(text) = text.match?(/\A-?\d+\z/) ? text.to_i : text.to_sym
+
+    def items(parser)
+      list = []
+      list << datum(parser) until parser.peek.type == :rparen
+      parser.expect(:rparen)
+      list
+    end
+
+    def tag(node)
+      node.class.name.split("::").last
+          .gsub(/([A-Z]+)([A-Z][a-z])/, '\1_\2')
+          .gsub(/([a-z\d])([A-Z])/, '\1_\2')
+          .downcase
+    end
+  end
+end

data/lib/lilt/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Lilt
+  VERSION = "0.1.0"
+end

data/lib/lilt.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+require_relative "lilt/version"
+require_relative "lilt/lexer"
+require_relative "lilt/parser"
+require_relative "lilt/sexp"

metadata

@@ -0,0 +1,52 @@
+--- !ruby/object:Gem::Specification
+name: lilt
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Phil Crissman
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies: []
+description: Lilt provides a table-driven lexer, a Pratt (top-down operator precedence)
+  parser engine extended with juxtaposition and multiple expression tables, position
+  tracking, and an s-expression reader and printer. Grammars are plain data; ASTs
+  are whatever your handlers build.
+email:
+- phil.crissman@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE.txt
+- README.md
+- lib/lilt.rb
+- lib/lilt/lexer.rb
+- lib/lilt/parser.rb
+- lib/lilt/sexp.rb
+- lib/lilt/version.rb
+homepage: https://github.com/philcrissman/lilt
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/philcrissman/lilt
+  source_code_uri: https://github.com/philcrissman/lilt
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.7
+specification_version: 4
+summary: A small toolkit for building lexers and Pratt parsers for little languages.
+test_files: []