antlr-gemerator 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 0b211050cf47ab23a191f9264309bf28ba7393c1a25711debd67b74d254e2026
+  data.tar.gz: 5f888d09bd2cc518860b394501c491646c7eca3f9315e32d663d8dafb1682be7
+SHA512:
+  metadata.gz: 6b78306e25a3051e1b70d749b1893c7a367bc883d822ec3f3411c1b76bd0d0c6f778fdfc6d99520cc9a11749d61c222c0d83625e582ca7de85990a67bf7e349c
+  data.tar.gz: 9baab78ca3ca863f4188450b0eac44a41cb0778dcb922f2d96fdff7a21f58f138d20f83f16db884994afb3299bcae5ed43e7b516d73254f6f542a1e1da4e97f3

data/Gemfile

@@ -0,0 +1,12 @@
+source 'https://rubygems.org'
+
+gemspec
+
+group :development do
+  gem 'pry-byebug'
+  gem 'antlr4-native'
+end
+
+group :development, :test do
+  gem 'rake'
+end

data/README.md

@@ -0,0 +1,143 @@
+# antlr-gemerator
+
+Generate a complete Rubygem from any ANTLR grammar.
+
+## What is this thing?
+
+This gem leverages the functionality in [antlr4-native](https://github.com/camertron/antlr4-native) to generate a complete rubygem from an ANTLR grammar. The resulting gem contains a working parser and visitor, and is ready to be published to rubygems.org.
+
+## Background
+
+ANTLR is a wonderful tool for generating parsers, and is capable of parsing just about anything. Parsers (and corresponding lexers) are described via a grammar file, which declaratively defines the rules of the language or format. ANTLR generates parser code in the desired target programming language (Java by default), which can subsequently be used to parse documents that use the syntax defined in the grammar.
+
+For example, let's say you want to parse some [Lua](https://www.lua.org/about.html) code. You'd obtain ANTLR (usually by downloading a .jar file from the ANTLR website), find a grammar for Lua, then run the ANTLR tool like this:
+
+```bash
+java -jar /path/to/antlr.jar -o /path/to/output/dir ./Lua.g4
+```
+
+ANTLR will emit a bunch of .java files you can include in your Java project to parse Lua source code.
+
+### ANTLR and Ruby
+
+ANTLR can generate parsers in a number of target programming languages, but unfortunately Ruby isn't one of them. A couple of attempts have been made over the years to add a Ruby target to ANTLR, including [this excellent one](https://github.com/MODLanguage/antlr4-ruby-runtime) from [@twalmsley](https://github.com/twalmsley). Unfortunately none of these attempts have been merged into ANTLR proper yet.
+
+The real problem with a Ruby target however is execution speed. The runtime linked above runs \~80% slower than the equivalent Java-based Python parser I'm working on.
+
+### Speeding things up with native extensions
+
+One way to speed things up is to generate a parser in a more performant language and somehow build it as a Ruby native extension or FFI-compatible library. As it happens, ANTLR can target C++... and Ruby extensions can be written in C++! From there it was just a matter of writing some C++ glue code, and voila! ANTLR parsers wrapped in a loving Ruby embrace.
+
+## Usage
+
+antlr-gemerator runs from the command-line. First, install the gem by running:
+
+```bash
+gem install antlr-gemerator
+```
+
+If you're using rbenv, don't forget to run `rbenv rehash` to make the `antlr-gemerator` executable available in your shell.
+
+Next, change directory to where you'd like to generate your new gem and invoke `antlr-gemerator`:
+
+```bash
+antlr-gemerator create \
+  --author 'Mickey Mouse' \
+  --desc 'A Lua parser for Ruby' \
+  --email 'mickey@disney.com' \
+  --homepage 'https://github.com/mickeymouse/lua-parser-rb' \
+  --grammar path/to/Lua.g4 \
+  --root chunk
+```
+
+**NOTE**: You can specify the `--grammar` option more than once, i.e. for each .g4 file. It's common for the parser and lexer in ANTLR grammars to exist in individual files.
+
+**NOTE**: The `--root` option tells antlr-gemerator which context represents the root of the parse tree. This context functions as the starting point for visitors. Look inside your .g4 file (the parser one if there is more than one) and find the first grammar element. For the Lua grammar, this root element is called `chunk`.
+
+You should see a bunch of console output as antlr-gemerator emits all the files necessary for a Lua parser gem. It will also add the ANTLR runtime as a git submodule and build the native extension for you.
+
+### Using your gem
+
+Now that your gem has been generated and built, try it out by parsing some Lua code. Save the following snippet as tester.rb and run it with `bundle exec ruby tester.rb`:
+
+```ruby
+require 'lua-parser'
+
+lua_code = <<~END
+  -- test for even number
+  if n % 2 == 0 then
+    print "The number is even"
+  end
+
+  -- test for odd number
+  if not (n % 2 == 0) then
+    print "The number is odd"
+  end
+END
+
+class MyFuncVisitor < LuaParser::Visitor
+  def visit_functioncall(ctx)
+    puts ctx.var_or_exp.text
+    visit_children(ctx)
+  end
+end
+
+parser = LuaParser::Parser.parse(lua_code)
+parser.visit(MyFuncVisitor.new)
+```
+
+You should see the following output:
+
+```
+$> bundle exec ruby tester.rb
+print
+print
+```
+
+The `MyFuncVisitor` instance passed to `Parser#visit` prints the name of each function call, then visits the child contexts in the parsed subtree.
+
+## Publishing your gem
+
+To package your gem into a .gem file, run:
+
+```bash
+bundle exec rake build
+```
+
+The .gem file will be built into the pkg/ directory. Publish it by running:
+
+```bash
+gem push pkg/lua-parser-1.0.0.gem
+```
+
+You'll need to be signed into rubygems.org before publishing. Take a look at this [handy guide](https://guides.rubygems.org/publishing/) for instructions.
+
+## Maintaining your gem
+
+Each gem created by antlr-gemerator comes with several rake tasks to help make maintenance easier.
+
+1. `build`: builds the gem into a .gem file suitable for publishing to rubygems.org.
+2. `compile`: builds the native extension (i.e. compiles all the generated C++ code and the ANTLR runtime).
+3. `generate`: regenerates the C++ code by invoking antlr4-native. It's like running `antlr-gemerator` all over again, but _after_ your gem has been created.
+
+For example, to compile the native extension, run:
+
+```bash
+bundle exec rake compile
+```
+
+## Caveats
+
+See the caveats listed in [antlr4-native's README](https://github.com/camertron/antlr4-native-rb#caveats).
+
+## System Requirements
+
+See the system requirements listed in [antlr4-native's README](https://github.com/camertron/antlr4-native-rb#system-requirements).
+
+## License
+
+Licensed under the MIT license. See LICENSE.txt for details.
+
+## Authors
+
+* Cameron C. Dutro: http://github.com/camertron

data/Rakefile

@@ -0,0 +1,4 @@
+require 'bundler'
+require 'antlr4-native'
+
+Bundler::GemHelper.install_tasks

data/antlr-gemerator.gemspec

@@ -0,0 +1,22 @@
+$:.unshift File.join(File.dirname(__FILE__), 'lib')
+require 'antlr-gemerator/version'
+
+Gem::Specification.new do |s|
+  s.name     = 'antlr-gemerator'
+  s.version  = ::AntlrGemerator::VERSION
+  s.authors  = ['Cameron Dutro']
+  s.email    = ['camertron@gmail.com']
+  s.homepage = 'http://github.com/camertron/antlr-gemerator'
+
+  s.description = s.summary = 'Generate a complete Rubygem from any ANTLR4 grammar.'
+
+  s.platform = Gem::Platform::RUBY
+
+  s.add_dependency 'antlr4-native', '~> 1.0'
+  s.add_dependency 'gli', '~> 2.0'
+
+  s.executables << 'antlr-gemerator'
+
+  s.require_path = 'lib'
+  s.files = Dir['{lib,spec}/**/*', 'Gemfile', 'README.md', 'Rakefile', 'antlr-gemerator.gemspec']
+end

data/bin/antlr-gemerator

@@ -0,0 +1,42 @@
+#! /usr/bin/env ruby
+
+require 'gli'
+require 'antlr-gemerator'
+require 'antlr-gemerator/version'
+
+include GLI::App
+
+program_desc 'Generate a complete Rubygem from an ANTLR grammar.'
+
+version AntlrGemerator::VERSION
+
+subcommand_option_handling :normal
+arguments :strict
+
+desc 'Generate a complete Rubygem from an ANTLR grammar.'
+command :create do |c|
+  c.desc 'Grammar file'
+  c.flag [:g, :grammar], type: String, multiple: true, required: true
+
+  c.desc 'Root ANTLR context'
+  c.flag [:r, :root], type: String, required: true
+
+  c.desc "Gem author's name"
+  c.flag [:a, :author], type: String, required: true
+
+  c.desc "Gem author's email address"
+  c.flag [:e, :email], type: String, required: true
+
+  c.desc "Gem's homepage"
+  c.flag [:h, :homepage], type: String, required: true
+
+  c.desc "Gem's description"
+  c.flag [:d, :desc], type: String, required: true
+
+  c.action do |global_options, options, args|
+    AntlrGemerator.create(options)
+  end
+end
+
+exit run(ARGV)
+

data/lib/antlr-gemerator.rb

@@ -0,0 +1,109 @@
+require 'fileutils'
+require 'antlr4-native'
+require 'etc'
+
+module AntlrGemerator
+  autoload :Template, 'antlr-gemerator/template'
+
+  class << self
+    def create(options)
+      root_dir = '.'
+      lib_dir = File.join(root_dir, 'lib')
+      ext_dir = File.join(root_dir, 'ext')
+
+      # ANTLR does weird things if the grammar file isn't in the current
+      # working directory
+      grammars = options[:grammar].map do |g|
+        local_g = File.join(root_dir, File.basename(g))
+        FileUtils.cp(g, local_g)
+        local_g
+      end
+
+      generator = Antlr4Native::Generator.new(
+        grammar_files:      grammars,
+        output_dir:         ext_dir,
+        parser_root_method: options[:root]
+      )
+
+      bindings = {
+        gem_name:            generator.gem_name,
+        ext_name:            generator.ext_name,
+        gem_namespace:       generator.parser_ns,
+        gem_author_name:     options[:author],
+        gem_author_email:    options[:email],
+        gem_homepage:        options[:homepage] || '',
+        gem_description:     options[:desc],
+        grammar_files_array: grammars,
+        root_method:         options[:root]
+      }
+
+      # root level files
+      render 'gitignore.erb', File.join(root_dir, '.gitignore'), bindings
+      render 'Gemfile.erb', File.join(root_dir, 'Gemfile'), bindings
+      render 'gemspec.erb', File.join(root_dir, "#{bindings[:gem_name]}.gemspec"), bindings
+      render 'Rakefile.erb', File.join(root_dir, 'Rakefile'), bindings
+
+      # lib
+      mkdir File.join(lib_dir, bindings[:gem_name])
+
+      render 'entrypoint.rb.erb', File.join(lib_dir, "#{bindings[:gem_name]}.rb"), bindings
+      render 'version.rb.erb', File.join(lib_dir, bindings[:gem_name], 'version.rb'), bindings
+
+      # ext
+      antlr_version = Antlr4Native::Generator::ANTLR_VERSION
+      antlr_upstream_dir = File.join(ext_dir, bindings[:gem_name], 'antlr4-upstream')
+
+      mkdir File.join(ext_dir, bindings[:gem_name])
+
+      extension_dir = File.join(ext_dir, bindings[:gem_name])
+      extconf_path = File.join(extension_dir, 'extconf.rb')
+      render 'extconf.rb.erb', extconf_path, bindings
+
+      run 'git init', root_dir
+
+      # git is very stupid and won't let me pass a tag to git submodule add
+      run "git submodule add git://github.com/antlr/antlr4 #{antlr_upstream_dir}", root_dir
+      Dir.chdir(antlr_upstream_dir) { run "git checkout #{antlr_version}" }
+
+      generator.generate
+
+      bx 'bundle install', root_dir
+
+      # build
+      bx "ruby extconf.rb", extension_dir
+      run "make -j #{Etc.nprocessors}", extension_dir
+    end
+
+    private
+
+    def render(src, dest, bindings)
+      puts "RENDER #{dest}"
+      tmpl = Template.new(File.read(File.join(template_dir, src)), bindings)
+      File.write(dest, tmpl.render)
+    end
+
+    def mkdir(dir)
+      "MKDIR #{dir}"
+      FileUtils.mkdir_p(dir)
+    end
+
+    def run(cmd, in_dir = '.')
+      puts "RUN #{cmd}"
+      Dir.chdir(in_dir) { system(cmd) }
+    end
+
+    def bx(cmd, in_dir)
+      runner = -> { run(cmd, in_dir) }
+
+      if Kernel.const_defined?(:Bundler)
+        Bundler.with_clean_env(&runner)
+      else
+        runner.call
+      end
+    end
+
+    def template_dir
+      @template_dir ||= File.expand_path(File.join('antlr-gemerator', 'templates'), __dir__)
+    end
+  end
+end

data/lib/antlr-gemerator/template.rb

@@ -0,0 +1,27 @@
+require 'erb'
+
+module AntlrGemerator
+  class Template
+    def initialize(tmpl, bindings)
+      @tmpl = tmpl
+      @bindings = bindings
+    end
+
+    def render
+      ERB.new(@tmpl).result(binding)
+    end
+
+    def method_missing(mtd, *args, &block)
+      if @bindings.include?(mtd)
+        @bindings[mtd]
+      else
+        raise NoMethodError, "no method `#{mtd}' for #{self.class}"
+      end
+    end
+
+    def respond_to?(mtd)
+      return true if @bindings.include?(mtd)
+      super
+    end
+  end
+end

data/lib/antlr-gemerator/templates/Gemfile.erb

@@ -0,0 +1,12 @@
+source 'https://rubygems.org'
+
+gemspec
+
+group :development do
+  gem 'pry-byebug'
+  gem 'antlr4-native', '~> 1.0'
+end
+
+group :development, :test do
+  gem 'rake'
+end

data/lib/antlr-gemerator/templates/Rakefile.erb

@@ -0,0 +1,23 @@
+require 'bundler'
+
+require 'antlr4-native'
+require 'etc'
+
+Bundler::GemHelper.install_tasks
+
+task :generate do
+  generator = Antlr4Native::Generator.new(
+    grammar_files:      <%= grammar_files_array.inspect %>,
+    output_dir:         'ext/',
+    parser_root_method: '<%= root_method %>'
+  )
+
+  generator.generate
+end
+
+task :compile do
+  Dir.chdir(File.join(%w(ext <%= gem_name %>))) do
+    load 'extconf.rb'
+    exec "make -j #{Etc.nprocessors}"
+  end
+end

data/lib/antlr-gemerator/templates/entrypoint.rb.erb

@@ -0,0 +1,2 @@
+# load the native extension
+require File.expand_path(File.join('..', 'ext', '<%= gem_name %>', '<%= ext_name %>'), __dir__)

data/lib/antlr-gemerator/templates/extconf.rb.erb

@@ -0,0 +1,34 @@
+require 'mkmf-rice'
+
+extension_name = '<%= ext_name %>'
+dir_config(extension_name)
+
+have_library('stdc++')
+
+$CFLAGS << ' -std=c++14'
+
+include_paths = [
+  '.',
+  'antlrgen',
+  'antlr4-upstream/runtime/Cpp/runtime/src',
+  'antlr4-upstream/runtime/Cpp/runtime/src/atn',
+  'antlr4-upstream/runtime/Cpp/runtime/src/dfa',
+  'antlr4-upstream/runtime/Cpp/runtime/src/misc',
+  'antlr4-upstream/runtime/Cpp/runtime/src/support',
+  'antlr4-upstream/runtime/Cpp/runtime/src/tree',
+  'antlr4-upstream/runtime/Cpp/runtime/src/tree/pattern',
+  'antlr4-upstream/runtime/Cpp/runtime/src/tree/xpath'
+]
+
+$srcs = []
+
+include_paths.each do |include_path|
+  $INCFLAGS << " -I#{include_path}"
+  $VPATH << include_path
+
+  Dir.glob("#{include_path}/*.cpp").each do |path|
+    $srcs << path
+  end
+end
+
+create_makefile(extension_name)

data/lib/antlr-gemerator/templates/gemspec.erb

@@ -0,0 +1,31 @@
+$:.unshift File.join(File.dirname(__FILE__), 'lib')
+require '<%= gem_name %>/version'
+
+Gem::Specification.new do |s|
+  s.name     = '<%= gem_name %>'
+  s.version  = ::<%= gem_namespace %>::VERSION
+  s.authors  = ['<%= gem_author_name %>']
+  s.email    = ['<%= gem_author_email %>']
+  s.homepage = '<%= gem_homepage %>'
+
+  s.description = s.summary = '<%= gem_description %>'
+
+  s.platform = Gem::Platform::RUBY
+
+  s.add_dependency 'rice', '~> 2.0'
+
+  s.extensions = File.join(*%w(ext <%= gem_name %> extconf.rb))
+
+  s.require_path = 'lib'
+  s.files = Dir[
+    '{lib,spec}/**/*',
+    'ext/<%= gem_name %>/*.{cpp,h}',
+    'ext/<%= gem_name %>/extconf.rb',
+    'ext/<%= gem_name %>/antlrgen/*',
+    'ext/<%= gem_name %>/antlr4-upstream/runtime/Cpp/runtime/src/**/*.{cpp,h}',
+    'Gemfile',
+    'README.md',
+    'Rakefile',
+    '<%= gem_name %>.gemspec'
+  ]
+end

data/lib/antlr-gemerator/templates/gitignore.erb

@@ -0,0 +1,7 @@
+Gemfile.lock
+pkg/
+*.o
+*.o.tmp
+ext/<%= ext_name %>/Makefile
+ext/<%= ext_name %>/mkmf.log
+*.bundle

data/lib/antlr-gemerator/templates/version.rb.erb

@@ -0,0 +1,3 @@
+module <%= gem_namespace %>
+  VERSION = '1.0.0'
+end

data/lib/antlr-gemerator/version.rb

@@ -0,0 +1,3 @@
+module AntlrGemerator
+  VERSION = '1.0.0'
+end

metadata

@@ -0,0 +1,86 @@
+--- !ruby/object:Gem::Specification
+name: antlr-gemerator
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Cameron Dutro
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2020-05-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: antlr4-native
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: gli
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+description: Generate a complete Rubygem from any ANTLR4 grammar.
+email:
+- camertron@gmail.com
+executables:
+- antlr-gemerator
+extensions: []
+extra_rdoc_files: []
+files:
+- Gemfile
+- README.md
+- Rakefile
+- antlr-gemerator.gemspec
+- bin/antlr-gemerator
+- lib/antlr-gemerator.rb
+- lib/antlr-gemerator/template.rb
+- lib/antlr-gemerator/templates/Gemfile.erb
+- lib/antlr-gemerator/templates/Rakefile.erb
+- lib/antlr-gemerator/templates/entrypoint.rb.erb
+- lib/antlr-gemerator/templates/extconf.rb.erb
+- lib/antlr-gemerator/templates/gemspec.erb
+- lib/antlr-gemerator/templates/gitignore.erb
+- lib/antlr-gemerator/templates/version.rb.erb
+- lib/antlr-gemerator/version.rb
+homepage: http://github.com/camertron/antlr-gemerator
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.0.6
+signing_key: 
+specification_version: 4
+summary: Generate a complete Rubygem from any ANTLR4 grammar.
+test_files: []