antlr4-native 1.0.0 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 51499ca3096bdcbe0ce2e2231e9574b170a96acedeebc813f0e2c69134aa04f1
-  data.tar.gz: 485568b468bbaf64646344e239d58191e7025238c824e21b639e8641d953b118
+  metadata.gz: f14d05febd863a51c3e111e7ef3493eb2fe1b72fe72b0b3d1c5c114a2947f7c7
+  data.tar.gz: e8f3fe771ae21c552a1074e2f0608f9c87da1e7bd9063f2f8e18363531c37e6b
 SHA512:
-  metadata.gz: 65ccf82166ff334ac080850448093501c6a371a04f2084a820e3a8298af061af50330fda845815173011c66c08a0ad3e8b2b1026b201d5b419c21d82db8d2b4c
-  data.tar.gz: fb9a2983e52969af587ccb1e6a401825fade06ed13f437a86f5cb8b6289848ee13c7c9284c13c558e17a5e24a60b7eaecf24854de4465ca7d4ef4bbddd10e0c7
+  metadata.gz: 64cf7e6628edbaff365daf278c1e70dccd7ca083128d604efe8fcb4850c77efc621ce204fe250a7ffb91adb5d6cb4d02de9f1f68bbbb32c4c8c79d725c308a26
+  data.tar.gz: 1c5ba3ec3908cc1a951794d0e70f87afa0f01694ef623b38269ad5f22569fd8b587c1c6a7e820d32de273f6c87e77d15abcf604e83b0edd13f2a0636c09c5d86

data/README.md

@@ -0,0 +1,98 @@
+# antlr4-native
+
+Create a Ruby native extension from (almost) any ANTLR4 grammar.
+
+## What is this thing?
+
+This gem generates native Ruby extensions from ANTLR grammars, enabling Ruby developers to generate parsers for numerous programming languages, file formats, etc.
+
+## Who needs this?
+
+If you're a Ruby programmer who wants to parse and traverse source code written in a plethora of programming languages, antlr4-native might be able to help you. A number of community-developed ANTLR grammars are available in ANTLR's [grammars-v4](https://github.com/antlr/grammars-v4) repo. Grab one, then use antlr4-native to generate a bunch of Ruby-compatible C++ code from it. The C++ code can be compiled and used as a native extension.
+
+Rather than use antlr4-native directly, consider using its sister project, the [antlr-gemerator](https://github.com/camertron/antlr-gemerator), which can generate a complete rubygem from an ANTLR grammar.
+
+## Code Generation
+
+Here's how to generate a native extension for a given lexer and parser (Python in this case), defined in two .g4 files:
+
+```ruby
+require 'antlr4-native'
+
+generator = Antlr4Native::Generator.new(
+  grammar_files:      ['Python3Lexer.g4', 'Python3Parser.g4'],
+  output_dir:         'ext',
+  parser_root_method: 'file_input'
+)
+
+generator.generate
+```
+
+In the example above, the output directory is set to the standard Ruby native extensions directory, 'ext'. Antlr4-native will generate code into ext/\<name\>, where \<name\> is the name of the parser as defined in the grammar file(s). In this case, PythonParser.g4 contains:
+
+```antlr
+parser grammar Python3Parser;
+```
+
+so antlr4-native will generate code into the ext/python3-parser directory.
+
+Finally, the `parser_root_method` option tells antlr4-native which context represents the root of the parse tree. This context functions as the starting point for visitors.
+
+## Using extensions in Ruby
+
+Parsers contain several methods for parsing source code. Use `#parse` to parse a string and `#parse_file` to parse the contents of a file:
+
+
+```ruby
+parser = Python3Parser::Parser.parse(File.read('path/to/file.py'))
+
+# equivalent to:
+parser = Python3Parser::Parser.parse_file('path/to/file.py')
+```
+
+Use the `#visit` method on an instance of `Parser` to make use of a visitor:
+
+```ruby
+visitor = MyVisitor.new
+parser.visit(visitor)
+```
+
+See the next section for more info regarding creating and using visitors.
+
+## Visitors
+
+A visitor class is automatically created during code generation. Visitors are just classes with a bunch of special methods, each corresponding to a specific part of the source language's syntax. The methods are essentially callbacks that are triggered in-order as the parser walks over the parse tree. For example, here's a visitor with a method that will be called whenever the parser walks over a Python function definition:
+
+
+```ruby
+class FuncDefVisitor < Python3Parser::Visitor
+  def visit_func_def(ctx)
+    puts ctx.NAME.text  # print the name of the method
+    visit_children(ctx)
+  end
+end
+```
+
+Make sure to always call `#visit_children` at some point in your `visit_*` methods. If you don't, the subtree under the current context won't get visited.
+
+Finally, if you override `#initialize` in your visitor subclasses, don't forget to call `super`. If you don't, you'll get a nice big segfault.
+
+## Caveats
+
+1. Avoid retaining references to contexts, tokens, etc anywhere in your Ruby code. Contexts (i.e. the `ctx` variables in the examples above) and other objects that are created by ANTLR's C++ runtime are automatically cleaned up without the Ruby interpreter's knowledge. You'll almost surely see a segfault if you retain a reference to one of these objects and try to use it after the call to `Parser#visit`.
+2. Due to an ANTLR limitation, parsers cannot be used in a multi-threaded environment, even if each parser instance is used entirely in the context of a single thread (i.e. parsers are not shared between threads). According to the ANTLR C++ developers, parsers should be threadsafe. Unfortunately firsthand experience has proven otherwise. Your mileage may vary.
+3. The description of this gem says "(almost) any ANTLR4 grammar" because many grammars contain target-specific code. For example, the Python3 grammar referenced in the examples above contains inline Java code that the C++ compiler won't understand. You'll need to port any such code to C++ before you'll be able to compile and use the native extension.
+
+## System Requirements
+
+* A Java runtime (version 1.6 or higher) is required to generate parsers, since ANTLR is a Java tool. The ANTLR .jar file is distributed inside the antlr4-native gem, so there's no need to download it separately. You can download a Java runtime [here](https://www.java.com/en/download/).
+* Ruby >= 2.3.
+* A C compiler (like gcc or clang) that supports C++14. If Ruby is working on your machine then you likely already have this.
+
+## License
+
+Licensed under the MIT license. See LICENSE.txt for details.
+
+## Authors
+
+* Cameron C. Dutro: http://github.com/camertron

data/antlr4-native.gemspec

@@ -11,8 +11,9 @@ Gem::Specification.new do |s|
   s.description = s.summary = 'Create a Ruby native extension from any ANTLR4 grammar.'
 
   s.platform = Gem::Platform::RUBY
-  s.has_rdoc = true
 
   s.require_path = 'lib'
-  s.files = Dir['{lib,spec}/**/*', 'Gemfile', 'README.md', 'Rakefile', 'antlr4-native.gemspec']
+  s.files = Dir['{lib,spec,vendor}/**/*', 'Gemfile', 'README.md', 'Rakefile', 'antlr4-native.gemspec']
+
+  s.add_runtime_dependency "rice", "~> 4.0"
 end

data/lib/antlr4-native/context.rb

@@ -51,16 +51,24 @@ module Antlr4Native
 
     def conversions
       @class_conversions ||= <<~END
-        template <>
-        Object to_ruby<#{parser_ns}::#{name}*>(#{parser_ns}::#{name}* const &x) {
-          if (!x) return Nil;
-          return Data_Object<#{parser_ns}::#{name}>(x, #{proxy_class_variable}, nullptr, nullptr);
-        }
+        namespace Rice::detail {
+          template <>
+          class To_Ruby<#{parser_ns}::#{name}*> {
+          public:
+            VALUE convert(#{parser_ns}::#{name}* const &x) {
+              if (!x) return Nil;
+              return Data_Object<#{parser_ns}::#{name}>(x, false, #{proxy_class_variable});
+            }
+          };
 
-        template <>
-        Object to_ruby<#{name}Proxy*>(#{name}Proxy* const &x) {
-          if (!x) return Nil;
-          return Data_Object<#{name}Proxy>(x, #{proxy_class_variable}, nullptr, nullptr);
+          template <>
+          class To_Ruby<#{name}Proxy*> {
+          public:
+            VALUE convert(#{name}Proxy* const &x) {
+              if (!x) return Nil;
+              return Data_Object<#{name}Proxy>(x, false, #{proxy_class_variable});
+            }
+          };
         }
       END
     end
@@ -88,7 +96,7 @@ module Antlr4Native
                 }
               }
 
-              return a;
+              return std::move(a);
             }
           END
         else
@@ -105,7 +113,7 @@ module Antlr4Native
               }
 
               for (auto child : getChildren()) {
-                if (ctx == from_ruby<ContextProxy>(child).getOriginal()) {
+                if (ctx == detail::From_Ruby<ContextProxy>().convert(child.value()).getOriginal()) {
                   return child;
                 }
               }
@@ -127,17 +135,17 @@ module Antlr4Native
               Array a;
 
               if (orig == nullptr) {
-                return a;
+                return std::move(a);
               }
 
               auto vec = ((#{parser_ns}::#{name}*)orig) -> #{token_mtd.name}(#{params});
 
               for (auto it = vec.begin(); it != vec.end(); it ++) {
                 TerminalNodeProxy proxy(*it);
-                a.push(proxy);
+                a.push(detail::To_Ruby<TerminalNodeProxy>().convert(proxy));
               }
 
-              return a;
+              return std::move(a);
             }
           END
         else
@@ -148,8 +156,13 @@ module Antlr4Native
               }
 
               auto token = ((#{parser_ns}::#{name}*)orig) -> #{token_mtd.name}(#{params});
+
+              if (token == nullptr) {
+                return Qnil;
+              }
+
               TerminalNodeProxy proxy(token);
-              return to_ruby(proxy);
+              return detail::To_Ruby<TerminalNodeProxy>().convert(proxy);
             }
           END
         end
@@ -159,8 +172,7 @@ module Antlr4Native
     def class_wrapper(module_var)
       @class_wrapper ||= begin
         lines = [
-          "#{proxy_class_variable} = #{module_var}",
-          ".define_class<#{name}Proxy, ContextProxy>(\"#{name}\")"
+          %(#{proxy_class_variable} = define_class_under<#{name}Proxy, ContextProxy>(#{module_var}, "#{name}"))
         ]
 
         each_context_method do |ctx_method|

data/lib/antlr4-native/generator.rb

@@ -63,18 +63,27 @@ module Antlr4Native
 
     def interop_code
       <<~END
-        #include "iostream"
+        #include <iostream>
 
-        #include "antlr4-runtime.h"
+        #include <antlr4-runtime.h>
 
-        #include "#{parser_ns}.h"
-        #include "#{antlr_ns}BaseVisitor.h"
-        #include "#{lexer_ns}.h"
+        #include "antlrgen/#{parser_ns}.h"
+        #include "antlrgen/#{antlr_ns}BaseVisitor.h"
+        #include "antlrgen/#{lexer_ns}.h"
 
-        #include "rice/Array.hpp"
-        #include "rice/Class.hpp"
-        #include "rice/Constructor.hpp"
-        #include "rice/Director.hpp"
+        #include <rice/rice.hpp>
+        #include <rice/stl.hpp>
+
+        #ifdef _WIN32
+        #undef OPTIONAL
+        #undef IN
+        #undef OUT
+        #endif
+
+        #undef FALSE
+        #undef TRUE
+
+        #undef TYPE
 
         using namespace std;
         using namespace Rice;
@@ -82,6 +91,35 @@ module Antlr4Native
 
         #{proxy_class_declarations}
 
+        namespace Rice::detail {
+          template <>
+          class To_Ruby<Token*> {
+          public:
+            VALUE convert(Token* const &x) {
+              if (!x) return Nil;
+              return Data_Object<Token>(x, false, rb_cToken);
+            }
+          };
+
+          template <>
+          class To_Ruby<tree::ParseTree*> {
+          public:
+            VALUE convert(tree::ParseTree* const &x) {
+              if (!x) return Nil;
+              return Data_Object<tree::ParseTree>(x, false, rb_cParseTree);
+            }
+          };
+
+          template <>
+          class To_Ruby<tree::TerminalNode*> {
+          public:
+            VALUE convert(tree::TerminalNode* const &x) {
+              if (!x) return Nil;
+              return Data_Object<tree::TerminalNode>(x, false, rb_cTerminalNode);
+            }
+          };
+        }
+
         class ContextProxy {
         public:
           ContextProxy(tree::ParseTree* orig) {
@@ -96,6 +134,18 @@ module Antlr4Native
             return orig -> getText();
           }
 
+          Object getStart() {
+            auto token = ((ParserRuleContext*) orig) -> getStart();
+
+            return detail::To_Ruby<Token*>().convert(token);
+          }
+
+          Object getStop() {
+            auto token = ((ParserRuleContext*) orig) -> getStop();
+
+            return detail::To_Ruby<Token*>().convert(token);
+          }
+
           Array getChildren() {
             if (children == nullptr) {
               children = new Array();
@@ -134,7 +184,7 @@ module Antlr4Native
 
           bool doubleEquals(Object other) {
             if (other.is_a(rb_cContextProxy)) {
-              return from_ruby<ContextProxy*>(other) -> getOriginal() == getOriginal();
+              return detail::From_Ruby<ContextProxy*>().convert(other) -> getOriginal() == getOriginal();
             } else {
               return false;
             }
@@ -155,6 +205,7 @@ module Antlr4Native
           TerminalNodeProxy(tree::ParseTree* tree) : ContextProxy(tree) { }
         };
 
+
         #{proxy_class_headers}
 
         #{conversions}
@@ -191,31 +242,7 @@ module Antlr4Native
     end
 
     def conversions
-      @conversions ||= begin
-        context_conversions = contexts.map(&:conversions).join("\n")
-
-        <<~END
-          template <>
-          Object to_ruby<Token*>(Token* const &x) {
-            if (!x) return Nil;
-            return Data_Object<Token>(x, rb_cToken, nullptr, nullptr);
-          }
-
-          template <>
-          Object to_ruby<tree::ParseTree*>(tree::ParseTree* const &x) {
-            if (!x) return Nil;
-            return Data_Object<tree::ParseTree>(x, rb_cParseTree, nullptr, nullptr);
-          }
-
-          template <>
-          Object to_ruby<tree::TerminalNode*>(tree::TerminalNode* const &x) {
-            if (!x) return Nil;
-            return Data_Object<tree::TerminalNode>(x, rb_cTerminalNode, nullptr, nullptr);
-          }
-
-          #{context_conversions}
-        END
-      end
+      @conversions ||= contexts.map(&:conversions).join("\n")
     end
 
     def proxy_class_methods
@@ -243,14 +270,21 @@ module Antlr4Native
             return parser;
           }
 
-          VALUE visit(VisitorProxy* visitor) {
-            visitor -> visit(this -> parser -> #{parser_root_method}());
+          Object #{parser_root_method}() {
+            auto ctx = this -> parser -> #{parser_root_method}();
+
+            #{capitalize(parser_root_method)}ContextProxy proxy((#{parser_ns}::#{capitalize(parser_root_method)}Context*) ctx);
+            return detail::To_Ruby<#{capitalize(parser_root_method)}ContextProxy>().convert(proxy);
+          }
+
+          Object visit(VisitorProxy* visitor) {
+            auto result = visitor -> visit(this -> parser -> #{parser_root_method}());
 
             // reset for the next visit call
             this -> lexer -> reset();
             this -> parser -> reset();
 
-            return Qnil;
+            return result;
           }
 
           ~ParserProxy() {
@@ -280,10 +314,15 @@ module Antlr4Native
           #{parser_ns}* parser;
         };
 
-        template <>
-        Object to_ruby<ParserProxy*>(ParserProxy* const &x) {
-          if (!x) return Nil;
-          return Data_Object<ParserProxy>(x, rb_cParser, nullptr, nullptr);
+        namespace Rice::detail {
+          template <>
+          class To_Ruby<ParserProxy*> {
+          public:
+            VALUE convert(ParserProxy* const &x) {
+              if (!x) return Nil;
+              return Data_Object<ParserProxy>(x, false, rb_cParser);
+            }
+          };
         }
       END
     end
@@ -294,38 +333,37 @@ module Antlr4Native
         void Init_#{ext_name}() {
           Module rb_m#{parser_ns} = define_module("#{parser_ns}");
 
-          rb_cToken = rb_m#{parser_ns}
-            .define_class<Token>("Token")
-            .define_method("text", &Token::getText);
-
-          rb_cParser = rb_m#{parser_ns}
-            .define_class<ParserProxy>("Parser")
-            .define_singleton_method("parse", &ParserProxy::parse)
-            .define_singleton_method("parse_file", &ParserProxy::parseFile)
-            .define_method("visit", &ParserProxy::visit);
+          rb_cToken = define_class_under<Token>(rb_m#{parser_ns}, "Token")
+            .define_method("text", &Token::getText)
+            .define_method("channel", &Token::getChannel)
+            .define_method("token_index", &Token::getTokenIndex);
 
-          rb_cParseTree = rb_m#{parser_ns}
-            .define_class<tree::ParseTree>("ParseTree");
+          rb_cParseTree = define_class_under<tree::ParseTree>(rb_m#{parser_ns}, "ParseTree");
 
-          rb_cContextProxy = rb_m#{parser_ns}
-            .define_class<ContextProxy>("Context")
+          rb_cContextProxy = define_class_under<ContextProxy>(rb_m#{parser_ns}, "Context")
             .define_method("children", &ContextProxy::getChildren)
             .define_method("child_count", &ContextProxy::childCount)
             .define_method("text", &ContextProxy::getText)
+            .define_method("start", &ContextProxy::getStart)
+            .define_method("stop", &ContextProxy::getStop)
             .define_method("parent", &ContextProxy::getParent)
             .define_method("==", &ContextProxy::doubleEquals);
 
-          rb_cTerminalNode = rb_mPython3Parser
-            .define_class<TerminalNodeProxy, ContextProxy>("TerminalNodeImpl");
+          rb_cTerminalNode = define_class_under<TerminalNodeProxy, ContextProxy>(rb_m#{parser_ns}, "TerminalNodeImpl");
 
-          rb_m#{parser_ns}
-            .define_class<#{visitor_generator.cpp_class_name}>("#{visitor_generator.class_name}")
+          define_class_under<#{antlr_ns}BaseVisitor>(rb_m#{parser_ns}, "#{visitor_generator.class_name}")
             .define_director<#{visitor_generator.cpp_class_name}>()
             .define_constructor(Constructor<#{visitor_generator.cpp_class_name}, Object>())
             .define_method("visit", &#{visitor_generator.cpp_class_name}::ruby_visit)
             .define_method("visit_children", &#{visitor_generator.cpp_class_name}::ruby_visitChildren)
         #{visitor_generator.visitor_proxy_methods('    ').join("\n")};
 
+          rb_cParser = define_class_under<ParserProxy>(rb_m#{parser_ns}, "Parser")
+            .define_singleton_function("parse", &ParserProxy::parse)
+            .define_singleton_function("parse_file", &ParserProxy::parseFile)
+            .define_method("#{parser_root_method}", &ParserProxy::#{parser_root_method})
+            .define_method("visit", &ParserProxy::visit);
+
         #{class_wrappers_str('  ')}
         }
       END
@@ -337,7 +375,7 @@ module Antlr4Native
           [
             "  #{idx == 0 ? 'if' : 'else if'} (antlrcpp::is<#{parser_ns}::#{context.name}*>(node)) {",
             "    #{context.name}Proxy proxy((#{parser_ns}::#{context.name}*)node);",
-            "    return to_ruby(proxy);",
+            "    return detail::To_Ruby<#{context.name}Proxy>().convert(proxy);",
             "  }"
           ]
         end
@@ -347,7 +385,7 @@ module Antlr4Native
           #{wrapper_branches.join("\n")}
             else if (antlrcpp::is<tree::TerminalNodeImpl*>(node)) {
               TerminalNodeProxy proxy(node);
-              return to_ruby(proxy);
+              return detail::To_Ruby<TerminalNodeProxy>().convert(proxy);
             } else {
               return Nil;
             }

data/lib/antlr4-native/version.rb

@@ -1,3 +1,3 @@
 module Antlr4Native
-  VERSION = '1.0.0'
+  VERSION = '2.0.0'
 end

data/lib/antlr4-native/visitor_generator.rb

@@ -46,13 +46,21 @@ module Antlr4Native
           #{cpp_class_name}(Object self) : Director(self) { }
 
           Object ruby_visit(ContextProxy* proxy) {
-            visit(proxy -> getOriginal());
-            return Nil;
+            auto result = visit(proxy -> getOriginal());
+            try {
+              return result.as<Object>();
+            } catch(std::bad_cast) {
+              return Qnil;
+            }
           }
 
           Object ruby_visitChildren(ContextProxy* proxy) {
-            visitChildren(proxy -> getOriginal());
-            return Nil;
+            auto result = visitChildren(proxy -> getOriginal());
+            try {
+              return result.as<Object>();
+            } catch(std::bad_cast) {
+              return Qnil;
+            }
           }
 
         #{vms.join("\n")}

metadata

@@ -1,15 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: antlr4-native
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 2.0.0
 platform: ruby
 authors:
 - Cameron Dutro
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-05-01 00:00:00.000000000 Z
-dependencies: []
+date: 2022-01-24 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rice
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
 description: Create a Ruby native extension from any ANTLR4 grammar.
 email:
 - camertron@gmail.com
@@ -18,6 +32,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - Gemfile
+- README.md
 - Rakefile
 - antlr4-native.gemspec
 - lib/antlr4-native.rb
@@ -28,10 +43,11 @@ files:
 - lib/antlr4-native/string_helpers.rb
 - lib/antlr4-native/version.rb
 - lib/antlr4-native/visitor_generator.rb
+- vendor/antlr4-4.8-1-complete.jar
 homepage: http://github.com/camertron/antlr4-native-rb
 licenses: []
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -46,9 +62,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.6.2
-signing_key: 
+rubygems_version: 3.2.22
+signing_key:
 specification_version: 4
 summary: Create a Ruby native extension from any ANTLR4 grammar.
 test_files: []