rexical 1.0.5 → 1.0.7

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d6b78dc8daf9900783617edca9228ca4d4640f808b6251775e7105f6f20d1e9b
+  data.tar.gz: 01dd220d9a85cc3236f79b6596fa6b64b9f95456b71d5340fb4284c48db20fa5
+SHA512:
+  metadata.gz: 6fcd99b73957780f3617ddc9317d24207dd9be43c9bb42d55863c1b4f7cfa591d2a48a9f9b0e200089303dbb0c8cf64628c1514c9181220e555721b1274c28f7
+  data.tar.gz: d6f222da2d448cbb7b5b2a849f7a1549cde2870301e557217825e428668fd57c02eb05ffcf8332f993d8ddf14ab6dca7d5a39e2ec0bb0fa5fe444b3bfff083b2

data/CHANGELOG.rdoc

@@ -1,4 +1,16 @@
-=== 1.0.5 / Not released
+=== 1.0.7 / 2019-08-06
+
+* Security
+
+  * prefer File.open to Kernel.open
+
+=== 1.0.6
+
+* Bug fixes
+
+  * scanner states work better.  Thanks Mat.
+
+=== 1.0.5
 
 * Bug fixes
 

data/DOCUMENTATION.en.rdoc

@@ -1,215 +1,211 @@
-
-= REX: Ruby Lex for Racc
-
-
-== About
-
-  Lexical Scanner Generator for Ruby, with Racc.
-
-
-== Usage
-
-  rex [options] grammarfile
-
-  -o  --output-file  filename   designated output filename.
-  -s  --stub                    append stub main for debug.
-  -i  --ignorecase              ignore char case
-  -C  --check-only              syntax check only.
-      --independent             independent mode.
+= REX: Ruby Lex for Racc
+
+
+== About
+
+  Lexical Scanner Generator used with Racc for Ruby
+
+
+== Usage
+
+  rex [options] grammarfile
+
+  -o  --output-file  filename   designate output filename
+  -s  --stub                    append stub main for debugging
+  -i  --ignorecase              ignore character case
+  -C  --check-only              only check syntax
+      --independent             independent mode
   -d  --debug                   print debug information
-  -h  --help                    print usage.
-      --version                 print version.
-      --copyright               print copyright.
-
-
-== Default Output Filename
-
-  It destinate from foo.rex  to foo.rex.rb.
-  This name is for a follow description.
-
-    require 'foo.rex' 
-
-
-== Grammar File Format
-
-  A definition is given in order of a header part, a rule part, 
-  and the footer part.  One or more sections are included in a rule part.
-  As for each section, the head of the sentence starts by the keyword.
-
-  Summary:
-
-    [Header Part]
-    "class" Foo
-    ["option"
-      [options] ]
-    ["inner"
-      [methods] ]
-    ["macro"
-      [macro-name  regular-expression] ]
-    "rule"
-      [start-state]  pattern  [actions]
-    "end"
-    [Footer Part]
-
-
-=== Grammar File Example
-
-    class Foo
-    macro
-      BLANK         \s+
-      DIGIT         \d+
-    rule
-      {BLANK}
-      {DIGIT}       { [:NUMBER, text.to_i] }
-      .             { [text, text] }
-    end
-
-
-== Header Part ( Optional )
-
-  All the contents described before the definition of a rule part are 
-  posted to head of the output file.
-
-
-== Footer Part ( Optional )
-
-  All the contents described after the definition of a rule part are 
-  posted to tail of the output file.
-
-
-== Rule Part
-
-  Rule part is from the line which begins from the "class" keyword 
-  to the line which begins from the "end" keyword.
-  The class name outputted after a keyword "class" is specified.
-  If embellished with a module name, it will become a class in a module.
-  The class which inherited Racc::Parser is generated.
-
-
-=== Rule Header Example
-
-    class Foo
-    class Bar::Foo
-
-
-== Option Section ( Optional )
-
-  "option" is start keyword.
-
-    "ignorecase"  when pattern match, ignore char case.
-    "stub"        append stub main for debug.
-    "independent" independent mode, for it is not inherited Racc.
-
-
-== Inner Section ( Optional )
-
-  "inner" is start keyword.
-  The contents defined here are defined by the inside of the class 
-  of the generated scanner.
-
-
-== Macro Section ( Optional )
-
-  "macro" is start keyword.
-  One regular expression is named.
-  A blank character (0x20) can be included by escaping by \ .
-
-
-=== Macro Section Example
-
-    DIGIT         \d+
-    IDENT         [a-zA-Z_][a-zA-Z0-9_]*
-    BLANK         [\ \t]+
-    REMIN         \/\*
-    REMOUT        \*\/
-
-
-== Rule Section
-
-  "rule" is start keyword.
-
-  [state]  pattern  [actions]
-
-
-=== state: Start State ( Optional )
-
-    A start state is expressed with the identifier which prefaces ":".
-    When the continuing alphabetic character is a capital letter, 
-    it will be in an exclusive start state.
-    When it is a small letter, it will be in an inclusive start state.
-    Initial value and default value of a start state is nil.
-
-
-=== pattern: String Pattern
-
-    The regular expression for specifying a character string.
-    The macro definition bundled with { } can be used for description 
-    of a regular expression.  A macro definition is used in order to use 
-    a regular expression including a blank.
-
-
-=== actions: Processing Actions ( Optional )
-
-    Action is performed when a pattern is suited.
-    The processing which creates a suitable token is defined.
-    The arrangement whose token has the second clause of classification 
-    and a value, or nil.
-    The following elements can be used in order to create a token.
-
-      lineno    Line number     ( Read Only )
-      text      Matched string  ( Read Only )
-      state     Start state     ( Read/Write )
-
-    Action is bundled with { }.  It is the block of Ruby.
-    Don't use the function to change the flow of control exceeding a block.
-    ( return, exit, next, break, ... )
-    If action is omitted, the character string which matched will be canceled 
-    and will progress to the next scan.
-
-
-=== Rule Part Example
-
-        {REMIN}                 { state = :REM ; [:REM_IN, text] }
-  :REM  {REMOUT}                { state = nil ; [:REM_OUT, text] }
-  :REM  (.+)(?={REMOUT})        { [:COMMENT, text] }
-        {BLANK}
-        -?{DIGIT}               { [:NUMBER, text.to_i] }
-        {WORD}                  { [:word, text] }
-        .                       { [text, text] }
-
-== Comment ( Optional )
-
-  From "#" to the end of the line becomes a comment in each line.
-
-
-== Usage for Generated Class
-
-=== scan_setup()
-
-    The event for initializing at the time of the execution start of a scanner.
-    It is redefined and used.
-
-
-=== scan_str( str )
-
-    Parse the string described by the defined grammar.
-    Token is stored in an inside.
-
-
-=== scan_file( filename )
-
-    Parse the file described by the defined grammar.
-    Token is stored in an inside.
-
-
-=== next_token
-
-    One token stored in the inside is taken out.
-    The last returns nil.
-
-
-== Notice
-
-  This specification is provisional and may be changed without a preliminary 
-  announcement.
-
+  -h  --help                    print usage
+      --version                 print version
+      --copyright               print copyright
+
+
+== Default Output Filename
+
+  The destination file for foo.rex is foo.rex.rb.
+  To use, include the following in the Ruby source code file.
+
+    require 'foo.rex' 
+
+
+== Grammar File Format
+
+  A definition consists of a header section, a rule section, 
+  and a footer section.  The rule section includes one or more clauses.
+  Each clause starts with a keyword.
+
+  Summary:
+
+    [Header section]
+    "class" Foo
+    ["option"
+      [options] ]
+    ["inner"
+      [methods] ]
+    ["macro"
+      [macro-name  regular-expression] ]
+    "rule"
+      [start-state]  pattern  [actions]
+    "end"
+    [Footer section]
+
+
+=== Grammar File Description Example
+
+    class Foo
+    macro
+      BLANK         \s+
+      DIGIT         \d+
+    rule
+      {BLANK}
+      {DIGIT}       { [:NUMBER, text.to_i] }
+      .             { [text, text] }
+    end
+
+
+== Header Section ( Optional )
+
+  All of the contents described before the definitions in the rule section are 
+  copied to the beginning of the output file.
+
+
+== Footer Section ( Optional )
+
+  All the contents described after the definitions in the rule section are 
+  copied to the end of the output file.
+
+
+== Rule Section
+
+  The rule section starts at the line beginning with the "class" keyword 
+  and ends at the line beginning with the "end" keyword.
+  The class name is specified after the "class" keyword.
+  If a module name is specified, the class will be included in the module.
+  A class that inherits Racc::Parser is generated.
+
+
+=== Example of Rule Section Definition
+
+    class Foo
+    class Bar::Foo
+
+
+== Option Section ( Optional )
+
+  This section begins with the "option" keyword.
+
+    "ignorecase"  ignore the character case when pattern matching
+    "stub"        append stub main for debugging
+    "independent" independent mode, do not inherit Racc.
+
+
+== Inner Section for User Code ( Optional )
+
+  This section begins with the "inner" keyword.
+  The contents defined here are defined by the contents of the class 
+  of the generated scanner.
+
+
+== Macro Section ( Optional )
+
+  This section begins with the "macro" keyword.
+  A name is assigned to one regular expression.
+  A space character (0x20) can be included by using a backslash \ to escape.
+
+
+=== Example of Macro Definition
+
+    DIGIT         \d+
+    IDENT         [a-zA-Z_][a-zA-Z0-9_]*
+    BLANK         [\ \t]+
+    REMIN         \/\*
+    REMOUT        \*\/
+
+
+== Rule Section
+
+  This section begins with the "rule" keyword.
+
+  [state]  pattern  [actions]
+
+
+=== state: Start State ( Optional )
+
+    A start state is indicated by an identifier beginning with ":", a Ruby symbol.
+    If uppercase letters follow the ":", the state becomes an exclusive start state.
+    If lowercase letters follow the ":", the state becomes an inclusive start state.
+    The initial value and the default value of a start state are nil.
+
+
+=== pattern: String Pattern
+
+    A regular expression specifies a character string.
+    A regular expression description may include a macro definition enclosed
+    by curly braces { }.
+    A macro definition is used when the regular expression includes whitespace.
+
+
+=== actions: Processing Actions ( Optional )
+
+    An action is executed when the pattern is matched.
+    The action defines the process for creating the appropriate token.
+    A token is a two-element array containing a type and a value, or is nil.
+    The following elements can be used to create a token.
+
+      lineno    Line number     ( Read Only )
+      text      Matched string  ( Read Only )
+      state     Start state     ( Read/Write )
+
+    The action is a block of Ruby code enclosed by { }.
+    Do not use functions that exit the block and change the control flow.
+    ( return, exit, next, break, ... )
+    If the action is omitted, the matched character string is discarded, 
+    and the process advances to the next scan.
+
+
+=== Example of Rule Section Definition
+
+        {REMIN}                 { state = :REM ; [:REM_IN, text] }
+  :REM  {REMOUT}                { state = nil ; [:REM_OUT, text] }
+  :REM  (.+)(?={REMOUT})        { [:COMMENT, text] }
+        {BLANK}
+        -?{DIGIT}               { [:NUMBER, text.to_i] }
+        {WORD}                  { [:word, text] }
+        .                       { [text, text] }
+
+== Comments ( Optional )
+
+  Any text following a "#" to the end of the line becomes a comment.
+
+
+== Using the Generated Class
+
+=== scan_setup( str )
+
+    Initializes the scanner with the str string argument.
+    This is redefined and used.
+
+
+=== scan_str( str )
+
+    Parses the string described in the defined grammar.
+    The tokens are stored internally.
+
+
+=== scan_file( filename )
+
+    Reads in a file described in the defined grammar.
+    The tokens are stored internally.
+
+
+=== next_token
+
+    The tokens stored internally are extracted one by one.
+    When there are no more tokens, nil is returned.
+
+
+== Notice
+
+  This specification is provisional and may be changed without prior notice.
+