parse_queue 0.1.0 → 0.2.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 2dbdae6528358e96af53ee8a1021337dd6d9c2d8
-  data.tar.gz: 90ce08195ef61d12475c4b66e7751e248e5a5452
+SHA256:
+  metadata.gz: e2a1861627deadb0bac32e7b807cad91fc82d90d5758c5b4969f3ede031bcbdb
+  data.tar.gz: 4cfd26f3b0c455a6394d10b325b12d73dce63f8b5d8e8271d3d4b73da49af594
 SHA512:
-  metadata.gz: 2dc54cd963668ab878f3fe69f6d2ffe7731dab1e5fbc9a49af619035b9d8ebe8df4f4aa93690c98810ae97dc660455932e9f001ed7c3e8d76ac4ca8db1663d4a
-  data.tar.gz: 749792d40d3992d2bdac14b70aecfa7021422127c71a589a2599dfa7173622136eb0f1f1e0c1d27bac5877ffc0ab13ea8631dcb205896925418788af413f2937
+  metadata.gz: 95750f6139a58734609f072f253bcd97eedfc7af81d5dceb6e714ee57e092bba7b71e42ce05bde2688f81a0c7636f02c612d29f4bd18673ff2171c5be10a119c
+  data.tar.gz: 436b508221175b802ae1dee447e1a6e76ca2e07850d1d718ffb8c9fc80a2f43fa6e2a82f79b2af893db91ff78530db190ba69a7230ddfc1d6e29adfc401642dc

data/README.md

@@ -7,6 +7,16 @@ simple queue, it supports backing up or falling back to earlier states allowing
 the parser to try other paths in the syntax tree when one path runs into a
 dead end.
 
+The parse queue was created to simplify the design of both the lexical analyzer
+and the parser. Parsers often have a built-in limited "look-ahead" of tokens.
+This can be seen in parser names like LL(0), LL(1), LR(1), LALR(n), etc. The
+parse queue provides for a flexible look-ahead removing this burden from other
+compiler components.
+
+The operation of the parse queue is summarized in the following diagram:
+
+![parse queue](./images/queue.png)
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -34,11 +44,21 @@ When creating a parse queue, an optional block parameter is passed in. This is
 called whenever more queue items are required. For example:
 
 ```ruby
-  pq = ParseQueue.new { lex.next }
+def open_file_tokenized(name)
+  txt = IO.readlines(name, nil)[0]
+  lex = LexicalAnalayzer.new(text: txt, rules: LEXICAL_RULES)
+  ParseQueue.new { lex.get }
+end
 ```
-If this block is omitted, then items will have to added to the parse queue
-using the add method. The add method accepts single items, multiple items or
-an array of items.
+This example above is a method that reads in the named file, creates an
+analyzer on it (see the
+[lexical_analyzer](https://rubygems.org/gems/lexical_analyzer)
+gem for more details) and then uses that as the source for the parse queue.
+The queue is returned for use by the compiler's parser.
+
+Note: The constant LEXICAL_RULES is a set of rules used to define the tokens
+extracted by the lexical analyzer. As such it is not discussed further here.
+See that gem for more details on how rules are constructed.
 
 #### Getting a queued item:
 
@@ -50,6 +70,12 @@ Getting an item from the queue is done with the get method. For example:
 This method returns the next unread item from the queue. Note that if no items
 are available, the exception **ParseQueueNoFwd** is raised.
 
+Note: The get! method is a get without backtracking. In effect it is a get
+followed by a shift (see Shifting below).
+
+```ruby
+  item = pq.get!
+```
 
 #### Backtracking:
 
@@ -82,7 +108,7 @@ a value saved off at an earlier point of the processing. For example:
 
 #### Shifting
 
-So far, items have been retained in the queue, even when are done being
+So far, items have been retained in the queue, even after they are done being
 processed. For large files, this may use a large amount of memory. To avoid
 this, used items need to be shifted out of the parse queue. This can be done as
 follows:
@@ -96,9 +122,9 @@ follows:
   }
 ```
 Note how the try! block returns a value called success. If this value is false
-or nil, the parse queue is rolled back to its condition at the start of the try
-block. Otherwise, any changes to the parse queue are retained and processed
-items are removed.
+or nil, the parse queue is rolled back to its condition at the start of the
+try! block. Otherwise, any changes to the parse queue are retained and
+processed items are removed.
 
 This too can be done manually as shown below:
 
@@ -116,11 +142,21 @@ This too can be done manually as shown below:
 Note that if an attempt is made to fall back to data that has been shifted out,
 a **ParseQueueNoRev** exception is raised.
 
+#### Exceptions
+
+The parse queue uses the following exception classes:
+
+    Exception              # From Ruby.
+      StandardError        # From Ruby.
+        ParseQueueError    # The abstract root of parse queue exceptions.
+          ParseQueueNoFwd  # Error: Can't go forward.
+          ParseQueueNoRev  # Error: Can't fall back.
+
 ## Contributing
 
 #### Plan A
 
-1. Fork it ( https://github.com/PeterCamilleri/parse_queue_dup/fork )
+1. Fork it ( https://github.com/PeterCamilleri/parse_queue/fork )
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
@@ -134,10 +170,10 @@ aspect that could use some TLC or a suggestion or an idea.
 ## License
 
 The gem is available as open source under the terms of the
-[MIT License](http://opensource.org/licenses/MIT).
+[MIT License](./LICENSE.txt).
 
 ## Code of Conduct
 
 Everyone interacting in the ParseQueue project’s codebases, issue trackers,
 chat rooms and mailing lists is expected to follow the
-[code of conduct](https://github.com/PeterCamilleri/parse_queue/blob/master/CODE_OF_CONDUCT.md).
+[code of conduct](./CODE_OF_CONDUCT.md).

data/lib/parse_queue.rb

@@ -1,54 +1,61 @@
-# coding: utf-8
-
 # The Ruby Compiler Toolkit Project - Parse Queue
 # A queue for compiler objects between parser layers.
 
 require_relative "parse_queue/exceptions"
 require_relative "parse_queue/version"
 
+# The RCTP queue for parser token flow with backtracking.
 class ParseQueue
 
   # The current read point of the queue.
   attr_reader :position
 
-  # The number of old items removed from the queue.
-  attr_reader :offset
+  #The default fetch block
+  DFB = Proc.new { false }
 
   # Set up the parser queue.
   def initialize(&fetch)
-    @fetch = fetch || lambda { false }
+    @fetch  = fetch
     @buffer = []
     @offset = @position = 0
   end
 
   # How many unread items are in this parse queue?
-  def unread
-    @buffer.length - @position + @offset
+  def fwd_count
+    index_limit - @position
   end
 
-  # Manually add items to the buffer
-  def add(*items)
-    @buffer += items.flatten
+  # How many already read items are still in this parse queue?
+  def rev_count
+    @position - @offset
   end
 
   # Get an item from the buffer.
   def get
-    if @position >= (@buffer.length + @offset)
-      item = @fetch.call
-      fail ParseQueueNoFwd unless item
-      @buffer << item
-    end
+    @buffer << fetch_one if @position == index_limit
 
-    result = @buffer[@position - @offset]
+    result = @buffer[rev_count]
     @position += 1
     result
   end
 
-  # Get all possible items
-  def read_all
+  # Get an item and shift the buffer.
+  def get!
+    result = get
+    shift
+    result
+  end
+
+  # Fetch all possible items.
+  def fetch_all
     loop do
       item = @fetch.call
-      return unless item
+
+      unless item
+        @fetch = DFB
+        return
+      end
+
       @buffer << item
     end
   end
@@ -60,40 +67,53 @@ class ParseQueue
   end
 
   # Undo the last get.
-  def back_up
-    @position -= 1
+  def unget(count=1)
+    @position -= count
     validate_position
   end
 
   # Release any items before the current item.
   def shift
-    @buffer.shift(@position - @offset)
+    @buffer.shift(rev_count)
     @offset = @position
   end
 
   # Try to process some items with roll back on failure.
   def try(&block)
     save = @position
-    @position = save unless block.call
-    validate_position
+    self.position = save unless (result = block.call)
+    result
   end
 
-  # Try to process some items with shift of success and roll back on failure.
+  # Process some items with a shift on success and a roll back on failure.
   def try!(&block)
-    save = @position
-
-    if block.call
-      shift
-    else
-      @position = save
-      validate_position
-    end
+    shift if (result = try(&block))
+    result
   end
 
+private
+
   # Is this a valid position?
   def validate_position
     fail ParseQueueNoRev if @position < @offset
-    fail ParseQueueNoFwd if @position >= (@buffer.length + @offset)
+    fail ParseQueueNoFwd if @position >= index_limit
+  end
+
+  # The first index past the end of the array
+  def index_limit
+    @buffer.length + @offset
+  end
+
+  # Fetch a single item.
+  def fetch_one
+    item = @fetch.call
+
+    unless item
+      @fetch = DFB
+      fail ParseQueueNoFwd
+    end
+
+    item
   end
 
 end

data/lib/parse_queue/exceptions.rb

@@ -1,5 +1,3 @@
-# coding: utf-8
-
 # Exception types for the parse queue.
 
 # The base error class for the parse queue.

data/lib/parse_queue/version.rb

@@ -1,5 +1,3 @@
-# coding: utf-8
-
 class ParseQueue
-  VERSION = "0.1.0"
+  VERSION = "0.2.5".freeze
 end

data/parse_queue.gemspec

@@ -18,8 +18,10 @@ Gem::Specification.new do |spec|
   spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test)/}) }
   spec.require_paths = ["lib"]
 
-  spec.add_development_dependency "bundler", "~> 1.15"
-  spec.add_development_dependency "rake", "~> 10.0"
+  spec.required_ruby_version = '>=2.3.0'
+
+  spec.add_development_dependency "bundler", ">= 2.1.0"
+  spec.add_development_dependency "rake", ">= 12.3.3"
   spec.add_development_dependency "minitest", "~> 5.0"
-  spec.add_development_dependency 'minitest_visible', "~> 0.1"
+  spec.add_development_dependency 'reek', "~> 5.0.2"
 end

data/rakefile.rb

@@ -8,3 +8,14 @@ Rake::TestTask.new(:test) do |t|
 end
 
 task :default => :test
+
+desc "Run a scan for smelly code!"
+task :reek do |t|
+  `reek --no-color lib > reek.txt`
+end
+
+desc "What version of parse queue is this?"
+task :vers do |t|
+  puts
+  puts "parse_queue version = #{ParseQueue::VERSION}"
+end

data/reek.txt

@@ -0,0 +1 @@
+0 total warnings

metadata

@@ -1,43 +1,43 @@
 --- !ruby/object:Gem::Specification
 name: parse_queue
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.5
 platform: ruby
 authors:
 - PeterCamilleri
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-08-22 00:00:00.000000000 Z
+date: 2021-05-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.15'
+        version: 2.1.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.15'
+        version: 2.1.0
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: 12.3.3
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: 12.3.3
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
@@ -53,19 +53,19 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '5.0'
 - !ruby/object:Gem::Dependency
-  name: minitest_visible
+  name: reek
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.1'
+        version: 5.0.2
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.1'
+        version: 5.0.2
 description: The RCTP object queue for moving compiler tokens with nestable backtrack
   capability.
 email:
@@ -79,11 +79,13 @@ files:
 - Gemfile
 - LICENSE.txt
 - README.md
+- images/queue.png
 - lib/parse_queue.rb
 - lib/parse_queue/exceptions.rb
 - lib/parse_queue/version.rb
 - parse_queue.gemspec
 - rakefile.rb
+- reek.txt
 homepage: https://github.com/PeterCamilleri/parse_queue
 licenses:
 - MIT
@@ -96,15 +98,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 2.3.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.5.2
+rubygems_version: 3.2.17
 signing_key: 
 specification_version: 4
 summary: The RCTP object queue for connecting parser steps.