puppet-lint 2.1.1 → 2.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 09c3216991dd5d40c797bb951adeef212f612456
-  data.tar.gz: ac37d47099b67bf5693ce27b8312fcff4648419e
+  metadata.gz: 3d618e442c08c4d3e79853fc370348c9ee8e8970
+  data.tar.gz: 82099093c21a374e137a39bd0323443e16015e95
 SHA512:
-  metadata.gz: f66775062fc637446250708ff5fb9fb5106a38bc758a6276bf325a7db6b97c4a50f528b964cc0bda44e1c550c8cc2b38ac0d83489dc1f97e9508c2a43c350337
-  data.tar.gz: 41b45bc46f3be42f046319ffdc588cdc70df842da61cdd098e032b44ba752d805128a88d95296b66b91d35313608230dff491de010e5e10f195ec9bb184e4498
+  metadata.gz: 179cd2b6f7ada144261bf630064a4ca26a212d3693fba9244a9a4fbd2b8c75ec78f6935f50175845bf2d743e53671f19c683e9e5ccb14b950e7d6e9e86c269d4
+  data.tar.gz: 8d5f9a973d53c47e8ce6a7caac88c4aa8ab7637e95221e95676b7adf40b8cc8c2ed9b615d09f19857aeadbc36d7629d86518ba644fbaf08845050ae2c2018201

data/CHANGELOG.md

@@ -1,5 +1,65 @@
 # Change Log
 
+## [2.2.0](https://github.com/rodjek/puppet-lint/tree/2.2.0) (2017-03-29)
+[Full Changelog](https://github.com/rodjek/puppet-lint/compare/2.1.1...2.2.0)
+
+**Closed issues:**
+
+- Missed bad file modes in file with multiple resource bodies [\#663](https://github.com/rodjek/puppet-lint/issues/663)
+- Provide helper methods to search for specific tokens [\#660](https://github.com/rodjek/puppet-lint/issues/660)
+- ensure\_first\_param fix can create invalid syntax [\#659](https://github.com/rodjek/puppet-lint/issues/659)
+- puppet-lint dies with inline\_template syntax [\#656](https://github.com/rodjek/puppet-lint/issues/656)
+- Variable use like "${$a}" isn't reported as an error and is changed with no message when run with --fix [\#655](https://github.com/rodjek/puppet-lint/issues/655)
+- Linter gets confused with arrays of hashes [\#654](https://github.com/rodjek/puppet-lint/issues/654)
+- 2.1.1 git tag [\#652](https://github.com/rodjek/puppet-lint/issues/652)
+- heredoc throws unhandled exception [\#649](https://github.com/rodjek/puppet-lint/issues/649)
+- Quoted boolean triggers on the command 'true' [\#646](https://github.com/rodjek/puppet-lint/issues/646)
+- Match function breaks puppet-lint [\#645](https://github.com/rodjek/puppet-lint/issues/645)
+- top-scope variable being used without an explicit namespace in a string with a lookup [\#635](https://github.com/rodjek/puppet-lint/issues/635)
+- unquoted file mode & mode should be represented as a 4 digit when file mode is done by a lookup [\#634](https://github.com/rodjek/puppet-lint/issues/634)
+- Namevars detected as optional parameters [\#633](https://github.com/rodjek/puppet-lint/issues/633)
+- 'Duplicate Parameter' warning on dynamic class parameter [\#627](https://github.com/rodjek/puppet-lint/issues/627)
+- double\_quoted\_strings-check issue with escaped character in the string [\#625](https://github.com/rodjek/puppet-lint/issues/625)
+- unable to disable 140chars check with control comments "block" [\#622](https://github.com/rodjek/puppet-lint/issues/622)
+- arrow\_alignment should only check the alignment of the first arrow on each line [\#609](https://github.com/rodjek/puppet-lint/issues/609)
+- unquoted\_node\_name crash when curly braces missing [\#582](https://github.com/rodjek/puppet-lint/issues/582)
+- Heredoc triggers exception in arrow\_alignment check [\#578](https://github.com/rodjek/puppet-lint/issues/578)
+- Each + With = Fake positive top-scope variable detection [\#576](https://github.com/rodjek/puppet-lint/issues/576)
+- Top-scope variable warning on inline lambda [\#549](https://github.com/rodjek/puppet-lint/issues/549)
+- Top-scope variable warning on nested each loops [\#548](https://github.com/rodjek/puppet-lint/issues/548)
+- Puppet-lint crash with new puppet 4 syntax [\#516](https://github.com/rodjek/puppet-lint/issues/516)
+- `arrow\_alignment --fix` doesn't indent keys when introducing line breaks; erroneously reports success [\#506](https://github.com/rodjek/puppet-lint/issues/506)
+- Top-scope warning when looping though an array of hashes [\#464](https://github.com/rodjek/puppet-lint/issues/464)
+- Array of hashes one-liner throws a "Indentation of =\> is not properly aligned" [\#446](https://github.com/rodjek/puppet-lint/issues/446)
+- heredoc escape gives syntax error [\#430](https://github.com/rodjek/puppet-lint/issues/430)
+- NoMethodError when multiple heredocs are used [\#395](https://github.com/rodjek/puppet-lint/issues/395)
+
+**Merged pull requests:**
+
+- Ignore selectors when finding resource type [\#678](https://github.com/rodjek/puppet-lint/pull/678) ([rodjek](https://github.com/rodjek))
+- Fix for arrow\_alignment bugs in \#506 [\#677](https://github.com/rodjek/puppet-lint/pull/677) ([rodjek](https://github.com/rodjek))
+- Support double quoted strings inside interpolated values in double quoted strings [\#676](https://github.com/rodjek/puppet-lint/pull/676) ([rodjek](https://github.com/rodjek))
+- Don't silently remove unnecessary $ from enclosed variables [\#674](https://github.com/rodjek/puppet-lint/pull/674) ([rodjek](https://github.com/rodjek))
+- Fix ensure\_first\_param fix method to retrieve the full value of the ensure parameter [\#673](https://github.com/rodjek/puppet-lint/pull/673) ([rodjek](https://github.com/rodjek))
+- Check that arrow is on the line of right operand [\#672](https://github.com/rodjek/puppet-lint/pull/672) ([Darhazer](https://github.com/Darhazer))
+- Restrict appveyor testing to Ruby versions that appveyor supports [\#670](https://github.com/rodjek/puppet-lint/pull/670) ([james-stocks](https://github.com/james-stocks))
+- Clear expected parameter column after processing each block when checking arrow alignment [\#669](https://github.com/rodjek/puppet-lint/pull/669) ([rodjek](https://github.com/rodjek))
+- Allow regexps to used as function arguments [\#665](https://github.com/rodjek/puppet-lint/pull/665) ([rodjek](https://github.com/rodjek))
+- Catch unhandled exception and provide debug info for issue [\#664](https://github.com/rodjek/puppet-lint/pull/664) ([rodjek](https://github.com/rodjek))
+- Fix showing of a failure [\#662](https://github.com/rodjek/puppet-lint/pull/662) ([Darhazer](https://github.com/Darhazer))
+- \(SDK-115\) Enable appveyor testing [\#653](https://github.com/rodjek/puppet-lint/pull/653) ([james-stocks](https://github.com/james-stocks))
+- Heredoc support [\#650](https://github.com/rodjek/puppet-lint/pull/650) ([rodjek](https://github.com/rodjek))
+- Warn when control comment blocks are not properly terminated [\#648](https://github.com/rodjek/puppet-lint/pull/648) ([rodjek](https://github.com/rodjek))
+- Prevent incomplete node blocks from crashing puppet-lint [\#642](https://github.com/rodjek/puppet-lint/pull/642) ([rodjek](https://github.com/rodjek))
+- Update the tokeniser to differentiate between unquoted strings and function names [\#640](https://github.com/rodjek/puppet-lint/pull/640) ([rodjek](https://github.com/rodjek))
+- Check for escaped backslashes too [\#639](https://github.com/rodjek/puppet-lint/pull/639) ([binford2k](https://github.com/binford2k))
+- Correctly handle function calls inside string interpolation [\#638](https://github.com/rodjek/puppet-lint/pull/638) ([hanazuki](https://github.com/hanazuki))
+- Correctly handle nested lambdas [\#637](https://github.com/rodjek/puppet-lint/pull/637) ([hanazuki](https://github.com/hanazuki))
+- Support for nested multiple-variable assignments [\#636](https://github.com/rodjek/puppet-lint/pull/636) ([hanazuki](https://github.com/hanazuki))
+- Add LoadError to fix broken tests [\#631](https://github.com/rodjek/puppet-lint/pull/631) ([davidmogar](https://github.com/davidmogar))
+- Deal with ruby 1.8.7 gem issues [\#630](https://github.com/rodjek/puppet-lint/pull/630) ([mterzo](https://github.com/mterzo))
+- Plugin review; disable unnecessary plugins [\#567](https://github.com/rodjek/puppet-lint/pull/567) ([rnelson0](https://github.com/rnelson0))
+
 ## [2.1.1](https://github.com/rodjek/puppet-lint/tree/2.1.1) (2017-02-13)
 [Full Changelog](https://github.com/rodjek/puppet-lint/compare/2.1.0...2.1.1)
 
@@ -15,6 +75,7 @@
 
 **Merged pull requests:**
 
+- puppet-lint 2.1.1 release [\#628](https://github.com/rodjek/puppet-lint/pull/628) ([rnelson0](https://github.com/rnelson0))
 - Correctly handle strings-with-variables as hash keys in arrow\_alignment check [\#621](https://github.com/rodjek/puppet-lint/pull/621) ([rodjek](https://github.com/rodjek))
 - Support array of variables on left side of an assign operation [\#617](https://github.com/rodjek/puppet-lint/pull/617) ([rodjek](https://github.com/rodjek))
 - Test against Ruby 2.4.0 [\#616](https://github.com/rodjek/puppet-lint/pull/616) ([rodjek](https://github.com/rodjek))
@@ -181,25 +242,6 @@
 ## [2.0.0](https://github.com/rodjek/puppet-lint/tree/2.0.0) (2016-06-22)
 [Full Changelog](https://github.com/rodjek/puppet-lint/compare/1.1.0...2.0.0)
 
-puppet-lint 2.0.0 is a breaking change. Specifically, the renaming of the line length test was changed from `80chars` to `140chars`. You may need to adjust your configuration and lint checks. For example:
-```ruby
-# Line length test is 80 chars in puppet-lint 1.1.0
-PuppetLint.configuration.send('disable_80chars')
-# Line length test is 140 chars in puppet-lint 2.x
-PuppetLint.configuration.send('disable_140chars')
-```
-
-You may also need to adjust your Gemfile if you are pointing directly at git:
-```ruby
-# old
-gem 'puppet-lint', :require => false, :git => 'https://github.com/rodjek/puppet-lint.git'
-
-# new
-gem 'puppet-lint', '~> 2.0'
-```
-
-If the additional gems you use for checks are pinned to 1.x, you should pin puppet-lint to `'~> 1.0'` or `'>= 1.0', '< 3.0'` until updated check gems are released.
-
 **Closed issues:**
 
 - Current package [\#471](https://github.com/rodjek/puppet-lint/issues/471)

data/Gemfile

@@ -4,7 +4,6 @@ gemspec
 
 group :test do
   gem 'rake', '~> 10.0'
-  gem 'rspec', '~> 3.0'
   gem 'rspec-its', '~> 1.0'
   gem 'rspec-collection_matchers', '~> 1.0'
 
@@ -15,10 +14,13 @@ group :test do
     gem 'json_pure', '= 1.8.3'
     # addressable 2.4.0 requires ruby 1.9.0. Lock to 2.3.8.
     gem 'addressable', '= 2.3.8'
+    gem 'diff-lcs', '< 1.3'
+    gem 'rspec', '<= 3.4'
   else
+    gem 'rspec', '~> 3.0'
     gem 'json'
   end
-  
+
   if RUBY_VERSION > '1.8'
     # requires ruby 1.9+, on 1.8 we'll fall back to the old regex parsing
     gem 'rspec-json_expectations', '~> 1.4'
@@ -27,7 +29,9 @@ end
 
 group :development do
   # For Changelog generation
-  gem 'github_changelog_generator',                                 :require => false if RUBY_VERSION >= '2.2.2'
-  gem 'github_changelog_generator', '~> 1.13.0',                    :require => false if RUBY_VERSION < '2.2.2'
-  gem 'rack', '~> 1.0',                                             :require => false if RUBY_VERSION < '2.2.2'
+  if RUBY_VERSION > '1.9'
+    gem 'github_changelog_generator',                                 :require => false if RUBY_VERSION >= '2.2.2'
+    gem 'github_changelog_generator', '~> 1.13.0',                    :require => false if RUBY_VERSION < '2.2.2'
+    gem 'rack', '~> 1.0',                                             :require => false if RUBY_VERSION < '2.2.2'
+  end
 end

data/appveyor.yml

@@ -0,0 +1,32 @@
+build: off
+
+branches:
+  only:
+    - master
+
+# ruby versions under test
+environment:
+  matrix:
+    - RUBY_VERSION: 193
+    - RUBY_VERSION: 200
+    - RUBY_VERSION: 21
+    - RUBY_VERSION: 22
+    - RUBY_VERSION: 23-x64
+
+install:
+  - SET PATH=C:\Ruby%RUBY_VERSION%\bin;%PATH%
+  - SET LOG_SPEC_ORDER=true
+  - bundle install --jobs 4 --retry 2 --without development
+
+before_test:
+  - type Gemfile.lock
+  - ruby -v
+  - gem -v
+  - bundle -v
+
+test_script:
+  - bundle exec rake test
+
+notifications:
+  email:
+    - tim@bombasticmonkey.com

data/lib/puppet-lint/checkplugin.rb

@@ -21,7 +21,7 @@ class PuppetLint::CheckPlugin
     check
 
     @problems.each do |problem|
-      if PuppetLint::Data.ignore_overrides[problem[:check]].has_key?(problem[:line])
+      if problem[:check] != :syntax && PuppetLint::Data.ignore_overrides[problem[:check]].has_key?(problem[:line])
         problem[:kind] = :ignored
         problem[:reason] = PuppetLint::Data.ignore_overrides[problem[:check]][problem[:line]]
         next
@@ -35,7 +35,7 @@ class PuppetLint::CheckPlugin
   #
   # Returns an Array of problem Hashes.
   def fix_problems
-    @problems.reject { |problem| problem[:kind] == :ignored }.each do |problem|
+    @problems.reject { |problem| problem[:kind] == :ignored || problem[:check] == :syntax }.each do |problem|
       if self.respond_to?(:fix)
         begin
           fix(problem)

data/lib/puppet-lint/checks.rb

@@ -65,6 +65,27 @@ class PuppetLint::Checks
     end
 
     @problems
+  rescue => e
+    puts <<-END
+Whoops! It looks like puppet-lint has encountered an error that it doesn't
+know how to handle. Please open an issue at https://github.com/rodjek/puppet-lint
+and paste the following output into the issue description.
+---
+puppet-lint version: #{PuppetLint::VERSION}
+ruby version: #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL}
+platform: #{RUBY_PLATFORM}
+file path: #{fileinfo}
+file contents:
+```
+#{File.read(fileinfo) if File.readable?(fileinfo)}
+```
+error:
+```
+#{e.class}: #{e.message}
+#{e.backtrace.join("\n")}
+```
+END
+    exit 1
   end
 
   # Internal: Get a list of checks that have not been disabled.

data/lib/puppet-lint/data.rb

@@ -105,38 +105,28 @@ class PuppetLint::Data
     #   :end   - An Integer position in the `tokens` Array pointing to the last
     #            Token of a resource declaration.
     def resource_indexes
-      @resource_indexes ||= Proc.new do
+      @resource_indexes ||= begin
+        marker = 0
         result = []
-        tokens.each_index do |token_idx|
-          if tokens[token_idx].type == :COLON
-            next_token = tokens[token_idx].next_code_token
-            depth = 1
-            if next_token && next_token.type != :LBRACE
-              tokens[(token_idx + 1)..-1].each_index do |idx|
-                real_idx = token_idx + idx + 1
-                if tokens[real_idx].type == :LBRACE
-                  depth += 1
-                elsif {:SEMIC => true, :RBRACE => true}.include? tokens[real_idx].type
-                  unless tokens[real_idx].type == :SEMIC && depth > 1
-                    depth -= 1
-                    if depth == 0
-                      result << {
-                        :start        => token_idx + 1,
-                        :end          => real_idx,
-                        :tokens       => tokens[(token_idx + 1)..real_idx],
-                        :type         => find_resource_type_token(token_idx),
-                        :param_tokens => find_resource_param_tokens(tokens[(token_idx + 1)..real_idx]),
-                      }
-                      break
-                    end
-                  end
-                end
-              end
-            end
+        tokens.select { |t| t.type == :COLON }.each do |colon_token|
+          if colon_token.next_code_token && colon_token.next_code_token != :LBRACE
+            start_idx = tokens.index(colon_token)
+            next if start_idx < marker
+            end_token = colon_token.next_token_of([:SEMIC, :RBRACE])
+            end_idx = tokens.index(end_token)
+
+            result << {
+              :start        => start_idx + 1,
+              :end          => end_idx,
+              :tokens       => tokens[start_idx..end_idx],
+              :type         => find_resource_type_token(start_idx),
+              :param_tokens => find_resource_param_tokens(tokens[start_idx..end_idx]),
+            }
+            marker = end_idx
           end
         end
         result
-      end.call
+      end
     end
 
     # Internal: Find the Token representing the type of a resource definition.
@@ -146,7 +136,10 @@ class PuppetLint::Data
     #
     # Returns a Token object.
     def find_resource_type_token(index)
-      tokens[tokens[0..index].rindex { |token| token.type == :LBRACE }].prev_code_token
+      lbrace_idx = tokens[0..index].rindex { |token|
+        token.type == :LBRACE && token.prev_code_token.type != :QMARK
+      }
+      tokens[lbrace_idx].prev_code_token
     end
 
     # Internal: Find all the Token objects representing the parameter names in
@@ -497,7 +490,6 @@ class PuppetLint::Data
 
           if command == 'ignore'
             check = split_control[2].to_sym
-
             if token.prev_token && !Set[:NEWLINE, :INDENT].include?(token.prev_token.type)
               # control comment at the end of the line, override applies to
               # a single line only
@@ -524,6 +516,10 @@ class PuppetLint::Data
         end
         stack << stack_add unless stack_add.empty?
       end
+
+      stack.each do |control|
+        puts "WARNING: lint:ignore:#{control[0][2]} comment on line #{control[0][0]} with no closing lint:endignore comment"
+      end
     end
   end
 end

data/lib/puppet-lint/lexer.rb

@@ -29,6 +29,7 @@ class PuppetLint
     def initialize
       @line_no = 1
       @column = 1
+      @@heredoc_queue ||= []
     end
 
     # Internal: A Hash whose keys are Strings representing reserved keywords in
@@ -83,17 +84,20 @@ class PuppetLint
       :LBRACK  => true,
       :IF      => true,
       :ELSIF   => true,
+      :LPAREN  => true,
     }
 
     # Internal: An Array of Arrays containing tokens that can be described by
     # a single regular expression.  Each sub-Array contains 2 elements, the
     # name of the token as a Symbol and a regular expression describing the
     # value of the token.
+    NAME_RE = /\A(((::)?[_a-z0-9][-\w]*)(::[a-z0-9][-\w]*)*)/
     KNOWN_TOKENS = [
       [:TYPE, /\A(Integer|Float|Boolean|Regexp|String|Array|Hash|Resource|Class|Collection|Scalar|Numeric|CatalogEntry|Data|Tuple|Struct|Optional|NotUndef|Variant|Enum|Pattern|Any|Callable|Type|Runtime|Undef|Default)\b/],
       [:CLASSREF, /\A(((::){0,1}[A-Z][-\w]*)+)/],
       [:NUMBER, /\A\b((?:0[xX][0-9A-Fa-f]+|0?\d+(?:\.\d+)?(?:[eE]-?\d+)?))\b/],
-      [:NAME, /\A(((::)?[_a-z0-9][-\w]*)(::[a-z0-9][-\w]*)*)/],
+      [:FUNCTION_NAME, /#{NAME_RE}\(/],
+      [:NAME, NAME_RE],
       [:LBRACK, /\A(\[)/],
       [:RBRACK, /\A(\])/],
       [:LBRACE, /\A(\{)/],
@@ -128,7 +132,6 @@ class PuppetLint
       [:COMMA, /\A(,)/],
       [:DOT, /\A(\.)/],
       [:COLON, /\A(:)/],
-      [:AT, /\A(@)/],
       [:SEMIC, /\A(;)/],
       [:QMARK, /\A(\?)/],
       [:BACKSLASH, /\A(\\)/],
@@ -176,12 +179,12 @@ class PuppetLint
             length = value.size
             if type == :NAME
               if KEYWORDS.include? value
-                tokens << new_token(value.upcase.to_sym, value, length)
+                tokens << new_token(value.upcase.to_sym, value)
               else
-                tokens << new_token(type, value, length)
+                tokens << new_token(type, value)
               end
             else
-              tokens << new_token(type, value, length)
+              tokens << new_token(type, value)
             end
             i += length
             found = true
@@ -192,28 +195,33 @@ class PuppetLint
         unless found
           if var_name = chunk[/\A\$((::)?(\w+(-\w+)*::)*\w+(-\w+)*(\[.+?\])*)/, 1]
             length = var_name.size + 1
-            tokens << new_token(:VARIABLE, var_name, length)
+            tokens << new_token(:VARIABLE, var_name)
 
           elsif chunk.match(/\A'(.*?)'/m)
             str_content = StringScanner.new(code[i+1..-1]).scan_until(/(\A|[^\\])(\\\\)*'/m)
             length = str_content.size + 1
-            tokens << new_token(:SSTRING, str_content[0..-2], length)
+            tokens << new_token(:SSTRING, str_content[0..-2])
 
           elsif chunk.match(/\A"/)
-            str_contents = StringScanner.new(code[i+1..-1]).scan_until(/(\A|[^\\])(\\\\)*"/m)
+            str_contents = slurp_string(code[i+1..-1])
             _ = code[0..i].split("\n")
             interpolate_string(str_contents, _.count, _.last.length)
             length = str_contents.size + 1
 
+          elsif heredoc_name = chunk[/\A@\(("?.+?"?(:.+?)?(\/.*?)?)\)/, 1]
+            @@heredoc_queue << heredoc_name
+            tokens << new_token(:HEREDOC_OPEN, heredoc_name)
+            length = heredoc_name.size + 3
+
           elsif comment = chunk[/\A(#.*)/, 1]
             length = comment.size
             comment.sub!(/#/, '')
-            tokens << new_token(:COMMENT, comment, length)
+            tokens << new_token(:COMMENT, comment)
 
           elsif slash_comment = chunk[/\A(\/\/.*)/, 1]
             length = slash_comment.size
             slash_comment.sub!(/\/\//, '')
-            tokens << new_token(:SLASH_COMMENT, slash_comment, length)
+            tokens << new_token(:SLASH_COMMENT, slash_comment)
 
           elsif mlcomment = chunk[/\A(\/\*.*?\*\/)/m, 1]
             length = mlcomment.size
@@ -221,32 +229,54 @@ class PuppetLint
             mlcomment.sub!(/\A\/\* ?/, '')
             mlcomment.sub!(/ ?\*\/\Z/, '')
             mlcomment.gsub!(/^ *\*/, '')
-            tokens << new_token(:MLCOMMENT, mlcomment, length)
-            tokens.last.raw = mlcomment_raw
+            tokens << new_token(:MLCOMMENT, mlcomment, :raw => mlcomment_raw)
 
           elsif chunk.match(/\A\/.*?\//) && possible_regex?
             str_content = StringScanner.new(code[i+1..-1]).scan_until(/(\A|[^\\])(\\\\)*\//m)
             length = str_content.size + 1
-            tokens << new_token(:REGEX, str_content[0..-2], length)
+            tokens << new_token(:REGEX, str_content[0..-2])
 
           elsif eolindent = chunk[/\A((\r\n|\r|\n)[ \t]+)/m, 1]
             eol = eolindent[/\A([\r\n]+)/m, 1]
-            indent = eolindent[/\A[\r\n]+([ \t]+)/m, 1]
-            tokens << new_token(:NEWLINE, eol, eol.size)
-            tokens << new_token(:INDENT, indent, indent.size)
-            length = indent.size + eol.size
+            tokens << new_token(:NEWLINE, eol)
+            length = eol.size
+
+            if @@heredoc_queue.empty?
+              indent = eolindent[/\A[\r\n]+([ \t]+)/m, 1]
+              tokens << new_token(:INDENT, indent)
+              length += indent.size
+            else
+              heredoc_tag = @@heredoc_queue.shift
+              heredoc_name = heredoc_tag[/\A"?(.+?)"?(:.+?)?(\/.*)?\Z/, 1]
+              str_contents = StringScanner.new(code[i+length..-1]).scan_until(/\|?\s*-?\s*#{heredoc_name}/)
+              interpolate_heredoc(str_contents, heredoc_tag)
+              length += str_contents.size
+            end
 
           elsif whitespace = chunk[/\A([ \t]+)/, 1]
             length = whitespace.size
-            tokens << new_token(:WHITESPACE, whitespace, length)
+            tokens << new_token(:WHITESPACE, whitespace)
 
           elsif eol = chunk[/\A(\r\n|\r|\n)/, 1]
             length = eol.size
-            tokens << new_token(:NEWLINE, eol, length)
+            tokens << new_token(:NEWLINE, eol)
+
+            unless @@heredoc_queue.empty?
+              heredoc_tag = @@heredoc_queue.shift
+              heredoc_name = heredoc_tag[/\A"?(.+?)"?(:.+?)?(\/.*)?\Z/, 1]
+              str_contents = StringScanner.new(code[i+length..-1]).scan_until(/\|?\s*-?\s*#{heredoc_name}/)
+              _ = code[0..i+length].split("\n")
+              interpolate_heredoc(str_contents, heredoc_tag)
+              length += str_contents.size
+            end
 
           elsif chunk.match(/\A\//)
             length = 1
-            tokens << new_token(:DIV, '/', length)
+            tokens << new_token(:DIV, '/')
+
+          elsif chunk.match(/\A@/)
+            length = 1
+            tokens << new_token(:AT, '@')
 
           else
             raise PuppetLint::LexerError.new(@line_no, @column)
@@ -259,6 +289,18 @@ class PuppetLint
       tokens
     end
 
+
+    def slurp_string(string)
+      dq_str_regexp = /(\$\{|(\A|[^\\])(\\\\)*")/m
+      scanner = StringScanner.new(string)
+      contents = scanner.scan_until(dq_str_regexp)
+      until scanner.matched.end_with?('"')
+        contents += scanner.scan_until(/\}/m)
+        contents += scanner.scan_until(dq_str_regexp)
+      end
+      contents
+    end
+
     # Internal: Given the tokens already processed, determine if the next token
     # could be a regular expression.
     #
@@ -282,14 +324,19 @@ class PuppetLint
     #
     # type   - The Symbol token type.
     # value  - The token value.
-    # length - The Integer length of the token's value.
     # opts   - A Hash of additional values required to determine line number and
     #         column:
     #   :line   - The Integer line number if calculated externally.
     #   :column - The Integer column number if calculated externally.
+    #   :raw    - The String raw value of the token (if necessary).
     #
     # Returns the instantiated PuppetLint::Lexer::Token object.
-    def new_token(type, value, length, opts = {})
+    def new_token(type, value, *args)
+      # This bit of magic is used instead of an "opts = {}" argument so that we
+      # can safely deprecate the old "length" parameter that might still be
+      # passed by 3rd party plugins that haven't updated yet.
+      opts = args.last.is_a?(Hash) ? args.last : {}
+
       column = opts[:column] || @column
       line_no = opts[:line] || @line_no
 
@@ -308,25 +355,24 @@ class PuppetLint
         end
       end
 
-      @column += length
-
-      # If creating a :VARIABLE token inside a double quoted string, add 3 to
-      # the column state in order to account for the ${} characters when
-      # rendering out to manifest.
-      if token.type == :VARIABLE
-        if !token.prev_code_token.nil? && [:DQPRE, :DQMID].include?(token.prev_code_token.type)
-          @column += 3
-        end
+      if opts[:raw]
+        token.raw = opts[:raw]
       end
 
       if type == :NEWLINE
         @line_no += 1
         @column = 1
-      end
-      if [:MLCOMMENT, :SSTRING, :STRING].include? type and /(?:\r\n|\r|\n)/.match(value)
-        lines = value.split(/(?:\r\n|\r|\n)/, -1)
-        @line_no += lines.length-1
-        @column = lines.last.length
+      else
+        lines = token.to_manifest.split(/(?:\r\n|\r|\n)/, -1)
+        @line_no += lines.length - 1
+        if lines.length > 1
+          # if the token renders to multiple lines, set the column state to the
+          # length of the last line plus 1 (because column numbers are
+          # 1 indexed)
+          @column = lines.last.size + 1
+        else
+          @column += (lines.last || "").size
+        end
       end
 
       token
@@ -364,34 +410,37 @@ class PuppetLint
       until value.nil?
         if terminator == "\""
           if first
-            tokens << new_token(:STRING, value, value.size + 2, :line => line, :column => column)
+            tokens << new_token(:STRING, value, :line => line, :column => column)
             first = false
           else
             line += value.scan(/(\r\n|\r|\n)/).size
             token_column = column + (ss.pos - value.size)
-            tokens << new_token(:DQPOST, value, value.size + 1, :line => line, :column => token_column)
+            tokens << new_token(:DQPOST, value, :line => line, :column => token_column)
+            @column = token_column + 1
+            @line_no = line
           end
         else
           if first
-            tokens << new_token(:DQPRE, value, value.size + 1, :line => line, :column => column)
+            tokens << new_token(:DQPRE, value, :line => line, :column => column)
             first = false
           else
             line += value.scan(/(\r\n|\r|\n)/).size
             token_column = column + (ss.pos - value.size)
-            tokens << new_token(:DQMID, value, value.size, :line => line, :column => token_column)
+            tokens << new_token(:DQMID, value, :line => line, :column => token_column)
           end
           if ss.scan(/\{/).nil?
             var_name = ss.scan(/(::)?(\w+(-\w+)*::)*\w+(-\w+)*/)
             if var_name.nil?
               token_column = column + ss.pos - 1
-              tokens << new_token(:DQMID, "$", 1, :line => line, :column => token_column)
+              tokens << new_token(:DQMID, "$", :line => line, :column => token_column)
             else
               token_column = column + (ss.pos - var_name.size)
-              tokens << new_token(:UNENC_VARIABLE, var_name, var_name.size, :line => line, :column => token_column)
+              tokens << new_token(:UNENC_VARIABLE, var_name, :line => line, :column => token_column)
             end
           else
             contents = ss.scan_until(/\}/)[0..-2]
-            if contents.match(/\A(::)?([\w-]+::)*[\w-]+(\[.+?\])*/)
+            raw = contents.dup
+            if contents.match(/\A(::)?([\w-]+::)*[\w-]+(\[.+?\])*/) && !contents.match(/\A\w+\(/)
               contents = "$#{contents}"
             end
             lexer = PuppetLint::Lexer.new
@@ -399,12 +448,98 @@ class PuppetLint
             lexer.tokens.each do |token|
               tok_col = column + token.column + (ss.pos - contents.size - 1)
               tok_line = token.line + line - 1
-              tokens << new_token(token.type, token.value, token.value.size, :line => tok_line, :column => tok_col)
+              tokens << new_token(token.type, token.value, :line => tok_line, :column => tok_col)
+            end
+            if lexer.tokens.length == 1 && lexer.tokens[0].type == :VARIABLE
+              tokens.last.raw = raw
             end
           end
         end
         value, terminator = get_string_segment(ss, '"$')
       end
     end
+
+    # Internal: Tokenise the contents of a heredoc.
+    #
+    # string - The String to be tokenised.
+    # name   - The String name/endtext of the heredoc.
+    #
+    # Returns nothing.
+    def interpolate_heredoc(string, name)
+      ss = StringScanner.new(string)
+      eos_text = name[/\A"?(.+?)"?(:.+?)?(\/.*)?\Z/, 1]
+      first = true
+      interpolate = name.start_with?('"')
+      value, terminator = get_heredoc_segment(ss, eos_text, interpolate)
+      until value.nil?
+        if terminator =~ /\A\|?\s*-?\s*#{Regexp.escape(eos_text)}/
+          if first
+            tokens << new_token(:HEREDOC, value, :raw => "#{value}#{terminator}")
+            first = false
+          else
+            tokens << new_token(:HEREDOC_POST, value, :raw => "#{value}#{terminator}")
+          end
+        else
+          if first
+            tokens << new_token(:HEREDOC_PRE, value)
+            first = false
+          else
+            tokens << new_token(:HEREDOC_MID, value)
+          end
+          if ss.scan(/\{/).nil?
+            var_name = ss.scan(/(::)?(\w+(-\w+)*::)*\w+(-\w+)*/)
+            if var_name.nil?
+              tokens << new_token(:HEREDOC_MID, "$")
+            else
+              tokens << new_token(:UNENC_VARIABLE, var_name)
+            end
+          else
+            contents = ss.scan_until(/\}/)[0..-2]
+            raw = contents.dup
+            if contents.match(/\A(::)?([\w-]+::)*[\w-]|(\[.+?\])*/) && !contents.match(/\A\w+\(/)
+              contents = "$#{contents}" unless contents.start_with?("$")
+            end
+
+            lexer = PuppetLint::Lexer.new
+            lexer.tokenise(contents)
+            lexer.tokens.each do |token|
+              tokens << new_token(token.type, token.value)
+            end
+            if lexer.tokens.length == 1 && lexer.tokens[0].type == :VARIABLE
+              tokens.last.raw = raw
+            end
+          end
+        end
+        value, terminator = get_heredoc_segment(ss, eos_text, interpolate)
+      end
+    end
+
+    # Internal: Splits a heredoc String into segments if it is to be
+    # interpolated.
+    #
+    # string      - The String heredoc.
+    # eos_text    - The String endtext for the heredoc.
+    # interpolate - A Boolean that specifies whether this heredoc can contain
+    #               interpolated values (defaults to True).
+    #
+    # Returns an Array consisting of two Strings, the String up to the first
+    # terminator and the terminator that was found.
+    def get_heredoc_segment(string, eos_text, interpolate=true)
+      if interpolate
+        regexp = /(([^\\]|^|[^\\])([\\]{2})*[$]+|\|?\s*-?#{Regexp.escape(eos_text)})/
+      else
+        regexp = /\|?\s*-?#{Regexp.escape(eos_text)}/
+      end
+
+      str = string.scan_until(regexp)
+      begin
+        str =~ /\A(.*?)([$]+|\|?\s*-?#{Regexp.escape(eos_text)})\Z/m
+        value = $1
+        terminator = $2
+        [value, terminator]
+      rescue
+        [nil, nil]
+      end
+    end
   end
 end