coderay 0.7.4.215 → 0.8.260

data/README

@@ -18,12 +18,12 @@ And with line numbers.
 * is what everybody should have on their website
 * solves all your problems and makes the girls run after you
 
-Version: 0.7.4 (2006.october.20)
+Version: 0.8.0
 Author:: murphy (Kornelius Kalnbach)
 Contact:: murphy rubychan de
 Website:: coderay.rubychan.de[http://coderay.rubychan.de]
 License:: GNU LGPL; see LICENSE file in the main directory.
-Subversion:: $Id: README 206 2006-10-19 19:46:56Z murphy $
+Subversion:: $Id: README 247 2008-08-28 05:06:01Z murphy $
 
 -----
 
@@ -40,8 +40,10 @@ Since CodeRay is still in beta stage, nightly buildy may be useful:
 
 === Dependencies
 
-CodeRay needs Ruby 1.8 and the strscan[http://www.ruby-doc.org/stdlib/libdoc/strscan/rdoc/index.htm] library
-(part of the standard library.)  It should also run with Ruby 1.9 and yarv.
+CodeRay needs Ruby 1.8 and the
+strscan[http://www.ruby-doc.org/stdlib/libdoc/strscan/rdoc/index.htm]
+library (part of the standard library.)  It should also run with Ruby 1.9 and
+yarv.
 
 
 == Example Usage
@@ -67,7 +69,8 @@ Please report errors in this documentation to <coderay cycnus de>.
 
 === Special Thanks to
 
-* licenser (Heinz N. Gies) for ending my QBasic career, inventing the Coder project and the input/output plugin system.
+* licenser (Heinz N. Gies) for ending my QBasic career, inventing the Coder
+  project and the input/output plugin system.
   CodeRay would not exist without him.
 
 === Thanks to
@@ -88,6 +91,8 @@ Please report errors in this documentation to <coderay cycnus de>.
 * Daniel and Dethix from ruby-mine.de
 * Dookie (who is no longer with us...) and Leonidas from
   http://www.python-forum.de
+* Andreas Schwarz for finding out that CaseIgnoringWordList was not case
+  ignoring! Such things really make you write tests.
 * matz and all Ruby gods and gurus
 * The inventors of: the computer, the internet, the true color display, HTML &
   CSS, VIM, RUBY, pizza, microwaves, guitars, scouting, programming, anime, 
@@ -97,21 +102,27 @@ Where would we be without all those people?
 
 === Created using
 
-* Ruby[http://ruby-lang.org/en/]
-* Chihiro (my Sony VAIO laptop) and Seras (my Athlon 2200+ tower)
-* VIM[http://vim.org]
+* Ruby[http://ruby-lang.org/]
+* Chihiro (my Sony VAIO laptop), Henrietta (my new MacBook) and
+  Seras (my Athlon 2200+ tower)
+* VIM[http://vim.org] and TextMate[http://macromates.com]
 * RDE[http://homepage2.nifty.com/sakazuki/rde_e.html]
-* Microsoft Windows (yes, I confess!)
-* Firefox[http://www.mozilla.org/products/firefox/] and Thunderbird[http://www.mozilla.org/products/thunderbird/]
+* Microsoft Windows (yes, I confess!) and MacOS X
+* Firefox[http://www.mozilla.org/products/firefox/] and
+  Thunderbird[http://www.mozilla.org/products/thunderbird/]
 * Rake[http://rake.rubyforge.org/]
 * RubyGems[http://docs.rubygems.org/]
-* {Subversion/TortoiseSVN}[http://tortoisesvn.tigris.org/] using Apache via XAMPP[http://www.apachefriends.org/en/xampp.html]
+* {Subversion/TortoiseSVN}[http://tortoisesvn.tigris.org/] using Apache via
+  XAMPP[http://www.apachefriends.org/en/xampp.html]
 * RDoc (though I'm quite unsatisfied with it)
-* GNUWin32, MinGW and some other tools to make the shell under windows a bit more useful
+* GNUWin32, MinGW and some other tools to make the shell under windows a bit
+  more useful
+* Term::ANSIColor[http://term-ansicolor.rubyforge.org/]
 
 ---
 
 * As you can see, CodeRay was created under heavy use of *free* software.
 * So CodeRay is also *free*.
-* If you use CodeRay to create software, think about making this software *free*, too.
+* If you use CodeRay to create software, think about making this software
+  *free*, too.
 * Thanks :)

data/bin/coderay

@@ -29,11 +29,11 @@ Example:
     if first[/-(\w+)/] == first
       lang = $1.to_sym
       input = $stdin.read
-      tokens = CodeRay.scan input, lang
+      tokens = :scan
     elsif first == '-'
       lang = $1.to_sym
       input = $stdin.read
-      tokens = CodeRay.scan input, lang
+      tokens = :scan
     else
       file = first
       tokens = CodeRay.scan_file file
@@ -54,8 +54,13 @@ Example:
     $stderr.puts 'No format given; setting to default (HTML Page)'
     format = :page
   end
-
-  output = tokens.encode format
+  
+  # TODO: allow streaming
+  if tokens == :scan
+    output = CodeRay::Duo[lang => format].highlight input  #, :stream => true
+  else
+    output = tokens.encode format
+  end
   out = $stdout
   if output_filename
     output_filename += '.' + CodeRay::Encoders[format]::FILE_EXTENSION

data/lib/coderay.rb

@@ -1,6 +1,6 @@
 # = CodeRay Library
 #
-# $Id: coderay.rb 213 2006-10-20 07:12:31Z murphy $
+# $Id: coderay.rb 260 2008-10-08 20:18:31Z murphy $
 #
 # CodeRay is a Ruby library for syntax highlighting.
 #
@@ -24,8 +24,8 @@
 #
 # == Usage
 #
-# Remember you need RubyGems to use CodeRay. Run Ruby with -rubygems option
-# if required.
+# Remember you need RubyGems to use CodeRay, unless you have it in your load path. Run Ruby with
+# -rubygems option if required.
 #
 # === Highlight Ruby code in a string as html
 # 
@@ -44,19 +44,15 @@
 # 
 # You can include this div in your page. The used CSS styles can be printed with
 # 
-#   % ruby -rcoderay -e "print CodeRay::Encoders[:html]::CSS"
+#   % coderay_stylesheet
 # 
 # === Highlight without typing too much
-#
+# 
 # If you are one of the hasty (or lazy, or extremely curious) people, just run this file:
-#
-#   % ruby -rubygems coderay.rb
 # 
-# If the output was to fast for you, try
+#   % ruby -rubygems /path/to/coderay/coderay.rb > example.html
 # 
-#   % ruby -rubygems coderay.rb > example.html
-#
-# and look at the file it created.
+# and look at the file it created in your browser.
 # 
 # = CodeRay Module
 #
@@ -111,7 +107,7 @@
 #
 # CodeRay.scan_stream:: Scan in stream mode.
 #
-#  == All-in-One Encoding
+# == All-in-One Encoding
 #
 # CodeRay.encode:: Highlight a string with a given input and output format.
 #
@@ -121,11 +117,16 @@
 # for this Encoder must only be done once.
 #
 # CodeRay.encoder:: Create an Encoder instance with format and options.
+# CodeRay.scanner:: Create an Scanner instance for lang, with '' as default code.
 #
-# There is no CodeRay.scanner method because Scanners are bound to an input string
-# on creation; you can't re-use them with another string.
+# To make use of CodeRay.scanner, use CodeRay::Scanner::code=.
 #
-#  The scanning methods provide more flexibility; we recommend to use these.
+# The scanning methods provide more flexibility; we recommend to use these.
+# 
+# == Reusing Scanners and Encoders
+# 
+# If you want to re-use scanners and encoders (because that is faster), see
+# CodeRay::Duo for the most convenient (and recommended) interface.
 module CodeRay
 
   # Version: Major.Minor.Teeny[.Revision]
@@ -133,7 +134,7 @@ module CodeRay
   # Minor: odd for beta, even for stable
   # Teeny: development state
   # Revision: Subversion Revision number (generated on rake)
-  VERSION = '0.7.4'
+  VERSION = '0.8'
 
   require 'coderay/tokens'
   require 'coderay/scanner'
@@ -314,6 +315,7 @@ end
 # Run a test script.
 if $0 == __FILE__
   $stderr.print 'Press key to print demo.'; gets
-  code = File.read($0)[/module CodeRay.*/m]
+  # Just use this file as an example of Ruby code.
+  code = File.read(__FILE__)[/module CodeRay.*/m]
   print CodeRay.scan(code, :ruby).html
 end

data/lib/coderay/duo.rb

@@ -4,26 +4,84 @@ module CodeRay
   #
   # $Id: scanner.rb 123 2006-03-21 14:46:34Z murphy $
   #
-  # TODO: Doc.
+  # A Duo is a convenient way to use CodeRay. You just create a Duo,
+  # giving it a lang (language of the input code) and a format (desired
+  # output format), and call Duo#highlight with the code.
+  # 
+  # Duo makes it easy to re-use both scanner and encoder for a repetitive
+  # task. It also provides a very easy interface syntax:
+  # 
+  #   require 'coderay'
+  #   CodeRay::Duo[:python, :div].highlight 'import this'
+  # 
+  # Until you want to do uncommon things with CodeRay, I recommend to use
+  # this method, since it takes care of everything.
   class Duo
 
-    attr_accessor :scanner, :encoder
-
-    def initialize lang, format, options = {}
-      @scanner = CodeRay.scanner lang, CodeRay.get_scanner_options(options)
-      @encoder = CodeRay.encoder format, options
+    attr_accessor :lang, :format, :options
+    
+    # Create a new Duo, holding a lang and a format to highlight code.
+    # 
+    # simple:
+    #   CodeRay::Duo[:ruby, :page].highlight 'bla 42'
+    # 
+    # streaming:
+    #   CodeRay::Duo[:ruby, :page].highlight 'bar 23', :stream => true
+    # 
+    # with options:
+    #   CodeRay::Duo[:ruby, :html, :hint => :debug].highlight '????::??'
+    # 
+    # alternative syntax without options:
+    #   CodeRay::Duo[:ruby => :statistic].encode 'class << self; end'
+    # 
+    # alternative syntax with options:
+    #   CodeRay::Duo[{ :ruby => :statistic }, :do => :something].encode 'abc'
+    # 
+    # The options are forwarded to scanner and encoder
+    # (see CodeRay.get_scanner_options).
+    def initialize lang = nil, format = nil, options = {}
+      if format == nil and lang.is_a? Hash and lang.size == 1
+        @lang = lang.keys.first
+        @format = lang[@lang]
+      else
+        @lang = lang
+        @format = format
+      end
+      @options = options
     end
 
     class << self
+      # To allow calls like Duo[:ruby, :html].highlight.
       alias [] new
     end
 
-    def encode code
-      @scanner.string = code
-      @encoder.encode_tokens(scanner.tokenize)
+    # The scanner of the duo. Only created once.
+    def scanner
+      @scanner ||= CodeRay.scanner @lang, CodeRay.get_scanner_options(@options)
+    end
+    
+    # The encoder of the duo. Only created once.
+    def encoder
+      @encoder ||= CodeRay.encoder @format, @options
+    end
+    
+    # Tokenize and highlight the code using +scanner+ and +encoder+.
+    #
+    # If the :stream option is set, the Duo will go into streaming mode,
+    # saving memory for the cost of time.
+    def encode code, options = { :stream => false }
+      stream = options.delete :stream
+      options = @options.merge options
+      if stream
+        encoder.encode_stream(code, @lang, options)
+      else
+        scanner.code = code
+        encoder.encode_tokens(scanner.tokenize, options)
+      end
     end
     alias highlight encode
 
   end
 
 end
+

data/lib/coderay/encoder.rb

@@ -1,3 +1,5 @@
+require "stringio"
+
 module CodeRay
 
   # This module holds the Encoder class and its subclasses.
@@ -40,7 +42,7 @@ module CodeRay
         # downcase class name instead.
         def const_missing sym
           if sym == :FILE_EXTENSION
-            sym.to_s.downcase
+            plugin_id
           else
             super
           end
@@ -130,13 +132,15 @@ module CodeRay
       # By default, it calls text_token or block_token, depending on
       # whether +text+ is a String.
       def token text, kind
-        if text.instance_of? ::String  # Ruby 1.9: :open.is_a? String
-          text_token text, kind
-        elsif text.is_a? ::Symbol
-          block_token text, kind
-        else
-          raise 'Unknown token text type: %p' % text
-        end
+        out =
+          if text.is_a? ::String  # Ruby 1.9: watch out, :open.is_a? String is true
+            text_token text, kind
+          elsif text.is_a? ::Symbol
+            block_token text, kind
+          else
+            raise 'Unknown token text type: %p' % text
+          end
+        @out << out if defined?(@out) && @out
       end
 
       def text_token text, kind
@@ -148,6 +152,10 @@ module CodeRay
           open_token kind
         when :close
           close_token kind
+        when :begin_line
+          begin_line kind
+        when :end_line
+          end_line kind
         else
           raise 'unknown block action: %p' % action
         end
@@ -164,7 +172,8 @@ module CodeRay
       # The already created +tokens+ object must be used; it can be a
       # TokenStream or a Tokens object.
       def compile tokens, options
-        tokens.each(&self)
+        tokens.each { |text, kind| token text, kind }  # FIXME for Ruby 1.9?
+        #tokens.each(&self)
       end
 
     end

data/lib/coderay/encoders/_map.rb

@@ -2,7 +2,8 @@ module CodeRay
 module Encoders
 
   map :stats => :statistic,
-    :plain => :text
+    :plain => :text,
+    :tex => :latex
 
 end
 end

data/lib/coderay/encoders/debug.rb

@@ -19,17 +19,12 @@ module Encoders
 
   protected
     def text_token text, kind
-      @out <<
-        if kind == :space
-          text
-        else
-          text = text.gsub(/[)\\]/, '\\\\\0')
-          "#{kind}(#{text})"
-        end
-    end
-
-    def block_token action, kind
-      @out << super
+      if kind == :space
+        text
+      else
+        text = text.gsub(/[)\\]/, '\\\\\0')  # escape ) and \
+        "#{kind}(#{text})"
+      end
     end
 
     def open_token kind
@@ -40,6 +35,14 @@ module Encoders
       ">"
     end
 
+    def begin_line kind
+      "#{kind}["
+    end
+
+    def end_line kind
+      "]"
+    end
+
   end
 
 end

data/lib/coderay/encoders/html.rb

@@ -86,7 +86,7 @@ module Encoders
       :hint => false,
     }
 
-    helper :classes, :output, :css
+    helper :output, :css
 
     attr_reader :css
 
@@ -119,15 +119,14 @@ module Encoders
         end
     }
 
-    TRANSPARENT_TOKEN_KINDS = Set[
+    TRANSPARENT_TOKEN_KINDS = [
       :delimiter, :modifier, :content, :escape, :inline_delimiter,
-    ]
+    ].to_set
 
     # Generate a hint about the given +classes+ in a +hint+ style.
     #
     # +hint+ may be :info, :info_long or :debug.
     def self.token_path_to_hint hint, classes
-      return '' unless hint
       title =
         case hint
         when :info
@@ -137,7 +136,7 @@ module Encoders
         when :debug
           classes.inspect
         end
-      " title=\"#{title}\""
+      title ? " title=\"#{title}\"" : ''
     end
 
     def setup options
@@ -159,29 +158,32 @@ module Encoders
 
       when :class
         @css_style = Hash.new do |h, k|
-          if k.is_a? Array
-            type = k.first
-          else
-            type = k
-          end
-          c = ClassOfKind[type]
+          c = CodeRay::Tokens::ClassOfKind[k.first]
           if c == :NO_HIGHLIGHT and not hint
-            h[k] = false
+            h[k.dup] = false
           else
-            title = HTML.token_path_to_hint hint, (k[1..-1] << k.first)
-            h[k] = '<span%s class="%s">' % [title, c]
+            title = if hint
+              HTML.token_path_to_hint(hint, k[1..-1] << k.first)
+            else
+              ''
+            end
+            if c == :NO_HIGHLIGHT
+              h[k.dup] = '<span%s>' % [title]
+            else
+              h[k.dup] = '<span%s class="%s">' % [title, c]
+            end
           end
         end
 
       when :style
         @css_style = Hash.new do |h, k|
-          if k.is_a? Array
+          if k.is_a? ::Array
             styles = k.dup
           else
             styles = [k]
           end
           type = styles.first
-          classes = styles.map { |c| ClassOfKind[c] }
+          classes = styles.map { |c| Tokens::ClassOfKind[c] }
           if classes.first == :NO_HIGHLIGHT and not hint
             h[k] = false
           else
@@ -224,13 +226,16 @@ module Encoders
           text = text.gsub(/#{HTML_ESCAPE_PATTERN}/o) { |m| @HTML_ESCAPE[m] }
         end
         @opened[0] = type
-        if style = @css_style[@opened]
+        if text != "\n" && style = @css_style[@opened]
           @out << style << text << '</span>'
         else
           @out << text
         end
       else
+        
         case text
+        
+        # token groups, eg. strings
         when :open
           @opened[0] = type
           @out << (@css_style[@opened] || '<span>')
@@ -246,6 +251,28 @@ module Encoders
             @out << '</span>'
             @opened.pop
           end
+        
+        # whole lines to be highlighted, eg. a deleted line in a diff
+        when :begin_line
+          @opened[0] = type
+          if style = @css_style[@opened]
+            @out << style.sub('<span', '<div')
+          else
+            @out << '<div>'
+          end
+          @opened << type
+        when :end_line
+          if @opened.empty?
+            # nothing to close
+          else
+            if $DEBUG and (@opened.size == 1 or @opened.last != type)
+              raise 'Malformed token stream: Trying to close a line (%p) \
+                that is not open. Open are: %p.' % [type, @opened[1..-1]]
+            end
+            @out << '</div>'
+            @opened.pop
+          end
+        
         when nil
           raise 'Token with nil as text was given: %p' % [[text, type]]
         else