syntax 1.0.0 → 1.1.0

data/CHANGELOG

@@ -0,0 +1,21 @@
+Syntax: a syntax highlighting library for Ruby.
+
+1.1.0 In Progress
+  Published from https://github.com/dblock/syntax, a now maintained fork.
+  The project builds again and runs, fixes by @dblock, @distler.
+
+1.0.0: 18 Jun 2005
+  Added user manual
+  Fixed various bugs
+
+0.7.0: 23 Mar 2005
+  Added Syntax.all to enumerate all supported syntaxes.
+  Much improved heredoc support.
+  Support for syntax "regions"
+  Added sample CSS files for the supported syntaxes
+  Added the example "to_html.rb" script
+  Bug fixes.
+
+0.5.0: 11 Jan 2005
+  First release.
+

data/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2004, Jamis Buck (jgb3@email.byu.edu)
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright notice,
+      this list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+
+    * The names of its contributors may not be used to endorse or promote
+      products derived from this software without specific prior written
+      permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.rdoc

@@ -0,0 +1,49 @@
+= Syntax
+
+A syntax highlighting a library for Ruby.
+
+This fork is maintained and version 1.1.0 has been published from it. However, there's currently none or not much new development going on here and the original author, @jamis[https://github.com/jamis], recommends using CodeRay[http://coderay.rubychan.de], over this library.
+
+== About
+
+This is a simple syntax highlighting library for Ruby. It is a naive syntax analysis tool, meaning that it does not "understand" the syntaxes of the languages it processes, but merely does some semi-intelligent pattern matching.
+
+== Usage
+
+There are primarily two uses for the Syntax library:
+
+# Convert text from a supported syntax to a supported highlight format (like HTML).
+# Tokenize text in a supported syntax and process the tokens directly.
+
+=== Highlighting a supported syntax
+
+  require 'syntax/convertors/html'
+
+  convertor = Syntax::Convertors::HTML.for_syntax "ruby"
+  puts convertor.convert( File.read( "file.rb" ) )
+
+The above snippet will emit HTML, using spans and CSS to indicate the different highlight "groups". (Sample CSS files are included in the "data" directory.)
+
+=== Tokenize text
+
+  require 'syntax'
+
+  tokenizer = Syntax.load "ruby"
+  tokenizer.tokenize( File.read( "file.rb" ) ) do |token|
+    puts "group(#{token.group}, #{token.instruction}) lexeme(#{token})"
+  end
+
+Tokenizing is straightforward process. Each time a new token is discovered by the tokenizer, it is yielded to the given block.
+
+* <tt>token.group</tt> is the lexical group to which the token belongs. Each supported syntax may have it's own set of lexical groups.
+* <tt>token.instruction</tt> is an instruction used to determine how this token should be treated. It will be <tt>:none</tt> for normal tokens, <tt>:region_open</tt> if the token starts a nested region, and <tt>:region_close</tt> if it closes the last opened region.
+* <tt>token</tt> is itself a subclass of String, so you can use it just as you would a string. It represents the lexeme that was actually parsed.
+
+== Releasing Syntax
+
+* Update version in <tt>lib/syntax/version.rb</tt>.
+
+* Build the gem, use Ruby 1.9.3.
+
+    rake clean
+    rake package

data/lib/syntax/lang/ansic.rb

@@ -0,0 +1,69 @@
+require 'syntax'
+
+module Syntax
+
+  class AnsiC < Tokenizer
+    
+    def self.add_type(name)
+      ANSIC_PREDEFINED_TYPES << name
+    end
+    
+    ANSIC_KEYWORDS =
+      Set.new %w{asm break case continue default do else for goto if return
+         switch while struct union enum typedef static register
+         auto extern sizeof volatile const inline restrict} unless const_defined?(:ANSIC_KEYWORDS)
+
+    ANSIC_PREDEFINED_TYPES =
+      Set.new %w{int long short char void signed unsigned 
+         float double bool complex} unless const_defined?(:ANSIC_PREDEFINED_TYPES)
+
+    ANSIC_PREDEFINED_CONSTANTS =
+      %w{EOF NULL true false} unless const_defined?(:ANSIC_PREDEFINED_CONSTANTS)
+
+    ANSIC_ESCAPE = / [rbfnrtv\n\\'"] | x[a-fA-F0-9]{1,2} | [0-7]{1,3} /x unless const_defined?(:ANSIC_ESCAPE)
+
+    def step
+      case
+      when scan(/\s+/)
+        start_group :normal, matched
+      when match = scan(/#\s*(\w*)/)
+        match << scan_until(/\n/)
+        start_group :preprocessor, match
+      when scan(/ L?' (?: [^\'\n\\] | \\ #{ANSIC_ESCAPE} )? '? /ox)
+        start_group :char, matched
+      when scan(/0[xX][0-9A-Fa-f]+/)
+        start_group :hex, matched
+      when scan(/(?:0[0-7]+)(?![89.eEfF])/)
+        start_group :oct, matched
+      when scan(/(?:\d+)(?![.eEfF])/)
+        start_group :integer, matched
+      when scan(/\d[fF]?|\d*\.\d+(?:[eE][+-]?\d+)?[fF]?|\d+[eE][+-]?\d+[fF]?/)
+        start_group :float, matched
+      when scan(/"(?:[^"\\]|\\.)*"/)
+        start_group :string, matched
+      when scan( %r{ ('(?: . | [\t\b\n] )') }x )
+        start_group :char, matched
+      when scan(/[a-z_][a-z_\d]+/)
+        if ANSIC_KEYWORDS.include?( matched )
+          start_group :keyword, matched
+        elsif ANSIC_PREDEFINED_TYPES.include?( matched )
+          start_group :predefined_types, matched
+        else
+          start_group :ident, matched
+        end
+      when scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
+        start_group :comment, matched
+      else
+        start_group :other, scan(/./x)
+      end
+    end
+
+  end
+
+  SYNTAX["ansic"] = AnsiC
+
+end
+
+
+
+

data/lib/syntax/lang/css21.rb

@@ -0,0 +1,324 @@
+require 'syntax'
+
+module Syntax
+
+  class CSS21 < Tokenizer
+
+    CSS21_PROPERTIES = Set.new %w{font-family font-style font-variant font-weight
+      font-size font background-color background-image
+      background-repeat background-attachment background-position
+      color background word-spacing letter-spacing
+      border-top-width border-right-width border-left-width
+      border-bottom-width border-width list-style-type
+      list-style-image list-style-position text-decoration
+      vertical-align text-transform text-align text-indent
+      line-height margin-top margin-right margin-bottom
+      margin-left margin padding-top padding-right padding-bottom
+      padding-left padding border-top border-right border-bottom
+      border-left border width height float clear display
+      list-style white-space border-style border-color
+      azimuth border-bottom-color border-bottom-style
+      border-collapse border-left-color border-left-style
+      border-right-color border-right-style border-top-color
+      border-top-style caption-side cell-spacing clip column-span
+      content cue cue-after cue-before cursor direction
+      elevation font-size-adjust marks max-height max-width
+      min-height min-width orphans overflow page-break-after
+      page-break-before pause pause-after pause-before pitch
+      pitch-range play-during position richness right row-span
+      size speak speak-date speak-header speak-punctuation
+      speak-time speech-rate stress table-layout text-shadow top
+      visibility voice-family volume
+      widows z-index quotes
+      marker-offset outline outline-color outline-style outline-width
+      border-spacing border-collapse
+      page-break-before page-break-after page-break-inside
+      orphans widows} unless const_defined?(:CSS21_PROPERTIES)
+
+    CSS21_KEYWORDS = Set.new %w{maroon red orange yellow olive purple
+      fuchsia white lime green navy blue aqua teal black silver gray
+      scroll fixed transparent none top center bottom
+      left right repeat repeat-x repeat-y no-repeat
+      thin medium thick dotted dashed solid double groove ridge
+      inset outset both block inline list-item
+      xx-small x-small small medium large x-large xx-large
+      smaller italic oblique small-caps bold bolder lighter auto
+      disc circle square decimal lower-roman upper-roman lower-alpha
+      upper-alpha inside outside justify underline overline line-through
+      blink capitalize uppercase lowercase baseline sub super
+      top text-top middle bottom text-bottom pre nowrap
+      compact run-in inherit caption icon menu message-box small-caption
+      status-bar marker
+      table inline-table table-column-group table-column
+      table-row-group table-row table-cell table-caption
+      table-header-group table-footer-group
+      screen print projection braille embosed aural tv
+      tty handheld cross hidden open-quote close-quote
+      absolute relative normal collapse
+      serif sans-serif monospace cursive
+      fantasy, always} unless const_defined?(:CSS21_KEYWORDS)
+
+    HTML_TAGS = Set.new %w{a abbr address area article aside
+      audio b base bdo blockquote body br button canvas caption
+      cite code col colgroup command datalist dd del details dfn
+      div dl dt em embed fieldset figure footer form h1 h2 h3
+      h4 h5 h6 head header hgroup hr html i iframe img input ins
+      kbd keygen label legend li link map mark menu meta meter
+      nav noscript object ol optgroup option output p param pre
+      progress q rp rt ruby samp script section select small
+      source span strong style sub sup table tbody td textarea
+      tfoot th thead time title tr ul var video
+      acronym applet big center frame frameset isindex marquee
+      noframes s tt u} unless const_defined?(:HTML_TAGS)
+
+    MATHML_TAGS = Set.new %w{annotation annotation-xml maction
+      malign maligngroup malignmark malignscope math menclose
+      merror mfenced mfrac mglyph mi mlabeledtr mlongdiv mmultiscripts
+      mn mo mover mpadded mphantom mprescripts mroot mrow ms mscarries
+      mscarry msgroup msline mspace msqrt msrow mstack mstyle msub
+      msubsup msup mtable mtd mtext mtr munder munderover none
+      semantics} unless const_defined?(:MATHML_TAGS)
+
+    SVG_TAGS = Set.new %w{a altGlyph altGlyphDef altGlyphItem animate
+      animateColor animateMotion animateTransform circle clipPath
+      color-profile cursor definition-src defs desc ellipse feBlend
+      feColorMatrix feComponentTransfer feComposite feConvolveMatrix
+      feDiffuseLighting feDisplacementMap feDistantLight feFlood feFuncA
+      feFuncB feFuncG feFuncR feGaussianBlur feImage feMerge feMergeNode
+      feMorphology feOffset fePointLight feSpecularLighting feSpotLight
+      feTile feTurbulence filter font font-face font-face-format
+      font-face-name font-face-src font-face-uri foreignObject g glyph
+      glyphRef hkern image line linearGradient marker mask metadata
+      missing-glyph mpath path pattern polygon polyline radialGradient
+      rect script set stop style svg switch symbol text textPath title
+      tref tspan use view vkern} unless const_defined?(:SVG_TAGS)
+      
+    MY_TAGS = HTML_TAGS + MATHML_TAGS + SVG_TAGS unless const_defined?(:MY_TAGS)
+
+    def setup
+      @selector = true
+      @macros = {}
+      @tokens = {}
+
+      # http://www.w3.org/TR/CSS21/syndata.html
+      macro(:h, /([0-9a-fA-F])/ ) # uppercase A-Z added?
+      macro(:nonascii, /([^\000-\177])/ )
+      macro(:nl, /(\n|\r\n|\r|\f)/ )
+      macro(:unicode, /(\\#{m(:h)}{1,6}(\r\n|[ \t\r\n\f])?)/ )
+      macro(:escape, /(#{m(:unicode)}|\\[^\r\n\f0-9a-f])/ )
+      macro(:nmstart, /([_a-z]|#{m(:nonascii)}|#{m(:escape)})/ )
+      macro(:nmchar, /([_a-z0-9-]|#{m(:nonascii)}|#{m(:escape)})/ )
+      macro(:string1, /(\"([^\n\r\f\\\"]|\\#{m(:nl)}|#{m(:escape)})*\")/ )
+      macro(:string2, /(\'([^\n\r\f\\']|\\#{m(:nl)}|#{m(:escape)})*\')/ )
+      macro(:invalid1, /(\"([^\n\r\f\\\"]|\\#{m(:nl)}|#{m(:escape)})*)/ )
+      macro(:invalid2, /(\'([^\n\r\f\\']|\\#{m(:nl)}|#{m(:escape)})*)/ )
+      macro(:comment, /(\/\*[^*]*\*+([^\/*][^*]*\*+)*\/)/ )
+      macro(:ident, /(-?#{m(:nmstart)}#{m(:nmchar)}*)/ )
+      macro(:name, /(#{m(:nmchar)}+)/ )
+      macro(:num, /([0-9]+|[0-9]*\.[0-9]+)/ )
+      macro(:string, /(#{m(:string1)}|#{m(:string2)})/ )
+      macro(:invalid, /(#{m(:invalid1)}|#{m(:invalid2)})/ )
+      macro(:s, /([ \t\r\n\f]+)/ )
+      macro(:w, /(#{m(:s)}?)/ )
+      macro(:A, /(a|\\0{0,4}(41|61)(\r\n|[ \t\r\n\f])?)/ )
+      macro(:C, /(c|\\0{0,4}(43|63)(\r\n|[ \t\r\n\f])?)/ )
+      macro(:D, /(d|\\0{0,4}(44|64)(\r\n|[ \t\r\n\f])?)/ )
+      macro(:E, /(e|\\0{0,4}(45|65)(\r\n|[ \t\r\n\f])?)/ )
+      macro(:G, /(g|\\0{0,4}(47|67)(\r\n|[ \t\r\n\f])?|\\g)/ )
+      macro(:H, /(h|\\0{0,4}(48|68)(\r\n|[ \t\r\n\f])?|\\h)/ )
+      macro(:I, /(i|\\0{0,4}(49|69)(\r\n|[ \t\r\n\f])?|\\i)/ )
+      macro(:K, /(k|\\0{0,4}(4b|6b)(\r\n|[ \t\r\n\f])?|\\k)/ )
+      macro(:M, /(m|\\0{0,4}(4d|6d)(\r\n|[ \t\r\n\f])?|\\m)/ )
+      macro(:N, /(n|\\0{0,4}(4e|6e)(\r\n|[ \t\r\n\f])?|\\n)/ )
+      macro(:O, /(o|\\0{0,4}(51|71)(\r\n|[ \t\r\n\f])?|\\o)/ )
+      macro(:P, /(p|\\0{0,4}(50|70)(\r\n|[ \t\r\n\f])?|\\p)/ )
+      macro(:R, /(r|\\0{0,4}(52|72)(\r\n|[ \t\r\n\f])?|\\r)/ )
+      macro(:S, /(s|\\0{0,4}(53|73)(\r\n|[ \t\r\n\f])?|\\s)/ )
+      macro(:T, /(t|\\0{0,4}(54|74)(\r\n|[ \t\r\n\f])?|\\t)/ )
+      macro(:X, /(x|\\0{0,4}(58|78)(\r\n|[ \t\r\n\f])?|\\x)/ )
+      macro(:Z, /(z|\\0{0,4}(5a|7a)(\r\n|[ \t\r\n\f])?|\\z)/ )
+
+      token(:COMMENT, /#{m(:comment)}/)
+
+      token(:HASH, /\#/)
+      token(:IDENT, /#{m(:ident)}/)
+      token(:LBRACE, /#{m(:w)}\{/)
+      token(:RBRACE, /#{m(:w)}\}/)
+
+      token(:S, /#{m(:s)}/)
+
+      token(:FUNCTION, /#{m(:ident)}(?=\()/)
+
+      token(:PLUS, /#{m(:w)}\+/)
+      token(:GREATER, /#{m(:w)}>/)
+      token(:COMMA, /#{m(:w)},/)
+
+      token(:CDO, /<!--/)
+      token(:CDC, /-->/)
+      token(:INCLUDES, /~=/)
+      token(:DASHMATCH, /\|=/)
+      token(:STRING, /#{m(:string)}/)
+      token(:INVALID, /#{m(:invalid)}/)
+      token(:IMPORT_SYM, /@#{m(:I)}#{m(:M)}#{m(:P)}#{m(:O)}#{m(:R)}#{m(:T)}/)
+      token(:PAGE_SYM, /@#{m(:P)}#{m(:A)}#{m(:G)}#{m(:E)}/)
+      token(:MEDIA_SYM, /@#{m(:M)}#{m(:E)}#{m(:D)}#{m(:I)}#{m(:A)}/)
+      token(:CHARSET_SYM, /@#{m(:C)}#{m(:H)}#{m(:A)}#{m(:R)}#{m(:S)}#{m(:E)}#{m(:T)}/)
+      token(:IMPORTANT_SYM, /!(#{m(:w)}|#{m(:comment)})*#{m(:I)}#{m(:M)}#{m(:P)}#{m(:O)}#{m(:R)}#{m(:T)}#{m(:A)}#{m(:N)}#{m(:T)}/)
+      token(:EMS, /#{m(:num)}#{m(:E)}#{m(:M)}/)
+      token(:EXS, /#{m(:num)}#{m(:E)}#{m(:X)}/)
+
+      token :LENGTH do |patterns|
+        patterns << /#{m(:num)}#{m(:P)}#{m(:X)}/
+        patterns << /#{m(:num)}#{m(:C)}#{m(:M)}/
+        patterns << /#{m(:num)}#{m(:M)}#{m(:M)}/
+        patterns << /#{m(:num)}#{m(:I)}#{m(:N)}/
+        patterns << /#{m(:num)}#{m(:P)}#{m(:T)}/
+        patterns << /#{m(:num)}#{m(:P)}#{m(:C)}/
+      end
+
+      token :ANGLE do |patterns|
+        patterns << /#{m(:num)}#{m(:D)}#{m(:E)}#{m(:G)}/
+        patterns << /#{m(:num)}#{m(:R)}#{m(:A)}#{m(:D)}/
+        patterns << /#{m(:num)}#{m(:G)}#{m(:R)}#{m(:A)}#{m(:D)}/
+      end
+
+      token :TIME do |patterns|
+        patterns << /#{m(:num)}#{m(:M)}#{m(:S)}/
+        patterns << /#{m(:num)}#{m(:S)}/
+      end
+
+      token :FREQ do |patterns|
+        patterns << /#{m(:num)}#{m(:H)}#{m(:Z)}/
+        patterns << /#{m(:num)}#{m(:K)}#{m(:H)}#{m(:Z)}/
+      end
+
+      token :URI do |patterns|
+        patterns << /url\(#{m(:w)}#{m(:string)}#{m(:w)}\)/
+        patterns << /url\(#{m(:w)}([!$%&*-~]|#{m(:nonascii)}|#{m(:escape)})*#{m(:w)}\)/
+      end
+
+      token(:DIMENSION, /#{m(:num)}#{m(:ident)}/)
+      token(:PERCENTAGE, /#{m(:num)}%/)
+      token(:HEXNUM, /##{m(:h)}{2,6}/)
+      token(:NUMBER, /#{m(:num)}/)
+
+    end
+
+    def step
+
+      case
+
+      # scanning selectors only
+      when @selector && scan(@tokens[:LBRACE])
+        @selector = false
+        start_group :normal, matched
+      when @selector && scan(@tokens[:IMPORT_SYM])
+        start_group :import, matched
+      when @selector && scan(@tokens[:PAGE_SYM])
+        start_group :page, matched
+      when @selector && scan(@tokens[:MEDIA_SYM])
+        start_group :media, matched
+      when @selector && scan(@tokens[:CHARSET_SYM])
+        start_group :charset, matched
+      when @selector && scan(@tokens[:HASH])
+        start_group :normal, matched
+      when @selector && scan(@tokens[:URI])
+        start_group :uri, matched
+
+      when @selector && scan(@tokens[:IDENT])
+        if MY_TAGS.include?( matched )
+          start_group :tag, matched
+        else
+          start_group :ident, matched
+        end
+
+      # scanning declarations only
+      when !@selector && scan(@tokens[:RBRACE])
+        @selector = true
+        start_group :normal, matched
+      when !@selector && scan(@tokens[:FUNCTION])
+        start_group :function, matched
+      when !@selector && scan(@tokens[:EMS])
+        start_group :ems, matched
+      when !@selector && scan(@tokens[:EXS])
+        start_group :exs, matched
+      when !@selector && scan(@tokens[:LENGTH])
+        start_group :length, matched
+      when !@selector && scan(@tokens[:ANGLE])
+        start_group :angle, matched
+      when !@selector && scan(@tokens[:TIME])
+        start_group :time, matched
+      when !@selector && scan(@tokens[:FREQ])
+        start_group :freq, matched
+      when !@selector && scan(@tokens[:PERCENTAGE])
+        start_group :percentage, matched
+      when !@selector && scan(@tokens[:DIMENSION])
+        start_group :dimension, matched
+      when !@selector && scan(@tokens[:NUMBER])
+        start_group :number, matched
+      when !@selector && scan(@tokens[:HEXNUM])
+        start_group :number, matched
+      when !@selector && scan(@tokens[:IMPORTANT_SYM])
+        start_group :important, matched
+
+      when !@selector && scan(@tokens[:IDENT])
+        if CSS21_PROPERTIES.include?( matched ) # are they disjoint?
+          start_group :property, matched
+        elsif CSS21_KEYWORDS.include?( matched )
+          start_group :keyword, matched
+        else
+          start_group :ident, matched
+        end
+
+      # scanning both
+      when scan(@tokens[:S])
+        start_group :normal, matched
+      when scan(@tokens[:COMMENT])
+        start_group :comment, matched
+      when scan(@tokens[:STRING])
+        start_group :string, matched
+      when scan(@tokens[:CDO])
+        start_group :cdo, matched
+      when scan(@tokens[:CDC])
+        start_group :cdc, matched
+      when scan(@tokens[:INVALID])
+        start_group :invalid, matched
+      else
+        start_group :normal, scan(/./x)
+      end
+
+    end
+
+    private
+
+    def macro(name, regex=nil)
+      regex ? @macros[name] = regex : @macros[name].source
+    end
+
+    def token(name, pattern=nil, &block)
+      raise ArgumentError, "name required" unless name
+
+      patterns = []
+      patterns << pattern if pattern
+      yield(patterns) if block_given?
+      if patterns.empty?
+        raise ArgumentError, "at least one pattern required"
+      end
+      patterns.collect! do |pattern|
+        source = pattern.source
+        source = "\\A#{source}"
+        Regexp.new(source, Regexp::IGNORECASE + Regexp::MULTILINE)
+      end
+
+      @tokens[name] = Regexp.union(*patterns)
+    end
+
+    alias :m :macro
+
+  end
+
+  SYNTAX["css21"] = CSS21
+
+end
+
+

data/lib/syntax/lang/fortran.rb

@@ -0,0 +1,149 @@
+# fortran.rb -- a free-form Fortran module for the Ruby Syntax library
+#
+# Copyright (C) 2009-2010 Jason Blevins
+# License: 3-clause BSD (the same as Syntax itself)
+
+require 'syntax'
+
+module Syntax
+
+# A tokenizer for free-form Fortran source code.
+class Fortran < Tokenizer
+
+    # Fortran keywords
+    LC_KEYWORDS =
+        %w{ assign backspace call close common continue data dimension do
+            else end endfile endif enddo entry equivalence external
+            format function goto if implicit inquire intrinsic open
+            parameter pause print program read return rewind
+            save stop subroutine then write } +
+        [ "block data", "go to" ] +
+    # Fortran 90 keywords
+        %w{ allocate allocatable case contains cycle deallocate
+            elsewhere exit include interface intent module namelist nullify
+            only operator optional pointer private procedure public
+            result recursive select sequence target type use while where } +
+    # Fortran 95 keywords
+        %w{ elemental forall pure } +
+    # Fortran 2003 keywords
+        %w{ abstract associate asynchronous bind class deferred enum enumerator
+            extends final flush generic import non_overridable nopass pass
+            protected value volatile wait } +
+    # Fortran 2008 keywords
+        %w{ block codimension concurrent contiguous critical submodule
+            lock unlock } +
+        [ "error stop", "sync all", "sync images", "sync memory" ]
+
+    # List of identifiers recognized as types
+    LC_TYPES = [ "character", "complex", "logical", "integer", "real",
+                 "double precision" ]
+
+    # Fortran intrinsic procedures
+    LC_INTRINSICS = %w{
+        abs achar acos acosh adjustl adjustr aimag
+        aint all allocated anint any asin asinh
+        associated atan atan2 atanh atomic_define atomic_ref
+        bessel_j0 bessel_j1 bessel_jn
+        bessel_y0 bessel_y1 bessel_yn
+        bit_size btest
+        bge bgt ble blt
+        c_associated c_f_pointer c_f_procpointer
+        c_funloc c_loc c_sizeof
+        ceiling char cmplx command_argument_count
+        compiler_version compiler_options
+        conjg cos cosh count cpu_time cshift
+        date_and_time dble digits dim dot_product dprod
+        dshiftl dshiftr
+        eoshift epsilon erf erfc erfc_scaled exp exponent
+        execute_command_line extends_type_of
+        findloc float floor fraction
+        gamma get_command get_command_argument
+        get_environment_variable
+        huge hypot
+        iachar iall iand iany ibclr ibits ibset ichar image_index
+        int ior iparity is_contiguous is_iostat_end is_iostat_eor
+        ishft ishftc
+        kind
+        lbound lcobound leadz len len_trim lge lgt lle llt
+        log log10 log_gamma logical
+        maskl maskr matmul max maxexponent maxloc maxval merge merge_bits
+        min minexponent minloc minval mod modulo
+        move_alloc mvbits
+        nearest new_line nint norm2 not null num_images
+        pack parity popcnt poppar precision present product
+        radix random_number random_seed range real
+        repeat reshape rrspacing
+        same_type_as scale scan selected_char_kind selected_int_kind
+        selected_real_kind set_exponent shape shiftl shiftr sign
+        sin sinh size sngl spacing spread sqrt storage_size sum system_clock
+        tan tanh this_image tiny trailz transfer transpose trim
+        ubound ucobound unpack
+        verify
+    }
+
+    # Also support all uppercase keywords, types, and procedures
+    KEYWORDS = Set.new LC_KEYWORDS + LC_KEYWORDS.map { |x| x.upcase }
+    TYPES = Set.new LC_TYPES + LC_TYPES.map { |x| x.upcase }
+    INTRINSICS = Set.new LC_INTRINSICS + LC_INTRINSICS.map { |x| x.upcase }
+
+    # Step through a single iteration of the tokenization process.
+    def step
+      case
+      when check( /program\s+/ )
+        start_group :keyword, scan( /program\s+/ )
+        start_group :function,  scan_until( /(?=[;(\s]|#{EOL})/ )
+      when check( /subroutine\s+/ )
+        start_group :keyword, scan( /subroutine\s+/ )
+        start_group :function,  scan_until( /(?=[;(\s]|#{EOL})/ )
+      when check( /function\s+/ )
+        start_group :keyword, scan( /function\s+/ )
+        start_group :function,  scan_until( /(?=[;(\s]|#{EOL})/ )
+      when check( /module\s+/ )
+        start_group :keyword, scan( /module\s+/ )
+        start_group :function,  scan_until( /(?=[;\s]|#{EOL})/ )
+      when check( /\.true\.|\.false\.|\.TRUE\.|\.FALSE\./ )
+        start_group :constant,
+        scan(/\.true\.|\.false\.|\.TRUE\.|\.FALSE\./)
+      when scan( /(\d+\.?\d*|\d*\.?\d+)([eEdDqQ][+-]?\d+)?(_\w+)?/ )
+        start_group :number, matched
+      when scan( /[bB]\'[01]+\'|[oO]\'[0-7]+\'|[zZ]\'[0-9a-fA-F]+\'/ )
+        start_group :number, matched
+      else
+        case peek(1)
+        when /[\n\r]/
+          start_group :normal, scan( /\s+/ )
+        when /\s/
+          start_group :normal, scan( /\s+/ )
+        when "!"
+          start_group :comment, scan( /![^\n\r]*/ )
+        when /[A-Za-z]/
+          word = scan( /\w+/ )
+          if KEYWORDS.include?(word)
+            start_group :keyword, word
+          elsif TYPES.include?(word)
+            start_group :type, word
+          elsif INTRINSICS.include?(word)
+            start_group :function, word
+          elsif
+            start_group :ident, word
+          end
+        when '"'
+          # allow for continuation characters within strings
+          start_group :string, scan(/"([^"]*(&[ ]*[\n\r]+)?)*"/)
+        when "'"
+          # allow for continuation characters within strings
+          start_group :string, scan(/'([^']*(&[ ]*[\n\r]+)?)*'/)
+        when /[-!?*\/+=<>()\[\]\{}:;,&|%]/
+          start_group :punct, scan(/./)
+        else
+          # pass everything else through
+          append getch
+        end
+      end
+    end
+
+  end
+
+  SYNTAX["fortran"] = Fortran
+
+end

data/lib/syntax/lang/javascript.rb

@@ -0,0 +1,58 @@
+require 'syntax'
+
+module Syntax
+
+  class Javascript < Tokenizer
+    
+    JAVASCRIPT_KEYWORDS =
+      Set.new %w{abstract break case catch class const continue
+         debugger default delete do else enum export extends
+         final finally for function goto if implements
+         import in instanceof interface native new
+         package private protected public return
+         static super switch synchronized this throw
+         throws transient try typeof
+         var void volatile while with} unless const_defined?(:JAVASCRIPT_KEYWORDS)
+
+    JAVASCRIPT_PREDEFINED_TYPES =
+      Set.new %w{boolean byte char double float int long short} unless const_defined?(:JAVASCRIPT_PREDEFINED_TYPES)
+
+    JAVASCRIPT_PREDEFINED_CONSTANTS =
+      %w{null true false} unless const_defined?(:JAVASCRIPT_PREDEFINED_CONSTANTS)
+    
+    def step
+      case
+      when scan(/\s+/)
+        start_group :normal, matched
+      when scan(/\\u[0-9a-f]{4}/i)
+        start_group :unicode, matched
+      when scan(/0[xX][0-9A-Fa-f]+/)
+        start_group :hex, matched
+      when scan(/(?:0[0-7]+)(?![89.eEfF])/)
+        start_group :oct, matched
+      when scan(/(?:\d+)(?![.eEfF])/)
+        start_group :integer, matched
+      when scan(/\d[fF]?|\d*\.\d+(?:[eE][+-]?\d+)?[fF]?|\d+[eE][+-]?\d+[fF]?/)
+        start_group :float, matched
+      when (scan(/"(?:[^"\\]|\\.)*"/) or scan(/'(?:[^'\\]|\\.)*'/) )
+        start_group :string, matched
+      when scan(/[a-z_$][a-z_\d]*/i)
+        if JAVASCRIPT_KEYWORDS.include?( matched )
+          start_group :keyword, matched
+        elsif JAVASCRIPT_PREDEFINED_TYPES.include?( matched )
+          start_group :predefined_types, matched
+        else
+          start_group :ident, matched
+        end
+      when scan(%r! // [^\n\\]* (?: \\. [^\n\\]* )* | /\* (?: .*? \*/ | .* ) !mx)
+        start_group :comment, matched
+      else
+        start_group :other, scan(/./x)
+      end
+    end
+    
+  end
+
+  SYNTAX["javascript"] = Javascript
+
+end

data/lib/syntax/lang/ruby.rb

@@ -9,7 +9,7 @@ module Syntax
 
     # The list of all identifiers recognized as keywords.
     KEYWORDS =
-      %w{if then elsif else end begin do rescue ensure while for
+      Set.new %w{if then elsif else end begin do rescue ensure while for
          class module def yield raise until unless and or not when
          case super undef break next redo retry in return alias
          defined?}

data/lib/syntax/lang/sqlite.rb

@@ -0,0 +1,77 @@
+require 'syntax'
+
+module Syntax
+
+  class SQLite < Tokenizer
+    
+    def self.add_function(name)
+      SQLite_PREDEFINED_FUNCTIONS << name
+    end
+
+    SQLite_PREDEFINED_FUNCTIONS = Set.new %w{abs avg coalesce count glob
+      hex ifnull last_insert_rowid length like load_extension lower 
+      ltrim max min sum nullif
+      quote random randomblob replace round
+      rtrim soundex sqlite_version substr total
+      trim typeof upper zeroblob
+      date time datetime julianday
+      strftime over} unless const_defined?(:SQLite_PREDEFINED_FUNCTIONS)
+
+    SQLite_KEYWORDS = Set.new %w{abort add after all alter analyze
+      asc attach autoincrement before by cascade change character
+      check commit conflict constraint create cross  collate
+      current_date current_time current_timestamp database default
+      deferrable deferred delete desc detach distinct drop each
+      escape except exclusive explain fail for foreign from
+      full group having if ignore immediate
+      index initially inner insert instead intersect into is
+      join key left limit modify natural of offset on or order
+      outer plan pragma primary query raise references reindex
+      rename replace restrict right rollback row select set
+      table temp temporary to transaction trigger union
+      unique update using vacuum values view virtual
+      where partition} unless const_defined?(:SQLite_KEYWORDS)
+
+    SQLite_DATATYPES = Set.new %w{null none text numeric integer
+      text blob int varchar char real float
+      double} unless const_defined?(:SQLite_DATATYPES)
+
+    SQLite_OPERATORS = Set.new %w{not escape isnull notnull between and
+      in exists case when then else begin end cast as
+      like glob regexp < >  || * / % + - << >>
+      & | <= >= = == != <>
+      match} unless const_defined?(:SQLite_OPERATORS)
+    
+    def step
+      case
+      when scan(/\s+/)
+        start_group :normal, matched
+      when scan(%r{ "(?: \\. | \\" | [^"\n])*" }x)
+        start_group :string, matched
+      when scan(%r{ '(?: \\. | \\' | [^'\n])*' }x )
+        start_group :string, matched
+      when (scan(/[a-z_][a-z_\d]+/i) or scan(/[<>\|\*%&!=]+/))
+        m = matched.downcase
+        if SQLite_PREDEFINED_FUNCTIONS.include?( m )
+          start_group :function, matched
+        elsif SQLite_KEYWORDS.include?( m )
+          start_group :keyword, matched
+        elsif SQLite_DATATYPES.include?( m )
+          start_group :datatype, matched
+        elsif SQLite_OPERATORS.include?( m )
+          start_group :operator, matched
+        else
+          start_group :ident, matched
+        end
+      when scan(/--.*$/)
+        start_group :comment, matched
+      else
+        start_group :other, scan(/./x)
+      end
+    end
+
+  end
+
+  SYNTAX["sqlite"] = SQLite
+
+end

data/lib/syntax/lang/yaml.rb

@@ -61,6 +61,7 @@ module Syntax
             indent = check(/ */)
             start_group :string
             loop do
+              break if eos?
               line = check_until(/[\n\r]|\Z/)
               break if line.nil?
               if line.chomp.length > 0

data/lib/syntax/version.rb

@@ -1,7 +1,7 @@
 module Syntax
   module Version
     MAJOR=1
-    MINOR=0
+    MINOR=1
     TINY=0
 
     STRING=[MAJOR,MINOR,TINY].join('.')

data/test/ALL-TESTS.rb

@@ -2,4 +2,4 @@
 $:.unshift "../lib"
 
 Dir.chdir File.dirname(__FILE__)
-Dir["**/tc_*.rb"].each { |file| load file }
+Dir["**/tc_*.rb"].each { |file| load File.expand_path(file) }

metadata

@@ -1,58 +1,67 @@
---- !ruby/object:Gem::Specification 
-rubygems_version: 0.8.10
-specification_version: 1
+--- !ruby/object:Gem::Specification
 name: syntax
-version: !ruby/object:Gem::Version 
-  version: 1.0.0
-date: 2005-06-18
-summary: Syntax is Ruby library for performing simple syntax highlighting.
-require_paths: 
-  - lib
-email: jamis@jamisbuck.org
-homepage: 
-rubyforge_project: 
-description: 
+version: !ruby/object:Gem::Version
+  version: 1.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Jamis Buck
 autorequire: syntax
-default_executable: 
 bindir: bin
-has_rdoc: true
-required_ruby_version: !ruby/object:Gem::Version::Requirement 
-  requirements: 
-    - 
-      - ">"
-      - !ruby/object:Gem::Version 
-        version: 0.0.0
-  version: 
-platform: ruby
-authors: 
-  - Jamis Buck
-files: 
-  - data/ruby.css
-  - data/xml.css
-  - data/yaml.css
-  - lib/syntax
-  - lib/syntax.rb
-  - lib/syntax/common.rb
-  - lib/syntax/convertors
-  - lib/syntax/lang
-  - lib/syntax/version.rb
-  - lib/syntax/convertors/abstract.rb
-  - lib/syntax/convertors/html.rb
-  - lib/syntax/lang/ruby.rb
-  - lib/syntax/lang/xml.rb
-  - lib/syntax/lang/yaml.rb
-  - test/ALL-TESTS.rb
-  - test/syntax
-  - test/tc_syntax.rb
-  - test/syntax/tc_ruby.rb
-  - test/syntax/tc_xml.rb
-  - test/syntax/tc_yaml.rb
-  - test/syntax/tokenizer_testcase.rb
-test_files: 
-  - test/ALL-TESTS.rb
-rdoc_options: []
-extra_rdoc_files: []
+cert_chain: []
+date: 2013-12-11 00:00:00.000000000 Z
+dependencies: []
+description: 
+email: jamis@jamisbuck.org
 executables: []
 extensions: []
+extra_rdoc_files: []
+files:
+- lib/syntax/common.rb
+- lib/syntax/convertors/abstract.rb
+- lib/syntax/convertors/html.rb
+- lib/syntax/lang/ansic.rb
+- lib/syntax/lang/css21.rb
+- lib/syntax/lang/fortran.rb
+- lib/syntax/lang/javascript.rb
+- lib/syntax/lang/ruby.rb
+- lib/syntax/lang/sqlite.rb
+- lib/syntax/lang/xml.rb
+- lib/syntax/lang/yaml.rb
+- lib/syntax/version.rb
+- lib/syntax.rb
+- test/ALL-TESTS.rb
+- test/syntax/tc_ruby.rb
+- test/syntax/tc_xml.rb
+- test/syntax/tc_yaml.rb
+- test/syntax/tokenizer_testcase.rb
+- test/tc_syntax.rb
+- README.rdoc
+- LICENSE
+- CHANGELOG
+homepage: 
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
 requirements: []
-dependencies: []
+rubyforge_project: 
+rubygems_version: 1.8.25
+signing_key: 
+specification_version: 3
+summary: Syntax is Ruby library for performing simple syntax highlighting.
+test_files:
+- test/ALL-TESTS.rb

data/data/ruby.css

@@ -1,18 +0,0 @@
-.ruby .normal {}
-.ruby .comment { color: #005; font-style: italic; }
-.ruby .keyword { color: #A00; font-weight: bold; }
-.ruby .method { color: #077; }
-.ruby .class { color: #074; }
-.ruby .module { color: #050; }
-.ruby .punct { color: #447; font-weight: bold; }
-.ruby .symbol { color: #099; }
-.ruby .string { color: #944; background: #FFE; }
-.ruby .char { color: #F07; }
-.ruby .ident { color: #004; }
-.ruby .constant { color: #07F; }
-.ruby .regex { color: #B66; background: #FEF; }
-.ruby .number { color: #F99; }
-.ruby .attribute { color: #7BB; }
-.ruby .global { color: #7FB; }
-.ruby .expr { color: #227; }
-.ruby .escape { color: #277; }

data/data/xml.css

@@ -1,8 +0,0 @@
-.xml .normal {}
-.xml .namespace { color: #B66; font-weight: bold; }
-.xml .tag { color: #F88; }
-.xml .comment { color: #005; font-style: italic; }
-.xml .punct { color: #447; font-weight: bold; }
-.xml .string { color: #944; }
-.xml .number { color: #F99; }
-.xml .attribute { color: #BB7; }

data/data/yaml.css

@@ -1,12 +0,0 @@
-.yaml .normal {}
-.yaml .document { font-weight: bold; color: #07F; }
-.yaml .type { font-weight: bold; color: #05C; }
-.yaml .key { color: #F88; }
-.yaml .comment { color: #005; font-style: italic; }
-.yaml .punct { color: #447; font-weight: bold; }
-.yaml .string { color: #944; }
-.yaml .number { color: #F99; }
-.yaml .time { color: #F99; }
-.yaml .date { color: #F99; }
-.yaml .ref { color: #944; }
-.yaml .anchor { color: #944; }