textpow1x 1.0.0 → 1.1.0

data/.travis.yml

@@ -0,0 +1,7 @@
+before_script:
+  - sudo apt-get install -y libonig-dev && bundle install
+script: "bundle exec rake"
+rvm:
+  - ree
+  - 1.9.2
+  - 1.9.3

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    textpow1x (1.0.0)
+    textpow1x (1.1.0)
       plist (>= 3.0.1)
 
 GEM

data/MIT-LICENSE.txt

@@ -0,0 +1,18 @@
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -1,4 +1,4 @@
-A library for parsing {TextMate}[http://macromates.com/] bundles.
+Read {TextMate}[http://macromates.com/] syntax files and parse text with them.
 
 For Ruby 1.8 and 1.9.
 
@@ -31,180 +31,53 @@ Install oniguruma
 
 3. Parse some text:
 
-  syntax.parse( text,  processor )
-
-
-=== INDEPTH:
-
-At the heart of syntax parsing are ..., well, syntax files. Lets see for instance
-the example syntax that appears in textmate's
-{documentation}[http://macromates.com/textmate/manual/language_grammars#language_grammars]:
-
-
-  {  scopeName = 'source.untitled';
-     fileTypes = ( txt );
-     foldingStartMarker = '\{\s*$';
-     foldingStopMarker = '^\s*\}';
-     patterns = (
-        {  name = 'keyword.control.untitled';
-           match = '\b(if|while|for|return)\b';
-        },
-        {  name = 'string.quoted.double.untitled';
-           begin = '"';
-           end = '"';
-           patterns = (
-              {  name = 'constant.character.escape.untitled';
-                 match = '\\.';
-              }
-           );
-        },
-     );
-  }
-
-But Textpow is not able to parse text pfiles. However, in practice this is not a problem,
-since it is possible to convert both text and binary pfiles to an XML format. Indeed, all
-the syntaxes in the Textmate syntax {repository}[http://macromates.com/svn/Bundles/trunk/Bundles/]
-are in XML format:
-
-  <?xml version="1.0" encoding="UTF-8"?>
-  <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
-  <plist version="1.0">
-  <dict>
-     <key>scopeName</key>
-     <string>source.untitled</string>
-     <key>fileTypes</key>
-     <array>
-        <string>txt</string>
-     </array>
-     <key>foldingStartMarker</key>
-     <string>\{\s*$</string>
-     <key>foldingStopMarker</key>
-     <string>^\s*\}</string>
-     <key>patterns</key>
-     <array>
-        <dict>
-           <key>name</key>
-           <string>keyword.control.untitled</string>
-           <key>match</key>
-           <string>\b(if|while|for|return)\b</string>
-        </dict>
-        <dict>
-           <key>name</key>
-           <string>string.quoted.double.untitled</string>
-           <key>begin</key>
-           <string>"</string>
-           <key>end</key>
-           <string>"</string>
-           <key>patterns</key>
-           <array>
-              <dict>
-                 <key>name</key>
-                 <string>constant.character.escape.untitled</string>
-                 <key>match</key>
-                 <string>\\.</string>
-              </dict>
-           </array>
-        </dict>
-     </array>
-  </dict>
-
-Of course, most people find XML both ugly and cumbersome. Fortunately, it is
-also possible to store syntax files in YAML format, which is much easier to
-read:
-
-  ---
-  fileTypes:
-  - txt
-  scopeName: source.untitled
-  foldingStartMarker: \{\s*$
-  foldingStopMarker: ^\s*\}
-  patterns:
-  - name: keyword.control.untitled
-    match: \b(if|while|for|return)\b
-  - name: string.quoted.double.untitled
-    begin: '"'
-    end: '"'
-    patterns:
-    - name: constant.character.escape.untitled
-      match: \\.
-
-==== Processors
-
-Until now we have talked about the parsing process without explaining what
-it is exactly. Basically, parsing consists in reading text from a string or
-file and applying tags to parts of the text according to what has been
-specified in the syntax file.
-
-In textpow, the process takes place line by line, from the beginning to the
-end and from left to right for every line. As the text is parsed, events are
-sent to a processor object when a tag is open or closed and so on.
-A processor is any object which implements one or more of the following
-methods:
-
-  class Processor
-     def open_tag name, position
-     end
-
-     def close_tag name, position
-     end
-
-     def new_line line
-     end
-
-     def start_parsing name
-     end
-
-     def end_parsing name
-     end
+  syntax.parse(text,  processor)
+
+=== Syntax-files
+
+Syntax files are written with {Textmate grammer}[http://macromates.com/textmate/manual/language_grammars#language_grammars].
+They are written in text/binary/xml/yaml format (all can be translated into each other).
+Textpow can understand the yaml form (ending in .syntax) and the xml form (ending in .tmSyntax or .plist) via plist library, but not the text or binary form.
+All syntax files shipped with textpow are in the yaml form, since its easiest to read and less verbose.
+
+Adding a new syntax:
+* use <tt>rake convert[filename]</tt> to convert it into yaml format
+* drop it into the lib/textpow/syntax folder
+* run tests <tt>rake</tt> to see if the syntax is parseable
+
+=== Processors
+A processors is a hook that is called while parsing a text.
+It must implement these methods:
+
+  class MyProcessor
+    def open_tag(tag_name, position_in_current_line); end
+    def close_tag(tag_name, position_in_current_line); end
+
+    # called before processing a line
+    def new_line(line_content); end
+
+    # called before parsing anything
+    def start_parsing(scope_name); end
+
+    # called before parsing everything
+    def end_parsing(scope_name); end
   end
 
-* <tt>open_tag</tt>. Is called when a new tag is opened, it receives the tag's name and
-  its position (relative to the current line).
-* <tt>close_tag</tt>. The same that <tt>open_tag</tt>, but it is called when a tag is closed.
-* <tt>new_line</tt>. Is called every time that a new line is processed, it receives the
-  line's contents.
-* <tt>start_parsing</tt>. Is called once at the beginning of the parsing process. It
-  receives the scope name for the syntax being used.
-* <tt>end_parsing</tt>. Is called once after all the input text has been parsed. It
-  receives the scope name for the syntax being used.
-
-Textpow ensures that the methods are called in parsing order, thus,
-for example, if there are two subsequent calls to <tt>open_tag</tt>, the first
+Textpow ensures that the methods are called in parsing order.
+If there are two subsequent calls to <tt>open_tag</tt>, the first
 having <tt>name="text.string", position=10</tt> and the second having
-<tt>name="markup.string", position=10</tt>, it should be understood that the
+<tt>name="markup.string", position=10</tt>, it means that the
 <tt>"markup.string"</tt> tag is inside the <tt>"text.string"</tt> tag.
 
-== CREDITS:
+== TODO
 
-* {Michael Grosser}[http://github.com/grosser]
-* {Chris Hoffman}[http://github.com/cehoffman]
-* {Spox}[http://github.com/spox]
-* Dizan Vasque
+* parse text plist files {example}[https://raw.github.com/kangax/textmate-js-language-syntax-file/master/JavaScript.plist]
+* parse tmLanguage files {example}[https://github.com/mattfarina/Ruby-Sass/blob/master/Ruby%20Sass.tmLanguage]
+* document / spec / fix include languages (they dont work very well)
 
-== LICENSE:
+== LICENSE: MIT
 
-(The MIT License)
-
-* Copyright (c) 2011 Michael Grosser
-* Copyright (c) 2010 Chris Hoffman
-* Copyright (c) 2009 Spox
+* Copyright (c) 2011 {Michael Grosser}[http://github.com/grosser]
+* Copyright (c) 2010 {Chris Hoffman}[http://github.com/cehoffman]
+* Copyright (c) 2009 {Spox}[http://github.com/spox]
 * Copyright (c) 2007-2008 Dizan Vasquez
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-'Software'), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -4,3 +4,25 @@ Bundler::GemHelper.install_tasks
 task :default do
   sh "rspec spec/"
 end
+
+desc "Convert a plist to a .syntax file"
+task :convert, :file do |t,args|
+  require 'plist'
+  require 'yaml'
+  yaml = Plist.parse_xml(args[:file]).to_yaml
+  File.open('out.syntax','w'){|f| f.write(yaml) }
+end
+
+rule /^version:bump:.*/ do |t|
+  sh "git status | grep 'nothing to commit'" # ensure we are not dirty
+  index = ['major', 'minor','patch'].index(t.name.split(':').last)
+  file = 'lib/textpow/version.rb'
+
+  version_file = File.read(file)
+  old_version, *version_parts = version_file.match(/(\d+)\.(\d+)\.(\d+)/).to_a
+  version_parts[index] = version_parts[index].to_i + 1
+  new_version = version_parts * '.'
+  File.open(file,'w'){|f| f.write(version_file.sub(old_version, new_version)) }
+
+  sh "bundle && git add #{file} Gemfile.lock && git commit -m 'bump version to #{new_version}'"
+end

data/examples/benchmark_js.rb

@@ -0,0 +1,11 @@
+$LOAD_PATH.unshift 'lib'
+require 'textpow'
+
+syntax = Textpow::SyntaxNode.load("#{Textpow.syntax_path}/javascript.syntax")
+text = File.read('examples/jquery.js')
+processor = Textpow::RecordingProcessor.new
+
+start = Time.now.to_f
+5.times{ syntax.parse(text,  processor) }
+puts Time.now.to_f - start
+