librariesio-gem-parser 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ee10e89cac8d75e23956264a511b98302f723179
+  data.tar.gz: 6cf5b395df0bab001f382b2bf3d2604eb0e768c7
+SHA512:
+  metadata.gz: 934211a80747662da08b4d997facdcf7086351297ae527b0466a62977708db039238d9bc1d63c5bf87eb0d74d80311a055e40e9c4dec92409192f1414cfbc5ac
+  data.tar.gz: ee17f4dc806bc5e49b2c791f4a5c145c6703fee288b7be5c53a93c00c90565e3f7a7c67a113f77ff31f52d7df818c147a4bd348f0186ebc0b02efd6e03334365

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.rvmrc
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/.travis.yml

@@ -0,0 +1,4 @@
+rvm:
+  - 1.9.2
+  - 1.9.3
+  - 2.0.0

data/CHANGELOG.md

@@ -0,0 +1,19 @@
+## 0.1.9 / 2012-11-19
+
+Bugfixes:
+
+* remove CR chars from content before parsing since it breaks patterns matching
+
+## 0.1.8 / 2012-11-18
+
+Bugfixes:
+
+* support :github option and exclude gems that use it
+* add CHANGELOG
+
+## 0.1.7 2012-10-17
+
+Bugfixes:
+
+* fix pattern to allow gem's
+ with a period

data/Gemfile

@@ -0,0 +1,3 @@
+source "https://rubygems.org"
+
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2011 Steve Richert
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,55 @@
+# Gemnasium::Parser
+
+[![Gem Version](https://badge.fury.io/rb/gemnasium-parser.png)](http://badge.fury.io/rb/gemnasium-parser)
+[![Build Status](https://travis-ci.org/gemnasium/gemnasium-parser.png?branch=master)](https://travis-ci.org/gemnasium/gemnasium-parser)
+[![Dependency Status](https://gemnasium.com/laserlemon/gemnasium-parser.png)](https://gemnasium.com/laserlemon/gemnasium-parser)
+[![Code Climate](https://codeclimate.com/github/laserlemon/gemnasium-parser.png)](https://codeclimate.com/github/laserlemon/gemnasium-parser)
+[![Coverage Status](https://coveralls.io/repos/laserlemon/gemnasium-parser/badge.png?branch=master)](https://coveralls.io/r/laserlemon/gemnasium-parser)
+
+The [Gemnasium](https://gemnasium.com/) parser determines gem dependencies from gemfiles and gemspecs, without evaluating the Ruby.
+
+## Why?
+
+[Bundler](http://gembundler.com/) is wonderful. It takes your gemfile and your gemspec (both just Ruby) and evaluates them, determining your gem dependencies. This works great locally and even on your production server… but only because you can be trusted!
+
+An untrustworthy character could put some pretty nasty stuff in a gemfile. If Gemnasium were to blindly evaluate that Ruby on its servers, havoc would ensue.
+
+### Solution #1
+
+If evaluating Ruby is so dangerous, just sandbox it! [Travis CI](http://travis-ci.org/) runs its builds inside isolated environments built with [Vagrant](http://vagrantup.com/). That way, if anything goes awry, it’s in a controlled environment.
+
+This is entirely possible with Gemnasium, but it’s impractical. Gemfiles often `require` other files in the repository. So to evaluate a gemfile, Gemnasium needs to clone the entire repo. That’s an expensive operation when only a couple files determine the dependencies.
+
+### Solution #2
+
+Parse Ruby like Ruby parses Ruby.
+
+Ruby 1.9 includes a library called Ripper. Ripper is a Ruby parsing library that can break down a gemfile or gemspec into bite-sized chunks, without evaluating the source. Then it can be searched for just the methods that matter.
+
+The problem is that it’s hard to make heads or tails from Ripper’s output, at least for me. I could see the Gemnasium parser one day moving to this strategy. But not today.
+
+### Third try’s the charm
+
+If we can’t evaluate the Ruby and Ripper’s output is unmanageable, how else can we find patterns in a gemfile or gemspec and get usable output?
+
+Regular expressions!
+
+The Gemnasium parser, for both gemfiles and gemspecs, is based on a number of Ruby regular expressions. These patterns match `gem` method calls in gemfiles and `add_dependency` calls in gemspecs.
+
+For a more comprehensive list of its abilities, see the [specs](https://github.com/laserlemon/gemnasium-parser/tree/master/spec).
+
+## Contributing <a name="contributing"></a>
+
+1. Fork it
+2. Create your branch (`git checkout -b my-bugfix`)
+3. Commit your changes (`git commit -am "Fix my bug"`)
+4. Push your branch (`git push origin my-bugfix`)
+5. Send a pull request
+
+## Problems?
+
+If you have a gemfile or gemspec that the Gemnasium parser screws up…
+
+1. Boil it down to its simplest problematic form
+2. Write a failing spec
+3. See [Contributing](#contributing)

data/Rakefile

@@ -0,0 +1,9 @@
+#!/usr/bin/env rake
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :test => :spec
+task :default => :spec

data/lib/gemnasium/parser.rb

@@ -0,0 +1,35 @@
+require "gemnasium/parser/configuration"
+require "gemnasium/parser/gemfile"
+require "gemnasium/parser/podfile"
+require "gemnasium/parser/gemspec"
+require "gemnasium/parser/podspec"
+
+module Gemnasium
+  module Parser
+    extend Configuration
+
+    def self.gemfile(content)
+      # Remove CR chars "\r" from content since it breaks Patterns matching
+      # TODO: Find something cleaner than this workaround
+      Gemnasium::Parser::Gemfile.new(content.gsub("\r",''))
+    end
+
+    def self.podfile(content)
+      # Remove CR chars "\r" from content since it breaks Patterns matching
+      # TODO: Find something cleaner than this workaround
+      Gemnasium::Parser::Podfile.new(content.gsub("\r",''))
+    end
+
+    def self.gemspec(content)
+      # Remove CR chars "\r" from content since it breaks Patterns matching
+      # TODO: Find something cleaner than this workaround
+      Gemnasium::Parser::Gemspec.new(content.gsub("\r",''))
+    end
+
+    def self.podspec(content)
+      # Remove CR chars "\r" from content since it breaks Patterns matching
+      # TODO: Find something cleaner than this workaround
+      Gemnasium::Parser::Podspec.new(content.gsub("\r",''))
+    end
+  end
+end

data/lib/gemnasium/parser/configuration.rb

@@ -0,0 +1,11 @@
+module Gemnasium
+  module Parser
+    module Configuration
+      attr_writer :runtime_groups
+
+      def runtime_groups
+        @runtime_groups ||= [:default, :production]
+      end
+    end
+  end
+end

data/lib/gemnasium/parser/gemfile.rb

@@ -0,0 +1,109 @@
+require "bundler"
+require "gemnasium/parser/patterns"
+
+module Gemnasium
+  module Parser
+    class Gemfile
+      attr_reader :content
+
+      def initialize(content)
+        @content = content
+      end
+
+      def dependencies
+        @dependencies ||= [].tap do |deps|
+          gem_matches.each do |match|
+            dep = dependency(match)
+            deps << dep if dep
+          end
+        end
+      end
+
+      def gemspec
+        @gemspec = if gemspec_match
+          opts = Patterns.options(gemspec_match["opts"])
+          path = opts["path"]
+          name = opts["name"] || "*"
+          File.join(*[path, "#{name}.gemspec"].compact)
+        end
+      end
+
+      def gemspec?
+        !!gemspec
+      end
+
+      private
+        def gem_matches
+          @gem_matches ||= matches(Patterns::GEM_CALL)
+        end
+
+        def matches(pattern)
+          [].tap{|m| content.scan(pattern){ m << Regexp.last_match } }
+        end
+
+        def dependency(match)
+          opts = Patterns.options(match["opts"])
+          clean!(match, opts)
+          name, reqs = match["name"], [match["req1"], match["req2"]].compact
+          Bundler::Dependency.new(name, reqs, opts).tap do |dep|
+            line = content.slice(0, match.begin(0)).count("\n") + 1
+            dep.instance_variable_set(:@line, line)
+          end
+        end
+
+        def groups(match)
+          group = group_matches.detect{|m| in_block?(match, m) }
+          group && Patterns.values(group[:grps])
+        end
+
+        def in_block?(inner, outer)
+          outer.begin(:blk) <= inner.begin(0) && outer.end(:blk) >= inner.end(0)
+        end
+
+        def group_matches
+          @group_matches ||= matches(Patterns::GROUP_CALL)
+        end
+
+        def git?(match, opts)
+          opts["git"] || in_git_block?(match)
+        end
+
+        def github?(match, opts)
+          opts["github"]
+        end
+
+        def in_git_block?(match)
+          git_matches.any?{|m| in_block?(match, m) }
+        end
+
+        def git_matches
+          @git_matches ||= matches(Patterns::GIT_CALL)
+        end
+
+        def path?(match, opts)
+          opts["path"] || in_path_block?(match)
+        end
+
+        def in_path_block?(match)
+          path_matches.any?{|m| in_block?(match, m) }
+        end
+
+        def path_matches
+          @path_matches ||= matches(Patterns::PATH_CALL)
+        end
+
+        def clean!(match, opts)
+          opts["group"] ||= opts.delete("groups")
+          opts["group"] ||= groups(match)
+          groups = Array(opts["group"]).flatten.compact
+          runtime = groups.empty? || !(groups & Parser.runtime_groups).empty?
+          opts["type"] ||= runtime ? :runtime : :development
+        end
+
+        def gemspec_match
+          return @gemspec_match if defined?(@gemspec_match)
+          @gemspec_match = content.match(Patterns::GEMSPEC_CALL)
+        end
+    end
+  end
+end

data/lib/gemnasium/parser/gemspec.rb

@@ -0,0 +1,38 @@
+module Gemnasium
+  module Parser
+    class Gemspec
+      attr_reader :content
+
+      def initialize(content)
+        @content = content
+      end
+
+      def dependencies
+        @dependencies ||= [].tap do |deps|
+          runtime_matches.each do |match|
+            dep = dependency(match)
+            deps << dep
+          end
+        end
+      end
+
+      private
+        def runtime_matches
+          @runtime_matches ||= matches(Patterns::ADD_DEPENDENCY_CALL)
+        end
+
+        def matches(pattern)
+          [].tap{|m| content.scan(pattern){ m << Regexp.last_match } }
+        end
+
+        def dependency(match)
+          name, reqs = match["name"], [match["req1"], match["req2"]].compact
+          type = match["type"] =~ /development/ ? :development : :runtime
+          Bundler::Dependency.new(name, reqs, "type" => type).tap do |dep|
+            line = content.slice(0, match.begin(0)).count("\n") + 1
+            dep.instance_variable_set(:@line, line)
+          end
+        end
+    end
+  end
+end

data/lib/gemnasium/parser/patterns.rb

@@ -0,0 +1,69 @@
+module Gemnasium
+  module Parser
+    module Patterns
+      GEM_NAME = /[a-zA-Z0-9\-_\.]+/
+      QUOTED_GEM_NAME = /(?:(?<gq>["'])(?<name>#{GEM_NAME})\k<gq>|%q<(?<name>#{GEM_NAME})>)/
+
+      QUOTED_POD_NAME = /(?:(?<gq>["'])(?<name>#{GEM_NAME})(\/[a-zA-Z0-9\-_\.]+)?\k<gq>|%q<(?<name>#{GEM_NAME})(\/[a-zA-Z0-9\-_\.]+)?>)/
+
+      MATCHER = /(?:=|!=|>|<|>=|<=|~>)/
+      VERSION = /[0-9]+(?:\.[a-zA-Z0-9]+)*/
+      REQUIREMENT = /[ \t]*(?:#{MATCHER}[ \t]*)?#{VERSION}[ \t]*/
+      REQUIREMENT_LIST = /(?<qr1>["'])(?<req1>#{REQUIREMENT})\k<qr1>(?:[ \t]*,[ \t]*(?<qr2>["'])(?<req2>#{REQUIREMENT})\k<qr2>)?/
+      REQUIREMENTS = /(?:#{REQUIREMENT_LIST}|\[[ \t]*#{REQUIREMENT_LIST}[ \t]*\])/
+
+      KEY = /(?::\w+|:?"\w+"|:?'\w+')/
+      SYMBOL = /(?::\w+|:"[^"#]+"|:'[^']+')/
+      STRING = /(?:"[^"#]*"|'[^']*')/
+      BOOLEAN = /(?:true|false)/
+      NIL = /nil/
+      ELEMENT = /(?:#{SYMBOL}|#{STRING})/
+      ARRAY = /\[(?:#{ELEMENT}(?:[ \t]*,[ \t]*#{ELEMENT})*)?\]/
+      VALUE = /(?:#{BOOLEAN}|#{NIL}|#{ELEMENT}|#{ARRAY}|)/
+      PAIR = /(?:(#{KEY})[ \t]*=>[ \t]*(#{VALUE})|(\w+):[ \t]+(#{VALUE}))/
+      OPTIONS = /#{PAIR}(?:[ \t]*,[ \t]*#{PAIR})*/
+      COMMENT = /(#[^\n]*)?/
+
+      GEM_CALL = /^[ \t]*gem\(?[ \t]*#{QUOTED_GEM_NAME}(?:[ \t]*,[ \t]*#{REQUIREMENT_LIST})?(?:[ \t]*,[ \t]*(?<opts>#{OPTIONS}))?[ \t]*\)?[ \t]*#{COMMENT}$/
+      POD_CALL = /^[ \t]*pod\(?[ \t]*#{QUOTED_GEM_NAME}(?:[ \t]*,[ \t]*#{REQUIREMENT_LIST})?(?:[ \t]*,[ \t]*(?<opts>#{OPTIONS}))?[ \t]*\)?[ \t]*#{COMMENT}$/
+
+      SYMBOLS = /#{SYMBOL}([ \t]*,[ \t]*#{SYMBOL})*/
+      GROUP_CALL = /^(?<i1>[ \t]*)group\(?[ \t]*(?<grps>#{SYMBOLS})[ \t]*\)?[ \t]+do[ \t]*?\n(?<blk>.*?)\n^\k<i1>end[ \t]*$/m
+
+      GIT_CALL = /^(?<i1>[ \t]*)git[ \(][^\n]*?do[ \t]*?\n(?<blk>.*?)\n^\k<i1>end[ \t]*$/m
+
+      PATH_CALL = /^(?<i1>[ \t]*)path[ \(][^\n]*?do[ \t]*?\n(?<blk>.*?)\n^\k<i1>end[ \t]*$/m
+
+      GEMSPEC_CALL = /^[ \t]*gemspec(?:\(?[ \t]*(?<opts>#{OPTIONS}))?[ \t]*\)?[ \t]*$/
+
+      ADD_DEPENDENCY_CALL = /^[ \t]*\w+\.add(?<type>_runtime|_development)?_dependency\(?[ \t]*#{QUOTED_GEM_NAME}(?:[ \t]*,[ \t]*#{REQUIREMENTS})?[ \t]*\)?[ \t]*#{COMMENT}$/
+      ADD_POD_DEPENDENCY_CALL = /^[ \t]*\w+\.dependency\(?[ \t]*#{QUOTED_POD_NAME}(?:[ \t]*,[ \t]*#{REQUIREMENTS})?[ \t]*\)?[ \t]*#{COMMENT}$/
+
+      def self.options(string)
+        {}.tap do |hash|
+          return hash unless string
+          pairs = Hash[*string.match(OPTIONS).captures.compact]
+          pairs.each{|k,v| hash[key(k)] = value(v) }
+        end
+      end
+
+      def self.key(string)
+        string.tr(%(:"'), "")
+      end
+
+      def self.value(string)
+        case string
+        when ARRAY then values(string.tr("[]", ""))
+        when SYMBOL then string.tr(%(:"'), "").to_sym
+        when STRING then string.tr(%("'), "")
+        when BOOLEAN then string == "true"
+        when NIL then nil
+        end
+      end
+
+      def self.values(string)
+        string.strip.split(/[ \t]*,[ \t]*/).map{|v| value(v) }
+      end
+    end
+  end
+end

data/lib/gemnasium/parser/podfile.rb

@@ -0,0 +1,91 @@
+require "bundler"
+require "gemnasium/parser/patterns"
+
+module Gemnasium
+  module Parser
+    class Podfile
+      attr_reader :content
+
+      def initialize(content)
+        @content = content
+      end
+
+      def dependencies
+        @dependencies ||= [].tap do |deps|
+          pod_matches.each do |match|
+            dep = dependency(match)
+            deps << dep if dep
+          end
+        end
+      end
+
+      private
+        def pod_matches
+          @pod_matches ||= matches(Patterns::POD_CALL)
+        end
+
+        def matches(pattern)
+          [].tap{|m| content.scan(pattern){ m << Regexp.last_match } }
+        end
+
+        def dependency(match)
+          opts = Patterns.options(match["opts"])
+          clean!(match, opts)
+          name, reqs = match["name"], [match["req1"], match["req2"]].compact
+          Bundler::Dependency.new(name, reqs, opts).tap do |dep|
+            line = content.slice(0, match.begin(0)).count("\n") + 1
+            dep.instance_variable_set(:@line, line)
+          end
+        end
+
+        def groups(match)
+          group = group_matches.detect{|m| in_block?(match, m) }
+          group && Patterns.values(group[:grps])
+        end
+
+        def in_block?(inner, outer)
+          outer.begin(:blk) <= inner.begin(0) && outer.end(:blk) >= inner.end(0)
+        end
+
+        def group_matches
+          @group_matches ||= matches(Patterns::GROUP_CALL)
+        end
+
+        def git?(match, opts)
+          opts["git"] || in_git_block?(match)
+        end
+
+        def github?(match, opts)
+          opts["github"]
+        end
+
+        def in_git_block?(match)
+          git_matches.any?{|m| in_block?(match, m) }
+        end
+
+        def git_matches
+          @git_matches ||= matches(Patterns::GIT_CALL)
+        end
+
+        def path?(match, opts)
+          opts["path"] || in_path_block?(match)
+        end
+
+        def in_path_block?(match)
+          path_matches.any?{|m| in_block?(match, m) }
+        end
+
+        def path_matches
+          @path_matches ||= matches(Patterns::PATH_CALL)
+        end
+
+        def clean!(match, opts)
+          opts["group"] ||= opts.delete("groups")
+          opts["group"] ||= groups(match)
+          groups = Array(opts["group"]).flatten.compact
+          runtime = groups.empty? || !(groups & Parser.runtime_groups).empty?
+          opts["type"] ||= runtime ? :runtime : :development
+        end
+    end
+  end
+end