RubyGems - extractify - Versions diffs - 0.0.1 - Mend

extractify 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (11) hide show

data/.gitignore ADDED

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile ADDED

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+# Specify your gem's dependencies in extractify.gemspec
+gemspec

data/LICENSE ADDED

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Brian John
+MIT License
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md ADDED

@@ -0,0 +1,105 @@
+# Extractify
+A trivial library to extract groups of data from XML or HTML.
+Do you have a crusty XML api or an HTML page that you want to do some quick and dirty scraping from?  Extractify might just be able to help you.
+Extractify uses the power of the mighty [Nokogiri](https://github.com/sparklemotion/nokogiri).
+## Installation
+Add this line to your application's Gemfile:
+    gem 'extractify'
+And then execute:
+    $ bundle
+Or install it yourself as:
+    $ gem install extractify
+## Usage
+Say you have some XML that you want to pull some data out of.
+```Ruby
+xml = '<?xml version="1.0"?>
+<movies>
+  <movie>
+    <title>Very Scary Movie</title>
+    <released>1999</released>
+    <length>117</length>
+    <director>Jane Doe</director>
+    <genre>Horror</genre>
+  </movie>
+  <movie>
+    <title>Funny Movie</title>
+    <released>2005</released>
+    <length>120</length>
+    <director>John Doe</director>
+    <genre>Comedy</genre>
+  </movie>
+  <movie>
+    <title>SciFi Movie</title>
+    <released>2009</released>
+    <length>105</length>
+    <director>Jim Doe</director>
+    <genre>Science Fiction</genre>
+  </movie>
+</movies>'
+Extractify.extract xml, '//movie' => { :name => '//title/text()', :runtime => '//length/text()' }
+=> [{:name=>"Very Scary Movie", :runtime=>"117"},
+    {:name=>"Funny Movie", :runtime=>"120"},
+    {:name=>"SciFi Movie", :runtime=>"105"}]
+```
+This also works with CSS selectors for HTML.
+```Ruby
+html = '
+<html>
+  <div class=movies>
+    <div class=movie>
+      <span class=title>Very Scary Movie</span>
+      <span class=released>1999</span>
+      <span class=length>117</span>
+      <span class=director>Jane Doe</span>
+      <span class=genre>Horror</span>
+    </div>
+    <div class=movie>
+      <span class=title>Funny Movie</span>
+      <span class=released>2005</span>
+      <span class=length>120</span>
+      <span class=director>John Doe</span>
+      <span class=genre>Comedy</span>
+    </div>
+    <div class=movie>
+      <span class=title>SciFi Movie</span>
+      <span class=released>2009</span>
+      <span class=length>105</span>
+      <span class=director>Jim Doe</span>
+      <span class=genre>Science Fiction</span>
+    </div>
+  </div>
+</html>'
+Extractify.extract html, '.movie' => { :name => '.title/text()', :runtime => '.length/text()' }
+=> [{:name=>"Very Scary Movie", :runtime=>"117"},
+    {:name=>"Funny Movie", :runtime=>"120"},
+    {:name=>"SciFi Movie", :runtime=>"105"}]
+```
+## Contributing
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile ADDED

	@@ -0,0 +1,2 @@
1	+ #!/usr/bin/env rake
2	+ require "bundler/gem_tasks"

data/extractify.gemspec ADDED

@@ -0,0 +1,21 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path('../lib/extractify/version', __FILE__)
+Gem::Specification.new do |gem|
+  gem.authors       = ["Brian John"]
+  gem.email         = ["brian@brianjohn.com"]
+  gem.description   = %q{extract grouped data from XML or HTML}
+  gem.summary       = %q{extract grouped data from XML or HTML}
+  gem.homepage      = "https://github.com/f1sherman/extractify"
+  gem.files         = `git ls-files`.split($\)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.name          = "extractify"
+  gem.require_paths = ["lib"]
+  gem.version       = Extractify::VERSION
+  gem.add_dependency 'nokogiri'
+  gem.add_development_dependency 'rspec'
+end

data/lib/extractify.rb ADDED

@@ -0,0 +1,37 @@
+require 'extractify/version'
+require 'nokogiri'
+module Extractify
+  def self.extract(doc, instructions = {})
+    results = []
+    doc = Nokogiri.parse(doc) unless doc.is_a?(Nokogiri::XML::Document)
+    results = extract_nodes(doc, instructions)
+    results
+  end
+  private
+  def self.extract_nodes(doc, instructions)
+    extracted_nodes = []
+    instructions.each_pair do |container_selector, element_selectors|
+      doc.search(container_selector).each do |container|
+        container_nodes = {}
+        element_selectors.each_pair do |label, selector|
+          selector = ".#{selector}" if selector[0] == '/'
+          node = container.search(selector).first
+          container_nodes[label] = node.to_s if node
+        end
+        extracted_nodes << container_nodes if container_nodes.any?
+      end
+    end
+    extracted_nodes
+  end
+end

data/lib/extractify/version.rb ADDED

@@ -0,0 +1,3 @@
+module Extractify
+  VERSION = "0.0.1"
+end

data/spec/extractify_spec.rb ADDED

@@ -0,0 +1,64 @@
+require 'spec_helper'
+describe Extractify do
+  describe :extract do
+    it 'should extract css selectors' do
+      Extractify.extract('<html><div class=bif>baz</div></html>', 'html' => { :bif => '.bif/text()' }).should == [{ :bif => 'baz' }]
+    end
+    it 'should extract xpath selectors' do
+      Extractify.extract('<foo><bar>baz</bar></foo>', 'foo' => { :bar => '//bar/text()' }).should == [{ :bar => 'baz' }]
+    end
+    it 'should handle multiple instructions' do
+      Extractify.extract('<html><div class=bif>baz</div><div class=baz>foo</div></html>', 'html' => { :bif => '.bif/text()', :baz => '.baz/text()' }).should == [{ :bif => 'baz', :baz => 'foo' }]
+    end
+    it 'should not find results outside of the container' do
+      Extractify.extract('<html><div class=bif>baz</div><div class=baz>foo</div></html>', 'doesnotexist' => { :bif => '.bif/text()', :baz => '.baz/text()' }).should == []
+    end
+    it 'should search only within the container' do
+      doc = '<parents>
+              <parent>
+                <child>child1</child>
+              </parent>
+              <parent>
+                <child>child2</child>
+              </parent>
+            </parents>'
+      Extractify.extract(doc, 'parent' => { :child => '//child/text()' }).should == [{:child => 'child1'}, {:child => 'child2'}]
+    end
+    it 'should group results in same order as instructions' do
+      doc = '<?xml version="1.0"?>
+              <movies>
+                <movie>
+                  <title>Very Scary Movie</title>
+                  <released>1999</released>
+                  <length>117</length>
+                  <director>Jane Doe</director>
+                  <genre>Horror</genre>
+                </movie>
+                <movie>
+                  <title>Funny Movie</title>
+                  <released>2005</released>
+                  <director>John Doe</director>
+                  <genre>Comedy</genre>
+                </movie>
+                <movie>
+                  <released>2009</released>
+                  <length>105</length>
+                  <director>Jim Doe</director>
+                  <genre>Science Fiction</genre>
+                </movie>
+              </movies>'
+        Extractify.extract(doc, '//movie' => { :name => '//title/text()', :runtime => '//length/text()' }).should == [
+                                                                                              {:name => 'Very Scary Movie', :runtime => '117'},
+                                                                                              {:name => 'Funny Movie'},
+                                                                                              {:runtime => '105'},
+                                                                                             ]
+    end
+    it 'should should allow Nokogiri documents to be passed in' do
+      Extractify.extract(Nokogiri.parse('<html><div class=bif>baz</div></html>'), 'html' => { :bif => '.bif/text()' }).should == [{ :bif => 'baz' }]
+    end
+  end
+end

data/spec/spec_helper.rb ADDED

	@@ -0,0 +1,2 @@
1	+ require 'rspec'
2	+ require 'extractify'

metadata ADDED

@@ -0,0 +1,89 @@
+--- !ruby/object:Gem::Specification
+name: extractify
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease:
+platform: ruby
+authors:
+- Brian John
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2012-07-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: extract grouped data from XML or HTML
+email:
+- brian@brianjohn.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- extractify.gemspec
+- lib/extractify.rb
+- lib/extractify/version.rb
+- spec/extractify_spec.rb
+- spec/spec_helper.rb
+homepage: https://github.com/f1sherman/extractify
+licenses: []
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 1.8.23
+signing_key:
+specification_version: 3
+summary: extract grouped data from XML or HTML
+test_files:
+- spec/extractify_spec.rb
+- spec/spec_helper.rb