extraloop 0.0.4 → 0.0.5

data/History.txt

@@ -1,3 +1,6 @@
+== 0.0.5  / 2011-01-14
+  * Refactored #extract, #loop_on, and #set_hook to make a more idematic use of ruby blocks
+
 == 0.0.4  / 2011-01-14
   * fixed a bug which prevented from subclassing `IterativeScraper` instances
 

data/README.md

@@ -8,70 +8,70 @@ for iterating over paginated datasets.
 
     gem install extraloop
 
-### Usage:
+### Sample scrapers:
 
-A basic scraper that fetches the top 25 websites from Alexa's daily top 100 list:
+A basic scraper that fetches the top 25 websites from [Alexa's daily top 100](www.alexa.com/topsites) list:
 
-    results = nil
-
-    ExtraLoop::ScraperBase.
+    alexa_scraper = ExtraLoop::ScraperBase.
       new("http://www.alexa.com/topsites").
       loop_on("li.site-listing").
         extract(:site_name, "h2").
         extract(:url, "h2 a").
         extract(:description, ".description").
-      on(:data, proc { |data| results = data }).
-      run()
+      on(:data) { |data| { |record| puts record.site_name } }
 
-An Iterative Scraper that fetches URL, title, and publisher from some 110 Google News articles mentioning the keyword 'Egypt'.
+    alexa_scraper.run
+
+An iterative Scraper that fetches URL, title, and publisher from some 110 Google News articles mentioning the keyword _'Egypt'_.
 
     results = []
 
     ExtraLoop::IterativeScraper.
-       new("https://www.google.com/search?tbm=nws&q=Egypt").
-       set_iteration(:start, (1..101).step(10)).
-       loop_on("h3", proc { |nodes| nodes.map(&:parent) }).
-         extract(:title, "h3.r a").
-         extract(:url, "h3.r a", :href).
-         extract(:source, "br", proc { |node| node.next.text.split("-").first }).
-       on(:data, proc { |data, response| data.each { |record| results << record } }).
-       run()
+      new("https://www.google.com/search?tbm=nws&q=Egypt").
+      set_iteration(:start, (1..101).step(10)).
+      loop_on("h3") { |nodes| nodes.map(&:parent) }.
+        extract(:title, "h3.r a").
+        extract(:url, "h3.r a", :href).
+        extract(:source, "br") { |node| node.next.text.split("-").first }.
+      on(:data) { |data, response| data.each { |record| results << record } }.
+      run()
 
 
-### Initializations
+### Scraper initialisation signature
 
-    #new(urls, scraper_options, httpclient_options)
+    #new(urls, scraper_options, http_options)
 
-- `urls` - single url, or array of several urls.
-- `scraper_options` - hash of scraper options (see below).
-- `httpclient_options` - hash of request options for `Typheous::Request#initialize` (see [API documentation](http://rubydoc.info/github/pauldix/typhoeus/master/Typhoeus/Request#initialize-instance_method) for details).
+- __urls__ - single url, or array of several urls.
+- __scraper_options__ - hash of scraper options (see below).
+- __http_options__ - hash of request options for `Typheous::Request#initialize` (see [API documentation](http://rubydoc.info/github/pauldix/typhoeus/master/Typhoeus/Request#initialize-instance_method) for details).
 
-#### Scraper options:
-* `format` - Specifies the scraped document format (valid values are :html, :xml, :json). 
-* `async` - Specifies whether the scraper's HTTP requests should be run in parallel or in series (defaults to false). **Note:** currently only GET requests can be run asynchronously.
-* `log` - Logging options hash:
-     * `loglevel`  - a symbol specifying the desired log level (defaults to `:info`).
-     * `appenders` - a list of Logging.appenders object (defaults to `Logging.appenders.sterr`).
+#### scraper options:
+
+* __format__ - Specifies the scraped document format (valid values are :html, :xml, :json). 
+* __async__ - Specifies whether the scraper's HTTP requests should be run in parallel or in series (defaults to false). **Note:** currently only GET requests can be run asynchronously.
+* __log__ - Logging options hash:
+     * __loglevel__  - a symbol specifying the desired log level (defaults to `:info`).
+     * __appenders__ - a list of Logging.appenders object (defaults to `Logging.appenders.sterr`).
 
 ### Extractors
 
 ExtraLoop allows to fetch structured data from online documents by looping through a list of elements matching a given selector.
 For each matched element, an arbitrary set of fields can be extracted. While the `loop_on` method sets up such loop, the `extract` 
-method extracts a piece of information from an element (e.g. a story's title) and and stores it into a record's field.
+method extracts a specific piece of information from an element (e.g. a story's title) and stores it into a record's field.
 
-    # using a CSS3 or an XPath selector
+    # looping over a set of document elements using a CSS3 (or XPath) selector
     loop_on('div.post')
 
-    # using a proc as a selector
+    # looping 
 
-    loop_on(proc { |doc| doc.search('div.post') })
+    loop_on { |doc| doc.search('div.post') }
 
-    # using both a selector and a proc (matched elements are passed in as the first argument of the proc )
+    # using both a selector and a proc (the matched element list is passed in to the proc as its first argument )
 
-    loop_on('div.post', proc { |posts| posts.reject { |post| post.attr(:class) == 'sticky' }})
+    loop_on('div.post') { |posts| posts.reject { |post| post.attr(:class) == 'sticky' } }
 
-Both the `loop_on` and the `extract` methods may be called with a selector, a proc or a combination of the two. By default, when parsing DOM documents, `extract` will call
-`Nokogiri::XML::Node#text()`. Alternatively, `extract` also accepts an attribute name or a proc as a third argument, this will be evaluated in the context of the matching element. 
+Both the `loop_on` and the `extract` methods may be called with a selector, a block or a combination of the two. By default, when parsing DOM documents, `extract` will call
+`Nokogiri::XML::Node#text()`. Alternatively, `extract` also accepts an attribute name and block, this will be evaluated in the context of the current iteration element. 
 
     # extract a story's title 
     extract(:title, 'h3')
@@ -80,34 +80,32 @@ Both the `loop_on` and the `extract` methods may be called with a selector, a pr
     extract(:url, "a.link-to-story", :href)
 
     # extract a description text, separating paragraphs with newlines 
-    extract(:description, "div.description", proc { |node|
-      node.css("p").map(&:text).join("\n") 
-    })
+    extract(:description, "div.description") { |node| node.css("p").map(&:text).join("\n") }
 
 #### Extracting from JSON Documents
 
 While processing each HTTP response, ExtraLoop tries to automatically detect the scraped document format by looking at 
-the `ContentType` header sent by the server (this value may be overriden by providing a `:format` key in the scraper's 
-initialization options). When the format is JSON, the document is parsed using the `yajl` parser and converted into a hash. 
-In this case, both the `loop_on` and the `extract` methods still behave as illustrated above, with the sole exception 
-of the CSS3/XPath selector string, which is specific of DOM documents. 
+the `ContentType` header sent by the server. This value can be overriden by providing a `:format` key in the scraper's 
+initialization options. When format is JSON, the document is parsed using the `yajl` JSON parser and converted into a hash. 
+In this case, both the `loop_on` and the `extract` methods still behave as illustrated above, except for the  
+CSS3/XPath selectors, which are specific to DOM documents. 
 
-When working with JSON data, you can just use a proc and return the document elements you want to loop on.
+When working with JSON data, you can just use a block and have it return the document elements you want to loop on.
 
     # Fetch a portion of a document using a proc
-    loop_on(proc { |data| data['query']['categorymembers'] })
+    loop_on  { |data| data['query']['categorymembers'] })
 
-Alternatively, the same loop can be defined by passing an array of keys pointing at a value located 
-at several levels of depth down into the parsed document hash.
+Alternatively, the same loop can be defined by passing an array of keys pointing at a hash value located 
+at several levels of depth down into the parsed document structure.
 
     # Fetch the same document portion above using a hash path
     loop_on(['query', 'categorymembers'])
 
-When fetching fields from a portion of a JSON document, `extract` will use the
-field name as a hash key if no key path or key string is provided.
+When fetching fields from a JSON document fragment, `extract` will often not need a block or an array of keys. If called with only
+one argument, it will in fact try to fetch a hash value using the provided field name as key.
 
     # current node:
-    # 
+    #
     # {
     #  'from_user' => "johndoe", 
     #  'text' => 'bla bla bla',
@@ -118,25 +116,27 @@ field name as a hash key if no key path or key string is provided.
     # => "johndoe"
 
 
-### Iteration methods:
+### Iteration methods
+
+The `IterativeScraper` class comes with two methods that allow scrapers to loop over paginated content.
+
+    set_iteration(iteration_parameter, array_range_or_block)
 
-The `IterativeScraper` class comes with two methods for defining how a scraper should loop over paginated content.
+* __iteration_parameter__ - A symbol identifying the request parameter that the scraper will use as offset in order to iterate over the paginated content.
+* __array_or_range_or_block__ - Either an explicit set of values or a block of code. If provided, the block is called with the parsed document object as its first argument. The block should return a non empty array, which will determine the value of the offset parameter during each iteration. If the block fails to return a non empty array, the iteration stops.
 
-_set_iteration(iteration_parameter, array_range_or_proc)_
 
-* `iteration_parameter` - A symbol identifying the request parameter that the scraper will use as offset in order to iterate over the paginated content.
-* `array_or_range_or_proc` - Either an explicit set of values or a block of code. If provided, the block is called with the parsed document as its first argument. Its return value is then used to shift, at each iteration, the value of the iteration parameter. If the block fails to return a non empty array, the iteration stops.
+The second iteration methods, `#continue_with`, allows to continue iterating untill an arbitrary block of code returns a positive, non-nil value (to be assigned to the iteration parameter).
 
-The second iteration methods, `#continue_with`, allows to continue iterating untill an arbitrary block of code returns a positive, non-nil value.
+    continue_with(iteration_parameter, &block)
 
-_continue_with(iteration_parameter, block)_
+* __iteration_parameter__ - the scraper' iteration parameter.
+* __&block__ - An arbitrary block of ruby code, its return value will be used to determine the value of the next iteration's offset parameter.
 
-* `iteration_parameter` - the scraper' iteration parameter.
-* `block` - An arbitrary block of ruby code, its return value will be used to determine the value of the next iteration's offset parameter.
 
 ### Running tests
 
 ExtraLoop uses `rspec` and `rr` as its testing framework. The test suite can be run by calling the `rspec` executable from within the `spec` directory:
 
-    cd spec 
+    cd spec
     rspec *

data/examples/google_news_scraper.rb

@@ -5,16 +5,15 @@ results = []
 ExtraLoop::IterativeScraper.new("https://www.google.com/search?tbm=nws&q=Egypt", :log => {
   :log_level => :debug,
   :appenders => [Logging.appenders.stderr ]
+
 }).set_iteration(:start, (1..101).step(10)).
-   loop_on("h3", proc { |nodes| nodes.map(&:parent) }).
+   loop_on("h3") { |nodes| nodes.map(&:parent) }.
      extract(:title, "h3.r a").
      extract(:url, "h3.r a", :href).
-     extract(:source, "br", proc { |node|
-       node.next.text.split("-").first
-     }).
-   on(:data, proc { |data, response|
+     extract(:source, "br") { |node| node.next.text.split("-").first }.
+   on(:data)  do |data, response|
      data.each { |record| results << record }
-   }).
+   end.
    run()
 
 results.each_with_index do |record, index|

data/lib/extraloop/hookable.rb

@@ -6,9 +6,8 @@ module ExtraLoop
       end
     end
 
-    def set_hook(hookname, handler)
+    def set_hook(hookname, &handler)
       @hooks ||= {}
-      raise Exceptions::HookArgumentError.new "handler must be a callable proc" unless handler.respond_to?(:call)
       @hooks[hookname.to_sym] ? @hooks[hookname.to_sym].push(handler) : @hooks[hookname.to_sym] = [handler]
       self
     end

data/lib/extraloop/iterative_scraper.rb

@@ -209,7 +209,7 @@ module ExtraLoop
     def update_request_params!
       offset = @iteration_set.at(@iteration_count) || default_offset
       @request_arguments[:params] ||= {}
-      @request_arguments[:params][@iteration_param.to_sym] = offset
+      @request_arguments[:params][@iteration_param.to_sym] = offset if @iteration_count > 0
     end
 
 
@@ -246,7 +246,7 @@ module ExtraLoop
 
     def issue_request(url)
       # remove continue argument if this is the first iteration
-      @request_arguments[:params].delete(@iteration_param.to_sym) if @continue_clause_args && @iteration_count == 0
+      #@request_arguments[:params].delete(@iteration_param.to_sym) if @continue_clause_args && @iteration_count == 0
       super(url)
       # clear previous value of iteration parameter
       @request_arguments[:params].delete(@iteration_param.to_sym) if @request_arguments[:params] && @request_arguments[:params].any?

data/lib/extraloop/scraper_base.rb

@@ -52,13 +52,15 @@ module ExtraLoop
     #
     #
     # selector  - The CSS3 selector identifying the node list over which iterate (optional).
-    # callback  - A block of code (optional).
     # attribute - An attribute name (optional).
     #
+    # callback  - A block of code (optional).
+    #
     # Returns itself.
     #
 
-    def loop_on(*args)
+    def loop_on(*args, &block)
+      args << block if block
       @loop_extractor_args = args.insert(0, nil, ExtractionEnvironment.new(self))
       self
     end
@@ -75,7 +77,8 @@ module ExtraLoop
     #
     #
 
-    def extract(*args)
+    def extract(*args, &block)
+      args << block if block
       @extractor_args << args.insert(1, ExtractionEnvironment.new(self))
       self
     end

data/spec/iterative_scraper_spec.rb

@@ -78,7 +78,7 @@ describe IterativeScraper do
         new("http://whatever.net/search-stuff").
         set_iteration(:p, iteration_proc).
         loop_on(".whatever").
-        set_hook(:data, proc { iteration_count += 1 }).
+        set_hook(:data, &proc { iteration_count += 1 }).
         run()
 
       @iteration_count = iteration_count
@@ -90,7 +90,7 @@ describe IterativeScraper do
       end
 
       it "should have sent p=1, p=2, p=3, p=4 as request parameters" do
-        @params_sent.should eql(["1", "2", "3", "4"])
+        @params_sent.should eql([nil, "2", "3", "4"])
       end
     end
   end

data/spec/scraper_base_spec.rb

@@ -24,16 +24,10 @@ describe ScraperBase do
   end
 
   describe "#set_hook" do
-    subject { @scraper.set_hook(:after, proc {}) }
+    subject { @scraper.set_hook(:after, &proc {}) }
     it { should be_an_instance_of(ScraperBase)  }
   end
 
-  describe "#set_hook" do
-    it "should raise exception if no proc is provided" do
-      expect { @scraper.set_hook(:after, :method) }.to raise_exception(ScraperBase::Exceptions::HookArgumentError)
-    end
-  end
-
   context "request params in both the url and the arguments hash" do
     describe "#run" do
       before do
@@ -76,7 +70,7 @@ describe ScraperBase do
           loop_on("ul li.file a").
             extract(:url, :href).
             extract(:filename).
-          set_hook(:data, proc { |records| records.each { |record| results << record }})
+          set_hook(:data, &proc { |records| records.each { |record| results << record }})
 
         @results = results
       end
@@ -110,7 +104,7 @@ describe ScraperBase do
           loop_on("ul li.file a").
             extract(:url, :href).
             extract(:filename).
-          set_hook(:data, proc { |records| records.each { |record| results << record } })
+          set_hook(:data, &proc { |records| records.each { |record| results << record } })
 
         @results = results
 
@@ -154,7 +148,7 @@ describe ScraperBase do
           loop_on("ul li.file a").
             extract(:url, :href).
             extract(:filename).
-          set_hook(:data, proc { |records| records.each { |record| results << record } })
+          set_hook(:data, &proc { |records| records.each { |record| results << record } })
 
         @results = results
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: extraloop
 version: !ruby/object:Gem::Version
-  version: 0.0.4
+  version: 0.0.5
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-01-14 00:00:00.000000000Z
+date: 2012-01-28 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: yajl-ruby
-  requirement: &10100300 !ruby/object:Gem::Requirement
+  requirement: &19726880 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -21,10 +21,10 @@ dependencies:
         version: 1.1.0
   type: :runtime
   prerelease: false
-  version_requirements: *10100300
+  version_requirements: *19726880
 - !ruby/object:Gem::Dependency
   name: nokogiri
-  requirement: &10098200 !ruby/object:Gem::Requirement
+  requirement: &19726420 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,10 +32,10 @@ dependencies:
         version: 1.5.0
   type: :runtime
   prerelease: false
-  version_requirements: *10098200
+  version_requirements: *19726420
 - !ruby/object:Gem::Dependency
   name: typhoeus
-  requirement: &10095680 !ruby/object:Gem::Requirement
+  requirement: &19725960 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,10 +43,10 @@ dependencies:
         version: 0.3.2
   type: :runtime
   prerelease: false
-  version_requirements: *10095680
+  version_requirements: *19725960
 - !ruby/object:Gem::Dependency
   name: logging
-  requirement: &10094320 !ruby/object:Gem::Requirement
+  requirement: &19725500 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -54,21 +54,21 @@ dependencies:
         version: 0.6.1
   type: :runtime
   prerelease: false
-  version_requirements: *10094320
+  version_requirements: *19725500
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &10092700 !ruby/object:Gem::Requirement
+  requirement: &19725040 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 2.7.1
+        version: 2.7.0
   type: :development
   prerelease: false
-  version_requirements: *10092700
+  version_requirements: *19725040
 - !ruby/object:Gem::Dependency
   name: rr
-  requirement: &10089100 !ruby/object:Gem::Requirement
+  requirement: &19724580 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -76,10 +76,10 @@ dependencies:
         version: 1.0.4
   type: :development
   prerelease: false
-  version_requirements: *10089100
+  version_requirements: *19724580
 - !ruby/object:Gem::Dependency
   name: pry
-  requirement: &10088040 !ruby/object:Gem::Requirement
+  requirement: &19724060 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -87,7 +87,7 @@ dependencies:
         version: 0.9.7.4
   type: :development
   prerelease: false
-  version_requirements: *10088040
+  version_requirements: *19724060
 description: A Ruby library for extracting data from websites and web based APIs.
   Supports most common document formats (i.e. HTML, XML, and JSON), and comes with
   a handy mechanism  for iterating over paginated datasets.