benschwarz-smoke 0.2.3 → 0.2.4

data/README.markdown

@@ -4,7 +4,11 @@ smoke is a Ruby based DSL that allows you to take data from YQL, RSS / Atom (and
 This "data" can then be re-represented, sorted and filtered. You can collect data from a multiude of sources, sort them on a common property
 and return a plain old ruby object or json (You could add in something to output XML too)
 
-In an early attempt to get feedback I created [this screencast](http://vimeo.com/4272804). I will do another once the library becomes closer to 1.0.0.
+## Media
+
+* [Presentation from Melbourne #roro](http://www.slideshare.net/benschwarz/smoke-1371124)
+* Early [screencast](http://vimeo.com/4272804) to get developer / peer feedback
+
 
 ## The concept
 
@@ -13,9 +17,8 @@ The concept comes from using [Yahoo Pipes](http://pipes.yahoo.com) to make littl
 ## How or what to contribute
 
 * Test everything you do
-* Add a source (I was thinking a source that took simply took a url and parsed whatever it got)
 * Add a way to output (XML, anyone?)
-* Examples of queries you'd like to be able to do
+* Examples of queries you'd like to be able to do (email / github message them to me)
 
 ## API Examples
 ### YQL
@@ -53,13 +56,20 @@ Integrity [is running for smoke](http://integrity.ffolio.net/smoke)
 
 ### TODO (working on, just mental notes)
 
-* Look at moving object transformations into the origin class
+* Clean up / simplify source activation process
+* Consider variable injection (eg, allow to query sending a username within some aspect of the request)
+* Implement a disaptch / request method that will actually thread property (event-machine)
+* Checkout experimental fakeweb version to stub out the yql specs
+
+#### Later / maybe
 * Allow for sources to explicitly set the content type being returned for those stupid content providers
-* Consider invokation methods (registering of sources, namespacing et al)
-* YQL Definitions
 * YQL w/oAuth
 * YQL Subqueries?
 
+#### For wiki pages (docs, later)
+* How to use `path`
+* YQL Definitions
+
 ### Copyright
 
 Copyright (c) 2009 Ben Schwarz. See LICENSE for details.

data/VERSION.yml

@@ -1,4 +1,4 @@
 --- 
 :major: 0
 :minor: 2
-:patch: 3
+:patch: 4

data/lib/smoke/origin.rb

@@ -5,10 +5,60 @@ module Smoke
     def initialize(name, &block)
       raise StandardError, "Sources must have a name" unless name
       @name = name
-      @items = []
+      @items, @transformation = [], []
+
       activate!
       instance_eval(&block) if block_given?
+    end
+    
+    # Output your items in a range of formats (:ruby, :json and :yaml currently)
+    # Ruby is the default format and will automagically yielded from your source
+    #
+    # Usage
+    # 
+    #   output(:json)
+    #   => "[{title: \"Ray\"}, {title: \"Peace\"}]"
+    def output(type = :ruby)
       dispatch if respond_to? :dispatch
+      output = (@items.length == 1) ? @items.first : @items
+      
+      case type
+      when :ruby
+        return output
+      when :json
+        return ::JSON.generate(output)
+      when :yaml
+        return YAML.dump(output)
+      end      
+    end
+    
+    def items=(items) # :nodoc:
+      @items = items.flatten.map{|i| i.symbolize_keys! }
+      invoke_transformation
+    end
+    
+    # Path allows you to traverse the tree of a the items returned to 
+    # only give you access to what you're interested in.
+    #
+    # Usage:
+    # path :down, :to, :the, :data
+    # 
+    # Will traverse through a tree as follows:
+    #
+    #   {
+    #     :down => {
+    #       :to => {
+    #         :the => {
+    #           :data => []
+    #         }
+    #       }
+    #     }
+    #   }
+    #
+    # You will need to help smoke find an array of 
+    # items that you're interested in. 
+    def path(*path)
+      @path = path
     end
     
     # Transform each item
@@ -18,7 +68,7 @@ module Smoke
     #     rename(:href => :link)
     #   end 
     def emit(&block)
-      (@transformations ||= []) << DelayedBlock.new(&block)
+      @transformation << DelayedBlock.new(&block)
     end
 
     # Re-sort items by a particular key
@@ -43,7 +93,7 @@ module Smoke
       @items.reject! {|i| (i[key] =~ matcher) ? false : true }
       return self
     end
-    
+
     # Discard items that do not match the regex
     # 
     # Usage (chained): 
@@ -57,7 +107,7 @@ module Smoke
       @items.reject! {|i| (i[key] =~ matcher) ? true : false }
       return self
     end
-    
+
     # Rename one or many keys at a time
     # Suggested that you run it within an each block, but it makes no difference
     # other than readability
@@ -71,25 +121,7 @@ module Smoke
       @items.each {|item| item.rename(*args) }
       return self
     end
-    
-    # Output your items in a range of formats (:ruby, :json and :yaml currently)
-    # Ruby is the default format and will automagically yielded from your source
-    #
-    # Usage
-    # 
-    #   output(:json)
-    #   => "[{title: \"Ray\"}, {title: \"Peace\"}]"
-    def output(type = :ruby)
-      case type
-      when :ruby
-        return @items
-      when :json
-        return ::JSON.generate(@items)
-      when :yaml
-        return YAML.dump(@items)
-      end
-    end
-    
+
     # Truncate your result set to this many objects
     #
     # Usage
@@ -101,14 +133,15 @@ module Smoke
       return self
     end
     
-    def items=(items)
-      @items = items.map{|i| i.symbolize_keys! }
-      invoke_transformations
+    private
+    def invoke_transformation
+      @transformation.each{|t| t.execute! } unless @transformation.nil?
     end
     
-    private
-    def invoke_transformations
-      @transformations.each{|t| t.execute! } unless @transformations.nil?
+    def drill(*path)
+      path.inject(nil) do |obj, pointer|
+        obj = obj.nil? ? pointer : obj[pointer]
+      end
     end
     
     def activate!

data/lib/smoke/request.rb

@@ -5,7 +5,7 @@ module Smoke
       
       def initialize(uri, msg)
         @uri = URI.parse(uri)
-        Smoke.log.error "Smoke Request: Failed to get from #{@uri.host} via #{@uri.scheme}\n#{msg}"
+        Smoke.log.error "Smoke Request: Failed to get from #{@uri} (#{msg})"
       end
     end
     

data/lib/smoke/source/data.rb

@@ -32,25 +32,11 @@ module Smoke
           @url = source_url
         end
         
-        # Path allows you to traverse the tree of a the items returned to 
-        # only give you access to what you're interested in. In the case
-        # of the comments in this document I traverse to the photos returned.
-        def path(*path)
-          @path = path
-        end
-        
         protected
         def dispatch
           @request = Smoke::Request.new(@url)
           self.items = (@path.nil?) ? @request.body : drill(@request.body, *@path)
         end
-        
-        private
-        def drill(*path)
-          path.inject(nil) do |obj, pointer|
-            obj = obj.nil? ? pointer : obj[pointer]
-          end
-        end
       end
     end
   end

data/lib/smoke/source/yql.rb

@@ -16,6 +16,7 @@ module Smoke
       #     where   :query, "ruby"
       #   end
       class YQL < Origin
+        API_BASE = "http://query.yahooapis.com/v1/public/yql"
         attr_reader :request
         
         # Select indicates what YQL will be selecting
@@ -26,7 +27,7 @@ module Smoke
         #   => "SELECT title"
         #   select :title, :description
         #   => "SELECT title, description"
-        def select(what)
+        def select(what = :all)
           @select = what.join(",") and return if what.is_a? Array
           @select = "*" and return if what == :all
           @select = what.to_s
@@ -53,20 +54,38 @@ module Smoke
           @where << "#{column.to_s} = '#{value}'"
         end
         
+        # `use` can be used to set the url location of the data-table
+        # that you want YQL to search upon
+        #
+        # Usage:
+        #   use "http://datatables.org/alltables.env"
+        def use(url)
+          params.merge!({:env => url})
+        end
+        
         protected
+        def params 
+          @params || @params = {}
+        end
+        
         def dispatch
           @request = Smoke::Request.new(build_uri)
-          self.items = @request.body[:query][:results][:result]
+          self.items = [(@path.nil?) ? @request.body : drill(@request.body, *@path)]
         end
         
         private
-        
         def build_uri
-          "http://query.yahooapis.com/v1/public/yql?q=#{build_query}&format=json"
+          chunks = []
+          default_opts = {
+            :q => build_query,
+            :format => "json"
+          }.merge(params).each {|k,v| chunks << "#{k}=#{v}" }
+          
+          return URI.encode(API_BASE + "?" + chunks.join("&"))
         end
         
         def build_query
-          URI.encode("SELECT #{@select} FROM #{@from} WHERE #{@where.join(" AND ")}")
+          "SELECT #{@select} FROM #{@from} WHERE #{@where.join(" AND ")}"
         end
       end
     end

data/lib/smoke.rb

@@ -12,7 +12,7 @@ $:<< File.join(File.dirname(__FILE__))
 require 'core_ext/hash.rb'
 
 module Smoke  
-  class << self
+  class << self 
     
     @@active_sources = {}
     attr_reader :active_sources

data/spec/smoke/origin_spec.rb

@@ -18,6 +18,10 @@ describe Smoke::Origin do
     it "should be ordered by title" do
       @origin.output.first[:title].should == "Kangaroo"
     end
+    
+    it "should output a single hash rather than a hash in an array when there is one item" do
+      Smoke[:test].truncate(1).output.should == {:title => "Kangaroo"}
+    end
   end
   
   describe "transformations" do
@@ -44,7 +48,7 @@ describe Smoke::Origin do
       end
       
       it "should only contain items that match" do
-        Smoke[:keep].keep(:head, /^K/).output.should == [{:head => "Kangaroo"}]
+        Smoke[:keep].keep(:head, /^K/).output.should == {:head => "Kangaroo"}
       end
       
       it "should discard items" do
@@ -52,25 +56,36 @@ describe Smoke::Origin do
       end
       
       it "should not contain items that match" do
-        Smoke[:discard].discard(:head, /^K/).output.should == [{:head => "Platypus"}]
+        Smoke[:discard].discard(:head, /^K/).output.should == {:head => "Platypus"}
       end
     end
   end
   
-  it "should output" do
-    @origin.output.should be_an_instance_of(Array)
-  end
-  
-  it "should output two items" do
-    @origin.output.size.should == 2
-  end
-  
-  it "should output json" do
-    @origin.output(:json).should =~ /^\[\{/
-  end
-  
-  it "should output yml" do
-    @origin.output(:yaml).should =~ /--- \n- :title:/
+  describe "output" do
+    it "should output" do
+      @origin.output.should be_an_instance_of(Array)
+    end
+
+    it "should output two items" do
+      @origin.output.size.should == 2
+    end
+
+    it "should output json" do
+      @origin.output(:json).should =~ /^\[\{/
+    end
+
+    it "should output yml" do
+      @origin.output(:yaml).should =~ /--- \n- :title:/
+    end
+    
+    it "should dispatch when output is called" do
+      TestSource.source(:no_dispatch)
+      Smoke[:no_dispatch].should_not_receive(:dispatch)
+
+      TestSource.source(:dispatch)
+      Smoke[:dispatch].should_receive(:dispatch)
+      Smoke[:dispatch].output
+    end
   end
   
   it "method chaining" do
@@ -81,4 +96,10 @@ describe Smoke::Origin do
   it "should softly error when attempting to sort on a key that doesn't exist" do
     TestSource.source(:chain).sort(:header).should_not raise_error("NoMethodError")
   end
+  
+  describe "variable injection" do
+    it "should allow variables to be left happlessly in your source definitions"
+    it "should allow setting of variables"
+    it "should error if output is called before a variable has been set"
+  end
 end

data/spec/smoke/source/feed_spec.rb

@@ -27,16 +27,22 @@ describe "Feed" do
     Smoke[:slashdot].should respond_to(:url)
   end
   
-  it "should accept multiple urls" do
-    Smoke[:slashdot].requests.should be_an_instance_of(Array)
-  end
-  
-  it "should hold the url used to query" do
-    Smoke[:slashdot].requests.collect{|r| r.uri }.should include("http://slashdot.org/index.rdf")
-  end
-  
-  it "should have renamed url to link" do
-    Smoke[:slashdot].items.first.should have_key(:url)
-    Smoke[:slashdot].items.first.should_not have_key(:link)
+  describe "after dispatch / query" do
+    before do
+      Smoke[:slashdot].output
+    end
+    
+    it "should accept multiple urls" do
+      Smoke[:slashdot].requests.should be_an_instance_of(Array)
+    end
+    
+    it "should hold the url used to query" do
+      Smoke[:slashdot].requests.collect{|r| r.uri }.should include("http://slashdot.org/index.rdf")
+    end
+    
+    it "should have renamed url to link" do
+      Smoke[:slashdot].output.first.should have_key(:url)
+      Smoke[:slashdot].output.first.should_not have_key(:link)
+    end
   end
 end

data/spec/smoke/source/yql_spec.rb

@@ -4,15 +4,13 @@ describe "YQL" do
   before :all do
     # Fake web does not yet support regex matched uris
     
-    #FakeWeb.register_uri("query.yahooapis.com/*") do |response|
-    #  response.body = File.read(File.join(SPEC_DIR, 'supports', 'search-web.yql'))
-    #  response.content_type "text/json"
-    #end
+    FakeWeb.register_uri("query.yahooapis.com/*", :response => File.join(SPEC_DIR, 'supports', 'search-web.yql'))
     
     Smoke.yql(:search) do
       select  :all
       from    "search.web"
       where   :query, "ruby"
+      path    :query, :results, :result
       
       emit do
         rename(:url => :link)
@@ -28,16 +26,50 @@ describe "YQL" do
     Smoke[:search].items.should be_an_instance_of(Array)
   end
   
-  it "should hold the url used to query" do
-    Smoke[:search].request.uri.should == "http://query.yahooapis.com/v1/public/yql?q=SELECT%20*%20FROM%20search.web%20WHERE%20query%20=%20'ruby'&format=json"
-  end
-  
-  it "should have renamed url to link" do
-    Smoke[:search].items.first.should have_key(:link)
-    Smoke[:search].items.first.should_not have_key(:href)
+  describe "after dispatch" do
+    before do
+      Smoke[:search].output
+    end
+    
+    it "should hold the url used to query" do
+      Smoke[:search].request.uri.should == "http://query.yahooapis.com/v1/public/yql?q=SELECT%20*%20FROM%20search.web%20WHERE%20query%20=%20'ruby'&format=json"
+    end
+    
+    it "should have renamed url to link" do
+      Smoke[:search].output.first.should have_key(:link)
+      Smoke[:search].output.first.should_not have_key(:url)
+    end
+
+    it "should output a ruby object" do
+      Smoke[:search].output.should be_an_instance_of(Array)
+    end
   end
   
-  it "should output a ruby object" do
-    Smoke[:search].output.should be_an_instance_of(Array)
+  describe "yql definitions" do
+    before do
+      Smoke.yql(:smoke) do
+        use "http://datatables.org/alltables.env"
+
+        select :all
+        from "github.repo"
+        where :id, "benschwarz"
+        where :repo, "smoke"
+        path :query, :results
+      end
+      
+      Smoke[:smoke].output # Force execution
+    end
+
+    it "should be a respository" do
+      Smoke[:smoke].output.should have_key(:repository)
+    end
+
+    it "should respond to use" do
+      Smoke[:smoke].should respond_to(:use)
+    end
+    
+    it "should contain 'env' within the query string" do
+      Smoke[:smoke].request.uri.should =~ /env=/
+    end
   end
 end

data/spec/spec.opts

@@ -1 +1 @@
--c --format specdoc
+-c

data/spec/supports/test_source.rb

@@ -1,8 +1,6 @@
 module TestSource
   def self.source(name, &block)
-    source = Smoke::Origin.allocate
-    source.stub!(:dispatch)
-    source.send(:initialize, name, &block || Proc.new {})
+    source = Smoke::Origin.new(name, &block || Proc.new {})
     source.items = [
       {:head => "Platypus"},
       {:head => "Kangaroo"}

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: benschwarz-smoke
 version: !ruby/object:Gem::Version 
-  version: 0.2.3
+  version: 0.2.4
 platform: ruby
 authors: 
 - Ben Schwarz
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-04-29 00:00:00 -07:00
+date: 2009-05-31 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency