smoke 0.0.3 → 0.5.9

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Ben Schwarz
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,101 @@
+# smoke
+
+smoke is a Ruby based DSL that allows you to query web services such as YQL, RSS / Atom and JSON or XML in an elegant manner.
+
+These "services" can then be re-represented, sorted and filtered. Data can be collected from multiple sources, sorted via a common property, chopped up and refried. 
+
+Then you can output as a plain ruby object (or JSON)
+
+## Examples of use
+
+* The `examples` directory has something to get you running straight away
+* I powered [my entire site](http://www.germanforblack.com) [using smoke](http://github.com/benschwarz/benschwarz-site/blob/44de70463c744d821d3ffd2cf940e6d3e415fbdd/lib/stream.rb), until further documentation exists, this is probably a good place to start.
+* Read further details in the [rdoc documentation](http://rdoc.info/projects/benschwarz/smoke) or [the wiki](http://wiki.github.com/benschwarz/smoke)
+
+## Media
+
+* [Presentation from Melbourne #roro](http://www.slideshare.net/benschwarz/smoke-1371124)
+* Early [screencast](http://vimeo.com/4272804) to get developer / peer feedback
+
+
+## The concept
+
+The concept comes from using [Yahoo Pipes](http://pipes.yahoo.com) to make web based mash ups: Get a list of tv shows for my torrent client, compile a recipe book or make tools to give me a list of albums that artists in my music library are about to be released.
+
+## How or what to contribute
+
+* Test everything you do
+* Add a way to output (XML, anyone?)
+* Examples of queries you'd like to be able to do (email / github message them to me)
+
+## API Examples
+### YQL
+    # This will use yahoo search to get an array of search results about Ruby
+    Smoke.yql(:ruby) do
+      select  :all
+      from    "search.web"
+      where   :query, "ruby"
+      
+      discard :title, /tuesday/i
+    end
+
+    Smoke.yql(:python) do
+      select  :all
+      from    "search.web"
+      where   :query, "python"
+    end
+
+### Join sources and use them together
+    Smoke.join(:ruby, :python)
+
+or even
+
+    Smoke.join(:python, :ruby) do
+      emit do
+        sort :title
+        rename :shit_name => :title
+      end
+    end
+    
+### Define a source allowing for variables to be injected later
+
+Source definition:
+
+    Smoke.feed :delicious do
+      prepare do
+        url "http://feeds.delicious.com/v2/rss/#{username}?count=15"
+      end
+    end
+
+Execution: 
+
+    Smoke[:delicious].username("bschwarz").output
+    
+### CI
+
+Integrity [is running for smoke](http://integrity.ffolio.net/smoke)
+
+
+### TODO (working on, just mental notes)
+#### Later / maybe
+* YQL w/oAuth
+* YQL Subqueries?
+* Implement basic auth for sources
+
+
+#### For wiki pages (docs, later)
+* Document all sources with their irrespective differential methods
+* How to use `path`
+* YQL Definitions
+* Tranformations
+* Insert
+* Joining
+* Variable injection
+* Sort, Reverse
+* Keep, Discard
+* Truncate
+* Manually setting the content type for a url
+
+### Copyright
+
+Copyright (c) 2009 Ben Schwarz. See LICENSE for details.

data/Rakefile

@@ -1,4 +1,53 @@
-require 'config/requirements'
-require 'config/hoe' # setup Hoe + all gem configuration
-
-Dir['tasks/**/*.rake'].each { |rake| load rake }
+require 'rubygems'
+require 'rake'
+require 'spec'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "smoke"
+    gem.summary = %Q{smoke is a Ruby based DSL that allows you to query web services such as YQL, RSS / Atom and JSON or XML in an elegant manner.}
+    gem.email = "ben.schwarz@gmail.com"
+    gem.homepage = "http://github.com/benschwarz/smoke"
+    gem.authors = ["Ben Schwarz"]
+    gem.files = FileList['lib/**/*.rb', 'rdoc/**/*', '[A-Z]*', 'spec/**/*', 'vendor/**/*'].to_a
+    gem.add_dependency("simple-rss", "1.2")
+    gem.add_dependency("json", "1.1.3")
+    gem.add_dependency("fakeweb", "1.2.5")
+    gem.add_dependency("crack", "0.1.1")
+    gem.add_dependency("moneta", "0.6.0")
+    gem.add_dependency("rest-client", "1.0.3")
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+begin
+  gem 'mislav-hanna', '>= 0.2.7'
+  require 'hanna/rdoctask'
+rescue LoadError
+  require 'rake/rdoctask'
+end
+Rake::RDocTask.new do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = 'smoke'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+require 'spec/rake/spectask'
+Spec::Rake::SpecTask.new(:test) do |t|
+  t.spec_opts = ['--options', "spec/spec.opts"]
+  t.spec_files = FileList['spec/**/*_spec.rb']
+end
+
+Spec::Rake::SpecTask.new(:rcov) do |spec|
+  spec.libs << 'lib' << 'spec'
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+task :spec => :test
+task :default => :test

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:patch: 9
+:major: 0
+:minor: 5

data/examples/web_search_joined.rb

@@ -0,0 +1,25 @@
+require 'lib/smoke'
+Smoke.configure do |c|
+  c[:enable_logging] = true
+  c[:cache][:enabled] = true
+  c[:cache][:store] = :memory
+end
+Smoke.yql(:python) do
+  select :all
+  from "search.web"
+  where :query, "python"
+  
+  path :query, :results, :result
+end
+Smoke[:python].output
+
+
+Smoke.yql(:ruby) do
+  select :all
+  from "search.web"
+  where :query, "ruby"
+  
+  path :query, :results, :result
+end
+
+Smoke.join(:ruby, :python)

data/examples/yql_web_search.rb

@@ -0,0 +1,11 @@
+Smoke.yql(:ruby) do
+  select :all
+  from "search.web"
+  where :query, "ruby"
+  
+  emit do
+    sort :title
+    rename :url => :link
+    truncate 5
+  end
+end

data/lib/core_ext/hash.rb

@@ -0,0 +1,20 @@
+class Hash # :nodoc: 
+  
+  # Thanks merb!
+  def symbolize_keys!
+    each do |k,v| 
+      sym = k.respond_to?(:to_sym) ? k.to_sym : k 
+      self[sym] = Hash === v ? v.symbolize_keys! : v 
+      delete(k) unless k == sym
+    end
+    self
+  end
+  
+  def rename(candidates)
+    candidates.each do |old_key, new_key|
+      self[new_key] = self.delete(old_key) if self.has_key?(old_key)
+    end
+    
+    return self
+  end
+end

data/lib/core_ext/string.rb

@@ -0,0 +1,6 @@
+class String
+  def camel_case
+    return self if self !~ /_/ && self =~ /[A-Z]+.*/
+    split('_').map{|e| e.capitalize}.join
+  end
+end

data/lib/smoke.rb

@@ -1,5 +1,105 @@
-$:.unshift File.dirname(__FILE__)
+require 'logger'
+require 'digest/md5'
 
-module Smoke
-  
+require 'simple-rss'
+require 'json'
+require 'crack'
+require 'moneta'
+require 'restclient'
+
+module Smoke  
+  class << self
+    @@active_sources = {}
+    @@config = {
+      :enable_logging => true,
+      :user_agent     => "Ruby/#{RUBY_VERSION}/Smoke",
+      :cache          => {
+        :enabled  => false,
+        :store    => :memory,
+        :options  => {},
+        :expiry   => 1800
+      }
+    }
+    
+    # Access registered smoke source instances
+    #
+    # Define your source:
+    #     Smoke.yql(:ruby) do ....
+    # Then access it:
+    #     Smoke[:ruby]
+    #     => #<Smoke::Source::YQL::0x18428d4...
+    def [](source)
+      active_sources[source]
+    end
+    
+    # Activates new instances of sources
+    # Source instances are stored within the 
+    # @@active_sources class variable for later use
+    def activate(name, source)
+      if active_sources.has_key?(name)
+        Smoke.log.warn "Smoke source activation: Source with idential name already initialized" 
+      end
+      active_sources.update({ name => source })
+    end
+    
+    # Returns all activated smoke sources
+    def active_sources
+      @@active_sources
+    end
+    
+    # Rename a source 
+    def rename(candidates)
+      candidates.each do |o, n| 
+        active_sources[o].name = n.to_s
+        active_sources.rename(o => n)
+        return active_sources[n]
+      end
+    end
+    
+    # Log for info, debug, error and warn with:
+    # 
+    #   Smoke.log.info "message"
+    #   Smoke.log.debug "message"
+    #   Smoke.log.error "message"
+    #   Smoke.log.warn "message"
+    def log
+      @@log ||= Logger.new(config[:enable_logging] ? $stdout : "/dev/null")
+    end
+    
+    # Set any configurable options
+    #
+    #     Smoke.configure do |c|
+    #       c[:user_agent] = "Some other site"
+    #     end
+    #
+    def configure(&block)
+      yield @@config
+    end
+    
+    # Access configuration options
+    #
+    #   Smoke.config[:option_name]
+    #   => true
+    def config
+      @@config 
+    end
+    
+    def yql(name, &block); Smoke::Source::YQL.new(name, &block); end
+    def data(name, &block); Smoke::Source::Data.new(name, &block); end
+    def feed(name, &block); Smoke::Source::Feed.new(name, &block); end
+    
+    # Join multiple sources together into a single feed
+    # Usage:
+    #   Smoke.join(:delicious, :twitter, :flickr) do
+    #     name :stream
+    #     path :photos, :photo
+    #   end
+    def join(*names, &block); Smoke::Source::Join.new(names, &block); end
+  end
+end
+
+%w(core_ext/hash core_ext/string smoke/cache smoke/request smoke/origin).each {|r| require File.join(File.dirname(__FILE__), r)}
+
+class Object # :nodoc: 
+  include Smoke
 end

data/lib/smoke/cache.rb

@@ -0,0 +1,56 @@
+module Smoke
+  class Cache
+    class << self
+      def fetch(uri, options)  
+        output = (enabled?) ? read(uri) : query(uri, options)
+        
+        unless output.keys.any?
+          Smoke.log.info "Cache miss"
+          output = query(uri, options)
+        end
+        
+        output
+      rescue
+        query(uri, options)
+      end
+
+      def enabled?
+        Smoke.config[:cache][:enabled]
+      end
+  
+      protected
+      def cache
+        Moneta.autoload(klass_name.to_sym, file_name)
+        @@cache ||= Moneta.const_get(klass_name).new(Smoke.config[:cache][:options])
+      end
+            
+      def file_name
+        "moneta/#{Smoke.config[:cache][:store].to_s}"
+      end
+      
+      def klass_name
+        Smoke.config[:cache][:store].to_s.camel_case
+      end
+      
+      def query(uri, options)
+        request = RestClient.get(uri, options)
+        write(uri, request, request.headers[:content_type]) if enabled?
+        {:body => request, :content_type => request.headers[:content_type]}
+      end
+      
+      def read(uri)
+        key = generate_key(uri)
+        return cache[key]
+      end
+      
+      def write(uri, body, content_type)
+        store = {:body => body, :content_type => content_type}
+        self.cache.store(generate_key(uri), store, :expire_in => Smoke.config[:cache][:expiry])
+      end
+  
+      def generate_key(key)
+        Digest::MD5.hexdigest(key.to_s)
+      end
+    end
+  end
+end

data/lib/smoke/origin.rb

@@ -0,0 +1,251 @@
+module Smoke
+  class Origin
+    attr_reader :items
+    attr_accessor :name
+    
+    def initialize(name, &block)
+      raise StandardError, "Sources must have a name" unless name
+      @name = name
+      @items, @prepare, @transformation = [], [], []
+
+      activate!
+      instance_eval(&block) if block_given?
+    end
+    
+    # Output your items in a range of formats (:ruby, :json and :yaml currently)
+    # Ruby is the default format and will automagically yielded from your source
+    #
+    # Usage
+    # 
+    #   output(:json)
+    #   => "[{title: \"Ray\"}, {title: \"Peace\"}]"
+    def output(type = :ruby)
+      prepare!
+      dispatch if respond_to? :dispatch
+      
+      case type
+      when :json
+        return ::JSON.generate(@items)
+      when :yaml
+        return YAML.dump(@items)
+      else
+        return @items
+      end      
+    end
+    
+    def items=(response) # :nodoc:
+      @items = [(@path.nil?) ? response : drill(response, *@path)]
+      symbolize_keys!
+      transform!
+    end
+    
+    # Path allows you to traverse the tree of a the items returned to 
+    # only give you access to what you're interested in.
+    #
+    # Usage:
+    # path :down, :to, :the, :data
+    # 
+    # Will traverse through a tree as follows:
+    #
+    #   {
+    #     :down => {
+    #       :to => {
+    #         :the => {
+    #           :data => []
+    #         }
+    #       }
+    #     }
+    #   }
+    #
+    # You will need to help smoke find an array of 
+    # items that you're interested in. 
+    def path(*path)
+      @path = path
+    end
+    
+    # Transform each item
+    #
+    # Usage:
+    #   emit do
+    #     rename(:href => :link)
+    #   end 
+    def emit(&block)
+      raise ArgumentError, "requires a block" unless block_given?
+      @transformation << block
+    end
+    
+    # Transform must be used inside an `emit` block.
+    # It can be used to alter named keys within the item set
+    #
+    # Usage:
+    #     emit do
+    #       transform :name, :description do |name|
+    #         name.gsub(/\302/, "")
+    #       end
+    #     end
+    #
+    # In quasi-english: The result of the block is returned and set to each 
+    # of the :name and :description keys in the result set.
+    def transform(*keys)
+      raise ArgumentError, "requires a block" unless block_given?
+      keys.each do |key|
+        items.each do |item|
+          item[key] = yield(item[key]) || item[key]
+        end
+      end
+    end
+    
+    # Insert must be used inside an `emit` block.
+    # It can be used to insert named keys to all items within the result set
+    # 
+    # Usage: 
+    #   
+    #   emit do
+    #     insert :source, "twitter"
+    #   end
+    # 
+    # Once output is called, all items will contain a key of :source with
+    # a value of "twitter"
+    def insert(key, value)
+      @items.each do |item|
+        item[key] = value
+      end
+    end
+
+    # Prepare is used when you'd like to provision for 
+    # arguments / variables to be set after the source definition.
+    # Eg, create a source definition for twitter, omitting the "username".
+    # Set the username using chaining later.
+    # 
+    # Usage:
+    #   # Definition
+    #   Smoke.feed :twitter do
+    #     prepare do
+    #       url "http://twitter.com/#{username}/rss"
+    #     end
+    #   end
+    #   
+    #   # End use
+    #   Smoke[:twitter].username(:benschwarz).output
+    def prepare(&block)
+      raise ArgumentError, "requires a block" unless block_given?
+      @prepare << block
+    end
+    
+    def method_missing(symbol, *args, &block)
+      ivar = "@#{symbol}"
+      
+      if args.empty?
+        return instance_variable_get(ivar) || super  
+      else
+        instance_variable_set(ivar, args.pop)
+      end
+
+      return self
+    end
+
+    # Re-sort items by a particular key
+    # Sort must be used inside an `emit` block.
+    def sort(key)
+      @items = @items.sort_by{|i| i[key] }
+    rescue NoMethodError => e
+      Smoke.log.info "You're trying to sort by \"#{key}\" but it does not exist in your item set"
+    end
+    
+    # Reverse the order of the items
+    # 
+    # Usage
+    #   Smoke[:ruby].output
+    # Returns [{:header => "Platypus"}, {:header => "Kangaroo"}]
+    #   Smoke.yql(:ruby) do
+    #     ... Define your source
+    #     emit do
+    #       reverse
+    #     end
+    #   end
+    # Returns [{:header => "Kangaroo"}, {:header => "Platypus"}]
+    # Reverse must be used inside an `emit` block.
+    def reverse
+      @items.reverse!
+    end
+
+    # Keep items that match the regex
+    # 
+    # Usage (block, during initialization):
+    #     Smoke.yql(:ruby) do
+    #       ...
+    #       emit do
+    #         keep(:title, /tuesday/i)
+    #       end
+    #     end
+    # Keep must be used inside an `emit` block.
+    def keep(key, matcher)
+      @items.reject! {|i| (i[key] =~ matcher) ? false : true }
+    end
+
+    # Discard items that do not match the regex
+    # 
+    # Usage (block, during initialization):
+    #     Smoke.yql(:ruby) do
+    #       ...
+    #       emit do
+    #         discard(:title, /tuesday/i)
+    #       end
+    #     end
+    # Discard must be used inside an `emit` block.
+    def discard(key, matcher)
+      @items.reject! {|i| (i[key] =~ matcher) ? true : false }
+    end
+
+    # Rename one or many keys at a time
+    #
+    # Usage
+    #   # Renames all items with a key of href to link
+    #   rename(:href => :link)
+    # or
+    #   rename(:href => :link, :description => :excerpt)
+    # Rename must be used inside an `emit` block.
+    def rename(*args)
+      @items.each {|item| item.rename(*args) }
+    end
+
+    # Truncate your result set to this many objects
+    #
+    # Usage
+    #   Smoke.yql(:ruby) do
+    #     ...
+    #     truncate(3)
+    #   end
+    #   Smoke[:ruby].output
+    #   => [{title => "Canon"}, {:title => "Nikon"}, {:title => "Pentax"}]
+    # Truncate must be used inside an `emit` block.
+    def truncate(length)
+      @items = @items[0..(length - 1)]
+    end
+    
+    private
+    def prepare!
+      @prepare.each{|p| p.call } unless @prepare.nil?
+    end
+    
+    def transform!
+      @transformation.each{|t| t.call } unless @transformation.nil?
+    end
+    
+    def symbolize_keys!
+      @items = items.flatten.map{|i| i.symbolize_keys! } if items.respond_to? :flatten
+    end
+    
+    def drill(*path)
+      path.inject(nil) do |obj, pointer|
+        obj = obj.nil? ? pointer : obj[pointer]
+      end
+    end
+    
+    def activate!
+      Smoke.activate(@name, self)
+    end
+  end
+end
+
+Dir["#{File.dirname(__FILE__)}/source/*.rb"].each &method(:require)