mida 0.2.0 → 0.3.0

data/CHANGELOG.rdoc

@@ -1,7 +1,23 @@
+== 0.3.0 (29th June 2011)
+* Merge +VocabularyDesc+ into +Vocabulary+
+* Vocabularies are now auto registered using +inherited+ hook
+* Removed vocabulary from <tt>Item#to_h</tt>
+* Deprecate +types+ to describe a Vocabulary property if favour of +extract+
+* Add +DataType+ so can use <tt>DataType::Text</tt> instead of +String+ for a
+  type
+* Add various <tt>DataType</tt>s: +Boolean+, +Float+, +Integer+, +Number+,
+  +ISO8601Date+, +Text+
+* Add Bundler support
+* Properties marked as <tt>has_one</tt> now output a single value instead of
+  an +Array+
+* <tt>Document#search</tt> now only uses a +Regexp+ to search with
+* +Document+ now includes +Enumerable+ Mixin
+
 == 0.2.0 (3rd May 2011)
 * Add ability to describe and conform to vocabularies
-* Rename Mida::Property to Mida::Itemprop to better reflect use
-* Make some of the Mida::Itemprop class methods private
+* Rename <tt>Mida::Property</tt> to <tt>Mida::Itemprop</tt> to better reflect
+  use
+* Make some of the <tt>Mida::Itemprop</tt> class methods private
 
 == 0.1.3 (18th April 2011)
 * Ensure itemprops are parsed properly if containing non-microdata elements

data/README.rdoc

@@ -1,6 +1,7 @@
 = Mida
 
-* {Mida Project Page}[https://github.com/LawrenceWoodman/mida]
+* {Mida Project Page}[http://lawrencewoodman.github.com/mida]
+* {Mida Github Repository}[https://github.com/LawrenceWoodman/mida]
 * {Mida Bug Tracker}[https://github.com/LawrenceWoodman/mida/issues]
 
 == Description
@@ -43,8 +44,8 @@ To return all the +Items+ that use one of Google's Review vocabularies:
   doc.search(%r{http://data-vocabulary\.org.*?review.*?}i)
 
 === Inspecting an +Item+
-Each +Item+ is a <tt>Mida::Item</tt> instance and has three main methods of
-interest, +type+, +properties+ and +id+.
+Each +Item+ is a <tt>Mida::Item</tt> instance and has four main methods of
+interest: +type+, +vocabulary+, +properties+ and +id+.
 
 To find out the +itemtype+ of the +Item+:
   puts doc.items.first.type
@@ -60,21 +61,31 @@ To see the +properties+ of the +Item+:
 
 === Working with Vocabularies
 Mida allows you to define vocabularies, so that input data can be constrained to match
-expected patterns.  By default a generic vocabulary (<tt>Mida::Vocabulary::Generic</tt>)
+expected patterns.  By default a generic vocabulary (<tt>Mida::GenericVocabulary</tt>)
 is registered which will match against any +itemtype+ with any number of properties.
 
-If you want to specify a vocabulary you create a class derived from <tt>Mida::VocabularyDesc</tt>
-and use +itemtype+, +has_one+, +has_many+ and +types+ to describe the vocabulary.
+If you want to specify a vocabulary you create a class derived from <tt>Mida::Vocabulary</tt>
+and use +itemtype+, +has_one+, +has_many+ and +extract+ to describe the vocabulary.
 
 As an example the following describes a subset of Google's Review vocabulary:
-  class Review < Mida::VocabularyDesc
-    itemtype %r{http://data-vocabulary.org/review}
+
+  class Rating < Mida::Vocabulary
+    itemtype %r{http://data-vocabulary.org/rating}i
+    has_one 'best'
+    has_one 'worst'
+    has_one 'value'
+  end
+
+  class Review < Mida::Vocabulary
+    itemtype %r{http://data-vocabulary.org/review}i
     has_one 'itemreviewed'
-    has_one 'rating'
+    has_one 'rating' do
+      extract Rating, Mida::DataType::Text
+    end
   end
 
-To register the above Vocabulary use:
-  Mida::Vocabulary.register(Review)
+When you create a subclass of <tt>Mida::Vocabulary</tt> it automatically
+registers the Vocabulary.
 
 Now if Mida is parsing some input and manages to match against the +Review+ +itemtype+, it
 will only allow the specified properties and will reject any that don't have the correct number.  It

data/Rakefile

@@ -6,10 +6,10 @@ spec = Gem::Specification.new do |s|
   s.name        = "mida"
   s.summary     = "A Microdata parser/extractor library"
   s.description = "A Microdata parser and extractor library, based on the latest published version of the Microdata Specification, dated 5th April 2011."
-  s.version     = "0.2.0"
+  s.version     = "0.3.0"
   s.author      = "Lawrence Woodman"
   s.email       = "lwoodman@vlifesystems.com"
-  s.homepage    = %q{http://github.com/LawrenceWoodman/mida}
+  s.homepage    = %q{http://lawrencewoodman.github.com/mida/}
   s.platform    = Gem::Platform::RUBY
   s.required_ruby_version = '>=1.9'
   s.files       = Dir['lib/**/*.rb'] + Dir['spec/**/*.rb'] + Dir['*.rdoc'] + Dir['Rakefile']

data/lib/mida.rb

@@ -6,4 +6,4 @@ module Mida
 
 end
 
-require_relative 'mida/vocabulary/generic'
+require 'mida/genericvocabulary'

data/lib/mida/datatype.rb

@@ -0,0 +1,15 @@
+module Mida
+  # Module to hold the various data types.
+  # Each DataType should be a module containing the class method: +extract+
+  # which returns the value extracted or raises an +ArgumentError+ exception
+  # if input value is not valid.
+  module DataType
+  end
+end
+
+require 'mida/datatype/boolean'
+require 'mida/datatype/float'
+require 'mida/datatype/integer'
+require 'mida/datatype/iso8601date'
+require 'mida/datatype/number'
+require 'mida/datatype/text'

data/lib/mida/datatype/boolean.rb

@@ -0,0 +1,18 @@
+module Mida
+  module DataType
+
+    # Boolean data type
+    module Boolean
+
+      # Returns the +value+ as a boolean
+      # or raises ArgumentError if not valid
+      def self.extract(value)
+        case value.downcase
+        when 'true' then true
+        when 'false' then false
+        else raise ArgumentError, 'Invalid value'
+        end
+      end
+    end
+  end
+end

data/lib/mida/datatype/float.rb

@@ -0,0 +1,15 @@
+module Mida
+  module DataType
+
+    # Float data type
+    module Float
+
+      # Returns the +value+ as a floating point number
+      # Relies on +Float+ to raise +ArgumentError+ if not valid
+      def self.extract(value)
+        Float(value)
+      end
+
+    end
+  end
+end

data/lib/mida/datatype/integer.rb

@@ -0,0 +1,15 @@
+module Mida
+  module DataType
+
+    # Integer data type
+    module Integer
+
+      # Returns the +value+ as an integer
+      # Relies on +Integer+ to raise +ArgumentError+ if not valid
+      def self.extract(value)
+        Integer(value)
+      end
+
+    end
+  end
+end

data/lib/mida/datatype/iso8601date.rb

@@ -0,0 +1,17 @@
+require 'date'
+
+module Mida
+  module DataType
+
+    # ISO 8601 Date data type
+    module ISO8601Date
+
+      # Returns the +value+ as a +DateTime+ instance
+      # Relies on <tt>DateTime#iso8601</tt> to raise
+      # +ArgumentError+ if not valid
+      def self.extract(value)
+        DateTime.iso8601(value)
+      end
+    end
+  end
+end

data/lib/mida/datatype/number.rb

@@ -0,0 +1,15 @@
+module Mida
+  module DataType
+
+    # Number data type
+    module Number
+
+      # Returns the +value+ as a number
+      # Relies on +Float+ to raise +ArgumentError+ if not valid
+      def self.extract(value)
+        Float(value)
+      end
+
+    end
+  end
+end

data/lib/mida/datatype/text.rb

@@ -0,0 +1,13 @@
+module Mida
+  module DataType
+
+    # Text data type
+    module Text
+
+      # Returns the value extracted
+      def self.extract(value)
+        value
+      end
+    end
+  end
+end

data/lib/mida/document.rb

@@ -4,6 +4,7 @@ module Mida
 
   # Class that holds the extracted Microdata
   class Document
+    include Enumerable
 
     # An Array of Mida::Item objects.  These are all top-level
     # and hence not properties of other Items
@@ -20,25 +21,27 @@ module Mida
       @items = extract_items
     end
 
-    # Returns an array of matching Mida::Item objects
-    #
-    # [vocabulary] A regexp to match the item types against
-    #              or a Class derived from Mida::VocabularyDesc
-    #              to match against
-    def search(vocabulary, items=@items)
-      found_items = []
-      regexp_passed = vocabulary.kind_of?(Regexp)
-      regexp = if regexp_passed then vocabulary else vocabulary.itemtype end
+    # Implements method for Enumerable
+    def each
+      @items.each {|item| yield(item)}
+    end
 
-      items.each do |item|
+    # Returns an array of matching <tt>Mida::Item</tt> objects
+    #
+    # This drills down through each +Item+ to find match items
+    #
+    # [itemtype] A regexp to match the item types against
+    # [items]    An array of items to search.  If no argument supplied, will
+    #            search through all items in the document.
+    def search(itemtype, items=@items)
+      items.each_with_object([]) do |item, found_items|
         # Allows matching against empty string, otherwise couldn't match
         # as item.type can be nil
-        if (item.type.nil? && "" =~ regexp) || (item.type =~ regexp)
+        if (item.type.nil? && "" =~ itemtype) || (item.type =~ itemtype)
           found_items << item
         end
-        found_items += search_values(item.properties.values, regexp)
+        found_items.concat(search_values(item.properties.values, itemtype))
       end
-      found_items
     end
 
   private
@@ -47,18 +50,17 @@ module Mida
       return nil unless itemscopes
 
       itemscopes.collect do |itemscope|
-        Item.new(itemscope, @page_url)
+        itemscope = Itemscope.new(itemscope, @page_url)
+        Item.new(itemscope)
       end
     end
 
     def search_values(values, vocabulary)
-      items = []
-      values.each do |value|
-        if value.is_a?(Mida::Item) then items += search(vocabulary, [value])
-        elsif value.is_a?(Array) then items += search_values(value, vocabulary)
+      values.each_with_object([]) do |value, items|
+        if value.is_a?(Item) then items.concat(search(vocabulary, [value]))
+        elsif value.is_a?(Array) then items.concat(search_values(value, vocabulary))
         end
       end
-      items
     end
 
   end

data/lib/mida/genericvocabulary.rb

@@ -0,0 +1,13 @@
+require 'mida/vocabulary'
+
+module Mida
+
+  # A Generic vocabulary that will match against anything
+  class GenericVocabulary < Mida::Vocabulary
+    itemtype %r{}
+    has_many :any do
+      extract :any
+    end
+  end
+
+end

data/lib/mida/item.rb

@@ -1,9 +1,11 @@
 require 'nokogiri'
+require 'mida'
 
 module Mida
 
-  # Class that holds each item/itemscope
+  # Class that holds a validated item
   class Item
+
     # The vocabulary used to interpret this item
     attr_reader :vocabulary
 
@@ -18,29 +20,26 @@ module Mida
     # or <tt>Mida::Item</tt> instances
     attr_reader :properties
 
-    # Create a new Item object
+    # Create a new Item object from an +Itemscope+ and validates
+    # its +properties+
     #
-    # [itemscope] The itemscope that you want to parse.
-    # [page_url] The url of target used for form absolute url.
-    def initialize(itemscope, page_url=nil)
-      @itemscope, @page_url = itemscope, page_url
-      @type, @id = extract_attribute('itemtype'), extract_attribute('itemid')
+    # [itemscope] The itemscope that has been parsed by +Itemscope+
+    def initialize(itemscope)
+      @type = itemscope.type
+      @id = itemscope.id
       @vocabulary = Mida::Vocabulary.find(@type)
-      @properties = {}
-      add_itemref_properties
-      parse_elements(extract_elements(@itemscope))
+      @properties = itemscope.properties
       validate_properties
     end
 
     # Return a Hash representation
     # of the form:
-    #   { vocabulary: 'http://example.com/vocab/review',
-    #     type: 'The item type',
+    #   { type: 'http://example.com/vocab/review',
     #     id: 'urn:isbn:1-934356-08-5',
     #     properties: {'a name' => 'avalue' }
     #   }
     def to_h
-      {vocabulary: @vocabulary, type: @type, id: @id, properties: properties_to_h(@properties)}
+      {type: @type, id: @id, properties: properties_to_h(@properties)}
     end
 
     def to_s
@@ -58,63 +57,101 @@ module Mida
     def validate_properties
       @properties =
       @properties.each_with_object({}) do |(property, values), hash|
-        if valid_property?(property, values)
-          hash[property] = valid_values(property, values)
-        end
+        valid_values = validate_values(property, values)
+        hash[property] = valid_values unless valid_values.nil?
       end
     end
 
-    # Return whether the number of values conforms to the spec
-    def valid_num_values?(property, values)
-      return false unless @vocabulary.prop_spec.has_key?(property)
-      property_spec = @vocabulary.prop_spec[property]
-      (property_spec[:num] == :many ||
-        (property_spec[:num] == :one && values.length == 1))
+    # Return whether the number of values conforms to +num+
+    def valid_num_values?(num, values)
+      num == :many || (num == :one && values.length == 1)
     end
 
+    # Return whether this property name is valid
     def valid_property?(property, values)
-      [property, :any].any? {|prop| valid_num_values?(prop, values)}
+      [property, :any].any? do |prop|
+        @vocabulary.properties.has_key?(prop)
+      end
     end
 
-    def valid_values(property, values)
-      prop_types = if @vocabulary.prop_spec.has_key?(property)
-        @vocabulary.prop_spec[property][:types]
+    # Return valid values, converted to the correct +DataType+
+    # or +Item+ and number if necessary
+    def validate_values(property, values)
+      return nil unless valid_property?(property, values)
+      prop_num = property_number(property)
+      return nil unless valid_num_values?(prop_num, values)
+      prop_types = property_types(property)
+
+      valid_values = values.each_with_object([]) do |value, valid_values|
+        new_value = validate_value(prop_types, value)
+        valid_values << new_value unless new_value.nil?
+      end
+
+      # Convert property to correct number
+      prop_num == :many ? valid_values : valid_values.first
+    end
+
+    # Returns value converted to correct +DataType+ or +Item+
+    # or +nil+ if not valid
+    def validate_value(prop_types, value)
+      if is_itemscope?(value)
+        valid_itemtype?(prop_types, value.type) ? Item.new(value) : nil
+      elsif (extract_value = datatype_extract(prop_types, value))
+        extract_value
+      elsif prop_types.include?(:any)
+        value
       else
-        @vocabulary.prop_spec[:any][:types]
+        nil
       end
+    end
 
-      values.select {|value| valid_type(prop_types, value) }
+    # Return the correct type for this property
+    def property_types(property)
+      if @vocabulary.properties.has_key?(property)
+        @vocabulary.properties[property][:types]
+      else
+        @vocabulary.properties[:any][:types]
+      end
     end
 
-    def valid_type(prop_types, value)
-      if value.respond_to?(:vocabulary)
-        if prop_types.include?(value.vocabulary) || prop_types.include?(:any)
-          return true
-        end
-      elsif prop_types.include?(value.class) || prop_types.include?(:any)
-        return true
+    # Return the correct number for this property
+    def property_number(property)
+      if @vocabulary.properties.has_key?(property)
+        @vocabulary.properties[property][:num]
+      else
+        @vocabulary.properties[:any][:num]
       end
-      false
     end
 
-    def extract_attribute(attribute)
-      (value = @itemscope.attribute(attribute)) ? value.value : nil
+    def is_itemscope?(object)
+      object.kind_of?(Itemscope)
     end
 
-    def extract_elements(itemscope)
-      itemscope.search('./*')
+    # Returns whether the +itemtype+ is a valid type
+    def valid_itemtype?(valid_types, itemtype)
+      return true if valid_types.include?(:any)
+
+      valid_types.find do |type|
+        type.respond_to?(:itemtype) && type.itemtype =~ itemtype
+      end
     end
 
-    # Find an element with a matching id
-    def find_with_id(id)
-      @itemscope.search("//*[@id='#{id}']")
+    # Returns the extracted value or +nil+ if none of the datatypes
+    # could extract the +value+
+    def datatype_extract(valid_types, value)
+      valid_types.find do |type|
+        begin
+          return type.extract(value) if type.respond_to?(:extract)
+        rescue ArgumentError
+        end
+      end
+      nil
     end
 
     # The value as it should appear in to_h()
     def value_to_h(value)
-      case
-      when value.is_a?(Array) then value.collect {|element| value_to_h(element)}
-      when value.is_a?(Item) then value.to_h
+      if value.is_a?(Array) then value.collect {|element| value_to_h(element)}
+      elsif value.is_a?(Item) then value.to_h
       else value
       end
     end
@@ -125,31 +162,6 @@ module Mida
       end
     end
 
-    # Add any properties referred to by 'itemref'
-    def add_itemref_properties
-      itemref = extract_attribute('itemref')
-      if itemref
-        itemref.split.each {|id| parse_elements(find_with_id(id))}
-      end
-    end
-
-    def parse_elements(elements)
-      elements.each {|element| parse_element(element)}
-    end
-
-    def parse_element(element)
-      itemscope = element.attribute('itemscope')
-      itemprop = element.attribute('itemprop')
-      internal_elements = extract_elements(element)
-      add_itemprop(element) if itemscope || itemprop
-      parse_elements(internal_elements) if internal_elements && !itemscope
-    end
-
-    def add_itemprop(itemprop)
-      properties = Itemprop.parse(itemprop, @page_url)
-      properties.each { |name, value| (@properties[name] ||= []) << value }
-    end
-
   end
 
 end