jm-calais 0.0.13

data/CHANGELOG.markdown

@@ -0,0 +1,63 @@
+# Changes
+
+## 0.0.13
+
+* load path fix
+
+## 0.0.12
+
+* added relevances to Geographies
+* improved doc
+* removed jeweler dependency and simplified Rakefile
+* bumped rspec requirement
+
+## 0.0.11
+
+* simple fix for some rubies not liking DateTime.parse without including date
+* tests for SocialTags
+* typo fix: SocailTag != SocialTag
+
+## 0.0.10
+
+* community patch to expose SocialTags
+
+## 0.0.9
+
+* updates related to API changes
+* community patches to support bundler, support ruby 1.9
+
+## 0.0.8
+
+* community patches to use nokogiri
+
+## 0.0.7
+* verified 4.0 API
+* moved gem packaging to `jeweler` and documentation to `yard`
+
+## 0.0.6
+* fully implemented 3.1 API
+
+## 0.0.5
+* fixed error where classes weren't being required in the proper order on Ubuntu (reported by Jon Moses)
+* New things coming back from the API. Fixing in tests.
+
+## 0.0.4
+* changed dependency from `hpricot` to `libxml`
+* unicode fun
+* cleanup all around
+
+## 0.0.3
+* pluginized the library for Rails (thanks [pius](http://gitorious.org/projects/calais-au-rails))
+* added helper methods name entity types from a response
+
+## 0.0.2
+* cleanup in the specs
+* cleaner parsing
+* location of named entities
+* more data in relationships
+* moved Names and Relationships
+
+## 0.0.1
+* Access to OpenCalais's Enlighten action
+* Single method to process a document
+    * Get relationships and names from a document

data/Gemfile

@@ -0,0 +1,3 @@
+source :gemcutter
+
+gemspec

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 Abhay Kumar info@opensynapse.net
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,55 @@
+# Calais #
+A Ruby interface to the [Open Calais Web Service](http://opencalais.com)
+
+## About this Fork ##
+Forked from https://github.com/abhay/calais version ~> 0.0.13
+to fix issues caused by the depreciation of iconv in ruby > 1.9.3
+
+## Features ##
+* Accepts documents in text/plain, text/xml and text/html format.
+* Basic access to the Open Calais API's Enlighten action.
+    * Output is RDF representation of input document.
+* Single function ability to extract names, entities and geographies from given text.
+  
+## Synopsis ##
+
+This is a very basic wrapper to the Open Calais API. It uses the POST endpoint and currently supports the Enlighten action. Here's a simple call:
+
+    Calais.enlighten(
+        :content => "The government of the United Kingdom has given corporations like fast food chain McDonald's the right to award high school qualifications to employees who complete a company training program.",
+        :content_type => :raw,
+        :license_id => 'your license id'
+    )
+
+This is the easiest way to get the RDF-formated response from the OpenCalais service.
+
+If you want to do something more fun like getting all sorts of fun information about a document, you can try this:
+
+    Calais.process_document(
+        :content => "The government of the United Kingdom has given corporations like fast food chain McDonald's the right to award high school qualifications to employees who complete a company training program.",
+        :content_type => :raw,
+        :license_id => 'your license id'
+    )
+
+This will return an object containing information extracted from the RDF response.
+
+## Requirements ##
+
+* [Ruby 1.8.5 or better](http://ruby-lang.org)
+* [nokogiri](http://nokogiri.rubyforge.org/nokogiri/), [libxml2](http://xmlsoft.org/), [libxslt](http://xmlsoft.org/xslt/)
+* [curb](http://curb.rubyforge.org/), [libcurl](http://curl.haxx.se/)
+* [json](http://json.rubyforge.org/)
+
+## Install ##
+
+You can install the Calais gem via Rubygems (`gem install calais`) or by building from source.
+
+## Authors ##
+
+* [Abhay Kumar](http://opensynapse.net) 
+
+## Acknowledgements ##
+
+* [Paul Legato](http://www.economaton.com/): Help all around with the new response processor and implementation of the 3.1 API.
+* [Ryan Ong](http://www.ryanong.net/)
+* [Juan Antonio Chavez](https://github.com/TheNaoX): Geographies relevance

data/Rakefile

@@ -0,0 +1,36 @@
+# -*- ruby -*-
+
+require 'rake'
+require 'rake/clean'
+
+require './lib/calais.rb'
+
+begin
+	require 'rspec/core/rake_task'
+
+	RSpec::Core::RakeTask.new(:spec)
+
+  task :default => :spec
+rescue LoadError
+  puts "RSpec, or one of its dependencies, is not available. Please install it."
+  exit(1)
+end
+
+begin
+  require 'yard'
+  require 'yard/rake/yardoc_task'
+
+  YARD::Rake::YardocTask.new do |t|
+    t.options = ["--verbose", "--markup=markdown", "--files=CHANGELOG.markdown,MIT-LICENSE"]
+  end
+  
+  task :rdoc => :yardoc
+  
+  CLOBBER.include 'doc'
+  CLOBBER.include '.yardoc'
+rescue LoadError
+  puts "Yard, or one of its dependencies is not available. Please install it."
+  exit(1)
+end
+
+# vim: syntax=Ruby

data/lib/calais/client.rb

@@ -0,0 +1,115 @@
+module Calais
+  class Client
+    # base attributes of the call
+    attr_accessor :content
+    attr_accessor :license_id
+
+    # processing directives
+    attr_accessor :content_type, :output_format, :reltag_base_url, :calculate_relevance, :omit_outputting_original_text
+    attr_accessor :store_rdf, :metadata_enables, :metadata_discards
+
+    # user directives
+    attr_accessor :allow_distribution, :allow_search, :external_id, :submitter
+
+    attr_accessor :external_metadata
+
+    attr_accessor :use_beta
+
+    def initialize(options={}, &block)
+      options.each {|k,v| send("#{k}=", v)}
+      yield(self) if block_given?
+    end
+
+    def enlighten
+      post_args = {
+        "licenseID" => @license_id,
+        "content" => RUBY_VERSION.to_f < 1.9 ?
+          Iconv.iconv('UTF-8//IGNORE', 'UTF-8',  "#{@content} ").first[0..-2] :
+          "#{@content} ".encode(Encoding::UTF_8, :invalid => :replace, :undef => :replace, :replace => '')[0 .. -2],
+        "paramsXML" => params_xml
+      }
+
+      do_request(post_args)
+    end
+
+    def params_xml
+      check_params
+      document = Nokogiri::XML::Document.new
+      
+      params_node = Nokogiri::XML::Node.new('c:params', document)
+      params_node['xmlns:c'] = 'http://s.opencalais.com/1/pred/'
+      params_node['xmlns:rdf'] = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'
+      
+      processing_node = Nokogiri::XML::Node.new('c:processingDirectives', document)
+      processing_node['c:contentType'] = AVAILABLE_CONTENT_TYPES[@content_type] if @content_type
+      processing_node['c:outputFormat'] = AVAILABLE_OUTPUT_FORMATS[@output_format] if @output_format
+      processing_node['c:calculateRelevanceScore'] = 'false' if @calculate_relevance == false
+      processing_node['c:reltagBaseURL'] = @reltag_base_url.to_s if @reltag_base_url
+      
+      processing_node['c:enableMetadataType'] = @metadata_enables.join(',') unless @metadata_enables.empty?
+      processing_node['c:docRDFaccessible'] = @store_rdf if @store_rdf
+      processing_node['c:discardMetadata'] = @metadata_discards.join(';') unless @metadata_discards.empty?
+      processing_node['c:omitOutputtingOriginalText'] = 'true' if @omit_outputting_original_text
+      
+      user_node = Nokogiri::XML::Node.new('c:userDirectives', document)
+      user_node['c:allowDistribution'] = @allow_distribution.to_s unless @allow_distribution.nil?
+      user_node['c:allowSearch'] = @allow_search.to_s unless @allow_search.nil?
+      user_node['c:externalID'] = @external_id.to_s if @external_id
+      user_node['c:submitter'] = @submitter.to_s if @submitter
+      
+      params_node << processing_node
+      params_node << user_node
+      
+      if @external_metadata
+        external_node = Nokogiri::XML::Node.new('c:externalMetadata', document)
+        external_node << @external_metadata
+        params_node << external_node
+      end
+      
+      params_node.to_xml(:indent => 2)
+    end
+
+    def url
+      @url ||= URI.parse(calais_endpoint)
+    end
+
+    private
+      def check_params
+        raise 'missing content' if @content.nil? || @content.empty?
+
+        content_length = @content.length
+        raise 'content is too small' if content_length < MIN_CONTENT_SIZE
+        raise 'content is too large' if content_length > MAX_CONTENT_SIZE
+
+        raise 'missing license id' if @license_id.nil? || @license_id.empty?
+
+        raise 'unknown content type' unless AVAILABLE_CONTENT_TYPES.keys.include?(@content_type) if @content_type
+        raise 'unknown output format' unless AVAILABLE_OUTPUT_FORMATS.keys.include?(@output_format) if @output_format
+
+        %w[calculate_relevance store_rdf allow_distribution allow_search].each do |variable|
+          value = self.send(variable)
+          unless NilClass === value || TrueClass === value || FalseClass === value
+            raise "expected a boolean value for #{variable} but got #{value}"
+          end
+        end
+
+        @metadata_enables ||= []
+        unknown_enables = Set.new(@metadata_enables) - KNOWN_ENABLES
+        raise "unknown metadata enables: #{unknown_enables.to_a.inspect}" unless unknown_enables.empty?
+
+        @metadata_discards ||= []
+        unknown_discards = Set.new(@metadata_discards) - KNOWN_DISCARDS
+        raise "unknown metadata discards: #{unknown_discards.to_a.inspect}" unless unknown_discards.empty?
+      end
+
+      def do_request(post_fields)
+        @request ||= Net::HTTP::Post.new(url.path)
+        @request.set_form_data(post_fields)
+        Net::HTTP.new(url.host, url.port).start {|http| http.request(@request)}.body
+      end
+
+      def calais_endpoint
+         @use_beta ? BETA_REST_ENDPOINT : REST_ENDPOINT
+      end
+  end
+end

data/lib/calais/error.rb

@@ -0,0 +1,3 @@
+class Calais::Error < StandardError
+  
+end

data/lib/calais/response.rb

@@ -0,0 +1,220 @@
+module Calais
+  class Response
+    MATCHERS = {
+      :docinfo => 'DocInfo',
+      :docinfometa => 'DocInfoMeta',
+      :defaultlangid => 'DefaultLangId',
+      :doccat => 'DocCat',
+      :entities => 'type/em/e',
+      :relations => 'type/em/r',
+      :geographies => 'type/er',
+      :instances => 'type/sys/InstanceInfo',
+      :relevances => 'type/sys/RelevanceInfo',
+    }
+
+    attr_accessor :submitter_code, :signature, :language, :submission_date, :request_id, :doc_title, :doc_date
+    attr_accessor :hashes, :entities, :relations, :geographies, :categories, :socialtags, :relevances
+
+    def initialize(rdf_string)
+      @raw_response = rdf_string
+
+      @hashes = []
+      @entities = []
+      @relations = []
+      @geographies = []
+      @relevances = {} # key = String hash, val = Float relevance
+      @categories = []
+      @socialtags = []
+      
+      extract_data
+    end
+
+    class Entity
+      attr_accessor :calais_hash, :type, :attributes, :relevance, :instances
+    end
+
+    class Relation
+      attr_accessor :calais_hash, :type, :attributes, :instances
+    end
+
+    class Geography
+      attr_accessor :name, :calais_hash, :attributes, :relevance
+    end
+
+    class Category
+      attr_accessor :name, :score
+    end
+
+    class SocialTag
+      attr_accessor :name, :importance
+    end
+    
+    class Instance
+      attr_accessor :prefix, :exact, :suffix, :offset, :length
+
+      # Makes a new Instance object from an appropriate Nokogiri::XML::Node.
+      def self.from_node(node)
+        instance = self.new
+        instance.prefix = node.xpath("c:prefix[1]").first.content
+        instance.exact  = node.xpath("c:exact[1]").first.content
+        instance.suffix = node.xpath("c:suffix[1]").first.content
+        instance.offset = node.xpath("c:offset[1]").first.content.to_i
+        instance.length = node.xpath("c:length[1]").first.content.to_i
+
+        instance
+      end
+    end
+
+    class CalaisHash
+      attr_accessor :value
+
+      def self.find_or_create(hash, hashes)
+        if !selected = hashes.select {|h| h.value == hash }.first
+          selected = self.new
+          selected.value = hash
+          hashes << selected
+        end
+
+        selected
+      end
+    end
+
+    private
+      def extract_data
+        doc = Nokogiri::XML(@raw_response)
+
+        if doc.root.xpath("/Error[1]").first
+          raise Calais::Error, doc.root.xpath("/Error/Exception").first.content
+        end        
+
+        doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:docinfometa]}')]/..").each do |node|
+          @language = node['language']
+          @submission_date =  DateTime.parse node['submissionDate']
+
+          attributes = extract_attributes(node.xpath("*[contains(name(), 'c:')]"))
+
+          @signature = attributes.delete('signature')
+          @submitter_code = attributes.delete('submitterCode')
+
+          node.remove
+        end
+
+        doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:docinfo]}')]/..").each do |node|
+          @request_id = node['calaisRequestID']
+
+          attributes = extract_attributes(node.xpath("*[contains(name(), 'c:')]"))
+
+          @doc_title = attributes.delete('docTitle')
+          @doc_date = Date.parse(attributes.delete('docDate')) 
+
+          node.remove
+        end
+
+        @socialtags = doc.root.xpath("rdf:Description/c:socialtag/..").map do |node|
+          tag = SocialTag.new
+          tag.name = node.xpath("c:name[1]").first.content
+          tag.importance = node.xpath("c:importance[1]").first.content.to_i
+          
+          node.remove if node.xpath("c:categoryName[1]").first.nil?
+          
+          tag
+        end
+
+        @categories = doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:doccat]}')]/..").map do |node|
+          category = Category.new
+          category.name = node.xpath("c:categoryName[1]").first.content
+          score = node.xpath("c:score[1]").first
+          category.score = score.content.to_f unless score.nil?
+
+          node.remove
+          category
+        end
+
+        @relevances = doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:relevances]}')]/..").inject({}) do |acc, node|
+          subject_hash = node.xpath("c:subject[1]").first[:resource].split('/')[-1]
+          acc[subject_hash] = node.xpath("c:relevance[1]").first.content.to_f
+
+          node.remove
+          acc
+        end
+
+        @entities = doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:entities]}')]/..").map do |node|
+          extracted_hash = node['about'].split('/')[-1] rescue nil
+
+          entity = Entity.new
+          entity.calais_hash = CalaisHash.find_or_create(extracted_hash, @hashes)
+          entity.type = extract_type(node)
+          entity.attributes = extract_attributes(node.xpath("*[contains(name(), 'c:')]"))
+
+          entity.relevance = @relevances[extracted_hash]
+          entity.instances = extract_instances(doc, extracted_hash)
+
+          node.remove
+          entity
+        end
+
+        @relations = doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:relations]}')]/..").map do |node|
+          extracted_hash = node['about'].split('/')[-1] rescue nil
+
+          relation = Relation.new
+          relation.calais_hash = CalaisHash.find_or_create(extracted_hash, @hashes)
+          relation.type = extract_type(node)
+          relation.attributes = extract_attributes(node.xpath("*[contains(name(), 'c:')]"))
+          relation.instances = extract_instances(doc, extracted_hash)
+
+          node.remove
+          relation
+        end
+
+        @geographies = doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:geographies]}')]/..").map do |node|
+          attributes = extract_attributes(node.xpath("*[contains(name(), 'c:')]"))
+
+          geography = Geography.new
+          geography.name = attributes.delete('name')
+          geography.calais_hash = attributes.delete('subject')
+          geography.attributes = attributes
+          geography.relevance = extract_relevance(geography.calais_hash.value)
+
+          node.remove
+          geography
+        end
+
+        doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:defaultlangid]}')]/..").each { |node| node.remove }
+        doc.root.xpath("./*").each { |node| node.remove }
+
+        return
+      end
+
+      def extract_instances(doc, hash)
+        doc.root.xpath("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:instances]}')]/..").select do |instance_node|
+          instance_node.xpath("c:subject[1]").first[:resource].split("/")[-1] == hash
+        end.map do |instance_node|
+          instance = Instance.from_node(instance_node)
+          instance_node.remove
+
+          instance
+        end
+      end
+
+      def extract_type(node)
+        node.xpath("*[name()='rdf:type']")[0]['resource'].split('/')[-1]
+      rescue
+        nil
+      end
+
+      def extract_attributes(nodes)
+        nodes.inject({}) do |hsh, node|
+          value = if node['resource']
+              extracted_hash = node['resource'].split('/')[-1] rescue nil
+              CalaisHash.find_or_create(extracted_hash, @hashes)
+            else
+              node.content
+            end
+          hsh.merge(node.name => value)
+        end
+      end
+      def extract_relevance(value)
+        return @relevances[value]
+      end
+  end
+end