calais 0.0.6 → 0.0.7

data/CHANGELOG.markdown

@@ -0,0 +1,33 @@
+# Changes
+
+## 0.0.7
+* verified 4.0 API
+* moved gem packaging to `jeweler` and documentation to `yard`
+
+## 0.0.6
+* fully implemented 3.1 API
+
+## 0.0.5
+* fixed error where classes weren't being required in the proper order on Ubuntu (reported by Jon Moses)
+* New things coming back from the API. Fixing in tests.
+
+## 0.0.4
+* changed dependency from `hpricot` to `libxml`
+* unicode fun
+* cleanup all around
+
+## 0.0.3
+* pluginized the library for Rails (thanks [pius](http://gitorious.org/projects/calais-au-rails))
+* added helper methods name entity types from a response
+
+## 0.0.2
+* cleanup in the specs
+* cleaner parsing
+* location of named entities
+* more data in relationships
+* moved Names and Relationships
+
+## 0.0.1
+* Access to OpenCalais's Enlighten action
+* Single method to process a document
+    * Get relationships and names from a document

data/README.markdown

@@ -0,0 +1,49 @@
+# Calais #
+A Ruby interface to the [Open Calais Web Service](http://opencalais.com)
+
+## Features ##
+* Accepts documents in text/plain, text/xml and text/html format.
+* Basic access to the Open Calais API's Enlighten action.
+    * Output is RDF representation of input document.
+* Single function ability to extract names, entities and geographies from given text.
+  
+## Synopsis ##
+
+This is a very basic wrapper to the Open Calais API. It uses the POST endpoint and currently supports the Enlighten action. Here's a simple call:
+
+    Calais.enlighten(
+        :content => "The government of the United Kingdom has given corporations like fast food chain McDonald's the right to award high school qualifications to employees who complete a company training program."
+        :content_type => :text, 
+        :license_id => 'your license id'
+    )
+
+This is the easiest way to get the RDF-formated response from the OpenCalais service.
+
+If you want to do something more fun like getting all sorts of fun information about a document, you can try this:
+
+    Calais.process_document(
+        :content => "The government of the United Kingdom has given corporations like fast food chain McDonald's the right to award high school qualifications to employees who complete a company training program.",
+        :content_type => :text,
+        :license_id => 'your license id'
+    )
+
+This will return an object containing information extracted from the RDF response.
+
+## Requirements ##
+
+* [Ruby 1.8.5 or better](http://ruby-lang.org)
+* [libxml-ruby](http://libxml.rubyforge.org/), [libxml2](http://xmlsoft.org/)
+* [curb](http://curb.rubyforge.org/), [libcurl](http://curl.haxx.se/)
+* [json](http://json.rubyforge.org/)
+
+## Install ##
+
+You can install the Calais gem via Rubygems (`gem install calais`) or by building from source.
+
+## Authors ##
+
+* [Abhay Kumar](http://opensynapse.net) 
+
+## Acknowledgements ##
+
+* [Paul Legato](http://www.economaton.com/): Help all around with the new response processor and implementation of the 3.1 API.

data/Rakefile

@@ -1,37 +1,97 @@
 # -*- ruby -*-
 
-require 'rubygems'
-require 'hoe'
-require 'spec/rake/spectask'
+require 'rake'
+require 'rake/clean'
 
 require './lib/calais.rb'
 
-Hoe.new('calais', Calais::VERSION) do |p|
-  p.rubyforge_name = 'calais'
-  p.author = 'Abhay Kumar'
-  p.summary = 'A Ruby interface to the Calais Web Service'
-  p.description = p.paragraphs_of('README.txt', 1..9).join("\n\n")
-  p.email = 'info@opensynapse.net'
-  p.url = 'http://calais.rubyforge.org'
-  p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
-  p.remote_rdoc_dir = ''
+begin
+  gem 'jeweler', '>= 1.0.1'
+  require 'jeweler'
+
+  Jeweler::Tasks.new do |s|
+    s.name = 'calais'
+    s.summary = 'A Ruby interface to the Calais Web Service'
+    s.email = 'info@opensynapse.net'
+    s.homepage = 'http://github.com/abhay/calais'
+    s.description = 'A Ruby interface to the Calais Web Service'
+    s.authors = ['Abhay Kumar']
+    s.files = FileList["[A-Z]*", "{bin,generators,lib,test}/**/*"]
+    s.rubyforge_project = 'calais'
+    s.add_dependency 'libxml-ruby', '>= 0.5.4'
+    s.add_dependency 'json', '>= 1.1.3'
+    s.add_dependency 'curb', '>= 0.1.4'
+  end
+rescue LoadError
+  puts "Jeweler, or one of its dependencies, is not available. Please install it."
+  exit(1)
 end
 
-desc "Run all specs"
-Spec::Rake::SpecTask.new do |t|
-  t.spec_files = FileList["spec/**/*_spec.rb"].sort
-  t.spec_opts = ["--options", "spec/spec.opts"]
+begin
+  require 'spec/rake/spectask'
+
+  desc "Run all specs"
+  Spec::Rake::SpecTask.new do |t|
+    t.spec_files = FileList["spec/**/*_spec.rb"].sort
+    t.spec_opts = ["--options", "spec/spec.opts"]
+  end
+
+  desc "Run all specs and get coverage statistics"
+  Spec::Rake::SpecTask.new('coverage') do |t|
+    t.spec_opts = ["--options", "spec/spec.opts"]
+    t.spec_files = FileList["spec/*_spec.rb"].sort
+    t.rcov_opts = ["--exclude", "spec", "--exclude", "gems"]
+    t.rcov = true
+  end
+
+  task :default => :spec
+rescue LoadError
+  puts "RSpec, or one of its dependencies, is not available. Please install it."
+  exit(1)
 end
 
-desc "Run all specs and get coverage statistics"
-Spec::Rake::SpecTask.new('coverage') do |t|
-  t.spec_opts = ["--options", "spec/spec.opts"]
-  t.spec_files = FileList["spec/*_spec.rb"].sort
-  t.rcov_opts = ["--exclude", "spec", "--exclude", "gems"]
-  t.rcov = true
+begin
+  require 'yard'
+  require 'yard/rake/yardoc_task'
+
+  YARD::Rake::YardocTask.new do |t|
+    t.options = ["--verbose", "--markup=markdown", "--files=CHANGELOG.markdown,MIT-LICENSE"]
+  end
+  
+  task :rdoc => :yardoc
+  
+  CLOBBER.include 'doc'
+  CLOBBER.include '.yardoc'
+rescue LoadError
+  puts "Yard, or one of its dependencies is not available. Please install it."
+  exit(1)
 end
 
-Rake::Task[:default].prerequisites.clear
-task :default => :spec
+begin
+  require 'rake/contrib/sshpublisher'
+  namespace :rubyforge do
+
+    desc "Release gem and RDoc documentation to RubyForge"
+    task :release => ["rubyforge:release:gem", "rubyforge:release:docs"]
+
+    namespace :release do
+      desc "Publish RDoc to RubyForge."
+      task :docs => [:yardoc] do
+        config = YAML.load(
+            File.read(File.expand_path('~/.rubyforge/user-config.yml'))
+        )
+
+        host = "#{config['username']}@rubyforge.org"
+        remote_dir = "/var/www/gforge-projects/calais/"
+        local_dir = 'doc'
+
+        Rake::SshDirPublisher.new(host, remote_dir, local_dir).upload
+      end
+    end
+  end
+rescue LoadError
+  puts "Rake SshDirPublisher is unavailable or your rubyforge environment is not configured."
+  exit(1)
+end
 
 # vim: syntax=Ruby

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:patch: 7
+:major: 0
+:minor: 0

data/lib/calais.rb

@@ -16,6 +16,7 @@ $:.unshift File.expand_path(File.dirname(__FILE__)) + '/calais'
 
 require 'client'
 require 'response'
+require 'error'
 
 module Calais
   REST_ENDPOINT = "http://api.opencalais.com/enlighten/rest/"
@@ -40,7 +41,7 @@ module Calais
 
   MAX_RETRIES = 5
   HTTP_TIMEOUT = 60
-  MIN_CONTENT_SIZE = 100
+  MIN_CONTENT_SIZE = 1
   MAX_CONTENT_SIZE = 100_000
 
   class << self
@@ -53,7 +54,3 @@ module Calais
     end
   end
 end
-
-module Calais
-  VERSION = '0.0.6'
-end

data/lib/calais/error.rb

@@ -0,0 +1,3 @@
+class Calais::Error < StandardError
+  
+end

data/lib/calais/response.rb

@@ -29,15 +29,15 @@ module Calais
     end
 
     class Entity
-      attr_accessor :hash, :type, :attributes, :relevance, :instances
+      attr_accessor :calais_hash, :type, :attributes, :relevance, :instances
     end
 
     class Relation
-      attr_accessor :hash, :type, :attributes, :instances
+      attr_accessor :calais_hash, :type, :attributes, :instances
     end
 
     class Geography
-      attr_accessor :name, :hash, :attributes
+      attr_accessor :name, :calais_hash, :attributes
     end
 
     class Category
@@ -78,6 +78,10 @@ module Calais
       def extract_data
         doc = XML::Parser.string(@raw_response).parse
 
+        if doc.root.find("/Error").first
+          raise Calais::Error, doc.root.find("/Error/Exception").first.content
+        end        
+
         doc.root.find("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:docinfometa]}')]/..").each do |node|
           @language = node['language']
           @submission_date =  DateTime.parse node['submissionDate']
@@ -104,7 +108,8 @@ module Calais
         @categories = doc.root.find("rdf:Description/rdf:type[contains(@rdf:resource, '#{MATCHERS[:doccat]}')]/..").map do |node|
           category = Category.new
           category.name = node.find_first("c:categoryName").content
-          category.score = node.find_first("c:score").content.to_f
+          score = node.find_first("c:score")
+          category.score = score.content.to_f unless score.nil?
 
           node.remove!
           category
@@ -122,7 +127,7 @@ module Calais
           extracted_hash = node['about'].split('/')[-1] rescue nil
 
           entity = Entity.new
-          entity.hash = CalaisHash.find_or_create(extracted_hash, @hashes)
+          entity.calais_hash = CalaisHash.find_or_create(extracted_hash, @hashes)
           entity.type = extract_type(node)
           entity.attributes = extract_attributes(node.find("*[contains(name(), 'c:')]"))
 
@@ -137,7 +142,7 @@ module Calais
           extracted_hash = node['about'].split('/')[-1] rescue nil
 
           relation = Relation.new
-          relation.hash = CalaisHash.find_or_create(extracted_hash, @hashes)
+          relation.calais_hash = CalaisHash.find_or_create(extracted_hash, @hashes)
           relation.type = extract_type(node)
           relation.attributes = extract_attributes(node.find("*[contains(name(), 'c:')]"))
           relation.instances = extract_instances(doc, extracted_hash)
@@ -151,7 +156,7 @@ module Calais
 
           geography = Geography.new
           geography.name = attributes.delete('name')
-          geography.hash = attributes.delete('subject')
+          geography.calais_hash = attributes.delete('subject')
           geography.attributes = attributes
 
           node.remove!

data/spec/calais/response_spec.rb

@@ -6,6 +6,14 @@ describe Calais::Response, :new do
   end
 end
 
+describe Calais::Response, :new do
+  it "should return error message in runtime error" do
+    lambda {
+      @response = Calais::Response.new(RESPONSE_WITH_EXCEPTION)
+    }.should raise_error(Calais::Error, "My Error Message") 
+  end
+end
+
 describe Calais::Response, :new do
   before :all do
     @response = Calais::Response.new(SAMPLE_RESPONSE)
@@ -14,21 +22,21 @@ describe Calais::Response, :new do
   it 'should extract document information' do
     @response.language.should == 'English'
     @response.submission_date.should be_a_kind_of(DateTime)
-    @response.signature.should == 'digestalg-1|iCEVI2NK1nAAvP+p5uaqnHISxdo=|U3QC5z6ZN1DLUJrqiP6gpTuxrdAxOaVOrjUQVuarCmb+zoqbm2fypA=='
-    @response.submitter_code.should == '4a388fbc-9897-def9-9233-efddbfbca363'
-    @response.request_id.should == '896ffd83-ad5f-4e4b-892b-4cc337a246af'
+    @response.signature.should be_a_kind_of(String)
+    @response.submitter_code.should be_a_kind_of(String)
+    @response.request_id.should be_a_kind_of(String)
     @response.doc_title.should == 'Record number of bicycles sold in Australia in 2006'
     @response.doc_date.should be_a_kind_of(Date)
   end
 
   it 'should extract entities' do
     entities = @response.entities
-    entities.map { |e| e.type }.sort.uniq.should == %w[City Continent Country IndustryTerm Organization Person ProvinceOrState]
+    entities.map { |e| e.type }.sort.uniq.should == %w[City Continent Country IndustryTerm Organization Person Position ProvinceOrState]
   end
 
   it 'should extract relations' do
     relations = @response.relations
-    relations.map { |e| e.type }.sort.uniq.should == %w[GenericRelations PersonAttributes PersonProfessional Quotation]
+    relations.map { |e| e.type }.sort.uniq.should == %w[GenericRelations PersonAttributes PersonCareer Quotation]
   end
 
   it 'should extract geographies' do
@@ -37,27 +45,11 @@ describe Calais::Response, :new do
   end
 
   it 'should extract relevances' do
-    @response.instance_variable_get("@relevances").size.should == 10
+    @response.instance_variable_get(:@relevances).should be_a_kind_of(Hash)
   end
 
   it 'should assign a floating-point relevance to each entity' do
-    @response.entities.each {|e| e.relevance.class.should == Float }
-  end
-
-  it 'should assign the correct relevance to each entity' do
-    correct_relevances = {
-      "84a34c48-25ac-327f-a805-7b81fd570f7d" => 0.725,
-      "9853f11e-5efa-3efc-90b9-0d0450f7d673" => 0.396,
-      "9fa3fb8a-f517-32c7-8a46-3c1506ea3a70" => 0.156,
-      "ed0e83f9-87e8-3da6-ab46-cd6be116357c" => 0.291,
-      "e05f3d33-1622-3172-836c-b48637a156d3" => 0.316,
-      "d0ca04b6-9cf5-3595-ad4b-7758a0b57997" => 0.156,
-      "0bb9cdb4-3cb7-342a-9901-6d1f12b32f6a" => 0.31,
-      "3979e581-0823-3e84-9257-1ca36db4665e" => 0.228,
-      "0c3d5340-106f-390e-92d3-a4aa18004fb8" => 0.158,
-      "3bcf2655-ff2a-3a80-8de4-558b9626ad21" => 0.644
-    }
-    @response.entities.each {|e| correct_relevances[e.hash.value].should == e.relevance }
+    @response.entities.each {|e| e.relevance.should be_a_kind_of(Float) }
   end
 
   it 'should find the correct document categories returned by OpenCalais' do
@@ -65,9 +57,18 @@ describe Calais::Response, :new do
   end
 
   it 'should find the correct document category scores returned by OpenCalais' do
-    @response.categories.map {|c| c.score }.should == [1.0, 1.0]
+    @response.categories.map {|c| c.score.should be_a_kind_of(Float) }
   end
-
+  
+  it "should not raise an error if no score is given by OpenCalais" do
+    lambda {Calais::Response.new(SAMPLE_RESPONSE_WITH_NO_SCORE)}.should_not raise_error
+  end
+  
+  it "should not raise an error if no score is given by OpenCalais" do
+    response = Calais::Response.new(SAMPLE_RESPONSE_WITH_NO_SCORE)
+    response.categories.map {|c| c.score }.should == [nil]
+  end
+  
   it 'should find instances for each entity' do
     @response.entities.each {|e|
       e.instances.size.should > 0
@@ -115,7 +116,7 @@ describe Calais::Response, :new do
     ## thorough test that tests for the instances of each of the many other
     ## relations in the sample doc is desirable in the future.
 
-    rel = @response.relations.select {|e| e.hash.value == "8f3936d9-cf6b-37fc-ae0d-a145959ae3b5" }.first
+    rel = @response.relations.select {|e| e.calais_hash.value == "8f3936d9-cf6b-37fc-ae0d-a145959ae3b5" }.first
     rel.instances.size.should == 1
 
     rel.instances.first.prefix.should == " manufacturers.\n\nThe Cycling Promotion Fund (CPF) "

data/spec/helper.rb

@@ -7,4 +7,6 @@ require File.dirname(__FILE__) + '/../lib/calais'
 FIXTURES_DIR = File.join File.dirname(__FILE__), %[fixtures]
 SAMPLE_DOCUMENT = File.read(File.join(FIXTURES_DIR, %[bicycles_australia.xml]))
 SAMPLE_RESPONSE = File.read(File.join(FIXTURES_DIR, %[bicycles_australia.response.rdf]))
+SAMPLE_RESPONSE_WITH_NO_SCORE = File.read(File.join(FIXTURES_DIR, %[twitter_tweet_without_score.response.rdf]))
+RESPONSE_WITH_EXCEPTION = File.read(File.join(FIXTURES_DIR, %[error.response.xml]))
 LICENSE_ID = YAML.load(File.read(File.join(FIXTURES_DIR, %[calais.yml])))['key']

metadata

@@ -1,85 +1,70 @@
 --- !ruby/object:Gem::Specification 
 name: calais
 version: !ruby/object:Gem::Version 
-  version: 0.0.6
+  version: 0.0.7
 platform: ruby
 authors: 
 - Abhay Kumar
 autorequire: 
 bindir: bin
-cert_chain: 
-- |
-  -----BEGIN CERTIFICATE-----
-  MIIDNjCCAh6gAwIBAgIBADANBgkqhkiG9w0BAQUFADBBMQ0wCwYDVQQDDARpbmZv
-  MRswGQYKCZImiZPyLGQBGRYLb3BlbnN5bmFwc2UxEzARBgoJkiaJk/IsZAEZFgNu
-  ZXQwHhcNMDgwMjAzMDUwODQzWhcNMDkwMjAyMDUwODQzWjBBMQ0wCwYDVQQDDARp
-  bmZvMRswGQYKCZImiZPyLGQBGRYLb3BlbnN5bmFwc2UxEzARBgoJkiaJk/IsZAEZ
-  FgNuZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCmaA3Od1p42luz
-  zDJepXD3VBFEmmeCUCOjs8rkGIlhRibBvAU8GB0hhkTUykeF6JvAp68FYtIqyTqM
-  EY7bnyYTWsvX7HrX/wGRshSKZPnxn2b0AnZ9T3QQZyUut1YQ5G+kBQrI76hz9ynA
-  l0mPCiGxrh+yUNTKt7KzOAzQbtPlqGiIzj+aYvzmdEsj24Ekm/11A/ntPnz+N/Wj
-  yS5c2tbfZdU8NfwfHCZQUBE4PROYCCjoly0QChvBQzKSZPrEpJB3EedMUyBc5m5E
-  TQ0u5aItr3isQchwo410x7ixzVveVzn4mchaGCZ3ZuPwaQkuI/7KSSWWH1LCouct
-  N7LsWR7jAgMBAAGjOTA3MAkGA1UdEwQCMAAwCwYDVR0PBAQDAgSwMB0GA1UdDgQW
-  BBRsRhkAWj4iWaut121ZcaOAKXG27DANBgkqhkiG9w0BAQUFAAOCAQEAFuqEVgKC
-  U6f237SZ/hzevOwRkaErF1EcaCEVzuNj+KNdbQOK9oOo+hHyos3jUo17TiUNDi+3
-  VJhw3cOkA/PEpa0ou0Vm8VIfXdp6dh62NhTKHBVwQ/qXHnn3aVuV/zIfOmi9WQ+t
-  mr7ehGTw7URly95GOESW4NKQ95p+iquAh/NGhtHGFt+nxjJGUkkYlnGVaxmmgof3
-  sP2hOrejIrD9jAoejiRhiA+IyEoaYJvlh+D+3MngvnyDFqHiFZgngM0fvTnMTsgT
-  avOOKhLsesocjiElkLMv8mwuY+L8P4tSvDTDKXxM9Bx/YagwgzYCqPoGtFdWI/GI
-  +keKvrmaTOJ7CQ==
-  -----END CERTIFICATE-----
+cert_chain: []
 
-date: 2008-12-18 00:00:00 -08:00
+date: 2009-06-08 00:00:00 -05:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
-  name: hoe
-  type: :development
+  name: libxml-ruby
+  type: :runtime
   version_requirement: 
   version_requirements: !ruby/object:Gem::Requirement 
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
-        version: 1.8.2
+        version: 0.5.4
     version: 
-description: "== Features  * Accepts documents in text/plain, text/xml and text/html format. * Basic access to the Open Calais API's Enlighten action. * Output is RDF representation of input document. * Single function ability to extract names, entities and geographies from given text.  == Synopsis  This is a very basic wrapper to the Open Calais API. It uses the POST endpoint and currently supports the Enlighten action. Here's a simple call:  Calais.enlighten(:content => \"The government of the United Kingdom has given corporations like fast food chain McDonald's the right to award high school qualifications to employees who complete a company training program.\", :content_type => :text, :license_id => LICENSE_ID)  This is the easiest way to get the RDF-formated response from the OpenCalais service.  If you want to do something more fun like getting all sorts of fun information about a document, you can try this:  Calais.process_document(:content => \"The government of the United Kingdom has given corporations like fast food chain McDonald's the right to award high school qualifications to employees who complete a company training program.\", :content_type => :text, :license_id => LICENSE_ID)  This will return an object containing information extracted from the RDF response.  == Requirements  * Ruby 1.8.5 or better * Uses the following standard libraries: digest/sha1, net/http, yaml, cgi * libxml2 / libxml-ruby * curb, libcurl"
+- !ruby/object:Gem::Dependency 
+  name: json
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.1.3
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: curb
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.1.4
+    version: 
+description: A Ruby interface to the Calais Web Service
 email: info@opensynapse.net
 executables: []
 
 extensions: []
 
 extra_rdoc_files: 
-- History.txt
-- Manifest.txt
-- README.txt
+- README.markdown
 files: 
-- History.txt
+- CHANGELOG.markdown
 - MIT-LICENSE
-- Manifest.txt
-- README.txt
+- README.markdown
 - Rakefile
-- init.rb
+- VERSION.yml
 - lib/calais.rb
 - lib/calais/client.rb
+- lib/calais/error.rb
 - lib/calais/response.rb
-- spec/calais/client_spec.rb
-- spec/calais/response_spec.rb
-- spec/calais_spec.rb
-- spec/fixtures/.gitignore
-- spec/fixtures/bicycles_australia.response.json
-- spec/fixtures/bicycles_australia.response.rdf
-- spec/fixtures/bicycles_australia.xml
-- spec/fixtures/calais.yml.sample
-- spec/fixtures/slovenia_euro.xml
-- spec/helper.rb
-- spec/spec.opts
 has_rdoc: true
-homepage: http://calais.rubyforge.org
+homepage: http://github.com/abhay/calais
 post_install_message: 
 rdoc_options: 
-- --main
-- README.txt
+- --charset=UTF-8
 require_paths: 
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement 
@@ -101,5 +86,7 @@ rubygems_version: 1.3.1
 signing_key: 
 specification_version: 2
 summary: A Ruby interface to the Calais Web Service
-test_files: []
-
+test_files: 
+- spec/helper.rb
+- spec/calais/response_spec.rb
+- spec/calais/client_spec.rb