dover_to_calais 0.1.0

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/.yardopts

@@ -0,0 +1,4 @@
+--no-private
+--title "DoverToCalais"
+lib/**/*.erb
+lib/**/*.rb

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in dover_to_calais.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Fred Heath
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,77 @@
+# DoverToCalais
+
+DoverToCalais allows the user to send a wide range of data sources (files & URLs)
+to [OpenCalais](http://www.opencalais.com/about) and receive asynchronous responses when [OpenCalais](http://www.opencalais.com/about) has finished processing
+the inputs. In addition, DoverToCalais enables the filtering of the response in order to
+find relevant tags and/or tag values.
+
+## What is OpenCalais?
+In short -and quoting the [OpenCalais](http://www.opencalais.com/about) creators:  
+
+*The OpenCalais Web Service automatically creates rich semantic metadata for the content you submit – in well under a second. Using natural language processing (NLP), machine learning and other methods, Calais analyzes your document and finds the entities within it. But, Calais goes well beyond classic entity identification and returns the facts and events hidden within your text as well.*
+
+## Why use OpenCalais?
+There are many reasons, mainly to:
+   
+ * incorporate tags into other applications, such as search, news aggregation, blogs, catalogs, etc.
+ * enrich search by looking for deeper, contextual meaning instead of merely phrases or keywords.
+ * help to discern relationships between semantic entities. 
+ * facilitate data processing and analysis by allowing easy filtering of relevant data sources and the discarding of irrelevant ones.
+
+
+## DoverToCalais Features
+1. **Supports many data sources**: Thanks to the power of [Yomu](https://github.com/Erol/yomu), DoverToCalais can process a vast range of files (and, of course, web pages), extract text from them and send
+them to OpenCalais for analysis and tag generation. 
+
+2. **Asynchronous responses (callbacks)**:
+Users can set callbacks to receive the processed meta-data, once the OpenCalais Web Service response has  been received.
+Furthermore, a user can set multiple callbacks for the same request (data source), thus enabling cleaner,
+more modular code. 
+
+3. **Result filtering**: DoverToCalais uses the OpenCalais [Simple XML Format](http://www.opencalais.com/documentation/calais-web-service-api/interpreting-api-response/simple-format) as its preferred response format. The user can work directly with the XML-formatted response, or -if feeling a bit lazy- can take advantage of the DoverToCalais filtering functionality and receive specific entities, optionally  based on specified conditions.
+
+For more details of the features and code samples, see [Usage](#usage).
+
+##Pre-requisites
+
+To use the OpenCalais Web Service  and -by extension- DoverToCalais, one needs to possess an OpenCalais API key, which is easily obtainable from the [OpenCalais web site](http://www.opencalais.com/APIkey).
+
+Also, DoverToCalais requires the presence of a working [JRE](http://en.wikipedia.org/wiki/JRE#Execution_environment). 
+
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'dover_to_calais'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install dover_to_calais
+
+## Dependencies
+DoverToCalais has been developed in Ruby 1.9.3 and requires the following gems (for development purposes only)
+
+ * 'nokogiri', 1.6.0
+ * 'eventmachine', 1.0.3
+ * 'em-http-request', 1.1.0
+ * 'open-uri', 
+ * 'yomu', 0.1.9 
+
+As [Yomu](https://github.com/Erol/yomu) depends on a working JRE in order to function, so does DoverToCalais.
+
+## Usage
+
+TODO: Write usage instructions here
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/dover_to_calais.gemspec

@@ -0,0 +1,33 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'dover_to_calais/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "dover_to_calais"
+  spec.version       = DoverToCalais::VERSION
+  spec.authors       = ["Fred Heath"]
+  spec.email         = ["fred@bootstrap.me.uk"]
+  spec.description   = %q{DoverToCalais allows the user to send a wide range of data sources (files & URLs)
+                          to OpenCalais and receive asynchronous responses when OpenCalais has finished processing
+                          the inputs. In addition, DoverToCalais enables the filtering of the response in order to
+                          find relevant tags and/or tag values. }
+  spec.summary       = %q{An easy-to-use wrapper round the OpenCalais semantic analysis web service. }
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+
+  spec.add_runtime_dependency "nokogiri", "~>1.6.0"
+  spec.add_runtime_dependency "eventmachine", "~>1.0.3"
+  spec.add_runtime_dependency "em-http-request", "~>1.1.0"
+  spec.add_runtime_dependency "yomu", "~>0.1.9"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+end

data/lib/dover_to_calais.rb

@@ -0,0 +1,279 @@
+require "dover_to_calais/version"   #gem lib file
+require "dover_to_calais/ontology"  #gem lib file
+require 'open-uri'  # in std library
+require 'nokogiri'
+require 'eventmachine'
+require 'em-http-request'
+require 'yomu'
+
+
+module DoverToCalais
+
+
+  PROXY = nil
+
+  # The ResponseItem structure holds all potential text and attribute values of an OpenCalais
+  # XML Simple format element.
+  class ResponseItem < Struct.new(:name, :value, :relevance, :count, :normalized, :importance, :originalValue)
+    #
+    # @!attribute [r] name
+    #   @return [String] the element's name.
+    #
+    #
+    # @!attribute [r] value
+    #   @return [String] the element's text value.
+    #
+    #
+    # @!attribute [r] relevance
+    #   The importance of the element in the context of the given input, in the range 0-1
+    #   (1 being the most relevant and important). The score has 3-digit precision after the decimal point.
+    #   @return [Float] the element's relevance, in the range 0-1.
+    #
+    #
+    # @!attribute [r] count
+    #   @return [Integer]  the count (frequency) of the element in the given input.
+    #
+    #
+    # @!attribute [r] normalized
+    #   If the element is one of: (Company, City, ProvinceOrState, Country) OpenCalais
+    #   provides disambiguation of its value in the 'normalized' attribute. For instance, the element
+    #   <City>Birmingham</City> may include the attribute normalized = "Birmingham, UK" in order to avoid
+    #   confusion with the city of Birmingham, USA.
+    #   @return [String] the element text's disambiguated value
+    #
+    #
+    # @!attribute [r] importance
+    #   Only applicable to the SocialTag element.
+    #   @return [1, 2] the importance of the SocialTag.
+    #
+    #
+    # @!attribute [r] originalValue
+    #   Only applicable to the SocialTag element.
+    #   @return [String] The original value of the SocialTag.
+    #
+  end
+
+
+  # This class is responsible for creating a response object that will be passed to {Dover}, after
+  # the data source has been analysed by OpenCalais. If the response contains valid data, the
+  # {#error} attribute will be nil. The response object will then contain the OpenCalais response
+  # as an XML string. The user can then call {#filter} to filter the response.
+  # If the response doesn't contain valid, processed data then the {#error} won't be nil (i.e. will
+  # be true). The {#error} attribute can then be read in order to find the cause of the error.
+  #
+  # @!attribute [r] error
+  #   @return [String, nil] any error that occurred as a result of the OpenCalais API call,
+  #     nil if none occurred
+  #
+  class ResponseData
+    attr_reader :error
+
+    # creates a new ResponseData object, passing the name of the data source to be processed
+    #
+    # @param xml_data [ Nokogiri::XML::NodeSet, nil] the xml data returned by OpenCalais
+    # @param error [ String, nil] an error description if the OpenCalais call has failed
+    def initialize(xml_data = nil, error = nil)
+      if xml_data
+        @raw = xml_data
+      else
+        @error = error
+      end
+    end
+
+    # Returns  the response data as an XML string or an error, if one has occurred.
+    #
+    # @return [String] an XML string
+    def to_s
+      @raw ?  @raw.to_s : @error
+    end
+
+
+    # Filters the response object to extract relevant data.
+    #
+    # @param params [Hash] a filter Hash (see code samples)
+    # @return [Array[ResponseItem]] a list of relevant response items
+    def filter(params)
+      result = Array.new
+      begin
+        if @raw
+
+          if params[:given]
+            found = @raw.xpath("//#{params[:given][:entity]}[contains(text(), #{params[:given][:value].inspect})]")
+            if found.size > 0
+              @raw.xpath("//#{params[:entity]}[contains(text(), #{params[:value].inspect})]").each do |node|
+                result <<  create_response_item(node)
+              end
+            end
+          else  # no conditional
+            @raw.xpath("//#{params[:entity]}[contains(text(), #{params[:value].inspect})]").each do |node|
+              result <<  create_response_item(node)
+            end
+          end
+
+          return result
+        else # no xml data
+          puts 'ERR: no valid xml data!'
+
+        end #if
+
+      rescue  Exception=>e
+        puts "ERR: #filter:  #{e}"
+
+      end
+
+      #return result
+
+    end  #method
+
+    def create_response_item(node)
+      node_relevance =  node.attribute('relevance').text.to_f if node.has_attribute?('relevance')
+      node_count =  node.attribute('count').text.to_i if node.has_attribute?('count')
+      node_normalized =  node.attribute('normalized').text if node.has_attribute?('normalized')
+      node_importance = node.attribute('importance').text.to_i if node.has_attribute?('importance')
+
+      ResponseItem.new(node.name,
+                       node.content,
+                       node_relevance,
+                       node_count,
+                       node_normalized,
+                       node_importance,
+                       node.xpath('originalValue').text )
+
+    end
+
+    public :filter
+    private :create_response_item
+
+  end #class
+
+
+  #====================================================================================
+
+  # This class is responsible for parsing, reading and sending to OpenCalais, text from a data source.
+  # The data source is passed to the class constructor and can be pretty much any form of document or URL.
+  # The class allows the user to specify one or more callbacks, to be called when the data source has been
+  # processed by OpenCalais ({#to_calais}).
+  #
+  # @!attribute [r] data_src
+  #   @return [String] the data source to be processed, either a file path or a URL.
+  #
+  # @!attribute [r] error
+  #   @return [String, nil] any error that occurred during data-source processing, nil if none occurred
+  #
+  class Dover
+
+    CALAIS_SERVICE = 'https://api.opencalais.com/tag/rs/enrich'
+
+    attr_reader :data_src, :error
+
+    # creates a new Dover object, passing the name of the data source to be processed
+    #
+    # @param data_src [String] the name of the data source to be processed
+    def initialize(data_src)
+      @data_src = data_src
+      @callbacks = []
+      analyse_this
+    end
+
+
+    # uses the {https://github.com/Erol/yomu yomu} gem to extract text from a number of document formats and URLs.
+    # If an exception occurs, it is written to the {@error} instance variable
+    #
+    # @param [String] src the name of the data source (file-path or URI)
+    # @return [String, nil] the extracted text, or nil if an exception occurred.
+    def get_src_data(src)
+      begin
+        yomu = Yomu.new src
+
+      rescue Exception=>e
+        @error = "ERR: #{e}"
+      else
+        yomu.text
+      end
+
+    end
+
+    # Defines the user callbacks. If the data source is successfully read, then this method will store a
+    # user-defined block which will be called on completion of the OpenCalais HTTP request. If the data source
+    # cannot be read -for whatever reason- then the block will immediately be called, passing the parameter that
+    # caused the read failure.
+    #
+    # @param block a user-defined block
+    # @return N/A
+    def to_calais(&block)
+      #fred rules ok
+      if @document
+        @callbacks << block
+      else
+        result = ResponseData.new nil, @error
+        block.call(result)
+      end
+
+    end #method
+
+    #
+    def analyse_this
+
+      @document = get_src_data(@data_src)
+      begin
+        if @document
+          response = nil
+
+          connection_options = {:inactivity_timeout => 0}
+
+
+          if DoverToCalais::PROXY &&
+              DoverToCalais::PROXY.class.eql?('Hash') &&
+              DoverToCalais::PROXY.keys[0].eql?(:proxy)
+
+            connection_options = connection_options.merge(DoverToCalais::PROXY)
+          end
+
+          request_options = {
+              :body => @document.to_s,
+              :head => {
+                  'x-calais-licenseID' => DoverToCalais::API_KEY,
+                  :content_type => 'TEXT/RAW',
+                  :enableMetadataType => 'GenericRelations,SocialTags',
+                  :outputFormat => 'Text/Simple'}
+          }
+
+          http = EventMachine::HttpRequest.new(CALAIS_SERVICE, connection_options ).post request_options
+
+
+          http.callback do
+            http.response.match(/<OpenCalaisSimple>/) do |m|
+              response = Nokogiri::XML('<OpenCalaisSimple>' + m.post_match)  do |config|
+                #strict xml parsing, disallow network connections
+                config.strict.nonet
+              end #block
+            end #block
+
+            result =   response ? ResponseData.new(response, nil) :  ResponseData.new(nil,'ERR: unable to find <OpenCalaisSimple> tag in response data')
+            @callbacks.each { |c| c.call(result) }
+          end  #callback
+
+
+          http.errback do
+
+            result = ResponseData.new nil, "#{http.error}"
+            @callbacks.each { |c| c.call(result) }
+          end  #errback
+
+
+        end  #if
+      rescue  Exception=>e
+        puts "ERR: #{e}"
+        exit 0
+      end
+
+    end  #method
+
+
+    public :to_calais
+    private :get_src_data, :analyse_this
+
+
+  end  #class
+
+end

data/lib/dover_to_calais/ontology.rb

@@ -0,0 +1,147 @@
+#  ontology.rb
+#  
+#  Copyright 2013 Fred <fred@fred-Veriton-X270>
+#  
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; either version 2 of the License, or
+#  (at your option) any later version.
+#  
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#  
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software
+#  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+#  MA 02110-1301, USA.
+#  
+#  
+
+
+module CalaisOntology
+  CALAIS_ENTITIES = %w(
+    Anniversary
+    City
+    Company
+    Continent
+    Country
+    Currency
+    EmailAddress
+    EntertainmentAwardEvent
+    Facility
+    FaxNumber
+    Holiday
+    IndustryTerm
+    MarketIndex
+    MedicalCondition
+    MedicalTreatment
+    Movie
+    MusicAlbum
+    MusicGroup
+    NaturalFeature
+    OperatingSystem
+    Organization
+    Person
+    PhoneNumber
+    PoliticalEvent
+    Product
+    ProgrammingLanguage
+    ProvinceOrState
+    PublishedMedium
+    RadioProgram
+    RadioStation
+    Region
+    SportsEvent
+    SportsGame
+    SportsLeague
+    TVShow
+    TVStation
+    Technology
+    URL
+  )
+
+CALAIS_EVENTS = %w(
+    Acquisition
+    Alliance
+    AnalystEarningsEstimate
+    AnalystRecommendation
+    Arrest
+    Bankruptcy
+    BonusSharesIssuance
+    BusinessRelation
+    Buybacks
+    CompanyAccountingChange
+    CompanyEarningsAnnouncement
+    CompanyEarningsGuidance
+    CompanyExpansion
+    CompanyForceMajeure
+    CompanyInvestment
+    CompanyLaborIssues
+    CompanyLayoffs
+    CompanyLegalIssues
+    CompanyListingChange
+    CompanyMeeting
+    CompanyNameChange
+    CompanyReorganization
+    CompanyRestatement
+    ConferenceCall
+    Conviction
+    CreditRating
+    DebtFinancing
+    DelayedFiling
+    DiplomaticRelations
+    Dividend
+    EmploymentChange
+    EnvironmentalIssue
+    Extinction
+    FDAPhase
+    Indictment
+    IPO
+    JointVenture
+    ManMadeDisaster
+    Merger
+    MovieRelease
+    MusicAlbumRelease
+    NaturalDisaster
+    PatentFiling
+    PatentIssuance
+    PersonTravel
+    PoliticalEndorsement
+    PoliticalRelationship
+    PollsResult
+    ProductIssues
+    ProductRecall
+    ProductRelease
+    SocialTags
+    SecondaryIssuance
+    PersonCommunication
+    StockSplit
+    Trial
+    VotingResult
+)
+
+CALAIS_TOPICS = %w(
+    Business_Finance
+    Disaster_Accident
+    Education
+    Entertainment_Culture
+    Environment
+    Health_Medical_Pharma
+    Hospitality_Recreation
+    Human_Interest
+    Labor
+    Law_Crime
+    Politics
+    Religion_Belief
+    Social_Issues
+    Sports
+    Technology_Internet
+    Weather
+    War_Conflict
+    Other
+)
+
+end
+

data/lib/dover_to_calais/version.rb

@@ -0,0 +1,3 @@
+module DoverToCalais
+  VERSION = "0.1.0"
+end

metadata

@@ -0,0 +1,156 @@
+--- !ruby/object:Gem::Specification
+name: dover_to_calais
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Fred Heath
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-10-04 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.6.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.6.0
+- !ruby/object:Gem::Dependency
+  name: eventmachine
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.0.3
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.0.3
+- !ruby/object:Gem::Dependency
+  name: em-http-request
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.1.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.1.0
+- !ruby/object:Gem::Dependency
+  name: yomu
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.1.9
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.1.9
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: ! "DoverToCalais allows the user to send a wide range of data sources
+  (files & URLs)\n                          to OpenCalais and receive asynchronous
+  responses when OpenCalais has finished processing\n                          the
+  inputs. In addition, DoverToCalais enables the filtering of the response in order
+  to\n                          find relevant tags and/or tag values. "
+email:
+- fred@bootstrap.me.uk
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .yardopts
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- dover_to_calais.gemspec
+- lib/dover_to_calais.rb
+- lib/dover_to_calais/ontology.rb
+- lib/dover_to_calais/version.rb
+homepage: ''
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: An easy-to-use wrapper round the OpenCalais semantic analysis web service.
+test_files: []