dover_to_calais 0.1.0 → 0.2.0

checksums.yaml

@@ -0,0 +1,15 @@
+---
+!binary "U0hBMQ==":
+  metadata.gz: !binary |-
+    OWJmOWEyMGFjNDk2ZjZiODYyNjQ1NDM2YjM0YjMyNzQ1MmUzZjg3MA==
+  data.tar.gz: !binary |-
+    MTllMDRiOTNlNDg2Y2FiNmY1MmQyMjAyMzViNWJiZWFmN2ZjYTY3ZA==
+SHA512:
+  metadata.gz: !binary |-
+    OGI3NDU4YWU1YzllMjBiNTVlMmU3NzNhMDUzYmNhNWYzODY0ZTE3MDQzZmMx
+    NmZiMDMxZjYzMTI3ZTdkYWU5MGNiNDc3ZTE2ZTRjYThjYjc5ZDQxNjFlZjU0
+    MGRmZWY5ZGM4NTAwYjAyZTEyZmY5M2I5MDdjNDA4NWQ1MDE4MDM=
+  data.tar.gz: !binary |-
+    ODUyZGFhN2JhYjdjZDAyNmMxMTNhZjY0MjJhNWQ5YjU2OTY0OTQyNmU4MDkz
+    NjljZTU1NDUzYTRhN2I2MTA3MmQ3MTM3MDYxODUyMjgzOGVkYTYzNzY4MjA1
+    YTgyZDNkNjE3YjI1NWJiYTJkMzNjN2RiYzEzN2M1MWNmYzFhMzU=

data/.gitignore

@@ -1,17 +1,33 @@
 *.gem
 *.rbc
-.bundle
-.config
-.yardoc
+/.config
+/coverage/
+/InstalledFiles
+/pkg/
+/spec/reports/
+/test/tmp/
+/test/version_tmp/
+/tmp/
+
+## Documentation cache and generated files:
+/.yardoc/
+/_yardoc/
+/doc/
+/rdoc/
+
+## Environment normalisation:
+/.bundle/
+/lib/bundler/man/
+
+# for a library or gem, you might want to ignore these files since the code is
+# intended to run in multiple environments; otherwise, check them in:
 Gemfile.lock
-InstalledFiles
-_yardoc
-coverage
-doc/
-lib/bundler/man
-pkg
-rdoc
-spec/reports
-test/tmp
-test/version_tmp
-tmp
+.ruby-version
+.ruby-gemset
+
+# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
+.rvmrc
+
+## Sublime Text project files
+*.sublime-project
+*.sublime-workspace

data/README.md

@@ -1,14 +1,19 @@
+
+
 # DoverToCalais
 
 DoverToCalais allows the user to send a wide range of data sources (files & URLs)
 to [OpenCalais](http://www.opencalais.com/about) and receive asynchronous responses when [OpenCalais](http://www.opencalais.com/about) has finished processing
-the inputs. In addition, DoverToCalais enables the filtering of the response in order to
-find relevant tags and/or tag values.
+the inputs. In addition, DoverToCalais enables response filtering in order to find relevant tags and/or tag values.
 
 ## What is OpenCalais?
 In short -and quoting the [OpenCalais](http://www.opencalais.com/about) creators:  
+> "*The OpenCalais Web Service automatically creates rich semantic metadata for the content you submit – in well under a second. Using natural language processing (NLP), machine learning and other methods, Calais analyzes your document and finds the entities within it. But, Calais goes well beyond classic entity identification and returns the facts and events hidden within your text as well.*"
+
+In general, OpenCalais Simple XML Format (the one used by DoverToCalais) returns three kinds of tags: [Entitites, Events](http://www.opencalais.com/documentation/calais-web-service-api/api-metadata/entity-index-and-definitions) and [Topics](http://www.opencalais.com/documentation/calais-web-service-api/api-metadata/document-categorization). ***Entities*** are static 'things', like Persons, Places, et al. that are involved in the textual context in some capacity. OpenCalais assigns a *relevance* score to each entity to indicate it's relevance within the context of the data source's general topic.  ***Events*** are facts or actions that pertain to one or more Entities.  ***Topics*** are a characterisation or generic description of the data source's context. 
+
+We can use these tags and the information within them to extract relevant information from the data or to draw useful conclusions about it. For example, if the data source tags include an *<Event>* with the value of *'CompanyExpansion'*, I can then look for the <City> or  <Company> tags to find out which company is expanding and if it's near my location (hint: they may be looking for more staff :))  Or, I could pick out all <Company>s involved in a <JointVenture>, or all  <Person>s implicated in an  <Arrest> in my  <City>, etc.
 
-*The OpenCalais Web Service automatically creates rich semantic metadata for the content you submit – in well under a second. Using natural language processing (NLP), machine learning and other methods, Calais analyzes your document and finds the entities within it. But, Calais goes well beyond classic entity identification and returns the facts and events hidden within your text as well.*
 
 ## Why use OpenCalais?
 There are many reasons, mainly to:
@@ -16,11 +21,11 @@ There are many reasons, mainly to:
  * incorporate tags into other applications, such as search, news aggregation, blogs, catalogs, etc.
  * enrich search by looking for deeper, contextual meaning instead of merely phrases or keywords.
  * help to discern relationships between semantic entities. 
- * facilitate data processing and analysis by allowing easy filtering of relevant data sources and the discarding of irrelevant ones.
+ * facilitate data processing and analysis by allowing easy identification of relevant or important data sources and the discarding of irrelevant ones.
 
 
 ## DoverToCalais Features
-1. **Supports many data sources**: Thanks to the power of [Yomu](https://github.com/Erol/yomu), DoverToCalais can process a vast range of files (and, of course, web pages), extract text from them and send
+1. **Multiple data source support**: Thanks to the power of [Yomu](https://github.com/Erol/yomu), DoverToCalais can process a vast range of files (and, of course, web pages), extract text from them and send
 them to OpenCalais for analysis and tag generation. 
 
 2. **Asynchronous responses (callbacks)**:
@@ -28,7 +33,7 @@ Users can set callbacks to receive the processed meta-data, once the OpenCalais
 Furthermore, a user can set multiple callbacks for the same request (data source), thus enabling cleaner,
 more modular code. 
 
-3. **Result filtering**: DoverToCalais uses the OpenCalais [Simple XML Format](http://www.opencalais.com/documentation/calais-web-service-api/interpreting-api-response/simple-format) as its preferred response format. The user can work directly with the XML-formatted response, or -if feeling a bit lazy- can take advantage of the DoverToCalais filtering functionality and receive specific entities, optionally  based on specified conditions.
+3. **Result filtering**: DoverToCalais uses the OpenCalais [Simple XML Format](http://www.opencalais.com/documentation/calais-web-service-api/interpreting-api-response/simple-format) as the preferred response format. The user can work directly with the XML-formatted response, or -if feeling a bit lazy- can take advantage of the DoverToCalais filtering functionality and receive specific entities, optionally  based on specified conditions.
 
 For more details of the features and code samples, see [Usage](#usage).
 
@@ -53,20 +58,325 @@ Or install it yourself as:
 
     $ gem install dover_to_calais
 
+
+
 ## Dependencies
-DoverToCalais has been developed in Ruby 1.9.3 and requires the following gems (for development purposes only)
+DoverToCalais has been developed in Ruby 1.9.3 and relies on the following gems to work (installation with the gem command will automatically install all dependencies)
 
  * 'nokogiri', 1.6.0
  * 'eventmachine', 1.0.3
  * 'em-http-request', 1.1.0
- * 'open-uri', 
  * 'yomu', 0.1.9 
 
 As [Yomu](https://github.com/Erol/yomu) depends on a working JRE in order to function, so does DoverToCalais.
 
 ## Usage
+Using DoverToCalais is extremely simple.
+
+### The Basics
+As DoverToCalais uses the awesome-ness of [EventMachine](http://rubyeventmachine.com/), code must be placed within an EM *run* block:
+
+```ruby
+EM.run do
+
+    # use Control + C to stop the EM
+    Signal.trap('INT')  { EventMachine.stop }
+    Signal.trap('TERM') { EventMachine.stop }
+
+    # we need an API key to use OpenCalais
+    DoverToCalais::API_KEY =  'my-opencalais-api-key'
+    # create a new dover
+    dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
+    # parse the text and send it to OpenCalais
+    dover.analyse_this
+    puts 'do some stuff....'
+    # set a callback for when we receive a response
+    dover.to_calais { |response| puts response.error ? response.error : response }
+
+    puts 'do some more stuff....'
+
+end
+```
+This will produce the following result:
+
+
+> do some stuff....  <br>
+> do some more stuff.... <br>
+> <?xml version="1.0"?>  <br>
+> &lt;OpenCalaisSimple&gt;  <br>
+> ..........    <br>
+> (the rest of the XML response from OpenCalais)  <br>
+
+
+As can be observed, the callback (#to_calais) is trigerred after the rest of the code has been executed and only when the OpenCalais request has been completed.
+
+Of course, we can analyse more than one sources at a time:
+
+```ruby
+EM.run do
+
+  # use Control + C to stop the EM
+  Signal.trap('INT')  { EventMachine.stop }
+  Signal.trap('TERM') { EventMachine.stop }
+
+  DoverToCalais::API_KEY =  'my-opencalais-api-key'
+
+  d1 =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
+  d2 =  DoverToCalais::Dover.new('/home/fred/Documents/RailsRecipes.pdf')
+  d3 =  DoverToCalais::Dover.new('//network-drive/annual_forecast.doc')
+
+  d1.analyse_this; d2.analyse_this; d3.analyse_this; 
+
+  puts 'do some stuff....'
+
+  d1.to_calais { |response| puts response.error ? response.error : response }
+  d2.to_calais { |response| puts response.error ? response.error : response }
+  d3.to_calais { |response| puts response.error ? response.error : response }
+
+  puts 'do some more stuff....'
+
+end
+```
+
+This will output the two *puts* statements followed by the three callbacks (d1, d2, d3) in the order in which they are triggered, i.e. the first callback to receive a response from OpenCalais will fire first.
+
+
+###Filtering the response
+Why parse the response XML ourselves when DoverToCalais can do it for us? We'll just use the *#filter* method on the response object, passing a filtering hash:
+
+ ```ruby
+     my_filter = {:entity => 'Entity1', :value => 'Value1', :given => {:entity => 'Entity2', :value => 'Value2'}}
+     reponse.filter(my_filter)
+ ```
+
+The above tells DoverToCalais to look in the reponse for an entity called 'Entity1' with a value of 'Value1', **only** if the response contains an entity called 'Entity2' which has a value of 'Value2'.
+
+The conditional clause (*:given*) is optional; the filtering hash can be used in pretty much any permutation.  For instance:
+
+```ruby
+EM.run do
+
+    DoverToCalais::API_KEY =  'my-opencalais-api-key'
+
+    dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
+    dover.analyse_this
+
+    dover.to_calais do |response|
+    if   response.error
+      puts  response.error
+    else
+      puts response.filter({:entity => 'Company'})
+    end
+    end
+
+end
+```
+
+This will pick out all entities tagged 'Company' from the data source. The output will be an Array of ResponseItem objects. 
+
+
+> &lt;struct DoverToCalais::ResponseItem name="Company", value="BBC News", relevance=0.654, count=13, normalized=nil, importance=nil, originalValue=nil&gt;<br>
+> &lt;struct DoverToCalais::ResponseItem name="Company", value="TV Radio", relevance=0.565, count=2, normalized="HERALD & WEEKLY-TV,RADIO OPS", importance=nil, originalValue=nil&gt;  <br>
+> &lt;struct DoverToCalais::ResponseItem name="Company", value="Reuters", relevance=0.255, count=2, normalized="THOMSON REUTERS GROUP LIMITED", importance=nil, originalValue=nil&gt;  <br>
+> &lt;struct DoverToCalais::ResponseItem name="Company", value="Twitter", relevance=0.395, count=1, normalized="TWITTER, INC.", importance=nil, originalValue=nil&gt;  <br>
+> &lt;struct DoverToCalais::ResponseItem name="Company", value="Huffington Post UK", relevance=0.136, count=1, normalized=nil, importance=nil, originalValue=nil&gt; <br>
+> &lt;struct DoverToCalais::ResponseItem name="Company", value="Ireland Kenya", relevance=0.144, count=1, normalized=nil, importance=nil, originalValue=nil&gt; <br>
+> &lt;struct DoverToCalais::ResponseItem name="Company", value="Yahoo! UK", relevance=0.144, count=1, normalized="YAHOO! UK LIMITED", importance=nil, originalValue=nil&gt; <br>
+
+
+If this output looks a bit cluttered, we can easily tidy it up:
+
+```ruby
+EM.run do
+
+  DoverToCalais::API_KEY =  'my-opencalais-api-key'
+
+  dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/world-africa-24412315')
+  dover.analyse_this
+
+  dover.to_calais do |response|
+    if   response.error
+      puts  response.error
+    else
+      items = response.filter({:entity => 'Company'})
+      items.each do |item|
+        puts "#{item.name}: #{item.value}, relevance = #{item.relevance}"
+      end
+    end
+  end
+
+end
+```
+
+Which will give us:
+
+
+> Company: BBC News, relevance = 0.656  <br>
+> Company: TV Radio, relevance = 0.566  <br>
+> Company: Reuters, relevance = 0.26  <br>
+> Company: Guardian.co.uk, relevance = 0.143  <br>
+> Company: Twitter, relevance = 0.399  <br>
+> Company: Huffington Post UK, relevance = 0.132  <br>
+> Company: Ireland Kenya, relevance = 0.139  <br>
+> Company: Yahoo! UK, relevance = 0.139  <br>
+
+
 
-TODO: Write usage instructions here
+Let's see if the data source refers to any business partnerships:
+
+```ruby
+EM.run do
+
+  DoverToCalais::API_KEY =  'my-opencalais-api-key'
+
+  dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/technology-24380202')
+  dover.analyse_this
+
+  dover.to_calais do |response|
+    if   response.error
+      puts  response.error
+    else
+      items = response.filter({:entity => 'Event', :value => 'Business Partnership'})
+      puts "There are #{items.length} events like that in the source"
+    end
+  end
+
+end
+```
+
+which will produce:
+
+> There are 1 events like that in the source
+
+
+Now let's find all companies involved in any business partnerships:
+
+```ruby
+EM.run do
+
+  DoverToCalais::API_KEY =  'my-opencalais-api-key'
+
+  dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/technology-24380202')
+  dover.analyse_this
+
+  dover.to_calais do |response|
+    if   response.error
+      puts  response.error
+    else
+      items = response.filter( {:entity => 'Company', :given => {:entity => 'Event',  :value => 'Business Partnership'}} )
+      items.each do |item|
+        puts "#{item.name}: #{item.value} a.k.a #{item.normalized}, relevance = #{item.relevance}"
+      end
+    end
+  end
+
+end
+```
+
+which gives us:
+
+> Company: BBC News a.k.a , relevance = 0.678 <br>
+> Company: Google a.k.a GOOGLE INC., relevance = 0.508 <br>
+> Company: Flutter a.k.a FLUTTER COM INC, relevance = 0.531 <br>
+> Company: TV Radio a.k.a HERALD & WEEKLY-TV,RADIO OPS, relevance = 0.558 <br>
+> Company: Microsoft a.k.a MICROSOFT CORPORATION, relevance = 0.303 <br>
+> Company: Adobe a.k.a ADOBE SYSTEMS INCORPORATED, relevance = 0.193 <br>
+> Company: Netflix a.k.a NETFLIX, INC., relevance = 0.301 <br>
+> Company: Y Combinator a.k.a Y Combinator, relevance = 0.258 <br>
+> Company: Nintendo a.k.a Nintendo Co., Ltd., relevance = 0.286 <br>
+> Company: Samsung a.k.a Samsung C&T Corporation, relevance = 0.285 <br>
+> Company: Glyndwr University a.k.a , relevance = 0.269 <br>
+
+
+
+At this point, someone may ask: "But what if we want to get more than one entity for a given condition? The filter hash doesn't allow that!"
+
+No it doesn't. However, given that filtering is done on the *whole* reponse *after* it's been received, we can apply many filters on the same response:
+
+```ruby
+EM.run do
+
+  DoverToCalais::API_KEY =  'my-opencalais-api-key'
+
+  dover =  DoverToCalais::Dover.new('http://www.bbc.co.uk/news/technology-24380202')
+  dover.analyse_this
+
+  dover.to_calais do |response|
+    if   response.error
+      puts  response.error
+    else
+      result1 = response.filter( {:entity => 'Company', :value => 'Google', :given => {:entity => 'Technology',  :value => 'gesture recognition'}} )
+      result2 = response.filter( {:entity => 'Product', :given => {:entity => 'Technology',  :value => 'gesture recognition'}} )
+      puts result1 | result2
+    end
+  end
+
+end
+```
+
+Which will give us all the gesture-recognition products that Google is associated with according to our data source: 
+
+> &lt;struct DoverToCalais::ResponseItem name="Company", value="Google", relevance=0.506, count=7, normalized="GOOGLE INC.", importance=nil, originalValue=nil&gt; <br>
+> &lt;struct DoverToCalais::ResponseItem name="Product", value="Xbox Kinect", relevance=0.286, count=1, normalized=nil, importance=nil, originalValue=nil&gt; <br>
+> &lt;struct DoverToCalais::ResponseItem name="Product", value="Galaxy S4 smartphone", relevance=0.282, count=1, normalized=nil, importance=nil, originalValue=nil&gt; <br>
+> &lt;struct DoverToCalais::ResponseItem name="Product", value="Wii", relevance=0.286, count=1, normalized=nil, importance=nil, originalValue=nil&gt; <br>
+> &lt;struct DoverToCalais::ResponseItem name="Product", value="Galaxy S4", relevance=0.282, count=1, normalized=nil, importance=nil, originalValue=nil&gt; <br>
+
+
+
+
+***PS***: If you're not sure about the names or values of the tags you want to filter, you can get a listing with the following Constants:
+
+```ruby
+CalaisOntology::CALAIS_ENTITIES
+CalaisOntology::CALAIS_EVENTS
+CalaisOntology::CALAIS_TOPICS
+```
+
+###Code samples
+
+More examples of using DoverToCalais can be found as GitHub Gists:
+
+[Using DoverToCalais to semantically tag all files in a directory](https://gist.github.com/RedFred7/6961349)  
+[Use DoverToCalais to find all Persons or Organizations with a relevance score greater than 0.1, if the data source contains an environmental event](https://gist.github.com/RedFred7/6961853)  
+
+
+### Using a Proxy
+
+If you're behind a corporate firewall and the only way to reach outside is through a proxy then you need to set the *DoverToCalais::PROXY* constant:
+
+```ruby
+    DoverToCalais::PROXY = 
+        :proxy => {
+           :host => 'www.myproxy.com',
+           :port => 8080,
+           :authorization => ['username', 'password'] #optional
+        }
+```
+
+
+If you're connecting through a SOCKS5 Proxy just set the *:type* key to :socks5.
+
+```ruby
+    DoverToCalais::PROXY = 
+        :proxy => {
+           :host => 'www.myproxy.com',
+           :port => 8080,
+           :type => :socks5
+        }
+```
+
+## Documentation
+
+Comprehensive documentation can be found at http://rubydoc.info/gems/dover_to_calais.
+
+## Testing 
+
+A list of Cucumber features and scenarios can be found in the *features* directory. The list is far from exhaustive, so feel free to add your own scenarios and steps.
+
+To run the tests, there is already a rake task set up. Just type:
+
+    rake features API_KEY='my_api_key'
 
 ## Contributing
 
@@ -75,3 +385,13 @@ TODO: Write usage instructions here
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
+
+
+##Changelog
+
+   * **07-Oct-2013** Version: 0.1.0  
+Initial release
+   * **10-Feb-2014** Version: 0.1.1 
+Improved Response error message
+   * **10-Feb-2014** Version: 0.2.0  
+Added #analyse_this to public interface

data/Rakefile

@@ -1 +1,7 @@
 require "bundler/gem_tasks"
+require 'cucumber'
+require 'cucumber/rake/task'
+
+Cucumber::Rake::Task.new(:features) do |t|
+  t.cucumber_opts = "features --format pretty API_KEY=#{ENV['API_KEY']}"
+end

data/dover_to_calais.gemspec

@@ -7,20 +7,21 @@ Gem::Specification.new do |spec|
   spec.name          = "dover_to_calais"
   spec.version       = DoverToCalais::VERSION
   spec.authors       = ["Fred Heath"]
-  spec.email         = ["fred@bootstrap.me.uk"]
+  spec.email         = ["fred_h@bootstrap.me.uk"]
   spec.description   = %q{DoverToCalais allows the user to send a wide range of data sources (files & URLs)
                           to OpenCalais and receive asynchronous responses when OpenCalais has finished processing
                           the inputs. In addition, DoverToCalais enables the filtering of the response in order to
                           find relevant tags and/or tag values. }
   spec.summary       = %q{An easy-to-use wrapper round the OpenCalais semantic analysis web service. }
-  spec.homepage      = ""
+  spec.homepage      = "https://github.com/RedFred7/dover_to_calais"
   spec.license       = "MIT"
 
 
-  spec.add_runtime_dependency "nokogiri", "~>1.6.0"
-  spec.add_runtime_dependency "eventmachine", "~>1.0.3"
-  spec.add_runtime_dependency "em-http-request", "~>1.1.0"
-  spec.add_runtime_dependency "yomu", "~>0.1.9"
+  spec.add_runtime_dependency "nokogiri", "~> 1.6"
+  spec.add_runtime_dependency "eventmachine", "~> 1.0", ">= 1.0.3"
+  spec.add_runtime_dependency "em-http-request", "~> 1.1"
+  spec.add_runtime_dependency "yomu", "~> 0.1", ">= 0.1.9"
+
 
   spec.files         = `git ls-files`.split($/)
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
@@ -29,5 +30,7 @@ Gem::Specification.new do |spec|
 
 
   spec.add_development_dependency "bundler", "~> 1.3"
-  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rake", "~> 0"
+  spec.add_development_dependency "cucumber", "~> 1.3", ">= 1.3.8"
+  spec.add_development_dependency "rspec", "~> 2.14", ">= 2.14.1"
 end

data/features/data_sources.feature

@@ -0,0 +1,14 @@
+Feature: Able to handle wide range of data formats as input
+  Scenario Outline:  Processing various data-source formats
+    Given  the file <input>
+    When DoverToCalais processes this file
+    Then the output should have no errors
+
+  Examples:
+  | input |
+  |test_file_1.doc |
+  |test_file_1.html|
+  |test_file_1.odt|
+  |test_file_1.pdf|
+  |test_file_1.rtf|
+  |test_file_1.txt|

data/features/filtering.feature

@@ -0,0 +1,24 @@
+Feature: Ability to select certain OpenCalais entities based on certain conditions
+
+  Background:
+    Given the file 'test_file_1.txt' is successfully processed
+
+
+  Scenario:  Select all entities with a specific name
+    When I filter on {:entity => 'EmailAddress'}
+    Then the output should have 2 entries
+    And All entries should be named 'EmailAddress'
+
+  Scenario:  Select an entity with a specific value
+    When I filter on {:entity => 'Event', :value => 'Meeting'}
+    Then the output should have 1 entries
+    And All entries should be named 'Event'
+    And All entries should have the value 'Meeting'
+
+
+  Scenario:  Select an entity only if another entity with a specific value exists in the data source
+    When I filter on {:entity => 'Person', :given => {:entity => 'Event', :value => 'Meeting'}}
+    Then the output should have 2 entries
+    And All entries should be named 'Person'
+    And One entry should have the value 'Roger Kay'
+    And One entry should have the value 'David Bailey'

data/features/step_definitions/data_sources_steps.rb

@@ -0,0 +1,40 @@
+
+require 'nokogiri'
+require 'eventmachine'
+require 'em-http-request'
+require 'yomu'
+require 'rspec'
+require File.expand_path('../../../lib/dover_to_calais', __FILE__)
+
+
+# N.B Cucumber must be run with the Environment variable 'API_KEY' set
+# to the OpenCalais API Key value.
+
+
+Given(/^the file (\w+\.\w{3,4})$/) do |arg1|
+  puts arg1
+ @input = Dir.pwd + '/test/' + arg1
+  @output = nil
+end
+
+
+
+When(/^DoverToCalais processes this file$/) do
+  EM.run {
+
+    DoverToCalais::API_KEY =  ENV['API_KEY']
+    d1 =  DoverToCalais::Dover.new(@input)
+    d1.analyse_this
+    d1.to_calais do |response|
+      @output = response
+      EM.stop
+    end
+
+  }
+end
+
+
+
+Then(/^the output should have no errors$/) do
+   @output.error.should be_nil
+end

data/features/step_definitions/filtering_steps.rb

@@ -0,0 +1,60 @@
+require 'nokogiri'
+require 'eventmachine'
+require 'em-http-request'
+require 'yomu'
+require 'rspec'
+require File.expand_path('../../../lib/dover_to_calais', __FILE__)
+
+
+# N.B Cucumber must be run with the Environment variable 'API_KEY' set
+# to the OpenCalais API Key value.
+
+
+
+Given(/^the file '(\w+\.\w{3,4})' is successfully processed$/) do |file|
+
+  steps %{
+        Given  the file #{file}
+        When DoverToCalais processes this file
+        Then the output should have no errors
+        }
+
+end
+
+
+When(/^I filter on ({.+})/) do  |filter|
+  @filtered_output = @output.filter(eval(filter))
+
+end
+
+Then(/^the output should have (\d+) entries$/) do |item_num|
+  @filtered_output.size.should ==  item_num.to_i
+end
+
+Then(/^All entries should be named '(\w+)'$/) do  |name|
+  @filtered_output.each do |item|
+    item.name.should == name
+  end
+
+end
+
+
+And(/^All entries should have the value '(\w+)'$/) do |value|
+  @filtered_output.each do |item|
+    item.value.match(value).should_not be_nil
+  end
+end
+
+
+And(/^One entry should have the value '(\w+\s*\w+)'$/) do  |value|
+  found = false
+  @filtered_output.each do |item|
+    if item.value.match(value)
+       found =true
+      break
+    end
+  end
+
+  fail("couldn't match value '#{value}'") unless found
+end
+

data/lib/dover_to_calais.rb

@@ -130,14 +130,15 @@ module DoverToCalais
       node_count =  node.attribute('count').text.to_i if node.has_attribute?('count')
       node_normalized =  node.attribute('normalized').text if node.has_attribute?('normalized')
       node_importance = node.attribute('importance').text.to_i if node.has_attribute?('importance')
+      node_orig_value =  node.xpath('originalValue').text if node.name.eql?('SocialTag')
 
       ResponseItem.new(node.name,
-                       node.content,
+                       node.text,
                        node_relevance,
                        node_count,
                        node_normalized,
                        node_importance,
-                       node.xpath('originalValue').text )
+                       node_orig_value )
 
     end
 
@@ -172,7 +173,6 @@ module DoverToCalais
     def initialize(data_src)
       @data_src = data_src
       @callbacks = []
-      analyse_this
     end
 
 
@@ -202,7 +202,7 @@ module DoverToCalais
     # @return N/A
     def to_calais(&block)
       #fred rules ok
-      if @document
+      if !@error
         @callbacks << block
       else
         result = ResponseData.new nil, @error
@@ -211,12 +211,19 @@ module DoverToCalais
 
     end #method
 
+    # Gets the source text parsed. If the parsing is successful, the data source is POSTed to OpenCalais
+    # via an EventMachine request and a callback is set to manage the OpenCalais response.
+    # All Dover object callbacks are then called with the request result yielded to them. 
     #
+    # @param N/A
+    # @return a {Class ResponseData} object
     def analyse_this
 
       @document = get_src_data(@data_src)
       begin
-        if @document
+        if @document[0..2].eql?('ERR')
+          raise 'Invalid data source'
+        else
           response = nil
 
           connection_options = {:inactivity_timeout => 0}
@@ -242,36 +249,47 @@ module DoverToCalais
 
 
           http.callback do
-            http.response.match(/<OpenCalaisSimple>/) do |m|
-              response = Nokogiri::XML('<OpenCalaisSimple>' + m.post_match)  do |config|
-                #strict xml parsing, disallow network connections
-                config.strict.nonet
+
+            if http.response_header.status == 200
+              http.response.match(/<OpenCalaisSimple>/) do |m|
+                response = Nokogiri::XML('<OpenCalaisSimple>' + m.post_match)  do |config|
+                  #strict xml parsing, disallow network connections
+                  config.strict.nonet
+                end #block
               end #block
-            end #block
 
-            result =   response ? ResponseData.new(response, nil) :  ResponseData.new(nil,'ERR: unable to find <OpenCalaisSimple> tag in response data')
+              result =   response ?
+                        ResponseData.new(response, nil) :
+                        ResponseData.new(nil,'ERR: cannot find <OpenCalaisSimple> tag in response data - source invalid?')
+            else #non-200 response header
+              result = ResponseData.new nil, 
+                            "ERR: OpenCalais service responded with #{http.response_header.status} - response body: '#{http.response}'"
+            end 
+
             @callbacks.each { |c| c.call(result) }
+
           end  #callback
 
 
           http.errback do
-
-            result = ResponseData.new nil, "#{http.error}"
+            result = ResponseData.new nil, "ERR: #{http.error}"
             @callbacks.each { |c| c.call(result) }
           end  #errback
 
 
         end  #if
       rescue  Exception=>e
-        puts "ERR: #{e}"
-        exit 0
+        #result = ResponseData.new nil,  "ERR: #{e}"
+        #@callbacks.each { |c| c.call(result) }
+        @error = "ERR: #{e}"
       end
 
     end  #method
 
 
-    public :to_calais
-    private :get_src_data, :analyse_this
+    alias_method :analyze_this, :analyse_this
+    public :to_calais, :analyse_this
+    private :get_src_data
 
 
   end  #class

data/lib/dover_to_calais/version.rb

@@ -1,3 +1,3 @@
 module DoverToCalais
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/test/test_file_1.html

@@ -0,0 +1,36 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
+<HTML>
+<HEAD>
+	<META HTTP-EQUIV="CONTENT-TYPE" CONTENT="text/html; charset=utf-8">
+	<TITLE></TITLE>
+	<META NAME="GENERATOR" CONTENT="LibreOffice 3.5  (Linux)">
+	<META NAME="CREATED" CONTENT="0;0">
+	<META NAME="CHANGED" CONTENT="0;0">
+	<STYLE TYPE="text/css">
+	<!--
+		@page { margin: 2cm }
+		P { margin-bottom: 0.21cm }
+		P.western { so-language: en-GB }
+		PRE.western { so-language: en-GB }
+		PRE.cjk { font-family: "WenQuanYi Micro Hei", monospace }
+		PRE.ctl { font-family: "Lohit Hindi", monospace }
+	-->
+	</STYLE>
+</HEAD>
+<BODY LANG="en-GB" DIR="LTR">
+<PRE CLASS="western">Tensleep Corporation (Other OTC:TENS.PK - News) (&quot;Tensleep&quot;) announced that with the acquisition of XSTV Media, Inc. (&quot;XSTV&quot;),
+                                                                                                         it will become an online independent sports company. The transaction is
+to close on or before September 15, 2007. Tensleep will, by the end of
+this week or early next week, call a special meeting of shareholders to
+approve the change name to &quot;XSTV Corporation.&quot;
+
+David Bailey, an analyst at Gerard Klauer Mattison who can be contacted at david at bailey dot com, said such cuts &quot;could include head count reductions.&quot;
+Layoffs to some degree are inevitable, said IDC analyst Roger Kay. For years,
+                                                                       the company enjoyed a lower cost structure than other PC makers because
+it sold computers directly.
+                      International Star Inc. (info@ITSR.com - OTC BB: ILST) announced that the annual meeting of
+shareholders of International Star Inc. will be held on May 19, 2008,
+                                                            at 3:00 p.m. (local time) at The Hilton Hotel, 104 Market Street,
+                                                                                                                      Shreveport, La., 71101.   </PRE>
+</BODY>
+</HTML>

data/test/test_file_1.rtf

@@ -0,0 +1,54 @@
+{\rtf1\ansi\deff3\adeflang1025
+{\fonttbl{\f0\froman\fprq2\fcharset0 Times New Roman;}{\f1\froman\fprq2\fcharset2 Symbol;}{\f2\fswiss\fprq2\fcharset0 Arial;}{\f3\froman\fprq2\fcharset128 Times New Roman;}{\f4\fswiss\fprq2\fcharset128 Arial;}{\f5\fmodern\fprq1\fcharset128 Droid Sans Mono;}{\f6\fnil\fprq2\fcharset128 Droid Sans;}{\f7\fmodern\fprq1\fcharset128 WenQuanYi Micro Hei;}{\f8\fnil\fprq2\fcharset128 Lohit Hindi;}{\f9\fnil\fprq0\fcharset128 Lohit Hindi;}{\f10\fmodern\fprq1\fcharset128 Lohit Hindi;}}
+{\colortbl;\red0\green0\blue0;\red0\green0\blue128;\red128\green0\blue0;\red128\green128\blue128;}
+{\stylesheet{\s0\snext0\nowidctlpar{\*\hyphen2\hyphlead2\hyphtrail2\hyphmax0}\cf0\hich\af6\langfe2052\dbch\af8\afs24\alang1081\loch\f3\fs24\lang2057 Normal;}
+{\*\cs15\snext15 Footnote Characters;}
+{\*\cs16\snext16 Endnote Characters;}
+{\*\cs17\snext17\cf2\ul\ulc0\langfe255\alang255\lang255 Internet Link;}
+{\*\cs18\snext18\cf3\ul\ulc0\langfe255\alang255\lang255 Visited Internet Link;}
+{\s19\sbasedon0\snext20\sb240\sa120\keepn\hich\af6\dbch\af8\afs28\loch\f4\fs28 Heading;}
+{\s20\sbasedon0\snext20\sb0\sa120 Text body;}
+{\s21\sbasedon20\snext21\sb0\sa120\dbch\af9 List;}
+{\s22\sbasedon0\snext22\sb120\sa120\noline\i\dbch\af9\afs24\ai\fs24 Caption;}
+{\s23\sbasedon0\snext23\noline\dbch\af9 Index;}
+{\s24\sbasedon0\snext24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20 Preformatted Text;}
+{\s25\sbasedon0\snext25\li567\ri0\lin567\rin0\fi0 List Contents;}
+}{\info{\creatim\yr0\mo0\dy0\hr0\min0}{\revtim\yr0\mo0\dy0\hr0\min0}{\printim\yr0\mo0\dy0\hr0\min0}{\comment LibreOffice}{\vern3500}}\deftab709
+
+{\*\pgdsctbl
+{\pgdsc0\pgdscuse195\pgwsxn11906\pghsxn16838\marglsxn1134\margrsxn1134\margtsxn1134\margbsxn1134\pgdscnxt0 Default;}}
+\formshade\paperh16838\paperw11906\margl1134\margr1134\margt1134\margb1134\sectd\sbknone\sectunlocked1\pgndec\pgwsxn11906\pghsxn16838\marglsxn1134\margrsxn1134\margtsxn1134\margbsxn1134\ftnbj\ftnstart1\ftnrstcont\ftnnar\aenddoc\aftnrstcont\aftnstart1\aftnnrlc
+\pgndec\pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch\loch
+Tensleep Corporation (Other OTC:TENS.PK - News) ("Tensleep") announced that with the acquisition of XSTV Media, Inc. ("XSTV"),}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch
+                                                                                                         }{\rtlch \ltrch\loch
+it will become an online independent sports company. The transaction is}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch\loch
+to close on or before September 15, 2007. Tensleep will, by the end of}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch\loch
+this week or early next week, call a special meeting of shareholders to}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch\loch
+approve the change name to "XSTV Corporation."}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20\rtlch \ltrch\loch
+
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch\loch
+David Bailey, an analyst at Gerard Klauer Mattison who can be contacted at david at bailey dot com, said such cuts "could include head count reductions."}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch\loch
+Layoffs to some degree are inevitable, said IDC analyst Roger Kay. For years,}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch
+                                                                       }{\rtlch \ltrch\loch
+the company enjoyed a lower cost structure than other PC makers because}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch\loch
+it sold computers directly.}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch
+                      }{\rtlch \ltrch\loch
+International Star Inc. (info@ITSR.com - OTC BB: ILST) announced that the annual meeting of}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch\loch
+shareholders of International Star Inc. will be held on May 19, 2008,}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch
+                                                            }{\rtlch \ltrch\loch
+at 3:00 p.m. (local time) at The Hilton Hotel, 104 Market Street,}
+\par \pard\plain \s24\sb0\sa0\hich\af7\dbch\af10\afs20\loch\f5\fs20{\rtlch \ltrch
+                                                                                                                      }{\rtlch \ltrch\loch
+Shreveport, La., 71101.   }
+\par }

data/test/test_file_1.txt

@@ -0,0 +1,14 @@
+Tensleep Corporation (Other OTC:TENS.PK - News) ("Tensleep") announced that with the acquisition of XSTV Media, Inc. ("XSTV"),
+                                                                                                         it will become an online independent sports company. The transaction is
+to close on or before September 15, 2007. Tensleep will, by the end of
+this week or early next week, call a special meeting of shareholders to
+approve the change name to "XSTV Corporation."
+
+David Bailey, an analyst at Gerard Klauer Mattison who can be contacted at david at bailey dot com, said such cuts "could include head count reductions."
+Layoffs to some degree are inevitable, said IDC analyst Roger Kay. For years,
+                                                                       the company enjoyed a lower cost structure than other PC makers because
+it sold computers directly.
+                      International Star Inc. (info@ITSR.com - OTC BB: ILST) announced that the annual meeting of
+shareholders of International Star Inc. will be held on May 19, 2008,
+                                                            at 3:00 p.m. (local time) at The Hilton Hotel, 104 Market Street,
+                                                                                                                      Shreveport, La., 71101.   

metadata

@@ -1,84 +1,86 @@
 --- !ruby/object:Gem::Specification
 name: dover_to_calais
 version: !ruby/object:Gem::Version
-  version: 0.1.0
-  prerelease: 
+  version: 0.2.0
 platform: ruby
 authors:
 - Fred Heath
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-10-04 00:00:00.000000000 Z
+date: 2014-02-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nokogiri
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 1.6.0
+        version: '1.6'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 1.6.0
+        version: '1.6'
 - !ruby/object:Gem::Dependency
   name: eventmachine
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
+    - - ! '>='
       - !ruby/object:Gem::Version
         version: 1.0.3
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
+    - - ! '>='
       - !ruby/object:Gem::Version
         version: 1.0.3
 - !ruby/object:Gem::Dependency
   name: em-http-request
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 1.1.0
+        version: '1.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 1.1.0
+        version: '1.1'
 - !ruby/object:Gem::Dependency
   name: yomu
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.1'
+    - - ! '>='
       - !ruby/object:Gem::Version
         version: 0.1.9
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.1'
+    - - ! '>='
       - !ruby/object:Gem::Version
         version: 0.1.9
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -86,7 +88,6 @@ dependencies:
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
@@ -94,26 +95,64 @@ dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: cucumber
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.3.8
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.3.8
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2.14'
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.14.1
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2.14'
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.14.1
 description: ! "DoverToCalais allows the user to send a wide range of data sources
   (files & URLs)\n                          to OpenCalais and receive asynchronous
   responses when OpenCalais has finished processing\n                          the
   inputs. In addition, DoverToCalais enables the filtering of the response in order
   to\n                          find relevant tags and/or tag values. "
 email:
-- fred@bootstrap.me.uk
+- fred_h@bootstrap.me.uk
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -125,32 +164,51 @@ files:
 - README.md
 - Rakefile
 - dover_to_calais.gemspec
+- features/data_sources.feature
+- features/filtering.feature
+- features/step_definitions/data_sources_steps.rb
+- features/step_definitions/filtering_steps.rb
 - lib/dover_to_calais.rb
 - lib/dover_to_calais/ontology.rb
 - lib/dover_to_calais/version.rb
-homepage: ''
+- test/test_file_1.doc
+- test/test_file_1.html
+- test/test_file_1.odt
+- test/test_file_1.pdf
+- test/test_file_1.rtf
+- test/test_file_1.txt
+homepage: https://github.com/RedFred7/dover_to_calais
 licenses:
 - MIT
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 2.2.0
 signing_key: 
-specification_version: 3
+specification_version: 4
 summary: An easy-to-use wrapper round the OpenCalais semantic analysis web service.
-test_files: []
+test_files:
+- features/data_sources.feature
+- features/filtering.feature
+- features/step_definitions/data_sources_steps.rb
+- features/step_definitions/filtering_steps.rb
+- test/test_file_1.doc
+- test/test_file_1.html
+- test/test_file_1.odt
+- test/test_file_1.pdf
+- test/test_file_1.rtf
+- test/test_file_1.txt