datasift 3.0.0.beta2 → 3.0.0.beta4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b56e8834e9e227fc90f26c08fac16521d50fc2ab
-  data.tar.gz: ab50e42f90b9aeaca168af3c7174160895cb558d
+  metadata.gz: f5a8e4b6cddffbd3b5608c4fe831bdf87213c772
+  data.tar.gz: e05742fa5831660e60c8395b812025d40bbdf0dd
 SHA512:
-  metadata.gz: dbd1f4890b2b0693afa5255e75ec6e7a8f40f5b923d1c6c034f5a3e7540965c432c25e50e29a138f45d964cf5967a90a1ac50f5f28223d14cc429313ca53e9cc
-  data.tar.gz: 3ee310b5822002750a6f517e2ef3ae218628753e4f4e55167f4cbc591542be962724db03fd06954ff56a5e51f3b6243ee7e1104e00014b3b9a9feecf201eff10
+  metadata.gz: 0a03568fe8b5bc5a650b8694f3763154ffd0c10fd13924221dc29cadfcc3befdd5a68b4ae325817c39fa84aea1b84946111230f3f73363d4682ae45f4e1445f9
+  data.tar.gz: f626e022353a188ba41d76d68680bacb60c92bd716921362c5c51400efbbe876ac6313b93caf7e7d9b1463bc53f4a4a339b11167ee781af34ca3cb7840b7c5a1

data/.gitignore

@@ -1,7 +1,7 @@
 .DS_Store
 sftp-config.json
-Gemfile.lock
 *.iml
 .idea/
 rdoc/
-pkg/*
+.bundle/
+Gemfile.lock

data/CHANGELOG.md

@@ -1,16 +1,6 @@
 CHANGELOG
 ================================
 
-v.3.0.0.beta2 (2013-12-02)
--------------------------
-
-Minor fixes to the 3.0.0 beta release
-
-* Requires websocket_td v.0.0.4+
-* Tests now run with Rake
-* Fixes a broken reference to the VERSION file
-* Renamed 'tests' directory to 'test'
-
 v.3.0.0.beta (2013-11-07)
 -------------------------
 

data/MIGRATING_TO_V.3.0.0.md

@@ -0,0 +1,262 @@
+MIGRATING TO V.3.0.0
+================================
+
+Breaking Changes
+----------------
+Earlier versions of the DataSift library are incompatible with 3.x.x. 3.0.0 is a complete re-design. In order to continue delivering better features and performance some architectural changes have been made which make backwards compatibility very difficult and in some cases impractical.
+
+Features
+--------
+* Live streaming now uses multi-threaded WebSockets, so you can subscribe and unsubscribe from stream hashes.
+* This update ensures that the Ruby client library now supports all API features that were missing from prior versions of the client.
+* This includes adding support for Historics Previews, and the [Pull Connector](http://dev.datasift.com/blog/pullingdata).
+* Previous versions made API requests through a ```DataSift::User``` object. From v.3.0.0, we have moved to a more semantically correct ```DataSift::Client``` object.
+
+Code
+====
+
+## Authentication
+From v.3.0.0 of the Ruby client, we have dropped the concept of the ```DataSift::User``` object, and now use a ```DataSift::Client``` object for all API calls.
+
+### Authentication: < 3.0.0
+
+```ruby
+config = YAML::load(File.open(File.join(File.dirname(__FILE__), '..', 'config.yml')))
+user = DataSift::User.new(config['username'], config['api_key'])
+```
+
+### Authentication: 3.0.0+
+From v.3.0.0+ you begin by providing a configuration object to the DataSift client. From here the client instance gives you access to the rest of the DataSift API. This is organized in a similar way to the DataSift [REST API documentation](http://dev.datasift.com/docs/rest-api) except where it didn't make sense to do so.
+
+```ruby
+@config = {:username => 'DATASIFT_USERNAME', :api_key => 'DATASIFT_API_KEY', :enable_ssl => true}
+@datasift = DataSift::Client.new(@config)
+```
+
+
+## Core
+* @datasift.valid? csdl
+* @datasift.compile csdl
+* @datasift.usage [period]
+* @datasift.dpu hash
+* @datasift.balance
+
+Below are examples of how to compile, then check the DPU cost of a CSDL statement, then check your API usage. These examples both assume you have correctly authenticated with the API.
+
+### Core: < 3.0.0
+```ruby
+csdl = 'interaction.content contains "datasift"'
+definition = user.createDefinition(csdl)
+dpu = definition.getDPUBreakdown()
+usage = user.getUsage()
+```
+
+### Core: 3.0.0+
+```ruby
+csdl = interaction.content contains "datasift"'
+stream = @datasift.compile csdl
+dpu = @datasift.dpu stream[:data][:hash]
+usage = @datasift.usage
+```
+
+
+## Live Streaming
+The Live Streaming API is now accessed via WebSockets using the [websocket_td](https://github.com/zcourts/websocket-td) gem, rather than streaming over HTTP. This allows us to use the ```stream.subscribe(hash, on_message)``` and ```stream.unsubscribe hash``` methods to asynchronously subscribe and unsubscribe from streams, while still streaming data.
+Please note, the examples below include only the mandatory callback methods, and do not include any additional error handling. The examples included in the client library itself do include some very basic error handling.
+
+### Core: < 3.0.0
+```ruby
+consumer.consume(true) do |interaction|
+  if interaction
+    puts interaction.to_s
+  end
+end
+```
+
+
+### Core: 3.0.0+
+```ruby
+def stream(hash)
+  on_delete  = lambda { |stream, m| puts m }
+  on_error   = lambda { |stream, e| puts "An error has occurred: #{message}" }
+  on_message = lambda { |message, stream, hash| puts message }
+
+  on_datasift_message = lambda do |stream, message, hash|
+    puts "DataSift Message #{hash} ==> #{message}"
+  end
+
+  conn = DataSift::new_stream(@config, on_delete, on_error, on_open, on_close)
+  conn.on_datasift_message = on_datasift_message
+  conn.stream.read_thread.join
+end
+```
+
+#### on_delete event
+on_delete is called when your stream receives a [delete notification](http://dev.datasift.com/docs/resources/twitter-deletes) from Twitter, notifying you that a Tweet you may have received has been deleted.
+
+#### on_error event
+on_error is called in cases where where an exception occurs during streaming.
+
+#### on_message event
+on_message is called when we receive [user status messages](http://dev.datasift.com/docs/resources/twitter-user-status-messages) from Twitter.
+
+
+## Push
+* @datasift.push.valid? @params
+* @datasift.create @params
+* @datasift.push.pause subscription_id
+* @datasift.push.resume subscription_id
+* @datasift.push.update @params.merge({:id => subscription_id, :name => 'Updated name'})
+* @datasift.push.stop subscription_id
+* @datasift.push.delete subscription_id
+* @datasift.push.logs
+* @datasift.push.get_by_subscription subscription_id
+* @datasift.push.get
+* @datasift.pull
+
+Below are some simple examples, showing you how to create, pause, resume, update, get, stop then delete a Push Subscription:
+
+### Push: < 3.0.0
+```ruby
+definition = env.user.createDefinition(csdl)
+
+pushdef = env.user.createPushDefinition()
+pushdef.output_type = output_type
+
+# Add your output parameters to your Push Definition
+while env.args.size() > 0
+  k, v = env.args.shift.split('=', 2)
+  pushdef.output_params[k] = v
+end
+
+sub = pushdef.subscribeDefinition(definition, name)
+
+sub.pause()
+sub.resume()
+sub.save()
+sub.stop()
+sub.delete()
+```
+
+
+### Push: 3.0.0+
+```ruby
+subscription = create_push(hash)
+subscription_id = subscription[:data][:id]
+
+@datasift.push.pause subscription_id
+@datasift.push.resume subscription_id
+@datasift.push.update @params.merge({:id => subscription_id, :name => 'New name'})
+@datasift.push.get_by_subscription subscription_id
+@datasift.push.stop subscription_id
+@datasift.push.delete subscription_id
+```
+
+
+## Historics
+* @datasift.historics.prepare(hash, start, end, 'My ruby historics')
+* @datasift.historics.start id
+* @datasift.historics.stop id
+* @datasift.historics.status(start, end_time)
+* @datasift.historics.update(id, 'The new name of my historics')
+* @datasift.historics.delete id
+* @datasift.historics.get_by_id id
+* @datasift.historics.get
+
+Below are some simple examples demonstrating how to check the status of the Historics archive for a given timeframe, prepare an Historic, then start, get, stop and delete the Historic.
+
+### Historics: < 3.0.0
+```ruby
+start_time = Time.now.to_i - 7200
+end_time   = start + 3600
+
+# /historics/status not implemented in < 3.0.0
+definition = env.user.createDefinition(csdl)
+historic = definition.createHistoric(start_time, end_time, sources, sample, name)
+
+historic.prepare()
+historic.start()
+user.getHistoric(historic.id)
+historic.stop()
+historic.delete()
+```
+
+### Historics: 3.0.0+
+```ruby
+start_time = Time.now.to_i - 7200
+end_time   = start + 3600
+
+@datasift.historics.status(start_time, end_time)
+
+historics = @datasift.historics.prepare(hash, start_time, end_time, 'My Historic')
+id = historics[:data][:id]
+
+create_push(id, true)
+
+@datasift.historics.start id
+@datasift.historics.get_by_id id
+@datasift.historics.stop id
+@datasift.historics.delete id
+```
+
+## Historics Preview
+* @datasift.historics_preview.create(hash, parameters, start, end)
+* @datasift.historics_preview.get id
+
+Historics preview was not available before v.3.0.0. The example below demonstrates how to create, then get the results of an Historics preview:
+
+### Hisotrics Preview: 3.0.0+
+```ruby
+parameters = 'interaction.author.link,targetVol,hour;interaction.type,freqDist,10'
+start      = Time.now.to_i - (3600 * 48) # 48hrs ago
+source     = @datasift.historics_preview.create(hash, parameters, start)
+@datasift.historics_preview.get source[:data][:id]
+```
+
+
+## Managed Sources
+* @datasift.managed_source.create(source_type, name, parameters, resources, auth)
+* @datasift.managed_source.update(id, source_type, name, parameters, resources, auth)
+* @datasift.managed_source.delete id
+* @datasift.managed_source.log id
+* @datasift.managed_source.get id
+* @datasift.managed_source.stop id
+* @datasift.managed_source.start id
+
+Below is a Managed Sources example, using each of the Managed Sources API endpoints:
+
+### Managed Sources < 3.0.0
+```ruby
+parameters = {:likes => true, :comments => true}
+resources  = [{:parameters => {:type => 'tag', :value => 'coffee'}}]
+auth       = [{:parameters => {:value => '10942112.1fb234f.8713bcf4d5b44ece801022f6fa4b9e1b'}}]
+
+user   = DataSift::User.new(config['username'], config['api_key'], false)
+source = user.createManagedSource(:source_type => 'instagram', :name => '#Coffee Pics', :parameters => parameters, :resources => resources, :auth => auth)
+
+source.start
+user.getManagedSource(source.managed_source_id)
+user.getManagedSourcesLog(source.managed_source_id)
+source.stop
+source.delete
+```
+
+### Managed Sources 3.0.0+
+```ruby
+parameters = {:likes => true, :comments => true}
+resources  = [{:parameters => {:type => 'tag', :value => 'coffee'}}]
+auth       = [{:parameters => {:value => '10942112.1fb234f.8713bcf4d5b44ece801022f6fa4b9e1b'}}]
+
+source = @datasift.managed_source.create('instagram', '#Coffee Pics', parameters, resources, auth)
+id     = source[:data][:id]
+
+@datasift.managed_source.start id
+source = @datasift.managed_source.get id
+# Note that in the line below, we pass the auth object returned from a /source/get call back into the /source/update statement. Passing the original auth object will fail
+@datasift.managed_source.update(id, 'instagram', 'Updated source name', parameters, resources, source[:data][:auth])
+@datasift.managed_source.log id
+@datasift.managed_source.stop id
+@datasift.managed_source.delete id
+```
+
+

data/VERSION

@@ -1 +1 @@
-3.0.0.beta2
+3.0.0.beta4

data/datasift.gemspec

@@ -12,14 +12,15 @@ Gem::Specification.new do |s|
   s.rubygems_version = %q{1.3.6}
   s.required_rubygems_version = Gem::Requirement.new(">= 1.3.6") if s.respond_to? :required_rubygems_version=
 
-  s.add_runtime_dependency('rest-client', '~> 1.6.3')
-  s.add_development_dependency('multi_json', '~> 1.8.0')
+  s.add_runtime_dependency('rest-client', '~> 1.6.7')
+  s.add_runtime_dependency('multi_json', '~> 1.8.0')
   s.add_runtime_dependency('websocket-td', '~> 0.0.4')
   s.add_development_dependency('rdoc', '> 0')
+  s.add_development_dependency('webmock', '~> 1.17.1')
   s.add_development_dependency('shoulda', '~> 2.11.3')
   s.add_development_dependency('test-unit', '>= 2.5.5')
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
   s.require_paths = ["lib"]
-end
+end

data/examples/auth.rb

@@ -1,10 +1,14 @@
 class DataSiftExample
-  require File.dirname(__FILE__) + '/../lib/datasift'
+  require '../lib/datasift'
 
   def initialize
-    @username = 'DATASIFT_USERNAME'
-    @api_key  = 'DATASIFT_API_KEY'
-    @config   ={:username => @username, :api_key => @api_key, :enable_ssl => false}
+    #only SSLv3 and TLSv1 currently supported, TLSv1 preferred
+    # this is fixed in REST client and is scheduled for the 1.7.0 release
+    # see https://github.com/rest-client/rest-client/pull/123
+    OpenSSL::SSL::SSLContext::DEFAULT_PARAMS[:ssl_version] = 'TLSv1'
+    @username = 'zcourts'
+    @api_key  ='9dc2e16075de005b1a23444fbc6dd269'
+    @config   ={:username => @username, :api_key => @api_key, :enable_ssl => true}
     @params   = {
         :output_type                       => 's3',
         'output_params.bucket'             => 'apitests',
@@ -41,4 +45,4 @@ class DataSiftExample
     puts 'Create push => ' + subscription.to_s
     subscription
   end
-end
+end

data/examples/cli.sh

@@ -0,0 +1,155 @@
+#!/bin/bash
+command -v jq >/dev/null 2>&1 || { echo >&2 "jq command must be available. See http://stedolan.github.io/jq/.  Aborting."; exit 1; }
+
+set -e
+
+DEFAULT_API='api.datasift.com'
+
+CC=${1:$CC}
+DU=${2:-$DU}
+DK=${3:-$DK}
+API=${4:-$DEFAULT_API}
+
+function ds(){
+    ${CC} -a ${DU} ${DK} --u ${API} "$@" #| jq .
+}
+
+# core API - validate our hash, compile it, check our usage, dpu and balance
+echo 'Validating CSDL'
+csdl='interaction.content contains "music"'
+valid=$(ds -e core -c validate -p csdl "$csdl" | jq .status)
+
+if [ ${valid} != 200 ]; then
+    echo "Validating CSDL failed"
+    echo ${valid}
+    exit -1
+fi
+
+echo 'Compiling'
+hash=$(ds -e core -c compile -p csdl "$csdl" | jq -r .body.hash)
+echo "Compiled and got $hash"
+
+echo 'Usage :'
+ds -e core -c usage | jq .
+
+echo 'DPU :'
+ds -e core -c dpu -p hash $hash | jq .
+
+echo 'Balance :'
+ds -e core -c usage | jq .
+
+echo 'Preparing Historic query'
+end=`expr $(date +%s) - 7200`
+start=`expr $end - 3600`
+
+historic=$(ds -e historics -c prepare -p start ${start} -p end ${end} -p name "Historics CLI @ $start" -p hash ${hash})
+echo ${historic} | jq .
+historic_id=$(echo ${historic} | jq -r .body.id)
+echo "Historic created with ID $historic_id"
+
+echo 'Validating Push subscription'
+push_v=$(ds -e push -c validate -p playback_id ${historic_id} -p name "Playback CLI @ $start" -p output_type http \
+-p output_params.method post -p output_params.url 'http://ec2-50-19-63-138.compute-1.amazonaws.com:80' \
+-p output_params.delivery_frequency 0 -p output_params.max_size 102400 -p output_params.auth.type none \
+-p output_params.verify_ssl false -p output_params.use_gzip true)
+push_status=$(echo ${push_v} | jq .status)
+echo ${push_v} | jq .
+
+if [ ${push_status} != 200 ]; then
+    echo "Validating Push subscription failed"
+    exit -1
+fi
+
+echo 'Creating Push from Historic'
+push=$(ds -e push -c create -p playback_id ${historic_id} -p name "Playback CLI @ $start" -p output_type http \
+-p output_params.method post -p output_params.url 'http://ec2-50-19-63-138.compute-1.amazonaws.com:80' \
+-p output_params.delivery_frequency 0 -p output_params.max_size 102400 -p output_params.auth.type none \
+-p output_params.verify_ssl false -p output_params.use_gzip true)
+
+echo "Created push subscription for historic"
+echo ${push} | jq .
+push_id=$(echo ${push} | jq -r .body.id)
+
+echo 'Starting Historic query'
+ds -e historics -c start -p id ${historic_id} | jq .
+
+echo 'Getting Historic status'
+ds -e historics -c status -p start ${start} -p end ${end} | jq .
+
+echo 'Getting Historics'
+ds -e historics -c get -p id ${historic_id} | jq .
+
+echo 'Updating historic'
+ds -e historics -c update -p id ${historic_id} -p name "Some name @ $start - CLI" | jq .
+
+echo 'Getting push'
+ds -e push -c get -p id ${push_id} | jq .
+
+echo 'Getting push logs'
+ds -e push -c log -p id ${push_id} | jq .
+
+echo 'Pausing push'
+ds -e push -c pause -p id ${push_id} | jq .
+
+echo 'Resuming push'
+ds -e push -c resume -p id ${push_id} | jq .
+
+echo 'Stopping Historic'
+ds -e historics -c stop -p id ${historic_id} | jq .
+
+echo 'Deleting Historic'
+ds -e historics -c delete -p id ${historic_id} | jq .
+
+echo 'Stopping push'
+ds -e push -c stop -p id ${push_id} | jq .
+
+echo 'Deleting push'
+ds -e push -c delete -p id ${push_id} | jq .
+#todo update push, pull
+
+echo "Attempting to create a Historics preview"
+preview=$(ds -e preview -c create -p start ${start} -p end ${end} -p hash ${hash} -p sources twitter \
+-p parameters 'interaction.author.link,targetVol,hour;interaction.type,freqDist,10')
+
+echo ${preview} | jq .
+preview_id=$(echo ${preview} | jq -r .body.id)
+
+
+echo "Getting the preview we created"
+ds -e preview -c get -p id ${preview_id} | jq .
+
+echo "Creating a managed source"
+source=$(ds -e managed_sources -c create -p source_type instagram -p name api \
+     -p auth "[{\"parameters\":{\"value\":\"$start$end\"}}]" \
+     -p resources '[{"parameters":{"value":"cats","type":"tag"}}]' \
+     -p parameters '{"comments":true,"likes":false}')
+echo ${source}
+source_id=$(echo ${source}| jq -r .body.id)
+echo ${source_id}
+
+echo "Starting managed source"
+ds -e managed_sources -c start -p source_id ${source_id} | jq .
+
+echo "Getting managed sources"
+ds -e managed_sources -c get | jq .
+
+echo "Getting Instagram sources"
+ds -e managed_sources -c get -p source_type instagram | jq .
+
+echo "Getting Facebook page sources"
+ds -e managed_sources -c get -p source_type facebook_page | jq .
+
+echo "Getting page 2 of instagram sources"
+ds -e managed_sources -c get -p source_type instagram -p page 2 | jq .
+
+echo "Getting source for $source_id"
+ds -e managed_sources -c get -p source_id ${source_id} | jq .
+
+echo "Getting logs for source $source_id"
+ds -e managed_sources -c log -p source_id ${source_id} -p page 2 -p per_page 1 | jq .
+
+echo "Stopping managed source"
+ds -e managed_sources -c stop -p source_id ${source_id} | jq .
+
+echo "Deleting managed source $source_id"
+ds -e managed_sources -c delete -p source_id ${source_id} | jq .