quandl 0.2.27 → 0.3.0

checksums.yaml

@@ -1,15 +1,15 @@
 ---
 !binary "U0hBMQ==":
   metadata.gz: !binary |-
-    OWE2ODc2YmIxMGQyNTJlNTUzY2Q1ZTgyNzI2ZjFhMTdhYmI2NTE1Mw==
+    MmI3MGNiMmJjYzNkNzU3ZWQzMjE3NTI0ZTlhNmZlMjMxMGNiMTk5NQ==
   data.tar.gz: !binary |-
-    NTc0MDA0MzU0YTY5ODJjNmQwYzM1NGI2NDEzYjY0YmQwNDUxYTc2Nw==
+    NzUwMGIyYjkxZGRkYTk5ZTc0ZTk3YTFjNzY5YzMzZDQ5OTY3ODNlYg==
 SHA512:
   metadata.gz: !binary |-
-    MmNiMzVmODUxNWE1YWJiNjUxY2IwNDRmZDM0Nzk5YzdjNzEwNzgxMzk2YjYx
-    ZGNlZTA1OWYzZWJhMDVkNmY1MDg0MDFjMzQ5NDRiN2FiZTA4NDdmZTNlNTZh
-    ZTE2NTAxNWIxZWFjZTg3MzQ1NDQzMTI5MGQ2MWFjZWJkZjhhM2E=
+    YmJjMTA3OGI1ODJhZWRjY2VkOWFkNzgxMTM4MTA5NmE0NzA1MDExZDFmNTBh
+    MmUzNTgyMzM0OGIxYWY5MmJlNjk0ODIxZDhmYTNjMTRlYzE4NzA2NzAzNWIx
+    NjU4ODMyOTgwZWYyYmU0OWQ4ZTg1ZWE3Y2NmZjk0YmFjZmM1NWQ=
   data.tar.gz: !binary |-
-    YjVjMjhhZmFkMjNlYmY2MWYyYmI3MzRiNzhhYjg2YzI3Y2M3NjllMDI4NzFl
-    NTNiNTVjOWNjYjgzZjg0NjZiN2I0MTgwOGM2MTcxMmEzOTI3MTM0ZWVhMjli
-    YjBmM2IwZDgyMjMyYWI4NDZmM2VkNjIzMjA1YTY4NTYyNmRiNzQ=
+    NWJiNzYyYWE5MmVkOGQxNGUwYjE5MTMzZGRiZTAxMzBlMTIxNmJkNDI4NGQz
+    MjkxMjFkYzlmNzljNTJhZjlkOWQyMmVkN2JhMTE4YzdiZTYyNGRjOGRjOGM4
+    ZTg0M2Q3YTM0MmQxY2U3ODE4NWYxYzNkN2MwY2RhM2ZkY2NhNTg=

data/README.md

@@ -1,16 +1,33 @@
-# Getting Started
+[![Code Climate](https://codeclimate.com/github/quandl/quandl_command.png)](https://codeclimate.com/github/quandl/quandl_command)
+
+# Quandl Toolbelt
+
+** The Quandl Toolbelt is currently in ALPHA TESTING.  You are nevertheless welcome to try it. **
 
 The Quandl toolbelt enables you to create and maintain time series data on Quandl.com.  The Quandl toolbelt is extremly simple to understand and use.  (We use it to maintain the 8 million datasets currently on the site.)
 
 ## Installation
 
-### Mac or Linux
+### Mac 
+
+**[quandl-toolbelt.pkg](http://s3.amazonaws.com/quandl-command/quandl-toolbelt.pkg)**
 
-    curl -s http://s3.amazonaws.com/quandl-command/install.sh | bash
 
 ### Windows
 
-[Executable Installer](http://s3.amazonaws.com/quandl-command/Quandl+Setup.exe)
+**[Quandl Setup.exe]( http://s3.amazonaws.com/quandl-command/Quandl+Setup.exe)**
+
+
+### Gemfile
+
+In your Gemfile, add:
+
+```ruby
+gem "quandl"
+```
+
+
+
 
 ## Login
 
@@ -23,15 +40,10 @@ Once the toolbelt is installed, the next step is to login to Quandl:
     username:   tammer1
     email:      tammer@quandl.com
 
-If you have a Quandl password, you can do this:
+If you have a Quandl password, you can also use `quandl login --method password`.  (You might not have a Quandl password if you log in using Github, Google, Linkedin or Twitter)
+
 
-    $ quandl login --method password
-    quandl login
-    Username or Email: Tammer1
-    Password:  ******
-    You've successfully authorized Tammer1!
 
-(You might not have a Quandl password if you log in using Github, Google, Linkedin or Twitter)
 
 ## Create a Dataset
 
@@ -52,6 +64,9 @@ Now send it to Quandl:
 
 You just created a dataset on Quandl.com: `www.quandl.com/<your-username>/AAA`
 
+
+
+
 ## Update a Dataset
 
 The new dataset will now exist on Quandl forever.  You can send new data and/or update metadata whenever you want.  For example, create [data_update.csv](https://raw2.github.com/quandl/toolbelt_help/master/data_udpate.csv):
@@ -74,6 +89,9 @@ Notice that the dataset now has three rows and a new description:
 You might also want to edit it more at [www.quandl.com/edit/<your user name>/FOO](#)
 -->
 
+
+
+
 ## Delete a Dataset
 
 You can delete the dataset:
@@ -86,6 +104,7 @@ You can delete the dataset:
 
 
 
+
 ## Scrapers and Other Data Producing Programs
 
 As long as your program outputs Quandl flavored CSV as above, it is ready for use with the Quandl toolbelt. Consider this scraper, written in both Ruby ([scraper1.rb](https://raw2.github.com/quandl/toolbelt_help/master/scraper1.rb)) and Python ([scraper1.py](https://raw2.github.com/quandl/toolbelt_help/master/scraper1.py)):
@@ -132,10 +151,16 @@ The scraper will be run daily at 17:30 (in your time zone).  Every day when it r
 You now have a dataset on Quandl that refreshes daily!
 -->
 
+
+
+
 ## Scheduling Your Scripts
 
 This feature is not ready for use yet.  When it is ready you will be able to send any script to Quandl.  Quandl will then run the script on a schedule and send the output to `quandl upload` for you.  You can (optionally) receive emails when the script succeeds or fails.
 
+
+
+
 ## Many Datasets via One Input Stream
 
 You can send  multiple datasets with a single call to `quandl upload`.  [Scraper2.rb](https://raw2.github.com/quandl/toolbelt_help/master/scraper2.rb) ([Scraper2.py](https://raw2.github.com/quandl/toolbelt_help/master/scraper2.py)) produces the most recent closing data for two stocks:
@@ -156,10 +181,14 @@ Creates or updates both `quandl.com/<your-username>/AAPL` and `quandl.com/<your-
 
 You can send an infinite number of datasets via one call to `quandl upload`.
 
+
+
+
 ## Quandl Flavored CSV
 
 Quandl "flavored" CSV is just just plain vanilla CSV prepended with metadata in [YAML](http://en.wikipedia.org/wiki/YAML) format.  Metadata is seperated from data by a single line containing one or more dashes "-".
 
+
 ### Quick Reference
 
 Here is the entire specification by example for quick reference:
@@ -184,6 +213,7 @@ Here is the entire specification by example for quick reference:
     Date,Price,Volume # if omitted on new dataset, default headings are created
     2012-01-01,32.23 # the csv data.  date can be almost any format you want
 
+
 ### Metadata Specifications
 
 |Field|Description|Required?|
@@ -196,11 +226,13 @@ Here is the entire specification by example for quick reference:
 |private|true or false; default is false|private data is visible to only you|
 
 
+
+
 ## Example Scrapers
 
 ### Shibor
 
-[www.shibor.org](http://www.shibor.org) publishes Shibor rates which Quandl republishes at [www.quandl.com/TAMMER1/SHIBOR](http://www.quandl.com/USER_YY/SHIBOR)
+[www.shibor.org](http://www.shibor.org) publishes Shibor rates which Quandl republishes at [www.quandl.com/TAMMER1/SHIBOR](http://www.quandl.com/TAMMER1/SHIBOR)
 
 This dataset is maintained via [this ruby script](https://github.com/tammer/scrapers/blob/master/shibor.rb) that fetches the 10 most recent days of data from Shibor.org.
 
@@ -218,9 +250,10 @@ The backfill for this dataset was manually downloaded and converted into a simpl
 
     quandl upload shibor_backfill.csv
 
+
 ### Hsieh Trend Following Factors
 
-Professor David Hsieh maintains hedge fund trend following risk factors at [faculty.fuqua.duke.edu/~dah7/HFRFData.htm](https://faculty.fuqua.duke.edu/~dah7/HFRFData.htm).  They are available on Quandl at [quandl.com/TAMMER1/TFRF](http://www.quandl.com/USER_YY/TFRF).
+Professor David Hsieh maintains hedge fund trend following risk factors at [faculty.fuqua.duke.edu/~dah7/HFRFData.htm](https://faculty.fuqua.duke.edu/~dah7/HFRFData.htm).  They are available on Quandl at [quandl.com/TAMMER1/TFRF](http://www.quandl.com/TAMMER1/TFRF).
 
 The data is maintained by running [hsieh.rb](https://github.com/tammer/scrapers/blob/master/hsieh.rb) every day.  To see the output of the script:
 
@@ -230,6 +263,7 @@ To keep the data up to date, we scheduled a daily run of:
 
     curl "https://raw.github.com/tammer/scrapers/master/hsieh.rb" | ruby | quandl upload
 
+
 ### Copyright Data
 
 Some data publishers provide data on the condition that you not republish it.  When scraping such sites, be sure to set the private flag to be true so that only you can see the data, at which point you should be in compliance, since you are simply storing a single copy on a private cloud based repository; (no different from storing a copy on Google Docs or Dropbox).
@@ -240,16 +274,23 @@ For example, if you happen to need the MSCI Far East Index on Quandl, you can sc
 
 Now you have the data you need on Quandl while remaining compliant with MSCI's terms of use.
 
+
 ### Additional Examples
 
 |Dataset|Scraper|
 |-------|----|
 | [Litecoin vs USD](http://quandl.com/TAMMER1/LTCUSD)| [litecoin.rb](https://github.com/tammer/scrapers/blob/master/litecoin.rb)|
 
+
+
+
 ## Full Reference
 
 Other features of the Toolbelt including `quandl download`, `quandl info`, `quandl list` and other minor features are documented in the [Quandl Toolbelt Reference](#) page.
 
+
+
+
 ## FAQ
 
 ### How can I use ":" in the name or description field?
@@ -270,7 +311,7 @@ or
 
 From Python:
 
-    print "description: \"I love colons : :   :\""    
+    print "description: \"I love colons : :   :\""
 
 
 ### Are the Datasets Publicly Accessible?
@@ -281,11 +322,13 @@ You decide.  By default it is public.  use:
 
 To make the dataset visible only to you.
 
+
 ### Can you handle high frequency (intra-day) data?
 
 No.
 
-### How to I including Blank or Nils
+
+### How do I including Blank or Nils
 
 This is how you include nil datums:
 
@@ -301,15 +344,25 @@ This is how you include nil datums:
 
 This dataset can be seen on Quandl right [here](http://www.quandl.com/TAMMER1/NIL)
 
+
 ### Your SHIBOR script seems to download the past 10 days' worth of data...
 
 ...Assuming that happens daily, then you'll have overlapping data (e.g., the most recent day's data is new, but the prior nine days worth of data should be in the database already). How does Quandl deal with that? What if the underlying data changes - will Quandl update the previous nine days of data? Will it record what the data used to be based on the 'original' dataset?
 
 Answer: If you upload data for dates where data already exists, the new data over-writes the old data.  Thus if you send redundant data, it is harmless.  Shibor.rb is written this way for two reason:  1) helpful in case the publisher changes something a few days later.  2) helpful if we miss run for a couple of days for some reason.
 
-### A given municipal bond doesn't trade every day. So, if I set up a separate 'id' for each bond, then each day there will be some bonds that get pricing updates and others that don't. Are there any issues with this, or can Quandl handle this kind of 'sparse' data?
 
-Sparse data is not a problem.
+### A given municipal bond doesn't trade every day...
+
+So, if I set up a separate 'id' for each bond, then each day there will be some bonds that get pricing updates and others that don't. Are there any issues with this, or can Quandl handle this kind of 'sparse' data?
+
+Answer: Sparse data is not a problem.
+
+
+### Why can't I find my dataset using search on Quandl.
+
+If it is private, it will not appear in search ever.  If it is public, it can take up to 1 hour before our index is updated with your new dataset.
+
 
 ### My Question is not answered!
 

data/Rakefile

@@ -21,90 +21,11 @@ RSpec::Core::RakeTask.new(:spec) do |task|
   task.pattern = "spec/**/*_spec.rb"
 end
 
-def checkout_master!
-  raise "You have unstaged commits." if %x{git status} =~ /Changes not staged for commit/
-  sh "git checkout master"
-  raise "Unable to checkout master" unless %x{git branch | grep '*'} =~ /master/
+require 'quandl/utility/rake_tasks'
+Quandl::Utility::Tasks.configure do |c|
+  c.name              = 'quandl'
+  c.version_path      = 'VERSION'
+  c.changelog_path    = 'UPGRADE.md'
+  c.tag_prefix        = 'v'
+  c.changelog_matching  = ['^QUGC','^WIKI']
 end
-
-def verify_version!(version)
-  raise ArgumentError, "Expected version format is: MAJOR.MINOR.PATCH, got: #{version}" unless version =~ /[0-9]+\.[0-9]+\.[0-9]+/
-  raise "Version '#{version}' has already been documented in UPGRADE.md" if file_contains_matching("UPGRADE.md", /## #{version}/)  
-end
-
-def file_contains_matching(file_path, matching)
-  matched = false
-  f = File.open(file_path)
-  f.each_line do |line|
-    if line =~ matching
-      matched = true
-      break
-    end
-  end
-  f.close
-  matched
-end
-
-namespace :quandl do
-  desc "bump version and generate UPGRADE documentation"
-  task :bump, :version do |t,args|
-    version = args['version']
-    checkout_master!
-    verify_version!(version)
-    # update UPGRADE.md with commits
-    Rake::Task["quandl:generate:documentation"].execute args.to_hash.stringify_keys!
-    Rake::Task["quandl:version:bump"].execute args.to_hash.stringify_keys!    
-  end
-  desc "build and push gem & distros"
-  task :release do |t,args|
-    checkout_master!
-    # tag git revision with version, build quandl.gem, push to rubygems
-    Rake::Task["release"].execute
-    # build windows exe, mac pkg, tarball
-    Rake::Task["toolbelt:release:all"].execute
-  end
-  namespace :version do
-    task :bump, :version do |t,args|
-      version_file = 'lib/quandl/command/version.rb'
-      IO.write(version_file, File.open(version_file) do |f|
-        f.read.gsub(Quandl::Command::VERSION, args['version'])
-      end
-      )
-      sh(%Q{git commit -am "Bump version to: #{args['version']}"})
-    end
-  end
-  namespace :generate do
-    task :documentation, :version do |t,args|
-      version = args['version']
-      verify_version!(version)
-      # collect commits that match JIRA syntax
-      commits = %x{ git --no-pager log --since="v#{Quandl::Command::VERSION}" --pretty=oneline --grep='^QUGC' }
-      # split newlines and exclude reference, select uniq
-      commits = commits.split("\n").collect{|c| "* #{c[41..-1]}" }.uniq
-      # compose prepend string
-      commits = "## #{version} \n\n" + commits.join("\n") + "\n\n\n\n"
-      # prepend to UPGRADE.md
-      File.write( 'UPGRADE.md', commits + File.read('UPGRADE.md') )
-    end
-  end
-end
-
-namespace :rubies do
-
-  rubies = [
-    'ruby-1.9.3-p194',
-    'ruby-1.9.3-p484',
-    'ruby-2.0.0-p353',
-  ]
-  
-  desc "rspec all the rubies"
-  task :spec do |t, args|
-    rubies.each do |ruby|
-      cmd = "rvm #{ruby} do bundle exec rspec"
-      puts(cmd)
-      system(cmd)
-    end
-  end
-  
-end
-

data/UPGRADE.md

@@ -1,3 +1,26 @@
+## 0.3.0 
+
+* QUGC-132 Add thread_pool locking so that requests for the same dataset are not made in parallel
+* QUGC-120 update Task::Threading to catch SIGINT and shutdown thread_pool. Second occurence of SIGINT will exit immediately.
+* QUGC-117 add option for specifying logfile output --stdout and --stderr
+* QUGC-118 add option to specify environment -E --environment
+* QUGC-119 only use thread pool when needed
+* QUGC-119 given 1 thread skip using thread pool so that we can debug more easily, but leave threadpool implemented otherwise
+* QUGC-55 POST crash input back to us. Quandl Toolbelt has crashed. Can we please send the crash report back to Quandl.com?
+* QUGC-113 bump client for error propagation
+* QUGC-109 load task modules into Task
+* QUGC-109 refactor quandl/command/task.rb into modules
+* QUGC-89 quandl list inherits from quandl search and implements the same paging behaviour.
+* QUGC-89 quandl search will return an unlimited number of results.
+* QUGC-99 Add 'quandl search' to Toolbelt
+* QUGC-104 bump gems to fix when upload fails output is not json
+* QUGC-98 Add basic presenter classes
+* QUGC-95 rename ruby package identifier to avoid clobbering other ruby packages
+* WIKI-153 bump quandl_client to 2.5.2 for scraper models
+* WIKI-153 Add schedule task with subtasks: add, delete, replace, list, show
+
+
+
 ## 0.2.27
 
 * add option to force update

data/VERSION

@@ -0,0 +1 @@
+0.3.0

data/lib/quandl/command.rb

@@ -10,10 +10,12 @@ end
 
 require 'quandl/lang'
 require "quandl/command/version"
-require 'quandl/command/qconfig'
+require 'quandl/command/config'
+require 'quandl/command/presenter'
 require 'quandl/command/tasks'
 
 Quandl::Logger.use(Quandl::Logger::Outputs)
+I18n.enforce_available_locales = false
 
 module Quandl::Command
   extend ActiveSupport::Concern
@@ -27,8 +29,14 @@ module Quandl::Command
     default_command :help
     
     global_option '-T', '--token STRING', 'secret token used to authenticate requests.'
-    global_option '-V', '--verbose', 'display detailed log messages'
-    global_option '-U', '--url STRING', 'API url.'
+    global_option '-V', '--verbose', 'use detailed log output'
+    global_option '-U', '--url STRING', 'the API host that will be used'
+    global_option '-F', '--output-format STRING', 'the output format [ pipes, json, qdf ]'
+    global_option '-C', '--output-column INTEGER', 'the column to output'
+    global_option '-E', '--environment INTEGER', 'the environment to load'
+    global_option '--threads INTEGER', 'how many threads to use for concurrent operations'
+    global_option '--stdout STRING', 'where to redirect stdout'
+    global_option '--stderr STRING', 'where to redirect stderr'
     global_option '--force-yes', 'force y/n with yes'
     
     Tasks.each{|t| t.configure(self) }

data/lib/quandl/command/config.rb

@@ -0,0 +1,88 @@
+require 'fileutils'
+require 'yaml'
+require 'ostruct'
+require 'quandl/support/attributes'
+
+module Quandl
+module Command
+class Config
+  
+  include Quandl::Support::Attributes
+  
+  define_attributes :token, :quandl_url, :last_checked_for_update, :output_format, :send_crash_reports, :stdout, :stderr
+  
+  attr_accessor :options
+
+  def initialize(opts={})
+    self.options = opts.symbolize_keys
+    convert_legacy_config
+    ensure_directory_is_present
+    @attributes = read_config_from_file.stringify_keys
+  end
+  
+  def stdout
+    options[:stdout] || read_attribute(:stdout)
+  end
+  
+  def stderr
+    options[:stderr] || read_attribute(:stderr)
+  end
+  
+  def quandl_url
+    return @quandl_url if @quandl_url.present?
+    @quandl_url = options[:url] if options[:url].present?
+    @quandl_url = ENV['QUANDL_URL'] if @quandl_url.blank?
+    @quandl_url = read_attribute(:quandl_url) if @quandl_url.blank?
+    @quandl_url = 'http://quandl.com/api/' if @quandl_url.blank?
+    @quandl_url
+  end
+  
+  def auth_token
+    return @auth_token if @auth_token.present?
+    @auth_token = options[:token] if options[:token].present?
+    @auth_token = ENV['QUANDL_TOKEN'] if @auth_token.blank?
+    @auth_token = read_attribute(:token) if @auth_token.blank?
+    @auth_token
+  end
+  
+  def file_path
+    options[:file_path]
+  end
+  
+  def file_dir
+    File.dirname(file_path)
+  end
+  
+  private
+  
+  def ensure_directory_is_present
+    return if File.exists?(file_path)
+    FileUtils.mkdir(file_dir) unless Dir.exists?(file_dir)
+    File.write( file_path, YAML.dump({}) )
+  end
+  
+  def convert_legacy_config
+    return if File.directory?(file_dir) || !File.exists?(file_dir)
+    # otherwise move the old .quandl into .quandl/config
+    FileUtils.mv(file_dir, "#{file_dir}.old")
+    FileUtils.mkdir(file_dir)
+    token = File.read("#{file_dir}.old")
+    File.write(file_path, "token: #{token}")
+  end
+  
+  def write_attribute(*args)
+    super(*args)
+    write_config_to_file
+  end
+  
+  def read_config_from_file
+    YAML.load(File.read(file_path))
+  end
+  
+  def write_config_to_file
+    File.write( file_path, YAML.dump(attributes) )
+  end
+  
+end
+end
+end