noaa_ncei_weather 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 53642b4bebf5f150c3022a688787ec632b5f9b1c
+  data.tar.gz: 6f04afc3ab6c0a17fcb8b75db46985ff16d93000
+SHA512:
+  metadata.gz: a58cd2a4d133bbfd8fb9079c808a52a53ce8fe7c77af8392244600a4ce993c0d5abf4d9852bb50e515a9f9e3d920a9815e742c2163624781361cfba36c99d2ff
+  data.tar.gz: fc8a428f3fc37523eb2d8ed2b9929f86c9bd0a332c23899e9888d6ebd1eea0fc6cc10a6e710aea6a69cc95fcb1aaeb915b9b73acf0c14300ffef5ff0e1729761

data/.gitignore

@@ -0,0 +1,10 @@
+/.bundle/
+/.yardoc
+Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+env.rb

data/.travis.yml

@@ -0,0 +1,7 @@
+language: ruby
+rvm:
+  - "2.0.0"
+  - "2.1" # latest 2.1.x
+  - "2.2" # latest 2.2.x
+  - "jruby-9.0.5.0"
+script: rake test

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in noaa_ncei_weather.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Jason B Thelen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,98 @@
+[![Build Status](https://travis-ci.org/JasonBThelen/noaa_ncei_weather.svg?branch=master)](https://travis-ci.org/JasonBThelen/noaa_ncei_weather)
+
+# NOAA NCEI Weather
+
+Ruby wrapper to the NOAA NCEI Weather API at [ncdc.noaa.gov](http://www.ncdc.noaa.gov/cdo-web/webservices/v2)
+
+Uses [Rest-Client](https://github.com/rest-client/rest-client) to query information from the National Oceanic and Atmospheric Administration Climate Data Web Services API. This provides free access to global historical weather information from around the globe. Many types of weather measurements can be pulled in time frames from the first records to near current (a few days ago).
+
+Use of the API requires the use of a free token. Register for your token on the [request page](http://www.ncdc.noaa.gov/cdo-web/token). You'll need a token to use this gem.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'noaa_ncei_weather'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install noaa_ncei_weather
+
+## Usage
+
+Start by setting the token
+
+```ruby
+NoaaNceiWeather::Connection.token = 'TOKEN'
+```
+
+Use the [NOAA](http://www.ncdc.noaa.gov/cdo-web/webservices/v2#gettingStarted) page as a reference for the data available and the paramaters that can be used. This gem provides a class for the data returned by each of the available endpoints.
+
+All of the endpoints (except `Data`) are what I'll call 'organizational' information used to filter down a query to the `Data` class. All of these have class methods for querying against the database.
+
+```ruby
+NoaaNceiWeather::LocationCategory.all # Returns a collection of LocationCategory objects
+NoaaNceiWeather::LocationCategory.where({limit: 42}) # Returns a collection of LocationCategory objects filtered by the parameters given
+NoaaNceiWeather::LocationCategory.first # Returns a single LocationCategory object
+NoaaNceiWeather::LocationCategory.find("CITY") # Returns a single LocationCategory with the given ID
+```
+
+Many of these endpoints also have relationships with one another. For example, each location category has many locations. This is shown as an optional parameter on the NOAA page for the [location endpoint](http://www.ncdc.noaa.gov/cdo-web/webservices/v2#locations). To reflect this relationship, `LocationCategory` has an instance method `.locations` to retrieve the related records. The same is true for the other classes where available via parameters shown on the NOAA documentation
+
+```ruby
+lc = NoaaNceiWeather::LocationCategory.first # returns a single instance of LocationCategory
+lc.locations # returns a collection of Location objects related to lc
+```
+
+Parameters passed to the class methods are parsed before being sent to the NOAA API for convenience.
+  1. Relationship parameters to another endpoint can be passed an object instance. For example:
+
+    ```ruby
+    lc = NoaaNceiWeather::LocationCategory.find('CITY') # an instance of LocationCategory
+    NoaaNceiWeather::Location.where(locationcategoryid: lc.id) # Parameter shown on the NOAA API Docs
+    NoaaNceiWeather::Location.where(locationcategory: lc) # alternative
+    ```
+
+  2. Date parameters shown in the NOAA parameters can be given as Ruby `Date`, `DateTime`, or `Time` instances rather than an ISO formatted string
+
+Limit is a parameter available to all the NOAA endpoints. Limit is being handled in the query so you can exceed the documented NOAA limit of a maximum 1000 per query. The `.all` method will return all of the available records, even if there are more than 1000. You may also set a limit of something over 1000 and that number will be returned to you. For example:
+
+```ruby
+data = NoaaNceiWeather::DataType.all
+data.count # => 1461 - the current count as of this writing
+data = NoaaNceiWeather::DataType.where(limit: 1200)
+data.count # => 1200 - passing a 1200 limit to the noaa api directly would raise a bad request error
+```
+
+
+The /data endpoint and corresponding `Data` class is where most of the real data is stored. The NOAA API required parameters are required in the class method, while the rest are passed with a hash as with the other classes. This returns a collection of Data objects.
+
+```ruby
+ds = NoaaNceiWeather::Dataset.first
+date = Date.today - 14
+data = NoaaNceiWeather::Data.where(ds, date, date, {offset: 0})
+```
+
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+Create an `env.rb` file and add a line `ENV['NOAA_TOKEN'] = 'TOKEN'`. The token will be loaded in when using `bin/console` so you don't have to set the token each time. It is also used in the test setup. The file is already included in `.gitignore`.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com//JasonBThelen/noaa_ncei_weather.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,11 @@
+require "bundler/gem_tasks"
+task :default => :spec
+
+
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.libs << "tests"
+  t.test_files = FileList['tests/test*.rb']
+  t.verbose = true
+end

data/bin/console

@@ -0,0 +1,19 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "noaa_ncei_weather"
+load 'env.rb' if File.exist?('./env.rb')
+NoaaNceiWeather::Connection.token = ENV['NOAA_TOKEN']
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+require "pry"
+Pry.start
+
+
+
+
+# require "irb"
+# IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/noaa_ncei_weather.rb

@@ -0,0 +1,13 @@
+require "rest-client"
+require "noaa_ncei_weather/version"
+
+require 'noaa_ncei_weather/connection'
+require 'noaa_ncei_weather/weather'
+require 'noaa_ncei_weather/data_category'
+
+require 'noaa_ncei_weather/location_category'
+require 'noaa_ncei_weather/data'
+require 'noaa_ncei_weather/station'
+require 'noaa_ncei_weather/data_type'
+require 'noaa_ncei_weather/dataset'
+require 'noaa_ncei_weather/location'

data/lib/noaa_ncei_weather/connection.rb

@@ -0,0 +1,92 @@
+module NoaaNceiWeather
+
+  # @abstract Contains common connection components shared between {Weather} and {Data}
+  #   used to make requests to the NOAA API.
+  module Connection
+    # Base URL for the NOAA API. Endppoints are appended in concrete classes.
+    @@url = 'http://www.ncdc.noaa.gov/cdo-web/api/v2/'
+
+    # Connection token required to make requests. Must be set before calling
+    # class methods from any NoaaNceiWeather module classes.
+    # @note Each Token is restricted to five request per second and 1,000 requests
+    #   per day per the NOAA documentation
+    @@token = ''
+
+    # Set the request token to be used in requests to NOAA. Token must be obtained
+    # from {http://www.ncdc.noaa.gov/cdo-web/token NOAA}. Use this before trying
+    # to make any requests:
+    #   NoaaNceiWeather::Connection.token = 'token'
+    #
+    # @!attribute [w] token
+    #   @return [String] Token required to be sent with any requests. This can be
+    #     obtained for free from {http://www.ncdc.noaa.gov/cdo-web/token NOAA}.
+    #     The token is good for 5 requests per second, 1,000 requests per day.
+    def self.token=(token)
+      @@token = token
+    end
+
+    # Parses params to the format expected by the API. Allows more flexibility in what can
+    # be sent in as a parameter. Objects, Dates, and Limits are converted
+    # into strings as expected by the API.
+    #
+    # @param params [Hash] Hash of params to be parsed into an API compatible version
+    # @return [Hash] Hash of params. Objects are converted into IDs, Dates are
+    #   converted into iso formatted strings, and max limit is set at 1000
+    def parse_params(params)
+      # Handle passing of weather objects as parameters for other api queries
+      objects = [:dataset, :datatype, :location, :station, :datacategory, :locationcategory]
+      objects.each do |object|
+        params[(object.to_s + "id").to_sym] = params.delete(object).id if params.has_key?(object)
+      end
+
+      # Handle Date, DateTime, or Time parameters
+      # Convert to formatted string the api is expecting
+      dates = [:startdate, :enddate]
+      dates.each do |date|
+        params[date] = params[date].iso8601 if params[date].respond_to?(:iso8601)
+      end
+
+      # Prep for handling requests for over the 1k NOAA limit
+      params[:limit] = 1000 unless params[:limit] && params[:limit] < 1000
+
+      #return modified params
+      params
+    end
+
+    # Raw request sent via RestClient to the API. Used by all other requests
+    #
+    # @param endpoint [String] Endpoint of the API to be used. This is set as a
+    #   class variable by each of the concerete classes in this gem.
+    # @param params [Hash] Hash of params to be passed through to the API
+    # @return [Hash] Hashified version of the response body received, includes
+    #   both actual data and metadata about the resultset
+    def request(endpoint, params = {})
+      url = @@url + endpoint
+      response = RestClient::Request.execute(method: 'get', url: url, headers: {token: @@token, params: params})
+      JSON.parse(response.body)
+    end
+
+    # Request with parameters used by child classes. Handles NOAA max limit of 1000
+    # records by looping through the request.
+    #
+    # @param endpoint [String] Endpoint of the API to be used. This is set as a
+    #   class variable by each of the concerete classes in this gem.
+    # @param params [Hash] Hash of params to be passed through to the API
+    # @return [Array<Hash>] Array of Hashes containing data only of the result (no metadata)
+    def where(endpoint, params = {})
+      limit = params[:limit] || Float::INFINITY
+      params = self.parse_params(params)
+      output = []
+      begin
+        response = self.request(endpoint, params)
+        break unless response.any?
+        meta = response['metadata']['resultset']
+        output.concat response['results']
+        count = meta['offset'] + meta['limit'] - 1
+        params[:offset] = count + 1
+        params[:limit] = limit - count if (limit - count) < 1000
+      end while count < meta['count'] && count < limit && limit > 1000
+      output
+    end
+  end
+end

data/lib/noaa_ncei_weather/data.rb

@@ -0,0 +1,87 @@
+# Namespace for classes and modules that handle requesting data
+# from the NOAA NCEI Historical Weather Database
+module NoaaNceiWeather
+
+  # Class for querying against the /data endpoint of the
+  # {http://www.ncdc.noaa.gov/cdo-web/webservices/v2 NOAA API}. This endpoint
+  # gives access to the actual measurements taken.
+  class Data
+    extend Connection
+
+    # Endpoint portion of the API URL, appended to the Connection URL for requests
+    @@endpoint = 'data'
+
+    # @!attribute [r] date
+    #   @return [DateTime] The date/time this data object was measured
+    # @!attribute [r] datatype
+    #   @return [String] The type of data measured, ID of {DataType}
+    # @!attribute [r] station
+    #   @return [String] The station at which the measurement was taken, ID of {Station}
+    # @!attribute [r] value
+    #   @return [Numeric] Numeric value of the measurement
+    attr_reader :date, :datatype, :station, :attributes, :value
+
+    # Creates a new Data object
+    def initialize(date, datatype, station, attributes, value)
+      @date = date
+      @datatype = datatype
+      @station = station
+      @attributes = attributes
+      @value = value
+    end
+
+    # Retrieves a collection of {Data} objects based on the params given.
+    #
+    # @param datasetid [String, Dataset] A String ID for a Dataset or a DataSet
+    #   object from the NOAA DB.
+    # @param startdate [String, Date] A Date or ISO8601 formatted date string for
+    #   the earliest data that should be retrieved
+    # @param enddate [String, Date] A Date or ISO8601 formatted date string for
+    #   the latest data that should be retrieved
+    # @param params [Hash] A hash including other params to set filters on the NOAA request
+    # @option params [String] :datatypeid String ID of a {DataType}
+    #   within the selected dataset that should be retrieved
+    # @option params [DataType] :datatype {DataType} object
+    # @option params [String] :locationid String ID of a {Location}
+    # @option params [Location] :location {Location} object
+    # @option params [String] :stationid String ID of a {Station}
+    # @option params [Station] :station {Station} object
+    # @option params [String] :units ('metric') Accepts string 'standard' or 'metric'
+    #   to set the unit of measurement returned in the value
+    # @option params [String] :sortfield ('id') Accepts string values 'id', 'name,
+    #   'mindate', 'maxdate', and 'datacoverage' to sort data before being returned
+    # @option params [String] :sortorder ('asc') Accepts 'asc' or 'desc' for sort order
+    # @option params [Integer] :limit Set a limit to the amount of records returned
+    # @option params [Integer] :offset (0) Used to offset the result list
+    # @return [Array<Data>] An array of Data objects
+    def self.where(datasetid, startdate, enddate, params = {})
+      datasetid = datasetid.id if datasetid.respond_to?(:id)
+      params[:datasetid] = datasetid
+      params[:includemetadata] = true
+
+      startdate = Date.parse startdate if startdate.kind_of? String
+      enddate = Date.parse enddate if enddate.kind_of? String
+      to_date = enddate
+      enddate = startdate + 365 if enddate - startdate > 365
+      limit = params[:limit] if params[:limit]
+
+      output = []
+      begin
+        params.merge!({startdate: startdate, enddate: enddate})
+
+        data = super(@@endpoint, params)
+        output.concat data
+        if limit && output.count >= limit
+          output = output[0...limit]
+          break
+        end
+        startdate = enddate + 1
+        enddate = startdate + 365
+        enddate = to_date if enddate > to_date
+        params[:limit] = limit
+        params[:offset] = nil
+      end while to_date > startdate
+      output.collect {|item| self.new DateTime.parse(item['date']), item['datatype'], item['station'], item['attributes'], item['value']}
+    end
+  end
+end

data/lib/noaa_ncei_weather/data_category.rb

@@ -0,0 +1,85 @@
+module NoaaNceiWeather
+
+  # Class for querying against the /datacategory endpoint of the NOAA API
+  class DataCategory < Weather
+
+    # Endpoint portion of the API URL, appended to the Connection URL for requests
+    @@endpoint = 'datacategories'
+
+    # @!attribute [r] id
+    #   @return [String] The unique Identifier of the {DataCategory}
+    # @!attribute [r] name
+    #   @return [String] The descriptive name of the {DataCategory}
+
+    # Retrieves the {DataType DataTypes} associated with a {DataCategory} object
+    # {DataCategory} has a one to many relationship with {DataType} (in rare
+    # cases a DataType may belong to more than one DataCategory)
+    #
+    # @param params [Hash] See {DataType.where} for valid param key/values
+    # @return [Array<DataType>] Array of the data types associated with this
+    #   {DataCategory} instance
+    def data_types(params = {})
+      params.merge!({datacategoryid: @id})
+      DataType.where(params)
+    end
+
+    # Retrieves the {Location Locations} associated with a {DataCategory} object.
+    # {Location} and {DataCategory} have a many to many relationship.
+    #
+    # @param params [Hash] See {Location.where} for valid param key/values
+    # @return [Array<DataType>] Array of the data types associated with this
+    #   DataCategory instance
+    def locations(params = {})
+      params.merge!({datacategoryid: @id})
+      Location.where(params)
+    end
+
+    # Retrieves the {Station Stations} associated with a {DataCategory} object
+    # {Station} and {DataCategory} have a many to many relationship.
+    #
+    # @param params [Hash] See {Station.where} for valid param key/values
+    # @return [Array<DataType>] Array of the data types associated with this
+    #   DataCategory instance
+    def stations(params = {})
+      params.merge!({datacategoryid: @id})
+      Station.where(params)
+    end
+
+    # Finds a specific instance of {DataCategory} by its ID
+    #
+    # @param id [String] String ID of the resource.
+    # @return [DataCategory, nil] Instance of {DataCategory}, or nil if none found.
+    def self.find(id)
+      data = super(@@endpoint + "/#{id}")
+      if data && data.any?
+        self.new data['id'], data['name']
+      else
+        nil
+      end
+    end
+
+    # Retrieves a set of {DataCategory DataCategories} based on the parameters given
+    #
+    # @param params [Hash] Hash to set filters on the request sent to the NOAA API
+    # @option params [String] :datasetid String ID of a {Dataset}
+    # @option params [Dataset] :dataset {Dataset} object
+    # @option params [String] :locationid String ID of a {Location}
+    # @option params [Location] :location {Location} object
+    # @option params [String] :stationid String ID of a {Station}
+    # @option params [Station] :station {Station} object
+    # @option params [String] :sortfield ('id') Accepts string values 'id', 'name,
+    #   'mindate', 'maxdate', and 'datacoverage' to sort data before being returned
+    # @option params [String] :sortorder ('asc') Accepts 'asc' or 'desc' for sort order
+    # @option params [Integer] :limit Set a limit to the amount of records returned
+    # @option params [Integer] :offset (0) Used to offset the result list
+    # @return [Array<DataCategory>] Array of {DataCategory} objects that match the filter.
+    def self.where(params = {})
+      data = super(@@endpoint, params)
+      if data && data.any?
+        data.collect { |item| self.new(item['id'], item['name']) }
+      else
+        []
+      end
+    end
+  end
+end