datapackage 0.0.4 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 577f0cccc791f7f81d19b087038ec6061dcee12a
-  data.tar.gz: 9cac27e6e3eaa0be69575afd50a7193e5a7fb23c
+  metadata.gz: 0dbfa361366c6830bb4624347821cc07c754d903
+  data.tar.gz: 4fc8b53abca06e6885d686cde540a21ad591aa49
 SHA512:
-  metadata.gz: ff2eaf11c9605f6fc20019b285747f952de6563da74e6fc2f5d0eb6d3f650fc32d1d40f44e44776a3f17f56152fa2479d6ff5a37d3f253eae4420144054db829
-  data.tar.gz: e83f013ca8d5be9c002b43abda33c89d906f9162722a4044267c652e2e96d3d5612d39c5727cc094941fc97e3f62e1f53862bd8d0499975a3d837dac45ab6f90
+  metadata.gz: 53d75e66e0b49d2a92350d826f8e33e371f76b162fd8c94ecccf566064aa3e0e5c6f98aa2ec89deafa7126a359d804383df03163e91035721f7f4c22efcbcda9
+  data.tar.gz: 4e4b96bef09d7e46f06e7eac048ebacf45aebf3bcb50a6b9f44bf1ff578e7153fc49e5cb64239b76d84a6c936128c5e8394d2f50c67de25203565f0d997cc7df

data/LICENSE.md

@@ -1,4 +1,6 @@
-Copyright 2013 The Open Data Institute
+##Copyright (c) 2016 The Open Data Institute
+
+#MIT License
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the
@@ -17,4 +19,4 @@ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,11 +1,14 @@
+[![Build Status](http://img.shields.io/travis/theodi/datapackage.rb.svg?style=flat-square)](https://travis-ci.org/theodi/datapackage.rb)
+[![Dependency Status](http://img.shields.io/gemnasium/theodi/datapackage.rb.svg?style=flat-square)](https://gemnasium.com/theodi/datapackage.rb)
+[![Coverage Status](http://img.shields.io/coveralls/theodi/datapackage.rb.svg?style=flat-square)](https://coveralls.io/r/theodi/datapackage.rb)
+[![Code Climate](http://img.shields.io/codeclimate/github/theodi/datapackage.rb.svg?style=flat-square)](https://codeclimate.com/github/theodi/datapackage.rb)
+[![Gem Version](http://img.shields.io/gem/v/datapackage.svg?style=flat-square)](https://rubygems.org/gems/datapackage)
+[![License](http://img.shields.io/:license-mit-blue.svg?style=flat-square)](http://theodi.mit-license.org)
+
 # DataPackage.rb
 
 A ruby library for working with [Data Packages](http://dataprotocols.org/data-packages/).
 
-[![Build Status](http://jenkins.theodi.org/job/datapackage.rb-master/badge/icon)](http://jenkins.theodi.org/job/datapackage.rb-master/)
-[![Code Climate](https://codeclimate.com/github/theodi/datapackage.rb.png)](https://codeclimate.com/github/theodi/datapackage.rb)
-[![Dependency Status](https://gemnasium.com/theodi/datapackage.rb.png)](https://gemnasium.com/theodi/datapackage.rb)
-
 The library is intending to support:
 
 * Parsing and using data package metadata and data
@@ -15,179 +18,115 @@ The library is intending to support:
 
 Add the gem into your Gemfile:
 
-    gem 'datapackage.rb', :git => "git://github.com/theodi/datapackage.rb.git"
+```
+gem 'datapackage.rb'
+```
 
 Or:
 
-	sudo gem install datapackage
+```
+gem install datapackage
+```
 
-## Basic Usage
+## Reading a Data Package
 
 Require the gem, if you need to:
 
-    require 'datapackage.rb'
-
-Parsing a datapackage from a remote location:
-
-    package = DataPackage::Package.new( "http://example.org/datasets/a" )
-    
-This assumes that `http://example.org/datasets/a/datapackage.json` exists, or specifically load a JSON file:
-
-    package = DataPackage::Package.new( "http://example.org/datasets/a/datapackage.json" )
-    
-Similarly you can load a package from a local JSON file, or specify a directory:
-
-    package = DataPackage::Package.new( "/my/data/package" )
-    package = DataPackage::Package.new( "/my/data/package/datapackage.json" )
-    
-There are a set of helper methods for accessing data from the package, e.g:
-
-    package = DataPackage::Package.new( "/my/data/package" )
-    package.name
-    package.title
-    package.licenses
-    package.resources
-    
-These currently just return the raw JSON structure, but this might change in future.
-
-## Package Validation
-
-The library supports validating packages. It can be used to validate both the metadata for the package (`datapackage.json`) 
-and the integrity of the package itself, e.g. whether the data files exist.
-
-### Validating a Package
-
-Quickly checking the validity of a package can be achieve as follows:
+```ruby
+require 'datapackage.rb'
+```
 
-    package.valid?
-    
-To expose more detail on errors and warnings:
+Parsing a Data Package from a remote location:
 
-    messages = package.validate() # or package.validate(:datapackage)
+```ruby
+package = DataPackage::Package.new( "http://example.org/datasets/a" )
+```
 
-This returns an object with two keys: `:errors` and `:warnings`. The values of these keys are arrays of message object. 
-Message objects are formatted as follows:
-
-    {
-        :type => :metadata|:integrity,
-        :message => "message for user",
-        :fragment => "/path/to/responsible/element"
-    }
-
-It is possible to treat all warnings as errors by performing strict validation:
-
-    package.valid?(true)
-
-Examples of warnings might include notes on missing metadata elements (e.g. package `licenses`) which are not required by the 
-DataPackage specification but which SHOULD be included.
-
-Warnings are currently generated for:
-
-* Missing `README.md` files from packages
-* Missing `licenses` key from `datapackage.json`
-* Missing `datapackage_version` key from `datapackage.json`
-
-### Selecting a Validation Profile
-
-The library contains two validation classes, one for the core Data Package specification and the other for the Simple Data Format 
-rules. By default the library uses the more liberal Data Package rules.
-
-The required profile can be specified in one of two ways. Either as a parameter to the validation methods:
-
-    package.valid?(:datapackage)
-    package.valid?(:simpledataformat)
-    package.validate(:datapackage)
-    package.validate(:simpledataformat)
-
-Or, by using a `DataPackage::Validation` class:
+This assumes that `http://example.org/datasets/a/datapackage.json` exists, or specifically load a JSON file:
 
-    validator = DataPackage::SimpleDataFormatValidator.new
-    validator.valid?( package )
-    validator.validate( package )
+```ruby
+package = DataPackage::Package.new( "http://example.org/datasets/a/datapackage.json" )
+```
 
-### Approach to Validation
+Similarly you can load a package from a local JSON file, or specify a directory:
 
-The library will support validating packages against the general rules specified in the 
-[DataPackage specification](http://dataprotocols.org/data-packages/) as well as the stricter requirements given in the 
-[Simple Data Format specification](http://dataprotocols.org/simple-data-format/) (SDF). 
- 
-SDF is essentially a profile of DataPackage which includes some additional restrictions on 
-how data should be published and described. For example all data is to be published as CSV files.
+```ruby
+package = DataPackage::Package.new( "/my/data/package" )
+package = DataPackage::Package.new( "/my/data/package/datapackage.json" )
+```
 
-The validation in the library is divided into two parts:
+There are a set of helper methods for accessing data from the package, e.g:
 
-* Metadata validation -- checking that the structure of the `datapackage.json` file is correcgt
-* Integrity checking -- checking that the overall package and data files appear to be in order
+```ruby
+package = DataPackage::Package.new( "/my/data/package" )
+package.name
+package.title
+package.description
+package.homepage
+package.license
+```
 
-#### Metadata Validation
+## Creating a Data Package
 
-The basic structure of `datapackage.json` files are validated using [JSON Schema](http://json-schema.org/). This provides a simple 
-declarative way to describe the expected structure of the package metadata. 
+```ruby
+package = DataPackage::Package.new
 
-The schema files can be found in the `etc` directory of the project and could be used in other applications. The schema files can 
-be customised to support local validation checks (see below).
+package.name = 'my_sleep_duration'
+package.resources =  [
+  {'name': 'data'}
+]
 
-#### Integrity Checking
+resource = package.resources[0]
+resource.descriptor['data'] = [
+  7, 8, 5, 6, 9, 7, 8
+]
 
-While the metadata for a package might be correct, there are other ways in which the package could be invalid. For example, 
-data files might be missing or incorrectly described.
+File.open('datapackage.json', 'w') do |f|
+  f.write(package.to_json)
+end
 
-The metadata validation is therefore supplemented with some custom code that performs some other checks:
+# {"name": "my_sleep_duration", "resources": [{"name": "data", "data": [7, 8, 5, 6, 9, 7, 8]}]}
+```
 
-* (Both profiles) All resources described in the package must be accessible, e.g. the local file exists or a URL responds successfully to a `HEAD`
-* (`:simpledataformat`) All resources must be CSV files
-* (`:simpledataformat`) All resources must have a valid JSON Table Schema
-* (`:simpledataformat`) CSV `dialect` descriptions must be valid
-* (`:simpledataformat`) All fields declared in the schema must be present in the CSV file
-* (`:simpledataformat`) All fields present in the CSV file must be present in the schema
+## Validating a Data Package
 
-### Customising the Validation Code
+```ruby
+package = DataPackage::Package.new('http://data.okfn.org/data/core/gdp/datapackage.json')
 
-The library provides several extension points for customising the way that packages are validated.
+package.valid?
+#=> true
+package.errors
+#=> [] # An array of errors
+```
 
-#### Supplying Custom JSON Schemas
+## Using a different schema
 
-Custom JSON schemas can be provided to allow validation to be tweaked for local conventions. An options hash can be 
-provided to the constructor of a `DataPackage::Validator` object, this can be used to map schema names to custom 
-schemas.
+By default, the gem uses the standard [Data Package Schema](http://specs.frictionlessdata.io/data-packages/), but alternative schemas are available.
 
-(Any options passed to the constructor of a `DataPackage::Package` object will also be passed to its validator)
-  
-For example to create a new validation profile called `my-validation-rules` and then apply it:
+### Schemas in the local cache
 
-    opts = {
-        :schema => {
-            :my-validation-rules => "/path/to/json/schema.json"
-        }
-    }
-    package = DataPackage::Package.new( url )
-    package.valid?(:my-validation-rules)
+The gem comes with schemas for the standard Data Package Schema, as well as the [Tabular Data Package Schema](http://specs.frictionlessdata.io/tabular-data-package/), and the [Fiscal Data Package Schema](http://fiscal.dataprotocols.org/spec/). These can be referred to via an identifier, expressed as a symbol.
 
-This will cause the code to create a custom `DataPackage::Validator` instance that will apply the supplied schema. This class 
-does not provide any integrity checks.
+```ruby
+package = DataPackage::Package.new(nil, :tabular) # Or :fiscal
+```
 
-To mix a custom schema with the existing integrity checking, you must manually create a `Validator` instance. E.g:
+### Schemas from elsewhere
 
-    opts = {
-        :schema => {
-            :my-validation-rules => "/path/to/json/schema.json"
-        }
-    }
-    validator = DataPackage::SimpleDataFormatValidator(:my-validation-rules, opts)
-    validator.valid?( package )
+If you have a schema stored in an alternative registry, you can pass a `registry_url` option to the initializer.
 
-Custom schemas must be valid JSON files that conforms to the JSON Schema v4 specification. The absolute path to the schema file must be 
-provided.
+```ruby
+package = DataPackage::Package.new(nil, :identifier, {registry_url: 'http://example.org/my-registry.csv'} )
+```
 
-Validation is performed using the [json-schema](https://github.com/hoxworth/json-schema) gem which has some documented restrictions.
-     
-The built-in schema files can also be overridden in this way, e.g. by specifying an alternate location for the `:datapackage` schema.
+## Developer notes
 
-#### Custom Integrity Checking
+These notes are intended to help people that want to contribute to this package itself. If you just want to use it, you can safely ignore them.
 
-Integrity checking can be customized by creating new sub-classes of `DataPackage::Validator` or one of the existing sub-classes. 
+### Updating the local schemas cache
 
-The following methods can be implemented:
+We cache the schemas from https://github.com/dataprotocols/schemas using git-subtree. To update it, use:
 
-* `validate_metadata(package, messages)` -- perform additional metadata checking after JSON schema is provided.
-* `validate_resource(package, resource, messages)` -- called for each resource in the package
+```
+  git subtree pull --prefix datapackage/schemas https://github.com/dataprotocols/schemas.git master --squash
+```

data/lib/datapackage/exceptions.rb

@@ -0,0 +1,12 @@
+module DataPackage
+  class RegistryError < StandardError; end
+
+  class SchemaException < Exception
+    attr_reader :status, :message
+
+    def initialize status
+      @status = status
+      @message = status
+    end
+  end
+end

data/lib/datapackage/package.rb

@@ -1,160 +1,181 @@
 require 'open-uri'
 
 module DataPackage
-    
-    class Package
-                
-        attr_reader :metadata, :opts
-                
-        # Parse a data package
-        #
-        # Supports reading data from JSON file, directory, and a URL
-        #
-        # package:: Hash or a String
-        # opts:: Options used to customize reading and parsing
-        def initialize(package, opts={})
-            @opts = opts
-            #TODO base directory/url            
-            if package.class == Hash
-                @metadata = package
-            else
-                if !package.start_with?("http") && File.directory?(package)
-                    package = File.join(package, opts[:default_filename] || "datapackage.json")
-                end
-                if package.start_with?("http") && !package.end_with?("datapackage.json")
-                    package = URI.join(package, "datapackage.json")
-                end                    
-                @location = package.to_s
-                @metadata = JSON.parse( open(package).read )                
-            end
-        end
-        
-        #Returns the directory for a local file package or base url for a remote
-        #Returns nil for an in-memory object (because it has no base as yet)
-        def base
-            #user can override base
-            return @opts[:base] if @opts[:base]
-            return "" unless @location
-            #work out base directory or uri
-            if local?
-                return File.dirname( @location )
-            else
-                return @location.split("/")[0..-2].join("/")
-            end
-        end
-        
-        #Is this a local package? Returns true if created from an in-memory object or a file/directory reference
-        def local?             
-            return !@location.start_with?("http") if @location
-            return true
-        end
-        
-        def name
-            @metadata["name"]
-        end
-        
-        def title
-            @metadata["title"]
-        end
-        
-        def description
-            @metadata["description"]
-        end
-        
-        def homepage
-            @metadata["homepage"]
-        end
-        
-        def licenses
-            @metadata["licenses"] || []
-        end
-        alias_method :licences, :licenses
-        
-        #What version of datapackage specification is this using?
-        def datapackage_version
-            @metadata["datapackage_version"]
-        end
-        
-        #What is the version of this specific data package?
-        def version
-            @metadata["version"]
-        end
-        
-        def sources
-            @metadata["sources"] || []
-        end
-        
-        def keywords
-            @metadata["keywords"] || []
-        end
-        
-        def last_modified
-            DateTime.parse @metadata["last_modified"] rescue nil 
-        end
-        
-        def image
-            @metadata["image"]
-        end
-        
-        def maintainers
-            @metadata["maintainers"] || []
-        end
-        
-        def contributors
-            @metadata["contributors"] || []
-        end
-        
-        def publisher
-            @metadata["publisher"] || []
-        end
-        
-        def resources
-            @metadata["resources"] || []
-        end
-        
-        def dependencies
-            @metadata["dependencies"]
-        end
-        
-        def property(property, default=nil)
-            @metadata[property] || default
-        end
-        
-        def valid?(profile=:datapackage, strict=false)
-            validator = DataPackage::Validator.create(profile, @opts)
-            return validator.valid?(self, strict)
-        end
-        
-        def validate(profile=:datapackage)
-            validator = DataPackage::Validator.create(profile, @opts)
-            return validator.validate(self)
-        end
-        
-        def resolve_resource(resource)
-            return resource["url"] || resolve( resource["path"] )
-        end
-        
-        def resolve(path)
-            if local?
-                return File.join( base , path) if base != ""
-                return path
-            else
-                return URI.join(base, path)
-            end
-        end
-        
-        def resource_exists?(location)
-            if !location.to_s.start_with?("http")                
-                return File.exists?( location )
-            else
-                begin
-                    status = RestClient.head( location ).code
-                    return status == 200 
-                rescue => e
-                    return false
-                end                
-            end
-        end
-             
+  class Package < Hash
+    attr_reader :opts, :errors
+    attr_writer :resources
+
+    # Parse or create a data package
+    #
+    # Supports reading data from JSON file, directory, and a URL
+    #
+    # package:: Hash or a String
+    # schema:: Hash, Symbol or String
+    # opts:: Options used to customize reading and parsing
+    def initialize(package = nil, schema = :base, opts = {})
+      @opts = opts
+      @schema = DataPackage::Schema.new(schema || :base)
+      @dead_resources = []
+
+      self.merge! parse_package(package)
+      define_properties!
+      load_resources!
+    end
+
+    def parse_package(package)
+      # TODO: base directory/url
+      if package.nil?
+        {}
+      elsif package.class == Hash
+        package
+      else
+        read_package(package)
+      end
+    end
+
+    # Returns the directory for a local file package or base url for a remote
+    # Returns nil for an in-memory object (because it has no base as yet)
+    def base
+      # user can override base
+      return @opts[:base] if @opts[:base]
+      return '' unless @location
+      # work out base directory or uri
+      if local?
+          return File.dirname(@location)
+      else
+          return @location.split('/')[0..-2].join('/')
+      end
+    end
+
+    # Is this a local package? Returns true if created from an in-memory object or a file/directory reference
+    def local?
+      return @local if @local
+      return !@location.start_with?('http') if @location
+      true
+    end
+
+    def resources
+      update_resources!
+      @resources
+    end
+
+    def property(property, default = nil)
+      self[property] || default
+    end
+
+    def valid?
+      validate
+      @valid
+    end
+
+    def validate
+      @errors = @schema.validation_errors(self)
+      @valid = @schema.valid?(self)
+    end
+
+    def resource_exists?(location)
+      @dead_resources.include?(location)
+    end
+
+    def to_json
+      self.to_json
+    end
+
+    private
+
+    def define_properties!
+      (@schema['properties'] || {}).each do |k, v|
+        next if k == 'resources'
+        define_singleton_method("#{k.to_sym}=", proc { |p| set_property(k, p) })
+        define_singleton_method(k.to_sym.to_s, proc { property k, default_value(v) })
+      end
+    end
+
+    def load_resources!
+      @resources = (self['resources'] || [])
+      update_resources!
+    end
+
+    def update_resources!
+      @resources.map! do |resource|
+        begin
+          load_resource(resource)
+        rescue
+          @dead_resources << resource['path']
+          nil
+        end
+      end
+    end
+
+    def load_resource(resource)
+      if resource.is_a?(Resource)
+        resource
+      else
+        Resource.load(resource, base)
+      end
+    end
+
+    def default_value(schema_data)
+      case schema_data['type']
+      when 'string'
+          nil
+      when 'array'
+          []
+      when 'object'
+          {}
+      end
+    end
+
+    def set_property(key, value)
+      self[key] = value
+    end
+
+    def read_package(package)
+      if is_directory?(package)
+          package = File.join(package, opts[:default_filename] || 'datapackage.json')
+      elsif is_containing_url?(package)
+          package = URI.join(package, 'datapackage.json')
+      end
+
+      @location = package.to_s
+
+      if File.extname(package.to_s) == '.zip'
+          unzip_package(package)
+      else
+          JSON.parse open(package).read
+      end
+    end
+
+    def is_directory?(package)
+      !package.start_with?('http') && File.directory?(package)
+    end
+
+    def is_containing_url?(package)
+      package.start_with?('http') && !package.end_with?('datapackage.json', 'datapackage.zip')
+    end
+
+    def write_to_tempfile(url)
+      tempfile = Tempfile.new('datapackage')
+      tempfile.write(open(url).read)
+      tempfile.rewind
+      tempfile
+    end
+
+    def unzip_package(package)
+      package = write_to_tempfile(package) if package.start_with?('http')
+      dir = Dir.mktmpdir
+      Zip::File.open(package) do |zip_file|
+          # Extract all the files
+          zip_file.each { |entry| entry.extract("#{dir}/#{File.basename entry.name}") }
+          # Get and parse the datapackage metadata
+          entry = zip_file.glob("*/#{opts[:default_filename] || 'datapackage.json'}").first
+          package = JSON.parse(entry.get_input_stream.read)
+      end
+      # Set the base dir to the directory we unzipped to
+      @opts[:base] = dir
+      # This is now a local file, not a URL
+      @local = true
+      package
     end
-end
+  end
+end