csvpack 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b9ea6e66b9c56d609a881d9edd84c19fa58c9f8e
+  data.tar.gz: 0a0b7df35fa79d3f06aceb7a100aeb20ee71bd0e
+SHA512:
+  metadata.gz: 1e5aa488c56683fbd1215da82c60cc8213b890c0e3b42cd126626d3005e2de7ffac156288bb91bcab42f234a34e8734731acd8c1f61694b7b1362e7d154d8fc9
+  data.tar.gz: 4eccdd36ca45df01c5d69f63e51abe71bb191a3144998fe0cf2de9ceb639aeb399df24b470f66752fa3a5e96e276b15f7863ff4b7cb258c5cef86520de8874d1

data/HISTORY.md

@@ -0,0 +1,4 @@
+### 0.0.1 / 2015-04-23
+
+* Everything is new. First release
+

data/Manifest.txt

@@ -0,0 +1,13 @@
+HISTORY.md
+Manifest.txt
+README.md
+Rakefile
+lib/csvpack.rb
+lib/csvpack/downloader.rb
+lib/csvpack/pack.rb
+lib/csvpack/version.rb
+test/helper.rb
+test/test_companies.rb
+test/test_countries.rb
+test/test_downloader.rb
+test/test_import.rb

data/README.md

@@ -0,0 +1,354 @@
+# csvpack
+
+work with tabular data packages using comma-separated values (CSV) datafiles in text with datapackage.json; download, read into and query comma-separated values (CSV) datafiles with your SQL database (e.g. SQLite, PostgreSQL, ...) of choice and much more
+
+
+* home  :: [github.com/csv11/csvpack](https://github.com/csv11/csvpack)
+* bugs  :: [github.com/csv11/csvpack/issues](https://github.com/csv11/csvpack/issues)
+* gem   :: [rubygems.org/gems/csvpack](https://rubygems.org/gems/csvpack)
+* rdoc  :: [rubydoc.info/gems/csvpack](http://rubydoc.info/gems/csvpack)
+* forum :: [ruby-talk@ruby-lang.org](http://www.ruby-lang.org/en/community/mailing-lists/)
+
+
+
+## Usage
+
+
+### What's a tabular data package?
+
+> Tabular Data Package is a simple structure for publishing and sharing
+> tabular data with the following key features:
+>
+> - Data is stored in CSV (comma separated values) files
+> - Metadata about the dataset both general (e.g. title, author)
+>   and the specific data files (e.g. schema) is stored in a single JSON file
+>   named `datapackage.json` which follows the Data Package format
+
+(Source: [Tabular Data Packages, Frictionless Data Initiative • Data Hub.io • Open Knowledge Foundation • Data Protocols.org](https://datahub.io/docs/data-packages/tabular))
+
+
+
+Here's a minimal example of a tabular data package holding two files, that is, `data.csv` and `datapackage.json`:
+
+`data.csv`:
+
+```
+Brewery,City,Name,Abv
+Andechser Klosterbrauerei,Andechs,Doppelbock Dunkel,7%
+Augustiner Bräu München,München,Edelstoff,5.6%
+Bayerische Staatsbrauerei Weihenstephan,Freising,Hefe Weissbier,5.4%
+Brauerei Spezial,Bamberg,Rauchbier Märzen,5.1%
+Hacker-Pschorr Bräu,München,Münchner Dunkel,5.0%
+Staatliches Hofbräuhaus München,München,Hofbräu Oktoberfestbier,6.3%
+...
+```
+
+`datapackage.json`:
+
+``` json
+{
+  "name": "beer",
+  "resources": [
+    {
+      "path": "data.csv",
+      "schema": {
+        "fields": [{ "name": "Brewery",   "type": "string" },
+                   { "name": "City",      "type": "string" },
+                   { "name": "Name",      "type": "string" },
+                   { "name": "Abv",       "type": "number" }]
+      }
+    }
+  ]
+}
+```
+
+
+
+### Where to find data packages?
+
+For some real world examples see the [Data Packages Listing](https://datahub.io/core) ([Sources](https://github.com/datasets)) at the Data Hub.io • Frictionless Data Initiative
+website for a start. Tabular data packages include:
+
+Name                     | Comments
+------------------------ | -------------
+`country-codes`          | Comprehensive country codes: ISO 3166, ITU, ISO 4217 currency codes and many more
+`language-codes`         | ISO Language Codes (639-1 and 693-2)
+`currency-codes`         | ISO 4217 Currency Codes
+`gdb`                    | Country, Regional and World GDP (Gross Domestic Product)
+`s-and-p-500-companies`  | S&P 500 Companies with Financial Information
+`un-locode`              | UN-LOCODE Codelist
+`gold-prices`            | Gold Prices (Monthly in USD)
+`bond-yields-uk-10y`     | 10 Year UK Government Bond Yields (Long-Term Interest Rate)
+
+
+
+and many more
+
+
+### Code, Code, Code - Script Your Data Workflow with Ruby
+
+
+``` ruby
+require 'csvpack'
+
+CsvPack.import(
+  's-and-p-500-companies',
+  'gdb'
+)
+```
+
+Using `CsvPack.import` will:
+
+1) download all data packages to the `./pack` folder
+
+2) (auto-)add all tables to an in-memory SQLite database using SQL `create_table`
+   commands via `ActiveRecord` migrations e.g.
+
+
+``` ruby
+create_table :constituents_financials do |t|
+  t.string :symbol          # Symbol         (string)
+  t.string :name            # Name           (string)
+  t.string :sector          # Sector         (string)
+  t.float  :price           # Price          (number)
+  t.float  :dividend_yield  # Dividend Yield (number)
+  t.float  :price_earnings  # Price/Earnings (number)
+  t.float  :earnings_share  # Earnings/Share (number)
+  t.float  :book_value      # Book Value     (number)
+  t.float  :_52_week_low    # 52 week low    (number)
+  t.float  :_52_week_high   # 52 week high   (number)
+  t.float  :market_cap      # Market Cap     (number)
+  t.float  :ebitda          # EBITDA         (number)
+  t.float  :price_sales     # Price/Sales    (number)
+  t.float  :price_book      # Price/Book     (number)
+  t.string :sec_filings     # SEC Filings    (string)
+end
+```
+
+3) (auto-)import all datasets using SQL inserts e.g.
+
+``` sql
+INSERT INTO constituents_financials
+  (symbol,
+   name,
+   sector,
+   price,
+   dividend_yield,
+   price_earnings,
+   earnings_share,
+   book_value,
+   _52_week_low,
+   _52_week_high,
+   market_cap,
+   ebitda,
+   price_sales,
+   price_book,
+   sec_filings)
+VALUES
+  ('MMM',
+   '3M Co',
+   'Industrials',
+   162.27,
+   2.11,
+   22.28,
+   7.284,
+   25.238,
+   123.61,
+   162.92,
+   104.0,
+   8.467,
+   3.28,
+   6.43,
+   'http://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=MMM')
+```
+
+4) (auto-)add ActiveRecord models for all tables.
+
+
+So what? Now you can use all the "magic" of ActiveRecord to query
+the datasets. Example:
+
+``` ruby
+pp Constituent.count
+
+# SELECT COUNT(*) FROM "constituents"
+# => 496
+
+
+pp Constituent.first
+
+# SELECT  "constituents".* FROM "constituents" ORDER BY "constituents"."id" ASC LIMIT 1
+# => #<Constituent:0x9f8cb78
+#         id:     1,
+#         symbol: "MMM",
+#         name:   "3M Co",
+#         sector: "Industrials">
+
+
+pp Constituent.find_by!( symbol: 'MMM' )
+
+# SELECT  "constituents".*
+#         FROM "constituents"
+#         WHERE "constituents"."symbol" = "MMM"
+#         LIMIT 1
+# => #<Constituent:0x9f8cb78
+#         id:     1,
+#         symbol: "MMM",
+#         name:   "3M Co",
+#         sector: "Industrials">
+
+
+pp Constituent.find_by!( name: '3M Co' )
+
+# SELECT  "constituents".*
+#          FROM "constituents"
+#          WHERE "constituents"."name" = "3M Co"
+#          LIMIT 1
+# => #<Constituent:0x9f8cb78
+#         id:     1,
+#         symbol: "MMM",
+#         name:   "3M Co",
+#         sector: "Industrials">
+
+
+pp Constituent.where( sector: 'Industrials' ).count
+
+# SELECT COUNT(*) FROM "constituents"
+#         WHERE "constituents"."sector" = "Industrials"
+# => 63
+
+
+pp Constituent.where( sector: 'Industrials' ).all
+
+# SELECT "constituents".*
+#         FROM "constituents"
+#         WHERE "constituents"."sector" = "Industrials"
+# => [#<Constituent:0x9f8cb78
+#          id:     1,
+#          symbol: "MMM",
+#          name:   "3M Co",
+#          sector: "Industrials">,
+#      #<Constituent:0xa2a4180
+#          id:     8,
+#          symbol: "ADT",
+#          name:   "ADT Corp (The)",
+#          sector: "Industrials">,...]
+```
+
+and so on
+
+
+
+### Frequently Asked Questions (F.A.Qs) and Answers
+
+
+#### Q: How to dowload a data package ("by hand")?
+
+Use the `CsvPack::Downloader` class to download a data package
+to your disk (by default data packages get stored in `./pack`).
+
+``` ruby
+dl = CsvPack::Downloader.new
+dl.fetch( 'language-codes' )
+dl.fetch( 's-and-p-500-companies' )
+dl.fetch( 'un-locode')
+```
+
+Will result in:
+
+```
+-- pack
+   |-- language-codes
+   |   |-- data
+   |   |   |-- language-codes-3b2.csv
+   |   |   |-- language-codes.csv
+   |   |   `-- language-codes-full.csv
+   |   `-- datapackage.json
+   |-- s-and-p-500-companies
+   |   |-- data
+   |   |   |-- constituents.csv
+   |   |   `-- constituents-financials.csv
+   |   `-- datapackage.json
+   `-- un-locode
+       |-- data
+       |   |-- code-list.csv
+       |   |-- country-codes.csv
+       |   |-- function-classifiers.csv
+       |   |-- status-indicators.csv
+       |   `-- subdivision-codes.csv
+       `-- datapackage.json
+```
+
+
+#### Q: How to add and import a data package ("by hand")?
+
+Use the `CsvPack::Pack` class to read-in a data package
+and add and import into an SQL database.
+
+``` ruby
+pack = CsvPack::Pack.new( './pack/un-locode/datapackage.json' )
+pack.tables.each do |table|
+  table.up!      # (auto-) add table  using SQL create_table via ActiveRecord migration
+  table.import!  # import all records using SQL inserts
+end
+```
+
+
+#### Q: How to connect to a different SQL database?
+
+You can connect to any database supported by ActiveRecord. If you do NOT
+establish a connection in your script - the standard (default fallback)
+is using an in-memory SQLite3 database.
+
+##### SQLite
+
+For example, to create an SQLite3 database on disk - lets say `mine.db` -
+use in your script (before the `CsvPack.import` statement):
+
+``` ruby
+ActiveRecord::Base.establish_connection( adapter:  'sqlite3',
+                                         database: './mine.db' )
+```
+
+##### PostgreSQL
+
+For example, to connect to a PostgreSQL database use in your script
+(before the `CsvPack.import` statement):
+
+``` ruby
+require 'pg'       ##  pull-in PostgreSQL (pg) machinery
+
+ActiveRecord::Base.establish_connection( adapter:  'postgresql'
+                                         username: 'ruby',
+                                         password: 'topsecret',
+                                         database: 'database' )
+```
+
+
+
+
+## Install
+
+Just install the gem:
+
+```
+$ gem install csvpack
+```
+
+
+
+## Alternatives
+
+See the "[Tools and Plugins for working with Data Packages](https://frictionlessdata.io/software)"
+page at the Frictionless Data Initiative.
+
+
+## License
+
+
+The `csvpack` scripts are dedicated to the public domain.
+Use it as you please with no restrictions whatsoever.
+
+## Questions? Comments?
+
+Send them along to the ruby-talk mailing list. Thanks!

data/Rakefile

@@ -0,0 +1,32 @@
+require 'hoe'
+require './lib/csvpack/version.rb'
+
+Hoe.spec 'csvpack' do
+
+  self.version = CsvPack::VERSION
+
+  self.summary = 'csvpack - work with tabular data packages using comma-separated values (CSV) datafiles in text with datapackage.json; download, read into and query comma-separated values (CSV) datafiles with your SQL database (e.g. SQLite, PostgreSQL, ...) of choice and much more'
+  self.description = summary
+
+  self.urls    = ['https://github.com/csv11/csvpack']
+
+  self.author  = 'Gerald Bauer'
+  self.email   = 'ruby-talk@ruby-lang.org'
+
+  # switch extension to .markdown for gihub formatting
+  self.readme_file  = 'README.md'
+  self.history_file = 'HISTORY.md'
+
+  self.extra_deps = [
+    ['logutils',     '>=0.6.1'],
+    ['fetcher',      '>=0.4.5'],
+    ['activerecord', '>=5.0.0'],
+  ]
+
+  self.licenses = ['Public Domain']
+
+  self.spec_extras = {
+    required_ruby_version: '>= 2.2.2'
+  }
+
+end

data/lib/csvpack.rb

@@ -0,0 +1,52 @@
+# encoding: utf-8
+
+
+require 'pp'
+require 'forwardable'
+
+### csv
+require 'csv'
+require 'json'
+require 'fileutils'
+
+
+### downloader
+require 'fetcher'
+
+### activerecord w/ sqlite3
+##  require 'active_support/all'    ## needed for String#binary? method
+require 'active_record'
+
+
+
+# our own code
+
+require 'csvpack/version'      ## let version always go first
+require 'csvpack/pack'
+require 'csvpack/downloader'
+
+module CsvPack
+
+  def self.import( *args )
+    ## step 1: download
+    dl = Downloader.new
+    args.each do |arg|
+      dl.fetch( arg )
+    end
+
+    ## step 2: up 'n' import
+    args.each do |arg|
+      pack = Pack.new( "./pack/#{arg}/datapackage.json" )
+      pack.tables.each do |table|
+        table.up!
+        table.import!
+      end
+    end
+  end
+
+end # module CsvPack
+
+
+
+# say hello
+puts CsvPack.banner    if defined?($RUBYLIBS_DEBUG) && $RUBYLIBS_DEBUG

data/lib/csvpack/downloader.rb

@@ -0,0 +1,62 @@
+# encoding: utf-8
+
+module CsvPack
+
+class Downloader
+
+  def initialize( cache_dir='./pack' )
+    @cache_dir = cache_dir   # todo: check if folder exists now (or on demand)?
+    @worker = Fetcher::Worker.new
+  end
+
+  SHORTCUTS = {
+    ## to be done
+  }
+
+  def fetch( name_or_shortcut_or_url )   ## todo/check: use (re)name to get/update/etc. why? why not??
+
+    name = name_or_shortcut_or_url
+
+    ##
+    ## e.g. try
+    ##   country-list
+    ##
+
+    ## url_base = "http://data.okfn.org/data/core/#{name}"
+    url_base = "https://datahub.io/core/#{name}"
+    url = "#{url_base}/datapackage.json"
+
+    dest_dir = "#{@cache_dir}/#{name}"
+    FileUtils.mkdir_p( dest_dir )
+
+    pack_path = "#{dest_dir}/datapackage.json"
+    @worker.copy( url, pack_path )
+
+    h = JSON.parse( File.read( pack_path ) )
+    pp h
+
+    ## copy resources (tables)
+    h['resources'].each do |r|
+      puts "== resource:"
+      pp r
+
+      res_url       = r['url']
+
+      res_name          = r['name']
+      res_relative_path = r['path']
+      if res_relative_path.nil?
+        res_relative_path = "#{res_name}.csv"
+      end
+
+      res_path = "#{dest_dir}/#{res_relative_path}"
+      puts "[debug] res_path: >#{res_path}<"
+      res_dir   = File.dirname( res_path )
+      FileUtils.mkdir_p( res_dir )
+
+      @worker.copy( res_url, res_path )
+    end
+  end
+
+end # class Downloader
+
+end # module CsvPack

data/lib/csvpack/pack.rb

@@ -0,0 +1,246 @@
+# encoding: utf-8
+
+
+## note: for now use in-memory sqlite3 db
+
+module CsvPack
+
+class Pack
+  ## load (tabular) datapackage into memory
+  def initialize( path )
+
+    ## convenience
+    ## - check: if path is a folder/directory
+    ##    (auto-)add  /datapackage.json
+
+    text = File.open( path, 'r:utf-8' ).read
+    @h = JSON.parse( text )
+
+    pack_dir = File.dirname(path)
+
+    ## pp @h
+
+    ## read in tables
+    @tables = []
+    @h['resources'].each do |r|
+      ## build table data
+      @tables << build_tab( r, pack_dir )
+    end
+
+    ## pp @tables
+  end
+
+  def name()    @h['name']; end
+  def title()   @h['title']; end
+  def license() @h['license']; end
+
+  def tables()  @tables; end
+  ## convenience method - return first table
+  def table()   @tables[0]; end
+
+  def build_tab( h, pack_dir )
+    name          = h['name']
+    relative_path = h['path']
+
+    if relative_path.nil?
+      relative_path = "#{name}.csv"
+      puts "  warn: no path defined; using fallback '#{relative_path}'"
+    end
+
+    puts "  reading resource (table) #{name} (#{relative_path})..."
+    pp h
+
+    path = "#{pack_dir}/#{relative_path}"
+    text = File.open( path, 'r:utf-8' ).read
+    tab = Tab.new( h, text )
+    tab
+  end
+end # class Pack
+
+
+class Tab
+  extend Forwardable
+
+  def initialize( h, text )
+    @h = h
+
+    ## todo parse csv
+    ##  note: use header options (first row MUST include headers)
+    @data = CSV.parse( text, headers: true )
+
+    pp @data[0]
+  end
+
+  def name()  @h['name']; end
+  def_delegators :@data, :[], :each
+
+  def pretty_print( printer )
+    printer.text "Tab<#{object_id} @data.name=#{name}, @data.size=#{@data.size}>"
+  end
+
+
+  def up!
+    # run Migration#up to create table
+    connect!
+    con = ActiveRecord::Base.connection
+
+    con.create_table sanitize_name( name ) do |t|
+      @h['schema']['fields'].each do |f|
+        column_name = sanitize_name(f['name'])
+        column_type = DATA_TYPES[f['type']]
+
+        puts "  #{column_type} :#{column_name}  =>  #{f['type']} - #{f['name']}"
+
+        t.send( column_type.to_sym, column_name.to_sym )   ## todo/check: to_sym needed?
+      end
+      t.string  :name
+    end
+  end
+
+  def import!
+    connect!
+    con = ActiveRecord::Base.connection
+
+    column_names = []
+    column_types = []
+    column_placeholders = []
+    @h['schema']['fields'].each do |f|
+      column_names << sanitize_name(f['name'])
+      column_types << DATA_TYPES[f['type']]
+      column_placeholders << '?'
+    end
+
+    sql_insert_into = "INSERT INTO #{sanitize_name(name)} (#{column_names.join(',')}) VALUES "
+    puts sql_insert_into
+
+    i=0
+    @data.each do |row|
+      i+=1
+      ## next if i > 3   ## for testing; only insert a couple of recs
+
+      ## todo: check if all string is ok; or number/date/etc. conversion needed/required?
+      values = []
+      row.fields.each_with_index do |value,index|   # get array of values
+        type = column_types[index]
+        ## todo add boolean ??
+        if value.blank?
+          values << 'NULL'
+        elsif [:number,:float,:integer].include?( type )
+          values << value           ## do NOT wrap in quotes (numeric)
+        else
+          esc_value = value.gsub( "'", "''" )  ## escape quotes e.g. ' becomse \'\', that is, double quotes
+          values << "'#{esc_value}'"    ## wrap in quotes
+        end
+      end
+      pp values
+
+      sql = "#{sql_insert_into} (#{values.join(',')})"
+      puts sql
+      con.execute( sql )
+    end
+  end # method import!
+
+
+  def import_v1!
+     ### note: import via sql for (do NOT use ActiveRecord record class for now)
+    con = ActiveRecord::Base.connection
+
+    column_names = []
+    column_types = []
+    column_placeholders = []
+    @h['schema']['fields'].each do |f|
+      column_names << sanitize_name(f['name'])
+      column_types << DATA_TYPES[f['type']]
+      column_placeholders << '?'
+    end
+
+    sql = "INSERT INTO #{sanitize_name(name)} (#{column_names.join(',')}) VALUES (#{column_placeholders.join(',')})"
+    puts sql
+
+    i=0
+    @data.each do |row|
+      i+=1
+      next if i > 3   ## for testing; only insert a couple of recs
+
+      ## todo: check if all string is ok; or number/date/etc. conversion needed/required?
+      params = row.fields   # get array of values
+      pp params
+      con.exec_insert( sql, 'SQL', params )  # todo/check: 2nd param name used for logging only??
+    end
+  end # method import!
+
+
+  ### note:
+  ## activerecord supports:
+  ##  :string, :text, :integer, :float, :decimal, :datetime, :time, :date, :binary, :boolean
+
+  ### mappings for data types
+  ##  from tabular data package to ActiveRecord migrations
+  ##
+  #  see http://dataprotocols.org/json-table-schema/   (section Field Types and Formats)
+  #
+  # for now supports these types
+
+  DATA_TYPES = {
+    'string'   => :string,    ## use text for larger strings ???
+    'number'   => :float,     ## note: use float for now
+    'integer'  => :integer,
+    'boolean'  => :boolean,
+    'datetime' => :datetime,
+    'date'     => :date,
+    'time'     => :time,
+  }
+
+  def dump_schema
+    ## try to dump schema (fields)
+    puts "*** dump schema:"
+
+    @h['schema']['fields'].each do |f|
+      puts "   #{f['name']} ( #{sanitize_name(f['name'])} ) : #{f['type']}} ( #{DATA_TYPES[f['type']]} )"
+    end
+
+  end
+
+
+  def sanitize_name( ident )
+    ##
+    ## if identifier starts w/ number add leading underscore (_)
+    ##  e.g. 52 Week Price  => becomes  _52_week_price
+
+    ident = ident.strip.downcase
+    ident = ident.gsub( /[\.\-\/]/, '_' )  ## convert some special chars to underscore (e.g. dash -)
+    ident = ident.gsub( ' ', '_' )
+    ident = ident.gsub( /[^a-z0-9_]/, '' )
+    ident = "_#{ident}"  if ident =~ /^[0-9]/
+    ident
+  end
+
+
+  def ar_clazz
+    @ar_clazz ||= begin
+      clazz = Class.new( ActiveRecord::Base ) do
+        ## nothing here for now
+      end
+      puts "set table_name to #{sanitize_name( name )}"
+      clazz.table_name = sanitize_name( name )
+      clazz
+    end
+    @ar_clazz
+  end
+
+private
+
+  ## helper to get connection; if not connection established use defaults
+  def connect!
+    ## todo: cache returned con - why, why not ??
+    unless ActiveRecord::Base.connected?
+      puts "note: no database connection established; using defaults (e.g. in-memory SQLite database)"
+      ActiveRecord::Base.establish_connection( adapter:  'sqlite3', database: ':memory:' )
+      ActiveRecord::Base.logger = Logger.new( STDOUT )
+    end
+    ActiveRecord::Base.connection
+  end
+
+end # class Tab
+
+end # module CsvPack

data/lib/csvpack/version.rb

@@ -0,0 +1,22 @@
+# encoding: utf-8
+
+module CsvPack
+
+  MAJOR = 0 ## todo: namespace inside version or something - why? why not??
+  MINOR = 1
+  PATCH = 0
+  VERSION = [MAJOR,MINOR,PATCH].join('.')
+
+  def self.version
+    VERSION
+  end
+
+  def self.banner
+    "csvpack/#{VERSION} on Ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE}) [#{RUBY_PLATFORM}]"
+  end
+
+  def self.root
+    File.expand_path( File.dirname(File.dirname(File.dirname(__FILE__))) )
+  end
+
+end  # module CsvPack

data/test/helper.rb

@@ -0,0 +1,7 @@
+
+## minitest setup
+require 'minitest/autorun'
+
+
+## our own code
+require 'csvpack'

data/test/test_companies.rb

@@ -0,0 +1,61 @@
+# encoding: utf-8
+
+###
+#  to run use
+#     ruby -I ./lib -I ./test test/test_companies.rb
+
+
+require 'helper'
+
+class TestCompanies < MiniTest::Test
+
+  def test_s_and_p_500_companies
+
+    pak = Datapak::Pak.new( './pak/s-and-p-500-companies/datapackage.json' )
+
+    puts "name: #{pak.name}"
+    puts "title: #{pak.title}"
+    puts "license: #{pak.license}"
+
+    pp pak.tables
+    pp pak.table[0]['Symbol']
+    pp pak.table[495]['Symbol']
+
+    ## pak.table.each do |row|
+    ##  pp row
+    ## end
+
+    puts pak.tables[0].dump_schema
+    puts pak.tables[1].dump_schema
+
+    # database setup 'n' config
+    ActiveRecord::Base.establish_connection( adapter:  'sqlite3', database: ':memory:' )
+    ActiveRecord::Base.logger = Logger.new( STDOUT )
+
+    pak.table.up!
+    pak.table.import!
+
+    pak.tables[1].up!
+    pak.tables[1].import!
+
+
+    pp pak.table.ar_clazz
+
+
+    company = pak.table.ar_clazz
+
+    puts "Company.count: #{company.count}"
+    pp company.first
+    pp company.find_by!( symbol: 'MMM' )
+    pp company.find_by!( name: '3M Co' )
+    pp company.where( sector: 'Industrials' ).count
+    pp company.where( sector: 'Industrials' ).all
+
+
+    ### todo: try a join w/ belongs_to ??
+
+    assert true  # if we get here - test success
+  end
+
+end # class TestCompanies
+

data/test/test_countries.rb

@@ -0,0 +1,40 @@
+# encoding: utf-8
+
+###
+#  to run use
+#     ruby -I ./lib -I ./test test/test_countries.rb
+
+
+require 'helper'
+
+class TestCountries < MiniTest::Test
+
+  def test_country_list
+    pak = Datapak::Pak.new( './pak/country-list/datapackage.json' )
+
+    puts "name: #{pak.name}"
+    puts "title: #{pak.title}"
+    puts "license: #{pak.license}"
+
+    pp pak.tables
+
+    ## pak.table.each do |row|
+    ##  pp row
+    ## end
+
+    puts pak.table.dump_schema
+
+    # database setup 'n' config
+    ActiveRecord::Base.establish_connection( adapter:  'sqlite3', database: ':memory:' )
+    ActiveRecord::Base.logger = Logger.new( STDOUT )
+
+    pak.table.up!
+    pak.table.import!
+
+    pp pak.table.ar_clazz
+
+    assert true  # if we get here - test success
+  end
+
+end # class TestCountries
+

data/test/test_downloader.rb

@@ -0,0 +1,32 @@
+# encoding: utf-8
+
+###
+#  to run use
+#     ruby -I ./lib -I ./test test/test_downloader.rb
+
+
+require 'helper'
+
+class TestDownloader < MiniTest::Test
+
+  def test_download
+    
+    names = [
+      'country-list',
+      'country-codes',
+      'language-codes',
+      'cpi',                    ## Annual Consumer Price Index (CPI) 
+      'gdp',                    ## Country, Regional and World GDP (Gross Domestic Product)
+      's-and-p-500-companies',  ## S&P 500 Companies with Financial Information 
+      'un-locode',              ## UN-LOCODE Codelist  - note: incl. country-codes.csv
+    ]
+
+    dl = Datapak::Downloader.new
+    names.each do |name|
+      dl.fetch( name )
+    end
+
+    assert true  # if we get here - test success
+  end
+
+end # class TestDownloader

data/test/test_import.rb

@@ -0,0 +1,22 @@
+# encoding: utf-8
+
+###
+#  to run use
+#     ruby -I ./lib -I ./test test/test_import.rb
+
+
+require 'helper'
+
+class TestImport < MiniTest::Test
+
+  def test_import
+
+    CsvPack.import(
+      'cpi',                    ## Annual Consumer Price Index (CPI)
+      'gdp',                    ## Country, Regional and World GDP (Gross Domestic Product)
+    )
+
+    assert true  # if we get here - test success
+  end
+
+end # class TestImport

metadata

@@ -0,0 +1,137 @@
+--- !ruby/object:Gem::Specification
+name: csvpack
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Gerald Bauer
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2018-08-07 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logutils
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.6.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.6.1
+- !ruby/object:Gem::Dependency
+  name: fetcher
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.4.5
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.4.5
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 5.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 5.0.0
+- !ruby/object:Gem::Dependency
+  name: rdoc
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: hoe
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.16'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.16'
+description: csvpack - work with tabular data packages using comma-separated values
+  (CSV) datafiles in text with datapackage.json; download, read into and query comma-separated
+  values (CSV) datafiles with your SQL database (e.g. SQLite, PostgreSQL, ...) of
+  choice and much more
+email: ruby-talk@ruby-lang.org
+executables: []
+extensions: []
+extra_rdoc_files:
+- HISTORY.md
+- Manifest.txt
+- README.md
+files:
+- HISTORY.md
+- Manifest.txt
+- README.md
+- Rakefile
+- lib/csvpack.rb
+- lib/csvpack/downloader.rb
+- lib/csvpack/pack.rb
+- lib/csvpack/version.rb
+- test/helper.rb
+- test/test_companies.rb
+- test/test_countries.rb
+- test/test_downloader.rb
+- test/test_import.rb
+homepage: https://github.com/csv11/csvpack
+licenses:
+- Public Domain
+metadata: {}
+post_install_message: 
+rdoc_options:
+- "--main"
+- README.md
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.2.2
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.2
+signing_key: 
+specification_version: 4
+summary: csvpack - work with tabular data packages using comma-separated values (CSV)
+  datafiles in text with datapackage.json; download, read into and query comma-separated
+  values (CSV) datafiles with your SQL database (e.g. SQLite, PostgreSQL, ...) of
+  choice and much more
+test_files: []