rstore 0.2.0

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2011 Stefan Rohlfing
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,253 @@
+# RStore 
+
+### A library for easy batch storage of csv data into a database
+
+Uses the CSV standard library for parsing, *Nokogiri* for URL handling, and *Sequel* ORM for database management.
+
+## Special Features
+
+* **Batch processing** of csv files  
+* Fetches data from different sources: **files, directories, URLs**  
+* **Customizable** using additional options (also see section *Available Options*)  
+* **Validation of field values**. At the moment validation of the following types is supported:  
+  * `String`, `Integer`, `Float`, `Date`, `DateTime`, `Time`, and `Boolean` 
+* **Descriptive error messages** pointing helping you to find any invalid data quickly.  
+* Only define your database and table classes once, then just `require` them when needed.  
+* **Safe and transparent data storage**: 
+  * Using database transactions: Either the data from all all files is stored or none (also see section *Database Requirements*)  
+  * To avoid double entry of data, the `run` method can only be run once on a single instance of `RStore::CSV`.
+
+
+## Database Requirements
+  
+1. Expects the database table to have an addition column storing an auto-incrementing primary key.
+2. **Requires the database to support transactions**:  
+   Most other database platforms support transactions natively.  
+   In MySQL, you'll need to be running `InnoDB` or `BDB` table types rather than the more common `MyISAM`.  
+   If you are using MySQL and the table has not been created yet, RStore::CSV will take care of using the  
+   correct table type upon creation.
+
+
+## Installation
+
+``` batch
+gem install 'rstore'
+```
+
+## Public API Documentation
+
+A detailed documentation of the public API can be found [here](http://rubydoc.info/github/bytesource/rstore).
+
+
+## Sample Usage
+
+Sample csv file
+
+> "product","quantity","price","created_at","min_demand","max_demand","on_stock"  
+> "toy1","1","1.12","2011-2-4","1:30","1:30am","true"  
+> "toy2","2","2.22","2012/2/4","2:30","2:30pm","false  
+> "toy3","3","3.33","2013/2/4","3:30","3:30 a.m.","True  
+> "toy4","4",,,"4:30","4:30 p.m.","False"  
+> "toy4","5","5.55","2015-2-4","5:30","5:30AM","1"  
+> "toy5","6","6.66","2016/2/4","6:30","6:30 P.M.","0"  
+> "toy6","7","7.77",,,,"false"  
+
+
+1) Load gem
+
+``` ruby    
+
+require 'rstore/csv'
+
+```
+2) Store database information in a subclass of `RStore::BaseDB`  
+Naming convention: name => NameDB
+
+``` ruby 
+
+class CompanyDB < RStore::BaseDB
+
+  # Same as Sequel.connect, except that you don't need to
+  # provide the :database key.
+  info(:adapter  => 'mysql', 
+       :host     => 'localhost',
+       :user     => 'root',
+       :password => 'xxx')
+
+end
+
+```
+
+3) Store table information in a subclass of `RStore::BaseTable`  
+Naming convention: name => NameTable
+
+``` ruby 
+
+class ProductsTable < RStore::BaseTable
+
+  # Specify the database table the same way
+  # you do in Sequel
+  create do
+    primary_key :id, :allow_null => false
+    String      :product
+    Integer     :quantity
+    Float       :price
+    Date        :created_at
+    DateTime    :min_demand
+    Time        :max_demand
+    Boolean     :on_stock, :allow_null => false, :default => false
+  end
+
+end
+
+```  
+
+**Note**:  
+You can put the database and table class definitions in separate files
+and `require` them when needed.
+
+
+4) Enter csv data into the database  
+The `from` method accepts a path to a file or directory as well as an URL.  
+The `to` metthod accepts a string of the form *db_name.table_name*  
+
+```ruby
+RStore::CSV.new do
+  from '../easter/children', :recursive => true                   # select a directory or
+  from '../christmas/children/toys.csv'                           # file, or
+  from 'www.example.com/sweets.csv', :selector => 'pre div.line'  # URL
+  to   'company.products'                                         # provide database and table name
+  run                                                             # run the program
+end
+
+```
+### Additional Features
+--- 
+
+You can change and reset the default options (see section *Available Options* below for details)
+
+``` ruby  
+# Search directories recursively and handle the first row of a file as data by default
+RStore::CSV.change_default_options(:recursive => true, :has_headers => false) 
+
+RStore::CSV.new do
+  from 'dir1'              
+  from 'dir2'
+  from 'dir3'
+  to   'company.products'
+  run
+end
+
+# Restore default options
+RStore::CSV.reset_default_options
+
+```
+
+There is also a convenience method enabling you to use  
+all of [Sequels query methods](http://sequel.rubyforge.org/rdoc/files/doc/querying_rdoc.html).
+
+``` ruby  
+RStore::CSV.query('company.products') do |table|    # table = Sequel::Dataset object 
+  table.all                                         # fetch everything 
+  table.all[3]                                      # fetch row number 4 (see output below)
+  table.filter(:id => 2).update(:on_stock => true)  # update entry
+  table.filter(:id => 3).delete                     # delete entry
+end
+
+```
+
+*)
+Output of `db[table.name].all[3]`
+
+``` ruby   
+# {:product     => "toy4",
+#  :quantity    => 4,
+#  :price       => nil,
+#  :created_at  => nil,
+#  :min_demand  => <DateTime: 2011-10-25T04:30:00+00:00 (39293755/16,0/1,2299161)>,
+#  :max_demand  => <DateTime: 2011-10-25T16:30:00+00:00 (39293763/16,0/1,2299161)>,
+#  :on_stock    => false}
+
+```
+
+Access all of Sequels functionality by using the convenience methods   
+`BaseDB.connect`, `BaseTable.name`, and `BaseTable.table_info`:  
+
+``` ruby  
+
+DB     = CompanyDB.connect           # Open connection to 'company' database
+name   = ProductTable.name           # Table name, :products, used as an argument to the following methods.
+layout = ProductsTable.table_info    # The Proc that was passed to ProductsTable.create
+
+DB.create_table(name, &layout)       # Create table
+
+DB.alter_table name do               # Alter table
+  drop_column :created_at
+  add_column  :entry_date, :date
+end
+
+DB.drop_table(name)                  # Drop table
+
+```
+
+
+## Available Options
+
+The method `from` accepts two kinds of options, file options and parse options:
+
+### File Options
+File options are used for fetching csv data from a source. The following options are recognized:
+
+* **:has_headers**, default: `true` 
+    * When set to false, the first line of a file is processed as data, otherwise it is discarded.
+* **:recursive**, default: `false` 
+    * When set to true and a directory is given, recursively search for files. Non-csv files are skipped. 
+* **:selector**, default: `""` 
+    * Mandatory css selector with an URL. For more details please see the section *Further Reading* below 
+ 
+  
+### Parse Options
+Parse options are arguments to `CSV::parse`. The following options are recognized:
+
+* **:col_sep**, default: `","`
+    * The String placed between each field.
+* **:row_sep**, default: `:auto`
+    * The String appended to the end of each row.
+* **:quote_char**, default: `'"'`
+    * The character used to quote fields.
+* **:field_size_limit**, default: `nil`
+    * The maximum size CSV will read ahead looking for the closing quote for a field.
+* **:skip_blanks**, default: `false`
+    * When set to a true value, CSV will skip over any rows with no content.
+
+For more information on the parse options, please see section *Further Reading* below.
+
+
+## Further Reading  
+
+* Sequel
+  * [Cheat sheet][sequel_cheat]
+  * [Connecting to a database][sequel_connect]
+  * [Querying][sequel_query]
+* CSV
+  * [Parse options documentation][csv_options]
+  * [Common Format and MIME Type][csv_standard] for Comma-Separated Values (CSV) Files
+* Nokogiri
+  * [Project Site][nokogiri_home]
+
+[sequel_cheat]: http://sequel.rubyforge.org/rdoc/files/doc/cheat_sheet_rdoc.html
+[sequel_connect]: http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html
+[sequel_query]: http://sequel.rubyforge.org/rdoc/files/doc/querying_rdoc.html
+[csv_options]: http://ruby-doc.org/stdlib-1.9.2/libdoc/csv/rdoc/CSV.html#method-c-new
+[csv_standard]: http://www.ietf.org/rfc/rfc4180.txt
+[nokogiri_home]: http://nokogiri.org/ 
+
+
+## Feedback
+
+Any suggestions or criticism are highly welcome! Whatever your feedback, it will help me make this gem better!
+
+
+
+
+

data/Rakefile

@@ -0,0 +1,40 @@
+# require 'spec/rake/spectask' # depreciated
+require 'rspec/core/rake_task'
+require 'rake/gempackagetask'
+require 'rdoc/task'
+
+# Build gem: rake gem
+# Push gem:  rake push
+
+task :default => [ :spec, :gem ]
+
+RSpec::Core::RakeTask.new do
+
+  message = <<-MESSAGE
+
+  ====================================
+  | NOTE:
+  | Make sure to provide the correct connection info for database 'PlastronicsDB'
+  | in file 'csv_spec.rb' (line 12).
+  ====================================
+
+  MESSAGE
+
+  puts message
+
+  :spec
+end
+
+gem_spec = eval(File.read('rstore.gemspec'))
+
+Rake::GemPackageTask.new( gem_spec ) do |t|
+  t.need_zip = true
+end
+
+#RDoc::Task.new do |rdoc|
+#
+#end
+
+task :push => :gem do |t|
+  sh "gem push pkg/#{gem_spec.name}-#{gem_spec.version}.gem"
+end

data/lib/rstore.rb

@@ -0,0 +1,5 @@
+# encoding: utf-8
+
+module RStore
+  require 'rstore/csv'
+end

data/lib/rstore/base_db.rb

@@ -0,0 +1,119 @@
+# encoding: utf-8
+
+require 'sequel'
+
+module RStore
+  class BaseDB
+
+    class << self
+      # A Hash holding subclasses of {RStore::BaseDB}
+      # @return [Hash{Symbol=>BaseDB}] All subclasses of {RStore::BaseDB} defined in the current namespace.  
+      #   Subclasses are added automatically via _self.inherited_.
+      # @example
+      #   class CompanyDB < RStore::BaseDB
+      #     info #...
+      #   end
+      #
+      #   class MyDB < RStore::BaseDB
+      #     info #...
+      #   end
+      #
+      #   RStore::BaseDB.db_classes
+      #   #=> {:company=>companyDB, :my=>MyDB}
+      attr_reader :db_classes # self = #<Class:RStore::BaseDB>
+    end
+
+    @db_classes = Hash.new     # self = RStore::BaseDB
+
+
+    def self.inherited subclass
+      BaseDB.db_classes[subclass.name] = subclass
+    end
+
+   
+    # Define the database connection.
+    # @note To be called when defining a _subclass_ of {RStore::BaseDB}
+    # Accepts the same _one_ _arity_ parameters as [Sequel.connect](http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html)
+    # @overload info(options)
+    #  @param [String, Hash] connection_info Either a connection string such as _postgres://user:password@localhost/blog_, or a `Hash` with the following options:
+    #  @option options [String] :adapter The SQL database used, such as _mysql_ or _postgres_ 
+    #  @option options [String] :host Example: 'localhost'
+    #  @option options [String] :user 
+    #  @option options [String] :password 
+    #  @option options [String] :database The database name. You don't need to provide this option, as its value will be inferred from the class name. 
+    #  @return [void]
+    # @example
+    #  # Using a connection string
+    #  class CompanyDB < RStore::BaseDB
+    #    info('postgres://user:password@localhost/blog')
+    #  end
+    #
+    #  # Using an options hash
+    #  class CompanyDB < RStore::BaseDB
+    #    info(adapter: 'mysql', 
+    #         host: 'localhost', 
+    #         user: 'root', 
+    #         password: 'xxx')
+    #  end
+    def self.info hash_or_string
+      class << self            # self = #<Class:CompanyDB>
+        attr_reader :connection_info
+      end
+                               # self = CompanyDB
+      @connection_info = hash_or_string.is_a?(Hash) ? hash_or_string.merge(:database => self.name.to_s): hash_or_string
+    end
+
+
+    # Uses the connection info from {.info} to connect to the database.  
+    # @note To be called when defining a _subclass_ of {RStore::BaseDB}
+    # @yieldparam [Sequel::Database] db The opened Sequel {http://sequel.rubyforge.org/rdoc/classes/Sequel/Database.html Database} object, which is closed when the block exits.
+    # @return [void]
+    # @example
+    #  class CompanyDB < RStore::BaseDB
+    #    info(adapter: 'mysql', 
+    #         host: 'localhost', 
+    #         user: 'root', 
+    #         password: 'xxx')
+    #  end
+    #
+    #  class DataTable < RStore::BaseTable
+    #    create do
+    #      primary_key :id, :allow_null => false
+    #      String      :col1
+    #      Integer     :col2
+    #      Float       :col3
+    #    end
+    #  end
+    #
+    #  name = DataTable.name
+    #
+    #  #Either
+    #  DB = CompanyDB.connect
+    #  DB.drop_table(name)
+    #  DB.disconnect
+    #
+    #  #Or
+    #  CompanyDB.connect do |db|
+    #    db.drop_table(name)
+    #  end
+    def self.connect &block
+      Sequel.connect(@connection_info, &block)
+    end
+
+
+
+    # @return [Symbol] The lower-case class name without the _DB_ postfix
+    # @example
+    #  class CompanyDB < RStore::BaseDB
+    #    info('postgres://user:password@localhost/blog')
+    #  end
+    #
+    #  CompanyDB.name
+    #  #=> :company
+    def self.name  
+      super.gsub!(/DB/,'').downcase.to_sym
+    end
+
+  end
+end
+ 

data/lib/rstore/base_table.rb

@@ -0,0 +1,92 @@
+# encoding: utf-8
+
+module RStore
+  class BaseTable
+
+    class << self
+      # A Hash holding subclasses of {RStore::BaseTable}
+      # @return [Hash{Symbol=>BaseTable}] All subclasses of {RStore::BaseTable} defined in the current namespace.   
+      #   Subclasses are added automatically via _self.inherited_.
+      # @example
+      #   class ProductsTable < RStore::BaseTable
+      #     create #...
+      #   end
+      #
+      #   class DataTable < RStore::BaseTable
+      #     create #...
+      #   end
+      #
+      #   RStore::BaseTable.table_classes
+      #   #=> {:products=>ProductsTable, :data=>DataTable} 
+      attr_reader :table_classes
+    end
+
+    @table_classes = Hash.new
+
+    def self.inherited subclass
+      BaseTable.table_classes[subclass.name] = subclass
+    end
+
+
+    # Define the table layout.
+    # @note To be called when defining of a _subclass_ of {RStore::BaseTable}
+    # @note You need to define an extra column for an auto-incrementing primary key.  
+    # @yield The same block as {http://sequel.rubyforge.org/rdoc/classes/Sequel/Database.html#method-i-create_table Sequel::Database.create_table}.  
+    # @return [void]
+    # @example
+    #  class ProductsTable < RStore::BaseTable
+    #
+    #    create do
+    #      primary_key :id, :allow_null => false
+    #      String      :product
+    #      Integer     :quantity
+    #      Float       :price
+    #      Date        :created_at
+    #      DateTime    :min_demand
+    #      Time        :max_demand
+    #      Boolean     :on_stock, :allow_null => false, :default => false
+    #    end
+    #  end
+    def self.create &block
+      class << self
+        attr_reader :table_info
+      end
+
+      @table_info = block
+    end
+
+
+    # @return [Symbol] The lower-case class name without the _DB_ postfix
+    # @example
+    #  class CompanyDB < RStore::BaseDB
+    #    info('postgres://user:password@localhost/blog')
+    #  end
+    #
+    #  CompanyDB.name
+    #  #=> :company
+
+    
+    # @return [Symbol] The lower-case class name without the _Table_ postfix
+    # @example
+    #  class ProductsTable < RStore::BaseTable
+    #
+    #    create do
+    #      primary_key :id, :allow_null => false
+    #      String      :product
+    #      Integer     :quantity
+    #      Float       :price
+    #      Date        :created_at
+    #      DateTime    :min_demand
+    #      Time        :max_demand
+    #      Boolean     :on_stock, :allow_null => false, :default => false
+    #    end
+    #  end
+    #
+    #  ProductsTable.name
+    #  #=> :products
+    def self.name  
+      super.gsub!(/Table/,'').downcase.to_sym
+    end
+  end
+end
+