conformist 0.0.3 → 0.1.0

data/{CHANGELOG.markdown → CHANGELOG.md}

@@ -1,5 +1,15 @@
 # CHANGELOG
 
+## 0.1.0 / 2012-01-05
+
+* Added anonymous schemas.
+* Added `Conformist::Schema::Methods#conform` for lazily applying schema to input.
+* Added capability to access columns with methods.
+* FasterCSV is no longer included, use `require 'fastercsv'` instead.
+* `include Conformist::Base` has been removed.
+* `Conformist.foreach` has been removed.
+* `Conformist::Base::ClassMethods#load` has been removed.
+
 ## 0.0.3 / 2011-05-07
 
 * Inheriting from a class which mixes in Conformist::Base gives you access to all of the superclasses' columns.

data/README.md

@@ -0,0 +1,299 @@
+# Conformist 
+
+Bend CSVs to your will. Stop using array indexing and start using declarative schemas. Declarative schemas are easier to understand, quicker to setup and independent of I/O.
+
+![](http://f.cl.ly/items/00191n3O1J2E1a342F1L/conformist.jpg)
+
+## Quick and Dirty Examples
+
+Open a CSV file and declare a schema.
+
+``` ruby
+require 'csv'
+require 'conformist'
+
+csv    = CSV.open '~/transmitters.csv'
+schema = Conformist.new do
+  column :callsign, 1
+  column :latitude, 1, 2, 3
+  column :longitude, 3, 4, 5
+  column :name, 0 do |value|
+    value.upcase
+  end
+end
+```
+
+Insert the transmitters into a SQLite database.
+
+``` ruby
+require 'sqlite3'
+
+db = SQLite3::Database.new 'transmitters.db'
+schema.conform(csv).each do |transmitter|
+  db.execute "INSERT INTO transmitters (callsign, ...) VALUES ('#{transmitter.callsign}', ...);"
+end
+```
+
+Only insert the transmitters with the name "Mount Cooth-tha" using ActiveRecord or DataMapper.
+
+``` ruby
+transmitters = schema.conform(csv).select do |transmitter|
+  transmitter.name == 'Mount Coot-tha'
+end
+transmitter.each do |transmitter|
+  Transmitter.create! transmitter.attributes
+end
+```
+
+Source from multiple, different input files and insert transmitters together into a single database.
+
+``` ruby
+require 'conformist'
+require 'csv'
+require 'sqlite3'
+
+au_schema = Conformist.new do
+  column :callsign, 8
+  column :latitude, 10
+end
+us_schema = Conformist.new do
+  column :callsign, 1
+  column :latitude, 1, 2, 3
+end
+
+au_csv = CSV.open '~/au/transmitters.csv'
+us_csv = CSV.open '~/us/transmitters.csv'
+
+db = SQLite3::Database.new 'transmitters.db'
+
+[au_schema.conform(au_csv), us_schema.conform(us_csv)].each do |schema|
+  schema.each do |transmitter|
+    db.execute "INSERT INTO transmitters (callsign, ...) VALUES ('#{transmitter.callsign}', ...);"
+  end
+end
+```
+
+For **more examples** see `test/fixtures`, `test/schemas` and `test/unit/integration_test.rb`.
+
+## Installation
+
+Conformist is available as a gem. Install it at the command line.
+
+``` sh
+$ [sudo] gem install conformist
+```
+
+Or add it to your Gemfile and run `$ bundle install`.
+
+``` ruby
+gem 'conformist'
+```
+
+## Usage
+
+### Anonymous Schema
+
+Anonymous schemas are quick to declare and don't have the overhead of creating an explicit class.
+
+``` ruby
+citizen = Conformist.new do
+  column :name, 0, 1
+  column :email, 2
+end
+
+citizen.conform [['Tate', 'Johnson', 'tate@tatey.com']]
+```
+
+### Class Schema
+
+Class schemas are explicit. Class schemas were the only type available in earlier versions of Conformist.
+
+``` ruby
+class Citizen
+  extend Conformist
+  
+  column :name, 0, 1
+  column :email, 2
+end
+
+Citizen.conform [['Tate', 'Johnson', 'tate@tatey.com']]
+```
+
+### Conform
+
+Conform is the principle method for lazily applying a schema to the given input.
+
+``` ruby
+enumerator = schema.conform CSV.open('~/file.csv')
+enumerator.each do |row|
+  puts row.attributes
+end
+```
+
+#### Input
+
+`#conform` expects any object that responds to `#each` to return an array-like object.
+
+``` ruby
+CSV.open('~/file.csv').responds_to? :each # => true
+[[], [], []].responds_to? :each           # => true
+```
+
+#### Enumerator
+
+`#conform` is lazy, returning an [Enumerator](http://www.ruby-doc.org/core-1.9.3/Enumerator.html). Input is not parsed until you call `#each`, `#map` or any method defined in [Enumerable](http://www.ruby-doc.org/core-1.9.3/Enumerable.html). That means schemas can be assigned now and evaluated later. `#each` has the lowest memory footprint because it does not build a collection.
+
+#### Struct
+
+The argument passed into the block is a struct-like object. You can access columns as methods or keys. Columns were only accessible as keys in earlier versions of Conformist. Methods are now the preferred syntax.
+
+``` ruby
+citizen[:name] # => "Tate Johnson"
+citizen.name   # => "Tate Johnson"
+```
+
+For convenience the `#attributes` method returns a hash of key-value pairs suitable for creating ActiveRecord or DataMapper records.
+
+``` ruby
+citizen.attributes # => {:name => "Tate Johnson", :email => "tate@tatey.com"}
+```
+
+### One Column
+
+Maps the first column in the input file to `:first_name`. Column indexing starts at zero.
+
+``` ruby
+column :first_name, 0
+```
+
+### Many Columns
+
+Maps the first and second columns in the input file to `:name`.
+
+``` ruby
+column :name, 0, 1
+```
+
+Indexing is completely arbitrary and you can map any combination.
+
+``` ruby
+column :name_and_city 0, 1, 2
+```
+
+Many columns are implicitly concatenated. Behaviour can be changed by passing a block. See *preprocessing*.
+
+### Preprocessing
+
+Sometimes values need to be manipulated before they're conformed. Passing a block gets access to values. The return value of the block becomes the conformed output.
+
+``` ruby
+column :name, 0, 1 do |values|
+  values.map(&:upcase) * ' '
+end
+```
+
+Works with one column too. Instead of getting a collection of objects, one object is passed to the block.
+
+``` ruby
+column :first_name, 0 do |value|
+  value.upcase
+end
+```
+
+### Virtual Columns
+
+Virtual columns are not sourced from input. Omit the index to create a virtual column. Like real columns, virtual columns are included in the conformed output.
+
+``` ruby
+column :day do 
+  1
+end
+```
+
+### Inheritance
+
+Inheriting from a schema gives access to all of the parent schema's columns.
+
+#### Anonymous Schema
+
+Anonymous inheritance takes inspiration from Ruby's syntax for [instantiating new classes](http://ruby-doc.org/core-1.9.3/Class.html#method-c-new).
+
+``` ruby
+parent = Conformist.new do
+  column :name, 0, 1
+end
+
+child = Conformist.new parent do
+  column :category do
+    'Child'
+  end
+end
+```
+
+#### Class Schema
+
+Classical inheritance works as expected.
+
+``` ruby
+class Parent
+  extend Conformist
+
+  column :name, 0, 1
+end
+
+class Child < Citizen
+  column :category do
+    'Child'
+  end
+end
+```
+
+## Upgrading from <= 0.0.3 to 0.1.0
+
+Where previously you had
+
+``` ruby
+class Citizen
+  include Conformist::Base
+
+  column :name, 0, 1
+end
+
+Citizen.load('~/file.csv').foreach do |citizen|
+  # ...
+end
+```
+
+You should now do
+
+``` ruby
+require 'fastercsv'
+
+class Citizen
+  extend Conformist
+
+  column :name, 0, 1
+end
+
+Citizen.conform(CSV.open('~/file.csv')).each do |citizen|
+  # ...
+end
+```
+
+See CHANGELOG.md for a full list of changes.
+
+## Compatibility and Dependancies
+
+* MRI 1.9.2+
+* MRI 1.8.7
+* JRuby 1.6.5
+
+Conformist has no explicit dependencies, although `CSV` or `FasterCSV` is commonly used.
+
+## Motivation
+
+Motivation for this project came from the desire to simplify importing data from various government organisations into [Antenna Mate](http://antennamate.com). The data from each government was similar, but had completely different formatting. Some pieces of data needed preprocessing while others simply needed to be concatenated together. Not wanting to write a parser for each new government organisation, I created Conformist.
+
+## Copyright
+
+Copyright © 2011 Tate Johnson. Conformist is released under the MIT license. See LICENSE for details.

data/Rakefile

@@ -1,21 +1,9 @@
-begin
-  require 'rubygems'
-  require 'bundler'
-rescue LoadError
-  raise 'Could not load the bundler gem. Install it with `gem install bundler`.'
-end
-
-begin
-  Bundler.setup
-rescue Bundler::GemNotFound
-  raise RuntimeError, "Bundler couldn't find some gems." +
-    "Did you run `bundle install`?"
-end
-
-Bundler::GemHelper.install_tasks
-
+require 'rubygems'
+require 'bundler/gem_tasks'
+require 'bundler/setup'
 require 'rake/testtask'
-Rake::TestTask.new(:test) do |test|
+
+Rake::TestTask.new :test do |test|
   test.libs << 'test'
   test.pattern = 'test/**/*_test.rb'
 end

data/conformist.gemspec

@@ -3,22 +3,30 @@ $:.push File.expand_path("../lib", __FILE__)
 require "conformist/version"
 
 Gem::Specification.new do |s|
-  s.name        = "conformist"
-  s.version     = Conformist::VERSION
-  s.platform    = Gem::Platform::RUBY
-  s.authors     = ["Tate Johnson"]
-  s.email       = ["tate@tatey.com"]
-  s.homepage    = "https://github.com/tatey/conformist"
-  s.summary     = %q{Let multiple, different input files conform to a single interface.}
-  s.description = %q{Conformist lets you bend CSVs to your will. Let multiple, different input files conform to a single interface without rewriting your parser each time.}
+  s.name                 = "conformist"
+  s.version              = Conformist::VERSION
+  s.platform             = Gem::Platform::RUBY
+  s.authors              = ["Tate Johnson"]
+  s.email                = ["tate@tatey.com"]
+  s.homepage             = "https://github.com/tatey/conformist"
+  s.summary              = %q{Bend CSVs to your will.}
+  s.description          = %q{Stop using array indexing and start using declarative schemas.}
+  s.post_install_message = <<-EOS
+********************************************************************************
+
+  Upgrading from <= 0.0.3? You should be aware of breaking changes. See
+  https://github.com/tatey/conformist and skip to "Upgrading from 0.0.3 to 
+  0.1.0" to learn more. Conformist will raise helpful messages where necessary.
+
+********************************************************************************  
+EOS
 
   s.rubyforge_project = "conformist"
 
   s.required_ruby_version = '>= 1.8.7'
   
-  s.add_development_dependency 'rake', '~> 0.8.7'
-  s.add_development_dependency 'minitest', '~> 2.1.0'
-  s.add_dependency 'fastercsv', "~> 1.5.4"
+  s.add_development_dependency 'rake'
+  s.add_development_dependency 'minitest'
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")

data/lib/conformist.rb

@@ -1,26 +1,24 @@
-if RUBY_VERSION >= '1.9.0'
-  require 'csv'
-else
-  require 'fastercsv'
-end
-
 require 'conformist/base'
+require 'conformist/builder'
 require 'conformist/column'
-require 'conformist/row'
+require 'conformist/hash_struct'
+require 'conformist/schema'
 
 module Conformist
-  CSV = RUBY_VERSION >= '1.9.0' ? ::CSV : FasterCSV
+  unless defined? Enumerator # Compatible with 1.8
+    require 'generator'
+    Enumerator = Generator
+  end
+  
+  def self.extended base
+    base.extend Schema
+  end
+  
+  def self.foreach *args, &block
+    raise "`Conformist.foreach` has been removed, use something like `[MySchema1.conform(file1), MySchema2.conform(file2)].each(&block)` instead (#{caller.first})"
+  end
   
-  # Enumerate over each row from multiple input files.
-  #
-  # Example:
-  #
-  #   Conformist::Base.foreach Input1.load('input.csv'), Input2.load('input.csv') do |row|
-  #     Model.create! row
-  #   end
-  #
-  # Returns nothing.
-  def self.foreach *bases, &block
-    bases.each { |base| base.foreach(&block) }
+  def self.new *args, &block
+    Class.new { include Schema }.new *args, &block
   end
 end

data/lib/conformist/base.rb

@@ -1,52 +1,7 @@
 module Conformist
   module Base
     def self.included base
-      base.class_eval do
-        extend ClassMethods
-        attr_accessor :path
-      end
+      raise "`include Conformist::Base` has been removed, `extend Conformist` instead (#{caller.first})"
     end
-
-    # Enumerate over each row in the input file.
-    #
-    # Example:
-    #
-    #   input1 = Input1.load 'input1.csv'
-    #   input1.foreach do |row|
-    #     Model.create! row
-    #   end
-    #
-    # Returns nothing.
-    def foreach &block
-      CSV.foreach(path, self.class.options) do |row| 
-        yield Row.new(self.class.columns, row).to_hash
-      end
-    end
-    
-    module ClassMethods      
-      def column name, *indexes, &preprocessor
-        columns << Column.new(name, *indexes, &preprocessor)
-      end
-
-      def columns
-        @columns ||= if superclass.ancestors.include? Conformist::Base
-          superclass.columns.dup
-        else
-          []
-        end
-      end
-
-      def option value
-        options.merge! value
-      end
-
-      def options
-        @options ||= {}
-      end
-            
-      def load path
-        new.tap { |object| object.path = path }
-      end
-    end
-  end
+  end  
 end