csv_builder 2.0.2 → 2.1.0

data/CHANGELOG.rdoc

@@ -1,3 +1,8 @@
+== 2.1.0 release 2011-11-25
+* Implemented streaming support. (fawce.com)
+* Temporarily renamed to csv_streamer to facilitate publishing to rubygems. (fawce.com)
+* Merged back fawce's patch so it can be used in mainline csv_builder. (nbudin)
+
 == 2.0.2 release 2011-09-26
 * Rails 3.1 deprecation warning fix (Scott Gonyea, Tom Anderson, Sergio Cambra)
 * Added dependencies and simplified contributor setup

data/README.md

@@ -8,14 +8,13 @@ if anyone has patches.
 The CSV Builder Rails plugin provides a simple templating system for serving dynamically generated CSV files from your
 application.
 
-
-
 ## Requirements
 
 The current version of CSV Builder works with:
 
 * Rails 3.x
 * Ruby 1.8 or 1.9
+* Unicorn _is required for streaming_ see [the example streaming app](https://github.com/fawce/test_csv_streamer) for more details.
 
 The legacy version (1.1.x) was originally developed and tested for Rails 2.1.  See [the legacy
 docs](https://github.com/econsultancy/csv_builder) for more details.
@@ -27,6 +26,8 @@ docs](https://github.com/econsultancy/csv_builder) for more details.
 ### Install as a gem (recommended)
 
     $ gem install csv_builder
+_or for streaming_
+	$ gem install csv_streamer 
 
 If you are using Bundler then [you know what to do](http://gembundler.com).
 
@@ -58,6 +59,10 @@ You can set `@csv_options` instance variable to define options for FasterCSV gen
 
     @csv_options = { :force_quotes => true, :col_sep => ';' }
 
+You can optionally stream your results line by line as they are generated. Results will stream if the underlying Rack server supports streaming, otherwise the results will be buffered and sent when the template finishes rendering. Just set `@streaming` to true:
+	
+	@streaming = true 
+
 You can respond with csv in your controller as well:
 
     respond_to do |format|
@@ -73,6 +78,8 @@ including a snippet like the following in your mailer method
       attachment.filename = 'report.csv'
     end
 
+## Streaming Support
+Many csv files are quite large, and need to be streamed rather than return in a single shot. Csv stream handling is based on [an epic answer on stackoverflow about rails and streaming.](http://stackoverflow.com/questions/3507594/ruby-on-rails-3-streaming-data-through-rails-to-client). Streaming requires configuration of your rails app - you need to use a Rack that supports streaming. I've tested with Unicorn, and created [a separate sample](https://github.com/fawce/test_csv_streamer) project to facilitate testing on Heroku. 
 
 
 ## Contributions
@@ -90,7 +97,6 @@ To install the main testing requirements.  Then return back to the root director
 I will also take patches for Rails 2.3.x, though I personally have no further need of that branch.
 
 
-
 ## Troubleshooting
 
 There's a known bug of encoding error in Ruby 1.9
@@ -98,4 +104,5 @@ There's a known bug of encoding error in Ruby 1.9
 For more details see https://rails.lighthouseapp.com/projects/8994/tickets/2188-i18n-fails-with-multibyte-strings-in-ruby-19-similar-to-2038
 
 
-Copyright (c) 2008 Econsultancy.com, 2009 Vidmantas Kabošis & 2011 Gabe da Silveira released under the MIT license
+Original content Copyright (c) 2008 Econsultancy.com, 2009 Vidmantas Kabošis & 2011 Gabe da Silveira released under the MIT license
+Updated content for streaming, Copyright (c) 2011 John Fawcett released under the MIT license

data/VERSION

@@ -1 +1 @@
-2.0.2
+2.1.0

data/{csv_builder.gemspec → csv_streamer.gemspec}

@@ -8,7 +8,7 @@ Gem::Specification.new do |s|
   s.version = "2.0.2"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
-  s.authors = [%q{Econsultancy}, %q{Vidmantas Kabosis}, %q{Gabe da Silveira}]
+  s.authors = [%q{Econsultancy}, %q{Vidmantas Kabosis}, %q{Gabe da Silveira}, %q{fawce.com}]
   s.date = %q{2011-09-26}
   s.description = %q{CSV template handler for Rails.  Enables :format => 'csv' in controllers, with templates of the form report.csv.csvbuilder.}
   s.email = %q{gabe@websaviour.com}
@@ -50,7 +50,7 @@ Gem::Specification.new do |s|
     "spec/templates/csv_builder_reports/simple.csv.csvbuilder",
     "spec/templates/csv_builder_reports/simple.html.erb"
   ]
-  s.homepage = %q{http://github.com/dasil003/csv_builder}
+  s.homepage = %q{https://github.com/fawce/csv_builder}
   s.licenses = [%q{MIT}]
   s.require_paths = [%q{lib}]
   s.requirements = [%q{iconv}, %q{Ruby 1.9.x or FasterCSV}]

data/lib/csv_builder/template_handler.rb

@@ -28,39 +28,105 @@ module CsvBuilder # :nodoc:
       include ActionView::Template::Handlers::Compilable
     end
   end
-
+  
+  # The ruby csv class will try to infer a separator to use, if the csv options
+  # do not set it. ruby's csv calls pos, eof?, read, and rewind to check the first line
+  # of the io to infer a separator. Rails' output object does not support these methods
+  # so we provide a mock implementation to satisfy csv.
+  #
+  # See code at https://github.com/ruby/ruby/blob/trunk/lib/csv.rb#L2021 - note that @io points
+  # to an object of this class.
+  class Yielder
+    def initialize(yielder)
+      @yielder = yielder
+    end
+    
+    # always indicate that we are at the start of the io stream
+    def pos
+      return 0
+    end
+    
+    # always indicate that we have reached the end of the file
+    def eof?
+      return true
+    end
+    
+    #do nothing, we haven't moved forward
+    def rewind
+    end
+    
+    #despite indicating that we have no data with pos and eof, we still need to return a newline
+    #otherwise CSV will enter an infinite loop with read.
+    def read(arg1)
+      return "\n"
+    end
+    
+    # this is the method that ultimately yields to the block with output.
+    # the block is passed by Rails into the Streamer class' each method.
+    # Streamer provides a Proc to this class, which simply invokes yield 
+    # from within the context of the each block.
+    def <<(data)
+      @yielder.call data
+    end
+    
+  end
+  
+  # Streamer implements an each method to facilitate streaming back through the Rails stack. It requires
+  # the template to be passed to it as a proc. An instance of this class is returned from the template handler's
+  # compile method, and will receive calls to each. Data is streamed by yielding back to the containing block.
+  class Streamer
+    def initialize(template_proc)
+      @template_proc = template_proc
+    end
+    
+    def each
+      yielder = CsvBuilder::Yielder.new(Proc.new{|data| yield data})
+      csv_stream = CsvBuilder::CSV_LIB.new(yielder, @csv_options || {}) 
+      csv = CsvBuilder::TransliteratingFilter.new(csv_stream, @input_encoding || 'UTF-8', @output_encoding || 'LATIN1')
+      @template_proc.call(csv)
+    end
+  end
+  
   class TemplateHandler
     def self.call(template)
+      
       <<-EOV
       begin
-        output = CsvBuilder::CSV_LIB.generate(@csv_options || {}) do |faster_csv|
-          csv = CsvBuilder::TransliteratingFilter.new(faster_csv, @input_encoding || 'UTF-8', @output_encoding || 'LATIN1')
-          #{template.source}
-        end
-
+           
         unless defined?(ActionMailer) && defined?(ActionMailer::Base) && controller.is_a?(ActionMailer::Base)
           @filename ||= "\#{controller.action_name}.csv"
           if controller.request.env['HTTP_USER_AGENT'] =~ /msie/i
-            controller.response.headers['Pragma'] = 'public'
-            controller.response.headers["Content-type"] = "text/plain"
-            controller.response.headers['Cache-Control'] = 'no-cache, must-revalidate, post-check=0, pre-check=0'
-            controller.response.headers['Content-Disposition'] = "attachment; filename=\#{@filename}"
-            controller.response.headers['Expires'] = "0"
+            response.headers['Pragma'] = 'public'
+            response.headers["Content-type"] = "text/plain"
+            response.headers['Cache-Control'] = 'no-cache, must-revalidate, post-check=0, pre-check=0'
+            response.headers['Content-Disposition'] = "attachment; filename=\#{@filename}"
+            response.headers['Expires'] = "0"
           else
-            controller.response.headers["Content-Type"] ||= 'text/csv'
-            controller.response.headers["Content-Disposition"] = "attachment; filename=\#{@filename}"
-            controller.response.headers["Content-Transfer-Encoding"] = "binary"
+            response.headers["Content-Type"] ||= 'text/csv'
+            response.headers["Content-Disposition"] = "attachment; filename=\#{@filename}"
+            response.headers["Content-Transfer-Encoding"] = "binary"
           end
         end
-
-        output
+        
+        if @streaming
+          template = Proc.new {|csv|
+            #{template.source}
+          }
+          CsvBuilder::Streamer.new(template)
+        else 
+          output = CsvBuilder::CSV_LIB.generate(@csv_options || {}) do |faster_csv|
+            csv = CsvBuilder::TransliteratingFilter.new(faster_csv, @input_encoding || 'UTF-8', @output_encoding || 'LATIN1')
+            #{template.source}
+          end
+          output
+        end
       rescue Exception => e
         Rails.logger.warn("Exception \#{e} \#{e.message} with class \#{e.class.name} thrown when rendering CSV")
         raise e
       end
       EOV
     end
-
+    
     def compile(template)
       self.class.call(template)
     end

data/spec/controllers/csv_builder_spec.rb

@@ -28,9 +28,21 @@ class CsvBuilderReportsController < ApplicationController
       format.csv { @output_encoding = 'UTF-16' }
     end
   end
+  
+  def massive
+    respond_to do |format|
+      @streaming = true
+      format.csv 
+    end
+  end
 
 end
-ActionController::Routing::Routes.draw { |map| map.connect ':controller/:action' }
+
+if defined?(Rails) and Rails.version < '3'
+  ActionController::Routing::Routes.draw { |map| map.connect ':controller/:action' }
+else
+  Rails.application.routes.draw { get ':controller/:action' }
+end
 
 
 describe CsvBuilderReportsController do
@@ -64,5 +76,12 @@ describe CsvBuilderReportsController do
       get 'complex', :format => 'csv'
       response.headers['Content-Disposition'].should match(/filename=some_complex_filename.csv/)
     end
+    
+    #TODO: unfortunately, this test only verifies that streaming will behave like single-shot response, because rspec's testresponse doesn't 
+    #support streaming. Streaming has to be manually verified with a browser and stand-alone test application. see https://github.com/fawce/test_csv_streamer 
+    it "handles very large downloads without timing out" do
+      get 'massive', :format => 'csv'
+      response.body.to_s.length.should == 24890
+    end
   end
 end

data/spec/templates/csv_builder_reports/massive.csv.csvbuilder

@@ -0,0 +1,3 @@
+1000.times do |i|
+   csv << ["this","is","a","scale","test",i.to_s]
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: csv_builder
 version: !ruby/object:Gem::Version
-  version: 2.0.2
+  version: 2.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -11,11 +11,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2011-09-26 00:00:00.000000000Z
+date: 2011-11-25 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
-  requirement: &70256232057580 !ruby/object:Gem::Requirement
+  requirement: &70319189612680 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -23,10 +23,10 @@ dependencies:
         version: 3.0.0
   type: :runtime
   prerelease: false
-  version_requirements: *70256232057580
+  version_requirements: *70319189612680
 - !ruby/object:Gem::Dependency
   name: rails
-  requirement: &70256232051300 !ruby/object:Gem::Requirement
+  requirement: &70319189612080 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -34,10 +34,10 @@ dependencies:
         version: 3.0.0
   type: :development
   prerelease: false
-  version_requirements: *70256232051300
+  version_requirements: *70319189612080
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70256232042780 !ruby/object:Gem::Requirement
+  requirement: &70319189611480 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -45,10 +45,10 @@ dependencies:
         version: '2.5'
   type: :development
   prerelease: false
-  version_requirements: *70256232042780
+  version_requirements: *70319189611480
 - !ruby/object:Gem::Dependency
   name: rspec-rails
-  requirement: &70256232034560 !ruby/object:Gem::Requirement
+  requirement: &70319189610880 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -56,10 +56,10 @@ dependencies:
         version: '2.5'
   type: :development
   prerelease: false
-  version_requirements: *70256232034560
+  version_requirements: *70319189610880
 - !ruby/object:Gem::Dependency
   name: jeweler
-  requirement: &70256232513000 !ruby/object:Gem::Requirement
+  requirement: &70319189610280 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -67,10 +67,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70256232513000
+  version_requirements: *70319189610280
 - !ruby/object:Gem::Dependency
   name: rack
-  requirement: &70256232512520 !ruby/object:Gem::Requirement
+  requirement: &70319189609620 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -78,10 +78,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70256232512520
+  version_requirements: *70319189609620
 - !ruby/object:Gem::Dependency
   name: sqlite3
-  requirement: &70256232512040 !ruby/object:Gem::Requirement
+  requirement: &70319189608960 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -89,7 +89,7 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70256232512040
+  version_requirements: *70319189608960
 description: CSV template handler for Rails.  Enables :format => 'csv' in controllers,
   with templates of the form report.csv.csvbuilder.
 email: gabe@websaviour.com
@@ -103,7 +103,7 @@ files:
 - README.md
 - Rakefile
 - VERSION
-- csv_builder.gemspec
+- csv_streamer.gemspec
 - lib/csv_builder.rb
 - lib/csv_builder/railtie.rb
 - lib/csv_builder/template_handler.rb
@@ -129,6 +129,7 @@ files:
 - spec/spec_helper.rb
 - spec/templates/csv_builder_reports/complex.csv.csvbuilder
 - spec/templates/csv_builder_reports/encoding.csv.csvbuilder
+- spec/templates/csv_builder_reports/massive.csv.csvbuilder
 - spec/templates/csv_builder_reports/simple.csv.csvbuilder
 - spec/templates/csv_builder_reports/simple.html.erb
 homepage: http://github.com/dasil003/csv_builder