csv_builder 2.0.2 → 2.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/CHANGELOG.rdoc +5 -0
- data/README.md +11 -4
- data/VERSION +1 -1
- data/{csv_builder.gemspec → csv_streamer.gemspec} +2 -2
- data/lib/csv_builder/template_handler.rb +83 -17
- data/spec/controllers/csv_builder_spec.rb +20 -1
- data/spec/templates/csv_builder_reports/massive.csv.csvbuilder +3 -0
- metadata +18 -17
data/CHANGELOG.rdoc
CHANGED
@@ -1,3 +1,8 @@
|
|
1
|
+
== 2.1.0 release 2011-11-25
|
2
|
+
* Implemented streaming support. (fawce.com)
|
3
|
+
* Temporarily renamed to csv_streamer to facilitate publishing to rubygems. (fawce.com)
|
4
|
+
* Merged back fawce's patch so it can be used in mainline csv_builder. (nbudin)
|
5
|
+
|
1
6
|
== 2.0.2 release 2011-09-26
|
2
7
|
* Rails 3.1 deprecation warning fix (Scott Gonyea, Tom Anderson, Sergio Cambra)
|
3
8
|
* Added dependencies and simplified contributor setup
|
data/README.md
CHANGED
@@ -8,14 +8,13 @@ if anyone has patches.
|
|
8
8
|
The CSV Builder Rails plugin provides a simple templating system for serving dynamically generated CSV files from your
|
9
9
|
application.
|
10
10
|
|
11
|
-
|
12
|
-
|
13
11
|
## Requirements
|
14
12
|
|
15
13
|
The current version of CSV Builder works with:
|
16
14
|
|
17
15
|
* Rails 3.x
|
18
16
|
* Ruby 1.8 or 1.9
|
17
|
+
* Unicorn _is required for streaming_ see [the example streaming app](https://github.com/fawce/test_csv_streamer) for more details.
|
19
18
|
|
20
19
|
The legacy version (1.1.x) was originally developed and tested for Rails 2.1. See [the legacy
|
21
20
|
docs](https://github.com/econsultancy/csv_builder) for more details.
|
@@ -27,6 +26,8 @@ docs](https://github.com/econsultancy/csv_builder) for more details.
|
|
27
26
|
### Install as a gem (recommended)
|
28
27
|
|
29
28
|
$ gem install csv_builder
|
29
|
+
_or for streaming_
|
30
|
+
$ gem install csv_streamer
|
30
31
|
|
31
32
|
If you are using Bundler then [you know what to do](http://gembundler.com).
|
32
33
|
|
@@ -58,6 +59,10 @@ You can set `@csv_options` instance variable to define options for FasterCSV gen
|
|
58
59
|
|
59
60
|
@csv_options = { :force_quotes => true, :col_sep => ';' }
|
60
61
|
|
62
|
+
You can optionally stream your results line by line as they are generated. Results will stream if the underlying Rack server supports streaming, otherwise the results will be buffered and sent when the template finishes rendering. Just set `@streaming` to true:
|
63
|
+
|
64
|
+
@streaming = true
|
65
|
+
|
61
66
|
You can respond with csv in your controller as well:
|
62
67
|
|
63
68
|
respond_to do |format|
|
@@ -73,6 +78,8 @@ including a snippet like the following in your mailer method
|
|
73
78
|
attachment.filename = 'report.csv'
|
74
79
|
end
|
75
80
|
|
81
|
+
## Streaming Support
|
82
|
+
Many csv files are quite large, and need to be streamed rather than return in a single shot. Csv stream handling is based on [an epic answer on stackoverflow about rails and streaming.](http://stackoverflow.com/questions/3507594/ruby-on-rails-3-streaming-data-through-rails-to-client). Streaming requires configuration of your rails app - you need to use a Rack that supports streaming. I've tested with Unicorn, and created [a separate sample](https://github.com/fawce/test_csv_streamer) project to facilitate testing on Heroku.
|
76
83
|
|
77
84
|
|
78
85
|
## Contributions
|
@@ -90,7 +97,6 @@ To install the main testing requirements. Then return back to the root director
|
|
90
97
|
I will also take patches for Rails 2.3.x, though I personally have no further need of that branch.
|
91
98
|
|
92
99
|
|
93
|
-
|
94
100
|
## Troubleshooting
|
95
101
|
|
96
102
|
There's a known bug of encoding error in Ruby 1.9
|
@@ -98,4 +104,5 @@ There's a known bug of encoding error in Ruby 1.9
|
|
98
104
|
For more details see https://rails.lighthouseapp.com/projects/8994/tickets/2188-i18n-fails-with-multibyte-strings-in-ruby-19-similar-to-2038
|
99
105
|
|
100
106
|
|
101
|
-
Copyright (c) 2008 Econsultancy.com, 2009 Vidmantas Kabošis & 2011 Gabe da Silveira released under the MIT license
|
107
|
+
Original content Copyright (c) 2008 Econsultancy.com, 2009 Vidmantas Kabošis & 2011 Gabe da Silveira released under the MIT license
|
108
|
+
Updated content for streaming, Copyright (c) 2011 John Fawcett released under the MIT license
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
2.0
|
1
|
+
2.1.0
|
@@ -8,7 +8,7 @@ Gem::Specification.new do |s|
|
|
8
8
|
s.version = "2.0.2"
|
9
9
|
|
10
10
|
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
11
|
-
s.authors = [%q{Econsultancy}, %q{Vidmantas Kabosis}, %q{Gabe da Silveira}]
|
11
|
+
s.authors = [%q{Econsultancy}, %q{Vidmantas Kabosis}, %q{Gabe da Silveira}, %q{fawce.com}]
|
12
12
|
s.date = %q{2011-09-26}
|
13
13
|
s.description = %q{CSV template handler for Rails. Enables :format => 'csv' in controllers, with templates of the form report.csv.csvbuilder.}
|
14
14
|
s.email = %q{gabe@websaviour.com}
|
@@ -50,7 +50,7 @@ Gem::Specification.new do |s|
|
|
50
50
|
"spec/templates/csv_builder_reports/simple.csv.csvbuilder",
|
51
51
|
"spec/templates/csv_builder_reports/simple.html.erb"
|
52
52
|
]
|
53
|
-
s.homepage = %q{
|
53
|
+
s.homepage = %q{https://github.com/fawce/csv_builder}
|
54
54
|
s.licenses = [%q{MIT}]
|
55
55
|
s.require_paths = [%q{lib}]
|
56
56
|
s.requirements = [%q{iconv}, %q{Ruby 1.9.x or FasterCSV}]
|
@@ -28,39 +28,105 @@ module CsvBuilder # :nodoc:
|
|
28
28
|
include ActionView::Template::Handlers::Compilable
|
29
29
|
end
|
30
30
|
end
|
31
|
-
|
31
|
+
|
32
|
+
# The ruby csv class will try to infer a separator to use, if the csv options
|
33
|
+
# do not set it. ruby's csv calls pos, eof?, read, and rewind to check the first line
|
34
|
+
# of the io to infer a separator. Rails' output object does not support these methods
|
35
|
+
# so we provide a mock implementation to satisfy csv.
|
36
|
+
#
|
37
|
+
# See code at https://github.com/ruby/ruby/blob/trunk/lib/csv.rb#L2021 - note that @io points
|
38
|
+
# to an object of this class.
|
39
|
+
class Yielder
|
40
|
+
def initialize(yielder)
|
41
|
+
@yielder = yielder
|
42
|
+
end
|
43
|
+
|
44
|
+
# always indicate that we are at the start of the io stream
|
45
|
+
def pos
|
46
|
+
return 0
|
47
|
+
end
|
48
|
+
|
49
|
+
# always indicate that we have reached the end of the file
|
50
|
+
def eof?
|
51
|
+
return true
|
52
|
+
end
|
53
|
+
|
54
|
+
#do nothing, we haven't moved forward
|
55
|
+
def rewind
|
56
|
+
end
|
57
|
+
|
58
|
+
#despite indicating that we have no data with pos and eof, we still need to return a newline
|
59
|
+
#otherwise CSV will enter an infinite loop with read.
|
60
|
+
def read(arg1)
|
61
|
+
return "\n"
|
62
|
+
end
|
63
|
+
|
64
|
+
# this is the method that ultimately yields to the block with output.
|
65
|
+
# the block is passed by Rails into the Streamer class' each method.
|
66
|
+
# Streamer provides a Proc to this class, which simply invokes yield
|
67
|
+
# from within the context of the each block.
|
68
|
+
def <<(data)
|
69
|
+
@yielder.call data
|
70
|
+
end
|
71
|
+
|
72
|
+
end
|
73
|
+
|
74
|
+
# Streamer implements an each method to facilitate streaming back through the Rails stack. It requires
|
75
|
+
# the template to be passed to it as a proc. An instance of this class is returned from the template handler's
|
76
|
+
# compile method, and will receive calls to each. Data is streamed by yielding back to the containing block.
|
77
|
+
class Streamer
|
78
|
+
def initialize(template_proc)
|
79
|
+
@template_proc = template_proc
|
80
|
+
end
|
81
|
+
|
82
|
+
def each
|
83
|
+
yielder = CsvBuilder::Yielder.new(Proc.new{|data| yield data})
|
84
|
+
csv_stream = CsvBuilder::CSV_LIB.new(yielder, @csv_options || {})
|
85
|
+
csv = CsvBuilder::TransliteratingFilter.new(csv_stream, @input_encoding || 'UTF-8', @output_encoding || 'LATIN1')
|
86
|
+
@template_proc.call(csv)
|
87
|
+
end
|
88
|
+
end
|
89
|
+
|
32
90
|
class TemplateHandler
|
33
91
|
def self.call(template)
|
92
|
+
|
34
93
|
<<-EOV
|
35
94
|
begin
|
36
|
-
|
37
|
-
csv = CsvBuilder::TransliteratingFilter.new(faster_csv, @input_encoding || 'UTF-8', @output_encoding || 'LATIN1')
|
38
|
-
#{template.source}
|
39
|
-
end
|
40
|
-
|
95
|
+
|
41
96
|
unless defined?(ActionMailer) && defined?(ActionMailer::Base) && controller.is_a?(ActionMailer::Base)
|
42
97
|
@filename ||= "\#{controller.action_name}.csv"
|
43
98
|
if controller.request.env['HTTP_USER_AGENT'] =~ /msie/i
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
99
|
+
response.headers['Pragma'] = 'public'
|
100
|
+
response.headers["Content-type"] = "text/plain"
|
101
|
+
response.headers['Cache-Control'] = 'no-cache, must-revalidate, post-check=0, pre-check=0'
|
102
|
+
response.headers['Content-Disposition'] = "attachment; filename=\#{@filename}"
|
103
|
+
response.headers['Expires'] = "0"
|
49
104
|
else
|
50
|
-
|
51
|
-
|
52
|
-
|
105
|
+
response.headers["Content-Type"] ||= 'text/csv'
|
106
|
+
response.headers["Content-Disposition"] = "attachment; filename=\#{@filename}"
|
107
|
+
response.headers["Content-Transfer-Encoding"] = "binary"
|
53
108
|
end
|
54
109
|
end
|
55
|
-
|
56
|
-
|
110
|
+
|
111
|
+
if @streaming
|
112
|
+
template = Proc.new {|csv|
|
113
|
+
#{template.source}
|
114
|
+
}
|
115
|
+
CsvBuilder::Streamer.new(template)
|
116
|
+
else
|
117
|
+
output = CsvBuilder::CSV_LIB.generate(@csv_options || {}) do |faster_csv|
|
118
|
+
csv = CsvBuilder::TransliteratingFilter.new(faster_csv, @input_encoding || 'UTF-8', @output_encoding || 'LATIN1')
|
119
|
+
#{template.source}
|
120
|
+
end
|
121
|
+
output
|
122
|
+
end
|
57
123
|
rescue Exception => e
|
58
124
|
Rails.logger.warn("Exception \#{e} \#{e.message} with class \#{e.class.name} thrown when rendering CSV")
|
59
125
|
raise e
|
60
126
|
end
|
61
127
|
EOV
|
62
128
|
end
|
63
|
-
|
129
|
+
|
64
130
|
def compile(template)
|
65
131
|
self.class.call(template)
|
66
132
|
end
|
@@ -28,9 +28,21 @@ class CsvBuilderReportsController < ApplicationController
|
|
28
28
|
format.csv { @output_encoding = 'UTF-16' }
|
29
29
|
end
|
30
30
|
end
|
31
|
+
|
32
|
+
def massive
|
33
|
+
respond_to do |format|
|
34
|
+
@streaming = true
|
35
|
+
format.csv
|
36
|
+
end
|
37
|
+
end
|
31
38
|
|
32
39
|
end
|
33
|
-
|
40
|
+
|
41
|
+
if defined?(Rails) and Rails.version < '3'
|
42
|
+
ActionController::Routing::Routes.draw { |map| map.connect ':controller/:action' }
|
43
|
+
else
|
44
|
+
Rails.application.routes.draw { get ':controller/:action' }
|
45
|
+
end
|
34
46
|
|
35
47
|
|
36
48
|
describe CsvBuilderReportsController do
|
@@ -64,5 +76,12 @@ describe CsvBuilderReportsController do
|
|
64
76
|
get 'complex', :format => 'csv'
|
65
77
|
response.headers['Content-Disposition'].should match(/filename=some_complex_filename.csv/)
|
66
78
|
end
|
79
|
+
|
80
|
+
#TODO: unfortunately, this test only verifies that streaming will behave like single-shot response, because rspec's testresponse doesn't
|
81
|
+
#support streaming. Streaming has to be manually verified with a browser and stand-alone test application. see https://github.com/fawce/test_csv_streamer
|
82
|
+
it "handles very large downloads without timing out" do
|
83
|
+
get 'massive', :format => 'csv'
|
84
|
+
response.body.to_s.length.should == 24890
|
85
|
+
end
|
67
86
|
end
|
68
87
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: csv_builder
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.0
|
4
|
+
version: 2.1.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -11,11 +11,11 @@ authors:
|
|
11
11
|
autorequire:
|
12
12
|
bindir: bin
|
13
13
|
cert_chain: []
|
14
|
-
date: 2011-
|
14
|
+
date: 2011-11-25 00:00:00.000000000Z
|
15
15
|
dependencies:
|
16
16
|
- !ruby/object:Gem::Dependency
|
17
17
|
name: actionpack
|
18
|
-
requirement: &
|
18
|
+
requirement: &70319189612680 !ruby/object:Gem::Requirement
|
19
19
|
none: false
|
20
20
|
requirements:
|
21
21
|
- - ! '>='
|
@@ -23,10 +23,10 @@ dependencies:
|
|
23
23
|
version: 3.0.0
|
24
24
|
type: :runtime
|
25
25
|
prerelease: false
|
26
|
-
version_requirements: *
|
26
|
+
version_requirements: *70319189612680
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
28
|
name: rails
|
29
|
-
requirement: &
|
29
|
+
requirement: &70319189612080 !ruby/object:Gem::Requirement
|
30
30
|
none: false
|
31
31
|
requirements:
|
32
32
|
- - ! '>='
|
@@ -34,10 +34,10 @@ dependencies:
|
|
34
34
|
version: 3.0.0
|
35
35
|
type: :development
|
36
36
|
prerelease: false
|
37
|
-
version_requirements: *
|
37
|
+
version_requirements: *70319189612080
|
38
38
|
- !ruby/object:Gem::Dependency
|
39
39
|
name: rspec
|
40
|
-
requirement: &
|
40
|
+
requirement: &70319189611480 !ruby/object:Gem::Requirement
|
41
41
|
none: false
|
42
42
|
requirements:
|
43
43
|
- - ~>
|
@@ -45,10 +45,10 @@ dependencies:
|
|
45
45
|
version: '2.5'
|
46
46
|
type: :development
|
47
47
|
prerelease: false
|
48
|
-
version_requirements: *
|
48
|
+
version_requirements: *70319189611480
|
49
49
|
- !ruby/object:Gem::Dependency
|
50
50
|
name: rspec-rails
|
51
|
-
requirement: &
|
51
|
+
requirement: &70319189610880 !ruby/object:Gem::Requirement
|
52
52
|
none: false
|
53
53
|
requirements:
|
54
54
|
- - ~>
|
@@ -56,10 +56,10 @@ dependencies:
|
|
56
56
|
version: '2.5'
|
57
57
|
type: :development
|
58
58
|
prerelease: false
|
59
|
-
version_requirements: *
|
59
|
+
version_requirements: *70319189610880
|
60
60
|
- !ruby/object:Gem::Dependency
|
61
61
|
name: jeweler
|
62
|
-
requirement: &
|
62
|
+
requirement: &70319189610280 !ruby/object:Gem::Requirement
|
63
63
|
none: false
|
64
64
|
requirements:
|
65
65
|
- - ! '>='
|
@@ -67,10 +67,10 @@ dependencies:
|
|
67
67
|
version: '0'
|
68
68
|
type: :development
|
69
69
|
prerelease: false
|
70
|
-
version_requirements: *
|
70
|
+
version_requirements: *70319189610280
|
71
71
|
- !ruby/object:Gem::Dependency
|
72
72
|
name: rack
|
73
|
-
requirement: &
|
73
|
+
requirement: &70319189609620 !ruby/object:Gem::Requirement
|
74
74
|
none: false
|
75
75
|
requirements:
|
76
76
|
- - ! '>='
|
@@ -78,10 +78,10 @@ dependencies:
|
|
78
78
|
version: '0'
|
79
79
|
type: :development
|
80
80
|
prerelease: false
|
81
|
-
version_requirements: *
|
81
|
+
version_requirements: *70319189609620
|
82
82
|
- !ruby/object:Gem::Dependency
|
83
83
|
name: sqlite3
|
84
|
-
requirement: &
|
84
|
+
requirement: &70319189608960 !ruby/object:Gem::Requirement
|
85
85
|
none: false
|
86
86
|
requirements:
|
87
87
|
- - ! '>='
|
@@ -89,7 +89,7 @@ dependencies:
|
|
89
89
|
version: '0'
|
90
90
|
type: :development
|
91
91
|
prerelease: false
|
92
|
-
version_requirements: *
|
92
|
+
version_requirements: *70319189608960
|
93
93
|
description: CSV template handler for Rails. Enables :format => 'csv' in controllers,
|
94
94
|
with templates of the form report.csv.csvbuilder.
|
95
95
|
email: gabe@websaviour.com
|
@@ -103,7 +103,7 @@ files:
|
|
103
103
|
- README.md
|
104
104
|
- Rakefile
|
105
105
|
- VERSION
|
106
|
-
-
|
106
|
+
- csv_streamer.gemspec
|
107
107
|
- lib/csv_builder.rb
|
108
108
|
- lib/csv_builder/railtie.rb
|
109
109
|
- lib/csv_builder/template_handler.rb
|
@@ -129,6 +129,7 @@ files:
|
|
129
129
|
- spec/spec_helper.rb
|
130
130
|
- spec/templates/csv_builder_reports/complex.csv.csvbuilder
|
131
131
|
- spec/templates/csv_builder_reports/encoding.csv.csvbuilder
|
132
|
+
- spec/templates/csv_builder_reports/massive.csv.csvbuilder
|
132
133
|
- spec/templates/csv_builder_reports/simple.csv.csvbuilder
|
133
134
|
- spec/templates/csv_builder_reports/simple.html.erb
|
134
135
|
homepage: http://github.com/dasil003/csv_builder
|