csv_preview 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.yardopts +8 -0
- data/CHANGELOG.md +25 -0
- data/LICENSE +21 -0
- data/README.md +587 -0
- data/SECURITY.md +51 -0
- data/fonts/Inter-Regular.ttf +0 -0
- data/fonts/Inter-SemiBold.ttf +0 -0
- data/fonts/OFL.txt +92 -0
- data/lib/csv_preview/active_storage_previewer.rb +97 -0
- data/lib/csv_preview/cell_kind.rb +75 -0
- data/lib/csv_preview/configuration.rb +183 -0
- data/lib/csv_preview/deadline.rb +31 -0
- data/lib/csv_preview/errors.rb +30 -0
- data/lib/csv_preview/layout.rb +590 -0
- data/lib/csv_preview/painter.rb +194 -0
- data/lib/csv_preview/parser.rb +456 -0
- data/lib/csv_preview/railtie.rb +54 -0
- data/lib/csv_preview/reader.rb +259 -0
- data/lib/csv_preview/sample.rb +85 -0
- data/lib/csv_preview/sanitizer.rb +31 -0
- data/lib/csv_preview/text.rb +621 -0
- data/lib/csv_preview/theme.rb +207 -0
- data/lib/csv_preview/version.rb +6 -0
- data/lib/csv_preview.rb +149 -0
- data/skills/csv-preview/SKILL.md +171 -0
- metadata +121 -0
data/SECURITY.md
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
Built for untrusted uploads: it reads at most 2 MiB, stops after 5 seconds, and never runs another program or opens a network connection.
|
|
4
|
+
|
|
5
|
+
csv_preview turns files that strangers upload into images, so its security matters. This page explains how to report a vulnerability, which versions receive fixes, and what csv_preview does and does not defend against.
|
|
6
|
+
|
|
7
|
+
## Reporting a vulnerability
|
|
8
|
+
|
|
9
|
+
Report vulnerabilities privately through GitHub: open the repository's **Security** tab and choose **Report a vulnerability** ([direct link](https://github.com/garyhtou/csv_preview/security/advisories/new)). Please do not open a public issue, pull request or discussion about a vulnerability.
|
|
10
|
+
|
|
11
|
+
Include what you can: the csv_preview, Ruby and libvips versions, the input that triggers the problem (or a script that generates it), the options used, and what happens. Keep real personal data out of the report; a generated file is better.
|
|
12
|
+
|
|
13
|
+
csv_preview is maintained by a volunteer, and reports are handled on a best-effort basis, so there is no promised response time. You will hear back once the report has been read, and again when there is a fix or a decision.
|
|
14
|
+
|
|
15
|
+
## Supported versions
|
|
16
|
+
|
|
17
|
+
Before 1.0, security fixes are released for the latest minor version only. Upgrade to the newest release to receive them.
|
|
18
|
+
|
|
19
|
+
| Version | Security fixes |
|
|
20
|
+
|---|---|
|
|
21
|
+
| 0.1.x | Yes |
|
|
22
|
+
|
|
23
|
+
## Disclosure
|
|
24
|
+
|
|
25
|
+
Disclosure is coordinated: a fix is prepared privately, released, and then announced. Each fix ships with a GitHub Security Advisory (with a CVE when warranted), which feeds Dependabot alerts, and with a pull request to the [Ruby Advisory Database](https://github.com/rubysec/ruby-advisory-db), which feeds `bundle audit`. Reporters are credited in the advisory unless they prefer not to be.
|
|
26
|
+
|
|
27
|
+
## Threat model
|
|
28
|
+
|
|
29
|
+
- **Who:** anyone who can upload a file to an application that previews it. Previews are usually generated automatically, in a background job or a web request, so nobody reviews a file before csv_preview reads it.
|
|
30
|
+
- **What they control:** every byte of the file.
|
|
31
|
+
- **Where the bytes go:** csv_preview's Ruby reader and parser (built on Ruby's `csv` library) read at most `max_input_bytes`. Only sanitised, escaped and truncated cell text then reaches Pango and HarfBuzz, through libvips, to be shaped and drawn. The output PNG goes to the application's variant processor and to browsers.
|
|
32
|
+
- **Where the bytes never go:** an image decoder, a subprocess, the network, `eval`, or a file written by csv_preview. (In Rails, Active Storage itself downloads a blob to a temporary file only for storage services that cannot serve ranged reads.)
|
|
33
|
+
- **What csv_preview guarantees for any input:** bounded bytes read; bounded work, memory and time; a bounded image size; formulas, markup and bidirectional control characters never interpreted; no side effects; and error messages that never include the input. The README's [Security](https://github.com/garyhtou/csv_preview#security) section lists each guarantee with the test that proves it, and the honest limits.
|
|
34
|
+
|
|
35
|
+
### In scope
|
|
36
|
+
|
|
37
|
+
- Crashes, hangs, or time or memory use beyond the documented bounds, caused by a file or by options.
|
|
38
|
+
- Cell text that is interpreted instead of drawn literally: markup, formulas, or bidirectional control characters that reorder what is shown.
|
|
39
|
+
- Anything that makes csv_preview start a process, open a socket, write a file, or pass input bytes to an image decoder.
|
|
40
|
+
- Error messages, log lines or instrumentation events that include file content.
|
|
41
|
+
- The Active Storage previewer reading more of a blob than `max_input_bytes` on services that support ranged reads.
|
|
42
|
+
|
|
43
|
+
### Out of scope
|
|
44
|
+
|
|
45
|
+
Report these to the project concerned, and tell us too if csv_preview could mitigate them:
|
|
46
|
+
|
|
47
|
+
- Bugs inside libvips, Pango, HarfBuzz, FreeType, cairo or fontconfig.
|
|
48
|
+
- ImageMagick in the application's own variant pipeline (csv_preview never uses it).
|
|
49
|
+
- Floods of uploads; rate limiting is the application's job.
|
|
50
|
+
- CSV injection when someone opens the original file in a spreadsheet program. csv_preview never modifies or re-exports the file.
|
|
51
|
+
- Who can see a preview image. It shows the first rows of the file, so applications must serve it with the same authorisation as the original.
|
|
Binary file
|
|
Binary file
|
data/fonts/OFL.txt
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
Copyright (c) 2016 The Inter Project Authors (https://github.com/rsms/inter)
|
|
2
|
+
|
|
3
|
+
This Font Software is licensed under the SIL Open Font License, Version 1.1.
|
|
4
|
+
This license is copied below, and is also available with a FAQ at:
|
|
5
|
+
http://scripts.sil.org/OFL
|
|
6
|
+
|
|
7
|
+
-----------------------------------------------------------
|
|
8
|
+
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
|
|
9
|
+
-----------------------------------------------------------
|
|
10
|
+
|
|
11
|
+
PREAMBLE
|
|
12
|
+
The goals of the Open Font License (OFL) are to stimulate worldwide
|
|
13
|
+
development of collaborative font projects, to support the font creation
|
|
14
|
+
efforts of academic and linguistic communities, and to provide a free and
|
|
15
|
+
open framework in which fonts may be shared and improved in partnership
|
|
16
|
+
with others.
|
|
17
|
+
|
|
18
|
+
The OFL allows the licensed fonts to be used, studied, modified and
|
|
19
|
+
redistributed freely as long as they are not sold by themselves. The
|
|
20
|
+
fonts, including any derivative works, can be bundled, embedded,
|
|
21
|
+
redistributed and/or sold with any software provided that any reserved
|
|
22
|
+
names are not used by derivative works. The fonts and derivatives,
|
|
23
|
+
however, cannot be released under any other type of license. The
|
|
24
|
+
requirement for fonts to remain under this license does not apply
|
|
25
|
+
to any document created using the fonts or their derivatives.
|
|
26
|
+
|
|
27
|
+
DEFINITIONS
|
|
28
|
+
"Font Software" refers to the set of files released by the Copyright
|
|
29
|
+
Holder(s) under this license and clearly marked as such. This may
|
|
30
|
+
include source files, build scripts and documentation.
|
|
31
|
+
|
|
32
|
+
"Reserved Font Name" refers to any names specified as such after the
|
|
33
|
+
copyright statement(s).
|
|
34
|
+
|
|
35
|
+
"Original Version" refers to the collection of Font Software components as
|
|
36
|
+
distributed by the Copyright Holder(s).
|
|
37
|
+
|
|
38
|
+
"Modified Version" refers to any derivative made by adding to, deleting,
|
|
39
|
+
or substituting -- in part or in whole -- any of the components of the
|
|
40
|
+
Original Version, by changing formats or by porting the Font Software to a
|
|
41
|
+
new environment.
|
|
42
|
+
|
|
43
|
+
"Author" refers to any designer, engineer, programmer, technical
|
|
44
|
+
writer or other person who contributed to the Font Software.
|
|
45
|
+
|
|
46
|
+
PERMISSION AND CONDITIONS
|
|
47
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
|
48
|
+
a copy of the Font Software, to use, study, copy, merge, embed, modify,
|
|
49
|
+
redistribute, and sell modified and unmodified copies of the Font
|
|
50
|
+
Software, subject to the following conditions:
|
|
51
|
+
|
|
52
|
+
1) Neither the Font Software nor any of its individual components,
|
|
53
|
+
in Original or Modified Versions, may be sold by itself.
|
|
54
|
+
|
|
55
|
+
2) Original or Modified Versions of the Font Software may be bundled,
|
|
56
|
+
redistributed and/or sold with any software, provided that each copy
|
|
57
|
+
contains the above copyright notice and this license. These can be
|
|
58
|
+
included either as stand-alone text files, human-readable headers or
|
|
59
|
+
in the appropriate machine-readable metadata fields within text or
|
|
60
|
+
binary files as long as those fields can be easily viewed by the user.
|
|
61
|
+
|
|
62
|
+
3) No Modified Version of the Font Software may use the Reserved Font
|
|
63
|
+
Name(s) unless explicit written permission is granted by the corresponding
|
|
64
|
+
Copyright Holder. This restriction only applies to the primary font name as
|
|
65
|
+
presented to the users.
|
|
66
|
+
|
|
67
|
+
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
|
|
68
|
+
Software shall not be used to promote, endorse or advertise any
|
|
69
|
+
Modified Version, except to acknowledge the contribution(s) of the
|
|
70
|
+
Copyright Holder(s) and the Author(s) or with their explicit written
|
|
71
|
+
permission.
|
|
72
|
+
|
|
73
|
+
5) The Font Software, modified or unmodified, in part or in whole,
|
|
74
|
+
must be distributed entirely under this license, and must not be
|
|
75
|
+
distributed under any other license. The requirement for fonts to
|
|
76
|
+
remain under this license does not apply to any document created
|
|
77
|
+
using the Font Software.
|
|
78
|
+
|
|
79
|
+
TERMINATION
|
|
80
|
+
This license becomes null and void if any of the above conditions are
|
|
81
|
+
not met.
|
|
82
|
+
|
|
83
|
+
DISCLAIMER
|
|
84
|
+
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
85
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
|
|
86
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
|
|
87
|
+
OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
|
|
88
|
+
COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
|
89
|
+
INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
|
|
90
|
+
DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
|
91
|
+
FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
|
|
92
|
+
OTHER DEALINGS IN THE FONT SOFTWARE.
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "stringio"
|
|
4
|
+
require "active_storage"
|
|
5
|
+
|
|
6
|
+
module CsvPreview
|
|
7
|
+
# An Active Storage previewer for CSV and TSV blobs. Register it with
|
|
8
|
+
# `config.active_storage.previewers.unshift CsvPreview::ActiveStoragePreviewer`.
|
|
9
|
+
#
|
|
10
|
+
# It reads only the first `max_input_bytes` bytes of the blob (a ranged read), renders them with the
|
|
11
|
+
# global {CsvPreview.config} and always yields a PNG.
|
|
12
|
+
#
|
|
13
|
+
# @example Register it in config/application.rb, inside the Application class
|
|
14
|
+
# config.active_storage.previewers.unshift CsvPreview::ActiveStoragePreviewer
|
|
15
|
+
class ActiveStoragePreviewer < ActiveStorage::Previewer
|
|
16
|
+
class << self
|
|
17
|
+
# @param blob [ActiveStorage::Blob]
|
|
18
|
+
# @return [Boolean] the content type is in `previewer_content_types` and libvips can draw text
|
|
19
|
+
# @example Pick a CSV-only variant in a view
|
|
20
|
+
# variant = CsvPreview::ActiveStoragePreviewer.accept?(document.file.blob) ? :table_thumb : :thumb
|
|
21
|
+
def accept?(blob)
|
|
22
|
+
content_type_accepted?(blob.content_type) && text_supported?
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Cheap and memoised: never renders. Logs one warning the first time it is false.
|
|
26
|
+
# @return [Boolean]
|
|
27
|
+
# @example A Rails health check
|
|
28
|
+
# CsvPreview::ActiveStoragePreviewer.text_supported? # => true
|
|
29
|
+
def text_supported?
|
|
30
|
+
return @text_supported unless @text_supported.nil?
|
|
31
|
+
|
|
32
|
+
@text_supported = CsvPreview::Text.supported?
|
|
33
|
+
unless @text_supported
|
|
34
|
+
ActiveStorage.logger&.warn("csv_preview: libvips has no text support (Pango), so CSV files " \
|
|
35
|
+
"are not previewable in this process. Install libvips with Pango, see " \
|
|
36
|
+
"#{CsvPreview::Help::INSTALLATION}")
|
|
37
|
+
end
|
|
38
|
+
@text_supported
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
private
|
|
42
|
+
|
|
43
|
+
def content_type_accepted?(content_type)
|
|
44
|
+
type = content_type.to_s.split(";", 2).first.to_s.strip.downcase
|
|
45
|
+
return false if type.empty?
|
|
46
|
+
|
|
47
|
+
CsvPreview.config.previewer_content_types.any? { |accepted| accepted.to_s.strip.downcase == type }
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# Renders the blob's preview image. Active Storage calls this; apps call `Blob#preview` or
|
|
52
|
+
# `#representation` instead.
|
|
53
|
+
#
|
|
54
|
+
# @param options [Hash] passed through to the yielded attachable (`service_name:`); never to render
|
|
55
|
+
# @yieldparam attachable [Hash] `io:`, `filename:`, `content_type:` plus `options`
|
|
56
|
+
# @return [Object] whatever the block returns
|
|
57
|
+
# @raise [ActiveStorage::PreviewError] when the core raises a {CsvPreview::Error}
|
|
58
|
+
# @example What an app writes; Active Storage then calls #preview once per blob
|
|
59
|
+
# document.file.preview(resize_to_limit: [300, 300]).processed
|
|
60
|
+
def preview(**options)
|
|
61
|
+
png = render_image
|
|
62
|
+
yield io: StringIO.new(png), filename: "#{blob.filename.base}.png", content_type: "image/png", **options
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
private
|
|
66
|
+
|
|
67
|
+
# Extension point: per-call options for CsvPreview.render. `format` is always forced to :png, and
|
|
68
|
+
# `max_input_bytes` also sets the size of the ranged read.
|
|
69
|
+
def render_options
|
|
70
|
+
{}
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def render_image
|
|
74
|
+
options = render_options.merge(format: :png)
|
|
75
|
+
if (head = read_head(options))
|
|
76
|
+
render(head, options)
|
|
77
|
+
else
|
|
78
|
+
download_blob_to_tempfile { |file| render(file, options) }
|
|
79
|
+
end
|
|
80
|
+
rescue CsvPreview::Error => error
|
|
81
|
+
raise ActiveStorage::PreviewError, "#{error.class.name}: #{error.message}"
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def render(source, options)
|
|
85
|
+
instrument(:preview, key: blob.key) { CsvPreview.render(source, **options) }
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
# The first max_input_bytes bytes of the blob, or nil when the service cannot serve ranges.
|
|
89
|
+
def read_head(options)
|
|
90
|
+
limit = options.fetch(:max_input_bytes) { CsvPreview.config.max_input_bytes }
|
|
91
|
+
length = [blob.byte_size.to_i, limit].min
|
|
92
|
+
StringIO.new(length.positive? ? blob.download_chunk(0...length) : "".b)
|
|
93
|
+
rescue NotImplementedError
|
|
94
|
+
nil
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module CsvPreview
|
|
4
|
+
# Classifies sanitized cells and columns as numbers or text (numbers are right-aligned).
|
|
5
|
+
# See docs/spec/parsing-and-safety.md section 6.
|
|
6
|
+
# @api private
|
|
7
|
+
module CellKind
|
|
8
|
+
CURRENCY_CODES = %w[
|
|
9
|
+
USD EUR GBP JPY CNY CAD AUD NZD CHF HKD SGD INR KRW TWD THB MYR IDR PHP VND
|
|
10
|
+
MXN BRL ARS CLP COP PEN ZAR SEK NOK DKK ISK PLN CZK HUF RON TRY ILS AED SAR
|
|
11
|
+
RUB UAH NGN KES EGP
|
|
12
|
+
].freeze
|
|
13
|
+
SYMBOL = '(?:US\$|CA\$|C\$|A\$|AU\$|NZ\$|HK\$|S\$|R\$|MX\$|[$€£¥₹₩₽₪₫₱₴₺₦₵₸₼₾¢฿]|kr|Kč|zł)'
|
|
14
|
+
CUR = "(?:#{SYMBOL}|#{CURRENCY_CODES.join("|")})"
|
|
15
|
+
SP = '[ \u00A0\u202F]?'
|
|
16
|
+
GROUP = '[ \u00A0\u202F\u2009]'
|
|
17
|
+
SIGN = '[-+−]'
|
|
18
|
+
NUM = "(?:" + [
|
|
19
|
+
'\d{1,3}(?:,\d{3})+(?:\.\d+)?',
|
|
20
|
+
'\d{1,3}(?:\.\d{3})+(?:,\d+)?',
|
|
21
|
+
'\d{1,2}(?:,\d{2})+,\d{3}(?:\.\d+)?',
|
|
22
|
+
"\\d{1,3}(?:#{GROUP}\\d{3})+(?:[.,]\\d+)?",
|
|
23
|
+
'\d{1,3}(?:[\'’]\d{3})+(?:\.\d+)?',
|
|
24
|
+
'\d+(?:[.,]\d+)?(?:[eE][-+]?\d{1,3})?',
|
|
25
|
+
'[.,]\d+'
|
|
26
|
+
].join("|") + ")"
|
|
27
|
+
NUMBER = Regexp.new('\A(?:' + [
|
|
28
|
+
"#{SIGN}?#{SP}#{NUM}#{SP}%",
|
|
29
|
+
"#{SIGN}?#{SP}(?:#{CUR}#{SP})?#{NUM}(?:#{SP}#{CUR})?",
|
|
30
|
+
"#{CUR}#{SP}#{SIGN}#{SP}#{NUM}",
|
|
31
|
+
"(?:#{CUR}#{SP})?#{NUM}-",
|
|
32
|
+
"\\(#{SP}(?:#{CUR}#{SP})?#{NUM}(?:#{SP}#{CUR})?#{SP}\\)",
|
|
33
|
+
"#{CUR}#{SP}\\(#{NUM}\\)"
|
|
34
|
+
].join("|") + ')\z')
|
|
35
|
+
BARE_INTEGER = /\A\d+\z/
|
|
36
|
+
PLACEHOLDER = /\A(?:[-‐-―−]{1,3}|n\/?a|null|none|nan)\z/i
|
|
37
|
+
MAX_NUMBER_LENGTH = 40
|
|
38
|
+
NUMBER_SHARE = 0.8
|
|
39
|
+
|
|
40
|
+
module_function
|
|
41
|
+
|
|
42
|
+
# @param s [String] a sanitized cell
|
|
43
|
+
# @return [Symbol] :empty, :number or :text
|
|
44
|
+
def cell(s)
|
|
45
|
+
return :empty if s.empty?
|
|
46
|
+
return :text if s.length > MAX_NUMBER_LENGTH || !NUMBER.match?(s)
|
|
47
|
+
return :text if BARE_INTEGER.match?(s) && (s.length > 15 || (s.length > 1 && s.start_with?("0")))
|
|
48
|
+
|
|
49
|
+
:number
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
# @param cells [Array<String>] sanitized data cells of one column (header excluded)
|
|
53
|
+
# @return [Symbol] :number or :text
|
|
54
|
+
def column(cells)
|
|
55
|
+
numbers = []
|
|
56
|
+
texts = 0
|
|
57
|
+
cells.each do |c|
|
|
58
|
+
next if c.empty? || PLACEHOLDER.match?(c)
|
|
59
|
+
|
|
60
|
+
cell(c) == :number ? numbers << c : texts += 1
|
|
61
|
+
end
|
|
62
|
+
return :text if numbers.empty? || numbers.size < NUMBER_SHARE * (numbers.size + texts)
|
|
63
|
+
return :text if identifier_like?(numbers)
|
|
64
|
+
|
|
65
|
+
:number
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# Every numeric cell is a bare integer, there are at least two, and all have the
|
|
69
|
+
# same number of digits, at least 4: MCC codes, last-4, years, phone numbers.
|
|
70
|
+
def identifier_like?(numbers)
|
|
71
|
+
numbers.size >= 2 && numbers.all? { |n| BARE_INTEGER.match?(n) } &&
|
|
72
|
+
numbers.map(&:length).uniq.then { |l| l.size == 1 && l[0] >= 4 }
|
|
73
|
+
end
|
|
74
|
+
end
|
|
75
|
+
end
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module CsvPreview
|
|
4
|
+
# Option defaults and validation. The global instance is edited with {CsvPreview.configure};
|
|
5
|
+
# each call takes a validated, frozen snapshot with {#resolve}.
|
|
6
|
+
#
|
|
7
|
+
# Every option except {#previewer_content_types} can also be passed per call to
|
|
8
|
+
# {CsvPreview.render}, {CsvPreview.sample} and {CsvPreview.layout}; per-call values win. Invalid
|
|
9
|
+
# values raise `ArgumentError` before any input is read.
|
|
10
|
+
#
|
|
11
|
+
# @example Set defaults once at boot
|
|
12
|
+
# CsvPreview.configure do |config|
|
|
13
|
+
# config.max_rows = 12
|
|
14
|
+
# config.previewer_content_types += ["text/x-csv"]
|
|
15
|
+
# end
|
|
16
|
+
class Configuration
|
|
17
|
+
# The options that can be passed per call.
|
|
18
|
+
OPTIONS = %i[
|
|
19
|
+
max_input_bytes max_rows max_columns max_cell_chars max_width_px max_height_px
|
|
20
|
+
scale timeout header_row delimiter encoding format theme font
|
|
21
|
+
].freeze
|
|
22
|
+
|
|
23
|
+
# Options that must be positive Integers.
|
|
24
|
+
POSITIVE_INTEGERS = %i[max_input_bytes max_rows max_columns max_cell_chars max_width_px max_height_px].freeze
|
|
25
|
+
# Allowed values of {#scale}.
|
|
26
|
+
SCALES = (1..4)
|
|
27
|
+
# Allowed values of {#format}.
|
|
28
|
+
FORMATS = %i[png jpeg webp].freeze
|
|
29
|
+
# Allowed values of {#theme}.
|
|
30
|
+
THEMES = %i[ledger].freeze
|
|
31
|
+
# {#max_width_px} must be at least this times {#scale}.
|
|
32
|
+
MIN_WIDTH_PER_SCALE = 160
|
|
33
|
+
# {#max_height_px} must be at least this times {#scale}.
|
|
34
|
+
MIN_HEIGHT_PER_SCALE = 110
|
|
35
|
+
# Characters that cannot be a {#delimiter}.
|
|
36
|
+
FORBIDDEN_DELIMITERS = ["\"", "\r", "\n", "\0"].freeze
|
|
37
|
+
|
|
38
|
+
# @return [Integer] bytes read from the start of the source; nothing past this is read.
|
|
39
|
+
# Default 2_097_152 (2 MiB).
|
|
40
|
+
attr_accessor :max_input_bytes
|
|
41
|
+
# @return [Integer] data rows shown (the header is not counted). Default 20.
|
|
42
|
+
attr_accessor :max_rows
|
|
43
|
+
# @return [Integer] most columns kept and shown; fewer are shown if they do not fit. Default 12.
|
|
44
|
+
attr_accessor :max_columns
|
|
45
|
+
# @return [Integer] characters kept per cell before measuring; longer cells end in "…". Default 60.
|
|
46
|
+
attr_accessor :max_cell_chars
|
|
47
|
+
# @return [Integer] cap on output width in pixels, not a target: small tables produce narrower
|
|
48
|
+
# images. At least 160 * scale. Default 2048.
|
|
49
|
+
attr_accessor :max_width_px
|
|
50
|
+
# @return [Integer] cap on output height in pixels. At least 110 * scale. Default 2600.
|
|
51
|
+
attr_accessor :max_height_px
|
|
52
|
+
# @return [Integer] pixel density from 1 to 4; theme sizes are logical pixels times this. Default 2.
|
|
53
|
+
attr_accessor :scale
|
|
54
|
+
# @return [Numeric] wall-clock budget in seconds for one call, checked at checkpoints (no
|
|
55
|
+
# `Timeout.timeout`); past it the call raises {TimeoutExceeded}. Default 5.
|
|
56
|
+
attr_accessor :timeout
|
|
57
|
+
# @return [:auto, Boolean] whether the first row is a header; `:auto` detects it. Default `:auto`.
|
|
58
|
+
attr_accessor :header_row
|
|
59
|
+
# @return [:auto, String] `:auto` sniffs comma, semicolon, tab and pipe; or any single ASCII
|
|
60
|
+
# character other than a quote, CR, LF or NUL. Default `:auto`.
|
|
61
|
+
attr_accessor :delimiter
|
|
62
|
+
# @return [:auto, String] `:auto` detects UTF-8, UTF-16, UTF-32 and Windows-1252; or an encoding
|
|
63
|
+
# name such as "Windows-1252". Default `:auto`.
|
|
64
|
+
attr_accessor :encoding
|
|
65
|
+
# @return [Symbol] `:png`, `:jpeg` or `:webp`. The Active Storage previewer always produces PNG.
|
|
66
|
+
# Default `:png`.
|
|
67
|
+
attr_accessor :format
|
|
68
|
+
# @return [Symbol] the look; only `:ledger` in this version. Default `:ledger`.
|
|
69
|
+
attr_accessor :theme
|
|
70
|
+
# @return [String, nil] nil uses the bundled Inter; a String names a system font family instead.
|
|
71
|
+
# Default nil.
|
|
72
|
+
attr_accessor :font
|
|
73
|
+
# @return [Array<String>] content types the Active Storage previewer accepts (global only, not a
|
|
74
|
+
# per-call option). Default ["text/csv", "text/tab-separated-values"].
|
|
75
|
+
attr_accessor :previewer_content_types
|
|
76
|
+
|
|
77
|
+
# A configuration with the factory defaults (not yet validated or frozen; see {#resolve}).
|
|
78
|
+
def initialize
|
|
79
|
+
@max_input_bytes = 2_097_152
|
|
80
|
+
@max_rows = 20
|
|
81
|
+
@max_columns = 12
|
|
82
|
+
@max_cell_chars = 60
|
|
83
|
+
@max_width_px = 2048
|
|
84
|
+
@max_height_px = 2600
|
|
85
|
+
@scale = 2
|
|
86
|
+
@timeout = 5
|
|
87
|
+
@header_row = :auto
|
|
88
|
+
@delimiter = :auto
|
|
89
|
+
@encoding = :auto
|
|
90
|
+
@format = :png
|
|
91
|
+
@theme = :ledger
|
|
92
|
+
@font = nil
|
|
93
|
+
@previewer_content_types = ["text/csv", "text/tab-separated-values"]
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
# @param overrides [Hash] per-call option values; they win over this configuration
|
|
97
|
+
# @return [Configuration] a validated, frozen snapshot
|
|
98
|
+
# @raise [ArgumentError] on an unknown option or an invalid value
|
|
99
|
+
def resolve(**overrides)
|
|
100
|
+
unknown = overrides.keys - OPTIONS
|
|
101
|
+
raise ArgumentError, "unknown option(s): #{unknown.map(&:inspect).join(", ")}" unless unknown.empty?
|
|
102
|
+
|
|
103
|
+
snapshot = dup
|
|
104
|
+
overrides.each { |key, value| snapshot.instance_variable_set(:"@#{key}", value) } # keys are in OPTIONS (checked above)
|
|
105
|
+
snapshot.send(:validate!)
|
|
106
|
+
snapshot.previewer_content_types = previewer_content_types.dup.freeze
|
|
107
|
+
snapshot.freeze_string_options
|
|
108
|
+
snapshot.freeze
|
|
109
|
+
end
|
|
110
|
+
|
|
111
|
+
# @api private
|
|
112
|
+
def freeze_string_options
|
|
113
|
+
@font = @font.dup.freeze if @font.is_a?(String)
|
|
114
|
+
@delimiter = @delimiter.dup.freeze if @delimiter.is_a?(String)
|
|
115
|
+
@encoding = @encoding.dup.freeze if @encoding.is_a?(String)
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
private
|
|
119
|
+
|
|
120
|
+
def validate!
|
|
121
|
+
POSITIVE_INTEGERS.each do |name|
|
|
122
|
+
value = instance_variable_get(:"@#{name}") # name is a POSITIVE_INTEGERS constant
|
|
123
|
+
raise ArgumentError, "#{name} must be a positive Integer" unless value.is_a?(Integer) && value.positive?
|
|
124
|
+
end
|
|
125
|
+
raise ArgumentError, "scale must be an Integer from 1 to 4" unless scale.is_a?(Integer) && SCALES.cover?(scale)
|
|
126
|
+
|
|
127
|
+
min_width = MIN_WIDTH_PER_SCALE * scale
|
|
128
|
+
min_height = MIN_HEIGHT_PER_SCALE * scale
|
|
129
|
+
raise ArgumentError, "max_width_px must be at least #{min_width} at scale #{scale}" if max_width_px < min_width
|
|
130
|
+
raise ArgumentError, "max_height_px must be at least #{min_height} at scale #{scale}" if max_height_px < min_height
|
|
131
|
+
|
|
132
|
+
validate_timeout!
|
|
133
|
+
validate_header_row!
|
|
134
|
+
validate_delimiter!
|
|
135
|
+
validate_encoding!
|
|
136
|
+
raise ArgumentError, "format must be one of #{FORMATS.inspect}" unless FORMATS.include?(format)
|
|
137
|
+
raise ArgumentError, "theme must be one of #{THEMES.inspect}" unless THEMES.include?(theme)
|
|
138
|
+
raise ArgumentError, "font must be nil or a String" unless font.nil? || font.is_a?(String)
|
|
139
|
+
validate_content_types!
|
|
140
|
+
end
|
|
141
|
+
|
|
142
|
+
def validate_content_types!
|
|
143
|
+
return if previewer_content_types.is_a?(Array) && previewer_content_types.all?(String)
|
|
144
|
+
|
|
145
|
+
raise ArgumentError, "previewer_content_types must be an Array of Strings"
|
|
146
|
+
end
|
|
147
|
+
|
|
148
|
+
def validate_timeout!
|
|
149
|
+
return if timeout.is_a?(Numeric) && timeout.real? && timeout.positive? && timeout.to_f.finite?
|
|
150
|
+
|
|
151
|
+
raise ArgumentError, "timeout must be a positive, finite Numeric"
|
|
152
|
+
end
|
|
153
|
+
|
|
154
|
+
def validate_header_row!
|
|
155
|
+
return if [:auto, true, false].include?(header_row)
|
|
156
|
+
|
|
157
|
+
raise ArgumentError, "header_row must be :auto, true or false"
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
def validate_delimiter!
|
|
161
|
+
return if delimiter == :auto
|
|
162
|
+
return if delimiter.is_a?(String) && delimiter.length == 1 && delimiter.ascii_only? &&
|
|
163
|
+
!FORBIDDEN_DELIMITERS.include?(delimiter)
|
|
164
|
+
|
|
165
|
+
raise ArgumentError, "delimiter must be :auto or a single ASCII character other than a quote, CR, LF or NUL"
|
|
166
|
+
end
|
|
167
|
+
|
|
168
|
+
def validate_encoding!
|
|
169
|
+
return if encoding == :auto
|
|
170
|
+
|
|
171
|
+
raise ArgumentError, "encoding must be :auto or an encoding name" unless encoding.is_a?(String)
|
|
172
|
+
|
|
173
|
+
raise ArgumentError, "unsupported" if %w[internal external locale filesystem].include?(encoding.downcase)
|
|
174
|
+
|
|
175
|
+
enc = Encoding.find(encoding)
|
|
176
|
+
raise ArgumentError, "unsupported" if enc.nil? || enc.dummy? || enc == Encoding::BINARY
|
|
177
|
+
|
|
178
|
+
Encoding::Converter.new(enc, Encoding::UTF_8) unless enc == Encoding::UTF_8
|
|
179
|
+
rescue ArgumentError, Encoding::ConverterNotFoundError
|
|
180
|
+
raise ArgumentError, "encoding must be :auto or a known encoding name"
|
|
181
|
+
end
|
|
182
|
+
end
|
|
183
|
+
end
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module CsvPreview
|
|
4
|
+
# A cooperative wall-clock budget for one call. Code calls {#check!} at checkpoints; nothing is
|
|
5
|
+
# interrupted asynchronously.
|
|
6
|
+
# @api private
|
|
7
|
+
class Deadline
|
|
8
|
+
# The default clock: monotonic seconds, unaffected by changes to the wall clock.
|
|
9
|
+
CLOCK = -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }
|
|
10
|
+
|
|
11
|
+
# @param seconds [Numeric] the budget
|
|
12
|
+
# @param clock [#call] monotonic clock returning seconds (for tests)
|
|
13
|
+
def initialize(seconds, clock: CLOCK)
|
|
14
|
+
@seconds = seconds
|
|
15
|
+
@clock = clock
|
|
16
|
+
@expires_at = clock.call + seconds
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# @param stage [String, Symbol] name of the checkpoint (never input content)
|
|
20
|
+
# @raise [TimeoutExceeded] when the budget is spent
|
|
21
|
+
def check!(stage)
|
|
22
|
+
return if @clock.call <= @expires_at
|
|
23
|
+
|
|
24
|
+
raise TimeoutExceeded, "csv_preview exceeded its #{@seconds}s timeout (#{stage}). Pass a larger timeout " \
|
|
25
|
+
"option or render previews in a background job, see #{Help::ERRORS}"
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# @return [Float] seconds left, never negative
|
|
29
|
+
def remaining = [@expires_at - @clock.call, 0.0].max
|
|
30
|
+
end
|
|
31
|
+
end
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module CsvPreview
|
|
4
|
+
# Base class for every error the gem raises on purpose. Messages never include input content, and
|
|
5
|
+
# each one says what to do next and links the README section that explains it.
|
|
6
|
+
class Error < StandardError; end
|
|
7
|
+
|
|
8
|
+
# The input is binary, undecodable, or not text at all (for example a PNG or XLSX file renamed
|
|
9
|
+
# `.csv`). Show a fallback for such files; in Rails, discard the preview job's
|
|
10
|
+
# `ActiveStorage::PreviewError`.
|
|
11
|
+
class UnreadableInput < Error; end
|
|
12
|
+
|
|
13
|
+
# The call ran past its `timeout`. The message names the stage that noticed.
|
|
14
|
+
class TimeoutExceeded < Error; end
|
|
15
|
+
|
|
16
|
+
# libvips is missing, lacks text (Pango) support, failed, or cannot encode the requested format.
|
|
17
|
+
class RenderFailed < Error; end
|
|
18
|
+
|
|
19
|
+
# README sections that error messages and warnings link to, so a person or a coding agent reading a
|
|
20
|
+
# log knows where the fix is documented.
|
|
21
|
+
# @api private
|
|
22
|
+
module Help
|
|
23
|
+
REPOSITORY = "https://github.com/garyhtou/csv_preview"
|
|
24
|
+
INSTALLATION = "#{REPOSITORY}#installation"
|
|
25
|
+
ERRORS = "#{REPOSITORY}#errors"
|
|
26
|
+
OPTIONS = "#{REPOSITORY}#options"
|
|
27
|
+
RAILS = "#{REPOSITORY}#rails-usage"
|
|
28
|
+
ISSUES = "#{REPOSITORY}/issues"
|
|
29
|
+
end
|
|
30
|
+
end
|