marcel 1.0.3 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d72cd32835e2c94cb305350ca65c70471ba0003730705c42f880571344b9a79c
-  data.tar.gz: bf87bf4e6739db822a9e474677d4785aad74a37add0ad88fba3c2daac24bb142
+  metadata.gz: 916c5ef797fe6c9f5b510ed333d0fa2d041625336d13c8bc0e79e4c85969130a
+  data.tar.gz: 9ea0baa6f2412b34f4bf7489ffa03cc58340161e794d52f97db0744d500c2dba
 SHA512:
-  metadata.gz: 002d7bf37923e0ce1da283ee7d41f36f604c454aa0924066013590805bb5bcaa878fab1cda9a884544afbcaf589384e0960a80774723f202a7c6dfa637bea05a
-  data.tar.gz: c3c361fc72ceec92d38e938613f8e1bbb7821cbe7221a59255d5d6eb8012aafd775b38f431c34df3fb5e9b443fedfdcda500fb76a2cb52e03cbe6028a042c26e
+  metadata.gz: 80bf2801abff8f03bc7a8189278187ffa61fc115281abbe0ae20b7a66d1488888d3a3d4810ce13187ea9de1a3502df5ee8d2a1393c7b7ce060917a46bbf02e3d
+  data.tar.gz: f1094603ebeabced48164200dcff42c2f5ba1c5fd068834f568f5620d12be13567db438ee87d67a978ed0b3bfa9368474c2c923ff88725d82510b977592832c4

data/README.md

@@ -1,8 +1,20 @@
 # Marcel
 
-Marcel attempts to choose the most appropriate content type for a given file by looking at the binary data, the filename, and any declared type (perhaps passed as a request header). This is done via the `Marcel::MimeType.for` method, and is used like this:
+Marcel chooses the most appropriate content type for a file by inspecting its contents, the declared MIME type (perhaps passed as a Content-Type header), and the file extension.
+
+Marcel checks, in order:
+
+1. The "magic bytes" sniffed from the file contents.
+2. The declared type, typically provided in a Content-Type header on an uploaded file, unless it's the `application/octet-stream` default.
+3. The filename extension.
+4. Safe fallback to the indeterminate `application/octet-stream` default.
+
+At each step, the most specific MIME subtype is selected. This allows the declared type and file extension to refine the parent type sniffed from the file contents, but not conflict with it. For example, if "file.csv" has declared type `text/plain`, `text/csv` is returned since it's a more specific subtype of `text/plain`. Similarly, Adobe Illustrator files are PDFs internally, so magic byte sniffing indicates `application/pdf` which is refined to `application/illustrator` by the `ai` file extension. But a PDF named "image.png" will still be detected as `application/pdf` since `image/png` is not a subtype.
+
+## Usage
 
 ```ruby
+# Magic bytes sniffing alone
 Marcel::MimeType.for Pathname.new("example.gif")
 #  => "image/gif"
 
@@ -11,37 +23,26 @@ File.open "example.gif" do |file|
 end
 #  => "image/gif"
 
+# Magic bytes with filename fallback
 Marcel::MimeType.for Pathname.new("unrecognisable-data"), name: "example.pdf"
 #  => "application/pdf"
 
+# File extension alone
 Marcel::MimeType.for extension: ".pdf"
 #  => "application/pdf"
 
+# Magic bytes, declared type, and filename together
 Marcel::MimeType.for Pathname.new("unrecognisable-data"), name: "example", declared_type: "image/png"
 #  => "image/png"
 
+# Safe fallback to application/octet-stream
 Marcel::MimeType.for StringIO.new(File.read "unrecognisable-data")
 #  => "application/octet-stream"
 ```
 
-By preference, the magic number data in any passed in file is used to determine the type. If this doesn't work, it uses the type gleaned from the filename, extension, and finally the declared type. If no valid type is found in any of these, "application/octet-stream" is returned.
-
-Some types aren't easily recognised solely by magic number data. For example Adobe Illustrator files have the same magic number as PDFs (and can usually even be viewed in PDF viewers!). For these types, Marcel uses both the magic number data and the file name to work out the type:
-
-```ruby
-Marcel::MimeType.for Pathname.new("example.ai"), name: "example.ai"
-#  => "application/illustrator"
-```
-
-This only happens when the type from the filename is a more specific type of that from the magic number. If it isn't the magic number alone is used.
-
-```ruby
-Marcel::MimeType.for Pathname.new("example.png"), name: "example.ai"
-#  => "image/png"
-# As "application/illustrator" is not a more specific type of "image/png", the filename is ignored
-```
+## Extending
 
-Custom file types not supported by Marcel can be added using `Marcel::MimeType.extend`. 
+Custom file types may be added with `Marcel::MimeType.extend`:
 
 ```ruby
 Marcel::MimeType.extend "text/custom", extensions: %w( customtxt )
@@ -51,17 +52,20 @@ Marcel::MimeType.for name: "file.customtxt"
 
 ## Motivation
 
-Marcel was extracted from Basecamp 3, in order to make our file detection logic both easily reusable but more importantly, easily testable. Test fixtures have been added for all of the most common file types uploaded to Basecamp, and other common file types too. We hope to expand this test coverage with other file types as and when problems are identified.
+Marcel was extracted from Basecamp's file detection heuristics. The aim is provide sensible, safe, "do what I expect" results for typical file handling. Test fixtures have been added for many common file types, including those typically encountered by Basecamp.
+
 
 ## Contributing
 
-Marcel generates MIME lookup tables with `bundle exec rake tables`. MIME types are seeded from data found in `data/*.xml`. Custom MIMEs may be added to `data/custom.xml`, while overrides to the standard MIME database may be added to `lib/marcel/mime_type/definitions.rb`.
+Marcel generates MIME lookup tables with `bundle exec rake update`. MIME types are seeded from data found in `data/*.xml`. Custom MIMEs may be added to `data/custom.xml`, while overrides to the standard MIME database may be added to `lib/marcel/mime_type/definitions.rb`.
 
 Marcel follows the same contributing guidelines as [rails/rails](https://github.com/rails/rails#contributing).
 
+
 ## Testing
 
-The main test fixture files are split into two folders, those that can be recognised by magic numbers, and those that can only be recognised by name. Even though strictly unnecessary, the fixtures in both folders should all be valid files of the type they represent.
+The main test fixture files are split into two folders, those that can be recognised by magic bytes, and those that can only be recognised by name. Even though strictly unnecessary, the fixtures in both folders should all be valid files of the type they represent.
+
 
 ## License
 

data/lib/marcel/magic.rb

@@ -119,7 +119,7 @@ module Marcel
 
       io.binmode if io.respond_to?(:binmode)
       io.set_encoding(Encoding::BINARY) if io.respond_to?(:set_encoding)
-      buffer = "".encode(Encoding::BINARY)
+      buffer = (+"").encode(Encoding::BINARY)
 
       MAGIC.send(method) { |type, matches| magic_match_io(io, matches, buffer) }
     end

data/lib/marcel/mime_type/definitions.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 Marcel::MimeType.extend "text/plain", extensions: %w( txt asc )
+Marcel::MimeType.extend "text/html", magic: [[0..64, "<!DOCTYPE HTML"], [0..64, "<!DOCTYPE html"], [0..64, "<!doctype HTML"], [0..64, "<!doctype html"]]
 
 Marcel::MimeType.extend "application/illustrator", parents: "application/pdf"
 Marcel::MimeType.extend "image/vnd.adobe.photoshop", magic: [[0, "8BPS"]], extensions: %w( psd psb )
@@ -43,6 +44,9 @@ Marcel::MimeType.extend "image/avif", magic: [[4, "ftypavif"]], extensions: %w(
 Marcel::MimeType.extend "image/heif", magic: [[4, "ftypmif1"]], extensions: %w( heif )
 Marcel::MimeType.extend "image/heic", magic: [[4, "ftypheic"]], extensions: %w( heic )
 
+Marcel::MimeType.extend "image/x-raw-sony", extensions: %w( arw ), parents: "image/tiff"
+Marcel::MimeType.extend "image/x-raw-canon",  extensions: %w( cr2 crw ), parents: "image/tiff"
+
 Marcel::MimeType.extend "video/mp4", magic: [[4, "ftypisom"], [4, "ftypM4V "]], extensions: %w( mp4 m4v )
 
 Marcel::MimeType.extend "audio/flac", magic: [[0, 'fLaC']], extensions: %w( flac ), parents: "audio/x-flac"

data/lib/marcel/mime_type.rb

@@ -60,7 +60,12 @@ module Marcel
         end
 
         def for_declared_type(declared_type)
-          parse_media_type(declared_type)
+          type = parse_media_type(declared_type)
+
+          # application/octet-stream is treated as an undeclared/missing type,
+          # allowing the type to be inferred from the filename. If there's no
+          # filename extension, then the type falls back to binary anyway.
+          type unless type == BINARY
         end
 
         def with_io(pathname_or_io, &block)