marcel 1.0.4 → 1.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a51f606ee622330ffe46d2163caca9452ad4b9c704f6733628bb5c228bbdd15c
-  data.tar.gz: f661a5a357dd426328906391a0997f1919d4f36da4120006bb3be0e4a32b87e2
+  metadata.gz: 4631da96baa71a625b60ec1f66ed05c4f6966f62340a0e6c522f67133dd39e48
+  data.tar.gz: 61c1b75b32db1aa7ab69716fb42f0eb7b12d05090d5e56738a29254311fa1a21
 SHA512:
-  metadata.gz: 5e4cf80453f5f398a4012dafcd7c60f4c0badebf5850c64d318d3705c2de486ddb14015965f9d067f45afa5ddf4e22fd12d01d5069959b49114c5fe7c219278a
-  data.tar.gz: e03fffed012927c64dcfff9cc37a61db4d66ff7f0eb39517d8eefe1c5a0089cdcc630dc7385fd7dd95a6760b7b5b855c0999f1182a63cae75e8b12f9aabc44d2
+  metadata.gz: f8e1a5b1fce489a8ea20dccbf88b2771bb9441f94631377155e30b8bb5be04738beb13688e0b007c75112283323cd2b577e398825b070195d0bf3f4ef770b140
+  data.tar.gz: cfa87feac79dfc6c1f462a85533295c51f837a7ba9423f461309e8b6c41690a2dc163bba8da60608ccea3a860e14c9d554e40a0edd94a5116cb9565b0d23d4e3

data/README.md

@@ -1,8 +1,20 @@
 # Marcel
 
-Marcel attempts to choose the most appropriate content type for a given file by looking at the binary data, the filename, and any declared type (perhaps passed as a request header). This is done via the `Marcel::MimeType.for` method, and is used like this:
+Marcel chooses the most appropriate content type for a file by inspecting its contents, the declared MIME type (perhaps passed as a Content-Type header), and the file extension.
+
+Marcel checks, in order:
+
+1. The "magic bytes" sniffed from the file contents.
+2. The declared type, typically provided in a Content-Type header on an uploaded file, unless it's the `application/octet-stream` default.
+3. The filename extension.
+4. Safe fallback to the indeterminate `application/octet-stream` default.
+
+At each step, the most specific MIME subtype is selected. This allows the declared type and file extension to refine the parent type sniffed from the file contents, but not conflict with it. For example, if "file.csv" has declared type `text/plain`, `text/csv` is returned since it's a more specific subtype of `text/plain`. Similarly, Adobe Illustrator files are PDFs internally, so magic byte sniffing indicates `application/pdf` which is refined to `application/illustrator` by the `ai` file extension. But a PDF named "image.png" will still be detected as `application/pdf` since `image/png` is not a subtype.
+
+## Usage
 
 ```ruby
+# Magic bytes sniffing alone
 Marcel::MimeType.for Pathname.new("example.gif")
 #  => "image/gif"
 
@@ -11,37 +23,26 @@ File.open "example.gif" do |file|
 end
 #  => "image/gif"
 
+# Magic bytes with filename fallback
 Marcel::MimeType.for Pathname.new("unrecognisable-data"), name: "example.pdf"
 #  => "application/pdf"
 
+# File extension alone
 Marcel::MimeType.for extension: ".pdf"
 #  => "application/pdf"
 
+# Magic bytes, declared type, and filename together
 Marcel::MimeType.for Pathname.new("unrecognisable-data"), name: "example", declared_type: "image/png"
 #  => "image/png"
 
+# Safe fallback to application/octet-stream
 Marcel::MimeType.for StringIO.new(File.read "unrecognisable-data")
 #  => "application/octet-stream"
 ```
 
-By preference, the magic number data in any passed in file is used to determine the type. If this doesn't work, it uses the type gleaned from the filename, extension, and finally the declared type. If no valid type is found in any of these, "application/octet-stream" is returned.
-
-Some types aren't easily recognised solely by magic number data. For example Adobe Illustrator files have the same magic number as PDFs (and can usually even be viewed in PDF viewers!). For these types, Marcel uses both the magic number data and the file name to work out the type:
-
-```ruby
-Marcel::MimeType.for Pathname.new("example.ai"), name: "example.ai"
-#  => "application/illustrator"
-```
-
-This only happens when the type from the filename is a more specific type of that from the magic number. If it isn't the magic number alone is used.
-
-```ruby
-Marcel::MimeType.for Pathname.new("example.png"), name: "example.ai"
-#  => "image/png"
-# As "application/illustrator" is not a more specific type of "image/png", the filename is ignored
-```
+## Extending
 
-Custom file types not supported by Marcel can be added using `Marcel::MimeType.extend`. 
+Custom file types may be added with `Marcel::MimeType.extend`:
 
 ```ruby
 Marcel::MimeType.extend "text/custom", extensions: %w( customtxt )
@@ -51,17 +52,20 @@ Marcel::MimeType.for name: "file.customtxt"
 
 ## Motivation
 
-Marcel was extracted from Basecamp 3, in order to make our file detection logic both easily reusable but more importantly, easily testable. Test fixtures have been added for all of the most common file types uploaded to Basecamp, and other common file types too. We hope to expand this test coverage with other file types as and when problems are identified.
+Marcel was extracted from Basecamp's file detection heuristics. The aim is provide sensible, safe, "do what I expect" results for typical file handling. Test fixtures have been added for many common file types, including those typically encountered by Basecamp.
+
 
 ## Contributing
 
-Marcel generates MIME lookup tables with `bundle exec rake tables`. MIME types are seeded from data found in `data/*.xml`. Custom MIMEs may be added to `data/custom.xml`, while overrides to the standard MIME database may be added to `lib/marcel/mime_type/definitions.rb`.
+Marcel generates MIME lookup tables with `bundle exec rake update`. MIME types are seeded from data found in `data/*.xml`. Custom MIMEs may be added to `data/custom.xml`, while overrides to the standard MIME database may be added to `lib/marcel/mime_type/definitions.rb`.
 
 Marcel follows the same contributing guidelines as [rails/rails](https://github.com/rails/rails#contributing).
 
+
 ## Testing
 
-The main test fixture files are split into two folders, those that can be recognised by magic numbers, and those that can only be recognised by name. Even though strictly unnecessary, the fixtures in both folders should all be valid files of the type they represent.
+The main test fixture files are split into two folders, those that can be recognised by magic bytes, and those that can only be recognised by name. Even though strictly unnecessary, the fixtures in both folders should all be valid files of the type they represent.
+
 
 ## License
 

data/lib/marcel/magic.rb

@@ -117,10 +117,7 @@ module Marcel
     def self.magic_match(io, method)
       return magic_match(StringIO.new(io.to_s), method) unless io.respond_to?(:read)
 
-      io.binmode if io.respond_to?(:binmode)
-      io.set_encoding(Encoding::BINARY) if io.respond_to?(:set_encoding)
-      buffer = "".encode(Encoding::BINARY)
-
+      buffer = "".b
       MAGIC.send(method) { |type, matches| magic_match_io(io, matches, buffer) }
     end
 
@@ -128,7 +125,9 @@ module Marcel
       matches.any? do |offset, value, children|
         match =
           if value
-            if Range === offset
+            if value.is_a?(Regexp)
+              match_regex(io, offset, value, buffer)
+            elsif Range === offset
               io.read(offset.begin, buffer)
               x = io.read(offset.end - offset.begin + value.bytesize, buffer)
               x && x.include?(value)
@@ -143,6 +142,15 @@ module Marcel
       end
     end
 
-    private_class_method :magic_match, :magic_match_io
+    def self.match_regex(io, offset, regexp, buffer)
+      start = offset.is_a?(Range) ? offset.begin : offset
+      io.read(start, buffer) if start > 0
+      data = io.read(256, buffer)
+      return false unless data
+
+      data.match?(regexp)
+    end
+
+    private_class_method :magic_match, :magic_match_io, :match_regex
   end
 end

data/lib/marcel/mime_type/definitions.rb

@@ -2,6 +2,28 @@
 
 Marcel::MimeType.extend "text/plain", extensions: %w( txt asc )
 
+Marcel::Magic.remove("text/html")
+Marcel::MimeType.extend "text/html", 
+  extensions: %w( html htm ),
+  magic: [
+    [0, "<!DOCTYPE html"],
+    [0, "<!DOCTYPE HTML"], 
+    [0, "<!doctype html"],
+    [0, "<!doctype HTML"],
+    [0, "<html"],
+    [0, "<HTML"],
+    [0, " <!DOCTYPE html"],
+    [0, "\n<!DOCTYPE html"],
+    [0, "\r<!DOCTYPE html"], 
+    [0, "\r\n<!DOCTYPE html"],
+    [0, "\t<!DOCTYPE html"],
+    [0, " <html"],
+    [0, "\n<html"],
+    [0, "\r<html"],
+    [0, "\r\n<html"],
+    [0, "\t<html"]
+  ]
+
 Marcel::MimeType.extend "application/illustrator", parents: "application/pdf"
 Marcel::MimeType.extend "image/vnd.adobe.photoshop", magic: [[0, "8BPS"]], extensions: %w( psd psb )
 
@@ -33,9 +55,11 @@ Marcel::MimeType.extend "application/vnd.apple.numbers", extensions: %w( numbers
 Marcel::MimeType.extend "application/vnd.apple.keynote", extensions: %w( key ), parents: "application/zip"
 
 Marcel::MimeType.extend "audio/aac", extensions: %w( aac ), parents: "audio/x-aac"
+Marcel::MimeType.extend("audio/ogg", extensions: %w( opus ), magic: [[0, 'OggS', [[28, 'OpusHead']]]])
 Marcel::MimeType.extend("audio/ogg", extensions: %w( ogg oga ), magic: [[0, 'OggS', [[29, 'vorbis']]]])
 
 Marcel::MimeType.extend "image/vnd.dwg", magic: [[0, "AC10"]]
+Marcel::MimeType.extend "application/pkcs8", magic: [[0, '-----BEGIN PRIVATE KEY-----']], extensions: %w( p8 )
 
 Marcel::MimeType.extend "application/x-x509-ca-cert", magic: [[0, '-----BEGIN CERTIFICATE-----']], extensions: %w( pem ), parents: "application/x-x509-cert;format=pem"
 
@@ -43,6 +67,9 @@ Marcel::MimeType.extend "image/avif", magic: [[4, "ftypavif"]], extensions: %w(
 Marcel::MimeType.extend "image/heif", magic: [[4, "ftypmif1"]], extensions: %w( heif )
 Marcel::MimeType.extend "image/heic", magic: [[4, "ftypheic"]], extensions: %w( heic )
 
+Marcel::MimeType.extend "image/x-raw-sony", magic: [[0, "II*\000", [[0..4096, 'SONY']]], [0, "MM\000*", [[0..4096, 'SONY']]]], extensions: %w( arw ), parents: "image/tiff"
+Marcel::MimeType.extend "image/x-raw-canon",  extensions: %w( cr2 crw ), parents: "image/tiff"
+
 Marcel::MimeType.extend "video/mp4", magic: [[4, "ftypisom"], [4, "ftypM4V "]], extensions: %w( mp4 m4v )
 
 Marcel::MimeType.extend "audio/flac", magic: [[0, 'fLaC']], extensions: %w( flac ), parents: "audio/x-flac"