format_parser 0.9.0 → 0.9.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 91bcb409bce9e0576a83680ef91cd6e910e6df9b4bc4953cb23de8d887021b59
-  data.tar.gz: ef6f74cd026bb3dd26e16c3282f9ec71e5b63f8e957d4b818cdac0acbc058c74
+  metadata.gz: cd4a3b56391cebff09efc933b5fd48188c67f913adeec46c77a832ff067fd870
+  data.tar.gz: 159c14df0b5740f627a99915f05750bf753c89620017733eab067c6e865e972d
 SHA512:
-  metadata.gz: c118d28ecce997d45058d3b19de89d1c8ab6a88df390c58875f4ad22b36043f4562913c78c66cea380357184d48fddf6cf5736a8f7f7eb62a83fea6bddc85b13
-  data.tar.gz: 70ee5504c514fbed060ca62cf0cd0e429f1f01535e805de6a974dbcd5a9154b81c859a70a09ad39fc2ef8c341391f3f1f2a2ed3566a19976884dea03b9d541d6
+  metadata.gz: 559b0f5709bd8fc23cb20468eed5c3840a4087a444a581591b530490278532dff8700a54652e7e9cc8ab82b89ac2db910bf8e79620c7de5cf5a172cd7285ade3
+  data.tar.gz: d5baa5e7b5aea3cce2acf2513509b4dac8ad40dceb29a69875a10e4322dd136214ae650e82254416d1d39907a33cb18ed0d40634af0fa974a68ce2b0dc99b7dd

data/CHANGELOG.md

@@ -1,3 +1,16 @@
+## 0.9.3
+* Fix a JPEG parsing regression introduced in 0.9.1
+
+## 0.9.2 (yanked)
+* Make sure MP3 parser returns `nil` when encountering infinite duration
+* Do not read JPEG APP1 markers that contain no EXIF data
+* Explicitly replace `Float::INFINITY` values in `AttributesJSON` with `nil` as per JSON convention
+* Make sure the cached pages in `Care` are explicitly deleted after each `parse` call (should help GC)
+* Raise the pagefaults restriction to 16 to cope with "too many useless markers in JPEGs" scenario once more
+
+## 0.9.1 (yanked)
+* Perf: Make JPEG parser bail out earlier if no marker is found while scanning through 1024 bytes of data
+
 ## 0.9.0
 * Add a parser for the BMP image file format
 

data/CONTRIBUTING.md

@@ -87,6 +87,82 @@ or no result as soon as possible (once you know the file is not fit for your spe
 Bear in mind that we enforce read budgets per-parser, so you will not be allowed to perform
 too many reads, or perform reads which are too large.
 
+In order to create new parsers, it is recommended to make a well-named class with an instance method `call`.
+
+`call` accepts the IO-ish object as an argument, parses data that it reads from it,
+and then returns the metadata for the file (if it could recover any) or `nil` if it couldn't. All files pass
+through all parsers by default, so if you are dealing with a file that is not "your" format - return `nil` from
+your method or `break` your Proc as early as possible. A blank `return` works fine too.
+
+The IO will at the minimum support the subset of the IO API defined in `IOConstraint`
+
+Your parser has to be registered using `FormatParser.register_parser` with the information on the formats
+and file natures it provides.
+
+Down below you can find the most basic parser implementation:
+
+```ruby
+MyParser = ->(io) {
+  # ... do some parsing with `io`
+  magic_bytes = io.read(4)
+  # breaking the block returns `nil` to the caller signaling "no match"
+  break if magic_bytes != 'IMGA'
+
+  parsed_witdh, parsed_height = io.read(8).unpack('VV')
+  # ...and return the FileInformation::Image object with the metadata.
+  FormatParser::Image.new(
+    format: :imga,
+    width_px: parsed_width,
+    height_px: parsed_height,
+  )
+}
+
+# Register the parser with the module, so that it will be applied to any
+# document given to `FormatParser.parse()`. The supported natures are currently
+#      - :audio
+#      - :document
+#      - :image
+#      - :video
+#      - :archive
+FormatParser.register_parser MyParser, natures: :image, formats: :imga
+```
+
+If you are using a class, this is the skeleton to use:
+
+```ruby
+class MyParser
+  def call(io)
+    # ... do some parsing with `io`
+    # The instance will be discarded after parsing, so using instance variables
+    # is permitted - they are not shared between calls to `call`
+    @magic_bytes = io.read(4)
+    break if @magic_bytes != 'IMGA'
+    parsed_witdh, parsed_height = io.read(8).unpack('VV')
+    FormatParser::Image.new(
+      format: :imga,
+      width_px: parsed_width,
+      height_px: parsed_height,
+    )
+  end
+
+  FormatParser.register_parser self, natures: :image, formats: :bmp
+end
+```
+
+### Calling convention for preparing parsers
+
+A parser that gets registered using `register_parser` must be either:
+
+1) An object that can be `call()`-ed itself, with an argument that conforms to `IOConstraint`
+2) An object that responds to `new` and returns something that can be `call()`-ed with with an argument that conforms to `IOConstraint`.
+
+The second opton is recommended  for most cases.
+
+FormatParser is made to be used in threaded environments, and if you use instance variables
+you need your parser to be isolated from it's siblings in other threads - therefore you can pass
+a Class on registration to have your parser instantiated for each `call()`, anew.
+
+
 ## Pull requests
 
 Good pull requests-patches, improvements, new features-are a fantastic

data/README.md

@@ -12,7 +12,24 @@ and [dimensions,](https://github.com/sstephenson/dimensions) borrowing from them
 
 ## Currently supported filetypes:
 
-`TIFF, CR2, PSD, PNG, MP3, JPEG, GIF, PDF, DPX, AIFF, WAV, FDX, MOV, MP4, M4A, FLAC, DOCX, PPTX, XLSX`
+* TIFF
+* CR2
+* PSD
+* PNG
+* MP3
+* JPEG
+* GIF
+* PDF
+* DPX
+* AIFF
+* WAV
+* FLAC
+* FDX
+* MOV
+* MP4
+* M4A
+* ZIP
+* DOCX, PPTX, XLSX
 
 ...with [more](https://github.com/WeTransfer/format_parser/issues?q=is%3Aissue+is%3Aopen+label%3Aformats) on the way!
 
@@ -32,7 +49,7 @@ match.orientation   #=> :top_left
 If you would rather receive all potential results from the gem, call the gem as follows:
 
 ```ruby
-FormatParser.parse(File.open("myimage.jpg", "rb"), results: :all)
+array_of_results = FormatParser.parse(File.open("myimage.jpg", "rb"), results: :all)
 ```
 
 You can also optimize the metadata extraction by providing hints to the gem:
@@ -50,69 +67,7 @@ JSON.pretty_generate(img_info) #=> ...
 
 ## Creating your own parsers
 
-In order to create new parsers, you have to write a method or a  Proc that accepts an IO and performs the
-parsing, and then returns the metadata for the file (if it could recover any) or `nil` if it couldn't. All files pass
-through all parsers by default, so if you are dealing with a file that is not "your" format - return `nil` from
-your method or `break` your Proc as early as possible. A blank `return` works fine too.
-
-The IO will at the minimum support the subset of the IO API defined in `IOConstraint`
-
-Strictly, a parser should be one of the two things:
-
-1) An object that can be `call()`-ed itself, with an argument that conforms to `IOConstraint`
-2) An object that responds to `new` and returns something that can be `call()`-ed with the same convention.
-
-The second opton is useful for parsers that are stateful and non-reentrant. FormatParser is made to be used in
-threaded environments, and if you use instance variables you need your parser to be isolated from it's siblings in
-other threads - therefore you can pass a Class on registration to have your parser instantiated for each `call()`,
-anew.
-
-Your parser has to be registered using `FormatParser.register_parser` with the information on the formats
-and file natures it provides.
-
-Down below you can find a basic parser implementation:
-
-```ruby
-MyParser = ->(io) {
-  # ... do some parsing with `io`
-  magic_bytes = io.read(4)
-  break if magic_bytes != 'XBMP'
-  # ... more parsing code
-  # ...and return the FileInformation::Image object with the metadata.
-  FormatParser::Image.new(
-    width_px: parsed_width,
-    height_px: parsed_height,
-  )
-}
-
-# Register the parser with the module, so that it will be applied to any
-# document given to `FormatParser.parse()`. The supported natures are currently
-#      - :audio
-#      - :document
-#      - :image
-#      - :video
-FormatParser.register_parser MyParser, natures: :image, formats: :bmp
-```
-
-If you are using a class, this is the skeleton to use:
-
-```ruby
-class MyParser
-  def call(io)
-    # ... do some parsing with `io`
-    magic_bytes = io.read(4)
-    return unless magic_bytes == 'XBMP'
-    # ... more parsing code
-    # ...and return the FileInformation::Image object with the metadata.
-    FormatParser::Image.new(
-      width_px: parsed_width,
-      height_px: parsed_height,
-    )
-  end
-
-  FormatParser.register_parser self, natures: :image, formats: :bmp
-end
-```
+See the [section on writing parsers in CONTRIBUTING.md](CONTRIBUTING.md#so-you-want-to-contribute-a-new-parser)
 
 ## Design rationale
 
@@ -151,6 +106,7 @@ Unless specified otherwise in this section the fixture files are MIT licensed an
 ### JPEG
 - `divergent_pixel_dimensions_exif.jpg` is used with permission from LiveKom GmbH
 - `extended_reads.jpg` has kindly been made available by Raphaelle Pellerin for use exclusively with format_parser
+- `too_many_APP1_markers_surrogate.jpg` was created by the project maintainers
 
 ### AIFF
 - fixture.aiff was created by one of the project maintainers and is MIT licensed

data/lib/attributes_json.rb

@@ -20,6 +20,7 @@ module FormatParser::AttributesJSON
     methods.grep(/\w\=$/).each_with_object(h) do |attr_writer_method_name, h|
       reader_method_name = attr_writer_method_name.to_s.gsub(/\=$/, '')
       value = public_send(reader_method_name)
+      value = nil if value == Float::INFINITY
       # When calling as_json on our members there is no need to pass the root: option given to us
       # by the caller
       h[reader_method_name] = value.respond_to?(:as_json) ? value.as_json : value

data/lib/format_parser.rb

@@ -141,6 +141,8 @@ module FormatParser
     end
 
     amount == 1 ? results.first : results
+  ensure
+    cached_io.clear if cached_io
   end
 
   def self.execute_parser_and_capture_expected_exceptions(parser, limited_io)

data/lib/format_parser/version.rb

@@ -1,3 +1,3 @@
 module FormatParser
-  VERSION = '0.9.0'
+  VERSION = '0.9.3'
 end

data/lib/parsers/jpeg_parser.rb

@@ -10,6 +10,7 @@ class FormatParser::JPEGParser
   SOS_MARKER  = 0xDA  # start of stream
   APP1_MARKER = 0xE1  # maybe EXIF
   EXIF_MAGIC_STRING = "Exif\0\0".b
+  MUST_FIND_NEXT_MARKER_WITHIN_BYTES = 1024
 
   def call(io)
     @buf = FormatParser::IOConstraint.new(io)
@@ -61,6 +62,8 @@ class FormatParser::JPEGParser
       end
     end
 
+    FormatParser::Measurometer.add_distribution_value('format_parser.JPEGParser.bytes_read_until_capture', @buf.pos)
+
     # Return at the earliest possible opportunity
     if @width && @height
       return  FormatParser::Image.new(
@@ -78,11 +81,18 @@ class FormatParser::JPEGParser
   end
 
   # Read a byte, if it is 0xFF then skip bytes as long as they are also 0xFF (byte stuffing)
-  # and return the first byte scanned that is not 0xFF
+  # and return the first byte scanned that is not 0xFF. Also applies limits so that we do not
+  # read for inordinate amount of time should we encounter a file where we _do_ have a SOI
+  # marker at the start and then no markers for a _very_ long time (happened with some PSDs)
   def read_next_marker
-    c = read_char while c != 0xFF
-    c = read_char while c == 0xFF
-    c
+    # We need to find a sequence of two bytes - the first one is 0xFF, the other is anything but 0xFF
+    a = read_char
+    (MUST_FIND_NEXT_MARKER_WITHIN_BYTES - 1).times do
+      b = read_char
+      return b if a == 0xFF && b != 0xFF # Caught the marker
+      a = b # Shift the tuple one byte forward
+    end
+    nil # Nothing found
   end
 
   def scan_start_of_frame
@@ -107,34 +117,42 @@ class FormatParser::JPEGParser
     # pry it out of the APP1 frame and parse it as the TIFF segment - which is what EXIFR
     # does under the hood.
     app1_frame_content_length = read_short - 2
-    app1_frame_bytes = safe_read(@buf, app1_frame_content_length)
-
-    maybe_exif_magic_str = app1_frame_bytes[0..5]
-    maybe_exif_data = app1_frame_bytes[6..-1]
-    if maybe_exif_magic_str == EXIF_MAGIC_STRING
-      FormatParser::Measurometer.add_distribution_value('format_parser.JPEGParser.bytes_sent_to_exif_parser', maybe_exif_data.bytesize)
-      scanner = FormatParser::EXIFParser.new(StringIO.new(maybe_exif_data))
-      scanner.scan_image_tiff
-
-      @exif_output = scanner.exif_data
-      @orientation = scanner.orientation unless scanner.orientation.nil?
-      @intrinsics[:exif_pixel_x_dimension] = @exif_output.pixel_x_dimension
-      @intrinsics[:exif_pixel_y_dimension] = @exif_output.pixel_y_dimension
-      # Save these two for later, when we decide to provide display width /
-      # display height in addition to pixel buffer width / height. These two
-      # are _different concepts_. Imagine you have an image shot with a camera
-      # in portrait orientation, and the camera has an anamorphic lens. That
-      # anamorpohic lens is a smart lens, and therefore transmits pixel aspect
-      # ratio to the camera, and the camera encodes that aspect ratio into the
-      # image metadata. If we want to know what size our _pixel buffer_ will be,
-      # and how to _read_ the pixel data (stride/interleaving) - we need the
-      # pixel buffer dimensions. If we want to know what aspect and dimensions
-      # our file is going to have _once displayed_ and _once pixels have been
-      # brought to the right orientation_ we need to work with **display dimensions**
-      # which can be remarkably different from the pixel buffer dimensions.
-      @exif_width = scanner.width
-      @exif_height = scanner.height
+
+    # Peek whether the contents of the marker starts with Exif\0
+    maybe_exif_magic_str = safe_read(@buf, EXIF_MAGIC_STRING.bytesize)
+
+    # If we could not find the magic Exif\0 string at the start of the marker,
+    # seek to the start of the next marker and return
+    unless maybe_exif_magic_str == EXIF_MAGIC_STRING
+      safe_skip(@buf, app1_frame_content_length - EXIF_MAGIC_STRING.bytesize)
+      return
     end
+
+    # ...and only then read the marker contents and parse it as EXIF
+    exif_data = safe_read(@buf, app1_frame_content_length - EXIF_MAGIC_STRING.bytesize)
+
+    FormatParser::Measurometer.add_distribution_value('format_parser.JPEGParser.bytes_sent_to_exif_parser', exif_data.bytesize)
+    scanner = FormatParser::EXIFParser.new(StringIO.new(exif_data))
+    scanner.scan_image_tiff
+
+    @exif_output = scanner.exif_data
+    @orientation = scanner.orientation unless scanner.orientation.nil?
+    @intrinsics[:exif_pixel_x_dimension] = @exif_output.pixel_x_dimension
+    @intrinsics[:exif_pixel_y_dimension] = @exif_output.pixel_y_dimension
+    # Save these two for later, when we decide to provide display width /
+    # display height in addition to pixel buffer width / height. These two
+    # are _different concepts_. Imagine you have an image shot with a camera
+    # in portrait orientation, and the camera has an anamorphic lens. That
+    # anamorpohic lens is a smart lens, and therefore transmits pixel aspect
+    # ratio to the camera, and the camera encodes that aspect ratio into the
+    # image metadata. If we want to know what size our _pixel buffer_ will be,
+    # and how to _read_ the pixel data (stride/interleaving) - we need the
+    # pixel buffer dimensions. If we want to know what aspect and dimensions
+    # our file is going to have _once displayed_ and _once pixels have been
+    # brought to the right orientation_ we need to work with **display dimensions**
+    # which can be remarkably different from the pixel buffer dimensions.
+    @exif_width = scanner.width
+    @exif_height = scanner.height
   rescue EXIFR::MalformedTIFF
     # Not a JPEG or the Exif headers contain invalid data, or
     # an APP1 marker was detected in a file that is not a JPEG

data/lib/parsers/mp3_parser.rb

@@ -83,6 +83,10 @@ class FormatParser::MP3Parser
     est_samples = est_frame_count * SAMPLES_PER_FRAME
     est_duration_seconds = est_samples / avg_sample_rate
 
+    # Safeguard for i.e. some JPEGs being recognized as MP3
+    # to prevent ambiguous recognition
+    return if est_duration_seconds == Float::INFINITY
+
     file_info.media_duration_seconds = est_duration_seconds
     file_info
   end

data/lib/read_limiter.rb

@@ -2,6 +2,8 @@
 class FormatParser::ReadLimiter
   NO_LIMIT = nil
 
+  attr_reader :seeks, :reads, :bytes
+
   class BudgetExceeded < StandardError
   end
 

data/lib/read_limits_config.rb

@@ -1,5 +1,5 @@
 class FormatParser::ReadLimitsConfig
-  MAX_PAGE_FAULTS = 8
+  MAX_PAGE_FAULTS = 16
 
   def initialize(total_bytes_available_per_parser)
     @max_read_bytes_per_parser = total_bytes_available_per_parser.to_i

data/spec/attributes_json_spec.rb

@@ -27,6 +27,21 @@ describe FormatParser::AttributesJSON do
     end
   end
 
+  it 'converts Float::INFINITY to nil' do
+    anon_class = Class.new do
+      include FormatParser::AttributesJSON
+      attr_accessor :some_infinity
+      def some_infinity
+        Float::INFINITY
+      end
+    end
+    instance = anon_class.new
+    output = JSON.dump(instance)
+    readback = JSON.parse(output, symbolize_names: true)
+    expect(readback).to have_key(:some_infinity)
+    expect(readback[:some_infinity]).to be_nil
+  end
+
   it 'provides a default implementation of to_json as well' do
     anon_class = Class.new do
       include FormatParser::AttributesJSON

data/spec/esoteric_formats_spec.rb

@@ -13,4 +13,11 @@ describe 'Parsing esoteric files and files causing ambiguous detection' do
     result = FormatParser.parse(File.open(jpeg_path, 'rb'))
     expect(result.nature).to eq(:archive)
   end
+
+  it 'returns a result for JPEG file that causes many reads due to too many APP1 markers' do
+    jpeg_path = fixtures_dir + '/JPEG/too_many_APP1_markers_surrogate.jpg'
+    result = FormatParser.parse(File.open(jpeg_path, 'rb'))
+    expect(result).not_to be_nil
+    expect(result.nature).to eq(:image)
+  end
 end

data/spec/format_parser_spec.rb

@@ -32,10 +32,11 @@ describe FormatParser do
     FormatParser.register_parser exploit, natures: :document, formats: :exploit
 
     sample_io = StringIO.new(Random.new.bytes(1024 * 1024 * 8))
-
-    expect(sample_io).to receive(:read).at_most(8).times.and_call_original
+    allow(sample_io).to receive(:read).and_call_original
 
     result = FormatParser.parse(sample_io, formats: [:exploit])
+
+    expect(sample_io).to have_received(:read).at_most(16).times
     expect(result).to be_nil
 
     FormatParser.deregister_parser(exploit)

data/spec/parsers/jpeg_parser_spec.rb

@@ -49,9 +49,47 @@ describe FormatParser::JPEGParser do
     expect(result.intrinsics).to eq(exif_pixel_x_dimension: 8214, exif_pixel_y_dimension: 5476)
   end
 
+  it 'reads an example with many APP1 markers at the beginning of which none are EXIF' do
+    fixture_path = fixtures_dir + '/JPEG/too_many_APP1_markers_surrogate.jpg'
+    io = FormatParser::ReadLimiter.new(File.open(fixture_path, 'rb'))
+
+    result = subject.call(io)
+
+    expect(result).not_to be_nil
+    expect(result.width_px).to eq(1920)
+    expect(result.height_px).to eq(1200)
+
+    expect(io.bytes).to be < (128 * 1024)
+    expect(io.reads).to be < (1024 * 4)
+  end
+
+  it 'does not continue parsing for inordinate amount of time if the file contains no 0xFF bytes' do
+    # Create a large fuzzed input that consists of any bytes except 0xFF,
+    # so that the marker detector has nothing to latch on to
+    bytes_except_byte_255 = 0x0..0xFE
+
+    # Start the blob with the usual SOI marker - 0xFF 0xD8, so that the parser does not
+    # bail out too early and actually "bites" into the blob
+    no_markers = ([0xFF, 0xD8] + (16 * 1024).times.map { rand(bytes_except_byte_255) }).pack('C*')
+
+    # Yes, assertions on a private method - but we want to ensure we do not read more
+    # single bytes than the restriction stipulates we may. At the same time we check that
+    # the method does indeed, get triggered
+    allow(subject).to receive(:read_char).and_call_original
+    result = subject.call(StringIO.new(no_markers))
+    expect(result).to be_nil
+    expect(subject).to have_received(:read_char).at_most(1026).times
+  end
+
   it 'does not return a result for a Keynote document' do
     key_path = fixtures_dir + '/JPEG/keynote_recognized_as_jpeg.key'
     result = subject.call(File.open(key_path, 'rb'))
     expect(result).to be_nil
   end
+
+  it 'parses the the marker structure correctly when marker bytes cannot be read in groups of 2' do
+    kitten_path = fixtures_dir + '/JPEG/off-cadence-markers.jpg'
+    result = subject.call(File.open(kitten_path, 'rb'))
+    expect(result).not_to be_nil
+  end
 end

data/spec/parsers/mp3_parser_spec.rb

@@ -44,4 +44,11 @@ describe FormatParser::MP3Parser do
 
     expect(parsed.intrinsics).not_to be_nil
   end
+
+  it 'avoids returning a result when the parsed duration is infinite' do
+    fpath = fixtures_dir + '/JPEG/too_many_APP1_markers_surrogate.jpg'
+    parsed = subject.call(File.open(fpath, 'rb'))
+
+    expect(parsed).to be_nil
+  end
 end

data/spec/read_limiter_spec.rb

@@ -18,6 +18,18 @@ describe FormatParser::ReadLimiter do
     expect(reader.pos).to eq(2)
   end
 
+  it 'exposes #reads, #seeks, #bytes' do
+    reader = FormatParser::ReadLimiter.new(io)
+    expect(reader.pos).to eq(0)
+    reader.read(2)
+    reader.seek(3)
+    reader.seek(4)
+
+    expect(reader.reads).to eq(1)
+    expect(reader.bytes).to eq(2)
+    expect(reader.seeks).to eq(2)
+  end
+
   it 'enforces the number of seeks' do
     reader = FormatParser::ReadLimiter.new(io, max_seeks: 4)
     4.times { reader.seek(1) }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: format_parser
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 0.9.3
 platform: ruby
 authors:
 - Noah Berman
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2018-04-18 00:00:00.000000000 Z
+date: 2018-04-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ks