format_parser 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ba98fc34f902ba77ee29c4e96c69ef5e583c86ac
+  data.tar.gz: 757999ef4e7d25fe268637f7b26843e41da7e342
+SHA512:
+  metadata.gz: 12dd8fe78b7b58d24e8e15098c1c825ceb0877f33b393b8757178a0d15ca569098c0830e157f069a45b649baef837d774020eac047792d9c27cbea8cc56a731b
+  data.tar.gz: 130256991758f0a7c6fd399bfc8ab50e6418dd7a8bdbe73580da4e47ceb1ca3ac337a020d9b93f423c656258725704dc56a9f02da640a16cc0831ea72811353f

data/.gitignore

@@ -0,0 +1,13 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.gem
+
+# rspec failure tracking
+.rspec_status

data/.rspec

@@ -0,0 +1,3 @@
+--format progress
+--order rand
+--color

data/.travis.yml

@@ -0,0 +1,11 @@
+rvm:
+- 2.3.0
+- 2.4.2
+- jruby-9.0
+sudo: false
+cache: bundler
+matrix:
+  allow_failures:
+    - rvm: jruby-9.0
+script:
+  - bundle exec rspec

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at julik@wetransfer.com and/or noah@wetransfer.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/CONTRIBUTING.md

@@ -0,0 +1,157 @@
+# Contributing to format_parser
+
+Please take a moment to review this document in order to make the contribution
+process easy and effective for everyone involved.
+
+Following these guidelines helps to communicate that you respect the time of
+the developers managing and developing this open source project. In return,
+they should reciprocate that respect in addressing your issue or assessing
+patches and features.
+
+## What do I need to know to help?
+
+If you are already familiar with the [Ruby Programming Language](https://www.ruby-lang.org/) you can start contributing code right away, otherwise look for issues labeled with *documentation* or *good first issue* to get started.
+
+If you are interested in contributing code and would like to learn more about the technologies that we use, check out the (non-exhaustive) list below. You can also get in touch with us via an issue or email to julik@wetransfer.com and/or noah@wetransfer.com to get additional information.
+
+ - [ruby](https://ruby-doc.org)
+ - [rspec](http://rspec.info/) (for testing)
+ 
+# How do I make a contribution?
+
+## Using the issue tracker
+
+The issue tracker is the preferred channel for [bug reports](#bug-reports),
+[feature requests](#feature-requests) and [submitting pull
+requests](#pull-requests), but please respect the following restrictions:
+
+* Please **do not** derail or troll issues. Keep the discussion on topic and respect the opinions of others. Adhere to the principles set out in the [Code of Conduct](https://github.com/WeTransfer/format_parser/blob/master/CODE_OF_CONDUCT.md).
+
+## Bug reports
+
+A bug is a _demonstrable problem_ that is caused by code in the repository.
+
+Good bug reports are extremely helpful-thank you!
+
+Guidelines for bug reports:
+
+1. **Use the GitHub issue search** – check if the issue has already been
+   reported.
+
+2. **Check if the issue has been fixed** – try to reproduce it using the
+   latest `master` branch in the repository.
+
+3. **Isolate the problem** – create a [reduced test
+   case](http://css-tricks.com/reduced-test-cases/) and a live example.
+
+A good bug report shouldn't leave others needing to chase you up for more
+information. Please try to be as detailed as possible in your report. What is
+your environment? What steps will reproduce the issue? What tool(s) or OS will
+experience the problem? What would you expect to be the outcome? All these
+details will help people to fix any potential bugs.
+
+Example:
+
+> Short and descriptive example bug report title
+>
+> A summary of the issue and the OS environment in which it occurs. If
+> suitable, include the steps required to reproduce the bug.
+>
+> 1. This is the first step
+> 2. This is the second step
+> 3. Further steps, etc.
+>
+> `<url>` - a link to the reduced test case, if possible. Feel free to use a [Gist](https://gist.github.com).
+>
+> Any other information you want to share that is relevant to the issue being
+> reported. This might include the lines of code that you have identified as
+> causing the bug, and potential solutions (and your opinions on their
+> merits).
+
+## Feature requests
+
+Feature requests are welcome. But take a moment to find out whether your idea
+fits with the scope and aims of the project. It's up to *you* to make a strong
+case to convince the project's developers of the merits of this feature. Please
+provide as much detail and context as possible.
+
+## So, you want to contribute a new parser
+
+That's awesome! Please do take care to add example files that fit your parser use case.
+Make sure that the file you are adding is licensed for use within an MIT-licensed piece
+of software. Ideally, this file is going to be something you have produced yourself
+and you are permitted to share under the MIT license provisions.
+
+When writing a parser, please try to ensure it returns a usable result as soon as possible,
+or no result as soon as possible (once you know the file is not fit for your specific parser).
+Bear in mind that we enforce read budgets per-parser, so you will not be allowed to perform
+too many reads, or perform reads which are too large.
+
+## Pull requests
+
+Good pull requests-patches, improvements, new features-are a fantastic
+help. They should remain focused in scope and avoid containing unrelated
+commits.
+
+**Please ask first** before embarking on any significant pull request (e.g.
+implementing features, refactoring code, porting to a different language),
+otherwise you risk spending a lot of time working on something that the
+project's developers might not want to merge into the project.
+
+Please adhere to the coding conventions used throughout the project (indentation,
+accurate comments, etc.) and any other requirements (such as test coverage).
+
+The test suite can be run with `bundle exec rspec`. 
+
+Follow this process if you'd like your work considered for inclusion in the
+project:
+
+1. [Fork](http://help.github.com/fork-a-repo/) the project, clone your fork,
+   and configure the remotes:
+
+   ```bash
+   # Clone your fork of the repo into the current directory
+   git clone git@github.com:WeTransfer/format_parser.git
+   # Navigate to the newly cloned directory
+   cd format_parser
+   # Assign the original repo to a remote called "upstream"
+   git remote add upstream git@github.com:WeTransfer/format_parser.git
+   ```
+
+2. If you cloned a while ago, get the latest changes from upstream:
+
+   ```bash
+   git checkout <dev-branch>
+   git pull upstream <dev-branch>
+   ```
+
+3. Create a new topic branch (off the main project development branch) to
+   contain your feature, change, or fix:
+
+   ```bash
+   git checkout -b <topic-branch-name>
+   ```
+
+4. Commit your changes in logical chunks and/or squash them for readability and
+   conciseness. Check out [this post](https://chris.beams.io/posts/git-commit/) or
+   [this other post](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html) for some tips re: writing good commit messages.
+
+5. Locally merge (or rebase) the upstream development branch into your topic branch:
+
+   ```bash
+   git pull [--rebase] upstream <dev-branch>
+   ```
+
+6. Push your topic branch up to your fork:
+
+   ```bash
+   git push origin <topic-branch-name>
+   ```
+
+7. [Open a Pull Request](https://help.github.com/articles/using-pull-requests/)
+    with a clear title and description.
+
+**IMPORTANT**: By submitting a patch, you agree to allow the project owner to
+license your work under the same license as that used by the project, which you
+can see by clicking [here](https://github.com/WeTransfer/format_parser/blob/master/LICENSE.txt).
+This provision also applies to the test files you include with the changed code as fixtures.

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Gem dependencies specified in the gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2017 WeTransfer
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,53 @@
+# format_parser
+
+is a Ruby library for prying open video, image, document, and audio files.
+It includes a number of parser modules that try to recover metadata useful for post-processing and layout while reading the absolute
+minimum amount of data possible.
+
+`format_parser` is inspired by [imagesize,](https://rubygems.org/gem/imagesize) [fastimage](https://github.com/sdsykes/fastimage)
+and [dimensions,](https://github.com/sstephenson/dimensions) borrowing from them where appropriate.
+
+## Basic usage
+
+Pass an IO object that responds to `read` and `seek` to `FormatParser`.
+
+```ruby
+file_info = FormatParser.parse(File.open("myimage.jpg", "rb"))
+file_info.file_nature           #=> :image
+file_info.file_format           #=> :JPG
+file_info.width_px              #=> 320
+file_info.height_px             #=> 240
+file_info.orientation           #=> :top_left
+```
+If nothing is detected, the result will be `nil`.
+
+## Design rationale
+
+We need to recover metadata from various file types, and we need to do so satisfying the following constraints:
+
+* The data in those files can be malicious and/or incomplete, so we need to be failsafe
+* The data will be fetched from a remote location, so we want to acquire it with as few HTTP requests as possible
+  and with fetches being sufficiently small - the number of HTTP requests being of greater concern due to the
+  fact that we rely on AWS, and data transfer is much cheaper than per-request fees.
+* The data can be recognized ambiguously and match more than one format definition (like TIFF sections of camera RAW)
+* The number of supported formats is only ever going to increase, not decrease
+* The library is likely to be used in multiple consumer applications
+* The information necessary is a small subset of the overall metadata available in the file
+
+Therefore we adapt the following approaches:
+
+* Modular parsers per file format, with some degree of code sharing between them (but not too much). Adding new formats
+  should be low-friction, and testing these format parsers should be possible in isolation
+* Modular and configurable IO stack that supports limiting reads/loops from the source entity.
+  The IO stack is isolated from the parsers, meaning parsers do not need to care about things
+  like fetches using `Range:` headers, GZIP compression and the like
+* A caching system that allows us to ideally fetch once, and only once, and as little as possible - but still accomodate formats
+  that have the important information at the end of the file or might need information from the middle of the file
+* Minimal dependencies, and if dependencies are to be used they should be very stable and low-level
+* Where possible, use small subsets of full-feature format parsers since we only care about a small subset of the data
+* Avoid using C libraries which are likely to contain buffer overflows/underflows - we stay memory safe
+
+## Fixture Sources
+
+- MIT licensed fixture files from the FastImage and Dimensions projects
+- fixture.aiff was created by one of the project maintainers and is MIT licensed

data/Rakefile

@@ -0,0 +1,12 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+require 'yard'
+
+YARD::Rake::YardocTask.new(:doc) do |t|
+  # The dash has to be between the two to "divide" the source files and
+  # miscellaneous documentation files that contain no code
+  t.files = ['lib/**/*.rb', '-', 'LICENSE.txt', 'IMPLEMENTATION_DETAILS.md']
+end
+
+RSpec::Core::RakeTask.new(:spec)
+task default: :spec

data/format_parser.gemspec

@@ -0,0 +1,43 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'format_parser/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "format_parser"
+  spec.version       = FormatParser::VERSION
+  spec.authors       = ['Noah Berman', 'Julik Tarkhanov']
+  spec.email         = ['noah@noahberman.org', 'me@julik.nl']
+  spec.licenses      = ['MIT']
+  spec.summary       = "A library for efficient parsing of file metadata"
+  spec.description   = "A Ruby library for prying open files you can convert to a previewable format, such as video, image and audio files. It includes
+  a number of parser modules that try to recover metadata useful for post-processing and layout while reading the absolute
+  minimum amount of data possible."
+  spec.homepage      = "https://github.com/WeTransfer/format_parser"
+  spec.license       = "MIT"
+
+  # Prevent pushing this gem to RubyGems.org. To allow pushes either set the 'allowed_push_host'
+  # to allow pushing to a single host or delete this section to allow pushing to any host.
+  if spec.respond_to?(:metadata)
+    spec.metadata['allowed_push_host'] = "https://rubygems.org"
+  else
+    raise "RubyGems 2.0 or newer is required to protect against public gem pushes."
+  end
+
+  spec.files = `git ls-files -z`.split("\x0").reject do |f|
+    # Make sure large fixture files are not packaged with the gem every time
+    f.match(%r{^spec/fixtures/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'exifr', '~> 1.0'
+  spec.add_dependency 'faraday', '~> 0.13'
+  
+  spec.add_development_dependency 'rspec', '~> 3.0'
+  spec.add_development_dependency 'rake', '~> 12'
+  spec.add_development_dependency 'simplecov', '~> 0.15'
+  spec.add_development_dependency 'pry', '~> 0.11'
+  spec.add_development_dependency 'yard', '~> 0.9'
+end

data/lib/care.rb

@@ -0,0 +1,123 @@
+# Care (Caching Reader) makes it more efficient to feed a
+# possibly remote IO to parsers that tend to read (and skip)
+# in very small increments. This way, with a remote source that
+# is only available via HTTP, for example, we can have less
+# fetches and have them return more data for one fetch
+class Care
+  DEFAULT_PAGE_SIZE = 16 * 1024
+
+  class IOWrapper
+    def initialize(io, cache=Cache.new(DEFAULT_PAGE_SIZE))
+      @io, @cache = io, cache
+      @pos = 0
+    end
+
+    def seek(to)
+      @pos = to
+    end
+
+    def read(n_bytes)
+      read = @cache.byteslice(@io, @pos, n_bytes)
+      return nil unless read && !read.empty?
+      @pos += read.bytesize
+      read
+    end
+
+    def clear
+      @cache.clear
+    end
+
+    def close
+      clear
+      @io.close if @io.respond_to?(:close)
+    end
+  end
+
+  # Stores cached pages of data from the given IO as strings.
+  # Pages are sized to be `page_size` or less (for the last page).
+  class Cache
+    def initialize(page_size = DEFAULT_PAGE_SIZE)
+      @page_size = page_size.to_i
+      raise ArgumentError, "The page size must be a positive Integer" unless @page_size > 0
+      @pages = {}
+      @lowest_known_empty_page = nil
+    end
+
+    # Returns the maximum possible byte string that can be
+    # recovered from the given `io` at the given offset.
+    # If the IO has been exhausted, `nil` will be returned
+    # instead. Will use the cached pages where available,
+    # or fetch pages where necessary
+    def byteslice(io, at, n_bytes)
+      if n_bytes < 1
+        raise ArgumentError, "The number of bytes to fetch must be a positive Integer" 
+      end
+
+      first_page = at / @page_size
+      last_page = (at + n_bytes) / @page_size
+
+      relevant_pages = (first_page..last_page).map{|i| hydrate_page(io, i) }
+
+      # Create one string combining all the pages which are relevant for
+      # us - it is much easier to address that string instead of piecing
+      # the output together page by page, and joining arrays of strings
+      # is supposed to be optimized.
+      slab = if relevant_pages.length > 1
+        # If our read overlaps multiple pages, we do have to join them, this is
+        # the general case
+        relevant_pages.join
+      else # We only have one page
+        # Optimize a little. If we only have one page that we need to read from
+        # - which is likely going to be the case *often* we can avoid allocating
+        # a new string for the joined pages and juse use the only page
+        # directly as the slab. Since it might contain a `nil` and we do
+        # not join (which casts nils to strings) we take care of that too
+        relevant_pages.first || ''
+      end
+
+      offset_in_slab = at % @page_size
+      slice = slab.byteslice(offset_in_slab, n_bytes)
+
+      # Returning an empty string from read() is very confusing for the caller,
+      # and no builtins do this - if we are at EOF we should return nil
+      if slice && !slice.empty?
+        slice
+      else
+        nil
+      end
+    end
+
+    def clear
+      @pages.clear
+    end
+
+    def hydrate_page(io, page_i)
+      # Avoid trying to read the page if we know there is no content to fill it
+      # in the underlying IO
+      if @lowest_known_empty_page && page_i >= @lowest_known_empty_page
+        return nil
+      end
+
+      @pages[page_i] ||= read_page(io, page_i)
+    end
+
+    def read_page(io, page_i)
+      io.seek(page_i * @page_size)
+      read_result = io.read(@page_size)
+
+      if read_result.nil?
+        # If the read went past the end of the IO the read result will be nil,
+        # so we know our IO is exhausted here
+        if @lowest_known_empty_page.nil? || @lowest_known_empty_page > page_i
+          @lowest_known_empty_page = page_i
+        end
+      elsif read_result.bytesize < @page_size
+        # If we read less than we initially wanted we know there are no pages
+        # to read following this one, so we can also optimize
+        @lowest_known_empty_page = page_i + 1
+      end
+        
+      read_result
+    end
+  end
+end