mime-types 2.99.3 → 3.3.1

data/{Licence.rdoc → Licence.md}

@@ -1,18 +1,10 @@
-== Licence
+## Licence
 
-This software is available under three licences: the GNU GPL version 2 (or at
-your option, a later version), the Perl Artistic License, or the MIT License.
-The licence preferred by me is the MIT licence.
+*   Copyright 2003–2019 Austin Ziegler and contributors.
 
-The original Perl MIME::Types was dually originally licensed with the Perl
-Artistic Licence and the GNU GPL ("the same terms as Perl itself") and given
-that the Ruby implementation originally hewed pretty closely to the Perl
-version, I chose to maintain the additional licensing terms.
+The software in this repository is made available under the MIT license.
 
-* Copyright 2003–2016 Austin Ziegler and other contributors.
-* Adapted from MIME::Types (Perl) by Mark Overmeer.
-
-=== MIT License
+### MIT License
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in
@@ -31,9 +23,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
-
-=== Perl Artistic License (version 2)
-See the file docs/artistic.txt in the main distribution.
-
-=== GNU GPL version 2
-See the file docs/COPYING.txt in the main distribution.

data/Manifest.txt

@@ -1,40 +1,23 @@
-Code-of-Conduct.rdoc
-Contributing.rdoc
-History-Types.rdoc
-History.rdoc
-Licence.rdoc
+Code-of-Conduct.md
+Contributing.md
+History.md
+Licence.md
 Manifest.txt
 README.rdoc
 Rakefile
-data/mime-types.json
-data/mime.content_type.column
-data/mime.docs.column
-data/mime.encoding.column
-data/mime.friendly.column
-data/mime.obsolete.column
-data/mime.registered.column
-data/mime.signature.column
-data/mime.use_instead.column
-data/mime.xrefs.column
-docs/COPYING.txt
-docs/artistic.txt
 lib/mime-types.rb
 lib/mime/type.rb
 lib/mime/type/columnar.rb
 lib/mime/types.rb
+lib/mime/types/_columnar.rb
 lib/mime/types/cache.rb
 lib/mime/types/columnar.rb
+lib/mime/types/container.rb
 lib/mime/types/deprecations.rb
+lib/mime/types/full.rb
 lib/mime/types/loader.rb
-lib/mime/types/loader_path.rb
 lib/mime/types/logger.rb
-support/apache_mime_types.rb
-support/benchmarks/load.rb
-support/benchmarks/load_allocations.rb
-support/benchmarks/object_counts.rb
-support/convert.rb
-support/convert/columnar.rb
-support/iana_registry.rb
+lib/mime/types/registry.rb
 test/bad-fixtures/malformed
 test/fixture/json.json
 test/fixture/old-data

data/README.rdoc

@@ -1,11 +1,12 @@
-= mime-types
+= mime-types for Ruby
 
 home :: https://github.com/mime-types/ruby-mime-types/
 code :: https://github.com/mime-types/ruby-mime-types/
 bugs :: https://github.com/mime-types/ruby-mime-types/issues
 rdoc :: http://rdoc.info/gems/mime-types/
-continuous integration :: {<img src="https://travis-ci.org/mime-types/ruby-mime-types.png" />}[https://travis-ci.org/mime-types/ruby-mime-types]
-test coverage :: {<img src="https://coveralls.io/repos/mime-types/ruby-mime-types/badge.png" alt="Coverage Status" />}[https://coveralls.io/r/mime-types/ruby-mime-types]
+clog :: https://github.com/mime-types/ruby-mime-types/blob/master/History.md
+continuous integration :: {<img src="https://travis-ci.org/mime-types/ruby-mime-types.svg?branch=master" alt="Build Status" />}[https://travis-ci.org/mime-types/ruby-mime-types]
+test coverage :: {<img src="https://coveralls.io/repos/mime-types/ruby-mime-types/badge.svg?branch=master&service=github" alt="Coverage Status" />}[https://coveralls.io/github/mime-types/ruby-mime-types?branch=master]
 
 == Description
 
@@ -14,19 +15,13 @@ MIME content type definitions. It can be used to determine defined filename
 extensions for MIME types, or to use filename extensions to look up the likely
 MIME type definitions.
 
-This is release 2.99.1, the first scheduled data update for mime-types 2.x. As
-of mime-types 2.99. deprecation warnings are noisy and data that has been
-deprecated is now no longer available. The data is both dropped from the data
-files and is stubbed out as empty or +nil+ values as appropriate.
-
-mime-types-2.6 was the last version of mime-types 2.x with newly available
-features, and mime-types 2.99 will only receive quarterly updates to the IANA
-registered MIME media types plus any security updates that may be required.
-
-If the loss of the deprecated data matters, be sure to set your dependency
-appropriately:
-
-   gem 'mime-types', '~> 2.6, < 2.99'
+Version 3.0 is a major release that requires Ruby 2.0 compatibility and removes
+deprecated functions. The columnar registry format introduced in 2.6 has been
+made the primary format; the registry data has been extracted from this library
+and put into {mime-types-data}[https://github.com/mime-types/mime-types-data].
+Additionally, mime-types is now licensed exclusively under the MIT licence and
+there is a code of conduct in effect. There are a number of other smaller
+changes described in the History file.
 
 === About MIME Media Types
 
@@ -41,17 +36,6 @@ recommendations, the {IANA Media Types
 registry}[https://www.iana.org/assignments/media-types/media-types.xhtml], and
 user contributions. It conforms to RFCs 2045 and 2231.
 
-=== mime-types 1.x End of Life
-
-mime-types 1.x has reached end of life and is no longer supported as of
-2015-10-27 and will receive no updates at all.
-
-=== mime-types 2.x End of Life
-
-mime-types 2.x is supported as 2.99.x receiving quarterly updates of the IANA
-registry and security updates for two years. It will reach full end of life on
-2017-11-21.
-
 === mime-types 3.x
 
 Users are encouraged to upgrade to mime-types 3.x as soon as is practical.
@@ -90,8 +74,8 @@ files). A MIME::Type stores the known information about one MIME type.
    puts text == 'text/x-plain'     # => false
    puts 'text/x-plain' == text     # => false
 
-   puts MIME::Type.simplified('x-appl/x-zip') # => 'appl/zip'
-   puts MIME::Type.i18n_key('x-appl/x-zip') # => 'appl.zip'
+   puts MIME::Type.simplified('x-appl/x-zip') # => 'x-appl/x-zip'
+   puts MIME::Type.i18n_key('x-appl/x-zip') # => 'x-appl.x-zip'
 
    puts text.like?('text/x-plain') # => true
    puts text.like?(MIME::Type.new('x-text/x-plain')) # => true
@@ -118,54 +102,61 @@ files). A MIME::Type stores the known information about one MIME type.
 
    xwingz = MIME::Types['application/x-Wingz'].first # => application/x-Wingz
    puts xwingz.content_type        # => 'application/x-Wingz'
-   puts xwingz.simplified          # => 'application/wingz'
+   puts xwingz.simplified          # => 'application/x-wingz'
 
 === Columnar Store
 
-mime-types 2.6 introduced an experimental columnar storage format that reduces
-the default memory footprint. It does this by selectively loading data. When a
-registry is first loaded from a columnar store, only the canonical MIME type
-and registered extensions will be loaded and the MIME type will be connected to
-its registry. When extended data is required (including +registered+,
-+obsolete+, and +use_instead+), that data is loaded from its own column file
-for all types in the registry. This load is done with a Mutex to ensure that
-the types are updated safely in a multithreaded environment.
+mime-types uses as its primary registry storage format a columnar storage
+format reducing the default memory footprint. This is done by selectively
+loading the data on a per-attribute basis. When the registry is first loaded
+from the columnar store, only the canonical MIME content type and known
+extensions and the MIME type will be connected to its loading registry. When
+other data about the type is required (including +preferred_extension+,
+<tt>obsolete?</tt>, and <tt>registered?</tt>) that data is loaded from its own
+column file for all types in the registry.
 
-Columnar storage is slated to become the default storage format for mime-types
-3.0, but until that is released, the default is still to use the JSON storage
-format. As such, columnar storage can only currently be loaded at an
-application level with the following specification in the application Gemfile:
+The load of any column data is performed with a Mutex to ensure that types are
+updated safely in a multithreaded environment. Benchmarks show that while
+columnar data loading is slower than the JSON store, it cuts the memory use by
+a third over the JSON store.
 
-   gem 'mime-types', require: 'mime/types/columnar'
+If you prefer to load all the data at once, this can be specified in your
+application Gemfile as:
 
-Projects that do not use Bundler, and libraries that wish to suggest this
-behaviour to applications are encouraged to require this directly, but only if
-you specify a dependency on mime-types 2.6.
+   gem 'mime-types', require: 'mime/types/full'
 
-   require 'mime/types/columnar'
+Projects that do not use Bundler should +require+ the same:
+
+   require 'mime/types/full'
 
-Although this require will not be necessary after mime-types 3, it will work
-through at least {version
+Libraries that use mime-types are discouraged from choosing the JSON store.
+
+For applications and clients that used mime-types 2.6 when the columnar store
+was introduced, the require used previously will still work through at least
+{version
 4}[https://github.com/mime-types/ruby-mime-types/pull/96#issuecomment-100725400]
-and possibly beyond.
+and possibly beyond; it is effectively an empty operation. You are recommended
+to change your Gemfile as soon as is practical.
+
+   require 'mime/types/columnar'
 
-Note that the new Columnar class (MIME::Type::Columnar) and module
-(MIME::Types::Columnar) are considered private variant implementations of
-MIME::Type and MIME::Types and the specific implementation should not be relied
-upon by consumers of the mime-types library. Instead, depend on the public
-implementations only.
+Note that MIME::Type::Columnar and MIME::Types::Columnar are considered private
+variant implementations of MIME::Type and MIME::Types and the specific
+implementation should not be relied upon by consumers of the mime-types
+library. Instead, depend on the public implementations (MIME::Type and
+MIME::Types) only.
 
 === Cached Storage
 
-Since version 2.0, mime-types has supported a cache of MIME types based on
-<tt>Marshal.dump</tt>. The cache is invalidated for each released version of
-mime-types so that version 2.5 is not reused for version 2.6. If the
+mime-types supports a cache of MIME types using <tt>Marshal.dump</tt>. The
+cache is invalidated for each version of the mime-types-data gem so that data
+version 3.2015.1201 will not be reused with data version 3.2016.0101. If the
 environment variable +RUBY_MIME_TYPES_CACHE+ is set to a cache file, mime-types
 will attempt to load the MIME type registry from the cache file. If it cannot,
 it will load the types normally and then saves the registry to the cache file.
 
-The current mime-types cache is not compatible with the columnar storage
-format. This will be resolved for mime-types 3.
+The caching works with both full stores and columnar stores. Only the data that
+has been loaded prior to saving the cache will be stored.
 
 == mime-types Modified Semantic Versioning
 
@@ -183,22 +174,20 @@ MAJOR.MINOR:
    incremented, MINOR will be set to zero, and PATCH will be reset to the
    implied zero.
 
-2. If an API (code) feature is added that does not break compatibilty OR if
-   there are MIME types added, removed, or changed in the registry, the MINOR
-   version will be incremented and PATCH will be reset to the implied zero.
+2. If an API (code) feature is added that does not break compatibility, the
+   MINOR version will be incremented and PATCH will be reset to the implied zero.
 
 3. If there is a bugfix to a feature added in the most recent MAJOR.MINOR
-   release, OR if purely typographical errors are fixed in MIME types, the
-   implied PATCH value will be incremented resulting in MAJOR.MINOR.PATCH.
+   release, the implied PATCH value will be incremented resulting in
+   MAJOR.MINOR.PATCH.
 
-In practical terms, there should be a MINOR release roughly monthly to track
-updated or changed MIME types from the official IANA registry. This does not
-indicate when new API features have been added, but all minor versions of
-mime-types 2.x will be backwards compatible; the interfaces marked deprecated
-will be removed in mime-types 3.x.
+In practical terms, there will be fewer releases of mime-types focussing on
+features because of the existence of the [mime-types-data][] gem, and if
+features are marked deprecated in the course of mime-types 3.x, they will not
+be removed until mime-types 4.x or possibly later.
 
-:include: Code-of-Conduct.rdoc
+{Code of Conduct}[Code-of-Conduct_md.html]
 
-:include: Contributing.rdoc
+{Contributing}[Contributing_md.html]
 
-:include: Licence.rdoc
+{Licence}[Licence_md.html]

data/Rakefile

@@ -15,13 +15,14 @@ spec = Hoe.spec 'mime-types' do
   developer('Austin Ziegler', 'halostatue@gmail.com')
   self.need_tar = true
 
-  require_ruby_version '>= 1.9.2'
+  require_ruby_version '>= 2.0'
 
-  self.history_file = 'History.rdoc'
+  self.history_file = 'History.md'
   self.readme_file = 'README.rdoc'
-  self.extra_rdoc_files = FileList['*.rdoc'].to_a
 
-  self.licenses = ['MIT', 'Artistic-2.0', 'GPL-2.0']
+  license 'MIT'
+
+  extra_deps << ['mime-types-data', '~> 3.2015']
 
   extra_dev_deps << ['hoe-doofus', '~> 1.0']
   extra_dev_deps << ['hoe-gemspec2', '~> 1.1']
@@ -31,28 +32,29 @@ spec = Hoe.spec 'mime-types' do
   extra_dev_deps << ['minitest', '~> 5.4']
   extra_dev_deps << ['minitest-autotest', '~> 1.0']
   extra_dev_deps << ['minitest-focus', '~> 1.0']
-  extra_dev_deps << ['rake', '~> 10.0']
-  extra_dev_deps << ['fivemat', '~> 1.3' ]
+  extra_dev_deps << ['minitest-bonus-assertions', '~> 3.0']
+  extra_dev_deps << ['minitest-hooks', '~> 1.4']
+  extra_dev_deps << ['rake', '>= 10.0', '< 13.0']
+  extra_dev_deps << ['fivemat', '~> 1.3']
   extra_dev_deps << ['minitest-rg', '~> 5.2']
-end
-
-task :support do
-  %w(lib support).each { |path|
-    $LOAD_PATH.unshift(File.join(Rake.application.original_dir, path))
-  }
-end
 
-task 'support:nokogiri' => :support do
-  begin
-    gem 'nokogiri'
-  rescue Gem::LoadError
-    raise 'Nokogiri is not installed. Please install it.'
+  if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('2.0')
+    extra_dev_deps << ['simplecov', '~> 0.7']
+    # if ENV['CI'] or ENV['TRAVIS']
+    #   extra_dev_deps << ['coveralls', '~> 0.8']
+    # end
   end
 end
 
 namespace :benchmark do
+  task :support do
+    %w(lib support).each { |path|
+      $LOAD_PATH.unshift(File.join(Rake.application.original_dir, path))
+    }
+  end
+
   desc 'Benchmark Load Times'
-  task :load, [ :repeats ] => :support do |_, args|
+  task :load, [:repeats] => 'benchmark:support' do |_, args|
     require 'benchmarks/load'
     Benchmarks::Load.report(
       File.join(Rake.application.original_dir, 'lib'),
@@ -61,7 +63,7 @@ namespace :benchmark do
   end
 
   desc 'Allocation counts'
-  task :allocations, [ :top_x, :mime_types_only ] => :support do |_, args|
+  task :allocations, [:top_x, :mime_types_only] => 'benchmark:support' do |_, args|
     require 'benchmarks/load_allocations'
     Benchmarks::LoadAllocations.report(
       top_x: args.top_x,
@@ -70,7 +72,7 @@ namespace :benchmark do
   end
 
   desc 'Columnar allocation counts'
-  task 'allocations:columnar', [ :top_x, :mime_types_only ] => :support do |_, args|
+  task 'allocations:columnar', [:top_x, :mime_types_only] => 'benchmark:support' do |_, args|
     require 'benchmarks/load_allocations'
     Benchmarks::LoadAllocations.report(
       columnar: true,
@@ -79,30 +81,132 @@ namespace :benchmark do
     )
   end
 
+  desc 'Columnar allocation counts (full load)'
+  task 'allocations:columnar:full', [:top_x, :mime_types_only] => 'benchmark:support' do |_, args|
+    require 'benchmarks/load_allocations'
+    Benchmarks::LoadAllocations.report(
+      columnar: true,
+      top_x: args.top_x,
+      mime_types_only: args.mime_types_only,
+      full: true
+    )
+  end
+
+  desc 'Memory profiler'
+  task :memory, [:top_x, :mime_types_only] => 'benchmark:support' do |_, args|
+    require 'benchmarks/memory_profiler'
+    Benchmarks::ProfileMemory.report(
+      mime_types_only: args.mime_types_only,
+      top_x: args.top_x
+    )
+  end
+
+  desc 'Columnar memory profiler'
+  task 'memory:columnar', [:top_x, :mime_types_only] => 'benchmark:support' do |_, args|
+    require 'benchmarks/memory_profiler'
+    Benchmarks::ProfileMemory.report(
+      columnar: true,
+      mime_types_only: args.mime_types_only,
+      top_x: args.top_x
+    )
+  end
+
+  desc 'Columnar allocation counts (full load)'
+  task 'memory:columnar:full', [:top_x, :mime_types_only] => 'benchmark:support' do |_, args|
+    require 'benchmarks/memory_profiler'
+    Benchmarks::ProfileMemory.report(
+      columnar: true,
+      full: true,
+      top_x: args.top_x,
+      mime_types_only: args.mime_types_only
+    )
+  end
+
   desc 'Object counts'
-  task objects: :support do
+  task objects: 'benchmark:support' do
     require 'benchmarks/object_counts'
     Benchmarks::ObjectCounts.report
   end
 
   desc 'Columnar object counts'
-  task 'objects:columnar' => :support do
+  task 'objects:columnar' => 'benchmark:support' do
     require 'benchmarks/object_counts'
     Benchmarks::ObjectCounts.report(columnar: true)
   end
+
+  desc 'Columnar object counts (full load)'
+  task 'objects:columnar:full' => 'benchmark:support' do
+    require 'benchmarks/object_counts'
+    Benchmarks::ObjectCounts.report(columnar: true, full: true)
+  end
 end
 
-namespace :mime do
-  desc 'Download the current MIME type registrations from IANA.'
-  task :iana, [ :destination ] => 'support:nokogiri' do |_, args|
-    require 'iana_registry'
-    IANARegistry.download(to: args.destination)
+namespace :profile do
+  directory 'tmp/profile'
+
+  CLEAN.add 'tmp'
+
+  def ruby_prof(script)
+    require 'pathname'
+    output = Pathname('tmp/profile').join(script)
+    output.mkpath
+    script = Pathname('support/profile').join("#{script}.rb")
+
+    args = [
+      '-W0',
+      '-Ilib',
+      '-S', 'ruby-prof',
+      '-R', 'mime/types',
+      '-s', 'self',
+      '-p', 'multi',
+      '-f', output.to_s,
+      script.to_s
+    ]
+    ruby args.join(' ')
   end
 
-  desc 'Download the current MIME type configuration from Apache.'
-  task :apache, [ :destination ] => 'support:nokogiri' do |_, args|
-    require 'apache_mime_types'
-    ApacheMIMETypes.download(to: args.destination)
+  task full: 'tmp/profile' do
+    ruby_prof 'full'
+  end
+
+  task columnar: :support do
+    ruby_prof 'columnar'
+  end
+
+  task 'columnar:full' => :support do
+    ruby_prof 'columnar_full'
+  end
+end
+
+if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('2.0')
+  namespace :test do
+    # Coveralls needs to be disabled for now because it transitively depends on
+    # an earlier version of mime-types.
+    # if ENV['CI'] or ENV['TRAVIS']
+    #   task :coveralls do
+    #     spec.test_prelude = [
+    #       'require "psych"',
+    #       'require "simplecov"',
+    #       'require "coveralls"',
+    #       'SimpleCov.formatter = Coveralls::SimpleCov::Formatter',
+    #       'SimpleCov.start("test_frameworks") { command_name "Minitest" }',
+    #       'gem "minitest"'
+    #     ].join('; ')
+    #     Rake::Task['test'].execute
+    #   end
+
+    #   Rake::Task['travis'].prerequisites.replace(%w(test:coveralls))
+    # end
+
+    desc 'Run test coverage'
+    task :coverage do
+      spec.test_prelude = [
+        'require "simplecov"',
+        'SimpleCov.start("test_frameworks") { command_name "Minitest" }',
+        'gem "minitest"'
+      ].join('; ')
+      Rake::Task['test'].execute
+    end
   end
 end
 
@@ -114,11 +218,11 @@ namespace :convert do
       @doc_converter ||= RDoc::Markup::ToMarkdown.new
     end
 
-    %w(README History History-Types).each do |name|
-      rdoc = "#{name}.rdoc"
-      mark = "#{name}.md"
+    FileList['*.rdoc'].each do |name|
+      rdoc = name
+      mark = "#{File.basename(name, '.rdoc')}.md"
 
-      file mark => [ rdoc, :setup ] do |t|
+      file mark => [rdoc, :setup] do |t|
         puts "#{rdoc} => #{mark}"
         File.open(t.name, 'wb') { |target|
           target.write @doc_converter.convert(IO.read(t.prerequisites.first))
@@ -127,41 +231,54 @@ namespace :convert do
 
       CLEAN.add mark
 
-      task run: [ mark ]
+      task run: [mark]
     end
   end
 
   desc 'Convert documentation from RDoc to Markdown'
   task docs: 'convert:docs:run'
+end
 
-  namespace :yaml do
-    desc 'Convert from YAML to JSON'
-    task :json, [ :source, :destination, :multiple_files ] => :support do |_, args|
-      require 'convert'
-      Convert.from_yaml_to_json(args)
-    end
+task 'deps:top', [:number] do |_, args|
+  require 'net/http'
+  require 'json'
 
-    desc 'Convert from YAML to Columnar'
-    task :columnar, [ :source, :destination ] => :support do |_, args|
-      require 'convert/columnar'
-      Convert::Columnar.from_yaml_to_columnar(args)
+  def rubygems_get(gem_name: '', endpoint: '')
+    path = File.join('/api/v1/gems/', gem_name, endpoint).chomp('/') + '.json'
+    Net::HTTP.start('rubygems.org', use_ssl: true) do |http|
+      JSON.parse(http.get(path).body)
     end
   end
 
-  namespace :json do
-    desc 'Convert from JSON to YAML'
-    task :yaml, [ :source, :destination, :multiple_files ] => :support do |_, args|
-      require 'convert'
-      Convert.from_json_to_yaml(args)
+  results = rubygems_get(
+    gem_name: 'mime-types',
+    endpoint: 'reverse_dependencies'
+  )
+
+  weighted_results = {}
+  results.each do |name|
+    begin
+      weighted_results[name] = rubygems_get(gem_name: name)['downloads']
+    rescue => e
+      puts "#{name} #{e.message}"
     end
   end
-end
 
-desc 'Default conversion from YAML to JSON and Columnar'
-task convert: [ 'convert:yaml:json', 'convert:yaml:columnar' ]
+  weighted_results.sort { |(_k1, v1), (_k2, v2)|
+    v2 <=> v1
+  }.first(args.number || 50).each_with_index do |(k, v), i|
+    puts "#{i}) #{k}: #{v}"
+  end
+end
 
-Rake::Task['gem'].prerequisites.unshift('convert')
-Rake::Task['gem'].prerequisites.unshift('git:manifest')
-Rake::Task['gem'].prerequisites.unshift('gemspec')
+task :console do
+  arguments = %w(pry)
+  arguments.push(*spec.spec.require_paths.map { |dir| "-I#{dir}" })
+  arguments.push("-r#{spec.spec.name.gsub('-', File::SEPARATOR)}")
+  unless system(*arguments)
+    error "Command failed: #{show_command}"
+    abort
+  end
+end
 
 # vim: syntax=ruby