suvii 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 5bf394d2a7c3da8466bf22f4d738f02034174fa7
+  data.tar.gz: 7664c6d478ba6b2f97de9961eb3d68a5b610d468
+SHA512:
+  metadata.gz: ecdf92f1be591c5e8b934376c0b3825747657fb58de474cad4e6250f3c3bb101ba2a80fa1f65cb2ac61b78d30042abd6d0b8cfcc11610f0d08053261ba5cbe3f
+  data.tar.gz: bba9d298fd9513fdcb5502b14abbdcc9284c0cb286a0bba201f471a359d1f4a66ec6d5ab13ef85f3c2bda748f0b4b3990407a44c266e3f9b3f0ab0ed8ee8af54

data/.gitignore

@@ -0,0 +1,14 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/.travis.yml

@@ -0,0 +1,10 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.0.0
+  - 2.1
+  - 2.2
+  - 2.3.0
+  - jruby-19mode
+  - rbx-2
+bundler_args: --without doc debug

data/.yardopts

@@ -0,0 +1 @@
+--no-private

data/CHANGELOG.md

@@ -0,0 +1 @@
+## Unreleased

data/CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing
+
+## Adding a feature
+
+1. Open an issue and explain what you're planning to do. It is better to discuss new idea first,
+   rather when diving into code.
+2. Add some tests.
+3. Write the code.
+4. Make sure all tests pass.
+5. Commit with detailed explanation what you've done in a message.
+6. Open pull request.
+
+## Breaking/removing a feature
+
+1. Add deprecation warning and fallback to old behaivour if possible.
+2. Explain how to migrate to the new code in CHANGELOG.
+3. Update/remove tests.
+4. Update the code.
+5. Make sure all tests pass.
+6. Commit with detailed explanation what you've done in a message.
+7. Open pull request.
+
+## Fixing a bug
+
+1. Add failing test.
+2. Fix the bug.
+3. Make sure all tests pass.
+4. Commit with detailed explanation what you've done in a message.
+5. Open pull request.
+
+## Fixing a typo
+
+1. Commit with a message that include "[ci skip]" remark.
+2. Open pull request.
+
+## Running the tests
+
+```
+rake test
+```
+
+## Working with documentation
+
+```
+yard server -dr
+open http://localhost:8808
+```

data/Gemfile

@@ -0,0 +1,6 @@
+source 'https://rubygems.org'
+
+gemspec
+
+gem "pry", :group => :debug
+gem "yard", "~> 0.8", :group => :doc

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2016 Andrii Malyshko
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,51 @@
+# Suvii
+
+[![Build Status](https://img.shields.io/travis/torba-rb/suvii.svg)](https://travis-ci.org/torba-rb/suvii)
+[![Gem version](https://img.shields.io/gem/v/suvii.svg)](https://rubygems.org/gems/suvii)
+
+**Suvii** is a library that abstracts content reading from a remote tar.gz/zip archive. It fetches
+the archive and extracts it to a temp directory.
+
+## Name origin
+
+"Сувій" [[suʋii̯][suvii-pronounce]] in Ukrainian can mean a scroll, something which is rolled up or
+even a package.
+
+## Status
+
+Production ready.
+
+## Documentation
+
+[Released version](http://rubydoc.info/gems/suvii/0.1.0)
+
+
+## Installation
+
+Add this line to your application's Gemfile and run `bundle`:
+
+```ruby
+gem 'suvii'
+```
+
+## Usage
+
+```ruby
+temp_directory = Suvii.fetch("https://registry.npmjs.org/coffee-script/-/coffee-script-2.0.0.tgz")
+# do whatever you need with extracted content
+Dir.glob("#{temp_directory}/**/*").each do |entry|
+  # ...
+end
+```
+
+Note, that fetching same URL twice will unpack to another temp directory. Such design decision was
+based on assumption, that generally you don't need everything in an archive, or even have to modify
+something. This extra copy/modify step should be totally independent from extraction, and you shouldn't
+worry about cleanup, or that another process could mess things up.
+
+## Origin
+
+Extracted from [Torba][torba-github] library since it looks more like a standalone component.
+
+[suvii-pronounce]: https://commons.wikimedia.org/wiki/File:Uk-%D1%81%D1%83%D0%B2%D1%96%D0%B9.ogg
+[torba-github]: https://github.com/torba-rb/torba

data/Rakefile

@@ -0,0 +1,7 @@
+require "bundler/gem_tasks"
+
+task :default => :test
+task :test do
+  $LOAD_PATH.unshift("test")
+  Dir.glob("./test/**/*_test.rb").each { |file| require file}
+end

data/lib/suvii.rb

@@ -0,0 +1,40 @@
+require "tmpdir"
+require "logger"
+
+require "suvii/cache"
+require "suvii/http"
+require "suvii/extract"
+
+# @since 0.1.0
+module Suvii
+  # Downloads and extracts an archive to a temp directory.
+  #
+  # @param url [String] URL of an archive to be fetched.
+  # @option (see Cache.fetch)
+  # @option (see Http.save)
+  # @option (see Extract#initialize)
+  # @return [String] path to a temp directory with the archive being extracted.
+  def self.fetch(url, options = {})
+    path_to_archive = Cache.fetch(url, options) do |path|
+      Http.save(url, path, options)
+    end
+
+    extractor = Extract.class_for(path_to_archive).new(path_to_archive, options)
+    extractor.extract_to(Dir.mktmpdir)
+  end
+
+  # @return [Logger] instance of Logger compatible class.
+  def self.logger
+    @logger ||= Logger.new(STDOUT).tap do |logger|
+      logger.level = Logger::INFO
+      logger.formatter = proc { |_, _, _, msg| msg }
+    end
+  end
+
+  # Overrides default logger instance.
+  #
+  # @param logger [Logger] instance of Logger compatible class.
+  def self.logger=(logger)
+    @logger = logger
+  end
+end

data/lib/suvii/cache.rb

@@ -0,0 +1,24 @@
+require "cgi"
+require "tmpdir"
+
+module Suvii
+  class Cache
+    # Maps an archive URL to a path to its cached file.
+    # @note This method doesn't write to a disk. It is a responsibility of a block implementation.
+    #
+    # @param url [String] URL of an archive to be processed.
+    # @option options [String] :cache_path (random temporary directory) where the downloaded archive
+    #   should be stored. You can provide path to a persistent folder to prevent downloading same
+    #   files again.
+    # @yield a block if the archive was not previously cached.
+    # @yieldparam path [String] full path to the archive to be stored.
+    # @return [String] full path to the stored archive.
+    def self.fetch(url, options = {})
+      cache_path = options[:cache_path] || Dir.mktmpdir
+      escaped_url = CGI.escape(url)
+      archive_path = File.join(cache_path, escaped_url)
+      yield archive_path unless File.exist?(archive_path)
+      archive_path
+    end
+  end
+end

data/lib/suvii/extract.rb

@@ -0,0 +1,58 @@
+module Suvii
+  class Extract
+    UnknownFormatError = Class.new(StandardError)
+
+    TARGZ_RE = /\.t(ar\.)?gz\z/
+    ZIP_RE = /\.zip\z/
+
+    # Detects proper class for given archive path.
+    #
+    # @return [Targz, Zip]
+    # @param source [String] local path to an archive.
+    # @raise [UnknownFormatError] when archive format is not supported.
+    def self.class_for(source)
+      case source
+      when TARGZ_RE then Targz
+      when ZIP_RE then Zip
+      else raise UnknownFormatError, "unknown format for #{source}"
+      end
+    end
+
+    # @return [String]
+    attr_reader :source
+
+    # @return [Integer, nil]
+    attr_reader :strip_components
+
+    # @param source [String] local path to an archive.
+    # @option options [Integer] :strip_components (nil, i.e. no skipping) specifies number of top-level
+    #   directories to be skipped during the archive extraction. Same as `strip-components` option
+    #   for GNU tar.
+    def initialize(source, options = {})
+      @source = source
+      @strip_components = options[:strip_components]
+    end
+
+    # Performs archive extraction.
+    #
+    # @param destination [String] directory where the archive should be extracted.
+    # @return [String] destination.
+    def extract_to(destination)
+      raise NotImplementedError
+    end
+
+    private
+
+    def path_with_stripped_components(path)
+      if strip_components
+        segments = path.split(File::SEPARATOR)
+        segments[strip_components..-1].join(File::SEPARATOR)
+      else
+        path
+      end
+    end
+  end
+end
+
+require "suvii/extract/targz"
+require "suvii/extract/zip"

data/lib/suvii/extract/targz.rb

@@ -0,0 +1,49 @@
+require "fileutils"
+require "zlib"
+require "rubygems/package"
+
+module Suvii
+  class Extract
+    class Targz < Extract
+      CHUNK_SIZE = 65_536
+      DIR_UP = ".."
+      PAX_GLOBAL_HEADER = "pax_global_header"
+
+      # can't use IO.copy_stream because TarReader::Entry#read has different arity when IO#read
+      def self.copy_stream(tar_io, destination)
+        File.open(destination, "wb") do |destination_file|
+          until tar_io.eof?
+            destination_file.write(tar_io.read(CHUNK_SIZE))
+          end
+        end
+      end
+
+      # (see Extract#extract_to)
+      def extract_to(destination)
+        Zlib::GzipReader.open(source) do |gz|
+          Gem::Package::TarReader.new(gz) do |tar|
+            tar.each do |tarfile|
+              next if tarfile.full_name == PAX_GLOBAL_HEADER
+
+              if tarfile.full_name.include?(DIR_UP)
+                raise SecurityError, "can't write outside of destination directory, entry filename is #{tarfile.full_name.inspect}"
+              end
+
+              path = path_with_stripped_components(tarfile.full_name)
+              destination_file = File.join(destination, path)
+
+              if tarfile.directory?
+                FileUtils.mkdir_p(destination_file)
+              else
+                destination_directory = File.dirname(destination_file)
+                FileUtils.mkdir_p(destination_directory)
+                Targz.copy_stream(tarfile, destination_file)
+              end
+            end
+          end
+        end
+        destination
+      end
+    end
+  end
+end

data/lib/suvii/extract/zip.rb

@@ -0,0 +1,19 @@
+require "zip"
+
+module Suvii
+  class Extract
+    class Zip < Extract
+      # (see Extract#extract_to)
+      def extract_to(destination)
+        ::Zip::File.open(source) do |zip_file|
+          zip_file.each do |entry|
+            path = path_with_stripped_components(entry.name)
+            destination_file = File.join(destination, path)
+            entry.extract(destination_file)
+          end
+        end
+        destination
+      end
+    end
+  end
+end

data/lib/suvii/http.rb

@@ -0,0 +1,31 @@
+require "open-uri"
+
+module Suvii
+  # Thin wrapper around {https://github.com/ruby/ruby/blob/trunk/lib/open-uri.rb OpenURI},
+  # therefore proxy can be configured via standard env variables.
+  class Http
+    # Saves a content of the url to a file.
+    #
+    # @param url [String] URL to be fetched.
+    # @param destination [String] path to a file to be created with the URL content.
+    # @option options [Integer] :max_attempts_for_200_response (1, i.e. no retries) number of times
+    #   to fetch a remote resource until get a 200 response.
+    # @raise [OpenURI::HTTPError]
+    # @return [true]
+    def self.save(url, destination, options = {})
+      attempts ||= 1
+      uri = URI.parse(url)
+      Suvii.logger.info("downloading '#{url}'")
+      IO.copy_stream(uri.open, destination)
+      true
+    rescue OpenURI::HTTPError => e
+      if attempts < (options[:max_attempts_for_200_response] || 1)
+        Suvii.logger.warn("failed to download '#{url}': #{e.message}")
+        attempts += 1
+        retry
+      else
+        raise
+      end
+    end
+  end
+end

data/suvii.gemspec

@@ -0,0 +1,23 @@
+Gem::Specification.new do |spec|
+  spec.name          = "suvii"
+  spec.version       = "0.1.0"
+  spec.authors       = ["Andrii Malyshko"]
+  spec.email         = ["mail@nashbridges.me"]
+  spec.description   = "Read from a remote tar.gz/zip archive easily"
+  spec.summary       = spec.description
+  spec.homepage      = "https://github.com/torba-rb/suvii"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "rubyzip", "~> 1.0"
+
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "bundler", "~> 1.6"
+  spec.add_development_dependency "minitest", "~> 5.4"
+  spec.add_development_dependency "webmock", "~> 1.24"
+  spec.add_development_dependency "assert_dirs_equal", "~> 0.2"
+end

data/test/acceptance_test.rb

@@ -0,0 +1,29 @@
+require "test_helper"
+
+module Suvii
+  class AcceptanceTest < Minitest::Test
+    attr_reader :destination
+
+    def teardown
+      FileUtils.rm_rf(destination)
+    end
+
+    def test_zip
+      @destination = Suvii.fetch(
+        "https://github.com/torba-rb/Trumbowyg/archive/1.1.7.zip",
+        strip_components: 1,
+        max_attempts_for_200_response: 2
+      )
+      assert_dirs_equal "test/fixtures/results/Trumbowyg-1.1.7", destination
+    end
+
+    def test_targz
+      @destination = Suvii.fetch(
+        "https://github.com/torba-rb/Trumbowyg/archive/1.1.7.tar.gz",
+        strip_components: 1,
+        max_attempts_for_200_response: 2
+      )
+      assert_dirs_equal "test/fixtures/results/Trumbowyg-1.1.7", destination
+    end
+  end
+end