pmirror 0.0.1

data/.gitignore

@@ -0,0 +1,4 @@
+.DS_Store
+results.html
+pkg
+html

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm: 
+  - 1.9.3
+  - 2.1.0

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in pmirror.gemspec
+gemspec

data/Gemfile.lock

@@ -0,0 +1,59 @@
+PATH
+  remote: .
+  specs:
+    pmirror (0.0.1)
+      methadone (~> 1.3.1)
+      nokogiri
+      progressbar
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    aruba (0.5.4)
+      childprocess (>= 0.3.6)
+      cucumber (>= 1.1.1)
+      rspec-expectations (>= 2.7.0)
+    builder (3.2.2)
+    childprocess (0.3.9)
+      ffi (~> 1.0, >= 1.0.11)
+    cucumber (1.3.10)
+      builder (>= 2.1.2)
+      diff-lcs (>= 1.1.3)
+      gherkin (~> 2.12)
+      multi_json (>= 1.7.5, < 2.0)
+      multi_test (>= 0.0.2)
+    diff-lcs (1.2.5)
+    ffi (1.9.3)
+    gherkin (2.12.2)
+      multi_json (~> 1.3)
+    json (1.8.1)
+    methadone (1.3.1)
+      bundler
+    mini_portile (0.5.2)
+    multi_json (1.8.4)
+    multi_test (0.0.3)
+    nokogiri (1.6.1)
+      mini_portile (~> 0.5.0)
+    progressbar (0.21.0)
+    rake (0.9.6)
+    rdoc (4.1.1)
+      json (~> 1.4)
+    rspec (2.14.1)
+      rspec-core (~> 2.14.0)
+      rspec-expectations (~> 2.14.0)
+      rspec-mocks (~> 2.14.0)
+    rspec-core (2.14.7)
+    rspec-expectations (2.14.4)
+      diff-lcs (>= 1.1.3, < 2.0)
+    rspec-mocks (2.14.4)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  aruba
+  bundler (~> 1.3)
+  pmirror!
+  rake (~> 0.9.2)
+  rdoc
+  rspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Aaron Nichols
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,97 @@
+# Pmirror
+
+[![Build
+Status](https://travis-ci.org/adnichols/pmirror.png?branch=master)](https://travis-ci.org/adnichols/pmirror)
+
+pmirror is a tool primarily intended to mirror a subset of files from a
+remote http repository. I created the tool because I wanted a way to
+mirror just some files from a remote RPM repository to a local
+repository. I also wanted to be able to perform operations on the
+locally downloaded files once the mirror operation was complete
+
+Was there already a tool out there that does this? Probably, I couldn't
+find it. 
+
+## Features
+
+The tool is very new so it only has the bare minimum feature set:
+
+- Specify multiple regex patterns to match against the remote directory
+- Provides a progressbar download status indicator
+- Specify a local directory to download files to
+- Specify a command to execute on the local directory once all files are
+  downloaded
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'pmirror'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install pmirror
+
+### Dependencies
+
+This gem makes use of the following other projects
+
+- [methadone](https://github.com/davetron5000/methadone)
+- [progressbar](https://github.com/peleteiro/progressbar)
+- [nokogiri](http://nokogiri.org/)
+
+## Usage
+
+```
+Usage: pmirror [options] url
+
+Options:
+    -h, --help                       Show command line help
+    -p, --pattern PAT1,PAT2,PAT3     Regex to match files in remote dir,
+may specify multiple patterns
+    -l, --localdir DIR               Local directory to mirror files to
+    -e, --exec CMD                   Execute command after completion
+    -d, --debug                      Enable debugging
+    -v, --version                    Show version
+```
+
+Usage should be pretty self explanatory but here are the details:
+
+`url` is the remote URL that you want to fetch files from. Right now
+this is assumed to be an un-authenticated url. We do not recurse into
+directories looking for files.
+
+`--pattern` allows you to specify a comma separated list of patterns to
+match on the remote directly. We will iterate over each pattern and
+build up the file list to fetch from the remote url. 
+
+`--localdir` is the location you want files downloaded to. It is also
+the directory in which any commands specified with `--exec` will be
+performed.
+
+`--exec` is used to perform actions on the download directory. The
+envisioned use case right now is simply to run a createrepo when the
+downloads are complete. This will change PWD to whatever directory is
+specified in `--localdir` and then will run the command you specify.
+There is not currently a way to pass the value of `--localdir` into a
+command other than to specify `.` in your command. 
+
+`--debug` would enable some extra output
+
+`--version` provides the version info
+
+`--help` should be self explanatory
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Add tests for your new feature to pmirror.feature
+4. Run tests (`rake`)
+5. Commit your changes (`git commit -am 'Add some feature'`)
+6. Push to the branch (`git push origin my-new-feature`)
+7. Create new Pull Request

data/README.rdoc

@@ -0,0 +1,19 @@
+= pmirror - DESCRIBE YOUR GEM
+
+Author::  YOUR NAME (YOUR EMAIL)
+Copyright:: Copyright (c) 2014 YOUR NAME
+
+
+DESCRIBE YOUR GEM HERE
+
+== Links
+
+* {Source on Github}[LINK TO GITHUB]
+* RDoc[LINK TO RDOC.INFO]
+
+== Install
+
+== Examples
+
+== Contributing
+

data/Rakefile

@@ -0,0 +1,67 @@
+def dump_load_path
+  puts $LOAD_PATH.join("\n")
+  found = nil
+  $LOAD_PATH.each do |path|
+    if File.exists?(File.join(path,"rspec"))
+      puts "Found rspec in #{path}"
+      if File.exists?(File.join(path,"rspec","core"))
+        puts "Found core"
+        if File.exists?(File.join(path,"rspec","core","rake_task"))
+          puts "Found rake_task"
+          found = path
+        else
+          puts "!! no rake_task"
+        end
+      else
+        puts "!!! no core"
+      end
+    end
+  end
+  if found.nil?
+    puts "Didn't find rspec/core/rake_task anywhere"
+  else
+    puts "Found in #{path}"
+  end
+end
+require 'bundler'
+require 'rake/clean'
+
+begin
+require 'rspec/core/rake_task'
+rescue LoadError
+dump_load_path
+raise
+end
+
+require 'cucumber'
+require 'cucumber/rake/task'
+gem 'rdoc' # we need the installed RDoc gem, not the system one
+require 'rdoc/task'
+
+include Rake::DSL
+
+Bundler::GemHelper.install_tasks
+
+
+RSpec::Core::RakeTask.new do |t|
+  # Put spec opts in a file named .rspec in root
+end
+
+
+PIDFILE = Dir.pwd + "/webrick.pid"
+CUKE_RESULTS = 'results.html'
+CLEAN << CUKE_RESULTS
+Cucumber::Rake::Task.new(:features) do |t|
+  t.cucumber_opts = "features --format html -o #{CUKE_RESULTS} --format pretty --no-source -x"
+  t.fork = false
+end
+
+Rake::RDocTask.new do |rd|
+  
+  rd.main = "README.rdoc"
+  
+  rd.rdoc_files.include("README.rdoc","lib/**/*.rb","bin/**/*")
+end
+
+task :default => [:spec,:features]
+

data/bin/pmirror

@@ -0,0 +1,43 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'methadone'
+require 'pmirror.rb'
+
+class App
+  include Methadone::Main
+  include Methadone::CLILogging
+
+  main do # Add args you want: |like,so|
+    # your program code here
+    # You can access CLI options via
+    # the options Hash
+  end
+
+  # supplemental methods here
+
+  # Declare command-line interface here
+
+  # description "one line description of your app"
+  #
+  # Accept flags via:
+  # on("--flag VAL","Some flag")
+  # options[flag] will contain VAL
+  #
+  # Specify switches via:
+  # on("--[no-]switch","Some switch")
+  #
+  # Or, just call OptionParser methods on opts
+  #
+  # Require an argument
+  # arg :some_arg 
+  #
+  # # Make an argument optional
+  # arg :optional_arg, :optional
+
+  version Pmirror::VERSION
+
+  use_log_level_option
+
+  go!
+end

data/features/pmirror.feature

@@ -0,0 +1,55 @@
+Feature: My bootstrapped app kinda works
+  In order to get going on coding my awesome app
+  I want to have aruba and cucumber setup
+  So I don't have to do it myself
+
+  Scenario: App just runs
+    When I get help for "pmirror"
+    Then the exit status should be 0
+    And the banner should be present
+    And the banner should document that this app takes options
+    And the following options should be documented:
+      |--pattern|
+      |--debug|
+      |--localdir|
+      |--exec|
+      |--version|
+    And the banner should document that this app's arguments are:
+      |url|which is required|
+
+
+  Scenario: Download a file
+    When I successfully run `pmirror -p meh -l ../foo http://localhost:55555`
+    Then the exit status should be 0
+    And the following files should exist:
+      |../foo/meh.txt|
+
+  Scenario: Execute on local directory
+    When I successfully run `pmirror -p meh -l ../foo -e "touch test" http://localhost:55555`
+    Then the exit status should be 0
+    And the following files should exist:
+      |../foo/meh.txt|
+      |../foo/test   |
+
+  Scenario: Match multiple files
+    When I successfully run `pmirror -p floo -l ../foo http://localhost:55555`
+    Then the exit status should be 0
+    And the following files should exist:
+      | ../foo/floober.txt|
+      | ../foo/floobah.txt|
+    And the following files should not exist:
+      | ../foo/mah.txt|
+      | ../foo/meh.txt|
+
+  Scenario: Match multiple patterns
+    When I successfully run `pmirror -p '^floo.*','^mah.*' -l ../foo http://localhost:55555`
+    Then the exit status should be 0
+    And the following files should exist:
+      | ../foo/floober.txt|
+      | ../foo/floobah.txt|
+      | ../foo/mah.txt|
+    And the following files should not exist:
+      | ../foo/meh.txt|
+
+
+

data/features/step_definitions/pmirror_steps.rb

@@ -0,0 +1 @@
+# Put your step definitions here

data/features/support/env.rb

@@ -0,0 +1,49 @@
+require 'aruba/cucumber'
+require 'methadone/cucumber'
+
+ENV['PATH'] = "#{File.expand_path(File.dirname(__FILE__) + '/../../bin')}#{File::PATH_SEPARATOR}#{ENV['PATH']}"
+LIB_DIR = File.join(File.expand_path(File.dirname(__FILE__)),'..','..','lib')
+
+def start_web
+  require 'webrick'
+  pid = fork do
+    web_server = WEBrick::HTTPServer.new(:Port => 55555, :DocumentRoot => Dir.pwd + '/web')
+    trap('INT') { web_server.shutdown }
+    web_server.start
+  end
+  File.open(PIDFILE, 'w') { |f| f.write(pid) }
+end
+
+def end_web
+  if File.exist?(PIDFILE)
+    pid = File.open(PIDFILE, 'r').read.to_i
+  else
+    puts "Server not started... exiting"
+    exit
+  end
+
+  begin
+    Process.kill('INT', pid)
+  rescue
+    puts "Process not running"
+  end
+  File.unlink(PIDFILE)
+end
+
+Before do
+  # Using "announce" causes massive warnings on 1.9.2
+  @puts = true
+  @original_rubylib = ENV['RUBYLIB']
+  ENV['RUBYLIB'] = LIB_DIR + File::PATH_SEPARATOR + ENV['RUBYLIB'].to_s
+
+  FileUtils.mkdir("tmp/foo")
+  start_web
+end
+
+After do
+  ENV['RUBYLIB'] = @original_rubylib
+
+  FileUtils.rm_r("tmp/foo")
+  end_web
+end
+

data/lib/pmirror.rb

@@ -0,0 +1,98 @@
+require "pmirror/version"
+require 'nokogiri'
+require 'open-uri'
+require 'progressbar'
+
+module Pmirror
+  class Pmirror
+    include Methadone::Main
+    include Methadone::CLILogging
+    include Methadone::SH
+
+    main do |url|
+
+      download_files( url, options[:localdir], get_download_list(url, options[:pattern]) )
+      execute(options[:exec]) if options[:exec]
+
+    end
+
+    description "Mirror files on a remote http server based on pattern match"
+    arg("url", "Url or remote site", :one, :required)
+    on("-p", "--pattern PAT1,PAT2,PAT3", Array,
+       "Regex to match files in remote dir, may specify multiple patterns"
+      )
+    on("-l", "--localdir DIR", "Local directory to mirror files to")
+    on("-e", "--exec CMD", "Execute command after completion")
+    on("-d", "--debug", "Enable debugging")
+    on("-v", "--version", "Show version")
+
+    def self.get_download_list(url, pattern)
+      info "Getting download list"
+      page = Nokogiri::HTML(open(url))
+      downloads = []
+
+      page.css("a").each do |link|
+        rpm_name = link.attributes['href'].value
+        pattern.each do |matcher|
+          if /#{matcher}/.match(rpm_name)
+            downloads << rpm_name
+          end
+        end
+      end
+      downloads
+    end
+
+    def self.download_files(url, local_dir, file_list=[])
+      file_list.each do |file|
+        local_fn = "#{local_dir}/#{file}"
+
+        unless Dir.exist? options[:localdir]
+          puts "PWD: #{Dir.pwd}"
+          puts Dir.open(Dir.pwd).read
+          puts "Destination directory '#{options[:localdir]}' does not exist!"
+          exit 1
+        end
+
+        remote_fn = "#{url}/#{file}"
+        unless File.exist?(local_fn)
+          puts "Downloading File: #{file}"
+          puts "#{remote_fn} ==> #{local_fn}"
+          http_to_file(local_fn, remote_fn)
+          # File.write(local_fn, open(remote_fn).read)
+          puts "Download Complete for #{file}"
+        else
+          puts "Skipping #{file}, already exists"
+        end
+      end
+    end
+
+    def self.http_to_file(filename,url)
+      pbar = nil
+      File.open(filename, 'wb') do |save_file|
+        open(url, 'rb',
+             :content_length_proc => lambda {|t|
+          if t && 0 < t
+            pbar = ProgressBar.new("=", t)
+            pbar.file_transfer_mode
+          end
+        },
+        :progress_proc => lambda {|s|
+          pbar.set s if pbar
+        }) {|f| save_file.write(f.read) }
+      end
+      puts
+    end
+
+    def self.execute(cmd)
+      puts "Executing: #{cmd}"
+      sh("cd #{options[:localdir]} && #{cmd}")
+    end
+
+    def self.update_repodata(local_dir)
+      puts "Running createrepo for dir '#{local_dir}'"
+      sh("createrepo -c /#{local_dir}/.cache -d #{local_dir}")
+    end
+
+    go!
+  end
+end

data/lib/pmirror/version.rb

@@ -0,0 +1,3 @@
+module Pmirror
+  VERSION = "0.0.1"
+end

data/pmirror.gemspec

@@ -0,0 +1,30 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'pmirror/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "pmirror"
+  spec.version       = Pmirror::VERSION
+  spec.authors       = ["Aaron Nichols"]
+  spec.email         = ["anichols@trumped.org"]
+  spec.description   = %q{Mirror files from remote http server by pattern}
+  spec.summary       = %q{Mirror files from remote http server by pattern}
+  spec.homepage      = ""
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency('rdoc')
+  spec.add_development_dependency('aruba')
+  spec.add_development_dependency('rake', '~> 0.9.2')
+  spec.add_dependency('methadone', '~> 1.3.1')
+  spec.add_dependency('nokogiri')
+  spec.add_dependency('progressbar')
+  spec.add_development_dependency('rspec')
+end

data/spec/something_spec.rb

@@ -0,0 +1,5 @@
+describe "TestSomething" do
+  it "should be true" do
+    true.should be_true
+  end
+end

metadata

@@ -0,0 +1,215 @@
+--- !ruby/object:Gem::Specification
+name: pmirror
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Aaron Nichols
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-01-19 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rdoc
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: aruba
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.9.2
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.9.2
+- !ruby/object:Gem::Dependency
+  name: methadone
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.3.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.3.1
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: progressbar
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Mirror files from remote http server by pattern
+email:
+- anichols@trumped.org
+executables:
+- pmirror
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .travis.yml
+- Gemfile
+- Gemfile.lock
+- LICENSE.txt
+- README.md
+- README.rdoc
+- Rakefile
+- bin/pmirror
+- features/pmirror.feature
+- features/step_definitions/pmirror_steps.rb
+- features/support/env.rb
+- lib/pmirror.rb
+- lib/pmirror/version.rb
+- pmirror.gemspec
+- spec/something_spec.rb
+- web/floobah.txt
+- web/floober.txt
+- web/mah.txt
+- web/meh.txt
+homepage: ''
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: Mirror files from remote http server by pattern
+test_files:
+- features/pmirror.feature
+- features/step_definitions/pmirror_steps.rb
+- features/support/env.rb
+- spec/something_spec.rb