matthewtodd-downloads 0.6.5

data/README.rdoc

@@ -0,0 +1,66 @@
+Downloads helps me get big files into Tanzania. It's been fun to write! Maybe you'll be able to make use of it, too.
+
+== Overview
+
+Since http downloads can be a little flaky over a slow connection, we take this basic approach:
+
+1. Ssh to a server with a fast connection and download the file to a special directory there.
+2. Later, rsync down the contents of that directory.
+
+Beyond that, it's all polish. There's a nice interactive shell, smart (non-)use of the network, and good tab completion all around.
+
+== Sample Workflow
+
+  $ downloads add http://media.railscasts.com/videos/143_paypal_security.mov
+  $ downloads sync
+  receiving incremental file list
+  143_paypal_security.mov
+        229376   0%    7.86kB/s    0:50:27
+  ...
+  $ downloads rm 143_paypal_security.mov
+  $
+
+== Installation
+
+  sudo gem install matthewtodd-downloads --source http://gems.github.com
+
+== Configuration
+
+You can see your current configuration via the "config" command:
+
+  downloads config
+
+To update these parameters, pass the key and the new value like so:
+
+  downloads config remote_host downloads.example.com
+
+== Commands
+
+  $ downloads help
+  add URL ...
+  attachments
+  config [KEY [VALUE]]
+  help [COMMAND]
+  ls
+  mv SOURCE TARGET
+  quit
+  rm FILE ...
+  shell
+  status
+  sync [kill]
+
+== Tips
+
+1. <tt>downloads shell</tt> re-uses the same ssh connection through your entire session, <b>saving the re-connection overhead</b> for every remote command.
+
+2. <b>Tab completion</b> is available for free in <tt>downloads shell</tt>. You can also get it at the command line thanks to Dr. Nic's excellent TabTab gem, http://github.com/drnic/tabtab/tree/REL-0.9.1/PostInstall.txt.
+
+3. I use the following <b>cron jobs</b> to <b>sync my downloads overnight</b>. (Restarting <tt>sync</tt> every half hour helps recover from dropped connections.)
+     */30 0-5,22-23 * * * downloads sync
+     30   6         * * * downloads sync kill
+
+4. I like to set up a <b>GeekTool window</b> showing the results of <tt>downloads status</tt>. If you like, you can bottom-justify this text by piping it through http://gist.github.com/43363.
+
+5. Install Downloads on your mail server, and you can set up a special email address, say <tt>big.files@downloads.example.com</tt>, that pipes all its messages through <tt>downloads attachments</tt>. Currently, the messages will be dropped on the floor, but <b>any attachments will be dropped into your downloads folder</b>, ready to be sync'ed down.
+
+6. I've been noodling with some <b>applescripts</b>. Check out the <tt>resources</tt> folder.

data/bin/downloads

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby -w
+
+$:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'downloads'
+
+command = Downloads::Commands.lookup(ARGV)
+if command.valid?
+  command.run
+else
+  puts "Usage: downloads #{command.usage}"
+  exit 1
+end

data/lib/downloads.rb

@@ -0,0 +1,3 @@
+require 'downloads/commands'
+require 'downloads/configuration'
+require 'downloads/servers'

data/lib/downloads/commands.rb

@@ -0,0 +1,81 @@
+module Downloads #:nodoc:
+  module Commands
+    def self.configuration
+      @@configuration ||= Configuration.new
+    end
+
+    def self.registry
+      @@registry ||= {}
+    end
+
+    def self.names
+      registry.keys.sort
+    end
+
+    def self.objects
+      registry.values.sort_by { |command| command.command_name }
+    end
+
+    # MAYBE we could use optparse at this top level as well, allowing for (1) overriding host & directory config and (2) faking remote interactions
+    def self.lookup(argv)
+      klass = registry[argv.shift] || Help
+      klass.new(configuration, argv)
+    end
+
+    class Base
+      def self.command_name
+        name.split('::').last.downcase
+      end
+
+      def self.usage
+        command_name
+      end
+
+      def self.inherited(command)
+        Commands.registry[command.command_name] = command
+      end
+
+      attr_reader :configuration, :local, :remote, :options
+
+      def initialize(configuration, argv)
+        @configuration = configuration
+        @local         = configuration.local_server
+        @remote        = configuration.remote_server
+        configure(argv)
+      end
+
+      def configure(argv)
+      end
+
+      def usage
+        self.class.usage
+      end
+
+      def run
+        raise NotImplementedError
+      end
+
+      def valid?
+        true
+      end
+
+      private
+
+      def shift_argument(argv)
+        argv.shift unless argv.first =~ /^-/
+      end
+    end
+  end
+end
+
+require 'downloads/commands/add'
+require 'downloads/commands/attachments'
+require 'downloads/commands/config'
+require 'downloads/commands/help'
+require 'downloads/commands/ls'
+require 'downloads/commands/mv'
+require 'downloads/commands/quit'
+require 'downloads/commands/rm'
+require 'downloads/commands/shell'
+require 'downloads/commands/status'
+require 'downloads/commands/sync'

data/lib/downloads/commands/add.rb

@@ -0,0 +1,36 @@
+require 'uri'
+
+module Downloads
+  module Commands
+    class Add < Base
+      attr_accessor :uris
+
+      def self.usage
+        "#{super} URL ..."
+      end
+
+      def configure(argv)
+        self.uris = []
+        while uri = shift_argument(argv)
+          self.uris << parse_uri(uri)
+        end
+      end
+
+      def run
+        remote.run("wget '#{uris.join("' '")}' --no-check-certificate")
+      end
+
+      def valid?
+        uris.any?
+      end
+
+      private
+
+      def parse_uri(uri)
+        URI.parse(uri)
+      rescue URI::InvalidURIError => error
+        puts error.message
+      end
+    end
+  end
+end

data/lib/downloads/commands/attachments.rb

@@ -0,0 +1,23 @@
+require 'rubygems'
+require 'tmail'
+
+module Downloads
+  module Commands
+    class Attachments < Base
+      attr_accessor :stream
+
+      def configure(argv)
+        self.stream = ARGF
+      end
+
+      # TODO forward original messages from attachment emails
+      def run
+        TMail::Mail.parse(stream.read).attachments.each do |attachment|
+          filename = File.join(local.directory, attachment.original_filename)
+          File.open(filename, 'wb') { |file| file.write(attachment.read) }
+          File.chmod(0644, filename)
+        end
+      end
+    end
+  end
+end

data/lib/downloads/commands/config.rb

@@ -0,0 +1,26 @@
+module Downloads
+  module Commands
+    class Config < Base
+      attr_accessor :key, :value
+
+      def self.usage
+        "#{super} [KEY [VALUE]]"
+      end
+
+      def configure(argv)
+        self.key   = shift_argument(argv)
+        self.value = shift_argument(argv)
+      end
+
+      def run
+        if value
+          configuration[key] = value
+        elsif key
+          puts configuration[key]
+        else
+          puts configuration.to_yaml
+        end
+      end
+    end
+  end
+end

data/lib/downloads/commands/help.rb

@@ -0,0 +1,23 @@
+module Downloads
+  module Commands
+    class Help < Base
+      attr_accessor :command
+
+      def self.usage
+        "#{super} [COMMAND]"
+      end
+
+      def configure(argv)
+        self.command = shift_argument(argv)
+      end
+
+      def run
+        if command
+          puts Commands.registry[command].usage
+        else
+          puts Commands.objects.map { |command| command.usage }
+        end
+      end
+    end
+  end
+end

data/lib/downloads/commands/ls.rb

@@ -0,0 +1,9 @@
+module Downloads
+  module Commands
+    class Ls < Base
+      def run
+        puts remote.filenames
+      end
+    end
+  end
+end

data/lib/downloads/commands/mv.rb

@@ -0,0 +1,25 @@
+module Downloads
+  module Commands
+    class Mv < Base
+      attr_accessor :source, :target
+
+      def self.usage
+        "#{super} SOURCE TARGET"
+      end
+
+      def configure(argv)
+        self.source = shift_argument(argv)
+        self.target = shift_argument(argv)
+      end
+
+      def run
+        remote.run("mv '#{source}' '#{target}'")
+        local.run("mv '#{source}' '#{target}'") if local.exists?(source)
+      end
+
+      def valid?
+        source && target
+      end
+    end
+  end
+end

data/lib/downloads/commands/quit.rb

@@ -0,0 +1,10 @@
+module Downloads
+  module Commands
+    class Quit < Base
+      def run
+        puts 'Goodbye.'
+        exit
+      end
+    end
+  end
+end

data/lib/downloads/commands/rm.rb

@@ -0,0 +1,26 @@
+module Downloads
+  module Commands
+    class Rm < Base
+      attr_accessor :filenames
+
+      def self.usage
+        "#{super} FILE ..."
+      end
+
+      def configure(argv)
+        self.filenames = []
+        while filename = shift_argument(argv)
+          self.filenames << filename
+        end
+      end
+
+      def run
+        remote.run("rm '#{filenames.join("' '")}'")
+      end
+
+      def valid?
+        filenames.any?
+      end
+    end
+  end
+end

data/lib/downloads/commands/shell.rb

@@ -0,0 +1,36 @@
+require 'tabtab' unless Object.const_defined?(:TabTab)
+require File.join(File.dirname(__FILE__), '..', 'tabtab_definitions')
+require 'readline'
+require 'shellwords'
+
+module Downloads
+  module Commands
+    class Shell < Base
+      def run
+        Readline.completer_word_break_characters = ''
+        Readline.completion_append_character = ' '
+        Readline.completion_proc = lambda do |line|
+          words = Shellwords.shellwords("downloads #{line}")
+          words << '' if line.split('')[-1] == ' '
+          previous_token, current_token = words[-2..-1]
+          completions = TabTab::Definition['downloads'].extract_completions(previous_token, current_token, {})
+          completions.map { |completion| (words[1..-2] + [completion]).join(' ') }
+        end
+
+        puts 'Type ctrl-d or quit to exit.'
+
+        loop do
+          line = Readline::readline('> ') || 'quit'
+          next if line.strip == ''
+          Readline::HISTORY.push(line)
+          command = Downloads::Commands.lookup(line.split(' '))
+          if command.valid?
+            command.run
+          else
+            puts command.usage
+          end
+        end
+      end
+    end
+  end
+end

data/lib/downloads/commands/status.rb

@@ -0,0 +1,40 @@
+module Downloads
+  module Commands
+    class Status < Base
+      def run
+        longest_filename = remote.filenames.max { |a, b| a.length <=> b.length }
+        remote.files.each do |file|
+          puts "%-#{longest_filename.length}s\t%3s%%\t%5s" % [file[:name], status(file), human_readable(file[:size])]
+        end
+      end
+
+      private
+
+      def status(remote_file)
+        local_file = local.exists?(remote_file[:name]) || { :size => 0 }
+        percent(local_file[:size], remote_file[:size])
+      end
+
+      def percent(numerator, denominator)
+        if denominator.zero? # oddly enough, this is happening for a .webloc file on my Desktop right now
+          '-'
+        else
+          (numerator.to_f * 100 / denominator).to_i
+        end
+      end
+
+      def human_readable(bytes)
+        case bytes
+        when (0...1024)
+          "#{bytes}B"
+        when (1024...1024**2)
+          "#{bytes/1024}K"
+        when (1024**2...1024**3)
+          "#{bytes/1024**2}M"
+        else
+          "#{bytes/1024**3}G"
+        end
+      end
+    end
+  end
+end

data/lib/downloads/commands/sync.rb

@@ -0,0 +1,40 @@
+module Downloads
+  module Commands
+    class Sync < Base
+      attr_accessor :kill
+
+      def self.usage
+        "#{super} [kill]"
+      end
+
+      def configure(argv)
+        self.kill = (shift_argument(argv) == 'kill')
+      end
+
+      def run
+        if File.exists?(pid_file)
+          `kill #{File.read(pid_file)}`
+          File.delete(pid_file)
+        end
+
+        unless kill
+          pid = fork { exec("rsync --recursive --progress --partial #{remote.rsync_path} #{local.rsync_path}") }
+          File.open(pid_file, 'w') { |file| file.write(pid) }
+
+          begin
+            Process.wait
+          rescue Interrupt
+            # we don't need to see the stacktrace
+            puts # but a blank line is nice
+          end
+        end
+      end
+
+      private
+
+      def pid_file
+        configuration.pid_file
+      end
+    end
+  end
+end

data/lib/downloads/configuration.rb

@@ -0,0 +1,61 @@
+require 'pathname'
+require 'yaml'
+
+module Downloads
+  class Configuration
+    DIRECTORY = Pathname.new(File.join(ENV['HOME'], '.downloads'))
+    DEFAULTS = { 'remote_host' => 'downloads', 'remote_directory' => 'downloads', 'local_directory' => File.join(ENV['HOME'], 'Desktop') }.freeze
+
+    attr_reader :local_server, :remote_server
+
+    def initialize
+      DIRECTORY.mkdir unless DIRECTORY.directory?
+
+      @values = DEFAULTS.dup
+      @values.merge!(YAML.load_file(path('config'))) if path('config').file?
+
+      @local_server  = build_local_server
+      @remote_server = build_remote_server
+    end
+
+    def [](key)
+      @values[key.to_s]
+    end
+
+    def []=(key, value)
+      @values[key.to_s] = value
+      File.open(path('config'), 'w') { |file| file.puts(to_yaml) }
+
+      case key.to_s
+      when /^remote/
+        File.delete(path('remote_cache')) if path('remote_cache').file?
+        @remote_server = build_remote_server
+      when /^local/
+        @local_server = build_local_server
+      end
+    end
+
+    def pid_file
+      path('pid')
+    end
+
+    def to_yaml
+      # Marginally nicer than @values.to_yaml, as it avoids the leading '---'
+      @values.map { |key, value| "#{key}: #{value}" }.join("\n")
+    end
+
+    private
+
+    def build_local_server
+      Servers::Local.new(@values['local_directory'])
+    end
+
+    def build_remote_server
+      Servers::Remote.new(@values['remote_host'], @values['remote_directory'], path('remote_cache'))
+    end
+
+    def path(path)
+      DIRECTORY + path
+    end
+  end
+end

data/lib/downloads/servers.rb

@@ -0,0 +1,28 @@
+module Downloads
+  module Servers #:nodoc:
+    class Base
+      def exists?(filename)
+        filenames.include?(filename)
+      end
+
+      def files
+        raise NotImplementedError
+      end
+
+      def filenames
+        files.map { |file| file[:name] }
+      end
+
+      def rsync_path
+        raise NotImplementedError
+      end
+
+      def run(command)
+        raise NotImplementedError
+      end
+    end
+  end
+end
+
+require 'downloads/servers/local'
+require 'downloads/servers/remote'

data/lib/downloads/servers/local.rb

@@ -0,0 +1,25 @@
+module Downloads
+  module Servers
+    class Local < Base
+      attr_reader :directory
+
+      def initialize(directory)
+        @directory = directory
+      end
+
+      def files
+        Dir.chdir(directory) do
+          Dir.glob('*').sort.map { |name| { :name => name, :size => File.size(name) } }
+        end
+      end
+
+      def rsync_path
+        "#{@directory}/"
+      end
+
+      def run(command)
+        `sh -c "cd #{@directory}; #{command}"`
+      end
+    end
+  end
+end

data/lib/downloads/servers/remote.rb

@@ -0,0 +1,52 @@
+require 'net/ssh'
+
+module Downloads
+  module Servers
+    class Remote < Base
+      def initialize(host, directory, cache_path)
+        @host       = host
+        @directory  = directory
+        @cache_path = cache_path
+        @connection = nil
+        @files      = YAML.load_file(@cache_path) if File.exists?(@cache_path)
+      end
+
+      def files
+        populate_file_cache unless @files
+        @files
+      end
+
+      def rsync_path
+        "#{@host}:#{@directory}/"
+      end
+
+      def run(command)
+        result = run_in_directory(command)
+        populate_file_cache
+        result
+      end
+
+      private
+
+      def connection
+        initialize_connection unless @connection
+        @connection
+      end
+
+      def initialize_connection
+        @connection = Net::SSH.start(@host, ENV['USER']) # TODO does it make sense to use ENV['USER']?
+        at_exit { @connection.close }
+      end
+
+      def run_in_directory(command)
+        connection.exec!("cd #{@directory}; #{command}")
+      end
+
+      def populate_file_cache
+        yaml = run_in_directory(%{ruby -ryaml -e "puts Dir.glob('*').sort.map { |name| { :name => name, :size => File.size(name) } }.to_yaml"})
+        File.open(@cache_path, 'w') { |file| file.write(yaml) }
+        @files = YAML.load(yaml)
+      end
+    end
+  end
+end

data/lib/downloads/tabtab_definitions.rb

@@ -0,0 +1,55 @@
+TabTab::Definition.register('downloads') do |c|
+  def clipboard_contents
+    [`pbpaste`.strip]
+  end
+
+  def commands
+    require File.join(File.dirname(__FILE__), 'commands') unless Object.const_defined?(:Downloads)
+    Downloads::Commands.names
+  end
+
+  def remote_files
+    require File.join(File.dirname(__FILE__), 'commands') unless Object.const_defined?(:Downloads)
+    Downloads::Commands.configuration.remote_server.filenames
+  end
+
+  # FIXME waiting for tabtab to support completion for multiple filenames
+  c.command(:add) do |add|
+    add.default { clipboard_contents.grep(/^http:/) }
+  end
+
+  c.command(:config) do |config|
+    config.command(:remote_host)      { }
+    config.command(:remote_directory) { }
+    config.command(:local_directory)  { }
+  end
+
+  c.command(:help) do |help|
+    help.default { commands }
+  end
+
+  c.command(:ls) do |ls|
+  end
+
+  c.command(:mv) do |mv|
+    mv.default { remote_files }
+  end
+
+  # FIXME waiting for tabtab to support completion for multiple filenames
+  c.command(:rm) do |rm|
+    rm.default { remote_files }
+  end
+
+  c.command(:quit) do |shell|
+  end
+
+  c.command(:shell) do |shell|
+  end
+
+  c.command(:status) do |status|
+  end
+
+  c.command(:sync) do |sync|
+    sync.command :kill
+  end
+end

metadata

@@ -0,0 +1,110 @@
+--- !ruby/object:Gem::Specification 
+name: matthewtodd-downloads
+version: !ruby/object:Gem::Version 
+  version: 0.6.5
+platform: ruby
+authors: 
+- Matthew Todd
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-01-06 00:00:00 -08:00
+default_executable: downloads
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: net-ssh
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 2.0.3
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: tabtab
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.9.1
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: tmail
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.2.2
+    version: 
+description: 
+email: matthew.todd@gmail.com
+executables: 
+- downloads
+extensions: []
+
+extra_rdoc_files: 
+- README.rdoc
+- TODO.rdoc
+- bin/downloads
+files: 
+- README.rdoc
+- TODO.rdoc
+- bin/downloads
+- lib/downloads/commands/add.rb
+- lib/downloads/commands/attachments.rb
+- lib/downloads/commands/config.rb
+- lib/downloads/commands/help.rb
+- lib/downloads/commands/ls.rb
+- lib/downloads/commands/mv.rb
+- lib/downloads/commands/quit.rb
+- lib/downloads/commands/rm.rb
+- lib/downloads/commands/shell.rb
+- lib/downloads/commands/status.rb
+- lib/downloads/commands/sync.rb
+- lib/downloads/commands.rb
+- lib/downloads/configuration.rb
+- lib/downloads/servers/local.rb
+- lib/downloads/servers/remote.rb
+- lib/downloads/servers.rb
+- lib/downloads/tabtab_definitions.rb
+- lib/downloads.rb
+- resources/applescripts
+- resources/applescripts/Add Downloads.app
+- resources/applescripts/folderaction_download_webloc.scpt
+- resources/applescripts/netnewswire_download_enclosure.scpt
+has_rdoc: true
+homepage: 
+post_install_message: 
+rdoc_options: 
+- --main
+- README.rdoc
+- --title
+- downloads-0.6.5
+- --inline-source
+- --line-numbers
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: 
+- rsync
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: Downloads uses net-ssh, rsync and tmail to reliably get big files into Tanzania.
+test_files: []
+