imw 0.2.0 → 0.2.1

data/README.rdoc

@@ -25,14 +25,13 @@ data.  It has the following goals:
 The Infinite Monkeywrench is a powerful tool but it is not always the
 right one to use.  IMW is **not** designed for
 
-* Scraping vast amounts of data (use Wuclan[http://github.com/infochimps/wuclan], Monkeyshines[http://github.com/infochimps/monkeyshines], and Edamame[http://github.com/infochimps/edamame].)
+* Scraping vast amounts of data (use Wuclan[http://github.com/infochimps/wuclan] and Monkeyshines[http://github.com/infochimps/monkeyshines])
 
 * Really, really big datasets (use Wukong[http://github.com/infochimps/wukong] and Hadoop[http://hadoop.apache.org])
 
-* Data mining
-
-* Data visualization
+* Data mining or statistical analysis
 
+* Visualization
 
 = Setup
 
@@ -47,21 +46,42 @@ and then install IMW
 
   $ sudo gem install imw
 
-= IMW Basics
+In all the examples that follow it is assumed that you've installed
+IMW and required it in a script via
 
-The central goal of IMW is to make workflow involved in processing a
-dataset from a raw source to a finished product as simple as possible.
+  require 'rubygems'
+  require 'imw'
 
-To help achieve this goal, IMW creates lots of convenient structures
-and methods.  The following sections provide a tour of these.  
+= Resources
 
-It is assumed that you've installed IMW and required it in a script
-via
+IMW is centered around processing resources.  A resource can be
+_anything_ with a URI and you create one using IMW.open.
 
-  require 'rubygems'
-  require 'imw'
+  csv     = IMW.open('/path/to/my_data.csv')
+  html    = IMW.open('http://www.infochimps.com')
+  tar_bz2 = IMW.open(
+
+IMW dynamically extends a resource with modules appropriate to it when
+you open it.  In the above case, +csv+ would be automatically extended
+by the IMW::Resources::Formats::Csv module, among others:
+
+  csv.resource_modules
+  => [IMW::Resources::LocalObj, IMW::Resources::LocalFile, IMW::Resources::Compressible, IMW::Resources::Formats::Csv]
+
+while +html+ will use a different set
+
+  html.resource_modules
+  => [IMW::Resources::LocalObj, IMW::Resources::LocalFile, IMW::Resources::Compressible, IMW::Resources::Formats::Csv]
+
+
+Consult the documentation for the modules a resource uses to learn
+what it can do.
+
+Since resources are built around the idea of URIs, you can explicitly i
+
+== Manipulating Paths
 
-== Paths
+You can p
 
 IMW holds a registry of paths that you can define on the fly or store
 in a configuration file.

data/Rakefile

@@ -21,7 +21,7 @@ end
 
 desc "Build tags"
 task :tags do
-  system "etags -R bin etc examples lib spec"
+  system "etags -R README.rdoc bin etc examples lib spec"
 end
 
 desc "Build docs"

data/VERSION

@@ -1 +1 @@
-0.2.0
+0.2.1

data/lib/imw.rb

@@ -26,12 +26,15 @@ require 'imw/utils'
 # Repositories are collections of datasets and it is on these
 # collections that the +imw+ command line tool operates.
 module IMW
-  autoload :Resource,   'imw/resource'
-  autoload :Resources,  'imw/resources'
-  autoload :Repository, 'imw/repository'
-  autoload :Dataset,    'imw/dataset'
-  autoload :Transforms, 'imw/transforms'
-  autoload :Parsers,    'imw/parsers'
+  autoload :Resource,        'imw/resource'
+  autoload :Schemes,         'imw/schemes'
+  autoload :Archives,        'imw/archives'
+  autoload :CompressedFiles, 'imw/compressed_files'
+  autoload :Formats,         'imw/formats'  
+  autoload :Tools,           'imw/tools'
+  autoload :Parsers,         'imw/parsers'
+  autoload :Dataset,         'imw/dataset'  
+  autoload :Repository,      'imw/repository'
 
   # Open a resource at the given +uri+.  The resource will
   # automatically be extended by modules which make sense given the

data/lib/imw/{resources/archive.rb → archives.rb}

@@ -1,13 +1,24 @@
 module IMW
-  module Resources
 
-    module Archives
-      autoload :Rar,    'imw/resources/archives_and_compressed/rar'
-      autoload :Tar,    'imw/resources/archives_and_compressed/tar'
-      autoload :Tarbz2, 'imw/resources/archives_and_compressed/tarbz2'
-      autoload :Targz,  'imw/resources/archives_and_compressed/targz'
-      autoload :Zip,    'imw/resources/archives_and_compressed/zip'
-    end
+  # Contains modules which define the behavior of archive files.
+  module Archives
+
+    # Handlers for archives.
+    HANDLERS = [
+                ["Archives::Tarbz2",     Proc.new { |r| r.is_local? && r.path =~ /\.tar\.bz2$/  } ],
+                ["Archives::Tarbz2",     Proc.new { |r| r.is_local? && r.path =~ /\.tbz2$/      } ],
+                ["Archives::Targz",      Proc.new { |r| r.is_local? && r.path =~ /\.tar\.gz$/   } ],
+                ["Archives::Targz",      Proc.new { |r| r.is_local? && r.path =~ /\.tgz$/       } ],
+                ["Archives::Tar",        Proc.new { |r| r.is_local? && r.path =~ /\.tar$/       } ],
+                ["Archives::Rar",        Proc.new { |r| r.is_local? && r.path =~ /\.rar$/       } ],
+                ["Archives::Zip",        Proc.new { |r| r.is_local? && r.path =~ /\.zip$/       } ]
+               ]
+
+    autoload :Rar,    'imw/archives/rar'
+    autoload :Tar,    'imw/archives/tar'
+    autoload :Tarbz2, 'imw/archives/tarbz2'
+    autoload :Targz,  'imw/archives/targz'
+    autoload :Zip,    'imw/archives/zip'
 
     # Defines methods for creating, appending to, extracting, and
     # listing an archive file.  This module isn't used to directly
@@ -15,7 +26,7 @@ module IMW
     # (e.g. - IMW::Resources::Archives::Tarbz2) include this module
     # and define the specific settings (command-line flags, &c.)
     # required to make things work.
-    module Archive
+    module Base
 
       attr_accessor :archive_settings
 
@@ -94,4 +105,3 @@ module IMW
   end
 end
 
-

data/lib/imw/archives/rar.rb

@@ -0,0 +1,19 @@
+module IMW
+  module Archives
+    module Rar
+      
+      include IMW::Archives::Base
+      
+      def archive_settings
+        @archive_settings ||= {
+          :program => :rar,
+          :create  => ['a', '-o+', '-inul'],
+          :append  => ['a', '-o+', '-inul'],
+          :list    => "vb",
+          :extract => ['x', '-o+', '-inul']
+        }
+      end
+    end
+  end
+end
+

data/lib/imw/archives/tar.rb

@@ -0,0 +1,19 @@
+module IMW
+  module Archives
+    module Tar
+
+      include IMW::Archives::Base
+      
+      def archive_settings
+        @archive_settings ||=  {
+          :create  => "-cf",
+          :append  => "-rf",
+          :list    => "-tf",
+          :extract => "-xf",
+          :program => :tar
+        }
+      end
+    end
+  end
+end
+

data/lib/imw/archives/tarbz2.rb

@@ -0,0 +1,73 @@
+module IMW
+  module Archives
+    module Tarbz2
+
+      #
+      # It's a compressed file
+      #
+
+      include IMW::CompressedFiles::Base
+
+      def compression_settings
+        @compression_settings ||= {
+          :program               => :bzip2,
+          :decompression_program => :bunzip2,
+          :decompress            => '',
+          :extension             => 'bz2'
+        }
+      end
+
+      #
+      # But it's also an archive
+      #
+      
+      include IMW::Archives::Base
+      
+      def archive_settings
+        @archive_settings ||= {
+          :program               => :tar,
+          :create                => '-cf',
+          :list                  => "-tjf",
+          :extract               => "-xjf"
+        }
+      end
+
+      # Overrides default behvaior of IMW::Files::Archive#create to
+      # compress files after creating them.
+      def create *input_paths
+        IMW.system(archive_settings[:program], archive_settings[:create], path_between_archive_and_compression, *input_paths.flatten)
+        IMW.open(path_between_archive_and_compression).compress!
+      end
+
+      def decompressed_basename
+        case extname
+        when '.tar.bz2' then basename[0..-5]               # .tar.bz2 => .tar
+        when '.tbz2'    then basename.gsub(/tbz2$/, 'tar') # .tbz2    => .tar
+        else                 basename[0..-(extname.size + 1)]
+        end
+      end
+      
+
+      protected
+      def path_between_archive_and_compression
+        File.join(dirname,name + '.tar')
+      end
+
+      public
+
+      #
+      # It's a compressed file AND an archive!
+      #
+      
+      def extname
+        case path
+        when /\.tar\.bz2$/ then '.tar.bz2'
+        when /\.tbz2$/     then '.tbz2'
+        else               File.extname(path)
+        end
+      end
+      
+    end
+  end
+end
+

data/lib/imw/archives/targz.rb

@@ -0,0 +1,73 @@
+module IMW
+  module Archives
+    module Targz
+
+      #
+      # It's a compressed file
+      #
+
+      include IMW::CompressedFiles::Base
+
+      def compression_settings
+        @compression_settings ||= {
+          :program               => :gzip,
+          :decompression_program => :gunzip,
+          :decompress            => '',
+          :extension             => 'gz'
+        }
+      end
+
+      #
+      # But it's also an archive
+      #
+      
+      include IMW::Archives::Base        
+      
+      def archive_settings
+        @archive_settings ||= {
+          :program               => :tar,        
+          :list                  => "-tzf",
+          :create                => '-cf',            
+          :extract               => "-xzf"
+        }
+      end
+
+      # Overrides default behvaior of IMW::Files::Archive#create to
+      # compress files after creating them.
+      def create *input_paths
+        IMW.system(archive_settings[:program], archive_settings[:create].split, path_between_archive_and_compression, *input_paths.flatten)
+        tar = IMW.open(path_between_archive_and_compression)
+        tar.compression_settings = compression_settings
+        tar.compress!
+      end
+
+      def decompressed_basename
+        case extname
+        when '.tar.gz' then basename[0..-4]              # .tar.gz => .tar
+        when '.tgz'    then basename.gsub(/tgz$/, 'tar') # .tgz    => .tar
+        else                basename[0..-(extname.size + 1)]
+        end
+      end
+      
+      protected
+      def path_between_archive_and_compression
+        File.join(dirname,name + '.tar')
+      end
+      public
+
+      #
+      # It's both an archive and a compressed file!
+      #
+
+      def extname
+        case path
+        when /\.tar\.gz$/ then '.tar.gz'
+        when /\.tgz$/     then '.tgz'
+        else              File.extname(path)
+        end
+      end
+
+    end
+  end
+end
+

data/lib/imw/archives/zip.rb

@@ -0,0 +1,51 @@
+module IMW
+  module Archives
+    module Zip
+
+      include IMW::Archives::Base
+
+      def archive_settings
+        @archive_settings ||= {
+          :program             => :zip,
+          :create              => "-qqr",
+          :append              => "-qqg",
+          :list                => "-l",
+          :extract             => "-qqo",
+          :unarchiving_program => :unzip
+        }
+      end
+
+      protected
+      
+      # The `unzip' program outputs data in a very annoying format:
+      #
+      #     Archive:  data.zip
+      #       Length     Date   Time    Name
+      #      --------    ----   ----    ----
+      #         18510  07-28-08 15:58   data/4d7Qrgz7.csv
+      #          3418  07-28-08 15:41   data/7S.csv
+      #         23353  07-28-08 15:41   data/g.csv
+      #           711  07-28-08 15:58   data/g.xml
+      #          1095  07-28-08 15:41   data/L.xml
+      #          2399  07-28-08 15:58   data/mTAu9H3.xml
+      #           152  07-28-08 15:58   data/vaHBS2t5R.dat
+      #      --------                   -------
+      #         49638                   7 files            
+      #
+      # which is parsed by this method.
+      def archive_contents_string_to_array string
+        rows = string.split("\n")
+        # ignore the first 3 lines of the output and also discared the
+        # last 2 (5 = 2 + 3)
+        file_rows = rows[3,(rows.length - 5)]
+        file_rows.map do |row|
+          if row
+            columns = row.lstrip.rstrip.split(' ')
+            # grab the filename in the fourth column
+            columns[3..-1].join(' ')
+          end
+        end.compact
+      end
+    end
+  end
+end

data/lib/imw/{resources/compressed_file.rb → compressed_files.rb}

@@ -1,10 +1,19 @@
 module IMW
-  module Resources
-    
-    module CompressedFiles
-      autoload :Bz2, 'imw/resources/archives_and_compressed/bz2'
-      autoload :Gz, 'imw/resources/archives_and_compressed/gz'
-    end
+
+  # Contains modules which define the behavior of compressed files.
+  module CompressedFiles
+    autoload :Bz2,          'imw/compressed_files/bz2'
+    autoload :Gz,           'imw/compressed_files/gz'
+    autoload :Compressible, 'imw/compressed_files/compressible'
+
+    # Handlers which include modules for compressed file formats as
+    # well as the IMW::CompressedFiles::Compressible module for
+    # compressing regular files.
+    HANDLERS = [
+                ["CompressedFiles::Compressible", Proc.new { |r| r.is_local? && r.is_file? && r.path != /\.(bz2|gz|tgz|tbz2)$/                       } ],
+                ["CompressedFiles::Gz",           Proc.new { |r| r.is_local? && r.path =~ /\.gz$/  && r.path !~ /\.tar\.gz$/  && r.path !~ /\.tgz$/  } ],
+                ["CompressedFiles::Bz2",          Proc.new { |r| r.is_local? && r.path =~ /\.bz2$/ && r.path !~ /\.tar\.bz2$/ && r.path !~ /\.tbz2$/ } ]    
+               ]
 
     # Defines methods for decompressing a compressed file.  This
     # module isn't used to directly extend an IMW::Resource --
@@ -12,7 +21,7 @@ module IMW
     # IMW::Resources::CompressedFiles::Bz2) include this module and
     # further define the command-line flags &c. needed to make
     # everything work.
-    module CompressedFile
+    module Base
 
       attr_accessor :compression_settings
 
@@ -80,10 +89,6 @@ module IMW
           copy.mv(path) if copy && copy.exist?
         end
       end
-
     end
   end
 end
-
-
-

data/lib/imw/compressed_files/bz2.rb

@@ -0,0 +1,16 @@
+module IMW
+  module CompressedFiles
+    module Bz2
+
+      include IMW::CompressedFiles::Base
+      
+      def compression_settings
+        @compression_settings ||= {
+          :decompression_program    => :bzip2,
+          :decompress               => '-fd'
+        }
+      end
+      
+    end
+  end
+end