vos 0.0.4 → 0.1.0

data/readme.md

@@ -1,21 +1,122 @@
-# Rsh - Tiny wrapper over Net::SSH/SFTP
+# Vfs - Virtual File System
 
-Because they are too hard to use and have terrible API design.
+Handy and simple abstraction over any storage that can represent concept of File and Directory (or at least part of it). 
+The Vfs for File System Storages is the same as ActiveRecord is for Relational Databases.
 
-    box = Rsh::Box.new host: 'webapp.com', ssh: {user: 'root', password: 'secret'}
+Currently, there are following implementations available:
+
+- local file system
+- remote file system (over ssh)
+
+## Goals
+
+- **handy, simple and clean** API.
+- **high performance** - the same as by using low-level storage API, there should be no extra calls **.
+- same API for different storages (Local FS, SSH, Hadoop, or any other , ...).
+- should work **simultaneously with different storages**.
+- small codebase, easy to extend by others.
+- simple storage-driver implementation, easy add new storage types (Hadoop DFS, LDAP, Document Oriented DB, In-Memory, ...).
+
+** all methods should have the same performance as native system calls, except for :move and :rename. Right now they are implemented 
+ASAP by using copy+destroy approach, will be fixed as soon as I'll have free time to do it.
+
+## Installation
+
+    $ gem install vfs
+    $ gem install vos
+
+## Code samples:
+    gem 'vfs'                                    # Virtual File System
+    require 'vfs'                              
+
+    gem 'vos'                                    # Virtual Operating System
+    require 'vos'
+
+
+    # Connections, let's deploy our 'cool_app' project from our local box to remote server
+    server = Vfs::Box.new(host: 'cool_app.com', ssh: {user: 'me', password: 'secret'})
+    me = '~'.to_dir
+
+    cool_app = server['apps/cool_app']
+    projects = me['projects']
+
+
+    # Working with dirs, copying dir from any source to any destination (local/remote/custom_storage_type)
+    projects['cool_app'].copy_to cool_app        
+
+
+    # Working with files
+    dbc = cool_app.file('config/database.yml')   # <= the 'config' dir not exist yet
+    dbc.write("user: root\npassword: secret")    # <= now the 'database.yml' and parent 'config' has been created
+    dbc.content =~ /database/                    # => false, we forgot to add the database
+    dbc.append("\ndatabase: mysql")              # let's do it
+
+    dbc.update do |content|                      # and add host info
+      content + "\nhost: cool_app.com "
+    end                                       
+
+    projects['cool_app/config/database.yml'].    # or just overwrite it with our local dev version
+      copy_to! dbc
+      
+    # there are also streaming support (read/write/append), please go to specs for docs
 
-    box.upload_directory '/my_project', '/apps/my_project'
-    box.bash 'nohup /apps/my_project/server_start'
-  
-Honestly my wrapper also not very good. I would like to make API looks like the ['rush'][rush] gem (made by Adam Wiggins)
-but it requires a lots of time, maybe I'll do it later.
-So, for now it's just a small wrapper to do ssh/io operations not so painfull.
 
+    # Checks
+    cool_app['config'].exist?                    # => true
+    cool_app.dir('config').exist?                # => true
+    cool_app.file('config').exist?               # => false
+
+    cool_app['config'].dir?                      # => true
+    cool_app['config'].file?                     # => false
+
+
+    # Navigation
+    config = cool_app['config']
+    config.parent                                # => </apps/cool_app>
+    config['../..']                              # => </>
+    config['../..'].dir?                         # => true
+
+    cool_app.entries                             # => list of dirs and files, also support &block
+    cool_app.files                               # => list of files, also support &block
+    cool_app.dirs                                # => list of dirs, also support &block
+
+
+    # For more please go to specs (create/update/move/copy/destroy/...)
+      
+## Integration with [Vos][vos] (Virtual Operating System)
+    
+    server['apps/cool_app'].bash 'rails production'
+
+For more details please go to [Vos][vos] project page.
+
+# Why?
+
+To easy my work: with local FS, remote FS (cluster management, deployment automation), and some specific systems like Hadoop DFS.
+
+Because the API of standard File/Dir/FileUtils classes are just terrible. And there's the reason for it - the goal of thouse tools
+is to provide 1-to-1 clone of underlying OS API, instead of provididing handy tool.
+
+And if you want to use remote FS - things are getting even worse and more complicated (Net::SSH & Net::SFTP use a little
+different API than local FS, and you has to remember all thouse little quirks).
+  
 ## TODO
 
-- remove ssh.remote and use only open/close
-- introduce Entity/Dir/File (the same as in Rush)
-- allow to move files between any Boxes, not only between local and remote.
-- add support for moving dirs.
+### v 0.1 (all done)
+
+- bash & basic support for 
+- File.append
+- list of entries/files/dirs
+- support for efficient copy for Local and SSH storages
+
+### v 0.2 (not started)
+
+- efficient (not copy/destroy) versions of move_to, rename
+- glob search for directories: Dir['**/*.yml']
+- access via attributes and helpers for unix chmod
+- add storages: remote FS over HTTP.
+
+### future
+
+- add storages: Hadoop DFS, MongoDB, Amazon S3
 
-[rush]: http://github.com/adamwiggins/rush
+[vfs]: http://github.com/alexeypetrushin/vfs

data/spec/box_spec.rb

@@ -1,109 +1,56 @@
 require 'spec_helper'
 
-describe Rsh::Box do  
-  with_tmp_spec_dir before: :each
-  
+describe Vos::Box do  
   before :each do
-    @box = Rsh::Box.new
-
-    @local_dir = spec_dir    
-    @remote_dir = @box.generate_tmp_dir_name        
-    
-    @box.remove_directory @remote_dir if @box.directory_exist? @remote_dir
-    @box.create_directory @remote_dir
-  end
-
-  after :each do
-    @box.remove_directory @remote_dir if @box.directory_exist? @remote_dir
+    @box = Vos::Box.new
+    @box.stub :puts
   end
   
-  describe "io" do
-    describe "files" do
-      before :each do
-        @local_file = "#{@local_dir}/local_file"
-        @check_file = "#{@local_dir}/check_file"
-        @remote_file = "#{@remote_dir}/remote_file"
-      end
-      
-      it "file_exist?" do
-        @box.file_exist?(@remote_file).should be_false
-        @box.upload_file(@local_file, @remote_file)
-        @box.file_exist?(@remote_file).should be_true
-      end
-
-      it "upload_file" do
-        @box.upload_file(@local_file, @remote_file)
-        @box.file_exist?(@remote_file).should be_true
-
-        lambda{@box.upload_file(@local_file, @remote_file)}.should raise_error(/exists/)
-
-        # upload with override
-        @box.upload_file(@local_file, @remote_file, override: true)
-        @box.file_exist?(@remote_file).should be_true
-      end
-
-      it "download_file" do
-        lambda{@box.download_file(@remote_file, @check_file)}.should raise_error(/not exists/)
-        @box.upload_file(@local_file, @remote_file)
-        @box.download_file(@remote_file, @check_file)
-        File.read(@local_file).should == File.read(@check_file)
-      end          
-
-      it "remove_file" do
-        lambda{@box.remove_file(@remote_file)}.should raise_error(/not exists/)
-        @box.upload_file(@local_file, @remote_file)
-        @box.file_exist?(@remote_file).should be_true
-        @box.remove_file(@remote_file)
-        @box.file_exist?(@remote_file).should be_false
-      end
+  describe 'vfs integration' do
+    it 'smoke test' do
+      @box['/'].exist?.should be_true      
     end
     
-    describe 'directories' do
-      before :each do 
-        @from_local, @remote_path, @to_local = "#{@local_dir}/dir", "#{@remote_dir}/upload", "#{@local_dir}/download"
-      end
-      
-      it "directory_exist?" do
-        @box.file_exist?(@remote_path).should be_false
-        @box.upload_directory(@from_local, @remote_path)
-        @box.file_exist?(@remote_path).should be_true
-      end
-
-      it "upload_directory" do
-        @box.upload_directory(@from_local, @remote_path)
-        @box.directory_exist?(@remote_path).should be_true
-    
-        lambda{@box.upload_directory(@from_local, @remote_path)}.should raise_error(/exists/)
-    
-        # upload with override
-        @box.upload_directory(@from_local, @remote_path, override: true)
-        @box.directory_exist?(@remote_path).should be_true
-      end
-    
-      it "download_directory" do
-        lambda{@box.download_directory(@remote_path, @to_local)}.should raise_error(/not exists/)
-        @box.upload_directory(@from_local, @remote_path)
-        @box.download_directory(@remote_path, @to_local)
-        File.exist?("#{@to_local}/dir2/file").should be_true
-      end          
-    
-      it "remove_directory" do
-        lambda{@box.remove_directory(@remote_path)}.should raise_error(/not exists/)
-        @box.upload_directory(@from_local, @remote_path)
-        @box.directory_exist?(@remote_path).should be_true
-        @box.remove_directory(@remote_path)
-        @box.directory_exist?(@remote_path).should be_false
-      end    
+    it 'vfs integration' do
+      @box['/'].bash("echo 'ok'").should == "ok\n"
     end
   end
   
   describe "shell" do
     it 'bash' do
       @box.bash("echo 'ok'").should == "ok\n"
-    end  
+    end
+    
+    it 'bash working dir should be /' do
+      @box.bash('pwd').should == "/\n"
+    end
+    
+    it 'check with regex' do
+      @box.bash "echo 'ok'", /ok/
+      -> {@box.bash "echo 'ok'", /no/}.should raise_error(/not match/)
+    end
     
     it "exec" do
       @box.exec("echo 'ok'").should == [0, "ok\n", ""]
     end
+    
+    it 'home' do
+      @box.home.should_not be_nil
+    end
+    
+    it 'env' do
+      @box.env.should == {}
+      @box.env = {a: 'b'}
+      
+      @box.env c: 'd' do
+        @box.env.should == {a: 'b', c: 'd'}
+      end
+      @box.env.should == {a: 'b'}
+      
+      @box.env(c: 'd')
+      @box.env.should == {a: 'b', c: 'd'}
+      
+      @box.env('ls').should == "a=b c=d && ls"
+    end
   end
 end

data/spec/drivers/local_spec.rb

@@ -0,0 +1,10 @@
+require 'drivers/spec_helper'
+
+describe Vos::Drivers::Local do
+  it_should_behave_like "vos driver"
+  it_should_behave_like "vfs storage"  
+  
+  before :each do
+    @storage = @driver = Vos::Drivers::Local.new
+  end
+end

data/spec/drivers/spec_helper.rb

@@ -0,0 +1,5 @@
+require 'spec_helper'
+require 'yaml'
+
+require 'vfs/storages/specification'
+require 'vos/drivers/specification'

data/spec/drivers/ssh_spec.rb

@@ -0,0 +1,15 @@
+require 'drivers/spec_helper'
+
+describe Vos::Drivers::Ssh do
+  it_should_behave_like "vos driver"    
+  it_should_behave_like "vfs storage"
+
+  before :all do
+    @storage = @driver = Vos::Drivers::Ssh.new(config[:remote_driver])
+    @driver.open
+  end
+  
+  after :all do
+    @driver.close
+  end  
+end

data/spec/spec_helper.rb

@@ -1,5 +1,6 @@
 require 'rspec_ext'
-require 'rsh'
+require 'ruby_ext'
+require 'vos'
 
 rspec do
   def config

metadata

@@ -4,9 +4,9 @@ version: !ruby/object:Gem::Version
   prerelease: false
   segments: 
   - 0
+  - 1
   - 0
-  - 4
-  version: 0.0.4
+  version: 0.1.0
 platform: ruby
 authors: 
 - Alexey Petrushin
@@ -14,10 +14,22 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-02-02 00:00:00 +03:00
+date: 2011-02-08 00:00:00 +03:00
 default_executable: 
-dependencies: []
-
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: vfs
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 0
+        version: "0"
+  type: :runtime
+  version_requirements: *id001
 description: 
 email: 
 executables: []
@@ -30,25 +42,25 @@ files:
 - Rakefile
 - readme.md
 - lib/old/ssh.rb
-- lib/rsh/box/marks.rb
-- lib/rsh/box.rb
-- lib/rsh/drivers/abstract.rb
-- lib/rsh/drivers/local.rb
-- lib/rsh/drivers/ssh.rb
-- lib/rsh/gems.rb
-- lib/rsh/support.rb
-- lib/rsh.rb
-- spec/abstract_driver/dir/dir2/file
-- spec/abstract_driver/local_file
-- spec/abstract_driver.rb
-- spec/box_spec/dir/dir2/file
-- spec/box_spec/local_file
+- lib/vos/box/marks.rb
+- lib/vos/box/shell.rb
+- lib/vos/box/vfs.rb
+- lib/vos/box.rb
+- lib/vos/drivers/abstract.rb
+- lib/vos/drivers/local.rb
+- lib/vos/drivers/specification.rb
+- lib/vos/drivers/ssh.rb
+- lib/vos/gems.rb
+- lib/vos/helpers/ubuntu.rb
+- lib/vos/support.rb
+- lib/vos.rb
 - spec/box_spec.rb
 - spec/config.example.yml
 - spec/config.yml
-- spec/local_driver_spec.rb
+- spec/drivers/local_spec.rb
+- spec/drivers/spec_helper.rb
+- spec/drivers/ssh_spec.rb
 - spec/spec_helper.rb
-- spec/ssh_driver_spec.rb
 has_rdoc: true
 homepage: http://github.com/alexeypetrushin/vos
 licenses: []
@@ -80,6 +92,6 @@ rubyforge_project:
 rubygems_version: 1.3.7
 signing_key: 
 specification_version: 3
-summary: Tiny wrapper over Net::SSH/SFTP + small rake addon for cluster configuration management
+summary: Virtual Operating System
 test_files: []
 

data/lib/rsh/box/marks.rb

@@ -1,29 +0,0 @@
-module Rsh
-  module Marks
-    def mark key
-      ensure_mark_requrements!
-      bash "touch #{marks_dir}/#{key}"
-    end
-
-    def has_mark? key
-      ensure_mark_requrements!
-      file_exist? "#{marks_dir}/#{key}"
-    end
-    
-    def clear_marks
-      bash "rm -r #{marks_dir}"
-    end
-    
-    protected
-      def marks_dir
-        home "/.marks"
-      end
-
-      def ensure_mark_requrements!
-        unless @ensure_mark_requrements
-          create_directory marks_dir unless directory_exist? marks_dir
-          @ensure_mark_requrements = true
-        end
-      end
-  end
-end

data/lib/rsh/box.rb

@@ -1,182 +0,0 @@
-module Rsh
-  class Box
-    include Marks
-    
-    attr_accessor :options
-    
-    def initialize options = {}
-      @options = options
-      options[:host] ||= 'localhost'
-    end
-    
-    def driver
-      unless @driver
-        klass = options[:host] == 'localhost' ? Drivers::Local : Drivers::Ssh
-        @driver = klass.new options
-      end
-      @driver
-    end
-    
-    def local_driver
-      @local_driver ||= Drivers::Local.new
-    end
-    
-    def bulk &b
-      driver.bulk &b
-    end
-    
-    def upload_file from_local_path, to_remote_path, options = {}
-      bulk do
-        raise "file '#{from_local_path}' not exists!" unless local_driver.file_exist? from_local_path
-        if driver.file_exist?(to_remote_path)
-          if options[:override]
-            driver.remove_file to_remote_path
-          else
-            raise "file '#{to_remote_path}' already exists!"
-          end
-        end        
-        driver.upload_file from_local_path, to_remote_path
-      end
-    end
-    
-    def download_file from_remote_path, to_local_path, options = {}
-      bulk do
-        raise "file #{from_remote_path} not exists!" unless driver.file_exist?(from_remote_path)
-        if local_driver.file_exist? to_local_path
-          if options[:override]
-            local_driver.remove_file to_local_path
-          else
-            raise "file #{to_local_path} already exists!" 
-          end
-        end
-        driver.download_file from_remote_path, to_local_path
-      end
-    end
-    
-    def remove_file remote_file_path, options = {}
-      bulk do
-        if driver.file_exist? remote_file_path
-          driver.remove_file remote_file_path
-        else
-          raise "file #{remote_file_path} not exists!" unless options[:silent]
-        end        
-      end
-    end
-    
-    def exist? remote_path
-      driver.exist? remote_path
-    end
-    
-    def file_exist? remote_file_path
-      driver.file_exist? remote_file_path
-    end
-    
-    def remove_directory remote_directory_path, options = {}
-      bulk do
-        if driver.directory_exist? remote_directory_path
-          driver.remove_directory remote_directory_path
-        else
-          raise "directory #{remote_directory_path} not exists!" unless options[:silent]
-        end
-      end
-    end
-    
-    def create_directory remote_path, options = {}
-      bulk do
-        if driver.directory_exist?(remote_path)
-          if options[:override]
-            driver.remove_directory remote_path
-            driver.create_directory remote_path
-          elsif options[:silent]
-            # do nothing
-          else
-            raise "directory '#{remote_path}' already exists!"
-          end
-        else
-          driver.create_directory remote_path
-        end        
-      end
-    end
-    
-    def directory_exist? remote_path
-      driver.directory_exist? remote_path
-    end
-    
-    def upload_directory from_local_path, to_remote_path, options = {}
-      bulk do
-        raise "directory '#{from_local_path}' not exists!" unless local_driver.directory_exist? from_local_path
-        if driver.directory_exist?(to_remote_path)
-          if options[:override]
-            driver.remove_directory to_remote_path
-          else
-            raise "directory '#{to_remote_path}' already exists!"
-          end
-        end        
-        driver.upload_directory from_local_path, to_remote_path
-      end
-    end
-    
-    def download_directory from_remote_path, to_local_path, options = {}
-      bulk do
-        raise "directory #{from_remote_path} not exists!" unless driver.directory_exist?(from_remote_path)
-        if local_driver.directory_exist? to_local_path
-          if options[:override]
-            local_driver.remove_directory to_local_path
-          else
-            raise "directory #{to_local_path} already exists!" 
-          end
-        end
-        driver.download_directory from_remote_path, to_local_path
-      end
-    end
-    
-    def with_tmp_dir &block
-      bulk do
-        tmp_dir = driver.generate_tmp_dir_name
-        begin   
-          remove_directory tmp_dir if directory_exist? tmp_dir
-          create_directory tmp_dir
-          block.call
-        ensure
-          remove_directory tmp_dir if directory_exist? tmp_dir
-        end
-      end
-    end
-    
-    def bash cmd, options = {}
-      ignore_stderr = options.delete :ignore_stderr
-      raise "invalid options :#{options.keys.join(', :')}" unless options.empty?
-      
-      code, stdout, stderr = exec cmd
-      unless code == 0
-        puts stdout
-        puts stderr
-        raise "can't execute '#{cmd}'!" 
-      end
-      unless stderr.empty? or ignore_stderr
-        puts stderr
-        raise "stderr not empty for '#{cmd}'!"
-      end
-      stdout + stderr
-    end
-    
-    def exec cmd
-      driver.exec cmd
-    end
-    
-    def home path = nil
-      @home ||= bash('cd ~; pwd').gsub("\n", '')    
-      "#{@home}#{path}"
-    end    
-    
-    def inspect
-      "<Box: #{options[:host]}>"
-    end
-    alias_method :to_s, :inspect
-
-    protected
-      def method_missing m, *a, &b
-        driver.send m, *a, &b
-      end
-  end
-end

data/lib/rsh/drivers/local.rb

@@ -1,48 +0,0 @@
-module Rsh
-  module Drivers
-    class Local < Abstract    
-      def upload_file from_local_path, to_remote_path
-        FileUtils.copy from_local_path, to_remote_path
-      end
-    
-      def download_file from_remote_path, to_local_path
-        FileUtils.copy from_remote_path, to_local_path
-      end
-    
-      def exist? remote_file_path
-        File.exist? remote_file_path
-      end
-    
-      alias_method :directory_exist?, :exist?
-      alias_method :file_exist?, :exist?
-    
-      def remove_file remote_file_path
-        File.delete remote_file_path
-      end    
-    
-      def create_directory path
-        Dir.mkdir path
-      end
-    
-      def remove_directory path
-        FileUtils.rm_r path
-      end
-      
-      def upload_directory from_local_path, to_remote_path
-        FileUtils.cp_r from_local_path, to_remote_path
-      end
-      
-      def download_directory from_remote_path, to_local_path
-        FileUtils.cp_r from_remote_path, to_local_path
-      end
-    
-      def exec command        
-        code, stdout, stderr = Open3.popen3 command do |stdin, stdout, stderr, waitth|  
-          [waitth.value.to_i, stdout.read, stderr.read]
-        end
-      
-        return code, stdout, stderr
-      end
-    end
-  end
-end