file_pool 0.2.0

data/README.md

@@ -0,0 +1,90 @@
+# FilePool
+
+FilePool helps to manage a large number of files in a Ruby project. It
+takes care of the storage of files in a balanced directory tree and
+generates unique identifiers for all files. It also comes in handy
+when dealing with only a few files.
+
+FilePool does not deal with file meta information. It's only purpose
+is to return a file's location given a file identifier, which was
+generated when the file was added to the pool.
+
+The identifiers are strings of UUID Type 4 (random), which are also
+used as file names. The directory tree is a 3 level structure using
+the 3 first hexadecimal digits of a UUID as path. For example:
+
+    0/d/6/0d6f8dd9-8deb-4500-bb85-2d0796241963
+    0/c/f/0cfb082a-fd57-490c-978b-e47d5948bc8b
+    6/1/d/61ddfe33-13f3-4f71-9234-5fbbf5c4fc2c
+
+FilePool is tested with Ruby 1.8.7 and 1.9.3.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'file_pool'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install file_pool
+
+## Usage
+
+### Setup
+The root path and optionally a Logger must be defined:
+
+    FilePool.setup '/var/lib/files'
+
+In a Rails project the file pool setup would be placed in an intializer:
+
+    config/initializers/file_pool.
+
+### Example Usage
+
+Adding files (perhaps after completed upload)
+
+    fid = FilePool.add('/Temp/p348dvhn4')
+
+Get location of previously added file
+
+    path = FilePool.path(fid)
+
+Remove a file
+
+    FilePool.remove(fid)
+
+### Maintenance
+
+FilePool has a straight forward way of storing files. It doesn't use
+any form of index. As long as you stick to directory structure
+outlined above you can:
+
+* move the entire pool somewhere else
+* split the pool using symbolic links or mount points to remote file systems
+* merge file pools by copying them into one
+
+There is no risk of overwriting, because UUID type 4 file names are
+unique. (up to an extremely small collision probability).
+
+### Notes
+
+Make sure to store the generated file identifiers safely. There is no
+way of identifying a file again when it's ID is lost. In doubt generate a hash
+value from the file and store it somewhere else.
+
+For large files the pool root should be on the same file system as the files
+added to the pool. Then adding a file returns immediately. Otherwise
+files will be copied which may take a significant time.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/lib/file_pool/version.rb

@@ -0,0 +1,3 @@
+module FilePool
+  VERSION = "0.2.0"
+end

data/lib/file_pool.rb

@@ -0,0 +1,253 @@
+# -*- coding: utf-8 -*-
+require 'file_pool/version'
+require 'uuidtools'
+=begin
+<em>Robert Anniés (2012)</em>
+
+== Introduction
+
+FilePool helps to manage a large number of files in a Ruby project. It
+takes care of the storage of files in a balanced directory tree and
+generates unique identifiers for all files. It also comes in handy
+when delaing with only a few files.
+
+FilePool does not deal with file meta information. It's only purpose
+is to return a file's location given a file identifier, which was
+generated when the file was added to the pool.
+
+The identifiers are strings of UUID Type 4 (random), which are also
+used as file names. The directory tree is a 3 level structure using
+the 3 first hexadecimal digits of a UUID as path. For example:
+
+  0/d/6/0d6f8dd9-8deb-4500-bb85-2d0796241963
+  0/c/f/0cfb082a-fd57-490c-978b-e47d5948bc8b
+  6/1/d/61ddfe33-13f3-4f71-9234-5fbbf5c4fc2c
+
+== Examples
+
+=== Setup
+The root path and optionally a Logger must be defined:
+
+  FilePool.setup '/var/lib/files'
+
+In a Rails project the file pool setup should be placed in an intializer:
+
+  config/initializers/file_pool.rb
+
+=== Usage
+
+Adding files (perhaps after completed upload)
+
+  fid = FilePool.add('/Temp/p348dvhn4')
+
+Get location of previously added file
+
+  path = FilePool.path(fid)
+
+Remove a file
+
+  FilePool.remove(fid)
+
+== Maintenance
+
+FilePool has a straight forward way of storing files. It doesn't use
+any form of index. As long as you stick to directory structure
+outlined above you can:
+
+* move the entire pool somewhere else
+* split the pool using symbolic links or mount points to remote file systems
+* merge file pools by copying them into one
+
+There is no risk of overwriting, because UUID type 4 file names are
+unique. (up to an extremely small collision probability).
+
+== Notes
+
+Make sure to store the generated file identifiers safely. There is no
+way of identifying a file again when it's ID is lost. In doubt generate a hash
+value from the file and store it somewhere else.
+
+For large files the pool root should be on the same file system as the files
+added to the pool. Then adding a file returns immediately. Otherwise
+files will be copied which may take a significant time.
+
+=end
+module FilePool
+
+  class InvalidFileId < Exception; end
+
+  #
+  # Setup the root directory of the file pool root and specify where
+  # to write log messages
+  #
+  # === Parameters:
+  #
+  # root (String)::
+  #   absolute path of the file pool's root directory under which all files will be stored.
+  def self.setup root
+    @@root = root
+  end
+
+  #
+  # Add a file to the file pool.
+  #
+  # Creates hard-links (ln) when file at +path+ is on same file system as
+  # pool, otherwise copies it. When dealing with large files the
+  # latter should be avoided, because it takes more time and space.
+  #
+  # Throws standard file exceptions when unable to store the file. See
+  # also FilePool.add to avoid it.
+  #
+  # === Parameters:
+  #
+  # path (String)::
+  #   path of the file to add.
+  #
+  # === Return Value:
+  #
+  # :: *String* containing a new unique ID for the file added.
+  def self.add! path
+    newid = uuid
+    target = path newid
+
+    FileUtils.mkpath(id2dir newid)
+    FileUtils.link(path, target)
+
+    return newid
+
+  rescue Errno::EXDEV
+    FileUtils.copy(path, target)
+    return newid
+  end
+
+  #
+  # Add a file to the file pool.
+  #
+  # Same as FilePool.add!, but doesn't throw exceptions.
+  #
+  # === Parameters:
+  #
+  # source (String)::
+  #   path of the file to add.
+  #
+  # === Return Value:
+  #
+  # :: *String* containing a new unique ID for the file added.
+  # :: +false+ when the file could not be stored.
+  def self.add path
+    self.add!(path)
+
+  rescue Exception => ex
+    return false
+  end
+
+  #
+  # Return the path of a previously added file by its ID.
+  #
+  # === Parameters:
+  #
+  # fid (String)::
+  #   File ID which was generated by a previous #add operation.
+  #
+  # === Return Value:
+  #
+  # :: *String*, absolute path of the file in the pool.
+  def self.path fid
+    raise InvalidFileId unless valid?(fid)
+    id2dir(fid) + "/#{fid}"
+  end
+
+  #
+  # Remove a previously added file by its ID. Same as FilePool.remove,
+  # but throws exceptions on failure.
+  #
+  # === Parameters:
+  #
+  # fid (String)::
+  #   File ID which was generated by a previous #add operation.
+  def self.remove! fid
+    FileUtils.rm path(fid)
+  end
+
+  #
+  # Remove a previously added file by its ID. Same as FilePool.remove!, but
+  # doesn't throw exceptions.
+  #
+  # === Parameters:
+  #
+  # fid (String)::
+  #   File ID which was generated by a previous #add operation.
+  #
+  # === Return Value:
+  #
+  # :: *Boolean*, +true+ if file was removed successfully, +false+ else
+  def self.remove fid
+    self.remove! fid
+  rescue Exception => ex
+    return false
+  end
+
+  #
+  # Returns some statistics about the current pool. (It may be slow if
+  # the pool contains very many files as it computes them from scratch.)
+  #
+  # === Return Value
+  #
+  # :: *Hash* with keys
+  #   :total_number (Integer)::
+  #     Number of files in pool
+  #   :total_size (Integer)::
+  #     Total number of bytes of all files
+  #   :median_size (Float)::
+  #     Median of file sizes (most frequent size)
+  #   :last_add (Time)::
+  #     Time and Date of last add operation
+
+  def self.stat
+    all_files = Dir.glob("#{root}/*/*/*/*")
+    all_stats = all_files.map{|f| File.stat(f) }
+
+    {
+      :total_size => all_stats.inject(0){|sum,stat| sum+=stat.size},
+      :median_size => median(all_stats.map{|stat| stat.size}),
+      :file_number => all_files.length,
+      :last_add => all_stats.map{|stat| stat.ctime}.max
+    }
+  end
+
+
+  private
+
+  def self.root
+    @@root rescue raise("FilePool: no root directory defined. Use FilePool#setup.")
+  end
+
+  # path from fid without file name
+  def self.id2dir fid
+    "#{root}/#{fid[0,1]}/#{fid[1,1]}/#{fid[2,1]}"
+  end
+
+  # return a new UUID type 4 (random) as String
+  def self.uuid
+    UUIDTools::UUID.random_create.to_s
+  end
+
+  # return +true+ if _uuid_ is a valid UUID type 4
+  def self.valid? uuid
+    begin
+      UUIDTools::UUID.parse(uuid).valid?
+    rescue TypeError, ArgumentError
+      return false
+    end
+  end
+
+  # median file size
+  def self.median(sizes)
+    arr = sizes
+    sortedarr = arr.sort
+    medpt1 = arr.length / 2
+    medpt2 = (arr.length+1)/2
+    (sortedarr[medpt1] + sortedarr[medpt2]).to_f / 2
+  end
+
+end

data/test/test_file_pool.rb

@@ -0,0 +1,77 @@
+require 'rubygems'
+require 'shoulda'
+require 'file_pool'
+
+class FilePoolTest < Test::Unit::TestCase
+
+  def setup
+    @test_dir = "#{File.dirname(__FILE__)}/files"
+    @pool_root = "#{File.dirname(__FILE__)}/fp_root"
+    FilePool.setup @pool_root
+  end
+
+  def teardown
+    FileUtils.rm_r(Dir.glob @pool_root+"/*")
+  end
+
+  context "File Pool" do
+    should "store files" do
+      fid = FilePool.add(@test_dir+"/a")
+
+      assert UUIDTools::UUID.parse(fid).valid?
+
+      md5_orig = Digest::MD5.hexdigest(File.open(@test_dir+"/a").read)
+      md5_pooled = Digest::MD5.hexdigest(File.open(@pool_root + "/#{fid[0,1]}/#{fid[1,1]}/#{fid[2,1]}/#{fid}").read)
+
+      assert_equal md5_orig, md5_pooled
+    end
+
+    should "return path from stored files" do
+
+      fidb = FilePool.add(@test_dir+"/b")
+      fidc = FilePool.add(@test_dir+"/c")
+      fidd = FilePool.add!(@test_dir+"/d")
+
+      assert_equal "#{@pool_root}/#{fidb[0,1]}/#{fidb[1,1]}/#{fidb[2,1]}/#{fidb}", FilePool.path(fidb)
+      assert_equal "#{@pool_root}/#{fidc[0,1]}/#{fidc[1,1]}/#{fidc[2,1]}/#{fidc}", FilePool.path(fidc)
+      assert_equal "#{@pool_root}/#{fidd[0,1]}/#{fidd[1,1]}/#{fidd[2,1]}/#{fidd}", FilePool.path(fidd)
+
+    end
+
+    should "remove files from pool" do
+
+      fidb = FilePool.add(@test_dir+"/b")
+      fidc = FilePool.add!(@test_dir+"/c")
+      fidd = FilePool.add!(@test_dir+"/d")
+
+      path_c = FilePool.path(fidc)
+      FilePool.remove(fidc)
+
+      assert !File.exist?(path_c)
+      assert File.exist?(FilePool.path(fidb))
+      assert File.exist?(FilePool.path(fidd))
+
+    end
+
+    should "throw excceptions when using add! and remove! on failure" do
+      assert_raises(FilePool::InvalidFileId) do
+        FilePool.remove!("invalid-id")
+      end
+
+      assert_raises(Errno::ENOENT) do
+        FilePool.remove!("61e9b2d1-1738-440d-9b3d-e3c64876f2b0")
+      end
+
+      assert_raises(Errno::ENOENT) do
+        FilePool.add!("/not/here/foo.png")
+      end
+
+    end
+
+    should "not throw exceptions when using add and remove on failure" do
+      assert !FilePool.remove("invalid-id")
+      assert !FilePool.remove("61e9b2d1-1738-440d-9b3d-e3c64876f2b0")
+      assert !FilePool.add("/not/here/foo.png")
+    end
+  end
+end

metadata

@@ -0,0 +1,103 @@
+--- !ruby/object:Gem::Specification 
+name: file_pool
+version: !ruby/object:Gem::Version 
+  hash: 23
+  prerelease: 
+  segments: 
+  - 0
+  - 2
+  - 0
+  version: 0.2.0
+platform: ruby
+authors: 
+- "robokopp (Robert Anni\xC3\xA9s)"
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2012-09-17 00:00:00 Z
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: shoulda
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: uuidtools
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 15
+        segments: 
+        - 2
+        - 1
+        - 2
+        version: 2.1.2
+  type: :runtime
+  version_requirements: *id002
+description: |
+  FilePool helps to manage a large number of files in a Ruby
+  project. It takes care of the storage of files in a balanced directory
+  tree and generates unique identifiers for all files.
+
+email: 
+- robokopp@fernwerk.net
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- README.md
+files: 
+- lib/file_pool.rb
+- lib/file_pool/version.rb
+- README.md
+- test/test_file_pool.rb
+homepage: https://github.com/robokopp/file_pool
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Manage a large number files in a pool
+test_files: 
+- test/test_file_pool.rb
+has_rdoc: