boatman 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.document +5 -0
- data/.gitignore +21 -0
- data/LICENSE +20 -0
- data/README.rdoc +166 -0
- data/Rakefile +44 -0
- data/VERSION +1 -0
- data/bin/boatman +11 -0
- data/bin/boatman~ +15 -0
- data/lib/boatman/copyable.rb +25 -0
- data/lib/boatman/ext/class.rb +48 -0
- data/lib/boatman/ext/fixnum.rb +19 -0
- data/lib/boatman/ext/string.rb +10 -0
- data/lib/boatman/monitored_directory.rb +83 -0
- data/lib/boatman/monitored_file.rb +40 -0
- data/lib/boatman.rb +83 -0
- data/spec/boatman_spec.rb +94 -0
- data/spec/data/filename_ending.rb +7 -0
- data/spec/data/filename_ending.yml +5 -0
- data/spec/data/filename_ending_regexp.rb +7 -0
- data/spec/data/filename_ending_regexp.yml +5 -0
- data/spec/data/filename_regexp.rb +7 -0
- data/spec/data/filename_regexp.yml +5 -0
- data/spec/data/folder.rb +7 -0
- data/spec/data/folder.yml +5 -0
- data/spec/data/modify.rb +17 -0
- data/spec/data/modify.yml +5 -0
- data/spec/data/no_destination_folder.rb +7 -0
- data/spec/data/no_destination_folder.yml +5 -0
- data/spec/data/rename.rb +9 -0
- data/spec/data/rename.yml +5 -0
- data/spec/spec.opts +1 -0
- data/spec/spec_helper.rb +11 -0
- metadata +104 -0
data/.document
ADDED
data/.gitignore
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2009 Bruz Marzolf
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.rdoc
ADDED
@@ -0,0 +1,166 @@
|
|
1
|
+
= boatman
|
2
|
+
|
3
|
+
Boatman is a simple Ruby DSL for polling directories and ferrying around / manipulating new files that appear in those folders. It was created as an attempt at something more elegant than having numerous scripts that all do very similar file transfer and manipulation tasks.
|
4
|
+
|
5
|
+
== Install
|
6
|
+
|
7
|
+
Install the boatman gem (assuming you have Ruby and RubyGems):
|
8
|
+
|
9
|
+
gem install boatman
|
10
|
+
|
11
|
+
== Example
|
12
|
+
|
13
|
+
Get a quick feel for what boatman does with this example.
|
14
|
+
|
15
|
+
Create a YAML file to define the task scripts and directories they'll operate on. Let's call it demo.yml:
|
16
|
+
|
17
|
+
tasks:
|
18
|
+
- demo.rb
|
19
|
+
directories:
|
20
|
+
fresh_text_folder: txt_output
|
21
|
+
text_storage_folder: storage
|
22
|
+
|
23
|
+
Now create two folders under the folder where you have demo.yml called "txt_output" and "storage".
|
24
|
+
|
25
|
+
Make a task file demo.rb:
|
26
|
+
|
27
|
+
fresh_text_folder.check_every 5.seconds do
|
28
|
+
age :greater_than => 1.second
|
29
|
+
|
30
|
+
files_ending_with "txt" do |file|
|
31
|
+
move file, :to => text_storage_folder
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
Now run your task:
|
36
|
+
|
37
|
+
boatman demo.yml
|
38
|
+
|
39
|
+
Now, while boatman is running, open another terminal or a file browser and create a file in the txt_output folder you created called "demo.txt". Wait a bit and it should get moved to the "storage" folder you made.
|
40
|
+
|
41
|
+
Hit Ctrl-C to exit out of boatman. You can take a look at the boatman.log file it creates to see a log of what operations have been performed.
|
42
|
+
|
43
|
+
== Creating tasks
|
44
|
+
|
45
|
+
=== Polling folders
|
46
|
+
|
47
|
+
The top level of a boatman task will usually be a directory polling loop. This is accomplished by running the check_every method on a directory specified in your YAML configuration file. The check_every method takes a time interval as its only argument other than a block. Using the example above, running
|
48
|
+
|
49
|
+
fresh_text_folder.check_every 5.seconds do
|
50
|
+
...
|
51
|
+
end
|
52
|
+
|
53
|
+
will run the provided do..end block every 5 seconds in the context of the fresh_text_folder directory.
|
54
|
+
|
55
|
+
=== Specifying file/folder criteria
|
56
|
+
|
57
|
+
It is often desirable to consider just a subset of the files in the folder being polled. Files/folders can be selected by age and file/folder name. Age can be specified inside the polling folders do..end block:
|
58
|
+
|
59
|
+
# age of the file must be greater than 1 minute
|
60
|
+
age :greater_than => 1.minute
|
61
|
+
|
62
|
+
# age of the file must be less than 24 hours
|
63
|
+
age :less_than => 24.hours
|
64
|
+
|
65
|
+
There are a few ways to select based on file name. Each of these methods accepts a block to run on each selected file/folder:
|
66
|
+
|
67
|
+
# select files by a string or regular expression
|
68
|
+
files_matching /\d+\.tif/ do |file|
|
69
|
+
...
|
70
|
+
end
|
71
|
+
|
72
|
+
# select files by a string or regular expression ending
|
73
|
+
files_ending_with "txt" do |file|
|
74
|
+
...
|
75
|
+
end
|
76
|
+
|
77
|
+
# select folders by a string or regular expression
|
78
|
+
folders_matching /\d+/ do |folder|
|
79
|
+
...
|
80
|
+
end
|
81
|
+
|
82
|
+
# select folders by a string or regular expression ending
|
83
|
+
folders_ending_with "log" do |folder|
|
84
|
+
...
|
85
|
+
end
|
86
|
+
|
87
|
+
=== Copying files/folders
|
88
|
+
|
89
|
+
Files/folders can be copied or moved inside the block provided to the file/folder matching methods:
|
90
|
+
|
91
|
+
# move selected files to destination_folder, which needs to be defined in the configuration YAML file.
|
92
|
+
files_ending_with "txt" do |file|
|
93
|
+
move file, :to => destination_folder
|
94
|
+
end
|
95
|
+
|
96
|
+
# same thing but copy the file instead of moving it
|
97
|
+
files_ending_with "txt" do |file|
|
98
|
+
copy file, :to => destination_folder
|
99
|
+
end
|
100
|
+
|
101
|
+
boatman will perform a checksum verification by default in order to catch errors in file transfers. This can be disabled with the disable_checksum_verification directive inside the file/folder matching block:
|
102
|
+
|
103
|
+
files_ending_with "txt" do |file|
|
104
|
+
disable_checksum_verification
|
105
|
+
|
106
|
+
move file, :to => destination_folder
|
107
|
+
end
|
108
|
+
|
109
|
+
Files can optionally be renamed using the :rename parameter for move or copy:
|
110
|
+
|
111
|
+
files_ending_with "txt" do |file|
|
112
|
+
# use the path method on the file
|
113
|
+
new_name = "renamed_" + File.basename(file.path)
|
114
|
+
|
115
|
+
# file will renamed, e.g. test.txt becomes renamed_test.txt
|
116
|
+
move file, :to => destination_folder, :rename => new_name
|
117
|
+
end
|
118
|
+
|
119
|
+
The file being copied/moved can also be modified by passing a block to the copy or move methods. The parameters for the block are the path to read the original file and a the path to write the modified file:
|
120
|
+
|
121
|
+
files_ending_with "txt" do |file|
|
122
|
+
move file, :to => destination_folder do |old_file_name, new_file_name|
|
123
|
+
old_file = File.new(old_file_name, "r")
|
124
|
+
new_file = File.new(new_file_name, "w")
|
125
|
+
|
126
|
+
old_file.readlines.each do |line|
|
127
|
+
new_file << "# #{line}"
|
128
|
+
end
|
129
|
+
end
|
130
|
+
end
|
131
|
+
|
132
|
+
== Configuration Files
|
133
|
+
|
134
|
+
YAML configuration files for boatman consist of two parts, tasks and directories.
|
135
|
+
|
136
|
+
Under tasks you can specify any number of task files to be loaded and run together. Note that boatman will take care of running each task on the interval it specifies, however the tasks are run serially so a long-running task will prevent subsequent tasks from running until it completes:
|
137
|
+
|
138
|
+
# config.yml
|
139
|
+
tasks:
|
140
|
+
- text_file_reformatter.rb
|
141
|
+
- raw_data_transfer.rb
|
142
|
+
directories:
|
143
|
+
...
|
144
|
+
|
145
|
+
Directories allow you to name directories you'd like to have available to your tasks. It is possible to specify both Windows- and POSIX-style paths:
|
146
|
+
|
147
|
+
# Windows-style
|
148
|
+
my_shared_folder: //mycomputer/myshare
|
149
|
+
|
150
|
+
# POSIX-style
|
151
|
+
my_shared_folder: /home/bmarzolf/share
|
152
|
+
|
153
|
+
== Note on Patches/Pull Requests
|
154
|
+
|
155
|
+
* Fork the project.
|
156
|
+
* Make your feature addition or bug fix.
|
157
|
+
* Add tests for it. This is important so I don't break it in a
|
158
|
+
future version unintentionally.
|
159
|
+
* Commit, do not mess with rakefile, version, or history.
|
160
|
+
(if you want to have your own version, that is fine but
|
161
|
+
bump version in a commit by itself I can ignore when I pull)
|
162
|
+
* Send me a pull request. Bonus points for topic branches.
|
163
|
+
|
164
|
+
== Copyright
|
165
|
+
|
166
|
+
Copyright (c) 2009 Bruz Marzolf. See LICENSE for details.
|
data/Rakefile
ADDED
@@ -0,0 +1,44 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'rake'
|
3
|
+
|
4
|
+
begin
|
5
|
+
require 'jeweler'
|
6
|
+
Jeweler::Tasks.new do |gem|
|
7
|
+
gem.name = "boatman"
|
8
|
+
gem.summary = %Q{Ruby DSL for ferrying around and manipulating files}
|
9
|
+
gem.description = %Q{}
|
10
|
+
gem.email = "bmarzolf@systemsbiology.org"
|
11
|
+
gem.homepage = "http://github.com/bmarzolf/boatman"
|
12
|
+
gem.authors = ["Bruz Marzolf"]
|
13
|
+
gem.add_development_dependency "rspec", ">= 1.2.9"
|
14
|
+
# gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
|
15
|
+
end
|
16
|
+
rescue LoadError
|
17
|
+
puts "Jeweler (or a dependency) not available. Install it with: sudo gem install jeweler"
|
18
|
+
end
|
19
|
+
|
20
|
+
require 'spec/rake/spectask'
|
21
|
+
Spec::Rake::SpecTask.new(:spec) do |spec|
|
22
|
+
spec.libs << 'lib' << 'spec'
|
23
|
+
spec.spec_files = FileList['spec/**/*_spec.rb']
|
24
|
+
end
|
25
|
+
|
26
|
+
Spec::Rake::SpecTask.new(:rcov) do |spec|
|
27
|
+
spec.libs << 'lib' << 'spec'
|
28
|
+
spec.pattern = 'spec/**/*_spec.rb'
|
29
|
+
spec.rcov = true
|
30
|
+
end
|
31
|
+
|
32
|
+
task :spec => :check_dependencies
|
33
|
+
|
34
|
+
task :default => :spec
|
35
|
+
|
36
|
+
require 'rake/rdoctask'
|
37
|
+
Rake::RDocTask.new do |rdoc|
|
38
|
+
version = File.exist?('VERSION') ? File.read('VERSION') : ""
|
39
|
+
|
40
|
+
rdoc.rdoc_dir = 'rdoc'
|
41
|
+
rdoc.title = "boatman #{version}"
|
42
|
+
rdoc.rdoc_files.include('README*')
|
43
|
+
rdoc.rdoc_files.include('lib/**/*.rb')
|
44
|
+
end
|
data/VERSION
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
0.1.0
|
data/bin/boatman
ADDED
data/bin/boatman~
ADDED
@@ -0,0 +1,15 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
puts "Starting boatman"
|
4
|
+
|
5
|
+
$LOAD_PATH.unshift(File.dirname(__FILE__) + '/../lib')
|
6
|
+
|
7
|
+
require 'rubygems'
|
8
|
+
require 'boatman'
|
9
|
+
|
10
|
+
puts "Logging to boatman.log"
|
11
|
+
$logger = Logger.new("boatman.log")
|
12
|
+
$logger.level = Logger::INFO
|
13
|
+
|
14
|
+
Boatman.load(ARGV)
|
15
|
+
Boatman.run
|
@@ -0,0 +1,25 @@
|
|
1
|
+
class Boatman
|
2
|
+
module Copyable
|
3
|
+
def copy(file, params, remove_original=false, &block)
|
4
|
+
source_path = file.path
|
5
|
+
base_name = params[:rename] || File.basename(source_path)
|
6
|
+
|
7
|
+
destination_path = File.expand_path(params[:to] + "/" + base_name)
|
8
|
+
|
9
|
+
return if File.exists?(destination_path)
|
10
|
+
|
11
|
+
begin
|
12
|
+
copy_entry(source_path, destination_path, &block)
|
13
|
+
FileUtils.rm_r source_path if remove_original
|
14
|
+
|
15
|
+
Boatman.logger.info "Successfully copied #{source_path} to #{destination_path}"
|
16
|
+
rescue Exception => e
|
17
|
+
Boatman.logger.error e.message
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def move(file, params, &block)
|
22
|
+
copy(file, params, remove_original=true, &block)
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
@@ -0,0 +1,48 @@
|
|
1
|
+
class Class
|
2
|
+
def cattr_reader(*syms)
|
3
|
+
syms.flatten.each do |sym|
|
4
|
+
next if sym.is_a?(Hash)
|
5
|
+
class_eval(<<-EOS, __FILE__, __LINE__)
|
6
|
+
unless defined? @@#{sym}
|
7
|
+
@@#{sym} = nil
|
8
|
+
end
|
9
|
+
|
10
|
+
def self.#{sym}
|
11
|
+
@@#{sym}
|
12
|
+
end
|
13
|
+
|
14
|
+
def #{sym}
|
15
|
+
@@#{sym}
|
16
|
+
end
|
17
|
+
EOS
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def cattr_writer(*syms)
|
22
|
+
options = syms.last.is_a?(Hash) ? syms.pop : {}
|
23
|
+
syms.flatten.each do |sym|
|
24
|
+
class_eval(<<-EOS, __FILE__, __LINE__)
|
25
|
+
unless defined? @@#{sym}
|
26
|
+
@@#{sym} = nil
|
27
|
+
end
|
28
|
+
|
29
|
+
def self.#{sym}=(obj)
|
30
|
+
@@#{sym} = obj
|
31
|
+
end
|
32
|
+
|
33
|
+
#{"
|
34
|
+
def #{sym}=(obj)
|
35
|
+
@@#{sym} = obj
|
36
|
+
end
|
37
|
+
" unless options[:instance_writer] == false }
|
38
|
+
EOS
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
def cattr_accessor(*syms)
|
43
|
+
cattr_reader(*syms)
|
44
|
+
cattr_writer(*syms)
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
|
@@ -0,0 +1,19 @@
|
|
1
|
+
class Fixnum
|
2
|
+
def seconds
|
3
|
+
self
|
4
|
+
end
|
5
|
+
def minutes
|
6
|
+
self * 60
|
7
|
+
end
|
8
|
+
def hours
|
9
|
+
self * 60 * 60
|
10
|
+
end
|
11
|
+
def days
|
12
|
+
self * 60 * 60 * 24
|
13
|
+
end
|
14
|
+
|
15
|
+
alias :second :seconds
|
16
|
+
alias :minute :minutes
|
17
|
+
alias :hour :hours
|
18
|
+
alias :day :days
|
19
|
+
end
|
@@ -0,0 +1,83 @@
|
|
1
|
+
class Boatman
|
2
|
+
class MonitoredDirectory
|
3
|
+
include Copyable
|
4
|
+
|
5
|
+
attr_accessor :path
|
6
|
+
attr_accessor :match_data
|
7
|
+
|
8
|
+
def initialize(path, match_data = nil)
|
9
|
+
@path = path
|
10
|
+
@match_data = match_data
|
11
|
+
end
|
12
|
+
|
13
|
+
def age(params)
|
14
|
+
params.each do |key, value|
|
15
|
+
case key
|
16
|
+
when :greater_than
|
17
|
+
@minimum_age = value
|
18
|
+
when :less_than
|
19
|
+
@maximum_age = value
|
20
|
+
end
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
def entries_matching(entry_pattern, type, &block)
|
25
|
+
@minimum_age ||= false
|
26
|
+
@maximum_age ||= false
|
27
|
+
|
28
|
+
entry_paths = Dir.entries(@path).grep(/#{entry_pattern}/).collect do |name|
|
29
|
+
"#{@path}/#{name}"
|
30
|
+
end
|
31
|
+
|
32
|
+
Boatman.logger.debug "Found #{entry_paths.size} entries in #{@path}"
|
33
|
+
entry_paths.each do |entry_path|
|
34
|
+
next if type == :file && !File.file?(entry_path)
|
35
|
+
next if type == :directory && !File.directory?(entry_path)
|
36
|
+
|
37
|
+
age = Time.now - File.mtime(entry_path)
|
38
|
+
next if @minimum_age && age < @minimum_age
|
39
|
+
next if @maximum_age && age > @maximum_age
|
40
|
+
|
41
|
+
match_data = entry_path.match(entry_pattern) if entry_pattern.is_a?(Regexp)
|
42
|
+
case type
|
43
|
+
when :file
|
44
|
+
entry = MonitoredFile.new(entry_path, match_data)
|
45
|
+
when :directory
|
46
|
+
entry = MonitoredDirectory.new(entry_path, match_data)
|
47
|
+
end
|
48
|
+
|
49
|
+
entry.instance_eval &block
|
50
|
+
end
|
51
|
+
end
|
52
|
+
|
53
|
+
def files_matching(file_pattern, &block)
|
54
|
+
return entries_matching(file_pattern, type = :file, &block)
|
55
|
+
end
|
56
|
+
|
57
|
+
def files_ending_with(file_ending, &block)
|
58
|
+
return files_matching(/#{file_ending}$/, &block)
|
59
|
+
end
|
60
|
+
|
61
|
+
def folders_matching(folder_pattern, &block)
|
62
|
+
return entries_matching(folder_pattern, type = :directory, &block)
|
63
|
+
end
|
64
|
+
|
65
|
+
def folders_ending_with(folder_ending, &block)
|
66
|
+
return folders_matching(/#{folder_ending}$/, &block)
|
67
|
+
end
|
68
|
+
|
69
|
+
private
|
70
|
+
|
71
|
+
def copy_entry(source_path, destination_path, &block)
|
72
|
+
FileUtils.mkdir_p File.dirname(destination_path)
|
73
|
+
FileUtils.cp_r source_path, destination_path
|
74
|
+
|
75
|
+
if block_given?
|
76
|
+
yield destination_path, "#{destination_path}.tmp"
|
77
|
+
FileUtils.cp_r "#{destination_path}.tmp", destination_path
|
78
|
+
FileUtils.rm_r "#{destination_path}.tmp"
|
79
|
+
end
|
80
|
+
end
|
81
|
+
|
82
|
+
end
|
83
|
+
end
|
@@ -0,0 +1,40 @@
|
|
1
|
+
class Boatman
|
2
|
+
class MonitoredFile
|
3
|
+
include Copyable
|
4
|
+
|
5
|
+
attr_accessor :path
|
6
|
+
attr_accessor :match_data
|
7
|
+
|
8
|
+
def initialize(file_path, match_data = nil)
|
9
|
+
@path = file_path
|
10
|
+
@match_data = match_data
|
11
|
+
end
|
12
|
+
|
13
|
+
def disable_checksum_verification
|
14
|
+
@checksum_verification_disabled = true
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def copy_entry(source_path, destination_path, &block)
|
20
|
+
FileUtils.mkdir_p File.dirname(destination_path)
|
21
|
+
FileUtils.cp source_path, destination_path
|
22
|
+
|
23
|
+
unless @checksum_verification_disabled
|
24
|
+
verify_checksum_matches(source_path, destination_path, &block)
|
25
|
+
end
|
26
|
+
|
27
|
+
if block_given?
|
28
|
+
yield destination_path, "#{destination_path}.tmp"
|
29
|
+
FileUtils.cp "#{destination_path}.tmp", destination_path
|
30
|
+
FileUtils.rm "#{destination_path}.tmp"
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
def verify_checksum_matches(file_1, file_2)
|
35
|
+
file_1_digest = Digest::MD5.hexdigest( File.read(file_1) )
|
36
|
+
file_2_digest = Digest::MD5.hexdigest( File.read(file_2) )
|
37
|
+
raise "Checksum verification failed when copying #{base_name}" unless file_1_digest == file_2_digest
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
data/lib/boatman.rb
ADDED
@@ -0,0 +1,83 @@
|
|
1
|
+
require "yaml"
|
2
|
+
require "digest/md5"
|
3
|
+
|
4
|
+
require "boatman/ext/class"
|
5
|
+
require "boatman/ext/string"
|
6
|
+
require "boatman/ext/fixnum"
|
7
|
+
|
8
|
+
require "boatman/copyable"
|
9
|
+
require "boatman/monitored_directory.rb"
|
10
|
+
require "boatman/monitored_file.rb"
|
11
|
+
|
12
|
+
class Boatman
|
13
|
+
cattr_accessor :tasks
|
14
|
+
cattr_accessor :logger
|
15
|
+
|
16
|
+
def self.load(args)
|
17
|
+
if(args.size < 1 || args.size > 2)
|
18
|
+
Boatman.print_usage
|
19
|
+
exit(0)
|
20
|
+
end
|
21
|
+
|
22
|
+
puts "Logging to boatman.log"
|
23
|
+
require 'logger'
|
24
|
+
Boatman.logger = Logger.new("boatman.log")
|
25
|
+
logger.level = Logger::INFO
|
26
|
+
|
27
|
+
@config_file = args[0]
|
28
|
+
|
29
|
+
@working_directory = args[1] || "."
|
30
|
+
Boatman.logger.info "Working directory: #{@working_directory}"
|
31
|
+
|
32
|
+
load_config_file(args[0])
|
33
|
+
end
|
34
|
+
|
35
|
+
def self.load_config_file(file)
|
36
|
+
Boatman.logger.info "Loading Config File: #{@config_file}"
|
37
|
+
config = YAML.load_file(@config_file)
|
38
|
+
|
39
|
+
@task_files = config["tasks"]
|
40
|
+
|
41
|
+
config["directories"].each do |key, value|
|
42
|
+
Object.class_eval do
|
43
|
+
define_method(key) do
|
44
|
+
return value
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
48
|
+
end
|
49
|
+
|
50
|
+
def self.print_usage
|
51
|
+
puts "Usage: boatman <config file> [<working directory>]"
|
52
|
+
end
|
53
|
+
|
54
|
+
def self.run
|
55
|
+
@task_files.each do |task_file|
|
56
|
+
require "#{@working_directory}/#{task_file}"
|
57
|
+
puts "Added task #{task_file}"
|
58
|
+
end
|
59
|
+
|
60
|
+
interrupted = false
|
61
|
+
trap("INT") { interrupted = true }
|
62
|
+
|
63
|
+
while true do
|
64
|
+
if interrupted
|
65
|
+
puts "Exiting from boatman"
|
66
|
+
return
|
67
|
+
end
|
68
|
+
|
69
|
+
tasks.each do |task|
|
70
|
+
if task[:last_run].nil? || Time.now - task[:last_run] > task[:time_interval]
|
71
|
+
# do everything in the context of the working directory
|
72
|
+
Dir.chdir(@working_directory) do
|
73
|
+
task[:directory].instance_eval &task[:block]
|
74
|
+
end
|
75
|
+
|
76
|
+
task[:last_run] = Time.now
|
77
|
+
end
|
78
|
+
end
|
79
|
+
|
80
|
+
sleep 1
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
@@ -0,0 +1,94 @@
|
|
1
|
+
require File.expand_path(File.dirname(__FILE__) + '/spec_helper')
|
2
|
+
|
3
|
+
describe "Moving new files from one location to another" do
|
4
|
+
|
5
|
+
before(:each) do
|
6
|
+
@working_directory = File.expand_path(File.dirname(__FILE__) + '/data')
|
7
|
+
FileUtils.mkdir_p(@working_directory + '/tmp/source')
|
8
|
+
FileUtils.mkdir_p(@working_directory + '/tmp/destination')
|
9
|
+
end
|
10
|
+
|
11
|
+
def run_boatman
|
12
|
+
thread = Thread.new do
|
13
|
+
Boatman.run
|
14
|
+
end
|
15
|
+
sleep 2
|
16
|
+
thread.exit
|
17
|
+
end
|
18
|
+
|
19
|
+
it "should move based on filename ending" do
|
20
|
+
FileUtils.touch(@working_directory + '/tmp/source/datafile.txt')
|
21
|
+
|
22
|
+
boatman = Boatman.load(["#{@working_directory}/filename_ending.yml", @working_directory])
|
23
|
+
run_boatman
|
24
|
+
|
25
|
+
File.exist?(@working_directory + '/tmp/destination/datafile.txt').should be_true
|
26
|
+
end
|
27
|
+
|
28
|
+
it "should create the destination folder if it does not yet exist" do
|
29
|
+
FileUtils.rm_rf(@working_directory + '/tmp/destination')
|
30
|
+
FileUtils.touch(@working_directory + '/tmp/source/datafile.txt')
|
31
|
+
|
32
|
+
boatman = Boatman.load(["#{@working_directory}/no_destination_folder.yml", @working_directory])
|
33
|
+
run_boatman
|
34
|
+
|
35
|
+
File.exist?(@working_directory + '/tmp/destination/datafile.txt').should be_true
|
36
|
+
end
|
37
|
+
|
38
|
+
it "should move based on regular expression match to filename ending" do
|
39
|
+
FileUtils.touch(@working_directory + '/tmp/source/datafile_R.tif')
|
40
|
+
FileUtils.touch(@working_directory + '/tmp/source/datafile.tif')
|
41
|
+
|
42
|
+
boatman = Boatman.load(["#{@working_directory}/filename_ending_regexp.yml", @working_directory])
|
43
|
+
run_boatman
|
44
|
+
|
45
|
+
File.exist?(@working_directory + '/tmp/destination/datafile_R.tif').should be_true
|
46
|
+
File.exist?(@working_directory + '/tmp/destination/datafile.tif').should be_false
|
47
|
+
end
|
48
|
+
|
49
|
+
it "should move and modify a file" do
|
50
|
+
datafile = File.new(@working_directory + '/tmp/source/datafile.txt', "w")
|
51
|
+
datafile << "Some text"
|
52
|
+
datafile.close
|
53
|
+
|
54
|
+
boatman = Boatman.load(["#{@working_directory}/modify.yml", @working_directory])
|
55
|
+
run_boatman
|
56
|
+
|
57
|
+
File.exist?(@working_directory + '/tmp/destination/datafile.txt').should be_true
|
58
|
+
File.new(@working_directory + '/tmp/destination/datafile.txt').readlines.should ==
|
59
|
+
["# Some text"]
|
60
|
+
end
|
61
|
+
|
62
|
+
it "should move and rename a file" do
|
63
|
+
FileUtils.touch(@working_directory + '/tmp/source/datafile.txt')
|
64
|
+
|
65
|
+
boatman = Boatman.load(["#{@working_directory}/rename.yml", @working_directory])
|
66
|
+
run_boatman
|
67
|
+
|
68
|
+
File.exist?(@working_directory + '/tmp/destination/renamed_datafile.txt').should be_true
|
69
|
+
end
|
70
|
+
|
71
|
+
it "should move a file based on a regular expression on the entire filename" do
|
72
|
+
FileUtils.touch(@working_directory + '/tmp/source/datafile.txt')
|
73
|
+
|
74
|
+
boatman = Boatman.load(["#{@working_directory}/filename_regexp.yml", @working_directory])
|
75
|
+
run_boatman
|
76
|
+
|
77
|
+
File.exist?(@working_directory + '/tmp/destination/datafile.txt').should be_true
|
78
|
+
end
|
79
|
+
|
80
|
+
it "should move a folder based on its name" do
|
81
|
+
FileUtils.mkdir(@working_directory + '/tmp/source/bob')
|
82
|
+
|
83
|
+
boatman = Boatman.load(["#{@working_directory}/folder.yml", @working_directory])
|
84
|
+
run_boatman
|
85
|
+
|
86
|
+
File.exist?(@working_directory + '/tmp/destination/bob').should be_true
|
87
|
+
File.directory?(@working_directory + '/tmp/destination/bob').should be_true
|
88
|
+
end
|
89
|
+
|
90
|
+
after(:each) do
|
91
|
+
FileUtils.rm_rf(@working_directory + '/tmp')
|
92
|
+
end
|
93
|
+
|
94
|
+
end
|
data/spec/data/folder.rb
ADDED
data/spec/data/modify.rb
ADDED
@@ -0,0 +1,17 @@
|
|
1
|
+
source_folder.check_every 1.second do
|
2
|
+
age :greater_than => 0.minutes
|
3
|
+
|
4
|
+
files_ending_with "txt" do |file|
|
5
|
+
move file, :to => destination_folder do |old_file_name, new_file_name|
|
6
|
+
old_file = File.new(old_file_name, "r")
|
7
|
+
new_file = File.new(new_file_name, "w")
|
8
|
+
|
9
|
+
old_file.readlines.each do |line|
|
10
|
+
new_file << "# #{line}"
|
11
|
+
end
|
12
|
+
|
13
|
+
old_file.close
|
14
|
+
new_file.close
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
data/spec/data/rename.rb
ADDED
data/spec/spec.opts
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
--color
|
data/spec/spec_helper.rb
ADDED
@@ -0,0 +1,11 @@
|
|
1
|
+
$LOAD_PATH.unshift(File.dirname(__FILE__))
|
2
|
+
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
|
3
|
+
require 'boatman'
|
4
|
+
require 'spec'
|
5
|
+
require 'spec/autorun'
|
6
|
+
|
7
|
+
BOATMAN_BIN = File.expand_path(File.dirname(__FILE__) + '/../bin/boatman')
|
8
|
+
|
9
|
+
Spec::Runner.configure do |config|
|
10
|
+
config.before(:each) { Boatman.tasks = nil }
|
11
|
+
end
|
metadata
ADDED
@@ -0,0 +1,104 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: boatman
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Bruz Marzolf
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
|
12
|
+
date: 2009-12-22 00:00:00 -08:00
|
13
|
+
default_executable:
|
14
|
+
dependencies:
|
15
|
+
- !ruby/object:Gem::Dependency
|
16
|
+
name: rspec
|
17
|
+
type: :development
|
18
|
+
version_requirement:
|
19
|
+
version_requirements: !ruby/object:Gem::Requirement
|
20
|
+
requirements:
|
21
|
+
- - ">="
|
22
|
+
- !ruby/object:Gem::Version
|
23
|
+
version: 1.2.9
|
24
|
+
version:
|
25
|
+
description: ""
|
26
|
+
email: bmarzolf@systemsbiology.org
|
27
|
+
executables:
|
28
|
+
- boatman
|
29
|
+
- boatman~
|
30
|
+
extensions: []
|
31
|
+
|
32
|
+
extra_rdoc_files:
|
33
|
+
- LICENSE
|
34
|
+
- README.rdoc
|
35
|
+
files:
|
36
|
+
- .document
|
37
|
+
- .gitignore
|
38
|
+
- LICENSE
|
39
|
+
- README.rdoc
|
40
|
+
- Rakefile
|
41
|
+
- VERSION
|
42
|
+
- bin/boatman
|
43
|
+
- lib/boatman.rb
|
44
|
+
- lib/boatman/copyable.rb
|
45
|
+
- lib/boatman/ext/class.rb
|
46
|
+
- lib/boatman/ext/fixnum.rb
|
47
|
+
- lib/boatman/ext/string.rb
|
48
|
+
- lib/boatman/monitored_directory.rb
|
49
|
+
- lib/boatman/monitored_file.rb
|
50
|
+
- spec/boatman_spec.rb
|
51
|
+
- spec/data/filename_ending.rb
|
52
|
+
- spec/data/filename_ending.yml
|
53
|
+
- spec/data/filename_ending_regexp.rb
|
54
|
+
- spec/data/filename_ending_regexp.yml
|
55
|
+
- spec/data/filename_regexp.rb
|
56
|
+
- spec/data/filename_regexp.yml
|
57
|
+
- spec/data/folder.rb
|
58
|
+
- spec/data/folder.yml
|
59
|
+
- spec/data/modify.rb
|
60
|
+
- spec/data/modify.yml
|
61
|
+
- spec/data/no_destination_folder.rb
|
62
|
+
- spec/data/no_destination_folder.yml
|
63
|
+
- spec/data/rename.rb
|
64
|
+
- spec/data/rename.yml
|
65
|
+
- spec/spec.opts
|
66
|
+
- spec/spec_helper.rb
|
67
|
+
has_rdoc: true
|
68
|
+
homepage: http://github.com/bmarzolf/boatman
|
69
|
+
licenses: []
|
70
|
+
|
71
|
+
post_install_message:
|
72
|
+
rdoc_options:
|
73
|
+
- --charset=UTF-8
|
74
|
+
require_paths:
|
75
|
+
- lib
|
76
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
77
|
+
requirements:
|
78
|
+
- - ">="
|
79
|
+
- !ruby/object:Gem::Version
|
80
|
+
version: "0"
|
81
|
+
version:
|
82
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
83
|
+
requirements:
|
84
|
+
- - ">="
|
85
|
+
- !ruby/object:Gem::Version
|
86
|
+
version: "0"
|
87
|
+
version:
|
88
|
+
requirements: []
|
89
|
+
|
90
|
+
rubyforge_project:
|
91
|
+
rubygems_version: 1.3.5
|
92
|
+
signing_key:
|
93
|
+
specification_version: 3
|
94
|
+
summary: Ruby DSL for ferrying around and manipulating files
|
95
|
+
test_files:
|
96
|
+
- spec/spec_helper.rb
|
97
|
+
- spec/boatman_spec.rb
|
98
|
+
- spec/data/folder.rb
|
99
|
+
- spec/data/modify.rb
|
100
|
+
- spec/data/no_destination_folder.rb
|
101
|
+
- spec/data/filename_ending_regexp.rb
|
102
|
+
- spec/data/filename_ending.rb
|
103
|
+
- spec/data/rename.rb
|
104
|
+
- spec/data/filename_regexp.rb
|