iron_worker_ng 0.1.14 → 0.2.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (25) hide show
  1. data/README.md +64 -44
  2. data/VERSION +1 -1
  3. data/bin/iron_worker_ng +0 -5
  4. data/examples/ruby/hello/hello.rb +26 -0
  5. data/examples/ruby/hello/hello_worker.rb +1 -0
  6. data/examples/ruby/master_slave/master_slave.rb +39 -0
  7. data/examples/ruby/master_slave/master_worker.rb +22 -0
  8. data/examples/ruby/master_slave/slave_worker.rb +1 -0
  9. data/lib/iron_worker_ng/api_client.rb +20 -5
  10. data/lib/iron_worker_ng/code/base.rb +4 -2
  11. data/lib/iron_worker_ng/code/java.rb +9 -0
  12. data/lib/iron_worker_ng/code/node.rb +7 -0
  13. data/lib/iron_worker_ng/code/ruby.rb +7 -0
  14. data/lib/iron_worker_ng/feature/common/merge_dir.rb +4 -0
  15. data/lib/iron_worker_ng/feature/common/merge_file.rb +4 -0
  16. data/lib/iron_worker_ng/feature/java/merge_jar.rb +4 -0
  17. data/lib/iron_worker_ng/feature/java/merge_worker.rb +9 -1
  18. data/lib/iron_worker_ng/feature/node/merge_worker.rb +9 -1
  19. data/lib/iron_worker_ng/feature/ruby/merge_gem.rb +34 -6
  20. data/lib/iron_worker_ng/feature/ruby/merge_gemfile.rb +2 -0
  21. data/lib/iron_worker_ng/feature/ruby/merge_worker.rb +10 -2
  22. data/lib/iron_worker_ng/logger.rb +38 -0
  23. data/lib/iron_worker_ng.rb +1 -0
  24. data/test/helpers.rb +1 -0
  25. metadata +10 -3
data/README.md CHANGED
@@ -1,10 +1,17 @@
1
1
  # Introduction
2
2
 
3
- To run your code in cloud you need to do two things - upload code package to be executed and queue or schedule it for execution. While you can use REST APIs for that, it's easier to use an IronWorker library created specifically for your language of choice, such as IronWorkerNG.
3
+ To run your code in cloud you need to do three things:
4
4
 
5
- # Preparing Environment
5
+ - **Create code package**
6
+ - **Upload code package**
7
+ - **Queue or schedule tasks** for execution
6
8
 
7
- You'll need to register at http://iron.io/ and get your credentials to use IronWorkerNG. Each account can have an unlimited number of projects, so take advantage of it by creating separate projects for development, testing and production. Each project is identified by a unique project_id and requires your access token to do any actions on it, like uploading or queuing workers.
9
+ While you can use [REST APIs](http://dev.iron.io/worker/reference/api) for that, it's easier to use an
10
+ IronWorker library created specifically for your language of choice, such as this gem, IronWorkerNG.
11
+
12
+ # Preparing Your Environment
13
+
14
+ You'll need to register at http://iron.io/ and get your credentials to use IronWorkerNG. Each account can have an unlimited number of projects, so take advantage of it by creating separate projects for development, testing and production. Each project is identified by a unique project ID and requires your access token before it will perform any action, like uploading or queuing workers.
8
15
 
9
16
  Also, you'll need a Ruby 1.9 interpreter and the IronWorkerNG gem. Install it using following command.
10
17
 
@@ -12,15 +19,15 @@ Also, you'll need a Ruby 1.9 interpreter and the IronWorkerNG gem. Install it us
12
19
  gem install iron_worker_ng
13
20
  ```
14
21
 
15
- We recommend that you install typhoeus gem as well for faster API interaction.
22
+ We recommend that you install the `typhoeus` gem as well for faster API interaction.
16
23
 
17
24
  ```sh
18
25
  gem install typhoeus
19
26
  ```
20
27
 
21
- # Creating Worker
28
+ # Creating A Worker
22
29
 
23
- IronWorkerNG Ruby worker is common Ruby code. It can be as simple as shown below or as complex as you want.
30
+ Each IronWorkerNG Ruby worker is just Ruby code. It can be as simple or as complex as you want. For example, the following is an acceptable worker:
24
31
 
25
32
  ```ruby
26
33
  puts "I'm worker"
@@ -29,11 +36,11 @@ puts "I'm executing inside #{@iron_io_project_id} and was queued using #{@iron_i
29
36
  puts "I got '#{params}' parameters"
30
37
  ```
31
38
 
32
- All output to stdout will be logged and available for your review when workers finish execution.
39
+ All output to `STDOUT` will be logged and available for your review when your worker finishes execution.
33
40
 
34
- # Creating Code Package
41
+ # Creating The Code Package
35
42
 
36
- As this code will be executed on the cloud, you'll need to supply it with all necessary gems and supplementary data. IronWorkerNG::Code::Ruby will help you to do this.
43
+ Because your worker will be executed in the cloud, you'll need to bundle all the necessary gems, supplementary data, and other dependencies with it. `IronWorkerNG::Code::Ruby` makes this easy.
37
44
 
38
45
  ```ruby
39
46
  code = IronWorkerNG::Code::Ruby.new
@@ -46,11 +53,11 @@ code.merge_gem 'activerecord'
46
53
 
47
54
  ## IronWorkerNG::Code::Ruby API
48
55
 
49
- Please note that this API will help you to create code package but to upload it to IronWorker servers you'll need to use IronWorkerNG::Client API.
56
+ The IronWorkerNG::Code::Ruby class will help you package your code for upload, but to upload it to the cloud, you'll need to use the `IronWorkerNG::Client` class.
50
57
 
51
58
  ### initialize(name = nil)
52
59
 
53
- Will create new code package with specified name. If name is omitted, camel-cased worker's file name will be used.
60
+ Create new code package with the specified name. If `name` is omitted, a camel-cased version of the worker's file name will be used.
54
61
 
55
62
  ```ruby
56
63
  code = IronWorkerNG::Code::Ruby.new
@@ -59,7 +66,7 @@ code_with_name = IronWorkerNG::Code::Ruby.new('CoolWorker')
59
66
 
60
67
  ### name()
61
68
 
62
- Will return code package name.
69
+ Return the code package's name.
63
70
 
64
71
  ```ruby
65
72
  puts code.name
@@ -67,24 +74,24 @@ puts code.name
67
74
 
68
75
  ### hash_string()
69
76
 
77
+ Return the hash string for the code package. If you want to prevent uploading unchanged code packages, you can use it to check if any changes were made. It's very efficient, so it shouldn't cause any performance impact.
78
+
70
79
  ```
71
80
  puts code.hash_string
72
81
  ```
73
82
 
74
- Will return code package hash string. If you want to prevent uploading unchanged code packages, you can use it to check if any changes were made. As it's very efficient, it shouldn't cause any performance impact.
75
-
76
83
  ### merge_file(path, dest = '')
77
84
 
85
+ Merge the file located at `path` into the code package. If `dest` is set, it will be used as the path to a directory within the zip, into which the file will be merged. If the directory does not exist, it will be automatically created.
86
+
78
87
  ```ruby
79
88
  code.merge_file '../config/database.yml' # will be in the same directory as worker
80
89
  code.merge_file 'clients.csv', 'information/clients' # will be in information/clients subdirectory
81
90
  ```
82
91
 
83
- Merges file located at path into the code package. You can use the optional dest parameter to set destination directory which will be automatically created.
84
-
85
92
  ### merge_dir(path, dest = '')
86
93
 
87
- Recursively merges the directory located at path into the code package.
94
+ Recursively merge the directory located at path into the code package. If `dest` is set, it will be used as the path to a directory within the zip, into which the directory specified by `path` will be merged. If `dest` is set but does not exist, it will be automatically created.
88
95
 
89
96
  ```ruby
90
97
  code.merge_dir '../config' # will be in the same directory as worker
@@ -93,7 +100,7 @@ code.merge_dir 'lib', 'utils' # will be in utils subdirectory, accessible as uti
93
100
 
94
101
  ### merge_worker(path, name = nil)
95
102
 
96
- Merges worker located at path. If name is omitted, camel-cased file name will be used. You can have only one worker merged per code package.
103
+ Merge the worker located at `path`. If `name` is omitted, a camel-cased version of the file name will be used. **You can have only one worker merged per code package.**
97
104
 
98
105
  ```ruby
99
106
  code.merge_worker 'my_worker.rb' # name will be MyWorker
@@ -101,7 +108,7 @@ code.merge_worker 'my_worker.rb' # name will be MyWorker
101
108
 
102
109
  ### merge_gem(name, version = '>= 0')
103
110
 
104
- Merges gem with dependencies. Please note that gems which contains binary extensions will not be merged at the moment, however we have sane set of such gems preinstalled at IronWorker servers. You can use version constrains if you need a specific gem version.
111
+ Merge a gem with dependencies. Please note that gems which contains binary extensions will not be merged for now, as binary extensions are not supported at this time; we have [a set](http://dev.iron.io/worker/reference/environment/?lang=ruby#ruby_gems_installed) of the most common gems with binary extensions preinstalled for your use. You can use version constrains if you need a specific gem version.
105
112
 
106
113
  ```ruby
107
114
  code.merge_gem 'activerecord'
@@ -110,30 +117,43 @@ code.merge_gem 'paperclip', '< 3.0.0,>= 2.1.0'
110
117
 
111
118
  ### merge_gemfile(path, *groups)
112
119
 
113
- Merges all gems from specified groups in Gemfile. Please note that it'll just merge gems and not auto require them when executing worker.
120
+ Merge all gems from specified the groups in a Gemfile. Please note that this will not auto-require the gems when executing the worker.
114
121
 
115
122
  ```ruby
116
123
  code.merge_gemfile '../Gemfile', 'common', 'worker' # merges gems from common and worker groups
117
124
  ```
118
125
 
119
- # Using IronWorker
126
+ # Upload Your Worker
120
127
 
121
- When you have your code package you are ready to run it on IronWorker servers.
128
+ When you have your code package, you are ready to upload and run it on the IronWorker cloud.
122
129
 
123
130
  ```ruby
131
+ # Initialize the client
124
132
  client = IronWorkerNG::Client.new(:token => 'IRON_IO_TOKEN', :project_id => 'IRON_IO_PROJECT_ID')
125
-
133
+ # Upload the code
126
134
  client.codes.create(code)
135
+ ```
136
+
137
+ **NOTE**: You only need to call `client.codes.create(code)` once for each time your code changes.
138
+
139
+ # Queue Up Tasks for Your Worker
140
+
141
+ Now that the code is uploaded, we can create/queue up tasks. You can call this over and over
142
+ for as many tasks as you want.
143
+
144
+ ```ruby
127
145
  client.tasks.create('MyWorker', {:client => 'Joe'})
128
146
  ```
129
147
 
130
- ## IronWorker::Client API
148
+ # The Rest of the IronWorker API
149
+
150
+ ## IronWorker::Client
131
151
 
132
- You can use IronWorkerNG::Client API to upload code packages, queue tasks, create schedules and more.
152
+ You can use the `IronWorkerNG::Client` class to upload code packages, queue tasks, create schedules, and more.
133
153
 
134
154
  ### initialize(options = {})
135
155
 
136
- Creates client object used for all interactions with IronWorker servers.
156
+ Create a client object used for all your interactions with the IronWorker cloud.
137
157
 
138
158
  ```ruby
139
159
  client = IronWorkerNG::Client.new(:token => 'IRON_IO_TOKEN', :project_id => 'IRON_IO_PROJECT_ID')
@@ -141,7 +161,7 @@ client = IronWorkerNG::Client.new(:token => 'IRON_IO_TOKEN', :project_id => 'IRO
141
161
 
142
162
  ### codes.list(options = {})
143
163
 
144
- Returns an array of information about uploaded codes. Visit http://dev.iron.io/worker/reference/api/#list_code_packages for more information about options and code information object format.
164
+ Return an array of information about uploaded code packages. Visit http://dev.iron.io/worker/reference/api/#list_code_packages for more information about the available options and the code package object format.
145
165
 
146
166
  ```ruby
147
167
  client.codes.list.each do |code|
@@ -151,7 +171,7 @@ end
151
171
 
152
172
  ### codes.get(code_id)
153
173
 
154
- Returns information about uploaded code with specified code_id. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_code_package for more information about code information object format.
174
+ Return information about an uploaded code package with the specified ID. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_code_package for more information about the code package object format.
155
175
 
156
176
  ```ruby
157
177
  puts client.codes.get('1234567890').name
@@ -159,7 +179,7 @@ puts client.codes.get('1234567890').name
159
179
 
160
180
  ### codes.create(code)
161
181
 
162
- Uploads IronWorkerNG::Code::Ruby object to IronWorker servers.
182
+ Upload an `IronWorkerNG::Code::Ruby` object to the IronWorker cloud.
163
183
 
164
184
  ```ruby
165
185
  client.codes.create(code)
@@ -167,7 +187,7 @@ client.codes.create(code)
167
187
 
168
188
  ### codes.delete(code_id)
169
189
 
170
- Deletes code with specified code_id from IronWorker servers.
190
+ Delete the code package specified by `code_id` from the IronWorker cloud.
171
191
 
172
192
  ```ruby
173
193
  client.codes.delete('1234567890')
@@ -175,7 +195,7 @@ client.codes.delete('1234567890')
175
195
 
176
196
  ### codes.revisions(code_id, options = {})
177
197
 
178
- Returns array of revision information for code package with specified code_id. Visit http://dev.iron.io/worker/reference/api/#list_code_package_revisions for more information about options and revision information object.
198
+ Get an array of revision information for the code package specified by `code_id`. Visit http://dev.iron.io/worker/reference/api/#list_code_package_revisions for more information about the available options and the revision objects.
179
199
 
180
200
  ```ruby
181
201
  client.codes.revisions('1234567890').each do |revision|
@@ -185,7 +205,7 @@ end
185
205
 
186
206
  ### codes.download(code_id, options = {})
187
207
 
188
- Downloads the code package with specified id and returns it to you as array of bytes. Visit http://dev.iron.io/worker/reference/api/#download_a_code_package for more information about options.
208
+ Download the code package specified by `code_id` and return it as an array of bytes. Visit http://dev.iron.io/worker/reference/api/#download_a_code_package for more information about the available options.
189
209
 
190
210
  ```ruby
191
211
  data = client.codes.download('1234567890')
@@ -193,7 +213,7 @@ data = client.codes.download('1234567890')
193
213
 
194
214
  ### tasks.list(options = {})
195
215
 
196
- Returns an array of information about tasks. Visit http://dev.iron.io/worker/reference/api/#list_tasks for more information about options and task information object format.
216
+ Retrieve an array of information about your workers' tasks. Visit http://dev.iron.io/worker/reference/api/#list_tasks for more information about the available options and the task object format.
197
217
 
198
218
  ```ruby
199
219
  client.tasks.list.each do |task|
@@ -203,7 +223,7 @@ end
203
223
 
204
224
  ### tasks.get(task_id)
205
225
 
206
- Returns information about task with specified task_id. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_task for more information about task information object format.
226
+ Return information about the task specified by `task_id`. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_task for more information about the task object format.
207
227
 
208
228
  ```ruby
209
229
  puts client.tasks.get('1234567890').code_name
@@ -211,7 +231,7 @@ puts client.tasks.get('1234567890').code_name
211
231
 
212
232
  ### tasks.create(code_name, params = {}, options = {})
213
233
 
214
- Queues a new task for code with specified code_name, passes params hash to it and returns task information object with only id field filled. Visit http://dev.iron.io/worker/reference/api/#queue_a_task for more information about options.
234
+ Queue a new task for the code package specified by `code_name`, passing the `params` hash to it as a payload and returning a task object with only the `id` field filled. Visit http://dev.iron.io/worker/reference/api/#queue_a_task for more information about the available options.
215
235
 
216
236
  ```ruby
217
237
  task = client.tasks.create('MyWorker', {:client => 'Joe'}, {:delay => 180})
@@ -220,7 +240,7 @@ puts task.id
220
240
 
221
241
  ### tasks.cancel(task_id)
222
242
 
223
- Cancels the task with specified task_id.
243
+ Cancel the task specified by `task_id`.
224
244
 
225
245
  ```ruby
226
246
  client.tasks.cancel('1234567890')
@@ -228,7 +248,7 @@ client.tasks.cancel('1234567890')
228
248
 
229
249
  ### tasks.cancel_all(code_id)
230
250
 
231
- Cancels all tasks for code package identified by specified code_id.
251
+ Cancel all tasks for the code package specified by `code_id`.
232
252
 
233
253
  ```ruby
234
254
  client.tasks.cancel_all('1234567890')
@@ -236,7 +256,7 @@ client.tasks.cancel_all('1234567890')
236
256
 
237
257
  ### tasks.log(task_id)
238
258
 
239
- Returns full task log for task with specified task_id. Please note that log is available only when task completed execution.
259
+ Retrieve the full task log for the task specified by `task_id`. Please note that log is available only after the task has completed execution. The log will include any output to `STDOUT`.
240
260
 
241
261
  ```ruby
242
262
  puts client.tasks.log('1234567890')
@@ -244,7 +264,7 @@ puts client.tasks.log('1234567890')
244
264
 
245
265
  ### tasks.set_progress(task_id, options = {})
246
266
 
247
- Sets task progress information for task with specified task_id. This should be used from within workers to inform you about worker execution status which you'll get via a tasks.get call. Visit http://dev.iron.io/worker/reference/api/#set_a_tasks_progress for more information about options.
267
+ Set the progress information for the task specified by `task_id`. This should be used from within workers to inform you about worker execution status, which you can retrieve with a `tasks.get` call. Visit http://dev.iron.io/worker/reference/api/#set_a_tasks_progress for more information about the available options.
248
268
 
249
269
  ```ruby
250
270
  client.tasks.set_progress('1234567890', {:msg => 'Still running...'})
@@ -252,7 +272,7 @@ client.tasks.set_progress('1234567890', {:msg => 'Still running...'})
252
272
 
253
273
  ### tasks.wait_for(task_id, options = {})
254
274
 
255
- Waits while task identified by specified task_id executes. Options can contain :sleep parameter used to sleep between API invocations which defaults to 5 seconds. If block is provided, it'll be yielded after each API call with task information object as parameter.
275
+ Wait (block) while the task specified by `task_id` executes. Options can contain a `:sleep` parameter used to modify the delay between API invocations; the default is 5 seconds. If a block is provided (as in the example below), it will be called after each API call with the task object as parameter.
256
276
 
257
277
  ```ruby
258
278
  client.tasks.wait_for('1234567890') do |task|
@@ -262,7 +282,7 @@ end
262
282
 
263
283
  ### schedules.list(options = {})
264
284
 
265
- Returns an array of information about schedules. Visit http://dev.iron.io/worker/reference/api/#list_scheduled_tasks for more information about options and schedule information object format.
285
+ Return an array of scheduled tasks. Visit http://dev.iron.io/worker/reference/api/#list_scheduled_tasks for more information about the available options and the scheduled task object format.
266
286
 
267
287
  ```ruby
268
288
  client.schedules.list.each do |schedule|
@@ -272,7 +292,7 @@ end
272
292
 
273
293
  ### schedules.get(schedule_id)
274
294
 
275
- Returns information about the schedule with specified schedule_id. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_scheduled_task for more information about schedule information object format.
295
+ Return information about the scheduled task specified by `schedule_id`. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_scheduled_task for more information about the scheduled task object format.
276
296
 
277
297
  ```ruby
278
298
  puts client.schedules.get('1234567890').last_run_time
@@ -280,7 +300,7 @@ puts client.schedules.get('1234567890').last_run_time
280
300
 
281
301
  ### schedules.create(code_name, params = {}, options = {})
282
302
 
283
- Creates a new schedule for code with specified code_name, passes params hash to it and returns schedule information object with only id field filled. Visit http://dev.iron.io/worker/reference/api/#schedule_a_task for more information about options.
303
+ Create a new scheduled task for the code package specified by `code_name`, passing the params hash to it as a data payload and returning a scheduled task object with only the `id` field filled. Visit http://dev.iron.io/worker/reference/api/#schedule_a_task for more information about the available options.
284
304
 
285
305
  ```ruby
286
306
  schedule = client.schedules.create('MyWorker', {:client => 'Joe'}, {:start_at => Time.now + 3600})
@@ -289,7 +309,7 @@ puts schedule.id
289
309
 
290
310
  ### schedules.cancel(schedule_id)
291
311
 
292
- Cancels the schedule with specified schedule_id.
312
+ Cancel the scheduled task specified by `schedule_id`.
293
313
 
294
314
  ```ruby
295
315
  client.schedules.cancel('1234567890')
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.1.14
1
+ 0.2.0
data/bin/iron_worker_ng CHANGED
@@ -14,11 +14,6 @@ end
14
14
  command = $*[0]
15
15
  $*.shift
16
16
 
17
- if ENV['IRON_IO_TOKEN'].nil? || ENV['IRON_IO_PROJECT_ID'].nil?
18
- puts "please set $IRON_IO_TOKEN and $IRON_IO_PROJECT_ID environment variables"
19
- exit 1
20
- end
21
-
22
17
  client = IronWorkerNG::Client.new(:token => ENV['IRON_IO_TOKEN'], :project_id => ENV['IRON_IO_PROJECT_ID'])
23
18
 
24
19
  if command == 'codes.create'
data/examples/ruby/hello/hello.rb ADDED
@@ -0,0 +1,26 @@
1
+ require 'iron_worker_ng'
2
+
3
+ # to run examples, you must specify iron.io authentication token and project id
4
+ token, project_id = [ ENV['IRON_IO_TOKEN'], ENV['IRON_IO_PROJECT_ID'] ]
5
+ raise("please set $IRON_IO_TOKEN and $IRON_IO_PROJECT_ID " +
6
+ "environment variables") unless token and project_id
7
+
8
+ # initializing api object with them
9
+ client = IronWorkerNG::Client.new(:token => token,
10
+ :project_id => project_id)
11
+
12
+ # create ruby code bundle
13
+ code = IronWorkerNG::Code::Ruby.new
14
+ code.merge_worker(File.dirname(__FILE__) + '/hello_worker.rb')
15
+
16
+ # upload it to iron.io
17
+ client.codes.create(code)
18
+
19
+ # create task to run the bundle
20
+ task_id = client.tasks.create('HelloWorker').id
21
+
22
+ # wait for the task to finish
23
+ client.tasks.wait_for(task_id)
24
+
25
+ # retriving task log
26
+ log = client.tasks.log(task_id) #> log == "hello\n" -- worker stdout is in log
data/examples/ruby/hello/hello_worker.rb ADDED
@@ -0,0 +1 @@
1
+ puts "hello"
data/examples/ruby/master_slave/master_slave.rb ADDED
@@ -0,0 +1,39 @@
1
+ require 'iron_worker_ng'
2
+
3
+ # to run examples, you must specify iron.io authentication token and project id
4
+ token, project_id = [ ENV['IRON_IO_TOKEN'], ENV['IRON_IO_PROJECT_ID'] ]
5
+ raise("please set $IRON_IO_TOKEN and $IRON_IO_PROJECT_ID " +
6
+ "environment variables") unless token and project_id
7
+
8
+ # initializing api object with them
9
+ client = IronWorkerNG::Client.new(:token => token,
10
+ :project_id => project_id)
11
+
12
+ # create master code bundle
13
+ master = IronWorkerNG::Code::Ruby.new
14
+ master.merge_worker(File.dirname(__FILE__) + '/master_worker.rb')
15
+ master.merge_gem('iron_worker_ng') # we need it to queue slave workers
16
+
17
+ # create slave code bundle
18
+ slave = IronWorkerNG::Code::Ruby.new
19
+ slave.merge_worker(File.dirname(__FILE__) + '/slave_worker.rb')
20
+
21
+ # upload both
22
+ client.codes.create(master)
23
+ client.codes.create(slave)
24
+
25
+ # run master task
26
+ task_id = client.tasks.create('MasterWorker',
27
+ {
28
+ :args => [ [1, 2, 3],
29
+ [4, 5, 6],
30
+ [7, 8, 9] ]
31
+ }).id
32
+
33
+ # wait for the task to finish
34
+ client.tasks.wait_for(task_id)
35
+
36
+ # retriving task log
37
+ log = client.tasks.log(task_id)
38
+
39
+ #> log.lines.find{ |l| l =~ /Sum =/ } == "Sum = 45\n" -- correct result in log
data/examples/ruby/master_slave/master_worker.rb ADDED
@@ -0,0 +1,22 @@
1
+ require 'iron_worker_ng'
2
+
3
+ # token and project id are available inside worker
4
+ client = IronWorkerNG::Client.new(:token => iron_io_token,
5
+ :project_id => iron_io_project_id)
6
+
7
+ log 'Running slave workers...'
8
+ task_ids = []
9
+ params[:args].each do |arg|
10
+ log "Queueing slave with arg=#{arg.to_s}"
11
+ task_ids << client.tasks.create('SlaveWorker', { :arg => arg }).id
12
+ end
13
+
14
+ log 'Retriving results from slaves logs'
15
+ results = task_ids.map do |id|
16
+ client.tasks.wait_for(id)
17
+ client.tasks.log(id).to_i
18
+ end
19
+
20
+ log "Sum = #{ results.inject(:+) }"
21
+
22
+ log 'Done'
data/examples/ruby/master_slave/slave_worker.rb ADDED
@@ -0,0 +1 @@
1
+ log params[:arg].inject(:+)
data/lib/iron_worker_ng/api_client.rb CHANGED
@@ -21,11 +21,16 @@ module IronWorkerNG
21
21
  @token = options[:token] || options['token']
22
22
  @project_id = options[:project_id] || options['project_id']
23
23
 
24
- @scheme = options[:scheme] || 'https'
25
- @host = options[:host] || IronWorkerNG::APIClient::AWS_US_EAST_HOST
26
- @port = options[:port] || 443
27
- @api_version = options[:api_version] || 2
28
- @user_agent = options[:user_agent] || 'iron_worker_ng-' + IronWorkerNG.version
24
+ if (not @token) || (not @project_id)
25
+ IronWorkerNG::Logger.error 'Both iron.io token and project_id must be specified'
26
+ raise 'Both iron.io token and project_id must be specified'
27
+ end
28
+
29
+ @scheme = options[:scheme] || options['scheme'] || 'https'
30
+ @host = options[:host] || options['host'] || IronWorkerNG::APIClient::AWS_US_EAST_HOST
31
+ @port = options[:port] || options['port'] || 443
32
+ @api_version = options[:api_version] || options['api_version'] || 2
33
+ @user_agent = options[:user_agent] || options['user_agent'] || 'iron_worker_ng-' + IronWorkerNG.version
29
34
 
30
35
  @url = "#{scheme}://#{host}:#{port}/#{api_version}/"
31
36
 
@@ -45,6 +50,8 @@ module IronWorkerNG
45
50
  request_hash[:headers] = common_request_hash
46
51
  request_hash[:params] = params
47
52
 
53
+ IronWorkerNG::Logger.debug "GET #{@url + method} with params='#{request_hash.to_s}'"
54
+
48
55
  @rest.get(@url + method, request_hash)
49
56
  end
50
57
 
@@ -53,6 +60,8 @@ module IronWorkerNG
53
60
  request_hash[:headers] = common_request_hash
54
61
  request_hash[:body] = params.to_json
55
62
 
63
+ IronWorkerNG::Logger.debug "POST #{@url + method} with params='#{request_hash.to_s}'"
64
+
56
65
  @rest.post(@url + method, request_hash)
57
66
  end
58
67
 
@@ -61,6 +70,8 @@ module IronWorkerNG
61
70
  request_hash[:headers] = common_request_hash
62
71
  request_hash[:params] = params
63
72
 
73
+ IronWorkerNG::Logger.debug "DELETE #{@url + method} with params='#{request_hash.to_s}'"
74
+
64
75
  @rest.delete(@url + method, request_hash)
65
76
  end
66
77
 
@@ -71,10 +82,14 @@ module IronWorkerNG
71
82
  request_hash[:data] = params.to_json
72
83
  request_hash[:file] = file
73
84
 
85
+ IronWorkerNG::Logger.debug "POST #{@url + method + "?oauth=" + @token} with params='#{request_hash.to_s}'"
86
+
74
87
  RestClient.post(@url + method + "?oauth=#{@token}", request_hash)
75
88
  end
76
89
 
77
90
  def parse_response(response, parse_json = true)
91
+ IronWorkerNG::Logger.debug "GOT #{response.code} with params='#{response.body}'"
92
+
78
93
  raise IronWorkerNG::APIClientError.new(response.body) if response.code != 200
79
94
 
80
95
  return response.body unless parse_json
data/lib/iron_worker_ng/code/base.rb CHANGED
@@ -34,7 +34,7 @@ module IronWorkerNG
34
34
  include IronWorkerNG::Feature::Common::MergeFile::InstanceMethods
35
35
  include IronWorkerNG::Feature::Common::MergeDir::InstanceMethods
36
36
 
37
- def initialize(name = nil)
37
+ def initialize(name = nil, options = {})
38
38
  @name = name
39
39
  @features = []
40
40
  end
@@ -71,7 +71,9 @@ module IronWorkerNG
71
71
  end
72
72
 
73
73
  zip_name = Dir.tmpdir + '/' + Dir::Tmpname.make_tmpname("iron-worker-ng-", "code.zip")
74
-
74
+
75
+ IronWorkerNG::Logger.debug "Creating code zip '#{zip_name}'"
76
+
75
77
  Zip::ZipFile.open(zip_name, Zip::ZipFile::CREATE) do |zip|
76
78
  bundle(zip)
77
79
  create_runner(zip, init_code)
data/lib/iron_worker_ng/code/java.rb CHANGED
@@ -8,6 +8,13 @@ module IronWorkerNG
8
8
  include IronWorkerNG::Feature::Java::MergeWorker::InstanceMethods
9
9
 
10
10
  def create_runner(zip, init_code)
11
+ IronWorkerNG::Logger.info 'Creating java runner'
12
+
13
+ unless @worker
14
+ IronWorkerNG::Logger.error 'No worker specified'
15
+ raise 'No worker specified'
16
+ end
17
+
11
18
  classpath_array = []
12
19
 
13
20
  @features.each do |f|
@@ -17,6 +24,8 @@ module IronWorkerNG
17
24
  end
18
25
 
19
26
  classpath = classpath_array.join(':')
27
+
28
+ IronWorkerNG::Logger.info "Collected '#{classpath}' classpath"
20
29
 
21
30
  zip.get_output_stream('runner.rb') do |runner|
22
31
  runner.write <<RUNNER
data/lib/iron_worker_ng/code/node.rb CHANGED
@@ -6,6 +6,13 @@ module IronWorkerNG
6
6
  include IronWorkerNG::Feature::Node::MergeWorker::InstanceMethods
7
7
 
8
8
  def create_runner(zip, init_code)
9
+ IronWorkerNG::Logger.info 'Creating node runner'
10
+
11
+ unless @worker
12
+ IronWorkerNG::Logger.error 'No worker specified'
13
+ raise 'No worker specified'
14
+ end
15
+
9
16
  zip.get_output_stream('runner.rb') do |runner|
10
17
  runner.write <<RUNNER
11
18
  # iron_worker_ng-#{IronWorkerNG.version}
data/lib/iron_worker_ng/code/ruby.rb CHANGED
@@ -10,6 +10,13 @@ module IronWorkerNG
10
10
  include IronWorkerNG::Feature::Ruby::MergeWorker::InstanceMethods
11
11
 
12
12
  def create_runner(zip, init_code)
13
+ IronWorkerNG::Logger.info 'Creating ruby runner'
14
+
15
+ unless @worker
16
+ IronWorkerNG::Logger.error 'No worker specified'
17
+ raise 'No worker specified'
18
+ end
19
+
13
20
  zip.get_output_stream('runner.rb') do |runner|
14
21
  runner.write <<RUNNER
15
22
  # iron_worker_ng-#{IronWorkerNG.version}
data/lib/iron_worker_ng/feature/common/merge_dir.rb CHANGED
@@ -22,6 +22,8 @@ module IronWorkerNG
22
22
  end
23
23
 
24
24
  def bundle(zip)
25
+ IronWorkerNG::Logger.debug "Bundling dir with path='#{@path}' and dest='#{@dest}'"
26
+
25
27
  Dir.glob(@path + '/**/**') do |path|
26
28
  zip.add(@dest + File.basename(@path) + path[@path.length .. -1], path)
27
29
  end
@@ -30,6 +32,8 @@ module IronWorkerNG
30
32
 
31
33
  module InstanceMethods
32
34
  def merge_dir(path, dest = '')
35
+ IronWorkerNG::Logger.info "Merging dir with path='#{path}' and dest='#{dest}'"
36
+
33
37
  @features << IronWorkerNG::Feature::Common::MergeDir::Feature.new(path, dest)
34
38
  end
35
39
 
data/lib/iron_worker_ng/feature/common/merge_file.rb CHANGED
@@ -16,12 +16,16 @@ module IronWorkerNG
16
16
  end
17
17
 
18
18
  def bundle(zip)
19
+ IronWorkerNG::Logger.debug "Bundling file with path='#{@path}' and dest='#{@dest}'"
20
+
19
21
  zip.add(@dest + File.basename(@path), @path)
20
22
  end
21
23
  end
22
24
 
23
25
  module InstanceMethods
24
26
  def merge_file(path, dest = '')
27
+ IronWorkerNG::Logger.info "Merging file with path='#{path}' and dest='#{dest}'"
28
+
25
29
  @features << IronWorkerNG::Feature::Common::MergeFile::Feature.new(path, dest)
26
30
  end
27
31
 
data/lib/iron_worker_ng/feature/java/merge_jar.rb CHANGED
@@ -14,6 +14,8 @@ module IronWorkerNG
14
14
  end
15
15
 
16
16
  def bundle(zip)
17
+ IronWorkerNG::Logger.debug "Bundling java jar with path='#{@path}'"
18
+
17
19
  zip.add(File.basename(@path), @path)
18
20
  end
19
21
 
@@ -24,6 +26,8 @@ module IronWorkerNG
24
26
 
25
27
  module InstanceMethods
26
28
  def merge_jar(path)
29
+ IronWorkerNG::Logger.info "Merging java jar with path='#{path}'"
30
+
27
31
  @features << IronWorkerNG::Feature::Java::MergeJar::Feature.new(path)
28
32
  end
29
33
 
data/lib/iron_worker_ng/feature/java/merge_worker.rb CHANGED
@@ -16,6 +16,8 @@ module IronWorkerNG
16
16
  end
17
17
 
18
18
  def bundle(zip)
19
+ IronWorkerNG::Logger.debug "Bundling java worker with path='#{@path}' and class='#{@klass}'"
20
+
19
21
  zip.add(File.basename(@path), @path)
20
22
  end
21
23
 
@@ -30,11 +32,17 @@ module IronWorkerNG
30
32
  def merge_worker(path, klass)
31
33
  @worker ||= nil
32
34
 
33
- return unless @worker.nil?
35
+ unless @worker.nil?
36
+ IronWorkerNG::Logger.warn "Ignoring attempt to merge java worker with path='#{path}' and class='#{klass}'"
37
+ return
38
+ end
34
39
 
35
40
  @name ||= klass.split('.')[-1]
36
41
 
37
42
  @worker = IronWorkerNG::Feature::Java::MergeWorker::Feature.new(path, klass)
43
+
44
+ IronWorkerNG::Logger.info "Merging java worker with path='#{path}' and class='#{klass}'"
45
+
38
46
  @features << @worker
39
47
  end
40
48
 
data/lib/iron_worker_ng/feature/node/merge_worker.rb CHANGED
@@ -14,6 +14,8 @@ module IronWorkerNG
14
14
  end
15
15
 
16
16
  def bundle(zip)
17
+ IronWorkerNG::Logger.debug "Bundling node worker with path='#{@path}'"
18
+
17
19
  zip.add(File.basename(@path), @path)
18
20
  end
19
21
 
@@ -28,11 +30,17 @@ module IronWorkerNG
28
30
  def merge_worker(path)
29
31
  @worker ||= nil
30
32
 
31
- return unless @worker.nil?
33
+ unless @worker.nil?
34
+ IronWorkerNG::Logger.warn "Ignoring attempt to merge node worker with path='#{path}'"
35
+ return
36
+ end
32
37
 
33
38
  @name ||= File.basename(path).gsub(/\.js$/, '').capitalize.gsub(/_./) { |x| x[1].upcase }
34
39
 
35
40
  @worker = IronWorkerNG::Feature::Node::MergeWorker::Feature.new(path)
41
+
42
+ IronWorkerNG::Logger.info "Merging node worker with path='#{path}'"
43
+
36
44
  @features << @worker
37
45
  end
38
46
 
data/lib/iron_worker_ng/feature/ruby/merge_gem.rb CHANGED
@@ -11,16 +11,35 @@ module IronWorkerNG
11
11
  @spec = spec
12
12
  end
13
13
 
14
+ def gem_path
15
+ path = @spec.full_gem_path
16
+
17
+ # when running under bundle exec it sometimes duplicates gem path suffix
18
+
19
+ suffix_index = path.rindex('/gems/')
20
+ suffix = path[suffix_index .. -1]
21
+
22
+ if path.end_with?(suffix + suffix)
23
+ path = path[0 .. suffix_index - 1]
24
+ end
25
+
26
+ path
27
+ end
28
+
14
29
  def hash_string
15
30
  Digest::MD5.hexdigest(@spec.full_name)
16
31
  end
17
32
 
18
33
  def bundle(zip)
19
34
  if @spec.extensions.length == 0
20
- zip.add('gems/' + @spec.full_name, @spec.full_gem_path)
21
- Dir.glob(@spec.full_gem_path + '/**/**') do |path|
22
- zip.add('gems/' + @spec.full_name + path[@spec.full_gem_path.length .. -1], path)
35
+ IronWorkerNG::Logger.debug "Bundling ruby gem with name='#{@spec.name}' and version='#{@spec.version}'"
36
+
37
+ zip.add('gems/' + @spec.full_name, gem_path)
38
+ Dir.glob(gem_path + '/**/**') do |path|
39
+ zip.add('gems/' + @spec.full_name + path[gem_path.length .. -1], path)
23
40
  end
41
+ else
42
+ IronWorkerNG::Logger.warn "Skipping ruby gem with name='#{@spec.name}' and version='#{@spec.version}' as it contains native extensions"
24
43
  end
25
44
  end
26
45
 
@@ -37,15 +56,20 @@ module IronWorkerNG
37
56
  attr_reader :merge_gem_reqs
38
57
 
39
58
  def merge_gem(name, version = '>= 0')
40
- @merge_gem_reqs ||= []
59
+ IronWorkerNG::Logger.info "Adding ruby gem dependency with name='#{name}' and version='#{version}'"
41
60
 
61
+ @merge_gem_reqs ||= []
42
62
  @merge_gem_reqs << Bundler::Dependency.new(name, version.split(', '))
43
63
  end
44
64
 
45
65
  def merge_gem_fixate
66
+ IronWorkerNG::Logger.info 'Fixating gems dependencies'
67
+
46
68
  @merge_gem_reqs ||= []
47
69
 
48
- @features.reject! { |f| f.class == IronWorkerNG::Feature::Ruby::MergeGem::Feature }
70
+ @features.reject! do |f|
71
+ f.class == IronWorkerNG::Feature::Ruby::MergeGem::Feature
72
+ end
49
73
 
50
74
  if @merge_gem_reqs.length > 0
51
75
  reqs = @merge_gem_reqs.map { |req| Bundler::DepProxy.new(req, Gem::Platform::RUBY) }
@@ -63,7 +87,11 @@ module IronWorkerNG
63
87
  spec_set = Bundler::Resolver.resolve(reqs, index)
64
88
 
65
89
  spec_set.to_a.each do |spec|
66
- @features << IronWorkerNG::Feature::Ruby::MergeGem::Feature.new(spec.__materialize__)
90
+ spec.__materialize__
91
+
92
+ IronWorkerNG::Logger.info "Merging ruby gem with name='#{spec.name}' and version='#{spec.version}'"
93
+
94
+ @features << IronWorkerNG::Feature::Ruby::MergeGem::Feature.new(spec)
67
95
  end
68
96
  end
69
97
  end
data/lib/iron_worker_ng/feature/ruby/merge_gemfile.rb CHANGED
@@ -23,6 +23,8 @@ module IronWorkerNG
23
23
  groups = groups.map { |g| g.to_sym }
24
24
  groups << :default if groups.length == 0
25
25
 
26
+ IronWorkerNG::Logger.info "Adding ruby gems dependencies from #{groups.join(', ')} group#{groups.size > 1 ? 's' : ''} of #{path}"
27
+
26
28
  specs = Bundler::Definition.build(path, path + '.lock', nil).specs_for(groups)
27
29
 
28
30
  specs.each do |spec|
data/lib/iron_worker_ng/feature/ruby/merge_worker.rb CHANGED
@@ -16,6 +16,8 @@ module IronWorkerNG
16
16
  end
17
17
 
18
18
  def bundle(zip)
19
+ IronWorkerNG::Logger.debug "Bundling ruby worker with path='#{path}' and class='#{klass}'"
20
+
19
21
  zip.add(File.basename(@path), @path)
20
22
  end
21
23
 
@@ -30,15 +32,21 @@ module IronWorkerNG
30
32
  def merge_worker(path, klass = nil)
31
33
  @worker ||= nil
32
34
 
33
- return unless @worker.nil?
34
-
35
35
  if klass == nil
36
36
  klass = File.basename(path).gsub(/\.rb$/, '').capitalize.gsub(/_./) { |x| x[1].upcase }
37
37
  end
38
38
 
39
+ unless @worker.nil?
40
+ IronWorkerNG::Logger.warn "Ignoring attempt to merge ruby worker with path='#{path}' and class='#{klass}'"
41
+ return
42
+ end
43
+
39
44
  @name ||= klass
40
45
 
41
46
  @worker = IronWorkerNG::Feature::Ruby::MergeWorker::Feature.new(path, klass)
47
+
48
+ IronWorkerNG::Logger.info "Merging ruby worker with path='#{path}' and class='#{klass}'"
49
+
42
50
  @features << @worker
43
51
  end
44
52
 
data/lib/iron_worker_ng/logger.rb ADDED
@@ -0,0 +1,38 @@
1
+ require 'logger'
2
+
3
+ module IronWorkerNG
4
+ module Logger
5
+ def self.logger
6
+ unless @logger
7
+ @logger = ::Logger.new(STDOUT)
8
+ @logger.level = ::Logger::INFO
9
+ end
10
+
11
+ @logger
12
+ end
13
+
14
+ def self.logger=(logger)
15
+ @logger = logger
16
+ end
17
+
18
+ def self.fatal(msg)
19
+ self.logger.fatal("IronWorkerNG") { msg }
20
+ end
21
+
22
+ def self.error(msg)
23
+ self.logger.error("IronWorkerNG") { msg }
24
+ end
25
+
26
+ def self.warn(msg)
27
+ self.logger.warn("IronWorkerNG") { msg }
28
+ end
29
+
30
+ def self.info(msg)
31
+ self.logger.info("IronWorkerNG") { msg }
32
+ end
33
+
34
+ def self.debug(msg)
35
+ self.logger.debug("IronWorkerNG") { msg }
36
+ end
37
+ end
38
+ end
data/lib/iron_worker_ng.rb CHANGED
@@ -1,4 +1,5 @@
1
1
  require_relative 'iron_worker_ng/version'
2
+ require_relative 'iron_worker_ng/logger'
2
3
  require_relative 'iron_worker_ng/client'
3
4
  require_relative 'iron_worker_ng/code/base'
4
5
  require_relative 'iron_worker_ng/code/ruby'
data/test/helpers.rb ADDED
@@ -0,0 +1 @@
1
+ require 'test/unit'
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: iron_worker_ng
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.14
4
+ version: 0.2.0
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -10,7 +10,7 @@ authors:
10
10
  autorequire:
11
11
  bindir: bin
12
12
  cert_chain: []
13
- date: 2012-03-29 00:00:00.000000000 Z
13
+ date: 2012-04-09 00:00:00.000000000 Z
14
14
  dependencies:
15
15
  - !ruby/object:Gem::Dependency
16
16
  name: zip
@@ -121,6 +121,11 @@ files:
121
121
  - README.md
122
122
  - VERSION
123
123
  - bin/iron_worker_ng
124
+ - examples/ruby/hello/hello.rb
125
+ - examples/ruby/hello/hello_worker.rb
126
+ - examples/ruby/master_slave/master_slave.rb
127
+ - examples/ruby/master_slave/master_worker.rb
128
+ - examples/ruby/master_slave/slave_worker.rb
124
129
  - lib/iron_worker_ng.rb
125
130
  - lib/iron_worker_ng/api_client.rb
126
131
  - lib/iron_worker_ng/api_client_error.rb
@@ -138,7 +143,9 @@ files:
138
143
  - lib/iron_worker_ng/feature/ruby/merge_gem.rb
139
144
  - lib/iron_worker_ng/feature/ruby/merge_gemfile.rb
140
145
  - lib/iron_worker_ng/feature/ruby/merge_worker.rb
146
+ - lib/iron_worker_ng/logger.rb
141
147
  - lib/iron_worker_ng/version.rb
148
+ - test/helpers.rb
142
149
  homepage: https://github.com/iron-io/iron_worker_ruby_ng
143
150
  licenses: []
144
151
  post_install_message:
@@ -153,7 +160,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
153
160
  version: '0'
154
161
  segments:
155
162
  - 0
156
- hash: 462212253
163
+ hash: 862609601
157
164
  required_rubygems_version: !ruby/object:Gem::Requirement
158
165
  none: false
159
166
  requirements: