automatic 12.3.0 → 12.3.1
Sign up to get free protection for your applications and to get access to all the features.
- data/Gemfile +1 -0
- data/README.md +54 -36
- data/Rakefile +14 -0
- data/VERSION +1 -1
- data/automatic.gemspec +17 -5
- data/bin/automatic +37 -3
- data/bin/automatic-config +77 -0
- data/config/default.yml +9 -12
- data/doc/ChangeLog +32 -8
- data/doc/PLUGINS +205 -0
- data/doc/PLUGINS.ja +2 -3
- data/doc/README +488 -0
- data/doc/README.ja +195 -131
- data/lib/automatic/feed_parser.rb +1 -9
- data/lib/automatic/log.rb +1 -9
- data/lib/automatic/opml.rb +239 -0
- data/lib/automatic/pipeline.rb +16 -10
- data/lib/automatic/recipe.rb +3 -4
- data/lib/automatic.rb +32 -38
- data/lib/config/validator.rb +83 -0
- data/plugins/custom_feed/svn_log.rb +1 -1
- data/plugins/filter/ignore.rb +9 -1
- data/plugins/notify/ikachan.rb +7 -6
- data/plugins/publish/hatena_bookmark.rb +6 -9
- data/script/build +63 -0
- data/spec/lib/automatic/pipeline_spec.rb +55 -0
- data/spec/lib/automatic_spec.rb +77 -0
- data/spec/lib/pipeline_spec.rb +67 -0
- data/spec/plugins/filter/ignore_spec.rb +16 -0
- data/spec/plugins/filter/image_spec.rb +4 -4
- data/spec/plugins/filter/tumblr_resize_spec.rb +4 -4
- data/spec/plugins/notify/ikachan_spec.rb +30 -0
- data/spec/plugins/publish/console_spec.rb +1 -2
- data/spec/plugins/publish/hatena_bookmark_spec.rb +36 -1
- data/spec/plugins/store/full_text_spec.rb +0 -2
- data/spec/plugins/store/permalink_spec.rb +0 -1
- data/spec/plugins/store/target_link_spec.rb +0 -1
- data/spec/plugins/subscription/feed_spec.rb +0 -1
- data/spec/spec_helper.rb +6 -4
- data/spec/user_dir/plugins/store/mock.rb +12 -0
- data/test/fixtures/sampleOPML.xml +11 -0
- data/test/integration/test_activerecord.yml +2 -2
- data/test/integration/test_fulltext.yml +3 -3
- data/test/integration/test_hatenabookmark.yml +6 -2
- data/test/integration/test_ignore.yml +4 -1
- data/test/integration/test_ignore2.yml +1 -4
- data/test/integration/test_image2local.yml +3 -5
- data/test/integration/test_svnlog.yml +2 -1
- data/test/integration/test_tumblr2local.yml +3 -3
- metadata +43 -22
- data/utils/auto_discovery.rb +0 -18
- data/utils/opml_parser.rb +0 -247
data/doc/README
ADDED
@@ -0,0 +1,488 @@
|
|
1
|
+
automaticruby
|
2
|
+
|
3
|
+
Name
|
4
|
+
Automatic Ruby - Ruby General Automation Framework
|
5
|
+
|
6
|
+
Syntax
|
7
|
+
$ automatic
|
8
|
+
|
9
|
+
Specify the start of any recipe
|
10
|
+
$ automatic -c <recipe>
|
11
|
+
|
12
|
+
To view the version number
|
13
|
+
$ automatic -v
|
14
|
+
|
15
|
+
Description
|
16
|
+
This is a general-purpose automatic processing
|
17
|
+
ruby framework which can extend the functionality
|
18
|
+
by plug-ins.
|
19
|
+
|
20
|
+
|
21
|
+
============
|
22
|
+
Installation
|
23
|
+
============
|
24
|
+
|
25
|
+
[Stable]
|
26
|
+
$ gem install automatic
|
27
|
+
|
28
|
+
[Development]
|
29
|
+
$ git clone git://github.com/id774/automaticruby.git
|
30
|
+
|
31
|
+
|
32
|
+
If the ruby 1.9, put following command.
|
33
|
+
$ script/bootstrap
|
34
|
+
|
35
|
+
If the ruby 1.8, necessary to manually install the gems in the Gemfile.
|
36
|
+
|
37
|
+
|
38
|
+
===========
|
39
|
+
Get Started
|
40
|
+
===========
|
41
|
+
|
42
|
+
To create of any recipe (see below).
|
43
|
+
$ automatic -c your_recipe.yml
|
44
|
+
|
45
|
+
-c option to run with the manual. If there is no problem
|
46
|
+
to automate, register it to cron.
|
47
|
+
|
48
|
+
|
49
|
+
This is the easy recipe to simply output blog's feed to
|
50
|
+
only console. It can be used to check the operation of
|
51
|
+
this software.
|
52
|
+
$ automatic -c config/feed2console.yml
|
53
|
+
|
54
|
+
|
55
|
+
=======
|
56
|
+
Recipes
|
57
|
+
=======
|
58
|
+
|
59
|
+
Automatic Ruby parses configuration file that was written
|
60
|
+
in the form of YAML which including variety of information
|
61
|
+
of associated plug-ins.
|
62
|
+
|
63
|
+
This YAML file is called "Recipe".
|
64
|
+
|
65
|
+
When you start automatic ruby without argument -c option,
|
66
|
+
the config/default.yml is called. You can use -c option for
|
67
|
+
specify a file name.
|
68
|
+
|
69
|
+
[Example]
|
70
|
+
$ automatic -c ~/recipes/your_recipe.yml
|
71
|
+
|
72
|
+
|
73
|
+
=====================
|
74
|
+
How to write a Recipe
|
75
|
+
=====================
|
76
|
+
|
77
|
+
The Recipe has an implicit naming convention.
|
78
|
+
|
79
|
+
[Syntax]
|
80
|
+
|
81
|
+
plugins:
|
82
|
+
- module: MODULE_NAME
|
83
|
+
config:
|
84
|
+
VARIABLES
|
85
|
+
|
86
|
+
|
87
|
+
The following is a example.
|
88
|
+
|
89
|
+
plugins:
|
90
|
+
- module: SubscriptionFeed
|
91
|
+
config:
|
92
|
+
feeds:
|
93
|
+
- http://reretlet.tumblr.com/rss
|
94
|
+
|
95
|
+
- module: StorePermalink
|
96
|
+
config:
|
97
|
+
db: tumblr.db
|
98
|
+
|
99
|
+
- module: FilterImage
|
100
|
+
|
101
|
+
- module: FilterTumblrResize
|
102
|
+
|
103
|
+
- module: StoreTargetLink
|
104
|
+
config:
|
105
|
+
path: /Users/yourname/Desktop/
|
106
|
+
interval: 1
|
107
|
+
|
108
|
+
In the case of sample this recipe.
|
109
|
+
1. Subscribe to RSS feeds of Tumblr by SubScriptionFeed.
|
110
|
+
2. Save permalink to database by StorePermalink.
|
111
|
+
3. Specify the URL of the image by FilterImage and FilterTumblrResize.
|
112
|
+
4. Downloading image file of Tumblr by StoreTargetLink.
|
113
|
+
|
114
|
+
|
115
|
+
Showing another example as follows.
|
116
|
+
|
117
|
+
plugins:
|
118
|
+
- module: SubscriptionFeed
|
119
|
+
config:
|
120
|
+
feeds:
|
121
|
+
- http://example.com/rss2
|
122
|
+
- http://hogefuga.com/feed
|
123
|
+
|
124
|
+
- module: FilterIgnore
|
125
|
+
config:
|
126
|
+
link:
|
127
|
+
- hoge
|
128
|
+
- fuga
|
129
|
+
|
130
|
+
- module: StorePermalink
|
131
|
+
config:
|
132
|
+
db: permalink.db
|
133
|
+
|
134
|
+
- module: PublishHatenaBookmark
|
135
|
+
config:
|
136
|
+
username: your_hatena_id
|
137
|
+
password: your_password
|
138
|
+
interval: 5
|
139
|
+
|
140
|
+
In the case of sample this recipe.
|
141
|
+
1. Subscribe to 2 RSS feeds by SubScriptionFeed.
|
142
|
+
2. Feed ignore the keyword by FilterIgnore.
|
143
|
+
3. Save permalink to database by StorePermalink.
|
144
|
+
4. Social Bookmarking by PublishHatenaBookmark.
|
145
|
+
|
146
|
+
(Hatena Bookmark is a most popular social bookmark service
|
147
|
+
in Japan.)
|
148
|
+
|
149
|
+
|
150
|
+
These have been realized by a combination of plug-ins
|
151
|
+
a series of processes.
|
152
|
+
|
153
|
+
In this way, by simply writing to the recipe and
|
154
|
+
configuration information for plug-ins, free processing can
|
155
|
+
be achieved by a combination of multiple plug-ins.
|
156
|
+
|
157
|
+
The possibility of infinite depending on the combination
|
158
|
+
of plug-ins can be realized.
|
159
|
+
|
160
|
+
All depending on whether you make a plug-in looks like!
|
161
|
+
|
162
|
+
|
163
|
+
=======
|
164
|
+
Plug-in
|
165
|
+
=======
|
166
|
+
|
167
|
+
Plug-ins from the plugins directory (see below) will be loaded.
|
168
|
+
|
169
|
+
If ~ /.automatic/plugins exists, is loaded into them as well.
|
170
|
+
|
171
|
+
Put your own plug-ins in your home directory.
|
172
|
+
(How to make the plug-in will be described later.)
|
173
|
+
|
174
|
+
You can automatically generate a sub-directory for your own plug-ins
|
175
|
+
with the following command.
|
176
|
+
|
177
|
+
$ automatic-config scaffold
|
178
|
+
(For a description of automatic-config will be described later.)
|
179
|
+
|
180
|
+
|
181
|
+
============================
|
182
|
+
Directory and file structure
|
183
|
+
============================
|
184
|
+
.
|
185
|
+
|
|
186
|
+
+-+ bin
|
187
|
+
| |
|
188
|
+
| +-- automatic
|
189
|
+
| | The main file for run.
|
190
|
+
| |
|
191
|
+
| +-- automatic-config
|
192
|
+
| The tool for the auxiliary.
|
193
|
+
|
|
194
|
+
+-+ config
|
195
|
+
| |
|
196
|
+
| +-- default.yml
|
197
|
+
| To describe the recipe information in the form of yaml.
|
198
|
+
| They called in the argument -c option of Automatic Ruby.
|
199
|
+
| If you write a new recipe, locate it in the config directory.
|
200
|
+
|
|
201
|
+
|
|
202
|
+
+-+ plugins
|
203
|
+
| |
|
204
|
+
| +-- subscription
|
205
|
+
| | Plug-ins to subscribe to feeds.
|
206
|
+
| |
|
207
|
+
| +-- customfeed
|
208
|
+
| | Plug-ins to generate a custom feed.
|
209
|
+
| |
|
210
|
+
| +-- filter
|
211
|
+
| | Plug-ins to filter the information.
|
212
|
+
| |
|
213
|
+
| +-- store
|
214
|
+
| | Plug-ins to store the information to internally.
|
215
|
+
| |
|
216
|
+
| +-- notify
|
217
|
+
| | Plug-ins to notify the information.
|
218
|
+
| |
|
219
|
+
| +-- publish
|
220
|
+
| Plug-ins to send information to external.
|
221
|
+
|
|
222
|
+
| If you write a new plug-in, locate it in the plugins directory.
|
223
|
+
|
|
224
|
+
|
|
225
|
+
+-+ lib
|
226
|
+
| |
|
227
|
+
| +-- automatic.rb
|
228
|
+
| | To the definition of automatic module.
|
229
|
+
| |
|
230
|
+
| +-+ automatic
|
231
|
+
| | |
|
232
|
+
| | +-- environment.rb
|
233
|
+
| | | Read the environmental information.
|
234
|
+
| | |
|
235
|
+
| | +-- pipeline.rb
|
236
|
+
| | | To use an object called a pipeline
|
237
|
+
| | | sequentially processing plug-ins in recipes.
|
238
|
+
| | |
|
239
|
+
| | +-- feed_parser.rb
|
240
|
+
| | | To parse the feed.
|
241
|
+
| | |
|
242
|
+
| | +-- log.rb
|
243
|
+
| | | To output logs.
|
244
|
+
| | |
|
245
|
+
| | +-- recipe.rb
|
246
|
+
| | Read a recipes.
|
247
|
+
| |
|
248
|
+
| +-- config
|
249
|
+
|
|
250
|
+
+-+ db
|
251
|
+
| |
|
252
|
+
| +-- permalink.db
|
253
|
+
| Have been collected in the plug-in Store::Permalink.
|
254
|
+
|
|
255
|
+
+-+ script
|
256
|
+
| |
|
257
|
+
| +-- bootstrap
|
258
|
+
| To set the environment, such as bundle install.
|
259
|
+
| (Only more than ruby 1.9)
|
260
|
+
|
|
261
|
+
+-- spec
|
262
|
+
| Directory for unit tests (Use RSpec).
|
263
|
+
|
|
264
|
+
+-+ test
|
265
|
+
| |
|
266
|
+
| +-- integration
|
267
|
+
| Directory for the Integration Test.
|
268
|
+
|
|
269
|
+
+-+ vendor
|
270
|
+
| |
|
271
|
+
| +-- gems
|
272
|
+
| Bundle installed gem packages.
|
273
|
+
|
|
274
|
+
+---+ doc
|
275
|
+
|
|
276
|
+
+-- COPYING
|
277
|
+
| The license for this software.
|
278
|
+
|
|
279
|
+
+-- PLUGINS
|
280
|
+
| Documentation for plug-ins.
|
281
|
+
|
|
282
|
+
+-- README
|
283
|
+
This document.
|
284
|
+
|
285
|
+
|
286
|
+
|
287
|
+
===========
|
288
|
+
Development
|
289
|
+
===========
|
290
|
+
|
291
|
+
Repository
|
292
|
+
https://github.com/id774/automaticruby
|
293
|
+
|
294
|
+
Issues
|
295
|
+
https://github.com/id774/automaticruby/issues
|
296
|
+
|
297
|
+
RubyForge
|
298
|
+
http://rubyforge.org/projects/automatic/
|
299
|
+
|
300
|
+
CI
|
301
|
+
http://id774.net/jenkins/
|
302
|
+
|
303
|
+
|
304
|
+
===========
|
305
|
+
How to Join
|
306
|
+
===========
|
307
|
+
|
308
|
+
1. Fork the repository.
|
309
|
+
2. Write new plugin or existing code improvement.
|
310
|
+
3. Send pull request!
|
311
|
+
|
312
|
+
After that, we will give you the right to commit for a quick fix finely.
|
313
|
+
|
314
|
+
|
315
|
+
===========
|
316
|
+
Coding Rule
|
317
|
+
===========
|
318
|
+
|
319
|
+
2 tabs (Soft Tabs).
|
320
|
+
Remove trailing spaces.
|
321
|
+
Namespace and conventions, see the existing code.
|
322
|
+
{} is recommended than do end (To avoid end end end,,).
|
323
|
+
RDoc Header is written by creator of the file.
|
324
|
+
Write tests with RSPec.
|
325
|
+
Aim at 100% coverage.
|
326
|
+
|
327
|
+
|
328
|
+
===================
|
329
|
+
Automatic::Pipeline
|
330
|
+
===================
|
331
|
+
|
332
|
+
This framework sequentially analyze a plug-ins on yaml recipe.
|
333
|
+
At this time, pipeline passed instantance as an argument. And then,
|
334
|
+
pipeline backs again as the return value.
|
335
|
+
|
336
|
+
Code is as follows.
|
337
|
+
|
338
|
+
pipeline = []
|
339
|
+
recipe.each_plugin { |plugin|
|
340
|
+
load_plugin(plugin.module)
|
341
|
+
klass = Automatic::Plugin.const_get(plugin.module)
|
342
|
+
pipeline = klass.new(plugin.config, pipeline).run
|
343
|
+
}
|
344
|
+
|
345
|
+
In this iteration, sequential processing can be realized.
|
346
|
+
|
347
|
+
This mechanism called "Automatic::Pipeline".
|
348
|
+
|
349
|
+
|
350
|
+
|
351
|
+
=============================
|
352
|
+
Implementation of the plug-in
|
353
|
+
=============================
|
354
|
+
|
355
|
+
Plug-in is loaded by the constructor of YAML.load
|
356
|
+
Class is recieved the array of hashes as the instance
|
357
|
+
variable of pipelines object.
|
358
|
+
|
359
|
+
Example implementation of the constructor that is recommended
|
360
|
+
is as follows.
|
361
|
+
|
362
|
+
def initialize(config, pipeline=[])
|
363
|
+
@config = config
|
364
|
+
@pipeline = pipeline
|
365
|
+
end
|
366
|
+
|
367
|
+
Run an instance method is called automatically. The return value
|
368
|
+
is on the instance variable @pipeline. And then, be handed over
|
369
|
+
to the next plug-in in the recipe.
|
370
|
+
|
371
|
+
|
372
|
+
=========
|
373
|
+
Unit Test
|
374
|
+
=========
|
375
|
+
|
376
|
+
Unit testing is performed in RSpec.
|
377
|
+
|
378
|
+
Test of the plug-in, pass the pipeline which will be the argument,
|
379
|
+
to check the return value.
|
380
|
+
|
381
|
+
|
382
|
+
============
|
383
|
+
Binding Test
|
384
|
+
============
|
385
|
+
|
386
|
+
To combine more than one recipe under the test/integration.
|
387
|
+
Put the recipe for the binding test.
|
388
|
+
|
389
|
+
Before you check-in to the repository, run following
|
390
|
+
command with ruby 1.9.
|
391
|
+
$ script/build
|
392
|
+
|
393
|
+
By this, all RSpec tests and binding tests will be conducted.
|
394
|
+
You can check-in to the repository after all test successed.
|
395
|
+
Otherwise, CI will fail.
|
396
|
+
|
397
|
+
|
398
|
+
======================
|
399
|
+
Continuous Integration
|
400
|
+
======================
|
401
|
+
|
402
|
+
CI is performed in Jenkins.
|
403
|
+
http://id774.net/jenkins/
|
404
|
+
|
405
|
+
|
406
|
+
===========================
|
407
|
+
Description of the plug-ins
|
408
|
+
===========================
|
409
|
+
|
410
|
+
See doc/PLUGINS.
|
411
|
+
|
412
|
+
|
413
|
+
|
414
|
+
================
|
415
|
+
automatic-config
|
416
|
+
================
|
417
|
+
|
418
|
+
automatic-config is the auxiliary tool.
|
419
|
+
|
420
|
+
[Syntax]
|
421
|
+
$ automatic-config
|
422
|
+
To view a list of sub-command.
|
423
|
+
|
424
|
+
$ automatic-config scaffold
|
425
|
+
To generate a ~/.automatic directory in your home directory.
|
426
|
+
|
427
|
+
$ automatic-config autodiscovery <url>
|
428
|
+
Return the URL of the feed of target detected by auto discovery.
|
429
|
+
(ex.)
|
430
|
+
$ automatic-config autodiscovery http://id774.net/blog
|
431
|
+
["http://id774.net/blog/feed/", "http://id774.net/blog/comments/feed/"]
|
432
|
+
|
433
|
+
$ automatic-config opmlparser <opml path>
|
434
|
+
To output the URLs to parse the OPML file.
|
435
|
+
(ex.)
|
436
|
+
$ automatic-config opmlparser opml.xml > feeds.txt
|
437
|
+
|
438
|
+
$ automatic-config feedparser <url>
|
439
|
+
Return the contents to parse the feed.
|
440
|
+
|
441
|
+
$ automatic-config log <level> <message>
|
442
|
+
To output log messages in the form of Automatic Ruby.
|
443
|
+
|
444
|
+
|
445
|
+
|
446
|
+
==============
|
447
|
+
Update History
|
448
|
+
==============
|
449
|
+
|
450
|
+
See doc/ChangeLog.
|
451
|
+
|
452
|
+
|
453
|
+
====
|
454
|
+
TODO
|
455
|
+
====
|
456
|
+
|
457
|
+
Refer to the issues of github
|
458
|
+
https://github.com/id774/automaticruby/issues
|
459
|
+
|
460
|
+
|
461
|
+
===========
|
462
|
+
Environment
|
463
|
+
===========
|
464
|
+
|
465
|
+
ruby 1.8 or later
|
466
|
+
|
467
|
+
|
468
|
+
The following packages depend on the gem.
|
469
|
+
|
470
|
+
sqlite3
|
471
|
+
activesupport
|
472
|
+
hashie
|
473
|
+
activerecord
|
474
|
+
gcalapi
|
475
|
+
xml-simple
|
476
|
+
feedbag
|
477
|
+
|
478
|
+
For more information, see Gemfile.
|
479
|
+
|
480
|
+
|
481
|
+
=====
|
482
|
+
Notes
|
483
|
+
=====
|
484
|
+
|
485
|
+
To avoid excessive scraping, use in the bounds of common sense.
|
486
|
+
This software licensed under the GNU GENERAL PUBLIC LICENSE, Version 3.0.
|
487
|
+
|
488
|
+
|