org-converge 0.0.4 → 0.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.org +112 -62
- data/TODO +20 -6
- data/examples/simple/setup.org +0 -9
- data/lib/org-converge/babel_output_buffer.rb +12 -7
- data/lib/org-converge/command.rb +27 -9
- data/lib/org-converge/version.rb +1 -1
- data/lib/org-converge.rb +9 -18
- data/org-converge.gemspec +1 -1
- data/spec/converge_examples/basic_run_example/setup.org +29 -17
- data/spec/converge_examples/runlist_example/setup.org +4 -1
- data/spec/converge_spec.rb +1 -2
- metadata +11 -12
- data/README.md +0 -157
data/README.org
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
# -*- mode: org; mode: auto-fill; -*-
|
2
|
-
#+TITLE: Org Converge
|
3
2
|
#+STARTUP: showeverything
|
4
3
|
|
5
|
-
*
|
4
|
+
* Org Converge
|
5
|
+
** Description
|
6
6
|
|
7
7
|
This attempts to be an experiment of using Org mode syntax to
|
8
8
|
describe, configure and setting up something, borrowing some ideas
|
9
9
|
of what is possible to do with tools like =chef-solo=, =puppet=,
|
10
10
|
=ansible=, etc...
|
11
11
|
|
12
|
-
|
12
|
+
** Motivation
|
13
13
|
|
14
14
|
The Org babel syntax has proven to be flexible enough to produce
|
15
15
|
/reproducible research/ papers. Then, I believe that configuring and setting up
|
@@ -17,11 +17,84 @@
|
|
17
17
|
the same syntax, given that /converging/ the configuration is something
|
18
18
|
that one ought to be able to reproduce.
|
19
19
|
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
20
|
+
** Usage
|
21
|
+
|
22
|
+
To run the blocks in parallel:
|
23
|
+
|
24
|
+
#+begin_src sh
|
25
|
+
org-converge path/to/setup-file.org
|
26
|
+
#+end_src
|
27
|
+
|
28
|
+
Or sequentially:
|
29
|
+
|
30
|
+
#+begin_src sh
|
31
|
+
org-converge path/to/setup-file.org --runmode=sequential
|
32
|
+
#+end_src
|
33
|
+
|
34
|
+
** Quick example
|
35
|
+
|
36
|
+
The following is an example of running 3 processes
|
37
|
+
in parallel by defining them as code blocks from
|
38
|
+
an Org mode file:
|
39
|
+
|
40
|
+
#+begin_src sh
|
41
|
+
,#+TITLE: Running Org babel processes in parallel
|
42
|
+
|
43
|
+
* Print with different languages
|
44
|
+
|
45
|
+
,#+name: hello_from_bash
|
46
|
+
,#+begin_src sh :shebang #!/bin/bash
|
47
|
+
while true; do echo "hello world from bash"; sleep 1; done
|
48
|
+
,#+end_src
|
49
|
+
|
50
|
+
,#+name: hello_from_ruby
|
51
|
+
,#+begin_src ruby :shebang #!/usr/local/bin/ruby
|
52
|
+
$stdout.sync = true
|
53
|
+
loop { puts "hello world from ruby" ; sleep 1 }
|
54
|
+
,#+end_src
|
55
|
+
|
56
|
+
,#+name: hello_from_python
|
57
|
+
,#+begin_src python :shebang #!/usr/bin/python
|
58
|
+
import time
|
59
|
+
import sys
|
60
|
+
for i in range(0,100):
|
61
|
+
print "hello world from python"
|
62
|
+
sys.stdout.flush()
|
63
|
+
time.sleep(1)
|
64
|
+
,#+end_src
|
65
|
+
#+end_src
|
66
|
+
|
67
|
+
We store this in a file named =hello.org= and then run it as follows:
|
68
|
+
|
69
|
+
#+begin_src sh
|
70
|
+
org-converge hello.org
|
71
|
+
#+end_src
|
72
|
+
|
73
|
+
This would produce an output similar to the following:
|
74
|
+
|
75
|
+
#+begin_src sh
|
76
|
+
I, [2014-04-17T23:48:44.663545 #1710] INFO -- : Tangling 0 files...
|
77
|
+
I, [2014-04-17T23:48:44.663698 #1710] INFO -- : Tangling succeeded!
|
78
|
+
I, [2014-04-17T23:48:44.664829 #1710] INFO -- : Running code blocks now! (4 runnable blocks found in total)
|
79
|
+
I, [2014-04-17T23:48:44.762899 #1710] INFO -- : hello_from_bash (1711) -- started with pid 1711
|
80
|
+
I, [2014-04-17T23:48:44.808717 #1710] INFO -- : hello_from_ruby (1712) -- started with pid 1712
|
81
|
+
I, [2014-04-17T23:48:44.837878 #1710] INFO -- : hello_from_python (1713) -- started with pid 1713
|
82
|
+
I, [2014-04-17T23:48:44.878336 #1710] INFO -- : (1714) -- started with pid 1714
|
83
|
+
I, [2014-04-17T23:48:44.879284 #1710] INFO -- : hello_from_bash (1711) -- hello world from bash
|
84
|
+
I, [2014-04-17T23:48:44.882778 #1710] INFO -- : hello_from_ruby (1712) -- hello world from ruby
|
85
|
+
I, [2014-04-17T23:48:44.883637 #1710] INFO -- : hello_from_python (1713) -- hello world from python
|
86
|
+
I, [2014-04-17T23:48:45.778681 #1710] INFO -- : hello_from_bash (1711) -- hello world from bash
|
87
|
+
I, [2014-04-17T23:48:45.792321 #1710] INFO -- : hello_from_ruby (1712) -- hello world from ruby
|
88
|
+
I, [2014-04-17T23:48:45.882020 #1710] INFO -- : hello_from_python (1713) -- hello world from python
|
89
|
+
I, [2014-04-17T23:48:46.787106 #1710] INFO -- : hello_from_bash (1711) -- hello world from bash
|
90
|
+
I, [2014-04-17T23:48:46.793347 #1710] INFO -- : hello_from_ruby (1712) -- hello world from ruby
|
91
|
+
I, [2014-04-17T23:48:46.883511 #1710] INFO -- : hello_from_python (1713) -- hello world from python
|
92
|
+
I, [2014-04-17T23:48:47.794846 #1710] INFO -- : hello_from_ruby (1712) -- hello world from ruby
|
93
|
+
I, [2014-04-17T23:48:47.796387 #1710] INFO -- : hello_from_bash (1711) -- hello world from bash
|
94
|
+
I, [2014-04-17T23:48:47.884196 #1710] INFO -- : hello_from_python (1713) -- hello world from python
|
95
|
+
#+end_src
|
96
|
+
|
97
|
+
** How it works
|
25
98
|
|
26
99
|
Org Converge uses an liberally extended version of Org Babel
|
27
100
|
features in order to give support for converging the configuration
|
@@ -30,10 +103,7 @@
|
|
30
103
|
For example, using Org Babel and macros we can easily spread config
|
31
104
|
files on a server by writing the following on a ~server.org~ file.
|
32
105
|
|
33
|
-
#+begin_src
|
34
|
-
,#+MACRO: multitenancy_enabled true
|
35
|
-
,#+MACRO: status_port true
|
36
|
-
|
106
|
+
#+begin_src sh
|
37
107
|
,#+begin_src yaml :tangle /etc/component.yml
|
38
108
|
multitenant: false
|
39
109
|
status_port: 10004
|
@@ -41,69 +111,49 @@
|
|
41
111
|
#+end_src
|
42
112
|
|
43
113
|
And then configure it by running it as follows, (considering we have
|
44
|
-
the correct permissions):
|
45
|
-
|
46
|
-
#+begin_src sh
|
47
|
-
org-converge server.org
|
48
|
-
#+end_src
|
49
|
-
|
50
|
-
This leverages on the syntax already provided by Org Babel, but one
|
51
|
-
difference here is that if we run it once again without changes...
|
114
|
+
the correct permissions for tangling at =/etc/component.yml=):
|
52
115
|
|
53
116
|
#+begin_src sh
|
54
|
-
org-converge server.org
|
117
|
+
sudo org-converge server.org
|
55
118
|
#+end_src
|
56
|
-
|
57
|
-
...it would finish soon since the configuration has already converged.
|
58
119
|
|
59
120
|
Next, let's say that we no only one want to set the configured templates,
|
60
121
|
but that we also want to install some packages. In that case, we
|
61
122
|
should be able to do the following:
|
62
123
|
|
63
|
-
#+begin_src
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
,#+begin_src sh
|
87
|
-
cd {{{project_path}}}
|
88
|
-
bundle install
|
89
|
-
,#+end_src
|
124
|
+
#+begin_src sh
|
125
|
+
,* Configuring the component
|
126
|
+
|
127
|
+
,#+begin_src yaml :tangle /etc/component.yml
|
128
|
+
multitenant: false
|
129
|
+
status_port: 10004
|
130
|
+
,#+end_src
|
131
|
+
|
132
|
+
,* Installing the dependencies
|
133
|
+
|
134
|
+
Need the following so that ~bundle install~ can compile
|
135
|
+
the native extensions correctly.
|
136
|
+
|
137
|
+
,#+begin_src sh
|
138
|
+
apt-get install build-essentials -y
|
139
|
+
,#+end_src
|
140
|
+
|
141
|
+
Then the following should work:
|
142
|
+
|
143
|
+
,#+begin_src sh
|
144
|
+
cd project_path
|
145
|
+
bundle install
|
146
|
+
,#+end_src
|
90
147
|
#+end_src
|
91
148
|
|
92
149
|
As long as the repo has been already checked out in the directory,
|
93
150
|
the previous example will succeed.
|
94
151
|
|
95
|
-
|
96
|
-
org-converge server.org
|
97
|
-
#+end_src
|
98
|
-
|
99
|
-
If that is not the case, then org-converge will fail
|
100
|
-
and pickup from that last step.
|
101
|
-
|
102
|
-
More practical examples can be found [[here]], more will be added as
|
152
|
+
More practical examples can be found [[https://github.com/wallyqs/org-converge/tree/master/examples][here]], more will be added as
|
103
153
|
long as dogfooding from this goes well.
|
104
154
|
|
105
|
-
|
155
|
+
** Contributing
|
106
156
|
|
107
|
-
The project is in very early development at this moment, but if you
|
108
|
-
feel that it is interesting enough, please create a ticket so start
|
109
|
-
the discussion.
|
157
|
+
The project is in very early development at this moment, but if you
|
158
|
+
feel that it is interesting enough, please create a ticket so start
|
159
|
+
the discussion.
|
data/TODO
CHANGED
@@ -2,9 +2,8 @@
|
|
2
2
|
#+TODO: TODO | DONE CANCELED
|
3
3
|
#+startup: showeverything
|
4
4
|
|
5
|
-
* [0/
|
5
|
+
* [0/7] > 0.0.9 version
|
6
6
|
|
7
|
-
- [ ] Support runlists notifications via =#+NAME:=
|
8
7
|
- [ ] Macros can be loaded and applied to the configuration
|
9
8
|
- [ ] Actually support converging and idempotency
|
10
9
|
+ Do not do an operation unless it is required
|
@@ -16,6 +15,21 @@
|
|
16
15
|
- [ ] Use :eval for evaling blocks (off by default)
|
17
16
|
- [ ] Can use :dir for running a process relative to that directory
|
18
17
|
|
18
|
+
* [0/3] 0.0.6 version
|
19
|
+
|
20
|
+
- [ ] Keyword to accumulate run blocks
|
21
|
+
- [ ] Do not run commented blocks
|
22
|
+
- [ ] Use rake task manager
|
23
|
+
|
24
|
+
* [2/2] 0.0.5 version
|
25
|
+
|
26
|
+
- [X] Do not run or tangle blocks without arguments
|
27
|
+
- [X] Keyword to activate runmode in the text itself
|
28
|
+
|
29
|
+
* [1/1] 0.0.4 version
|
30
|
+
|
31
|
+
- [X] Support runlists notifications via =#+NAME:=
|
32
|
+
|
19
33
|
* [2/2] 0.0.3 version
|
20
34
|
|
21
35
|
- [X] Support sequential and parallel execution
|
@@ -39,7 +53,7 @@ Need some basic functionality of what Org babel offers first.
|
|
39
53
|
- [X] ~--tangle~ flag
|
40
54
|
- [X] Support a root dir for when not running relative to the directory
|
41
55
|
|
42
|
-
* [
|
56
|
+
* [3/19] Ideas
|
43
57
|
** CANCELED How to set the permissions from the directory from the file that is being tangled when it does not exists?
|
44
58
|
|
45
59
|
By default, this would be 0644, but we also need to specify the
|
@@ -131,7 +145,7 @@ Kind of like the chef-solo runlist, a headline like this...
|
|
131
145
|
org-converge fluentd.org -t config
|
132
146
|
#+end_src
|
133
147
|
|
134
|
-
**
|
148
|
+
** DONE Managing dependencies: could be handled with ~#+include~ directives
|
135
149
|
|
136
150
|
One idea is that it would be useful to compile different manuals
|
137
151
|
that are dependent among each other.
|
@@ -181,7 +195,7 @@ sudo service td-agent start
|
|
181
195
|
sudo /etc/register-to-balancer
|
182
196
|
#+end_src
|
183
197
|
|
184
|
-
**
|
198
|
+
** DONE ~#+NAME:~ could be used in the logger for identifying the process
|
185
199
|
** TODO Support caching?
|
186
200
|
|
187
201
|
#+begin_src emacs-lisp :cache yes :exports results
|
@@ -199,6 +213,6 @@ sudo /etc/register-to-balancer
|
|
199
213
|
|
200
214
|
* Links
|
201
215
|
|
202
|
-
Super useful notes:
|
203
216
|
http://eschulte.github.io/org-scraps
|
204
217
|
|
218
|
+
http://doc.norang.ca/org-mode.html
|
data/examples/simple/setup.org
CHANGED
@@ -7,20 +7,11 @@ It does not get more simple than this:
|
|
7
7
|
#+begin_src conf :tangle /tmp/my-conf
|
8
8
|
<source>
|
9
9
|
bind 0.0.0.0
|
10
|
-
|
11
|
-
|
12
|
-
|
13
10
|
port 4224
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
11
|
</source>
|
19
12
|
|
20
13
|
<match **>
|
21
14
|
type file
|
22
|
-
|
23
|
-
|
24
15
|
path /var/tmp/a.log
|
25
16
|
</match>
|
26
17
|
#+end_src
|
@@ -2,10 +2,13 @@ module Orgmode
|
|
2
2
|
class BabelOutputBuffer < OutputBuffer
|
3
3
|
attr_reader :tangle
|
4
4
|
attr_reader :scripts
|
5
|
+
attr_reader :in_buffer_settings
|
5
6
|
|
6
|
-
def initialize(output)
|
7
|
+
def initialize(output, opts={})
|
7
8
|
super(output)
|
8
9
|
|
10
|
+
@in_buffer_settings = opts[:in_buffer_settings]
|
11
|
+
|
9
12
|
# ~@tangle~ files are put in the right path
|
10
13
|
# : @tangle['/path'] = [Lines]
|
11
14
|
@tangle = Hash.new {|h,k| h[k] = {:lines => '', :header => {}, :lang => ''}}
|
@@ -14,7 +17,6 @@ module Orgmode
|
|
14
17
|
# : @scripts = [text, text, ...]
|
15
18
|
@scripts = Hash.new {|h,k| h[k] = {:lines => '', :header => {}, :lang => ''}}
|
16
19
|
@scripts_counter = 0
|
17
|
-
|
18
20
|
@buffer = ''
|
19
21
|
end
|
20
22
|
|
@@ -29,7 +31,7 @@ module Orgmode
|
|
29
31
|
end
|
30
32
|
|
31
33
|
def insert(line)
|
32
|
-
# We try to get the lang from #+BEGIN_SRC blocks
|
34
|
+
# We try to get the lang from #+BEGIN_SRC and #+BEGIN_EXAMPLE blocks
|
33
35
|
if line.begin_block?
|
34
36
|
case
|
35
37
|
when line.block_header_arguments[':tangle']
|
@@ -39,7 +41,8 @@ module Orgmode
|
|
39
41
|
:mkdirp => line.block_header_arguments[':mkdirp']
|
40
42
|
}
|
41
43
|
@tangle[@current_tangle][:lang] = line.block_lang
|
42
|
-
when line.
|
44
|
+
when line.properties['block_name']
|
45
|
+
# unnamed blocks are not run
|
43
46
|
@current_tangle = nil
|
44
47
|
@buffer = ''
|
45
48
|
|
@@ -50,11 +53,12 @@ module Orgmode
|
|
50
53
|
:name => line.properties['block_name']
|
51
54
|
}
|
52
55
|
@scripts[@scripts_counter][:lang] = line.block_lang
|
53
|
-
|
54
56
|
# TODO: have a way to specify which are the default binaries to be used per language
|
55
57
|
# when binary_detected?(@block_lang)
|
56
58
|
else
|
57
|
-
#
|
59
|
+
# reset tangling
|
60
|
+
@current_tangle = nil
|
61
|
+
@buffer = ''
|
58
62
|
end
|
59
63
|
end
|
60
64
|
|
@@ -66,7 +70,8 @@ module Orgmode
|
|
66
70
|
# When a tangle is not going on, it means that the lines would go
|
67
71
|
# into a runnable script
|
68
72
|
@buffer << line.output_text << "\n"
|
69
|
-
when (!@buffer.empty? and not (line.begin_block? or
|
73
|
+
when (!@buffer.empty? and not (line.begin_block? or \
|
74
|
+
line.assigned_paragraph_type == :code))
|
70
75
|
# Fix indentation and remove fix commas from Org mode before flushing
|
71
76
|
strip_code_block!
|
72
77
|
@scripts[@scripts_counter][:lines] << @buffer
|
data/lib/org-converge/command.rb
CHANGED
@@ -37,14 +37,24 @@ module OrgConverge
|
|
37
37
|
|
38
38
|
def converge!
|
39
39
|
tangle!
|
40
|
-
|
40
|
+
if @options['--runmode']
|
41
|
+
dispatch_runmode(@options['--runmode'])
|
42
|
+
else
|
43
|
+
# Try to find one in the buffer
|
44
|
+
runmode = ob.in_buffer_settings['RUNMODE']
|
45
|
+
dispatch_runmode(runmode)
|
46
|
+
end
|
47
|
+
end
|
48
|
+
|
49
|
+
def dispatch_runmode(runmode)
|
50
|
+
case runmode
|
41
51
|
when 'parallel'
|
42
52
|
run_blocks_in_parallel!
|
43
53
|
when 'sequential'
|
44
54
|
run_blocks_sequentially!
|
45
55
|
when 'runlist'
|
46
56
|
# TODO
|
47
|
-
else
|
57
|
+
else # parallel by default
|
48
58
|
run_blocks_in_parallel!
|
49
59
|
end
|
50
60
|
end
|
@@ -52,7 +62,7 @@ module OrgConverge
|
|
52
62
|
def tangle!
|
53
63
|
results = babel.tangle!
|
54
64
|
rescue Orgmode::Babel::TangleError
|
55
|
-
logger.error "Cannot converge because there were errors during tangle step".red
|
65
|
+
logger.error "Cannot converge because there were errors during tangle step".fg 'red'
|
56
66
|
end
|
57
67
|
|
58
68
|
def run_blocks_sequentially!
|
@@ -66,14 +76,18 @@ module OrgConverge
|
|
66
76
|
|
67
77
|
while not runlist_stack.empty?
|
68
78
|
key, script = runlist_stack.shift
|
79
|
+
|
80
|
+
# Decision: Only run blocks which have a name
|
81
|
+
next unless script[:header][:name]
|
82
|
+
|
83
|
+
display_name = script[:header][:name]
|
69
84
|
with_running_engine do |engine|
|
70
85
|
file = File.expand_path("#{@run_dir}/#{key}")
|
71
86
|
cmd = "#{script[:lang]} #{file}"
|
72
|
-
display_name = script[:header][:name] || script[:lang]
|
73
87
|
engine.register display_name, cmd, { :cwd => @root_dir, :logger => logger }
|
74
88
|
end
|
75
89
|
end
|
76
|
-
logger.info "Run has completed successfully.".green
|
90
|
+
logger.info "Run has completed successfully.".fg 'green'
|
77
91
|
end
|
78
92
|
|
79
93
|
# TODO: Too much foreman has made this running blocks in parallel the default behavior.
|
@@ -84,12 +98,16 @@ module OrgConverge
|
|
84
98
|
babel.ob.scripts.each do |key, script|
|
85
99
|
file = File.expand_path("#{@run_dir}/#{key}")
|
86
100
|
cmd = "#{script[:lang]} #{file}"
|
87
|
-
|
101
|
+
|
102
|
+
# Decision: Only run blocks which have a name
|
103
|
+
next unless script[:header][:name]
|
104
|
+
|
105
|
+
display_name = script[:header][:name]
|
88
106
|
@engine.register display_name, cmd, { :cwd => @root_dir, :logger => logger }
|
89
107
|
end
|
90
108
|
logger.info "Running code blocks now! (#{babel.ob.scripts.count} runnable blocks found in total)"
|
91
109
|
@engine.start
|
92
|
-
logger.info "Run has completed successfully.".green
|
110
|
+
logger.info "Run has completed successfully.".fg 'green'
|
93
111
|
end
|
94
112
|
|
95
113
|
def with_running_engine
|
@@ -104,12 +122,12 @@ module OrgConverge
|
|
104
122
|
|
105
123
|
def showfiles
|
106
124
|
ob.tangle.each do |file, block|
|
107
|
-
puts "---------- #{file} --------------".green
|
125
|
+
puts "---------- #{file} --------------".fg 'green'
|
108
126
|
puts block[:lines]
|
109
127
|
end
|
110
128
|
|
111
129
|
ob.scripts.each do |index, block|
|
112
|
-
puts "---------- script: #{index} to be run with: #{block[:header][:shebang]} --------------".green
|
130
|
+
puts "---------- script: #{index} to be run with: #{block[:header][:shebang]} --------------".fg 'green'
|
113
131
|
puts block[:lines]
|
114
132
|
end
|
115
133
|
end
|
data/lib/org-converge/version.rb
CHANGED
data/lib/org-converge.rb
CHANGED
@@ -14,7 +14,10 @@ module Orgmode
|
|
14
14
|
# Feed the parsed contens and create the necessary internal structures
|
15
15
|
# for doing babel like features
|
16
16
|
output = ''
|
17
|
-
|
17
|
+
babel_options = {
|
18
|
+
:in_buffer_settings => @in_buffer_settings
|
19
|
+
}
|
20
|
+
ob = BabelOutputBuffer.new(output, babel_options)
|
18
21
|
translate(@header_lines, ob)
|
19
22
|
@headlines.each do |headline|
|
20
23
|
translate(headline.body_lines, ob)
|
@@ -25,20 +28,8 @@ module Orgmode
|
|
25
28
|
end
|
26
29
|
end
|
27
30
|
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
def magenta; colorize("\e[0m\e[35m"); end
|
34
|
-
def cyan; colorize("\e[0m\e[36m"); end
|
35
|
-
def white; colorize("\e[0m\e[37m"); end
|
36
|
-
def bright_black; colorize("\e[0m\e[30m"); end
|
37
|
-
def bright_magenta; colorize("\e[0m\e[35m"); end
|
38
|
-
def bold; colorize("\e[0m\e[1m"); end
|
39
|
-
def colorize(color_code); "#{color_code}#{self}\e[0m"; end
|
40
|
-
end
|
41
|
-
|
42
|
-
class String
|
43
|
-
include StringWithColors
|
44
|
-
end
|
31
|
+
require 'tco'
|
32
|
+
conf = Tco.config
|
33
|
+
conf.names["green"] = "#02a552"
|
34
|
+
conf.names["red"] = "#eb443b"
|
35
|
+
Tco.reconfigure conf
|
data/org-converge.gemspec
CHANGED
@@ -15,7 +15,7 @@ Gem::Specification.new do |gem|
|
|
15
15
|
gem.require_paths = ["lib"]
|
16
16
|
gem.version = OrgConverge::VERSION
|
17
17
|
gem.add_runtime_dependency('docopt', '0.5.0')
|
18
|
-
gem.add_runtime_dependency('org-ruby', '~> 0.9.
|
18
|
+
gem.add_runtime_dependency('org-ruby', '~> 0.9.4')
|
19
19
|
gem.add_runtime_dependency('foreman', '~> 0.63.0')
|
20
20
|
gem.add_runtime_dependency('tco', '~> 0.1.0')
|
21
21
|
end
|
@@ -1,33 +1,45 @@
|
|
1
|
-
#+
|
1
|
+
#+TITLE: Running scripts in parallel
|
2
2
|
|
3
3
|
In the following example, the following 3 scripts should be run in parallel
|
4
4
|
with all output being flushed to the screen.
|
5
5
|
|
6
6
|
- Count some numbers with bash
|
7
7
|
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
8
|
+
#+name: bash_counter
|
9
|
+
#+begin_src sh :shebang #!/bin/bash
|
10
|
+
echo '' > out.log
|
11
|
+
for i in `seq 1 5`; do
|
12
|
+
echo "Writing! $i"
|
13
|
+
echo "hello $i" >> out.log
|
14
|
+
sleep $(($RANDOM % 5))
|
15
|
+
done
|
16
|
+
#+end_src
|
16
17
|
|
17
18
|
- Count some numbers with ruby
|
18
19
|
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
20
|
+
#+name: ruby_counter
|
21
|
+
#+begin_src ruby :shebang #!/usr/bin/ruby
|
22
|
+
$stdout.sync = true
|
23
|
+
sleep 0.2
|
24
|
+
10.times do |n|
|
25
|
+
puts "And now writing! #{n}"
|
26
|
+
File.open("out.log", "a") {|f| f.puts "Hello again #{n}" }
|
27
|
+
sleep rand.round(2)
|
28
|
+
end
|
29
|
+
#+end_src
|
27
30
|
|
28
31
|
- Print some numbers with python
|
29
32
|
|
33
|
+
#+name: python_counter
|
30
34
|
#+begin_src python :shebang #!/usr/bin/python
|
31
35
|
for i in range(0,3):
|
32
36
|
print i
|
33
37
|
#+end_src
|
38
|
+
|
39
|
+
- Block that should not be run
|
40
|
+
|
41
|
+
#+begin_src js
|
42
|
+
{
|
43
|
+
"hello": "world"
|
44
|
+
}
|
45
|
+
#+end_src
|
@@ -1,8 +1,10 @@
|
|
1
1
|
# -*- mode: org; mode: auto-fill; -*-
|
2
|
-
#+TITLE:
|
2
|
+
#+TITLE: Runlist example
|
3
|
+
#+runmode: sequential
|
3
4
|
|
4
5
|
In this example, we will have scripts being chained one after another.
|
5
6
|
|
7
|
+
#+name: first
|
6
8
|
#+begin_src sh :shebang #!/bin/bash
|
7
9
|
sleep 5
|
8
10
|
echo "first" > out.log
|
@@ -10,6 +12,7 @@ echo "first" > out.log
|
|
10
12
|
|
11
13
|
The script below will only be called once the one above is run.
|
12
14
|
|
15
|
+
#+name: second
|
13
16
|
#+begin_src ruby :shebang #!/usr/local/bin/ruby
|
14
17
|
File.open("out.log", 'a') {|f| f.puts "second" }
|
15
18
|
#+end_src
|
data/spec/converge_spec.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: org-converge
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.5
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,11 +9,11 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2014-04-
|
12
|
+
date: 2014-04-20 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: docopt
|
16
|
-
requirement: &
|
16
|
+
requirement: &70344197198100 !ruby/object:Gem::Requirement
|
17
17
|
none: false
|
18
18
|
requirements:
|
19
19
|
- - =
|
@@ -21,21 +21,21 @@ dependencies:
|
|
21
21
|
version: 0.5.0
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
|
-
version_requirements: *
|
24
|
+
version_requirements: *70344197198100
|
25
25
|
- !ruby/object:Gem::Dependency
|
26
26
|
name: org-ruby
|
27
|
-
requirement: &
|
27
|
+
requirement: &70344197197620 !ruby/object:Gem::Requirement
|
28
28
|
none: false
|
29
29
|
requirements:
|
30
30
|
- - ~>
|
31
31
|
- !ruby/object:Gem::Version
|
32
|
-
version: 0.9.
|
32
|
+
version: 0.9.4
|
33
33
|
type: :runtime
|
34
34
|
prerelease: false
|
35
|
-
version_requirements: *
|
35
|
+
version_requirements: *70344197197620
|
36
36
|
- !ruby/object:Gem::Dependency
|
37
37
|
name: foreman
|
38
|
-
requirement: &
|
38
|
+
requirement: &70344196892060 !ruby/object:Gem::Requirement
|
39
39
|
none: false
|
40
40
|
requirements:
|
41
41
|
- - ~>
|
@@ -43,10 +43,10 @@ dependencies:
|
|
43
43
|
version: 0.63.0
|
44
44
|
type: :runtime
|
45
45
|
prerelease: false
|
46
|
-
version_requirements: *
|
46
|
+
version_requirements: *70344196892060
|
47
47
|
- !ruby/object:Gem::Dependency
|
48
48
|
name: tco
|
49
|
-
requirement: &
|
49
|
+
requirement: &70344196890840 !ruby/object:Gem::Requirement
|
50
50
|
none: false
|
51
51
|
requirements:
|
52
52
|
- - ~>
|
@@ -54,7 +54,7 @@ dependencies:
|
|
54
54
|
version: 0.1.0
|
55
55
|
type: :runtime
|
56
56
|
prerelease: false
|
57
|
-
version_requirements: *
|
57
|
+
version_requirements: *70344196890840
|
58
58
|
description: A light configuration management tool for Org mode
|
59
59
|
email:
|
60
60
|
- waldemar.quevedo@gmail.com
|
@@ -67,7 +67,6 @@ files:
|
|
67
67
|
- .travis.yml
|
68
68
|
- Gemfile
|
69
69
|
- LICENSE
|
70
|
-
- README.md
|
71
70
|
- README.org
|
72
71
|
- Rakefile
|
73
72
|
- TODO
|
data/README.md
DELETED
@@ -1,157 +0,0 @@
|
|
1
|
-
Org Converge
|
2
|
-
------------
|
3
|
-
[](https://travis-ci.org/wallyqs/org-converge)
|
4
|
-
|
5
|
-
# Description
|
6
|
-
|
7
|
-
A tool which uses Org mode syntax to describe, configure and
|
8
|
-
setup something, borrowing some ideas of what is possible to
|
9
|
-
do with tools like `chef-solo`, `puppet`, `ansible`, etc...
|
10
|
-
|
11
|
-
Can also be used for tangling code blocks like Org Babel does.
|
12
|
-
|
13
|
-
# Installing
|
14
|
-
|
15
|
-
$ gem install org-converge
|
16
|
-
|
17
|
-
# Motivation
|
18
|
-
|
19
|
-
The Org babel syntax has proven to be flexible enough to produce
|
20
|
-
*reproducible research* papers. Then, I believe that configuring and setting up
|
21
|
-
a server for example is something that could be also be done using
|
22
|
-
the same syntax, given that *converging* the configuration is something
|
23
|
-
that one ought to be able to reproduce.
|
24
|
-
|
25
|
-
# Example usage
|
26
|
-
|
27
|
-
Inspecting the files that would be tangled:
|
28
|
-
|
29
|
-
$ org-converge spec/converge_examples/basic_tangle/setup.org --showfiles
|
30
|
-
|
31
|
-
---------- conf.yml --------------
|
32
|
-
bind: 0.0.0.0
|
33
|
-
port: 2442
|
34
|
-
|
35
|
-
mysql:
|
36
|
-
db: users
|
37
|
-
host: somewhere-example.local
|
38
|
-
password: 111111111
|
39
|
-
|
40
|
-
Executing the tangle:
|
41
|
-
|
42
|
-
$ org-converge spec/converge_examples/basic_tangle/setup.org --tangle
|
43
|
-
|
44
|
-
I, [2014-03-24T00:21:08.073506 #660] INFO -- : Tangling 1 files...
|
45
|
-
I, [2014-03-24T00:21:08.073668 #660] INFO -- : BEGIN(conf.yml): Tangling 7 lines at 'conf.yml'
|
46
|
-
I, [2014-03-24T00:21:08.075562 #660] INFO -- : END(conf.yml): done.
|
47
|
-
I, [2014-03-24T00:21:08.075638 #660] INFO -- : Tangling succeeded!
|
48
|
-
|
49
|
-
Run the code blocks in sequence:
|
50
|
-
|
51
|
-
$ org-converge spec/converge_examples/runlist_example/setup.org --runmode=sequential
|
52
|
-
I, [2014-04-02T01:18:17.336255 #65462] INFO -- : Tangling 0 files...
|
53
|
-
I, [2014-04-02T01:18:17.336376 #65462] INFO -- : Tangling succeeded!
|
54
|
-
I, [2014-04-02T01:18:17.336698 #65462] INFO -- : Tangling 2 scripts within directory: /Users/mariko/repos/org-converge/run...
|
55
|
-
I, [2014-04-02T01:18:17.340638 #65462] INFO -- : sh(65466) -- started with pid 65466
|
56
|
-
I, [2014-04-02T01:18:22.365128 #65462] INFO -- : sh(65466) -- exited with code 0
|
57
|
-
I, [2014-04-02T01:18:22.398983 #65462] INFO -- : ruby(65469) -- started with pid 65469
|
58
|
-
I, [2014-04-02T01:18:23.013195 #65462] INFO -- : ruby(65469) -- exited with code 0
|
59
|
-
I, [2014-04-02T01:18:23.013354 #65462] INFO -- : Run has completed successfully.
|
60
|
-
|
61
|
-
Running the code blocks in parallel mode is also possible:
|
62
|
-
|
63
|
-
$ org-converge spec/converge_examples/basic_run_example/setup.org --runmode=parallel
|
64
|
-
|
65
|
-
I, [2014-04-02T01:16:27.660259 #65126] INFO -- : Tangling 0 files...
|
66
|
-
I, [2014-04-02T01:16:27.660390 #65126] INFO -- : Tangling succeeded!
|
67
|
-
I, [2014-04-02T01:16:27.661640 #65126] INFO -- : Running code blocks now! (3 runnable blocks found in total)
|
68
|
-
I, [2014-04-02T01:16:27.732659 #65126] INFO -- : sh(65157) -- started with pid 65157
|
69
|
-
I, [2014-04-02T01:16:27.732819 #65126] INFO -- : ruby(65158) -- started with pid 65158
|
70
|
-
I, [2014-04-02T01:16:27.732915 #65126] INFO -- : python(65160) -- started with pid 65160
|
71
|
-
I, [2014-04-02T01:16:27.733565 #65126] INFO -- : sh(65157) -- Writing! 1
|
72
|
-
I, [2014-04-02T01:16:27.771819 #65126] INFO -- : sh(65157) -- Writing! 2
|
73
|
-
I, [2014-04-02T01:16:27.914407 #65126] INFO -- : python(65160) -- 0
|
74
|
-
I, [2014-04-02T01:16:27.914674 #65126] INFO -- : python(65160) -- 1
|
75
|
-
I, [2014-04-02T01:16:27.914887 #65126] INFO -- : python(65160) -- 2
|
76
|
-
I, [2014-04-02T01:16:27.921333 #65126] INFO -- : python(65160) -- exited with code 0
|
77
|
-
I, [2014-04-02T01:16:33.226998 #65126] INFO -- : ruby(65158) -- And now writing! 0
|
78
|
-
I, [2014-04-02T01:16:33.227257 #65126] INFO -- : ruby(65158) -- And now writing! 1
|
79
|
-
I, [2014-04-02T01:16:33.227458 #65126] INFO -- : ruby(65158) -- And now writing! 2
|
80
|
-
I, [2014-04-02T01:16:33.227673 #65126] INFO -- : ruby(65158) -- And now writing! 3
|
81
|
-
I, [2014-04-02T01:16:33.248160 #65126] INFO -- : ruby(65158) -- And now writing! 4
|
82
|
-
I, [2014-04-02T01:16:33.248378 #65126] INFO -- : ruby(65158) -- And now writing! 5
|
83
|
-
I, [2014-04-02T01:16:33.248552 #65126] INFO -- : ruby(65158) -- And now writing! 6
|
84
|
-
I, [2014-04-02T01:16:33.301512 #65126] INFO -- : ruby(65158) -- And now writing! 7
|
85
|
-
I, [2014-04-02T01:16:33.301877 #65126] INFO -- : ruby(65158) -- And now writing! 8
|
86
|
-
I, [2014-04-02T01:16:33.302186 #65126] INFO -- : ruby(65158) -- And now writing! 9
|
87
|
-
I, [2014-04-02T01:16:33.363135 #65126] INFO -- : ruby(65158) -- exited with code 0
|
88
|
-
I, [2014-04-02T01:16:36.736210 #65126] INFO -- : sh(65157) -- Writing! 3
|
89
|
-
I, [2014-04-02T01:16:41.747375 #65126] INFO -- : sh(65157) -- Writing! 4
|
90
|
-
I, [2014-04-02T01:16:47.754810 #65126] INFO -- : sh(65157) -- Writing! 5
|
91
|
-
I, [2014-04-02T01:16:50.764513 #65126] INFO -- : sh(65157) -- exited with code 0
|
92
|
-
I, [2014-04-02T01:16:50.764636 #65126] INFO -- : Run has completed successfully.
|
93
|
-
|
94
|
-
# How it works
|
95
|
-
|
96
|
-
Org Converge uses an liberally extended version of Org Babel
|
97
|
-
features in order to give support for converging the configuration
|
98
|
-
of a server.
|
99
|
-
|
100
|
-
For example, using Org Babel and macros we can easily spread config
|
101
|
-
files on a server by writing the following on a `server.org` file.
|
102
|
-
|
103
|
-
```org
|
104
|
-
#+begin_src yaml :tangle /etc/component.yml
|
105
|
-
multitenant: false
|
106
|
-
status_port: 10004
|
107
|
-
#+end_src
|
108
|
-
```
|
109
|
-
|
110
|
-
And then configure it by running it as follows, (considering we have
|
111
|
-
the correct permissions):
|
112
|
-
|
113
|
-
```sh
|
114
|
-
$ org-converge server.org
|
115
|
-
```
|
116
|
-
|
117
|
-
Next, let's say that we no only one want to set the configured templates,
|
118
|
-
but that we also want to install some packages. In that case, we
|
119
|
-
should be able to do the following:
|
120
|
-
|
121
|
-
```org
|
122
|
-
* Configuring the component
|
123
|
-
|
124
|
-
#+begin_src yaml :tangle /etc/component.yml
|
125
|
-
multitenant: false
|
126
|
-
status_port: 10004
|
127
|
-
#+end_src
|
128
|
-
|
129
|
-
* Installing the dependencies
|
130
|
-
|
131
|
-
Need the following so that ~bundle install~ can compile
|
132
|
-
the native extensions correctly.
|
133
|
-
|
134
|
-
#+begin_src sh :shebang #!/bin/bash
|
135
|
-
apt-get install build-essentials -y
|
136
|
-
#+end_src
|
137
|
-
|
138
|
-
Then the following should work:
|
139
|
-
|
140
|
-
#+begin_src sh :shebang #!/bin/bash
|
141
|
-
cd {{{project_path}}}
|
142
|
-
bundle install
|
143
|
-
#+end_src
|
144
|
-
```
|
145
|
-
|
146
|
-
As long as the repo has been already checked out in the directory,
|
147
|
-
the previous example will succeed. Note that we are also setting
|
148
|
-
the `:shebang` on the scripts so that the script is executable.
|
149
|
-
|
150
|
-
More practical examples can be found in the examples directory.
|
151
|
-
Many more will be added as long as dogfooding from this goes well.
|
152
|
-
|
153
|
-
# Contributing
|
154
|
-
|
155
|
-
The project is in very early development at this moment, but if you
|
156
|
-
feel that it is interesting enough, please create a ticket to start
|
157
|
-
the discussion.
|