org-converge 0.0.1 → 0.0.2

data/.travis.yml

@@ -0,0 +1,5 @@
+rvm:
+  - 1.9.3
+  - 2.0.0
+notifications:
+  disabled: true

data/README.md

@@ -0,0 +1,130 @@
+Org Converge
+------------
+[![Build Status](https://travis-ci.org/wallyqs/org-converge.svg?branch=master)](https://travis-ci.org/wallyqs/org-converge)
+
+# Description
+
+A tool which uses Org mode syntax to describe, configure and
+setup something, borrowing some ideas of what is possible to 
+do with tools like `chef-solo`, `puppet`, `ansible`, etc...
+
+Can also be used for tangling code blocks like Org Babel does.
+
+# Installing
+
+    $ gem install org-converge
+
+# Motivation
+
+The Org babel syntax has proven to be flexible enough to produce
+*reproducible research* papers. Then, I believe that configuring and setting up
+a server for example is something that could be also be done using
+the same syntax, given that *converging* the configuration is something
+that one ought to be able to reproduce.
+
+# Example usage
+
+	$ org-converge spec/converge_examples/basic_tangle/setup.org --showfiles 
+	
+	---------- conf.yml --------------
+	bind: 0.0.0.0
+	port: 2442
+
+	mysql:
+	 db: users
+     host: somewhere-example.local
+	 password: 111111111
+	 
+	$ org-converge spec/converge_examples/basic_tangle/setup.org --tangle
+	
+	I, [2014-03-24T00:21:08.073506 #660]  INFO -- : Tangling 1 files...
+    I, [2014-03-24T00:21:08.073668 #660]  INFO -- : BEGIN(conf.yml): Tangling 7 lines at 'conf.yml'
+    I, [2014-03-24T00:21:08.075562 #660]  INFO -- : END(conf.yml): done.
+    I, [2014-03-24T00:21:08.075638 #660]  INFO -- : Tangling succeeded!
+
+# How it works
+
+Org Converge uses an liberally extended version of Org Babel
+features in order to give support for converging the configuration
+of a server.
+
+For example, using Org Babel and macros we can easily spread config
+files on a server by writing the following on a `server.org` file.
+
+```org
+    #+MACRO: multitenancy_enabled true
+    #+MACRO: status_port true
+    
+    #+begin_src yaml :tangle /etc/component.yml
+    multitenant: false
+    status_port: 10004
+    #+end_src
+```
+
+And then configure it by running it as follows, (considering we have
+the correct permissions):
+
+```sh
+  $ org-converge server.org
+```
+
+This leverages on the syntax already provided by Org Babel, but one
+difference here is that if we run it once again without changes...
+
+```sh
+  $ org-converge server.org
+```
+
+...it would finish soon since the configuration has already converged.
+
+Next, let's say that we no only one want to set the configured templates,
+but that we also want to install some packages. In that case, we
+should be able to do the following:
+
+```org
+    #+macro: multitenancy_enabled true
+    #+macro: status_port  true
+    #+macro: project_path path/to/project
+
+    * Configuring the component
+
+    #+begin_src yaml :tangle /etc/component.yml
+    multitenant: false
+    status_port: 10004
+    #+end_src  
+
+    * Installing the dependencies
+
+    Need the following so that ~bundle install~ can compile 
+    the native extensions correctly.
+
+    #+begin_src sh
+    apt-get install build-essentials -y
+    #+end_src
+
+    Then the following should work:
+
+    #+begin_src sh
+    cd {{{project_path}}}
+    bundle install
+    #+end_src
+```
+
+As long as the repo has been already checked out in the directory,
+the previous example will succeed.
+
+```sh
+  org-converge server.org
+```
+
+If that is not the case, then org-converge will fail
+and pickup from that last step.
+
+More practical examples can be found in the examples directory, more will be added as
+long as dogfooding from this goes well.
+
+# Contributing
+
+The project is in very early development at this moment, but if you
+feel that it is interesting enough, please create a ticket to start
+the discussion.

data/Rakefile

@@ -8,3 +8,5 @@ RSpec::Core::RakeTask.new(:spec) do |spec|
   spec.pattern = FileList['spec/**/*_spec.rb']
   spec.rspec_opts = ['--format', 'documentation', '--colour']
 end
+
+task :default => :spec

data/TODO

@@ -2,21 +2,28 @@
 #+TODO:    TODO | DONE CANCELED 
 #+startup: showeverything
 
-* [0/2] 0.0.3 version
+* [/] 0.0.4 version
 
 - [ ] Macros can be loaded and applied to the configuration
 - [ ] Actually support converging and idempotency
   + Do not do an operation unless it is required
   + Abort in case there was a failure in executing the script.
+- [ ] Support SETUPFILE
+- [ ] Heuristics for determining which binary to use for running the script
+- [ ] Display how the run would look like without making changes
+  : org-converge setupfile.org --dry-run
+
+* [0/3] 0.0.3 version
+
+- [ ] Support run lists for sequential execution
 
-* [0/2] 0.0.2 version
+* [2/2] 0.0.2 version of org-converge
 
 Run the code blocks!
 
-- [ ] Code blocks are executed by default after tangling
-  Only when :shebang exists as an argument
-- [ ] Display how the run would look like
-  : org-converge setupfile.org --dry-run
+- [X] Code blocks are executed by default after tangling in temp dir
+  Only when ~:shebang~ exists as an argument
+- [X] Default parallel execution of blocks
 
 * [3/3] 0.0.1 version
 
@@ -28,20 +35,7 @@ Need some basic functionality of what Org babel offers first.
 - [X] ~--tangle~ flag
 - [X] Support a root dir for when not running relative to the directory
 
-* [2/14] Ideas
-** CANCELED We don't need to create the directories in most cases
-
-Something like this is not required because the ~:tangle~ blocks
-would create the necessary directories behind the scenes.
-
-#+begin_src org
-  ,We need to prepare some directories for the configuration:
-  
-  ,#+begin_src converge
-  ,mkdir -p etc/fluentd/config
-  ,#+end_src
-#+end_src
-
+* [1/18] Ideas
 ** CANCELED How to set the permissions from the directory from the file that is being tangled when it does not exists?
 
 By default, this would be 0644, but we also need to specify the
@@ -53,6 +47,22 @@ hmm dont't like this syntax for folders
 
 Let's keep it simple and just use a babel block that shells out to create the directories
 until I can think of something better.
+** TODO Clarify which ones of the header arguments to implement
+
+http://orgmode.org/manual/Specific-header-arguments.html#Specific-header-arguments
+
+** TODO We don't need to create the directories in most cases (:mkdirp yes)
+
+Something like this is not required because the ~:tangle~ blocks
+would create the necessary directories behind the scenes.
+
+#+begin_src org
+  ,We need to prepare some directories for the configuration:
+  
+  ,#+begin_src converge
+  ,mkdir -p etc/fluentd/config
+  ,#+end_src
+#+end_src
 
 ** TODO By default, it should use current dir for tangling
 ** TODO Converging: Only do an operation when it didn't finish
@@ -165,3 +175,25 @@ sudo service td-agent start
 #+begin_src sh
 sudo /etc/register-to-balancer
 #+end_src
+
+** TODO ~#+NAME:~ could be used in the logger for identifying the process
+** TODO Support caching?
+
+#+begin_src emacs-lisp :cache yes :exports results
+  (random)
+#+end_src
+
+#+RESULTS[db54597aed193d861d01bf92110e10f28f8f40d4]:
+: 842438499349743708
+
+** TODO Support :eval ?
+
+#+begin_src sh :eval (print "Really doing this...")
+  echo "Going ahead with operation X!"
+#+end_src
+
+* Links
+
+Super useful notes:
+http://eschulte.github.io/org-scraps
+

data/lib/org-converge.rb

@@ -2,6 +2,7 @@ require 'org-ruby'
 require 'org-converge/babel_output_buffer'
 require 'org-converge/babel'
 require 'org-converge/command'
+require 'org-converge/engine'
 require 'org-converge/version'
 
 module Orgmode
@@ -25,9 +26,15 @@ module Orgmode
 end
 
 module StringWithColors
-  def red;    colorize("\e[0m\e[31m");  end
-  def green;  colorize("\e[0m\e[32m");  end
-  def yellow; colorize("\e[0m\e[33m");  end
+  def red;            colorize("\e[0m\e[31m");  end
+  def green;          colorize("\e[0m\e[32m");  end
+  def yellow;         colorize("\e[0m\e[33m");  end
+  def blue;           colorize("\e[0m\e[34m");  end
+  def magenta;        colorize("\e[0m\e[35m"); end
+  def cyan;           colorize("\e[0m\e[36m"); end
+  def white;          colorize("\e[0m\e[37m"); end
+  def bright_black;   colorize("\e[0m\e[30m"); end
+  def bright_magenta; colorize("\e[0m\e[35m"); end
   def bold;   colorize("\e[0m\e[1m");   end
   def colorize(color_code); "#{color_code}#{self}\e[0m"; end
 end

data/lib/org-converge/babel.rb

@@ -29,7 +29,9 @@ module Orgmode
 
         logger.info "BEGIN(#{tangle_file}): Tangling #{lines.count} lines at '#{file}'"
         # TODO: should abort when the directory does not exists
+        #       Org mode blocks have :mkdirp true
         # TODO: should abort when the directory failed because of permissions
+        # TODO: should apply :tangle-mode for permissions
         if not Dir.exists?(File.dirname(file))
           logger.error "Could not tangle #{file} because directory does not exists!"
           raise TangleError
@@ -52,7 +54,26 @@ module Orgmode
         logger.info "END(#{file}): done."
       end
 
-      logger.info "Tangling succeeded!".green
+      logger.info "Tangling succeeded!"
+    end
+
+    def tangle_runnable_blocks!(run_dir='run')
+      FileUtils.mkdir_p(run_dir)
+
+      logger.info "Tangling #{ob.scripts.count} scripts within directory: #{run_dir}..."
+
+      ob.scripts.each_pair do |script_key, script|
+        file = script_key.to_s
+        if File.exists?(file)
+          logger.warn "File already exists at #{file}, it will be overwritten"
+        end
+
+        File.open(File.join(run_dir, file), 'w') do |f|
+          script[:lines].each_line do |line|
+            f.puts line
+          end
+        end
+      end
     end
 
     class TangleError < Exception; end

data/lib/org-converge/babel_output_buffer.rb

@@ -12,12 +12,13 @@ module Orgmode
 
       # ~@scripts~ are tangled in order and ran
       # : @scripts = [text, text, ...]
-      @scripts = []
+      @scripts = Hash.new {|h,k| h[k] = {:lines => '', :header => {}, :lang => ''}}
+      @scripts_counter = 0
 
       @buffer = ''
     end
 
-    def push_mode(mode, indent)
+    def push_mode(mode, indent);
       super(mode, indent)
     end
 
@@ -30,13 +31,21 @@ module Orgmode
     def insert(line)
       # We try to get the lang from #+BEGIN_SRC blocks
       if line.begin_block?
-        @block_lang = line.block_lang
-        if line.block_header_arguments[':tangle']
+        case
+        when line.block_header_arguments[':tangle']
           @current_tangle = line.block_header_arguments[':tangle']
-        else
-          @shebang = line.block_header_arguments[':shebang']
+        when line.block_header_arguments[':shebang']
           @current_tangle = nil
           @buffer = ''
+
+          # Need to keep track of the options from a block before running it
+          @scripts[@scripts_counter][:header] = { 
+            :shebang => line.block_header_arguments[':shebang']
+          }
+          @scripts[@scripts_counter][:lang] = line.block_lang
+
+        # TODO: have a way to specify which are the default binaries to be used per language
+        # when binary_detected?(@block_lang)
         end
       end
 
@@ -51,8 +60,9 @@ module Orgmode
       when (!@buffer.empty? and not (line.begin_block? or line.assigned_paragraph_type == :code))
         # Fix indentation and remove pre fix commas from Org mode before flushing
         strip_code_block!
-        @scripts << @buffer
+        @scripts[@scripts_counter][:lines] << @buffer
         @buffer = ''
+        @scripts_counter += 1
       end
 
       @output_type = line.assigned_paragraph_type || line.paragraph_type
@@ -66,6 +76,7 @@ module Orgmode
     # TODO: This should be in the parent class....
     def output_footnotes!; false; end
 
+    # TODO: This should be part of some utils package from OrgRuby
     def strip_code_block!
       if @code_block_indent and @code_block_indent > 0
         strip_regexp = Regexp.new("^" + " " * @code_block_indent)

data/lib/org-converge/command.rb

@@ -3,19 +3,21 @@ module OrgConverge
     attr_reader :dotorg
     attr_reader :logger
     attr_reader :ob
+    attr_reader :engine
 
     def initialize(options)
       @options = options
       @dotorg  = options['<org_file>']
       @logger  = Logger.new(options['--log'] || STDOUT)
       @root_dir = options['--root-dir']
-      @ob = Orgmode::Parser.new(File.read(dotorg)).babelize
+      @run_dir  = File.expand_path('run')
+      @ob    = Orgmode::Parser.new(File.read(dotorg)).babelize
+      @babel = nil
     end
 
     def execute!
       case
       when @options['--showfiles']
-
         showfiles
       when @options['--tangle']
         tangle!
@@ -25,20 +27,37 @@ module OrgConverge
 
       true
     rescue => e
+      @logger.error e
       false
     end
 
     def converge!
       tangle!
+      run_blocks!
     end
 
     def tangle!
-      begin
-        babel = Orgmode::Babel.new(ob, { :logger => logger, :root_dir => @root_dir })
-        results = babel.tangle!
-      rescue Orgmode::Babel::TangleError
-        logger.error "Cannot converge because there were errors during tangle step".red
+      results = babel.tangle!
+    rescue Orgmode::Babel::TangleError
+      logger.error "Cannot converge because there were errors during tangle step".red
+    end
+
+    # TODO: Too much foreman has made this running blocks in parallel the default behavior.
+    #       We should actually be supporting run lists instead, but liking this experiment so far.
+    def run_blocks!
+      @engine = OrgConverge::Engine.new(:logger => @logger, :babel => @babel)
+      babel.tangle_runnable_blocks!(@run_dir)
+      babel.ob.scripts.each do |key, script|
+        file = File.expand_path("#{@run_dir}/#{key}")
+        cmd = "#{script[:lang]} #{file}"
+        @engine.register script[:lang], cmd, { :cwd => @root_dir, :logger => logger }
       end
+      logger.info "Running code blocks now! (#{babel.ob.scripts.count} runnable blocks found in total)"
+      @engine.start
+    end
+
+    def babel
+      @babel ||= Orgmode::Babel.new(ob, { :logger => @logger, :root_dir => @root_dir })
     end
 
     def showfiles
@@ -49,9 +68,9 @@ module OrgConverge
         end
       end
 
-      ob.scripts.each_with_index do |script, index|
-        puts "---------- script: #{index} --------------".green
-        puts script
+      ob.scripts.each do |index, block|
+        puts "---------- script: #{index} to be run with: #{block[:header][:shebang]} --------------".green
+        puts block[:lines]
       end
     end
   end

data/lib/org-converge/engine.rb

@@ -0,0 +1,87 @@
+#
+# Re-use most of the proven tricks from Foreman for this
+# and just customize to watch the output from runnable code blocks
+#
+require 'foreman/engine'
+require 'foreman/process'
+
+module OrgConverge
+  class Engine < Foreman::Engine
+
+    attr_reader :logger
+    attr_reader :babel
+
+    def initialize(options={})
+      super(options)
+      @logger = options[:logger] || Logger.new(STDOUT)
+      @babel  = options[:babel]
+    end
+
+    # We allow other processes to exit with 0 status
+    # to continue with the runlist
+    def start
+      register_signal_handlers
+      spawn_processes
+      watch_for_output
+      sleep 0.1
+      begin
+        status = watch_for_termination
+      end while @running.count > 0
+      logger.info "Run has completed successfully.".green
+    end
+
+    # Overriden: we do not consider process formations
+    def spawn_processes
+      @processes.each do |process|
+        reader, writer = create_pipe
+        begin
+          pid = process.run(:output => writer)
+          @names[process] = "#{@names[process]}(#{pid})"
+          writer.puts "started with pid #{pid}"
+        rescue Errno::ENOENT
+          writer.puts "unknown command: #{process.command}"
+        end
+        @running[pid] = [process]
+        @readers[pid] = reader
+      end
+    end
+
+    def register(name, command, options={})
+      options[:env] ||= env
+      options[:cwd] ||= File.dirname(command.split(" ").first)
+      process = OrgConverge::CodeBlockProcess.new(command, options)
+      @names[process] = name
+      @processes << process
+    end
+
+    def output(name, data)
+      data.to_s.lines.map(&:chomp).each do |message|
+
+        # FIXME: In case the process has finished before its lines where flushed
+        ps = name.empty? ? '<defunct>' : name.split('.').first
+        output  = "#{pad_process_name(ps)}".yellow
+        output += " -- "
+        output += message
+        # FIXME: When the process has stopped already,
+        # the name of the process does not appear
+        logger.info output
+      end
+    rescue Errno::EPIPE
+      terminate_gracefully
+    end
+
+    private
+    def name_padding
+      @name_padding ||= begin
+                          name_padding  = @names.values.map { |n| n.length }.sort.last
+                          [ 9, name_padding ].max
+                        end
+    end
+
+    def pad_process_name(name)
+      name.ljust(name_padding, " ")
+    end
+  end
+
+  class CodeBlockProcess < Foreman::Process; end
+end

data/lib/org-converge/version.rb

@@ -1,3 +1,3 @@
 module OrgConverge
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

data/org-converge.gemspec

@@ -16,4 +16,5 @@ Gem::Specification.new do |gem|
   gem.version       = OrgConverge::VERSION
   gem.add_runtime_dependency('docopt', '0.5.0')
   gem.add_runtime_dependency('org-ruby', '~> 0.9.2')
+  gem.add_runtime_dependency('foreman', '~> 0.63.0')
 end

data/spec/converge_examples/basic_run_example/setup.org

@@ -0,0 +1,31 @@
+#+title: Running scripts in parallel
+
+In the following example, the following 3 scripts should be run in parallel
+with all output being flushed to the screen.
+
+- Count some numbers with bash
+
+  #+begin_src sh :shebang #!/bin/bash
+  for i in `seq 1 5`; do 
+  echo "Writing! $i"
+  echo "hello $i" >> out.log
+  sleep $(($RANDOM % 10))
+  done
+  #+end_src
+
+- Count some numbers with ruby
+
+  #+begin_src ruby :shebang #!/usr/bin/ruby
+  10.times do |n|
+    puts "And now writing! #{n}"
+    File.open("out.log", "a") {|f| f.puts "Hello again #{n}" }
+    sleep rand.round(2)
+  end
+  #+end_src
+
+- Print some numbers with python
+
+#+begin_src python :shebang #!/usr/bin/python
+for i in range(0,3):
+  print i
+#+end_src

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: org-converge
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-03-23 00:00:00.000000000 Z
+date: 2014-03-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: docopt
-  requirement: &70317759431040 !ruby/object:Gem::Requirement
+  requirement: &70243331730100 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - =
@@ -21,10 +21,10 @@ dependencies:
         version: 0.5.0
   type: :runtime
   prerelease: false
-  version_requirements: *70317759431040
+  version_requirements: *70243331730100
 - !ruby/object:Gem::Dependency
   name: org-ruby
-  requirement: &70317759430520 !ruby/object:Gem::Requirement
+  requirement: &70243331729260 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,7 +32,18 @@ dependencies:
         version: 0.9.2
   type: :runtime
   prerelease: false
-  version_requirements: *70317759430520
+  version_requirements: *70243331729260
+- !ruby/object:Gem::Dependency
+  name: foreman
+  requirement: &70243331728140 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.63.0
+  type: :runtime
+  prerelease: false
+  version_requirements: *70243331728140
 description: A light configuration management tool for Org mode
 email:
 - waldemar.quevedo@gmail.com
@@ -42,8 +53,10 @@ extensions: []
 extra_rdoc_files: []
 files:
 - .gitmodules
+- .travis.yml
 - Gemfile
 - LICENSE
+- README.md
 - README.org
 - Rakefile
 - TODO
@@ -56,8 +69,10 @@ files:
 - lib/org-converge/babel.rb
 - lib/org-converge/babel_output_buffer.rb
 - lib/org-converge/command.rb
+- lib/org-converge/engine.rb
 - lib/org-converge/version.rb
 - org-converge.gemspec
+- spec/converge_examples/basic_run_example/setup.org
 - spec/converge_examples/basic_tangle/conf.yml.expected
 - spec/converge_examples/basic_tangle/setup.org
 - spec/converge_spec.rb
@@ -88,7 +103,9 @@ specification_version: 3
 summary: Provides an 'org-converge' command which can be used for tangling and running
   Org mode code blocks
 test_files:
+- spec/converge_examples/basic_run_example/setup.org
 - spec/converge_examples/basic_tangle/conf.yml.expected
 - spec/converge_examples/basic_tangle/setup.org
 - spec/converge_spec.rb
 - spec/spec_helper.rb
+has_rdoc: