org-converge 0.0.2 → 0.0.3

data/TODO

@@ -2,8 +2,9 @@
 #+TODO:    TODO | DONE CANCELED 
 #+startup: showeverything
 
-* [/] 0.0.4 version
+* [0/6] 0.0.4 version
 
+- [ ] Support runlists notifications via =#+NAME:=
 - [ ] Macros can be loaded and applied to the configuration
 - [ ] Actually support converging and idempotency
   + Do not do an operation unless it is required
@@ -12,10 +13,13 @@
 - [ ] Heuristics for determining which binary to use for running the script
 - [ ] Display how the run would look like without making changes
   : org-converge setupfile.org --dry-run
+- [ ] Use :eval for evaling blocks (off by default)
+- [ ] Can use :dir for running a process relative to that directory
 
-* [0/3] 0.0.3 version
+* [2/2] 0.0.3 version
 
-- [ ] Support run lists for sequential execution
+- [X] Support sequential and parallel execution
+- [X] Can use :mkdirp for 0644 permissions
 
 * [2/2] 0.0.2 version of org-converge
 
@@ -35,7 +39,7 @@ Need some basic functionality of what Org babel offers first.
 - [X] ~--tangle~ flag
 - [X] Support a root dir for when not running relative to the directory
 
-* [1/18] Ideas
+* [1/19] Ideas
 ** CANCELED How to set the permissions from the directory from the file that is being tangled when it does not exists?
 
 By default, this would be 0644, but we also need to specify the
@@ -51,6 +55,7 @@ until I can think of something better.
 
 http://orgmode.org/manual/Specific-header-arguments.html#Specific-header-arguments
 
+** TODO Use sshkit for running remote processes
 ** TODO We don't need to create the directories in most cases (:mkdirp yes)
 
 Something like this is not required because the ~:tangle~ blocks

data/bin/org-converge

@@ -8,7 +8,7 @@ doc = <<OPTIONS
 org-converge: A light configuration management tool for Org mode
 
 Usage:
-  org-converge <org_file> [--tangle] [--showfiles] [--log=<logfile>] [--root-dir=<root_dir>]
+  org-converge <org_file> [--tangle] [--showfiles] [--log=<logfile>] [--root-dir=<root_dir>] [--runmode=<runmode>]
 
 Options:
 

data/examples/fluentd/setup.org

@@ -6,21 +6,26 @@
 
 The following will setup and configure ~fluentd~ on a node.
 
-* Settings
-
-These can be defined with Org mode macro directives, but thinking on
-other ways to do it as well.
-
-#+macro: fluentd_port     4224
-#+macro: fluentd_filepath here.log
-
 * Setup 							      :setup:
 
 Fluentd can be installed either as a package from treasure data or
 just as a gem. In this example, we only install as a gem:
 
-#+begin_src sh
-gem install fluentd
+# Fails because of permissions
+#+begin_src sh :shebang #!/bin/bash
+gem list | grep fluentd
+fluentd_installed=$?
+if [ $fluentd_installed -ne 0 ]; then
+  echo "Installing fluentd...";
+  gem install fluentd
+fi
+
+gem list | grep fluent-logger
+fluentd_logger_installed=$?
+if [ $fluentd_logger_installed -ne 0 ]; then
+  echo "Installing fluent-logger for Ruby...";
+  gem install fluent-logger
+fi
 #+end_src
 
 * Configuration 						     :config:
@@ -28,23 +33,49 @@ gem install fluentd
 Set Fluentd to listen on the ~fluentd_port~ port and to write logs
 to ~fluentd_filepath~.
 
-#+begin_src conf :tangle etc/fluentd.conf
+#+begin_src conf :tangle fluentd.conf
 <source>
   type forward
-  port {{{ fluentd_port }}}
+  port 4224
 </source>
 
 <match **>
   type file
-  path {{{ fluentd_filepath }}}
+  path here.log
+  flush_interval 0s
 </match>
 #+end_src
 
 # A script to daemonize the process could be written here, as well as
 # any /etc/default/* required settings...
 
+* Flusher
+
+Send some logs:
+
+#+begin_src ruby :shebang #!/usr/local/bin/ruby
+require 'fluent-logger'
+
+sleep 2 # Wait for fluentd to start at the beginning
+
+Fluent::Logger::FluentLogger.open(nil, :host => 'localhost', :port => 4224)
+
+100.times do |n|
+  Fluent::Logger.post("myapp.access", {"agent"=>"foo - #{n}"})
+  sleep 0.1
+end
+#+end_src
+
+* Tailer
+
+And yet another process which will be tailing the file that we are sending logs too:
+
+#+begin_src sh :shebang #!/bin/bash
+tail -f here.*
+#+end_src
+
 * Start which was configured here 				      :start:
 
-#+begin_src sh
-fluentd -c etc/fluentd.conf -vvv
+#+begin_src bash :shebang #!/bin/bash
+fluentd -c fluentd.conf -vvv
 #+end_src

data/lib/org-converge/babel.rb

@@ -16,25 +16,33 @@ module Orgmode
       @root_dir = options[:root_dir]
     end
 
-    # TODO: should be able to tangle relatively to a dir
     def tangle!
       logger.info "Tangling #{ob.tangle.keys.count} files..."
 
-      ob.tangle.each do |tangle_file, lines|
+      ob.tangle.each do |tangle_file, script|
         file = if @root_dir
                  File.join(@root_dir, tangle_file)
                else
                  tangle_file
                end
 
-        logger.info "BEGIN(#{tangle_file}): Tangling #{lines.count} lines at '#{file}'"
-        # TODO: should abort when the directory does not exists
-        #       Org mode blocks have :mkdirp true
+        logger.info "BEGIN(#{tangle_file}): Tangling #{script[:lines].split('\n').count} lines at '#{file}'"
         # TODO: should abort when the directory failed because of permissions
         # TODO: should apply :tangle-mode for permissions
-        if not Dir.exists?(File.dirname(file))
-          logger.error "Could not tangle #{file} because directory does not exists!"
-          raise TangleError
+        directory = File.expand_path(File.dirname(file))
+        if not Dir.exists?(directory)
+          begin
+            if script[:header][:mkdirp] == 'true'
+              p script
+              logger.info "Create dir for #{file} since it does not exists..."
+              FileUtils.mkdir_p(File.dirname(file), :mode => 0755)
+            else
+              logger.warn "Cannot tangle #{file} because directory does not exists!"
+            end
+          rescue => e
+            p e
+            raise TangleError
+          end
         end
 
         if File.exists?(file)
@@ -43,7 +51,7 @@ module Orgmode
 
         begin
           File.open(file, 'w') do |f|
-            lines.each do |line|
+            script[:lines].each_line do |line|
               f.puts line
             end
           end
@@ -64,11 +72,10 @@ module Orgmode
 
       ob.scripts.each_pair do |script_key, script|
         file = script_key.to_s
-        if File.exists?(file)
-          logger.warn "File already exists at #{file}, it will be overwritten"
-        end
+        logger.warn "File already exists at #{file}, it will be overwritten" if File.exists?(file)
 
-        File.open(File.join(run_dir, file), 'w') do |f|
+        # Files with :shebang are executable by default
+        File.open(File.join(run_dir, file), 'w', 0755) do |f|
           script[:lines].each_line do |line|
             f.puts line
           end

data/lib/org-converge/babel_output_buffer.rb

@@ -8,7 +8,7 @@ module Orgmode
 
       # ~@tangle~ files are put in the right path
       # : @tangle['/path'] = [Lines]
-      @tangle  = Hash.new {|h,k| h[k] = []}
+      @tangle  = Hash.new {|h,k| h[k] = {:lines => '', :header => {}, :lang => ''}}
 
       # ~@scripts~ are tangled in order and ran
       # : @scripts = [text, text, ...]
@@ -18,7 +18,7 @@ module Orgmode
       @buffer = ''
     end
 
-    def push_mode(mode, indent);
+    def push_mode(mode, indent)
       super(mode, indent)
     end
 
@@ -34,13 +34,19 @@ module Orgmode
         case
         when line.block_header_arguments[':tangle']
           @current_tangle = line.block_header_arguments[':tangle']
+          @tangle[@current_tangle][:header] = {
+            :shebang => line.block_header_arguments[':shebang'],
+            :mkdirp  => line.block_header_arguments[':mkdirp']
+          }
+          @tangle[@current_tangle][:lang] = line.block_lang
         when line.block_header_arguments[':shebang']
           @current_tangle = nil
           @buffer = ''
 
           # Need to keep track of the options from a block before running it
           @scripts[@scripts_counter][:header] = { 
-            :shebang => line.block_header_arguments[':shebang']
+            :shebang => line.block_header_arguments[':shebang'],
+            :mkdirp  => line.block_header_arguments[':mkdirp']
           }
           @scripts[@scripts_counter][:lang] = line.block_lang
 
@@ -52,7 +58,7 @@ module Orgmode
       case
       when (line.assigned_paragraph_type == :code and @current_tangle)
         # Need to keep track of the current tangle to buffer its lines
-        @tangle[@current_tangle] << line
+        @tangle[@current_tangle][:lines] << line.output_text << "\n"
       when (line.assigned_paragraph_type == :code)
         # When a tangle is not going on, it means that the lines would go
         # into a runnable script

data/lib/org-converge/command.rb

@@ -10,7 +10,11 @@ module OrgConverge
       @dotorg  = options['<org_file>']
       @logger  = Logger.new(options['--log'] || STDOUT)
       @root_dir = options['--root-dir']
-      @run_dir  = File.expand_path('run')
+      @run_dir  = if @root_dir
+                    File.expand_path(File.join(@root_dir, 'run'))
+                  else
+                    File.expand_path('run')
+                  end
       @ob    = Orgmode::Parser.new(File.read(dotorg)).babelize
       @babel = nil
     end
@@ -33,7 +37,16 @@ module OrgConverge
 
     def converge!
       tangle!
-      run_blocks!
+      case @options['--runmode']
+      when 'parallel'
+        run_blocks_in_parallel!
+      when 'sequential'
+        run_blocks_sequentially!
+      when 'runlist'
+        # TODO
+      else
+        run_blocks_in_parallel!
+      end
     end
 
     def tangle!
@@ -42,9 +55,29 @@ module OrgConverge
       logger.error "Cannot converge because there were errors during tangle step".red
     end
 
+    def run_blocks_sequentially!
+      @engine = OrgConverge::Engine.new(:logger => @logger, :babel => @babel)
+      babel.tangle_runnable_blocks!(@run_dir)
+
+      runlist_stack = []
+      babel.ob.scripts.each do |key, script|
+        runlist_stack << [key, script]
+      end
+
+      while not runlist_stack.empty?
+        key, script = runlist_stack.shift
+        with_running_engine do |engine|
+          file = File.expand_path("#{@run_dir}/#{key}")
+          cmd = "#{script[:lang]} #{file}"
+          engine.register script[:lang], cmd, { :cwd => @root_dir, :logger => logger }
+        end
+      end
+      logger.info "Run has completed successfully.".green
+    end
+
     # TODO: Too much foreman has made this running blocks in parallel the default behavior.
     #       We should actually be supporting run lists instead, but liking this experiment so far.
-    def run_blocks!
+    def run_blocks_in_parallel!
       @engine = OrgConverge::Engine.new(:logger => @logger, :babel => @babel)
       babel.tangle_runnable_blocks!(@run_dir)
       babel.ob.scripts.each do |key, script|
@@ -54,6 +87,13 @@ module OrgConverge
       end
       logger.info "Running code blocks now! (#{babel.ob.scripts.count} runnable blocks found in total)"
       @engine.start
+      logger.info "Run has completed successfully.".green
+    end
+
+    def with_running_engine
+      engine = OrgConverge::Engine.new(:logger => @logger, :babel => @babel)
+      yield engine
+      engine.start
     end
 
     def babel
@@ -61,11 +101,9 @@ module OrgConverge
     end
 
     def showfiles
-      ob.tangle.each do |file, lines|
+      ob.tangle.each do |file, block|
         puts "---------- #{file} --------------".green
-        lines.each do |line|
-          puts line
-        end
+        puts block[:lines]
       end
 
       ob.scripts.each do |index, block|

data/lib/org-converge/engine.rb

@@ -25,9 +25,8 @@ module OrgConverge
       watch_for_output
       sleep 0.1
       begin
-        status = watch_for_termination
+        status = watch_for_termination        
       end while @running.count > 0
-      logger.info "Run has completed successfully.".green
     end
 
     # Overriden: we do not consider process formations

data/lib/org-converge/version.rb

@@ -1,3 +1,3 @@
 module OrgConverge
-  VERSION = "0.0.2"
+  VERSION = "0.0.3"
 end

data/spec/converge_examples/basic_run_example/setup.org

@@ -6,6 +6,7 @@ with all output being flushed to the screen.
 - Count some numbers with bash
 
   #+begin_src sh :shebang #!/bin/bash
+  echo '' > out.log
   for i in `seq 1 5`; do 
   echo "Writing! $i"
   echo "hello $i" >> out.log
@@ -16,6 +17,7 @@ with all output being flushed to the screen.
 - Count some numbers with ruby
 
   #+begin_src ruby :shebang #!/usr/bin/ruby
+  sleep 0.2
   10.times do |n|
     puts "And now writing! #{n}"
     File.open("out.log", "a") {|f| f.puts "Hello again #{n}" }

data/spec/converge_examples/basic_tangle/setup.org

@@ -16,3 +16,10 @@ mysql:
   host: somewhere-example.local
   password: 111111111
 #+end_src
+
+When directory does not exists, it should create
+when =:mkdirp= is specified.
+
+#+begin_src yaml :tangle config/hello.yml :mkdirp true
+config: 'hello'
+#+end_src

data/spec/converge_examples/runlist_example/setup.org

@@ -0,0 +1,15 @@
+# -*- mode: org; mode: auto-fill; -*-
+#+TITLE: Runlist example
+
+In this example, we will have scripts being chained one after another.
+
+#+begin_src sh :shebang #!/bin/bash
+sleep 5
+echo "first" > out.log
+#+end_src
+
+The script below will only be called once the one above is run.
+
+#+begin_src ruby :shebang #!/usr/local/bin/ruby
+File.open("out.log", 'a') {|f| f.puts "second" }
+#+end_src

data/spec/converge_spec.rb

@@ -1,28 +1,58 @@
 require 'spec_helper'
 
 describe OrgConverge::Command do
-  context "when converging 'basic_tangle'" do
-    example_dir = File.join(EXAMPLES_DIR, 'basic_tangle')
 
-    it "should tangle a 'conf.yml' file" do
-      setup_file = File.join(example_dir, 'setup.org')
-      o = OrgConverge::Command.new({ 
-                                     '<org_file>' => setup_file,
-                                     '--root-dir' => example_dir
-                                   })
-      success = o.execute!
-      success.should == true
+  it "should converge 'basic_tangle'" do
+    example_dir = File.join(EXAMPLES_DIR, 'basic_tangle')
 
-      expected_contents = File.read(File.join(example_dir, 'conf.yml.expected'))
+    setup_file = File.join(example_dir, 'setup.org')
+    o = OrgConverge::Command.new({ 
+                                   '<org_file>' => setup_file,
+                                   '--root-dir' => example_dir
+                                 })
+    success = o.execute!
+    success.should == true
 
-      resulting_file = File.join(example_dir, 'conf.yml')
-      File.exists?(resulting_file).should == true
+    expected_contents = File.read(File.join(example_dir, 'conf.yml.expected'))
+    resulting_file = File.join(example_dir, 'conf.yml')
+    File.exists?(resulting_file).should == true
 
-      result = File.read(resulting_file)
+    result = File.read(resulting_file)
+    result.should == expected_contents
 
-      result.should == expected_contents
-    end
+    resulting_file = File.join(example_dir, 'config/hello.yml')
+    File.exists?(resulting_file).should == true
   end
-end
 
+  it "should converge 'basic_run_example'" do
+    example_dir = File.join(EXAMPLES_DIR, 'basic_run_example')
+    setup_file = File.join(example_dir, 'setup.org')
+    o = OrgConverge::Command.new({ 
+                                   '<org_file>' => setup_file,
+                                   '--root-dir' => example_dir
+                                 })
+    success = o.execute!
+    success.should == true
+
+    expected_contents = File.read(File.join(example_dir, 'out.log'))
+    expected_contents.lines.count.should == 16
+  end
 
+  it "should converge 'runlist_example' sequentially" do
+    example_dir = File.join(EXAMPLES_DIR, 'runlist_example')
+    setup_file = File.join(example_dir, 'setup.org')
+
+    o = OrgConverge::Command.new({ 
+                                   '<org_file>' => setup_file,
+                                   '--root-dir' => example_dir,
+                                   '--runmode'  => 'sequential'
+                                 })
+    success = o.execute!
+    success.should == true
+
+    File.executable?(File.join(example_dir, 'run/0')).should == true
+    File.executable?(File.join(example_dir, 'run/1')).should == true
+    expected_contents = "first\nsecond\n"
+    File.read(File.join(example_dir, 'out.log')).should == expected_contents
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: org-converge
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-03-30 00:00:00.000000000 Z
+date: 2014-04-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: docopt
-  requirement: &70243331730100 !ruby/object:Gem::Requirement
+  requirement: &70113563924580 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - =
@@ -21,10 +21,10 @@ dependencies:
         version: 0.5.0
   type: :runtime
   prerelease: false
-  version_requirements: *70243331730100
+  version_requirements: *70113563924580
 - !ruby/object:Gem::Dependency
   name: org-ruby
-  requirement: &70243331729260 !ruby/object:Gem::Requirement
+  requirement: &70113563923840 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,10 +32,10 @@ dependencies:
         version: 0.9.2
   type: :runtime
   prerelease: false
-  version_requirements: *70243331729260
+  version_requirements: *70113563923840
 - !ruby/object:Gem::Dependency
   name: foreman
-  requirement: &70243331728140 !ruby/object:Gem::Requirement
+  requirement: &70113563921200 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,7 +43,7 @@ dependencies:
         version: 0.63.0
   type: :runtime
   prerelease: false
-  version_requirements: *70243331728140
+  version_requirements: *70113563921200
 description: A light configuration management tool for Org mode
 email:
 - waldemar.quevedo@gmail.com
@@ -75,6 +75,7 @@ files:
 - spec/converge_examples/basic_run_example/setup.org
 - spec/converge_examples/basic_tangle/conf.yml.expected
 - spec/converge_examples/basic_tangle/setup.org
+- spec/converge_examples/runlist_example/setup.org
 - spec/converge_spec.rb
 - spec/spec_helper.rb
 homepage: https://github.com/wallyqs/org-converge
@@ -106,6 +107,7 @@ test_files:
 - spec/converge_examples/basic_run_example/setup.org
 - spec/converge_examples/basic_tangle/conf.yml.expected
 - spec/converge_examples/basic_tangle/setup.org
+- spec/converge_examples/runlist_example/setup.org
 - spec/converge_spec.rb
 - spec/spec_helper.rb
 has_rdoc: