dust-deploy 0.2.3 → 0.3.0

data/README.md

@@ -1,5 +1,4 @@
-dust - a ssh/facter only server deployment tool
-=============
+# dust - a ssh only server deployment tool
 
 dust is a deployment tool, kinda like sprinkle. but cooler (at least for me).
 it's for those, who would like to maintain their servers with a tool like puppet or chef, but are scared by the thought of having configuration files, credentials and other stuff centrally on a server reachable via the internet.
@@ -7,21 +6,19 @@ it's for those, who would like to maintain their servers with a tool like puppet
 although the tool is not as versatile and elite as puppet, it's still cool for most use cases, and easily extendable.
 
 
-installing
-------------
+## installing
 
 installation is quite simple. just 
 
     # gem install dust-deploy
 
 
-using
-------------
+## quickstart
 
 let's start by creating a new directory skeleton
 
     $ dust new mynetwork
-      - spawning new dust directory skeleton into 'mynetwork.dust' [ ok ]
+      - spawning new dust directory skeleton into 'mynetwork.dust' [ ok ]
 
 this will create a directory called mynetwork.dust, the nodes, templates and recipes subdirectories and will copy over example templates and node configurations. hop into your new dust directory and see what's going on:
 
@@ -53,208 +50,20 @@ you can then save the file, and tell dust to get to work:
 
     $ dust deploy
 
-    [ yourhost.example.com ]
+    [ yourhost.example.com ]
 
     |packages|
-     - checking if vim is installed [ ok ]
-     - checking if git-core is installed [ failed ]
-       - installing git-core [ ok ]
-     - checking if rsync is installed [ ok ]
+     - checking if vim is installed [ ok ]
+     - checking if git-core is installed [ failed ]
+       - installing git-core [ ok ]
+     - checking if rsync is installed [ ok ]
 
 you should see dust connecting to the node, checking if the requested packages are installed, and if not, install them.
 dust works with aptitude, yum and emerge systems at the moment (testet with ubuntu, debian, gentoo, scientificlinux, centos).
 feel free to contribute to dust, so that your system is also supported. contribution is easy! just send me a github pull request. You can find the repository here: https://github.com/kechagia/dust-deploy
 
 
-inheritance
-------------
-
-because sometimes you will have similar configuration files for multiple systems, you can create templates.
-i usually start filenames of templates with an underscore, but that's not a must.
-
-    $ vi nodes/_default.yaml
-
-this template defines some general settings, usually used by most hosts
-
-    domain: example.com
-    port: 22
-    user: root
-
-
-and another one:
-
-    $ vi nodes/_debian.yaml
-
-in this template, i put in some debian specific settings
-
-    # you can add custom fields like "group"
-    # and filter on which hosts to deploy later
-    group: debian
-    
-    recipes:
-      locale: en_US.UTF-8
-      debsecan: default
-      repositories:
-        default:
-          url: "http://ftp.de.debian.org/debian/"
-          components: "main contrib non-free"
-    
-
-you can then inherit these templates in your yourhost.yaml:
-
-    hostname: yourhost
-    inherits: [ _default, _debian ]
-
-    recipes:
-      packages: [ 'vim', 'git-core', 'rsync' ]
-
-
-running dust now, will use the inherited settings as well.
-you can also overwrite settings in the template with the ones in yourhost.yaml
-
-**NOTE:** hashes will be deep merged with inherited hashes, other types will be overwritten!
-
-    $ dust deploy
-
-    [ yourhost ]
-
-    |repositories|
-     - determining whether node uses apt [ ok ]
-     - deleting old repositories [ ok ]
-
-     - deploying default repository [ ok ]
-
-    |packages|
-     - checking if vim is installed [ ok ]
-     - checking if git-core is installed [ ok ]
-     - checking if rsync is installed [ ok ]
-
-    |locale|
-     - setting locale to 'en_US.UTF-8' [ ok ]
-
-    |debsecan|
-     - checking if debsecan is installed [ ok ]
-     - configuring debsecan [ ok ]
-
-
-
-filters and proxy
-------------
-
-because that's not awesome enough, you can also filter your hosts using the --filter flag
-
-     $ dust deploy --filter hostname:myhost-1,otherhost
-
-     $ dust deploy --filter group:debian
-
-
-and even more, it supports socks proxys, so you can maintain your whole infrastructure without setting up a vpn from the outside via ssh
-
-     $ ssh user@gateway.yourcompany.net -D 1080
-
-     $ dust deploy --proxy localhost:1080
-
-
-
-using recipes (and their templates)
-------------
-
-dust comes with a set of predifined, (almost) ready to use recipes managing a lot of stuff for you, including the following:
-
--   ssh authorized keys
--   email aliases file
--   /etc/hosts
--   /etc/motd
--   /etc/resolv.conf
--   install basic system tools and pushing .configuration files for root
--   iptables firewall
--   debian/ubuntu debsecan security notifications
--   debian/ubuntu repositories
--   duplicity backups
--   mysql server configuration
--   postgresql server configuration (including corosync scripts)
--   nginx configuration
--   zabbix agent
--   debian/ubuntu unattended upgrades
--   newrelic system monitoring daemon
-
-
-writing your own recipes
-------------
-
-because the above recipes will probably not in all cases fulfill your needs, it's pretty easy to write your own recipes. You can either file them in using a git pull request (if you think it's a generic one which others might use as well), or place them locally in the "recipes" folder in your mynetwork.dust directory.
-
-dust comes with a set of predefined functions to perform system tasks, which you can (and should!) use.
-
-### the server.rb methods you can (and should!) use
-
-almost all functions understand the quiet=true and indend=integer arguments
-
-#### exec command
-#### write target, text, quiet=false, indent=1
-#### append target, text, quiet=false, indent=1
-#### scp source, destination, quiet=false, indent=1
-#### symlink source, destination, quiet=false, indent=1
-#### chmod mode, file, quiet=false, indent=1
-#### chown user, file, quiet=false, indent=1
-#### rm file, quiet=false, indent=1
-#### mkdir dir, quiet=false, indent=1
-#### restorecon path, quiet=false, indent=1
-#### get_system_users quiet=false
-#### package_installed? packages, quiet=false, indent=1
-#### install_package package, quiet=false, indent=1, env=""
-#### remove_package package, quiet=false, indent=1
-#### update_repos quiet=false, indent=1
-#### system_update quiet=false, indent=1
-#### uses_apt? quiet=false, indent=1
-#### uses_rpm? quiet=false, indent=1
-#### uses_emerge? quiet=false, indent=1
-#### is_executable? file, quiet=false, indent=1
-#### file_exists? file, quiet=false, indent=1
-#### dir_exists? dir, quiet=false, indent=1
-#### autostart_service service, quiet=false, indent=1
-#### restart_service service, quiet=false, indent=1
-#### reload_service service, quiet=false, indent=1
-#### user_exists? user, quiet=false, indent=1
-#### create_user user, home=nil, shell=nil, quiet=false, indent=1
-
-#### collect_facts quiet=false, indent=1
-#### is_os? os_list, quiet=false, indent=1
-#### is_debian? quiet=false, indent=1
-#### is_ubuntu? quiet=false, indent=1
-#### is_gentoo? quiet=false, indent=1
-#### is_centos? quiet=false, indent=1
-#### is_scientific? quiet=false, indent=1
-#### is_fedora? quiet=false, indent=1
-
-
-### example recipes
-
-The best is to have a look at dusts build-in recipes: https://github.com/kechagia/dust-deploy/tree/master/lib/dust/recipes
-
-this is the basic skeletton of a recipe file, placed in recipes/your_task.rb
-
-    class YourTask < Thor
-      desc 'your_task:deploy', 'example task: displays a message and does basically nothing'
-      def deploy node, ingredients, options
-
-        ::Dust.print_msg 'this is a test example. welcome! output of uname -a below:'
-        puts node.exec('uname -a')[:stdout]
-
-        node.uses_apt?
-
-        node.restart_service 'your-service' if options.restart?
-      end
-
-      desc 'your_task:status', 'example status: displays the status of this recipe (optional)'
-      def status, node, ingredients, options
-        ::Dust.print_msg "displaying status of this example recipe!"
-      end
-    end
-
-
-contributing
-------------
-
-you have a cool contribution or bugfix? yippie! just file in a pull-request!
+## documentation
 
+for further documentation (this README only covers the very basics), head over to the github wiki:
+https://github.com/kechagia/dust-deploy/wiki

data/bin/dust

@@ -12,35 +12,77 @@ module  Dust
 
     default_task :list
 
-    desc 'deploy [server.yaml] [--filter key=value,value2] [--recipes recipe1 recipe2] [--proxy host:port]',
+    desc 'deploy [--yaml server.yaml] [--filter key=value,value2] [--recipes recipe1 recipe2] [--proxy host:port]',
          'deploy all recipes to the node(s) specified in server.yaml or to all nodes defined in ./nodes/'
 
-    method_options :filter => :hash, :recipes => :array, :proxy => :string,
+    method_options :yaml => :string, :filter => :hash, :recipes => :array, :proxy => :string,
                    :restart => :boolean, :reload => :boolean
 
-    def deploy yaml=''
+    def deploy
       return unless check_dust_dir
       initialize_thorfiles
-      Dust.print_failed 'no servers match this filter' if load_servers(yaml).empty?
+      Dust.print_failed 'no servers match this filter' if load_servers.empty?
 
       run_recipes 'deploy'
     end
 
 
-    desc 'status [server.yaml] [--filter key=value,value2] [--recipes recipe1 recipe2] [--proxy host:port]',
+    desc 'status [--yaml server.yaml] [--filter key=value,value2] [--recipes recipe1 recipe2] [--proxy host:port]',
          'display status of recipes specified by filter'
 
-    method_options :filter => :hash, :recipes => :array, :proxy => :string
+    method_options :yaml => :string, :filter => :hash, :recipes => :array, :proxy => :string
 
-    def status yaml=''
+    def status
       return unless check_dust_dir
       initialize_thorfiles
-      Dust.print_failed 'no servers match this filter' if load_servers(yaml).empty?
+      Dust.print_failed 'no servers match this filter' if load_servers.empty?
 
       run_recipes 'status'
     end
 
 
+    desc 'system_update [--yaml server.yaml] [--filter key=value,vale2] [--proxy host:port]',
+         'perform a full system upgrade (using aptitude, emerge, yum)'
+
+    method_options :yaml => :string, :filter => :hash, :proxy => :string
+
+    def system_update
+      return unless check_dust_dir
+      initialize_thorfiles
+      Dust.print_failed 'no servers match this filter' if load_servers.empty?
+
+      @nodes.each do |node|
+        # connect to server
+        server = Server.new node
+        next unless server.connect
+        server.system_update
+        server.disconnect
+      end
+    end
+
+
+    desc 'exec <command> [--yaml server.yaml] [--filter key=value,vale2] [--proxy host:port]',
+         'run a command on the server'
+
+    method_options :yaml => :string, :filter => :hash, :proxy => :string
+
+    def exec cmd, yaml=''
+      return unless check_dust_dir
+      initialize_thorfiles
+      Dust.print_failed 'no servers match this filter' if load_servers.empty?
+
+      @nodes.each do |node|
+        # connect to server
+        server = Server.new node
+        next unless server.connect
+        ret = server.exec cmd
+        puts "#{Dust.grey}#{ret[:stdout]}#{Dust.none}"
+        puts "#{Dust.red}#{ret[:stderr]}#{Dust.none}"
+        server.disconnect
+      end
+    end
+
+
     # creates directory skeleton for a dust setup
     desc 'new <name>', 'creates a dust directory skeleton for your network'
     def new name
@@ -122,17 +164,20 @@ module  Dust
     end
 
     # loads servers
-    def load_servers yaml=''
+    def load_servers
       @nodes = []
 
       # if the argument is empty, load all yaml files in the ./nodes/ directory
       # if the argument is a directory, load yaml files in this directory
       # if the argument is a file, load the file.
-      if yaml.empty?
-        yaml_files = Dir['./nodes/**/*.yaml']
+      if options[:yaml]
+        if File.directory? options[:yaml]
+          yaml_files = Dir["#{options[:yaml]}/**/*.yaml"]
+        elsif File.exists? options[:yaml]
+          yaml_files = options[:yaml]
+        end
       else
-        yaml_files = Dir["#{yaml}/**/*.yaml"] if File.directory? yaml
-        yaml_files = yaml if File.exists? yaml
+        yaml_files = Dir['./nodes/**/*.yaml']
       end
 
       unless yaml_files

data/changelog.md

@@ -1,6 +1,19 @@
 Changelog
 =============
 
+0.3.0
+------------
+
+refactoring was done, you may have to upgrade your recipes.
+-  options for all node.* functions having an options={:quiet => false, :indent => 1} paremeter now, instead of plain arguments
+-  adds 2 new dust options
+   -  dust exec 'command'  # executes a command on all servers
+   -  dust system_update   # runs a full system upgrade on all servers
+-  small adjustments and improvements
+
+if you encounter bugs, please report.
+
+
 0.2.3
 ------------
 

data/lib/dust/examples/templates/postgres/pacemaker.sh.erb

@@ -12,7 +12,7 @@ RECOVERY=$PG_DATA/recovery.conf
 RECOVERY_DONE=$PG_DATA/recovery.done
 
 # path to postgresql init script
-% if node.is_gentoo? true
+% if node.is_gentoo?
 PG_INIT=/etc/init.d/postgresql-<%= config['version'] %>
 % else
 PG_INIT=/etc/init.d/postgresql

data/lib/dust/examples/templates/zabbix_agent/zabbix_agentd.conf.erb

@@ -59,22 +59,22 @@ DebugLevel=3
 
 # Name of PID file
 
-% if node.uses_apt? true
+% if node.uses_apt?
 PidFile=/var/run/zabbix-agent/zabbix_agentd.pid
-% elsif node.uses_emerge? true
+% elsif node.uses_emerge?
 PidFile=/var/run/zabbix/zabbix_agentd.pid
-% elsif node.uses_rpm? true
+% elsif node.uses_rpm?
 PidFile=/var/run/zabbix/zabbix_agentd.pid
 % end
 
 # Name of log file.
 # If not set, syslog will be used
 
-% if node.uses_apt? true
+% if node.uses_apt?
 LogFile=/var/log/zabbix-agent/zabbix_agentd.log
-% elsif node.uses_emerge? true
+% elsif node.uses_emerge?
 LogFile=/var/log/zabbix/zabbix_agentd.log
-% elsif node.uses_emerge? true
+% elsif node.uses_emerge?
 LogFile=/var/log/zabbix/zabbix_agentd.log
 % end
 
@@ -91,17 +91,17 @@ Timeout=30
 # Note that shell command must not return empty string or EOL only
 
 # system updates
-% if node.uses_apt? true
+% if node.uses_apt?
 UserParameter=debian.updates,aptitude search '~U' |wc -l
 UserParameter=debian.security,debsecan --suite squeeze --only-fixed --format packages |wc -l
 
-% elsif node.uses_emerge? true
+% elsif node.uses_emerge?
 UserParameter=gentoo.security,glsa-check -t all 2>/dev/null | wc -l
 UserParameter=gentoo.updates,emerge -uNDp @world | grep ebuild|wc -l
 UserParameter=gentoo.portage,emerge --info| grep 'Timestamp of tree' | sed -e s/'Timestamp of tree':// -e 's/\n//' | xargs -I {} date --date={} +%s |xargs -I {} expr $(date +%s) - {}
 UserParameter=gentoo.config,find /etc/ -name '._cfg*' 2>/dev/null|wc -l
 
-% elsif node.uses_rpm? true
+% elsif node.uses_rpm?
 UserParameter=centos.updates,yum check-update -q |wc -l
 % end
 

data/lib/dust/print_status.rb

@@ -12,45 +12,55 @@ module Dust
 
   $stdout.sync = true # autoflush
 
-  def self.print_result ret, quiet=false
+  def self.print_result ret, options={:quiet => false, :indent => 1}
     if ret == 0 or ret == true
-      print_ok unless quiet
+      print_ok '', options
       return true
     else
-      print_failed unless quiet
+      print_failed '', options
       return false
     end
   end
 
-  def self.print_ok string="", level=0
-    print_msg "#{string} #{blue}[ ok ]#{none}\n", level
+  def self.print_ok string='', options={:quiet => false, :indent => 1}
+    options[:indent] = 0 if string.empty?
+    print_msg "#{string} #{blue}[ ok ]#{none}\n", options
+    true
   end
 
-  def self.print_failed string="", level=0
-    print_msg "#{string} #{red}[ failed ]#{none}\n", level
+  def self.print_failed string='', options={:quiet => false, :indent => 1}
+    options[:indent] = 0 if string.empty?
+    print_msg "#{string} #{red}[ failed ]#{none}\n", options
+    false
   end
 
-  def self.print_warning string="", level=0
-    print_msg "#{string} #{yellow}[ warning ]#{none}\n", level
+  def self.print_warning string='', options={:quiet => false, :indent => 1}
+    options[:indent] = 0 if string.empty?
+    print_msg "#{string} #{yellow}[ warning ]#{none}\n", options
   end
 
-  def self.print_hostname hostname, level=0
-    print_msg "\n[ #{blue}#{hostname}#{none} ]\n\n", level
+  def self.print_hostname hostname, options={:quiet => false, :indent => 0}
+    print_msg "\n[ #{blue}#{hostname}#{none} ]\n\n", options
   end
 
-  def self.print_recipe recipe, level=0
-    print_msg "#{green}|#{recipe}|#{none}\n", level
+  def self.print_recipe recipe, options={:quiet => false, :indent => 1}
+    print_msg "#{green}|#{recipe}|#{none}\n", options
   end
 
-  # indent according to level
-  # level 0
-  #  - level 1
-  #    - level 2
-  def self.print_msg string, level=1
-    if level == 0
+  # indent according to options[:indent]
+  # indent 0
+  #  - indent 1
+  #    - indent 2
+  def self.print_msg string, options={:quiet => false, :indent => 1}
+    # just return if in quiet mode
+    return if options[:quiet]
+
+    options[:indent] ||= 1
+
+    if options[:indent] == 0
       print string
     else
-      print ' ' + '  ' * (level - 1) + '- ' + string
+      print ' ' + '  ' * (options[:indent] - 1) + '- ' + string
     end
   end