automateit 0.70923

data/TESTING.txt

@@ -0,0 +1,57 @@
+== AutomateIt's self-test
+
+AutomateIt ensures that it works correctly, and stays working, by using a rigorous self-test suite. This suite is written using the RSpec tool and its files are stored in the software installation's +spec+ directory. You must install the +rake+ and +rspec+ Gems to run the tests.
+
+=== Types of tests
+
+<b>Unit tests</b> are quick and safe because they don't write to the disk or alter your system.
+
+For example, <tt>spec/unit/template_manager_erb_spec.rb</tt> exercises AutomateIt::TemplateManager::ERB by rendering templates to memory. Because it doesn't write to the disk, you can safely run it without worrying about it damaging your system.
+
+<b>Integration tests</b> are slow and potentially dangerous because they write to disk and alter your system. Great care is taken to make sure these don't cause damage, but because they modify the system, there is a potential risk. These tests are designed to fail before doing anything if they detect that they might cause damage. The tests will clean up after themselves when done to remove any changes made to the system. To put "dangerous" in context, these tests are run before every release, so the likelihood of actual damage is very low.
+
+For example, the <tt>spec/integration/account_manager_spec.rb</tt> exercises the AutomateIt::AccountManager by adding and deleting users and groups. To make sure it doesn't cause harm, it uses dummy users with names like +automateit_testuser+. Before it starts the test, it will check for these users and fail the entire test if they're present. This ensures that it won't destroy an existing account in the unlikely event that you have a user with such a name.
+
+=== Understanding warnings
+
+When the tests can't check something, they'll print warning messages. These warnings are normal and expected. They are *not* errors and do not imply that something is broken.
+
+For example, if you run the integration test suite as a non-root user, it will warn you that it can't test some commands because you don't have the necessary privileges:
+
+ NOTE: Must be root to check 'chown' in ./spec/integration/shell_manager_spec.rb
+
+You will also get warnings about drivers that are not available on your platform. For example, Ubuntu systems don't include the YUM package manager, so there's no way to test it on an Ubuntu system. So if you run the integration tests on an Ubuntu system, you'll get a warning like:
+
+ NOTE: Can't check AutomateIt::PackageManager::YUM on this platform, ./spec/integration/package_manager_spec.rb
+
+=== Running an individual test
+
+Run an individual test, like <tt>spec/unit/template_manager_erb_spec.rb</tt>, by executing the Unix shell command:
+
+ spec spec/unit/template_manager_erb_spec.rb
+
+=== Running test suites
+
+To run the +unit+ test suite, execute the following command from the Unix shell:
+
+ rake spec
+
+To run all the test suites, including +unit+ and +integration+, execute the following command from the Unix shell:
+
+ rake spec:all
+
+The integration test can take a few minutes and will pause for long periods of time while appearing to do nothing. This is normal, expected and there's nothing that can be done to "fix" this. For example, when testing the PackageManager::Gem driver, the test must wait for the +gem+ program to download fresh package indexes and this can take a long time.
+
+=== Code coverage reports
+
+You can generate a code coverage report for the +unit+ test suite by running:
+
+ rake rcov
+
+Or a report for all the suites, including +unit+ and +integration+, by running:
+
+ rake rcov:all
+
+These tasks will create a +coverage+ directory with the report files.
+
+Note that because no single platform can run all the code, you'll never be able to get 100% coverage.

data/TODO.txt

@@ -0,0 +1,26 @@
+KEY: Important? Urgent? Easy? 1=yes, 0=no
+
+#===[ App ]=============================================================
+
+101 Download -- Add OpenURI wrapper from gTag
+100 Shell -- Expand glob patterns, e.g. chown_R(500, 500, "*")
+
+101 CLI -- Add tags via command-line
+100 Edit -- Display summary of edits, return with :details as [rv, list]
+100 Shell -- Incorporate which.cmd so #which can work on Windows
+100 Interpreter#invoke and HelpfulERB -- Extract error context code into class
+
+000 Shell#chperm -- With symbolic mode, wrap `chmod -v` as temporary workaround?
+000 Changes -- Write parser and new format so I can use "#" symbol
+000 FreeBSD -- Write driver for PackageManager::Ports
+000 Darwin -- Write drivers, e.g. PackageManager for Fink, MacPorts, Pkgsrc
+
+??? Shell -- Return false or list, never single item?
+??? Shell -- Extract FileManager? change docs
+??? Driver -- How to determine if a manager or driver method is available? Manager#available?, Manager#available and Driver#suitable? only say if it should be a default.
+
+#===[ Web ]=============================================================
+100 Index: Call for pilot users, driver authors
+000 CSS: Highlight active menu by setting a class, e.g., current_about.
+000 Kattare is redirecting to www, why?
+000 Add error page

data/TUTORIAL.txt

@@ -0,0 +1,390 @@
+== A hands-on tutorial for learning AutomateIt
+
+<em>AutomateIt is an open-source tool for automating the setup and maintenance of servers.</em>
+
+This hands-on guide will teach you to use AutomateIt and explain where to find more detailed instructions.
+
+It's recommended that you see the Screenshots[http://automateit.org/screenshots] of AutomateIt in action to get a quick idea of what AutomateIt is all about.
+
+AutomateIt is feature-complete, exceeds the capabilities of similar products, and ensures its quality with a self-test suite. However, this is a young product and users are expected to be technically proficient, willing to accept rough spots, to work through problems and upgrade frequently.
+
+Please sign up for RSS change notifications[http://automateit.org/changes]  so you know when upgrades are available.
+
+=== Ruby
+
+AutomateIt is written using the Ruby programming language. If you haven't used Ruby, you'll find it easy to learn and a pleasure to use. If you already know the basics of Perl, Python, PHP or Java, you'll be able to pick up Ruby almost instantly. Although AutomateIt provides much of the structure and commands needed, you still need to know the basic Ruby syntax to get by.
+
+Some Ruby resources:
+* Ruby's official documentation page: http://www.ruby-lang.org/en/documentation/
+* The online "Ruby user's guide" is a gentle introduction to the Ruby language: http://www.ruby-doc.org/docs/UsersGuide/rg/
+* The online "Ruby syntax" is a condensed, one-page reference of the language's syntax: http://www.ruby-doc.org/docs/ruby-doc-bundle/Manual/man-1.4/syntax.html
+* The online "Programming Ruby" site provides a fairly detailed walk-through of the language: http://www.ruby-doc.org/docs/ProgrammingRuby/
+* The "Practical Ruby for System Administration" book is a gentle introduction for sysadmins, which may be much easier for them to understand than a book aimed at software engineers, and it covers many system administration automation topics that'll be of use for Automateit recipes: http://www.apress.com/book/view/1590598210
+* The "Programming Ruby" book provides a much more complete language reference than any online reference: http://www.pragmaticprogrammer.com/titles/ruby/index.html
+
+=== Typographical conventions
+
+AutomateIt's technical documentation uses the following typographical conventions:
+
+* +automateit+ -- A file, command or variable.
+* <tt>:verbosity</tt> -- A symbol, usually used in an options hash.
+* AutomateIt::ShellManager -- A class, with a link to its documentation. You can also find a link to this class's documentation in the "Classes" pane on the left.
+* AutomateIt::ShellManager#sh -- A method, with a link to its documentation. You can also find a link to this method's documentation in the "Methods" pane on the left.
+* Ruby code:
+ puts "Ruby code"
+* Unix shell command, although the prompt may be left off when obvious:
+ you@host:myproject> echo "Unix command run from the 'myproject' directory"
+* AutomateIt interactive shell command, although the prompt may be left off when obvious:
+ ai> puts "I'm in the Interpreter"
+
+=== Glossary
+
+AutomateIt uses a number of unique terms:
+
+* *Recipe* -- A file that contains AutomateIt commands.
+* *Project* -- A special directory that contains related recipes and helper files.
+* *Interpreter* -- The part of AutomateIt that runs commands.
+* *Plugin* -- A part of AutomateIt that describes related features.
+* *Driver* -- An implementation of a plugin. There are often multiple drivers per plugin.
+
+=== Interactive shell
+
+AutomateIt comes with an interactive shell, which is useful for exploring commands and developing recipes.
+
+Here's what an interactive shell session looks like (text following the <tt>#</tt> symbol explains the line and is not actually part of the session):
+
+ you@host:tmp> automateit      # Start the AutomateIt interactive shell
+ => AutomateIt Shell v0.5.0    # Welcome message from AutomateIt
+ ai> puts "Hello world!"       # AutomateIt's prompt and a user command
+ Hello world!                  # Output from the "puts" command
+ => nil                        # Return value of the "puts" command
+ ai> self.class                # Prompt and another user command
+ => AutomateIt::Interpreter    # Return value of "self.class"
+ ai> <CTRL-D>                  # Press <CTRL-D> to exit the shell
+ you@host:tmp>                 # We're back to the Unix shell
+
+=== Recipe files
+
+The Interpreter can run recipe files that contain AutomateIt commands.
+
+For example, create a <tt>/tmp/hello.rb</tt> file that contains:
+
+ puts "Hello, I'm an #{self.class}"
+
+Then run the recipe:
+
+ you@host:tmp> automateit /tmp/hello.rb
+ Hello, I'm an AutomateIt::Interpreter
+
+=== String evaluation
+
+The Interpreter can also evaluate commands as strings:
+
+ you@host:tmp> automateit -e 'puts self.class'
+ AutomateIt::Interpreter
+
+=== Exploring the Interpreter's unique methods
+
+The Interpreter provides some unique methods:
+
+ ai> unique_methods
+ => ["account_manager", "address_manager", "cd", "chmod", ...
+
+The names are Interpreter methods. Names ending with +_manager+ are methods for accessing plugins, while the rest are normal methods. The AutomateIt::Interpreter documentation provides a list of these methods and links to their individual documentation.
+
+For example, one of the commands listed is +pwd+, which can be used like this:
+
+ ai> pwd
+ => "/tmp"
+
+AutomateIt provides many commands that work just like the Unix shell commands you already know, so you'll be productive quickly.
+
+=== Conditional execution
+
+AutomateIt executes only the commands needed to achieve a desired state, which makes recipes repeatable, reusable and maintainable.
+
+Eliminating the distinction between setup and maintenance means the same command can perform both tasks without concern for the host's condition. A properly-written recipe can be run and re-run immediately afterwards, and it will do nothing the second time because all changes have already been applied.
+
+An example can make this clearer -- again the text following the <tt>#</tt> symbol explains the commands and is not part of the session:
+
+ you@host:tmp> automateit   # Start the AutomateIt interactive shell
+ => AutomateIt Shell v0.5.0 # Welcome message
+ ai> mkdir "asdfasdf"       # Create a directory called "asdfasdf"
+ ** mkdir asdfasdf          # Message showing that directory was created
+ => ["asdfasdf"]            # Return value with array of directories created
+ ai> mkdir "asdfasdf"       # Try to create the same directory again
+ => false                   # Nothing happened, the directory already exists
+ ai> rmdir "asdfasdf"       # Remove the directory
+ ** rmdir asdfasdf          # Message showing that directory was removed
+ => ["asdfasdf"]            # Return value with array of directories removed
+ ai> rmdir "asdfasdf"       # Try to remove the directory again
+ => false                   # Nothing happened, there's no directory to remove
+
+Notice how the second time +mkdir+ was run above, it returned +false+ and didn't create a directory? That's the conditional execution in action, realizing the action has already been performed. Similarly, the +rmdir+ only ran the first time, but returned +false+ and took no action when the directory was already gone. You can use the return values of these commands to write sophisticated logic of your own based on what actions took place.
+
+=== Plugins
+
+AutomateIt uses an extensible plugin architecture to group together related commands:
+
+* AutomateIt::AccountManager -- Manipulates users and groups.
+* AutomateIt::AddressManager -- Manipulates host's network addresses.
+* AutomateIt::EditManager -- Edits files and strings.
+* AutomateIt::FieldManager -- Queries configuration variables.
+* AutomateIt::PackageManager -- Manipulates software packages.
+* AutomateIt::PlatformManager -- Queries platform, such as its OS version.
+* AutomateIt::ServiceManager -- Manipulates services, such as Unix daemons.
+* AutomateIt::ShellManager -- Manipulates files and executes Unix commands.
+* AutomateIt::TagManager -- Groups hosts by role and queries membership.
+* AutomateIt::TemplateManager -- Renders templates to files.
+
+Plugins can be accessed from the Interpreter like this:
+
+ ai> shell_manager.pwd
+ => "/tmp"
+
+The most common plugin methods have aliased shortcuts. For example, +pwd+ is the alias for <tt>shell_manager.pwd</tt>. These are documented in AutomateIt::Interpreter's "Aliased methods."
+
+=== Drivers
+
+Each plugin has one or more drivers that implement its functionality.
+
+For example:
+
+* AutomateIt::ShellManager -- A plugin for running shell commands.
+* AutomateIt::ShellManager::Portable -- A portable but limited-functionality driver that implements the ShellManager's methods.
+* AutomateIt::ShellManager::Unix -- A full-featured driver implementing ShellManager's methods that only runs on Unix-like systems.
+
+=== Plugins provide APIs, drivers provide implementations
+
+Plugins describe consistent interfaces for related features, and drivers implement this API for different tools.
+
+For example, AutomateIt provides a common API for all packaging tools: AutomateIt::PackageManager. It provides drivers for packaging tools like APT, YUM, Gem, Egg and others. To install a package called +foo+ with the +apt+ driver:
+
+ ai> package_manager.install "foo", :with => :apt
+
+AutomateIt will check if +foo+ is installed, install it if needed, or do nothing if the package is present. This API is the same for all packaging tools, making it easy to get work done by using high-level AutomateIt commands instead of cryptic tool-specific commands. Although AutomateIt uses the low-level tools, it uses them with best-practices approaches and hides the senseless complexity from the user.
+
+What's the big deal? Consider how one would install packages from the Unix shell. Most packaging tools are pathologically dysfunctional and make it bafflingly difficult to programmatically install or uninstall packages, or tell if a package is installed. Many don't use exit values and require complex output parsing. Others require user-input even when it's obvious that none is needed. Almost all make it necessary to write conditional code because they'll either fail with errors if told to install an existing package, destroy an existing setup, or install duplicate packages. Writing Unix shell code to handle all these quirks is frustrating and risky.
+
+Here's a sample Unix shell command for installing a package using one of the simplest, most reasonable tools available -- although note that unlike AutomateIt, this shell command can't handle multiple packages, is slower, can't be previewed, and has no consistent error handling:
+
+ if dpkg-query -W --showformat '${Status}\n' foo 2>&1 | \
+     egrep -q '(^| )installed'; then
+   apt-get install -y -q some_package_name < /dev/null
+ done
+
+Now compare that hideous command to the simple, clear and consistent AutomateIt command before it. AutomateIt's plugins and drivers are easy to install and write. As more are written, more people will hopefully be freed from needlessly convoluted low-level commands, and able to get simple things done simply. AutomateIt's consistent API, multiple drivers, sane defaults and conditional-checking make it easier to write clear and maintainable recipes than using the low-level directly commands from the Unix shell.
+
+=== Driver auto-detection
+
+AutomateIt can automatically detect the most suitable driver for each plugin command.
+
+For example, the AutomateIt::PackageManager plugin has drivers called AutomateIt::PackageManager::APT and AutomateIt::PackageManager::YUM. On a Debian system, which uses the <tt>apt-get</tt> packaging tool, AutomateIt will default to using the AutomateIt::PackageManager::APT driver:
+
+ ai> package_manager.installed? "apache2"
+ => true
+
+=== Using a specific driver
+
+Sometimes it's necessary to specify the driver to use. The recommended way to do this is to pass a <tt>:with => :driver_name</tt> option to the plugin command.
+
+For example, tell the package manager to use the Gem driver:
+
+ ai> package_manager.installed? "automateit", :with => :gem
+ => true
+
+You can also completely bypass the plugin and its auto-detection to directly interact with the driver:
+
+ ai> package_manager[:gem].installed? "automateit"
+ => true
+
+=== Projects
+
+A project is a special directory that contains related recipes and helper files. Although it's possible to run recipe files without a project, a project provides many useful features, described in the AutomateIt::Project documentation.
+
+A project is created by specifying the directory to create:
+
+ you@host:tmp> cd /tmp
+ you@host:tmp> automateit --create myproject
+ ** mkdir -p myproject
+ => Creating AutomateIt project at: myproject
+ ** cd myproject
+ ** mkdir config
+ ** cd config
+ => Rendering 'tags.yml' because of it doesn't exist
+ => Rendering 'fields.yml' because of it doesn't exist
+ => Rendering 'automateit_env.rb' because of it doesn't exist
+ ** cd -
+ ** mkdir dist
+ ** cd dist
+ => Rendering 'README.txt' because of it doesn't exist
+ ** cd -
+ ** mkdir lib
+ ** cd lib
+ => Rendering 'README.txt' because of it doesn't exist
+ ** cd -
+ ** mkdir recipes
+ ** cd recipes
+ => Rendering 'README.txt' because of it doesn't exist
+ ** cd -
+ => DONE!
+
+This creates a <tt>/tmp/myproject</tt> directory with the newly-created project. It contains directories, each with a <tt>README.txt</tt> file explaining the directory's purpose, and individual files like <tt>tags.yml</tt> that contain comments with basic usage instructions.
+
+The project creator is an AutomateIt recipe, so it's smart enough to only execute the commands needed. If the project creator is re-run against an existing project, it won't make any changes because none are needed:
+
+ you@host:tmp> automateit --create myproject
+ => Found AutomateIt project at: myproject
+ => DONE!
+
+=== Project recipes
+
+To associate a recipe with a project, put it into the project's +recipes+ directory.
+
+For example, create a <tt>recipes/hello_project.rb</tt> file with these contents:
+
+ puts "Hello, this is: " + project
+
+And run it:
+
+ you@host:myproject> automateit recipes/hello_project.rb
+ Hello, this is: /tmp/myproject
+
+The Interpreter automatically loads the project. The +project+ method contains the project path and is only available when executing a recipe associated with a project.
+
+=== Fields
+
+A project's <tt>config/fields.yml</tt> file is meant to store configuration constants, like custom paths for applications. Fields abstract configuration variables for recipes, improving maintainability by separating data and logic. Fields can also be easily queried from Unix using the +aifield+ command. More information about fields and +aifield+ can be found in AutomateIt::FieldManager.
+
+For example, consider a fields file with the following contents:
+
+ myuser: dhh
+ myapp:
+   path: /var/www/rails
+
+These fields can be used from the interactive shell like this:
+
+ you@host:myproject> pwd
+ /tmp/myproject
+ you@host:myproject> automateit -p .    # (1) Load project from current directory
+ => AutomateIt Shell v0.5.0
+ ai> lookup :myuser                     # (2) Lookup string value by symbol key
+ => "dhh"
+ ai> lookup "myuser"                    # (3) Lookup string value by string key
+ => "dhh"
+ ai> lookup "myapp"                     # (4) Lookup hash value by string key
+ => {"path"=>"/var/www/rails"}
+ ai> lookup "myapp#path"                # (5) Lookup string value by compound key
+ => "/var/www/rails"
+
+To load a project's fields, the AutomateIt interactive shell needs to know which project to load. On the line annotated (1), <tt>automateit -p .</tt> tells it to load the project from the "." directory, the current directory. The argument can be an absolute or relative path. The command accepts other arguments and environmental options, run <tt>automateit --help</tt> for details.
+
+Fields can be queried by string (2) or symbol (3) keys. The +lookup+ command can return any associated data, usually a string, but it can be a complex type, such as a hash (4). The compound-key (5) syntax is a convenient syntax for looking up nested keys in a hash. For example, <tt>lookup "myapp#path"</tt> is a shortcut for <tt>lookup("myapp")["path"] -- these query the +myapp+ hash and return the value of its +path+ key.
+
+The Interpreter loads a project and its fields automatically for recipes, so there is no need to specify the <tt>-p</tt> option. For example, a project recipe file <tt>recipes/hello_fields.rb</tt> contains:
+
+ puts lookup("myapp#path")
+
+This recipe will load its project and fields automatically:
+
+ you@host:myproject> automateit recipes/hello_fields.rb
+ /var/www/rails
+
+=== Tags
+
+A project's <tt>config/tags.yml</tt> file describes tags assigned to hosts, grouping together hosts by their roles and attributes.
+
+For example, this tags file describes a "desktops" tag with three hosts named "satori", "sunyata" and "michiru", and another tag named "notebooks" with other hosts:
+  desktops:
+    - satori
+    - sunyata
+    - michiru
+  notebooks:
+    - rheya
+    - avijja
+
+The tags can be accessed like this when run on a host named "satori":
+
+ you@satori:myproject> automateit -p .
+ ai> tags
+ => ["satori", "desktops", "localhost", ...]        # Tags for this host
+ ai> tagged?("desktops")                            # Is this host tagged with "desktops"?
+ => true
+ ai> tagged?("notebooks")
+ => false
+ ai> tagged?(:satori)                               # Strings and symbols are treated the same.
+ => true
+ ai> tagged?("satori")
+ => true
+ ai> tagged?("satori || desktops")                  # Query by simple boolean expression
+ => true
+ ai> tagged?("(satori || desktops) && !notebooks")  # Query by complex boolean expression
+ => true
+
+Role-based behavior can be demonstrated by creating a file called <tt>recipes/hello_tags.rb</tt>:
+
+ if tagged?("localhost")
+  puts "I'm a localhost!"
+ end
+ if tagged?("desktops")
+  puts "Special commands to execute on desktops"
+ elsif tagged?("notebooks")
+  puts "Special commands to execute on notebooks"
+ end
+
+Here are the results when executed on a host called "satori":
+
+ you@satori:myproject> automateit recipes/hello_tags.rb
+ I'm a localhost!
+ Special commands to execute on desktops
+
+Notice how the +notebooks+ section wasn't executed? Tags make it possible to create sophisticated recipes that run commands on only the appropriate systems.
+
+More documentation on tags can be found in AutomateIt::TagManager.
+
+=== Previewing commands
+
+AutomateIt lets you preview commands the recipe will run without actually letting them actually modify the system.
+
+The Interpreter has a boolean that determines if it'll make changes to your system. This one variable can be called from two different ways:
+* <tt>writing?</tt> -- Will it write changes?
+* <tt>noop?</tt> -- Will it not write changes and just preview them? The +noop+ means "no-operation"
+
+Here's an example with the interactive shell, with comments for annotations:
+
+ ai> noop true                # Enter noop mode
+ ai> noop?                    # Currently in noop mode?
+ => true                      # Yes, in noop mode
+ ai> writing?                 # Writing to disk? The opposite of noop
+ => false                     # No, not writing
+ ai> mkdir "foo"              # Try to create a directory
+ ** mkdir foo                 # Message showing directory will be creating
+ => ["foo"]                   # Return value with directories to create
+ ai> File.directory? "foo"    # Was the directory actually made?
+ => false                     # No, noop mode prevented the change
+
+Recipes can take advantage of the noop mode as well, consider a <tt>mkdir_example.rb</tt> recipe:
+
+ mkdir "foo"
+
+To run this recipe with noop mode, pass the <tt>-n</tt> option to the Interpreter shell:
+
+ you@host:tmp> automateit -n mkdir_example.rb
+ ** mkdir foo
+ you@host:tmp> ls foo
+ ls: foo: No such file or directory
+
+Notice how the directory wasn't actually created? This is great because you can see exactly what commands AutomateIt will run without actually having it apply the changes.
+
+*WARNING*: Previewing code can be dangerous. Read
+previews.txt[link:files/docs/previews_txt.html] for instructions on how to
+write code that can be safely previewed.
+
+=== Conclusion
+
+I hope you enjoy working with AutomateIt and look forward to hearing about your
+experiences with it. Drivers, patches, documentation and ideas are welcome.
+
+Thank you for taking the time to read this!
+
+- Igal Koshevoy (igal@pragmaticraft.com)