software_smithy 1.1

data/README.rdoc

@@ -0,0 +1,114 @@
+Smithy is a tool that aims to replicate and improve upon functionality of
+{SWTools}[http://www.olcf.ornl.gov/center-projects/swtools/].
+
+Smithy's goals are:
+
+* Provide a simplified and intuitive command line interface for installing software.
+* Reduce the number of steps necessary for each function
+
+= Installation
+
+== Installing ruby
+
+Smithy requires ruby 1.9.2 or later. Most distrubutions only ship version 1.8.7
+and you may need build your own copy. The simplest way is using the excellent
+{ruby-build}[https://github.com/sstephenson/ruby-build] script. As an example,
+this will install ruby with a prefix of <tt>/sw/xk6/ruby/1.9.3-p286/sles11.1_gnu4.3.4</tt>
+
+    curl -L https://github.com/sstephenson/ruby-build/archive/master.zip -o ruby-build.zip
+    unzip ruby-build.zip
+    cd ruby-build-master
+    ./bin/ruby-build -h
+    ./bin/ruby-build --definitions
+    ./bin/ruby-build 1.9.3-p286 /sw/xk6/ruby/1.9.3-p286/sles11.1_gnu4.3.4
+
+Many sites use {Environment Modules}[http://modules.sourceforge.net/] to allow
+users to load and unload software into their environment. Here is sample
+modulefile for ruby installed in the previous example.
+
+    #%Module1.0
+    proc ModulesHelp { } {
+      puts stderr "Ruby 1.9.3 patch 286"
+      puts stderr "The gem command will install gems to the ~/.gem directory."
+    }
+    module-whatis "Ruby 1.9.3-p286"
+
+    set PREFIX /sw/xk6/ruby/1.9.3-p286/sles11.1_gnu4.3.4
+    prepend-path PATH            $PREFIX/bin
+    prepend-path LD_LIBRARY_PATH $PREFIX/lib
+    prepend-path MANPATH         $PREFIX/share/man
+    setenv       GEM_HOME        $env(HOME)/.gem/ruby/1.9.1
+    setenv       GEM_PATH        $env(HOME)/.gem/ruby/1.9.1:$PREFIX/lib/ruby/gems/1.9.1
+    prepend-path PATH            $env(HOME)/.gem/ruby/1.9.1/bin:$PREFIX/lib/ruby/gems/1.9.1/bin
+
+This file should be saved to <tt>$MODULEPATH/ruby/1.9.3</tt>
+
+== Installing smithy
+
+=== Choosting an install location
+
+There are two ways to install smithy. The simplest is by simply installing the
+<tt>software_smithy</tt> gem. This will install smithy and it's required gems in
+your home directory:
+
+    gem install software_smithy
+
+If you are installing in production for all users you will want to install
+smithy somewhere everyone can access. There are two places you might want to do
+this:
+
+==== Ruby's default GEM_PATH
+
+If you installed manually using the examples above, this will be something like
+<tt>PREFIX/lib/ruby/gems/1.9.1</tt> This will give access to smithy whenever
+ruby is loaded into a users environment using the modulefile above. To install
+to ruby's default GEM_PATH (following the above example):
+
+    export GEM_HOME=/sw/xk6/ruby/1.9.3-p286/sles11.1_gnu4.3.4/lib/ruby/gems/1.9.1
+    gem install software_smithy --no-rdoc --no-ri
+
+==== A different location of your choosing.
+
+This is useful if you want a single smithy install location for more than one
+install of ruby (typically on separate machines). This method is a bit more
+complicated and requires users to load smithy into their environment manually in
+addition to ruby.
+
+Assuming you use environment modules you can install smithy to a separate
+directory e.g. <tt>/sw/tools/smithy</tt> with the install_smithy script. This
+will setup a folder containing the smithy gem, a modulefile and a script that
+sets up shell completion. Users can load smithy into their environment by
+running:
+
+    source /sw/tools/smithy/environment.sh
+
+=== Loading smithy into your environment
+
+Smithy depends on a config file to define it's behavior. For an example see
+{etc/smithyrc}[https://github.com/AnthonyDiGirolamo/smithy/blob/master/etc/smithyrc]
+This can be set via the <tt>$SMITHY_CONFIG</tt> environment variable.
+
+You may wish to set this using a modulefile or a shell script. Examples are
+provided in
+{modulefiles/smithy/1.0}[https://github.com/AnthonyDiGirolamo/smithy/blob/master/modulefiles/smithy/1.0]
+and
+{environment.sh}[https://github.com/AnthonyDiGirolamo/smithy/blob/master/environment.sh]
+
+= License
+
+Smithy is based on the ideas created in SWTools and uses a BSD license. See
+LICENSE for the exact text.
+
+= Contributing and Support
+
+The smithy {github repo}[https://github.com/AnthonyDiGirolamo/smithy] contains
+all development files. Please fork and send me a pull request with any additions
+or changes.
+
+If you encounter any issues please {open an
+issue}[https://github.com/AnthonyDiGirolamo/smithy/issues] on github. Or send me
+an email.
+
+= smithy command line usage
+
+:include:smithy.rdoc

data/bin/smithy

@@ -0,0 +1,586 @@
+#!/usr/bin/env ruby
+# Smithy is freely available under the terms of the BSD license given below. {{{
+#
+# Copyright (c) 2012. UT-BATTELLE, LLC. All rights reserved.
+#
+# Produced by the National Center for Computational Sciences at Oak Ridge
+# National Laboratory. Smithy is a based on SWTools, more information on SWTools
+# can be found at: http://www.olcf.ornl.gov/center-projects/swtools/
+#
+# This product includes software produced by UT-Battelle, LLC under Contract No.
+# DE-AC05-00OR22725 with the Department of Energy.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# - Redistributions of source code must retain the above copyright notice, this
+#   list of conditions and the following disclaimer.
+#
+# - Redistributions in binary form must reproduce the above copyright notice, this
+#   list of conditions and the following disclaimer in the documentation and/or
+#   other materials provided with the distribution.
+#
+# - Neither the name of the UT-BATTELLE nor the names of its contributors may
+#   be used to endorse or promote products derived from this software without
+#   specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# }}}
+
+# Absolute path to the smithy gem prefix
+@@smithy_bin_root = File.expand_path(File.dirname(File.realpath(__FILE__))+ '/../')
+# Add the smithy lib dir to the ruby library search path
+$: << File.expand_path(@@smithy_bin_root + '/lib')
+
+require 'English'
+require 'fileutils'
+
+require 'rubygems'
+require 'open4'
+require 'rainbow'
+require 'readline'
+require 'terminal-table'
+require 'erb'
+require 'uri'
+require 'kramdown'
+require 'active_support/core_ext/hash'
+require 'active_support/core_ext/string'
+# require 'debugger'      #XXX
+# require 'awesome_print' #XXX
+
+require 'gli'
+include GLI::App
+
+require 'smithy'
+include Smithy
+
+version Smithy::VERSION
+
+program_desc "Smithy will help you build, test, and install software with ease."
+
+desc "Machine architecure to operate on"
+arg_name "NAME"
+flag [:arch]
+
+desc "Group name for files created by smithy"
+arg_name "GROUP"
+flag ["file-group-name"]
+
+desc "Disable group writable file creation"
+switch ["disable-group-writable"], :negatable => false
+
+desc "The root level directory for software"
+arg_name "PATH"
+flag ["software-root"]
+
+desc "The prefix of programming environments for this machine"
+arg_name "NAME"
+flag ["prgenv-prefix"]
+
+desc "The root level directory for published web files"
+arg_name "PATH"
+flag ["web-root"]
+
+desc "The root level directory global description files"
+arg_name "PATH"
+flag ["descriptions-root"]
+
+desc "Disable or enable color output (default: enabled)"
+switch "colors"
+
+desc "Alternate config file, also set by $SMITHY_CONFIG"
+arg_name "FILE"
+flag [:"config-file"]
+
+desc "Always overwrite existing files"
+switch [:f, :force], :negatable => false
+
+desc "Search currently installed software"
+arg_name "PATTERN"
+command ['search'] do |c|
+  output_formats = {
+    'path'     => Format::Path.new,
+    'name'     => Format::Name.new,
+    'table'    => Format::Table.new,
+    'csv'      => Format::CSV.new,
+    'dokuwiki' => Format::Doku.new
+  }
+  c.desc 'Format of the output'
+  c.arg_name output_formats.keys.join('|')
+  c.default_value output_formats.keys.first
+  c.flag :format
+
+  #c.desc 'Only search packages ready for web publishing'
+  #c.switch [:w, "web-enabled"]
+
+  #sorting = %w{name date user}
+  #c.desc 'Field to sort in ascending order'
+  #c.arg_name sorting.join('|')
+  #c.default_value sorting.first
+  #c.flag :sort
+
+  c.action do |global_options,options,args|
+    swroot = Smithy::Config.full_root
+
+    formatter = output_formats[options[:format]]
+    raise "Unknow format \"#{options[:format]}\" valid formats include: #{output_formats.keys.join(',')}" if formatter.nil?
+
+    notice "Searching in #{swroot}" unless formatter.class == Format::Doku
+
+    software = Package.all :root => swroot
+
+    count = 0
+    formatter.before
+    if args.empty?
+      formatter.format(software, swroot)
+      count += software.count
+    else
+      args.each do |a|
+        result = software.select { |s| s =~ /#{a}/ }
+        formatter.format(result, swroot)
+        count += result.count
+      end
+    end
+    formatter.after
+
+    notice "#{count} Found" unless formatter.class == Format::Doku
+  end
+end
+
+desc "Test software"
+command ['test'] do |c|
+  c.desc 'Test log file name located within the software prefix.'
+  c.arg_name 'FILE'
+  c.default_value 'test.log'
+  c.flag ["log-name"]
+
+  c.desc "Disable logging"
+  c.switch ["disable-log"], :negatable => false
+
+  c.desc 'Suppress messages from STDOUT.'
+  c.switch [:s, "suppress-stdout"], :negatable => false
+
+  c.desc 'Ignore .lock file and run anyway'
+  c.switch [:f, "force"], :negatable => false
+
+  c.desc 'See what scripts will be run without running them'
+  c.switch [:n, "dry-run"], :negatable => false
+
+  c.action do |global_options,options,args|
+    packages = args.dup
+    if args.empty?
+      notice "Reading package names from STDIN..."
+      packages = STDIN.readlines.map{|p| p.chomp}
+    end
+
+    raise "You must supply at least one package to test" if packages.empty?
+
+    packages.each do |package|
+      p = Package.new :path => package
+      p.valid?
+      p.run_script :script => :test,
+                  :dry_run => options[:"dry-run"],
+          :disable_logging => options[:"disable-log"],
+                 :log_name => options[:"log-name"],
+          :suppress_stdout => options[:"suppress-stdout"],
+                    :force => options[:"force"]
+    end
+  end
+end
+
+
+desc "Build software"
+long_desc "The software to build may be either the absolute path or the full name of the software. The full name includes version numbers and build names using the format: NAME/VERSION/BUILD."
+arg_name "PATH"
+command ['build'] do |c|
+  c.desc 'Build log file name located within the software prefix.'
+  c.arg_name 'FILE'
+  c.default_value 'build.log'
+  c.flag ["log-name"]
+
+  c.desc "Disable logging"
+  c.switch ["disable-log"], :negatable => false
+
+  c.desc 'Suppress messages from STDOUT.'
+  c.switch [:s, "suppress-stdout"], :negatable => false
+
+  c.desc 'Ignore .lock file and run anyway'
+  c.switch [:f, "force"], :negatable => false
+
+  c.desc 'See what scripts will be run without running them'
+  c.switch [:n, "dry-run"], :negatable => false
+
+  c.action do |global_options,options,args|
+    packages = args.dup
+    if args.empty?
+      notice "Reading package names from STDIN..."
+      packages = STDIN.readlines.map{|p| p.chomp}
+    end
+
+    raise "You must supply at least one package to build" if packages.empty?
+
+    packages.each do |package|
+      p = Package.new :path => package
+      p.valid?
+      p.run_script :script => :build,
+                  :dry_run => options[:"dry-run"],
+          :disable_logging => options[:"disable-log"],
+                 :log_name => options[:"log-name"],
+          :suppress_stdout => options[:"suppress-stdout"],
+                    :force => options[:"force"]
+    end
+  end
+end
+
+desc "Generate a new build and all necessary files"
+long_desc <<-EOS
+The new command will create all necessary files needed to add a new software package. Some care should be given to naming new packages. Some considerations are package names, version numbers, and build names. New package names should be of the format NAME/VERSION/BUILD
+
+- NAME of the package should be all lower case and one word. If multiple words are necessary separate them with dashes '-'.
+
+- VERSION numbers should be standard numbers separated by periods. If another format is necessary ensure that the numbers can be lexigraphically sorted in order of oldest release to newest.
+
+- BUILD names should consist of two pieces of information separated by an underscore '_', the package's intended operating system and compiler.
+
+EXAMPLES:
+
+silo/4.8/sles11.1_gnu4.5.3
+fftw/3.2.2/cle4.0_pgi11.10.0
+EOS
+arg_name "NAME"
+command ['new'] do |c|
+  c.desc "Create description file for website"
+  c.switch ["web-description"], :negatable => false
+
+  c.desc "See what files will be created when running new without creating them"
+  c.switch [:n, "dry-run"], :negatable => false
+
+  c.desc "Skip modulefile generation"
+  c.switch ["skip-modulefile"], :negatable => false
+
+  c.desc "Provide a tarball to unpack, either a file or URL (optional)"
+  c.arg_name "FILE|URL"
+  c.flag [:t, :tarball]
+  c.action do |global_options,options,args|
+    raise "You must supply a name to create new packages" if args.empty?
+
+    p = Package.new :path => args.first
+
+    p.valid?
+
+    if options[:tarball] =~ URI::ABS_URI
+      url = options[:tarball]
+    elsif options[:tarball]
+      archive = File.absolute_path options[:tarball]
+      raise "The archive #{archive} does not exist" unless File.exists? archive
+    end
+    p.create :dry_run => options[:"dry-run"],
+                 :web => options[:"web-description"]
+
+    if options[:tarball]
+      downloaded = p.download(url) if url
+      archive = downloaded if downloaded
+      p.extract :archive => archive, :dry_run => options[:"dry-run"]
+    end
+
+    unless options[:"skip-modulefile"]
+      m = ModuleFile.new :package => p
+      m.create :dry_run => options[:"dry-run"]
+    end
+
+    notice "Next Steps"
+    notice_command " smithy edit last"            , "Edit build scripts or modulefile"
+    notice_command " smithy build last"           , "Run build script"
+    notice_command " smithy module create"        , "Generate a modulefile" if options[:"skip-modulefile"]
+    notice_command " smithy module deploy last"   , "Install modulefile"
+    notice_command " smithy publish last"         , "Publish web description"
+  end
+end
+
+desc "Edit package support files"
+arg_name "build|test|env|modulefile NAME"
+command ['edit'] do |c|
+  c.desc "Editor for opening script files"
+  c.arg_name "BINARY"
+  c.flag [:e, :editor]
+
+  c.desc "Split editing window with requested file and the environment (remodule) file"
+  c.switch [:s, :split], :negatable => false
+
+  c.action do |global_options,options,args|
+    operations = [:build, :environment, :modulefile, :test, :config]
+    arguments = args.dup
+    operation = nil
+    operations.each do |o|
+      if arguments.include? o.to_s
+        operation = o
+        arguments.delete o.to_s
+        break
+      end
+    end
+
+    raise "You must supply a package name to edit" if arguments.empty?
+
+    if operation == :config
+      file_name = @smithy_config_file
+      status = Kernel::system(editor, file_name)
+    else
+      p = Package.new :path => arguments.first
+      p.valid?
+      p.prefix_exists!
+      p.rebuild_script_exists!
+
+      # if no file, prompt
+      if operation.nil?
+        puts "Edit which file? (default is build)"
+        width = 25
+        notice_command "(b)uild", p.rebuild_script, width
+        notice_command "(e)nvironment", p.remodule_script, width
+        notice_command "(m)odulefile", ModuleFile.new(:package => p).module_file, width
+        notice_command "(t)est", p.retest_script, width
+
+        while operation.nil? do
+          prompt = Readline.readline("> ")
+          prompt.downcase!
+
+          if prompt =~ /(1|b|build)/
+            operation = :build
+          elsif prompt =~ /(2|e|environment)/
+            operation = :environment
+          elsif prompt =~ /(3|m|modulefile)/
+            operation = :modulefile
+          elsif prompt =~ /(4|t|test)/
+            operation = :test
+          else
+            operation = :build
+          end
+        end
+      end
+
+      file_names = []
+      case operation
+      when :build
+        file_names << p.rebuild_script
+      when :test
+        file_names << p.retest_script
+      when :environment
+        file_names << p.remodule_script
+      when :modulefile
+        file_names << ModuleFile.new(:package => p).module_file
+      else
+        file_names << p.rebuild_script
+      end
+
+      case operation
+      when :build || :test
+        file_names << p.remodule_script if options[:split]
+      end
+
+      launch_editor :editor => options[:editor],
+        :split => options[:split],
+        :files => file_names
+    end
+  end
+end
+
+desc "Repair a package"
+long_desc "Repair permissions and check that files exist"
+arg_name "NAME"
+command ['repair'] do |c|
+  c.desc "Verify permissions only"
+  c.switch [:n, "dry-run"], :negatable => false
+
+  c.action do |global_options,options,args|
+    packages = args.dup
+    if args.empty?
+      notice "Reading package names from STDIN..."
+      packages = STDIN.readlines.map{|p| p.chomp}
+    end
+
+    raise "You must supply at least one package to repair" if packages.empty?
+
+    packages.each do |package|
+      p = Package.new :path => package
+      p.valid?
+      p.repair :dry_run => options[:"dry-run"]
+    end
+  end
+end
+
+desc "Publish packages to web"
+arg_name "NAME"
+command ['publish'] do |c|
+  c.desc "See what files will be created without creating them"
+  c.switch [:n, "dry-run"], :negatable => false
+
+  c.action do |global_options,options,args|
+    www_root = Smithy::Config.web_root
+    raise """Cannot determine where to publish web descriptions.\n       Please add a 'web-root: PATH' entry to the smithy config file.""" if www_root.nil?
+
+    packages = args.dup
+    if args.empty?
+      notice "Reading package names from STDIN..."
+      packages = STDIN.readlines.map{|p| p.chomp}
+    end
+
+    raise "You must supply at least one package to publish" if packages.empty?
+
+    packages = Package.all_web if packages.include?("all")
+
+    notice "Publishing #{packages.size} package#{'s' if packages.size>1}#{' (dry-run)' if options[:"dry-run"]}"
+
+    descriptions = []
+
+    packages.each do |package|
+      d = Description.new :package => package
+      if d.valid? && d.publishable?
+        d.deploy :dry_run => options[:"dry-run"]
+        descriptions << d
+      else
+        notice_fail "Flagged for no publishing, remove 'noweb' from #{d.exceptions_file} and re-publish."
+      end
+    end
+
+    ["alphabetical","all"].each do |p|
+      Description.update_page p,
+        :descriptions => descriptions,
+        :dry_run      => options[:"dry-run"]
+    end
+  end
+end
+
+
+desc "Manage modulefiles for a package"
+arg_name "NAME"
+command ['module'] do |c|
+  c.desc "See what files will be created without creating them"
+  c.switch [:n, "dry-run"], :negatable => false
+
+  c.desc "Run the proper module command to add a package's modulefile to the MODULEPATH. This allows you to edit and test a modulefile in isolation. By adding the correct directory to MODULEPATH you may run module commands as you normally would."
+  c.command :use do |use|
+    use.action do |global_options,options,args|
+      raise "You must supply a package name" if args.empty?
+      p = Package.new :path => args.first
+      p.valid?
+      m = ModuleFile.new :package => p
+
+      mp = m.module_path
+      if Dir.exist?(mp)
+        puts "module use #{mp}"
+      else
+        raise "No modulefiles found in #{mp}"
+      end
+    end
+  end
+
+  c.desc "Generate a modulefile for a given package. This operation normally happens after creating a new package. It is provided separately here for use with existing packages."
+  c.command :create do |create|
+    create.action do |global_options,options,args|
+      raise "You must supply a package name" if args.empty?
+      p = Package.new :path => args.first
+      p.valid?
+      m = ModuleFile.new :package => p
+
+      m.create :dry_run => options[:"dry-run"]
+      notice "Next Steps"
+      notice_command " smithy edit modulefile last" , "Edit generated modulefile"
+      notice_command " smithy module deploy last"   , "Install modulefile"
+    end
+  end
+
+  c.desc "Copy a package's modulefile to the system module path. This is typically done after you have tested a new module in isolation."
+  c.command :deploy do |deploy|
+    deploy.action do |global_options,options,args|
+      raise "You must supply a package name" if args.empty?
+      p = Package.new :path => args.first
+      p.valid?
+      m = ModuleFile.new :package => p
+
+      m.deploy :dry_run => options[:"dry-run"]
+    end
+  end
+
+  c.desc "alias for 'smithy edit modulefile'"
+  c.command :edit do |edit|
+    edit.action do |global_options,options,args|
+      raise "You must supply a package name" if args.empty?
+      p = Package.new :path => args.first
+      p.valid?
+      m = ModuleFile.new :package => p
+
+      launch_editor :files => [m.module_file]
+    end
+  end
+
+end
+
+desc "Display internal smithy values"
+command ['show'] do |c|
+  c.desc "Display the package name used in the last smithy command. This is stored in the '~/.smithyrc' file."
+  c.command :last do |last|
+    last.action do
+      puts Smithy::Config.last_prefix
+    end
+  end
+
+  c.desc "List all architectures know to smithy."
+  c.command :arch do |arch|
+    arch.action do
+      Smithy::Config.architectures
+    end
+  end
+end
+
+# command ['publish_cray'] do |c|
+#   c.desc "See what files will be created without creating them"
+#   c.switch [:n, "dry-run"], :negatable => false
+
+#   c.desc "Exclude directories specified in $MODULEPATH"
+#   c.flag [:except]
+
+#   c.desc "Parse given directories only, $MODULEPATH is not checked"
+#   c.flag [:only]
+
+#   c.action do |global_options,options,args|
+#     packages, default_names = ModuleFile.get_module_names(options)
+#     Package.create_stubs_from_modules(packages, default_names, options)
+#   end
+# end
+
+pre do |global,command,options,args|
+  # ap global #XXX
+  Sickill::Rainbow.enabled = false if global[:"colors"] == false
+
+  Smithy::Config.config_file_name = File.join(@@smithy_bin_root,"etc/smithyrc")
+  Smithy::Config.load_configuration(global)
+
+  # Pre logic here
+  # Return true to proceed; false to abort and not call the chosen command
+  # Use skips_pre before a command to skip this block on that command only
+  true
+end
+
+post do |global,command,options,args|
+  # Post logic here
+  # Use skips_post before a command to skip this
+  # block on that command only
+end
+
+on_error do |exception|
+  # Error logic here
+  # return false to skip default error handling
+  true
+end
+
+exit run(ARGV)
+
+# vim: ft=ruby