software_smithy 1.1 → 1.6

data/lib/smithy.rb

@@ -42,4 +42,7 @@ require 'smithy/format'
 require 'smithy/package'
 require 'smithy/module_file'
 require 'smithy/description'
+require 'smithy/download_cache'
 require 'smithy/file_operations'
+require 'smithy/formula'
+require 'smithy/formula_command'

data/lib/smithy_version.rb

@@ -36,5 +36,5 @@
 # }}}
 
 module Smithy
-  VERSION = '1.1'
+  VERSION = '1.6'
 end

data/man/man1/smithy.1

@@ -1,60 +1,254 @@
 .\" generated with Ronn/v0.7.3
 .\" http://github.com/rtomayko/ronn/tree/0.7.3
 .
-.TH "SMITHY" "1" "November 2012" "" ""
+.TH "SMITHY" "1" "August 2013" "" ""
 .
 .SH "NAME"
-\fBsmithy\fR \- build, test, and install software with ease\.
+\fBsmithy\fR \- build, test, and install software with ease
 .
 .SH "SYNOPSIS"
 \fBsmithy\fR \fBnew\fR APPLICATION/VERSION/BUILD
 .
 .br
-\fBsmithy\fR \fBedit\fR last
+\fBsmithy\fR \fBedit\fR APPLICATION/VERSION/BUILD
 .
 .br
-\fBsmithy\fR \fBbuild\fR last
+\fBsmithy\fR \fBbuild\fR APPLICATION/VERSION/BUILD
+.
+.br
+\fBsmithy\fR \fBformula install\fR APPLICATION/VERSION/BUILD
 .
 .br
 .
 .SH "DESCRIPTION"
-\fBsmithy\fR will help you get software installed quickly and painlessly\. This tool follows all the same conventions as swtools\. That is, directory structures, support files, and scripts are all in the same locations\. Smithy\'s goal is to make using those conventions easier\.
+\fBsmithy\fR is a command line software installation tool that borrows ideas heavily from the excellent homebrew \fIhttp://brew\.sh/\fR package management system for Mac OS X and SWTools \fIhttp://www\.olcf\.ornl\.gov/center\-projects/swtools/\fR\.
 .
 .P
-This is still a work in progress\. If you have any questions, suggestions, or encounter any errors please send me an email: anthonyd@ornl\.gov
+This is still a work in progress\. If you have any questions, suggestions, or encounter any errors please open an issue on github at \fIhttps://github\.com/AnthonyDiGirolamo/smithy/issues\fR\.
 .
-.SH "COMMANDS"
+.SH "COMMAND LINE HELP"
 For help on all options and commands run \fBsmithy help\fR
 .
-.SH "EXAMPLES"
-This section shows the typical work\-flow for adding a new software build\. Let\'s imagine we want to install petsc 3\.2 using the cray compiler\. We must create a place for the software build, build it, and create a module\.
+.br
+For help on a specific command run \fBsmithy help COMMAND\fR or \fBsmithy help COMMAND SUBCOMMAND\fR
+.
+.SH "SOFTWARE INSTALLATION METHODS"
+There are two ways of installing software using smithy:
+.
+.IP "\(bu" 4
+\fBbuild scripts\fR (similar to swtools)
+.
+.IP "\(bu" 4
+\fBformulas\fR (similar to homebrew)
+.
+.IP "" 0
+.
+.SH "BUILD SCRIPTS"
+Build scripts are shell scripts that live inside of a package\'s PREFIX and execute the steps required to perform the compilation\. A hierarchy might look something like this:
+.
+.IP "" 4
+.
+.nf
+
+/sw/xk6/subversion/1\.6\.17
+`\-\-\- sles11\.1_gnu4\.3\.4
+   `\-\-\- bin
+   `\-\-\- build\-notes
+   `\-\-\- dependencies
+   `\-\-\- include
+   `\-\-\- lib
+   `\-\-\- rebuild
+   `\-\-\- relink
+   `\-\-\- remodule
+   `\-\-\- retest
+   `\-\-\- share
+   `\-\-\- subversion\-1\.6\.17
+.
+.fi
+.
+.IP "" 0
 .
 .P
-Note: all commands, options, and arguments have tab\-completion when using bash or zsh
+The prefix for the above package is \fB/sw/xk6/subversion/1\.6\.17/sles11\.1_gnu4\.3\.4\fR within this directory you can see the usual \fBbin/include/lib/share\fR folders\. Additionally there is the source directory \fBsubversion\-1\.6\.17\fR and three build scripts \fBrebuild\fR, \fBrelink\fR, and \fBremodule\fR\. The \fBrebuild\fR script will compile the software and set the prefix while the remodule script sets up the environment including loading modulefiles\.
 .
-.SS "1\. Creating The Build"
+.SH "FORMULAS"
+The Problem with build scripts is that they are duplicated for every software installation\. This makes installing new software difficult because you will have to go back and look at old ones and copy relevant steps to a new rebuild script\. So much for don\'t repeat yourself!
+.
+.P
+An alternative is to use formulas\. This idea is heavily borrowed from the wonderful homebrew \fIhttp://brew\.sh/\fR package management system for Mac OS X\. With a formula you can specify the modules to load, extra package dependencies, installation steps, and the modulefile in a single file\. The formulas are written in ruby and can be as flexible and dynamic as you like\.
+.
+.P
+A package installed with a formula has a simpler prefix directory structure\. The only addition is the source files under the source directory\.
+.
+.IP "" 4
+.
+.nf
+
+/sw/xk6/subversion/1\.7\.8
+`\-\-\- sles11\.1_gnu4\.3\.4
+   `\-\-\- bin
+   `\-\-\- include
+   `\-\-\- lib
+   `\-\-\- source
+   `\-\-\- share
+.
+.fi
+.
+.IP "" 0
+.
+.SH "NAMING SOFTWARE BUILDS"
 All software builds have the following name format: \fBAPPLICATION/VERSION/BUILD\fR That is three different parts separated by forward slashes \fB/\fR Each part consists of:
 .
-.TP
-\fBAPPLICATION\fR
+.SS "APPLICATION"
 The name using lowercase characters
 .
-.TP
-\fBVERSION\fR
-Numbers with periods\. I recommend that whatever you choose as a version allows it to be lexigraphically sorted from oldest to newest\.
+.SS "VERSION"
+Numbers with periods\. I recommend that whatever you choose as a version allows it to be lexigraphically sorted from oldest to newest\. For example use 2013\-02\-14 instead of Feb14\-2013\.
 .
-.TP
-\fBBUILD\fR
-Build is somewhat different, it consists of the intended operating system and compiler separated by underscores (\fB_\fR)\. It is important to use compiler version numbers that correspond to module versions\. This will allow smithy to auto\-generate modulefiles for software with multiple builds\. As an example:
+.SS "BUILD"
+Build is somewhat different, it consists of the intended operating system and compiler separated by underscores \fB_\fR\. It is important to use compiler version numbers that correspond to module versions\. This will allow smithy to auto\-generate modulefiles for software with multiple builds\.
 .
-.IP
+.P
 \fBsles11\.1_gnu4\.6\.2\fR corresponds to SuSE Linux Enterprise Server 11\.1 and the GNU gcc compiler 4\.6\.2
 .
-.IP
+.P
 \fBcle4\.0_pgi12\.1\.0\fR corresponds to Cray Linux Environment 4\.0 and the PGI 12\.1\.0 compile
 .
 .P
-Let\'s use \fBpetsc/3\.2/cle4\.0_cray8\.0\.1\fR as the name\. This would be petsc designed to run on a cle4\.0 compute node and compiled with the cray cce 8\.0\.1 compiler\. This can be accomplished by running:
+You may also add any other details relevant to the build\. For instance, add python2\.7 to the build name of a python module compiled for python2\.7 and python3\.3 for python3\.3\.
+.
+.SH "INSTALLATIONS WITH FORMULAS"
+For details on writing formulas please see the smithyformula(5) man page\. This section covers only how to install software using existing formulas\.
+.
+.P
+Smithy formula sub\-commands include:
+.
+.TP
+\fBnew\fR
+Create a new formula
+.
+.TP
+\fBlist\fR
+List known formulas
+.
+.TP
+\fBwhich\fR
+Display a formula location
+.
+.TP
+\fBdisplay\fR
+Display a formula
+.
+.TP
+\fBinstall\fR
+Install a package using a formula
+.
+.TP
+\fBcreate_modulefile\fR
+Create a modulefile for a given package
+.
+.P
+Once a formula has been written installing is straightforward using the install command\. For example to install subversion to your software root under the \fBsubversion/1\.7\.8/sles11\.1_gnu4\.3\.4\fR directory you might run:
+.
+.IP "" 4
+.
+.nf
+
+smithy formula install subversion/1\.7\.8/sles11\.1_gnu4\.3\.4
+.
+.fi
+.
+.IP "" 0
+.
+.P
+See \fINAMING SOFTWARE BUILDS\fR for details on naming a software build\. The format of the install sub\-command is: \fBsmithy formula install [command options] APPLICATION | APPLICATION/VERSION | APPLICATION/VERSION/BUILD\fR\. The options and arguments include:
+.
+.TP
+\fB\-\-[no\-]clean\fR
+This will delete all existing files in the target directory before performing the installation\.
+.
+.TP
+\fB\-\-formula\-name\fR
+By default smithy will guess the formula name based on the target directory APPLICATION/VERSION/BUILD argument\. You may wish you install to a location named differently than a formula\. In this case, use \fB\-\-formula\-name\fR to define which formula to use\.
+.
+.TP
+\fBAPPLICATION | APPLICATION/VERSION | APPLICATION/VERSION/BUILD\fR
+This is the destination directory that the software will be installed in\. If you omit the BUILD or VERSION/BUILD directories smithy will try to guess the version based on the formula, and the build based on the operating system and version of gcc available\. If in doubt, specify the full destination\.
+.
+.P
+Assuming we install subversion with the previous command and our software\-root is \fB/sw/xk6/\fR the finished directory structure would look like:
+.
+.IP "" 4
+.
+.nf
+
+/sw/xk6/subversion/1\.7\.8
+`\-\-\- modulefile
+|  `\-\-\- subversion
+|     `\-\-\- 1\.7\.8
+`\-\-\- sles11\.1_gnu4\.3\.4
+   `\-\-\- bin
+   `\-\-\- include
+   `\-\-\- lib
+   `\-\-\- source
+   `\-\-\- share
+.
+.fi
+.
+.IP "" 0
+.
+.P
+A \fBmodulefile\fR folder will be created alongside the \fBsles11\.1_gnu4\.3\.4\fR build directory so that the modulefile can be tested\. To test the modulefile you need to add the modulefile folder to the \fB$MODULEPATH\fR environment variable\. Running any of the following will do this:
+.
+.IP "" 4
+.
+.nf
+
+smithy module use last
+smithy module use subversion/1\.7\.8/sles11\.1_gnu4\.3\.4
+module use /sw/xk6/subversion/1\.7\.8/modulefile/subversion
+.
+.fi
+.
+.IP "" 0
+.
+.P
+\fBlast\fR is an alias to the last software build smithy worked on\. \fBsmithy show last\fR will display the last build you worked on\.
+.
+.P
+Once loaded, you should be able to interact with the new module file as normal using:
+.
+.IP "" 4
+.
+.nf
+
+module avail subversion/1\.7\.8
+module display subversion/1\.7\.8
+module load subversion/1\.7\.8
+.
+.fi
+.
+.IP "" 0
+.
+.P
+You can now deploy the module to make it available to other users\. This should be done once you\'re confident the modulefile is working properly\. To do so run either:
+.
+.IP "" 4
+.
+.nf
+
+smithy module deploy last
+smithy module deploy subversion/1\.7\.8/sles11\.1_gnu4\.3\.4
+.
+.fi
+.
+.IP "" 0
+.
+.SH "INSTALLATIONS WITH BUILD SCRIPTS"
+This section shows the typical work\-flow for adding a new software build using build scripts\. Let\'s imagine we want to install petsc 3\.2 using the cray compiler\. We must create a place for the software build, build it, and create a module\.
+.
+.SS "1\. Creating The Build"
+Let\'s use \fBpetsc/3\.2/cle4\.0_cray8\.0\.1\fR as the name for our new package\. See \fINAMING SOFTWARE BUILDS\fR for details on naming a software build\. This would be petsc designed to run on a cle4\.0 compute node and compiled with the cray cce 8\.0\.1 compiler\. This can be accomplished by running:
 .
 .IP "" 4
 .
@@ -67,13 +261,13 @@ smithy new petsc/3\.2/cle4\.0_cray8\.0\.1
 .IP "" 0
 .
 .P
-You can save yourself some extra time by telling smithy where the tar file for petsc is:
+You can save yourself some extra time by telling smithy where the source tar file for petsc is:
 .
 .IP "" 4
 .
 .nf
 
-smithy new \-t /sw/sources/petsc/3\.2/petsc\-lite\-3\.2\-p7\.tar\.gz petsc/3\.2/cle4\.0_cray8\.0\.1
+smithy new \-t petsc\-3\.2\-p7\.tar\.gz petsc/3\.2/cle4\.0_cray8\.0\.1
 .
 .fi
 .
@@ -83,9 +277,25 @@ smithy new \-t /sw/sources/petsc/3\.2/petsc\-lite\-3\.2\-p7\.tar\.gz petsc/3\.2/
 When using the \fB\-t\fR, \fB\-\-tarfile=\fR option smithy will extract the given archive to the \fBsource\fR directory\. For the petsc example above this would be \fB/sw/xk6/petsc/3\.2/cle4\.0_cray8\.0\.1/source\fR
 .
 .P
-If this is a brand new piece of software add the \fB\-\-web\-description\fR switch\. This will create the application description files too\.
+The \fB\-t\fR option can also download an archive from a given URL\. The archive is saved along side the source directory\. As an example:
+.
+.IP "" 4
+.
+.nf
+
+smithy new \-t http://ftp\.mcs\.anl\.gov/pub/petsc/release\-snapshots/petsc\-3\.2\-p7\.tar\.gz petsc/3\.2/cle4\.0_cray8\.0\.1
+.
+.fi
+.
+.IP "" 0
+.
+.P
+This command will download petsc\-3\.2\-p7\.tar\.gz, save it to \fB/sw/xk6/petsc/3\.2/cle4\.0_cray8\.0\.1/petsc\-3\.2\-p7\.tar\.gz\fR and extract it to \fB/sw/xk6/petsc/3\.2/cle4\.0_cray8\.0\.1/source\fR\. This feature requires the \fBcurl\fR command to work properly\.
+.
+.P
+Additionally, if this is a brand new piece of software add the \fB\-\-web\-description\fR switch\. This will create the application description files too\.
 .
-.SS "2\. Editing and Building the Software:"
+.SS "2\. Editing and Building the Software"
 Once you have created the build you may need to update the build (\fBrebuild\fR) and environment (\fBremodule\fR) scripts before building the software\. Both files live within the software prefix\. For our example it is located in \fB/sw/xk6/petsc/3\.2/cle4\.0_cray8\.0\.1\fR\. You can edit this and other related files using the \fBedit\fR command:
 .
 .IP "" 4
@@ -130,7 +340,7 @@ smithy build last
 .P
 The results of the run will be shown on the screen and simultaneously logged to \fBbuild\.log\fR withing the software prefix folder\.
 .
-.SS "3\. Create and edit a modulefile:"
+.SS "3\. Create and edit a modulefile"
 This step is best done after all builds for a particular application have been created\. When you create a new software build a modulefile is created too\. For our petsc install it lives in: \fB/sw/xk6/petsc/3\.2/modulefile\fR All builds of a particular application share a single modulefile\.
 .
 .P
@@ -207,7 +417,7 @@ smithy module deploy last
 For out petsc example, this command will copy \fB/sw/xk6/petsc/3\.2/modulefile/petsc/3\.2\fR to \fB/sw/xk6/modulefiles/petsc/3\.2\fR
 .
 .SS "4\. Website Description"
-If this is a new application you will need to add some information to the description file\. For petsc this lives in: \fB/sw/xk6/petsc/description\fR This is an html formatted file\. Alternatively, it can live in \fB/sw/xk6/petsc/description\.markdown\fR this file is in markdown format and is a bit simpler to write than html\. See http://kramdown\.rubyforge\.org/quickref\.html for more information on markdown syntax\. If both files exist, the markdown file takes precedence\.
+If this is a new application you will need to add some information to the description file\. For petsc this lives in: \fB/sw/xk6/petsc/description\fR This is an html formatted file\. Alternatively, it can live in \fB/sw/xk6/petsc/description\.markdown\fR this file is in markdown format and is a bit simpler to write than html\. See \fIhttp://kramdown\.rubyforge\.org/quickref\.html\fR for more information on markdown syntax\. If both files exist, the markdown file takes precedence\.
 .
 .P
 If the description file is missing you can generate one by running:
@@ -235,28 +445,5 @@ smithy publish petsc
 .
 .IP "" 0
 .
-.SH "LICENSE"
-Smithy is freely available under the terms of the BSD license given below\.
-.
-.P
-Copyright (c) 2012\. UT\-BATTELLE, LLC\. All rights reserved\.
-.
-.P
-Produced at the National Center for Computational Sciences in Oak Ridge National Laboratory\.
-.
-.P
-Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-.
-.IP "\(bu" 4
-Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer\.
-.
-.IP "\(bu" 4
-Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution\.
-.
-.IP "\(bu" 4
-Neither the name of the UT\-BATTELLE nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission\.
-.
-.IP "" 0
-.
-.P
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED\. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE\.
+.SH "SEE ALSO"
+smithyformula(5)

data/smithy.rdoc

@@ -1,6 +1,6 @@
 == smithy - Smithy will help you build, test, and install software with ease.
 
-v1.1
+v1.6
 
 === Global Options
 === --arch NAME
@@ -53,7 +53,7 @@ The root level directory for published web files
 
 
 === --[no-]colors
-Disable or enable color output (default: enabled)
+Disable or enable color output
 
 
 
@@ -72,16 +72,21 @@ Show this message
 
 
 
-=== --version
+=== -v|--verbose
+Be more verbose
+
 
 
+=== --version
+Display the program version
+
 
 
 === Commands
 ==== Command: <tt>build  PATH</tt>
 Build software
 
-The software to build may be either the absolute path or the full name of the software. The full name includes version numbers and build names using the format: NAME/VERSION/BUILD.
+The software to build may be either the absolute path or the full name of the software. The full name includes version numbers and build names using the format: APPLICATION/VERSION/BUILD.
 ===== Options
 ===== --log-name FILE
 
@@ -127,6 +132,84 @@ Split editing window with requested file and the environment (remodule) file
 
 
 
+==== Command: <tt>formula </tt>
+Install software from predefined formulas
+
+
+===== Options
+===== -d|--directories PATH
+
+Specify one or more additional formula directories separated with commas
+
+[Default Value] None
+
+
+===== Commands
+====== Command: <tt>create_modulefile  APPLICATION | APPLICATION/VERSION | APPLICATION/VERSION/BUILD</tt>
+Create a modulefile for a given package
+
+
+======= Options
+======= -f|--formula-name NAME
+
+Formula name
+
+[Default Value] None
+
+
+====== Command: <tt>display  FORMULA</tt>
+Display a formula
+
+
+====== Command: <tt>install  APPLICATION | APPLICATION/VERSION | APPLICATION/VERSION/BUILD</tt>
+Install a package using a formula
+
+
+======= Options
+======= -f|--formula-name NAME
+
+Formula name
+
+[Default Value] None
+
+
+======= -c|--[no-]clean
+Clean exiting install prefix
+
+
+
+======= -m|--modulefile
+Create modulefiles as well
+
+
+
+====== Command: <tt>list </tt>
+List known formulas
+
+
+====== Command: <tt>new  URL</tt>
+Create a new formula
+
+
+======= Options
+======= -h|--homepage URL
+
+Formula homepage
+
+[Default Value] None
+
+
+======= -n|--name NAME
+
+Formula name
+
+[Default Value] None
+
+
+====== Command: <tt>which  FORMULA</tt>
+Display a formula location
+
+
 ==== Command: <tt>help  command</tt>
 Shows a list of commands or help for one command
 
@@ -167,9 +250,9 @@ Run the proper module command to add a package's modulefile to the MODULEPATH. T
 ==== Command: <tt>new  NAME</tt>
 Generate a new build and all necessary files
 
-The new command will create all necessary files needed to add a new software package. Some care should be given to naming new packages. Some considerations are package names, version numbers, and build names. New package names should be of the format NAME/VERSION/BUILD
+The new command will create all necessary files needed to add a new software package. Some care should be given to naming new packages. Some considerations are package names, version numbers, and build names. New package names should be of the format APPLICATION/VERSION/BUILD
 
-- NAME of the package should be all lower case and one word. If multiple words are necessary separate them with dashes '-'.
+- APPLICATION of the package should be all lower case and one word. If multiple words are necessary separate them with dashes '-'.
 
 - VERSION numbers should be standard numbers separated by periods. If another format is necessary ensure that the numbers can be lexigraphically sorted in order of oldest release to newest.
 
@@ -180,7 +263,14 @@ EXAMPLES:
 silo/4.8/sles11.1_gnu4.5.3
 fftw/3.2.2/cle4.0_pgi11.10.0
 ===== Options
-===== -t|--tarball FILE|URL
+===== -e|--existing-scripts PATH
+
+Use an existing software's build scripts
+
+[Default Value] None
+
+
+===== -t|--tarball|--tarfile FILE|URL
 
 Provide a tarball to unpack, either a file or URL (optional)
 
@@ -243,6 +333,16 @@ Display internal smithy values
 List all architectures know to smithy.
 
 
+======= Options
+======= -a|--all
+list all architectures
+
+
+
+====== Command: <tt>example_config </tt>
+Display an example config file.
+
+
 ====== Command: <tt>last </tt>
 Display the package name used in the last smithy command. This is stored in the '~/.smithyrc' file.