omnibus 2.0.2 → 3.0.0

data/docs/Building on OSX.md

@@ -0,0 +1,66 @@
+Building Omnibus Packages for Mac OSX
+=====================================
+This document details the steps and configurables for building omnibus packages
+on Mac OSX. Unlike Linux-based systems, the process for building a `.pkg` and
+a `.dmg` requires some manual preparation of assets.
+
+
+Building a .pkg
+---------------
+In Mac OSX, a `.pkg` is a special file that is read by Installer.app that
+contains the set of instructions for installating a piece of software on a
+target system.
+
+### Requirements
+The omnibus generator will create some stock files in `files/mac_pkg`:
+
+- `background.png` - the background image for the installer. We recommend this
+image has a light background color (otherwise, the text will be difficult to
+read).
+- `license.html` - the full HTML document for the license
+- `welcome.html` - the full HTML document for the welcome screen
+
+You should use these stock files and templates as a starting point for building
+your custom pkg.
+
+
+Building a .dmg
+---------------
+In Mac OSX, a `.dmg` is a compressed wrapper around a collection of resources,
+often including a `.pkg`. The possibilities for creating and customizing a DMG
+are endless, but Omnibus provides a "basic" starter case that will generate a
+pretty DMG that contains the `.pkg` file it creates.
+
+The following tunables are available:
+
+- `dmg_window_bounds` - the starting and ending (x,y) coordinates for the opened
+DMG
+- `dmg_pkg_position` - the (x,y) coordinate for the `.pkg` file inside the
+opened DMG window
+
+### Requirements
+The omnibus generator will create some stock files in `files/mac_dmg`:
+
+- `background.png` - the background image to use for the DMG. We recommend using
+a high-resolution image that is slightly larger than the final length of your
+window (as determined by the `dmg_window_bounds`)
+- `icon.png` - a 1024x1024 @ 300 icon to use for the DMG. We will automatically
+create an icns and scale to smaller sizes
+
+You should use these stock files and templates as a starting point for building
+your custom dmg.
+
+### Disabling dmg building
+DMG creation is enabled by default, but you can disable DMG creation by setting
+the `build_dmg` omnibus configuration option to false:
+
+```ruby
+# omnibus.rb
+build_dmg false
+```
+
+And run as you normally would:
+
+```bash
+$ ./bin/omnibus build project <name>
+```

data/docs/omnibus-build-cache.md

@@ -0,0 +1,75 @@
+# The Omnibus Build Cache #
+
+The omnibus project includes a project build caching mechanism that
+reduces the time it takes to rebuild a project when only a few
+components need to be rebuilt.
+
+The cache uses git to snapshot the project tree after each software
+component build. The entire contents of the project's `install_path`
+is included in the snaphot.
+
+When rebuilding, omnibus walks the linearized component list and
+replays the cache until the first item is encountered that needs to be
+rebuilt. Items are rebuilt if the code changes (different version or
+different git SHA) or if the build instructions change (edit to
+config/software/$COMPONENT).
+
+## Location of Cache Data ##
+
+The default location of the cache (which is just a bare git
+repository) is
+`/var/cache/omnibus/cache/install_path/$INSTALL_PATH`. You can
+customize the location of the cache in the `omnibus.rb` config file
+using the key `install_path_cache_dir`. For example:
+
+    install_path_cache_dir "/opt/ominbus-caches"
+
+## How It Works ##
+
+Omnibus begins by loading all projects (see
+`lib/omnibus.rb#expand_software`). The dependencies of each project
+are loaded using `recursively_load_dependency` to capture the
+transitive dependencies of a given project dependency. The result is
+that each project has a list of components in "install order" such
+that all dependencies come before the things that depend on
+them. Components are de-duplicated on the way in and not added if
+already present (which will occur when two components share a common
+dependency).
+
+The actual build order is determined by `library.rb#build_order` which
+does a small reordering to ensure that components that are explicitly
+listed in the project file come last. Since the first cache miss
+causes the system to rebuild everything further to the right in the
+component list, you want to have your most frequently changed
+components last to minimize rebuild churn.
+
+Lightweight git tags are used as cache keys. After building a
+component, a tag name is computed in `install_path_cache.rb#tag`. The
+tag name has the format `#{name}-#{version}-#{digest}` where name and
+version map to the component that was just built and digest is a
+SHA256 value. The digest is computed by string-ifying the name/version
+pairs of all components built prior to this component (keeping the
+order of course) and prepending the entire contents of the component's
+`config/software/$NAME.rb` file. Important aspects of the cache key
+scheme:
+
+* A change in the order or in the name/version of component built
+  before the component will invalidate the cache key.
+* A change in the component's software config (build instructions)
+  invalidates the cache. This is done conservatively; the entire file
+  contents are included in the digest computation so even a whitespace
+  change will invalidate the key.
+* A change in the version of the component itself will invalidate the
+  key.
+
+You can inspect the cache keys like this:
+
+    git --git-dir=$CACHE_PATH tag -l
+
+You can manually remove a cache entry using `git tag --delete
+$TAG`. In theory, you can share the cache among build slaves using
+`git clone/fetch`.
+
+When running a build vm on OS X, note that you will likely run into
+trouble if you attempt to locate your cache on your local
+filesystem if it is not case sensitive.

data/lib/omnibus.rb

@@ -32,6 +32,7 @@ require 'omnibus/project'
 require 'omnibus/fetchers'
 require 'omnibus/health_check'
 require 'omnibus/build_version'
+require 'omnibus/sugar'
 require 'omnibus/overrides'
 require 'omnibus/version'
 
@@ -40,6 +41,12 @@ require 'pathname'
 module Omnibus
   DEFAULT_CONFIG_FILENAME = 'omnibus.rb'.freeze
 
+  module Packager
+    autoload :Base,   'omnibus/packagers/base'
+    autoload :MacDmg, 'omnibus/packagers/mac_dmg'
+    autoload :MacPkg, 'omnibus/packagers/mac_pkg'
+  end
+
   # Configure Omnibus.
   #
   # After this has been called, the {Omnibus::Config} object is
@@ -83,7 +90,6 @@ module Omnibus
   def self.process_configuration
     Config.validate
     process_dsl_files
-    generate_extra_rake_tasks
   end
 
   # All {Omnibus::Project} instances that have been created.
@@ -222,17 +228,6 @@ module Omnibus
     expand_software(overrides, final_software_map)
   end
 
-  # Creates some additional Rake tasks beyond those generated in the
-  # process of reading in the DSL files.
-  #
-  # @return [void]
-  #
-  # @todo Not so sure I like how this is being done, but at least it
-  #   isolates the Rake stuff.
-  def self.generate_extra_rake_tasks
-    require 'omnibus/clean_tasks'
-  end
-
   # Return a list of all the Ruby files (i.e., those with an "rb"
   # extension) in the given directory
   #
@@ -301,11 +296,12 @@ module Omnibus
     end
 
     dep_software = Omnibus::Software.load(dep_file, project, overrides)
-    project.library.component_added(dep_software)
 
     # load any transitive deps for the component into the library also
     dep_software.dependencies.each do |dep|
       recursively_load_dependency(dep, project, overrides, software_map)
     end
+
+    project.library.component_added(dep_software)
   end
 end

data/lib/omnibus/artifact.rb

@@ -97,9 +97,7 @@ module Omnibus
         'version' => build_version,
         'basename' => File.basename(path),
         'md5' => md5,
-        'sha1' => sha1,
         'sha256' => sha256,
-        'sha512' => sha512,
       }
     end
 
@@ -131,21 +129,11 @@ module Omnibus
       @md5 ||= digest(Digest::MD5)
     end
 
-    # @return [String] hex encoded SHA1 of the package
-    def sha1
-      @sha1 ||= digest(Digest::SHA1)
-    end
-
-    # @return [String] hex encoded SHA-256 of the package
+    # @return [String] hex encoded SHA2-256 of the package
     def sha256
       @sha256 ||= digest(Digest::SHA256)
     end
 
-    # @return [String] hex encoded SHA-512 of the package
-    def sha512
-      @sha512 ||= digest(Digest::SHA512)
-    end
-
     private
 
     def digest(digest_class)

data/lib/omnibus/builder.rb

@@ -17,6 +17,7 @@
 
 require 'forwardable'
 require 'omnibus/exceptions'
+require 'omnibus/install_path_cache'
 require 'ostruct'
 
 module Omnibus

data/lib/omnibus/cli/application.rb

@@ -77,9 +77,9 @@ module Omnibus
         template(File.join('software', 'ruby-example.rb.erb'), File.join(config_software, 'ruby-example.rb'), opts)
 
         # Kitchen build environment
-        template(File.join('.kitchen.local.yml.erb'), File.join(target, '.kitchen.local.yml'), opts)
-        template(File.join('.kitchen.yml.erb'), File.join(target, '.kitchen.yml'), opts)
-        template(File.join('Berksfile.erb'), File.join(target, 'Berksfile'), opts)
+        template('.kitchen.local.yml.erb', File.join(target, '.kitchen.local.yml'), opts)
+        template('.kitchen.yml.erb', File.join(target, '.kitchen.yml'), opts)
+        template('Berksfile.erb', File.join(target, 'Berksfile'), opts)
 
         # render out stub packge scripts
         %w(makeselfinst preinst prerm postinst postrm).each do |package_script|
@@ -88,8 +88,21 @@ module Omnibus
           # render the package script
           template(template_path, script_path, opts)
           # ensure the package script is executable
-          FileUtils.chmod(0755, script_path)
+          chmod(script_path, 0755)
         end
+
+        # mac pkg stock assets
+        mac_pkg_resources = File.expand_path("#{target}/files/mac_pkg/Resources")
+        empty_directory(mac_pkg_resources)
+        template(File.join('mac_pkg', 'license.html.erb'), File.join("#{mac_pkg_resources}", 'license.html'), opts)
+        template(File.join('mac_pkg', 'welcome.html.erb'), File.join("#{mac_pkg_resources}", 'welcome.html'), opts)
+        copy_file(File.join('mac_pkg', 'background.png'), File.join("#{mac_pkg_resources}", 'background.png'))
+
+        # mac dmg stock assets
+        mac_dmg_resources = File.expand_path("#{target}/files/mac_dmg/Resources")
+        empty_directory(mac_dmg_resources)
+        copy_file(File.join('mac_dmg', 'background.png'), File.join("#{mac_dmg_resources}", 'background.png'))
+        copy_file(File.join('mac_dmg', 'icon.png'), File.join("#{mac_dmg_resources}", 'icon.png'))
       end
 
       desc 'version', 'Display version information'

data/lib/omnibus/cli/build.rb

@@ -38,24 +38,26 @@ module Omnibus
       desc 'project PROJECT', 'Build the given Omnibus project'
       def project(project_name)
         project = load_project!(project_name)
-        project_task_name = "projects:#{project.name}"
 
         unless options[:timestamp]
           say("I won't append a timestamp to the version identifier.", :yellow)
         end
         say("Building #{project.name} #{project.build_version}", :green)
 
-        Rake::Task[project_task_name].invoke
+        project.build_me
       end
 
       desc 'software PROJECT SOFTWARE', 'Build the given software component'
       def software(project_name, software_name)
         project = load_project!(project_name)
-        software_task_name = "projects:#{project.name}:software:#{software_name}"
+
+        software = project.library.components.find do |s|
+          s.name == software_name
+        end
 
         say("Building #{software_name} for #{project.name} project", :green)
 
-        Rake::Task[software_task_name].invoke
+        software.build_me
       end
     end
   end

data/lib/omnibus/config.rb

@@ -40,6 +40,15 @@ module Omnibus
     #   @return [String]
     default :cache_dir, '/var/cache/omnibus/cache'
 
+    # @!attribute [rw] install_path_cache_dir
+    #   The absolute path to the directory on the virtual machine where
+    #   install paths will be progressively cached.
+    #
+    #   Defaults to `"/var/cache/omnibus/cache/install_path"`.
+    #
+    #   @return [String]
+    default :install_path_cache_dir, '/var/cache/omnibus/cache/install_path'
+
     # @!attribute [rw] source_dir
     #   The absolute path to the directory on the virtual machine where
     #   source code will be downloaded.
@@ -118,6 +127,30 @@ module Omnibus
 
     # @!endgroup
 
+    # @!group DMG configuration options
+
+    # @!attribute [rw] build_dmg
+    #   Package OSX pkg files inside a DMG
+    #
+    # @return [Boolean]
+    default :build_dmg, true
+
+    # @!attribute [rw] dmg_window_bounds
+    #   Indicate the starting x,y and ending x,y positions for the created DMG
+    #   window.
+    #
+    # @return [String]
+    default :dmg_window_bounds, '100, 100, 750, 600'
+
+    # @!attribute [rw] dmg_pkg_position
+    #   Indicate the starting x,y position where the .pkg file should live in
+    #   the DMG window.
+    #
+    # @return [String]
+    default :dmg_pkg_position, '535, 50'
+
+    # @!endgroup
+
     # @!group S3 Caching Configuration Parameters
 
     # @!attribute [rw] use_s3_caching

data/lib/omnibus/exceptions.rb

@@ -1,4 +1,24 @@
 module Omnibus
+  class AbstractMethod < RuntimeError
+    def initialize(signature)
+      @signature = signature
+    end
+
+    def to_s
+      "'#{@signature}' is an abstract method and must be overridden!"
+    end
+  end
+
+  class MissingAsset < RuntimeError
+    def initialize(path)
+      @path = path
+    end
+
+    def to_s
+      "Missing asset! '#{@path}' is not present on disk."
+    end
+  end
+
   class InvalidS3Configuration < RuntimeError
     def initialize(s3_bucket, s3_access_key, s3_secret_key)
       @s3_bucket, @s3_access_key, @s3_secret_key = s3_bucket, s3_access_key, s3_secret_key
@@ -176,20 +196,6 @@ module Omnibus
     end
   end
 
-  class MissingMacPkgResource < StandardError
-    def initialize(missing_file_paths)
-      @missing_file_paths = missing_file_paths
-    end
-
-    def to_s
-      <<-E
-      Your omnibus repo is missing the following files required to build Mac
-      packages:
-      #{@missing_file_paths.map { |p| "* #{p}" }.join("\n      ")}
-E
-    end
-  end
-
   class UnresolvableGitReference < RuntimeError
   end
 end

data/lib/omnibus/health_check.rb

@@ -66,6 +66,7 @@ module Omnibus
       /libcrypto.so/,
       /libcurses\.so/,
       /libdoor\.so/,
+      /libgcc_s\.so\.1/,
       /libgen\.so/,
       /libmd5\.so/,
       /libmd\.so/,
@@ -122,6 +123,7 @@ module Omnibus
       /libncurses\.5\.4\.dylib/,
       /libiconv/,
       /libstdc\+\+\.6\.dylib/,
+      /libc\+\+\.1\.dylib/,
     ]
 
     FREEBSD_WHITELIST_LIBS = [

data/lib/omnibus/install_path_cache.rb

@@ -0,0 +1,106 @@
+#
+# Copyright:: Copyright (c) 2014 Chef Software, Inc.
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+require 'omnibus/util'
+require 'omnibus/config'
+require 'fileutils'
+require 'digest'
+
+module Omnibus
+  class InstallPathCache
+    include Util
+
+    def initialize(install_path, software)
+      @install_path = install_path
+      @software = software
+    end
+
+    # The path to the full install_path cache for the project
+    def cache_path
+      File.join(Omnibus::Config.install_path_cache_dir, @install_path)
+    end
+
+    # Whether the cache_path above exists
+    def cache_path_exists?
+      File.directory?(cache_path)
+    end
+
+    # Creates the full path if it does not exist already
+    def create_cache_path
+      FileUtils.mkdir_p(File.dirname(cache_path))
+      shellout!("git --git-dir=#{cache_path} init -q") unless cache_path_exists?
+      true
+    end
+
+    # Computes the tag for this cache entry
+    def tag
+      name = @software.name
+      version = @software.version
+
+      # Accumulate an array of all the software projects that come before
+      # the name and version we are tagging. So if you have
+      #
+      # build_order = [ 1, 2, 3, 4, 5 ]
+      #
+      # And we are tagging 3, you would get dep_list = [ 1, 2 ]
+      dep_list = @software.project.library.build_order.take_while do |dep|
+        if dep.name == @software.name && dep.version == @software.version
+          false
+        else
+          true
+        end
+      end
+      dep_string = dep_list.map { |i| "#{i.name}-#{i.version}" }.join('-')
+      # digest the content of the software's config so that changes to
+      # build params invalidate cache.
+      dep_string = IO.read(@software.source_config) + dep_string
+      digest = Digest::SHA256.hexdigest(dep_string)
+      "#{name}-#{version}-#{digest}"
+    end
+
+    # Create an incremental install path cache for the software step
+    def incremental
+      create_cache_path
+      shellout!("git --git-dir=#{cache_path} --work-tree=#{@install_path} add -A -f")
+      begin
+        shellout!("git --git-dir=#{cache_path} --work-tree=#{@install_path} commit -q -m 'Backup of #{tag}'")
+      rescue Mixlib::ShellOut::ShellCommandFailed => e
+        if e.message !~ /nothing to commit/
+          raise
+        end
+      end
+      shellout!("git --git-dir=#{cache_path} --work-tree=#{@install_path} tag -f '#{tag}'")
+    end
+
+    def restore
+      create_cache_path
+      cmd = shellout("git --git-dir=#{cache_path} --work-tree=#{@install_path} tag -l #{tag}")
+
+      restore_me = false
+      cmd.stdout.each_line do |line|
+        restore_me = true if tag == line.chomp
+      end
+
+      if restore_me
+        shellout!("git --git-dir=#{cache_path} --work-tree=#{@install_path} checkout -f '#{tag}'")
+        true
+      else
+        false
+      end
+    end
+  end
+end