librarian 0.0.14 → 0.0.15

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+## 0.0.15
+
+* Rewrite the README.
+
+* \#44, \#49. Better updating of cached git sources without using local merges.
+
 ## 0.0.14
 
 * \#39 Fixes a regression induced by using `git reset --hard SHA`.

data/README.md

@@ -1,42 +1,160 @@
 Librarian
 =========
 
-A tool to resolve recursively a set of specifications and fetch and install the fully resolved specifications.
+Librarian is a framework for writing bundlers, which are tools that resolve,
+fetch, install, and isolate a project's dependencies, in Ruby.
 
-Librarian::Mock
----------------
+Librarian ships with Librarian-Chef, which is a bundler for your Chef-based
+infrastructure repositories. In the future, Librarian-Chef will be a separate
+project.
 
-An adapter for Librarian for unit testing the general features.
-The mock source is in-process and in-memory and does not touch the filesystem or the network.
+A bundler written with Librarian will expect you to provide a specfile listing
+your project's declared dependencies, including any version constraints and
+including the upstream sources for finding them. Librarian can resolve the spec,
+write a lockfile listing the full resolution, fetch the resolved dependencies,
+install them, and isolate them in your project. This is what Bundler does for
+projects that depend on Ruby gems.
 
-Librarian::Chef
+Librarian-Chef
 ---------------
 
-An adapter for Librarian applying to Chef cookbooks in a Chef Repository. When used with Chef, Librarian is really for pulling in the 50 or so finished third-party cookbooks that you're using, not the 1 or 2 cookbooks you're actively working on.
+Librarian-Chef is a bundler for infrastructure repositories using Chef. You can
+use Librarian-Chef to resolve your infrastructure's cookbook dependencies and
+fetch them and install them into your infrastructure.
 
-## Install librarian:
+Librarian-Chef is for resolving and fetching third-party, publicly-released
+cookbooks, and installing them into your infrastructure repository. It is not
+for dealing with the cookbooks you're actively working on within your
+infrastructure repository.
 
-    $ gem install librarian
+Librarian-Chef *takes over* your `cookbooks/` directory and manages it for you
+based on your `Cheffile`. Your `Cheffile` becomes the authoritative source for
+what cookbooks your infrastructure repository depends on. You should not modify
+the contents of your `cookbooks/` directory when using Librarian-Chef. If you
+have custom cookbooks specific to your infrastructure repository that you need,
+those should go in your `site-cookbooks/` directory.
+
+### The Cheffile
+
+Every infrastruture repository that uses Librarian-Chef will have a file named
+`Cheffile` in the root directory of the repository. The full specification for
+which third-party, publicly-rleased cookbooks your infrastructure repository
+depends will go here.
+
+Here's an example `Cheffile`:
+
+    site "http://community.opscode.com/api/v1"
+
+    cookbook "ntp"
+    cookbook "timezone", "0.0.1"
+
+    cookbook "rvm",
+      :git => "https://github.com/fnichol/chef-rvm",
+      :ref => "v0.7.1"
+
+    cookbook "cloudera",
+      :path => "vendor/cookbooks/cloudera-cookbook"
+
+Here's how it works:
+
+We start off by declaring the *default source* for this `Cheffile`.
+
+    site "http://community.opscode.com/api/v1"
+
+This default source in this example is the Opscode Community Site API. This is
+most likely what you will want for your default source. However, you can
+certainly set up your own API-compatible HTTP endpoint if you want more control.
+
+Any time we declare a cookbook dependency without also declaring a source for
+that cookbook dependency, Librarian-Chef assumes we want it to look for that
+cookbook in the default source.
+
+Any time we declare a cookbook dependency that has subsidiary cookbook
+dependencies of its own, Librarian-Chef assumes we want it to look for the
+subsidiary cookbook dependencies in the default source.
+
+    cookbook "ntp"
+
+Our infrastructure repository depends on the `ntp` cookbook from the default
+source. Any version of the `ntp` cookbook will fulfill our requirements.
+
+    cookbook "timezone", "0.0.1"
+
+Our infrastructure repository depends on the `timezone` cookbook from the
+default source. But only version `0.0.1` of that cookbook will do.
+
+    cookbook "rvm",
+      :git => "https://github.com/fnichol/chef-rvm",
+      :ref => "v0.7.1"
+
+Our infrastructure repository depends on the `rvm` cookbook, but not the one
+from the default source. Instead, the cookbook is to be fetched from the
+specified Git repository and from the specified Git tag only.
+
+We do not have to use a `:ref =>`. If we do not, then Librarian-Chef will assume
+we meant the branch `master`. (In the future, this will be changed to whatever
+branch is the default branch according to the Git remote, which may not be
+`master`.)
+
+If we use a `:ref =>`, we can use anything that Git will recognize as a ref.
+This includes any branch name, tag name, SHA, or SHA unique prefix. If we use a
+branch, we can later ask Librarian-Chef to update the cookbook by fetching the
+most recent version of the cookbook from that same branch.
+
+The Git source also supports a `:path =>` option. If we use the path option,
+Librarian-Chef will navigate down into the Git repository and only use the
+specified subdirectory. Many people have the havit of having a single repository
+with many cookbooks in it. If we need a cookbook from such a repository, we can
+use the `:path =>` option here to help Librarian-Chef drill down and find the
+cookbook subdirectory.
 
+    cookbook "cloudera",
+      :path => "vendor/cookbooks/cloudera-cookbook"
 
-__Make sure your cookbooks directory is gitignored__
+Our infrastructure repository depends on the `cloudera` cookbook, which we have
+downloaded and copied into our repository. In this example, `vendor/cookbooks/`
+is only for use with Librarian-Chef. This directory should not appear in the
+`.chef/knife.rb`. Librarian-Chef will, instead, copy this cookbook from where
+we vendored it in our repository into the `cookbooks/` directory for us.
+
+The `:path =>` source won't be confused with the `:git =>` source's `:path =>`
+option.
+
+### How to Use
+
+Install librarian-chef:
+
+    $ gem install librarian
+
+Prepare your infrastructure repository:
 
     $ cd ~/path/to/chef-repo
-    $ git rm -r cookbooks # if the directory is present
+    $ git rm -r cookbooks
     $ echo cookbooks >> .gitignore
     $ echo tmp >> .gitignore
 
-Note that librarian *takes over* your cookbooks directory
-  and manages it for you based on your Cheffile. Your
-  Cheffile becomes the authoritative source for what
-  cookbooks you have, rather than the directories in your
-  cookbooks directory.
+Librarian-Chef takes over your `cookbooks/` directory, and will always reinstall
+the cookbooks listed the `Cheffile.lock` into your `cookbooks/` directory. Hence
+you do not need your `cookbooks/` directory to be tracked in Git. If you
+nevertheless want your `cookbooks/` directory to be tracked in Git, simple don't
+`.gitignore` the directory.
 
-__Make a Cheffile__
+If you are manually tracking/vendoring outside cookbooks within the repository,
+put them in another directory such as `vendor/cookbooks/` and use the `:path =>`
+source when declaring these cookbooks in your `Cheffile`. Most people will
+typically not be manually tracking/vendoring outside cookbooks.
+
+Librarian-Chef uses your `tmp/` directory for tempfiles and caches. You do not
+need to track this directory in Git.
+
+Make a Cheffile:
 
     $ librarian-chef init
 
-__Add dependencies and their sources to Cheffile__
+This creates an empty `Cheffile` with the Opscode Community Site API as the
+default source.
+
+Add dependencies and their sources to the `Cheffile`:
 
     $ cat Cheffile
         site 'http://community.opscode.com/api/v1'
@@ -45,17 +163,41 @@ __Add dependencies and their sources to Cheffile__
         cookbook 'rvm',
           :git => 'https://github.com/fnichol/chef-rvm',
           :ref => 'v0.7.1'
+        cookbook 'cloudera',
+          :path => 'vendor/cookbooks/cloudera-cookbook'
 
-__install dependencies into ./cookbooks__
+This is the same `Cheffile` we saw above.
 
     $ librarian-chef install [--clean] [--verbose]
 
-__Check your Cheffile.lock into version control__
+This command looks at each `cookbook` declaration and fetches the cookbook from
+the source specified for that cookbook, or from the default source if none is
+provided for that cookbook.
+
+Each cookbook is inspected and its dependencies determined, and each dependency
+is also fetched. For example, if you declare `cookbook 'nagios'`, and that
+cookbook depends on other cookbooks such as `'php'`, then those other cookbooks
+including `'php'` will be fetched. This goes all the way down the chain of
+dependencies.
+
+This command writes the complete resolution into `Cheffile.lock`.
 
+This command then copies all of the fetched cookbooks into your `cookbooks/`
+directory, overwriting whatever was there before. You can then use `knife
+cookbook upload -all` to upload the cookbooks to your chef-server, if you are
+using the client-server model.
+
+Check your `Cheffile` and `Cheffile.lock` into version control:
+
+    $ git add Cheffile
     $ git add Cheffile.lock
     $ git commit -m "I want these particular versions of these particular cookbooks from these particular."
 
-__Update your cheffile with new/changed/removed constraints/sources/dependencies__
+Make sure you check your Cheffile.lock into version control. This will ensure
+dependencies do not need to be resolved every run, greatly reducing dependency
+resolution time.
+
+Update your cheffile with new/changed/removed constraints/sources/dependencies:
 
     $ cat Cheffile
         site 'http://community.opscode.com/api/v1'
@@ -72,43 +214,56 @@ __Update your cheffile with new/changed/removed constraints/sources/dependencies
     $ git add Cheffile.lock
     $ git commit -m "I also want these additional cookbooks."
 
-__Update the version of a dependency__
+Update the version of a dependency:
 
     $ librarian-chef update ntp timezone monit [--verbose]
     $ git diff Cheffile.lock
     $ git add Cheffile.lock
     $ git commit -m "I want updated versions of these cookbooks."
 
-__Push your changes to the git repository__
+Push your changes to the git repository:
 
     $ git push origin master
 
-__Upload the cookbooks to your chef-server__
+Upload the cookbooks to your chef-server:
 
     $ knife cookbook upload --all
 
-You should `.gitignore` your `./cookbooks` directory.
-If you are manually tracking/vendoring outside cookbooks within the repository,
-  put them in another directory such as `./cookbooks-sources` and use the `:path` source.
-  You should typically not need to do this.
+### Knife Integration
+
+You can integrate your `knife.rb` with Librarian-Chef.
 
-You can integrate your `knife.rb` with Librarian. Stick the following in your `knife.rb`:
+Stick the following in your `knife.rb`:
 
     require 'librarian/chef/integration/knife'
-    cookbook_path Librarian::Chef.install_path, "chef-repo/site-cookbooks"
+    cookbook_path Librarian::Chef.install_path,
+                  "/path/to/chef-repo/site-cookbooks"
+
+In the above, do *not* to include the path to your `cookbooks/` directory. If
+you have additional cookbooks directories in your chef-repo that you use for
+vendored cookbooks (where you use the `:path =>` source in your `Cheffile`),
+make sure *not* to include the paths to those additional cookbooks directories
+either.
+
+You still need to include your `site-cookbooks/` directory in the above list.
+
+What this integration does is whenever you use any `knife` command, it will:
+
+* Enforce that your `Cheffile` and `Cheffile.lock` are in sync
+* Install the resolved cookbooks to a temporary directory
+* Configure Knife to look in the temporary directory for the installed cookbooks
+  and not in the normal `cookbooks/` directory.
+
+When you use this integration, any changes you make to anything in the
+`cookbooks/` directory will be ignored by Knife, because Knife won't look in
+that directory for your cookbooks.
 
-In the above, make sure *not* to include the path to your `chef-repo/cookbooks`. If you
-  have additional cookbooks directories in your chef-repo that you use for `:path`-sourced
-  cookbooks in your `Cheffile`, make sure *not* to include the paths to those additional
-  cookbooks directories either in your chef-repo. Since your `chef-repo/site-cookbooks`
-  directory is for overrides (monkey-patches) to external cookbooks, and since you should
-  not have any `:path`-sourced cookbooks in your `Cheffile` sourced from that directory,
-  you still need to include your `chef-repo/site-cookbooks` directory in the above list.
+Reporting Issues
+----------------
 
-What this integration does is when you use `knife`, it will enforce that your `Cheffile`
-  and `Cheffile.lock` are in sync. When you `knife cookbook upload`, it will be sure to
-  upload the same cookbook as is in your `Cheffile.lock`, regardless of what you've done
-  to your `chef-repo/cookbooks` directory.
+Please include relevant `Cheffile` and `Cheffile.lock` files. Please run the
+`librarian-chef` commands in verbose mode by using the `--verbose` flag, and
+include the verbose output in the bug report as well.
 
 License
 -------
@@ -117,4 +272,5 @@ Written by Jay Feldblum.
 
 Copyright (c) 2011 ApplicationsOnline, LLC.
 
-Released under the terms of the MIT License.
+Released under the terms of the MIT License. For further information, please see
+the file `MIT-LICENSE`.

data/lib/librarian/source/git.rb

@@ -79,12 +79,16 @@ module Librarian
           repository.path.mkpath
           repository.clone!(uri)
         end
+        repository.reset_hard!
         unless sha == repository.current_commit_hash
-          repository.fetch!(:tags => true)
-          repository.fetch!
-          repository.merge_all_remote_branches!
-          repository.checkout!(repository.hash_from(sha || ref), :force => true)
-          @sha ||= repository.current_commit_hash
+          remote = repository.default_remote
+          repository.fetch!(remote)
+          repository.fetch!(remote, :tags => true)
+
+          new_sha = repository.hash_from(remote, sha || ref)
+          repository.checkout!(new_sha)
+
+          @sha ||= new_sha
         end
       end
 

data/lib/librarian/source/git/repository.rb

@@ -33,6 +33,10 @@ module Librarian
           path.join('.git').exist?
         end
 
+        def default_remote
+          "origin"
+        end
+
         def clone!(repository_url)
           within do
             command = "clone #{repository_url} ."
@@ -48,48 +52,61 @@ module Librarian
           end
         end
 
-        def fetch!(options = { })
+        def fetch!(remote, options = { })
           within do
-            command = "fetch"
+            command = "fetch #{remote}"
             command << " --tags" if options[:tags]
             run!(command)
           end
         end
 
-        def merge!(reference)
+        def reset_hard!
           within do
-            command = "merge #{reference}"
+            command = "reset --hard"
             run!(command)
           end
         end
 
-        def hash_from(reference)
+        def remote_names
           within do
-            command = "rev-parse #{reference}"
-            run!(command).strip
+            command = "remote"
+            run!(command, false).strip.lines.map(&:strip)
           end
         end
 
-        def current_commit_hash
+        def remote_branch_names
+          remotes = remote_names.sort_by(&:length).reverse
+
           within do
-            command = "rev-parse HEAD"
-            run!(command).strip!
+            command = "branch -r"
+            names = run!(command, false).strip.lines.map(&:strip).to_a
+            names.each{|n| n.gsub!(/\s*->.*$/, "")}
+            names.reject!{|n| n =~ /\/HEAD$/}
+            Hash[remotes.map do |r|
+              matching_names = names.select{|n| n.start_with?("#{r}/")}
+              matching_names.each{|n| names.delete(n)}
+              matching_names.each{|n| n.slice!(0, r.size + 1)}
+              [r, matching_names]
+            end]
           end
         end
 
-        def merge_all_remote_branches!
-          remote_branches.each do |branch|
-            checkout!(branch.slice(%r{[^/]+$}), :force => true)
-            merge! branch
+        def hash_from(remote, reference)
+          branch_names = remote_branch_names[remote]
+          if branch_names.include?(reference)
+            reference = "#{remote}/#{reference}"
+          end
+
+          within do
+            command = "rev-parse #{reference}"
+            run!(command).strip
           end
         end
 
-        def remote_branches
+        def current_commit_hash
           within do
-            command ="branch -r --no-color"
-            run!(command, false).split("\n  ").reject do |r|
-              r.include? '->' #delete pointers like origin/HEAD -> origin/master
-            end.collect {|r|r.strip}
+            command = "rev-parse HEAD"
+            run!(command).strip!
           end
         end
       private

data/lib/librarian/version.rb

@@ -1,3 +1,3 @@
 module Librarian
-  VERSION = "0.0.14"
+  VERSION = "0.0.15"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: librarian
 version: !ruby/object:Gem::Version
-  version: 0.0.14
+  version: 0.0.15
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-02-13 00:00:00.000000000 Z
+date: 2012-04-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
-  requirement: &15281700 !ruby/object:Gem::Requirement
+  requirement: &12899900 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *15281700
+  version_requirements: *12899900
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &15281180 !ruby/object:Gem::Requirement
+  requirement: &12899480 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *15281180
+  version_requirements: *12899480
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &15280600 !ruby/object:Gem::Requirement
+  requirement: &12899060 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *15280600
+  version_requirements: *12899060
 - !ruby/object:Gem::Dependency
   name: cucumber
-  requirement: &15279900 !ruby/object:Gem::Requirement
+  requirement: &12898640 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *15279900
+  version_requirements: *12898640
 - !ruby/object:Gem::Dependency
   name: aruba
-  requirement: &15278980 !ruby/object:Gem::Requirement
+  requirement: &12898220 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *15278980
+  version_requirements: *12898220
 - !ruby/object:Gem::Dependency
   name: webmock
-  requirement: &15278400 !ruby/object:Gem::Requirement
+  requirement: &12897800 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *15278400
+  version_requirements: *12897800
 - !ruby/object:Gem::Dependency
   name: chef
-  requirement: &15277240 !ruby/object:Gem::Requirement
+  requirement: &12897300 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -87,10 +87,10 @@ dependencies:
         version: '0.10'
   type: :runtime
   prerelease: false
-  version_requirements: *15277240
+  version_requirements: *12897300
 - !ruby/object:Gem::Dependency
   name: highline
-  requirement: &15276740 !ruby/object:Gem::Requirement
+  requirement: &12896880 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -98,7 +98,7 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *15276740
+  version_requirements: *12896880
 description: Librarian
 email:
 - y_feldblum@yahoo.com
@@ -216,7 +216,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: librarian
-rubygems_version: 1.8.10
+rubygems_version: 1.8.17
 signing_key: 
 specification_version: 3
 summary: Librarian