trackrepos 1.0.0

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in track-repos.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Concord Consortium
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,116 @@
+# trackrepos
+
+Command line tool that parses a yaml file in a directory to update and report on a collection of repositories.
+
+## Installation
+
+Install the Ruby Gem `track-repos`:
+
+    gem install track-repos
+
+## Setup
+
+Create a `.tracked-repos.yaml` file to update and keep track of either existing git repositores or new ones you intend to add.
+
+There are currently three types of git repositores that can be tracked andupdated:
+
+1. Regular git repositories.
+2. Git clones of Subversion repositories
+3. Git clones of git mirrors of subversion repositories (need to explicity request new tags).
+
+Here's an example yaml configuration file that tracks the Ruby Gems `aasm`, and `builder`.
+Normally the master branch is fetched hower the specifiction for thr `builder` gem indicates that
+the `trunk` branch should be checked out and tracked.In addition the `arduino` project which
+is located in a Subersion repository is checked out as a git clone of a subversion repository.
+
+    ---
+    :git:
+    - :path: aasm
+      :remote: http://github.com/rubyist/aasm.git
+    - :path: buildr
+      :branch: trunk
+      :remote: git://github.com/apache/buildr.git
+    :git_svn:
+    - :path: arduino
+      :remote: http://arduino.googlecode.com/svn
+    :git_clones_of_git_clones_of_svn_repos:
+    - :path: ruby
+      :branch: trunk
+      :remote: git://github.com/ruby/ruby.git
+
+The last item in the list specifies that the source code for Ruby should be checked out. In
+this case it is being cloned from a git repository on github which is a miirror of the main
+Ruby Subversion repository. This tpe if differentiated from a normal git repository because an
+additional action is taken to manually fetch any updated tags.
+
+The `.tracked-repos.yaml` file is a YAML serialization of the kind of data expressed in this Ruby Hash.
+Each type of external repositor consists of an array of repository specificationns with keys for
+`:path` and `:remote` git url, as well as an optional key for the `:branch` that should be tracked
+The defatult is the master branch and does not need to be specified.
+
+    { :git =>
+        [
+          { :path=>"aasm-git",
+            :remote=>"http://github.com/rubyist/aasm.git" },
+          { :path=>"buildr-git",
+            :branch=>"trunk",
+            :remote=>"git://github.com/apache/buildr.git" }
+        ],
+      :git_svn =>
+        [
+          { :path => "arduino-svn-git",
+            :remote => "http://arduino.googlecode.com/svn" }
+        ],
+      :git_clones_of_git_svn_mirrors_of_svn_repos =>
+        [
+          { :path => "ruby-git",
+            :branch => "trunk",
+            :remote => "git://github.com/ruby/ruby.git" }
+        ]
+    }
+
+## Usage
+
+Change to the directory with the `.tracked-repos.yaml` file and run the command-line tool:
+
+    track-repos
+
+### help
+
+    $ bin/trackrepos --help
+    Usage: trackrepos
+        -s YAMLFILE,                     Use specification file YAMLFILE instead of '.tracked-repos.yaml'
+            --specification-file
+        -d YAMLDIRECTORYFILE,            Use specification file YAMLDIRECTORYFILE instead of '.tracked-directories.yaml'
+            --directory-file
+        -v, --verbose                    Display yaml specification file
+        -g, --generate                   Generate and display yaml specification suitable for a '.tracked-repos.yaml'
+                                         file from existing repositories
+
+### Tracking directories
+
+You can also use a yaml specification file that just has lists of paths with yaml repository specifications.
+
+The defautt name for a directory tracking file: `.tracked-directories.yaml`
+
+This directory tracking file would cause `trackrepos` to process yaml repository specifications
+in both `dir1/` and `dir2/`
+
+    ---
+    - dir1
+    - dir2
+
+### Specifying different names for yaml specification files
+
+This example looks in the file `tracked-directories.yaml` in the current directory
+and looks for the file `tracked-repos.yaml` in each of the directroies it references.
+
+    $ trackrepos -d tracked-directories.yaml -s tracked-repos.yaml
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,2 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"

data/bin/trackrepos

@@ -0,0 +1,35 @@
+#!/usr/bin/env ruby
+$:.unshift(File.dirname(__FILE__) + '/../lib/')
+
+require 'optparse'
+require 'trackrepos'
+
+if %w(--version -v).include? ARGV.first
+  puts "trackrepos #{TrackRepos::VERSION}"
+  exit(0)
+end
+
+options = {}
+OptionParser.new do |opts|
+  opts.banner = "Usage: trackrepos"
+  opts.on("-s", "--specification-file YAMLFILE",
+    "Use specification file YAMLFILE instead of '#{TrackRepos::Repos::TRACKED_YAML}'") do |s|
+    options[:tracked_filename] = s
+  end
+  opts.on("-d", "--directory-file YAMLDIRECTORYFILE",
+    "Use specification file YAMLDIRECTORYFILE instead of '#{TrackRepos::Repos::TRACKED_DIRECTORIES_YAML}'") do |d|
+    options[:tracked_directories] = d
+  end
+  opts.on("-v", "--verbose",
+    "Display yaml specification file") do |v|
+    options[:verbose] = v
+  end
+
+  opts.on("-g", "--generate",
+    "Generate and display yaml specification suitable for a '#{TrackRepos::Repos::TRACKED_YAML}' file from existing repositories") do |g|
+    options[:generate] = g
+  end
+end.parse!
+
+repos = TrackRepos::Repos.new options
+repos.track

data/lib/trackrepos/repos.rb

@@ -0,0 +1,303 @@
+require 'yaml'
+require 'time'
+
+module TrackRepos
+  class Repos
+
+    PATH_FORMAT_STR   = "    %-56s%-30s%-24s"
+    COMMIT_FORMAT_STR = "  %-24s%s"
+
+    TRACKED_YAML = '.tracked-repos.yaml'
+    TRACKED_DIRECTORIES_YAML = '.tracked-directories.yaml'
+
+    GIT_COMMANDS = {
+      :git => [
+        ['git pull'],
+        'Updating local git clones of external git repositories'
+      ],
+
+      :git_svn_hybrid => [
+        ['git svn rebase'],
+        'Updating local git svn hybrid clones of external subversion repositories'
+      ],
+      :git_svn => [
+        ['git svn rebase'],
+        'Updating local git clones of external subversion repositories'
+      ],
+      :git_clones_of_git_clones_of_svn_repos => [
+        ['git pull', 'git fetch -t'],
+        'Updating local git clones of external git repositories which themselves are created from git-svn clones of svn repos'
+      ]
+    }
+
+    def initialize(options)
+      @options = options || {}
+      @tracked_filename = @options[:tracked_filename] || TRACKED_YAML
+      @tracked_directories = @options[:tracked_directories] || TRACKED_DIRECTORIES_YAML
+      if File.exists?(@tracked_directories)
+        @tracked_directories = YAML.load_file(@tracked_directories)
+      else
+        @tracked_directories = [ Dir.pwd ]
+      end
+      @tracked_collections = []
+      @tracked_directories.each do |dir|
+        dir_path = File.expand_path(dir)
+        unless File.exists?(dir_path)
+          raise <<-HEREDOC
+
+
+*** ERROR : Directory: '#{dir}' doesn't exist.
+
+          HEREDOC
+        end
+        tracking_spec_path = File.join(dir_path, @tracked_filename)
+        unless File.exists?(tracking_spec_path)
+          raise <<-HEREDOC
+
+
+*** ERROR: YAML specification file does not exist: '#{tracking_spec_path}'
+
+          HEREDOC
+        end
+        tracking_spec = YAML.load_file(tracking_spec_path)
+        @tracked_collections <<  { 
+          :dir => dir,
+          :tracking_spec => tracking_spec,
+          :repos => [],
+          :repos_with_update_errors => { "git error" => [], "dir doesn't exist" => [] }
+        }
+      end
+    end
+
+    def track
+      @tracked_collections.each do |collection|
+        track_collection(collection)
+        # FIXME: why are there duplicates ???
+        collection[:repos].uniq!
+        collection[:repos].sort! { |r1, r2| r2[1][:updated_date] <=> r1[1][:updated_date] }
+      end
+      @tracked_collections.each do |collection|
+        report_results(collection)
+      end
+
+    end
+
+    def track_collection(collection)
+      dir = collection[:dir]
+      tracking_spec = collection[:tracking_spec]
+      Dir.chdir File.expand_path(dir) do
+        puts <<-HEREDOC
+
+=======================================================================================
+
+Tracking git repositories in directory: #{dir}
+        HEREDOC
+        first_time = true
+        tracking_spec.each do |type, tracked_repos|
+          unless first_time
+            puts <<-HEREDOC
+
+-----------------------------------------------------------------------------------
+            HEREDOC
+          end
+          commands = GIT_COMMANDS[type][0]
+          desc = GIT_COMMANDS[type][1]
+          update_local_gits(collection, type, tracked_repos, commands, desc)
+          first_time = false
+        end
+      end
+    end
+
+    def update_local_gits(collection, type, tracked_repos, commands, message)
+      dir = collection[:dir]
+      if tracked_repos
+        puts <<-HEREDOC
+
+  #{message}
+  commands: #{commands.join('; ')}
+
+        HEREDOC
+        tracked_repos.each do |tracked_repo|
+          path = tracked_repo[:path]
+          branch = tracked_repo[:branch] || 'master'
+          remote = tracked_repo[:remote]
+          if !File.exists?(path) && remote
+            case type
+            when :git
+              cmd = "mkdir -p #{path}; git clone #{remote} #{path}"
+            when :git_svn
+              cmd = "mkdir -p #{path}; git svn clone #{remote} #{path}"
+            end
+            puts <<-HEREDOC
+
+  Checking out new project #{remote}
+  Into directory:          #{File.expand_path(dir, path)}
+
+  command: #{cmd}
+
+            HEREDOC
+            `#{cmd}`
+            puts
+          end
+          if File.exists?(path)
+            git_remote = `git config --file #{path}/.git/config --get remote.origin.url`.strip
+            git_svn_remote = `git config --file #{path}/.git/config --get svn-remote.svn.url`.strip
+            if git_remote.empty?
+              tracked_repo[:remote] = git_svn_remote
+            else
+              tracked_repo[:remote] = git_remote
+            end
+            git_command(dir, path, commands, collection, branch)
+            if File.exists?(File.join(path, '.gitmodules'))
+              git_command(dir, path, ['git submodule update --init --recursive'], collection, branch, { :quiet => true })
+            end
+          else
+            puts <<-HEREDOC
+
+  ERROR : Path: '#{path}' doesn't exist and failed to create new repository.
+
+            HEREDOC
+            collection[:repos_with_update_errors]["dir doesn't exist"] << path
+          end
+        end
+      end
+    end
+
+
+    def report_results(collection)
+      dir = collection[:dir]
+      if collection[:repos].length > 20
+        puts <<-HEREDOC
+
+=======================================================================================
+
+  Git repositories in directory: #{dir}
+  updated in the last week:
+
+        HEREDOC
+        collection[:repos].find_all {|r| r[1][:updated_in_last_week] }.uniq.each  do |repo|
+          puts sprintf(PATH_FORMAT_STR, repo[0].gsub(dir+ '/', ""), repo[1][:updated_relative], repo[1][:desc])
+        end
+        puts <<-HEREDOC
+
+Least recently updated:
+
+        HEREDOC
+        collection[:repos][-15..-1].each  do |repo|
+          puts sprintf(PATH_FORMAT_STR, repo[0].gsub(dir+ '/', ""), repo[1][:updated_relative], repo[1][:desc])
+        end
+      else
+        puts <<-HEREDOC
+
+=======================================================================================
+
+  Git repositories sorted by when last updated : #{dir}
+
+        HEREDOC
+        collection[:repos].each  do |repo|
+          puts sprintf(PATH_FORMAT_STR, repo[0].gsub(dir+ '/', ""), repo[1][:updated_relative], repo[1][:desc])
+        end
+      end
+
+      if @options[:verbose]
+        puts <<-HEREDOC
+
+---------------------------------------------------------------------------------------
+
+  YAML form of '#{collection[:dir]}/#{@tracked_filename}':
+
+          HEREDOC
+        puts collection[:tracking_spec].to_yaml
+      end
+      unless collection[:repos_with_update_errors].values.flatten.empty?
+        puts <<-HEREDOC
+
+=======================================================================================
+
+  Error summary:
+
+  The following git repositories were not updated because of errors:
+        HEREDOC
+        collection[:repos_with_update_errors].each do |error, paths|
+          unless paths.empty?
+            puts <<-HEREDOC
+
+    #{error}:
+    - #{paths.join("\n  - ")}
+            HEREDOC
+          end
+        end
+      end
+      puts
+    end
+
+    def gem_desc_in_rakefile
+      if File.exists?('Rakefile')
+        desc = File.read('Rakefile')[/summary\s*=\s*\"(.*)\"/i, 1]
+      else
+        nil
+      end
+    end
+
+    def gem_desc_in_config_hoe
+      if File.exists?('config/hoe.rb')
+        File.read('config/hoe.rb')[/summary\s*=\s*\"(.*)\"/i, 1]
+      else
+        nil
+      end
+    end
+
+    def look_for_description
+      desc = nil
+      gemspec = Dir["*.gemspec"]
+      unless gemspec.empty?
+        desc = File.read(gemspec[0])[/summary\s*=\s*\"(.*)\"/i, 1]
+        return desc
+      end
+      readme = Dir["*"].find_all {|p| p[/.*readme.*/i]}
+      unless readme.empty?
+        desc = File.readlines(readme[0])[0].strip
+        return desc
+      end
+      desc
+    end
+
+    def git_command(dir, path, commands, collection, branch='master', options={})
+      Dir.chdir File.expand_path(path) do
+        response = `git checkout #{branch} 2>&1`
+        commands.each do |command|
+          response += `#{command} 2>&1`
+        end
+        desc = look_for_description
+        unless options[:quiet]
+          if response =~ /Already up-to-date|Current branch master is up to date/
+            puts sprintf(PATH_FORMAT_STR, path, `git log -1 --pretty=format:"%cr"`, desc)
+          else
+            puts sprintf(PATH_FORMAT_STR, path, `git log -1 --pretty=format:"%cr"`, desc)
+            puts "\n#{response}\n"
+          end
+        end
+        if response =~/^(error:|fatal:)/
+          puts <<-HEREDOC
+  Error updating: #{path}
+
+          HEREDOC
+          collection[:repos_with_update_errors]["git error"] << path
+        else
+          collection[:repos] << [
+            path,
+            { :desc => desc,
+              :updated_date => Time.rfc2822(`git log -1 --pretty=format:"%cD"`),
+              :updated_relative => `git log -1 --pretty=format:"%cr"`,
+              :updated_in_last_day => !`git log HEAD --no-merges --reverse --since='1 day'`.empty?,
+              :updated_in_last_week => !`git log HEAD --no-merges --reverse --since='1 weeks'`.empty?,
+              :updated_in_last_two_weeks => !`git log HEAD --no-merges --reverse --since='2 weeks'`.empty?,
+              :commit_subject => `git log -1 --pretty=format:"%s %cn"`,
+              :commit_author => `git log -1 --pretty=format:"%cn"` }
+          ]
+        end
+      end
+    end
+
+  end
+end

data/lib/trackrepos/version.rb

@@ -0,0 +1,3 @@
+module TrackRepos
+  VERSION = "1.0.0"
+end

data/lib/trackrepos.rb

@@ -0,0 +1,2 @@
+require "trackrepos/version"
+require "trackrepos/repos"

data/trackrepos.gemspec

@@ -0,0 +1,61 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib/', __FILE__)
+$:.unshift lib unless $:.include?(lib)
+
+require 'trackrepos/version'
+
+Gem::Specification.new do |gem|
+  gem.authors       = ["Stephen Bannasch"]
+  gem.email         = ["sbannasch@concord.org"]
+  gem.description   = %q{Uses yaml configuration files to track and update collections of external git repositories}
+  gem.summary       = %q{Useful for tracking large numbers of external git repositories}
+  gem.homepage      = "https://github.com/concord-consortium/trackrepos"
+  gem.files         = `git ls-files`.split($\)
+  gem.executable    = "trackrepos"
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.name          = "trackrepos"
+  gem.require_paths = ["lib"]
+  gem.version       = TrackRepos::VERSION
+
+  gem.post_install_message = <<-HEREDOC
+
+trackrepos
+
+Create a '.tracked-repos.yaml' file to update and keep track of either existing git
+repositores or new ones you intend to add.
+
+There are currently three types of git repositores that can be tracked andupdated:
+
+1. Regular git repositories.
+2. Git clones of Subversion repositories
+3. Git clones of git mirrors of subversion repositories (need to explicity request new tags).
+
+Here's an example yaml configuration file that tracks the Ruby Gems `aasm`, and `builder`.
+Normally the master branch is fetched hower the specifiction for thr `builder` gem indicates that
+the `trunk` branch should be checked out and tracked.In addition the `arduino` project which
+is located in a Subersion repository is checked out as a git clone of a subversion repository.
+
+file: .tracked-repos.yaml
+
+    ---
+    :git:
+    - :path: aasm
+      :remote: http://github.com/rubyist/aasm.git
+    - :path: buildr
+      :branch: trunk
+      :remote: git://github.com/apache/buildr.git
+    :git_svn:
+    - :path: arduino
+      :remote: http://arduino.googlecode.com/svn
+    :git_clones_of_git_clones_of_svn_repos:
+    - :path: ruby
+      :branch: trunk
+      :remote: git://github.com/ruby/ruby.git
+
+The last item in the list specifies that the source code for Ruby should be checked out. In
+this case it is being cloned from a git repository on github which is a miirror of the main
+Ruby Subversion repository. This tpe if differentiated from a normal git repository because an
+additional action is taken to manually fetch any updated tags.
+
+  HEREDOC
+end

metadata

@@ -0,0 +1,75 @@
+--- !ruby/object:Gem::Specification
+name: trackrepos
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+  prerelease: 
+platform: ruby
+authors:
+- Stephen Bannasch
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-06-10 00:00:00.000000000 Z
+dependencies: []
+description: Uses yaml configuration files to track and update collections of external
+  git repositories
+email:
+- sbannasch@concord.org
+executables:
+- trackrepos
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- bin/trackrepos
+- lib/trackrepos.rb
+- lib/trackrepos/repos.rb
+- lib/trackrepos/version.rb
+- trackrepos.gemspec
+homepage: https://github.com/concord-consortium/trackrepos
+licenses: []
+post_install_message: ! "\ntrackrepos\n\nCreate a '.tracked-repos.yaml' file to update
+  and keep track of either existing git\nrepositores or new ones you intend to add.\n\nThere
+  are currently three types of git repositores that can be tracked andupdated:\n\n1.
+  Regular git repositories.\n2. Git clones of Subversion repositories\n3. Git clones
+  of git mirrors of subversion repositories (need to explicity request new tags).\n\nHere's
+  an example yaml configuration file that tracks the Ruby Gems `aasm`, and `builder`.\nNormally
+  the master branch is fetched hower the specifiction for thr `builder` gem indicates
+  that\nthe `trunk` branch should be checked out and tracked.In addition the `arduino`
+  project which\nis located in a Subersion repository is checked out as a git clone
+  of a subversion repository.\n\nfile: .tracked-repos.yaml\n\n    ---\n    :git:\n
+  \   - :path: aasm\n      :remote: http://github.com/rubyist/aasm.git\n    - :path:
+  buildr\n      :branch: trunk\n      :remote: git://github.com/apache/buildr.git\n
+  \   :git_svn:\n    - :path: arduino\n      :remote: http://arduino.googlecode.com/svn\n
+  \   :git_clones_of_git_clones_of_svn_repos:\n    - :path: ruby\n      :branch: trunk\n
+  \     :remote: git://github.com/ruby/ruby.git\n\nThe last item in the list specifies
+  that the source code for Ruby should be checked out. In\nthis case it is being cloned
+  from a git repository on github which is a miirror of the main\nRuby Subversion
+  repository. This tpe if differentiated from a normal git repository because an\nadditional
+  action is taken to manually fetch any updated tags.\n\n"
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Useful for tracking large numbers of external git repositories
+test_files: []