polisher 0.5.1 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b1f11ddfcc8473926f86d964a61adf4d625a2823
-  data.tar.gz: 7d26ce93f45697ba9b4c2bd834999a36d5cc91b3
+  metadata.gz: 376315592c88c3de00fca0125ae64583fb660764
+  data.tar.gz: 4d11de1d33a201b7195c090c4831d8e8ee61783c
 SHA512:
-  metadata.gz: eae476631daceb0550ddd0b2bda37e1654e9a8bcaa2da0f5d18ead56e401d58cf09f6a43cea3445751cc9cd1eca4d45583f81ee050bd0dbbb965143b8210c25c
-  data.tar.gz: e229d48dfdc81f7d4c3df1f299f792fb6c95e5983ebc14457f672c53307d0b95b2a2c2307f317945ddc050d518661157c9d89a4ae3f7c425cce2ab67f19bd961
+  metadata.gz: 6217a46af865d179c17d64e8c7cc030f624470b35b480e04f5a4cf2907161acd1afff8b57f283a8c394e0fd401baba77bef07e9318c57b9ac39e7360a461bc66
+  data.tar.gz: fa067ed1e2161d7b185108476c588c24f58f2bc26ce894f499b6b212563366fe68df5e431528d1ef6ce252263e4b930e2a4939585ba90eb908b5b1ca4b3794fa

data/README.md

@@ -29,11 +29,114 @@ Provided are a series of tools geared towards querying rubygems.org and other up
 ruby sources for metadata as well as downstream sources such as the Fedora and Debian
 distributions to cross reference various supported stacks vendored by each.
 
-Also prodivided are tools to integrate and hook into various build and install workflows
+Polisher also includes utilities to integrate and hook into various build and install workflows
 used by both upstream and downstream developers to synergize operations and streamline
 the packaging and support process.
 
-The project is still in its early / conceptual stages, and all are welcome to assist
-shaping its direction. See documentation and spec suite for specific usage.
+## Dependencies
+
+Polisher is made available through [rubygems](http://rubygems.org/gems/polisher).
+
+Polisher dependends on the 'curb' rubygem which in return depends on the curl-devel
+headers. The user should install those and the development tools for their platform
+or install the curb gem packaged by their distribution like so:
+
+    # install the ruby & curl headers & development tools...
+    yum install ruby-devel libcurl-devel
+    yum group install "Development Tools"
+
+    # ... or install the gem manually
+    yum install rubygem-curb
+
+Replace the 'yum install' commands with the corresponding ones on alternate platforms.
+
+Various polisher subcomponents depend on various command line utilities, these include:
+
+* /usr/bin/git - to checkout git repos
+* /usr/bin/koji - to query and build against koji
+* /usr/bin/yum - to query yum
+* /usr/bin/sed - to manipulated metadata
+* /usr/bin/md5sum - to generate required metadata
+* /usr/bin/fedpkg - to query fedora
+
+Some of these are pre-installed on many platforms and some are available via a quick
+'yum install' / 'apt-get' or other. Not all are required for all utilities, see the
+specific executables and modules for details.
+
+## Installation
+
+To install the latest release and all ruby dependencies simply run:
+
+    gem install polisher
+
+See the bin/ directory for all executables available, pass '-h' to any to
+see specific command line options. To run any command from a local git checkout 
+of polisher, run the following beforehand:
+
+    export RUBYLIB='lib'
+
+A few select utilities are highlighted below.
+
+### gem_dependency_checker
+
+A utility to cross reference rubygems and bundler gemfiles against various downstream
+resources including the koji build system, distgit (as used by Fedora), and more.
+
+For example to check a specified ruby app for compatability in koji and yum:
+
+    gem_dependency_checker.rb --gemfile ~/myapp/Gemfile -k -y
+
+### git_gem_updater
+
+A script that clones a rubygem maintained by distgit and update it to the latest
+version of the package available on rubygems (or the specified version if given).
+A scratch built will be run and if all goes will a commit staged so that the end
+user just has to 'git push' the updated package to the distro.
+
+Simply specify the name of the gem to update like so:
+
+    git_gem_updater.rb -n rails
+
+Alternatively if "-u" is specified with a Fedora username, all the packages the
+user owns will be checked out and updated.
+
+### ruby_rpm_spec_updater
+
+A tool to update a given ruby gem or application based rpm spec to the specified
+new source (as specified by a gem, gemspec, or gemfile).
+
+Simply pass it the location of the spec to update, and optionally a upstream
+source which to update from and the utility will print out the updated spec
+to STDOUT.
+
+    ruby_rpm_spec_updater.rb ~/rpmbuild/SPECS/rubygem-rails.spec rails-5.0.0.gem
+
+### Other
+
+There are various other tools and utilities provided and in conceptual phases.
+See the bin dir and TODO document for more detailed listing. 
+
+### polisher
+
+The core library behind these utilities, polisher provides reusable modules to
+query and interface with upstream ruby sources and downstream vendors and systems.
+
+## Documentation and Spec Suite
+
+Polisher comes with complete documentation and a full test suite.
+
+Documentation relies on the 'yard' gem / documentation system, to
+generate run:
+
+    rake yard
+
+The test suite is based on rspec, to run:
+
+    rake spec
+
+## Legal & Other
 
 Polisher is Licensed under the MIT License, Copyright (C) 2013 Red Hat, Inc.
+
+See the commit log for authors of the project. All feedback and contributions
+are more than welcome.

data/Rakefile

@@ -3,15 +3,26 @@
 # Licensed under the MIT license
 # Copyright (C) 2013 Red Hat, Inc.
 
-require "rspec/core/rake_task"
-
-desc "Run all specs"
-RSpec::Core::RakeTask.new(:spec) do |spec|
-  spec.pattern = 'specs/**/*_spec.rb'
-  spec.rspec_opts = ['--backtrace', '-fd', '-c']
+begin
+  require "rspec/core/rake_task"
+  desc "Run all specs"
+  RSpec::Core::RakeTask.new(:spec) do |spec|
+    spec.pattern = 'specs/**/*_spec.rb'
+    spec.rspec_opts = ['--backtrace', '-fd', '-c']
+  end
+rescue LoadError
 end
 
 desc "build the polisher gem"
 task :build do
   system "gem build polisher.gemspec"
 end
+
+begin
+  require "yard"
+  YARD::Rake::YardocTask.new do |t|
+    #t.files   = ['lib/**/*.rb', OTHER_PATHS]   # optional
+    #t.options = ['--any', '--extra', '--opts'] # optional
+  end
+rescue LoadError
+end

data/lib/polisher/bodhi.rb

@@ -8,6 +8,9 @@ require 'pkgwat'
 module Polisher
   class Bodhi
     def self.versions_for(name, &bl)
+      # XXX issue w/ retreiving packages from pkgwat causing issues:
+      # https://github.com/fedora-infra/fedora-packages/issues/55
+
       # fedora pkgwat provides a frontend to bodhi
       updates = Pkgwat.get_updates("rubygem-#{name}", 'all', 'all') # TODO set timeout
       updates.reject! { |u|

data/lib/polisher/errata.rb

@@ -3,6 +3,7 @@
 # Licensed under the MIT license
 # Copyright (C) 2013 Red Hat, Inc.
 
+require 'json'
 require 'curb'
 
 module Polisher

data/lib/polisher/fedora.rb

@@ -6,18 +6,24 @@
 require 'curb'
 require 'pkgwat'
 
+require 'polisher/bodhi'
+
 module Polisher
   class Fedora
     PACKAGE_LIST = 'https://admin.fedoraproject.org/pkgdb/users/packages/'
 
+    def self.client
+      @client ||= Curl::Easy.new
+    end
+
     # Retrieve list of gems owned by the specified user
     #
     # @param [String] user Fedora username to lookup
     # @return [Array<String>] list of gems which the user owns/has access to
     def self.gems_owned_by(user)
-      curl = Curl::Easy.new("#{PACKAGE_LIST}}#{user}")
-      curl.http_get
-      packages = curl.body_str
+      client.url = "#{PACKAGE_LIST}#{user}"
+      client.http_get
+      packages = client.body_str
       # TODO instantiate Polisher::Gem instances & return
       Nokogiri::HTML(packages).xpath("//a[@class='PackageName']").
                                select { |i| i.text =~ /rubygem-.*/ }.
@@ -31,13 +37,10 @@ module Polisher
     # @param [Callable] bl optional callback to invoke with versions retrieved
     # @return [Array<String>] list of versions in Fedora
     def self.versions_for(name, &bl)
-      # XXX bug w/ python-pkgwat, some html content
-      # is being returned w/ versions, need to look into
-      versions = Pkgwat.get_versions(name)
-      versions.reject! { |pkg| pkg['stable_version'] == "None" }
-      versions = versions.collect { |pkg| pkg['stable_version'] }
-      bl.call(:fedora, name, versions) unless(bl.nil?) 
-      versions
+      # simply dispatch to bodhi to get latest updates
+      Polisher::Bodhi.versions_for name do |target,name,versions|
+        bl.call(:fedora, name, versions) unless(bl.nil?) 
+      end
     end
   end # class Fedora
 end # module Polisher

data/lib/polisher/gem.rb

@@ -5,6 +5,7 @@
 
 require 'curb'
 require 'json'
+require 'yaml'
 require 'tempfile'
 require 'pathname'
 require 'rubygems/installer'
@@ -14,6 +15,9 @@ require 'polisher/version_checker'
 
 module Polisher
   class Gem
+    GEM_CMD      = '/usr/bin/gem'
+
+    attr_accessor :spec
     attr_accessor :name
     attr_accessor :version
     attr_accessor :deps
@@ -21,6 +25,7 @@ module Polisher
     attr_accessor :files
 
     def initialize(args={})
+      @spec     = args[:spec]
       @name     = args[:name]
       @version  = args[:version]
       @deps     = args[:deps]     || []
@@ -51,19 +56,35 @@ module Polisher
 
       if args.is_a?(String)
         specj     = JSON.parse(args)
-        metadata[:name]     = specj['name']
-        metadata[:version]  = specj['version']
-        metadata[:deps]     = specj['dependencies']['runtime'].collect { |d| d['name'] }
-        metadata[:dev_deps] = specj['dependencies']['development'].collect { |d| d['name'] }
+        metadata[:spec]    = specj
+        metadata[:name]    = specj['name']
+        metadata[:version] = specj['version']
+
+        metadata[:deps] =
+          specj['dependencies']['runtime'].collect { |d|
+            ::Gem::Dependency.new d['name'], *d['requirements'].split(',')
+          }
+
+        metadata[:dev_deps] =
+          specj['dependencies']['development'].collect { |d|
+            ::Gem::Dependency.new d['name'], d['requirements']
+          }
 
       elsif args.has_key?(:gemspec)
         gemspec  = ::Gem::Specification.load(args[:gemspec])
-        metadata[:name]     = gemspec.name
-        metadata[:version]  = gemspec.version.to_s
-        metadata[:deps]     =
-          gemspec.dependencies.select { |dep| dep.type == :runtime }.collect { |dep| dep.name }
+        metadata[:spec]    = gemspec # TODO to json
+        metadata[:name]    = gemspec.name
+        metadata[:version] = gemspec.version.to_s
+
+        metadata[:deps] =
+          gemspec.dependencies.select { |dep|
+            dep.type == :runtime
+          }.collect { |dep| dep }
+
         metadata[:dev_deps] =
-          gemspec.dependencies.select { |dep| dep.type == :development }.collect { |dep| dep.name }
+          gemspec.dependencies.select { |dep|
+            dep.type == :development
+          }.collect { |dep| dep }
 
       elsif args.has_key?(:gem)
         # TODO
@@ -82,6 +103,27 @@ module Polisher
       gemf = curl.body_str
     end
 
+    # Refresh the gem spec
+    def refresh_spec
+      gemf = download_gem
+      tgem = Tempfile.new(@name)
+      tgem.write gemf
+      tgem.close
+
+      @spec = YAML.load `#{GEM_CMD} specification #{tgem.path}`
+
+      # update deps, dev_deps
+      @deps =
+        @spec.dependencies.select { |dep|
+          dep.type == :runtime
+        }.collect { |dep| dep }
+
+      @dev_deps =
+        @spec.dependencies.select { |dep|
+          dep.type == :development
+        }.collect { |dep| dep }
+    end
+
     # Retrieve the list of files in the gem
     #
     # @return [Array<String>] list of files in the gem
@@ -115,7 +157,7 @@ module Polisher
       gem
     end
 
-    # Retreive versions of gem available on rubygems.org
+    # Retreive versions of gem available in all configured targets (optionally recursively)
     #
     # @param [Hash] args hash of options to configure retrieval
     # @option args [Boolean] :recursive indicates if versions of dependencies
@@ -134,16 +176,16 @@ module Polisher
 
       if recursive
         self.deps.each { |dep|
-          unless versions.has_key?(dep)
-            gem = Polisher::Gem.retrieve(dep)
+          unless versions.has_key?(dep.name)
+            gem = Polisher::Gem.retrieve(dep.name)
             versions.merge! gem.versions(args, &bl)
           end
         }
 
         if dev_deps
           self.dev_deps.each { |dep|
-            unless versions.has_key?(dep)
-              gem = Polisher::Gem.retrieve(dep)
+            unless versions.has_key?(dep.name)
+              gem = Polisher::Gem.retrieve(dep.name)
               versions.merge! gem.versions(args, &bl)
             end
           }

data/lib/polisher/git.rb

@@ -5,6 +5,8 @@
 
 # TODO use ruby git api and others
 
+require 'tmpdir'
+require 'awesome_spawn'
 require 'polisher/rpmspec'
 
 module Polisher
@@ -50,7 +52,7 @@ module Polisher
       rpm_name = "#{RPM_PREFIX}#{name}"
 
       unless File.directory? rpm_name
-        `#{PKG_CMD} clone #{rpm_name}`
+        AwesomeSpawn.run "#{PKG_CMD} clone #{rpm_name}"
       end
       
       # cd into working directory
@@ -62,9 +64,9 @@ module Polisher
       
       # checkout the latest rawhide
       # TODO allow other branches to be specified
-      `#{GIT_CMD} checkout master`
-      `#{GIT_CMD} reset HEAD~ --hard`
-      `#{GIT_CMD} pull`
+      AwesomeSpawn.run "#{GIT_CMD} checkout master"
+      AwesomeSpawn.run "#{GIT_CMD} reset HEAD~ --hard"
+      AwesomeSpawn.run "#{GIT_CMD} pull"
 
       self.new :name => name
     end
@@ -74,19 +76,19 @@ module Polisher
     # @param [Polisher::Gem] gem instance of gem containing metadata to update to
     def update_to(gem)
       # TODO use Polisher::RPMSpec to update spec
-      `#{SED_CMD} -i "s/Version.*/Version: #{gem.version}/" #{spec}`
-      `#{SED_CMD} -i "s/Release:.*/Release: 1%{?dist}/" #{spec}`
-      `#{MD5SUM_CMD} #{gem.name}-#{gem.version}.gem > sources`
+      AwesomeSpawn.run "#{SED_CMD} -i 's/Version.*/Version: #{gem.version}/' #{spec}"
+      AwesomeSpawn.run "#{SED_CMD} -i 's/Release:.*/Release: 1%{?dist}/' #{spec}"
+      AwesomeSpawn.run "#{MD5SUM_CMD} #{gem.name}-#{gem.version}.gem > sources"
       File.open(".gitignore", "w") { |f| f.write "#{gem.name}-#{gem.version}.gem" }
     end
 
     # Build the locally cloned package using the configured build command
     def build
       # build srpm
-      `#{PKG_CMD} srpm`
+      AwesomeSpawn.run "#{PKG_CMD} srpm"
       
       # attempt to build packages
-      `#{BUILD_CMD} build --scratch #{BUILD_TGT} #{srpm}`
+      AwesomeSpawn.run "#{BUILD_CMD} build --scratch #{BUILD_TGT} #{srpm}"
       # TODO if build fails, spit out error, exit
     end
 
@@ -94,7 +96,7 @@ module Polisher
     #
     # @return [Boolean] true/false depending on whether or not spec has %check
     def has_check?
-      open(spec, "r") do |spec|
+      File.open(spec, "r") do |spec|
         spec.lines.any? { |line| line.include?("%check") }
       end 
     end
@@ -102,9 +104,9 @@ module Polisher
     # Command the local package to the local git repo
     def commit
       # git add spec, git commit w/ message
-      `#{GIT_CMD} add #{spec} sources .gitignore`
+      AwesomeSpawn.run "#{GIT_CMD} add #{spec} sources .gitignore"
       #`git add #{gem_name}-#{version}.gem`
-      `#{GIT_CMD} commit -m 'updated to #{self.version}'`
+      AwesomeSpawn.run "#{GIT_CMD} commit -m 'updated to #{self.version}'"
     end
 
     # Retrieve list of the version of the specified package in git
@@ -119,7 +121,7 @@ module Polisher
       Dir.mktmpdir do |dir|
         Dir.chdir(dir) do |path|
           version = nil
-          `#{GIT_CMD} clone #{DIST_GIT_URL}#{rpm_name}.git . >& /dev/null`
+          AwesomeSpawn.run "#{GIT_CMD} clone #{DIST_GIT_URL}#{rpm_name}.git ."
           begin
             spec = Polisher::RPMSpec.parse(File.read("#{rpm_name}.spec"))
             version = spec.version

data/lib/polisher/koji.rb

@@ -35,6 +35,22 @@ module Polisher
       end
     end
 
+    # Return bool indiciating if koji has a build exactly
+    # matching the specified version
+    def self.has_build?(name, version)
+      versions = self.versions_for name
+      versions.include?(version)
+    end
+
+    # Return bool indicating if koji has a build which
+    # satisfies the specified ruby dependency
+    def self.has_build_satisfying?(name, version)
+      dep = ::Gem::Dependency.new name, version
+      self.versions_for(name).any? { |v|
+        dep.match?(name, v)
+      }
+    end
+
     # Retrieve list of the version of the specified package in koji
     #
     # @param [String] name name of package to lookup