polisher 0.3 → 0.4

data/README.rdoc

@@ -1,35 +1,105 @@
-== Rubygem Polisher
+== Polisher
 Copyright (C) 2010 Red Hat, Inc.
 
 Written by Mohammed Morsi <mmorsi@redhat.com>
 
 === Intro
-Polisher is a webapp utility greated towards processing rubygems after they
-have been published. It allows the end user to configure events to be run when
-specified gems are updated.
+Polisher is a project release management tool which can be used to register various event
+handlers to be invoked on specific versions of project / source releases. Polisher provides
+a simple {REST}[http://en.wikipedia.org/wiki/Representational_State_Transfer] interface which
+to create projects w/ sources and dependencies, and to download and transform them into any
+target output format.
 
-Polisher uses the gemcutter webhook API in conjunction with the sinatra/rack
-web framework to subscribe to gem updates and run any specified/arbitrary event
-handler callbacks.
+Polisher also provides a simple {DSL}[http://en.wikipedia.org/wiki/Domain-specific_language]
+frontend to the REST interface, able to be used to create and trigger elegant release
+workflows for any number of projects simultaneously. Workflows for many major Ruby based projects
+have been setup, transforming various versions into {Fedora}[http://fedoraproject.org/] repos and
+can be found at [http://github.com/movitto/polisher-scripts].
 
-=== Running
+
+== Running
 To install simply run:
 
   'gem install polisher'
 
+
 Alternatively checkout the source from github via
 
 'git clone git://github.com/movitto/polisher.git'
 
-To run the server, run 'bin/server' in the project root.
+
+Polisher requires various rubygems to work. gem install should pull in all dependencies
+but if running polisher from source, be sure to gem install:
+
+* rubygems
+* sinatra
+* libxml
+* curb
+* activerecord
+* libxml
+
+For the complete list of dependencies, see the
+{Rakefile}[http://github.com/movitto/polisher/blob/master/Rakefile].
+
+
+Polisher currently pulls config from the config/ subdir, the database
+params are defined in database.yml (currently defaulting to a sqlite flat file
+db residing in db/data), and the general application config resides in
+polisher.yml. The only general app config currently used is 'gem_api_key',
+which will be read from ~/.gem/credentials if missing from polisher.yml.
+
+To run the server, run './bin/server' in the project root.
 
 To run the spec suite simply run 'rake spec'
 
-Before running anything make sure to set any neccessary configuration
-in config/polisher.yml (particularily your gem api key)
+Config is environment specific, the server runs in the 'development' environment
+by default and the spec suite runs in 'test' by default. To change the environment
+simply run
+
+  'export RACK_ENV="production"'
+
+before './bin/server' or 'rake spec', etc
 
 === Using
- See the generated API docs and spec suite for more detailed usage.
+Run 'rake spec' in the project root to generate the API docs in the doc/ subdir.
+Also see the spec suite and polisher scripts for more detailed usage.
+
+When running, the server can be accessed by navigating to 'http://localhost:3000'.
+REST requests may be issued against that URI and the actions defined in polisher.rb.
+The DSL sits on top of this REST interface and provides a nice / simple means which
+to setup projects, sources, release workflows, etc, in an easily reproducable manner.
+
+At typical DSL use case might look like:
+
+  project :name => "ruby" do |proj|
+   proj.add_archive :name => 'ruby_source', :uri => "ftp://ftp.ruby-lang.org/pub/ruby/%{rubyxver}/ruby-%{arcver}.tar.bz2" do |archive|
+     archive.version "1.8.6", :rubyxver => "1.8", :arcver => "1.8.7-p311", :corresponds_to => proj.version("1.8.6")
+     proj.version "1.9.1", :corresponds_to => archive.version("1.9.1", :rubyxver => "1.9", :arcver => "1.8.7-p300")
+   end
+   proj.add_patch "http://cvs.fedoraproject.org/viewvc/rpms/ruby/F-13/ruby-deadcode.patch?view=markup"
+   # etc...
+
+   proj.on_version "*", "create package"
+   proj.on_version "=",  "1.8.6",  "update repo", "stable"
+   proj.on_version ">=", "1.9.1",  "update repo", "devel"
+  end
+
+Polisher is a work in progress, and features are being added / removed as neccessary.
+Comments, bug reports, and patches are all very welcome.
+
+== Extending
+Out of the box Polisher is able to handle {Rubygem}[http://rubygems.org/] based projects and
+generate rpm packages and yum repos on project releases. The
+{Gemcutter API}[http://rubygems.org/pages/api_docs],
+{gem2rpm}[http://rubyforge.org/projects/gem2rpm/], and rpm / yum tools are used to
+accomplish this, though Polisher can be easily extended to support any input / output format.
+
+The code itself currently has to be modified (most likely will be changed in the future),
+but to add support for downloading another input source format, extend
+{source.rb::download_to}[http://github.com/movitto/polisher/blob/master/db/models/source.rb]
+and optionally add another add_source_type method to the
+{Polisher::Project DSL}[http://github.com/movitto/polisher/blob/master/lib/dsl.rb].
 
- Polisher is a work in progress, and features are being added / removed as neccessary.
- Comments, bug reports, and patches are all very welcome.
+To add another event handler, simple add a new method to the
+{EventHandlers}[http://github.com/movitto/polisher/blob/master/lib/event_handlers.rb] module.
+Polisher will take care of the rest.

data/Rakefile

@@ -14,23 +14,23 @@ env ||= 'development'
 logger = Logger.new(STDOUT)
 
 GEM_NAME='polisher'
-PKG_VERSION='0.3'
+PKG_VERSION='0.4'
 
 namespace :db do
+  desc "Migrate the database"
   task :migrate do
-    desc "Migrate the database"
     Polisher::DB.connect Polisher::DB.load_config('./config/database.yml', env), logger
     Polisher::DB.migrate './db/migrations'
   end
 
+  desc "Rollback the database"
   task :rollback do
-    desc "Rollback the database"
     Polisher::DB.connect Polisher::DB.load_config('./config/database.yml', env), logger
     Polisher::DB.rollback './db/migrations'
   end
 
+  desc "Drop all tables in the database"
   task :drop_tables do
-    desc "Drop all tables in the database"
     Polisher::DB.connect Polisher::DB.load_config('./config/database.yml', env), logger
     Polisher::DB.drop_tables './db/migrations'
   end
@@ -46,24 +46,14 @@ Spec::Rake::SpecTask.new('spec' => ['test_env', 'db:drop_tables', 'db:migrate'])
   t.spec_files = FileList['spec/*_spec.rb']
 end
 
-task :rdoc do
-  desc "Create RDoc documentation"
-  system "rdoc --title 'Polisher documentation' lib/"
-end
-
 Rake::RDocTask.new do |rd|
     rd.main = "README.rdoc"
     rd.rdoc_dir = "doc/site/api"
-    rd.rdoc_files.include("README.rdoc", "polisher.rb", "db/**/*.rb", "lib/**/*.rb")
+    rd.rdoc_files.include("README.rdoc", "polisher.rb", "db/models/*.rb", "lib/**/*.rb")
 end
 
-task :create_gem do
-  desc "Create a new gem"
-  system "gem build polisher.gemspec"
-end
-
-PKG_FILES = FileList['bin/*', 'config/*.yml', 'config.ru', 'COPYING', 
-'db/**/*.rb', 'lib/**/*.rb', 'LICENSE', 'polisher.rb', 'public/**/*', 'Rakefile', 
+PKG_FILES = FileList['bin/*', 'config/*.yml', 'config.ru', 'COPYING',
+'db/**/*.rb', 'lib/**/*.rb', 'LICENSE', 'polisher.rb', 'public/**/*', 'Rakefile',
 'README.rdoc', 'spec/**/*.rb', 'TODO', 'views/**/*.haml']
 
 DIST_FILES = FileList[
@@ -78,11 +68,23 @@ SPEC = Gem::Specification.new do |s|
     s.required_ruby_version = '>= 1.8.1'
     s.required_rubygems_version = Gem::Requirement.new(">= 1.3.3")
 
+    s.add_dependency('sinatra',       '>= 0.9.4')
+    s.add_dependency('thin',          '>= 1.2.7')
+    s.add_dependency('activerecord',  '>= 2.3.5')
+    s.add_dependency('haml',          '>= 2.2.20')
+    s.add_dependency('curb',          '>= 0.6.7') # eg curl
+    s.add_dependency('libxml-ruby',   '>= 1.1.3')
+    s.add_dependency('rest-client',   '>= 1.4.2')
+    s.add_dependency('json_pure',     '>= 1.2.0')
+    s.add_dependency('gem2rpm',       '>= 0.6.0')
+    s.add_dependency('rspec',         '>= 1.3.0')
+    s.add_dependency('rack-test',     '>= 0.5.3')
+
     s.author = "Mohammed Morsi"
     s.email = "mmorsi@redhat.com"
     s.date = %q{2010-03-11}
-    s.description = "Ruby gem polisher"
-    s.summary = "Ruby gem polisher"
+    s.summary = "A project release management tool"
+    s.description = "Polisher provides simple REST and DSL interfaces allowing you to configure event workflows to be invoked on specific versions of project/source releases"
     s.homepage = "http://github.com/movitto/polisher"
 end
 

data/TODO

@@ -1,6 +1,7 @@
 - Remaining TODO's, FIXME's, etc
-- Syncronization locks needed! (on all shared artifacts, db objects)
-- More event handlers, generic handler interface, 
+- Parallelism w/ syncronization locks needed! (on all shared artifacts, db objects)
+- More event handlers, generic handler interface,
   workflow integration to automatically chain handlers together
-- multiple user support, RBAC and ACL mechanisms
-- simple command line utility / api interface
+- Generic source download interface, easily be able to download sources via and
+  number of mechanisms, and perform pre-release transformations (taring a source
+  checkout for example)

data/bin/server

@@ -1,3 +1,4 @@
 #!/bin/sh
+mkdir -p db/data/
 rake db:migrate
 thin start --debug -R config.ru 

data/config/polisher.yml

@@ -1,20 +1,5 @@
 development:
-  gem_api_key:       "YOUR GEMCUTTER API KEY HERE"
-  email_from:        "foo@example.host"
-  email_server:      "localhost"
-  email_subject:     "gem #{gem_name} updated"
-  email_body:        "gem #{gem_name} updated"
 
 test:
-  gem_api_key:       "YOUR GEMCUTTER API KEY HERE"
-  email_from:        "foo@example.host"
-  email_server:      "localhost"
-  email_subject:     "gem #{gem_name} updated"
-  email_body:        "gem #{gem_name} updated"
 
 production:
-  gem_api_key:       "YOUR GEMCUTTER API KEY HERE"
-  email_from:        "foo@example.host"
-  email_server:      "localhost"
-  email_subject:     "gem #{gem_name} updated"
-  email_body:        "gem #{gem_name} updated"

data/db/connection.rb

@@ -7,9 +7,9 @@
 # it under the terms of the GNU Affero General Public License
 # as published by the Free Software Foundation, either version 3
 # of the License, or (at your option) any later version.
-# 
+#
 # You should have received a copy of the the GNU Affero
-# General Public License, along with Polisher. If not, see 
+# General Public License, along with Polisher. If not, see
 # <http://www.gnu.org/licenses/>
 
 require 'active_record'
@@ -18,28 +18,31 @@ Dir[File.dirname(__FILE__) + '/models/*.rb'].each { |model| require model }
 
 module Polisher
   module DB
-  
+
+    # Load db config from specified file / environment
     def self.load_config(file_path, env = 'development')
       YAML::load(File.open(file_path))[env.to_s]
     end
-  
+
+    # Connect to database specified in config
     def self.connect(config, logger)
       ActiveRecord::Base.logger = logger
       ActiveRecord::Base.establish_connection config
     end
-  
+
+    # Perform any outstanding db migrations
     def self.migrate(migrations_dir)
       ActiveRecord::Migration.verbose = true
       ActiveRecord::Migrator.migrate(migrations_dir)
-    end 
-  
-    # perform a single polisher db rollback
+    end
+
+    # Perform a single polisher db rollback
     def self.rollback(migrations_dir)
       ActiveRecord::Migration.verbose = true
       ActiveRecord::Migrator.rollback(migrations_dir)
-    end 
+    end
 
-    # drop all tables in the db
+    # Drop all tables in the db
     def self.drop_tables(migrations_dir)
       ActiveRecord::Migration.verbose = true
       ActiveRecord::Migrator.down(migrations_dir, 0)

data/db/migrations/{002_create_managed_gems.rb → 001_create_projects.rb}

@@ -10,15 +10,14 @@
 # General Public License, along with Polisher. If not, see 
 # <http://www.gnu.org/licenses/>
 
-class CreateManagedGems < ActiveRecord::Migration
+class CreateProjects < ActiveRecord::Migration
   def self.up
-    create_table :managed_gems do |t|
-      t.string        :name
-      t.references    :source
+    create_table :projects do |t|
+      t.string :name
     end
   end
 
   def self.down
-    drop_table :managed_gems
+    drop_table :projects
   end
 end

data/db/migrations/{001_create_sources.rb → 002_create_sources.rb}

@@ -14,11 +14,9 @@ class CreateSources < ActiveRecord::Migration
   def self.up
     create_table :sources do |t|
       t.string :name
+      t.string :source_type
       t.string :uri
     end
-
-    # create entry for the official gemcutter repo
-    Source.create :name => "gemcutter", :uri => "http://gemcutter.org"
   end
 
   def self.down

data/db/migrations/003_create_project_source_versions.rb

@@ -0,0 +1,28 @@
+# Copyright (C) 2010 Red Hat, Inc.
+# Written by Mohammed Morsi <mmorsi@redhat.com>
+#
+# This program is free software, you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License
+# as published by the Free Software Foundation, either version 3
+# of the License, or (at your option) any later version.
+# 
+# You should have received a copy of the the GNU Affero
+# General Public License, along with Polisher. If not, see 
+# <http://www.gnu.org/licenses/>
+
+class CreateProjectSourceVersions < ActiveRecord::Migration
+  def self.up
+    create_table :project_source_versions do |t|
+      t.references    :project
+      t.references    :source
+      t.boolean       :primary_source, :default => false
+      t.string        :project_version
+      t.string        :source_version
+      t.string        :source_uri_params
+    end
+  end
+
+  def self.down
+    drop_table :project_source_versions
+  end
+end

data/db/migrations/{003_create_events.rb → 004_create_events.rb}

@@ -13,9 +13,9 @@
 class CreateEvents < ActiveRecord::Migration
   def self.up
     create_table :events do |t|
-      t.references :managed_gem
+      t.references :project
       t.string     :process
-      t.string     :gem_version
+      t.string     :version
       t.string     :version_qualifier
       t.string     :process_options
     end

data/db/migrations/005_create_project_dependencies.rb

@@ -0,0 +1,28 @@
+# Copyright (C) 2010 Red Hat, Inc.
+# Written by Mohammed Morsi <mmorsi@redhat.com>
+#
+# This program is free software, you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License
+# as published by the Free Software Foundation, either version 3
+# of the License, or (at your option) any later version.
+# 
+# You should have received a copy of the the GNU Affero
+# General Public License, along with Polisher. If not, see 
+# <http://www.gnu.org/licenses/>
+
+class CreateProjectDependencies < ActiveRecord::Migration
+  def self.up
+    create_table :project_dependencies do |t|
+      t.references    :project
+      t.string        :project_version, :default => nil
+
+      t.integer       :depends_on_project_id
+      t.string        :depends_on_project_version, :default => nil
+      t.string        :depends_on_project_params
+    end
+  end
+
+  def self.down
+    drop_table :project_dependencies
+  end
+end

data/db/models/event.rb

@@ -10,32 +10,48 @@
 # General Public License, along with Polisher. If not, see 
 # <http://www.gnu.org/licenses/>
 
+require 'lib/event_handlers'
+
 class Event < ActiveRecord::Base
-   belongs_to :managed_gem
-   alias :gem :managed_gem
+   include EventHandlers
+
+   belongs_to :project
 
-   validates_presence_of   :managed_gem_id, :process
+   validates_presence_of :project_id
+   validates_presence_of :process
 
-   # TODO right mow just returning a fixed list, at some point dynamically generate
+   # Dynamically generate from methods we define in the lib/event_handlers module.
+   # Add more methods to that module for them to appear here
    def self.processes
-      ["create_package", "update_repo", "run_test_suite", "notify_subscribers"]
+     EventHandlers.public_instance_methods.collect { |m| m.to_s }
    end
 
-   #  version qualifiers
+   # XXX FIXME we need this for security
+   #validates_inclusion_of :process, :in => Event.processes
+
+   # Version qualifiers
    VERSION_QUALIFIERS = ['', '=', '>', '<', '>=', '<=']
 
    validates_inclusion_of :version_qualifier, :in => VERSION_QUALIFIERS, 
                           :if => Proc.new { |e| !e.version_qualifier.nil? }
 
-   validates_presence_of :gem_version,
+   # nil version and version_qualifier indicates event will be run for _all_ versons
+
+   validates_presence_of :version,
                          :if => Proc.new { |e| !e.version_qualifier.nil? }
 
    validates_presence_of :version_qualifier,
-                         :if => Proc.new { |e| !e.gem_version.nil? }
+                         :if => Proc.new { |e| !e.version.nil? }
+
+   # Determine if event applies to specified version
+   def applies_to_version?(cversion)
+     raise ArgumentError, "valid event version #{version} and version #{cversion} required" unless (version.nil? || version.class == String) && cversion.class == String
+
+     # XXX remove any non-numeric chars from the version number (eg if a version has a '-beta' or whatnot, not pretty but works for now
+     cversion.gsub(/[a-zA-Z]*/, '')
 
-   # determine if event applies to specified version
-   def applies_to_version?(version)
-     gv, ev = version.to_f, gem_version.to_f
+     # TODO this will evaluate to false "1.1" = "1.1.0" here, is this correct? (implement this in a more robust way, eg split version into array around delims, compare each array elements w/ special case handling)
+     gv, ev = cversion, version
      return (["", nil].include? version_qualifier ) ||
             (version_qualifier == "="  && gv == ev) ||
             (version_qualifier == ">"  && gv >  ev) ||
@@ -44,18 +60,28 @@ class Event < ActiveRecord::Base
             (version_qualifier == "<=" && gv <= ev)
    end
 
-   # run the event
-   def run
-      # covert process to method name
-      handler = method(process.intern)
+   # Run the event, :version should be passed in via args; any other keys/values are optional
+   # and will be forwarded onto the event handler
+   def run(args = {})
+      version = args[:version]
+      raise ArgumentError, "must specify version when running event" if version.nil?
 
-      # generate array of params from semi-colon seperated options
-      params  = process_options.nil? ? [] : process_options.split(';')
+      handler = nil
+      begin
+        # covert process to method name
+        handler = method(process.intern)
+      rescue NameError => e
+        raise ArgumentError, "could not find event handler #{process}"
+      end
 
-      # first param is always the gem
-      params.unshift gem
+      # generate array of event params from project, the event itself, the version being run, and any optional params passed in
+      event_params  = [self, version, args]
 
-      # invoke
-      handler.call *params
+      begin
+        # invoke
+        handler.call *event_params
+      rescue Exception => e
+        raise RuntimeError, "error when running event handler #{process}: #{e}"
+      end
    end
 end