kpm 0.0.15 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f98562e91b2bb5a44b0d93bcaa886991bf09e0fd
-  data.tar.gz: 0d539a26c054a22aa1c13b8146bc13f642b346e2
+  metadata.gz: cfdf51795bb3bfc5198c442337ecd902616e357c
+  data.tar.gz: 5221a817f4b30fc91fa9214c582136d5f00e7532
 SHA512:
-  metadata.gz: a1acc6025cfdec57db5b56f6267fa14f7a62ec86cfc2ed9b7be88a42c3786d4ad2f572f6d74277259507ee7323dc4e3ab944230f8e32201bccf770ff26e000a5
-  data.tar.gz: e6881c699c95ff89b0303f032954ca9bb315f272d5c5e79c0d9ae879165030381d613e61cba295b1a21a72bb3d40b99816c231b2e56209a72207d9c0f1465246
+  metadata.gz: 3995b7421e68b14b4e932e27833202f49dc503433536a314a7a72d26b24b021b8dce192ce3fca680ac36a088cfe6eec04dd45e93ebdf74a4344f68c69eb4f287
+  data.tar.gz: 4087053ae2f9ce4673e8dfa3635331613242e8328bd98edb353c60a0d112806510971324e6b4572320ae2ec934a3c60a2361fb3df3ba7632d494c4edad905046

data/README.md

@@ -1,6 +1,6 @@
 # KPM: the Kill Bill Package Manager
 
-The goal of KPM is to facilitate the installation of Kill Bill and its plugins.
+The goal of KPM is to facilitate the installation of Kill Bill, its plugins and Kaui.
 
 kpm can be used interactively to search and download individual artifacts (Kill Bill war, plugins, etc.) or to perform an automatic Kill Bill installation using a configuration file.
 
@@ -12,55 +12,76 @@ Ruby 2.1+ or JRuby 1.7.11+ is recommended.
 
 ## Quick start
 
-Create a kpm.yml file as follows:
+The following commands
+
+    mkdir killbill
+    cd killbill
+    kpm install
+
+will setup Kill Bill and Kaui, i.e.:
+
+* Tomcat (open-source Java web server) is setup in the `killbill` directory
+* The Kill Bill application (war) is installed in the `killbill/webapps` directory
+* The Kill Bill UI (Kaui war) is installed in the `killbill/webapps` directory
+* Default plugins are installed in the `/var/tmp/bundles` directory, among them:
+ * `jruby.jar`, required to run Ruby plugins
+ * the [KPM plugin](https://github.com/killbill/killbill-kpm-plugin), required to (un-)install plugins at runtime
+
+To start Kill Bill, simply run
+
+    ./bin/catalina.sh run
+
+You can then verify Kill Bill is running by going to http://127.0.0.1:8080/kaui.
+
+## Configuration
+
+You can configure the installation process using a kpm.yml file.
+
+For example:
 
     killbill:
-      version: 0.12.1
-      webapp_path: /opt/tomcat/webapps/ROOT
-      plugins_dir: /var/tmp/bundles
+      version: 0.14.0
       plugins:
         java:
-          - name: analytics-plugin
-            version: 0.7.2
+          - name: analytics
         ruby:
-          - name: stripe-plugin
-            version: 0.2.1
-    kaui:
-      version: LATEST
-      webapp_path: /opt/tomcat/webapps/kaui
+          - name: stripe
 
 This instructs kpm to:
-* Download the Kill Bill war (version 0.12.1) and install it as `/opt/tomcat/webapps/ROOT`
-* Setup the Analytics (Java) plugin (version 0.7.2) and the Stripe (Ruby) plugin (version 0.2.1) under `/var/tmp/bundles`
-* Download the latest Kaui war and install it as `/opt/tomcat/webapps/kaui`
+
+* Download Kill Bill version 0.14.0
+* Setup the Analytics (Java) plugin and the Stripe (Ruby) plugin
 
 To start the installation:
 
     kpm install kpm.yml
 
-To help you with discovery of plugins, you can run
+Common configuration options:
 
-    kpm search_for_plugins
+* `jvm`: JVM properties
+* `killbill`: Kill Bill properties
+* `plugins_dir`: OSGI bundles and plugins base directory
+* `webapp_path`: path for the Kill Bill war (if specified, Tomcat isn't downloaded)
 
-This will list available (official) plugins. We maintain a list of recommended versions to use at https://github.com/killbill/killbill-cloud/blob/master/kpm/lib/kpm/plugins_directory.yml.
+There are many more options you can specify. Take a look at the configuration file used in the [Docker](https://github.com/killbill/killbill-cloud/blob/master/docker/templates/killbill/latest/kpm.yml.erb) image for example.
 
 ## Internals and Advanced Commands
 
-
 ### Caching
 
 KPM relies on the kpm.yml file to know what to install, and as it installs the pieces it keeps track of what was installed so that if it is invoked again, it does not download again the same binaries:
-* For the kilbill artifact itself, kpm will first extract the sha1 from the remote repository and compare with the cuurent sha1 installed under the default `webapp_path`; if those are the same, the file is not downloaded again.
+
+* For the Kill Bill artifact itself, kpm will first extract the sha1 from the remote repository and compare with the current sha1 installed under the default `webapp_path`; if those are the same, the file is not downloaded again.
 * For the plugins, since those are downloaded as an archive (*.tgz) and then decompressed/expanded, kpm will use an internal file, `<plugins_dir>/sha1.yml` to keep track of the sha1 archive. If there is an entry and the sha1 matches, then the file is not downloaded again.
 
 Note that you can override that behavior with the `--force-download` switch.
 
 ### Custom Downloads
 
-Note that you can also download specific versions/artifacts directly with the followinhg commands -- bypassing the kpm.yml file:
+Note that you can also download specific versions/artifacts directly with the following commands -- bypassing the kpm.yml file:
+
 * `kpm pull_kaui_war <version>`
 * `kpm pull_kb_server_war <version>`
 * `kpm pull_ruby_plugin <artifact_id>`
 
 For more details see `kpm --help`.
-

data/Rakefile

@@ -7,14 +7,20 @@ Bundler::GemHelper.install_tasks
 # Install test tasks
 require 'rspec/core/rake_task'
 namespace :test do
+  desc 'Run RSpec tests'
+  RSpec::Core::RakeTask.new do |task|
+    task.name = 'spec'
+    task.pattern = './spec/*/unit/*_spec.rb'
+  end
+
   namespace :remote do
     desc 'Run RSpec remote tests'
     RSpec::Core::RakeTask.new do |task|
-      task.name    = 'spec'
+      task.name = 'spec'
       task.pattern = './spec/*/remote/*_spec.rb'
     end
   end
 end
 
 # Run tests by default
-task :default => 'test:remote:spec'
+task :default => 'test:spec'

data/lib/kpm.rb

@@ -2,9 +2,12 @@ module KPM
   autoload :Utils, 'kpm/utils'
   autoload :BaseArtifact, 'kpm/base_artifact'
   autoload :Sha1Checker, 'kpm/sha1_checker'
+  autoload :TomcatManager, 'kpm/tomcat_manager'
   autoload :KillbillServerArtifact, 'kpm/killbill_server_artifact'
   autoload :KillbillPluginArtifact, 'kpm/killbill_plugin_artifact'
   autoload :KauiArtifact, 'kpm/kaui_artifact'
+  autoload :PluginsManager, 'kpm/plugins_manager'
+  autoload :BaseInstaller, 'kpm/base_installer'
   autoload :Installer, 'kpm/installer'
   autoload :Tasks, 'kpm/tasks'
   autoload :Cli, 'kpm/cli'
@@ -19,4 +22,4 @@ module KPM
       @ui ||= Thor::Shell::Color.new
     end
   end
-end
+end

data/lib/kpm/base_artifact.rb

@@ -40,6 +40,10 @@ module KPM
         pull_and_put_in_place(logger, coordinates, destination_path, is_ruby_plugin_and_should_skip_top_dir(group_id, artifact_id), sha1_file, force_download, verify_sha1, overrides, ssl_verify)
       end
 
+      def pull_from_fs(logger, file_path, destination_path=nil)
+        pull_from_fs_and_put_in_place(logger, file_path, destination_path)
+      end
+
       def nexus_remote(overrides={}, ssl_verify=true)
         nexus_remote ||= NexusCli::RemoteFactory.create(nexus_defaults.merge(overrides || {}), ssl_verify)
       end
@@ -54,38 +58,60 @@ module KPM
       protected
 
       def pull_and_put_in_place(logger, coordinates, destination_path=nil, skip_top_dir=true, sha1_file=nil, force_download=false, verify_sha1=true, overrides={}, ssl_verify=true)
-        destination_path = KPM::root if destination_path.nil?
+        # Build artifact info
+        artifact_info = artifact_info(coordinates, overrides, ssl_verify)
 
-        # Create the destination directory
-        if path_looks_like_a_directory(destination_path)
-          destination_dir = destination_path
-        else
-          destination_dir = File.dirname(destination_path)
-        end
-        FileUtils.mkdir_p(destination_dir)
+        populate_fs_info(artifact_info, destination_path)
 
-        # Build artifact info
-        artifact_info = artifact_info(coordinates, destination_path, overrides, ssl_verify)
+        # Return early if there's nothing to do
         if !force_download && skip_if_exists(artifact_info, coordinates, sha1_file)
-          logger.info "Skipping installation of #{coordinates} to #{artifact_info[:file_path]}, file already exists"
+          logger.info "  Skipping installation of #{coordinates} to #{artifact_info[:file_path]}, file already exists"
           artifact_info[:skipped] = true
           return artifact_info
         end
 
+        # Create the destination directory
+        FileUtils.mkdir_p(artifact_info[:dir_name])
+
         # Download the artifact in a temporary directory in case of failures
         Dir.mktmpdir do |tmp_destination_dir|
           logger.info "      Starting download of #{coordinates} to #{tmp_destination_dir}"
 
-          downloaded_artifact_info  = pull_and_verify(logger, artifact_info[:sha1], coordinates, tmp_destination_dir, sha1_file, verify_sha1, overrides, ssl_verify)
+          downloaded_artifact_info = pull_and_verify(logger, artifact_info[:sha1], coordinates, tmp_destination_dir, sha1_file, verify_sha1, overrides, ssl_verify)
           if artifact_info[:is_tgz]
-            Utils.unpack_tgz(downloaded_artifact_info[:file_path], destination_path, skip_top_dir)
+            artifact_info[:bundle_dir] = Utils.unpack_tgz(downloaded_artifact_info[:file_path], artifact_info[:dir_name], skip_top_dir)
             FileUtils.rm downloaded_artifact_info[:file_path]
           else
-            FileUtils.mv downloaded_artifact_info[:file_path], destination_path
+            FileUtils.mv downloaded_artifact_info[:file_path], artifact_info[:file_path]
+            artifact_info[:bundle_dir] = artifact_info[:dir_name]
             artifact_info[:size] = downloaded_artifact_info[:size]
           end
-          logger.info "Successful installation of #{coordinates} to #{artifact_info[:file_path]}"
+          logger.info "Successful installation of #{coordinates} to #{artifact_info[:bundle_dir]}"
+        end
+        artifact_info
+      end
+
+      # Logic similar than pull_and_put_in_place above
+      def pull_from_fs_and_put_in_place(logger, file_path, destination_path=nil)
+        artifact_info = {
+            :skipped => false,
+            :repository_path => file_path,
+            :is_tgz => file_path.end_with?('.tar.gz') || file_path.end_with?('.tgz')
+        }
+
+        populate_fs_info(artifact_info, destination_path)
+
+        # Create the destination directory
+        FileUtils.mkdir_p(artifact_info[:dir_name])
+
+        if artifact_info[:is_tgz]
+          artifact_info[:bundle_dir] = Utils.unpack_tgz(file_path, artifact_info[:dir_name], true)
+        else
+          FileUtils.cp file_path, artifact_info[:dir_name]
+          artifact_info[:bundle_dir] = artifact_info[:dir_name]
         end
+        logger.info "Successful installation of #{file_path} to #{artifact_info[:bundle_dir]}"
+
         artifact_info
       end
 
@@ -112,33 +138,50 @@ module KPM
         local_sha1 == artifact_info[:sha1]
       end
 
+      def artifact_info(coordinates, overrides={}, ssl_verify=true)
+        info = {
+            :skipped => false
+        }
 
-      def artifact_info(coordinates, destination_path, overrides={}, ssl_verify=true)
-
-        info = {}
         nexus_info = nexus_remote(overrides, ssl_verify).get_artifact_info(coordinates)
 
         xml = REXML::Document.new(nexus_info)
-        repository_path = xml.elements['//repositoryPath'].text unless xml.elements['//repositoryPath'].nil?
-        sha1 = xml.elements['//sha1'].text unless xml.elements['//sha1'].nil?
-        version = xml.elements['//version'].text unless xml.elements['//version'].nil?
-
-        info[:sha1] = sha1
-        info[:version] = version
-        info[:is_tgz] = repository_path.end_with?('.tar.gz') || repository_path.end_with?('.tgz')
-        if File.directory?(destination_path) && !info[:is_tgz]
-          destination = File.join(File.expand_path(destination_path), File.basename(repository_path))
-          info[:file_name] = File.basename(repository_path)
+        info[:sha1] = xml.elements['//sha1'].text unless xml.elements['//sha1'].nil?
+        info[:version] = xml.elements['//version'].text unless xml.elements['//version'].nil?
+        info[:repository_path] = xml.elements['//repositoryPath'].text unless xml.elements['//repositoryPath'].nil?
+        info[:is_tgz] = info[:repository_path].end_with?('.tar.gz') || info[:repository_path].end_with?('.tgz')
+
+        info
+      end
+
+      def update_destination_path(info, destination_path)
+        # In case LATEST was specified, use the actual version as the directory name
+        destination_path = KPM::root if destination_path.nil?
+        plugin_dir, version_dir = File.split(destination_path)
+        destination_path = Pathname.new(plugin_dir).join(info[:version]).to_s if version_dir == 'LATEST' && !info[:version].nil?
+        destination_path
+      end
+
+      def populate_fs_info(info, destination_path)
+        destination_path = update_destination_path(info, destination_path)
+
+        if path_looks_like_a_directory(destination_path) && !info[:is_tgz]
+          info[:dir_name] = File.expand_path(destination_path)
+          info[:file_name] = File.basename(info[:repository_path])
+          info[:file_path] = File.expand_path(File.join(info[:dir_name], File.basename(info[:repository_path])))
         else
           # The destination was a fully specified path or this is an archive and we keep the directory
-          destination = destination_path
-          info[:file_name] = File.basename(destination_path) if !info[:is_tgz]
+          if info[:is_tgz]
+            info[:dir_name] = File.expand_path(destination_path)
+          else
+            info[:dir_name] = File.dirname(destination_path)
+            info[:file_name] = File.basename(destination_path)
+          end
+          info[:file_path] = File.expand_path(destination_path)
         end
-        info[:file_path] = File.expand_path(destination)
-        info[:skipped] = false
-        info
-      end
 
+        destination_path
+      end
 
       def pull_and_verify(logger, remote_sha1, coordinates, destination_dir, sha1_file, verify_sha1, overrides={}, ssl_verify=true)
         info = nexus_remote(overrides, ssl_verify).pull_artifact(coordinates, destination_dir)

data/lib/kpm/base_installer.rb

@@ -0,0 +1,183 @@
+require 'pathname'
+
+module KPM
+  class BaseInstaller
+
+    LATEST_VERSION = 'LATEST'
+    SHA1_FILENAME = 'sha1.yml'
+    DEFAULT_BUNDLES_DIR = Pathname.new('/var').join('tmp').join('bundles').to_s
+
+    def initialize(logger, nexus_config, nexus_ssl_verify)
+      @logger = logger
+      @nexus_config = nexus_config
+      @nexus_ssl_verify = nexus_ssl_verify
+    end
+
+    def install_killbill_server(specified_group_id=nil, specified_artifact_id=nil, specified_packaging=nil, specified_classifier=nil, specified_version=nil, specified_webapp_path=nil, force_download=false, verify_sha1=true)
+      group_id = specified_group_id || KPM::BaseArtifact::KILLBILL_GROUP_ID
+      artifact_id = specified_artifact_id || KPM::BaseArtifact::KILLBILL_ARTIFACT_ID
+      packaging = specified_packaging || KPM::BaseArtifact::KILLBILL_PACKAGING
+      classifier = specified_classifier || KPM::BaseArtifact::KILLBILL_CLASSIFIER
+      version = specified_version || LATEST_VERSION
+      webapp_path = specified_webapp_path || KPM::root
+
+      @logger.debug("Installing Kill Bill server: group_id=#{group_id} artifact_id=#{artifact_id} packaging=#{packaging} classifier=#{classifier} version=#{version} webapp_path=#{webapp_path}")
+      KPM::KillbillServerArtifact.pull(@logger,
+                                       group_id,
+                                       artifact_id,
+                                       packaging,
+                                       classifier,
+                                       version,
+                                       webapp_path,
+                                       nil,
+                                       force_download,
+                                       verify_sha1,
+                                       @nexus_config,
+                                       @nexus_ssl_verify)
+    end
+
+    def install_kaui(specified_group_id=nil, specified_artifact_id=nil, specified_packaging=nil, specified_classifier=nil, specified_version=nil, specified_webapp_path=nil, force_download=false, verify_sha1=true)
+      group_id = specified_group_id || KPM::BaseArtifact::KAUI_GROUP_ID
+      artifact_id = specified_artifact_id || KPM::BaseArtifact::KAUI_ARTIFACT_ID
+      packaging = specified_packaging || KPM::BaseArtifact::KAUI_PACKAGING
+      classifier = specified_classifier || KPM::BaseArtifact::KAUI_CLASSIFIER
+      version = specified_version || LATEST_VERSION
+      webapp_path = specified_webapp_path || KPM::root
+
+      @logger.debug("Installing Kaui: group_id=#{group_id} artifact_id=#{artifact_id} packaging=#{packaging} classifier=#{classifier} version=#{version} webapp_path=#{webapp_path}")
+      KPM::KauiArtifact.pull(@logger,
+                             group_id,
+                             artifact_id,
+                             packaging,
+                             classifier,
+                             version,
+                             webapp_path,
+                             nil,
+                             force_download,
+                             verify_sha1,
+                             @nexus_config,
+                             @nexus_ssl_verify)
+    end
+
+    def install_plugin(specified_group_id, specified_artifact_id, specified_packaging=nil, specified_classifier=nil, specified_version=nil, bundles_dir=nil, specified_type='java', force_download=false, verify_sha1=true)
+      looked_up_group_id, looked_up_artifact_id, looked_up_packaging, looked_up_classifier, looked_up_version, looked_up_type = KPM::PluginsDirectory.lookup(specified_artifact_id, true)
+
+      # Specified parameters have always precedence except for the artifact_id (to map stripe to stripe-plugin)
+      artifact_id = looked_up_artifact_id || specified_artifact_id
+      if artifact_id.nil?
+        @logger.warn("Aborting installation: unable to lookup plugin #{specified_artifact_id}")
+        return nil
+      end
+
+      bundles_dir = Pathname.new(bundles_dir || DEFAULT_BUNDLES_DIR).expand_path
+      plugins_dir = bundles_dir.join('plugins')
+
+      type = specified_type || looked_up_type
+      if type.to_s == 'java'
+        group_id = specified_group_id || looked_up_group_id || KPM::BaseArtifact::KILLBILL_JAVA_PLUGIN_GROUP_ID
+        packaging = specified_packaging || looked_up_packaging || KPM::BaseArtifact::KILLBILL_JAVA_PLUGIN_PACKAGING
+        classifier = specified_classifier || looked_up_classifier || KPM::BaseArtifact::KILLBILL_JAVA_PLUGIN_CLASSIFIER
+        version = specified_version || looked_up_version || LATEST_VERSION
+        destination = plugins_dir.join('java').join(artifact_id).join(version)
+      else
+        group_id = specified_group_id || looked_up_group_id || KPM::BaseArtifact::KILLBILL_RUBY_PLUGIN_GROUP_ID
+        packaging = specified_packaging || looked_up_packaging || KPM::BaseArtifact::KILLBILL_RUBY_PLUGIN_PACKAGING
+        classifier = specified_classifier || looked_up_classifier || KPM::BaseArtifact::KILLBILL_RUBY_PLUGIN_CLASSIFIER
+        version = specified_version || looked_up_version || LATEST_VERSION
+        destination = plugins_dir.join('ruby')
+      end
+      sha1_file = "#{bundles_dir}/#{SHA1_FILENAME}"
+
+      @logger.debug("Installing plugin: group_id=#{group_id} artifact_id=#{artifact_id} packaging=#{packaging} classifier=#{classifier} version=#{version} destination=#{destination}")
+      artifact_info = KPM::KillbillPluginArtifact.pull(@logger,
+                                                       group_id,
+                                                       artifact_id,
+                                                       packaging,
+                                                       classifier,
+                                                       version,
+                                                       destination,
+                                                       sha1_file,
+                                                       force_download,
+                                                       verify_sha1,
+                                                       @nexus_config,
+                                                       @nexus_ssl_verify)
+
+      mark_as_active(plugins_dir, artifact_info, artifact_id)
+
+      artifact_info
+    end
+
+    def install_plugin_from_fs(file_path, name, version, bundles_dir=nil, type='java')
+      bundles_dir = Pathname.new(bundles_dir || DEFAULT_BUNDLES_DIR).expand_path
+      plugins_dir = bundles_dir.join('plugins')
+
+      if type.to_s == 'java'
+        destination = plugins_dir.join('java').join(name).join(version)
+      else
+        destination = plugins_dir.join('ruby')
+      end
+
+      artifact_info = KPM::KillbillPluginArtifact.pull_from_fs(@logger, file_path, destination)
+      artifact_info[:version] ||= version
+
+      mark_as_active(plugins_dir, artifact_info)
+
+      artifact_info
+    end
+
+    def install_default_bundles(bundles_dir, specified_version=nil, kb_version=nil, force_download=false, verify_sha1=true)
+      group_id = 'org.kill-bill.billing'
+      artifact_id = 'killbill-platform-osgi-bundles-defaultbundles'
+      packaging = 'tar.gz'
+      classifier = nil
+
+      version = specified_version
+      if version.nil?
+        info = KPM::KillbillServerArtifact.info(kb_version || LATEST_VERSION, @nexus_config, @nexus_ssl_verify)
+        version = info['killbill-platform']
+      end
+      version ||= LATEST_VERSION
+
+      bundles_dir = Pathname.new(bundles_dir || DEFAULT_BUNDLES_DIR).expand_path
+      destination = bundles_dir.join('platform')
+      sha1_file = bundles_dir.join(SHA1_FILENAME)
+
+      @logger.debug("Installing default bundles: group_id=#{group_id} artifact_id=#{artifact_id} packaging=#{packaging} classifier=#{classifier} version=#{version} destination=#{destination}")
+      info = KPM::BaseArtifact.pull(@logger,
+                                    group_id,
+                                    artifact_id,
+                                    packaging,
+                                    classifier,
+                                    version,
+                                    destination,
+                                    sha1_file,
+                                    force_download,
+                                    verify_sha1,
+                                    @nexus_config,
+                                    @nexus_ssl_verify)
+
+      # The special JRuby bundle needs to be called jruby.jar
+      # TODO .first - code smell
+      unless info[:skipped]
+        File.rename Dir.glob("#{destination}/killbill-platform-osgi-bundles-jruby-*.jar").first, destination.join('jruby.jar')
+      end
+
+      info
+    end
+
+    private
+
+    def mark_as_active(plugins_dir, artifact_info, artifact_id=nil)
+      # Mark this bundle as active
+      plugins_manager = PluginsManager.new(plugins_dir, @logger)
+      if artifact_info[:bundle_dir].nil?
+        # In case the artifact on disk already existed and the installation is skipped,
+        # we don't know the plugin name on disk (arbitrary if it's a .tar.gz). That being said,
+        # we can guess it for Kill Bill plugins (using our naming conventions)
+        plugins_manager.set_active(plugins_manager.guess_plugin_name(artifact_id), artifact_info[:version])
+      else
+        plugins_manager.set_active(artifact_info[:bundle_dir])
+      end
+    end
+  end
+end