nexus_client 0.3.0

data/README.md

@@ -0,0 +1,129 @@
+[![Build Status](https://travis-ci.org/logicminds/nexus-client.svg)](https://travis-ci.org/logicminds/nexus-client)
+
+
+# NexusClient
+The nexus client is a ruby wrapper around the nexus REST API for downloading maven artifacts.
+It features the ability to cache artifacts and also performs artifact checksums to ensure you only
+download the artifact once.  This gem was originally designed for use with configuration management software like puppet.
+
+This gem does not require maven or any of the maven settings files.  It was originally created to use with 
+configuration management software to download artifacts to hundreds of servers.  A CLI tool was created for this purpose
+to ease downloading of artifacts on any system.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'nexus_client'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install nexus_client
+
+## Features
+### Cache repository  
+The cache repository feature if enabled can be used to cache artifacts much like maven.  However since this
+gem does not use maven you are free to store the artifacts where ever you want. This works great for downloading the same
+artifact on the same system.  However, the cache feature was built to be used across a shared file system like NFS
+so if multiple systems are downloading the same artifact you can reduce the time and bandwidth needed to download large
+artifacts.  This feature alone is like having a mini nexus proxy on your network!
+
+### Automatic artifact checksums 
+This gem will grab the sha1 checksum from the nexus server and compare the checksum 
+with the downloaded artifact.  If the checksums are different and error will be raised.  If the checksums match, a file
+will be created with a extension of sha1 after the artifact is downloaded.  This sha1 file additionally contains the sha1 checksum of the file.
+This sha1 file is created as a trigger mechanism for configuration management software and also to speed up sha1 computation time of the artifact.
+```shell
+Coreys-MacBook-Pro-2:tmp$ nexus-client --nexus-host https://repository.jboss.org/nexus -e -c /tmp/cache -g 'org.glassfish.main.external:ant:4.0:central::pom' -d /tmp
+Coreys-MacBook-Pro-2:tmp$ ls -l
+   -rw-r--r--  1 user1      wheel    8853 Oct 22 14:26 ant-4.0.pom
+   -rw-r--r--  1 user1      wheel      40 Oct 22 14:26 ant-4.0.pom.sha1
+   
+Coreys-MacBook-Pro-2:tmp$ more ant-4.0.pom.sha1
+   387951c0aa333024b25085d76a8ad77441b9e55f
+```
+
+### Smart Artifact Retrieval 
+This gem will use artifact checksums to ensure the artifact is only downloaded once.  This
+is really important during configuration management runtime when the artifact downloading process is expected run multiple times.
+This is even more important when you use artifact snapshots.  Artifacts that are snapshots can contain different checksums
+at any time so its important that we download only when a new snapshot is detected by comparing the checksums.
+
+### Cache Analytics(Experimental) 
+This feature records the cache usage and shows just how much bandwidth has been saved and what
+artifacts are currently cached.  This feature is experimental and is not feature complete.  It was originally designed
+to show historically analysis of artifacts and make this information available to shell scripts, graphite and other reporting mechanisms.
+This could be used later to send alerts when artifact sizes between versions/snapshots are significantly different from each other.
+
+
+## Ruby Usage
+
+```ruby
+client = Nexus::Client.new
+client.download_gav('/tmp/ant-4.0.pom', 'org.glassfish.main.external:ant:4.0:central::pom')
+```
+or
+
+```ruby
+Nexus::Client.download_gav('/tmp/ant-4.0.pom', 'org.glassfish.main.external:ant:4.0:central::pom')
+```
+
+## CLI Usage
+We have also created a simple CLI tool that makes it easy to download artifacts from any nexus server.
+
+```shell
+nexus-client --help
+Options:
+   --destination, -d <s>:   destination directory to download file to
+    --gav-string, -g <s>:   The nexus GAV value: group:artifact:version:repository:classifier:extension
+     --cache-dir, -c <s>:   The directory to cache files to
+      --enable-cache, -e:   Enable cache
+    --nexus-host, -n <s>:   Nexus host url, if left blank reads from ~/.nexus_host
+  --enable-analytics, -a:   Enable cache analytics, requires sqlite3 gem (experimental!)
+              --help, -h:   Show this message
+              
+nexus-client --nexus-host https://repository.jboss.org/nexus -e -c /tmp/cache -g 'org.glassfish.main.external:ant:4.0:central::pom' -d /tmp
+```
+
+## Tips
+
+Create a nexus host file to store the nexus host url.  This can same time if your nexus host url is the same.  By default
+the nexus-client CLI will override the stored url in your nexus_host file when passing in the --nexus-host argument.
+
+```shell
+   Coreys-MacBook-Pro-2:~$ pwd
+   /Users/user1
+   Coreys-MacBook-Pro-2: echo 'https://repository.jboss.org/nexus' > ~/.nexus_host
+   Coreys-MacBook-Pro-2:~$ more .nexus_host
+   https://repository.jboss.org/nexus
+```
+## TODO
+* Extend usage to other supported maven server solutions (artifactory, archiva)
+* Implement a feature toggle to enable parallel downloads.
+* Finish analytics feature
+* Remove usage of typheous and other C compiled dependent gems  (conflicts with parallel feature)
+* Add feature to upload artifacts to nexus server
+* Add basic user authentication
+
+
+## OS Support
+Should work on all *nix platforms.  Windows may also be supported but it has never been tested.
+If you use this gem on windows please let me know so I can update this doc.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+
+## Examples
+
+
+

data/Rakefile

@@ -0,0 +1,12 @@
+require "bundler/gem_tasks"
+
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new('spec')
+
+task :default => :spec
+
+desc "Build gem"
+task :build do
+  `gem build nexus_client.gemspec`
+end

data/bin/nexus-client

@@ -0,0 +1,38 @@
+#!/usr/bin/env ruby
+
+require 'nexus_client'
+require 'trollop'
+require 'etc'
+
+# read host will read ~/.nexus_host file and
+def read_host(filename="#{Etc.getpwuid.dir}/.nexus_host")
+  fn = File.expand_path(filename)
+  abort("Please create the file #{filename} and add your nexus host") if not File.exists?(filename)
+  begin
+    File.open(fn, 'r') { |f| f.read }.strip
+  rescue Exception => e
+    raise(e)
+  end
+end
+
+opts = Trollop::options do
+  opt :destination, "destination directory to download file to " , :type => :string, :required => true
+  opt :gav_string, "The nexus GAV value: group:artifact:version:repository:classifier:extension" , :type => :string, :required => true
+  opt :cache_dir, "The directory to cache files to" , :type => :string
+  opt :enable_cache, "Enable cache", :default => false, :type => :boolean
+  opt :nexus_host, "Nexus host url, if left blank reads from ~/.nexus_host", :type => :string
+  opt :enable_analytics, "Enable cache analytics, requires sqlite3 gem (experimental!)", :type => :boolean, :default => false
+end
+
+if opts[:nexus_host].nil? or opts[:nexus_host].empty?
+  opts[:nexus_host] = read_host
+end
+
+value = Nexus::Client.download(opts[:destination], opts[:gav_string],
+                      opts[:cache_dir], opts[:enable_cache],
+                      opts[:enable_analytics], opts[:nexus_host])
+
+if value
+  exit(0)
+end
+exit(1)

data/lib/nexus_client.rb

@@ -0,0 +1,194 @@
+require "nexus_client/version"
+require "nexus_client/gav"
+require 'nexus_client/cache'
+require 'nexus_client/analytics'
+require "tmpdir"
+require 'typhoeus'
+require 'json'
+require 'etc'
+require 'fileutils'
+
+module Nexus
+  class Client
+    attr_reader :host, :cache
+    attr_accessor :use_cache, :log
+
+    def initialize(nexus_host=nil, cache_dir='/tmp/cache', enable_cache=true, enable_analytics=false,logger=nil)
+      @log = logger
+      @host = nexus_host || default_host
+      @host = @host.gsub(/\/nexus$/, '') # just in case user enters /nexus
+      @use_cache = enable_cache
+      if @use_cache
+        @cache_base = cache_dir
+        @cache = Nexus::Cache.new(@cache_base, enable_analytics, log)
+      end
+      #Typhoeus::Config.verbose = true
+
+    end
+
+    # read host will read ~/.nexus_host file and
+    def read_host(filename="#{Etc.getpwuid.dir}/.nexus_host")
+      fn = File.expand_path(filename)
+      abort("Please create the file #{filename} and add your nexus host") if not File.exists?(filename)
+      begin
+        File.open(fn, 'r') { |f|  f.read }.strip
+      rescue Exception => e
+        raise(e)
+      end
+    end
+
+    def default_host
+      read_host
+    end
+
+    def log
+      if @log.nil?
+        @log = Logger.new(STDOUT)
+      end
+      @log
+    end
+
+    def self.download(destination, gav_str, cache_dir='/tmp/cache', enable_cache=false,enable_analytics=false,host=nil)
+      client = Nexus::Client.new(host, cache_dir, enable_cache,enable_analytics)
+      client.download_gav(destination, gav_str)
+    end
+
+    def download_gav(destination, gav_str)
+      gav = Nexus::Gav.new(gav_str)
+      download(destination, gav)
+    end
+
+  def create_target(destination)
+    destination = File.expand_path(destination)
+    if ! File.directory?(destination)
+      begin
+        FileUtils.mkdir_p(destination) if not File.exists?(destination)
+      rescue SystemCallError => e
+        raise e, 'Cannot create directory'
+      end
+    end
+
+  end
+
+    # retrieves the attributes of the gav
+    def gav_data(gav)
+      res = {}
+      request = Typhoeus::Request.new(
+        "#{host}/nexus/service/local/artifact/maven/resolve",
+        :params  => gav.to_hash,:connecttimeout => 5,
+        :headers => { 'Accept' => 'application/json' }
+      )
+      request.on_failure do |response|
+        raise("Failed to get gav data for #{gav.to_s}")
+      end
+      request.on_complete do |response|
+        res = JSON.parse(response.response_body)
+      end
+      request.run
+
+      res['data']
+    end
+
+    # returns the sha1 of the file
+    def sha(file, use_sha_file=false)
+      if use_sha_file and File.exists?("#{file}.sha1")
+        # reading the file is faster than doing a hash, so we keep the hash in the file
+        # then we read back and compare.  There is no reason to perform sha1 everytime
+        begin
+          File.open("#{file}.sha1", 'r') { |f| f.read().strip}
+        rescue
+          Digest::SHA1.file(File.expand_path(file)).hexdigest
+        end
+      else
+        Digest::SHA1.file(File.expand_path(file)).hexdigest
+      end
+    end
+
+    # sha_match? returns bool by comparing the sha1 of the nexus gav artifact and the local file
+    def sha_match?(file, gav, use_sha_file=false)
+      if File.exists?(file)
+        if gav.sha1.nil?
+          gav.sha1 = gav_data(gav)['sha1']
+        end
+        sha(file,use_sha_file) == gav.sha1
+      else
+        false
+      end
+    end
+
+    private
+
+    # writes the sha1 a file if and only if the contents of the file do not match
+    def write_sha1(file,sha1)
+      shafile = "#{file}.sha1"
+      File.open(shafile, 'w') { |f| f.write(sha1)  }
+    end
+
+    # downloads the gav to the destination, returns the file if download was successful
+    # if cache is on then it will use the cache and if file is new will also cache the new file
+    # TODO need a timeout when host is unreachable
+    def download(destination, gav)
+      raise 'Download destination must not be empty' if destination.empty?
+      create_target(destination) # ensure directory path is created
+      destination = File.expand_path(destination)
+      if File.directory?(destination)
+        dstfile = File.expand_path("#{destination}/#{gav.filename}")
+      else
+        dstfile = File.expand_path(destination)
+      end
+      # if the file already exists at the destination path than we don't need to download it again
+      if sha_match?(dstfile, gav)
+        # create a file that stores the sha1 for faster file comparisions later
+        # This will only get created when the sha1 matches
+        write_sha1("#{dstfile}", gav.sha1)
+        if use_cache and not cache.exists?(gav)
+          cache.add_file(gav, dstfile)
+        end
+        return true
+      end
+      # remove the previous sha1 file if it already exists
+      FileUtils.rm("#{dstfile}.sha1") if File.exists?("#{dstfile}.sha1")
+
+      if gav.sha1.nil?
+        gav.sha1 = gav_data(gav)['sha1']
+      end
+      # use the cache if the file is in the cache
+      if use_cache and cache.exists?(gav)
+        cache_file_path = cache.file_path(gav)
+        FileUtils.copy(cache_file_path, dstfile)
+        cache.record_hit(gav)
+      else
+        request = Typhoeus::Request.new(
+          "#{host}/nexus/service/local/artifact/maven/redirect",
+          :params  => gav.to_hash,
+          :connecttimeout => 5,
+          :followlocation => true
+        )
+        request.on_failure do |response|
+          raise("Failed to download #{gav.to_s}")
+        end
+
+        # when complete, lets write the data to the file
+        # first lets compare the sha matches
+        # if the gav we thought we downloaded has the same checksum, were are good
+        request.on_complete do |response|
+          File.open(dstfile, 'wb') { |f| f.write(response.body) } unless ! response.success?
+          if not sha_match?(dstfile, gav, false)
+            raise("Error sha1 mismatch gav #{gav.sha1} != #{sha(dstfile)}")
+          end
+          gav.attributes[:size] =  File.size(dstfile)
+          gav.attributes[:total_time] =  response.options[:total_time]
+
+          # lets cache the file if cache is on
+          if use_cache
+            cache.add_file(gav, dstfile)
+          end
+          dstfile
+        end
+        request.run
+        dstfile
+      end
+      dstfile
+    end
+  end
+end

data/lib/nexus_client/analytics.rb

@@ -0,0 +1,187 @@
+require 'json'
+
+module Nexus
+  class Analytics
+    attr_reader :db, :data, :db_file
+    attr_accessor :a_file, :log
+
+    # new method
+    def initialize(database_dir='./',db_filename='cache-analytics.db', logger=nil)
+      @log = logger
+
+      filename ||= db_filename || 'cache-analytics.db'
+
+      log.warn "Filename is nil" if filename.nil?
+      @db_file = File.join(database_dir,filename)
+      begin
+        require 'sqlite3'
+        @db = SQLite3::Database.new( @db_file)
+        init_tables
+        total_view
+      rescue LoadError => e
+        log.error 'The sqlite3 gem must be installed before using the analytics class'
+        raise(e.message)
+      end
+    end
+
+    def log
+      if @log.nil?
+        @log = Logger.new(STDOUT)
+      end
+      @log
+    end
+
+    def add_item(gav, file_path)
+      begin
+        db.execute("insert into artifacts (sha1,artifact_gav,filesize,request_time, modified, file_location) "+
+                     "values ('#{gav.sha1}','#{gav.to_s}', #{gav.attributes[:size]}, #{gav.attributes[:total_time]},"+
+                     "'#{Time.now.to_i}', '#{file_path}')")
+      rescue
+        log.warn("Ignoring Duplicate entry #{file_path}")
+      end
+
+    end
+
+    def gavs
+      db.execute("select artifact_gav from artifacts").flatten
+    end
+
+    def update_item(gav)
+      count = hit_count(gav)
+      db.execute <<SQL
+      UPDATE artifacts SET hit_count=#{count + 1}, modified=#{Time.now.to_i}
+      WHERE sha1='#{gav.sha1}'
+SQL
+    end
+
+    def totals
+      db.execute("select * from totals")
+    end
+
+    def total(gav)
+      # TODO fix NoMethodError: undefined method `sha1' for #<String:0x7f1a0f387720>
+      # when type is not a gav or sha1 is not available
+      data = db.execute("select * from totals where sha1 = '#{gav.sha1}'")
+      db.results_as_hash = false
+      data
+    end
+
+
+    def hit_count(gav)
+      row = db.execute("select hit_count from totals where sha1 = '#{gav.sha1}'").first
+      if row.nil?
+        0
+      else
+        row.first
+      end
+    end
+
+    def total_time(gav)
+      row = db.execute("select total_time_saved from totals where sha1 = '#{gav.sha1}'").first
+      if row.nil?
+        0
+      else
+        row.first
+      end
+
+    end
+
+    def total_bytes(gav, pretty=false)
+      row = db.execute("select total_bytes_saved from totals where sha1 = '#{gav.sha1}'").first
+      if row.nil?
+        0
+      else
+        if pretty
+          Filesize.from("#{row.first} B").pretty
+        else
+          row.first
+        end
+      end
+
+    end
+
+    # returns the totals view as json
+    # the results as hash returns extra key/values we don't want so
+    # we had to create our own hash
+    # there are better ways of doing this but this was simple to create
+    def to_json(pretty=true)
+      db.results_as_hash = false
+      totals = db.execute("select * from totals")
+      hash_total = []
+      totals.each do |row|
+        h = {}
+        (0...row.length).each do |col|
+          h[total_columns[col]] = row[col]
+        end
+        hash_total << h
+      end
+      if pretty
+        JSON.pretty_generate(hash_total)
+      else
+        hash_total.to_json
+      end
+    end
+
+    # removes old items from the database that are older than mtime
+    def remove_old_items(mtime)
+      db.execute <<SQL
+      DELETE from artifacts where modified < #{mtime}
+SQL
+    end
+
+    # get items older than mtime, defaults to 2 days ago
+    def old_items(mtime=(Time.now.to_i)-(172800))
+      data = db.execute <<SQL
+        SELECT * from artifacts where modified < #{mtime}
+SQL
+      data || []
+    end
+
+    # returns the top X most utilized caches
+    def top_x(amount=10)
+      db.execute <<SQL
+        SELECT * FROM totals
+        ORDER BY hit_count desc
+        LIMIT #{amount}
+SQL
+    end
+
+    private
+
+    def total_columns
+      %w(sha1 artifact_gav filesize request_time modified file_location hit_count total_bytes_saved total_time_saved)
+    end
+
+    def artifact_columns
+      %w(sha1 artifact_gav filesize, request_time hit_count modified file_location )
+
+    end
+
+    def init_tables
+      # id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
+      db.execute <<SQL
+      CREATE TABLE IF NOT EXISTS artifacts (
+       sha1 VARCHAR(40) PRIMARY KEY NOT NULL,
+       artifact_gav VARCHAR(255),
+       filesize BIGINT default 0,
+       request_time FLOAT default 0,
+       hit_count INTEGER default 0,
+       modified BIGINT default '#{Time.now.to_i}',
+       file_location VARCHAR(255)
+      );
+SQL
+    end
+
+    def total_view
+      db.execute <<SQL
+        CREATE VIEW IF NOT EXISTS totals AS
+        SELECT sha1, artifact_gav, filesize, request_time, modified, file_location,
+               hit_count, sum(hit_count * filesize) AS 'total_bytes_saved',
+        sum(hit_count * request_time) as 'total_time_saved'
+        FROM artifacts
+SQL
+    end
+
+   end
+end
+