itchy 0.2.7 → 0.2.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: fdda0b2aa4e9eb5cc2ea91f70fac7563a84549dd
-  data.tar.gz: a0af4183e2155fc184c60dec6dbbf6bfb845d703
+  metadata.gz: 10b5ecfa7e3bc071fecc195d65a0169f0b7545ea
+  data.tar.gz: 29c643ccf9dce7e83730e68edefb64ab8e70768d
 SHA512:
-  metadata.gz: 7443a124bc70d22664b774c31ebc5e2da9b773b136c0b8a335365478f5813cb27224f152a0272dc1a098a79717cbdb0c99802ec20f4cf370b80176dfc3bd2d07
-  data.tar.gz: c3c4bc6f42d601077ca9f91390ae9ec77e335fa0819f407bb28e969cb345b021d3318958b1fd02871304ced73a833c7394c429abb51be3d75a8414c4f20fd93a
+  metadata.gz: 8359f10ef113911af9947c0eaafc580da3c7040b081d2198eb83783f09a0d129d52e5ed595128c0555bea5a40447b7f7485381edde921d96181a6da4c817e629
+  data.tar.gz: ab5fb5cf07d9c7f8e38a0c82300e391eda804bb6e9a948baef2e35376ad37d1c4f11269cc7a6ea4b637837a1d0fe6b4431bfd3b7f7f5396599a243e013aa5119

data/README.md

@@ -3,10 +3,127 @@
 [![Gem Version](https://fury-badge.herokuapp.com/rb/onevmcatcher.png)](https://badge.fury.io/rb/itchy)
 [![Code Climate](https://codeclimate.com/github/arax/onevmcatcher.png)](https://codeclimate.com/github/kosoburak/itchy)
 
-# Itchy
+# Itchy - vIrTual applianCe Handling utiliTY
 
-Handling images from HEPIX vmcatcher for Nifty.
+ITCHY is tool for handling virtual appliances.
 
+## Related tools
+ITCHY is part of tool chain used for automated virtual applience handling.
+Because of character of this tool, separated usage is not really helpfull.
+
+Part of ITCHY is designed to be an event handler for [vmcatcher](https://github.com/hepix-virtualisation/vmcatcher).
+
+Files created by ITCHY are intended to be uploated and registered in a cloud storage.
+This can be done by our other tool [nifty](https://github.com/CESNET/nifty).
+
+## What and how does ITCHY do?
+
+It basically converts heterogeneous output from vmcatcher and prepare all necessary files for upload an registration into a cloud storage.
+
+ITCHY consists of two cooperating tasks. One serves for archiving vmcatcher events, second one for further processing. This concept was needed because of two reasons. First reason is that vmcatcher run as a cron job, regularly. There is a possibility, that image processing time would be greater than time period between two vmcatcher runs. In this case, the whole process would fail. The second reason is that we can't interrupt vmcatcher event processing and start over. So, the first part of ITCHY running with vmcatcher as event handler is so simple as it can be. The other acts are separated and they can be restarted or delayed if there is a need to do so, and it won't affect vmcatcher.
+
+## Instalation
+
+###From distribution specific packages
+Distribution specific packages can be created with [omnibus packaging for ITCHY](https://github.com/CESNET/omnibus-itchy). When installing via packages you don't have to install neither ruby nor rubygems. Packages contain embedded ruby and all the necessary gems and libraries witch will not effect your system ruby, gems and libraries.
+
+Currently supported distributions:
+
+    Ubuntu 12.04
+    Ubuntu 14.04
+    Debian 7.6
+    Debian 8.2
+    CentOS 6.5
+    CentOS 7.1
+
+###From RubyGems.org
+To install the most recent stable version
+``bash
+gem install nifty
+``
+
+###From source (dev)
+**Installation from source should never be your first choice! Especially, if you are not
+familiar with RVM, Bundler, Rake and other dev tools for Ruby!**
+
+**However, if you wish to contribute to our project, this is the right way to start.**
+
+To build and install the bleeding edge version from master
+
+```bash
+git clone git://github.com/CESNET/itchy.git
+cd itchy
+gem install bundler
+bundle install
+bundle exec rake spec
+```
+
+## Configuration
+###Create a configuration file for ITCHY
+Configuration file can be read by ITCHY from these
+three locations:
+
+* `~/.itchy/itchy.yml`
+* `/etc/itchy/itchy.yml`
+* `PATH_TO_GEM_DIR/config/itchy.yml`
+
+The default configuration file can be found at the last location
+`PATH_TO_GEM_DIR/config/itchy.yml`.
+
+## Usage
+ITCHY starts with executable `itchy`. For further assistance run `itchy help`:
+```bash
+$ itchy help
+
+Commands:
+  itchy archive         # Handle an incoming vmcatcher event and store it for further processing
+  itchy help [COMMAND]  # Describe available commands or one specific command
+  itchy process         # Process stored events
+
+Options:
+  -b, [--log-level=LOG_LEVEL]  # Logging level
+                               # Default: error
+                               # Possible values: debug, info, warn, error, fatal, unknown
+  -d, [--debug], [--no-debug]  # Enable debugging
+```
+### ARCHIVE
+This command is used as an event handler for `vmcatcher`.
+For proper running it needs to have set required env variables by `vmcatcher`. 
+```bash
+$ itchy help archive
+
+```
+### PROCESS
+
+```bash
+$itchy help process
+
+Usage:
+  itchy process
+
+Options:
+  -m, [--metadata-dir=METADATA_DIR]          # Path to a metadata directory for stored events
+                                             # Default: /var/spool/itchy/metadata
+  -f, [--required-format=REQUIRED_FORMAT]    # Required output format of converted images
+                                             # Default: qcow2
+  -o, [--output-dir=OUTPUT_DIR]              # Path to a directory where processed events descriptors will be stored
+                                             # Default: /var/spool/itchy/output
+  -t, [--temp-image-dir=TEMP_IMAGE_DIR]      # Path to a directory where images will be temporary stored while being processed
+                                             # Default: /var/spool/itchy/temp
+  -e, [--descriptor-dir=DESCRIPTOR_DIR]      # Path to a directory where appliance descriptors will be stored
+                                             # Default: /var/spool/itchy/descriptors
+  -p, [--file-permissions=FILE_PERMISSIONS]  # Sets permissions for all created files
+                                             # Default: 0664
+  -l, [--log-to=LOG_TO]                      # Logging output, file path or stderr/stdout
+                                             # Default: /var/log/itchy/process.log
+  -q, [--qemu-img-binary=QEMU_IMG_BINARY]    # Path to qemu-img command binary, if not used, ITCHY will look for it in PATH
+  -b, [--log-level=LOG_LEVEL]                # Logging level
+                                             # Default: error
+                                             # Possible values: debug, info, warn, error, fatal, unknown
+  -d, [--debug], [--no-debug]                # Enable debugging
+
+Process stored events
+```
 ## Contributing
 
 1. Fork it ( https://github.com/kosoburak/itchy/fork )

data/bin/itchy

@@ -33,12 +33,17 @@ class ItchyRunnable < Thor
   AVAILABLE_AUTH_METHODS = %w(none basic).freeze
   ERROR_EXIT_CODE = 1
 
-  shared_option_log_level = [:log_level, { enum: AVAILABLE_LOG_LEVELS,
-                                           default: Itchy::Settings.log_level,
-                                           aliases: '-b', desc: 'Logging level' }]
-
-  shared_option_debug = [:debug, { type: :boolean, default: Itchy::Settings.debug,
-                                   aliases: '-d', desc: 'Enable debugging' }]
+  class_option  :log_level,
+                enum: AVAILABLE_LOG_LEVELS,
+                default: Itchy::Settings.log_level,
+                aliases: '-b',
+                desc: 'Logging level'
+
+  class_option  :debug,
+                type: :boolean,
+                default: Itchy::Settings.debug,
+                aliases: '-d',
+                desc: 'Enable debugging'
 
   # Static method required by Thor to make certain
   # actions work.
@@ -57,8 +62,8 @@ class ItchyRunnable < Thor
   method_option :log_to,
                 type: :string,
                 default: Itchy::Settings.log_to.archive_log,
-                aliasses: '-l',
-                desc: 'Logging output, file path or stderr/stdout'
+                aliases: '-l',
+                desc: 'Logging output for archive command, file path or stderr/stdout'
   
   method_option :file_permissions,
                 type: :string,
@@ -66,9 +71,6 @@ class ItchyRunnable < Thor
                 aliases: '-p',
                 desc: 'Sets permissions for all created files'
   
-  method_option *shared_option_log_level
-  method_option *shared_option_debug
-
   def archive
     check_restrictions
     opts = normalize_options
@@ -125,7 +127,7 @@ class ItchyRunnable < Thor
                 type: :string,
                 default: Itchy::Settings.log_to.process_log,
                 aliases: '-l',
-                desc: 'Logging output, file path or stderr/stdout'
+                desc: 'Logging output for process command, file path or stderr/stdout'
   
   
   method_option :qemu_img_binary,
@@ -134,9 +136,6 @@ class ItchyRunnable < Thor
                 aliases: '-q',
                 desc: 'Path to qemu-img command binary, if not used, ITCHY will look for it in PATH'
 
-  method_option *shared_option_log_level
-  method_option *shared_option_debug
-
   def process
     check_restrictions
     opts = normalize_options

data/itchy.gemspec

@@ -29,6 +29,7 @@ Gem::Specification.new do |gem|
   gem.add_dependency 'hashie', '~> 3.3', '>= 3.3.1'
   gem.add_dependency 'thor', '~> 0.19', '>= 0.19.1'
   gem.add_dependency 'cloud-appliance-descriptor', '~> 0.2'
+  gem.add_dependency 'nokogiri', '~>1.6', '>= 1.6.7'
 
   gem.add_development_dependency 'bundler', '~> 1.6'
   gem.add_development_dependency 'rake', '~> 10.0'

data/lib/itchy.rb

@@ -1,4 +1,6 @@
 require 'active_support/all'
+require 'rubygems'
+require 'nokogiri'
 require 'settingslogic'
 require 'multi_json'
 require 'mixlib/shellout'
@@ -7,12 +9,17 @@ require 'uri'
 require 'erb'
 require 'hashie/mash'
 require 'cloud-appliance-descriptor'
+require 'rubygems/package'
 
 # Wraps all internals of the handler.
 module Itchy
   BASIC_QEMU_COMMAND = 'qemu-img'
+  GEM_DIR = File.realdirpath(File.join(File.dirname(__FILE__), '..'))
+  XSD_SCHEMA = File.join(GEM_DIR, 'resources', 'dsp8023_2.0.1.xsd')  
+  XSD_DIR = File.join(GEM_DIR, 'resources')
 end
 
+
 require 'itchy/version'
 require 'itchy/settings'
 require 'itchy/log'

data/lib/itchy/image_transformer.rb

@@ -8,6 +8,9 @@ module Itchy
     # Archive format string msg
     ARCHIVE_STRING = 'POSIX tar archive'
 
+    # raw boot sector message
+    REAL_RAW_STRING = 'boot sector'
+
     # REGEX pattern for getting image format
     FORMAT_PATTERN = /format:\s(.*?)$/
 
@@ -34,12 +37,20 @@ module Itchy
     def transform!(metadata, vmcatcher_configuration)
       Itchy::Log.info "[#{self.class.name}] Transforming image format " \
                              "for #{metadata.dc_identifier.inspect}"
+
+      image_file = orig_image_file(metadata, vmcatcher_configuration)
+
+      unless File.file?(image_file)
+        Itchy::Log.error "[#{self.class.name}] Event image file - #{image_file}] - does not exist!"
+        fail Itchy::Errors::ImageTransformationError
+      end
+
       begin
-        if archived?(metadata.dc_identifier.inspect)
-          unpacking_dir = unpack_archived!(metadata, vmcatcher_configuration)
-          file_format = inspect_unpacked_dir(unpacking_dir, metadata)
+        if archived?(image_file)
+          unpacking_dir = process_archive(metadata, vmcatcher_configuration)
+          file_format = format("#{unpacking_dir}/#{metadata.dc_identifier}")
         else
-          file_format = format(orig_image_file(metadata, vmcatcher_configuration))
+          file_format = format(image_file)
           unpacking_dir = copy_unpacked!(metadata, vmcatcher_configuration)
         end
         if file_format == @options.required_format
@@ -82,9 +93,85 @@ module Itchy
         Itchy::Log.error "Image format #{file_format} is unknown and not supported!"
         fail Itchy::Errors::FileInspectError
       end
+      if file_format.eql? "raw" then
+        unless check_real_raw(file)
+	  Itchy::Log.error "Image format is not a real RAW, it has no boot sector!"
+          fail Itchy::Errors::FileInspectError
+        end
+      end
+      
       file_format
     end
 
+    def check_real_raw(file)
+      file_command(file).include? REAL_RAW_STRING
+    end 
+
+    def process_archive(metadata, vmcatcher_configuration)
+      unpacking_dir = prepare_image_temp_dir(metadata, vmcatcher_configuration)
+      File.open(orig_image_file(metadata, vmcatcher_configuration), "rb") do |file|
+        Gem::Package::TarReader.new(file) do |archive|
+          disk_name = nil
+          archive.each do |entry|
+            Itchy::Log.debug "EACH NAME: #{entry.full_name}"
+            if File.extname(entry.full_name).eql? ".ovf" then
+              File.open("#{unpacking_dir}/#{entry.full_name}", "wb") do |f|
+                f.write(entry.read)
+              end
+              disk_name = process_ovf("#{unpacking_dir}/#{entry.full_name}")
+              Itchy::Log.debug "FOUND DISK NAME FROM OVF: #{disk_name}"
+              break        
+            end
+          end
+          begin
+	  archive.rewind()
+          archive.seek(disk_name) do |disk|
+            Itchy::Log.debug "Seeked disk - with name: #{disk.full_name}"
+            File.open("#{unpacking_dir}/#{metadata.dc_identifier}", "wb") do |f|
+              until disk.eof?
+		output = disk.read(50000000)
+		f.write(output)
+              end
+            end
+          end
+          rescue => e
+            Ichy::Log.error e.message
+            Itchy::Log.error e.backtrace.join("\n")
+            fail e
+          end
+        end
+      end
+      
+      unpacking_dir
+    end
+
+    def process_ovf(ovf_file)
+      Itchy::Log.debug "PROCESSING OVF FILE: #{ovf_file}"
+      doc = Nokogiri::XML(File.open(ovf_file))
+      #validate_ovf(doc)
+      if doc.css("Envelope DiskSection Disk").count != 1
+        Itchy::Log.error "[#{self.class.name}] Unsupported ova, contains 0 or more than one disk!"
+        fail Itchy::Errors::FileInspectError
+      end
+      # return the name of disk
+      doc.css("Envelope References File").attr("href").value
+    end
+
+    # Validation is not used for now
+    def validate_ovf(doc)
+      Itchy::Log.debug "VALIDATING from DIR: #{XSD_DIR} with schema #{XSD_SCHEMA}"
+      Dir.chdir(XSD_DIR) do
+        xsd = Nokogiri::XML::Schema(File.read(XSD_SCHEMA))
+        xsd.validate(doc).each do |error|
+          Itchy::Log.error "VALIDATION ERROR: #{error.message}"
+        end
+        unless xsd.valid?(doc) then
+          Itchy::Log.error "[#{self.class.name}] OVF validation failed!"
+          fail Itchy::Errors::FileInspectError
+        end
+      end
+    end
+
     #
     #
     # @param metadata [Itchy::VmcatcherEvent] event metadata
@@ -215,14 +302,11 @@ module Itchy
                                 "#{ex.message}"
         fail Itchy::Errors::PrepareEnvError, ex
       end
+      temp_dir
     end
 
-    # Checks if file is archived image (format ova or tar)
-    #
-    # @param file [String] inspected file name
-    # @return [Boolean] archived or not
-    def archived?(file)
-      image_format_tester = Mixlib::ShellOut.new("file #{file}")
+    def file_command(file)
+      image_format_tester = Mixlib::ShellOut.new("file --brief #{file}")
       image_format_tester.run_command
       begin
         image_format_tester.error!
@@ -231,9 +315,17 @@ module Itchy
         Itchy::Log.error "[#{self.class.name}] Checking file format for" \
           "#{file} failed with #{image_format_tester.stderr}"
         fail Itchy::Errors::FileInspectError, ex
-                                
       end
-      temp = image_format_tester.stdout
+      image_format_tester.stdout
+    end
+
+
+    # Checks if file is archived image (format ova or tar)
+    #
+    # @param file [String] inspected file name
+    # @return [Boolean] archived or not
+    def archived?(file)
+      temp = file_command(file)
       temp.include? ARCHIVE_STRING
     end
   end

data/lib/itchy/version.rb

@@ -1,3 +1,3 @@
 module Itchy
-  VERSION = '0.2.7' unless defined?(::Itchy::VERSION)
+  VERSION = '0.2.8' unless defined?(::Itchy::VERSION)
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: itchy
 version: !ruby/object:Gem::Version
-  version: 0.2.7
+  version: 0.2.8
 platform: ruby
 authors:
 - Lubomir Kosaristan
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-03-31 00:00:00.000000000 Z
+date: 2016-05-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: opennebula
@@ -205,6 +205,26 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.2'
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.6.7
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.6'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.6.7
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement