fog-hadoop 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d6cf0fbcd7d3d76be2f843557741cfa219ce415f
+  data.tar.gz: b2562198c3186f9f4e8180163af038ac01c96cd1
+SHA512:
+  metadata.gz: e46ebef80dc8833b7497ca0a73125a692c802989f6dd9210d1038c37a7c5212325d6a56c92520cd582b17ae1d8f9f0a7e819eabc7ac45d9c4ce5f3c727995939
+  data.tar.gz: 110329ecf4257de26a988fea03469865412d621fc1a227a0b9d5be3f38f5d3eb5c6e522cad3e20556b49bd783c939784b47914dbaa7aca107df0eb3ff9ef27a9

data/.gitignore

@@ -0,0 +1,12 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+gemfiles/Gemfile-1.9.lock
+.DS_Store
+.idea/

data/.rubocop.yml

@@ -0,0 +1,38 @@
+# Custom Enforced Rubocop Styles
+
+LineLength:
+  Enabled: false
+MethodLength:
+  Enabled: false
+ClassLength:
+  Enabled: false
+SignalException:
+  Enabled: false
+RaiseArgs:
+  Enabled: false
+ParameterLists:
+  Enabled: false
+AbcSize:
+  Enabled: false
+RedundantBegin:
+  Enabled: false
+PerceivedComplexity:
+  Enabled: false
+CyclomaticComplexity:
+  Enabled: false
+RescueException:
+  Enabled: false
+NestedParenthesizedCalls:
+  Enabled: false
+
+# Custom Enforced Hound Ruby Styles
+
+CollectionMethods:
+  PreferredMethods:
+    detect: find
+StringLiterals:
+  EnforcedStyle: single_quotes
+SignalException:
+  EnforcedStyle: only_raise
+Style/FrozenStringLiteralComment:
+  Enabled: false

data/.ruby-gemset

@@ -0,0 +1 @@
+fog-hadoop

data/.ruby-version

@@ -0,0 +1 @@
+2.3.1

data/.travis.yml

@@ -0,0 +1,8 @@
+sudo: false
+language: ruby
+rvm:
+- 2.3.0
+script:
+- bundle install
+- bundle exec rake travis
+after_script: bundle exec codeclimate-test-reporter

data/CONTRIBUTING.md

@@ -0,0 +1,40 @@
+## Getting Involved
+
+New contributors are always welcome, when it doubt please ask questions. We strive to be an open and welcoming community. Please be nice to one another.
+
+For structure information of fog click [here](https://github.com/fog/fog-hadoop/blob/master/lib/fog/hadoop/docs/structure.md)
+
+### Coding
+
+* Pick a task:
+  * Offer feedback on open [pull requests](https://github.com/fog/fog-hadoop/pulls).
+  * Review open [issues](https://github.com/fog/fog-hadoop/issues) for things to help on.
+  * [Create an issue](https://github.com/fog/fog-hadoop/issues/new) to start a discussion on additions or features.
+* Fork the project, add your changes and tests to cover them in a topic branch.
+  * [Fork](https://github.com/fog/fog-hadoop/fork)
+  * Create your feature branch (`git checkout -b my-new-feature`)
+  * Commit your changes (`git commit -am 'Add some feature'`)
+  * Push to the branch (`git push origin my-new-feature`)
+  * Create a new pull request
+* Commit your changes and rebase against `fog/fog-azure-rm` to ensure everything is up to date.
+* [Submit a pull request](https://github.com/fog/fog-hadoop/compare/).
+
+### Non-Coding
+
+* Offer feedback on open [issues](https://github.com/fog/fog-hadoop/issues).
+* Organize or volunteer at events.
+* Write and help edit [documentation](https://github.com/fog/fog-hadoop/tree/master/lib/fog/aljesusg/docs).
+
+## Testing
+You can run `Minitest` tests by running this command in the root directory.
+
+```shell
+$ rake test
+```
+
+## Code Coverage
+We are using Simplecov Gem for code coverage. You can see the report of  Code Coverage by running this command in the root directory.
+
+```shell
+$ rake coverage
+```

data/CONTRIBUTORS.md

@@ -0,0 +1 @@
+* Alberto Gutiérrez <aljesusg@gmail.com>

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+# Specify your gem's dependencies in fog-azure-rm.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Alberto Jesus Gutierrez Juanes
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,51 @@
+# fog-hadoop
+[![Build Status](https://travis-ci.org/fog/fog-hadoop.svg?branch=master)](https://travis-ci.org/fog/fog-hadoop) [![Code Climate](https://lima.codeclimate.com/github/fog/fog-hadoop/badges/gpa.svg)](https://lima.codeclimate.com/github/fog/fog-hadoop) [![Test Coverage](https://lima.codeclimate.com/github/fog/fog-hadoop/badges/coverage.svg)](https://lima.codeclimate.com/github/fog/fog-hadoop/coverage)
+
+## Fog connector for Hadoop
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'fog-hadoop'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install fog-hadoop
+## Usage
+
+### Initial Setup
+
+Require the gem:
+
+```ruby
+require "fog/hadoop"
+```
+
+
+Connection parameters:
+
+```ruby
+@connection_params = {
+  hadoop_compute_api_url:     "http://<resource_manager_host>:8088/"
+}
+```
+
+* Use [Yarn](docs/yarn.md)    
+
+# Support
+
+Check in this [link](supported.md)
+
+# Contribute
+Read the [contribute](docs/CONTRIBUTING.md) documentation
+
+# Development
+
+You can use this [Vagrant Hadoop](https://github.com/vangj/vagrant-hadoop-2.4.1-spark-1.0.1) repo to have a environment to dev.

data/Rakefile

@@ -0,0 +1,112 @@
+require 'bundler/setup'
+require "bundler/gem_tasks"
+require 'rake/testtask'
+require 'date'
+require 'rubygems'
+require 'rubygems/package_task'
+require 'yard'
+require File.dirname(__FILE__) + '/lib/fog/hadoop'
+
+#############################################################################
+#
+# Helper functions
+#
+#############################################################################
+
+def name
+  @name ||= Dir['*.gemspec'].first.split('.').first
+end
+
+def version
+  Fog::VERSION
+end
+
+def date
+  Date.today.to_s
+end
+
+def rubyforge_project
+  name
+end
+
+def gemspec_file
+  "#{name}.gemspec"
+end
+
+def gem_file
+  "#{name}-#{version}.gem"
+end
+
+def replace_header(head, header_name)
+  head.sub!(/(\.#{header_name}\s*= ').*'/) { "#{$1}#{send(header_name)}'"}
+end
+
+#############################################################################
+#
+# Standard tasks
+#
+#############################################################################
+
+GEM_NAME = "#{name}"
+task :default => :test
+task :travis  => ['test', 'test:travis']
+
+Rake::TestTask.new do |t|
+  t.pattern = File.join("spec", "**", "*_spec.rb")
+end
+
+namespace :test do
+  mock = ENV['FOG_MOCK'] || 'true'
+  task :travis do
+    sh("export FOG_MOCK=#{mock} && bundle exec shindont")
+  end
+  task :vsphere do
+    sh("export FOG_MOCK=#{mock} && bundle exec shindont tests/vsphere")
+  end
+  task :openvz do
+    sh("export FOG_MOCK=#{mock} && bundle exec shindont tests/openvz")
+  end
+end
+
+desc 'Run mocked tests for a specific provider'
+task :mock, :provider do |t, args|
+  if args.to_a.size != 1
+    fail 'USAGE: rake mock[<provider>]'
+  end
+  provider = args[:provider]
+  sh("export FOG_MOCK=true && bundle exec shindont tests/#{provider}")
+end
+
+desc 'Run live tests against a specific provider'
+task :live, :provider do |t, args|
+  if args.to_a.size != 1
+    fail 'USAGE: rake live[<provider>]'
+  end
+  provider = args[:provider]
+  sh("export FOG_MOCK=false PROVIDER=#{provider} && bundle exec shindont tests/#{provider}")
+end
+
+task :nuke do
+  Fog.providers.each do |provider|
+    next if ['Vmfusion'].include?(provider)
+    begin
+      compute = Fog::Compute.new(:provider => provider)
+      for server in compute.servers
+        Fog::Formatador.display_line("[#{provider}] destroying server #{server.identity}")
+        server.destroy rescue nil
+      end
+    rescue
+    end
+    begin
+      dns = Fog::DNS.new(:provider => provider)
+      for zone in dns.zones
+        for record in zone.records
+          record.destroy rescue nil
+        end
+        Fog::Formatador.display_line("[#{provider}] destroying zone #{zone.identity}")
+        zone.destroy rescue nil
+      end
+    rescue
+    end
+  end
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "fog/hadoop"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/docs/yarn.md

@@ -0,0 +1,91 @@
+#Yarn use samples
+
+```ruby
+@connection_params = {
+  hadoop_compute_api_url:     "http://<resource_manager_host>:8088/"
+}
+```
+
+## Get Metrics
+
+```
+compute = Fog::Compute::Hadoop.new(@connection_params)
+compute.get_metrics
+```
+
+```
+# => {"appsSubmit`ted"=>0,
+#     "appsCompleted"=>0, 
+#     "appsPending"=>0,
+#     "appsRunning"=>0,
+#     "appsFailed"=>0,
+#     "appsKilled"=>0,
+#     "reservedMB"=>0,
+#     "availableMB"=>16384,
+#     "allocatedMB"=>0,
+#     "containersAllocated"=>0,
+#     "containersReserved"=>0,
+#     "containersPending"=>0,
+#     "totalMB"=>16384,
+#     "totalNodes"=>2,
+#     "lostNodes"=>0,
+#     "unhealthyNodes"=>0,
+#     "decommissionedNodes"=>0,
+#     "rebootedNodes"=>0,
+#     "activeNodes"=>2}
+```
+
+
+## Get Info
+
+```
+compute = Fog::Compute::Hadoop.new(@connection_params)
+compute.get_info
+```
+
+```
+# => {"id"=>1489170864142,
+#     "startedOn"=>1489170864142,
+#     "state"=>"STARTED",
+#     "haState"=>"ACTIVE",
+#     "resourceManagerVersion"=>"2.4.1",
+#     "resourceMan`agerBuildVersion"=>"2.4.1 from 1604318 by jenkins source checksum f74113c8e511baadac8fbc14827d8b85",
+#     "resourceManagerVersionBuiltOn"=>"2014-06-21T06:01Z",
+#     "hadoopVersion"=>"2.4.1",
+#     "hadoopBuildVersion"=>"2.4.1 from 1604318 by jenkins source checksum bb7ac0a3c73dc131f4844b873c74b630",
+#     "hadoopVersionBuiltOn"=>"2014-06-21T05:43Z"}
+```
+
+## Get Stats
+
+```
+compute = Fog::Compute::Hadoop.new(@connection_params)
+compute.get_app_stats
+```
+
+```
+=> [{"state"=>"SUBMITTED",
+#  "type"=>"*",
+#  "count"=>0},
+#  {"state"=>"NEW",
+#  "type"=>"*",
+#  "count"=>0},
+#  {"state"=>"KILLED",
+#  "type"=>"*",
+#  "count"=>0},
+#  {"state"=>"NEW_SAVING",
+#  "type"=>"*",
+#  "count"=>0},
+#  {"state"=>"RUNNING",
+#  "type"=>"*",
+#  "count"=>0},
+#  {"state"=>"FAILED",
+#  "type"=>"*",
+#  "count"=>0},
+#  {"state"=>"ACCEPTED",
+#  "type"=>"*",
+#  "count"=>0},
+#  {"state"=>"FINISHED",
+#  "type"=>"*",
+#  "count"=>0}]
+```

data/fog-hadoop.gemspec

@@ -0,0 +1,60 @@
+# Copyright (c) Microsoft Open Technologies, Inc.  All rights reserved.
+#
+# The MIT License (MIT)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'fog/hadoop/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "fog-hadoop"
+  spec.version       = Fog::Hadoop::VERSION
+  spec.authors       = ["Alberto Gutierrez"]
+  spec.email         = ["aljesusg@gmail.com"]
+  spec.summary       = %q{Module for the 'fog' gem to support Hadoop services.}
+  spec.description   = %q{This library can be used as a module for `fog` or as standalone provider
+                        to use the Hadoop services in applications..}
+  spec.homepage      = "http://github.com/fog/fog-hadoop"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'fog-core'
+  spec.add_dependency 'fog-json'
+
+  spec.add_development_dependency "bundler", "~> 1.5"
+  ## List your development dependencies here. Development dependencies are
+  ## those that are only needed during development
+  spec.add_development_dependency('minitest')
+  spec.add_development_dependency('rake')
+  spec.add_development_dependency('rbvmomi')
+  spec.add_development_dependency('yard')
+  spec.add_development_dependency('thor')
+  spec.add_development_dependency('rbovirt', '0.0.24')
+  spec.add_development_dependency('shindo', '~> 0.3.4')
+  spec.add_development_dependency('fission')
+  spec.add_development_dependency('pry')
+  spec.add_development_dependency('rubocop') if RUBY_VERSION > "1.9"
+  spec.add_development_dependency('codeclimate-test-reporter', '~> 1.0.0')
+end