smart_proxy_onboard 0.1.1

data/README.md

@@ -0,0 +1,105 @@
+# Smart Proxy - Onboard
+
+[![Gem Version](https://badge.fury.io/rb/smart_proxy_onboard.svg)](https://badge.fury.io/rb/smart_proxy_onboard)
+[![Build Status](https://travis-ci.org/Deltik/smart_proxy_onboard.svg?branch=develop)](https://travis-ci.org/Deltik/smart_proxy_onboard)
+[![Code Climate](https://codeclimate.com/github/Deltik/smart_proxy_onboard/badges/gpa.svg)](https://codeclimate.com/github/Deltik/smart_proxy_onboard)
+
+The **Smart Proxy Onboard Plugin** (**`smart_proxy_onboard`**) exposes useful API methods for the earliest part of the server lifecycle management process: onboarding new servers into [Foreman](https://github.com/theforeman/foreman).
+
+This plugin is used for the onboarding process before server discovery.  It plugs into [Smart Proxy](https://github.com/theforeman/smart-proxy) to cover the same network ranges as the Smart Proxy.
+
+[Rails DCIM Portal](https://github.com/buddwm/Rails_DCIM_Portal) uses the API methods in this plugin to scan a range of IP addresses to see which ones respond to IPMI.  Rails DCIM Portal then iterates over the list, finding IPMI credentials that work for each IP address.  These IPMI hosts are then considered scanned and can subsequently be discovered into Foreman by PXE booting into the discovery image from [`foreman_discovery`](https://github.com/theforeman/foreman_discovery).  Finally, while Foreman provisions the discovered hosts, Rails DCIM Portal can write inventory facts into the provisioned hosts to complete the onboarding process.
+
+## Installation
+
+Follow the ["Advanced Installation from Gems" instructions in the Foreman plugins documentation](https://theforeman.org/plugins/#2.3AdvancedInstallationfromGems) to install this plugin.
+
+The gem name is `smart_proxy_onboard`.
+
+## Configuration
+
+All configurable options for this plugin are documented at [settings.d/onboard.yml.example](settings.d/onboard.yml.example) and should be copied to `/etc/foreman-proxy/settings.d/onboard.yml`.
+
+## Compatible Software
+
+This plugin is made for the following combination of software:
+
+ - [**Rails DCIM Portal**](https://github.com/buddwm/Rails_DCIM_Portal)
+ - [**Smart Proxy**](https://github.com/theforeman/smart-proxy) (>= 1.16)
+ - [**Foreman - Discovery**](https://github.com/theforeman/foreman_discovery)
+ - [**Smart Proxy - Discovery**](https://github.com/theforeman/smart_proxy_discovery)
+
+## Features
+
+### Currently Implemented
+
+ - IPMI IP range scanner
+
+### Planned
+
+No additional features have been planned so far.
+
+## API
+
+### `GET /bmc/scan`
+
+Shows available resources `/bmc/scan/range` and `/bmc/scan/cidr`
+
+    {
+      "available_resources": [
+        "range",
+        "cidr"
+      ]
+    }
+
+### `GET /bmc/scan/range`
+
+Shows usage on specifying a beginning IP address and an ending IP address for making a scan request
+
+    {
+      "message": "You need to supply a range with /bmc/scan/range/:address_first/:address_last"
+    }
+
+### `GET /bmc/scan/cidr`
+
+Shows usage on specifying an IP address and its netmask in dot decimal format or prefixlen format for making a scan request
+
+    {
+      "message": "You need to supply a CIDR with /bmc/scan/cidr/:address/:netmask (e.g. \"192.168.1.1/24\" or \"192.168.1.1/255.255.255.0\")"
+    }
+
+### `GET /bmc/scan/range/:address_first/:address_last`
+
+Performs an IPMI ping scan from `:address_first` to `:address_last` and returns the result in key "`result`" of a JSON hash
+
+Sample output for `/bmc/scan/range/10.246.0.65/10.246.0.71`:
+
+    {
+      "result": [
+        "10.246.0.65",
+        "10.246.0.69",
+        "10.246.0.70",
+        "10.246.0.66",
+        "10.246.0.68",
+        "10.246.0.71",
+        "10.246.0.67"
+      ]
+    }
+
+### `GET /bmc/scan/cidr/:address/:netmask`
+
+Performs an IPMI ping scan in the CIDR range of `:address`/`:netmask`, where `:netmask` is in decimal format (e.g. "`255.255.255.0`") or in prefixlen format (e.g. "`24`")
+
+Sample output for `/bmc/scan/cidr/10.246.0.65/29`:
+
+    {
+      "result": [
+        "10.246.0.69",
+        "10.246.0.70",
+        "10.246.0.65",
+        "10.246.0.66",
+        "10.246.0.67",
+        "10.246.0.68",
+        "10.246.0.71"
+      ]
+    }

data/Rakefile

@@ -0,0 +1,10 @@
+require 'bundler/gem_tasks'
+require 'rake/testtask'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'test'
+  t.libs << 'lib'
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+task default: :test

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'smart_proxy_onboard'
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require 'irb'
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/bmc/basescanner.rb

@@ -0,0 +1,53 @@
+require 'smart_proxy_onboard'
+require 'ipaddr'
+
+module Proxy
+  module Onboard
+    module BMC
+      # This is the interface for scanning BMC IP address ranges.
+      class BaseScanner
+        def initialize(args = { })
+          if args.key? :address_first
+            address_first = IPAddr.new args[:address_first]
+            address_last  = IPAddr.new args[:address_last]
+            @range = (address_first..address_last)
+          elsif args.key?(:address) && args.key?(:netmask)
+            @range = IPAddr.new("#{args[:address]}/#{args[:netmask]}").to_range
+          else
+            @range = IPAddr.new(args[:address]).to_range
+          end
+          # Disallow range too large
+          scanner_max_range_size = max_range_size
+          if @range.first(scanner_max_range_size+1).size > scanner_max_range_size
+            @range = nil
+            @invalid_reason = "Range too large. Batch supports only #{scanner_max_range_size} IP addresses at a time."
+          end
+        rescue
+          @range = nil
+          if args.is_a?(Hash) && args.key?(:address)
+            @invalid_reason = "Invalid CIDR provided"
+          else
+            @invalid_reason = "Invalid IP address provided"
+          end
+        end
+  
+        def valid?
+          @range.is_a? Range
+        end
+  
+        def error_string
+          @invalid_reason
+        end
+  
+        def max_range_size
+          Proxy::Onboard::Plugin.settings.bmc_scanner_max_range_size || 65_536
+        end
+  
+        # Run the scanner and return results as array
+        def scan_to_list
+          raise NotImplementedError.new
+        end
+      end
+    end
+  end
+end

data/lib/bmc/ipmiscanner.rb

@@ -0,0 +1,83 @@
+require 'bmc/basescanner'
+require 'concurrent'
+
+module Proxy
+  module Onboard
+    module BMC
+      class IPMIScanner < BaseScanner
+        include Proxy::Log
+        include Proxy::Util
+  
+        # This is an IPMI ping, not an ICMP ping.
+        def address_pings?(address)
+          begin
+            socket = UDPSocket.new
+          rescue Errno::EMFILE
+            logger.warn "IPMIScanner: Ran out of free file descriptors while creating UDPSocket! Consider increasing file open limit."
+            retry
+          end
+          socket.connect(address.to_s, 623)
+          socket.send([0x6, 0x0, 0xff, 0x6, 0x0, 0x0, 0x11, 0xbe, 0x80, 0x0, 0x0, 0x0].pack('C*'), 0)
+          selections = IO.select([socket], nil, nil, (Proxy::Onboard::Plugin.settings.bmc_scanner_socket_timeout_seconds || 1))
+          socket.close
+          !selections.nil?
+        end
+  
+        # Not used because of slowness
+        def scan_unthreaded_to_list
+          return false if !valid?
+          pinged = Array.new
+          @range.each do |address|
+            if address_pings?(address)
+              pinged << address
+            end
+          end
+          pinged
+        end
+  
+        # Determine maximum number of threads
+        def calculate_max_threads
+          max_threads = Proxy::Onboard::Plugin.settings.bmc_scanner_max_threads_per_request || 500
+          begin
+            sockets = Array.new
+            # @range.first(max_threads).size performs much better than @range.count if @range is large.
+            (1..[max_threads, @range.first(max_threads).size].min).each do
+              socket = UDPSocket.new
+              sockets << socket
+            end
+          rescue Errno::EMFILE
+            # Running low on free file descriptors; only allow use of half of the remaining
+            max_threads = [sockets.length / 2, 1].max
+            # Clean up sockets
+            sockets.each do |sock|
+              sock.close
+            end
+            logger.warn "IPMIScanner: Running low on free file descriptors! Can only allocate #{sockets.length}, so using #{max_threads} to avoid hitting the limit."
+          end
+          max_threads
+        end
+  
+        def scan_threaded_to_list
+          return false if !valid?
+          pinged = Array.new
+          pool = Concurrent::ThreadPoolExecutor.new(max_threads: calculate_max_threads)
+          @range.each do |address|
+            pool.post do
+              if address_pings?(address)
+                pinged << address
+              end
+            end
+          end
+          pool.shutdown
+          pool.wait_for_termination
+          pinged
+        end
+  
+        def scan_to_list
+          return false if !valid?
+          scan_threaded_to_list
+        end
+      end
+    end
+  end
+end

data/lib/smart_proxy_onboard/api_bmc.rb

@@ -0,0 +1,49 @@
+require 'smart_proxy'
+
+module Proxy::Onboard
+  class ApiBmc < Sinatra::Base
+    helpers ::Proxy::Helpers
+  
+    # Scan IP range for BMC hosts
+    # Returns a list of available scanning options
+    get "/scan" do
+      { :available_resources => %w[range cidr] }.to_json
+    end
+
+    # Returns a helpful message that the user should supply a beginning IP and ending IP
+    get "/scan/range" do
+      { :message => "You need to supply a range with /bmc/scan/range/:address_first/:address_last"}.to_json
+    end
+
+    # Returns a helpful message that the user should supply a CIDR
+    get "/scan/cidr" do
+      { :message => "You need to supply a CIDR with /bmc/scan/cidr/:address/:netmask (e.g. \"192.168.1.1/24\" or \"192.168.1.1/255.255.255.0\")"}.to_json
+    end
+
+    ["/scan/range/:address_first/?:address_last?",
+     "/scan/cidr/:address/?:netmask?"].each do |path|
+      get path do
+        scanner_setup
+        if !@scanner.valid?
+          { :error => @scanner.error_string}.to_json
+        else
+          { :result => @scanner.scan_to_list}.to_json
+        end
+      end
+    end
+
+    def scanner_setup
+      args = {}
+      # /scan/cidr/:address/:netmask
+      if params.key? "address"
+        args = { :address => params[:address],
+                 :netmask => params[:netmask] }
+      # /scan/range/:address_first/:address_last
+      elsif params.key? "address_first"
+        args = { :address_first => params[:address_first],
+                 :address_last  => params[:address_last] }
+      end
+      @scanner = Proxy::Onboard::BMC::IPMIScanner.new(args)
+    end
+  end
+end

data/lib/smart_proxy_onboard/http_config.ru

@@ -0,0 +1,5 @@
+require 'smart_proxy_onboard/api_bmc'
+
+map '/bmc' do
+  run Proxy::Onboard::ApiBmc.new
+end

data/lib/smart_proxy_onboard/onboard.rb

@@ -0,0 +1,13 @@
+module Proxy::Onboard
+  # Plugin definition
+  class Plugin < ::Proxy::Plugin
+    plugin 'onboard', Proxy::Onboard::VERSION
+    default_settings bmc_scanner_max_range_size: 65_536,
+                     bmc_scanner_max_threads_per_request: 500,
+                     bmc_scanner_socket_timeout_seconds: 1
+
+    http_rackup_path File.expand_path('http_config.ru', File.expand_path('../', __FILE__))
+    https_rackup_path File.expand_path('http_config.ru', File.expand_path('../', __FILE__))
+    default_settings node_scheme: 'https', node_port: 8443
+  end
+end

data/lib/smart_proxy_onboard/version.rb

@@ -0,0 +1,5 @@
+module Proxy
+  module Onboard
+    VERSION = '0.1.1'.freeze
+  end
+end

data/lib/smart_proxy_onboard.rb

@@ -0,0 +1,2 @@
+require 'smart_proxy_onboard/version'
+require 'smart_proxy_onboard/onboard'

data/settings.d/onboard.yml.example

@@ -0,0 +1,19 @@
+---
+:enabled: true
+
+#####################################
+# Settings for scanner at /bmc/scan #
+#####################################
+# The largest number of IP addresses that may be scanned in one request. Set
+# this value to 0 to turn off the scanner.  Defaults to 65536, which covers a
+# CIDR prefixlen of /16.
+:bmc_scanner_max_range_size: 65536
+
+# The maximum number of IP addresses to scan at the same time, per request.
+# Defaults to 500.  Before increasing this number, you should increase the
+# maximum number of open files for Smart Proxy.
+:bmc_scanner_max_threads_per_request: 500
+
+# How many seconds to wait for each scanned IP address to respond before timing
+# out. Defaults to 1.
+:bmc_scanner_socket_timeout_seconds: 1

data/smart_proxy_onboard.gemspec

@@ -0,0 +1,33 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'smart_proxy_onboard/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'smart_proxy_onboard'
+  spec.version       = Proxy::Onboard::VERSION
+  spec.authors       = ['Nick Liu']
+  spec.email         = ['deltik@gmx.com']
+
+  spec.summary       = %q{Support functions for onboarding new servers into Foreman}
+  spec.description   = %q{This plugin exposes API calls that can be used to onboard new hosts in bulk into Foreman through PXE boot and the foreman_discovery image.}
+  spec.homepage      = 'https://github.com/theforeman/smart_proxy_onboard'
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = 'exe'
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ['lib']
+  spec.license       = 'GPLv3'
+
+  spec.add_development_dependency 'bundler', '~> 1.14'
+  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'test-unit'
+  spec.add_development_dependency 'rack'
+  spec.add_development_dependency 'rack-test'
+  spec.add_development_dependency 'mocha'
+  spec.add_development_dependency 'webmock'
+
+  spec.add_runtime_dependency 'concurrent-ruby'
+end

metadata

@@ -0,0 +1,176 @@
+--- !ruby/object:Gem::Specification
+name: smart_proxy_onboard
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- Nick Liu
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2017-05-06 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.14'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.14'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: test-unit
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rack
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rack-test
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: mocha
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: webmock
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: concurrent-ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This plugin exposes API calls that can be used to onboard new hosts in
+  bulk into Foreman through PXE boot and the foreman_discovery image.
+email:
+- deltik@gmx.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .rubocop.yml
+- .rubocop_todo.yml
+- .travis.yml
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/bmc/basescanner.rb
+- lib/bmc/ipmiscanner.rb
+- lib/smart_proxy_onboard.rb
+- lib/smart_proxy_onboard/api_bmc.rb
+- lib/smart_proxy_onboard/http_config.ru
+- lib/smart_proxy_onboard/onboard.rb
+- lib/smart_proxy_onboard/version.rb
+- settings.d/onboard.yml.example
+- smart_proxy_onboard.gemspec
+homepage: https://github.com/theforeman/smart_proxy_onboard
+licenses:
+- GPLv3
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.6.11
+signing_key: 
+specification_version: 4
+summary: Support functions for onboarding new servers into Foreman
+test_files: []