havox 0.11.1

data/README.md

@@ -0,0 +1,82 @@
+# Havox
+
+Havox is a rule generator for Autonomous Systems supported by OpenFlow switches. Based on orchestration directives (or instructions) written in its own DSL, Havox generates a set of OpenFlow rules to be installed on the underlying switches.
+
+As part of a bigger architecture, Havox is designed to run as a web API that receives requests containing the directives, the topology graph description and some parameters for rule formatting. Once the rules are generated, the API responds with a JSON message containing all the rules, each one indicating which OpenFlow switch it should be installed on.
+
+The mentioned architecture is composed of Havox, [RouteFlow](https://github.com/routeflow/RouteFlow) and [Merlin](https://github.com/merlin-lang/merlin), and currently tested in a [Mininet](https://github.com/mininet/mininet) environment with support of [MiniNext](https://github.com/USC-NSL/miniNExT) implementing Quagga routing engines. RouteFlow requests rules from Havox and therefore is a client, whereas Merlin is a dependency which effectively calculates the paths between the AS exits.
+
+## Development status
+
+Havox is still experimental, born from a [master's thesis](http://www2.uniriotec.br/ppgi/banco-de-dissertacoes-ppgi-unirio/ano-2017/havox-uma-arquitetura-para-orquestracao-de-trafego-em-redes-openflow/view), so it is being actively developed and improved. There is yet a lot of new functionalities to come.
+
+## How it works
+
+Havox is implemented in Ruby and is strongly based on metaprogramming concepts. The orchestration directives that forms its DSL are methods executed by the Ruby interpreter, having both values and entire blocks of code as parameters. The following code shows some Havox directives:
+
+```ruby
+topology 'example.dot'
+associate :rfvmE, :s5
+
+exit(:s4) { destination_port 80 }
+
+exit(:s3) {
+  source_ip '200.156.0.0/16'
+  destination_ip '200.20.0.0/16'
+}
+```
+
+The first one, `topology <file_name>` is a topology file definition directive. It takes a string and specifies the topology description file that should be considered, which usually goes together with the directives file in the same request.
+
+The second, `associate <router_name>, <switch_name>` is the router-switch association directive. It takes a string or a [symbol](https://ruby-doc.org/core-2.4.0/Symbol.html) that specifies the RouteFlow container name that runs the target routing instance and its associated OpenFlow switch name. In this case, it associates the routing container _rfvmE_ to the switch _s5_.
+
+Next, there are two orchestration directives that instructs the matching flow to leave the network by the indicated switch. The exit directive, `exit(<switch_name>) { <openflow_fields_and_values> }`, takes the switch name as a string or symbol and a block containing the supported OpenFlow fields and their respective matching values, which can be strings or integers depending on the field.
+
+For example, the first `exit` directive tells that any matching traffic with destination port 80 should leave the domain by the switch _s4_. The second `exit` directive tells that any packet that comes from the network 200.156/16 and heads to the network 200.20/16 must leave the domain by the switch _s3_.
+
+The topology file is a graph description file written in [DOT language](https://en.wikipedia.org/wiki/DOT_(graph_description_language)) and describes how the network topology is organized. This file is processed by both Havox and its dependency, Merlin. Each node has informations like its name, its internal IP address from one of its interfaces and how it is connected to the other nodes. It is important to know that this IP address is used by Havox to infer the associations between routing containers and switches using OSPF routes. Otherwise, it would be required to manually associate each pair using the association directive described above.
+
+The exit directives from the request will be transcompiled to Merlin blocks of code, written in a Merlin policy file. This file is sent with the topology file to the Merlin process, where they will be evaluated for paths calculation. The generated raw rules are then post-processed, formatted and outputted by the API.
+
+More details about the transcompilation, parsing and formatting processes can be found [here](http://www2.uniriotec.br/ppgi/banco-de-dissertacoes-ppgi-unirio/ano-2017/havox-uma-arquitetura-para-orquestracao-de-trafego-em-redes-openflow/view) (master's thesis in brazilian portuguese).
+
+## Installation
+
+Install Havox by:
+
+1. Cloning the repository: `$ git clone https://github.com/rodrigosoares/havox`
+2. Accessing the directory: `$ cd havox`
+3. Installing gem dependencies: `$ bin/setup`
+
+Besides, Havox requires [Merlin](https://github.com/merlin-lang/merlin) and a [modified version of RouteFlow](https://github.com/rodrigosoares/RouteFlow) that implements IP source and VLAN ID matching, subnet mask support and RFHavox module. The latter is an extra RouteFlow module designed to request rules from the remote Havox web API and enqueue them into the RouteFlow IPC system.
+
+If the tests will be run in a experimental network, [Mininet](https://github.com/mininet/mininet) and [MiniNext](https://github.com/USC-NSL/miniNExT) are also required. Gists containing example Quagga BGP configurations and Mininet topology files can be found [here](https://gist.github.com/rodrigosoares/53ca13f0376ade1fa7b7221328dae3ce).
+
+It is *strongly recommended* to install Merlin, RouteFlow and Mininet/MiniNext in isolated virtual machines, each. Make sure that all the VMs ping each other successfully. The recommendation is to use [VirtualBox](https://www.virtualbox.org/) with _host-only network_ adapters.
+
+Setup all projects following their respective installation instructions. If using RouteFlow with Mininet, follow the [instructions](https://github.com/routeflow/RouteFlow/wiki/Tutorial-2:-rftest2) on setting Mininet up with RouteFlow remote controller.
+
+## Usage
+
+HVX, MN and RF are the environments running Havox, Mininet and RouteFlow, respectively.
+
+1. (HVX) Setup Merlin and RouteFlow IPs and etcetera in `config.rb`.
+2. (HVX) Start Havox API web server: `$ havox`.
+3. (MN) Start Mininet, pointing its remote controller to the RouteFlow VM.
+4. (RF) Start RouteFlow: `sudo ./rftest2`.
+
+If everything runs fine, RouteFlow will send a POST request to Havox, which in turn will log it and respond.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rspec --format d` to run the tests and read a little documentation about what each component does. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/rodrigosoares/havox.
+
+Be sure to follow [Ruby best practices](https://github.com/bbatsov/ruby-style-guide) and to create a spec test case for each new method, module or component you create.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "havox"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/config.rb

@@ -0,0 +1,4 @@
+Havox.configure do |config|
+  config.merlin_host = '192.168.56.102'
+  config.rf_lxc_names = %w(rfvmA rfvmB rfvmC rfvmD rfvmE rfvmF rfvmG rfvmH rfvmI)
+end

data/examples/example_3x3.dot

@@ -0,0 +1,81 @@
+digraph g1 {
+  h1 [type = host, mac = "00:00:00:00:00:01", ip = "172.31.1.100"];
+  h2 [type = host, mac = "00:00:00:00:00:02", ip = "172.31.2.100"];
+  h3 [type = host, mac = "00:00:00:00:00:03", ip = "172.31.3.100"];
+  h4 [type = host, mac = "00:00:00:00:00:04", ip = "172.31.4.100"];
+  h5 [type = host, mac = "00:00:00:00:00:05", ip = "172.31.5.100"];
+  h6 [type = host, mac = "00:00:00:00:00:06", ip = "172.31.6.100"];
+  h7 [type = host, mac = "00:00:00:00:00:07", ip = "172.31.7.100"];
+  h8 [type = host, mac = "00:00:00:00:00:08", ip = "172.31.8.100"];
+  h9 [type = host, mac = "00:00:00:00:00:09", ip = "172.31.9.100"];
+
+  s1 [type = switch, ip = "172.31.1.1", id = 1];
+  s2 [type = switch, ip = "172.31.2.1", id = 2];
+  s3 [type = switch, ip = "172.31.3.1", id = 3];
+  s4 [type = switch, ip = "172.31.4.1", id = 4];
+  s5 [type = switch, ip = "172.31.5.1", id = 5];
+  s6 [type = switch, ip = "172.31.6.1", id = 6];
+  s7 [type = switch, ip = "172.31.7.1", id = 7];
+  s8 [type = switch, ip = "172.31.8.1", id = 8];
+  s9 [type = switch, ip = "172.31.9.1", id = 9];
+
+  s1 -> h1 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h1 -> s1 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s2 -> h2 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h2 -> s2 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s3 -> h3 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h3 -> s3 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s4 -> h4 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h4 -> s4 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s5 -> h5 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h5 -> s5 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s6 -> h6 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h6 -> s6 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s7 -> h7 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h7 -> s7 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s8 -> h8 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h8 -> s8 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s9 -> h9 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+  h9 -> s9 [src_port = 1, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s1 -> s2 [src_port = 2, dst_port = 2, cost = 1, capacity = "1Gbps"];
+  s1 -> s4 [src_port = 3, dst_port = 2, cost = 1, capacity = "1Gbps"];
+
+  s2 -> s1 [src_port = 2, dst_port = 2, cost = 1, capacity = "1Gbps"];
+  s2 -> s3 [src_port = 3, dst_port = 2, cost = 1, capacity = "1Gbps"];
+  s2 -> s5 [src_port = 4, dst_port = 1, cost = 1, capacity = "1Gbps"];
+
+  s3 -> s2 [src_port = 2, dst_port = 3, cost = 1, capacity = "1Gbps"];
+  s3 -> s6 [src_port = 3, dst_port = 2, cost = 1, capacity = "1Gbps"];
+
+  s4 -> s1 [src_port = 2, dst_port = 3, cost = 1, capacity = "1Gbps"];
+  s4 -> s5 [src_port = 4, dst_port = 2, cost = 1, capacity = "1Gbps"];
+  s4 -> s7 [src_port = 3, dst_port = 3, cost = 1, capacity = "1Gbps"];
+
+  s5 -> s2 [src_port = 1, dst_port = 4, cost = 1, capacity = "1Gbps"];
+  s5 -> s4 [src_port = 2, dst_port = 4, cost = 1, capacity = "1Gbps"];
+  s5 -> s6 [src_port = 3, dst_port = 4, cost = 1, capacity = "1Gbps"];
+  s5 -> s8 [src_port = 4, dst_port = 4, cost = 1, capacity = "1Gbps"];
+
+  s6 -> s3 [src_port = 2, dst_port = 3, cost = 1, capacity = "1Gbps"];
+  s6 -> s5 [src_port = 4, dst_port = 3, cost = 1, capacity = "1Gbps"];
+  s6 -> s9 [src_port = 3, dst_port = 3, cost = 1, capacity = "1Gbps"];
+
+  s7 -> s4 [src_port = 3, dst_port = 3, cost = 1, capacity = "1Gbps"];
+  s7 -> s8 [src_port = 2, dst_port = 2, cost = 1, capacity = "1Gbps"];
+
+  s8 -> s5 [src_port = 4, dst_port = 4, cost = 1, capacity = "1Gbps"];
+  s8 -> s7 [src_port = 2, dst_port = 2, cost = 1, capacity = "1Gbps"];
+  s8 -> s9 [src_port = 3, dst_port = 2, cost = 1, capacity = "1Gbps"];
+
+  s9 -> s6 [src_port = 3, dst_port = 3, cost = 1, capacity = "1Gbps"];
+  s9 -> s8 [src_port = 2, dst_port = 3, cost = 1, capacity = "1Gbps"];
+}

data/examples/example_3x3.hvx

@@ -0,0 +1,5 @@
+Havox::Network.define do
+  topology 'example_3x3.dot'
+  associate :rfvmE, :s5
+  exit(:s4) { destination_port 80 }
+end

data/exe/havox

@@ -0,0 +1,107 @@
+#!/usr/bin/env ruby
+
+require 'havox'
+require 'optparse'
+require 'ostruct'
+require_relative '../lib/havox/app/api'
+
+# Defines command options and parameters.
+args = OpenStruct.new
+OptionParser.new do |opt|
+  opt.separator ''
+  opt.separator <<-DESCRIPTION
+  Havox is a rule generator for Autonomous Systems supported by OpenFlow switches.
+  Based on a directives file written in its own DSL and a graph topology file, Havox
+  generates a set of OpenFlow rules to be installed in the underlying switches. Just
+  run the following command to start the Havox API web server:
+
+  havox
+
+  This will run the server on port 4567. It listens for HTTP requests to the /rules
+  route, POST method. The request should contain the directives file, the topology
+  file and a set of configuration parameters that define how the rules should be
+  formatted.
+
+  An alternative way to run Havox is by calling its rules-only mode (prints the rules
+  to the STDOUT) or using the Trema controller (yet experimental). See the options below.
+  DESCRIPTION
+  opt.separator ''
+
+  opt.on('-c', '--trema-topo CONF', 'Trema topology configuration file') { |o| args.trema_topology = o }
+  opt.on('-t', '--merlin-topo TOPO', 'Merlin topology file') { |o| args.merlin_topology = o }
+  opt.on('-p', '--merlin-policy POLICY', 'Merlin policy file') { |o| args.merlin_policy = o }
+  opt.on('-d', '--directives DIRECTIVES_FILE', 'Havox directives file') { |o| args.havox_directives = o }
+  opt.on('-s', '--syntax trema,ovs,routeflow', [:trema, :ovs, :routeflow], 'Output syntax for generated rules (default: trema)') { |o| args.syntax = o }
+  opt.on('-f', '--[no-]force', 'Forces Havox to ignore field conflicts in rules generated by Merlin') { |o| args.force = o }
+  opt.on('-b', '--[no-]basic-policies', 'Automatically appends policies for basic protocols in the policies file') { |o| args.basic = o }
+  opt.on('-x', '--[no-]expand', 'Expands rules from VLAN-based to full predicates') { |o| args.expand = o }
+  opt.on('-o', '--[no-]output', 'Switches all occurrences of Enqueue action to Output action') { |o| args.output = o }
+  opt.on('--rules-only', 'Tells Havox to just output rules without loading anything') { |o| args.rules_only = o }
+  opt.on('--trema', 'Tells Havox to load Trema environment') { |o| args.trema = o }
+end.parse!
+
+# Sets all required environment variables to be read in the controller.
+def set_environment_vars(args)
+  ENV['HAVOX_FORCE'] = args.force.to_s
+  ENV['HAVOX_BASIC'] = args.basic.to_s
+  ENV['HAVOX_EXPAND'] = args.expand.to_s
+  ENV['HAVOX_OUTPUT'] = args.output.to_s
+  ENV['HAVOX_SYNTAX'] = args.syntax.to_s
+  ENV['HAVOX_MERLIN_TOPOLOGY'] = args.merlin_topology
+  ENV['HAVOX_MERLIN_POLICY'] = args.merlin_policy
+end
+
+# Defines the Trema execution method.
+def start_trema!(args)
+  puts 'Loading Havox with Trema...'
+  cmd = 'trema run lib/trema/controllers/main_controller.rb' \
+  " -c #{args.trema_topology}"
+  system(cmd)
+rescue Interrupt
+  puts 'Closing.'
+end
+
+# Checks arguments for rules generation.
+def files_ok?(args)
+  !(args.havox_directives.nil? && args.merlin_policy.nil?) &&
+    !args.merlin_topology.nil?
+end
+
+# Checks arguments for Trema initialization.
+def can_start_trema?(args)
+  !args.trema_topology.nil? && files_ok?(args)
+end
+
+# Sets environment variables and starts Havox.
+def set_env_and_start_trema!(args)
+  if can_start_trema?(args)
+    set_environment_vars(args)
+    start_trema!(args)
+  end
+end
+
+# Transcompiles Havox directives to Merlin blocks and returns the resulting Merlin file.
+def transcompile_to_merlin(args)
+  mln_filename = "./#{File.basename(args.havox_directives, '.hvx')}.mln"
+  eval File.read(args.havox_directives)
+  mln_blocks = Havox::Network.transcompile(args)
+  File.open(mln_filename, 'w') { |f| f.write(mln_blocks.join("\n")) }
+  mln_filename
+end
+
+# Calls the rules generation and outputs the rules.
+def output_rules!(args)
+  if files_ok?(args)
+    args.merlin_policy = transcompile_to_merlin(args) unless args.havox_directives.nil?
+    puts Havox::Policy.new(args.to_h).to_json
+  end
+end
+
+# After everything is defined, main execution flux starts here.
+if args.rules_only
+  output_rules!(args)
+elsif args.trema
+  set_env_and_start_trema!(args)
+else
+  Havox::API.run!
+end

data/havox.gemspec

@@ -0,0 +1,33 @@
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'havox/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = 'havox'
+  spec.version       = Havox::VERSION
+  spec.authors       = ['Rodrigo Soares']
+  spec.email         = ['silvars1@gmail.com']
+
+  spec.summary       = %q{A gem for making the generation of OpenFlow network rules easier.}
+  spec.description   = %q{This gem offers a friendly way to deal with OpenFlow network policies generated by Merlin, integrating it with RouteFlow through a simple orchestration DSL.}
+  spec.homepage      = 'https://github.com/rodrigosoares/havox'
+  spec.license       = 'GPL-3.0'
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = 'exe'
+  spec.executables   = 'havox'
+  spec.require_paths = ['lib']
+
+  spec.add_development_dependency 'bundler', '~> 1.11'
+  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'rspec', '~> 3.0'
+  spec.add_development_dependency 'factory_girl', '~> 4.8'
+  spec.add_development_dependency 'awesome_print', '~> 1.7'
+  spec.add_runtime_dependency     'net-ssh', '~> 4.0'
+  spec.add_runtime_dependency     'net-scp', '~> 1.2'
+  spec.add_runtime_dependency     'trema', '~> 0.10.1'
+  spec.add_runtime_dependency     'colorize', '~> 0.8'
+  spec.add_runtime_dependency     'sinatra', '~> 2.0'
+end

data/lib/havox.rb

@@ -0,0 +1,51 @@
+require 'json'
+require 'ipaddr'
+require 'singleton'
+require 'havox/version'
+require 'havox/exceptions'
+require 'havox/configuration'
+require 'havox/modules/command'
+require 'havox/modules/routeflow'
+require 'havox/modules/merlin'
+require 'havox/modules/field_parser'
+require 'havox/modules/openflow10/ovs/actions'
+require 'havox/modules/openflow10/ovs/matches'
+require 'havox/modules/openflow10/routeflow/actions'
+require 'havox/modules/openflow10/routeflow/matches'
+require 'havox/modules/openflow10/trema/actions'
+require 'havox/modules/openflow10/trema/matches'
+require 'havox/classes/topology'
+require 'havox/classes/node'
+require 'havox/classes/edge'
+require 'havox/classes/modified_policy'
+require 'havox/classes/policy'
+require 'havox/classes/rib'
+require 'havox/classes/route'
+require 'havox/classes/route_filler'
+require 'havox/classes/rule'
+require 'havox/classes/rule_sanitizer'
+require 'havox/classes/rule_expander'
+require 'havox/classes/translator'
+require 'havox/dsl/directive.rb'
+require 'havox/dsl/directive_proxy.rb'
+require 'havox/dsl/network.rb'
+require 'net/ssh'
+require 'net/scp'
+
+module Havox
+  class << self
+    attr_accessor :configuration
+  end
+
+  def self.configuration
+    @configuration ||= Configuration.new
+  end
+
+  def self.reset
+    @configuration = Configuration.new
+  end
+
+  def self.configure
+    yield(configuration)
+  end
+end

data/lib/havox/app/api.rb

@@ -0,0 +1,57 @@
+require 'bundler/setup' # TODO: Remove this line after Havox is released to RubyGems.
+require 'sinatra'
+require 'colorize'
+require 'havox'
+require_relative '../../../config'
+require_relative 'helpers/methods'
+
+module Havox
+  class API < Sinatra::Base
+    set :bind, '0.0.0.0'
+
+    before { content_type :json }
+
+    # TODO: Make this read files from anywhere in the file system.
+    get '/rules/:name' do
+      base_dir = '/home/rod/Projetos/Mestrado/havox/lib/merlin'
+      opts = {
+        merlin_topology: "#{base_dir}/topologies/#{params[:name]}.dot",
+        merlin_policy: "#{base_dir}/policies/#{params[:name]}.mln",
+        force: params[:force].eql?('true'),
+        basic: params[:basic].eql?('true'),
+        expand: params[:expand].eql?('true'),
+        output: params[:output].eql?('true'),
+        syntax: params[:syntax]&.to_sym
+      }
+      Havox::Policy.new(opts).to_json
+    end
+
+    post '/rules' do
+      dot_filepath = handle_file(params[:dot_file])
+      hvx_filepath = handle_file(params[:hvx_file])
+      pre_opts = {
+        qos: params[:qos],
+        preferred: params[:preferred],
+        arbitrary: params[:arbitrary]
+      }
+      mln_filepath = run_network(dot_filepath, hvx_filepath, pre_opts)
+
+      if mln_filepath.nil?
+        [].to_json
+      else
+        pos_opts = {
+          merlin_topology: dot_filepath,
+          merlin_policy: mln_filepath,
+          force: params[:force].eql?('true'),
+          basic: params[:basic].eql?('true'),
+          expand: params[:expand].eql?('true'),
+          output: params[:output].eql?('true'),
+          syntax: params[:syntax]&.to_sym
+        }
+        policy = Havox::Policy.new(pos_opts)
+        print_policy(policy, pos_opts)
+        policy.to_json
+      end
+    end
+  end
+end