dorothy2 0.0.1

data/README.md

@@ -0,0 +1,231 @@
+# Dorothy2
+
+A malware/botnet analysis framework written in Ruby.
+
+
+##Introduction
+
+Dorothy2 is a framework created for mass malware analysis. Currently, it is mainly based on analyzing the network behavior of a virtual machine where a suspicious executable was executed.
+However, static binary analysis and system behavior analysis will be shortly introduced in the next version.
+
+Dorothy2 is a continuation of my degree's final project (Dorothy: inside the Storm [1] ) that I presented on Feb 2009.
+The main framework's structure remained almost the same, and it has been fully detailed in my degree's final project or in this short paper [2]. More information about the whole project can be found on the Italian Honeyproject website [3].
+
+
+The framework is manly composed by four big elements that can be even executed separately:
+
+* The Dorothy analysis engine (this gem)
+* The Network analysis module (included in this module, but still not working perfectly at the time of this writing)
+* The Webgui (Coded in Rails by Andrea Valerio, and not yet included in this gem)
+* The Java Dorothy Drone (Mainly coded by Patrizia Martemucci and Domenico Chiarito, but not part of this gem and not publicly available.)
+
+The first three modules are (or will be soon) publicly released under GPL 2/3 license as tribute to the the Honeynet Project Alliance[4].
+All the information generated by the framework - i.e. binary info, timestamps, dissected network analysis - are stored into a postgres DB (Dorothive) in order to be used for further analysis.
+A no-SQL database (CouchDB) is also used to mass strore all the traffic dumps thanks to the pcapr/xtractr[5] technology.
+
+I started to code this project in late 2009 while learning Ruby at the same time. Since then, I´ve been changing/improving it as long as my Ruby coding skills were improving. Because of that, you may find some parts of code not-really-tidy :)
+
+[1] https://www.honeynet.it/wp-content/uploads/Dorothy/The_Dorothy_Project.pdf
+[2] http://www.honeynet.it/wp-content/uploads/Dorothy/EC2ND-Dorothy.pdf
+[3] http://www.honeynet.it
+[4] http://www.honeynet.org
+[5] https://code.google.com/p/pcapr/wiki/Xtractr
+
+##Requirements
+
+>WARNING:
+The current version of Dorothy only utilizes VMWare ESX5 as its Virtual Sandbox Module (VSM). Thus, the free version of ESXi is not supported due to its limitations in using the
+vSphere 5 API.
+However, the overall framework could be easily customized in order to use another virtualization engine. Dorothy2 is
+very modular,and any customization or modification is very welcome.
+
+Dorothy needs the following software (not expressly in the same host) in order to be executed:
+
+* VMWare ESX >= 5.0  (tip: if you download ESXi, you can evaluate ESX for 30 days)
+* Ruby 1.8.7
+* Postgres >= 9.0
+* At least one Windows virtual machine
+* One unix-like machine dedicated to the Network Analysis Engine(NAM) (tcpdump/ssh needed)
+* [pcapr-local](https://github.com/mudynamics/pcapr-local )  (only used by doroParser)
+* MaxMind libraries (only used by doroParser)
+
+
+## Installation
+
+It is recommended to follow this step2step process:
+
+1. Set your ESX environment
+  * Sample setup
+2. Install the required software
+3. Install Dorothy and libmagic libraries
+4. Start Dorothy, and configure it
+5. Use Dorothy
+
+### 1. Set your ESX environment
+1. Basic configuration (ssh)
+ * From vSphere:
+
+            Configuration->Security Profile->Services->Proprieties->SSH->Options->Start and Stop with host->Start->OK
+
+2. Configure two separate virtual networks, one dedicated exclusively to the SandBoxes (See Sample Setups)
+
+3. Configure the Windows VMs used for sandboxing
+
+ * Disable Windows firewall (preferred)
+ * VMWare Tools must be installed in the Windows guest system.
+ * Configure a static IP
+ * After configuring everything on the Guest OS, create a snapshot of the sandbox VM from vSphere console. Dorothy will use it when reverting the VM after a binary execution.
+
+3. Configure the unix VM dedicated to the NAM
+     * Configure the NIC on the virtual machine that will be used for the network sniffing purpose (NAM).
+       >The vSwitch where the vNIC resides must allow the promisc mode, to enable it from vSphere:
+
+       >Configuration->Networking->Proprieties on the vistualSwitch used for the analysis->Double click on the virtual network used for the analysis->Securiry->Tick "Promiscuous Mode", then select "Accept" from the list menu.
+
+     * Install tcpdump and sudo
+
+                #apt-get install tcpdump sudo
+
+     * Create a dedicated user for dorothy (e.g. "dorothy")
+
+                #useradd dorothy
+     * Add dorothy's user permission to execute/kill tcpdump to the sudoers file:
+
+                #visudo
+                add the following line:
+                dorothy  ALL = NOPASSWD: /usr/sbin/tcpdump, /bin/kill
+
+#### * Sample Setups
+1. Basic setup
+> In the following example, the Dorothy gem is installed in the same host where Dorothive (the DB) resides.
+> This setup is strongly recommended
+
+    >![dorothy.basicsetup](http://www.honeynet.it/wp-content/uploads/2011/04/Dorothy-Basic.pdf)
+
+2. Advanced setup
+> This setup is recommended if Dorothy is going to be installed in a Corporate environment.
+> By leveraging a private VPN, all the sandbox traffics exits from the Corporate network with an external IP addresses.
+
+ >![dorothy.basicsetup](http://www.honeynet.it/wp-content/uploads/2011/04/Setup-Advanced.pdf)
+
+### 2. Install the required software
+
+
+1. Install postgres
+
+        $sudo apt-get install postgresql-9.1
+or
+
+        http://www.postgresql.org/download/
+
+2. Configure a dedicated postgres user for Dorothy (or use root user instead, up to you :)
+
+3. Install the following packages
+
+        $sudo apt-get install ruby1.8 rubygems postgresql-server-dev-9.1 libxml2-dev  libxslt1-dev libmagic-dev
+
+>For OSX users: all the above software are available through mac ports. A tip for libmagic: use brew instead:
+>
+        $ brew install libmagic
+        $ brew link libmagic
+
+Add a user dedicated to dorothy (or use the root one, up to you :)
+
+### 3. Install Dorothy gem
+
+*Install Dorothy gem
+
+        $ sudo gem install dorothy2
+
+### 4. Start Dorothy, and configure it!
+
+0. Install MaxMind libraries
+    * [GeoLiteCity](http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz)
+    * [GeoLite ASN](http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz)
+    * Copy GeoLiteCity.dat and GeoIPASNum.dat into Dorothy's etc/geo/ folder
+
+1. Start Dorothy
+
+        $ dorothy_start -v
+The following message should appear
+
+        [WARNING] It seems that the Dorothy configuration file is not present,
+        please answer to the following question in order to create it now.
+
+2. Follow the instruction to configure
+    * The environment variables (db, esx server, etc)
+    * The Dorothy sources (where to get new binaries)
+    * The ESX Virtual machines used for the analysis
+
+The first time you execute Dorothy, it will ask you to fill those information in order to create the required configuration files into the etc/ folder. However, you are free to modify/create such files directly - configuration example files can be found there too.
+
+###5. Use Dorothy
+1. Copy a .exe or .bat file into $yourdorothyhome/opt/bins/manual/
+2. Execute dorothy with the malwarefolder source type (if you left the default one)
+
+    $ dorothy_start -v -s malwarefolder
+
+
+## Usage
+
+	Usage:
+	$./dorothy_start [options]
+	where [options] are:
+       --verbose, -v:   Enable verbose mode
+      --infoflow, -i:   Print the analysis flow
+    --source, -s <s>:   Choose a source (from the ones defined in etc/sources.yml)
+        --daemon, -d:   Stay in the background, by constantly pooling datasources
+ --SandboxUpdate, -S:   Update Dorothive with the new Sandbox file
+ --DorothiveInit, -D:   (RE)Install the Dorothy Database (Dorothive)
+          --help, -h:   Show this message
+
+
+ >Example
+
+ >   $dorothy_start -v -s malwarefolder
+ >   $dorothy_stop
+
+###6. Debugging problems
+
+I recognize that setting up Dorothy is not the easiest task of the world.
+By considering that the whole framework consists in the union of several 3rd pats, it is very likely that one of them will fail during the process.
+Below there are some tips about how understand the root-cause of your crash.
+
+1. Execute the Dorothy UnitTest (tc_dorothy_full.rb) that resides in its gem home directory
+
+ >Example
+
+    $cd /opt/local/lib/ruby/gems/1.8/gems/dorothy2-0.0.1/test/
+    $ruby tc_dorothy_full.rb
+
+2. Set the verbose flag (-v) while executing dorothy
+
+> $dorothy_start -v -s malwarefolder
+
+3. Drop an email to info at honeynet.it with the output of your errors :)
+
+------------------------------------------
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+Every contribution is more than welcome!
+For any help, please don't hesitate in contacting us at :
+info at honeynet.it
+
+## License
+
+Dorothy is copyrighted by Marco Riccardi and is licensed under the
+following GNU General Public License version 3.
+
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/bin/dorothy_start

@@ -0,0 +1,176 @@
+#!/usr/bin/env ruby
+
+# Copyright (C) 2010-2013 marco riccardi.
+# This file is part of Dorothy - http://www.honeynet.it/dorothy
+# See the file 'LICENSE' for copying permission.
+
+require 'rubygems'
+require 'trollop'
+require 'dorothy2'          #comment for testing/developmnet
+
+#load '../lib/dorothy2.rb'  #uncomment for testing/developmnet
+
+include Dorothy
+
+
+opts = Trollop.options do
+  banner <<-EOS
+
+   ####################################################
+   ##                                                ##
+   ##  The Dorothy Malware Analysis Framework 2.0    ##
+   ##                                                ##
+   ####################################################
+
+        marco.riccardi@honeynet.it
+	www.honeynet.it/dorothy
+
+
+	Usage:
+  dorothy_start [options]
+	where [options] are:
+  EOS
+
+
+  opt :verbose, "Enable verbose mode"
+  opt :infoflow, "Print the analysis flow"
+  opt :source, "Choose a source (from the ones defined in etc/sources.yml)", :type => :string
+  opt :daemon, "Stay in the backround, by constantly pooling datasources"
+  opt :SandboxUpdate, "Update Dorothive with the new Sandbox file"
+  opt :DorothiveInit, "(RE)Install the Dorothy Database (Dorothive)"
+
+end
+
+if opts[:infoflow]
+  puts "
+	The Dorothy Malware Analysis Framework 2.0
+	---------------Execution Flow-------------
+	#0) Fetch new malwares
+	#1) Start VM
+	#2) Copy File to VM
+	#3) Start Sniffer
+	#4) Execute file into VM
+	#5) Make screenshot
+	#6) Wait X minutes (configure X in the conf file)
+	#7) Stop Sniffer
+	#8) Download Screenshot and trafficdump
+	#9) Try to retreive malware info from VirusTotal
+	#10) Insert data into Dorothy-DB
+	------------------------------------------
+	"
+  exit(0)
+end
+
+puts "
+
+   ####################################################
+   ##                                                ##
+   ##  The Dorothy Malware Analysis Framework 2.0    ##
+   ##                                                ##
+   ####################################################
+
+"
+
+#VARS
+HOME = File.expand_path("..",File.dirname(__FILE__))
+VERBOSE = (opts[:verbose] ? true : false)
+daemon = (opts[:daemon] ? true : false)
+
+#DEFAULT CONF FILES
+#conf = HOME + '/etc/dorothy.yml'
+
+conf = "#{File.expand_path("~")}/.dorothy.yml"
+
+#LOAD ENV
+if Util.exists?(conf)
+  DoroSettings.load!(conf)
+  else
+  DoroConfig.create
+  exit(0)
+end
+
+#Logging
+logout = (daemon ? DoroSettings.env[:logfile] : STDOUT)
+LOGGER = DoroLogger.new(logout, DoroSettings.env[:logage])
+LOGGER.sev_threshold = DoroSettings.env[:loglevel]
+
+home = DoroSettings.env[:home]
+#check homefolder
+unless Util.exists?(home)
+  DoroConfig.init_home(home)
+end
+
+sfile = home + '/etc/sources.yml'
+sboxfile = home + '/etc/sandboxes.yml'
+
+#INIT DB Connector
+begin
+  db = Insertdb.new
+rescue => e
+  if e.inspect =~ /exist/
+    puts "WARNING".yellow + " The database doesn't exist yet. Press Enter to load the ddl into the DB"
+    gets
+    Util.init_db(true)
+    exit(0)
+  else
+    puts "ERROR".red + " Can't connect to the database"
+    puts e
+    exit(0)
+  end
+end
+
+
+if opts[:DorothiveInit]
+  Util.init_db
+  exit(0)
+end
+
+if opts[:SandboxUpdate]
+  puts "[Dorothy]".yellow + " Loading #{sboxfile} into Dorothive"
+  DoroConfig.init_sandbox(sboxfile)
+  puts "[Dorothy]".yellow + " Done."
+  exit(0)
+end
+
+if Util.exists?(sfile)
+  sources = YAML.load_file(sfile)
+  #check if all the source directories exist
+  sources.keys.each do |s|
+    unless Util.exists?("#{sources[s]["localdir"]}")
+      LOGGER.warn "INIT", "Warning, the source's localdir #{s} doesn't exist yet, I'm going to create it"
+      Dir.mkdir("#{sources[s]["localdir"]}")
+    end
+  end
+else
+  puts "[WARNING]".red + " A source file doesn't exist, please crate one into #{home}/etc. See the example file in #{HOME}/etc/sandboxes.yml.example"
+  exit(0)
+end
+
+unless Util.exists?(sboxfile)
+  puts "[WARNING]".red + " There is no sandbox configured yet. Please do it now."
+  DoroConfig.create_sandbox(sboxfile)
+  DoroConfig.init_sandbox(sboxfile)
+end
+
+#Check DB sandbox data
+if db.table_empty?("sandboxes")
+  puts "[WARNING]".red + " No sandbox found in Dorothive, the DB will be filled with " + sboxfile
+  DoroConfig.init_sandbox(sboxfile)
+end
+
+if opts[:source] && !sources.key?(opts[:source])
+  puts "[WARNING]".red + " The selected source is not yet configured.\nThe available sources are: "
+  puts sources.keys
+  exit(0)
+end
+
+db.close
+
+begin
+  Dorothy.start sources[opts[:source]], daemon
+rescue => e
+  LOGGER.error "Dorothy", "An error occurred: " + $!
+  LOGGER.debug "Dorothy", "#{e.inspect} --BACKTRACE:  #{e.backtrace}"
+  LOGGER.error "Dorothy", "Dorothy has been stopped"
+end
+

data/bin/dorothy_stop

@@ -0,0 +1,28 @@
+#!/usr/bin/env ruby
+
+# Copyright (C) 2013 marco riccardi.
+# This file is part of Dorothy - http://www.honeynet.it/dorothy
+# See the file 'LICENSE' for copying permission.
+
+
+
+require 'rubygems'
+require 'trollop'
+require 'dorothy2'
+
+#load '../lib/dorothy2.rb'
+
+include Dorothy
+
+conf = "#{File.expand_path("~")}/.dorothy.yml"
+DoroSettings.load!(conf)
+
+#Logging
+
+LOGGER = DoroLogger.new(DoroSettings.env[:logfile], DoroSettings.env[:logage])
+LOGGER.sev_threshold = DoroSettings.env[:loglevel]
+
+
+Dorothy.stop
+
+

data/bin/dparser_start

@@ -0,0 +1,66 @@
+#!/usr/bin/env ruby
+
+# Copyright (C) 2013 marco riccardi.
+# This file is part of Dorothy - http://www.honeynet.it/dorothy
+# See the file 'LICENSE' for copying permission.
+
+require 'rubygems'
+require 'trollop'
+require 'dorothy2'
+require 'doroParser'
+
+load '../lib/doroParser'
+
+include DoroParser
+
+
+
+opts = Trollop.options do
+  banner <<-EOS
+	
+	The Dorothy Malware Analysis Framework 2.0   
+	
+	www.honeynet.it 
+	
+	Usage:
+	"do-dissector.rb" [options]
+	where [options] are:
+  EOS
+
+
+  opt :verbose, "Enable verbose mode"
+  opt :nonetbios, "Hide Netbios communication"
+  opt :daemon, "Stay in the backroud, by constantly pooling datasources"
+
+end
+
+def get_time
+  time = Time.new
+  return time.utc.strftime("%Y-%m-%d %H:%M:%S")
+end
+
+
+NONETBIOS = opts[:nonetbios] ? true : false
+VERBOSE = opts[:verbose] ? true : false
+daemon = opts[:daemon] ? true : false
+
+
+conf = "#{File.expand_path("~")}/.dorothy.yml"
+DoroSettings.load!(conf)
+
+#Logging
+logout = (daemon ? DoroSettings.env[:logfile_parser] : STDOUT)
+LOGGER = DoroLogger.new(logout, DoroSettings.env[:logage])
+LOGGER.sev_threshold = DoroSettings.env[:loglevel]
+
+begin
+  DoroParser.start daemon
+rescue => e
+  LOGGER.error "Parser", "An error occurred: " + $!
+  LOGGER.debug "Parser", "#{e.inspect} --BACKTRACE: #{e.backtrace}"
+  LOGGER.error "Parser", "Dorothy-Parser has been stopped"
+end
+
+
+
+

data/bin/dparser_stop

@@ -0,0 +1,23 @@
+#!/usr/bin/env ruby
+
+# Copyright (C) 2013 marco riccardi.
+# This file is part of Dorothy - http://www.honeynet.it/dorothy
+# See the file 'LICENSE' for copying permission.
+
+
+require 'rubygems'
+require 'trollop'
+require 'dorothy2'
+require 'doroParser'
+
+#load '../lib/doroParser'
+
+include DoroParser
+
+
+#Logging
+LOGGER_PARSER = DoroLogger.new(LOGFILE_PARSER, LOGAGE)
+LOGGER_PARSER.sev_threshold = LOGLEVEL
+
+DoroParser.stop
+

data/dorothy2.gemspec

@@ -0,0 +1,30 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'dorothy2/version'
+
+Gem::Specification.new do |gem|
+  gem.name          = "dorothy2"
+  gem.version       = Dorothy2::VERSION
+  gem.authors       = ["marco riccardi"]
+  gem.email         = ["marco.riccardi@honeynet.it"]
+  gem.description   = %q{A malware/botnet analysis framework written in Ruby.}
+  gem.summary       = %q{More info at http://www.honeynet.it}
+  gem.homepage      = "https://github.com/m4rco-/dorothy2"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+  gem.add_dependency(%q<net-scp>, [">= 1.0.4"])
+  gem.add_dependency(%q<net-ssh>, [">= 2.2.1"])
+  gem.add_dependency(%q<trollop>, [">= 1.16.2"])
+  gem.add_dependency(%q<rest-client>, [">= 1.6.1"])
+  gem.add_dependency(%q<mime-types>, [">= 1.16"])
+  gem.add_dependency(%q<colored>, [">= 1.2"])
+  gem.add_dependency(%q<ruby-pg>, [">= 0.7.9.2008.01.28"])
+  gem.add_dependency(%q<virustotal>, [">= 2.0.0"])
+  gem.add_dependency(%q<rbvmomi>, [">= 1.3.0"])
+  gem.add_dependency(%q<ruby-filemagic>, [">= 0.4.2"])
+
+end