ronin-web 0.2.1 → 0.3.0.pre1

data/{History.txt → ChangeLog.md}

@@ -1,4 +1,35 @@
-=== 0.2.1 / 2009-10-18
+### 0.3.0 / 2011-05-27
+
+* Upgraded to the GPL-3 license.
+* Require nokogiri ~> 1.4.
+* Require mechanize ~> 1.0.
+* Require spidr ~> 0.2.
+* Require sinatra ~> 1.2.
+* Require data_paths ~> 0.3.
+* Require ronin-support ~> 0.2.
+* Require ronin ~> 1.1.
+* Added {Ronin::Web::Middleware::Base}.
+* Added {Ronin::Web::Middleware::Request}.
+* Added {Ronin::Web::Middleware::Response}.
+* Added {Ronin::Web::Middleware::Helpers}.
+* Added {Ronin::Web::Middleware::Files}.
+* Added {Ronin::Web::Middleware::Directories}.
+* Added {Ronin::Web::Middleware::Rule}.
+* Added {Ronin::Web::Middleware::Filters::CampaignFilter}.
+* Added {Ronin::Web::Middleware::Filters::IPFilter}.
+* Added {Ronin::Web::Middleware::Filters::PathFilter}.
+* Added {Ronin::Web::Middleware::Filters::RefererFilter}.
+* Added {Ronin::Web::Middleware::Filters::UserAgentFilter}.
+* Added {Ronin::Web::Middleware::Filters::VHostFilter}.
+* Added {Ronin::Web::Middleware::Router}.
+* Added {Ronin::Web::Middleware::Proxy}.
+* Renamed `Ronin::Helpers::Web` to {Ronin::Network::Mixins::Web}.
+* Renamed `Ronin::Web::Server::Proxy` to {Ronin::Web::Proxy}.
+* Switched from [Jeweler](https://github.com/technicalpickles/jeweler)
+  to [Ore](http://github.com/ruby-ore/ore) and [Bundler](http://gembundler.com).
+* Opted into [test.rubygems.org](http://test.rubygems.org/).
+
+### 0.2.1 / 2009-10-18
 
 * Require spidr >= 0.2.0.
 * Added Ronin::Scanners::Web.
@@ -6,7 +37,7 @@
 * Renamed Ronin::Web.proxy to Ronin::Web.proxy_server, to not conflict
   with the original Ronin::Web.proxy method.
 
-=== 0.2.0 / 2009-09-24
+### 0.2.0 / 2009-09-24
 
 * Require ronin >= 0.3.0.
 * Require mechanize >= 0.9.3.
@@ -31,7 +62,7 @@
 * Refactored the Ronin::Web::Server to build ontop of Sinatra.
 * Refactored Ronin::Network::Helpers::Web.
 
-=== 0.1.3 / 2009-07-02
+### 0.1.3 / 2009-07-02
 
 * Use Hoe >= 2.0.0.
 * Require spidr >= 0.1.9.
@@ -49,7 +80,7 @@
 * Removed Server.run.
 * Removed Server#config.
 
-=== 0.1.2 / 2009-03-28
+### 0.1.2 / 2009-03-28
 
 * Added Ronin::Web::Proxy.
 * Added diagnostic messages to Ronin::Web::Spider.
@@ -59,10 +90,10 @@
     standard env Hash.
 * Updated specs for Ronin::Web::Server.
 
-=== 0.1.1 / 2009-02-23
+### 0.1.1 / 2009-02-23
 
-* Added a git style sub-command (<tt>ronin-web</tt>) which starts the Ronin
-  console with <tt>ronin/web</tt> preloaded.
+* Added a git style sub-command (`ronin-web`) which starts the Ronin
+  console with `ronin/web` preloaded.
 * Require Nokogiri >= 1.2.0.
 * Require Ronin >= 0.2.1.
 * Updated Ronin::Web::Server:
@@ -72,7 +103,7 @@
 * Fixed a bug when loading the Nokogiri extensions with Nokogiri >= 1.2.0.
 * Updated README.txt.
 
-=== 0.1.0 / 2009-01-22
+### 0.1.0 / 2009-01-22
 
 * Initial release.
   * Provides Web.html and Web.xml methods, which use Nokogiri::HTML and

data/Gemfile

@@ -0,0 +1,49 @@
+source 'https://rubygems.org'
+
+DM_URI = 'http://github.com/datamapper'
+DM_VERSION = '~> 1.1.0'
+RONIN_URI = 'http://github.com/ronin-ruby'
+
+gemspec
+
+# gem 'rack',         '~> 1.2', :git => 'http://github.com/rack/rack.git'
+# gem 'sinatra',      '~> 1.2', :git => 'http://github.com/sinatra/sinatra.git'
+
+# Ronin dependencies
+# gem 'ronin-support',	'~> 0.2', :git => "#{RONIN_URI}/ronin-support.git"
+# gem 'ronin',		      '~> 1.1', :git => "#{RONIN_URI}/ronin.git"
+
+group :test do
+  gem 'rack',         '~> 1.2', :git => 'http://github.com/rack/rack.git'
+  gem 'rack-test',    '~> 0.6.0'
+end
+
+group :development do
+  gem 'rake',		      '~> 0.8.7'
+
+  gem 'ore-tasks',    '~> 0.4'
+  gem 'rspec',        '~> 2.4'
+
+  gem 'kramdown',     '~> 0.12'
+end
+
+#
+# To enable additional DataMapper adapters for development work or for
+# testing purposes, simple set the ADAPTER or ADAPTERS environment
+# variable:
+#
+#     export ADAPTER="postgres"
+#     bundle install
+#
+#     ./bin/ronin --database postgres://ronin@localhost/ronin
+#
+require 'set'
+
+DM_ADAPTERS = Set['postgres', 'mysql', 'oracle', 'sqlserver']
+
+adapters = (ENV['ADAPTER'] || ENV['ADAPTERS']).to_s
+adapters = Set.new(adapters.to_s.tr(',',' ').split)
+
+(DM_ADAPTERS & adapters).each do |adapter|
+  gem "dm-#{adapter}-adapter", DM_VERSION #, :git => "#{DM_URI}/dm-#{adapter}-adapter.git"
+end

data/{README.txt → README.md}

@@ -1,11 +1,12 @@
-= Ronin Web
+# Ronin Web
 
-* http://ronin.rubyforge.org/web/
-* http://github.com/postmodern/ronin-web
+* [Source](http://github.com/ronin-ruby/ronin-web)
+* [Issues](http://github.com/ronin-ruby/ronin-web/issues)
+* [Documentation](http://rubydoc.info/github/ronin-ruby/ronin-web/frames)
+* [Mailing List](http://groups.google.com/group/ronin-ruby)
 * irc.freenode.net #ronin
-* Postmodern (postmodern.mod3 at gmail.com)
 
-== DESCRIPTION:
+## Description
 
 Ronin Web is a Ruby library for Ronin that provides support for web
 scraping and spidering functionality.
@@ -14,56 +15,65 @@ Ronin is a Ruby platform for exploit development and security research.
 Ronin allows for the rapid development and distribution of code, exploits
 or payloads over many common Source-Code-Management (SCM) systems.
 
-=== Ruby
+### Ruby
 
 Ronin's Ruby environment allows security researchers to leverage Ruby with
 ease. The Ruby environment contains a multitude of convenience methods
 for working with data in Ruby, a Ruby Object Database, a customized Ruby
 Console and an extendable command-line interface.
 
-=== Extend
+### Extend
 
 Ronin's more specialized features are provided by additional Ronin
 libraries, which users can choose to install. These libraries can allow
 one to write and run Exploits and Payloads, scan for PHP vulnerabilities,
 perform Google Dorks  or run 3rd party scanners.
 
-=== Publish
+### Publish
 
 Ronin allows users to publish and share code, exploits, payloads or other
 data via Overlays. Overlays are directories of code and data that can be
 hosted on any SVN, Hg, Git or Rsync server. Ronin makes it easy to create,
 install or update Overlays.
 
-== FEATURES:
+## Features
 
-* Web access (utilizing Mechanize and Nokogiri).
-* Integrates Spidr into Ronin::Web::Spider.
-* Provides Ronin::Web::Server, a customizable Sinatra based Web Server that
-  supports path and host-name routing.
-* Provides Ronin::Web::Proxy, a configurable Sinatra based Web Proxy.
+* Web access (utilizing [Mechanize](http://rubydoc.info/gems/mechanize/frames)
+  and [Nokogiri](http://rubydoc.info/gems/nokogiri/frames).
+* Integrates [Spidr](http://github.com/postmodern/spidr#readme) into
+  {Ronin::Web::Spider}.
+* Provides a collection of useful [Rack](http://rack.rubyforge.org/)
+  Middleware:
+  * {Ronin::Web::Middleware::Files}
+  * {Ronin::Web::Middleware::Directories}
+  * {Ronin::Web::Middleware::Proxy}
+  * {Ronin::Web::Middleware::Router}
+* Provides {Ronin::Web::Server}, a [Sinatra](http://sinatrarb.com/) based
+  Web Server.
+* Provides {Ronin::Web::Proxy}, a [Sinatra](http://sinatrarb.com/) based
+  Web Proxy.
 
-== SYNOPSIS:
+## Synopsis
 
-* Start the Ronin console with Ronin Web preloaded:
+Start the Ronin console with Ronin Web preloaded:
 
     $ ronin-web
 
-== EXAMPLES:
+## Examples
 
-* Get a web-page:
+Get a web-page:
 
     Web.get('http://www.rubyinside.com/')
 
-* Get only the body of the web-page:
+Get only the body of the web-page:
 
     Web.get_body('http://www.rubyinside.com/')
 
-* Get a WWW::Mechanize agent:
+Get a [Mechanize agent](http://rubydoc.info/gems/mechanize/1.0.0/Mechanize):
 
     agent = Web.agent
 
-* Parse HTML:
+Parse HTML:
 
     Web.html(open('some_file.html'))
     # => <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
@@ -73,7 +83,7 @@ install or update Overlays.
     #   </head>
     # </html>
 
-* Build a HTML document:
+Build a HTML document:
 
     doc = Web.build_html do
       html {
@@ -87,7 +97,7 @@ install or update Overlays.
     # <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
     # <html><head><script src="redirect.js" type="text/javascript"></script></head></html>
 
-* Parse XML:
+Parse XML:
 
     Web.xml(some_text)
     # => <?xml version="1.0"?>
@@ -99,7 +109,7 @@ install or update Overlays.
     # </users>
 
 
-* Build a XML document:
+Build a XML document:
 
     doc = Web.build_xml do
       playlist {
@@ -123,7 +133,7 @@ install or update Overlays.
     #   </mp3>
     # </playlist>
 
-* Spider a web site:
+Spider a web site:
 
     Web::Spider.host('www.example.com') do |spider|
       spider.every_url do |url|
@@ -135,7 +145,7 @@ install or update Overlays.
       end
     end
 
-* Serve files via a Web Server:
+Serve files via a Web Server:
 
     require 'ronin/web/server'
 
@@ -148,34 +158,40 @@ install or update Overlays.
       'Test 1 2 1 2'
     end
 
-== REQUIREMENTS:
+## Requirements
 
-* {mechanize}[http://mechanize.rubyforge.org/] >= 0.9.3
-* {spidr}[http://spidr.rubyforge.org/] >= 0.2.0
-* {sinatra}[http://www.sinatrarb.com/] >= 0.9.4
-* {ronin}[http://ronin.rubyforge.org/] >= 0.3.0
+* [nokogiri](http://github.com/tenderlove/nokogiri) ~> 1.4
+  * [libxml2](http://xmlsoft.org/)
+  * [libxslt1](http://xmlsoft.org/XSLT/)
+* [mechanize](http://github.com/tenderlove/mechanize) ~> 1.0
+* [spidr](http://github.com/postmodern/spidr) ~> 0.2
+* [sinatra](http://github.com/sinatra/sinatra) ~> 1.1
+* [data_paths](http://github.com/postmodern/data_paths) ~> 0.3
+* [ronin-support](http://github.com/ronin-ruby/ronin-support) ~> 0.2
+* [ronin](http://github.com/ronin-ruby/ronin) ~> 1.1
 
-== INSTALL:
+## Install
 
-  $ sudo gem install ronin-web
+    $ gem install ronin-web
 
-== LICENSE:
+## License
 
 Ronin Web - A Ruby library for Ronin that provides support for web
 scraping and spidering functionality.
 
-Copyright (c) 2006-2009 Hal Brodigan (postmodern.mod3 at gmail.com)
+Copyright (c) 2006-2011 Hal Brodigan (postmodern.mod3 at gmail.com)
 
-This program is free software; you can redistribute it and/or modify
+This file is part of Ronin Web.
+
+Ronin is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
-the Free Software Foundation; either version 2 of the License, or
+the Free Software Foundation, either version 3 of the License, or
 (at your option) any later version.
 
-This program is distributed in the hope that it will be useful,
+Ronin is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.
 
 You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+along with Ronin.  If not, see <http://www.gnu.org/licenses/>.

data/Rakefile

@@ -1,29 +1,30 @@
-# -*- ruby -*-
-
 require 'rubygems'
-require 'hoe'
-require 'hoe/signing'
-require './tasks/spec.rb'
-require './tasks/yard.rb'
-
-Hoe.spec('ronin-web') do
-  self.rubyforge_name = 'ronin'
-  self.developer('Postmodern', 'postmodern.mod3@gmail.com')
-  self.remote_rdoc_dir = 'docs/ronin-web'
-  self.extra_deps = [
-    ['mechanize', '>=0.9.3'],
-    ['spidr', '>=0.2.0'],
-    ['sinatra', '>=0.9.4'],
-    ['ronin', '>=0.3.0']
-  ]
 
-  self.extra_dev_deps = [
-    ['rspec', '>=1.2.8'],
-    ['test-unit', '=1.2.3'],
-    ['rack-test', '>=0.4.1']
-  ]
+begin
+  require 'bundler'
+rescue LoadError => e
+  STDERR.puts e.message
+  STDERR.puts "Run `gem install bundler` to install Bundler."
+  exit e.status_code
+end
 
-  self.spec_extras = {:has_rdoc => 'yard'}
+begin
+  Bundler.setup(:development)
+rescue Bundler::BundlerError => e
+  STDERR.puts e.message
+  STDERR.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
 end
 
-# vim: syntax=Ruby
+require 'rake'
+
+require 'ore/tasks'
+Ore::Tasks.new
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new
+task :test => :spec
+task :default => :spec
+
+require 'yard'
+YARD::Rake::YardocTask.new

data/bin/ronin-web

@@ -2,13 +2,26 @@
 
 require 'rubygems'
 
-lib_dir = File.expand_path(File.join(File.dirname(__FILE__),'..','lib'))
-unless $LOAD_PATH.include?(lib_dir)
-  $LOAD_PATH << lib_dir
+root_dir = File.expand_path(File.join(File.dirname(__FILE__),'..'))
+if File.directory?(File.join(root_dir,'.git'))
+  Dir.chdir(root_dir) do |path|
+    require 'bundler'
+
+    begin
+      Bundler.setup(:default)
+    rescue Bundler::BundlerError => e
+      STDERR.puts e.message
+      STDERR.puts "Run `bundle install` to install missing gems"
+      exit e.status_code
+    end
+  end
 end
 
-require 'ronin/ui/command_line/commands/console'
+lib_dir = File.join(root_dir,'lib')
+$LOAD_PATH << lib_dir unless $LOAD_PATH.include?(lib_dir)
+
 require 'ronin/ui/console'
+require 'ronin/ui/cli/commands/console'
 
 Ronin::UI::Console.auto_load << 'ronin/web'
-Ronin::UI::CommandLine::Commands::Console.start
+Ronin::UI::CLI::Commands::Console.start

data/gemspec.yml

@@ -0,0 +1,30 @@
+name: ronin-web
+summary:
+  A Ruby library for Ronin that provides support for web scraping and
+  spidering functionality.
+description:
+  Ronin Web is a Ruby library for Ronin that provides support for web
+  scraping and spidering functionality.
+
+license: GPL-3
+authors: Postmodern
+email: postmodern.mod3@gmail.com
+homepage: http://github.com/ronin-ruby/ronin-web
+has_yard: true
+
+dependencies:
+  nokogiri: ~> 1.4
+  mechanize: ~> 1.0
+  spidr: ~> 0.2
+  sinatra: ~> 1.2
+  # data_paths: ~> 0.3
+  data_paths: ~> 0.3.0.rc1
+  # Ronin dependencies:
+  # ronin-support: ~> 0.2
+  ronin-support: ~> 0.2.0.rc1
+  # ronin: ~> 1.1
+  ronin: ~> 1.1.0.rc1
+
+development_dependencies:
+  bundler: ~> 1.0.10
+  yard: ~> 0.7.0