ronin-web 0.1.3 → 0.2.0

data/History.txt

@@ -1,3 +1,28 @@
+=== 0.2.0 / 2009-09-24
+
+* Require ronin >= 0.3.0.
+* Require mechanize >= 0.9.3.
+* Require sinatra >= 0.9.4.
+* Require rspec >= 1.2.8.
+* Require test-unit >= 1.2.3.
+* Require rack-test >= 0.4.1.
+* Require yard >= 0.2.3.5.
+* Added Ronin::Web::Server::Base.
+* Added Ronin::Web::Server::Files.
+* Added Ronin::Web::Server::Hosts.
+* Added Ronin::Web::Server::Proxy.
+* Added Ronin::Web::Server::Helpers.
+* Added Ronin::Web::Server::Helpers::Files.
+* Added Ronin::Web::Server::Helpers::Hosts.
+* Added Ronin::Web::Server::Helpers::Proxy.
+* Added Ronin::Web::Server::App.
+* Added Ronin::Web.server.
+* Renamed Ronin::Sessions::Web to Ronin::Network::Helpers::Web.
+* Moved to YARD based documentation.
+* Updated the project summary and 3-point description for Ronin Web.
+* Refactored the Ronin::Web::Server to build ontop of Sinatra.
+* Refactored Ronin::Network::Helpers::Web.
+
 === 0.1.3 / 2009-07-02
 
 * Use Hoe >= 2.0.0.

data/Manifest.txt

@@ -1,10 +1,8 @@
 History.txt
 Manifest.txt
 README.txt
-TODO.txt
 Rakefile
 bin/ronin-web
-lib/ronin/sessions/web.rb
 lib/ronin/web.rb
 lib/ronin/web/extensions.rb
 lib/ronin/web/extensions/nokogiri.rb
@@ -14,16 +12,50 @@ lib/ronin/web/extensions/nokogiri/xml/text.rb
 lib/ronin/web/extensions/nokogiri/xml/attr.rb
 lib/ronin/web/extensions/nokogiri/xml/element.rb
 lib/ronin/web/extensions/nokogiri/xml/document.rb
-lib/ronin/web/fingerprint.rb
 lib/ronin/web/server.rb
+lib/ronin/web/server/helpers.rb
+lib/ronin/web/server/helpers/rendering.rb
+lib/ronin/web/server/helpers/files.rb
+lib/ronin/web/server/helpers/hosts.rb
+lib/ronin/web/server/helpers/proxy.rb
+lib/ronin/web/server/helpers/files.rb
+lib/ronin/web/server/helpers/hosts.rb
+lib/ronin/web/server/base.rb
+lib/ronin/web/server/files.rb
+lib/ronin/web/server/hosts.rb
+lib/ronin/web/server/proxy.rb
+lib/ronin/web/server/app.rb
+lib/ronin/web/server/web.rb
 lib/ronin/web/proxy.rb
+lib/ronin/web/proxy/base.rb
+lib/ronin/web/proxy/app.rb
+lib/ronin/web/proxy/web.rb
 lib/ronin/web/spider.rb
 lib/ronin/web/web.rb
 lib/ronin/web/version.rb
+lib/ronin/network/helpers/web.rb
 tasks/spec.rb
+tasks/yard.rb
 spec/spec_helper.rb
 spec/web/helpers/server.rb
 spec/web/helpers/root/index.html
 spec/web/helpers/root/test.txt
 spec/web/extensions/nokogiri_spec.rb
-spec/web/server_spec.rb
+spec/web/server/classes/public1/static1.txt
+spec/web/server/classes/public2/static2.txt
+spec/web/server/classes/files/dir/file.txt
+spec/web/server/classes/files/dir/index.html
+spec/web/server/classes/files/dir2/file2.txt
+spec/web/server/classes/files/dir3/page.xml
+spec/web/server/classes/files/file.txt
+spec/web/server/classes/sub_app.rb
+spec/web/server/classes/test_app.rb
+spec/web/server/classes/files_app.rb
+spec/web/server/classes/hosts_app.rb
+spec/web/server/classes/proxy_app.rb
+spec/web/server/helpers/server.rb
+spec/web/server/base_spec.rb
+spec/web/server/files_spec.rb
+spec/web/server/hosts_spec.rb
+spec/web/server/proxy_spec.rb
+spec/web/proxy/base_spec.rb

data/README.txt

@@ -2,7 +2,7 @@
 
 * http://ronin.rubyforge.org/web/
 * http://github.com/postmodern/ronin-web
-* irc.freenode.net ##ronin
+* irc.freenode.net #ronin
 * Postmodern (postmodern.mod3 at gmail.com)
 
 == DESCRIPTION:
@@ -10,52 +10,38 @@
 Ronin Web is a Ruby library for Ronin that provides support for web
 scraping and spidering functionality.
 
-Ronin is a Ruby platform designed for information security and data
-exploration tasks. Ronin allows for the rapid development and distribution
-of code over many of the common Source-Code-Management (SCM) systems.
+Ronin is a Ruby platform for exploit development and security research.
+Ronin allows for the rapid development and distribution of code, exploits
+or payloads over many common Source-Code-Management (SCM) systems.
 
-=== Free
+=== Ruby
 
-All source code within Ronin is licensed under the GPL-2, therefore no user
-will ever have to pay for Ronin or updates to Ronin. Not only is the
-source code free, the Ronin project will not sell enterprise grade security
-snake-oil solutions, give private training classes or later turn Ronin into
-commercial software.
+Ronin's Ruby environment allows security researchers to leverage Ruby with
+ease. The Ruby environment contains a multitude of convenience methods
+for working with data in Ruby, a Ruby Object Database, a customized Ruby
+Console and an extendable command-line interface.
 
-=== Modular
+=== Extend
 
-Ronin was not designed as one monolithic framework but instead as a
-collection of libraries which can be individually installed. This allows
-users to pick and choose what functionality they want in Ronin.
+Ronin's more specialized features are provided by additional Ronin
+libraries, which users can choose to install. These libraries can allow
+one to write and run Exploits and Payloads, scan for PHP vulnerabilities,
+perform Google Dorks  or run 3rd party scanners.
 
-=== Decentralized
+=== Publish
 
-Ronin does not have a central repository of exploits and payloads which
-all developers contribute to. Instead Ronin has Overlays, repositories of
-code that can be hosted on any CVS/SVN/Git/Rsync server. Users can then use
-Ronin to quickly install or update Overlays. This allows developers and
-users to form their own communities, independent of the main developers
-of Ronin.
+Ronin allows users to publish and share code, exploits, payloads or other
+data via Overlays. Overlays are directories of code and data that can be
+hosted on any SVN, Hg, Git or Rsync server. Ronin makes it easy to create,
+install or update Overlays.
 
-== FEATURES/PROBLEMS:
+== FEATURES:
 
 * Web access (utilizing Mechanize and Nokogiri).
 * Integrates Spidr into Ronin::Web::Spider.
-* Provides Ronin::Web::Server, a customizable Rack based Web Server that
+* Provides Ronin::Web::Server, a customizable Sinatra based Web Server that
   supports path and host-name routing.
-* Provides Ronin::Web::Proxy, a configurable Rack based Web Proxy.
-
-== REQUIREMENTS:
-
-* {nokogiri}[http://nokogiri.rubyforge.org/] >= 1.1.0
-* {mechanize}[http://mechanize.rubyforge.org/] >= 0.9.0
-* {spidr}[http://spidr.rubyforge.org/] >= 0.1.3
-* {rack}[http://rack.rubyforge.org/] >= 0.9.1
-* {ronin}[http://ronin.rubyforge.org/] >= 0.2.2
-
-== INSTALL:
-
-  $ sudo gem install ronin-web
+* Provides Ronin::Web::Proxy, a configurable Sinatra based Web Proxy.
 
 == SYNOPSIS:
 
@@ -81,11 +67,11 @@ of Ronin.
 
     Web.html(open('some_file.html'))
     # => <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
-    <html>
-      <head>
-        <script type="text/javascript" src="redirect.js"></script>
-      </head>
-    </html>
+    # <html>
+    #   <head>
+    #     <script type="text/javascript" src="redirect.js"></script>
+    #   </head>
+    # </html>
 
 * Build a HTML document:
 
@@ -98,19 +84,19 @@ of Ronin.
     end
     
     puts doc.to_html
-    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
-    <html><head><script src="redirect.js" type="text/javascript"></script></head></html>
+    # <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
+    # <html><head><script src="redirect.js" type="text/javascript"></script></head></html>
 
 * Parse XML:
 
     Web.xml(some_text)
-    => <?xml version="1.0"?>
-    <users>
-      <user>
-        <name>admin</name>
-        <password>0mni</password>
-      </user>
-    </users>
+    # => <?xml version="1.0"?>
+    # <users>
+    #   <user>
+    #     <name>admin</name>
+    #     <password>0mni</password>
+    #   </user>
+    # </users>
 
 
 * Build a XML document:
@@ -127,35 +113,52 @@ of Ronin.
     end
     
     puts doc.to_xml
-    <?xml version="1.0"?>
-    <playlist>
-      <mp3>
-        <file>02 THE WAIT.mp3</file>
-        <artist>Evil Nine</artist>
-        <track>The Wait feat David Autokratz</track>
-        <duration>1000000000</duration>
-      </mp3>
-    </playlist>
+    # <?xml version="1.0"?>
+    # <playlist>
+    #   <mp3>
+    #     <file>02 THE WAIT.mp3</file>
+    #     <artist>Evil Nine</artist>
+    #     <track>The Wait feat David Autokratz</track>
+    #     <duration>1000000000</duration>
+    #   </mp3>
+    # </playlist>
 
 * Spider a web site:
 
     Web::Spider.host('www.example.com') do |spider|
       spider.every_url do |url|
-        ...
+        # ...
       end
 
       spider.every_page do |page|
-        ...
+        # ...
       end
     end
 
 * Serve files via a Web Server:
 
-    Web::Server.start do |server|
-      server.file '/opensearch.xml', '/tmp/test.xml'
-      server.directory '/download/', '/tmp/download/'
+    require 'ronin/web/server'
+
+    Web.server do
+      file '/opensearch.xml', '/tmp/test.xml'
+      directory '/download/', '/tmp/download/'
     end
 
+    Web.server.get '/test' do
+      'Test 1 2 1 2'
+    end
+
+== REQUIREMENTS:
+
+* {mechanize}[http://mechanize.rubyforge.org/] >= 0.9.3
+* {spidr}[http://spidr.rubyforge.org/] >= 0.1.9
+* {sinatra}[http://www.sinatrarb.com/] >= 0.9.4
+* {ronin}[http://ronin.rubyforge.org/] >= 0.3.0
+
+== INSTALL:
+
+  $ sudo gem install ronin-web
+
 == LICENSE:
 
 Ronin Web - A Ruby library for Ronin that provides support for web

data/Rakefile

@@ -4,17 +4,26 @@ require 'rubygems'
 require 'hoe'
 require 'hoe/signing'
 require './tasks/spec.rb'
+require './tasks/yard.rb'
 
 Hoe.spec('ronin-web') do
   self.rubyforge_name = 'ronin'
   self.developer('Postmodern', 'postmodern.mod3@gmail.com')
   self.remote_rdoc_dir = 'docs/ronin-web'
   self.extra_deps = [
-    ['mechanize', '>=0.9.0'],
+    ['mechanize', '>=0.9.3'],
     ['spidr', '>=0.1.9'],
-    ['rack', '>=1.0.0'],
-    ['ronin', '>=0.2.4']
+    ['sinatra', '>=0.9.4'],
+    ['ronin', '>=0.3.0']
   ]
+
+  self.extra_dev_deps = [
+    ['rspec', '>=1.2.8'],
+    ['test-unit', '=1.2.3'],
+    ['rack-test', '>=0.4.1']
+  ]
+
+  self.spec_extras = {:has_rdoc => 'yard'}
 end
 
 # vim: syntax=Ruby

data/bin/ronin-web

@@ -11,4 +11,4 @@ require 'ronin/ui/command_line/commands/console'
 require 'ronin/ui/console'
 
 Ronin::UI::Console.auto_load << 'ronin/web'
-Ronin::UI::CommandLine::Commands::Console.run(*ARGV)
+Ronin::UI::CommandLine::Commands::Console.start

data/lib/ronin/network/helpers/web.rb

@@ -0,0 +1,221 @@
+#
+# Ronin Web - A Ruby library for Ronin that provides support for web
+# scraping and spidering functionality.
+#
+# Copyright (c) 2006-2009 Hal Brodigan (postmodern.mod3 at gmail.com)
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+
+require 'ronin/network/helpers/helper'
+require 'ronin/network/http/proxy'
+require 'ronin/web/web'
+
+module Ronin
+  module Network
+    module Helpers
+      module Web
+        include Helper
+
+        protected
+
+        #
+        # Combines the proxy information set by the +@web_proxy_host+,
+        # +@web_proxy_port+, +@web_proxy_user+ and +@web_proxy_password+
+        # instance variables.
+        #
+        # @return [Network::HTTP::Proxy]
+        #   The current proxy information.
+        #
+        def web_proxy
+          HTTP::Proxy.new(
+            :host => @web_proxy_host,
+            :port => @web_proxy_port,
+            :user => @web_proxy_user,
+            :password => @web_proxy_password
+          )
+        end
+
+        #
+        # Provides a persistant Mechanize agent.
+        #
+        # @param [Hash] options
+        #   Additional options for initializing the agent.
+        #
+        # @option options [Hash] :proxy (web_proxy)
+        #   Proxy information.
+        #
+        # @option options [String] :user_agent (+@web_user_agent+)
+        #   User-Agent string to use.
+        #
+        # @return [WWW::Mechanize]
+        #   The agent.
+        #
+        def web_agent(options={},&block)
+          unless @web_agent
+            options[:proxy] ||= web_proxy
+            options[:user_agent] ||= @web_user_agent
+
+            @web_agent = Ronin::Web.agent(options,&block)
+          end
+
+          return @web_agent
+        end
+
+        #
+        # Creates a Mechanize Page for the contents at a given URL.
+        #
+        # @param [URI::Generic, String] url
+        #   The URL to request.
+        #
+        # @param [Hash] options
+        #   Additional options to initialize the agent with.
+        #
+        # @option options [Hash] :proxy (web_proxy)
+        #   Proxy information.
+        #
+        # @option options [String] :user_agent (+@web_user_agent+)
+        #   User-Agent string to use.
+        #
+        # @yield [page]
+        #   If a block is given, it will be passed the page for the
+        #   requested URL.
+        #
+        # @yieldparam [WWW::Mechanize::Page] page
+        #   The requested page.
+        #
+        # @return [WWW::Mechanize::Page]
+        #   The requested page.
+        #
+        def web_get(url,options={},&block)
+          page = web_agent(options).get(url)
+
+          block.call(page) if block
+          return page
+        end
+
+        #
+        # Requests the body of the Mechanize Page created from the response
+        # of the given URL.
+        #
+        # @param [URI::Generic, String] url
+        #   The URL to request.
+        #
+        # @param [Hash] options
+        #   Additional options to initialize the agent with.
+        #
+        # @option options [Hash] :proxy (web_proxy)
+        #   Proxy information.
+        #
+        # @option options [String] :user_agent (+@web_user_agent+)
+        #   User-Agent string to use.
+        #
+        # @yield [body]
+        #   If a block is given, it will be passed the body of the page.
+        #
+        # @yieldparam [String] body
+        #   The requested body of the page.
+        #
+        # @return [String]
+        #   The requested body of the page.
+        #
+        def web_get_body(url,options={},&block)
+          web_get(url,options) do |page|
+            body = page.body
+
+            block.call(body) if block
+            return body
+          end
+        end
+
+        #
+        # Posts to a given URL and creates a Mechanize Page from the
+        # response.
+        #
+        # @param [URI::Generic, String] url
+        #   The URL to post to.
+        #
+        # @param [Hash] options
+        #   Additional options to initialize the agent with.
+        #
+        # @option options [Hash] :query
+        #   Additional query parameters to post with.
+        #
+        # @option options [Hash] :proxy (web_proxy)
+        #   Proxy information.
+        #
+        # @option options [String] :user_agent (+@web_user_agent+)
+        #   User-Agent string to use.
+        #
+        # @yield [page]
+        #   If a block is given, it will be passed the page for the
+        #   requested URL.
+        #
+        # @yieldparam [WWW::Mechanize::Page] page
+        #   The requested page.
+        #
+        # @return [WWW::Mechanize::Page]
+        #   The requested page.
+        #
+        def web_post(url,options={},&block)
+          query = {}
+          query.merge!(options[:query]) if options[:query]
+
+          page = web_agent(options).post(url)
+
+          block.call(page) if block
+          return page
+        end
+
+        #
+        # Posts to a given URL and returns the body of the Mechanize Page
+        # created from the response.
+        #
+        # @param [URI::Generic, String] url
+        #   The URL to post to.
+        #
+        # @param [Hash] options
+        #   Additional options to initialize the agent with.
+        #
+        # @option options [Hash] :query
+        #   Additional query parameters to post with.
+        #
+        # @option options [Hash] :proxy (web_proxy)
+        #   Proxy information.
+        #
+        # @option options [String] :user_agent (+@web_user_agent+)
+        #   User-Agent string to use.
+        #
+        # @yield [body]
+        #   If a block is given, it will be passed the body of the page.
+        #
+        # @yieldparam [WWW::Mechanize::Page] page
+        #   The body of the requested page.
+        #
+        # @return [String]
+        #   The requested body of the page.
+        #
+        def web_post_body(url,options={},&block)
+          web_post(url,options) do |page|
+            body = page.body
+
+            block.call(body) if block
+            return body
+          end
+        end
+      end
+    end
+  end
+end