ronin-web 0.1.0

data/History.txt

@@ -0,0 +1,15 @@
+=== 0.1.0 / 2009-01-22
+
+* Initial release.
+  * Provides Web.html and Web.xml methods, which use Nokogiri::HTML and
+    Nokogiri::XML respectively, to parse HTML and XML content.
+  * Supports custom Proxy and User-Agent strings, through
+    the Web.proxy and Web.user_agent methods respectively.
+  * Provides Web.agent, Web.get and Weg.post methods which access
+    WWW::Mechanize.
+  * Integrates Spidr::Agent into Web::Spider.
+  * Provides Web::Server, a customizable Rack web server that supports path
+    and host-name routing.
+    * Web::Server will use Mongrel, only if Mongrel is installed, otherwise
+      WEBrick will be used.
+

data/Manifest.txt

@@ -0,0 +1,25 @@
+History.txt
+Manifest.txt
+README.txt
+Rakefile
+lib/ronin/sessions/web.rb
+lib/ronin/web.rb
+lib/ronin/web/extensions.rb
+lib/ronin/web/extensions/nokogiri.rb
+lib/ronin/web/extensions/nokogiri/xml.rb
+lib/ronin/web/extensions/nokogiri/xml/node.rb
+lib/ronin/web/extensions/nokogiri/xml/text.rb
+lib/ronin/web/extensions/nokogiri/xml/attr.rb
+lib/ronin/web/extensions/nokogiri/xml/element.rb
+lib/ronin/web/extensions/nokogiri/xml/document.rb
+lib/ronin/web/server.rb
+lib/ronin/web/spider.rb
+lib/ronin/web/web.rb
+lib/ronin/web/version.rb
+tasks/spec.rb
+spec/spec_helper.rb
+spec/web/helpers/server.rb
+spec/web/helpers/root/index.html
+spec/web/helpers/root/test.txt
+spec/web/extensions/nokogiri_spec.rb
+spec/web/server_spec.rb

data/README.txt

@@ -0,0 +1,74 @@
+= Ronin Web
+
+* http://ronin.rubyforge.org/web/
+* http://github.com/postmodern/ronin-web
+* irc.freenode.net ##ronin
+* Postmodern (postmodern.mod3 at gmail.com)
+
+== DESCRIPTION:
+
+Ronin Web is a Ruby library for Ronin that provides support for web
+scraping and spidering functionality.
+
+Ronin is a Ruby platform designed for information security and data
+exploration tasks. Ronin allows for the rapid development and distribution
+of code over many of the common Source-Code-Management (SCM) systems.
+
+=== Free
+
+All source code within Ronin is licensed under the GPL-2, therefore no user
+will ever have to pay for Ronin or updates to Ronin. Not only is the
+source code free, the Ronin project will not sell enterprise grade security
+snake-oil solutions, give private training classes or later turn Ronin into
+commercial software.
+
+=== Modular
+
+Ronin was not designed as one monolithic framework but instead as a
+collection of libraries which can be individually installed. This allows
+users to pick and choose what functionality they want in Ronin.
+
+=== Decentralized
+
+Ronin does not have a central repository of exploits and payloads which
+all developers contribute to. Instead Ronin has Overlays, repositories of
+code that can be hosted on any CVS/SVN/Git/Rsync server. Users can then use
+Ronin to quickly install or update Overlays. This allows developers and
+users to form their own communities, independent of the main developers
+of Ronin.
+
+== FEATURES/PROBLEMS:
+
+* Web access (utilizing Mechanize and Hpricot).
+* Integrates Spidr into Ronin::Web::Spider.
+* Provides a custom Rack web-server based on Rack::Builder.
+
+== REQUIREMENTS:
+
+* nokogiri >= 1.1.0
+* mechanize >= 0.9.0
+* spidr >= 0.1.3
+* rack >= 0.9.1
+* ronin >= 0.1.4
+
+== INSTALL:
+
+  $ sudo gem install ronin-web
+
+== LICENSE:
+
+Copyright (c) 2006-2009 Hal Brodigan (postmodern.mod3 at gmail.com)
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

data/Rakefile

@@ -0,0 +1,21 @@
+# -*- ruby -*-
+
+require 'rubygems'
+require 'hoe'
+require './tasks/spec.rb'
+require './lib/ronin/web/version.rb'
+
+Hoe.new('ronin-web', Ronin::Web::VERSION) do |p|
+  p.rubyforge_name = 'ronin'
+  p.developer('Postmodern', 'postmodern.mod3@gmail.com')
+  p.remote_rdoc_dir = 'docs/ronin-web'
+  p.extra_deps = [
+    ['nokogiri', '>=1.1.0'],
+    ['mechanize', '>=0.9.0'],
+    ['spidr', '>=0.1.3'],
+    ['rack', '>=0.9.1'],
+    ['ronin', '>=0.1.4']
+  ]
+end
+
+# vim: syntax=Ruby

data/lib/ronin/sessions/web.rb

@@ -0,0 +1,80 @@
+#
+#--
+# Ronin Web - A Ruby library for Ronin that provides support for web
+# scraping and spidering functionality.
+#
+# Copyright (c) 2006-2009 Hal Brodigan (postmodern.mod3 at gmail.com)
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#++
+#
+
+require 'ronin/sessions/session'
+require 'ronin/web/web'
+
+module Ronin
+  module Sessions
+    module Web
+      include Session
+
+      setup_session do
+        parameter :web_proxy,
+                  :default => lambda { Ronin::Web.proxy },
+                  :description => 'Web Proxy'
+
+        parameter :web_user_agent,
+                  :default => lambda { Ronin::Web.user_agent },
+                  :description => 'Web User-Agent'
+      end
+
+      protected
+
+      def web_agent(options={},&block)
+        options[:proxy] ||= @web_proxy
+        options[:user_agent] ||= @web_user_agent
+
+        return Ronin::Web.agent(options,&block)
+      end
+
+      def web_get(uri,options={},&block)
+        page = web_agent(options).get(uri)
+
+        block.call(page) if block
+        return page
+      end
+
+      def web_get_body(uri,options={},&block)
+        body = web_agent(options).get(uri).body
+
+        block.call(body) if block
+        return body
+      end
+
+      def web_post(uri,options={},&block)
+        page = web_agent(options).post(uri)
+
+        block.call(page) if block
+        return page
+      end
+
+      def web_post_body(uri,options={},&block)
+        body = web_agent(options).post(uri).body
+
+        block.call(body) if block
+        return body
+      end
+    end
+  end
+end

data/lib/ronin/web/extensions/nokogiri/xml/attr.rb

@@ -0,0 +1,13 @@
+require 'nokogiri/xml/attr'
+
+module Nokogiri
+  module XML
+    class Attr < Node
+
+      def similar?(other)
+        super(other) && (self.value == other.value)
+      end
+
+    end
+  end
+end

data/lib/ronin/web/extensions/nokogiri/xml/document.rb

@@ -0,0 +1,18 @@
+require 'nokogiri/xml/document'
+
+module Nokogiri
+  module XML
+    class Document < Node
+
+      #
+      # Returns the total count of all sub-children of the document.
+      #
+      def total_children
+        return 0 unless root
+
+        1 + root.total_children
+      end
+
+    end
+  end
+end

data/lib/ronin/web/extensions/nokogiri/xml/element.rb

@@ -0,0 +1,23 @@
+require 'nokogiri/xml/element'
+
+module Nokogiri
+  module XML
+    class Element < Node
+
+      def similar?(other)
+        return false unless super(other)
+        return false unless attribute_nodes.length == other.attribute_nodes.length
+
+        (0...attribute_nodes.length).each do |index|
+          attr1 = attribute_nodes[index]
+          attr2 = other.attribute_nodes[index]
+
+          return false unless attr1.similar?(attr2)
+        end
+
+        true
+      end
+
+    end
+  end
+end

data/lib/ronin/web/extensions/nokogiri/xml/node.rb

@@ -0,0 +1,47 @@
+require 'nokogiri/xml/node'
+
+module Nokogiri
+  module XML
+    class Node
+
+      #
+      # Returns the total count of all sub-children of the node.
+      #
+      def total_children
+        count = 0
+        first = self.child
+
+        return count unless first
+
+        while first
+          count += (1 + first.total_children)
+
+          first = first.next
+        end
+
+        count
+      end
+
+      def traverse_text(&block)
+        block.call(self) if text?
+
+        first = self.child
+
+        while first
+          first.traverse_text(&block)
+
+          first = first.next
+        end
+
+        self
+      end
+
+      def similar?(other)
+        return false unless other
+
+        (self.type == other.type) && (self.name == other.name)
+      end
+
+    end
+  end
+end

data/lib/ronin/web/extensions/nokogiri/xml/text.rb

@@ -0,0 +1,13 @@
+require 'nokogiri/xml/text'
+
+module Nokogiri
+  module XML
+    class Text < Node
+
+      def similar?(other)
+        super(other) && (self.content == other.content)
+      end
+
+    end
+  end
+end

data/lib/ronin/web/extensions/nokogiri/xml.rb

@@ -0,0 +1,5 @@
+require 'ronin/web/extensions/nokogiri/xml/node'
+require 'ronin/web/extensions/nokogiri/xml/text'
+require 'ronin/web/extensions/nokogiri/xml/attr'
+require 'ronin/web/extensions/nokogiri/xml/element'
+require 'ronin/web/extensions/nokogiri/xml/document'

data/lib/ronin/web/extensions/nokogiri.rb

@@ -0,0 +1,24 @@
+#
+#--
+# Ronin Web - A Ruby library for Ronin that provides support for web
+# scraping and spidering functionality.
+#
+# Copyright (c) 2006-2009 Hal Brodigan (postmodern.mod3 at gmail.com)
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#++
+#
+
+require 'ronin/web/extensions/nokogiri/xml'

data/lib/ronin/web/extensions.rb

@@ -0,0 +1,24 @@
+#
+#--
+# Ronin Web - A Ruby library for Ronin that provides support for web
+# scraping and spidering functionality.
+#
+# Copyright (c) 2006-2009 Hal Brodigan (postmodern.mod3 at gmail.com)
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#++
+#
+
+require 'ronin/web/extensions/nokogiri'