ronin-web 0.3.0.rc1 → 1.0.0.beta1

data/README.md

@@ -1,199 +1,213 @@
-# Ronin Web
+# ronin-web
 
-* [Source](http://github.com/ronin-ruby/ronin-web)
-* [Issues](http://github.com/ronin-ruby/ronin-web/issues)
-* [Documentation](http://rubydoc.info/github/ronin-ruby/ronin-web/frames)
-* [Mailing List](http://groups.google.com/group/ronin-ruby)
-* [irc.freenode.net #ronin](http://webchat.freenode.net/?channels=ronin&uio=Mj10cnVldd)
+[![CI](https://github.com/ronin-rb/ronin-web/actions/workflows/ruby.yml/badge.svg)](https://github.com/ronin-rb/ronin-web/actions/workflows/ruby.yml)
+[![Code Climate](https://codeclimate.com/github/ronin-rb/ronin-web.svg)](https://codeclimate.com/github/ronin-rb/ronin-web)
 
-## Description
-
-Ronin Web is a Ruby library for Ronin that provides support for web
-scraping and spidering functionality.
-
-Ronin is a Ruby platform for exploit development and security research.
-Ronin allows for the rapid development and distribution of code, exploits
-or payloads over many common Source-Code-Management (SCM) systems.
-
-### Ruby
+* [Source](https://github.com/ronin-rb/ronin-web)
+* [Issues](https://github.com/ronin-rb/ronin-web/issues)
+* [Documentation](https://ronin-rb.dev/docs/ronin-web/frames)
+* [Discord](https://discord.gg/6WAb3PsVX9) |
+  [Twitter](https://twitter.com/ronin_rb) |
+  [Mastodon](https://infosec.exchange/@ronin_rb)
 
-Ronin's Ruby environment allows security researchers to leverage Ruby with
-ease. The Ruby environment contains a multitude of convenience methods
-for working with data in Ruby, a Ruby Object Database, a customized Ruby
-Console and an extendable command-line interface.
-
-### Extend
-
-Ronin's more specialized features are provided by additional Ronin
-libraries, which users can choose to install. These libraries can allow
-one to write and run Exploits and Payloads, scan for PHP vulnerabilities,
-perform Google Dorks  or run 3rd party scanners.
+## Description
 
-### Publish
+ronin-web is a collection of useful Web-related libraries, classes, and
+CLI commands.
 
-Ronin allows users to publish and share code, exploits, payloads or other
-data via Overlays. Overlays are directories of code and data that can be
-hosted on any SVN, Hg, Git or Rsync server. Ronin makes it easy to create,
-install or update Overlays.
+ronin-web is part of the [ronin-rb] project, a [Ruby] toolkit for security
+research and development.
 
 ## Features
 
-* Provides popular [User Agent strings](https://github.com/ronin-ruby/ronin-web/blob/master/data/ronin/web/user_agents.yml).
-* Web access (utilizing [Mechanize](http://rubydoc.info/gems/mechanize/frames)
-  and [Nokogiri](http://rubydoc.info/gems/nokogiri/frames).
-* Integrates [Spidr](http://github.com/postmodern/spidr#readme) into
-  {Ronin::Web::Spider}.
-* Provides a collection of useful [Rack](http://rack.rubyforge.org/)
-  Middleware:
-  * {Ronin::Web::Middleware::Files}
-  * {Ronin::Web::Middleware::Directories}
-  * {Ronin::Web::Middleware::Proxy}
-  * {Ronin::Web::Middleware::Router}
-* Provides {Ronin::Web::Server}, a [Sinatra](http://sinatrarb.com/) based
-  Web Server.
-* Provides {Ronin::Web::Proxy}, a [Sinatra](http://sinatrarb.com/) based
-  Web Proxy.
+* HTML/XML parsing/building (using [Nokogiri][nokogiri]).
+  * Also provides additional extensions to [Nokogiri][nokogiri] using
+    [nokogiri-ext].
+* Supports diffing HTML/XML documents using [nokogiri-diff].
+* Automated Web Browsing using [Mechanize][mechanize].
+* Supports random `User-Agent` generation using [ronin-web-user_agents].
+* Provides an easy to use [Sinatra][sintra] based web server using
+  [ronin-web-server].
+* Provides an easy to use web spider using [ronin-web-spider].
+* Provides a CLI for common web related tasks.
+* Has 98% documentation coverage.
+* Has 89% test coverage.
 
 ## Synopsis
 
-Start the Ronin console with Ronin Web preloaded:
+```
+Usage: ronin-web [options] [COMMAND [ARGS...]]
 
-    $ ronin-web
+Options:
+    -h, --help                       Print help information
+
+Arguments:
+    [COMMAND]                        The command name to run
+    [ARGS ...]                       Additional arguments for the command
+
+Commands:
+    diff
+    help
+    html
+    irb
+    new
+    reverse-proxy
+    server
+    spider
+```
 
 ## Examples
 
 Get a web-page:
 
-    Web.get('http://www.rubyinside.com/')
+```ruby
+Web.get('http://www.rubyinside.com/')
+```
 
 Get only the body of the web-page:
 
-    Web.get_body('http://www.rubyinside.com/')
+```ruby
+Web.get_body('http://www.rubyinside.com/')
+```
 
-Get a [Mechanize agent](http://rubydoc.info/gems/mechanize/1.0.0/Mechanize):
+Get a [Mechanize agent][mechanize]:
 
-    agent = Web.agent
+```ruby
+agent = Web.agent
+```
 
 Parse HTML:
 
-    Web.html(open('some_file.html'))
-    # => <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
-    # <html>
-    #   <head>
-    #     <script type="text/javascript" src="redirect.js"></script>
-    #   </head>
-    # </html>
+```ruby
+HTML.parse(open('some_file.html'))
+# => <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
+# <html>
+#   <head>
+#     <script type="text/javascript" src="redirect.js"></script>
+#   </head>
+# </html>
+```
 
 Build a HTML document:
 
-    doc = Web.build_html do
-      html {
-        head {
-          script(:type => 'text/javascript', :src => 'redirect.js')
-        }
-      }
-    end
-    
-    puts doc.to_html
-    # <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
-    # <html><head><script src="redirect.js" type="text/javascript"></script></head></html>
+```ruby
+doc = HTML.build do
+  html {
+    head {
+      script(:type => 'text/javascript', :src => 'redirect.js')
+    }
+  }
+end
 
-Parse XML:
+puts doc.to_html
+# <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
+# <html><head><script src="redirect.js" type="text/javascript"></script></head></html>
+```
 
-    Web.xml(some_text)
-    # => <?xml version="1.0"?>
-    # <users>
-    #   <user>
-    #     <name>admin</name>
-    #     <password>0mni</password>
-    #   </user>
-    # </users>
+Parse XML:
 
+```ruby
+XML.parse(some_text)
+# => <?xml version="1.0"?>
+# <users>
+#   <user>
+#     <name>admin</name>
+#     <password>0mni</password>
+#   </user>
+# </users>
+```
 
 Build a XML document:
 
-    doc = Web.build_xml do
-      playlist {
-        mp3 {
-          file { text('02 THE WAIT.mp3') }
-          artist { text('Evil Nine') }
-          track { text('The Wait feat David Autokratz') }
-          duration { text('1000000000') }
-        }
-      }
-    end
-    
-    puts doc.to_xml
-    # <?xml version="1.0"?>
-    # <playlist>
-    #   <mp3>
-    #     <file>02 THE WAIT.mp3</file>
-    #     <artist>Evil Nine</artist>
-    #     <track>The Wait feat David Autokratz</track>
-    #     <duration>1000000000</duration>
-    #   </mp3>
-    # </playlist>
-
-Spider a web site:
-
-    Web::Spider.host('www.example.com') do |spider|
-      spider.every_url do |url|
-        # ...
-      end
-
-      spider.every_page do |page|
-        # ...
-      end
-    end
-
-Serve files via a Web Server:
-
-    require 'ronin/web/server'
-
-    Web.server do
-      file '/opensearch.xml', '/tmp/test.xml'
-      directory '/download/', '/tmp/download/'
-    end
-
-    Web.server.get '/test' do
-      'Test 1 2 1 2'
-    end
+```ruby
+doc = XML.build do
+  playlist {
+    mp3 {
+      file { text('02 THE WAIT.mp3') }
+      artist { text('Evil Nine') }
+      track { text('The Wait feat David Autokratz') }
+      duration { text('1000000000') }
+    }
+  }
+end
+
+puts doc.to_xml
+# <?xml version="1.0"?>
+# <playlist>
+#   <mp3>
+#     <file>02 THE WAIT.mp3</file>
+#     <artist>Evil Nine</artist>
+#     <track>The Wait feat David Autokratz</track>
+#     <duration>1000000000</duration>
+#   </mp3>
+# </playlist>
+```
 
 ## Requirements
 
-* [nokogiri](http://github.com/tenderlove/nokogiri) ~> 1.4
-  * [libxml2](http://xmlsoft.org/)
-  * [libxslt1](http://xmlsoft.org/XSLT/)
-* [mechanize](http://github.com/tenderlove/mechanize) ~> 1.0
-* [spidr](http://github.com/postmodern/spidr) ~> 0.2
-* [rack](http://github.com/rack/rack) ~> 1.3
-* [sinatra](http://github.com/sinatra/sinatra) ~> 1.2
-* [data_paths](http://github.com/postmodern/data_paths) ~> 0.3
-* [ronin-support](http://github.com/ronin-ruby/ronin-support) ~> 0.2
-* [ronin](http://github.com/ronin-ruby/ronin) ~> 1.1
+* [Ruby] >= 3.0.0
+* [nokogiri] ~> 1.4
+  * [libxml2]
+  * [libxslt1]
+* [nokogiri-ext] ~> 0.1
+* [nokogiri-diff] ~> 0.2
+* [mechanize] ~> 2.0
+* [open_namespace] ~> 0.4
+* [ronin-support] ~> 1.0
+* [ronin-web-server] ~> 0.1
+* [ronin-web-spider] ~> 0.1
+* [ronin-web-user_agents] ~> 0.1
+* [ronin-core] ~> 0.1
 
 ## Install
 
-    $ gem install ronin-web
+```shell
+$ gem install ronin-web
+```
 
-## License
+## Development
+
+1. [Fork It!](https://github.com/ronin-rb/ronin-web/fork)
+2. Clone It!
+3. `cd ronin-web`
+4. `bundle install`
+5. `git checkout -b my_feature`
+6. Code It!
+7. `bundle exec rake spec`
+8. `git push origin my_feature`
 
-Ronin Web - A Ruby library for Ronin that provides support for web
-scraping and spidering functionality.
+## License
 
-Copyright (c) 2006-2011 Hal Brodigan (postmodern.mod3 at gmail.com)
+ronin-web - A collection of useful web helper methods and commands.
 
-This file is part of Ronin Web.
+Copyright (c) 2006-2022 Hal Brodigan (postmodern.mod3 at gmail.com)
 
-Ronin is free software: you can redistribute it and/or modify
+ronin-web is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation, either version 3 of the License, or
 (at your option) any later version.
 
-Ronin is distributed in the hope that it will be useful,
+ronin-web is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.
 
 You should have received a copy of the GNU General Public License
-along with Ronin.  If not, see <http://www.gnu.org/licenses/>.
+along with ronin-web.  If not, see <https://www.gnu.org/licenses/>.
+
+[ronin-rb]: https://ronin-rb.dev
+[Ruby]: https://www.ruby-lang.org
+
+[nokogiri]: https://nokogiri.org/
+[nokogiri-ext]: https://github.com/postmodern/nokogiri-ext#readme
+[nokogiri-diff]: https://github.com/postmodern/nokogiri-diff#readme
+[libxml2]: http://xmlsoft.org/
+[libxslt1]: http://xmlsoft.org/XSLT/
+[mechanize]: https://github.com/sparklemotion/mechanize#readme
+[open_namespace]: https://github.com/postmodern/open_namespace#readme
+[ronin-support]: https://github.com/ronin-rb/ronin-support#readme
+[ronin-core]: https://github.com/ronin-rb/ronin-core#readme
+[ronin-web-server]: https://github.com/ronin-rb/ronin-web-server#readme
+[ronin-web-spider]: https://github.com/ronin-rb/ronin-web-spider#readme
+[ronin-web-user_agents]: https://github.com/ronin-rb/ronin-web-user_agents#readme
+[ronin]: https://github.com/ronin-rb/ronin#readme

data/Rakefile

@@ -18,13 +18,22 @@ end
 
 require 'rake'
 
-require 'ore/tasks'
-Ore::Tasks.new
+require 'rubygems/tasks'
+Gem::Tasks.new(sign: {checksum: true, pgp: true})
 
 require 'rspec/core/rake_task'
 RSpec::Core::RakeTask.new
-task :test => :spec
+task :test    => :spec
 task :default => :spec
 
+namespace :spec do
+  RSpec::Core::RakeTask.new(:network) do |t|
+    t.rspec_opts = '--tag network'
+  end
+end
+
 require 'yard'
 YARD::Rake::YardocTask.new
+
+require 'kramdown/man/task'
+Kramdown::Man::Task.new

data/bin/ronin-web

@@ -2,26 +2,18 @@
 
 require 'rubygems'
 
-root_dir = File.expand_path(File.join(File.dirname(__FILE__),'..'))
-if File.directory?(File.join(root_dir,'.git'))
-  Dir.chdir(root_dir) do |path|
-    require 'bundler'
-
+root = File.expand_path(File.join(File.dirname(__FILE__),'..'))
+if File.directory?(File.join(root,'.git'))
+  Dir.chdir(root) do
     begin
-      Bundler.setup(:default)
-    rescue Bundler::BundlerError => e
+      require 'bundler/setup'
+    rescue LoadError => e
       warn e.message
-      warn "Run `bundle install` to install missing gems"
-      exit e.status_code
+      warn "Run `gem install bundler` to install Bundler"
+      exit -1
     end
   end
 end
 
-lib_dir = File.join(root_dir,'lib')
-$LOAD_PATH << lib_dir unless $LOAD_PATH.include?(lib_dir)
-
-require 'ronin/ui/console'
-require 'ronin/ui/cli/commands/console'
-
-Ronin::UI::Console.auto_load << 'ronin/web'
-Ronin::UI::CLI::Commands::Console.start
+require 'ronin/web/cli'
+Ronin::Web::CLI.start

data/data/new/nokogiri.rb.erb

@@ -0,0 +1,12 @@
+#!/usr/bin/env ruby
+
+require 'nokogiri'
+require 'open-uri'
+
+<%- if @url -%>
+doc = Nokogiri::HTML(URI.open(<%= @url -%>))
+<%- else -%>
+doc = Nokogiri::HTML(URI.open(ARGV[0]))
+<%- end -%>
+
+puts doc.search('//a[@href]')

data/data/new/server.rb.erb

@@ -0,0 +1,22 @@
+#!/usr/bin/env ruby
+
+require 'ronin/web/server'
+
+class App < Ronin::Web::Server::Base
+<%- if (@host || @port) -%>
+
+  <%- if @host -%>
+  set :host, <%= @host.inspect %>
+  <%- end -%>
+  <%- if @port -%>
+  set :port, <%= @port.inspect %>
+  <%- end -%>
+<%- end -%>
+
+  get '/' do
+    "hello world"
+  end
+
+end
+
+App.run! if __FILE__ == $0

data/data/new/spider.rb.erb

@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+
+require 'ronin/web/spider'
+
+Ronin::Web::Spider.<%= entry_point_method -%>(<%= entry_point_argument -%>) do |agent|
+  <%- unless @callbacks.empty? -%>
+  <%-   @callbacks.each_with_index do |(method,arguments,yield_arg),index| -%>
+  <%-     unless arguments.empty? -%>
+  agent.<%= method -%>(<%= arguments.map(&:inspect).join(', ') -%>) do |<%= yield_arg -%>|
+    # ...
+  end
+  <%-       else -%>
+  agent.<%= method -%> do |<%= yield_arg -%>|
+    # ...
+  end
+  <%-     end -%>
+  <%-     unless index == (@callbacks.length-1) -%>
+
+  <%-     end -%>
+  <%-   end -%>
+  <%- else -%>
+  agent.ever_page do |page|
+    puts page.url
+  end
+  <%- end -%>
+end

data/data/new/webapp/.gitignore

@@ -0,0 +1,15 @@
+/Gemfile.lock
+/vendor/bundle
+/.bundle
+.DS_Store
+*.db
+*.log
+*.swp
+*~
+*.sqlite3
+*.db
+*.gz
+*.bz2
+*.xz
+*.tar
+*.zip

data/data/new/webapp/.ruby-version.erb

@@ -0,0 +1 @@
+<%= @ruby_version -%>

data/data/new/webapp/Dockerfile.erb

@@ -0,0 +1,11 @@
+FROM ruby:<%= @ruby_version -%>
+
+RUN apt-get update -qq -y && gem update bundler
+
+WORKDIR /app
+COPY . /app
+RUN bundle install
+
+EXPOSE <%= @port -%>
+
+CMD ["bundle", "exec", "rackup", "--host", "0.0.0.0", "-p", "<%= @port -%>"]

data/data/new/webapp/Gemfile

@@ -0,0 +1,6 @@
+source 'https://rubygems.org'
+
+gem 'puma', '~> 6.0'
+
+# gem 'ronin-support', '~> 1.0' #, github: 'ronin-rb/ronin-support'
+gem 'ronin-web-server', '~> 0.1' #, github: 'ronin-rb/ronin-web-server'

data/data/new/webapp/app.rb.erb

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'ronin/web/server'
+$LOAD_PATH.unshift(File.join(__dir__,'lib'))
+
+class App < Ronin::Web::Server::Base
+
+  get '/' do
+    "hello world"
+  end
+
+end
+
+App.run! if __FILE__ == $0

data/data/new/webapp/config.ru

@@ -0,0 +1,4 @@
+#!/usr/bin/env rackup
+
+require './app'
+run App

data/data/new/webapp/docker-compose.yml.erb

@@ -0,0 +1,9 @@
+version: "3.8"
+services:
+  app:
+    build: .
+    command: bundle exec rackup --host 0.0.0.0 -p <%= @port -%>
+    ports:
+      - "<%= @port -%>:<%= @port -%>"
+    volumes:
+      - .:/app

data/gemspec.yml

@@ -1,28 +1,46 @@
 name: ronin-web
 summary:
-  A Ruby library for Ronin that provides support for web scraping and
+  A Ruby library that provides support for web scraping and
   spidering functionality.
+
 description:
-  Ronin Web is a Ruby library for Ronin that provides support for web
-  scraping and spidering functionality.
+  ronin-web is a Ruby library that provides support for web scraping and
+  spidering functionality.
 
-license: GPL-3
+license: GPL-3.0
 authors: Postmodern
 email: postmodern.mod3@gmail.com
-homepage: http://github.com/ronin-ruby/ronin-web
+homepage: https://github.com/ronin-rb/ronin-web#readme
 has_yard: true
 
+required_ruby_version: ">= 3.0.0"
+
+generated_files:
+  - man/ronin-web.1
+  - man/ronin-web-diff.1
+  - man/ronin-web-html.1
+  - man/ronin-web-irb.1
+  - man/ronin-web-new.1
+  - man/ronin-web-new-nokogiri.1
+  - man/ronin-web-new-server.1
+  - man/ronin-web-new-spider.1
+  - man/ronin-web-new-webapp.1
+  - man/ronin-web-server.1
+  - man/ronin-web-spider.1
+  - man/ronin-web-reverse-proxy.1
+
 dependencies:
   nokogiri: ~> 1.4
-  mechanize: ~> 1.0
-  spidr: ~> 0.2
-  rack: ~> 1.3
-  sinatra: ~> 1.2
-  data_paths: ~> 0.3
+  nokogiri-ext: ~> 0.1
+  nokogiri-diff: ~> 0.2
+  mechanize: ~> 2.0
+  open_namespace: ~> 0.4
   # Ronin dependencies:
-  ronin-support: ~> 0.2
-  ronin: ~> 1.1
+  ronin-support: ~> 1.0.0.beta1
+  ronin-web-server: ~> 0.1.0.beta1
+  ronin-web-spider: ~> 0.1.0.beta2
+  ronin-web-user_agents: ~> 0.1.0.beta1
+  ronin-core: ~> 0.1.0.beta1
 
 development_dependencies:
-  bundler: ~> 1.0.10
-  yard: ~> 0.7.0
+  bundler: ~> 2.0

data/lib/ronin/web/cli/command.rb

@@ -0,0 +1,36 @@
+#
+# ronin-web - A collection of useful web helper methods and commands.
+#
+# Copyright (c) 2006-2022 Hal Brodigan (postmodern.mod3 at gmail.com)
+#
+# ronin-web is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# ronin-web is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with ronin-web.  If not, see <https://www.gnu.org/licenses/>.
+#
+
+require 'ronin/web/root'
+
+require 'ronin/core/cli/command'
+
+module Ronin
+  module Web
+    class CLI
+      class Command < Core::CLI::Command
+
+        man_dir File.join(ROOT,'man')
+
+        bug_report_url 'https://github.com/ronin-rb/ronin-web/issues/new'
+
+      end
+    end
+  end
+end