ronin-recon 0.1.0.rc1

data/README.md

@@ -0,0 +1,391 @@
+# ronin-recon
+
+[![CI](https://github.com/ronin-rb/ronin-recon/actions/workflows/ruby.yml/badge.svg)](https://github.com/ronin-rb/ronin-recon/actions/workflows/ruby.yml)
+[![Code Climate](https://codeclimate.com/github/ronin-rb/ronin-recon.svg)](https://codeclimate.com/github/ronin-rb/ronin-recon)
+
+* [Website](https://ronin-rb.dev/)
+* [Source](https://github.com/ronin-rb/ronin-recon)
+* [Issues](https://github.com/ronin-rb/ronin-recon/issues)
+* [Documentation](https://ronin-rb.dev/docs/ronin-recon)
+* [Discord](https://discord.gg/6WAb3PsVX9) |
+  [Mastodon](https://infosec.exchange/@ronin_rb)
+
+## Description
+
+ronin-recon is a micro-framework and tool for performing reconnaissance.
+ronin-recon uses multiple workers which process different value types
+(ex: IP, host, URL, etc) and produce new values. ronin-recon contains built-in
+recon workers and supports loading additional 3rd-party workers from Ruby
+files or 3rd-party git repositories. ronin-recon has a unique queue design
+and uses asynchronous I/O to maximize efficiency.
+
+## Features
+
+* Uses asynchronous I/O and fibers.
+* Supports defining recon modules as plain old Ruby class.
+* Provides built-in recon workers for:
+  * IP range enumeration.
+  * DNS lookup of host-names.
+  * Querying nameservers.
+  * Querying mailservers.
+  * DNS reverse lookup of IP addresses.
+  * DNS SRV record enumeration.
+  * DNS subdomain enumeration.
+  * Service/port scanning with `nmap`.
+  * Enumerates the Common Name (`CN`) and `subjectAltName`s within all SSL/TLS
+    certificates.
+  * Web spidering.
+  * HTTP directory enumeration.
+* Supports loading additional recon modules from Ruby files or from installed
+  [3rd-party git repositories][ronin-repos].
+* Builds a network graph of all discovered assets.
+* Provides a simple CLI for listing workers or performing recon.
+* Supports many different output file formats:
+  * TXT
+  * CSV
+  * JSON
+  * [NDJSON](http://ndjson.org/)
+  * [GraphViz][graphviz]
+    * DOT
+    * SVG
+    * PNG
+    * PDF
+* Supports automatically saving recon results into [ronin-db].
+
+## Anti-Features
+
+* Does not require API keys to run.
+* Not just a script that runs a bunch of other recon tools.
+
+## Synopsis
+
+```
+$ ronin-recon
+Usage: ronin-recon [options]
+
+Options:
+    -V, --version                    Prints the version and exits
+    -h, --help                       Print help information
+
+Arguments:
+    [COMMAND]                        The command name to run
+    [ARGS ...]                       Additional arguments for the command
+
+Commands:
+    completion
+    help
+    irb
+    new
+    run
+    test
+    worker
+    workers
+```
+
+List all available recon workers:
+
+```shell
+$ ronin-recon workers
+  dns/lookup
+  dns/mailservers
+  dns/nameservers
+  dns/reverse_lookup
+  dns/srv_enum
+  dns/subdomain_enum
+  dns/suffix_enum
+  net/cert_enum
+  net/cert_grab
+  net/cert_sh
+  net/ip_range_enum
+  net/port_scan
+  net/service_id
+  web/dir_enum
+  web/email_addresses
+  web/spider
+```
+
+Print info about a specific recon worker:
+
+```shell
+$ ronin-recon worker dns/lookup
+[ dns/lookup ]
+
+  Summary: Looks up the IPs of a host-name
+  Description:
+
+    Resolves the IP addresses of domains, host names, nameservers,
+    and mailservers.
+
+  Accepts:
+
+    * domains
+    * hosts
+    * nameservers
+    * mailservers
+
+```
+
+Run the recon engine on a single domain:
+
+```shell
+$ ronin-recon run example.com
+```
+
+Run the recon engine on a single host-name:
+
+```shell
+$ ronin-recon run www.example.com
+```
+
+Run the recon engine on a single IP address:
+
+```shell
+$ ronin-recon run 1.1.1.1
+```
+
+Run the recon engine on an IP range:
+
+```shell
+$ ronin-recon run 1.1.1.1/24
+```
+
+Run the recon engine on multiple targets:
+
+```shell
+$ ronin-recon run example1.com example2.com secret.foo.example1.com secret.bar.example2.com 1.1.1.1/24
+```
+
+Run the recon engine and ignore specific hosts, IPs, URLs, etc.:
+
+```shell
+$ ronin-recon run --ignore staging.example.com example.com
+```
+
+Save the recon results to a plain-text file:
+
+```shell
+$ ronin-recon run -o output.txt example.com
+```
+
+Save the recon results to a directory of multiple plain-text files:
+
+```shell
+$ ronin-recon run -o output_dir example.com
+```
+
+Save the recon results to a CSV file:
+
+```shell
+$ ronin-recon run -o output.csv example.com
+```
+
+Save the recon results to a JSON file:
+
+```shell
+$ ronin-recon run -o output.json example.com
+```
+
+Save the recon results to a NDJSON file:
+
+```shell
+$ ronin-recon run -o output.ndjson example.com
+```
+
+Save the recon results to a PNG image:
+
+```shell
+$ ronin-recon run -o output.png example.com
+```
+
+Save the recon results to a SVG image:
+
+```shell
+$ ronin-recon run -o output.svg example.com
+```
+
+Save the recon results to a PDF image:
+
+```shell
+$ ronin-recon run -o output.pdf example.com
+```
+
+Generate a boilerplate recon worker file, with some custom information:
+
+```shell
+$ ronin-recon new example_worker.rb \
+                  --name Example \
+                  --authors Postmodern \
+                  --description "This is an example."
+```
+
+Generate a ronin repository of your own payloads (or exploits):
+
+```shell
+$ ronin-repos new my-repo
+$ cd my-repo/
+$ mkdir recon
+$ ronin-recon new recon/my_recon.rb \
+                  --name MyRecon \
+                  --authors You \
+                  --description "This is my payload."
+$ vim recon/my_recon.rb
+$ git add recon/my_recon.rb
+$ git commit
+$ git push
+```
+
+## Examples
+
+Defining a custom recon worker:
+
+```ruby
+require 'ronin/recon/worker'
+
+module Ronin
+  module Recon
+    module DNS
+      class FooBar
+
+        register 'dns/foo_bar'
+
+        summary 'My DNS recon technique'
+        description <<~DESC
+          This recon worker uses the foo-bar technique.
+          Bla bla bla bla.
+        DESC
+        author 'John Smith', email: '...'
+
+        accepts Domain
+        outputs Host
+        intensity :passive
+
+        param :wordlist, String, desc: 'Optional wordlist to use'
+
+        def process(value)
+          # ...
+          yield Host.new(discovered_host_name)
+          # ...
+        end
+
+      end
+    end
+  end
+end
+```
+
+Manually running the recon engine:
+
+```ruby
+require 'ronin/recon/engine'
+
+domain = Ronin::Recon::Values::Domain.new('github.com')
+
+Ronin::Recon::Engine.run([domain], max_depth: 3) do |value,parent|
+  case value
+  when Ronin::Recon::Values::Domain
+    puts "Found domain #{value} for #{parent}"
+  when Ronin::Recon::Values::Nameserver
+    puts "Found nameserver #{value} for #{parent}"
+  when Ronin::Recon::Values::Mailserver
+    puts "Found mailserver #{value} for #{parent}"
+  when Ronin::Recon::Values::Host
+    puts "Found host #{value} for #{parent}"
+  when Ronin::Recon::Values::IP
+    puts "Found IP address #{value} for #{parent}"
+  end
+end
+```
+
+## Requirements
+
+* [Ruby] >= 3.1.0
+* [nmap] >= 5.00
+* [GraphViz][graphviz] (for SVG, PNG, or PDF output)
+* [thread-local] ~> 1.0
+* [async-io] ~> 1.0
+* [async-dns] ~> 1.0
+* [async-http] ~> 0.60
+* [wordlist] ~> 1.0, >= 1.0.3
+* [ronin-support] ~> 1.1
+* [ronin-core] ~> 0.2
+* [ronin-db] ~> 0.2
+* [ronin-repos] ~> 0.1
+* [ronin-masscan] ~> 0.1
+* [ronin-nmap] ~> 0.1
+* [ronin-web-spider] ~> 0.2
+
+## Install
+
+```shell
+$ gem install ronin-recon
+```
+
+### Gemfile
+
+```ruby
+gem 'ronin-recon', '~> 0.1'
+```
+
+### gemspec
+
+```ruby
+gem.add_dependency 'ronin-recon', '~> 0.1'
+```
+
+## Post-Install
+
+### Running `nmap` / `masscan` without `sudo`
+
+You can configure `nmap` and `masscan` to run without `sudo` by setting their
+capabilities:
+
+```shell
+sudo setcap cap_net_raw,cap_net_admin,cap_net_bind_service+eip $(which nmap)
+sudo setcap cap_net_raw,cap_net_admin,cap_net_bind_service+eip $(which masscan)
+```
+
+## Development
+
+1. [Fork It!](https://github.com/ronin-rb/ronin-recon/fork)
+2. Clone It!
+3. `cd ronin-recon/`
+4. `./scripts/setup`
+5. `git checkout -b my_feature`
+6. Code It!
+7. `bundle exec rake spec`
+8. `git push origin my_feature`
+
+## License
+
+ronin-recon - A micro-framework and tool for performing reconnaissance.
+
+Copyright (c) 2023-2024 Hal Brodigan (postmodern.mod3@gmail.com)
+
+ronin-recon is free software: you can redistribute it and/or modify
+it under the terms of the GNU Lesser General Public License as published
+by the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+ronin-recon is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public License
+along with ronin-recon.  If not, see <https://www.gnu.org/licenses/>.
+
+[Ruby]: https://www.ruby-lang.org
+[graphviz]: https://graphviz.org/
+[nmap]: http://www.insecure.org/
+[thread-local]: https://github.com/socketry/thread-local#readme
+[async-io]: https://github.com/socketry/async-io#readme
+[async-dns]: https://github.com/socketry/async-dns#readme
+[async-http]: https://github.com/socketry/async-http#readme
+[wordlist]: https://github.com/postmodern/wordlist.rb#readme
+[ronin-support]: https://github.com/ronin-rb/ronin-support#readme
+[ronin-core]: https://github.com/ronin-rb/ronin-core#readme
+[ronin-db]: https://github.com/ronin-rb/ronin-db#readme
+[ronin-repos]: https://github.com/ronin-rb/ronin-repos#readme
+[ronin-masscan]: https://github.com/ronin-rb/ronin-masscan#readme
+[ronin-nmap]: https://github.com/ronin-rb/ronin-nmap#readme
+[ronin-web-spider]: https://github.com/ronin-rb/ronin-web-spider#readme

data/Rakefile

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+begin
+  require 'bundler'
+rescue LoadError => e
+  warn e.message
+  warn "Run `gem install bundler` to install Bundler"
+  exit(-1)
+end
+
+begin
+  Bundler.setup(:development)
+rescue Bundler::BundlerError => e
+  warn e.message
+  warn "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+
+require 'rake'
+
+require 'rubygems/tasks'
+Gem::Tasks.new(sign: {checksum: true, pgp: true})
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new
+
+namespace :spec do
+  RSpec::Core::RakeTask.new(:network) do |t|
+    t.rspec_opts = '--tag network'
+  end
+end
+
+task :test    => :spec
+task :default => :spec
+
+require 'yard'
+YARD::Rake::YardocTask.new
+task :docs => :yard
+
+require 'kramdown/man/task'
+Kramdown::Man::Task.new
+
+directory 'data/wordlists'
+
+file 'data/wordlists/subdomains-1000.txt' => 'data/wordlists' do
+  sh 'wget -O data/wordlists/subdomains-1000.txt https://raw.githubusercontent.com/rbsec/dnscan/master/subdomains-1000.txt'
+end
+
+file 'data/wordlists/subdomains-1000.txt.gz' => 'data/wordlists/subdomains-1000.txt' do
+  sh 'gzip -f data/wordlists/subdomains-1000.txt'
+end
+
+file 'data/wordlists/raft-small-directories.txt' => 'data/wordlists' do
+  sh 'wget -O data/wordlists/raft-small-directories.txt https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/raft-small-directories.txt'
+end
+
+file 'data/wordlists/raft-small-directories.txt.gz' => 'data/wordlists/raft-small-directories.txt' do
+  sh 'gzip -f data/wordlists/raft-small-directories.txt'
+end
+
+desc 'Generate built-in wordlists'
+task :wordlists => %w[
+  data/wordlists/subdomains-1000.txt.gz
+  data/wordlists/raft-small-directories.txt.gz
+]
+
+require 'command_kit/completion/task'
+CommandKit::Completion::Task.new(
+  class_file:  'ronin/recon/cli',
+  class_name:  'Ronin::Recon::CLI',
+  output_file: 'data/completions/ronin-recon'
+)
+
+task :setup => %w[wordlists man command_kit:completion]

data/bin/ronin-recon

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+root = File.expand_path(File.join(__dir__,'..'))
+if File.file?(File.join(root,'Gemfile.lock'))
+  Dir.chdir(root) do
+    require 'bundler/setup'
+  rescue LoadError => e
+    warn e.message
+    warn "Run `gem install bundler` to install Bundler"
+    exit(-1)
+  end
+end
+
+require 'ronin/recon/cli'
+Ronin::Recon::CLI.start

data/data/completions/ronin-recon

@@ -0,0 +1,95 @@
+# ronin-recon completion                                   -*- shell-script -*-
+
+# This bash completions script was generated by
+# completely (https://github.com/dannyben/completely)
+# Modifying it manually is not recommended
+
+_ronin-recon_completions_filter() {
+  local words="$1"
+  local cur=${COMP_WORDS[COMP_CWORD]}
+  local result=()
+
+  if [[ "${cur:0:1}" == "-" ]]; then
+    echo "$words"
+  
+  else
+    for word in $words; do
+      [[ "${word:0:1}" != "-" ]] && result+=("$word")
+    done
+
+    echo "${result[*]}"
+
+  fi
+}
+
+_ronin-recon_completions() {
+  local cur=${COMP_WORDS[COMP_CWORD]}
+  local compwords=("${COMP_WORDS[@]:1:$COMP_CWORD-1}")
+  local compline="${compwords[*]}"
+
+  case "$compline" in
+    'run'*'--config-file')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'run'*'--worker-file')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'worker'*'--file')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'run'*'--output')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'test'*'--file')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'completion'*)
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -W "$(_ronin-recon_completions_filter "--print --install --uninstall")" -- "$cur" )
+      ;;
+
+    'worker'*'-f')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'test'*'-f')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'run'*'-C')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'run'*'-o')
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -A file -- "$cur" )
+      ;;
+
+    'worker'*)
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -W "$(_ronin-recon_completions_filter "--file -f --verbose -v")" -- "$cur" )
+      ;;
+
+    'test'*)
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -W "$(_ronin-recon_completions_filter "--file -f --debug -D --param -p")" -- "$cur" )
+      ;;
+
+    'new'*)
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -W "$(_ronin-recon_completions_filter "--type -t --author -a --author-email -e --summary -S --description -D --reference -R --accepts -A --outputs -O --intensity -I")" -- "$cur" )
+      ;;
+
+    'run'*)
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -W "$(_ronin-recon_completions_filter "--debug -D --db --db-uri --db-file --config-file -C --worker -w --enable -e --disable -d --worker-file --param -p --concurrency -c --intensity --max-depth --output -o --output-format -F --import --ignore -I")" -- "$cur" )
+      ;;
+
+    *)
+      while read -r; do COMPREPLY+=( "$REPLY" ); done < <( compgen -W "$(_ronin-recon_completions_filter "--version -V help completion irb new run test worker workers")" -- "$cur" )
+      ;;
+
+  esac
+} &&
+complete -F _ronin-recon_completions ronin-recon
+
+# ex: filetype=sh

data/data/templates/worker.rb.erb

@@ -0,0 +1,67 @@
+#!/usr/bin/env -S ronin-recon test -f
+
+require 'ronin/recon/<%= @worker_type[:file] -%>'
+
+module Ronin
+  module Recon
+    class <%= @class_name -%> < <%= @worker_type[:class]  %>
+
+      register '<%= @file_name -%>'
+
+      <%- if @author_name -%>
+      <%-   if @author_email -%>
+      author <%= @author_name.inspect %>, email: <%= @author_email.inspect %>
+      <%-   else -%>
+      author <%= @author_name.inspect %>
+      <%-   end -%>
+      <%- else -%>
+      author "FIX ME", email: "FIXME@example.com"
+      <%- end -%>
+      <%- if @summary -%>
+      summary <%= @summary.inspect %>
+      <%- else -%>
+      summary "FIX ME"
+      <%- end -%>
+      <%- if @description -%>
+      description <<~DESC
+        <%= @description %>
+      DESC
+      <%- else -%>
+      description <<~DESC
+        FIX ME
+      DESC
+      <%- end -%>
+      <%- unless @references.empty? -%>
+      references [
+        <%- @references.each_with_index do |url,index| -%>
+        <%=   url.inspect -%><% if index < @references.length-1 %>,<% end %>
+        <%- end -%>
+      ]
+      <%- else -%>
+      # references [
+      #   "https://...",
+      #   "https://..."
+      # ]
+      <%- end -%>
+
+      <%- unless @accepts.empty? -%>
+      accepts <%= @accepts.join(', ') %>
+      <%- else -%>
+      accepts FIXME
+      <%- end -%>
+      <%- unless @outputs.empty? -%>
+      outputs <%= @outputs.join(', ') %>
+      <%- else -%>
+      outputs FIXME
+      <%- end -%>
+      <%- if @intensity -%>
+      intensity <%= @intensity.inspect %>
+      <%- end -%>
+
+      def process(value)
+        # ...
+      end
+
+    end
+  end
+end

data/examples/recon.rb

@@ -0,0 +1,24 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'bundler/setup'
+require 'ronin/recon/engine'
+
+domain = Ronin::Recon::Values::Domain.new('example.com')
+
+Ronin::Recon::Engine.run([domain], max_depth: 3) do |engine|
+  engine.on(:value) do |value,parent|
+    case value
+    when Ronin::Recon::Values::Domain
+      puts ">>> Found new domain #{value} for #{parent}"
+    when Ronin::Recon::Values::Nameserver
+      puts ">>> Found new nameserver #{value} for #{parent}"
+    when Ronin::Recon::Values::Mailserver
+      puts ">>> Found new mailserver #{value} for #{parent}"
+    when Ronin::Recon::Values::Host
+      puts ">>> Found new host #{value} for #{parent}"
+    when Ronin::Recon::Values::IP
+      puts ">>> Found new IP address #{value} for #{parent}"
+    end
+  end
+end