ronin-web 1.0.2 → 2.0.0.rc1

data/man/ronin-web-wordlist.1.md

@@ -0,0 +1,263 @@
+# ronin-web-spider 1 "2022-01-01" Ronin Web "User Manuals"
+
+## NAME
+
+ronin-web-wordlist - Builds a wordlist by spidering a website
+
+## SYNOPSIS
+
+`ronin-web wordlist` [*options*] {`--host` *HOST* \| `--domain` *DOMAIN* \| `--site` *URL*}
+
+## DESCRIPTION
+
+Builds a wordlist by spidering a website.
+
+## OPTIONS
+
+`-o`, `--output` *PATH*
+: The wordlist file to write to.
+
+`-X`, `--content-xpath` *XPATH*
+: The XPath expression for where the content exists in each HTML page.
+
+`-C`, `--content-css-path` *CSS-path*
+: The CSS-path expression for where the content exists in each HTML page.
+
+`--meta-tags`
+: Parses `keywords` and `description` `<meta>` tags while spidering HTML pages.
+  This is enabled by default.
+
+`--no-meta-tags`
+: Ignore `<meta>` tags while spidering HTML pages.
+
+`--comments`
+: Parses HTML comments while spidering HTML pages.
+  This is enabled by default.
+
+`--no-comments`
+: Ignores HTML comments while spidering HTML pages.
+
+`--alt-tags`
+: Parses `alt=` attribute tags on `<img>`, `<area>`, and `<input>`.
+
+`--no-alt-tags`
+: Ignore `alt=` attribute tags while spidering HTML pages.
+
+`--paths`
+: Parses the directory names from all spidered URLs.
+
+`--query-param-names`
+: Parses the query param names from all spidered URLs.
+
+`--query-param-values`
+: Parses the query param values from all spidered URLs.
+
+`--only-paths`
+: Only parse the directory names from all spidered URLs.
+
+`--only-query-param-names`
+: Only parse the query param names from all spidered URLs.
+
+`--query-param-values`
+: Only parse the query param values from all spidered URLs.
+
+`-f`, `--format` `txt`|`gz`|`bzip2`|`xz`
+: Specifies the format of the wordlist file that will be created.
+
+`-A`, `--append`
+: Append new words to an existing wordlist file instead of overwriting the file.
+
+### TEXT PARSING OPTIONS
+
+`-L`, `--lang` *LANG*
+: The language of the text to parse. Defaults to the current language set by the
+  `LANG` environment variable.
+
+`--stop-word` *WORD*
+: Defines a custom "stop word" (ex: "the", "is", "a") to be ignored.
+  If not specified, a default list of "stop words" will be selected based on
+  either `--lang` or the current language set by the `LANG` environment
+  variable.
+
+`--ignore-word` *WORD*
+: Adds the word to the list of words to ignore while parsing text.
+
+`--digits`
+: Accepts words contining digits (0-9) while parsing text. This is the default
+  behavior.
+
+`--no-digits`
+: Ignores words containing digits (0-9) while parsing text.
+
+`--special-char` *CHAR*
+: Allows a specific special character to exist within words. If not specified,
+  only the characters `_`, `-`, `'` are allowed by default.
+
+`--numbers`
+: Accepts whole numbers as words while parsing text.
+
+`--no-numbers`
+: Ignores whole numbers while parsing text. This is the default behavior.
+
+`--acronyms`
+: Treat acronyms (ex: `A.B.C.`) as words while parsing text.
+  This is the default behavior.
+
+`--no-acronyms`
+: Ignores acronyms (ex: `A.B.C.`) while parsing text.
+
+`--normalize-case`
+: Converts all words to lowercase while parsing text.
+
+`--no-normalize-case`
+: Preserves the case of words letters while parsing text. This is the default
+  behavior. This is the default behavior.
+
+`--normalize-apostrophes`
+: Removes apostrophes from words (ex: `It's` -> `Its`) while parsing text.
+
+`--no-normalize-apostrophes`
+: Preserves apostrophes in words (ex: `It's`). This is the default behavior.
+  This is the default behavior.
+
+`--normalize-acronyms`
+: Removes the periods from acronyms (ex: `A.B.C.` -> `ABC`) while parsing text.
+
+`--no-normalize-acronyms`
+: Preserves the periods in acronyms (ex: `A.B.C.`) while parsing text.
+  This is the default behavior.
+
+`-h`, `--help`
+: Print help information.
+
+### SPIDER OPTIONS
+
+`--open-timeout` *SECS*
+: Sets the connection open timeout.
+
+`--read-timeout` *SECS*
+: Sets the read timeout.
+
+`--ssl-timeout` *SECS*
+: Sets the SSL connection timeout.
+
+`--continue-timeout` *SECS*
+: Sets the continue timeout.
+
+`--keep-alive-timeout` *SECS*
+: Sets the connection keep alive timeout.
+
+`-P`, `--proxy` *PROXY*
+: Sets the proxy to use.
+
+`-H`, `--header` "*NAME*`:` *VALUE*"
+: Sets a default header.
+
+`--host-header` *NAME*=*VALUE*
+: Sets a default header.
+
+`-u`, `--user-agent` `chrome-linux`|`chrome-macos`|`chrome-windows`|`chrome-iphone`|`chrome-ipad`|`chrome-android`|`firefox-linux`|`firefox-macos`|`firefox-windows`|`firefox-iphone`|`firefox-ipad`|`firefox-android`|`safari-macos`|`safari-iphone`|`safari-ipad`|`edge`
+: The `User-Agent` to use.
+
+`-U`, `--user-agent-string` *STRING*
+: The raw `User-Agent` string to use.
+
+`-R`, `--referer` *URL*
+: Sets the `Referer` URL.
+
+`--delay` *SECS*
+: Sets the delay in seconds between each request.
+
+`-l`, `--limit` *COUNT*
+: Only spiders up to *COUNT* pages.
+
+`-d`, `--max-depth` *DEPTH*
+: Only spiders up to max depth.
+
+`--enqueue` *URL*
+: Adds the URL to the queue.
+
+`--visited` *URL*
+: Marks the URL as previously visited.
+
+`--strip-fragments`
+: Enables/disables stripping the fragment component of every URL.
+
+`--strip-query`
+: Enables/disables stripping the query component of every URL.
+
+`--visit-host` *HOST*
+: Visit URLs with the matching host name.
+
+`--visit-hosts-like` `/`*REGEX*`/`
+: Visit URLs with hostnames that match the *REGEX*.
+
+`--ignore-host` *HOST*
+: Ignore the host name.
+
+`--ignore-hosts-like` `/`*REGEX*`/`
+: Ignore the host names matching the *REGEX*.
+
+`--visit-port` *PORT*
+: Visit URLs with the matching port number.
+
+`--visit-ports-like` `/`*REGEX*`/`
+: Visit URLs with port numbers that match the *REGEX*.
+
+`--ignore-port` *PORT*
+: Ignore the port number.
+
+`--ignore-ports-like` `/`*REGEX*`/`
+: Ignore the port numbers matching the *REGEXP*.
+
+`--visit-link` *URL*
+: Visit the *URL*.
+
+`--visit-links-like` `/`*REGEX*`/`
+: Visit URLs that match the *REGEX*.
+
+`--ignore-link` *URL*
+: Ignore the *URL*.
+
+`--ignore-links-like` `/`*REGEX*`/`
+: Ignore URLs matching the *REGEX*.
+
+`--visit-ext` *FILE_EXT*
+: Visit URLs with the matching file ext.
+
+`--visit-exts-like` `/`*REGEX*`/`
+: Visit URLs with file exts that match the *REGEX*.
+
+`--ignore-ext` *FILE_EXT*
+: Ignore the URLs with the file ext.
+
+`--ignore-exts-like` `/`*REGEX*`/`
+: Ignore URLs with file exts matching the REGEX.
+
+`-r`, `--robots`
+: Specifies whether to honor `robots.txt`.
+
+`--host` *HOST*
+: Spiders the specific *HOST*.
+
+`--domain` *DOMAIN*
+: Spiders the whole *DOMAIN*.
+
+`--site` *URL*
+: Spiders the website, starting at the *URL*.
+
+## ENVIRONMENT
+
+*HTTP_PROXY*
+: Sets the global HTTP proxy.
+
+*RONIN_HTTP_PROXY*
+: Sets the HTTP proxy for Ronin.
+
+## AUTHOR
+
+Postmodern <postmodern.mod3@gmail.com>
+
+## SEE ALSO
+
+[ronin-web-spider](ronin-web-spider.1.md)

data/man/ronin-web-xml.1

@@ -0,0 +1,43 @@
+.\" Generated by kramdown-man 1.0.1
+.\" https://github.com/postmodern/kramdown-man#readme
+.TH ronin-web-xml 1 "2022-01-01" Ronin "User Manuals"
+.SH NAME
+.PP
+ronin\-web\-xml \- Performs XPath queries on a URL or XML file
+.SH SYNOPSIS
+.PP
+\fBronin\-web xml\fR \[lB]\fIoptions\fP\[rB] \[lC]\fIURL\fP \[or] \fIFILE\fP\[rC] \[lB]\fIXPATH\fP\[rB]
+.SH DESCRIPTION
+.PP
+Performs XPath queries on a URL or HTML file\.
+.SH ARGUMENTS
+.TP
+\fIURL\fP
+The \fBhttp:\[sl]\[sl]\fR or \fBhttps:\[sl]\[sl]\fR URL to fetch and parse\.
+.TP
+\fIFILE\fP
+The local XML file to parse\.
+.TP
+\fIXPATH\fP
+The XPath query expression\.
+.SH OPTIONS
+.TP
+\fB\-X\fR, \fB\-\-xpath\fR \fIXPATH\fP
+The XPath query to perform\.
+.TP
+\fB\-F\fR, \fB\-\-first\fR
+Only print the first match\.
+.TP
+\fB\-t\fR, \fB\-\-text\fR
+Prints the inner\-text of the matching elements\.
+.TP
+\fB\-h\fR, \fB\-\-help\fR
+Print help information\.
+.SH AUTHOR
+.PP
+Postmodern 
+.MT postmodern\.mod3\[at]gmail\.com
+.ME
+.SH SEE ALSO
+.PP
+.BR ronin\-web\-html (1)

data/man/ronin-web-xml.1.md

@@ -0,0 +1,46 @@
+# ronin-web-xml 1 "2022-01-01" Ronin "User Manuals"
+
+## NAME
+
+ronin-web-xml - Performs XPath queries on a URL or XML file
+
+## SYNOPSIS
+
+`ronin-web xml` [*options*] {*URL* \| *FILE*} [*XPATH*]
+
+## DESCRIPTION
+
+Performs XPath queries on a URL or HTML file.
+
+## ARGUMENTS
+
+*URL*
+: The `http://` or `https://` URL to fetch and parse.
+
+*FILE*
+: The local XML file to parse.
+
+*XPATH*
+: The XPath query expression.
+
+## OPTIONS
+
+`-X`, `--xpath` *XPATH*
+: The XPath query to perform.
+
+`-F`, `--first`
+: Only print the first match.
+
+`-t`, `--text`
+: Prints the inner-text of the matching elements.
+
+`-h`, `--help`
+: Print help information.
+
+## AUTHOR
+
+Postmodern <postmodern.mod3@gmail.com>
+
+## SEE ALSO
+
+[ronin-web-html](ronin-web-html.1.md)

data/man/ronin-web.1

@@ -1,41 +1,90 @@
-.\" Generated by kramdown-man 0.1.8
+.\" Generated by kramdown-man 1.0.1
 .\" https://github.com/postmodern/kramdown-man#readme
 .TH ronin-web 1 "2022-01-01" Ronin Web "User Manuals"
-.LP
+.SH NAME
+.PP
+ronin\-web \- a collection of useful web commands
 .SH SYNOPSIS
-.LP
-.HP
-\fBronin-web\fR \[lB]\fIoptions\fP\[rB] \[lB]\fICOMMAND\fP \[lB]\fIARGS\fP\[rB]\[rB]
-.LP
+.PP
+\fBronin\-web\fR \[lB]\fIoptions\fP\[rB] \[lB]\fICOMMAND\fP \[lB]\fIARGS\fP\[rB]\[rB]
 .SH DESCRIPTION
-.LP
 .PP
 A collection of useful web commands\.
-.LP
 .SH ARGUMENTS
-.LP
 .TP
 \fICOMMAND\fP
 The sub\-command to execute\.
-.LP
 .TP
 \fIARGS\fP
 Additional arguments for the sub\-command\.
-.LP
 .SH OPTIONS
-.LP
 .TP
-\fB-h\fR, \fB--help\fR
+\fB\-h\fR, \fB\-\-help\fR
 Prints help information\.
-.LP
+.SH COMMANDS
+.TP
+\fBbrowser\fR
+Automates a web browser\.
+.TP
+\fBcompletion\fR
+Manages the shell completion rules for \fBronin\-web\fR\.
+.TP
+\fBdiff\fR
+Diffs two web pages\.
+.TP
+\fBhtml\fR
+Performs XPath\[sl]CSS\-path queries on a URL or HTML file\.
+.TP
+\fBirb\fR
+Starts an interactive Ruby shell with ronin\-web loaded\.
+.TP
+\fBnew\fR
+Creates new projects or scripts\.
+.TP
+\fBreverse\-proxy\fR
+Starts a HTTP proxy server\.
+.TP
+\fBscreenshot\fR
+Screenshots one or more URLs\.
+.TP
+\fBserver\fR
+Starts a web server\.
+.TP
+\fBsession\-cookie\fR
+Parses and deserializes various session cookie formats\.
+.TP
+\fBspider\fR
+Spiders a website\.
+.TP
+\fBuser\-agent\fR
+Generates a random User\-Agent string\.
+.TP
+\fBwordlist\fR
+Builds a wordlist by spidering a website\.
+.TP
+\fBxml\fR
+Performs XPath queries on a URL or XML file\.
+.TP
+\fBhelp\fR
+Prints help information about this command or another command\.
 .SH AUTHOR
-.LP
 .PP
 Postmodern 
 .MT postmodern\.mod3\[at]gmail\.com
 .ME
-.LP
 .SH SEE ALSO
-.LP
 .PP
-ronin\-web\-diff(1) ronin\-web\-html(1) ronin\-web\-server(1) ronin\-web\-spider(1) ronin\-web\-proxy(1)
+.BR ronin\-web\-browser (1)
+.BR ronin\-web\-completion (1)
+.BR ronin\-web\-diff (1)
+.BR ronin\-web\-html (1)
+.BR ronin\-web\-irb (1)
+.BR ronin\-web\-new (1)
+.BR ronin\-web\-reverse\-proxy (1)
+.BR ronin\-web\-screenshot (1)
+.BR ronin\-web\-server (1)
+.BR ronin\-web\-session\-cookie (1)
+.BR ronin\-web\-spider (1)
+.BR ronin\-web\-user\-agent (1)
+.BR ronin\-web\-wordlist (1)
+.BR ronin\-web\-xml (1)

data/man/ronin-web.1.md

@@ -1,5 +1,9 @@
 # ronin-web 1 "2022-01-01" Ronin Web "User Manuals"
 
+## NAME
+
+ronin-web - a collection of useful web commands
+
 ## SYNOPSIS
 
 `ronin-web` [*options*] [*COMMAND* [*ARGS*]]
@@ -11,15 +15,62 @@ A collection of useful web commands.
 ## ARGUMENTS
 
 *COMMAND*
-  The sub-command to execute.
+: The sub-command to execute.
 
 *ARGS*
-  Additional arguments for the sub-command.
+: Additional arguments for the sub-command.
 
 ## OPTIONS
 
 `-h`, `--help`
-  Prints help information.
+: Prints help information.
+
+## COMMANDS
+
+`browser`
+: Automates a web browser.
+
+`completion`
+: Manages the shell completion rules for `ronin-web`.
+
+`diff`
+: Diffs two web pages.
+
+`html`
+: Performs XPath/CSS-path queries on a URL or HTML file.
+
+`irb`
+: Starts an interactive Ruby shell with ronin-web loaded.
+
+`new`
+: Creates new projects or scripts.
+
+`reverse-proxy`
+: Starts a HTTP proxy server.
+
+`screenshot`
+: Screenshots one or more URLs.
+
+`server`
+: Starts a web server.
+
+`session-cookie`
+: Parses and deserializes various session cookie formats.
+
+`spider`
+: Spiders a website.
+
+`user-agent`
+: Generates a random User-Agent string.
+
+`wordlist`
+: Builds a wordlist by spidering a website.
+
+`xml`
+: Performs XPath queries on a URL or XML file.
+
+`help`
+: Prints help information about this command or another command.
 
 ## AUTHOR
 
@@ -27,4 +78,4 @@ Postmodern <postmodern.mod3@gmail.com>
 
 ## SEE ALSO
 
-ronin-web-diff(1) ronin-web-html(1) ronin-web-server(1) ronin-web-spider(1) ronin-web-proxy(1)
+[ronin-web-browser](ronin-web-browser.1.md) [ronin-web-completion](ronin-web-completion.1.md) [ronin-web-diff](ronin-web-diff.1.md) [ronin-web-html](ronin-web-html.1.md) [ronin-web-irb](ronin-web-irb.1.md) [ronin-web-new](ronin-web-new.1.md) [ronin-web-reverse-proxy](ronin-web-reverse-proxy.1.md) [ronin-web-screenshot](ronin-web-screenshot.1.md) [ronin-web-server](ronin-web-server.1.md) [ronin-web-session-cookie](ronin-web-session-cookie.1.md) [ronin-web-spider](ronin-web-spider.1.md) [ronin-web-user-agent](ronin-web-user-agent.1.md) [ronin-web-wordlist](ronin-web-wordlist.1.md) [ronin-web-xml](ronin-web-xml.1.md)

data/scripts/setup

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+#
+# Prints a log message.
+#
+function log()
+{
+	if [[ -t 1 ]]; then
+		echo -e "\x1b[1m\x1b[32m>>>\x1b[0m \x1b[1m$1\x1b[0m"
+	else
+		echo ">>> $1"
+	fi
+}
+
+#
+# Prints a warn message.
+#
+function warn()
+{
+	if [[ -t 1 ]]; then
+		echo -e "\x1b[1m\x1b[33m***\x1b[0m \x1b[1m$1\x1b[0m" >&2
+	else
+		echo "*** $1" >&2
+	fi
+}
+
+#
+# Prints an error message.
+#
+function error()
+{
+	if [[ -t 1 ]]; then
+		echo -e "\x1b[1m\x1b[31m!!!\x1b[0m \x1b[1m$1\x1b[0m" >&2
+	else
+		echo "!!! $1" >&2
+	fi
+}
+
+#
+# Prints an error message and exists with -1.
+#
+function fail()
+{
+	error "$@"
+	exit -1
+}
+
+# default to installing gems into vendor/bundle
+if [[ ! -f .bundle/config ]]; then
+	bundle config set --local path vendor/bundle >/dev/null || \
+	  fail "Failed to run 'bundle config'"
+fi
+
+log "Installing gems ..."
+bundle install || fail "Failed to run 'bundle install'!"
+
+log "Setting up the project ..."
+bundle exec rake setup || "Failed to run 'rake setup'!"