nightcrawler-mitm 0.3.0__tar.gz

nightcrawler_mitm-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Paolo Perego - paolo@codiceinsicuro.it
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

nightcrawler_mitm-0.3.0/PKG-INFO

@@ -0,0 +1,162 @@
+Metadata-Version: 2.4
+Name: nightcrawler-mitm
+Version: 0.3.0
+Summary: A mitmproxy addon for background passive analysis, crawling, and basic active scanning.
+Author-email: Paolo Perego <paolo@codiceinsicuro.it>
+License: MIT License
+        
+        Copyright (c) 2020 Paolo Perego - paolo@codiceinsicuro.it
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/thesp0nge/nightcrawler-mitm
+Project-URL: Repository, https://github.com/thesp0nge/nightcrawler-mitm
+Project-URL: Bug Tracker, https://github.com/thesp0nge/nightcrawler-mitm/issues
+Keywords: mitmproxy,security,scanner,proxy,pentest,xss,sqli,crawler,addon
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: Proxy Servers
+Classifier: Topic :: Security
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mitmproxy>=10.0.0
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: beautifulsoup4>=4.10.0
+Dynamic: license-file
+
+# nightcrawler-mitm
+
+A mitmproxy addon for background passive analysis, crawling, and basic active
+scanning, designed as a security researcher's sidekick.
+
+**WARNING: Beta Stage - Use with caution, especially active scanning features**
+
+## FEATURES
+
+- Acts as an HTTP/HTTPS proxy.
+- Performs passive analysis (security headers, cookie attributes, basic info
+  disclosure).
+- Crawls the target application to discover new endpoints based on visited
+  pages.
+- Runs basic active scans for low-hanging fruit (Reflected XSS, basic SQLi -
+  Error/Time-based) in the background.
+- All output and logs are directed to the console.
+- Target scope is configurable via command-line argument.
+
+## INSTALLATION
+
+You can install `nightcrawler` directly from PyPI using pip:
+
+pip install nightcrawler-mitm
+
+It's recommended to install it in a virtual environment.
+
+## USAGE
+
+Once installed, a new command `nightcrawler` becomes available. This command
+wraps `mitmdump`, automatically loading the nightcrawler addon. You MUST specify
+the target scope using the `--nc-scope` option.
+
+You can pass any other valid `mitmdump` arguments to the `nightcrawler` command.
+
+1. Configure your Browser/Client: Set your browser (or system) to use 127.0.0.1
+   on port 8080 (or the port you specify using -p) as its HTTP and HTTPS proxy.
+
+2. Install Mitmproxy CA Certificate: For HTTPS interception, ensure the
+   mitmproxy CA certificate is installed and trusted in your browser/system.
+   While the proxy is running, visit <http://mitm.it> and follow the
+   instructions.
+
+3. Run Nightcrawler:
+
+   - Specify Target Scope (REQUIRED!): nightcrawler --nc-scope example.com
+
+   - Multiple domains (comma-separated, no spaces): nightcrawler --nc-scope
+     example.com,sub.example.com,another.net
+
+   - Common Options: (Specify port and scope) nightcrawler -p 8081 --nc-scope
+     example.com
+
+     (Disable upstream certificate verification + scope - USE WITH CAUTION!)
+     nightcrawler --ssl-insecure --nc-scope internal-site.local
+
+     (Increase verbosity + scope - Use -v for DEBUG logs) nightcrawler -v
+     --nc-scope example.com
+
+     (Combine options) nightcrawler -p 8080 --ssl-insecure -v --nc-scope
+     dev.example.com
+
+   NOTE: If --nc-scope is not provided, Nightcrawler will run but will not
+   process any requests.
+
+4. Browse: Start Browse the target application(s) specified in the scope. Output
+   from passive analysis, crawling, and active scans will appear in the terminal
+   where `nightcrawler` is running. Look for [Passive Scan], [CRAWLER
+   DISCOVERY],
+   [SQLi FOUND?], [XSS FOUND?] messages.
+
+## CONFIGURATION
+
+- Target Scope (Required): Set via the `--nc-scope` command-line argument
+  (comma-separated domains).
+
+- Other Settings: Max concurrency (MAX_CONCURRENT_SCANS) and User-Agent
+  (USER_AGENT) are currently defined in the `nightcrawler/config.py` file within
+  the installed package. Modifying these requires editing the installed file
+  (future versions may use command-line options or a separate config file). You
+  can find the installation location using `pip show nightcrawler-mitm`.
+
+## LIMITATIONS
+
+- Basic Active Scans: The SQLi and XSS scanners are very basic and intended only
+  for obvious low-hanging fruit. They CANNOT detect complex vulnerabilities
+  (e.g., Stored XSS, blind SQLi beyond time-based, DOM XSS, template injection,
+  etc.). DO NOT rely solely on this tool for comprehensive vulnerability
+  assessment.
+
+- Stored XSS: The current XSS scanner only checks for immediate reflection and
+  CANNOT detect Stored XSS.
+
+- Resource Usage: Background crawling and scanning can consume significant
+  network bandwidth, CPU, and memory resources. Adjust MAX_CONCURRENT_SCANS in
+  `config.py` if needed.
+
+- False Positives/Negatives: Expect potential false positives (especially from
+  passive checks or simple XSS reflection) and many false negatives
+  (vulnerabilities missed by the basic scanners).
+
+## LICENSE
+
+This project is licensed under the [MIT License]. See the LICENSE file for details.
+
+## CONTRIBUTING (Optional)
+
+Contributions are welcome! Please open an issue or submit a pull request on the
+GitHub repository: [https://github.com/thesp0nge/nightcrawler-mitm]

nightcrawler_mitm-0.3.0/README.md

@@ -0,0 +1,109 @@
+# nightcrawler-mitm
+
+A mitmproxy addon for background passive analysis, crawling, and basic active
+scanning, designed as a security researcher's sidekick.
+
+**WARNING: Beta Stage - Use with caution, especially active scanning features**
+
+## FEATURES
+
+- Acts as an HTTP/HTTPS proxy.
+- Performs passive analysis (security headers, cookie attributes, basic info
+  disclosure).
+- Crawls the target application to discover new endpoints based on visited
+  pages.
+- Runs basic active scans for low-hanging fruit (Reflected XSS, basic SQLi -
+  Error/Time-based) in the background.
+- All output and logs are directed to the console.
+- Target scope is configurable via command-line argument.
+
+## INSTALLATION
+
+You can install `nightcrawler` directly from PyPI using pip:
+
+pip install nightcrawler-mitm
+
+It's recommended to install it in a virtual environment.
+
+## USAGE
+
+Once installed, a new command `nightcrawler` becomes available. This command
+wraps `mitmdump`, automatically loading the nightcrawler addon. You MUST specify
+the target scope using the `--nc-scope` option.
+
+You can pass any other valid `mitmdump` arguments to the `nightcrawler` command.
+
+1. Configure your Browser/Client: Set your browser (or system) to use 127.0.0.1
+   on port 8080 (or the port you specify using -p) as its HTTP and HTTPS proxy.
+
+2. Install Mitmproxy CA Certificate: For HTTPS interception, ensure the
+   mitmproxy CA certificate is installed and trusted in your browser/system.
+   While the proxy is running, visit <http://mitm.it> and follow the
+   instructions.
+
+3. Run Nightcrawler:
+
+   - Specify Target Scope (REQUIRED!): nightcrawler --nc-scope example.com
+
+   - Multiple domains (comma-separated, no spaces): nightcrawler --nc-scope
+     example.com,sub.example.com,another.net
+
+   - Common Options: (Specify port and scope) nightcrawler -p 8081 --nc-scope
+     example.com
+
+     (Disable upstream certificate verification + scope - USE WITH CAUTION!)
+     nightcrawler --ssl-insecure --nc-scope internal-site.local
+
+     (Increase verbosity + scope - Use -v for DEBUG logs) nightcrawler -v
+     --nc-scope example.com
+
+     (Combine options) nightcrawler -p 8080 --ssl-insecure -v --nc-scope
+     dev.example.com
+
+   NOTE: If --nc-scope is not provided, Nightcrawler will run but will not
+   process any requests.
+
+4. Browse: Start Browse the target application(s) specified in the scope. Output
+   from passive analysis, crawling, and active scans will appear in the terminal
+   where `nightcrawler` is running. Look for [Passive Scan], [CRAWLER
+   DISCOVERY],
+   [SQLi FOUND?], [XSS FOUND?] messages.
+
+## CONFIGURATION
+
+- Target Scope (Required): Set via the `--nc-scope` command-line argument
+  (comma-separated domains).
+
+- Other Settings: Max concurrency (MAX_CONCURRENT_SCANS) and User-Agent
+  (USER_AGENT) are currently defined in the `nightcrawler/config.py` file within
+  the installed package. Modifying these requires editing the installed file
+  (future versions may use command-line options or a separate config file). You
+  can find the installation location using `pip show nightcrawler-mitm`.
+
+## LIMITATIONS
+
+- Basic Active Scans: The SQLi and XSS scanners are very basic and intended only
+  for obvious low-hanging fruit. They CANNOT detect complex vulnerabilities
+  (e.g., Stored XSS, blind SQLi beyond time-based, DOM XSS, template injection,
+  etc.). DO NOT rely solely on this tool for comprehensive vulnerability
+  assessment.
+
+- Stored XSS: The current XSS scanner only checks for immediate reflection and
+  CANNOT detect Stored XSS.
+
+- Resource Usage: Background crawling and scanning can consume significant
+  network bandwidth, CPU, and memory resources. Adjust MAX_CONCURRENT_SCANS in
+  `config.py` if needed.
+
+- False Positives/Negatives: Expect potential false positives (especially from
+  passive checks or simple XSS reflection) and many false negatives
+  (vulnerabilities missed by the basic scanners).
+
+## LICENSE
+
+This project is licensed under the [MIT License]. See the LICENSE file for details.
+
+## CONTRIBUTING (Optional)
+
+Contributions are welcome! Please open an issue or submit a pull request on the
+GitHub repository: [https://github.com/thesp0nge/nightcrawler-mitm]

nightcrawler_mitm-0.3.0/nightcrawler/__init__.py

@@ -0,0 +1,6 @@
+# nightcrawler/__init__.py
+# This file marks the directory as a Python package.
+# It can be empty or define the package's public API.
+# Example: from .addon import MainAddon
+
+__version__ = "0.3.0"  # Define the version here for programmatic access