secator 0.6.0__tar.gz → 0.7.0__tar.gz

{secator-0.6.0 → secator-0.7.0}/CHANGELOG.md

@@ -1,5 +1,52 @@
 # Changelog
 
+## [0.7.0](https://github.com/freelabz/secator/compare/v0.6.0...v0.7.0) (2024-11-13)
+
+
+### Features
+
+* **`bup`:** add proxy option and progress indicator ([#444](https://github.com/freelabz/secator/issues/444)) ([d5c63c8](https://github.com/freelabz/secator/commit/d5c63c81c6465a142ce70e4800effc02d526a243))
+* **`dnsx`:** add IP output type on dnsx A record  ([#426](https://github.com/freelabz/secator/issues/426)) ([629439e](https://github.com/freelabz/secator/commit/629439e459a6eefd5cbe68e9fc3a317371ba7987))
+* **`naabu`/`nmap`:** help for defaults and change workflow opts ([#438](https://github.com/freelabz/secator/issues/438)) ([4dd0055](https://github.com/freelabz/secator/commit/4dd00556648e691a093887d294325b00409ac04a))
+* **`nmap`:** add udp scan (`-sU`) and change default scan opts ([#418](https://github.com/freelabz/secator/issues/418)) ([36c6ff3](https://github.com/freelabz/secator/commit/36c6ff3766f88ac311c1bfea86a1b5e8686dd94e))
+* add url_bypass workflow based on bup ([e96b1bc](https://github.com/freelabz/secator/commit/e96b1bc9906cd2f9aa3eb5b3770594811f242abd))
+* chunk dalfox input by 1 ([#443](https://github.com/freelabz/secator/issues/443)) ([26c38d7](https://github.com/freelabz/secator/commit/26c38d79e89be3d35f464e89c6973b7beadb6ac4))
+* **cli:** misc bug fixes and features ([#445](https://github.com/freelabz/secator/issues/445)) ([fccfdb8](https://github.com/freelabz/secator/commit/fccfdb8ca38dcd3a2c559429a7d58d46ecac49a6))
+* **hooks:** explicit output type yield in static hooks ([#439](https://github.com/freelabz/secator/issues/439)) ([2d1f8e6](https://github.com/freelabz/secator/commit/2d1f8e6b7b77210028efe2c2c56866efbd6b0152))
+* **katana:** add form_fill option ([#419](https://github.com/freelabz/secator/issues/419)) ([bebddb1](https://github.com/freelabz/secator/commit/bebddb1e2fae460403adda2d84b9ae515ca977aa))
+* **refactor:** improve performance, add on_interval hook, rework CLI opts ([#473](https://github.com/freelabz/secator/issues/473)) ([4a22a70](https://github.com/freelabz/secator/commit/4a22a7082fe1edf50644034cfc54b11653b47aa4))
+* **runner:** add GCS driver and secator threads ([#476](https://github.com/freelabz/secator/issues/476)) ([cae475a](https://github.com/freelabz/secator/commit/cae475a2fe15742ccd80d40c28ad41aa1ffc5348))
+* **runner:** add skip_if_no_inputs to workflows ([#482](https://github.com/freelabz/secator/issues/482)) ([5546b82](https://github.com/freelabz/secator/commit/5546b82756d6aad0d227072d5a3b1149c44306e8))
+* **runner:** on serialized hooks ([#424](https://github.com/freelabz/secator/issues/424)) ([fde6cd7](https://github.com/freelabz/secator/commit/fde6cd7f6cba015b08b370bfd14b0aca3f4a4018))
+* **runner:** rework Celery core and mix fixes ([#450](https://github.com/freelabz/secator/issues/450)) ([b72f152](https://github.com/freelabz/secator/commit/b72f15286bb29ae60568309907d4dad41d4fbacb))
+* sudo prompt check test ([#432](https://github.com/freelabz/secator/issues/432)) ([f45b123](https://github.com/freelabz/secator/commit/f45b1230fd6313342ebdda5a359c1285f2d80aa8))
+* sudo prompts in non-tty mode ([#431](https://github.com/freelabz/secator/issues/431)) ([0e26b55](https://github.com/freelabz/secator/commit/0e26b55c168bfd69c212bc7667ef1b97e89e6bd5))
+* **tasks:** bbot integration ([#375](https://github.com/freelabz/secator/issues/375)) ([2f0dea4](https://github.com/freelabz/secator/commit/2f0dea4f4cac3370129d0adf0000c8d0efa54361))
+* **tasks:** bup integration ([#398](https://github.com/freelabz/secator/issues/398)) ([ed636aa](https://github.com/freelabz/secator/commit/ed636aad7d90baa7b3b73baebc8f5be002dd796a))
+
+
+### Bug Fixes
+
+* **cli:** proper opts override for workflows ([#436](https://github.com/freelabz/secator/issues/436)) ([1d1eaa3](https://github.com/freelabz/secator/commit/1d1eaa3283b3d5e9650b452e3865476e6a60a086))
+* **dalfox:** restore input_chunk_size to default ([8f0a3b4](https://github.com/freelabz/secator/commit/8f0a3b4976e20afc2fb708483c7f8885b2b9f3d9))
+* dnsx parsing output loading error ([#422](https://github.com/freelabz/secator/issues/422)) ([b9e98da](https://github.com/freelabz/secator/commit/b9e98da2b5378957076e1d8f0afd3948d5bcb5f6))
+* empty CVE should pass ([#478](https://github.com/freelabz/secator/issues/478)) ([0644d68](https://github.com/freelabz/secator/commit/0644d68ccb92a4c38e8210e39f14f0850d84348d))
+* gcs bug with empty paths ([549ac4c](https://github.com/freelabz/secator/commit/549ac4c8e7391a829cf1a6c5a43ad291bec1b34a))
+* gcs bug with empty paths ([2d57e1a](https://github.com/freelabz/secator/commit/2d57e1ad4669587cf0abb0a59b0918cf72107d72))
+* get_opt_value default value and reorg hooks ([#429](https://github.com/freelabz/secator/issues/429)) ([a44a36d](https://github.com/freelabz/secator/commit/a44a36d37f888787927ec6dfc891e86dab071aa4))
+* mix bugfixes for stable release ([b743925](https://github.com/freelabz/secator/commit/b7439258c9cdadc7bd14a0a0b49e2db2d0f5b537))
+* nmap defaults ([396f68a](https://github.com/freelabz/secator/commit/396f68a325a5a8f1a9379d314979dbf85a9c95c7))
+* **nmap:** undefined service name ([#437](https://github.com/freelabz/secator/issues/437)) ([596f1af](https://github.com/freelabz/secator/commit/596f1aff53e9add73e1587497aee82465d212300))
+* runner opts processing ([#477](https://github.com/freelabz/secator/issues/477)) ([d788e9d](https://github.com/freelabz/secator/commit/d788e9d3e508a849119d418bcc5ce371c6c53c6c))
+* runner toDict() errors ([#475](https://github.com/freelabz/secator/issues/475)) ([b43c866](https://github.com/freelabz/secator/commit/b43c8669808651368536fa121be2ce79de7556aa))
+* **runner:** bug with no inputs ([#483](https://github.com/freelabz/secator/issues/483)) ([4db7b46](https://github.com/freelabz/secator/commit/4db7b460a949e6b74b5837f0f1e3b5ca51b39094))
+* **url_vuln:** repair bad condition ([214c8ab](https://github.com/freelabz/secator/commit/214c8abf7cad4916c8301ff056d894cc0bc26b28))
+
+
+### Documentation
+
+* add package json ([#415](https://github.com/freelabz/secator/issues/415)) ([f9a7c2f](https://github.com/freelabz/secator/commit/f9a7c2fc5df11506cce0d81babf1f7790b80465a))
+
 ## [0.6.0](https://github.com/freelabz/secator/compare/v0.5.2...v0.6.0) (2024-07-25)
 
 

{secator-0.6.0 → secator-0.7.0}/Dockerfile

@@ -37,7 +37,8 @@ COPY . /code/
 RUN pipx install .
 RUN secator install tools
 RUN secator install addons worker
-RUN secator install addons google
+RUN secator install addons gdrive
+RUN secator install addons gcs
 RUN secator install addons mongodb
 RUN secator install addons redis
 RUN secator install addons dev

{secator-0.6.0 → secator-0.7.0}/PKG-INFO

@@ -1,11 +1,10 @@
 Metadata-Version: 2.3
 Name: secator
-Version: 0.6.0
+Version: 0.7.0
 Summary: The pentester's swiss knife.
 Project-URL: Homepage, https://github.com/freelabz/secator
 Project-URL: Issues, https://github.com/freelabz/secator/issues
 Author-email: FreeLabz <sales@freelabz.com>
-License-File: LICENSE
 Keywords: automation,cybersecurity,pentest,recon,vulnerability
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
@@ -25,16 +24,20 @@ Requires-Dist: cpe<2
 Requires-Dist: dotmap<2
 Requires-Dist: free-proxy<2
 Requires-Dist: furl<3
+Requires-Dist: greenlet<4
 Requires-Dist: humanize<5
 Requires-Dist: ifaddr<1
 Requires-Dist: jinja2<4
 Requires-Dist: packaging<25
+Requires-Dist: psutil<7
 Requires-Dist: pydantic<3
 Requires-Dist: python-dotenv<2
 Requires-Dist: pyyaml<7
 Requires-Dist: requests<3
+Requires-Dist: retry<1
 Requires-Dist: rich-click<1.7
 Requires-Dist: rich<14
+Requires-Dist: tldextract<6
 Requires-Dist: typing-extensions<5
 Requires-Dist: validators<1
 Requires-Dist: xmltodict<1
@@ -44,10 +47,13 @@ Provides-Extra: dev
 Requires-Dist: asciinema-automation<1; extra == 'dev'
 Requires-Dist: coverage<8; extra == 'dev'
 Requires-Dist: flake8<8; extra == 'dev'
+Requires-Dist: pytest<9; extra == 'dev'
 Requires-Dist: watchdog<3; extra == 'dev'
-Provides-Extra: google
-Requires-Dist: google-api-python-client<3; extra == 'google'
-Requires-Dist: gspread<7; extra == 'google'
+Provides-Extra: gcs
+Requires-Dist: google-cloud-storage<3; extra == 'gcs'
+Provides-Extra: gdrive
+Requires-Dist: google-api-python-client<3; extra == 'gdrive'
+Requires-Dist: gspread<7; extra == 'gdrive'
 Provides-Extra: mongodb
 Requires-Dist: pymongo<5; extra == 'mongodb'
 Provides-Extra: redis

secator-0.7.0/pyproject.toml

@@ -0,0 +1,97 @@
+[build-system]
+requires = ['hatchling']
+build-backend = 'hatchling.build'
+
+[project]
+name = 'secator'
+version = "0.7.0"
+authors = [{ name = 'FreeLabz', email = 'sales@freelabz.com' }]
+readme = 'README.md'
+description = "The pentester's swiss knife."
+requires-python = '>=3.8'
+keywords = [
+  'cybersecurity',
+  'recon',
+  'vulnerability',
+  'pentest',
+  'automation'
+]
+classifiers = [
+  'Development Status :: 3 - Alpha',
+  'Intended Audience :: Developers',
+  'Intended Audience :: Information Technology',
+  'License :: Free for non-commercial use',
+  'Operating System :: Unix',
+  'Programming Language :: Python',
+  'Programming Language :: Python :: 3',
+  'Programming Language :: Python :: 3.8',
+  'Programming Language :: Python :: 3.9',
+  'Programming Language :: Python :: 3.10',
+  'Programming Language :: Python :: 3.11',
+]
+dependencies = [
+  'beautifulsoup4 <= 5',
+  'celery < 6',
+  'cpe < 2',
+  'dotmap < 2',
+  'free-proxy < 2',
+  'furl < 3',
+  'greenlet < 4',
+  'humanize < 5',
+  'ifaddr < 1',
+  'jinja2 < 4',
+  'packaging < 25',
+  'python-dotenv < 2',
+  'pyyaml < 7',
+  'pydantic < 3',
+  'requests < 3',
+  'rich < 14',
+  'rich-click < 1.7',
+  'psutil < 7',
+  'retry < 1',
+  'tldextract < 6',
+  'typing_extensions < 5',
+  'validators < 1',
+  'xmltodict < 1'
+]
+
+[project.optional-dependencies]
+dev = [
+  'coverage < 8',
+  'flake8 < 8',
+  'pytest < 9',
+  'watchdog < 3',
+  'asciinema-automation < 1',
+]
+build = [
+  'hatch < 2',
+]
+trace = [
+  'memray < 2',
+  'pyinstrument < 5',
+]
+worker = [
+  'gevent < 25',
+  'eventlet < 1',
+  'flower < 3',
+]
+redis = [
+  'redis < 6',
+]
+mongodb = [
+  'pymongo < 5',
+]
+gdrive = [
+  'google-api-python-client < 3',
+  'gspread < 7'
+]
+gcs = [
+	'google-cloud-storage < 3'
+]
+
+[project.scripts]
+secator = 'secator.cli:cli'
+
+[project.urls]
+Homepage = 'https://github.com/freelabz/secator'
+Issues = 'https://github.com/freelabz/secator/issues'

secator-0.7.0/secator/celery.py

@@ -0,0 +1,344 @@
+import gc
+import logging
+import sys
+import uuid
+
+from time import time
+
+from celery import Celery, chain, chord, signals
+from celery.app import trace
+
+from rich.logging import RichHandler
+from retry import retry
+
+from secator.config import CONFIG
+from secator.output_types import Info, Warning, Error
+from secator.rich import console
+from secator.runners import Scan, Task, Workflow
+from secator.runners._helpers import run_extractors
+from secator.utils import (debug, deduplicate, flatten, should_update)
+
+IN_CELERY_WORKER_PROCESS = sys.argv and ('secator.celery.app' in sys.argv or 'worker' in sys.argv)
+
+#---------#
+# Logging #
+#---------#
+
+rich_handler = RichHandler(rich_tracebacks=True)
+rich_handler.setLevel(logging.INFO)
+logging.basicConfig(
+	level='NOTSET',
+	format="%(threadName)s:%(message)s",
+	datefmt="[%X]",
+	handlers=[rich_handler],
+	force=True)
+logging.getLogger('kombu').setLevel(logging.ERROR)
+logging.getLogger('celery').setLevel(logging.INFO if CONFIG.debug.level > 6 else logging.WARNING)
+logger = logging.getLogger(__name__)
+trace.LOG_SUCCESS = "Task %(name)s[%(id)s] succeeded in %(runtime)ss"
+
+
+#------------#
+# Celery app #
+#------------#
+
+app = Celery(__name__)
+app.conf.update({
+	# Content types
+	'accept_content': ['application/x-python-serialize', 'application/json'],
+
+	# Broker config
+	'broker_url': CONFIG.celery.broker_url,
+	'broker_transport_options': {
+		'data_folder_in': CONFIG.dirs.celery_data,
+		'data_folder_out': CONFIG.dirs.celery_data,
+		'control_folder': CONFIG.dirs.celery_data,
+		'visibility_timeout': CONFIG.celery.broker_visibility_timeout,
+	},
+	'broker_connection_retry_on_startup': True,
+	'broker_pool_limit': CONFIG.celery.broker_pool_limit,
+	'broker_connection_timeout': CONFIG.celery.broker_connection_timeout,
+
+	# Result backend config
+	'result_backend': CONFIG.celery.result_backend,
+	'result_expires': CONFIG.celery.result_expires,
+	'result_extended': True,
+	'result_backend_thread_safe': True,
+	'result_serializer': 'pickle',
+	# 'result_backend_transport_options': {'master_name': 'mymaster'}, # for Redis HA backend
+
+	# Task config
+	'task_acks_late': False,
+	'task_compression': 'gzip',
+	'task_create_missing_queues': True,
+	'task_eager_propagates': False,
+	'task_reject_on_worker_lost': False,
+	'task_routes': {
+		'secator.celery.run_workflow': {'queue': 'celery'},
+		'secator.celery.run_scan': {'queue': 'celery'},
+		'secator.celery.run_task': {'queue': 'celery'},
+		'secator.hooks.mongodb.tag_duplicates': {'queue': 'mongodb'}
+	},
+	'task_store_eager_result': True,
+	# 'task_send_sent_event': True,  # TODO: consider enabling this for Flower monitoring
+	'task_serializer': 'pickle',
+
+	# Worker config
+	# 'worker_direct': True,  # TODO: consider enabling this to allow routing to specific workers
+	'worker_max_tasks_per_child': 10,
+	# 'worker_max_memory_per_child': 100000  # TODO: consider enabling this
+	'worker_pool_restarts': True,
+	'worker_prefetch_multiplier': 1,
+	# 'worker_send_task_events': True,  # TODO: consider enabling this for Flower monitoring
+})
+app.autodiscover_tasks(['secator.hooks.mongodb'], related_name=None)
+
+
+def maybe_override_logging():
+	def decorator(func):
+		if CONFIG.celery.override_default_logging:
+			return signals.setup_logging.connect(func)
+		else:
+			return func
+	return decorator
+
+
+@maybe_override_logging()
+def void(*args, **kwargs):
+	"""Override celery's logging setup to prevent it from altering our settings.
+	github.com/celery/celery/issues/1867
+	"""
+	pass
+
+
+@retry(Exception, tries=3, delay=2)
+def update_state(celery_task, task, force=False):
+	"""Update task state to add metadata information."""
+	if task.sync:
+		return
+	if not force and not should_update(CONFIG.runners.backend_update_frequency, task.last_updated_celery):
+		return
+	task.last_updated_celery = time()
+	debug(
+		'',
+		sub='celery.state',
+		id=celery_task.request.id,
+		obj={task.unique_name: task.status, 'count': task.self_findings_count},
+		obj_after=False,
+		verbose=True
+	)
+	return celery_task.update_state(
+		state='RUNNING',
+		meta=task.celery_state
+	)
+
+
+def revoke_task(task_id, task_name=None):
+	message = f'Revoked task {task_id}'
+	if task_name:
+		message += f' ({task_name})'
+	app.control.revoke(task_id, terminate=True)
+	console.print(Info(message=message))
+
+
+#--------------#
+# Celery tasks #
+#--------------#
+
+
+def chunker(seq, size):
+	return (seq[pos:pos + size] for pos in range(0, len(seq), size))
+
+
+def break_task(task, task_opts, targets, results=[], chunk_size=1):
+	"""Break a task into multiple of the same type."""
+	chunks = targets
+	if chunk_size > 1:
+		chunks = list(chunker(targets, chunk_size))
+	debug(
+		'',
+		obj={task.unique_name: 'CHUNKED', 'chunk_size': chunk_size, 'chunks': len(chunks), 'target_count': len(targets)},
+		obj_after=False,
+		sub='celery.state',
+		verbose=True
+	)
+
+	# Clone opts
+	opts = task_opts.copy()
+
+	# Build signatures
+	sigs = []
+	task.ids_map = {}
+	for ix, chunk in enumerate(chunks):
+		if not isinstance(chunk, list):
+			chunk = [chunk]
+		if len(chunks) > 0:  # add chunk to task opts for tracking chunks exec
+			opts['chunk'] = ix + 1
+			opts['chunk_count'] = len(chunks)
+		task_id = str(uuid.uuid4())
+		opts['has_parent'] = True
+		opts['enable_duplicate_check'] = False
+		sig = type(task).s(chunk, **opts).set(queue=type(task).profile, task_id=task_id)
+		full_name = f'{task.name}_{ix + 1}'
+		task.add_subtask(task_id, task.name, f'{task.name}_{ix + 1}')
+		info = Info(message=f'Celery chunked task created: {task_id}', _source=full_name, _uuid=str(uuid.uuid4()))
+		task.add_result(info)
+		sigs.append(sig)
+
+	# Build Celery workflow
+	workflow = chain(
+		forward_results.s(results).set(queue='io'),
+		chord(
+			tuple(sigs),
+			forward_results.s().set(queue='io'),
+		)
+	)
+	if task.sync:
+		task.print_item = False
+		task.results = workflow.apply().get()
+	else:
+		result = workflow.apply_async()
+		task.celery_result = result
+
+
+@app.task(bind=True)
+def run_task(self, args=[], kwargs={}):
+	kwargs['context']['celery_id'] = self.request.id
+	task = Task(*args, **kwargs)
+	task.run()
+
+
+@app.task(bind=True)
+def run_workflow(self, args=[], kwargs={}):
+	kwargs['context']['celery_id'] = self.request.id
+	workflow = Workflow(*args, **kwargs)
+	workflow.run()
+
+
+@app.task(bind=True)
+def run_scan(self, args=[], kwargs={}):
+	if 'context' not in kwargs:
+		kwargs['context'] = {}
+	kwargs['context']['celery_id'] = self.request.id
+	scan = Scan(*args, **kwargs)
+	scan.run()
+
+
+@app.task(bind=True)
+def run_command(self, results, name, targets, opts={}):
+	chunk = opts.get('chunk')
+	sync = opts.get('sync', True)
+
+	# Set Celery request id in context
+	context = opts.get('context', {})
+	context['celery_id'] = self.request.id
+	opts['context'] = context
+	opts['print_remote_info'] = False
+	opts['results'] = results
+
+	# If we are in a Celery worker, print everything, always
+	if IN_CELERY_WORKER_PROCESS:
+		opts.update({
+			'print_item': True,
+			'print_line': True,
+			'print_cmd': True
+		})
+
+	# Flatten + dedupe results
+	results = flatten(results)
+	results = deduplicate(results, attr='_uuid')
+
+	# Get expanded targets
+	if not chunk and results:
+		targets, opts = run_extractors(results, opts, targets)
+		debug('after extractors', obj={'targets': targets, 'opts': opts}, sub='celery.state')
+
+	try:
+		# Get task class
+		task_cls = Task.get_task_class(name)
+
+		# Check if chunkable
+		many_targets = len(targets) > 1
+		targets_over_chunk_size = task_cls.input_chunk_size and len(targets) > task_cls.input_chunk_size
+		has_file_flag = task_cls.file_flag is not None
+		chunk_it = (sync and many_targets and not has_file_flag) or (not sync and many_targets and targets_over_chunk_size)
+		task_opts = opts.copy()
+		task_opts.update({
+			'print_remote_info': False,
+			'has_children': chunk_it,
+		})
+		if chunk_it:
+			task_opts['print_cmd'] = False
+		task = task_cls(targets, **task_opts)
+		debug(
+			'',
+			obj={
+				f'{task.unique_name}': 'CHUNK STATUS',
+				'chunk_it': chunk_it,
+				'sync': task.sync,
+				'many_targets': many_targets,
+				'targets_over_chunk_size': targets_over_chunk_size,
+			},
+			obj_after=False,
+			id=self.request.id,
+			sub='celery.state',
+			verbose=True
+		)
+
+		# Chunk task if needed
+		if chunk_it:
+			chunk_size = task_cls.input_chunk_size if has_file_flag else 1
+			break_task(
+				task,
+				opts,
+				targets,
+				results=results,
+				chunk_size=chunk_size)
+
+		# Update state before starting
+		update_state(self, task)
+
+		# Update state for each item found
+		for _ in task:
+			update_state(self, task)
+
+	except BaseException as e:
+		error = Error.from_exception(e)
+		error._source = task.unique_name
+		error._uuid = str(uuid.uuid4())
+		task.add_result(error, print=True)
+		task.stop_celery_tasks()
+
+	finally:
+		update_state(self, task, force=True)
+		gc.collect()
+		debug('', obj={task.unique_name: task.status, 'results': task.results}, sub='celery.results', verbose=True)
+		return task.results
+
+
+@app.task
+def forward_results(results):
+	if isinstance(results, list):
+		for ix, item in enumerate(results):
+			if isinstance(item, dict) and 'results' in item:
+				results[ix] = item['results']
+	elif 'results' in results:
+		results = results['results']
+	results = flatten(results)
+	results = deduplicate(results, attr='_uuid')
+	return results
+
+#--------------#
+# Celery utils #
+#--------------#
+
+
+def is_celery_worker_alive():
+	"""Check if a Celery worker is available."""
+	result = app.control.broadcast('ping', reply=True, limit=1, timeout=1)
+	result = bool(result)
+	if result:
+		console.print(Info(message='Celery worker is alive !'))
+	else:
+		console.print(Warning(message='No Celery worker alive.'))
+	return result