rekos 1.3.0__tar.gz

rekos-1.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 REKOS contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rekos-1.3.0/PKG-INFO

@@ -0,0 +1,223 @@
+Metadata-Version: 2.4
+Name: rekos
+Version: 1.3.0
+Summary: Terminal-native passive OSINT CLI for local-first public-source investigation workspaces
+Author: VladTepes84
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/VladTepes84/Rekos
+Project-URL: Repository, https://github.com/VladTepes84/Rekos
+Project-URL: Issues, https://github.com/VladTepes84/Rekos/issues
+Keywords: osint,passive-osint,cli,sqlite,investigation,public-source
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Information Technology
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet
+Classifier: Topic :: Security
+Classifier: Topic :: Utilities
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: rich>=13.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Dynamic: license-file
+
+# REKOS
+![REKOS quickstart](assets/rekos-quickstart.png)
+
+REKOS is a terminal-native passive OSINT CLI for local-first public-source investigation workspaces. It helps organize targets, evidence, source outputs, entities, relationships, normalized findings, and correlation-quality scores in a SQLite-backed case folder.
+
+REKOS is designed for passive public-source workflows:
+
+- Public-source investigation workspace
+- Target and evidence organizer
+- Username, profile, domain, URL, and indicator correlation tool
+- Local-first OSINT case workspace
+- No login automation, bypass, credential collection, or active exploitation
+
+A case is a local workspace for one public-source research thread. Cases are stored under `~/rekos_cases/<case_name>` by default. Each case keeps its own SQLite database, source outputs, evidence artifacts, graph records, findings, and exports.
+
+## Installation
+
+Install with pipx:
+
+```bash
+pipx install rekos
+```
+
+Users run REKOS commands such as `rekos investigate username <case> <username>` and `rekos investigate domain <case> <domain>`. REKOS calls available passive sources through its source adapters and continues cleanly when optional external tools are absent.
+
+Install from a local checkout:
+
+```bash
+git clone <repo-url>
+cd rekos
+pipx install .
+rekos --help
+```
+
+For development, use an editable install:
+
+```bash
+git clone <repo-url>
+cd rekos
+python -m venv .venv
+. .venv/bin/activate
+python -m pip install -e ".[dev]"
+pytest
+rekos --help
+```
+
+## Optional Integrations
+
+- `sherlock` enables the `sherlock_username` source when the `sherlock` command is installed
+- `maigret` enables the `maigret_username` source when available in the REKOS runtime
+- `exiftool` or `mediainfo` for file metadata collection
+- Playwright is optional for URL screenshots; HTTP snapshots still work without it
+
+Users always run `rekos`, not Sherlock or Maigret directly. `rekos investigate username <case> <username>` automatically uses the username sources available in the current environment.
+
+## Quick Start
+
+```bash
+rekos new-case social_test
+rekos investigate username social_test username
+rekos findings social_test
+rekos score social_test
+rekos graph-summary social_test
+rekos export-case social_test --output social_test.zip
+```
+
+Normal workflow:
+
+1. Create a case with `rekos new-case`.
+2. Add or investigate a target with `rekos investigate username`, `rekos investigate domain`, or `rekos snapshot-url`.
+3. Review normalized results with `rekos findings`.
+4. Score correlation quality with `rekos score`.
+5. Inspect relationships with `rekos graph-summary` or `rekos list-entities`.
+6. Export the workspace with `rekos export-case`.
+
+Most users only need these commands:
+
+```bash
+rekos quickstart
+rekos new-case acme-osint
+rekos investigate username acme-osint alice.example
+rekos investigate domain acme-osint example.com
+rekos snapshot-url acme-osint https://example.com/profile/alice
+rekos findings acme-osint
+rekos score acme-osint
+rekos search acme-osint example.com
+rekos graph-summary acme-osint
+rekos export-case acme-osint --output ./acme-osint.zip
+```
+
+Users run only `rekos`. Sherlock and Maigret are optional integrations that REKOS orchestrates internally when available.
+
+During `rekos investigate username <case> <username>`, REKOS generates safe username variants, runs available passive username sources, stores raw source output, normalizes discovered profile URLs into findings, updates the entity graph, records timeline events, and computes correlation-quality scores. Results are correlation indicators, not proof of identity ownership.
+
+During `rekos investigate domain <case> <domain>`, REKOS runs passive DNS, RDAP with registry/WHOIS fallback, HTTP/HTTPS endpoint checks, TLS certificate metadata collection, and crt.sh certificate transparency lookup when available. It records registration evidence, DNS records, web endpoint metadata, redirects, TLS certificate summaries, SPF/mail-security summaries, provider hints from TXT records, and certificate transparency findings.
+
+Domain, URL, and snapshot workflows reject localhost, private/internal IP ranges, link-local addresses, metadata-service IPs, reserved, multicast, and unspecified IP targets. REKOS is for public-source targets only.
+
+## How REKOS Works
+
+- Target input: user-provided usernames, domains, URLs, files, notes, and indicators are stored in a local case.
+- Source orchestration: REKOS runs passive adapters such as username sources, DNS, RDAP/WHOIS fallback, web/TLS checks, crt.sh, Wayback, metadata tools, and HTTP snapshots when available.
+- Findings normalization: raw source output is converted into normalized findings such as discovered profiles, URLs, domains, metadata records, archive records, and registration records.
+- Graph correlation: entities and relationships connect usernames, profiles, domains, URLs, files, and notes.
+- Quality scoring: REKOS scores correlation quality from source confidence, exact or normalized matches, duplicate source confirmation, evidence presence, and graph relationships.
+- Evidence export: raw outputs, artifacts, reports, SQLite data, and manifests can be exported with `rekos export-case`.
+
+## Supported Sources
+
+| Source              | Target types    | Dependencies                     | Notes                                                                           |
+|---------------------|-----------------|----------------------------------|---------------------------------------------------------------------------------|
+| `sherlock_username` | `username`      | `sherlock` binary                | Runs Sherlock with safe subprocess arguments and parses public profile URLs.     |
+| `maigret_username`  | `username`      | optional `maigret` package/tool  | Runs Maigret when installed; REKOS continues without it.                        |
+| `wmn_username`      | `username`      | none                             | Checks local public profile URL templates with conservative passive HTTP validation. |
+| `http_snapshot`     | `url`           | none                             | Captures public HTTP response artifacts and optional Playwright screenshot.      |
+| `rdap_domain`       | `domain`        | none                             | Uses public HTTPS RDAP lookup with registry and WHOIS fallback where available.  |
+| `dns_domain`        | `domain`        | none                             | Fetches public DNS A/AAAA/MX/NS/TXT records and extracts SPF/provider hints.     |
+| `web_domain`        | `domain`        | none                             | Performs passive HTTP/HTTPS endpoint and TLS certificate metadata checks.         |
+| `crtsh_domain`      | `domain`        | none                             | Queries the public crt.sh certificate transparency endpoint.                     |
+| `wayback_url`       | `url`, `domain` | none                             | Queries public Wayback CDX data and records archive URLs.                        |
+
+Source utilities:
+
+```bash
+rekos sources list
+rekos sources check
+rekos sources run acme-osint rdap_domain example.com
+```
+
+## Core Commands
+
+```bash
+rekos add-entity acme-osint --type domain --value example.com
+rekos relate-entities acme-osint --from <entity_uuid> --to <entity_uuid> --type related_to --confidence medium
+rekos list-targets acme-osint
+rekos list-sources acme-osint
+rekos show-investigation acme-osint
+rekos report acme-osint --format md
+```
+
+## Safety And Ethics
+
+REKOS is passive-only OSINT tooling. Use it only for lawful, authorized, and ethical public-source research.
+
+REKOS must not be used for:
+
+- Logging into accounts or automating authenticated sessions
+- Bypassing access controls, paywalls, CAPTCHAs, bot protection, or rate limits
+- Credential collection, phishing, account abuse, or social engineering
+- Exploitation, destructive operations, or aggressive crawling
+- Claiming identity ownership from correlation results
+
+Scores are correlation-quality indicators only. A high score means stronger local correlation support, not proof of identity, ownership, compromise, or intent.
+
+## Local Data Model
+
+REKOS stores:
+
+- Case metadata in SQLite
+- Targets, entities, relationships, notes, timeline events
+- Raw source outputs under `exports/`
+- Evidence and snapshot artifacts
+- Normalized findings with correlation-quality scores
+- Case ZIP exports with manifest data
+
+## Development
+
+```bash
+python -m pip install -e ".[dev]"
+pytest
+rekos --help
+```
+
+Before submitting a change:
+
+```bash
+pytest
+python -m compileall rekos
+git diff --check
+```
+
+## Roadmap
+
+- More passive source adapters with explicit safety boundaries
+- Stronger report templates and case export validation
+- Improved graph summaries and finding explainability
+- Better import/export interoperability
+- Optional UI views while keeping the CLI and local-first storage as the core
+
+## License
+
+MIT License. See [LICENSE](LICENSE).

rekos-1.3.0/README.md

@@ -0,0 +1,192 @@
+# REKOS
+![REKOS quickstart](assets/rekos-quickstart.png)
+
+REKOS is a terminal-native passive OSINT CLI for local-first public-source investigation workspaces. It helps organize targets, evidence, source outputs, entities, relationships, normalized findings, and correlation-quality scores in a SQLite-backed case folder.
+
+REKOS is designed for passive public-source workflows:
+
+- Public-source investigation workspace
+- Target and evidence organizer
+- Username, profile, domain, URL, and indicator correlation tool
+- Local-first OSINT case workspace
+- No login automation, bypass, credential collection, or active exploitation
+
+A case is a local workspace for one public-source research thread. Cases are stored under `~/rekos_cases/<case_name>` by default. Each case keeps its own SQLite database, source outputs, evidence artifacts, graph records, findings, and exports.
+
+## Installation
+
+Install with pipx:
+
+```bash
+pipx install rekos
+```
+
+Users run REKOS commands such as `rekos investigate username <case> <username>` and `rekos investigate domain <case> <domain>`. REKOS calls available passive sources through its source adapters and continues cleanly when optional external tools are absent.
+
+Install from a local checkout:
+
+```bash
+git clone <repo-url>
+cd rekos
+pipx install .
+rekos --help
+```
+
+For development, use an editable install:
+
+```bash
+git clone <repo-url>
+cd rekos
+python -m venv .venv
+. .venv/bin/activate
+python -m pip install -e ".[dev]"
+pytest
+rekos --help
+```
+
+## Optional Integrations
+
+- `sherlock` enables the `sherlock_username` source when the `sherlock` command is installed
+- `maigret` enables the `maigret_username` source when available in the REKOS runtime
+- `exiftool` or `mediainfo` for file metadata collection
+- Playwright is optional for URL screenshots; HTTP snapshots still work without it
+
+Users always run `rekos`, not Sherlock or Maigret directly. `rekos investigate username <case> <username>` automatically uses the username sources available in the current environment.
+
+## Quick Start
+
+```bash
+rekos new-case social_test
+rekos investigate username social_test username
+rekos findings social_test
+rekos score social_test
+rekos graph-summary social_test
+rekos export-case social_test --output social_test.zip
+```
+
+Normal workflow:
+
+1. Create a case with `rekos new-case`.
+2. Add or investigate a target with `rekos investigate username`, `rekos investigate domain`, or `rekos snapshot-url`.
+3. Review normalized results with `rekos findings`.
+4. Score correlation quality with `rekos score`.
+5. Inspect relationships with `rekos graph-summary` or `rekos list-entities`.
+6. Export the workspace with `rekos export-case`.
+
+Most users only need these commands:
+
+```bash
+rekos quickstart
+rekos new-case acme-osint
+rekos investigate username acme-osint alice.example
+rekos investigate domain acme-osint example.com
+rekos snapshot-url acme-osint https://example.com/profile/alice
+rekos findings acme-osint
+rekos score acme-osint
+rekos search acme-osint example.com
+rekos graph-summary acme-osint
+rekos export-case acme-osint --output ./acme-osint.zip
+```
+
+Users run only `rekos`. Sherlock and Maigret are optional integrations that REKOS orchestrates internally when available.
+
+During `rekos investigate username <case> <username>`, REKOS generates safe username variants, runs available passive username sources, stores raw source output, normalizes discovered profile URLs into findings, updates the entity graph, records timeline events, and computes correlation-quality scores. Results are correlation indicators, not proof of identity ownership.
+
+During `rekos investigate domain <case> <domain>`, REKOS runs passive DNS, RDAP with registry/WHOIS fallback, HTTP/HTTPS endpoint checks, TLS certificate metadata collection, and crt.sh certificate transparency lookup when available. It records registration evidence, DNS records, web endpoint metadata, redirects, TLS certificate summaries, SPF/mail-security summaries, provider hints from TXT records, and certificate transparency findings.
+
+Domain, URL, and snapshot workflows reject localhost, private/internal IP ranges, link-local addresses, metadata-service IPs, reserved, multicast, and unspecified IP targets. REKOS is for public-source targets only.
+
+## How REKOS Works
+
+- Target input: user-provided usernames, domains, URLs, files, notes, and indicators are stored in a local case.
+- Source orchestration: REKOS runs passive adapters such as username sources, DNS, RDAP/WHOIS fallback, web/TLS checks, crt.sh, Wayback, metadata tools, and HTTP snapshots when available.
+- Findings normalization: raw source output is converted into normalized findings such as discovered profiles, URLs, domains, metadata records, archive records, and registration records.
+- Graph correlation: entities and relationships connect usernames, profiles, domains, URLs, files, and notes.
+- Quality scoring: REKOS scores correlation quality from source confidence, exact or normalized matches, duplicate source confirmation, evidence presence, and graph relationships.
+- Evidence export: raw outputs, artifacts, reports, SQLite data, and manifests can be exported with `rekos export-case`.
+
+## Supported Sources
+
+| Source              | Target types    | Dependencies                     | Notes                                                                           |
+|---------------------|-----------------|----------------------------------|---------------------------------------------------------------------------------|
+| `sherlock_username` | `username`      | `sherlock` binary                | Runs Sherlock with safe subprocess arguments and parses public profile URLs.     |
+| `maigret_username`  | `username`      | optional `maigret` package/tool  | Runs Maigret when installed; REKOS continues without it.                        |
+| `wmn_username`      | `username`      | none                             | Checks local public profile URL templates with conservative passive HTTP validation. |
+| `http_snapshot`     | `url`           | none                             | Captures public HTTP response artifacts and optional Playwright screenshot.      |
+| `rdap_domain`       | `domain`        | none                             | Uses public HTTPS RDAP lookup with registry and WHOIS fallback where available.  |
+| `dns_domain`        | `domain`        | none                             | Fetches public DNS A/AAAA/MX/NS/TXT records and extracts SPF/provider hints.     |
+| `web_domain`        | `domain`        | none                             | Performs passive HTTP/HTTPS endpoint and TLS certificate metadata checks.         |
+| `crtsh_domain`      | `domain`        | none                             | Queries the public crt.sh certificate transparency endpoint.                     |
+| `wayback_url`       | `url`, `domain` | none                             | Queries public Wayback CDX data and records archive URLs.                        |
+
+Source utilities:
+
+```bash
+rekos sources list
+rekos sources check
+rekos sources run acme-osint rdap_domain example.com
+```
+
+## Core Commands
+
+```bash
+rekos add-entity acme-osint --type domain --value example.com
+rekos relate-entities acme-osint --from <entity_uuid> --to <entity_uuid> --type related_to --confidence medium
+rekos list-targets acme-osint
+rekos list-sources acme-osint
+rekos show-investigation acme-osint
+rekos report acme-osint --format md
+```
+
+## Safety And Ethics
+
+REKOS is passive-only OSINT tooling. Use it only for lawful, authorized, and ethical public-source research.
+
+REKOS must not be used for:
+
+- Logging into accounts or automating authenticated sessions
+- Bypassing access controls, paywalls, CAPTCHAs, bot protection, or rate limits
+- Credential collection, phishing, account abuse, or social engineering
+- Exploitation, destructive operations, or aggressive crawling
+- Claiming identity ownership from correlation results
+
+Scores are correlation-quality indicators only. A high score means stronger local correlation support, not proof of identity, ownership, compromise, or intent.
+
+## Local Data Model
+
+REKOS stores:
+
+- Case metadata in SQLite
+- Targets, entities, relationships, notes, timeline events
+- Raw source outputs under `exports/`
+- Evidence and snapshot artifacts
+- Normalized findings with correlation-quality scores
+- Case ZIP exports with manifest data
+
+## Development
+
+```bash
+python -m pip install -e ".[dev]"
+pytest
+rekos --help
+```
+
+Before submitting a change:
+
+```bash
+pytest
+python -m compileall rekos
+git diff --check
+```
+
+## Roadmap
+
+- More passive source adapters with explicit safety boundaries
+- Stronger report templates and case export validation
+- Improved graph summaries and finding explainability
+- Better import/export interoperability
+- Optional UI views while keeping the CLI and local-first storage as the core
+
+## License
+
+MIT License. See [LICENSE](LICENSE).

rekos-1.3.0/pyproject.toml

@@ -0,0 +1,56 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rekos"
+dynamic = ["version"]
+description = "Terminal-native passive OSINT CLI for local-first public-source investigation workspaces"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [
+    {name = "VladTepes84"},
+]
+keywords = ["osint", "passive-osint", "cli", "sqlite", "investigation", "public-source"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Information Technology",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Internet",
+    "Topic :: Security",
+    "Topic :: Utilities",
+    "Typing :: Typed",
+]
+dependencies = [
+    "rich>=13.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/VladTepes84/Rekos"
+Repository = "https://github.com/VladTepes84/Rekos"
+Issues = "https://github.com/VladTepes84/Rekos/issues"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+]
+
+[project.scripts]
+rekos = "rekos.cli:console_main"
+
+[tool.setuptools.packages.find]
+include = ["rekos*"]
+
+[tool.setuptools.package-data]
+rekos = ["py.typed", "adapters/*.json"]
+
+[tool.setuptools.dynamic]
+version = {attr = "rekos.__version__"}

rekos-1.3.0/rekos/__init__.py

@@ -0,0 +1,5 @@
+"""REKOS core package."""
+
+__all__ = ["__version__"]
+
+__version__ = "1.3.0"

rekos-1.3.0/rekos/adapters/__init__.py

@@ -0,0 +1,18 @@
+"""Passive OSINT source adapters."""
+
+from .base import AdapterResult, BaseSourceAdapter, SourceRunResult
+from .maigret import MaigretAdapter
+from .sherlock import SherlockAdapter, SherlockUsernameAdapter
+from .web_osint import DnsDomainAdapter
+from .wmn import WmnUsernameAdapter
+
+__all__ = [
+    "AdapterResult",
+    "BaseSourceAdapter",
+    "DnsDomainAdapter",
+    "MaigretAdapter",
+    "SherlockAdapter",
+    "SherlockUsernameAdapter",
+    "SourceRunResult",
+    "WmnUsernameAdapter",
+]

rekos-1.3.0/rekos/adapters/base.py

@@ -0,0 +1,104 @@
+"""Base interface for passive OSINT source adapters."""
+
+from __future__ import annotations
+
+import shutil
+import re
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from rekos.storage import CaseStore
+
+
+@dataclass(frozen=True)
+class AdapterResult:
+    source: str
+    target: str
+    url: str
+    platform: str
+    confidence: str
+    raw_reference: str
+
+
+@dataclass(frozen=True)
+class SourceRunResult:
+    source: str
+    target: str
+    raw_output: str
+    results: list[AdapterResult]
+    artifacts: list[Path]
+    skipped: bool = False
+
+
+class BaseSourceAdapter:
+    name: str = ""
+    description: str = ""
+    supported_target_types: tuple[str, ...] = ()
+    passive_only: bool = True
+    external_dependencies: tuple[str, ...] = ()
+
+    def dependency_status(self) -> dict[str, bool]:
+        return {
+            dependency: shutil.which(dependency) is not None
+            for dependency in self.external_dependencies
+        }
+
+    def missing_dependencies(self) -> list[str]:
+        return [
+            dependency
+            for dependency, available in self.dependency_status().items()
+            if not available
+        ]
+
+    def execute(self, case: str, target: str, store: CaseStore) -> SourceRunResult:
+        missing = self.missing_dependencies()
+        if missing:
+            from rekos.errors import ExternalToolMissingError
+
+            raise ExternalToolMissingError(
+                f"Missing dependencies for {self.name}: {', '.join(missing)}."
+            )
+        raw_output = self.run(case, target)
+        artifact_path = self._write_source_output(case, target, store, raw_output)
+        results = self.parse_results(target, raw_output)
+        store.add_adapter_results(case, results)
+        store.add_timeline_event(case, "source.run", f"Ran source {self.name} for {target}")
+        return SourceRunResult(
+            source=self.name,
+            target=target,
+            raw_output=raw_output,
+            results=results,
+            artifacts=[artifact_path],
+        )
+
+    def run(self, case: str, target: str) -> str:
+        raise NotImplementedError
+
+    def parse_results(self, target: str, raw_output: str) -> list[AdapterResult]:
+        raise NotImplementedError
+
+    def _write_source_output(
+        self,
+        case: str,
+        target: str,
+        store: CaseStore,
+        raw_output: str,
+    ) -> Path:
+        sources_folder = store.exports_folder(case) / "sources"
+        sources_folder.mkdir(exist_ok=True)
+        stem = f"{int(time.time())}-{self.name}-{_safe_export_name(target)}"
+        path = sources_folder / f"{stem}.txt"
+        counter = 2
+        while path.exists():
+            path = sources_folder / f"{stem}-{counter}.txt"
+            counter += 1
+        path.write_text(raw_output, encoding="utf-8")
+        return path
+
+
+def _safe_export_name(value: str) -> str:
+    cleaned = re.sub(r"[^A-Za-z0-9_.-]+", "-", value.strip()).strip(".-")
+    return (cleaned or "target")[:80]