osintkit 0.1.1 → 0.1.2

package/README.md

@@ -1,86 +1,117 @@
 # osintkit
 
-OSINT CLI tool for personal digital footprint analysis.
+OSINT CLI for personal digital footprint analysis. Input an email, phone, username, or name — get a risk-scored report saved locally as JSON, HTML, and Markdown.
+
+MIT licensed. No server. Everything stays on your machine.
 
 ## Installation
 
 ```bash
-# From source
-cd osintkit-oss
-npm install -g .
-
-# Or use directly without installation
-PYTHONPATH=. python3 -m osintkit.cli <command>
+npm install -g osintkit
 ```
 
-## Quick Start
-
-```bash
-# First-time setup (configure API keys)
-osintkit setup
-
-# Create a new profile
-osintkit new
-
-# List all profiles
-osintkit list
+This automatically installs all Python dependencies (core + optional OSINT tools) via the postinstall script. Just run `osintkit new` when it's done.
 
-# Run a scan
-osintkit refresh <profile_id>
+**Requirements:** Python 3.10+, Node.js 16+
 
-# View profile details
-osintkit open <profile_id>
+## Quick Start
 
-# Export data
-osintkit export <profile_id>
+```bash
+osintkit setup      # Configure API keys (optional, all have free tiers)
+osintkit new        # Create a profile and run a scan
+osintkit list       # View all profiles
+osintkit refresh    # Re-run scan on a profile
+osintkit open       # View profile details + open latest report
 ```
 
 ## Commands
 
 | Command | Description |
 |---------|-------------|
-| `osintkit new` | Create new profile |
-| `osintkit list` | List all profiles |
-| `osintkit refresh <id>` | Run scan for profile |
-| `osintkit open <id>` | Show profile details |
-| `osintkit export <id>` | Export as JSON/Markdown |
+| `osintkit new` | Create a new profile and optionally run a scan |
+| `osintkit list` | List all profiles with last risk score |
+| `osintkit refresh [id]` | Re-run scan for a profile |
+| `osintkit open [id]` | Show profile details and open latest report |
+| `osintkit export [id]` | Export as JSON or Markdown |
 | `osintkit setup` | Configure API keys |
-| `osintkit delete <id>` | Delete profile |
+| `osintkit delete [id]` | Delete a profile |
 | `osintkit version` | Show version |
 
-## Features
+## What It Checks
+
+### Stage 1 — No API keys needed, works out of the box
 
-- **Profile Management** - Store and manage multiple target profiles
-- **Duplicate Detection** - Warns when creating profiles with existing info
-- **OSINT Modules** - 10 integrated OSINT data sources
-- **Risk Scoring** - 0-100 risk score based on findings
-- **Export** - JSON and Markdown report formats
+| Module | Input | What it does |
+|--------|-------|-------------|
+| Maigret | username | 3000+ sites |
+| Sherlock | username | 400+ sites |
+| Holehe | email | 120+ platform registrations |
+| HIBP k-anonymity | email | Password breach check (no key) |
+| Gravatar | email | Profile existence + avatar |
+| theHarvester | email/domain | Web presence, subdomains |
+| crt.sh | email/domain | Certificate transparency |
+| Wayback CDX | email | Historical web appearances |
+| libphonenumber | phone | Carrier, region, line type (offline) |
+| Paste search | email | Paste site appearances |
+| Data brokers | name/email | Google CSE broker scan |
+| Dark web | email | Ahmia / public index |
+| Breach lookup | email | BreachDirectory |
 
-## API Keys (Optional)
+### Stage 2 — Optional free API keys, runs first when configured
 
-Free API keys available for enhanced functionality:
+| Service | Input | Free Tier |
+|---------|-------|-----------|
+| HaveIBeenPwned | email | Free w/ key |
+| LeakCheck | email/phone/user | Free tier |
+| NumVerify | phone | 100/month |
+| Hunter.io | name + domain | 50/month |
+| GitHub API | username | 5000/hr w/ key |
+| SecurityTrails | domain | Free tier |
 
-| Service | Free Limit | Purpose |
-|---------|------------|---------|
-| Have I Been Pwned | 10/min | Breach database |
-| NumVerify | 100/month | Phone validation |
-| Intelbase | 100/month | Dark web + paste |
-| BreachDirectory | 50/day | Breach lookups |
-| Google CSE | 100/day | Data broker detection |
+Stage 2 always runs first when a key is configured. If rate-limited or key missing, falls back to Stage 1 automatically.
 
-## Requirements
+## Output
 
-- Python 3.11+
-- Node.js 12+
-- pip
+Each scan creates a folder at `~/osint-results/<target>_<date>/` containing:
+- `report.html` — rendered report with risk score
+- `findings.json` — full structured data
+- `findings.md` — markdown summary
 
-## External Tools (Optional)
+Risk score is 0–100 based on breach exposure, social footprint, data broker listings, and dark web/paste appearances.
+
+## API Keys (All Optional)
+
+Run `osintkit setup` to configure. All keys are optional — the tool works without any of them.
+
+```
+~/.osintkit/config.yaml
+```
+
+| Service | Get key at |
+|---------|-----------|
+| HaveIBeenPwned | haveibeenpwned.com/API/Key |
+| LeakCheck | leakcheck.io |
+| NumVerify | numverify.com |
+| Hunter.io | hunter.io |
+| GitHub | github.com/settings/tokens |
+| Intelbase | intelbase.is |
+| BreachDirectory | rapidapi.com |
+| Google CSE | developers.google.com/custom-search |
+| SecurityTrails | securitytrails.com |
+
+## Run from Source
 
-For full functionality, install:
 ```bash
-pip install maigret holehe theHarvester
+git clone https://github.com/diesesschnitzel/osintkit.git
+cd osintkit
+pip install -r requirements.txt -r requirements-tools.txt
+PYTHONPATH=. python3 -m osintkit.cli new
 ```
 
+## Ethics
+
+Only use osintkit on targets you have explicit permission to investigate. GDPR applies to EU subjects. A disclaimer is shown before every scan.
+
 ## License
 
-MIT
+MIT

package/bin/osintkit.js

@@ -3,10 +3,24 @@ const { spawn } = require('child_process');
 const path = require('path');
 const fs = require('fs');
 
-// Prefer python3.11, fall back to python3, then python
+// The npm package root is one level up from bin/
+const packageDir = path.dirname(path.dirname(__filename));
+
+// Prefer venv Python (installed by postinstall), fall back to system Python
 function findPython() {
-    const candidates = ['python3.11', 'python3', 'python'];
-    for (const bin of candidates) {
+    const venvCandidates = [
+        path.join(packageDir, '.venv', 'bin', 'python3'),
+        path.join(packageDir, '.venv', 'bin', 'python'),
+        path.join(packageDir, 'venv', 'bin', 'python3'),
+        path.join(packageDir, 'venv', 'bin', 'python'),
+    ];
+    for (const p of venvCandidates) {
+        if (fs.existsSync(p)) return p;
+    }
+
+    // Fall back to system Python
+    const systemCandidates = ['python3.11', 'python3', 'python'];
+    for (const bin of systemCandidates) {
         try {
             require('child_process').execSync(`${bin} --version`, { stdio: 'ignore' });
             return bin;
@@ -15,14 +29,25 @@ function findPython() {
     return 'python3';
 }
 
-const python = spawn(findPython(), ['-m', 'osintkit', ...process.argv.slice(2)], {
+const pythonBin = findPython();
+
+// Always set PYTHONPATH so the package is importable regardless of install method
+const env = {
+    ...process.env,
+    PYTHONPATH: process.env.PYTHONPATH
+        ? `${packageDir}:${process.env.PYTHONPATH}`
+        : packageDir,
+};
+
+const child = spawn(pythonBin, ['-m', 'osintkit', ...process.argv.slice(2)], {
     stdio: 'inherit',
-    env: process.env
+    cwd: packageDir,
+    env,
 });
 
-python.on('error', () => {
+child.on('error', () => {
     console.error('Error: Python 3.10+ required. Install from https://python.org');
     process.exit(1);
 });
 
-python.on('close', (code) => process.exit(code || 0));
+child.on('close', (code) => process.exit(code || 0));

package/osintkit/__init__.py

@@ -1,3 +1,3 @@
 """osintkit - OSINT CLI tool for personal digital footprint analysis."""
 
-__version__ = "0.1.0"
+__version__ = "0.1.2"

package/osintkit/cli.py

@@ -3,6 +3,7 @@
 import sys
 import json
 import logging
+import threading
 from pathlib import Path
 from typing import Optional
 from datetime import datetime
@@ -14,14 +15,67 @@ from rich.table import Table
 from rich.prompt import Prompt, Confirm
 from rich.progress import Progress, SpinnerColumn, TextColumn
 
+from osintkit import __version__
 from osintkit.scanner import Scanner
 from osintkit.config import load_config, Config, APIKeys
 from osintkit.profiles import Profile, ProfileStore, ScanHistory
 
-app = typer.Typer(help="OSINT CLI for personal digital footprint analysis")
+app = typer.Typer(help="OSINT CLI for personal digital footprint analysis", invoke_without_command=True)
 console = Console()
 store = ProfileStore()
 logger = logging.getLogger(__name__)
+_update_thread = None
+
+
+@app.callback()
+def _startup(ctx: typer.Context):
+    """Start background version check on every invocation."""
+    global _update_thread
+    _update_thread = _start_update_check()
+
+# ---- Version update check ----
+
+_update_available: Optional[str] = None  # Set to newer version string if one exists
+
+
+def _check_for_update_bg():
+    """Check npm registry for a newer version in a background thread (non-blocking)."""
+    global _update_available
+    try:
+        import httpx
+        resp = httpx.get(
+            "https://registry.npmjs.org/osintkit/latest",
+            timeout=3.0,
+            headers={"Accept": "application/json"},
+        )
+        if resp.status_code == 200:
+            latest = resp.json().get("version", "")
+            if latest and latest != __version__:
+                from packaging.version import Version
+                if Version(latest) > Version(__version__):
+                    _update_available = latest
+    except Exception:
+        pass  # Never crash the app over a version check
+
+
+def _start_update_check():
+    """Start the background version check thread."""
+    t = threading.Thread(target=_check_for_update_bg, daemon=True)
+    t.start()
+    return t
+
+
+def _print_update_notice():
+    """Print update notice if a newer version was found."""
+    if _update_available:
+        console.print(Panel(
+            f"[bold cyan]osintkit {_update_available}[/bold cyan] is available "
+            f"[dim](you have {__version__})[/dim]\n"
+            "[dim]Run:[/dim] [bold]npm install -g osintkit[/bold]",
+            title="[bold yellow]⬆  Update available[/bold yellow]",
+            border_style="yellow",
+            padding=(0, 2),
+        ))
 
 
 def _print_ethics_banner():
@@ -123,6 +177,7 @@ api_keys:
   epieos: ""
 """
         config_path.write_text(config_content)
+        config_path.chmod(0o600)  # API keys must not be world-readable
         console.print(f"\n[green]✓[/green] Config saved to {config_path}")
         return True
     
@@ -288,6 +343,7 @@ api_keys:
   epieos: ""
 """
     config_path.write_text(config_content)
+    config_path.chmod(0o600)  # API keys must not be world-readable
     console.print(f"\n[green]✓[/green] Config saved: {config_path}")
 
 
@@ -402,6 +458,9 @@ def list():
     
     console.print(table)
     console.print()
+    if _update_thread:
+        _update_thread.join(timeout=4)
+    _print_update_notice()
 
 
 @app.command()
@@ -440,6 +499,7 @@ def refresh(profile_ref: str = typer.Argument(None, help="Profile ID or name")):
     )
     store.add_scan_result(profile.id, scan_record)
     console.print(f"\n[green]✓[/green] Scan saved to profile history")
+    _print_update_notice()
 
 
 @app.command()
@@ -605,8 +665,10 @@ def delete(profile_ref: str = typer.Argument(None, help="Profile ID or name")):
 @app.command()
 def version():
     """Show version."""
-    from osintkit import __version__
+    if _update_thread:
+        _update_thread.join(timeout=4)
     console.print(f"osintkit v{__version__}")
+    _print_update_notice()
 
 
 if __name__ == "__main__":

package/osintkit/modules/hibp_kanon.py

@@ -1,4 +1,17 @@
-"""HIBP k-anonymity password exposure check using SHA1 prefix API."""
+"""HIBP k-anonymity password exposure check using SHA1 prefix API.
+
+Mock format for tests
+---------------------
+The API returns plain text, one entry per line: ``HASHSUFFIX:COUNT``
+The suffix is uppercase hex (35 chars = SHA1 40 minus 5-char prefix).
+
+Example mock response.text for email "test@example.com":
+    sha1 = hashlib.sha1(b"test@example.com").hexdigest().upper()
+    prefix, suffix = sha1[:5], sha1[5:]
+    mock_text = f"{suffix}:42\\nDEADBEEFCAFE00000000000000000000000:1\\n"
+
+The module checks each line for a matching suffix and reads the count.
+"""
 
 import hashlib
 import logging

package/osintkit/modules/wayback.py

@@ -1,4 +1,22 @@
-"""Wayback Machine CDX API lookup for email domain and username."""
+"""Wayback Machine CDX API lookup for email domain and username.
+
+Mock format for tests
+---------------------
+The CDX API returns JSON: a list of lists where the FIRST row is a header
+and subsequent rows are data. Each row matches the ``fl`` fields requested.
+This module requests ``fl=original,timestamp``, so:
+
+    [
+        ["original", "timestamp"],                          # header row
+        ["http://example.com/page", "20210315120000"],      # data row
+        ["http://example.com/other", "20200101000000"],
+    ]
+
+An empty result (or only the header row) is treated as no findings::
+
+    []          # API returned nothing
+    [["original", "timestamp"]]   # header only — no data
+"""
 
 from typing import Any, Dict, List
 

package/osintkit/profiles.py

@@ -54,6 +54,21 @@ class ProfileStore:
         with open(self.store_path, "w") as f:
             json.dump(profiles, f, indent=2, default=str)
     
+    def find_duplicate(self, profile: Profile) -> Optional[Profile]:
+        """Return first existing profile that shares email, username, or phone with the given profile.
+
+        Used before create() to warn the user about potential duplicates.
+        """
+        profiles = self._load()
+        for p in profiles.values():
+            if profile.email and p.get("email") and profile.email.lower() == p["email"].lower():
+                return Profile(**p)
+            if profile.username and p.get("username") and profile.username.lower() == p["username"].lower():
+                return Profile(**p)
+            if profile.phone and p.get("phone") and profile.phone == p["phone"]:
+                return Profile(**p)
+        return None
+
     def create(self, profile: Profile) -> Profile:
         """Create a new profile."""
         profiles = self._load()

package/osintkit/risk.py

@@ -32,11 +32,20 @@ def calculate_risk_score(findings: Dict[str, List]) -> int:
     dark_count = len(findings.get("dark_web", [])) + len(findings.get("paste_sites", []))
     score += min(15, dark_count * 5)
     
-    # Password exposure - typically 0 since we don't have passwords
-    # If data contains count, scale it
-    password_data = findings.get("password_exposure", [])
-    if password_data and isinstance(password_data[0], dict) and password_data[0].get("data", {}).get("count"):
-        pw_count = password_data[0]["data"]["count"]
-        score += min(15, pw_count // 1000)
-    
+    # Password / hash exposure (15 points max)
+    # Covers both: HIBP full API ("password_exposure") and k-anonymity check ("hibp_kanon")
+    pw_score = 0
+    for key in ("password_exposure", "hibp_kanon"):
+        pw_data = findings.get(key, [])
+        if pw_data and isinstance(pw_data[0], dict):
+            count = pw_data[0].get("data", {}).get("count", 0)
+            if count:
+                pw_score = max(pw_score, min(15, count // 1000))
+    # Even a single confirmed exposure without a count is worth some points
+    if pw_score == 0:
+        all_pw = findings.get("password_exposure", []) + findings.get("hibp_kanon", [])
+        if all_pw:
+            pw_score = 5
+    score += pw_score
+
     return min(100, score)

package/osintkit/setup.py

@@ -121,11 +121,13 @@ def run_setup_wizard():
     config_path = config_dir / "config.yaml"
     with open(config_path, "w") as f:
         yaml.dump(config, f, default_flow_style=False)
-    
+    config_path.chmod(0o600)  # owner read/write only — API keys must not be world-readable
+
     # Create profiles file
     profiles_path = config_dir / "profiles.json"
     if not profiles_path.exists():
         profiles_path.write_text("{}")
+    profiles_path.chmod(0o600)
     
     console.print(f"\n[green]✓[/green] Config saved to: {config_path}")
     console.print("[green]✓[/green] Ready to use!")

package/package.json

@@ -1,19 +1,21 @@
 {
   "name": "osintkit",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "description": "OSINT CLI for personal digital footprint analysis",
   "bin": {
     "osintkit": "./bin/osintkit.js"
   },
   "scripts": {
-    "postinstall": "pip3 install -e . --quiet || true"
+    "postinstall": "node postinstall.js"
   },
   "files": [
     "bin/",
     "osintkit/",
     "pyproject.toml",
     "requirements.txt",
-    "README.md"
+    "requirements-tools.txt",
+    "README.md",
+    "postinstall.js"
   ],
   "keywords": ["osint", "security", "privacy", "cli", "recon"],
   "author": "",

package/postinstall.js

@@ -0,0 +1,75 @@
+#!/usr/bin/env node
+/**
+ * osintkit postinstall — installs all Python dependencies automatically.
+ *
+ * Runs after `npm install -g osintkit`.
+ * Installs:
+ *   1. Core Python deps (typer, rich, httpx, pydantic, etc.) from requirements.txt
+ *   2. Optional OSINT tools (maigret, holehe, sherlock) from requirements-tools.txt
+ *
+ * Uses --break-system-packages on Linux/macOS where needed (Python 3.11+).
+ * Falls back gracefully if pip isn't available.
+ */
+
+const { execSync } = require('child_process');
+const path = require('path');
+const fs = require('fs');
+
+const packageDir = __dirname;
+const req = path.join(packageDir, 'requirements.txt');
+const reqTools = path.join(packageDir, 'requirements-tools.txt');
+
+function findPip() {
+    for (const bin of ['pip3', 'pip']) {
+        try {
+            execSync(`${bin} --version`, { stdio: 'ignore' });
+            return bin;
+        } catch (_) {}
+    }
+    return null;
+}
+
+function pipInstall(pip, requirementsFile, label) {
+    if (!fs.existsSync(requirementsFile)) {
+        console.log(`  ⚠️  ${label}: requirements file not found, skipping`);
+        return;
+    }
+
+    const flags = ['-r', requirementsFile, '-q', '--disable-pip-version-check'];
+
+    // Try with --break-system-packages first (needed on modern Linux/macOS)
+    try {
+        execSync(`${pip} install ${flags.join(' ')} --break-system-packages`, {
+            stdio: 'pipe',
+            cwd: packageDir,
+        });
+        console.log(`  ✅  ${label} installed`);
+        return;
+    } catch (_) {}
+
+    // Fall back without the flag (works on older systems / virtual envs)
+    try {
+        execSync(`${pip} install ${flags.join(' ')}`, {
+            stdio: 'pipe',
+            cwd: packageDir,
+        });
+        console.log(`  ✅  ${label} installed`);
+        return;
+    } catch (err) {
+        console.log(`  ⚠️  ${label}: could not install (${err.message.split('\n')[0]})`);
+    }
+}
+
+console.log('\nosintkit: installing Python dependencies...\n');
+
+const pip = findPip();
+if (!pip) {
+    console.log('  ⚠️  pip not found — skipping Python dependency install.');
+    console.log('  Run manually: pip3 install -r requirements.txt -r requirements-tools.txt');
+    process.exit(0);
+}
+
+pipInstall(pip, req, 'Core dependencies');
+pipInstall(pip, reqTools, 'OSINT tools (maigret, holehe, sherlock)');
+
+console.log('\nosintkit ready. Run: osintkit new\n');

package/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "osintkit"
-version = "0.1.1"
+version = "0.1.2"
 description = "OSINT CLI tool for personal digital footprint analysis"
 authors = ["Your Name <you@example.com>"]
 license = "MIT"
@@ -17,6 +17,7 @@ pyyaml = "^6.0.1"
 pydantic = "^2.5.0"
 aiofiles = "^23.2.1"
 phonenumbers = ">=8.13"
+packaging = ">=23.0"
 questionary = ">=2.0"
 
 [tool.poetry.group.dev.dependencies]

package/requirements-tools.txt

@@ -0,0 +1,12 @@
+# Optional subprocess-based OSINT tools used by osintkit modules.
+# Install these to enable social profile enumeration, email account checks,
+# and username search:
+#
+#   pip install -r requirements-tools.txt
+#
+# They are not listed as poetry dependencies because they are standalone CLI
+# tools invoked via subprocess rather than imported as Python libraries.
+
+maigret>=0.4.4
+holehe>=1.1.9
+sherlock-project>=0.14.3

package/requirements.txt

@@ -1,4 +1,5 @@
 typer[all]>=0.9
+packaging>=23.0
 rich>=13.7
 httpx>=0.27
 pyyaml>=6.0