darknet-toolkit 1.1.0__tar.gz

darknet_toolkit-1.1.0/CHANGELOG.md

@@ -0,0 +1,222 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and
+this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+_Nothing yet._
+
+## [1.1.0] — 2026-11-08
+
+Second feature release: four new capability packages, a flat package-root
+namespace, new CLI command groups and the analysis/storage extras. Still
+zero runtime dependencies.
+
+### Added
+
+**`dark_net.crypto`** (55 exports)
+
+- Authenticated encryption: `encrypt`/`decrypt` (framed MAGIC+nonce+tag),
+  `encrypt_text`/`decrypt_text` (base64 armour), `encrypt_stream`/`decrypt_stream`
+  and `encrypt_file`/`decrypt_file` (per-chunk HMAC-derived nonces, EOF tag),
+  `generate_key`, `is_encrypted`.
+- Hashing: `digest`, `stream_digest`, `shake_digest`, `digest_algorithms`,
+  `adler32`/`crc32`/`crc16`.
+- MAC/KDF: `hmac_digest`, `keyed_hash`, `hkdf` (+ extract/expand),
+  `scrypt_derive`.
+- JWT: `encode_jwt`, `decode_jwt`, `decode_jwt_unverified`, `build_claims`.
+- Encodings: hex/base32/58/62/64/85, PEM, percent, rot13, UUID helpers.
+- CSPRNG: `random_bytes`, `random_int`, `random_token`, `random_choice`,
+  `secure_shuffle`.
+- Analysis: `shannon_entropy`, `byte_histogram`, `looks_random`.
+
+**`dark_net.storage`** (41 exports)
+
+- `SQLiteStore` (SAVEPOINT-nested `transaction()`, identifier policy,
+  `IntegrityError`/`DataError` mapping, `backup_to`), `open_db`, `db_info`,
+  `list_tables`, `table_exists`, `sql_explain`, `upsert`.
+- JSONL: read/write/append/merge/split/filter/sample/validate/`tail_jsonl`.
+- Query layer: `compile_filter`/`apply_filter` (quoted, type-aware grammar),
+  `select`, `top_n`, `inner_join`/`left_join`, `distinct_values`.
+- Migrations: `Migration`, `MigrationRunner`, `run_migrations` (per-migration
+  transaction + journal).
+- Tabular: sniff/clean/stats/convert/merge/transpose CSV/TSV helpers.
+
+**`dark_net.concurrent`** (25 exports)
+
+- `parallel_map`/`parallel_iter`/`parallel_starmap`/`process_map`,
+  `batched`/`window`/`round_robin`/`map_chunks`.
+- `async_map`, `gather_first`, `TokenBucket`, `SlidingWindowLimiter`,
+  `Throttle`, `BackgroundTask`, `run_in_background`, `WorkerPool`.
+- `AtomicCounter`, `Once`, `RWLock`, `ObjectPool`, `Deadline`, `SingleFlight`,
+  `CircuitBreaker`/`CircuitOpenError`, fluent `Pipeline`.
+
+**`dark_net.automation`** (33 exports)
+
+- Processes: `exec_command`, `start_process`, `ProcessHandle`/`ProcessResult`,
+  `terminate_process`, `is_pid_running` (zombie-aware), `wait_for_pid`,
+  `child_pids` — never `shell=True`.
+- Schedules: `parse_cron` (5-field, POSIX OR day rules), `CronSpec`,
+  `Schedule`, `run_schedule`.
+- Watching: `FileWatcher`, `tree_snapshot`/`diff_snapshots`,
+  `wait_for_change`/`wait_for_path`.
+- `FileLock` (O_EXCL + stale steal), clipboard and desktop-notification
+  best-effort helpers, `capture_output`/`shell_quote`/`shell_join`/
+  `command_exists`, `working_directory` and `env_overlay` contexts.
+
+**Flat namespace**
+
+- Every public name from every package is re-exported at the package root
+  (620 names): `dark_net.decrypt(...)` is `dark_net.crypto.decrypt(...)`.
+
+**CLI**
+
+- New command groups: `crypto` (keygen/digest/encrypt/decrypt/entropy),
+  `storage` (tables/tail/filter/explain/upsert),
+  `automation` (exec/exists/cron/notify) and `batch` (parallel per-line
+  execution, `{}` placeholder, exit 11 on any failure).
+
+**Docs & examples**
+
+- `docs/api/{crypto,storage,concurrent,automation}.md`, expanded
+  `docs/cli.md`, new examples.
+
+### Changed
+
+- Version bumped to **1.1.0**.
+
+
+## [1.0.0] — 2026-11-08
+
+First stable release. Eleven feature packages plus `core`/`utils` foundations,
+one hardened CLI, zero runtime dependencies.
+
+### Added
+
+**Package layout**
+
+- `dark_net.core` — `Config` (layered, immutable, validated), `CONFIG_FIELDS`
+  registry, the `DarkNetError` hierarchy (22 exception types, each with `code`,
+  `message`, `details`), `ExitCode`, constants and reusable validators.
+- `dark_net.system` — platform/CPU/memory/disk/hardware introspection (stdlib
+  only, `/proc` and `sysctl` fallbacks), process listing with a `psutil` fast
+  path, `EnvManager` with redaction, and `run_diagnostics()` (`dark-net doctor`).
+- `dark_net.network` — IP parsing/classification, CIDR helpers, DNS resolution,
+  URL parsing and SSRF validation, a hardened `HTTPClient`, interface
+  enumeration, TCP connectivity/latency probes, bounded port scanning and
+  opt-in public-IP lookup (`ipify`, `ifconfig.co`, `icanhazip`, `ipinfo`).
+- `dark_net.files` — atomic read/write helpers, JSON/YAML/CSV formats, hashing
+  and duplicate detection, integrity manifests, safe ZIP archive handling
+  (zip-slip protection) and consent-gated backup/verify/restore/prune.
+- `dark_net.security` — secret detection and redaction, password generation,
+  strength/entropy scoring and policy advice, token/passphrase/API-key
+  generation, HMAC signing, key derivation, checksums and configuration /
+  environment / file-permission audits.
+- `dark_net.text` — slugify and case conversion, statistics and readability
+  scores, word/character frequencies, extraction helpers (URLs, emails,
+  mentions, hashtags), encoding detection, base64 helpers and unicode utilities.
+- `dark_net.datetime_utils` — parsing, formatting, timezone conversion, date
+  arithmetic, business-day counts, ranges and human duration formatting.
+- `dark_net.developer` — retry policies (sync + async), TTL/LRU/disk caches,
+  structured logging (JSON formatter, redaction filter), profiling and timing
+  helpers, serialization, schema validation and async utilities.
+- `dark_net.web` — `JSONAPIClient` on top of `HTTPClient`, a stdlib-only
+  `AsyncHTTPClient`, response parsing (JSON/text/CSV), `extract_field` path
+  queries, pagination detection and URL manipulation helpers.
+- `dark_net.data` — mapping/record transforms: deep merge, flatten, pick/omit,
+  pluck, group/sort, dedupe, chunk, type coercion and table orientation.
+- `dark_net.integrations` — installed-package inventory, requirements
+  snapshots, `pip-audit` delegation and a PyPI "outdated" fallback.
+- `dark_net.utils` — paths (`safe_join`, `resolve_within`, secure temp files),
+  formatting (`format_bytes`, `format_table`, `pluralize`) and an allow-listed
+  subprocess runner.
+- `dark_net.cli` — the `dark-net` console script: `system`, `network`, `file`,
+  `security`, `text`, `datetime`, `config`, `doctor`, with `--json` output on
+  every command and semantic exit codes.
+- Package-level re-exports now cover the full public surface, including
+  `network.is_always_blocked`, `network.iter_hosts`, `network.blocked_target`,
+  `files.hash_stream`, `files.load_manifest`/`dump_manifest`,
+  `files.file_exists`, `files.iter_directory`, `files.touch_file`,
+  `files.safe_join_path`, `files.load_yaml`/`dump_yaml`,
+  `files.iter_csv_rows` and `files.prune_backups`.
+- `tests/unit/test_public_api.py`, a contract test asserting that every name in
+  each package's `__all__` resolves and is unique, so a dropped re-export fails
+  the suite instead of users' imports.
+- `py.typed` marker; strict `mypy` configuration; ruff lint/format settings;
+  pytest markers (`integration`, `security`, `slow`, `cli`); coverage gate at
+  90 %.
+
+**Documentation, examples and packaging**
+
+- `README.md` with install/CLI/library/security overviews; `docs/` with
+  `index`, `quickstart`, `cli`, `configuration`, `library`, `security`,
+  `architecture`, `faq`, `troubleshooting` and per-package API references.
+- Seven runnable examples (`examples/01_system_report.py` …
+  `07_cli_extras.py`) and developer scripts (`scripts/check.sh`,
+  `scripts/smoke_test.sh`, `scripts/build_dist.sh`).
+- `LICENSE` (MIT), `CHANGELOG.md`, `CONTRIBUTING.md`, `SECURITY.md`,
+  `CODE_OF_CONDUCT.md`, `MANIFEST.in`, `.gitignore` and a GitHub Actions
+  workflow running lint, types, tests with the 90 % coverage gate, and a
+  build/twine/venv-install verification.
+
+**Safety features**
+
+- Deny-by-default SSRF protection applied to every request *and* every redirect
+  hop; link-local and cloud metadata ranges stay blocked even when private
+  access is explicitly allowed.
+- Manual redirect handling that strips `Authorization`, `Cookie` and
+  `Proxy-Authorization` on cross-origin hops.
+- TLS verification on by default (TLS ≥ 1.2), disableable only by an explicit
+  argument.
+- Consent gates (`consent=True` / `--yes`) on recursive deletes, backups,
+  restores, pruning and port probing.
+- Secret redaction across CLI output, logs and environment dumps.
+- Bounded work everywhere: response sizes, archive expansion, traversal depth,
+  file counts, port counts, retry budgets and timeouts.
+
+### Fixed
+
+Defects found and corrected while writing the test suite:
+
+- `dark-net network public-ip` exited with a usage error unless
+  `DARK_NET_PUBLIC_IP_PROVIDERS` was set; it now defaults to the `ipify`
+  provider (`--provider` and the environment variable still override it).
+- `dark-net security redact` refused piped input ("provide text or --file");
+  it now reads stdin when nothing is passed and the stream is not a TTY, so
+  `echo 'password=…' | dark-net security redact` works.
+- An invalid configuration value (for example `DARK_NET_TIMEOUT=abc`) escaped as
+  a raw traceback with exit code 1; configuration is now loaded inside the CLI's
+  error boundary, producing `error: timeout must be a number, got 'abc'` and
+  exit code 2 (`USAGE_ERROR`), including for `--json`.
+- `dark-net security strength --stdin` was unusable because the positional
+  password argument was still required; `--stdin` now works on its own and
+  treats empty input as a usage error (exit 2).
+- `dark-net system` crashed with `AttributeError: 'Namespace' object has no
+  attribute 'hardware'` when no subcommand was supplied.
+- `security redact`/dependency findings printed the wrong severity for
+  informational entries, and duration output under `--json` returned an
+  unordered payload.
+- macOS uptime parsing in `dark_net.system.info` matched the `usec` component
+  of `kern.boottime` and rewound the value to "unknown" whenever the microsecond
+  field failed to parse.
+- `hash_file(..., algorithm="SHA-256")` and other dashed spellings were rejected
+  even though the validator advertised dash-insensitivity; aliases such as
+  `sha-256` and `sha3-256` now resolve to their `hashlib` names.
+- `Console.secret()` was missing from the CLI output helper used by redaction
+  commands.
+- IP/URL handling: `get_ip_info` keys now consistently report `is_private` /
+  `is_global`, and public-IP provider failures return a `PublicIPResult` with
+  `found=False` and the provider error instead of raising.
+
+### Security
+
+- No credentials are logged; `sanitize_url_for_logging` strips queries and
+  userinfo, and redaction covers header, token and password shapes.
+- Subprocesses use argv lists with `shell=False`, an executable allow-list and
+  explicit timeouts.
+
+[Unreleased]: https://github.com/dark-net/dark-net/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/dark-net/dark-net/releases/tag/v1.0.0

darknet_toolkit-1.1.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,101 @@
+# Contributor Code of Conduct
+
+## Our pledge
+
+We as members, contributors and maintainers pledge to make participation in the
+`dark-net` project a harassment-free experience for everyone, regardless of age,
+body size, visible or invisible disability, ethnicity, sex characteristics,
+gender identity and expression, level of experience, education, socio-economic
+status, nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive and healthy community.
+
+## Our standards
+
+Examples of behaviour that contributes to a positive environment:
+
+- Demonstrating empathy and kindness toward other people.
+- Being respectful of differing opinions, viewpoints and experiences.
+- Giving and gracefully accepting constructive feedback.
+- Accepting responsibility and apologising to those affected by our mistakes,
+  and learning from the experience.
+- Focusing on what is best for the community and the project, not just for us as
+  individuals.
+- Assuming good faith in technical disagreements, and arguing about the code
+  rather than the person.
+
+Examples of unacceptable behaviour:
+
+- The use of sexualised language or imagery, and sexual attention or advances of
+  any kind.
+- Trolling, insulting or derogatory comments, and personal or political attacks.
+- Public or private harassment.
+- Publishing others' private information, such as a physical or email address,
+  without their explicit permission.
+- Deliberately submitting code intended to harm users (backdoors, data
+  exfiltration, credential harvesting, telemetry without consent), or
+  pressuring maintainers to weaken the security guarantees documented in
+  `SECURITY.md`.
+- Other conduct which could reasonably be considered inappropriate in a
+  professional setting.
+
+## Enforcement responsibilities
+
+Maintainers are responsible for clarifying and enforcing these standards and
+will take appropriate and fair corrective action in response to any behaviour
+they deem inappropriate, threatening, offensive or harmful.
+
+Maintainers have the right and responsibility to remove, edit or reject
+comments, commits, code, wiki edits, issues and other contributions that are
+not aligned with this Code of Conduct, and will communicate reasons for
+moderation decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all project spaces (issues, pull requests,
+discussions, chat, code review) and also applies when an individual is
+officially representing the project in public spaces.
+
+## Reporting
+
+Report abusive, harassing or otherwise unacceptable behaviour to
+`conduct@dark-net.invalid`. All complaints will be reviewed and investigated
+promptly and fairly. Maintainers are obligated to respect the privacy and
+security of the reporter of any incident.
+
+Reports about security vulnerabilities should follow [SECURITY.md](SECURITY.md)
+instead.
+
+## Enforcement guidelines
+
+Maintainers will follow these community impact guidelines when determining the
+consequences of any action:
+
+**1. Correction** — *Impact: inappropriate language or other behaviour deemed
+unprofessional or unwelcome.* A private written warning from a maintainer,
+providing clarity around the nature of the violation and an explanation of why
+the behaviour was inappropriate. A public apology may be requested.
+
+**2. Warning** — *Impact: a violation through a single incident or series of
+actions.* A warning with consequences for continued behaviour. No interaction
+with the people involved for a specified period. Violating these terms may lead
+to a temporary or permanent ban.
+
+**3. Temporary ban** — *Impact: a serious violation of community standards,
+including sustained inappropriate behaviour.* A temporary ban from any sort of
+interaction or public communication with the community for a specified period.
+Violating these terms may lead to a permanent ban.
+
+**4. Permanent ban** — *Impact: demonstrating a pattern of violation of community
+standards, including sustained inappropriate behaviour, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.* A
+permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the
+[Contributor Covenant](https://www.contributor-covenant.org), version 2.1,
+available at
+<https://www.contributor-covenant.org/version/2/1/code_of_conduct.html>.

darknet_toolkit-1.1.0/CONTRIBUTING.md

@@ -0,0 +1,88 @@
+# Contributing to dark-net
+
+Thanks for helping out. This project favours **small, well-tested, dependency-free
+code** and a **security-first default posture**, so please keep those two
+constraints in mind for every change.
+
+## Getting started
+
+```bash
+git clone https://github.com/dark-net/dark-net.git
+cd dark-net
+python -m pip install -e ".[dev]"
+python -m pytest -q                 # full suite (offline)
+```
+
+Supported interpreters: CPython **3.10 – 3.13** on Linux, macOS and Windows.
+
+## Before you open a pull request
+
+Run the same checks CI runs:
+
+```bash
+python -m ruff check .              # lint (must be clean)
+python -m ruff format --check .     # formatting (run `ruff format .` to fix)
+python -m mypy src/                 # strict typing, no ignores without a comment
+python -m pytest -q                 # all tests
+python -m pytest -q --cov=dark_net  # coverage must stay >= 90 %
+```
+
+Or just run the script that does all of it: `./scripts/check.sh --cov`.
+
+## Coding conventions
+
+- **Standard library only** in `src/dark_net`. Optional third-party packages
+  (`psutil`, `PyYAML`, `httpx`, `pip-audit`) must be imported lazily inside the
+  function that needs them and raise `DependencyMissingError` with an install
+  hint when absent.
+- **Public API is re-exported** from the package `__init__` and listed in
+  `__all__`. Add new names there, plus a docstring example where useful.
+- **Raise `DarkNetError` subclasses**, never bare `Exception`. Include a
+  machine-readable `details` mapping; never put secrets in `details`.
+- **Return typed results**, not tuples of loose values: prefer a frozen
+  dataclass with `to_dict()`.
+- **Bound the work.** Any function that walks a tree, accepts a body, loops over
+  ports or retries must accept an explicit limit with a safe default.
+- **Never add a destructive default.** Anything that deletes, overwrites or
+  restores data takes `consent=True` (or `--yes`) and raises
+  `ConsentRequiredError` otherwise.
+- **Type annotations everywhere** — `mypy --strict` is enforced, including tests
+  that define helpers.
+- Keep lines ≤ 120 characters, use `from __future__ import annotations`, and
+  prefer `pathlib` over string paths.
+
+## Tests
+
+- Put unit tests in `tests/unit`, integration tests (loopback HTTP, filesystem,
+  environment) in `tests/integration`, security-focused tests in
+  `tests/security`, and CLI subprocess tests in `tests/cli`.
+- Tests must not require network access. Use the `http_server` /
+  `local_client` fixtures from `tests/conftest.py` for HTTP.
+- Monkeypatch rather than mock when possible; assert on observable behaviour
+  (return values, exceptions, exit codes, files) rather than call counts.
+- Mark slow tests with `@pytest.mark.slow` and network/filesystem tests with
+  `@pytest.mark.integration`.
+- Coverage is measured with `branch = true`; new code should be covered by
+  behaviour tests, not just executed.
+
+## Commits and pull requests
+
+- One logical change per pull request; keep the diff reviewable.
+- Write imperative commit subjects ("add IPv6 classification helper").
+- Describe **what changed, why, and how it was verified** in the PR body.
+- Update `CHANGELOG.md` under `[Unreleased]` for user-visible changes.
+- Update the README and `docs/` when the public API or CLI changes.
+
+## Security issues
+
+Do **not** open a public issue for a vulnerability. Follow
+[SECURITY.md](SECURITY.md).
+
+## Code of conduct
+
+Participation is governed by [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).
+
+## License
+
+By contributing you agree that your contribution is licensed under the MIT
+License (see [LICENSE](LICENSE)).

darknet_toolkit-1.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 dark-net contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

darknet_toolkit-1.1.0/MANIFEST.in

@@ -0,0 +1,16 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include CONTRIBUTING.md
+include SECURITY.md
+include CODE_OF_CONDUCT.md
+include pyproject.toml
+include Makefile
+
+recursive-include src/dark_net py.typed
+recursive-include tests *.py
+recursive-include docs *.md *.py
+recursive-include examples *.py *.md
+recursive-include scripts *.py *.sh
+
+global-exclude __pycache__/* *.py[cod] *.egg-info .DS_Store .mypy_cache/* .pytest_cache/* .ruff_cache/* .coverage