deepsecrets 2.1.1__tar.gz → 2.3.0__tar.gz

deepsecrets-2.3.0/LICENSE

@@ -0,0 +1,235 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+--------------------------------------------------------------------------------
+
+PORTIONS UNDER THE MIT LICENSE
+
+DeepSecrets was first released at https://github.com/avito-tech/deepsecrets
+under the MIT License below. Code from that release, and contributions made
+to the project under the MIT License before version 2.3.0, remain available
+under it; the rest of DeepSecrets is licensed under the Apache License,
+Version 2.0, above. See NOTICE.
+
+MIT License
+
+Copyright (c) 2023 Avito
+Copyright (c) 2023-2026 Nikolai Khechumov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

deepsecrets-2.3.0/NOTICE

@@ -0,0 +1,10 @@
+DeepSecrets
+Copyright 2023-2026 Nikolai Khechumov
+
+Licensed under the Apache License, Version 2.0, from version 2.3.0; see LICENSE.
+
+This product includes software written by Nikolai Khechumov at Avito and first
+released at https://github.com/avito-tech/deepsecrets, Copyright (c) 2023 Avito,
+and contributions made to the project before version 2.3.0. Those portions are
+licensed under the MIT License, whose text is at the end of LICENSE. Releases up
+to and including 2.2.0 were published under the MIT License alone.

deepsecrets-2.3.0/PKG-INFO

@@ -0,0 +1,338 @@
+Metadata-Version: 2.4
+Name: deepsecrets
+Version: 2.3.0
+Summary: Offline secrets scanner that understands code: credentials found by variable names, entropy and provider formats, with confidence tiers, tags and SARIF output
+License-Expression: Apache-2.0 AND MIT
+License-File: LICENSE
+License-File: NOTICE
+Keywords: secrets,secret-scanning,secrets-detection,credentials,api-keys,passwords,leak-detection,security,appsec,devsecops,sast,static-analysis,sarif,owasp
+Author: Nikolai Khechumov
+Author-email: khechumov@gmail.com
+Maintainer: Nikolai Khechumov
+Maintainer-email: khechumov@gmail.com
+Requires-Python: >=3.11,<4.0.0
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Security
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Dist: aenum (==3.1.17)
+Requires-Dist: jschema-to-python (==1.2.3)
+Requires-Dist: mmh3 (==5.3.0)
+Requires-Dist: ordered-set (==4.1.0)
+Requires-Dist: puppetparser (==0.2.14)
+Requires-Dist: pydantic (==2.13.5)
+Requires-Dist: pygments (==2.21.0)
+Requires-Dist: pyyaml (==6.0.3)
+Requires-Dist: regex (==2026.9.3)
+Requires-Dist: rich (==15.0.0)
+Requires-Dist: sarif-om (==1.0.4)
+Project-URL: Changelog, https://github.com/ntoskernel/deepsecrets/releases
+Project-URL: Documentation, https://github.com/ntoskernel/deepsecrets#readme
+Project-URL: Docker image, https://hub.docker.com/r/ntoskernel/deepsecrets
+Project-URL: Homepage, https://owasp.org/projects/deepsecrets
+Project-URL: Issues, https://github.com/ntoskernel/deepsecrets/issues
+Project-URL: Source, https://github.com/ntoskernel/deepsecrets
+Description-Content-Type: text/markdown
+
+# [OWASP](https://owasp.org/projects/deepsecrets) DeepSecrets — a better tool for secrets scanning
+
+
+![Tests Status](https://github.com/ntoskernel/deepsecrets/actions/workflows/run-tests.yml/badge.svg)
+
+## What is it? Another token-wasting CLI proxy to an AI API?
+
+Absolutely not!
+
+In our LLM-hype era, DeepSecrets still runs entirely on your machine — giving you great results offline, securely, and for free.
+
+
+## So why yet another tool?
+Most existing scanners don't actually "understand" code. Instead, they just parse texts and have bad coverage.
+
+DeepSecrets bridges the gap between classic regex scanners and full-scale commercial SAST tools. It extends the classic regex-based scanning strategy by heavily relying on semantic code analysis, dangerous variable detection, and context-aware entropy analysis.
+Secret candidates come from what the code says, its names, assignments and calls, not from text alone. DeepSecrets reads code with lexers for about 600 languages and formats (Pygments), and finds variables with detection rules for 14 languages and a generic rule for the rest.
+
+DeepSecrets also introduces a new way to find credentials with zero knowledge: the HashedSecret Engine. Just provide the hashed values of your known production secrets, and the tool will find them exposed in plain text within your code.
+
+### Performance & Benchmarks (SecretBench)
+
+DeepSecrets 2.3 was evaluated (November 2026) on the **SecretBench** benchmark: every file scanned, every labelled row counted.
+
+* **93% Recall** and **91% Precision** (2.2: 93% and 86%)
+* **1.6% False Positive Rate** on SecretBench scope (2.2: 2.7%)
+* About **2,000 more secrets** that SecretBench does not label, among its findings outside the labels
+
+Counted the way the June 2026 evaluation counted, on its own set of rows, 2.3 has 94% recall and 85% precision (2.0 in June 2026: 93% and 67%).
+
+#### Reproduce these numbers
+
+The figures come from [dsbench](https://github.com/ntoskernel/dsbench), an open-source benchmark harness that runs a scanner as a black box over a labelled corpus and counts what it reports. With SecretBench (available from [its authors](https://github.com/setu1421/SecretBench) on request) and dsbench, anyone can reproduce the per-row figures above and the comparison below:
+
+```bash
+dsbench prepare                        # load SecretBench's labels and the corpus listing
+dsbench run --tool-ref v2.3.0          # DeepSecrets 2.3.0 from its tag, in a venv from its own poetry.lock
+dsbench tools fetch                    # the pinned gitleaks and Betterleaks releases, checked by SHA-256
+dsbench run --tool gitleaks
+dsbench run --tool betterleaks
+dsbench compare RUN RUN ...            # side by side, with 95% intervals
+```
+
+Every run scans every file, with each tool's path exclusions turned off. The June-way figures also need the June 2026 evaluation's table of rows, and the count of secrets beyond the labels needs our judged verdicts; neither is published, because both point at leaked credentials.
+
+*(You can read the full under-the-hood story and benchmark breakdown in my HackerNoon article [here](https://hackernoon.com/deepsecrets-20-catching-93percent-of-secretbenchs-valids-while-filtering-92percent-of-noise-and-10k-extra))*
+
+
+# Quick Start Guide
+
+## Installation
+
+From Github via pip
+
+```bash
+$ pip install git+https://github.com/ntoskernel/deepsecrets.git
+```
+
+From PyPi
+
+```bash
+$ pip install deepsecrets
+```
+
+DeepSecrets needs Python 3.11 or later.
+
+
+## Scanning
+The easiest way to run a scan:
+
+```bash
+$ deepsecrets --target-dir /path/to/your/code --outformat dojo-sarif --outfile report.json
+```
+
+This will run a scan against `/path/to/your/code` using the default configuration:
+- Regex using the built-in ruleset
+- Semantic checks (variable detection, entropy checks)
+
+A report in SARIF format (compatible with DefectDojo and GitHub Security) will be saved to report.json.
+
+### Fine-Tuning
+The `--help` command is always ready to guide you, but here are the key flags you can use to tailor the scan to your environment:
+* `--regex-rules /path/to/rules.json`: Supply your own custom regex ruleset.
+* `--hashed-values /path/to/hashes.json`: Provide a list of pre-hashed known production secrets to search for them securely.
+* `--excluded-paths /path/to/exclusions.json`: Override or extend the default paths ignored during scanning.
+* `--deep-max-size BYTES`: Files larger than this (default 250,000 bytes) get the fast analysis: the regex rules and the variable search, without the language lexer. Big files are mostly generated or minified, the lexer costs most of the scan time on them and rarely finds more there. `0` gives every file the full analysis.
+* `--skip-bundles`: Skip minified JavaScript, source maps and bundles (`*.min.js`, `*.map`, `*.bundle.js`) entirely, on top of the other path exclusions.
+* `--confidence-level {all,low,medium,high,very-high}`: The lowest confidence to report (default `low`, every finding). `medium`, `high` and `very-high` keep confidence 5, 6 and 9 or more, the findings of the SARIF tiers of the same names and above; `all` also reports the regex matches the scanner judged not to be secrets (placeholders, examples), with an `-INFO` rule id.
+* `--min-confidence N`: Report findings of confidence `N` or more, `N` from 0 to 10: any cut, where `--confidence-level` offers the floors of the tiers. Like every level but `all`, it leaves out the regex matches judged not to be secrets. Use one flag or the other, not both.
+* `--exclude-tags TAG [TAG ...]`: Leave out the findings that carry any of these tags. A tag says what sort of secret a finding is, where its confidence says how likely it is one: `weak-password` (a weak value under a password-like name), `publishable` (a key made to ship to clients: reCAPTCHA and other site keys, Stripe `pk_` keys, Mapbox `pk.` tokens, player, analytics and maps keys), `test-context` (a test, spec, fixture, mock, example or sample path, relative to the target directory), `recorded-traffic` (VCR cassettes, session recordings, HAR files), `identifier` (an AWS id of a user, role or policy, which grants nothing) and `web-page` (an HTML page). Every finding is still reported by default; e.g. `--exclude-tags test-context` when you hunt live secrets.
+* `--report-diagnostics`: Add every file found under the target to the SARIF report, with its status, scan time, skip reason and analysis depth.
+* `--regex-candidate-scoring-rules /path/to/rules.json`: Supply your own rules for judging regex matches before they are reported. Give the flag no value to report regex matches unjudged.
+* `--multiprocessing-context {forkserver,spawn,fork}`: How worker processes start. The default is `forkserver` where the platform has it (Linux, macOS): workers share the scanner already loaded in a server process. Windows uses `spawn`. All three give the same findings.
+* `--disable-masking`: Keep potential secrets unmasked in the output report *(see caution below)*.
+* `--ci` / `--no-ci`: Turn CI mode on or off. CI mode prints a plain progress line every 30 seconds instead of the live progress display. It turns on by itself inside CI services (GitHub Actions, GitLab CI, Jenkins and others) and whenever the output is not a terminal.
+
+
+### Github Actions Integration
+
+eq. `.github/workflows/deepsecrets.yml`
+
+```yaml
+name: DeepSecrets Scan
+on: [push, pull_request]
+
+jobs:
+  scan:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.14'
+
+      - name: Install DeepSecrets
+        run: pip install deepsecrets
+
+      - name: Run Scan
+        run: deepsecrets --target-dir . --outformat dojo-sarif --outfile report.sarif
+        continue-on-error: true
+
+      - name: Upload SARIF report
+        uses: github/codeql-action/upload-sarif@v3
+        with:
+          sarif_file: report.sarif
+```
+
+### Masking Secrets in Reports
+
+As of v1.3.0, potential secrets are automatically masked inside reports to protect your pipeline artifacts. Turn this off via the `--disable-masking` flag if necessary.
+
+Masking doesn't break the deduplication logic of downstream platforms (like Github), as the `partialFingerprints` section in the report is correctly populated based on the raw data.
+
+> [!Caution]  
+> If you integrate DeepSecrets into your CI pipeline with masking disabled, you will likely re-leak your secrets inside your CI logs and artifacts.
+
+### SARIF Reports & Dynamic Confidence
+
+Every finding gets a confidence score. However, different security platforms parse SARIF metrics differently. To ensure compatibility across modern ASPM dashboards, DeepSecrets does the following:
+
+* **Virtual Subrules (`rules[]`)**: Dynamically generates rules by confidence tier, from `S105-LOW` (confidence 0 to 4) through `-MEDIUM` (5) and `-HIGH` (6 to 8) to `S105-VERY-HIGH` (9 and 10), and `-INFO` for the rejected matches `--confidence-level all` reports. This forces GitHub Security and DefectDojo to map semantic precision variance properly without breaking native parsers.
+
+* **Deterministic Result Level**: The tool always explicitly sets `level: error` in the `results[]` model. This acts as a universal fallback for CI/CD pipelines and older SAST parsers, ensuring that exposed secrets reliably break builds or block Pull Requests regardless of individual rule interpretations.
+
+* **Tags**: Each result carries its tags, if it has any, in SARIF's standard `properties.tags` array (`weak-password`, `publishable`, `test-context`, `recorded-traffic`, `identifier`, `web-page`). Rule ids, severity and fingerprints do not depend on them.
+
+* **Contextual Messages**: Injects the raw numeric confidence score natively into `result.message.text` so security analysts see it immediately on their dashboards.
+
+
+## Building rulesets
+
+### Regex
+
+The built-in ruleset for regex checks is located in `/deepsecrets/rules/regexes.json`. You're free to follow the format and create a custom ruleset.
+
+### HashedSecret (Zero-Knowledge Scanning)
+
+Example ruleset for hashed checks is located in `/tests/fixtures/hashed_secrets.json`. You're free to follow the format and create a custom ruleset.
+
+#### HashedSecret Ruleset Example
+
+To look for known production secrets without exposing them in plaintext inside your repository, provide a JSON containing their hashes:
+
+```json
+[
+  {
+    "name": "KNOWN-PROD-DATABASE-PASSWORD",
+    "hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+    "length": 12,
+    "algorithm": "sha1"
+  }
+]
+```
+Run with `--hashed-values /path/to/hashes.json`. DeepSecrets will automatically hash string candidates on the fly during its lexing stage to match them.
+
+## Contacts
+
+- Nikolai Khechumov ([@ntoskernel](https://github.com/ntoskernel)) — creator and maintainer
+
+## FAQ
+> Pff, is it still regex-based?
+
+Yes and no. Of course, it uses regexes to find typed secrets like any other tool. But language understanding (the lexing stage) and variable detection also use regexes under the hood. Regex is an instrument, not the problem. The problem is applying regex blindly without semantic context.
+
+> But what about Semgrep Secrets? Looks like you're cloning their thing.
+
+DeepSecrets was originally released in April 2023 — six months before Semgrep Secrets launched. We share similar principles, but DeepSecrets is 100% free/open-source and leverages a significantly broader multi-language tracking surface.
+
+### DeepSecrets vs. Other Scanners
+
+While other tools scan only what they know, DeepSecrets leverages lexers. This allows it to surface hidden, dangerous credentials in rare configuration formats and custom code blocks that benchmarks may not have datasets for.
+
+#### Tool comparison based on SecretBench Results
+
+| Feature / Capability | **DeepSecrets 2.3** | **Gitleaks 8.30** | **Betterleaks 1.8** | **TruffleHog** | **Semgrep Secrets** |
+| :--- | :---: | :---: | :---: | :---: | :---: |
+| **SecretBench Accuracy** | **93% Recall<br>91% Precision** | 85% Recall<br>70% Precision | 85% Recall<br>61% Precision | *Not re-measured* | *Not Evaluated* |
+| **Price & Licensing** | **Free / Open-Source** | Free / Open-Source | Free / Open-Source | Free / Open-Source | Commercial / Paid |
+| **Analysis Type** | **Semantic / Regex** | Flat-text Regex / Entropy | Flat-text Regex / Entropy | Flat-text Regex / Entropy | Semantic |
+| **Language Support** | **~600 lexed<br>(variables in 14)** | Context-agnostic (Text) | Context-agnostic (Text) | Context-agnostic (Text) | Limited subset |
+| **Pre-hashed Validation** | **Yes (via Hashed Engine)** | No | No | No | No |
+| **Context-Aware Entropy**| **Yes (Assigned values)** | No (Entire file text) | No (Entire file text) | No (Entire file text) | Yes |
+| **Advanced SARIF Output**| **Yes (Confidence tiers and tags)** | Basic | Basic | Basic | Yes |
+
+_SecretBench, every file scanned, every labelled row counted, with [dsbench](https://github.com/ntoskernel/dsbench) (see [Reproduce these numbers](#reproduce-these-numbers)). DeepSecrets 2.3, Gitleaks 8.30.1 and Betterleaks 1.8.1 measured on the same harness in October and November 2026, Gitleaks and Betterleaks with their default rules and their global path allowlist removed so that every file is scanned, as for DeepSecrets. TruffleHog was measured in June 2026 by another method (52% recall, 6% precision) and not since._
+
+
+
+> Why don't you build true abstract syntax trees? It's academically more correct!
+
+DeepSecrets tries to keep a balance between complexity and effectiveness. Building a true AST across hundreds of languages is incredibly complex and simply overkill for the secrets detection. The tool follows the generic SAST approach to code analysis but optimizes the AST stage for maximum speed and width.
+
+> I'd like to build my own semantic rules. How do I do that?
+
+Semantic rules are now effectively "variable evaluation rules". You can find them [here](https://github.com/ntoskernel/deepsecrets/blob/main/deepsecrets/rules/variable_scoring_rules.json).
+
+> I still have a question
+
+Feel free to contact the developer directly using the emails listed in [pyproject.toml](https://github.com/ntoskernel/deepsecrets/blob/main/pyproject.toml#L6-L8)
+
+
+## Contributing & Core Concepts
+
+### Under the hood
+There are several core concepts:
+
+- `File`
+- `Tokenizer`
+- `Token`
+- `Engine`
+- `Finding`
+- `ScanMode`
+
+### File
+Just a pythonic representation of a file with all needed methods for management.
+
+### Tokenizer
+Breaks the content of a file into pieces - Tokens - by its logic. There are four types of tokenizers available:
+
+- `FullContentTokenizer`: treats all content as a single token. Useful for regex-based search.
+- `PerWordTokenizer`: breaks given content by words and line breaks.
+- `LexerTokenizer`: uses language-specific smarts to break code into semantically correct pieces with additional context for each token.
+- `CheapVarDetectorTokenizer`: uses tight regexes to cover limitations of semantic variable detection.
+
+
+### Token
+A string with additional information about its semantic role, corresponding file, and location inside it.
+
+### Engine
+A component performing secrets search for a single token by its own logic. Returns a set of Findings. There are three engines available:
+
+- `RegexEngine`: checks tokens' values through a special ruleset
+- `SemanticEngine`: checks tokens produced by the LexerTokenizer using additional context - variable names and values
+- `HashedSecretEngine`: checks tokens' values by hashing them and trying to find coinciding hashes inside a special ruleset
+
+### Finding
+This is a data structure representing a problem detected inside code. Features information about the precise location inside a file and a rule that found it.
+
+### ScanMode
+This component is responsible for the scan process.
+
+- Defines the scope of analysis for a given work directory respecting exceptions
+- Allows declaring a `PerFileAnalyzer` - the method called against each file, returning a list of findings. The primary usage is to initialize necessary engines, tokenizers, and rulesets.
+- Runs the scan: a multiprocessing pool analyzes every file in parallel.
+- Prepares results for output and outputs them.
+
+The current implementation has a `CliScanMode` built by the user-provided config through the cli args.
+
+### Local development
+
+The project is supposed to be developed using VSCode and 'Remote containers' feature.
+
+Steps:
+1. Clone the repository
+2. Open the cloned folder with VSCode
+3. Select "Reopen in Container" when prompted
+4. Wait for the automated environment build to complete. You are ready to develop.
+
+## History and license
+
+DeepSecrets was created by Nikolai Khechumov at Avito in 2022 and open-sourced in April 2023 at
+[avito-tech/deepsecrets](https://github.com/avito-tech/deepsecrets). It has been an OWASP project since 2023 and has
+been developed in this repository since November 2023.
+
+From version 2.3.0 DeepSecrets is licensed under the [Apache License 2.0](LICENSE). Code from the original release
+and contributions made before 2.3.0 remain available under the MIT License, whose text is at the end of
+[LICENSE](LICENSE); [NOTICE](NOTICE) says which is which. Releases up to and including 2.2.0 were published under the
+MIT License.
+