envshield 4.7.1__tar.gz → 4.7.3__tar.gz

{envshield-4.7.1 → envshield-4.7.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: envshield
-Version: 4.7.1
+Version: 4.7.3
 Summary: EnvShield: environment variables as a version-controlled configuration contract.
 Author-email: Rabbil Yasar Sajal <rabbilyasar@gmail.com>
 License: MIT License
@@ -308,7 +308,25 @@ jobs:
           envshield undeclared "$BASE" HEAD
 ```
 
-Both commands exit non-zero on a real finding — nothing beyond the exit code is required to gate a PR. Add `--json` if you want to post a custom summary instead.
+Both commands exit non-zero on a real finding, and also on an error that prevented evaluation (a bad revision, a schema that failed to parse) — never a silent "clean" for something it couldn't actually check. Add `--json` if you want to post a custom summary instead.
+
+### SARIF output (`undeclared --sarif`)
+
+`envshield undeclared` also supports `--sarif` — a standard [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/) log, for GitHub Code Scanning or any other SARIF-consuming CI tool, so a missing declaration is annotated on its exact file and line rather than only appearing in a CI log:
+
+```yaml
+      - run: |
+          BASE=$(git merge-base "origin/${{ github.base_ref }}" HEAD)
+          envshield undeclared "$BASE" HEAD --sarif > envshield.sarif
+        continue-on-error: true   # upload the report even when it found something
+      - uses: github/codeql-action/upload-sarif@v3
+        with:
+          sarif_file: envshield.sarif
+```
+
+`--sarif` only exists for `undeclared`, deliberately: each finding there has a real file and line to point at. `schema diff` findings describe the *contract* (a variable added, removed, or changed) rather than a source location, so forcing them into SARIF would mean inventing a line number that isn't real — use its `--json`/exit-code output instead, or annotate it yourself (e.g. a `jq` step over the JSON, emitting `::error::` workflow commands).
+
+An error that prevents evaluating a service (a bad revision, a schema that fails to parse) is never turned into a fabricated SARIF result — it's reported as a SARIF `invocation` with `executionSuccessful: false`, so a consumer can't mistake "couldn't check" for "checked and clean."
 
 ---
 
@@ -410,6 +428,34 @@ A few things worth knowing:
 
 ---
 
+## Docker Compose base + override layers (`files:`)
+
+A manifest registration normally points at one file:
+
+```yaml
+manifests:
+  - file: docker-compose.yml
+    containers:
+      api: api
+```
+
+A real Compose project commonly splits its configuration across a base file plus an override that Compose applies automatically (`docker-compose.yml` + `docker-compose.override.yml`, or any other `-f a -f b` combination). Registering only the base misses whatever the override adds; registering the override alone reports the base's own variables as falsely missing, since an override file is deliberately just a delta. `files:` registers the whole ordered layer set as **one** logical manifest instead:
+
+```yaml
+manifests:
+  - files:
+      - docker-compose.yml
+      - docker-compose.override.yml
+    containers:
+      api: api
+```
+
+Order matters — later files override earlier ones, exactly like `docker compose -f docker-compose.yml -f docker-compose.override.yml` itself. `check`/`doctor`/`explain` all validate the merged result as a single source, under one combined label (`"docker-compose.yml + docker-compose.override.yml"`).
+
+This is Compose-specific merge behavior for the fields EnvShield already understands (`environment:`, `env_file:`) — **not** a general YAML merge engine. Volumes, networks, ports, build, and every other Compose field are never inspected or merged. `files:` is unrelated to `completeness: union` above: union combines *presence across different kinds of source* (a local file and a manifest); `files:` merges *values across ordered layers of one Compose source*. A manifest entry can declare `file:` or `files:`, never both.
+
+---
+
 ## Supported languages / discovery
 
 Source-code discovery (what powers `undeclared` and `explain`'s "used in source" section) is AST-based for **Python** (`os.environ.get`, `os.getenv`, `os.environ[...]`) and pattern-based for **JavaScript/TypeScript** (`process.env.X`, `process.env["X"]`, `import.meta.env.X`, including single-level destructuring). No other language is discovered yet — the schema and validation commands work with any stack, but `undeclared`/`explain` can only see what's read from these two.
@@ -422,7 +468,7 @@ Source-code discovery (what powers `undeclared` and `explain`'s "used in source"
 
 A supporting check alongside the contract, not the product itself. `secret = true` on a field tells EnvShield (and `setup`, and code generation) that it's sensitive — the schema records that fact, never the actual value. On top of that:
 
-- **`envshield scan`** looks for hardcoded secrets by pattern (Stripe, AWS, GitHub tokens, and more), and for env-var reads the schema doesn't declare, in one pass. Values are always redacted in output — a Stripe *publishable* key (`pk_...`) is never flagged, since detection matches the secret-key pattern (`sk_...`) by shape, not the variable's name.
+- **`envshield scan`** looks for hardcoded secrets by pattern (Stripe, AWS, GitHub tokens, and more), and for env-var reads the schema doesn't declare, in one pass. Values are always redacted in output — a Stripe *publishable* key (`pk_...`) is never flagged, since detection matches the secret-key pattern (`sk_...`) by shape, not the variable's name. Python candidates are context-aware: code-shaped patterns like `key=CONSTANT` or `api_key: str` may be suppressed when they're clearly not secret values, while secret-shaped string literals remain detectable regardless of context.
 - **`envshield hook install`** wires `scan --staged` into a pre-commit hook, so a real secret is caught before it's committed, not after.
 
 ---

{envshield-4.7.1 → envshield-4.7.3}/README.md

@@ -249,7 +249,25 @@ jobs:
           envshield undeclared "$BASE" HEAD
 ```
 
-Both commands exit non-zero on a real finding — nothing beyond the exit code is required to gate a PR. Add `--json` if you want to post a custom summary instead.
+Both commands exit non-zero on a real finding, and also on an error that prevented evaluation (a bad revision, a schema that failed to parse) — never a silent "clean" for something it couldn't actually check. Add `--json` if you want to post a custom summary instead.
+
+### SARIF output (`undeclared --sarif`)
+
+`envshield undeclared` also supports `--sarif` — a standard [SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/) log, for GitHub Code Scanning or any other SARIF-consuming CI tool, so a missing declaration is annotated on its exact file and line rather than only appearing in a CI log:
+
+```yaml
+      - run: |
+          BASE=$(git merge-base "origin/${{ github.base_ref }}" HEAD)
+          envshield undeclared "$BASE" HEAD --sarif > envshield.sarif
+        continue-on-error: true   # upload the report even when it found something
+      - uses: github/codeql-action/upload-sarif@v3
+        with:
+          sarif_file: envshield.sarif
+```
+
+`--sarif` only exists for `undeclared`, deliberately: each finding there has a real file and line to point at. `schema diff` findings describe the *contract* (a variable added, removed, or changed) rather than a source location, so forcing them into SARIF would mean inventing a line number that isn't real — use its `--json`/exit-code output instead, or annotate it yourself (e.g. a `jq` step over the JSON, emitting `::error::` workflow commands).
+
+An error that prevents evaluating a service (a bad revision, a schema that fails to parse) is never turned into a fabricated SARIF result — it's reported as a SARIF `invocation` with `executionSuccessful: false`, so a consumer can't mistake "couldn't check" for "checked and clean."
 
 ---
 
@@ -351,6 +369,34 @@ A few things worth knowing:
 
 ---
 
+## Docker Compose base + override layers (`files:`)
+
+A manifest registration normally points at one file:
+
+```yaml
+manifests:
+  - file: docker-compose.yml
+    containers:
+      api: api
+```
+
+A real Compose project commonly splits its configuration across a base file plus an override that Compose applies automatically (`docker-compose.yml` + `docker-compose.override.yml`, or any other `-f a -f b` combination). Registering only the base misses whatever the override adds; registering the override alone reports the base's own variables as falsely missing, since an override file is deliberately just a delta. `files:` registers the whole ordered layer set as **one** logical manifest instead:
+
+```yaml
+manifests:
+  - files:
+      - docker-compose.yml
+      - docker-compose.override.yml
+    containers:
+      api: api
+```
+
+Order matters — later files override earlier ones, exactly like `docker compose -f docker-compose.yml -f docker-compose.override.yml` itself. `check`/`doctor`/`explain` all validate the merged result as a single source, under one combined label (`"docker-compose.yml + docker-compose.override.yml"`).
+
+This is Compose-specific merge behavior for the fields EnvShield already understands (`environment:`, `env_file:`) — **not** a general YAML merge engine. Volumes, networks, ports, build, and every other Compose field are never inspected or merged. `files:` is unrelated to `completeness: union` above: union combines *presence across different kinds of source* (a local file and a manifest); `files:` merges *values across ordered layers of one Compose source*. A manifest entry can declare `file:` or `files:`, never both.
+
+---
+
 ## Supported languages / discovery
 
 Source-code discovery (what powers `undeclared` and `explain`'s "used in source" section) is AST-based for **Python** (`os.environ.get`, `os.getenv`, `os.environ[...]`) and pattern-based for **JavaScript/TypeScript** (`process.env.X`, `process.env["X"]`, `import.meta.env.X`, including single-level destructuring). No other language is discovered yet — the schema and validation commands work with any stack, but `undeclared`/`explain` can only see what's read from these two.
@@ -363,7 +409,7 @@ Source-code discovery (what powers `undeclared` and `explain`'s "used in source"
 
 A supporting check alongside the contract, not the product itself. `secret = true` on a field tells EnvShield (and `setup`, and code generation) that it's sensitive — the schema records that fact, never the actual value. On top of that:
 
-- **`envshield scan`** looks for hardcoded secrets by pattern (Stripe, AWS, GitHub tokens, and more), and for env-var reads the schema doesn't declare, in one pass. Values are always redacted in output — a Stripe *publishable* key (`pk_...`) is never flagged, since detection matches the secret-key pattern (`sk_...`) by shape, not the variable's name.
+- **`envshield scan`** looks for hardcoded secrets by pattern (Stripe, AWS, GitHub tokens, and more), and for env-var reads the schema doesn't declare, in one pass. Values are always redacted in output — a Stripe *publishable* key (`pk_...`) is never flagged, since detection matches the secret-key pattern (`sk_...`) by shape, not the variable's name. Python candidates are context-aware: code-shaped patterns like `key=CONSTANT` or `api_key: str` may be suppressed when they're clearly not secret values, while secret-shaped string literals remain detectable regardless of context.
 - **`envshield hook install`** wires `scan --staged` into a pre-commit hook, so a real secret is caught before it's committed, not after.
 
 ---

envshield-4.7.3/envshield/__init__.py

@@ -0,0 +1 @@
+__version__ = "4.7.3"

envshield-4.7.3/envshield/classifier/__init__.py

@@ -0,0 +1,12 @@
+# envshield/classifier/__init__.py
+"""
+Context classifier for false-positive reduction in secret scanning.
+
+This module provides the tokenizer and context classifier used to distinguish
+code patterns (function keyword arguments, type annotations, destructuring)
+from actual secret values, reducing false positives in the Generic API Key
+detector.
+
+See Milestone 2 (tokenizer) and Milestone 3 (classifier) documentation for
+design rationale and limitations.
+"""

envshield-4.7.3/envshield/classifier/context_classifier.py

@@ -0,0 +1,313 @@
+"""
+Context Classifier for EnvShield Secret Scanner
+
+Three-level classification model optimized for Git-hook blocking behavior:
+- LIKELY_SECRET: High confidence secret value (e.g., string literal)
+- LIKELY_CODE: High confidence code pattern (e.g., type annotation, function arg)
+- AMBIGUOUS: Uncertain context (preserve uncertainty for policy layer)
+
+Key principle: Reason about syntactic role, not identifier names.
+"""
+
+from dataclasses import dataclass
+from enum import Enum
+from typing import Any, Dict, Tuple
+
+# Import tokenizer from same module
+from .tokenizer import extract_context
+
+
+class Classification(Enum):
+    """Three-level classification for secret candidates."""
+
+    LIKELY_SECRET = "likely_secret"  # High confidence: looks like a secret value
+    LIKELY_CODE = "likely_code"  # High confidence: syntactic code pattern
+    AMBIGUOUS = "ambiguous"  # Uncertain: could be either
+
+
+@dataclass
+class ClassificationResult:
+    """
+    Result of classifying a secret candidate.
+
+    Attributes:
+        classification: The classification level (LIKELY_SECRET, LIKELY_CODE, AMBIGUOUS)
+        confidence: Confidence level ("HIGH", "MEDIUM", "LOW")
+        reason: Human-readable explanation of the classification
+        rule_name: Name of the rule that matched (for debugging/testing)
+        context_summary: Key context fields (never includes secret values)
+    """
+
+    classification: Classification
+    confidence: str
+    reason: str
+    rule_name: str
+    context_summary: Dict[str, Any]
+
+    def __repr__(self) -> str:
+        return (
+            f"ClassificationResult("
+            f"classification={self.classification.value}, "
+            f"confidence={self.confidence}, "
+            f"reason='{self.reason}', "
+            f"rule_name={self.rule_name})"
+        )
+
+
+class ContextClassifier:
+    """
+    Classifies regex match candidates based on syntactic context.
+
+    Three-level classification model:
+    - LIKELY_SECRET: High confidence secret value (e.g., string literal)
+    - LIKELY_CODE: High confidence code pattern (e.g., type annotation)
+    - AMBIGUOUS: Uncertain context
+
+    Designed for Git-hook blocking: preserves uncertainty for policy layer.
+
+    Example:
+        classifier = ContextClassifier()
+
+        # Function keyword argument (identifier) → LIKELY_CODE
+        result = classifier.classify("func(key=CONSTANT,)", 9, 17)
+        assert result.classification == Classification.LIKELY_CODE
+
+        # Function argument with string literal → LIKELY_SECRET
+        result = classifier.classify("func(key='secret')", 9, 17)
+        assert result.classification == Classification.LIKELY_SECRET
+
+        # Type annotation → LIKELY_CODE
+        result = classifier.classify("def foo(token: str):", 15, 18)
+        assert result.classification == Classification.LIKELY_CODE
+    """
+
+    def classify(
+        self, line: str, match_start: int, match_end: int
+    ) -> ClassificationResult:
+        """
+        Classify a regex match candidate based on syntactic context.
+
+        Args:
+            line: Source line containing the match
+            match_start: Character offset of match start (0-indexed)
+            match_end: Character offset of match end (exclusive)
+
+        Returns:
+            ClassificationResult with classification, confidence, reason, rule name
+
+        Classification rules (highest priority first):
+            1. String literal → LIKELY_SECRET (HIGH confidence)
+            2. Type annotation → LIKELY_CODE (HIGH confidence)
+            3. Function keyword argument (identifier) → LIKELY_CODE (HIGH confidence)
+            4. Multi-line keyword argument heuristic → LIKELY_CODE (MEDIUM-HIGH confidence)
+            5. Destructuring → LIKELY_CODE (HIGH confidence)
+            6. Identifier reference → LIKELY_CODE (MEDIUM confidence)
+            7. Default → AMBIGUOUS
+        """
+        # Extract syntactic context from tokenizer
+        context = extract_context(line, match_start, match_end)
+
+        # Rule 1: String literal → LIKELY_SECRET
+        if context.is_string_literal:
+            return ClassificationResult(
+                classification=Classification.LIKELY_SECRET,
+                confidence="HIGH",
+                reason="string literal value",
+                rule_name="string_literal",
+                context_summary={"is_string_literal": True},
+            )
+
+        # Rule 2: Type annotation → LIKELY_CODE
+        if context.is_type_annotation:
+            return ClassificationResult(
+                classification=Classification.LIKELY_CODE,
+                confidence="HIGH",
+                reason="type annotation (: Type syntax)",
+                rule_name="type_annotation",
+                context_summary={
+                    "is_type_annotation": True,
+                    "preceding_operator": context.preceding_operator,
+                },
+            )
+
+        # Rule 3: Function keyword argument (identifier) → LIKELY_CODE
+        # Example: func(key=CONSTANT,) or func(key=CONSTANT)
+        if (
+            context.inside_function_call
+            and not context.is_string_literal
+            and context.preceding_operator == "="
+            and context.following_char in (",", ")")
+        ):
+            return ClassificationResult(
+                classification=Classification.LIKELY_CODE,
+                confidence="HIGH",
+                reason="function keyword argument (identifier, not literal)",
+                rule_name="function_keyword_argument",
+                context_summary={
+                    "inside_function_call": True,
+                    "preceding_operator": "=",
+                    "following_char": context.following_char,
+                    "is_string_literal": False,
+                },
+            )
+
+        # Rule 4: Multi-line keyword argument heuristic → LIKELY_CODE
+        # Catches cases like:
+        #     key=SESSION_COOKIE_NAME,
+        # where opening paren is on previous line
+        if (
+            not context.is_string_literal
+            and context.preceding_operator == "="
+            and context.following_char in (",", ")")
+        ):
+            return ClassificationResult(
+                classification=Classification.LIKELY_CODE,
+                confidence="MEDIUM-HIGH",
+                reason="keyword argument pattern (multi-line heuristic)",
+                rule_name="multiline_keyword_argument",
+                context_summary={
+                    "preceding_operator": "=",
+                    "following_char": context.following_char,
+                    "is_string_literal": False,
+                },
+            )
+
+        # Rule 5: Destructuring → LIKELY_CODE
+        # Example: const {key, token} = config
+        if context.inside_braces and not context.is_string_literal:
+            return ClassificationResult(
+                classification=Classification.LIKELY_CODE,
+                confidence="HIGH",
+                reason="destructuring assignment",
+                rule_name="destructuring",
+                context_summary={"inside_braces": True, "is_string_literal": False},
+            )
+
+        # Rule 6: REMOVED - Identifier reference overlaps with BL-129
+        # Original intent: api_key = DEFAULT_KEY → LIKELY_CODE
+        # Problem: This does semantic analysis (is DEFAULT_KEY defined locally?)
+        # which requires cross-reference tracking the tokenizer doesn't have.
+        # BL-129 already handles this correctly for cases where the identifier
+        # has a local definition. For external references (no local definition),
+        # they SHOULD remain findings - the classifier was suppressing them
+        # incorrectly. Removing this rule to let BL-129 handle all identifier
+        # reference cases with proper semantic context.
+        #
+        # if (
+        #     not context.is_string_literal
+        #     and context.preceding_operator == '='
+        #     and context.following_char not in (',', ')')
+        # ):
+        #     return ClassificationResult(
+        #         classification=Classification.LIKELY_CODE,
+        #         confidence="MEDIUM",
+        #         reason="identifier reference (not literal)",
+        #         rule_name="identifier_reference",
+        #         context_summary={
+        #             "preceding_operator": "=",
+        #             "is_string_literal": False
+        #         }
+        #     )
+
+        # Rule 7: Default → AMBIGUOUS
+        # Insufficient evidence for high-confidence classification
+        return ClassificationResult(
+            classification=Classification.AMBIGUOUS,
+            confidence="LOW",
+            reason="insufficient syntactic context for confident classification",
+            rule_name="default",
+            context_summary={
+                "is_string_literal": context.is_string_literal,
+                "preceding_operator": context.preceding_operator,
+                "following_char": context.following_char,
+                "inside_function_call": context.inside_function_call,
+                "inside_braces": context.inside_braces,
+                "is_type_annotation": context.is_type_annotation,
+            },
+        )
+
+    def is_likely_secret(self, line: str, match_start: int, match_end: int) -> bool:
+        """
+        Convenience method: returns True if classification is LIKELY_SECRET.
+
+        Args:
+            line: Source line containing the match
+            match_start: Character offset of match start (0-indexed)
+            match_end: Character offset of match end (exclusive)
+
+        Returns:
+            True if classification is LIKELY_SECRET, False otherwise
+        """
+        return (
+            self.classify(line, match_start, match_end).classification
+            == Classification.LIKELY_SECRET
+        )
+
+    def is_likely_code(self, line: str, match_start: int, match_end: int) -> bool:
+        """
+        Convenience method: returns True if classification is LIKELY_CODE.
+
+        Args:
+            line: Source line containing the match
+            match_start: Character offset of match start (0-indexed)
+            match_end: Character offset of match end (exclusive)
+
+        Returns:
+            True if classification is LIKELY_CODE, False otherwise
+        """
+        return (
+            self.classify(line, match_start, match_end).classification
+            == Classification.LIKELY_CODE
+        )
+
+    def is_ambiguous(self, line: str, match_start: int, match_end: int) -> bool:
+        """
+        Convenience method: returns True if classification is AMBIGUOUS.
+
+        Args:
+            line: Source line containing the match
+            match_start: Character offset of match start (0-indexed)
+            match_end: Character offset of match end (exclusive)
+
+        Returns:
+            True if classification is AMBIGUOUS, False otherwise
+        """
+        return (
+            self.classify(line, match_start, match_end).classification
+            == Classification.AMBIGUOUS
+        )
+
+    def classify_batch(
+        self, candidates: list[Tuple[str, int, int]]
+    ) -> list[ClassificationResult]:
+        """
+        Classify multiple candidates in batch.
+
+        Args:
+            candidates: List of (line, match_start, match_end) tuples
+
+        Returns:
+            List of ClassificationResult objects in same order as input
+        """
+        return [self.classify(line, start, end) for line, start, end in candidates]
+
+
+# Convenience function for one-shot classification
+def classify_match(line: str, match_start: int, match_end: int) -> ClassificationResult:
+    """
+    One-shot convenience function: classify a single match.
+
+    Args:
+        line: Source line containing the match
+        match_start: Character offset of match start (0-indexed)
+        match_end: Character offset of match end (exclusive)
+
+    Returns:
+        ClassificationResult
+
+    Example:
+        result = classify_match("api_key = 'sk_live_abc123'", 10, 27)
+        assert result.classification == Classification.LIKELY_SECRET
+    """
+    classifier = ContextClassifier()
+    return classifier.classify(line, match_start, match_end)