nova-hunting 0.1.0__tar.gz

nova_hunting-0.1.0/LICENCE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Thomas Roccia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

nova_hunting-0.1.0/PKG-INFO

@@ -0,0 +1,99 @@
+Metadata-Version: 2.2
+Name: nova-hunting
+Version: 0.1.0
+Summary: Prompt Pattern Matching Framework for Generative AI
+Home-page: https://github.com/fr0gger/nova-framework
+Author: Thomas Roccia
+Author-email: contact@securitybreak.io
+License: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENCE
+Requires-Dist: mkdocs-material
+Requires-Dist: mkdocs
+Requires-Dist: mkdocs-material[imaging]
+Requires-Dist: sentence-transformers
+Requires-Dist: transformers
+Requires-Dist: requests
+Requires-Dist: pyyaml
+Requires-Dist: colorama
+Requires-Dist: openai
+Requires-Dist: anthropic
+Requires-Dist: pytest
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# NOVA: The Prompt Pattern Matching
+
+![GitHub](https://img.shields.io/github/fr0gger/nova-framework)
+
+<p align="center">
+    <img src="nova_doc/docs/nova.svg" alt="NOVA Logo">
+</p>
+
+Generative AI systems are rapidly being adopted and deployed across organizations. While they enhance productivity and efficiency, they also expand the attack surface.
+
+How do you detect abusive usage of your system? How do you hunt for malicious prompts? Whether it is identifying jailbreaking attempts, preventing reputational damage, or spotting unexpected behaviors, tracking prompt TTPs can be very useful to track the usage of your AI systems.
+
+That's where NOVA comes in!
+
+🚧 **Disclaimer:** NOVA is currently in beta. Expect potential bugs, incomplete features, and ongoing improvements. If you identify a bug, please [report it here](https://github.com/fr0gger/nova-framework/issues).
+
+NOVA is an open-source prompt pattern matching system combining keyword detection, semantic similarity, and LLM-based evaluation to analyze and detect prompt content.
+
+## Features
+
+- 🔍 **Keyword Detection:** Flag suspicious prompts using predefined keywords or regex.
+- 💬 **Semantic Similarity:** Identify pattern variations using configurable thresholds.
+- ✨ **LLM Matching:** Create matching rules using natural language evaluated by LLM.
+
+Inspired by YARA syntax, NOVA rules are readable and flexible, ideal for prompt hunting and threat detection.
+
+## Anatomy of a NOVA Rule
+
+```bash
+rule RuleName
+{
+    meta:
+        description = "Rule description"
+        author = "Author name"
+
+    keywords:
+        $keyword1 = "exact text"
+        $keyword2 = /regex pattern/i
+
+    semantics:
+        $semantic1 = "semantic pattern" (0.6)
+
+    llm:
+        $llm_check = "LLM evaluation prompt" (0.7)
+
+    condition:
+        keywords.$keyword1 or semantics.$semantic1 or llm.$llm_check
+}
+```
+
+## Getting Started
+
+```bash
+pip install nova-framework
+```
+
+## License
+
+This project is licensed under the [MIT License](LICENSE).
+
+## Credits
+
+Created and maintained by [fr0gger](https://github.com/fr0gger).

nova_hunting-0.1.0/README.md

@@ -0,0 +1,63 @@
+# NOVA: The Prompt Pattern Matching
+
+![GitHub](https://img.shields.io/github/fr0gger/nova-framework)
+
+<p align="center">
+    <img src="nova_doc/docs/nova.svg" alt="NOVA Logo">
+</p>
+
+Generative AI systems are rapidly being adopted and deployed across organizations. While they enhance productivity and efficiency, they also expand the attack surface.
+
+How do you detect abusive usage of your system? How do you hunt for malicious prompts? Whether it is identifying jailbreaking attempts, preventing reputational damage, or spotting unexpected behaviors, tracking prompt TTPs can be very useful to track the usage of your AI systems.
+
+That's where NOVA comes in!
+
+🚧 **Disclaimer:** NOVA is currently in beta. Expect potential bugs, incomplete features, and ongoing improvements. If you identify a bug, please [report it here](https://github.com/fr0gger/nova-framework/issues).
+
+NOVA is an open-source prompt pattern matching system combining keyword detection, semantic similarity, and LLM-based evaluation to analyze and detect prompt content.
+
+## Features
+
+- 🔍 **Keyword Detection:** Flag suspicious prompts using predefined keywords or regex.
+- 💬 **Semantic Similarity:** Identify pattern variations using configurable thresholds.
+- ✨ **LLM Matching:** Create matching rules using natural language evaluated by LLM.
+
+Inspired by YARA syntax, NOVA rules are readable and flexible, ideal for prompt hunting and threat detection.
+
+## Anatomy of a NOVA Rule
+
+```bash
+rule RuleName
+{
+    meta:
+        description = "Rule description"
+        author = "Author name"
+
+    keywords:
+        $keyword1 = "exact text"
+        $keyword2 = /regex pattern/i
+
+    semantics:
+        $semantic1 = "semantic pattern" (0.6)
+
+    llm:
+        $llm_check = "LLM evaluation prompt" (0.7)
+
+    condition:
+        keywords.$keyword1 or semantics.$semantic1 or llm.$llm_check
+}
+```
+
+## Getting Started
+
+```bash
+pip install nova-framework
+```
+
+## License
+
+This project is licensed under the [MIT License](LICENSE).
+
+## Credits
+
+Created and maintained by [fr0gger](https://github.com/fr0gger).

nova_hunting-0.1.0/nova/__init__.py

@@ -0,0 +1,28 @@
+"""
+NOVA: The Prompt Pattern Matching
+Author: Thomas Roccia 
+twitter: @fr0gger_
+License: MIT License
+Version: 1.0.0
+Description: Main Nova framework package initialization
+"""
+
+__version__ = "1.0.0"
+
+from nova.core.rules import (
+    KeywordPattern,
+    SemanticPattern,
+    LLMPattern,
+    NovaRule
+)
+from nova.core.matcher import NovaMatcher
+from nova.core.parser import NovaParser
+
+__all__ = [
+    'KeywordPattern',
+    'SemanticPattern',
+    'LLMPattern',
+    'NovaRule',
+    'NovaMatcher',
+    'NovaParser',
+]

nova_hunting-0.1.0/nova/core/__init__.py

@@ -0,0 +1,29 @@
+"""
+NOVA: The Prompt Pattern Matching
+Author: Thomas Roccia 
+twitter: @fr0gger_
+License: MIT License
+Version: 1.0.0
+Description: Core components package initialization
+"""
+
+from nova.core.rules import (
+    KeywordPattern,
+    SemanticPattern,
+    LLMPattern,
+    NovaRule
+)
+from nova.core.matcher import NovaMatcher
+from nova.core.parser import NovaParser, NovaRuleFileParser
+from nova.core.scanner import NovaScanner
+
+__all__ = [
+    'KeywordPattern',
+    'SemanticPattern',
+    'LLMPattern',
+    'NovaRule',
+    'NovaMatcher',
+    'NovaParser',
+    'NovaRuleFileParser',
+    'NovaScanner',
+]

nova_hunting-0.1.0/nova/core/matcher.py

@@ -0,0 +1,198 @@
+"""
+NOVA: The Prompt Pattern Matching
+Author: Thomas Roccia 
+twitter: @fr0gger_
+License: MIT License
+Version: 1.0.0
+Description: Core matcher implementation for Nova rules
+"""
+
+from typing import Dict, List, Tuple, Optional, Any
+import re
+
+from nova.core.rules import NovaRule, KeywordPattern, SemanticPattern, LLMPattern
+from nova.evaluators.keywords import DefaultKeywordEvaluator
+from nova.evaluators.semantics import DefaultSemanticEvaluator
+from nova.evaluators.llm import OpenAIEvaluator
+from nova.evaluators.condition import evaluate_condition
+
+
+class NovaMatcher:
+    """
+    Matcher for Nova rules.
+    Evaluates text against rules using different pattern types.
+    """
+    
+    def __init__(self, 
+                 rule: NovaRule,
+                 keyword_evaluator: Optional[DefaultKeywordEvaluator] = None,
+                 semantic_evaluator: Optional[DefaultSemanticEvaluator] = None,
+                 llm_evaluator: Optional[OpenAIEvaluator] = None):
+        """
+        Initialize the matcher with a rule and optional custom evaluators.
+        
+        Args:
+            rule: The NovaRule to match against
+            keyword_evaluator: Custom keyword evaluator (uses DefaultKeywordEvaluator if None)
+            semantic_evaluator: Custom semantic evaluator (uses DefaultSemanticEvaluator if None)
+            llm_evaluator: Custom LLM evaluator (uses OpenAIEvaluator if None)
+        """
+        self.rule = rule
+        
+        # Initialize evaluators
+        self.keyword_evaluator = keyword_evaluator or DefaultKeywordEvaluator()
+        self.semantic_evaluator = semantic_evaluator or DefaultSemanticEvaluator()
+        self.llm_evaluator = llm_evaluator or OpenAIEvaluator()
+        
+        # Pre-compile keyword patterns for performance
+        self._precompile_patterns()
+    
+    def _precompile_patterns(self):
+        """Pre-compile regex patterns for better performance."""
+        for key, pattern in self.rule.keywords.items():
+            if pattern.is_regex:
+                self.keyword_evaluator.compile_pattern(key, pattern)
+    
+    def check_prompt(self, prompt: str) -> Dict[str, Any]:
+        """
+        Check if a prompt matches the rule.
+        
+        Args:
+            prompt: The prompt text to check
+            
+        Returns:
+            Dictionary containing match results and details
+        """
+        # Get all keyword matches for debugging
+        all_keyword_matches = {}
+        for key, pattern in self.rule.keywords.items():
+            all_keyword_matches[key] = self.keyword_evaluator.evaluate(pattern, prompt, key)
+        
+        # Get all semantic matches for debugging
+        all_semantic_matches = {}
+        all_semantic_scores = {}
+        for key, pattern in self.rule.semantics.items():
+            matched, score = self.semantic_evaluator.evaluate(pattern, prompt)
+            all_semantic_matches[key] = matched
+            all_semantic_scores[key] = score
+        
+        # Get all LLM matches for debugging
+        all_llm_matches = {}
+        all_llm_scores = {}
+        for key, pattern in self.rule.llms.items():
+            # Use the pattern's threshold as the temperature parameter
+            temperature = pattern.threshold
+            matched, confidence, details = self.llm_evaluator.evaluate_prompt(pattern.pattern, prompt, temperature=temperature)
+            all_llm_matches[key] = matched  # Don't compare confidence to threshold anymore
+            all_llm_scores[key] = confidence
+        
+        # For the condition evaluation, only use variables explicitly referenced
+        condition = self.rule.condition
+        
+        # Extract variables directly referenced in the condition
+        keyword_matches = {}
+        semantic_matches = {}
+        llm_matches = {}
+        
+        # Check for direct variable references with wildcards (section.$var*)
+        for section, var_dict, target_dict in [
+            ('keywords', all_keyword_matches, keyword_matches),
+            ('semantics', all_semantic_matches, semantic_matches),
+            ('llm', all_llm_matches, llm_matches)
+        ]:
+            # Match exact references like "section.$var"
+            pattern = rf'{section}\.\$([a-zA-Z0-9_]+)(?!\*)'
+            for match in re.finditer(pattern, condition):
+                var_name = f"${match.group(1)}"
+                if var_name in var_dict:
+                    target_dict[var_name] = var_dict[var_name]
+                    
+            # Match wildcard references like "section.$var*"
+            wildcard_pattern = rf'{section}\.\$([a-zA-Z0-9_]+)\*'
+            for match in re.finditer(wildcard_pattern, condition):
+                prefix = match.group(1)
+                # Add all variables matching this prefix
+                for var, value in var_dict.items():
+                    if var[1:].startswith(prefix):  # Remove $ from var name
+                        target_dict[var] = value
+        
+        # Check for standalone variables ($var)
+        var_pattern = r'(?<![a-zA-Z0-9_\.])(\$[a-zA-Z0-9_]+)(?!\*)'
+        for match in re.finditer(var_pattern, condition):
+            var_name = match.group(1)
+            
+            # Skip if it's already handled as a section.$ reference
+            if any(var_name in d for d in [keyword_matches, semantic_matches, llm_matches]):
+                continue
+                
+            # Try to find where this variable is defined
+            if var_name in all_keyword_matches:
+                keyword_matches[var_name] = all_keyword_matches[var_name]
+            elif var_name in all_semantic_matches:
+                semantic_matches[var_name] = all_semantic_matches[var_name]
+            elif var_name in all_llm_matches:
+                llm_matches[var_name] = all_llm_matches[var_name]
+        
+        # Handle "any of" wildcards if present
+        any_of_pattern = r'any\s+of\s+\(\$([a-zA-Z0-9_]+)\*\)'
+        for match in re.finditer(any_of_pattern, condition):
+            prefix = match.group(1)
+            
+            # Add variables matching this prefix from all sections
+            for var, value in all_keyword_matches.items():
+                if var[1:].startswith(prefix):  # Remove $ from var name
+                    keyword_matches[var] = value
+                    
+            for var, value in all_semantic_matches.items():
+                if var[1:].startswith(prefix):
+                    semantic_matches[var] = value
+                    
+            for var, value in all_llm_matches.items():
+                if var[1:].startswith(prefix):
+                    llm_matches[var] = value
+        
+        # Process section wildcards (keywords.*, semantics.*, llm.*)
+        if "keywords.*" in condition:
+            keyword_matches.update(all_keyword_matches)
+        if "semantics.*" in condition:
+            semantic_matches.update(all_semantic_matches)
+        if "llm.*" in condition:
+            llm_matches.update(all_llm_matches)
+        
+        # Evaluate condition if provided
+        has_match = False
+        condition_result = None
+        
+        if self.rule.condition:
+            # Use the condition evaluator with filtered match types
+            condition_result = evaluate_condition(
+                self.rule.condition, 
+                keyword_matches, 
+                semantic_matches, 
+                llm_matches
+            )
+            has_match = condition_result
+        else:
+            # Fall back to original behavior if no condition is specified
+            has_match = any(keyword_matches.values()) or any(semantic_matches.values()) or any(llm_matches.values())
+        
+        # Build results with matching variables only
+        results = {
+            'matched': has_match,
+            'rule_name': self.rule.name,
+            'meta': self.rule.meta,
+            'matching_keywords': {k: v for k, v in keyword_matches.items() if v},
+            'matching_semantics': {k: v for k, v in semantic_matches.items() if v},
+            'matching_llm': {k: v for k, v in llm_matches.items() if v},
+            'semantic_scores': all_semantic_scores,
+            'llm_scores': all_llm_scores,
+            'debug': {
+                'condition': self.rule.condition,
+                'condition_result': condition_result,
+                'all_keyword_matches': all_keyword_matches,
+                'all_semantic_matches': all_semantic_matches,
+                'all_llm_matches': all_llm_matches
+            }
+        }
+        
+        return results