yara-ffi 4.0.0 → 4.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fbff0f40a5c38903311cc68831538ee87fd1e88089188187560c25b401886ac4
-  data.tar.gz: c554613d1be85c26e2e7ddc87ac9509a60f7b5b25296de64017b7b067ef13c4f
+  metadata.gz: f964b7eb475719ea6ceb456125b6ead06be8d9263a8839267ee96cdbe072b470
+  data.tar.gz: 36cc2f79374b0d00f245329f5d3e18423b6f16740e92f893c07d662f2183ce67
 SHA512:
-  metadata.gz: f09c36c15253e02a2267373561064434b77fc9dcacdd445a1ec28357fa4e01cc09940496f5b645b1dbd920bc2f6c822d8d8215361ff71c8bebe73f42e01897ce
-  data.tar.gz: 52e22658f2b1eb0f9adddc3642db98f715ed0db7c2d8291808547fa4be86e9f088998e3785204fe1655f2935d5e3dff62588f285120c9375a1603da36458193b
+  metadata.gz: 5636bffbece91a2b3d48568e7bcac3e75d0f7713231c1a0abfbcbf83a0436b1d950ca5dccb5dae98b82c6ab3686521f61b561a2802f1c26145f887ea64fccf37
+  data.tar.gz: b10394734d3c00d986a5f12a6a2682087a7641e2abb62108728fc699f1e79f395f4bf2c5d94842c440fb15c960690773760907be56b4f14edbe5c86c3589aa3c

data/.github/copilot-instructions.md

@@ -1,6 +1,6 @@
 # yara-ffi AI Coding Instructions
 
-This Ruby gem provides FFI bindings to YARA-X (Rust-based YARA implementation) for malware/pattern detection.
+This Ruby gem provides FFI bindings to YARA-X (Rust-based YARA implementation) for malware/pattern detection with advanced pattern matching analysis, rule compilation, serialization, and metadata support.
 
 ## Quick Development Guide
 
@@ -10,13 +10,84 @@ This Ruby gem provides FFI bindings to YARA-X (Rust-based YARA implementation) f
 3. Scanner lifecycle: `add_rule()` → `compile()` → `scan()` → `close()`
 4. Always use resource-safe patterns: `Scanner.open { |s| ... }` or manual `close()`
 5. Interactive testing: `docker run -it --mount type=bind,src="$(pwd)",dst=/app yara-ffi bin/console`
+6. **Documentation**: See `USAGE.md` for comprehensive examples and patterns
 
 ## Core Components (Read These Files First)
 
-- `lib/yara/scanner.rb`: Main API - compile-then-scan workflow, resource management
+- `lib/yara/scanner.rb`: Main API - compile-then-scan workflow, resource management, global variables
+- `lib/yara/compiler.rb`: Advanced rule compilation with globals, error diagnostics, serialization
+- `lib/yara/scan_result.rb`: Enhanced result parsing with pattern matches, metadata, tags, namespaces
+- `lib/yara/pattern_match.rb`: Detailed pattern match information with offsets and data extraction
 - `lib/yara/ffi.rb`: Raw FFI bindings with error codes (`YRX_SUCCESS = 0`)
-- `lib/yara/scan_result.rb`: Result parsing (temporary regex-based metadata extraction)
-- Tests in `test/scanner_test.rb`: Working examples of all patterns
+- Tests in `test/`: Comprehensive test coverage for all features
+  - `test/scanner_test.rb`: Basic scanner patterns
+  - `test/scanner_pattern_match_test.rb`: Pattern matching analysis
+  - `test/compiler_test.rb`: Advanced compilation features
+  - `test/serialize_test.rb`: Rule serialization/deserialization
+  - `test/metadata_test.rb`: Metadata extraction
+  - `test/tags_test.rb`: Tag support
+  - `test/namespace_test.rb`: Namespace functionality
+
+## Key Features & APIs
+
+### Pattern Matching Analysis (NEW)
+```ruby
+# Detailed pattern match information
+results = scanner.scan(data)
+result = results.first
+
+# Access specific pattern matches
+matches = result.matches_for_pattern(:$suspicious)
+matches.each do |match|
+  puts "At offset #{match.offset}: #{match.matched_data(data)}"
+end
+
+# Pattern match convenience methods
+result.pattern_matched?(:$api_call)  # => true/false
+result.total_matches                 # => 5
+result.all_matches                   # => [PatternMatch, ...]
+```
+
+### Advanced Rule Compilation (NEW)
+```ruby
+# Use Compiler for complex scenarios
+compiler = Yara::Compiler.new
+compiler.define_global_str("ENV", "production")
+compiler.define_global_bool("DEBUG", false)
+compiler.add_source(rule1, "rule1.yar")
+compiler.add_source(rule2, "rule2.yar")
+
+# Build and serialize
+serialized = compiler.build_serialized
+scanner = Yara::Scanner.from_serialized(serialized)
+```
+
+### Global Variables (NEW)
+```ruby
+# Set individual globals
+scanner.set_global_str("ENV", "production")
+scanner.set_global_int("MAX_SIZE", 1000)
+scanner.set_global_bool("DEBUG", false)
+
+# Bulk setting with error handling
+scanner.set_globals({
+  "ENV" => "production",
+  "RETRIES" => 3
+}, strict: false)
+```
+
+### Metadata & Tags (NEW)
+```ruby
+# Access metadata with type safety
+result.rule_meta[:author]           # Raw access
+result.metadata_string(:author)     # Type-safe String
+result.metadata_int(:severity)      # Type-safe Integer
+
+# Tag support
+result.tags                         # => ["malware", "trojan"]
+result.has_tag?("malware")         # => true
+result.qualified_name               # => "namespace.rule_name"
+```
 
 ## Critical FFI Patterns
 
@@ -113,11 +184,45 @@ scanner.close  # Required!
 ```
 
 **Error Handling:** Custom exceptions for different failure modes:
-- `CompilationError` - YARA rule syntax issues
-- `ScanError` - Runtime scanning failures
-- `NotCompiledError` - Scanning before compilation
+- `Scanner::CompilationError` - YARA rule syntax issues
+- `Scanner::ScanError` - Runtime scanning failures
+- `Scanner::NotCompiledError` - Scanning before compilation
+- `Compiler::CompileError` - Compilation errors with structured diagnostics
+
+**Enhanced Result Processing:** ScanResult now provides:
+- Structured metadata access via YARA-X API
+- Detailed pattern match information with offsets/lengths
+- Tag extraction and querying
+- Namespace support
+- Pattern match convenience methods
+
+## Performance & Advanced Features
+
+**Rule Serialization for Production:**
+```ruby
+# Compile once, use many times
+compiler = Yara::Compiler.new
+compiler.add_source(ruleset)
+serialized = compiler.build_serialized
+
+# Create multiple scanners from same rules
+scanners = 10.times.map { Yara::Scanner.from_serialized(serialized) }
+```
 
-**Metadata Parsing:** ScanResult parses YARA rule metadata and strings via regex from rule source (temporary solution until YARA-X API improvements).
+**Timeout Configuration:**
+```ruby
+scanner.set_timeout(10000)  # 10 seconds
+```
+
+**Error Diagnostics:**
+```ruby
+begin
+  compiler.build
+rescue Yara::Compiler::CompileError
+  errors = compiler.errors_json
+  warnings = compiler.warnings_json
+end
+```
 
 ## Adding New FFI Functions
 
@@ -138,8 +243,21 @@ end
 - `yrx_compile(src, rules_ptr)` - Compile rules from string
 - `yrx_scanner_create(rules, scanner_ptr)` - Create scanner from compiled rules
 - `yrx_scanner_scan(scanner, data, len)` - Scan data
+- `yrx_scanner_set_global_*()` - Set global variables on scanner
+- `yrx_scanner_set_timeout()` - Configure scan timeout
+- `yrx_compiler_*()` - Advanced compilation functions
+- `yrx_rules_serialize/deserialize()` - Rule serialization
+- `yrx_rule_iter_*()` - Iterate rule components (patterns, metadata, tags)
+- `yrx_pattern_iter_matches()` - Extract pattern match details
 - `yrx_last_error()` - Get last error message
-- Cleanup: `yrx_rules_destroy()`, `yrx_scanner_destroy()`
+- Cleanup: `yrx_rules_destroy()`, `yrx_scanner_destroy()`, `yrx_compiler_destroy()`
+
+## Documentation Structure
+
+- `README.md`: Project overview, installation, minimal usage example
+- `USAGE.md`: Comprehensive usage guide with quick reference + detailed examples
+- `DEVELOPMENT.md`: Development setup and contribution workflow
+- `.github/copilot-instructions.md`: This file - AI coding guidance
 
 ## Dependencies & Constraints
 

data/CHANGELOG.md

@@ -1,5 +1,36 @@
 ## [Unreleased]
 
+## [4.1.0] - 2025-08-20
+
+- **NEW**: Added advanced `Yara::Compiler` API for complex rule compilation scenarios
+  - `Compiler.new` - Create a new compiler instance
+  - `Compiler#define_global_*` methods for setting globals before compilation
+  - `Compiler#add_source` for adding rules from multiple sources
+  - `Compiler#build` and `Compiler#build_serialized` for creating compiled rules
+  - `Compiler#errors_json` and `Compiler#warnings_json` for detailed diagnostics
+- **NEW**: Added rule serialization and deserialization support
+  - `Scanner.from_serialized` - Create scanner from serialized rules
+  - `Scanner.from_rules` - Create scanner from pre-compiled rules
+  - Enables compile-once, use-many-times pattern for production deployments
+- **NEW**: Enhanced pattern matching analysis with `Yara::PatternMatch`
+  - Detailed pattern match information with offsets and lengths
+  - `PatternMatch#offset`, `PatternMatch#length`, `PatternMatch#matched_data`
+  - `ScanResult#matches_for_pattern` - Get matches for specific patterns
+  - `ScanResult#pattern_matched?` - Check if specific pattern matched
+  - `ScanResult#total_matches` and `ScanResult#all_matches` for match analysis
+- **NEW**: Added comprehensive metadata and tag support
+  - Type-safe metadata accessors: `metadata_string`, `metadata_int`, `metadata_bool`
+  - `ScanResult#tags` - Access rule tags as array
+  - `ScanResult#has_tag?` - Check for specific tags
+  - `ScanResult#qualified_name` - Get namespaced rule name
+- **NEW**: Added global variable support for scanners
+  - `Scanner#set_global_str`, `Scanner#set_global_int`, `Scanner#set_global_bool`, `Scanner#set_global_float`
+  - `Scanner#set_globals` - Bulk setting with error handling options
+  - Enable dynamic rule behavior based on runtime variables
+- **NEW**: Added scanner timeout configuration via `Scanner#set_timeout`
+- **IMPROVED**: Enhanced documentation with comprehensive usage examples in `USAGE.md`
+- **IMPROVED**: Updated development documentation and AI coding instructions
+
 ## [4.0.0] - 2025-08-19
 
 - **BREAKING**: Migrated from legacy libyara FFI bindings to YARA-X C API (`libyara_x_capi.so`) ([#24](https://github.com/jonmagic/yara-ffi/pull/24))

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    yara-ffi (4.0.0)
+    yara-ffi (4.1.0)
       ffi
 
 GEM

data/README.md

@@ -7,6 +7,22 @@ A Ruby library for using [YARA-X](https://virustotal.github.io/yara-x/) via FFI
 - Ruby 3.0 or later
 - YARA-X C API library (`libyara_x_capi`) installed on your system
 
+## Major Features
+
+**🔍 Pattern Matching Analysis**: Extract detailed pattern match information with exact offsets, lengths, and matched data - perfect for forensic analysis.
+
+**🛠️ Advanced Rule Compilation**: Use the `Yara::Compiler` class for complex scenarios with global variables, structured error reporting, and multiple rule sources.
+
+**💾 Rule Serialization**: Compile rules once, serialize for persistence or transport, then deserialize for instant scanning - eliminating compilation overhead.
+
+**🏷️ Metadata & Tags**: Full access to rule metadata with type safety and tag-based rule categorization and filtering.
+
+**🌐 Global Variables**: Set string, boolean, integer, and float globals at runtime to customize rule behavior dynamically.
+
+**📁 Namespace Support**: Organize rules logically, avoid naming conflicts, and access qualified rule names in large rule sets.
+
+**⚡ Performance**: Configurable scan timeouts, efficient resource management with automatic cleanup, and parallel scanning support.
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -23,138 +39,31 @@ Or install it yourself as:
 
     $ gem install yara-ffi
 
-## Usage
-
-### Quick scanning with convenience methods
-
-```ruby
-rule = <<-RULE
-rule ExampleRule
-{
-  meta:
-    description = "Example rule for testing"
-
-  strings:
-    $text_string = "we were here"
-    $text_regex = /were here/
-
-  condition:
-    $text_string or $text_regex
-}
-RULE
-
-# Test a rule against data
-results = Yara.test(rule, "one day we were here and then we were not")
-puts results.first.match?  # => true
-puts results.first.rule_name  # => "ExampleRule"
-
-# Scan with a block for processing results
-Yara.scan(rule, "sample data") do |result|
-  puts "Matched: #{result.rule_name}"
-end
-```
-
-### Manual scanner usage
+## Quick Start
 
 ```ruby
-rule = <<-RULE
-rule ExampleRule
-{
-  meta:
-    string_meta = "an example rule for testing"
-    int_meta = 42
-    bool_meta = true
-
-  strings:
-    $my_text_string = "we were here"
-    $my_text_regex = /were here/
-
-  condition:
-    $my_text_string or $my_text_regex
-}
-RULE
-
-scanner = Yara::Scanner.new
-scanner.add_rule(rule)
-scanner.compile
-
-results = scanner.scan("one day we were here and then we were not")
-result = results.first
-
-puts result.match?           # => true
-puts result.rule_name        # => "ExampleRule"
-puts result.rule_meta        # => {:string_meta=>"an example rule for testing", :int_meta=>42, :bool_meta=>true}
-puts result.rule_strings     # => {:"$my_text_string"=>"we were here", :"$my_text_regex"=>"were here"}
-
-scanner.close  # Clean up resources when done
-```
+require 'yara'
 
-### Block-based scanner usage
+# Simple test
+results = Yara.test(rule_string, data)
+puts "Matched: #{results.first.rule_name}" if results.first&.match?
 
-```ruby
-# Automatically handles resource cleanup
+# Resource-managed scanning
 Yara::Scanner.open(rule) do |scanner|
   scanner.compile
-  results = scanner.scan("test data")
-  # scanner is automatically closed when block exits
+  results = scanner.scan(data)
 end
 ```
 
-### Multiple rules
-
-```ruby
-rule1 = <<-RULE
-rule RuleOne
-{
-  strings:
-    $a = "pattern one"
-  condition:
-    $a
-}
-RULE
-
-rule2 = <<-RULE
-rule RuleTwo
-{
-  strings:
-    $b = "pattern two"
-  condition:
-    $b
-}
-RULE
-
-scanner = Yara::Scanner.new
-scanner.add_rule(rule1)
-scanner.add_rule(rule2)
-scanner.compile
-
-results = scanner.scan("text with pattern one and pattern two")
-puts results.map(&:rule_name)  # => ["RuleOne", "RuleTwo"]
-scanner.close
-```
-
-## API Reference
-
-### Yara module methods
+**📖 For comprehensive usage examples, advanced features, and API documentation, see [USAGE.md](USAGE.md).**
 
-- `Yara.test(rule_string, data)` - Quick test of a rule against data, returns array of ScanResult objects
-- `Yara.scan(rule_string, data, &block)` - Scan data with rule, optionally yielding each result to block
+## API Overview
 
-### Scanner class
+**Core Classes**: `Yara`, `Yara::Scanner`, `Yara::Compiler`, `Yara::ScanResult`, `Yara::ScanResults`, `Yara::PatternMatch`
 
-- `Scanner.new` - Create a new scanner instance
-- `Scanner.open(rule_string, namespace: nil, &block)` - Create scanner with optional rule and namespace, auto-cleanup with block
-- `#add_rule(rule_string, namespace: nil)` - Add a YARA rule to the scanner
-- `#compile` - Compile all added rules (required before scanning)
-- `#scan(data, &block)` - Scan data and return ScanResults, or yield each result to block
-- `#close` - Free scanner resources
+**Key Methods**: `Yara.test()`, `Yara.scan()`, `Scanner.open()`, `Scanner#scan()`, `ScanResult#pattern_matches`
 
-### ScanResult class
-
-- `#match?` - Returns true if rule matched
-- `#rule_name` - Name of the matched rule
-- `#rule_meta` - Hash of rule metadata (keys are symbols)
-- `#rule_strings` - Hash of rule strings (keys are symbols with $ prefix)
+**📖 For detailed API documentation, examples, and advanced usage patterns, see [USAGE.md](USAGE.md).**
 
 ## Installing YARA-X
 
@@ -170,12 +79,8 @@ See [DEVELOPMENT.md](DEVELOPMENT.md) for detailed development setup instructions
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/jonmagic/yara-ffi. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/jonmagic/yara-ffi/blob/main/CODE_OF_CONDUCT.md).
+Bug reports and pull requests are welcome on GitHub at https://github.com/jonmagic/yara-ffi. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](CODE_OF_CONDUCT.md).
 
 ## License
 
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
-
-## Code of Conduct
-
-Everyone interacting in the yara-ffi project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/jonmagic/yara-ffi/blob/main/CODE_OF_CONDUCT.md).
+The gem is available as open source under the terms of the [MIT License](LICENSE.txt).