yara-ffi 3.1.0 → 4.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8093d210271e152cfd1d66f69169f9efaf46838ad93dd5473a0b11a242bc0164
-  data.tar.gz: a73687b9d7a3e7d098d4706c7c558954f1963a8bf8b3206984cdccd8dd757d2d
+  metadata.gz: fbff0f40a5c38903311cc68831538ee87fd1e88089188187560c25b401886ac4
+  data.tar.gz: c554613d1be85c26e2e7ddc87ac9509a60f7b5b25296de64017b7b067ef13c4f
 SHA512:
-  metadata.gz: 879454866631eb296874a07aed2848a3252d547b45b593245114a0bb57404b5bea0badfecb553f77bd0029727112bfd9cde527f8722c3dfcf55e73f16a0cb6f7
-  data.tar.gz: 6cc31a9d0e330897c3133fd17d90ddac3e7b54a1b073af31235185e3e70a2738fa8041e9adc8c087e5867d34dc4ffaed3f75c7c54d4be08bdfe60059d5c0e672
+  metadata.gz: f09c36c15253e02a2267373561064434b77fc9dcacdd445a1ec28357fa4e01cc09940496f5b645b1dbd920bc2f6c822d8d8215361ff71c8bebe73f42e01897ce
+  data.tar.gz: 52e22658f2b1eb0f9adddc3642db98f715ed0db7c2d8291808547fa4be86e9f088998e3785204fe1655f2935d5e3dff62588f285120c9375a1603da36458193b

data/.github/copilot-instructions.md

@@ -0,0 +1,148 @@
+# yara-ffi AI Coding Instructions
+
+This Ruby gem provides FFI bindings to YARA-X (Rust-based YARA implementation) for malware/pattern detection.
+
+## Quick Development Guide
+
+**Start Here for New Features:**
+1. Run `script/test` (if Docker image missing, run `script/bootstrap` first)
+2. Follow **Red-Green-Refactor** cycle with small semantic commits after each cycle
+3. Scanner lifecycle: `add_rule()` → `compile()` → `scan()` → `close()`
+4. Always use resource-safe patterns: `Scanner.open { |s| ... }` or manual `close()`
+5. Interactive testing: `docker run -it --mount type=bind,src="$(pwd)",dst=/app yara-ffi bin/console`
+
+## Core Components (Read These Files First)
+
+- `lib/yara/scanner.rb`: Main API - compile-then-scan workflow, resource management
+- `lib/yara/ffi.rb`: Raw FFI bindings with error codes (`YRX_SUCCESS = 0`)
+- `lib/yara/scan_result.rb`: Result parsing (temporary regex-based metadata extraction)
+- Tests in `test/scanner_test.rb`: Working examples of all patterns
+
+## Critical FFI Patterns
+
+**Memory Management (ALWAYS Required):**
+```ruby
+# Preferred - auto-cleanup
+Scanner.open(rule_string) do |scanner|
+  scanner.compile
+  results = scanner.scan(data)
+end
+
+# Manual - MUST call close()
+scanner = Scanner.new
+# ... use scanner
+scanner.close  # Memory leak without this!
+```
+
+**Error Handling - Check These First:**
+```ruby
+result = Yara::FFI.yrx_compile(@rule_source, @rules_pointer)
+if result != Yara::FFI::YRX_SUCCESS
+  error_msg = Yara::FFI.yrx_last_error
+  raise CompilationError, "Failed: #{error_msg}"
+end
+```
+
+**Library Loading Strategy (Multiple Fallbacks):**
+```ruby
+ffi_lib "/usr/local/lib/aarch64-linux-gnu/libyara_x_capi.so"  # Specific first
+ffi_lib "yara_x_capi"  # System library fallback
+```
+
+## Development Environment
+
+**Docker-First Development:** All development happens in Docker container with YARA-X pre-built:
+- `script/test` - runs tests (builds image automatically if needed)
+- `script/bootstrap` - only run if `script/test` fails due to missing Docker image
+- Interactive: `docker run -it --mount type=bind,src="$(pwd)",dst=/app yara-ffi bin/console`
+
+**TDD Workflow:** Follow Red-Green-Refactor with small semantic commits:
+1. **Red**: Write failing test
+2. **Green**: Make test pass with minimal code
+3. **Refactor**: Clean up while keeping tests green
+4. **Commit**: Small semantic commit describing the feature/fix
+
+**Testing:** Uses Minitest. Tests in `test/` directory focus on Scanner lifecycle and rule matching.
+
+## Common YARA Rule Patterns
+
+**Basic Rule Template:**
+```ruby
+rule = <<-RULE
+rule ExampleRule
+{
+  meta:
+    description = "Example rule"
+    author = "test"
+
+  strings:
+    $text = "pattern"
+    $regex = /regex pattern/
+
+  condition:
+    $text or $regex
+}
+RULE
+```
+
+**Multiple Rules Pattern:**
+```ruby
+scanner = Scanner.new
+scanner.add_rule(rule1)
+scanner.add_rule(rule2)
+scanner.compile
+results = scanner.scan(data)  # Returns array of ScanResult objects
+```
+
+## Code Patterns
+
+**Resource Management:**
+```ruby
+# Preferred block pattern
+Scanner.open(rule_string) do |scanner|
+  scanner.compile
+  results = scanner.scan(data)
+end  # Auto-cleanup
+
+# Manual pattern - must call close()
+scanner = Scanner.new
+scanner.add_rule(rule)
+scanner.compile
+# ... use scanner
+scanner.close  # Required!
+```
+
+**Error Handling:** Custom exceptions for different failure modes:
+- `CompilationError` - YARA rule syntax issues
+- `ScanError` - Runtime scanning failures
+- `NotCompiledError` - Scanning before compilation
+
+**Metadata Parsing:** ScanResult parses YARA rule metadata and strings via regex from rule source (temporary solution until YARA-X API improvements).
+
+## Adding New FFI Functions
+
+**Pattern to Follow:**
+```ruby
+# In lib/yara/ffi.rb
+attach_function :yrx_new_function, [:param_types], :return_type
+
+# In lib/yara/scanner.rb - always check return codes
+result = Yara::FFI.yrx_new_function(params)
+if result != Yara::FFI::YRX_SUCCESS
+  error_msg = Yara::FFI.yrx_last_error
+  raise ScanError, "Operation failed: #{error_msg}"
+end
+```
+
+**Available FFI Functions (key ones):**
+- `yrx_compile(src, rules_ptr)` - Compile rules from string
+- `yrx_scanner_create(rules, scanner_ptr)` - Create scanner from compiled rules
+- `yrx_scanner_scan(scanner, data, len)` - Scan data
+- `yrx_last_error()` - Get last error message
+- Cleanup: `yrx_rules_destroy()`, `yrx_scanner_destroy()`
+
+## Dependencies & Constraints
+
+**Docker Dependencies:** Container includes Rust toolchain + cargo-c for building YARA-X from source.
+
+When adding features, maintain the resource-managed Scanner pattern and ensure proper C memory cleanup.

data/.github/workflows/ruby.yml

@@ -1,10 +1,4 @@
-# This workflow uses actions that are not certified by GitHub.
-# They are provided by a third-party and are governed by
-# separate terms of service, privacy policy, and support
-# documentation.
-# This workflow will download a prebuilt Ruby version, install dependencies and run tests with Rake
-# For more information see: https://github.com/marketplace/actions/setup-ruby-jruby-and-truffleruby
-
+# GitHub Actions workflow for Ruby gem testing and validation
 name: Ruby
 
 on:
@@ -13,28 +7,86 @@ on:
   pull_request:
     branches: [ main ]
 
+permissions:
+  contents: read
+
 jobs:
   test:
-
+    name: Test Ruby ${{ matrix.ruby-version }}
     runs-on: ubuntu-latest
+
     strategy:
+      fail-fast: false
       matrix:
-        ruby-version: ['2.6', '2.7', '3.0']
+        ruby-version: ['3.2', '3.3']
 
     steps:
-    - uses: actions/checkout@v2
+    - name: Checkout code
+      uses: actions/checkout@v4
+
     - name: Set up Ruby
-    # To automatically get bug fixes and new Ruby versions for ruby/setup-ruby,
-    # change this to (see https://github.com/ruby/setup-ruby#versioning):
       uses: ruby/setup-ruby@v1
       with:
         ruby-version: ${{ matrix.ruby-version }}
         bundler-cache: true # runs 'bundle install' and caches installed gems automatically
-    - name: Install dependencies
+
+    - name: Install system dependencies
       run: |
         sudo apt-get update -y
-        sudo apt-get install -y libyara-dev
-        sudo gem install bundler -v 2.2.14
-        bundle install
+        sudo apt-get install -y curl git unzip build-essential
+
+    - name: Install Rust, cargo-c, and build YARA-X C API library
+      run: |
+        # Install Rust
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+        source $HOME/.cargo/env
+
+        # Set default Rust toolchain
+        rustup default stable
+
+        # Install cargo-c
+        cargo install cargo-c
+
+        # Build and install YARA-X C API library
+        git clone --depth 1 --branch v1.5.0 https://github.com/VirusTotal/yara-x.git /tmp/yara-x
+        cd /tmp/yara-x
+
+        # Ensure rustup default is set before running cargo with sudo
+        source $HOME/.cargo/env
+        rustup default stable
+        sudo env "PATH=$HOME/.cargo/bin:$PATH" "RUSTUP_HOME=$HOME/.rustup" "CARGO_HOME=$HOME/.cargo" $HOME/.cargo/bin/cargo cinstall -p yara-x-capi --release
+        sudo ldconfig
     - name: Run tests
-      run: bundle exec rake
+      run: bundle exec rake test
+
+    - name: Run RuboCop (if present)
+      run: |
+        if bundle list rubocop > /dev/null 2>&1; then
+          bundle exec rubocop
+        else
+          echo "RuboCop not found, skipping..."
+        fi
+      continue-on-error: true
+
+  # lint:
+  #   name: Lint and Security Check
+  #   runs-on: ubuntu-latest
+
+  #   steps:
+  #   - name: Checkout code
+  #     uses: actions/checkout@v4
+
+  #   - name: Set up Ruby
+  #     uses: ruby/setup-ruby@v1
+  #     with:
+  #       ruby-version: '3.3'
+  #       bundler-cache: true
+
+  #   - name: Run bundle audit (if present)
+  #     run: |
+  #       if bundle list bundle-audit > /dev/null 2>&1; then
+  #         bundle exec bundle-audit check --update
+  #       else
+  #         echo "bundle-audit not found, skipping..."
+  #       fi
+  #     continue-on-error: true

data/CHANGELOG.md

@@ -1,5 +1,63 @@
 ## [Unreleased]
 
+## [4.0.0] - 2025-08-19
+
+- **BREAKING**: Migrated from legacy libyara FFI bindings to YARA-X C API (`libyara_x_capi.so`) ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+  - Removed all legacy FFI struct definitions (`YrRule`, `YrMeta`, `YrString`, etc.)
+  - Replaced incremental rule compilation with single-step compilation via `yrx_compile`
+  - Eliminated dependency on `Yara.start` and `Yara.stop` lifecycle methods
+- **BREAKING**: Changed `Scanner#call` to `Scanner#scan` method ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- **BREAKING**: Require Ruby >= 3.0.0 ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- **BREAKING**: Remove `ScanResult` return for non-matching scans ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- Added `Yara::ScanResults` enumerable collection for managing scan results ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- Added `Scanner.open` for block-based resource management ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- Added streaming scan API support with block yielding ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- Modernized CI workflow with Ruby 3.0-3.3 matrix testing and YARA-X build support ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- Added comprehensive development documentation in `DEVELOPMENT.md` ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- Updated Docker environment to Ruby 3.3 with YARA-X v1.5.0 ([#24](https://github.com/jonmagic/yara-ffi/pull/24))
+- Improved error handling for compilation and scanning with better exception handling
+- Preserved backward compatibility in `ScanResult` interface via fallback parsing
+- Removed obsolete helper files: `user_data.rb`, `yr_meta.rb`, `yr_string.rb`, `yr_namespace.rb`, `yr_rule.rb`
+
+## [3.1.0] - 2022-04-18
+
+- Minor documentation fix for `Scanner::call` return value ([#20](https://github.com/jonmagic/yara-ffi/pull/20))
+- Fix FFI type compatibility issues on ARM64 Linux by converting integer types ([#21](https://github.com/jonmagic/yara-ffi/pull/21))
+
+## [3.0.0] - 2021-10-21
+
+- **BREAKING**: Introduced new `Yara::Scanner` API for better memory management and control ([#17](https://github.com/jonmagic/yara-ffi/pull/17))
+- Added proper memory cleanup with `yr_compiler_destroy` and `yr_rules_destroy` calls
+- Moved core functionality to `Yara::Scanner` class
+
+## [2.1.1] - 2021-08-31
+
+- Fix memory leak by calling destroy methods ([#11](https://github.com/jonmagic/yara-ffi/pull/11))
+
+## [2.1.0] - 2021-08-30
+
+- Use struct hash access and `Struct.ptr` where possible ([#14](https://github.com/jonmagic/yara-ffi/pull/14))
+- Improved struct member access and performance optimizations
+
+## [2.0.1] - 2021-08-30
+
+- Bug fixes and improvements
+
+## [2.0.0] - 2021-08-24
+
+- **BREAKING**: Changed interface to support rule metas ([#4](https://github.com/jonmagic/yara-ffi/pull/4))
+- `Yara.test` now returns `Yara::ScanResult` objects instead of rule names
+- Added support for accessing rule metadata as hash of name => value
+- Return rule metas in scan results
+
+## [1.0.0] - 2021-08-16
+
+- Wire up basic Yara functionality ([#3](https://github.com/jonmagic/yara-ffi/pull/3))
+- Added `Yara.test(rules_string, string_to_scan)` functionality
+- Initial FFI bindings to libyara
+
 ## [0.1.0] - 2021-03-11
 
-- Initial release
+- Initial release with project structure ([#1](https://github.com/jonmagic/yara-ffi/pull/1), [#2](https://github.com/jonmagic/yara-ffi/pull/2))
+- Set up GitHub Actions CI
+- Configured RuboCop

data/DEVELOPMENT.md

@@ -0,0 +1,188 @@
+# Development Guide
+
+This guide covers setting up the development environment and working on the yara-ffi gem.
+
+## Requirements
+
+- Docker (for containerized development environment)
+
+## Quick Start
+
+After checking out the repo, run the bootstrap script to set up the development environment:
+
+```bash
+script/bootstrap
+```
+
+This will build a Docker image with all the necessary dependencies, including the YARA-X C API library.
+
+## Development Scripts
+
+The project includes several convenience scripts in the `script/` directory:
+
+- `script/bootstrap` - Sets up the development environment (builds Docker image)
+- `script/test` - Runs the test suite in the Docker container
+
+## Running Tests
+
+To run the full test suite:
+
+```bash
+script/test
+```
+
+This runs `bundle exec rake` inside the Docker container with all dependencies properly configured.
+
+You can also run tests manually inside the container:
+
+```bash
+docker run -it --mount type=bind,src="$(pwd)",dst=/app yara-ffi bundle exec rake
+```
+
+Or run specific test files:
+
+```bash
+docker run -it --mount type=bind,src="$(pwd)",dst=/app yara-ffi bundle exec ruby -Itest test/scanner_test.rb
+```
+
+Alternatively, you can use rake to run specific tests:
+
+```bash
+docker run -it --mount type=bind,src="$(pwd)",dst=/app yara-ffi bundle exec rake test TEST=test/scanner_test.rb
+```
+
+## Interactive Development
+
+For an interactive development session, you can start a console in the container:
+
+```bash
+docker run -it --mount type=bind,src="$(pwd)",dst=/app yara-ffi bin/console
+```
+
+This gives you an IRB session with the gem loaded for experimentation.
+
+## Development Environment Details
+
+The development environment uses Docker to provide a consistent setup with:
+
+- Ruby 3.3 (latest stable)
+- YARA-X C API library v1.5.0 built from source
+- All necessary system dependencies
+- Bundler with locked gem versions
+
+### Docker Image
+
+The `Dockerfile` sets up:
+
+1. Base Ruby 3.3 image
+2. System dependencies (curl, git, unzip)
+3. Rust toolchain and cargo-c for building YARA-X
+4. YARA-X C API library compiled and installed
+5. Ruby gem dependencies via Bundler
+
+### Manual Setup (without Docker)
+
+If you prefer not to use Docker, you'll need to manually install:
+
+1. Ruby 3.0+
+2. YARA-X C API library (see [Installation section in README](README.md#installing-yara-x))
+3. System dependencies for building native gems
+
+Then run:
+
+```bash
+bundle install
+rake test
+```
+
+## Code Structure
+
+The gem is organized as follows:
+
+- `lib/yara.rb` - Main entry point and convenience methods
+- `lib/yara/ffi.rb` - FFI bindings to YARA-X C API
+- `lib/yara/scanner.rb` - Scanner class for rule compilation and scanning
+- `lib/yara/scan_result.rb` - Individual scan result wrapper
+- `lib/yara/scan_results.rb` - Collection of scan results
+- `lib/yara/version.rb` - Gem version constant
+
+## Testing
+
+Tests are located in the `test/` directory:
+
+- `test/yara_test.rb` - Tests for main module convenience methods
+- `test/scanner_test.rb` - Tests for Scanner class functionality
+- `test/test_helper.rb` - Shared test setup and utilities
+
+The test suite uses Minitest and includes tests for:
+
+- Rule compilation and validation
+- Data scanning with various rule types
+- Memory management and resource cleanup
+- Error handling and edge cases
+
+## Release Process
+
+To release a new version of the gem:
+
+1. Update the version number in `lib/yara/version.rb`
+2. Update the `CHANGELOG.md` with release notes
+3. Commit the changes
+4. Create and push a git tag:
+   ```bash
+   git tag v<version>
+   git push origin v<version>
+   ```
+5. Build and push the gem:
+   ```bash
+   gem build yara-ffi.gemspec
+   gem push yara-ffi-<version>.gem
+   ```
+
+## Contributing Guidelines
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b my-new-feature`)
+3. Make your changes with appropriate tests
+4. Run the test suite (`script/test`) to ensure all tests pass
+5. Commit your changes (`git commit -am 'Add some feature'`)
+6. Push to the branch (`git push origin my-new-feature`)
+7. Create a Pull Request
+
+Please ensure your code follows the existing style and includes tests for new functionality.
+
+## Debugging
+
+For debugging FFI-related issues:
+
+1. Enable FFI debugging by setting the `RUBY_FFI_DEBUG` environment variable
+2. Use `puts` statements or `binding.pry` (if pry is available) for Ruby debugging
+3. Check YARA-X C API documentation for expected behavior
+4. Verify memory management - ensure all resources are properly freed
+
+## Common Issues
+
+### YARA-X Library Not Found
+
+If you see errors about missing YARA-X libraries, ensure:
+
+1. The YARA-X C API library is properly installed
+2. The library path is in your system's library search path
+3. You're using the correct version (v1.5.0 is tested)
+
+### Docker Build Issues
+
+If Docker builds fail:
+
+1. Ensure you have sufficient disk space
+2. Try rebuilding without cache: `docker build --no-cache . -t yara-ffi`
+3. Check internet connectivity for downloading dependencies
+
+### Test Failures
+
+If tests fail:
+
+1. Ensure all dependencies are properly installed
+2. Check that YARA-X C API library is available
+3. Verify Ruby version compatibility (3.0+ required)
+4. Run tests individually to isolate issues

data/Dockerfile

@@ -1,17 +1,25 @@
-FROM ruby:2.6.6
 
-RUN apt-get update -qq
-RUN apt-get install -y flex bison
+FROM ruby:3.3
+
+RUN apt-get update -qq \
+  && apt-get install -y curl git unzip
 
 WORKDIR /app
 
 COPY . ./
-RUN gem install bundler:2.2.15
-RUN bundle install
+RUN gem install bundler:2.2.15 \
+  && bundle install
+
+# Install Rust and cargo-c for building YARA-X C API
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y \
+  && . $HOME/.cargo/env \
+  && cargo install cargo-c
+
+# Build and install YARA-X C API library
+RUN . $HOME/.cargo/env \
+  && git clone --depth 1 --branch v1.5.0 https://github.com/VirusTotal/yara-x.git /tmp/yara-x \
+  && cd /tmp/yara-x \
+  && cargo cinstall -p yara-x-capi --release \
+  && rm -rf /tmp/yara-x
 
-RUN git clone --recursive --branch v4.1.1 https://github.com/VirusTotal/yara.git /tmp/yara && \
-  cd /tmp/yara/ && \
-  ./bootstrap.sh && \
-  ./configure && \
-  make && \
-  make install
+ENV PATH="/usr/local/bin:$PATH"

data/Gemfile.lock

@@ -1,44 +1,59 @@
 PATH
   remote: .
   specs:
-    yara-ffi (3.0.0)
+    yara-ffi (4.0.0)
       ffi
 
 GEM
   remote: https://rubygems.org/
   specs:
-    ast (2.4.2)
+    ast (2.4.3)
     coderay (1.1.3)
-    ffi (1.15.5)
-    method_source (1.0.0)
-    minitest (5.14.4)
-    parallel (1.20.1)
-    parser (3.0.0.0)
+    ffi (1.17.2)
+    ffi (1.17.2-aarch64-linux-gnu)
+    ffi (1.17.2-arm64-darwin)
+    ffi (1.17.2-x86_64-darwin)
+    ffi (1.17.2-x86_64-linux-gnu)
+    json (2.13.2)
+    language_server-protocol (3.17.0.5)
+    lint_roller (1.1.0)
+    method_source (1.1.0)
+    minitest (5.25.5)
+    parallel (1.27.0)
+    parser (3.3.9.0)
       ast (~> 2.4.1)
-    pry (0.14.0)
+      racc
+    prism (1.4.0)
+    pry (0.15.2)
       coderay (~> 1.1)
       method_source (~> 1.0)
-    rainbow (3.0.0)
-    rake (13.0.3)
-    regexp_parser (2.1.1)
-    rexml (3.2.4)
-    rubocop (1.11.0)
+    racc (1.8.1)
+    rainbow (3.1.1)
+    rake (13.3.0)
+    regexp_parser (2.11.2)
+    rubocop (1.79.2)
+      json (~> 2.3)
+      language_server-protocol (~> 3.17.0.2)
+      lint_roller (~> 1.1.0)
       parallel (~> 1.10)
-      parser (>= 3.0.0.0)
+      parser (>= 3.3.0.2)
       rainbow (>= 2.2.2, < 4.0)
-      regexp_parser (>= 1.8, < 3.0)
-      rexml
-      rubocop-ast (>= 1.2.0, < 2.0)
+      regexp_parser (>= 2.9.3, < 3.0)
+      rubocop-ast (>= 1.46.0, < 2.0)
       ruby-progressbar (~> 1.7)
-      unicode-display_width (>= 1.4.0, < 3.0)
-    rubocop-ast (1.4.1)
-      parser (>= 2.7.1.5)
-    ruby-progressbar (1.11.0)
-    unicode-display_width (2.0.0)
+      unicode-display_width (>= 2.4.0, < 4.0)
+    rubocop-ast (1.46.0)
+      parser (>= 3.3.7.2)
+      prism (~> 1.4)
+    ruby-progressbar (1.13.0)
+    unicode-display_width (3.1.5)
+      unicode-emoji (~> 4.0, >= 4.0.4)
+    unicode-emoji (4.0.4)
 
 PLATFORMS
   aarch64-linux
   arm64-darwin-21
+  ruby
   x86_64-darwin-19
   x86_64-linux
 
@@ -50,4 +65,4 @@ DEPENDENCIES
   yara-ffi!
 
 BUNDLED WITH
-   2.2.32
+   2.7.1