compliance_engine 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bbedee4752b0b7fd84e25194dd8fca02ce5bcf95fe9e36cadae328d82914da63
-  data.tar.gz: 53536b5e5f5d53ba4d6d7a7679bee7f1b4256e57d7877562f2a336c53588d3ba
+  metadata.gz: ca4c111afd1a2840ec2a266bea8e12f195a9cc8c262eef7676280382b63868b5
+  data.tar.gz: a59bc32c198332f65882d2a3950cfa2a89c8f48d0da58bf93d77d9e2cc555427
 SHA512:
-  metadata.gz: 15a7b6c0d11717ff5b0da2f89fd14207bda9f6a245a0e424a63f0693f8add5aa48e7798911b1058071f024ce45cebc5ce24266302c3dd22c4db402fb7dece58e
-  data.tar.gz: 9b4354ce4d0bc54fd2284f6c9ff082eb135d1e342e6889673341ff1f66965ad60de504d53638abb05bd4cf7b09d23649e6cf56024da4639068a5f1e3e0b64065
+  metadata.gz: 6503e8068deaec3d98bb6783689ae599fa149b3fe2299f1e101ca23f46488db680f4685f36a7670f68c3480977ce9ed539c96d0bb5b6df0f952ea5267d3ed935
+  data.tar.gz: 0a0afadab618fa6d86fece7ebb270342e59b788d84b8e321f97c053758a8d0e2b7e68127ccaf4228e2ba65978566fd6ff3bec05b137f829e58ebfc15314b937e

data/AGENTS.md

@@ -0,0 +1,134 @@
+# AGENTS.md
+
+This file provides guidance to AI agents when working with code in this repository.
+
+## Overview
+
+This is a Ruby gem (`compliance_engine`) that parses and works with [Sicura/SIMP Compliance Engine (SCE)](https://simp-project.com/docs/sce/) data. It also ships as a Puppet module providing a Hiera backend (`compliance_engine::enforcement`) for enforcing compliance profiles in Puppet environments.
+
+## Commands
+
+### Testing
+```bash
+# Run all tests and rubocop (default task)
+bundle exec rake
+
+# Run just spec tests (with fixture prep/cleanup)
+bundle exec rake spec
+
+# Run spec tests standalone (no fixture prep)
+bundle exec rake spec:standalone
+
+# Run rubocop linting
+bundle exec rake rubocop
+
+# Run a single spec file
+bundle exec rspec spec/classes/compliance_engine/data_spec.rb
+
+# Run tests in parallel (used in CI for Ruby < 4.0)
+bundle exec rake parallel_spec
+```
+
+### Development
+```bash
+# Install dependencies
+bundle install
+
+# Open interactive shell with compliance data loaded
+bundle exec compliance_engine inspect --module /path/to/module
+
+# CLI usage examples
+bundle exec compliance_engine profiles --modulepath /path/to/modules
+bundle exec compliance_engine hiera --profile my_profile --modulepath /path/to/modules
+bundle exec compliance_engine lookup some::class::param --profile my_profile --module /path/to/module
+```
+
+## Architecture
+
+### Data Model
+
+Compliance data lives in YAML/JSON files at `<module>/SIMP/compliance_profiles/*.yaml` or `<module>/simp/compliance_profiles/*.yaml`. Files are structured with four top-level keys: `profiles`, `ce` (Compliance Elements), `checks`, and `controls`.
+
+The library models this data with a two-layer class hierarchy:
+
+**Collections** (`ComplianceEngine::Collection` subclass) hold named groups of components:
+- `ComplianceEngine::Profiles` — keyed by `'profiles'` in source data
+- `ComplianceEngine::Ces` — keyed by `'ce'` in source data
+- `ComplianceEngine::Checks` — keyed by `'checks'` in source data
+- `ComplianceEngine::Controls` — keyed by `'controls'` in source data
+
+**Components** (`ComplianceEngine::Component` subclass) represent individual named entries within those collections:
+- `ComplianceEngine::Profile` — a named compliance profile
+- `ComplianceEngine::Ce` — a Compliance Element (CE)
+- `ComplianceEngine::Check` — a single compliance check; only `type: puppet-class-parameter` checks produce Hiera data via `Check#hiera`
+- `ComplianceEngine::Control` — a compliance control
+
+A component can have multiple **fragments** (one per source file), which are deep-merged together via `deep_merge`. Confinement logic in `Component` filters fragments based on Puppet facts, module presence/version, and remediation risk level.
+
+### Central Data Object
+
+`ComplianceEngine::Data` is the primary entry point. It:
+1. Loads files via `open(*paths)` which delegates to `ModuleLoader` → `DataLoader::Yaml/Json`
+2. Uses Ruby's `Observable` pattern — `DataLoader` objects notify `Data` of changes
+3. Lazily constructs and caches the four collection objects; invalidates all caches when facts, enforcement_tolerance, modulepath, or environment_data change
+4. Exposes `Data#hiera(profiles)` which walks the check_mapping of requested profiles to produce a flat Hiera-compatible hash
+
+### Business Logic: From Profiles to Hiera
+
+**`Data#hiera(profile_names)`** is the primary output method. It:
+1. Resolves each name to a `Profile` object (logs and skips unknown names).
+2. Calls `Data#check_mapping(profile)` for each profile to find all associated checks.
+3. Filters to checks with `type: 'puppet-class-parameter'`.
+4. Calls `Check#hiera` on each, which returns `{ settings['parameter'] => settings['value'] }`.
+5. Deep-merges all results into a single flat hash and caches it.
+
+**`Data#check_mapping(profile_or_ce)`** is the correlation engine that links profiles (or CEs) to checks. A check is included if **any** of the following hold (evaluated via `Data#mapping?`):
+
+| Condition | What it checks |
+|-----------|---------------|
+| Shared **control** | `check.controls` and `profile.controls` share a key set to `true` |
+| Shared **CE** | `check.ces` and `profile.ces` share a key set to `true` |
+| CE→Control overlap | Any of `check.ces`' CEs has a control that also appears in `profile.controls` |
+| Direct reference | `profile.checks[check_key]` is truthy |
+
+`check_mapping` can also be called with CE objects (in addition to profiles). Results are cached by `"#{object.class}:#{object.key}"`.
+
+### Loading Pipeline
+
+```
+paths → EnvironmentLoader → ModuleLoader (one per module dir)
+                                      → DataLoader::Yaml / DataLoader::Json
+                                              ↓ (Observable notify)
+                                        ComplianceEngine::Data#update
+```
+
+- `EnvironmentLoader` scans a Puppet modulepath for module directories
+- `EnvironmentLoader::Zip` handles zip-archived environments
+- `ModuleLoader` reads a module's `metadata.json` and discovers compliance data files
+- `DataLoader` (and its subclasses) read and parse individual files; they use the Observable pattern to push updates to `Data`
+
+### Puppet Hiera Backend
+
+`lib/puppet/functions/compliance_engine/enforcement.rb` implements the Hiera `lookup_key` function. It:
+- Resolves profiles from `compliance_engine::enforcement` and optionally `compliance_markup::enforcement` Hiera keys
+- Creates and caches a `ComplianceEngine::Data` object on the Puppet lookup context
+- Calls `data.hiera(profiles)` and bulk-caches results for subsequent lookups
+- Supports `compliance_markup` backwards compatibility via `compliance_markup_compatibility` option
+
+### Confinement and Enforcement Tolerance
+
+`Component#fragments` filters source fragments based on:
+- **Fact confinement** (`confine` key): dot-notation Puppet facts (e.g. `os.release.major`). Values may be a string (exact match), a string prefixed with `!` (negation), or an array (any match). Implemented in `Component#fact_match?`. Fact confinement is skipped when `facts` is `nil`.
+- **Module confinement** (`confine.module_name` + `confine.module_version`): checks against `environment_data` (a `{module_name => version}` hash) using semantic versioning. Module confinement only runs when `environment_data` is set (e.g. by `ComplianceEngine::Data#open`).
+- **Remediation risk** (`remediation.risk`): when `enforcement_tolerance` is a positive `Integer`, drops fragments where risk level ≥ `enforcement_tolerance` and drops disabled remediations. Only applies to `Check` components.
+
+In practice, only fact confinement is bypassed when `facts` is `nil`; module confinement still applies whenever `environment_data` is available. All confinement and risk/disabled-remediation filtering are effectively bypassed only when both `facts` and `environment_data` are unset and `enforcement_tolerance` is not a positive `Integer` (every fragment is then included). This is useful for offline analysis where system context and enforcement settings are unavailable.
+
+### Code Style
+
+Rubocop is configured via `.rubocop.yml` inheriting from `voxpupuli-test`. Key style choices:
+- `compact` class/module nesting style (e.g. `class ComplianceEngine::Data` not nested modules)
+- Trailing commas on multiline args/arrays
+- Leading dot position for method chaining
+- `braces_for_chaining` block delimiters
+- Max line length: 200

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+### 0.3.0 / 2026-03-19
+* Hash-like Collection methods return Collection objects (#37)
+
+### 0.2.2 / 2026-03-02
+* Ensure that cloned/duped objects get independent collection instances
+
 ### 0.2.1 / 2026-02-12
 * Turn off debugging in compliance_engine::enforcement function
 

data/README.md

@@ -42,6 +42,54 @@ Options:
 
 See the [`ComplianceEngine::Data`](https://rubydoc.info/gems/compliance_engine/ComplianceEngine/Data) class for details.
 
+## Concepts
+
+### Data Model
+
+Compliance data is expressed across four entity types that live in YAML/JSON files inside Puppet modules (`<module>/SIMP/compliance_profiles/*.yaml`):
+
+| Entity | Key | Purpose |
+|--------|-----|---------|
+| **Profile** | `profiles` | A named compliance standard (e.g. `nist_800_53_rev4`). References CEs, checks, and/or controls that together constitute that standard. |
+| **CE** (Compliance Element) | `ce` | A single, named compliance capability (e.g. "enable audit logging"). Bridges profiles to checks via a shared vocabulary. |
+| **Check** | `checks` | A verifiable assertion about a system setting. Checks of `type: puppet-class-parameter` carry a `parameter` and `value` that become Hiera data. |
+| **Control** | `controls` | A cross-reference label from an external framework (e.g. `nist_800_53:rev4:AU-2`). Profiles and checks both annotate themselves with controls to express alignment. |
+
+### From Profiles to Hiera Data
+
+The central operation of the library is `Data#hiera(profiles)`, which converts a list of profile names into a flat hash of Puppet class parameters and their enforced values:
+
+```
+profile names
+    ↓  check_mapping: find all checks that belong to each profile
+checks (type: puppet-class-parameter only)
+    ↓  Check#hiera: extract { 'class::param' => value }
+deep-merged hash  →  { 'widget_spinner::audit_logging' => true, ... }
+```
+
+**How check_mapping works** — a check is considered part of a profile if any of the following are true:
+
+1. The check and profile share a **control** label (`nist_800_53:rev4:AU-2`).
+2. The check and profile share a **CE** reference.
+3. The check's CE and the profile share a **control** label.
+4. The profile explicitly lists the check by key under its `checks:` map.
+
+This layered matching lets compliance authors express mappings at different levels of abstraction and have the engine resolve them automatically.
+
+### Confinement
+
+A component (profile, CE, check, or control) may be defined across multiple source files. Each file contributes a **fragment**. Before fragments are merged, they are filtered by:
+
+- **Facts** (`confine:` key): dot-notation Puppet facts, optionally negated with a `!` prefix. A fragment is dropped if its confinement does not match the current system's facts.
+- **Module presence/version** (`confine.module_name` / `confine.module_version`): fragment is dropped if the required module is absent or the wrong version.
+- **Remediation risk/status** (`remediation.risk` / `remediation.disabled`): when `enforcement_tolerance` is set to a positive Integer, a fragment is dropped if remediation is explicitly `disabled` or if its risk level is ≥ `enforcement_tolerance`.
+
+If `facts` is `nil`, all fact/module confinement is skipped; fragments are still subject to remediation-based filtering when `enforcement_tolerance` is set.
+
+### Enforcement Tolerance
+
+`enforcement_tolerance` is an optional integer threshold that controls how cautiously the engine applies remediations. When it is set to a positive Integer, fragments whose `remediation.risk.level` meets or exceeds the threshold, or whose remediation is explicitly `disabled`, are silently excluded from the merged result, allowing operators to tune aggressiveness (e.g. apply only low-risk remediations in production, all remediations in a test environment). When `enforcement_tolerance` is `nil` or not a positive Integer, no remediation-based filtering occurs and `remediation.risk` / `remediation.disabled` do not affect fragment inclusion.
+
 ## Using as a Puppet Module
 
 The Compliance Engine can be used as a Puppet module to provide a Hiera backend for compliance data. This allows you to enforce compliance profiles through Hiera lookups within your Puppet manifests.

data/lib/compliance_engine/collection.rb

@@ -32,6 +32,27 @@ class ComplianceEngine::Collection
     nil
   end
 
+  # Ensure that cloned/duped objects get independent component instances.
+  #
+  # Ruby's default clone/dup is a shallow copy, so @collection would be
+  # shared between the source and the copy, and every Component inside it
+  # would be the same object.  Calling invalidate_cache on one copy would
+  # then propagate its facts into those shared components, silently
+  # affecting the other copy's view of the data.
+  #
+  # Duping each Component (which triggers Component#initialize_copy) gives
+  # every copy of the Collection its own independent components.  Any
+  # derived caches on the Collection itself (e.g. @by_oval_id in Ces) are
+  # cleared so each copy rebuilds them from its own component set.
+  #
+  # @return [NilClass]
+  def initialize_copy(_source)
+    super
+    @collection = @collection.transform_values(&:dup)
+    (instance_variables - (context_variables + [:@collection])).each { |var| instance_variable_set(var, nil) }
+    nil
+  end
+
   # Converts the object to a hash representation
   #
   # @return [Hash] the hash representation of the object
@@ -46,6 +67,13 @@ class ComplianceEngine::Collection
     to_h.keys
   end
 
+  # Returns the values of the collection
+  #
+  # @return [Array] the values of the collection
+  def values
+    to_h.values
+  end
+
   # Return a single value from the collection
   #
   # @param key [String] the key of the value to return
@@ -57,22 +85,34 @@ class ComplianceEngine::Collection
   # Iterates over the collection
   #
   # @param block [Proc] the block to execute
+  # @return [self, Enumerator]
   def each(&block)
+    return to_enum(:each) unless block
+
     to_h.each(&block)
+    self
   end
 
   # Iterates over values in the collection
   #
   # @param block [Proc] the block to execute
+  # @return [self, Enumerator]
   def each_value(&block)
+    return to_enum(:each_value) unless block
+
     to_h.each_value(&block)
+    self
   end
 
   # Iterates over keys in the collection
   #
   # @param block [Proc] the block to execute
+  # @return [self, Enumerator]
   def each_key(&block)
+    return to_enum(:each_key) unless block
+
     to_h.each_key(&block)
+    self
   end
 
   # Return true if any of the values in the collection match the block
@@ -94,17 +134,35 @@ class ComplianceEngine::Collection
   # Select values in the collection
   #
   # @param block [Proc] the block to execute
-  # @return [Hash] the filtered hash
+  # @return [ComplianceEngine::Collection, Enumerator] the filtered collection or an Enumerator when no block is given
   def select(&block)
-    to_h.select(&block)
+    return to_enum(:select) unless block_given?
+
+    result = dup
+    result.collection = result.to_h.select(&block)
+    result
   end
 
+  alias filter select
+
   # Filter out values in the collection
   #
   # @param block [Proc] the block to execute
-  # @return [Hash] the filtered hash
+  # @return [ComplianceEngine::Collection, Enumerator] the filtered collection or an Enumerator when no block is given
   def reject(&block)
-    to_h.reject(&block)
+    return to_enum(:reject) unless block_given?
+
+    result = dup
+    result.collection = result.to_h.reject(&block)
+    result
+  end
+
+  # Transform values in the collection
+  #
+  # @param block [Proc] the block to execute
+  # @return [Hash, Enumerator] a hash with transformed values, or an Enumerator when no block is given
+  def transform_values(&block)
+    to_h.transform_values(&block)
   end
 
   private

data/lib/compliance_engine/component.rb

@@ -26,6 +26,29 @@ class ComplianceEngine::Component
     nil
   end
 
+  # Ensure that cloned/duped objects get independent fragment stores.
+  #
+  # Ruby's default clone/dup is a shallow copy, so @component (and the
+  # fragments hash nested within it) would be shared between the source
+  # and the copy.  Calling add on either would write into the same
+  # fragments hash, making new fragments visible on both objects.
+  # Pre-computed @element and @fragments caches would also be shared,
+  # returning stale fact-filtered results on the copy even after facts
+  # are changed.
+  #
+  # Duping @component and its inner fragments hash ensures each copy has
+  # an independent fragment store.  Cache variables are cleared so each
+  # copy rebuilds them lazily from its own fragments on first access.
+  #
+  # @return [NilClass]
+  def initialize_copy(_source)
+    super
+    @component = @component.dup
+    @component[:fragments] = @component[:fragments].transform_values { |fragment| Marshal.load(Marshal.dump(fragment)) }
+    cache_variables.each { |var| instance_variable_set(var, nil) }
+    nil
+  end
+
   # Adds a value to the fragments array of the component.
   #
   # @param value [Object] The value to be added to the fragments array.
@@ -250,7 +273,7 @@ class ComplianceEngine::Component
     @element = {}
 
     fragments.each_value do |fragment|
-      @element = DeepMerge.deep_merge!(fragment, @element)
+      @element = DeepMerge.deep_merge!(Marshal.load(Marshal.dump(fragment)), @element)
     end
 
     @element

data/lib/compliance_engine/data.rb

@@ -88,6 +88,42 @@ class ComplianceEngine::Data
     (instance_variables - (data_variables + context_variables)).each { |var| instance_variable_set(var, nil) }
   end
 
+  # Ensure that cloned/duped objects get independent collection instances.
+  #
+  # Ruby's default clone/dup is a shallow copy, so the collection instance
+  # variables (@ces, @profiles, @checks, @controls) would otherwise point to
+  # the same objects as the source.  When facts= is later called on either the
+  # source or the clone, invalidate_cache propagates facts into the shared
+  # collection, causing the other object to silently adopt the wrong facts.
+  #
+  # Nilling the collection variables here forces each clone to lazily rebuild
+  # its own collections the first time they are accessed, using its own context
+  # (facts, enforcement_tolerance, etc.).  Cache variables that depend on those
+  # collections are cleared for the same reason.
+  #
+  # @return [NilClass]
+  def initialize_copy(_source)
+    super
+    # Give each clone its own outer @data hash and its own per-file inner
+    # hashes so that new files opened on one clone (via open/update) are not
+    # visible to other clones or the source, and so that a loader refresh on
+    # the source (which mutates the inner hash in-place via Data#update) does
+    # not silently affect a clone that has not yet built its lazy collections.
+    # The inner per-file content values (read-only parsed data) stay shared.
+    #
+    # :loader is additionally cleared (set to nil) so the copy does not hold
+    # a reference to the source's DataLoader object.  If it did, the copy
+    # calling update(key_string) for an already-known file would invoke
+    # loader.refresh, which notifies the source (the registered Observable
+    # observer) and overwrites source.data[key][:content] while the copy's
+    # inner hash stays stale.  With :loader nil the copy creates its own
+    # independent loader (and registers itself as observer) on next access.
+    @data = @data.transform_values { |entry| entry.merge(loader: nil) }
+    collection_variables.each { |var| instance_variable_set(var, nil) }
+    cache_variables.each { |var| instance_variable_set(var, nil) }
+    nil
+  end
+
   # Scan a Puppet environment from a zip file
   # @param path [String] The Puppet environment archive file
   # @return [NilClass]
@@ -191,8 +227,13 @@ class ComplianceEngine::Data
     else
       data[filename.key] ||= {}
 
-      # Assume filename is a loader object
-      unless data[filename.key]&.key?(:loader)
+      # Register as an observer only when no loader is currently attached.
+      # Checking the :loader value (rather than key presence) is important
+      # after clone/dup: initialize_copy sets :loader to nil so the copy does
+      # not share the source's loader, but the key still exists.  Checking
+      # key presence would see the nil as "already registered" and skip
+      # add_observer, leaving the copy deaf to future loader refreshes.
+      unless data[filename.key][:loader]
         data[filename.key][:loader] = filename
         data[filename.key][:loader].add_observer(self, :update)
       end

data/lib/compliance_engine/data_loader.rb

@@ -20,13 +20,17 @@ class ComplianceEngine::DataLoader
 
   # Set the data for the data loader
   #
-  # @param value [Hash] The new value for the data loader
+  # The hash and all nested hashes, arrays, and strings within it are
+  # deep-frozen so that parsed compliance data is treated as read-only
+  # once loaded.  Callers must not retain a mutable reference to the
+  # hash after calling this method.
   #
+  # @param value [Hash] The new value for the data loader
   # @raise [ComplianceEngine::Error] If the value is not a Hash
   def data=(value)
     raise ComplianceEngine::Error, 'Data must be a hash' unless value.is_a?(Hash)
 
-    @data = value
+    @data = deep_freeze(value)
     changed
     notify_observers(self)
   end
@@ -43,4 +47,24 @@ class ComplianceEngine::DataLoader
     require 'securerandom'
     @key = "#{data.class}:#{SecureRandom.uuid}"
   end
+
+  private
+
+  # Recursively freezes a Hash or Array and all nested objects.
+  #
+  # Parsed compliance data is read-only once loaded; deep-freezing it makes
+  # that invariant explicit and surfaces any accidental in-place mutation
+  # immediately as a FrozenError rather than silent data corruption.
+  #
+  # @param obj [Object] the object to freeze
+  # @return [Object] the frozen object (modified in-place)
+  def deep_freeze(obj)
+    case obj
+    when Hash
+      obj.each_value { |v| deep_freeze(v) }
+    when Array
+      obj.each { |v| deep_freeze(v) }
+    end
+    obj.freeze
+  end
 end

data/lib/compliance_engine/version.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module ComplianceEngine
-  VERSION = '0.2.1'
+  VERSION = '0.3.0'
 
   # Handle supported compliance data versions
   class Version

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: compliance_engine
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - Steven Pritchard
@@ -120,6 +120,7 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
+- AGENTS.md
 - CHANGELOG.md
 - LICENSE
 - README.md
@@ -153,7 +154,7 @@ licenses:
 metadata:
   homepage_uri: https://simp-project.com/docs/sce/
   source_code_uri: https://github.com/simp/rubygem-simp-compliance_engine
-  changelog_uri: https://github.com/simp/rubygem-simp-compliance_engine/releases/tag/0.2.1
+  changelog_uri: https://github.com/simp/rubygem-simp-compliance_engine/releases/tag/0.3.0
   bug_tracker_uri: https://github.com/simp/rubygem-simp-compliance_engine/issues
 rdoc_options: []
 require_paths:
@@ -169,7 +170,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.3
+rubygems_version: 4.0.6
 specification_version: 4
 summary: Parser for Sicura Compliance Engine data
 test_files: []