loom-core 0.0.6 → 0.0.9

data/lib/loom/pattern/dsl.rb

@@ -1,13 +1,59 @@
+# IN Progress:
+# [wip-patternbuilder]
+# * Add a phase to the pattern execution sequence to collect calls to factset and loom
+#   objects. Results collected from this ""pre-execute"" can be analyzed for errors, optimization,
+#   success assertions, verification, etc. The loom file then executes in 2 passes, analyze &
+#   execute.
+#   -- pre-fact collection - inject the facts (or loom) object as a recorder
+#   -- (like a mock in record mode) # instead of the factual fact set. no need
+#   -- to change any loom files.
+#   -- only run fact providers which are accessed in the pattern set
+#   -- only load modules accessed in the pattern set
+#
+# * Replace Pattern "mods" and "mod specs" in 80 Loom::Pattern::ReferenceSet
+#   with usages of a builder instead. Currently the internal data model in
+#   ReferenceSet is confusing, but luckily it's the only client of pattern
+#   modules, that have used Loom::Pattern::DSL. Change calls to
+#   DSL#pattern/report/weave (anythin else that creates a pattern) to add a new
+#   PatternBuilder to the module. Use the builder to implement the TODO above
+#   ("Add a phase..."). Implement analysis on the builder.
+
+# [master]
+# * ... ongoing ... ways to test and +verify+ pattern execution
+#   - IDEA: launch a container on host before executing a pattern, w/ an overlay
+#     fs of "/". Use this to canary a command sequence before executing on the
+#     actual host. This way no rollback, or idempotence checks, or state
+#     management, or history needs to be kept. Only the volatile container image
+#     needs to be torn down.
+
 # TODO: DSL extensions:
-# - a way to test and verify pattern execution.... I still don't trust this
-#   enough. this starts with fixing error reporting.
+# - More Mods! .... ondeck:
+#   * bkblz
+#   * cassandra
+#   * apache (nginx?)
+#   * digital ocean
+#   * systemd-nspawj
+
+# - Models Future:
+#   Notice each ondeck module above (bkblz, cassanadra, apache, digital ocean,
+#   system-nspawn): covers a unique infrastructure area, i.e.: bkblz:object and
+#   cold storage, cassandra:hyper-scale data storage, apache/nginx:content
+#   serving, digital-ocean/aws/gcp:remote virtual hosting, containers:local
+#   virtual hosting
+#   I could generalize each of the above infrastructure area into a high level
+#   mod. Should I? Maybe not.
+
+# - auto mod documentation in the CLI
+
 # - Pattern+non_idempotent+ marks a pattern as explicitly not idempotent, this
 #   let's additional warnings and checks to be added
+
 # - A history module, store a log of each executed command, a hash of the .loom
 #   file, and the requisite facts (the let declarations) for each executed pattern
 #   on the host it executes. /var/log/loom/history? Create this log on startup.
 #   -- add a new set of history commands through the CLI and a history
 #      FactProvider exposing host/loom/pattern_slug execution stats.
+
 # - Provide automatic command reversion support with a =Module= DSL that ties in
 #   with local revision history.
 #   -- allow Module actions/mods to define an "undo" command of itself given the
@@ -24,6 +70,7 @@
 #      accesses to fact_set be done in let expressions (enforce this maybe?)
 #   -- Later... before/after hooks can ensure the entire loom execution sequence
 #      was "revertable"
+
 # - A mechanism to allow mods to register CLI flags and actions. Using an action predicate
 #   mechanisms can add flags at the global or action levels. All CLI flags set config values.
 #   -- Mods can also register for action namespaces similar to git. This is consistent with mod
@@ -31,33 +78,22 @@
 #   -- Best way is to migrate loom/mods/module and loom/mods/action_proxy into base classes of
 #      themselves. The isolate the shell specific behavior into a subclass of each to preserve the
 #      current behavior. A new "cli" module and "cli" action proxy would enable the implementation.
-# - Add a phase to the pattern execution sequence to collect calls to factset and loom
-#   objects. Results collected from this ""pre-execute"" can be analyzed for errors, optimization,
-#   success assertions, verification, etc. The loom file then executes in 2 passes, analyze &
-#   execute.
-#   -- pre-fact collection - inject the facts (or loom) object as a recorder (like a mock in record mode)
-#      instead of the factual fact set. no need to change any loom files.
-#   -- only run fact providers which are accessed in the pattern set
-#   -- only load modules accessed in the pattern set
-# - Replace Pattern "mods" and "mod specs" in Loom::Pattern::ReferenceSet with usages of a builder
-#   instead. Currently the internal data model in ReferenceSet is confusing, but luckily it's the
-#   only client of pattern modules, that have used Loom::Pattern::DSL. Change calls to
-#   DSL#pattern/report/weave (anythin else that creates a pattern) to add a new PatternBuilder to
-#   the module. Use the builder to implement the TODO above ("Add a phase..."). Implement analysis
-#   on the builder.
+#
+# - Add a "groups" flag to `loom i` to display inventory groupings
 
 =begin
 
 ## .loom File DSL
 
 See:
-* spec/test.loom for a valid .loom file.
+* spec/.loom/*.loom for a lots of valid .loom files.
+  * run these specs with `rspec spec/test_loom_spec.rb`
 * spec/loom/pattern/dsl_spec.rb for other examples
 
 
-I've tried to take inspriation from several ruby DSLs, including (but not
-limited to) RSpec, Thor, Commander, Sinatra... so hopefully it feels
-comfortable.
+I've tried to take inspriation from several ruby DSLs, including RSpec, Thor,
+Commander, Sinatra... so hopefully it feels comfortable. Thank you to all of
+those projects.
 
 Loom::Pattern::DSL is the mixin that defines the declarative API for all .loom
 file defined modules. It is included into Loom::Pattern by default. The outer
@@ -83,15 +119,19 @@ end
 
 It declares a reference set with slugs:
 
-* top_level
+* cmd
 * outer:first
 * outer:inner:second
 
-Defining the same pattern slug twice raises a DuplicatPatternRef error.
+Defining the same pattern slug twice raises a `DuplicatePatternRef` error.
 
 #### Code Details
 
-To follow the code path for .loom file loading see:
+Loom::DSL is a facade available to all .loom file ::Modules that include
+Loom::Pattern. It provides Module singleton methods listed at
+Loom::DSL::DSL_METHODS.
+
+Code path for .loom file loading:
 
     Loom::Runner#load
       -> Loom::Pattern::Loader.load
@@ -103,12 +143,48 @@ file. A ReferenseSet being a collection of references with uniquely named
 slugs. The slug of a reference is computed from the module namespace and
 instance method name.
 
+### `weave`
+
+The `weave` creates a specialized pattern, that allows aliasing a sequence of
+pattern slugs as a single pattern name. Pattern execution will be flattened and
+run sequentially before or after any other patterns in the `$ loom` invocation.
+
+``` ~ruby
+pattern :step_1 { ... }
+pattern :step_2 { ... }
+
+module OtherTasks
+...
+end
+
+weave :do_it, [ :step_1, :step_2, :other_tasks:* ]
+```
+
+This creates pattern :do_it, which when run `$ loom do_it` will run :step_1,
+:step_2, and all slugs that match /other_tasks.*/. Recursive expansion is
+explicitly disallowed, only pattern names (not weaves), are allowed in the list
+of weave pattern slugs.
+
+#### Code Details
+
+Weave expansion to pattern slugs is accomplished by creating a
+Loom::Pattern::ExpandingReference via the Loom::Pattern::Loader+load+ path
+invoked via Loom::Runner+load+. Expansion happens on read via
+Loom::Pattern::Loader+patterns+, thus the list of patterns is constant
+throughout all phases of pattern execution.
+
 ### `report`
 
-Use `report` to create a pattern that outputs a fact, other value, or result of
-a block to yaml, json, or any other format.
+Use `report` to create another specialized pattern that prints a fact, value,
+or result of a block to yaml, json, or any other format to STDOUT.
 
-### `let`, `before`, and `after`
+### `let`, `before`, and `after`: for examples spec/.loom/parent_context.loom
+
+`let`, does the same as in RSpec (including the context details). It creates an
+alias for a value, available in all patterns.
+
+`before` and `after` are similar to the same in RSpec. Each before/after block
+is run before/after respectively to EACH pattern.
 
 Module::Inner, from above, inherits all +let+ declarations from its outer
 contexts, both :: (root) and ::Outer. +before+ hooks are run from a top-down
@@ -179,31 +255,6 @@ Loom::Facts::FactSet as parameters.
 See Loom::Pattern::DefinitionContext for evaluation of `let` blocks and
 before/after context ordering.
 
-### `weave`
-
-The `weave` keyword allows aliasing a sequence of patterns a single
-name. Pattern execution will be flattened and run sequentially before or after
-any other patterns in the `$ loom` invocation.
-
-``` ~ruby
-pattern :step_1 { ... }
-pattern :step_2 { ... }
-
-weave :do_it, [ :step_1, :step_2 ]
-```
-
-This creates pattern :do_it, which when run `$ loom do_it` will run :step_1,
-:step_2. Recursive expansion is explicitly disallowed, only pattern names (not
-weaves), are allowed in the 2nd param of `weave`.
-
-#### Code Details
-
-Weave expansion to pattern slugs is accomplished by creating a
-Loom::Pattern::ExpandingReference via the Loom::Pattern::Loader+load+ path
-invoked via Loom::Runner+load+. Expansion happens on read via
-Loom::Pattern::Loader+patterns+, thus the list of patterns is constant
-throughout all phases of pattern execution.
-
 #### Pattern Execution Sequence
 
 Once hosts and patterns are identified in earlier Loom::Runner phases,
@@ -250,14 +301,55 @@ module Loom::Pattern
 
   PatternDefinitionError = Class.new Loom::LoomError
 
-  # TODO: clarify this DSL to only export:
-  # - description
-  # - pattern
-  # - let
-  # - after/before
-  # other methods are utility methods used to process and run patterns.
+  DSL_METHODS = [
+    :desc,
+    :description,
+
+    :pattern,
+    :weave,
+    :report,
+
+    :with_facts,
+    :let,
+    :before,
+    :after,
+
+    :namespace
+  ]
+
+  ##
+  # The Loom DSL definition. See documentation above.
   module DSL
+    DSL_METHODS.each do |m|
+      define_method m do |*args, **kwargs, &block|
+        Loom.log.debug1(self) { "delegating Pattern::DSL call to DSLBuilder+#{m}+" }
+        @dsl_builder.send(m, *args, **kwargs, &block)
+      end
+    end
+
+    attr_reader :dsl_builder
+
+    class << self
+      def extended(receiving_mod)
+        # NB: Using Forwardable was awkward here due to the scope of extended, and
+        # the scope of where the fordwardable instance variable would live.
+        dsl_builder = PatternBuilder.new
+        receiving_mod.instance_variable_set :@dsl_builder, dsl_builder
+      end
+    end
+  end
+
+  class DSL::PatternBuilder
+
+    def initialize
+      @pattern_map = {}
+      @fact_map = {}
+      @let_map = {}
+      @hooks = []
+      @next_description = nil
+    end
 
+    # BEGIN DSL Implementation
     loom_accessor :namespace
 
     def description(description)
@@ -266,37 +358,33 @@ module Loom::Pattern
     alias_method :desc, :description
 
     def with_facts(**new_facts, &block)
-      @facts ||= {}
-      @facts.merge! new_facts
-      yield_result = yield @facts if block_given?
-      @facts = yield_result if yield_result.is_a? Hash
+      @fact_map.merge! new_facts
+      yield_result = yield @fact_map if block_given?
+      @fact_map = yield_result if yield_result.is_a? Hash
     end
 
     def let(name, default: nil, &block)
       raise "malformed let expression: missing block" unless block_given?
-
-      @let_map ||= {}
       @let_map[name.to_sym] = LetMapEntry.new default, &block
     end
 
     def pattern(name, &block)
-      Loom.log.debug1(self) { "defining pattern => #{name}" }
-      define_pattern_internal(name, &block)
+      define_pattern_internal(name, kind: :pattern, &block)
     end
 
     ##
     # @param format[:yaml|:json|:raw] default is :yaml
     def report(name, format: :yaml, &block)
-      Loom.log.debug1(self) { "defining reporting pattern => #{name}" }
-      define_pattern_internal(name) do |loom, facts|
-        # TODO: I don't like all of this logic in the dsl.
+      define_pattern_internal(name, kind: :report) do |loom, facts|
+        # TODO: I don't like all of this logic here. It feels like it belongs in
+        # a mod.
         result = if block_given?
                    Loom.log.debug(self) { "report[#{name}] from block" }
-                   self.instance_exec(loom, facts, &block)
+                   instance_exec(loom, facts, &block)
                  elsif !Loom::Facts.is_empty?(facts[name])
                    Loom.log.debug(self) { "report[#{name}] from facts[#{name}]" }
                    facts[name]
-                 elsif self.respond_to?(name) && !self.send(name).nil?
+                 elsif respond_to?(name) && !self.send(name).nil?
                    Loom.log.debug(self) { "report[#{name}] from let{#{name}}" }
                    self.send name
                  else
@@ -318,15 +406,10 @@ module Loom::Pattern
     end
 
     def weave(name, pattern_slugs)
-      Loom.log.debug1(self) { "defining weave => #{name}" }
-      @weave_slugs ||= {}
-      @weave_slugs[name.to_sym] = pattern_slugs.map { |s| s.to_s }
-
       unless @next_description
         @next_description = "Weave runs patterns: %s" % pattern_slugs.join(", ")
       end
-
-      define_pattern_internal(name) { |_, _| true }
+      define_pattern_internal(name, kind: :weave, expanded_slugs: pattern_slugs) { true }
     end
 
     def before(&block)
@@ -336,68 +419,50 @@ module Loom::Pattern
     def after(&block)
       hook :after, &block
     end
+    # END DSL Implementation
 
-    def weave_slugs
-      @weave_slugs || {}
-    end
-
-    def is_weave?(name)
-      !!weave_slugs[name]
-    end
-
-    def pattern_methods
-      @pattern_methods || []
-    end
-
-    def pattern_description(name)
-      @pattern_descriptions[name]
-    end
-
-    def pattern_method(name)
-      raise UnknownPatternMethod, name unless @pattern_method_map[name]
-      instance_method name
+    def patterns
+      @pattern_map.values
     end
 
     def hooks
-      @hooks || []
+      @hooks
     end
 
     def facts
-      @facts || {}
+      @fact_map
     end
 
     def let_map
-      @let_map || {}
+      @let_map
     end
 
     private
-    def define_pattern_internal(name, &loom_file_block)
-      @pattern_methods ||= []
-      @pattern_method_map ||= {}
-      @pattern_descriptions ||= {}
-
-      method_name = name.to_sym
+    # TODO: Let mods introduce new pattern handlers. A pattern is effectively a
+    # named wrapper around a pattern execution block. This would be an advanced
+    # usage when before and after blocks aren't scalable. It could also provided
+    # additional filtering for pattern selection at weave time.
+    def define_pattern_internal(name, kind: :pattern, **kwargs, &loom_file_block)
+      unless block_given?
+        raise PatternDefinitionError, "missing block for pattern[#{name}]"
+      end
+      unless Pattern::KINDS[kind]
+        raise "unknown pattern kind: #{kind}"
+      end
 
-      @pattern_methods << method_name
-      @pattern_method_map[method_name] = true
-      @pattern_descriptions[method_name] = @next_description
+      desc = @next_description
+      unless desc.is_a?(String) || desc.nil?
+        raise PatternDefinitionError, "description must be a string: #{desc.class}"
+      end
       @next_description = nil
+      name = name.intern
 
-      ##
-      # ```ruby
-      # pattern :xyz do |loom, facts|
-      #   loom.x :dostuff
-      # end
-      # ```
-      # Patterns declared in the .loom file are defined here:
-      define_method method_name do |loom, facts|
-        Loom.log.debug2(self) { "calling .loom file pattern: #{name}" }
-        self.instance_exec(loom, facts, &loom_file_block)
-      end
+      Loom.log.debug(self) { "defined .loom pattern[kind:#{kind}]: #{name}" }
+      @pattern_map[name] = Pattern.new(
+        name: name, description: desc, kind: kind, **kwargs, &loom_file_block)
     end
 
     def hook(scope, &block)
-      @hooks ||= []
       @hooks << Hook.new(scope, &block)
     end
   end # DSL

data/lib/loom/pattern/expanding_reference.rb

@@ -1,16 +1,92 @@
+require "pry"
+
 module Loom::Pattern
   class ExpandingReference
 
-    attr_reader :slug, :reference_slugs, :source_file, :desc
+    RecursiveExpansionError = Class.new Loom::LoomError
+
+    # TODO: Ensure ExpandingReference and Reference stay in sync. Maybe create
+    # an inheritance hierarchy.
+    attr_reader :slug, :reference_slugs, :source_file, :desc, :pattern
 
-    def initialize(slug, reference_slugs, source_file, description)
+    ##
+    # @param slug [String]: flattened colon separated slug name
+    # @param pattern [Loom::Pattern::Pattern]: a pattern responding to +expanded_slugs+
+    def initialize(slug, pattern, reference_set)
       @slug = slug
-      @reference_slugs = reference_slugs
-      @desc = description
+      @reference_set = reference_set
+      # TODO: Hmm... I tried to abstract the "weave" keyword from the
+      # "ExpandingReference" concept... but it leaked through. Think the
+      # `pattern.kind` based method name over.
+      @reference_slugs = pattern.weave.expanded_slugs
+      @desc = pattern.description
+      @pattern
     end
 
-    def is_expanding?
-      true
+    def expand_slugs
+      # O(MN) :(
+      expanded_slugs = @reference_slugs.flat_map do |my_slug|
+        matcher = Matcher.get_matcher(my_slug)
+        @reference_set.slugs.select { |your_slug| matcher.match? your_slug }
+      end.uniq
+      Loom.log.debug3(self) { "Loom::Pattern::ExpandingReference@reference_slugs+: #{@reference_slugs.join(",")}"}
+      Loom.log.debug3(self) { "Loom::Pattern::ExpandingReference+expanded_slugs+: #{expanded_slugs.join(",")}"}
+
+      expanded_refs = expanded_slugs.map { |s| @reference_set[s] }
+      expanded_refs.each do |r|
+        if r.is_a? ExpandingReference
+          Loom.log.error "recursive expansion for pattern[#{r.slug}] in weave[#{@slug}], i.e. only patterns are allowed in weaves"
+          raise RecursiveExpansionError, @slug
+        end
+      end
+
+      Loom.log.info { "expanded slug[#{@slug}] => #{expanded_slugs.join(",")}"}
+      expanded_slugs
+    end
+
+    private
+    # TODO: This can be made common to some utility directory if one emerges.
+    class Matcher
+
+      def self.get_matcher(slug)
+        matcher_module = [
+          GlobMatcher,
+          EqualityMatcher
+        ].first { |m| m.handles_pattern? slug }
+
+        Class.new(Matcher).include(matcher_module).new(slug)
+      end
+
+      def initialize(loom_pattern_slug)
+        @my_slug = loom_pattern_slug
+      end
+
+      private
+      module GlobMatcher
+        MATCH_P = /(\*)$/
+
+        def self.handles_pattern?(my_slug)
+          res = my_slug.match? MATCH_P
+          Loom.log.debug2(self) { "#{p}.match? #{MATCH_P} = #{res}" }
+          res
+        end
+
+        def match?(your_pattern)
+          prefix = @my_slug.to_s.gsub(MATCH_P, "")
+          Loom.log.debug2(self) { "GlobMatcher+match?+ #{@my_slug} #{your_pattern}, prefix: #{prefix}"}
+          your_pattern.to_s.start_with? prefix
+        end
+      end
+
+      module EqualityMatcher
+        def self.handles_pattern?(p)
+          true
+        end
+
+        def match?(your_pattern)
+          @my_slug == your_pattern
+        end
+      end
     end
   end
 end

data/lib/loom/pattern/loader.rb

@@ -1,7 +1,6 @@
 module Loom::Pattern
 
   SiteFileNotFound = Class.new Loom::LoomError
-  RecursiveExpansionError = Class.new Loom::LoomError
 
   class Loader
     class << self
@@ -37,7 +36,7 @@ module Loom::Pattern
 
     def load_patterns
       @loom_pattern_files.each do |f|
-        raise SiteFileNotFound, f unless File.exists? f
+        raise SiteFileNotFound, f unless File.exist? f
         load_pattern_file f
       end
     end
@@ -48,22 +47,7 @@ module Loom::Pattern
     end
 
     def expand_refs(refs)
-      refs.flat_map do |ref|
-        if ref.is_expanding?
-          expanded_refs = ref.reference_slugs.map { |s| get_pattern_ref(s) }
-          expanded_refs.each do |exp_ref|
-            if exp_ref.is_expanding?
-              Loom.log.error "error expanding pattern[#{exp_ref.slug}] in weave[#{ref.slug}], i.e. only patterns are allowed in weaves"
-              raise RecursiveExpansionError, ref.slug
-            end
-          end
-          Loom.log.info(
-            "expanded pattern #{ref.slug} to patterns: #{expanded_refs.map(&:slug).join(",")}")
-          expanded_refs
-        else
-          ref
-        end
-      end
+      refs.flat_map { |r| r.expand_slugs }.map { |s| @reference_set[s] }
     end
   end
 end

data/lib/loom/pattern/pattern.rb

@@ -0,0 +1,52 @@
+module Loom::Pattern
+  # A value object represnting the .loom file pattern declarations. The
+  # difference between a Loom::Pattern::Pattern and Loom::Pattern::Reference is
+  # a pattern has no association to the context it should run in. It is simply a
+  # value object with pointers to its assigned values from the .loom
+  # file. However, a Reference includes it's DefinitionContext, including nested
+  # before/after/with_facts/let hooks.
+  class Pattern
+
+    KINDS = %i[pattern weave report]
+      .reduce({}) { |memo, k| memo.merge k => true }.freeze
+
+    # effectively, a list of `attr_readers` for the Pattern kind. but also used
+    # for validation
+    KIND_KWARGS = {
+      weave: [:expanded_slugs]
+    }.freeze
+
+    attr_reader :name, :description, :kind, :pattern_block
+
+    def initialize(name: nil, description: nil, kind: nil, **kind_kwargs, &block)
+      @name = name
+      @description = description
+      @kind = kind
+      @pattern_block = block
+
+      @valid_kwargs = KIND_KWARGS[kind]
+      kind_kwargs.each do |k, _|
+        raise "unknown kind_kwarg: #{k}" unless @valid_kwargs.include? k
+      end
+      @kind_properties_struct = OpenStruct.new kind_kwargs
+    end
+
+    # Adds methods:
+    # :is_weave?, is_pattern?, :is_reported?
+    # :weave, :patttern, :reported => returns the OpenStruct of KIND_KWARGS.
+    KINDS.keys.each do |k|
+      is_k_name = "is_#{k}?".intern
+      Loom.log.debug3(self) { "defining method Loom::Pattern::Pattern+#{is_k_name}+" }
+      define_method(is_k_name) { @kind == k }
+
+      k_name = k.intern
+      Loom.log.debug1(self) { "defining method Loom::Pattern::Pattern+#{k_name}+" }
+      define_method(k_name) do
+        if @kind != k_name
+          raise "invalid kwarg +#{k_name}+ for Pattern[#{@kind}]"
+        end
+        @kind_properties_struct.dup
+      end
+    end
+  end
+end