inspec-core 2.1.84 → 2.2.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b8f1a62efaa756d818227d18b3b61aa106f61c597a9cd9b46bd28ea0dc9eb66a
-  data.tar.gz: ee17dec7f433f4e9581c61e22692f91d45a7afb48e7af52e195f6a9a0a76661e
+  metadata.gz: d5a0147fcfec73a7c62fafd050d13b9cf3696d33892a769295cae623af5126fc
+  data.tar.gz: ea24534b1a9887a625310aabbab3054884d0d54ab0462378b2e1adb6d6a9b8fa
 SHA512:
-  metadata.gz: a6c8d5e7e4a4e2e89df4fcce36dc81745a05f071bd55089ac9e3ae8bf8a7edb6fe067db0f810307c23e5e1b75863bdf5553985f5ca1404ce1ee5280346c6553a
-  data.tar.gz: fc0b64a7a2d49a9dd65f8db7f2c171ca2fa877abcadd82a2331487f45cc7e378f3f3fadec7f20bad01232f62b1f2d6eca5d4228efde9dc9cc61e65ec5a32592b
+  metadata.gz: 4a2da0d531f2611b278ad64d6d877a48ab6e70226e4a535c409b3a05c5e81582fc049800de66eae9690848f85822903d32275b82ea6c5cfab830a2db2304cef0
+  data.tar.gz: 9d86cf80e4fc09a27fce83455c57ea349f10ed301f12ef30ac3d987da3b2cc18173d903574ade9ca702d7cb3eb76f766207a2511f5047adb0949c4cfc670a4ef

data/CHANGELOG.md

@@ -1,26 +1,49 @@
 # Change Log
 <!-- usage documentation: http://expeditor-docs.es.chef.io/configuration/changelog/ -->
-<!-- latest_release 2.1.84 -->
-## [v2.1.84](https://github.com/inspec/inspec/tree/v2.1.84) (2018-05-31)
+<!-- latest_release 2.2.8 -->
+## [v2.2.8](https://github.com/inspec/inspec/tree/v2.2.8) (2018-06-07)
 
-#### Merged Pull Requests
-- Generate new org builds [#3087](https://github.com/inspec/inspec/pull/3087) ([jquick](https://github.com/jquick))
+#### New Resources
+- Add aws_elb and aws_elbs resources [#3079](https://github.com/inspec/inspec/pull/3079) ([clintoncwolfe](https://github.com/clintoncwolfe))
 <!-- latest_release -->
 
-<!-- release_rollup since=2.1.83 -->
-### Changes since 2.1.83 release
+<!-- release_rollup since=2.1.84 -->
+### Changes since 2.1.84 release
+
+#### New Resources
+- Add aws_elb and aws_elbs resources [#3079](https://github.com/inspec/inspec/pull/3079) ([clintoncwolfe](https://github.com/clintoncwolfe)) <!-- 2.2.8 -->
+- Adds a aws_flow_log resource with unit and integration testing. [#2906](https://github.com/inspec/inspec/pull/2906) ([miah](https://github.com/miah)) <!-- 2.2.7 -->
+
+#### Bug Fixes
+- Update documentation for shadow resource. [#3042](https://github.com/inspec/inspec/pull/3042) ([miah](https://github.com/miah)) <!-- 2.2.6 -->
+- Two fixes to FilterTable `where` criteria handling [#3045](https://github.com/inspec/inspec/pull/3045) ([clintoncwolfe](https://github.com/clintoncwolfe)) <!-- 2.2.0 -->
+
+#### Enhancements
+- Allow custom resources to access all other resources [#3108](https://github.com/inspec/inspec/pull/3108) ([jquick](https://github.com/jquick)) <!-- 2.2.3 -->
+- Add common methods to FilterTable [#3104](https://github.com/inspec/inspec/pull/3104) ([clintoncwolfe](https://github.com/clintoncwolfe)) <!-- 2.2.2 -->
+- Detect windows packages with trailing/leading spaces [#3106](https://github.com/inspec/inspec/pull/3106) ([jquick](https://github.com/jquick)) <!-- 2.1.86 -->
 
 #### Merged Pull Requests
-- Generate new org builds [#3087](https://github.com/inspec/inspec/pull/3087) ([jquick](https://github.com/jquick)) <!-- 2.1.84 -->
+- Clean up issues in documentation [#3058](https://github.com/inspec/inspec/pull/3058) ([miah](https://github.com/miah)) <!-- 2.2.5 -->
+- Update Junit.rb to add failures attribute [#3086](https://github.com/inspec/inspec/pull/3086) ([scboucher](https://github.com/scboucher)) <!-- 2.2.4 -->
+- Add lazy-loading to FilterTable [#3093](https://github.com/inspec/inspec/pull/3093) ([clintoncwolfe](https://github.com/clintoncwolfe)) <!-- 2.2.1 -->
+- Refactor: Perform internal rename and add comments to FilterTable [#3047](https://github.com/inspec/inspec/pull/3047) ([clintoncwolfe](https://github.com/clintoncwolfe)) <!-- 2.1.88 -->
+- Spellcheck FilterTable Developer Documentation [#3111](https://github.com/inspec/inspec/pull/3111) ([jerryaldrichiii](https://github.com/jerryaldrichiii)) <!-- 2.1.87 -->
+- support local npm package searches [#3105](https://github.com/inspec/inspec/pull/3105) ([arlimus](https://github.com/arlimus)) <!-- 2.1.85 -->
 <!-- release_rollup -->
 
 <!-- latest_stable_release -->
+## [v2.1.84](https://github.com/inspec/inspec/tree/v2.1.84) (2018-05-31)
+
+#### Merged Pull Requests
+- Generate new org builds [#3087](https://github.com/inspec/inspec/pull/3087) ([jquick](https://github.com/jquick))
+<!-- latest_stable_release -->
+
 ## [v2.1.83](https://github.com/chef/inspec/tree/v2.1.83) (2018-05-18)
 
 #### Merged Pull Requests
 - Update ruby required version. [#3070](https://github.com/chef/inspec/pull/3070) ([jquick](https://github.com/jquick))
 - Test new gem builds [#3071](https://github.com/chef/inspec/pull/3071) ([jquick](https://github.com/jquick))
-<!-- latest_stable_release -->
 
 ## [v2.1.81](https://github.com/chef/inspec/tree/v2.1.81) (2018-05-17)
 

data/README.md

@@ -451,3 +451,4 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License. 
+ 

data/docs/dev/filtertable-internals.md

@@ -0,0 +1,353 @@
+# Internals of FilterTable
+
+If you just want to _use_ FilterTable, see filtertable-usage.md .  Reading this may make you more confused, not less.
+
+## What makes this hard?
+
+FilterTable was created in 2016 in an attempt to consolidate the pluralization features of several resources.  They each had slightly different feature-sets, and were all in the wild, so FilterTable exposes some extensive side-effects to provide those features.
+
+Additionally, the ways in which the classes relate is not straightforward.
+
+## Where is the code?
+
+The main FilterTable code is in [utils/filter.rb](https://github.com/chef/inspec/blob/master/lib/utils/filter.rb).
+
+Also educational is the unit test for Filtertable, at test/unit/utils/filter_table_test.rb .  Recent work has focused on using functional tests to exercise FilterTable; see test/unit/mocks/profiles/filter_table and test/functional/filter_table_test.rb .
+
+The file utils/filter_array.rb appears to be unrelated.
+
+## What are the classes involved?
+
+### FilterTable::Factory
+
+This class is responsible for the definition of the filtertable.  It provides the methods that are used by the resource author to configure the filtertable.
+
+FilterTable::Factory initializes three instance variables:
+```
+  @filter_methods = []
+  @custom_properties = {}
+  @resource = nil # This appears to be unused
+```
+
+### FilterTable::Table
+
+This is the actual innards of the implementation.  The Factory's goal is to configure a Table subclass and attach it to the resource you are authoring.  The table is a container for the raw data your resource provides, and performs filtration services.
+
+### FilterTable::ExceptionCatcher
+
+TODO
+
+## What are the major entry points? (FilterTable::Factory)
+
+A resource class using FilterTable typically will call a sequence similar to this, in the class body:
+
+```
+  filter = FilterTable.create
+        .register_column(:thing_ids, field: :thing_id)
+        .install_filter_methods_on_resource(self, :table)
+```
+
+Legacy code might look like this:
+```
+  filter = FilterTable.create
+  filter.add_accessor(:entries)
+        .add(:exists?) { |x| !x.entries.empty? }
+        .add(:count) { |x| x.entries.count }
+        .add(:thing_ids, field: :thing_id)
+        .connect(self, :table)
+```
+
+### create
+
+Returns a blank instance of a FilterTable::Factory.  It also adds a default implementation of `where` `raw_data`, and `entries` using `register_filter_method`, `count` using `register_custom_property`, and `exist?` using `register_custom_matcher`.
+
+### register\_filter\_method
+
+Legacy name (alias): `add_accessor`
+
+This simply pushes the provided method name onto the `@filter_methods` instance variable array.  See "filter_method" behavior section below for what this does.
+
+After adding the method name to the array, it returns `self` - the FilterTable::Factory instance - so that method chaining will work.
+
+### register\_column
+
+Legacy name (alias): `add`
+
+This is currently simply an alias for `register_custom_property`.  See it for details.  By calling it with a distinctive name, we'll be able to add functionality in the future (especially around introspection).
+
+### register\_custom\_matcher
+
+Legacy name (alias): `add`
+
+This is currently simply an alias for `register_custom_property`.  See it for details. By calling it with a distinctive name, we'll be able to add functionality in the future (especially around introspection).
+
+### register\_custom\_property
+
+Legacy name (alias): `add`
+
+This method has very complex behavior, ans should likely be split into several use cases.  `register_custom_property` requires a symbol (which will be used as a method name _to be added to the resource class_), then also accepts a block and/or additional args.  These things - name, block, and opts - are packed into a simple Struct called a CustomPropertyType. The name stored in the Struct will be `opts[:field]` if provided, and the method name if not.
+
+The CustomPropertyType Struct is then appended to the Hash `@custom_properties`, keyed on the method name provided.  `self` is then returned for method chaining.
+
+The implementation of the custom property method is generated by `create_custom_property_body`, and varies based on whether a block was provided to `register_custom_property`.
+
+#### Behavior when a block is provided
+
+This behavior is implemented by lines 398-404.
+
+If a block is provided, it is turned into a Lambda and used as the method body.
+
+The block will be provided two arguments (though most users only use the first):
+1. The FilterTable::Table instance that wraps the raw data.
+2. An optional value used as an additional opportunity to filter.
+
+For example, this is common in legacy code:
+```
+filter.add(:exists?) { |x| !x.entries.empty? }
+```
+
+Here, `x` is the Table instance, which exposes the `entries` method (which returns an array, one entry for each raw data row).
+
+You could also implement a more sophisticated property, which semantically should re-filter the table based on the candidate value, and return the new table.
+
+```
+filter.add(:smaller_than) { |table, threshold| table.where { some_field <= threshold } }
+```
+
+```
+things.smaller_than(12)
+```
+
+If you provide _both_ a block and opts, only the block is used, and the options are ignored.
+
+#### Behavior when no block is provided
+
+If you do not provide a block, you _must_ provide a `:field` option (though that does no appear to be enforced). The behavior is to define a method with the name provided, that has a conditional return type. The method body is defined in lines 306-423.
+
+If called without arguments, it returns an array of the values in the raw data for that column.
+```
+things.thing_ids => [1,2,3,4]
+```
+
+If called with an argument, it instead calls `where` passing the name of the field and the argument, effectively filtering.
+```
+things.thing_ids(2) => FilterTable::Table that only contains a row where thing_id = 2
+```
+
+If called with a block, it passes the block to where.
+```
+things.thing_ids { some_code } => Same as things.where { some_code }
+```
+
+POSSIBLE BUG: I think this case is broken; it certainly seems ill-advised.
+
+#### Known Options
+
+You can provide options to `register_custom_property` / `add`, after the desired method name.
+
+##### field
+
+This is the most common option, and is mandatory if a block is not provided.  It selects an implementation in which the desired method will be defined such that it returns an array of the row values using the specified key.  In other words, this acts as a "column fetcher", like in SQL: "SELECT some_column FROM some_table"
+
+Internally, (line 271-278), a Struct type is created to represent a row of raw data.  The struct's attribute list is taken from the `field` options passed to `register_custom_property` / `add`.  This new type is stored as `row_eval_context_type`.  It is used as the evaluation context for block-mode `where` calls. 
+
+* No checking is performed to see if the field name is actually a column in the raw data (the raw data hasn't been fetched yet, so we can't check).
+* You can't have two `register_custom_property` / `add` calls that reference the same field, because the Struct would see that as a duplicate attribute.
+
+POSSIBLE BUG: We could deduplicate the field names when defining the Struct, thus allowing multiple properties to use the same field.
+
+##### style
+
+The `style` option is intended to effect post-processing of the return value from the generated method.  To date there is only one recognized value, `:simple`, which `flatten`s, `uniq`s, and `compact`s the array value of the property.  This is implemented on line 416.
+
+No other values for `:style` have been seen.
+
+##### lazy
+
+This option implements column-wise lazy loading. The value of the option is expected to a lambda expecting 3 arguments: row (a Hash representing a row of the raw data), condition (a sought value to filter for), and table (a reference to the FilterTable::Table subclass, which may be used for context).
+
+See the usage guide for details on the usage of the lazy mechanism; this document will examine the internals.
+
+### install_filter_methods_on_resource
+
+Legacy name (alias): connect
+
+This method is called like this:
+
+```
+filter.install_filter_methods_on_resource(self, :data_fetching_method_name)
+```
+
+`filter` is an instance of FilterTable::Factory.  `self` is a reference to the resource class you are authoring. `data_fetching_method_name` is a symbol, the name of a method that will return the actual data to be processed by the FilterTable - as an array of hashes.
+
+Note that 'connect' does not refer to Connectors.
+
+`register_custom_property` and `register_filter_method` did nothing other than add register names for methods that we'd like to have added to the resource class.  No filtering ability is present, nor are the methods defined, at this point.
+
+So, `install_filter_methods_on_resource`/`connect`'s job is to actually install everything.
+
+#### Defines a special Struct type to support block-mode where
+
+At lines 270-278, a new Struct type `row_eval_context_type` is defined, with attributes for each of the known table fields.  We're careful to filter out any fields that have a block implementation, because they cannot be accessed as a row value, and so should not be present on the row context. The motivation for this struct type is to implement the block-mode behavior of `where`.  Because each struct represents a row, and it has the attributes (accessors) for the fields, block-mode `where` is implemented by `instance_eval`ing against each row as a struct.
+
+Additionally, an instance variable, `@criteria_string` is defined, with an accessor. `to_s` is implemented, using `@criteria_string`, or `super` if not defined.  I guess we then rely on the `Struct` class to stringify.
+
+`@criteria_string` is a trace - a string indicating the filter criteria used to create the table.  I found no location where this per-row trace data was used.
+
+Additionally, an instance variable is setup to refer to the filter table later.  This is required for lazy-loading columns.
+
+Table fields are determined by listing the `field_name`s of the CustomProperties.
+
+BUG: this means that any `register_custom_property` / `add` call that uses a block but not options will end up with an attribute in the row Struct.  Thus, `filter.add(:exists?) { ... }` results in a row Struct that includes an attribute named `exists?` which may be undesired. This attribute will never have a value, because when the structs are instantiated, the block for the field is not called.
+
+#### Re-pack the "connectors"
+
+On lines 280-282, the list of custom properties ("connectors", registered using the `register_custom_property` / `add` method) are repacked into an array of hashes of two elements - the desired method name and the lambda that will be used as the method body.  The lambda is created by the private method `create_custom_property_body`; see `register_custom_property` for discussion about how the implementation behaves.
+
+#### Subclass FilterTable::Table into an anonymous class
+
+At line 286, create the local var `table_class`, which refers to an anonymous class that subclasses FilterTable::Table.  The class is opened and two groups of methods are defined.
+
+Lines 288-290 install the "custom_property" methods, using the names and lambdas determined on line 281.
+
+Lines 292-294 allow the Table subclass to introspect on the CustomProperties by slipping a reference to it in the class body, forming a closure.
+
+Line 296-303 define a method, `create_eval_context_for_row`.  This is used when executing a block-mode `where`; see line 120.
+
+#### Setup the row context struct for lazy loading
+
+If you have a lazy field named `color` and it has not yet been populated, we need to trigger it to populate the first time it is read.  If a block-mode where is used (`my_resource.where { color == :red }`), then we have to intercept the Struct's default `getter method`, and call the lazy column's lambda.
+
+Lines 306-329 do exactly that, by defining methods on the Struct subclass we're using for context.  We continue to rely on the default Struct setter (`[]=`) and getter at the end (`[]`).
+
+#### Install methods on the resource
+
+Lines 337-348 install the "filter_methods" and "custom properties" methods onto the resource that you are authoring.
+
+Line 337-338 collects the names of the methods to define - by agglomerating the names of the "filter_methods" and "custom properties" methods.  They are treated the same.
+
+Line 339 uses `send` with a block to call `define_method` on the resource class that you're authoring.  Using a block with `send` is undocumented, but is treated as an implicit argument (per StackOverflow) , so the end result is that the block is used as the body for the new method being defined.
+
+The method body is wrapped in an exception-catching facility that catches skipped or failed resource exceptions and wraps them in a specialized exception catcher class. TBD: understand this better.
+
+Line 342 constructs an instance of the anonymous FilterTable::Table subclass defined at 284.  It passes three args:
+
+1. `self`. A reference to the resource instance.
+2. The return value of calling the data fetcher method (that is an array of hashes, the raw data).
+3. The string ' with', which is probably informing the criteria stringification. The extra space is intentional, as it follows the resource name: 'my_things with color == :red' might be a result.
+
+On line 343, we then immediately call a method on that "FilterTable::Table subclass instance". The method name is the same as the one we're defining on the resource - but we're calling it on the Table.  Recall we defined all the "custom_property" methods on the Table subclass at line 288-290.  The method gets called with any args or block passed, and since it's the last thing, it provides the return value.
+
+## What is its behavior? (FilterTable::Table)
+
+Assume that your resource has a method, `fetch_data`, which returns a fixed array:
+
+```
+ [
+   { id: 1, name: 'Dani', color: 'blue' },
+   { id: 2, name: 'Mike', color: 'red' },
+   { id: 3, name: 'Erika', color: 'green' },
+ ]
+```
+
+Assume that you then perform this sequence in your resource class body:
+
+```
+filter = FilterTable.create
+filter.register_filter_method(:entries)
+filter.register_filter_method(:where)
+filter.register_custom_property(:exists?) { |x| !x.exists.empty? }
+filter.register_custom_property(:names, field: :name)
+filter.install_filter_methods_on_resource(self, :fetch_data)
+```
+
+Legacy code equivalent:
+
+```
+filter = FilterTable.create
+filter.add_accessor(:entries)
+filter.add_accessor(:where)
+filter.add(:exists?) { |x| !x.exists.empty? }
+filter.add(:names, field: :name)
+filter.connect(self, :fetch_data)
+```
+
+We know from the above exploration of `install_filter_methods_on_resource` / `connect` that we now have several new methods on the resource class, all of which delegate to the FilterTable::Table implementation.
+
+### FilterTable::Table constructor and internals
+
+Factory calls the FilterTable::Table constructor at 87-93 with three args. Table stores them into instance vars:
+ * @resource_instance - this was passed in as `self` from line 342
+ * @raw_data - an array of hashes
+ * @criteria_string - This looks to be stringification trace data; the string ' with' was passed in by Factory.
+ * @populated_lazy_columns = a hash, by lazy field name, with boolean values.  This is set true if `populate_lazy_field` is called on a field.
+
+The first three get exposed via `attr_reader`s.
+
+### `entries` behavior
+
+From usage, I expect entries to return a structure that resembles an array of hashes representing the (filtered) data.
+
+#### A new method `entries` is defined on the resource class
+
+That is performed by Factory#connect line 339.
+
+#### It delegates to FilterTable::Table#entries
+
+This is a real method defined in filter.rb line 155.
+
+It loops over the provided raw data (@raw_data) and builds an array, calling `create_eval_context_for_row` (see Factory lines 297-303) on each row; also appending a stringification trace to each entry.  The array is returned.
+
+#### `entries` conclusion
+
+Not Surprising: It does behave as expected - an array of Hash-like structs representing the table.  I don't know why it adds in the per-row stringification data - I've never seen that used.
+
+Surprising: this is a real method with a concrete implementation.  That means that you can't call `filter.add_accessor` with arbitrary method names - `:entries` means something very specific.
+
+Surprising: I would not recommend this method be used for data access; instead I would recommend using `raw_data`.  
+
+### `where` behavior
+
+From usage, I expect this to take either method params or a block (both of which are magical), perform filtering, and return some object that contains only the filtered rows.
+
+So, what happens when you call `register_filter_method(:where)` and then call `resource.where`?
+
+#### A new method `where` is defined on the resource class
+
+That is performed by Factory#connect line 339.
+
+#### It delegates to FilterTable::Table#where
+
+Like `entries`, this is a real implemented method on FilterTable::Table, at line 98.
+
+The method accepts all params as the local var `conditions` which defaults to an empty Hash. A block, if any, is also explicitly assigned the name `block`.
+
+The implementation opens with two guard clauses, both of which will return `self` (which is the FilterTable::Table subclass instance).
+
+MISFEATURE: The first guard clause simply returns the Table if `conditions` is not a Hash.  That would mean that someone called it like: `thing.where(:apples, :bananas, :cantaloupes)`.  That misuse is silently ignored; I think we should probably throw a ResourceFailed or something.
+
+The second guard clause is a sensible degenerate case - return the existing Table if there are no conditions and no block.  So `thing.where` is OK.
+
+Line 103 initializes a local var, `new_criteria_string`, which again is a stringification tracker.
+
+Line 104 initializes a var to track the `filtered_raw_data`.
+
+Lines 108-113 loop over the provided Hash `conditions`. If the requested field is lazy, it requests that it be populated (note that `populate_lazy_field` is idempotent - it won't fetch a field twice).  Next, it repeatedly down-filters `filtered_raw_data` by calling the private method `filter_raw_data` on it.  `filter_raw_data` does some syntactic sugaring for common types, Integers and Floats and Regexp matching.  Additionally, the line 108-113 loop builds up the stringification tracker, `new_criteria_string`, by stringifying the field name and target value.
+
+Line 118-135 begins work if a filtration block has been provided.  At this point, `filtered_raw_data` has been initialized with the raw data, and (if method params were provided) has also been filtered down.
+
+Line 120 filters the rows of the raw data using an interesting approach. Each row is inflated to a Struct using `create_eval_context_for_row` (see line 297). Then the provided block is `instance_eval`'d against the Struct.  Because the Struct was defined with attributes (that is, accessor methods) for each declared field name (from FilterTable::Factory#register_custom_property), you can use field names in the block, and each row-as-struct will be able to respond.  If the field happened to be lazy, we'll call our custom getter from lines 306-329.
+
+_That just explained a major spooky side-effect for me._
+
+Lines 125-134 do something with stringification tracing.  TODO.
+
+Finally, at line 137, the FilterTable::Table anonymous subclass is again used to construct a new instance, passing on the resource reference, the newly filtered raw data table, and the newly adjusted stringification tracer.
+
+That new Table instance is returned, and thus `where` allows you to chain.
+
+#### `where` conclusion
+
+Unsurprising: How where works with method params.
+Surprising: How where works in block mode, `instance_eval`'ing against each row-as-Struct.
+Surprising: You can use method-mode and block-mode together if you want.

data/docs/dev/filtertable-usage.md

@@ -0,0 +1,533 @@
+# Using FilterTable to write a Resource
+
+## When do I use FilterTable?
+
+FilterTable is intended to help you author "plural" resources.
+
+Plural resources examine platform objects in bulk. For example, sorting through which packages are installed on a system, or which virtual machines are on a cloud provider. You don't know the identifiers of the objects, but you may know some of their properties, and you may want to filter the objects based on those - for example, all processes running more than an hour, or all VMs on a particular subnet.
+
+Singular resources, in contrast, are designed to examine a particular platform object in detail, _when you know an identifier_. For example, you would use a singular resource to fetch a VM by its ID, then interrogate its networking configuration. Singular resources are able to provide richer properties and matchers than plural resources, because the semantics are clearer.
+
+If you can't tell if the resource you are authoring is singular or plural, STOP and consult with a team member. This is a fundamental design question, and while we have had some resources that "straddle the fence" in the past, they are very difficult to use and maintain.
+
+### Should I use FilterTable to represent list properties?
+
+Suppose you have a person, and you want to represent that person's shoes.  Should you use FilterTable for that?
+
+NO.  FilterTable is intended to represent pluralities inherent to the resource itself, not a property of the resource.  So, you would use FilterTable to represent _people_.  To represent shoes, you could have a simple, dumb array-of-strings property on Person.  Or, you could create a new resource, Shoe, or Shoes, which has a person_name or person_id property. Or expose a complex structure as a low-level property, and create mid-level properties/matchers that compute on the values internally (`shoe_fit?`, `has_shoes_for_occasion?('red_carpet')`)
+
+### May I have multiple FilterTable installations on a class?
+
+In theory, yes - that would be used to implement different data fetching / caching strategies.  It is a very advanced usage, and no core resources currently do this, as far as I know.
+
+## Example Usage
+
+Suppose you are writing a resource, `things`.  You want it to behave like any plural resource (we'll explore what that means in a moment).  That is the basic expected behavior of any plural resource.
+
+### How do I declare my interaction with FilterTable?
+
+```ruby
+
+require 'utils/filter'
+
+class Thing < Inspec.resource(1)
+  #... other Resource DSL work goes here ...
+
+  # FilterTable setup
+  filter_table_config = FilterTable.create
+  filter_table_config.register_column(:thing_ids, field: :thing_id)
+  filter_table_config.register_column(:colors, field: :color, style: :simple)
+  filter_table_config.install_filter_methods_on_resource(self, :fetch_data)
+
+  def fetch_data
+    # This method should return an array of hashes - the raw data.  We'll hardcode it here.
+    [
+      { thing_id: 1, color: :red },
+      { thing_id: 2, color: :blue, tackiness: 'very' },
+      { thing_id: 3, color: :red },
+    ]
+  end
+
+  def some_other_property
+    # We'll examine this later
+  end
+
+end
+```
+
+Note that all of the methods on `filter_table_config` support chaining, so you will sometimes see it as:
+```ruby
+  filter_table_config = FilterTable.create
+                        .register_column(:thing_ids, field: :thing_id)
+                        .register_column(:colors, field: :color, style: :simple)
+                        .install_filter_methods_on_resource(self, :fetch_data)
+```
+etc.
+
+## Standard behavior
+
+With a (fairly standard) implementation like that above, what behavior do you get out of the box?
+
+### Some things are defined for you
+
+In the past, you needed to request certain methods be installed.  These are now installed automatically: `where`, `entries`, `raw_data`, `count`, and `exist?`.  You only have to declare your columns unique to your resource, and then attach the data fetcher.
+
+### Your class is still just a Resource class
+
+Nothing special immediately happens to your class or instances of it.  The data fetcher is not called yet.
+
+### Instances of your class can create a specialized FilterTable::Table instance
+
+When most of the following methods are called, it may trigger the instantiation of a FilterTable::Table anonymous subclass.  That instance will have called the raw data fetcher, and will wrap the raw data inside it.  Many of the following methods return the Table instance.
+
+### A `where` method you can call with nil to get the Table
+
+The resource class gains a method, `where`.  If called with a single `nil` param or no params, it will call the data fetcher method, wrap it up, and return the Table instance.  Calling `where` in other modes will do the same thing, but will filter the data.
+
+```ruby
+  describe things.where(nil)
+    it { should exist }
+    its('count') { should cmp 3 }
+  end
+
+  # This works, too, but for different internal reasons
+  describe things.where
+    it { should exist }
+    its('count') { should cmp 3 }
+  end
+```
+
+### A `where` method you can call with hash params, with loose matching
+
+If you call `where` as a method with no block and passing hash params, with keys you know are in the raw data, it will fetch the raw data, then filter row-wise and return the resulting Table.
+
+Multiple criteria are joined with a logical AND.
+
+The filtering is fancy, not just straight equality.
+
+```ruby
+  describe things.where(color: :red) do
+    its('count') { should cmp 2 }
+  end
+
+  # Regexes
+  describe things.where(color: /^re/) do
+    its('count') { should cmp 2 }
+  end
+
+  # It eventually falls out to === comparison
+  # Here, range membership 1..2
+  describe things.where(thing_id: (1..2)) do
+    its('count') { should cmp 2 }
+  end
+
+  # Things that don't exist are silently ignored, but do not match
+  # See https://github.com/chef/inspec/issues/2943
+  describe things.where(none_such: :nope) do
+    its('count') { should cmp 0 }
+  end
+
+  # irregular rows are supported
+  # Only one row has the :tackiness key, with value 'very'.
+  describe things.where(tackiness: 'very') do
+    its('count') { should cmp 1 }
+  end
+
+```
+
+### A `where` method you can call with a block, referencing some fields
+
+You can also call the `where` method with a block. The block is executed row-wise. If it returns truthy, the row is included in the results. register_custom_propertyitionally, within the block each field declared with the `register_custom_property` configuration method is available as a data accessor.
+
+```ruby
+
+  # You can have any logic you want in the block
+  describe things.where { true } do
+    its('count') { should cmp 3 }
+  end
+
+  # You can access any fields you declared using `register_column`
+  describe things.where { thing_id > 2 } do
+    its('count') { should cmp 1 }
+  end
+```
+
+### You can chain off of `where` or any other Table without re-fetching raw data
+
+The first time `where` is called, the data fetcher method is called.  `where` performs filtration on the raw data table.  It then constructs a new FilterTable::Table, directly passing in the filtered raw data; this is then the return value from `where`.
+
+```ruby
+  # This only calls fetch_data once
+  describe things.where(color: :red).where { thing_id > 2 } do
+    its('count') { should cmp 1 }
+  end
+```
+
+Some other methods return a Table object, and they may be chained without a re-fetch as well.
+
+### An `entries` method that will return an array of Structs
+
+The other `register_filter_method` call enables a pre-defined method, `entries`.  `entries` is much simpler than `where` - in fact, its behavior is unrelated.  It returns an encapsulated version of the raw data - a plain array, containing Structs as row-entries.  Each struct has an attribute for each time you called `register_column`.
+
+Overall, in my opinion, `entries` is less useful than `params` (which returns the raw data).  Wrapping in Structs does not seem to add much benefit.
+
+Importantly, note that the return value of `entries` is not the resource, nor the Table - in other words, you cannot chain it. However, you can call `entries` on any Table.
+
+If you call `entries` without chaining it after `where`, calling entries will trigger the call to the data fetching method.
+
+```ruby
+
+  # Access the entries array
+  describe things.entries do
+    # This is Array#count, not the resource's `count` method
+    its('count') { should cmp 3}
+  end
+
+  # Access the entries array after chaining off of where
+  describe things.where(color: :red).entries do
+    # This is Array#count, not the resource's or table's `count` method
+    its('count') { should cmp 2}
+  end
+
+  # You can access the struct elements as a method, as a hash keyed on symbol, or as a hash keyed on string
+  describe things.entries.first.color do
+    it { should cmp :red }
+  end
+  describe things.entries.first[:color] do
+    it { should cmp :red }
+  end
+  describe things.entries.first['color'] do
+    it { should cmp :red }
+  end
+```
+
+### You get an `exist?` matcher defined on the resource and the table
+
+This `register_custom_matcher` call:
+```ruby
+filter_table_config.register_custom_matcher(:exist?) { |filter_table| !filter_table.entries.empty? }
+```
+
+causes a new method to be defined on both the resource class and the Table class.  The body of the method is taken from the block that is provided.  When the method it called, it will receive the FilterTable::Table instance as its first parameter.  (It may also accept a second param, but that doesn't make sense for this method - see thing_ids).
+
+As when you are implementing matchers on a singular resource, the only thing that distinguishes this as a matcher is the fact that it ends in `?`.
+
+```ruby
+  # Bare call on the matcher (called as a method on the resource)
+  describe things do
+    it { should exist }
+  end
+
+  # Chained on where (called as a method on the Table)
+  describe things.where(color: :red) do
+    it { should exist }
+  end
+```
+
+### You get an `count` property defined on the resource and the table
+
+This `register_custom_property` call:
+```ruby
+filter_table_config.register_custom_property(:count) { |filter_table| filter_table.entries.count }
+```
+
+causes a new method to be defined on both the resource class and the Table class.  As with `exists?`, the body is taken from the block.
+
+```ruby
+  # Bare call on the property (called as a method on the resource)
+  describe things do
+    its('count') { should cmp 3 }
+  end
+
+  # Chained on where (called as a method on the Table)
+  describe things.where(color: :red) do
+    its('count') { should cmp 2 }
+  end
+```
+
+### A `thing_ids` method that will return an array of plain values when called without params
+
+This `register_column` call:
+```ruby
+filter_table_config.register_column(:thing_ids, field: :thing_id)
+```
+
+will cause a method to be defined on both the resource and the Table. Note that this `register_column` call does not provide a block; so FilterTable::Factory generates a method body.  The `:field` option specifies which column to access in the raw data (that is, which hash key in the array-of-hashes).
+
+The implementation provided by Factory changes behavior based on calling pattern.  If no params or block is provided, a simple array is returned, containing the column-wise values in the raw data.
+
+```ruby
+
+  # Use it to check for presence / absence of a member
+  # This retains nice output formatting - we're testing on a Table associated with a Things resource
+  describe things.where(color: :red) do
+    its('thing_ids') { should include 3 }
+  end
+
+  # Equivalent but with poor formatting - we're testing an anonymous array
+  describe things.where(color: :red).thing_ids do
+    it { should include 3 }
+  end
+
+  # Use as a test-less enumerator
+  things.where(color: :red).thing_ids.each do |thing_id|
+    # Do something with thing_id, maybe
+    # describe thing(thing_id) do ...
+  end
+
+  # Can be used without where - enumerates all Thing IDs with no filter
+  things.thing_ids.each do |thing_id|
+    # Do something with thing_id, maybe
+    # describe thing(thing_id) do ...
+  end
+
+```
+
+### A `colors` method that will return a flattened and uniq'd array of values
+
+This method behaves just like `thing_ids`, except that it returns the values of the `color` column.  In addition, the `style: :simple` option causes it to flatten and uniq the array of values when called without args or a block.
+
+```ruby
+  # Three rows in the data: red, blue, red
+  describe things.colors do
+    its('count') { should cmp 2 }
+    it { should include :red }
+    it { should include :blue }
+  end
+
+```
+
+### A `colors` method that can filter on a value and return a Table
+
+You also get this for `thing_ids`.  This is unrelated to `style: :simple` for `colors`.
+
+People definitely use this in the wild.  It reads badly to me; I think this is a legacy usage that we should consider deprecating. To me, this seems to imply that there is a sub-resource (here, colors) we are auditing.  At least two core resouces (`xinetd_conf` and `users`) advocate this as their primary use.
+
+```ruby
+  # Filter on colors
+  describe things.colors(:red) do
+    its('count') { should cmp 2 }
+  end
+
+  # Same, but doesn't imply we're now operating on some 'color' resource
+  describe things.where(color: :red) do
+    its('count') { should cmp 2 }
+  end
+```
+
+### A `colors` method that can filter on a block and return a Table
+
+You also get this for `thing_ids`.  This is unrelated to `style: :simple` for `colors`.
+
+I haven't seen this used in the wild, but its existence gives me a headache.
+
+```ruby
+  # Example A, B, C, and D are semantically the same
+
+  # A: Filter both on colors and the block
+  describe things.colors(:red) { thing_id < 2 } do
+    its('count') { should cmp 1 }
+    its('thing_ids') { should include 1 }
+  end
+
+  # B use one where block
+  describe things.where { color == :red && thing_id < 2 } do
+    its('count') { should cmp 1 }
+    its('thing_ids') { should include 1 }
+  end
+
+  # C use two where blocks
+  describe things.where { color == :red }.where { thing_id < 2 } do
+    its('count') { should cmp 1 }
+    its('thing_ids') { should include 1 }
+  end
+
+  # D use a where param and a where block
+  describe things.where(color: :red) { thing_id < 2 } do
+    its('count') { should cmp 1 }
+    its('thing_ids') { should include 1 }
+  end
+
+  # This has nothing to do with colors at all, and may be broken - the lack of an arg to `colors` may make it never match
+  describe things.colors { thing_id < 2 } do
+    its('count') { should cmp 1 }
+  end
+```
+
+### You can call `params` or `raw_data` on any Table to get the raw data
+
+People _definitely_ use this out in the wild. Unlike `entries`, which wraps each row in a Struct and omits undeclared fields, `raw_data` simply returns the actual raw data array-of-hashes.  It is not `dup`'d.
+
+```ruby
+  tacky_things = things.where(color: :blue).raw_data.select { |row| row[:tackiness] }
+  tacky_things.map { |row| row[:thing_id] }.each do |thing_id|
+    # Use to audit a singular Thing
+    describe thing(thing_id) do
+      it { should_not be_paisley }
+    end
+  end
+```
+
+### You can call `resource_instance` on any Table to get the resource instance
+
+You could use this to do something fairly complicated.
+
+```ruby
+  describe things.where do # Just getting a Table object
+    its('resource_instance.some_method') { should cmp 'some_value' }
+  end
+```
+
+However, the resource instance won't know about the filtration, so I'm not sure what good this does.  Chances are, someone is doing something horrid using this feature in the wild.
+
+## Lazy Loading
+
+### What is Lazy Loading
+
+In some cases, the raw data may require multiple actions to populate.  For example, if you wanted a list of processes, and their open files, you might need to call 'ps' once, then 'lsof' one or more times.  That would become slow, and so you would only want to do it if you knew it was going to be used.
+
+Lazy loaded columns are absent in the raw data, until they are accessed (either by method-where, block-where, or a list property).  When they are accessed, a user-provided Lambda is called, which populates one or more columns.  FilterTable remembers which lazy columns have been populated, and will not call the lambda again.
+
+### Declaring a lazy field
+
+You declare a field to be lazy by providing an option, `lazy`, whose value is the lambda to be called.
+
+You can use the 'stabby lambda' syntax:
+
+```ruby
+filter_table_config.register_column(
+  :open_files,
+  field: :files,
+  lazy: ->() {|r,c,t| r[:files] = lookup_files_for_pid(r[:pid])},
+)
+```
+
+You can also refer to a *class* method.  You cannot use an instance method, because FilterTable binds to the resource class, not the resource instance.
+
+```ruby
+def self.populate_lsof(row, criteria, table)
+  row[:files] = ...
+end
+filter_table_config.register_column(
+  :open_files,
+  field: :files,
+  lazy: method(:populate_lsof),
+)
+```
+
+### Arguments to the fetcher lambda
+
+The lambda will be provided three arguments:
+
+1. `row`.  This is a Hash, the current row of the raw_data. You will likely need to examine this to find an ID value or other field that will act as a search key for your fetch.  You are expected to add one or more entries to this hash, as a result of your fetch.
+2. `condition`.  In some cases, a condition (desired value) is provided; the semantics of this are up to you.
+3. `table`. A reference to the FilterTable.  You can use this to access other context - including the entire raw data (`table.raw_data`) or the resource instance (`table.resource_instance`).
+
+### Clobbering
+
+Lazy-loading will not clobber an existing value in raw data.  For example:
+
+```ruby
+# Your raw data table:
+[
+  { id: 1 },
+  { id: 2, color: :blue },
+  { id: 3 },
+]
+
+# On lazy load, set all rows to color red
+filter_table_config.register_column(
+  :colors,
+  field: :color,
+  lazy: ->() { |r,c,t| r[:color] = :red },
+)
+
+# Trigger a fetch
+my_resource.colors => [:red, :blue, :red]
+
+# Raw data now:
+[
+  { id: 1, color: :red },
+  { id: 2, color: :blue },
+  { id: 3, color: :red },
+]
+```
+Note that not only was the `:color` blue not overwritten, in fact the fetcher lambda was only called twice.
+
+### Can I set multiple columns at once?
+
+Yes.  If your fetching action provides you with data to populate multiple columns, you are free to set any columns you wish in the `row`.
+
+You can even have multiple lazy columns share an implementation; the first one to be called will populate all the columns that share that implementation, and if any of the others are later triggered, the no-clobber effect will kick in, and the fetcher will not be called again.
+
+### Can I set multiple rows at once?
+
+Yes.  Using `table.raw_data`, you could perform a column-at-once population.  After the fetcher was called for the first row, all other rows would already be populated, so the fetcher would not be called again due to the no-clobber effect.
+
+## Gotchas and Surprises
+
+### Methods defined with `register_column` will change their return type based on their call pattern
+
+To me, calling things.thing_ids should always return the same type of value.  But if you call it with args or a block, it not only runs a filter, but also changes its return type to Table.
+
+```ruby
+
+  # This is an Array of color values (symbols, here)
+  things.colors
+
+  # This is a FilterTable::Table and these are equivalent
+  things.colors(:red)
+  things.where(color: :red)
+
+  # This is a FilterTable::Table and these are equivalent
+  things.colors { color == :red } # I think there is a bug here which makes this never match
+  things.where(color: :red)
+
+```
+
+### `entries` will not have fields present in the raw data
+
+`entries` will only know about the fields declared by `register_column` with `field:`. And...
+
+### `entries` will have things that are not fields
+
+Each time you call `register_custom_property`, `register_custom_matcher` or `register_column` - even for things like `count` and `exists?` - that will add an attribute to the Struct that is used to represent a row.  Those attributes will always be nil.
+
+### `register_column` does not know about what fields are in the raw data
+
+This is because the raw data fetcher is not called until as late as possible.  That's good - it might be expensive - but it also means we can't scan it for columns.  There are ways around that.
+
+### `where` param-mode and raw data access is sensitive to symbols vs strings
+
+### You can't call resource methods on a Table directly
+
+### You can't use a column name in a `where` block unless it was declared as a field using `register_column`
+
+```ruby
+  # This will give a NameError - :tackiness is in the raw
+  # data hash but not declared using `register_custom_property`.
+  describe things.where { tackiness == 'very' } do
+    its('count') { should cmp 1 }
+  end
+  # NameError: undefined local variable or method `tackiness' for #<struct :exists?=nil, count=nil, id=nil>
+
+  # But this works:
+  describe things.where(tackiness: 'very') do
+    its('count') { should cmp 1 }
+  end
+```
+
+### The eval context of a where block is an anonymous Struct class
+
+You can't get to the resource or the table from there.  (It's the Entry Struct type).
+
+### You can in fact filter for `nil` values
+
+### There is no obvious accessor for the Table instance
+
+You can in fact get the FilterTable::Table instance by calling `where` with no args.  But that is not obvious.
+
+### There is no way to get the FilterTable::Factory object used to configure the resource
+
+Especially while developing in inspec shell, it would be nice to be able to get at the FilterTable::Factory object, perhaps to add more accessors.
+