lazy_graph 0.1.0

data/README.md

@@ -0,0 +1,562 @@
+<table>
+  <tr>
+    <td><img src="./logo.png" alt="logo" width="150"></td>
+    <td>
+      <h1 align="center">LazyGraph</h1>
+      <p align="center">
+        <a href="https://rubygems.org/gems/lazy_graph">
+          <img alt="GEM Version" src="https://img.shields.io/gem/v/lazy_graph?color=168AFE&include_prereleases&logo=ruby&logoColor=FE1616">
+        </a><br>
+        <a href="https://rubygems.org/gems/lazy_graph">
+          <img alt="GEM Downloads" src="https://img.shields.io/gem/dt/lazy_graph?color=168AFE&logo=ruby&logoColor=FE1616">
+        </a>
+      </p>
+    </td>
+  </tr>
+</table>
+
+# LazyGraph
+
+<details>
+  <summary><strong>Table of Contents</strong></summary>
+
+1. [Introduction](#introduction)
+2. [Features](#features)
+3. [Installation](#installation)
+4. [Getting Started](#getting-started)
+   - [Defining a Schema](#defining-a-schema)
+   - [Providing Input](#providing-input)
+   - [Querying the Graph](#querying-the-graph)
+5. [Advanced Usage](#advanced-usage)
+   - [Builder DSL](#builder-dsl)
+   - [Derived Properties and Dependency Resolution](#derived-properties-and-dependency-resolution)
+   - [Debug Mode & Recursive Dependency Detection](#debug-mode--recursive-dependency-detection)
+   - [Advanced Path Syntax](#advanced-path-syntax)
+   - [LazyGraph Server](#lazygraph-server)
+6. [API Reference and Helpers](#api-reference-and-helpers)
+7. [Contributing](#contributing)
+8. [License](#license)
+
+</details>
+
+## Introduction
+
+**LazyGraph** is an ideal tool for building efficient, complex rules engines and optionally exposing these as stateless HTTP services.
+Unlike traditional rules engines which utilize facts to manipulate stateful memory,
+LazyGraph encodes rules into a stateless and declarative knowledge graph.
+
+A LazyGraph is a similar to a limited [**JSON Schema**](https://json-schema.org/), which allows a single structure
+to define both the shape of your data domain, and all of the rules that should run within it.
+
+To use it, you can:
+
+1. Define a **JSON Schema-like** structure describing how data in your domain is structured, containing required, optional and derived properties.
+   - **Rules**: All properties that can be computed are marked with a `rule` property, which defines how to compute the value in plain old Ruby code.
+     The rule can reference other properties in the schema, allowing for complex, nested, and recursive computations.
+
+2. Feed an **input document** (JSON) that partially fills out the schema with actual data.
+
+3. Query the `LazyGraph` for computed outputs.
+
+`LazyGraph` will:
+* Validate that the input conforms to the schema’s structure and then
+* Allow you to intelligently query the graph for information, lazily triggering the evaluation of rules (resolving dependencies in a single pass and caching results).
+
+The final output is the queried slice of your JSON schema, filled with computed outputs.
+
+The `LazyGraph` library also includes:
+
+- A **Builder DSL** to dynamically compose targeted JSON Schemas in Ruby.
+- An optional **HTTP server** to serve dynamic computations from a simple endpoint
+(serve any collection of LazyGraph builders as a stateless rules microservice, no additional code required)!.
+
+## Features
+
+- **Lazy Evaluation & Caching**
+  Derived fields are efficiently calculated, at-most-once, on-demand.
+
+- **Recursive Dependency Check**
+  Automatically detects cycles in your derived fields and logs warnings/debug info if recursion is found.
+
+- **Debug Trace**
+  The order in which recursive calculations are processed is not always obvious.
+  LazyGraph is able to provide a detailed trace of exactly, when and how each value was computed.
+
+- **Rich Querying Syntax**
+  Extract exactly what you need from your model with an intuitive path-like syntax. Support for nested properties, arrays, indexing, ranges, wildcards, and more.
+
+- **Composable Builder DSL**
+  Support dynamic creation of large, composable schemas in Ruby with a simplified syntax.
+
+- **Optional HTTP Server**
+  Spin up an efficient server that exposes several dynamic lazy-graphs as endpoints, allowing you to:
+    * Select a dynamic schema
+    * Feed in inputs and a query
+    * Receive the computed JSON output
+  all at lightning speed.
+
+## Installation
+
+Add this line to your application’s Gemfile:
+
+```ruby
+gem 'lazy_graph'
+```
+
+And then execute:
+
+```bash
+bundle
+```
+
+Or install it yourself:
+
+```bash
+gem install lazy_graph
+```
+
+## Getting Started
+
+In this section, we’ll explore how to set up a minimal LazyGraph use case:
+
+1. **Define** a JSON Schema (with LazyGraph’s extended properties and types).
+2. **Provide** an input document.
+3. **Query** the graph to retrieve computed data.
+
+### Defining a Schema
+
+A LazyGraph schema looks like a standard JSON Schema, but with a few key differences:
+* The `rule` property, which defines how to compute derived fields.
+  - `rule:`
+    Defines a that computes this property’s value, if not given, referencing other fields in the graph.
+* The schema also does not support computation across node types of `oneOf`, `allOf`, `anyOf`, `not`, or references
+(read examples for alternative mechanisms for achieving similar flexibility in your live schema)
+
+Any field (even object and array fields) can have a `rule` property.
+
+Values at this property are lazily computed by the rule, if not present in the input.
+However, if the value is present in the input, the rule is not triggered.
+This makes a LazyGraph highly flexible.
+(you can override *absolutely any* step in a complex computation graph, if you choose).
+
+Here’s a simple **shopping cart** example:
+
+```ruby
+require 'lazy_graph'
+
+cart_schema = {
+  type: 'object',
+  properties: {
+    cart: {
+      type: 'object',
+      properties: {
+        items: {
+          type: 'array',
+          items: {
+            type: 'object',
+            properties: {
+              name: {
+                type: 'string'
+              },
+              price: {
+                type: 'number',
+                default: 1.0
+              },
+              quantity: {
+                type: 'number',
+                default: 1
+              },
+              total: {
+                type: 'number',
+                rule: '${price} * ${quantity}'
+              }
+            }
+          },
+          required: ['name']
+        },
+        cart_total: {
+          type: 'number',
+          rule: {
+            inputs: {item_totals: 'items.total'},
+            calc: 'item_totals.sum'
+          }
+        }
+      }
+    }
+  }
+}
+
+cart_graph = cart_schema.to_lazy_graph
+```
+
+### Providing Input
+
+Next, we create an **input document** that partially fills out the schema. For instance:
+
+```ruby
+input_data = {
+  cart: {
+    items: [
+      { name: 'Widget', price: 5.0, quantity: 2 },
+      { name: 'Gadget' }
+    ]
+  }
+}
+```
+
+- `Widget` is fully specified with `price=5.0` and `quantity=2`.
+- `Gadget` is missing `price` and `quantity`, so it will use defaults (`1.0` for price and `1` for quantity).
+
+### Querying the Graph
+
+To compute derived fields and extract results, we can do:
+
+```ruby
+# Create the graph and run the query:
+graph_context = cart_graph.context(input_data)
+
+# If we query '' (empty string), we get the entire graph with computed values:
+whole_output = graph_context['']
+puts JSON.pretty_generate whole_output
+# {
+#   "output": {
+#     "cart": {
+#       "items": [
+#         {
+#           "name": "Widget",
+#           "price": 5.0,
+#           "quantity": 2,
+#           "total": 10.0
+#         },
+#         {
+#           "name": "Gadget",
+#           "price": 1.0,
+#           "quantity": 1,
+#           "total": 1.0
+#         }
+#       ],
+#       "cart_total": 11.0
+#     }
+#   },
+#   "debug_trace": null
+# }
+
+# Query a specific path, e.g. "cart.items[0].total"
+graph_context["cart.items[0].total"]
+# => {output: 10.0, debug_trace: nil}
+
+# e.g. "cart.items.total"
+graph_context["cart.items.total"]
+# => {output: [10.0, 1.0], debug_trace: nil}
+
+
+# e.g. "cart.items[name, total]"
+all_item_name_and_totals = graph_context["cart.items[name, total]"]
+# => {output: [{name: "Widget", total: 10.0}, {name: "Gadget", total: 1.0}], debug_trace: nil}
+```
+
+LazyGraph **recursively computes** any derived fields, referenced in the query.
+If you query only a subset of the graph, only the necessary computations are triggered, allowing for very fast responses,
+even on graphs with millions of interdependent nodes.
+
+## Advanced Usage
+
+### Builder DSL
+
+Rather than manually writing JSON schemas, LazyGraph also offers a **Ruby DSL** for building them.
+This is useful if your schema needs to be dynamic (variable based on inputs), has repeated patterns, or is built across multiple modules,
+which you wish to allow a user to combine in different ways.
+
+```ruby
+module ShoppingCart
+  class CartBuilder < LazyGraph::Builder
+    rules_module :cart_base do
+      object :cart do
+        array :items, required: true do
+          items do
+            string :name, required: true
+            number :price, default: 1.0
+            number :quantity, default: 1
+            number :total, rule: '${price} * ${quantity}'
+          end
+        end
+
+        number :cart_total, rule: '${items.total}.sum'
+      end
+    end
+  end
+end
+
+# Then we can build a cart schema, feed input, and query:
+cart_schema = ShoppingCart::CartBuilder.cart_base
+context = cart_schema.context({
+  cart: {
+    items: [
+      { name: 'Widget', price: 10.0, quantity: 2 },
+      { name: 'Thingamajig' }
+    ]
+  }
+})
+
+puts context['cart.cart_total'] # => 21.0
+```
+
+### Rules and Dependency Resolution
+
+Rules let you define logic for computing new values from existing ones. LazyGraph:
+
+1. Gathers each property’s **dependencies** (in the example, `total` depends on `price` and `quantity`).
+2. Computes them **on-demand**, in a topological order, ensuring that any fields they rely on are already resolved.
+3. Caches the results so subsequent references don’t trigger re-computation.
+
+This graph-based approach means you can nest derived fields deeply without worrying about explicit ordering. If a derived field depends on another derived field, LazyGraph naturally handles the chain.
+
+#### Derived Rules DSL
+There are several different ways to define Rules in a LazyGraph.
+The most portable way (using plain old JSON) is to define a rule as a string that references other properties in the schema using
+${} placeholder syntax.
+E.g.
+
+```ruby
+rule: '${price} * ${quantity}'
+```
+
+You can define the inputs separately from the calculation, which can be useful for more complex rules:
+
+As an array if you do not need to map paths
+```ruby
+rule: {
+  inputs: %[quantity],
+  calc: 'quantity.sum'
+}
+```
+
+As a hash of input names to resolution paths
+```ruby
+rule: {
+  inputs: {item_totals: 'items.total'},
+  calc: 'item_totals.sum'
+}
+```
+
+The most expressive way to define rules is to use a Ruby block, proc or lambda.
+The arguments to the block are automatically resolved as inputs into the block.
+You can use keyword arguments to map paths to the input names.
+
+*Note:* Block rules cannot be define from inside a REPL, as LazyGraph needs to read these from disk
+to be able to include the source code inside debug outputs.
+```ruby
+# The input price is resolved to the value at path: 'price'
+# The input quantity is resolved to the value at path:  'quantity'
+# The input store_details is resolved to the value at path: 'store.details'
+rule: ->(price, quantity, store_details: store.details) {
+  price * quantity * store_details.discount
+}
+```
+
+Resolutions are relative to the current node.
+I.e. it will look for the path in the current node, and then in the parent node, and so on, until it finds a match.
+If you wish to make an absolute reference, you can prefix the path with a `$` to indicate that it should start at the root of the schema.
+Note, inside lambda rules, absolute references begin with a _ instead.
+
+E.g.
+```ruby
+rule: ->(store_details: _.store.details) {
+  store_details.discount * price * quantity
+}
+```
+
+Inside rules, the scope is set such that you are able to freely access any other node in the computation graph.
+Just type a variable by name and it will automatically be recursively resolved to the correct value.
+*However* it is essential that you explicitly define all inputs to the rule to ensure resolution is correct,
+as LazyGraph will not automatically resolve any variables that are dynamically accessed.
+This is advanced functionality, and should be used with caution. In general, it is best to define all input dependencies explicitly.
+
+### Debug Mode & Recursive Dependency Detection
+
+If you pass `debug: true`, the output will contain an "output_trace" array,
+containing a detailed, ordered log of how each derived field was computed (inputs, calc and outputs).
+
+```ruby
+context = cart_schema.to_graph_ctx({
+  cart: {
+    items: [
+      { name: 'Widget', price: 10.0, quantity: 2 },
+      { name: 'Thingamajig' }
+    ]
+  }
+}, debug: true)
+result = context['cart.cart_total'] # triggers computations
+
+puts result[:debug_trace]
+# =>
+# [{output: :"$.cart.items[0].total",
+#   result: 20.0,
+#   inputs: {aa22c03893cb0018e: 10.0, a2fb52b44379212e4: 2},
+#   calc: ["aa22c03893cb0018e * a2fb52b44379212e4"],
+#   location: "$.cart.items[0]"},
+#  {output: :"$.cart.items[1].total",
+#   result: 1.0,
+#   inputs: {aa22c03893cb0018e: 1.0, a2fb52b44379212e4: 1},
+#   calc: ["aa22c03893cb0018e * a2fb52b44379212e4"],
+#   location: "$.cart.items[1]"},
+#  {output: :"$.cart.cart_total", result: 21.0, inputs: {item_totals: [20.0, 1.0]}, calc: ["item_totals.sum"], location: "$.cart"}]
+```
+
+In cases where you accidentally create **circular dependencies**, LazyGraph will log warnings to the debug logs, and detect and break infinite loops
+in the dependency resolution, ensuring that the remainder of the graph is still computed correctly.
+
+### Advanced Path Syntax
+
+LazyGraph’s query engine supports a flexible path notation:
+
+- **Nested properties**: `"cart.items[0].total"`, `"cart.items[*].name"`, etc.
+- **Object bracket expansions**: If a property is enclosed in `[keyName]`, the key and value are kept together when extracting partial JSON. For example, `"cart[items]"` yields `{"cart"=>{"items"=>[...]}}`.
+- **Array indexing**: `"items[0]"`, `"items[0..2]"`, `"items[*]"` (all items), or `"items[0, 2, 4]"` for picking multiple indexes.
+- **Ranged queries**: `"items[1..3]"` returns a slice from index 1 to 3 inclusive.
+- **Root Queries**: 'Useful inside rules for depdencies that are not relative to the current node. E.g. `"$.cart.items[0].total"` (or `_.cart.items[0].total` inside proc rules).
+
+### LazyGraph Server
+
+For situations where you want to serve rules over HTTP:
+
+1. **Define** your schema(s) with the DSL or standard JSON approach.
+2. **Implement** a small server that:
+   - Instantiates the schema.
+   - Takes JSON input from a request.
+   - Runs a query (passed via a query parameter or request body).
+   - Returns the computed JSON object.
+
+A minimal example might look like:
+
+[TODO] Verify example.
+```ruby
+require 'lazy_graph'
+require 'lazy_graph/server'
+
+module CartAPI
+  VERSION = '0.1.0'
+
+  # Add all classes that you want to expose in the API, as constants to the builder group module.
+  # These will turn into downcased, nested endpoints.
+  # E.g.
+  # /cart/v1
+  # - GET: Get module info
+  # - POST: Run a query
+  #   Inputs: A JSON object with the following keys:
+  #
+  #    - modules: { cart: {} } # The modules to merged into the combined schema,
+  #                            # you can pass multiple modules here to merge them into a single schema.
+                               # The keys inside each module object, are passed as arguments to the rules_module dsl method
+                               # to allow you to dynamically adjust your output schema based on API inputs.
+  #    - query: "cart.cart_total" # The query to run. Can be a string, an array of strings or empty (in which case entire graph is returned)
+  #    - context: { cart: { items: [ { name: "Widget", price: 2.5, quantity: 4 } ] } } # The input data to the schema
+  module Cart
+    class V1 < LazyGraph::Builder
+      rules_module :cart_base do |foo:, bar:|
+        object :cart do
+          array :items, required: true do
+            items do
+              string :name, required: true
+              number :price, default: 1.0
+              number :quantity, default: 1
+              number :total, rule: '${price} * ${quantity}'
+            end
+          end
+
+          number :cart_total, rule: '${items.total}.sum'
+        end
+      end
+    end
+  end
+
+  # Bootstrap our builder group.
+  include LazyGraph::BuilderGroup.bootstrap!(reload_paths: File.join(__dir__, 'cart_api/**/*.rb'))
+end
+```
+
+`config.ru`
+```ruby
+require_relative 'lib/paybun'
+
+run CartAPI.server
+```
+
+Then send requests like:
+
+```bash
+curl -X POST http://localhost:9292/cart/v1 \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "modules": { "cart" : {} },
+    "query": "cart.cart_total",
+    "context": {
+      "cart": {
+        "items": [
+          { "name": "Widget", "price": 2.5, "quantity": 4 }
+        ]
+      }
+    }
+  }'
+```
+
+Response:
+
+```json
+{
+  "type": "success",
+  "result": {
+    "output": 10.0
+  }
+}
+```
+
+## Where LazyGraph Fits
+
+If you’re coming from a background in any of the following technologies, you might find several familiar ideas reflected in LazyGraph:
+
+* *Excel*: Formulas in spreadsheets operate similarly to derived fields—they reference other cells (properties) and recalculate automatically when dependencies change.
+LazyGraph extends that principle to hierarchical or deeply nested data, making it easier to manage complex relationships than typical flat spreadsheet cells.
+*	*Terraform*: Terraform is all about declarative configuration and automatic dependency resolution for infrastructure.
+LazyGraph brings a similar ethos to your application data—you declare how fields depend on one another, and the engine takes care of resolving values in the correct order, even across multiple modules or partial inputs.
+*	*JSON Schema*: At its core, LazyGraph consumes standard JSON Schema features (type checks, required fields, etc.) but introduces extended semantics (like derived or default).
+If you’re already comfortable with JSON Schema, you’ll find the basic structure familiar—just expanded to support making your data dynamic and reactive.
+*	*Rules Engines* / Expert Systems: Traditional rule engines (like Drools or other Rete-based systems) let you define sets of conditional statements that trigger when facts change.
+In many scenarios, LazyGraph can serve as a lighter-weight alternative: each “rule” (i.e., derived property) is defined right where you need it in the schema, pulling its dependencies from anywhere else in the data. You get transparent, on-demand evaluation without the overhead of an external rules engine. Plus, debug traces show exactly how each value was computed, so there’s no confusion about which rules fired or in which order.
+*	*JSON* Path, GraphQL, jq: These tools are great for querying JSON data, but they don't handle automatic lazy dependency resolution.
+LazyGraph’s querying syntax is designed to be expressive and powerful, allowing you to extract the data you need (triggering only the required calculations) from a complex graph of computed values.
+
+### Typical Use Cases
+You can leverage these concepts in a variety of scenarios:
+* Rules Engines: Encode complex, nested and interdepdendent rules into a single, easy to reason about, JSON schema.
+* Complex Chained Computation: Where multiple interdependent properties (e.g. finance calculations, policy checks) must auto-update when a single input changes.
+* Dynamic Configuration: Generate derived settings based on partial overrides without duplicating business logic everywhere.
+* Form or Wizard-Like Data: Reactively compute new fields as users provide partial inputs; only relevant properties are evaluated.
+* Stateless Microservices / APIs: Provide an HTTP endpoint that accepts partial data, automatically fills in defaults and computed fields, and returns a fully resolved JSON object.
+
+## API Reference and Helpers
+
+For more fine-grained integration without the DSL or server, you can use the lower-level Ruby API entry points:
+
+E.g. where `my_schema` is a LazyGraph compliant JSON Schema:
+
+```ruby
+# 1. Turn the schema hash into a lazy graph. (From which you can generate contexts)
+graph = my_schema.to_lazy_graph() # Optional args debug: false, validate: true
+
+# 2. Immediately create a context from the graph and input data.
+ctx = my_schema.to_graph_ctx(input_hash) # Optional args debug: false, validate: true
+
+# 3. Immediately evaluate a query on context given the graph and input data.
+result = my_schema.eval!(input_hash, 'some.query') # Optional args debug: false, validate: true
+```
+
+These methods allow you to embed LazyGraph with minimal overhead if you already have your own project structure.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at [https://github.com/wouterken/lazy_graph](https://github.com/wouterken/lazy_graph). This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to follow our [code of conduct](./CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+require 'bundler/gem_tasks'
+task default: %i[]

data/examples/performance_tests.rb

@@ -0,0 +1,117 @@
+require 'vernier'
+require 'memory_profiler'
+require 'benchmark/ips'
+require 'lazy_graph'
+
+class PerformanceBuilder < LazyGraph::Builder
+  rules_module :performance, {} do
+    integer :employees_count, rule: { inputs: 'employees', calc: 'employees.size' }
+
+    array :employees, required: true do
+      items do
+        string :id
+        array :positions, required: true, default: [] do
+          items do
+            string :pay_schedule_id, invisible: true
+            string :position_id, invisible: true
+
+            object :position, rule: :"${$.positions[position_id]}" do
+              number :base_rate
+              number :salary
+              number :bonus, rule: :'${base_rate} * 0.1'
+            end
+            object :pay_schedule, rule: :'${pay_schedules[pay_schedule_id]}'
+            number :base_rate, rule: :"${position.base_rate}"
+            string :employee_id, rule: :id
+          end
+        end
+      end
+    end
+
+    object :positions do
+      object :".*", pattern_property: true do
+        number :base_rate
+        number :salary, default: 100_000
+      end
+    end
+
+    object :pay_schedules do
+      object :".*", pattern_property: true do
+        string :payment_frequency, enum: %w[weekly biweekly semi-monthly monthly],
+                                   description: 'Payment frequency for this pay schedule.'
+      end
+    end
+  end
+end
+
+def gen_employees(n, m = 10)
+  {
+    employees: n.times.map do |i|
+      {
+        id: i.to_s,
+        positions: Random.rand(0..4).times.map do
+          {
+            position_id: Random.rand(1...10).to_s,
+            pay_schedule_id: Random.rand(1...10).to_s
+          }
+        end
+      }
+    end,
+    pay_schedules: [*1..m].map do |i|
+      [i, {
+        payment_frequency: %w[monthly weekly].sample
+      }]
+    end.to_h,
+    positions: [*1..m].map do |i|
+      [i, {
+        base_rate: Random.rand(10..100)
+      }]
+    end.to_h
+  }
+end
+
+def profile_n(n, debug: false, validate: false)
+  employees = gen_employees(n)
+  graph = PerformanceBuilder.performance.build!(debug: debug, validate: validate)
+  Vernier.profile(out: './examples/time_profile.json') do
+    start = Time.now
+    graph.context(employees).query('')
+    ends = Time.now
+    puts "Time elapsed: #{ends - start}"
+  end
+end
+
+def memory_profile_n(n, debug: false, validate: false)
+  employees = gen_employees(n)
+  graph = PerformanceBuilder.performance.build!(debug: debug, validate: validate)
+  report = MemoryProfiler.report do
+    graph.context(employees).query('')
+  end
+  report.pretty_print
+end
+
+def benchmark_ips_n(n, debug: false, validate: false)
+  graph = PerformanceBuilder.performance.build!(debug: debug, validate: validate)
+  employees = gen_employees(n)
+  Benchmark.ips do |x|
+    x.report('performance') do
+      graph.context(employees).query('')
+    end
+    x.compare!
+  end
+end
+
+def console_n(n, debug: false, validate: false)
+  require 'debug'
+  graph = PerformanceBuilder.performance.build!(debug: debug, validate: validate)
+  employees = gen_employees(n)
+  result = graph.context(employees).query('')
+  binding.b
+end
+
+case ARGV[0]
+when 'ips' then benchmark_ips_n(ARGV.fetch(1, 1000).to_i)
+when 'memory' then memory_profile_n(ARGV.fetch(1, 1000).to_i)
+when 'console' then console_n(ARGV.fetch(1, 1000).to_i, debug: true)
+else profile_n(ARGV.fetch(1, 100_000).to_i)
+end