pytrilogy 0.3.142__cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

trilogy/core/processing/README.md

@@ -0,0 +1,94 @@
+
+## Query Planning
+
+Query planning is divided into 3 core phases.
+
+The first phase builds an abstract node tree by looping through every combination of
+output concept and keys in the output query grain and recursively searching for sources.
+
+It will begin with aggregations if those exist, then window functions, then filtration functions,
+rowsets, and finally look for bare selects.
+
+Each type of complex node will generate a new recursive node search for required parents,
+until a set of terminal nodes with base concept selection is reached. 
+
+A default merge node is injected between every recursion. The overall loop will terminate early 
+if an output node is returned with all required query concepts. If not, the merge node will
+handle a join between the returned subtrees. If there are not multiple nodes to merge,
+the merge node will simply return the single parent node and prune itself from the graph.
+
+In the second pass, each node is resolved to an abstract CTE. At this phase, CTEs that 
+reference the same tables, parent CTEs, and filtering can be merged.
+
+Finally, in query rendering each CTE is rendered to a backend appropriate query. The final
+CTE, or the `base`, will contain all required columns for the final output. The last
+select will only apply any query level filters + ordering, no joins will take place.
+
+## Aug 2023 Update
+
+For complex derivations, propogating the "full" context upstream is an issue. Instead, we need to adjust logic to prune the optional nodes in each search pattern.
+
+For filter nodes -> we should have these generate a node with _just_ the filtered column + the new concept. 
+
+## Debugging
+
+Base query derivation accepts the 'DebugHook' defined under hooks, which will print to console
+each step of the plan. This is a great first step to figure out what might be going
+wrong with discovery in a query. 
+
+Example usage
+
+```python
+from preql import parse
+from preql.core.query_processor import process_query
+from preql.hooks.query_debugger import DebuggingHook
+from preql.core.models import Select
+
+declarations = """
+key user_id int metadata(description="the description");
+property user_id.display_name string metadata(description="The display name ");
+property user_id.about_me string metadata(description="User provided description");
+
+
+key post_id int;
+metric post_count <-count(post_id);
+
+
+datasource posts (
+user_id: user_id,
+id: post_id
+)
+grain (post_id)
+address bigquery-public-data.stackoverflow.post_history
+;
+
+select
+user_id,
+count(post_id) -> user_post_count
+;
+
+metric avg_user_post_count <- avg(user_post_count);
+
+
+datasource users (
+id: user_id,
+display_name: display_name,
+about_me: about_me,
+)
+grain (user_id)
+address bigquery-public-data.stackoverflow.users
+;
+
+
+select
+avg_user_post_count
+;
+
+
+"""
+env, parsed = parse(declarations)
+select: Select = parsed[-1]
+
+query = process_query(statement=select, environment=env, hooks=[DebuggingHook()])
+
+```

trilogy/core/processing/READMEv2.md

@@ -0,0 +1,121 @@
+
+
+## Execution Plan
+
+
+Query discovery is a recursive loop.
+
+Order of concepts to be discovered is prioritized by lineage type, with an additional sort so that if one concept
+is derived from another concept, the parent is delayed in order.
+
+This ensures that the parent is typically included in discovery of the child node path and results in a more 'ergonomic' query.
+
+In rare cases, a node may return partial results. Then the discovery loop will attempt to merge those.
+
+If it cannot merge, it will attempt to discover new concepts to inject into the search path that will result in a mergable graph.
+
+[see concept injection for more.]
+
+## Filtering
+Filtering via where clauses is always pushed up as high as possible by passing the condition object through to sourcing. 
+
+If at any point we have a discovery loop where the contents of the where clause are included in the filtering, we need to immediately inject them.
+
+If we never hit that point, filtering will be injected when we have only root or constant nodes left.
+
+## Psuedonyms
+
+Pseudonyms should always be handled by a node returning the pseudonymous type.
+
+Ex: if A* has been merged into B*, and B* cannot be found, but A* can, the node returning A* should return A* and let the merge loop reconcile. 
+
+
+To be more specific about desired behavior:
+
+Nodes must accurately track the *actual* concept being returned, because pseudonyms can have different rendering.
+
+Completion checks should always check for the matching concepts _or_ matching pseudonyms.
+
+The final query should map the output label to whatever the user actually specified.
+
+Things that might need work:
+- merging concepts with partial modifiers relies on datasources being updated with partial. If you merge a calculation, we should have additional tests that upstream discoveyr of the calculation
+verifies all partial constraints. 
+
+
+
+## Always pass up local optional
+## pass up filter
+Eg
+
+where z = 2,
+x, sum(y)
+
+
+loop 1
+start with sum(y), pass up x and condition z
+
+loop 2
+only scalar left
+
+get x, y, u via scalar resolution
+
+
+## With Complex in Where caluse
+
+where count(z) = 2,
+x, sum(y)
+
+
+loop 1
+start with sum(y), pass up x and condition z
+
+loop 2
+only scalar left for output
+x, y
+
+but condition has count(z) = 2
+
+need one more loop with all concepts
+
+loop 3:
+
+x, y, z
+
+get x,y,z via scalar resolution
+
+
+## Where Clause With Output Condition
+
+This is a tricky one
+
+where count(z) = a
+select
+    x,
+    count(z)
+
+First pass
+
+We need to get count(z), x AND a because we need to evaluate the condition here
+
+so required becomes x, a, count(z) by x
+
+loop 1:
+    count(z) optional x, a no filter
+
+
+[Agg Updates]
+
+18 Failing tests to start
+
+16 after fixing pseudonym binding
+
+17 after updating select nodes
+
+6 failing after using partial/full mappings
+
+3 failing after fixing persisted rebind handling
+
+3 [different] failing after fixing the materialized_cache
+
+1 failing after fixing order of injecting materialized roots

trilogy/core/processing/VIRTUAL_UNNEST.md

@@ -0,0 +1,30 @@
+# Virtual Unnest Nodes
+
+
+## The Problem
+
+We want to create a dynamic list - let's say a list of quarters. We can use an unnest or a union to create a constant field with [1,2,3,4].
+
+Now we have data - let's say quarter:sales that doesn't have all quarters. Maybe it has {2:3, 3:4} How do we merge this?
+
+If we run a naive merge 
+
+```
+auto quarters <- unnest([1,2,3,4]);
+
+merge data_quarter into ~quarters;
+
+Default discovery will say that since the unnest has no dependencies, we can just create it de-novo after fetching the sales.
+
+So we'll end up with 
+
+{1:3, 2:3, 3:3, 4:3, 1:4, 2:4, 3:4, 4:4}.
+
+When we want:
+
+{1:null,2:3, 3:4, 4: null}
+
+
+## Fix
+
+We need to regard a unnest node constant as a unique datasource.