pytrilogy 0.0.3.67__py3-none-any.whl → 0.0.3.69__py3-none-any.whl

{pytrilogy-0.0.3.67.dist-info → pytrilogy-0.0.3.69.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pytrilogy
-Version: 0.0.3.67
+Version: 0.0.3.69
 Summary: Declarative, typed query language that compiles to SQL.
 Home-page: 
 Author: 
@@ -40,46 +40,61 @@ Dynamic: summary
 [![Website](https://img.shields.io/badge/INTRO-WEB-orange?)](https://trilogydata.dev/)
 [![Discord](https://img.shields.io/badge/DISCORD-CHAT-red?logo=discord)](https://discord.gg/Z4QSSuqGEd)
 
-pytrilogy is an experimental implementation of the Trilogy language, a higher-level SQL that replaces tables/joins with a lightweight semantic binding layer.
+The Trilogy language is an experiment in better SQL for analytics - a streamlined SQL that replaces tables/joins with a lightweight semantic binding layer and provides easy reuse and composability. It compiles to SQL - making it easy to debug or integrate into existing workflows - and can be run against any supported SQL backend.  
 
-Trilogy looks like SQL, but simpler. It's a modern SQL refresh targeted at SQL lovers who want more reusability and composability without losing the expressiveness and iterative value of SQL. It compiles to SQL - making it easy to debug or integrate into existing workflows - and can be run against any supported SQL backend.  
+[pytrilogy](https://github.com/trilogy-data/pytrilogy) is the reference implementation, written in Python.
+
+Trilogy concretely solves these common problems in karge, SQL based analytics teams:
+- decoupling consumption code from specific physical assets
+- better testability and change management
+- reduced boilerplate and opportunity for OLAP style optimization at scale
+
+Trilogy can be especially powerful as a frontend consumption language, since the decoupling from the physical layout makes dynamic and interactive dashboards backed by SQL tables much easier to create.
 
 > [!TIP]
-> Try it online in a hosted [open-source studio](https://trilogydata.dev/trilogy-studio-core/). To get an overview of the language and run interactive examples, head to the [documentation](https://trilogydata.dev/).
+> You can try Trilogy in a [open-source studio](https://trilogydata.dev/trilogy-studio-core/). More details on the language can be found on the [documentation](https://trilogydata.dev/).
 
-Installation: `pip install pytrilogy`
+We recommend the studio as the fastest way to explore Trilogy. For deeper work and integration, `pytrilogy` can be run locally to parse and execute trilogy model [.preql] files using the `trilogy` CLI tool, or can be run in python by importing the `trilogy` package.
 
-`pytrilogy` can be run locally to parse and execute trilogy model [.preql] files using the `trilogy` CLI tool, or can be run in python by importing the `trilogy` package.
+Installation: `pip install pytrilogy`
 
-You can read more about the project [here](https://trilogydata.dev/) and try out an interactive demo [here](https://trilogydata.dev/demo/). 
+### Trilogy Looks Like SQL
 
-Trilogy looks like SQL:
 ```sql
+import names;
+
+const top_names <- ['Elvis', 'Elvira', 'Elrond', 'Sam'];
+
+def initcap(word) -> upper(substring(word, 1, 1)) || substring(word, 2, len(word));
+
 WHERE 
-    name like '%lvis%'
+    @initcap(name) in top_names
 SELECT
     name,
-    count(name) as name_count
+    sum(births) as name_count
 ORDER BY
     name_count desc
 LIMIT 10;
 ```
 ## Goals
-And aims to:
+Versus SQL, Trilogy aims to: 
 
-Preserve:
+Keep:
 - Correctness
 - Accessibility
 
-Enhance:
+Improve:
 - Simplicity
-- Understandability
 - Refactoring/mantainability
 - Reusability
 
 Maintain:
 - Acceptable performance
 
+Remove:
+- Lower-level procedural features
+- Transactional optimizations/non-analytics features
+
 ## Hello World
 
 Save the following code in a file named `hello.preql`
@@ -141,10 +156,7 @@ SELECT
     sentences.text
 ;
 
-
-
 ```
-
 Run the following from the directory the file is in.
 
 ```bash
@@ -157,11 +169,15 @@ trilogy run hello.trilogy duckdb
 
 The current Trilogy implementation supports these backends:
 
+### Core
 - Bigquery
-- SQL Server
 - DuckDB
 - Snowflake
 
+### Experimental
+- SQL Server
+- Presto
+
 ## Basic Example - Python
 
 Trilogy can be run directly in python through the core SDK. Trilogy code can be defined and parsed inline or parsed out of files.
@@ -170,7 +186,6 @@ A bigquery example, similar to bigquery [the quickstart](https://cloud.google.co
 
 ```python
 
-
 from trilogy import Dialects, Environment
 
 environment = Environment()

{pytrilogy-0.0.3.67.dist-info → pytrilogy-0.0.3.69.dist-info}/RECORD

@@ -1,5 +1,5 @@
-pytrilogy-0.0.3.67.dist-info/licenses/LICENSE.md,sha256=5ZRvtTyCCFwz1THxDTjAu3Lidds9WjPvvzgVwPSYNDo,1042
-trilogy/__init__.py,sha256=kIt0gE_lZ8Zmqhi-fcE9krdwlETcz0sP0kiCjnsBQog,303
+pytrilogy-0.0.3.69.dist-info/licenses/LICENSE.md,sha256=5ZRvtTyCCFwz1THxDTjAu3Lidds9WjPvvzgVwPSYNDo,1042
+trilogy/__init__.py,sha256=cI0erWMqoxAN4RoZk4oMIaGKoFySaz8SJ8_yrbpiYwA,303
 trilogy/compiler.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 trilogy/constants.py,sha256=lv_aJWP6dn6e2aF4BAE72jbnNtceFddfqtiDSsvzno0,1692
 trilogy/engine.py,sha256=OK2RuqCIUId6yZ5hfF8J1nxGP0AJqHRZiafcowmW0xc,1728
@@ -20,11 +20,11 @@ trilogy/core/functions.py,sha256=poVfAwet1xdxTkC7WL38UmGRDpUVO9iSMNWSagl9_r4,293
 trilogy/core/graph_models.py,sha256=wIT-oBchHWE46GLDkgN5K7EzhOBEo8LfaeWV5G5cYcE,3302
 trilogy/core/internal.py,sha256=iicDBlC6nM8d7e7jqzf_ZOmpUsW8yrr2AA8AqEiLx-s,1577
 trilogy/core/optimization.py,sha256=ojpn-p79lr03SSVQbbw74iPCyoYpDYBmj1dbZ3oXCjI,8860
-trilogy/core/query_processor.py,sha256=jA1lek3Kbpi_-NamDPjJuJanUdY7KM3ODB7tS_qqxH4,20311
+trilogy/core/query_processor.py,sha256=t91tYQS0o89r-acdxo8uDUpB1elNcqAhLWkbM8BFjho,20283
 trilogy/core/utility.py,sha256=3VC13uSQWcZNghgt7Ot0ZTeEmNqs__cx122abVq9qhM,410
 trilogy/core/models/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-trilogy/core/models/author.py,sha256=44VyEInWied287lJp99T14Ahq8RSQK4q3Y-AzAByMTY,77818
-trilogy/core/models/build.py,sha256=eNjbFwg_mEcwPeagQpkIybIluIrRCKJLAHdpZjm01e8,64857
+trilogy/core/models/author.py,sha256=Wz_6yEZS5EyXJ5wHHl-C44ikFka5XftBYOjNRK98Hfw,77790
+trilogy/core/models/build.py,sha256=BNHzfqD5NWCKHntvnswvDrtCD1JFgvPedx23cPq942U,65405
 trilogy/core/models/build_environment.py,sha256=s_C9xAHuD3yZ26T15pWVBvoqvlp2LdZ8yjsv2_HdXLk,5363
 trilogy/core/models/core.py,sha256=EMAuWTngoNVGCdfNrAY7_k6g528iodNQLwPRVip-8DA,10980
 trilogy/core/models/datasource.py,sha256=6RjJUd2u4nYmEwFBpJlM9LbHVYDv8iHJxqiBMZqUrwI,9422
@@ -37,24 +37,24 @@ trilogy/core/optimizations/predicate_pushdown.py,sha256=g4AYE8Aw_iMlAh68TjNXGP75
 trilogy/core/processing/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 trilogy/core/processing/concept_strategies_v3.py,sha256=zy5VZa9LITOws6aIILfv_bSR2-jR1Ndldy-nmwMyQ5w,23144
 trilogy/core/processing/discovery_loop.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-trilogy/core/processing/discovery_node_factory.py,sha256=73fViHQfKsrFbuk8zfx8YIiaT66pF5AFvw4suV7bXbM,14985
+trilogy/core/processing/discovery_node_factory.py,sha256=I3JJxoF-u8OVvqXXAOhvMg2h-KdpHQwg6EpCeQtxGCI,15123
 trilogy/core/processing/discovery_utility.py,sha256=3xdd1ypKappSDm0SJs7WtW5YegL80SlYhDQlkNePp4E,4549
 trilogy/core/processing/discovery_validation.py,sha256=fGWJmKpgEd1f4RkK-fYOBUT1cwsJnahwXFAdRlou7MI,5365
 trilogy/core/processing/graph_utils.py,sha256=8QUVrkE9j-9C1AyrCb1nQEh8daCe0u1HuXl-Te85lag,1205
-trilogy/core/processing/utility.py,sha256=mrfR9pgek-xjxoDQSlvPqOW9dpmREjgzqn4AGoqpGeM,22774
+trilogy/core/processing/utility.py,sha256=b1F3NT7-MP_-U4KmpC52BOAwLu6mybfndeA1iiZwChw,22016
 trilogy/core/processing/node_generators/__init__.py,sha256=w8TQQgNhyAra6JQHdg1_Ags4BGyxjXYruu6UeC5yOkI,873
 trilogy/core/processing/node_generators/basic_node.py,sha256=luN8LftafZepoFgDRv4gmvEGFlOI2j0icJ5fz4UT7uo,5165
 trilogy/core/processing/node_generators/common.py,sha256=PdysdroW9DUADP7f5Wv_GKPUyCTROZV1g3L45fawxi8,9443
 trilogy/core/processing/node_generators/filter_node.py,sha256=oRRq2-T3ufgn4D23uQsc58f20eFk-djs4QI3WKA75K8,10908
-trilogy/core/processing/node_generators/group_node.py,sha256=S7mzEAVmjc7rnHrJ9PAp4Ld8njcBSrvAu3K1cxsLQdY,6140
+trilogy/core/processing/node_generators/group_node.py,sha256=1QJhRxsTklJ5xq8wHlAURZaN9gL9FPpeCa1OJ7IwXnY,6769
 trilogy/core/processing/node_generators/group_to_node.py,sha256=jKcNCDOY6fNblrdZwaRU0sbUSr9H0moQbAxrGgX6iGA,3832
 trilogy/core/processing/node_generators/multiselect_node.py,sha256=GWV5yLmKTe1yyPhN60RG1Rnrn4ktfn9lYYXi_FVU4UI,7061
 trilogy/core/processing/node_generators/node_merge_node.py,sha256=dSqfqWp2SolhDB16nkPaaTXgNQo4QquEufPdf7q0Tb4,17398
 trilogy/core/processing/node_generators/recursive_node.py,sha256=l5zdh0dURKwmAy8kK4OpMtZfyUEQRk6N-PwSWIyBpSM,2468
 trilogy/core/processing/node_generators/rowset_node.py,sha256=2BiSsegbRF9csJ_Xl8P_CxIm4dAAb7dF29u6v_Odr-A,6709
-trilogy/core/processing/node_generators/select_merge_node.py,sha256=fisCkMyzNDEdDIi9BMuekpIJT5lBF4h0z_pEwR14x9s,21438
+trilogy/core/processing/node_generators/select_merge_node.py,sha256=2KIljy2TFLT9kjKUfhkFwSDkcu4hv4yRVTKOGzEe-NM,21763
 trilogy/core/processing/node_generators/select_node.py,sha256=Ta1G39V94gjX_AgyZDz9OqnwLz4BjY3D6Drx9YpziMQ,3555
-trilogy/core/processing/node_generators/synonym_node.py,sha256=F9DWaKEmJDYnwAmQduTuQP2LCCHqAMDA3oDERRDN2pU,3773
+trilogy/core/processing/node_generators/synonym_node.py,sha256=AnAsa_Wj50NJ_IK0HSgab_7klYmKVrv0WI1uUe-GvEY,3766
 trilogy/core/processing/node_generators/union_node.py,sha256=VNo6Oey4p8etU9xrOh2oTT2lIOTvY6PULUPRvVa2uxU,2877
 trilogy/core/processing/node_generators/unnest_node.py,sha256=ueOQtoTf2iJHO09RzWHDFQ5iKZq2fVhGf2KAF2U2kU8,2677
 trilogy/core/processing/node_generators/window_node.py,sha256=GP3Hvkbb0TDA6ef7W7bmvQEHVH-NRIfBT_0W4fcH3g4,6529
@@ -76,8 +76,8 @@ trilogy/core/statements/build.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hS
 trilogy/core/statements/common.py,sha256=KxEmz2ySySyZ6CTPzn0fJl5NX2KOk1RPyuUSwWhnK1g,759
 trilogy/core/statements/execute.py,sha256=rqfuoMuXPcH7L7TmE1dSiZ_K_A1ohB8whVMfGimZBOk,1294
 trilogy/dialect/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-trilogy/dialect/base.py,sha256=_EkBijBaJGF_d0SBNBCf_mbmDD7N9HrnO2XbRWM8ypg,43629
-trilogy/dialect/bigquery.py,sha256=6ghCqy-k7UioIJc1EEQ7gRo_PHaO8Vm7yYbiQ-kgpzs,3629
+trilogy/dialect/base.py,sha256=SpNv1D9eig-XDOui2xoeMvdTfRpejAfJ_HaAMkuPW9w,45112
+trilogy/dialect/bigquery.py,sha256=e19dGcarapgA0x5_Xmq2StyHzuDWPOOPaR4elkWXwug,4203
 trilogy/dialect/common.py,sha256=hhzuMTFW9QQIP7TKLT9BlJy6lw2R03a68jKQ-7t4-2c,6070
 trilogy/dialect/config.py,sha256=olnyeVU5W5T6b9-dMeNAnvxuPlyc2uefb7FRME094Ec,3834
 trilogy/dialect/dataframe.py,sha256=RUbNgReEa9g3pL6H7fP9lPTrAij5pkqedpZ99D8_5AE,1522
@@ -93,11 +93,11 @@ trilogy/hooks/graph_hook.py,sha256=5BfR7Dt0bgEsCLgwjowgCsVkboGYfVJGOz8g9mqpnos,4
 trilogy/hooks/query_debugger.py,sha256=1npRjww94sPV5RRBBlLqMJRaFkH9vhEY6o828MeoEcw,5583
 trilogy/metadata/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 trilogy/parsing/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-trilogy/parsing/common.py,sha256=_-KWPP3NvNAHiB4B6jyGSnYKqlLs2XqXeTZiX4MCtWU,30861
+trilogy/parsing/common.py,sha256=_5UEnLtu40VQ8gb6wg3GtSrxf6IONhEOntmdsm0X4lU,30961
 trilogy/parsing/config.py,sha256=Z-DaefdKhPDmSXLgg5V4pebhSB0h590vI0_VtHnlukI,111
 trilogy/parsing/exceptions.py,sha256=Xwwsv2C9kSNv2q-HrrKC1f60JNHShXcCMzstTSEbiCw,154
 trilogy/parsing/helpers.py,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
-trilogy/parsing/parse_engine.py,sha256=O7aM5nZ4SjKlqO2x8XWefI1BMCW06jYYLhABU4k1HCI,72430
+trilogy/parsing/parse_engine.py,sha256=fKsV5eGe31oTVDEjeGSv_pb-222XaVRHtVgzFSnixZk,72439
 trilogy/parsing/render.py,sha256=gGCFj2ue0UoaU2MR6qHGMAHXkYRMkTmHjnBowdcgFMY,19603
 trilogy/parsing/trilogy.lark,sha256=x9D1BXtE1E9Kxatx5Kt7xCaid8zgedabwca_B7j7L7o,14331
 trilogy/scripts/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
@@ -110,8 +110,8 @@ trilogy/std/money.preql,sha256=XWwvAV3WxBsHX9zfptoYRnBigcfYwrYtBHXTME0xJuQ,2082
 trilogy/std/net.preql,sha256=-bMV6dyofskl4Kvows-iQ4JCxjVUwsZOeWCy8JO5Ftw,135
 trilogy/std/ranking.preql,sha256=LDoZrYyz4g3xsII9XwXfmstZD-_92i1Eox1UqkBIfi8,83
 trilogy/std/report.preql,sha256=LbV-XlHdfw0jgnQ8pV7acG95xrd1-p65fVpiIc-S7W4,202
-pytrilogy-0.0.3.67.dist-info/METADATA,sha256=q69gBB9Ympec6F7E9ASBDjTfuRs06HmUws7QvAxVBeA,9095
-pytrilogy-0.0.3.67.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-pytrilogy-0.0.3.67.dist-info/entry_points.txt,sha256=ewBPU2vLnVexZVnB-NrVj-p3E-4vukg83Zk8A55Wp2w,56
-pytrilogy-0.0.3.67.dist-info/top_level.txt,sha256=cAy__NW_eMAa_yT9UnUNlZLFfxcg6eimUAZ184cdNiE,8
-pytrilogy-0.0.3.67.dist-info/RECORD,,
+pytrilogy-0.0.3.69.dist-info/METADATA,sha256=Je-V6aoBf1WujzpfMUFzAt1JduQTZ1ItagHgTRtubmU,9734
+pytrilogy-0.0.3.69.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+pytrilogy-0.0.3.69.dist-info/entry_points.txt,sha256=ewBPU2vLnVexZVnB-NrVj-p3E-4vukg83Zk8A55Wp2w,56
+pytrilogy-0.0.3.69.dist-info/top_level.txt,sha256=cAy__NW_eMAa_yT9UnUNlZLFfxcg6eimUAZ184cdNiE,8
+pytrilogy-0.0.3.69.dist-info/RECORD,,

trilogy/__init__.py

@@ -4,6 +4,6 @@ from trilogy.dialect.enums import Dialects
 from trilogy.executor import Executor
 from trilogy.parser import parse
 
-__version__ = "0.0.3.67"
+__version__ = "0.0.3.69"
 
 __all__ = ["parse", "Executor", "Dialects", "Environment", "CONFIG"]

trilogy/core/models/author.py

@@ -2303,7 +2303,7 @@ class CustomFunctionFactory:
         ]
         return self
 
-    def __call__(self, *creation_args: list[Expr]):
+    def __call__(self, *creation_args: Expr):
         nout = (
             self.function.model_copy(deep=True)
             if isinstance(self.function, BaseModel)
@@ -2389,7 +2389,6 @@ Expr = (
     | int
     | str
     | float
-    | list
     | date
     | datetime
     | TupleWrapper
@@ -2430,7 +2429,6 @@ FuncArgs = (
     | ListType
     | MapType
     | NumericType
-    | list
     | ListWrapper[Any]
     | TupleWrapper[Any]
     | Comparison

trilogy/core/models/build.py

@@ -155,6 +155,7 @@ def concepts_to_build_grain_concepts(
             pconcepts.append(c)
         elif environment:
             pconcepts.append(environment.concepts[c])
+
         else:
             raise ValueError(
                 f"Unable to resolve input {c} without environment provided to concepts_to_grain call"
@@ -248,7 +249,7 @@ def get_concept_arguments(expr) -> List["BuildConcept"]:
     return output
 
 
-class BuildParamaterizedConceptReference(BaseModel):
+class BuildParamaterizedConceptReference(DataTyped, BaseModel):
     concept: BuildConcept
 
     def __str__(self):
@@ -258,6 +259,10 @@ class BuildParamaterizedConceptReference(BaseModel):
     def safe_address(self) -> str:
         return self.concept.safe_address
 
+    @property
+    def output_datatype(self) -> DataType:
+        return self.concept.output_datatype
+
 
 class BuildGrain(BaseModel):
     components: set[str] = Field(default_factory=set)
@@ -1810,8 +1815,8 @@ class Factory:
             right_c, _ = self.instantiate_concept(base.right)
             right = right_c
         return BuildSubselectComparison.model_construct(
-            left=self.build(base.left),
-            right=self.build(right),
+            left=self.handle_constant(self.build(base.left)),
+            right=self.handle_constant(self.build(right)),
             operator=base.operator,
         )
 
@@ -1916,7 +1921,17 @@ class Factory:
         where_factory = Factory(
             grain=Grain(), environment=self.environment, local_concepts={}
         )
-
+        where_clause = (
+            where_factory.build(base.where_clause) if base.where_clause else None
+        )
+        # if the where clause derives new concepts
+        # we need to ensure these are accessible from the general factory
+        # post resolution
+        for bk, bv in where_factory.local_concepts.items():
+            # but do not override any local cahced grains
+            if bk in materialized:
+                continue
+            materialized[bk] = bv
         final: List[BuildConcept] = []
         for original in base.selection:
             new = original
@@ -1943,9 +1958,7 @@ class Factory:
                 factory.build(base.having_clause) if base.having_clause else None
             ),
             # this uses a different grain factory
-            where_clause=(
-                where_factory.build(base.where_clause) if base.where_clause else None
-            ),
+            where_clause=where_clause,
         )
 
     @build.register

trilogy/core/processing/discovery_node_factory.py

@@ -307,16 +307,17 @@ class RootNodeHandler:
     def _resolve_root_concepts(
         self, root_targets: List[BuildConcept]
     ) -> Optional[StrategyNode]:
-        synonym_node = self._try_synonym_resolution(root_targets)
-        if synonym_node:
-            logger.info(
-                f"{depth_to_prefix(self.ctx.depth)}{LOGGER_PREFIX} "
-                f"resolved root concepts through synonyms"
-            )
-            return synonym_node
         expanded_node = self._try_merge_expansion(root_targets)
         if expanded_node:
             return expanded_node
+        if self.ctx.accept_partial:
+            synonym_node = self._try_synonym_resolution(root_targets)
+            if synonym_node:
+                logger.info(
+                    f"{depth_to_prefix(self.ctx.depth)}{LOGGER_PREFIX} "
+                    f"resolved root concepts through synonyms"
+                )
+                return synonym_node
 
         return None
 
@@ -468,6 +469,7 @@ def generate_node(
         Derivation.GROUP_TO: lambda: _generate_group_to_node(context),
         Derivation.BASIC: lambda: _generate_basic_node(context),
         Derivation.ROOT: lambda: RootNodeHandler(context).generate(),
+        Derivation.CONSTANT: lambda: RootNodeHandler(context).generate(),
     }
 
     handler = derivation_handlers.get(concept.derivation)

trilogy/core/processing/node_generators/group_node.py

@@ -22,6 +22,24 @@ from trilogy.utility import unique
 LOGGER_PREFIX = "[GEN_GROUP_NODE]"
 
 
+def get_aggregate_grain(
+    concept: BuildConcept, environment: BuildEnvironment
+) -> BuildGrain:
+    parent_concepts: List[BuildConcept] = unique(
+        resolve_function_parent_concepts(concept, environment=environment), "address"
+    )
+    if (
+        concept.grain
+        and len(concept.grain.components) > 0
+        and not concept.grain.abstract
+    ):
+        grain_components = [environment.concepts[c] for c in concept.grain.components]
+        parent_concepts += grain_components
+        return BuildGrain.from_concepts(parent_concepts)
+    else:
+        return BuildGrain.from_concepts(parent_concepts)
+
+
 def gen_group_node(
     concept: BuildConcept,
     local_optional: List[BuildConcept],
@@ -51,7 +69,7 @@ def gen_group_node(
     ):
         grain_components = [environment.concepts[c] for c in concept.grain.components]
         parent_concepts += grain_components
-        build_grain_parents = BuildGrain.from_concepts(parent_concepts)
+        build_grain_parents = get_aggregate_grain(concept, environment)
         output_concepts += grain_components
         for possible_agg in local_optional:
 
@@ -70,6 +88,7 @@ def gen_group_node(
                     possible_agg,
                     environment=environment,
                 )
+                comp_grain = get_aggregate_grain(possible_agg, environment)
                 if set([x.address for x in agg_parents]).issubset(
                     set([x.address for x in parent_concepts])
                 ):
@@ -77,7 +96,7 @@ def gen_group_node(
                     logger.info(
                         f"{padding(depth)}{LOGGER_PREFIX} found equivalent group by optional concept {possible_agg.address} for {concept.address}"
                     )
-                elif BuildGrain.from_concepts(agg_parents) == build_grain_parents:
+                elif comp_grain == build_grain_parents:
                     extra = [x for x in agg_parents if x.address not in parent_concepts]
                     parent_concepts += extra
                     output_concepts.append(possible_agg)
@@ -86,7 +105,7 @@ def gen_group_node(
                     )
                 else:
                     logger.info(
-                        f"{padding(depth)}{LOGGER_PREFIX} cannot include optional agg {possible_agg.address}; mismatched parent grain {BuildGrain.from_concepts(agg_parents)} vs local parent {BuildGrain.from_concepts(parent_concepts)}"
+                        f"{padding(depth)}{LOGGER_PREFIX} cannot include optional agg {possible_agg.address}; it has mismatched parent grain {comp_grain } vs local parent {build_grain_parents}"
                     )
     if parent_concepts:
         logger.info(

trilogy/core/processing/node_generators/select_merge_node.py

@@ -231,7 +231,10 @@ def create_pruned_concept_graph(
 
 
 def resolve_subgraphs(
-    g: nx.DiGraph, relevant: list[BuildConcept], conditions: BuildWhereClause | None
+    g: nx.DiGraph,
+    relevant: list[BuildConcept],
+    conditions: BuildWhereClause | None,
+    depth: int = 0,
 ) -> dict[str, list[str]]:
     """When we have multiple distinct subgraphs within our matched
     nodes that can satisfy a query, resolve which one of those we should
@@ -241,7 +244,7 @@ def resolve_subgraphs(
     discarding duplicates.
     Duplicate subgraphs will be resolved based on which
     ones are most 'optimal' to use, a hueristic
-    that can evolve in the future but is currently based on
+    that can evolve in the future but is currently based on datasource
     cardinality."""
     datasources = [n for n in g.nodes if n.startswith("ds~")]
     subgraphs: dict[str, list[str]] = {
@@ -261,7 +264,7 @@ def resolve_subgraphs(
     pruned_subgraphs = {}
 
     def score_node(input: str):
-        logger.debug(f"scoring node {input}")
+        logger.debug(f"{padding(depth)}{LOGGER_PREFIX} scoring node {input}")
         grain = grain_length[input]
         # first - go for lowest grain
         # but if the object we want is in the grain, treat that as "free"
@@ -275,7 +278,7 @@ def resolve_subgraphs(
             len(subgraphs[input]),
             input,
         )
-        logger.debug(score)
+        logger.debug(f"{padding(depth)}{LOGGER_PREFIX} node {input} has score {score}")
         return score
 
     for key, nodes in subgraphs.items():
@@ -296,7 +299,7 @@ def resolve_subgraphs(
                 if len(value) < len(other_value):
                     is_subset = True
                     logger.debug(
-                        f"Dropping subgraph {key} with {value} as it is a subset of {other_key} with {other_value}"
+                        f"{padding(depth)}{LOGGER_PREFIX} Dropping subgraph {key} with {value} as it is a subset of {other_key} with {other_value}"
                     )
                 elif len(value) == len(other_value) and len(all_concepts) == len(
                     other_all_concepts
@@ -305,7 +308,9 @@ def resolve_subgraphs(
                     matches.add(key)
         if matches and not is_subset:
             min_node = min(matches, key=score_node)
-            logger.debug(f"minimum source score is {min_node}")
+            logger.debug(
+                f"{padding(depth)}{LOGGER_PREFIX} minimum source score is {min_node}"
+            )
             is_subset = key is not min(matches, key=score_node)
         if not is_subset:
             pruned_subgraphs[key] = nodes
@@ -330,7 +335,9 @@ def resolve_subgraphs(
                 > 1
             )
         if not keep:
-            logger.debug(f"Pruning node {node} as irrelevant after subgraph resolution")
+            logger.debug(
+                f"{padding(depth)}{LOGGER_PREFIX} Pruning node {node} as irrelevant after subgraph resolution"
+            )
             pruned_subgraphs = {
                 k: [n for n in v if n != node] for k, v in pruned_subgraphs.items()
             }
@@ -493,7 +500,7 @@ def create_select_node(
     else:
 
         candidate = bcandidate
-    assert candidate.resolve().output_concepts == all_concepts
+
     return candidate
 
 
@@ -536,7 +543,12 @@ def gen_select_merge_node(
             force_group=False,
             conditions=conditions.conditional if conditions else None,
         )
-    for attempt in [False, True]:
+    attempts = [
+        False,
+    ]
+    if accept_partial:
+        attempts.append(True)
+    for attempt in attempts:
         pruned_concept_graph = create_pruned_concept_graph(
             g,
             non_constant,
@@ -556,7 +568,7 @@ def gen_select_merge_node(
         return None
 
     sub_nodes = resolve_subgraphs(
-        pruned_concept_graph, relevant=non_constant, conditions=conditions
+        pruned_concept_graph, relevant=non_constant, conditions=conditions, depth=depth
     )
 
     logger.info(f"{padding(depth)}{LOGGER_PREFIX} fetching subgraphs {sub_nodes}")

trilogy/core/processing/node_generators/synonym_node.py

@@ -3,8 +3,8 @@ from collections import defaultdict
 from typing import List
 
 from trilogy.constants import logger
-from trilogy.core.enums import FunctionType
-from trilogy.core.models.build import BuildConcept, BuildFunction, BuildWhereClause
+from trilogy.core.enums import Derivation
+from trilogy.core.models.build import BuildConcept, BuildWhereClause
 from trilogy.core.models.build_environment import BuildEnvironment
 from trilogy.core.processing.nodes import History, StrategyNode
 from trilogy.core.processing.utility import padding
@@ -12,13 +12,6 @@ from trilogy.core.processing.utility import padding
 LOGGER_PREFIX = "[GEN_SYNONYM_NODE]"
 
 
-def is_union(c: BuildConcept):
-    return (
-        isinstance(c.lineage, BuildFunction)
-        and c.lineage.operator == FunctionType.UNION
-    )
-
-
 def gen_synonym_node(
     all_concepts: List[BuildConcept],
     environment: BuildEnvironment,
@@ -41,7 +34,6 @@ def gen_synonym_node(
                 synonyms[x.address].append(parent)
                 has_synonyms = True
         for y in x.pseudonyms:
-
             if y in environment.alias_origin_lookup:
                 synonyms[x.address].append(environment.alias_origin_lookup[y])
                 has_synonyms = True
@@ -59,11 +51,14 @@ def gen_synonym_node(
         itertools.product(*(synonyms[obj] for obj in sorted_keys))
     )
 
-    def similarity_sort_key(combo):
+    def similarity_sort_key(combo: tuple[BuildConcept, ...]):
         addresses = [x.address for x in combo]
 
         # Calculate similarity score - count how many pairs share prefixes
         similarity_score = 0
+        roots = sum(
+            [1 for x in combo if x.derivation in (Derivation.ROOT, Derivation.CONSTANT)]
+        )
         for i in range(len(addresses)):
             for j in range(i + 1, len(addresses)):
                 # Find common prefix length
@@ -77,8 +72,8 @@ def gen_synonym_node(
                         break
                 similarity_score += common_prefix_len
 
-        # Sort by similarity (descending), then by addresses (ascending) for ties
-        return (-similarity_score, addresses)
+        # Sort by roots, similarity (descending), then by addresses (ascending) for ties
+        return (-roots, -similarity_score, addresses)
 
     combinations_list.sort(key=similarity_sort_key)
     for combo in combinations_list:

trilogy/core/processing/utility.py

@@ -322,8 +322,10 @@ def resolve_instantiated_concept(
     for k in concept.pseudonyms:
         if k in datasource.output_concepts:
             return [x for x in datasource.output_concepts if x.address == k].pop()
+        if any(k in x.pseudonyms for x in datasource.output_concepts):
+            return [x for x in datasource.output_concepts if k in x.pseudonyms].pop()
     raise SyntaxError(
-        f"Could not find {concept.address} in {datasource.identifier} output {[c.address for c in datasource.output_concepts]}"
+        f"Could not find {concept.address} in {datasource.identifier} output {[c.address for c in datasource.output_concepts]}, acceptable synonyms {concept.pseudonyms}"
     )
 
 
@@ -601,19 +603,28 @@ def find_nullable_concepts(
 
 
 def sort_select_output_processed(
-    cte: CTE | UnionCTE, query: ProcessedQuery
+    cte: CTE | UnionCTE, query: SelectStatement | MultiSelectStatement | ProcessedQuery
 ) -> CTE | UnionCTE:
-    output_addresses = [c.address for c in query.output_columns]
+    if isinstance(query, ProcessedQuery):
+        targets = query.output_columns
+        hidden = query.hidden_columns
+    else:
+        targets = query.output_components
+        hidden = query.hidden_components
+
+    output_addresses = [c.address for c in targets]
 
     mapping = {x.address: x for x in cte.output_columns}
 
     new_output: list[BuildConcept] = []
-    for x in query.output_columns:
+    for x in targets:
         if x.address in mapping:
             new_output.append(mapping[x.address])
         for oc in cte.output_columns:
             if x.address in oc.pseudonyms:
                 # create a wrapper BuildConcept to render the pseudonym under the original name
+                if any(x.address == y for y in mapping.keys()):
+                    continue
                 new_output.append(
                     BuildConcept(
                         name=x.name,
@@ -636,10 +647,7 @@ def sort_select_output_processed(
         [
             c.address
             for c in cte.output_columns
-            if (
-                c.address not in query.output_columns
-                or c.address in query.hidden_columns
-            )
+            if (c.address not in targets or c.address in hidden)
         ]
     )
     cte.output_columns = new_output
@@ -650,38 +658,4 @@ def sort_select_output(
     cte: CTE | UnionCTE, query: SelectStatement | MultiSelectStatement | ProcessedQuery
 ) -> CTE | UnionCTE:
 
-    if isinstance(query, ProcessedQuery):
-        return sort_select_output_processed(cte, query)
-
-    mapping = {x.address: x for x in cte.output_columns}
-
-    new_output: list[BuildConcept] = []
-    for x in query.output_components:
-        if x.address in mapping:
-            new_output.append(mapping[x.address])
-        else:
-            for oc in cte.output_columns:
-                if x.address in oc.pseudonyms:
-                    # create a wrapper BuildConcept to render the pseudonym under the original name
-                    new_output.append(
-                        BuildConcept(
-                            name=x.name,
-                            namespace=x.namespace,
-                            pseudonyms={oc.address},
-                            datatype=oc.datatype,
-                            purpose=oc.purpose,
-                            grain=oc.grain,
-                            build_is_aggregate=oc.build_is_aggregate,
-                        )
-                    )
-                    break
-    cte.output_columns = new_output
-    cte.hidden_concepts = set(
-        [
-            c.address
-            for c in query.output_components
-            if c.address in query.hidden_components
-        ]
-    )
-
-    return cte
+    return sort_select_output_processed(cte, query)

trilogy/core/query_processor.py

@@ -395,14 +395,13 @@ def get_query_node(
     if not statement.output_components:
         raise ValueError(f"Statement has no output components {statement}")
     history = history or History(base_environment=environment)
-    print(
+    logger.info(
         f"{LOGGER_PREFIX} building query node for {statement.output_components} grain {statement.grain}"
     )
     build_statement: BuildSelectLineage | BuildMultiSelectLineage = Factory(
         environment=environment,
     ).build(statement)
 
-    # build_statement = statement
     build_environment = environment.materialize_for_select(
         build_statement.local_concepts
     )

trilogy/dialect/base.py

@@ -12,6 +12,7 @@ from trilogy.constants import (
 )
 from trilogy.core.constants import UNNEST_NAME
 from trilogy.core.enums import (
+    ComparisonOperator,
     DatePart,
     FunctionType,
     UnnestMode,
@@ -276,7 +277,13 @@ ORDER BY{% for order in order_by %}
 )
 
 
-def safe_get_cte_value(coalesce, cte: CTE | UnionCTE, c: BuildConcept, quote_char: str):
+def safe_get_cte_value(
+    coalesce,
+    cte: CTE | UnionCTE,
+    c: BuildConcept,
+    quote_char: str,
+    render_expr: Callable,
+) -> Optional[str]:
     address = c.address
     raw = cte.source_map.get(address, None)
 
@@ -287,6 +294,9 @@ def safe_get_cte_value(coalesce, cte: CTE | UnionCTE, c: BuildConcept, quote_cha
         return f"{quote_char}{raw}{quote_char}.{safe_quote(rendered, quote_char)}"
     if isinstance(raw, list) and len(raw) == 1:
         rendered = cte.get_alias(c, raw[0])
+        if isinstance(rendered, FUNCTION_ITEMS):
+            # if it's a function, we need to render it as a function
+            return f"{render_expr(rendered, cte=cte, raise_invalid=True)}"
         return f"{quote_char}{raw[0]}{quote_char}.{safe_quote(rendered, quote_char)}"
     return coalesce(
         sorted(
@@ -499,6 +509,7 @@ class BaseDialect:
                     cte,
                     c,
                     self.QUOTE_CHARACTER,
+                    self.render_expr,
                 )
                 if not rval:
                     # unions won't have a specific source mapped; just use a generic column reference
@@ -515,6 +526,17 @@ class BaseDialect:
                         )
         return rval
 
+    def render_array_unnest(
+        self,
+        left,
+        right,
+        operator: ComparisonOperator,
+        cte: CTE | UnionCTE | None = None,
+        cte_map: Optional[Dict[str, CTE | UnionCTE]] = None,
+        raise_invalid: bool = False,
+    ):
+        return f"{self.render_expr(left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {operator.value} {self.render_expr(right, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}"
+
     def render_expr(
         self,
         e: Union[
@@ -556,6 +578,7 @@ class BaseDialect:
         raise_invalid: bool = False,
     ) -> str:
         if isinstance(e, SUBSELECT_COMPARISON_ITEMS):
+
             if isinstance(e.right, BuildConcept):
                 # we won't always have an existnce map
                 # so fall back to the normal map
@@ -585,10 +608,22 @@ class BaseDialect:
                     info = cte.inlined_ctes[target]
                     return f"{self.render_expr(e.left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {e.operator.value} (select {target}.{self.QUOTE_CHARACTER}{e.right.safe_address}{self.QUOTE_CHARACTER} from {info.new_base} as {target} where {target}.{self.QUOTE_CHARACTER}{e.right.safe_address}{self.QUOTE_CHARACTER} is not null)"
                 return f"{self.render_expr(e.left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {e.operator.value} (select {target}.{self.QUOTE_CHARACTER}{e.right.safe_address}{self.QUOTE_CHARACTER} from {target} where {target}.{self.QUOTE_CHARACTER}{e.right.safe_address}{self.QUOTE_CHARACTER} is not null)"
-
+            elif isinstance(e.right, BuildParamaterizedConceptReference):
+                if isinstance(e.right.concept.lineage, BuildFunction) and isinstance(
+                    e.right.concept.lineage.arguments[0], ListWrapper
+                ):
+                    return self.render_array_unnest(
+                        e.left,
+                        e.right,
+                        e.operator,
+                        cte=cte,
+                        cte_map=cte_map,
+                        raise_invalid=raise_invalid,
+                    )
+                return f"{self.render_expr(e.left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {e.operator.value} {self.render_expr(e.right, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}"
             elif isinstance(
                 e.right,
-                (ListWrapper, TupleWrapper, BuildParenthetical, list),
+                (ListWrapper, TupleWrapper, BuildParenthetical),
             ):
                 return f"{self.render_expr(e.left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {e.operator.value} {self.render_expr(e.right, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}"
 
@@ -692,13 +727,11 @@ class BaseDialect:
             return f"'{e}'"
         elif isinstance(e, (int, float)):
             return str(e)
-        elif isinstance(e, ListWrapper):
-            return f"[{','.join([self.render_expr(x, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid) for x in e])}]"
         elif isinstance(e, TupleWrapper):
             return f"({','.join([self.render_expr(x, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid) for x in e])})"
         elif isinstance(e, MapWrapper):
             return f"MAP {{{','.join([f'{self.render_expr(k, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}:{self.render_expr(v, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}' for k, v in e.items()])}}}"
-        elif isinstance(e, list):
+        elif isinstance(e, ListWrapper):
             return f"{self.FUNCTION_MAP[FunctionType.ARRAY]([self.render_expr(x, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid) for x in e])}"
         elif isinstance(e, DataType):
             return self.DATATYPE_MAP.get(e, e.value)
@@ -762,6 +795,7 @@ class BaseDialect:
             ] + [
                 f"{self.QUOTE_CHARACTER}{c.safe_address}{self.QUOTE_CHARACTER}"
                 for c in cte.join_derived_concepts
+                if c.address not in cte.hidden_concepts
             ]
         elif self.UNNEST_MODE in (UnnestMode.CROSS_JOIN_UNNEST, UnnestMode.PRESTO):
             select_columns = [
@@ -772,6 +806,7 @@ class BaseDialect:
             ] + [
                 f"{UNNEST_NAME} as {self.QUOTE_CHARACTER}{c.safe_address}{self.QUOTE_CHARACTER}"
                 for c in cte.join_derived_concepts
+                if c.address not in cte.hidden_concepts
             ]
         else:
             # otherwse, assume we are unnesting directly in the select

trilogy/dialect/bigquery.py

@@ -1,11 +1,17 @@
-from typing import Any, Callable, Mapping
+from typing import Any, Callable, Dict, Mapping, Optional
 
 from jinja2 import Template
 
-from trilogy.core.enums import FunctionType, UnnestMode, WindowType
+from trilogy.core.enums import (
+    ComparisonOperator,
+    FunctionType,
+    UnnestMode,
+    WindowType,
+)
 from trilogy.core.models.core import (
     DataType,
 )
+from trilogy.core.models.execute import CTE, UnionCTE
 from trilogy.dialect.base import BaseDialect
 
 WINDOW_FUNCTION_MAP: Mapping[WindowType, Callable[[Any, Any, Any], str]] = {}
@@ -99,3 +105,14 @@ class BigqueryDialect(BaseDialect):
     SQL_TEMPLATE = BQ_SQL_TEMPLATE
     UNNEST_MODE = UnnestMode.CROSS_JOIN_UNNEST
     DATATYPE_MAP = DATATYPE_MAP
+
+    def render_array_unnest(
+        self,
+        left,
+        right,
+        operator: ComparisonOperator,
+        cte: CTE | UnionCTE | None = None,
+        cte_map: Optional[Dict[str, CTE | UnionCTE]] = None,
+        raise_invalid: bool = False,
+    ):
+        return f"{self.render_expr(left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {operator.value} unnest({self.render_expr(right, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)})"

trilogy/parsing/common.py

@@ -44,6 +44,7 @@ from trilogy.core.models.author import (
     RowsetLineage,
     SubselectComparison,
     TraitDataType,
+    TupleWrapper,
     UndefinedConcept,
     WhereClause,
     WindowItem,
@@ -172,7 +173,7 @@ def concept_list_to_keys(
 
 
 def constant_to_concept(
-    parent: ListWrapper | MapWrapper | list | int | float | str,
+    parent: ListWrapper | TupleWrapper | MapWrapper | int | float | str,
     name: str,
     namespace: str,
     metadata: Metadata | None = None,
@@ -183,7 +184,7 @@ def constant_to_concept(
         output_purpose=Purpose.CONSTANT,
         arguments=[parent],
     )
-    assert const_function.arguments[0] == parent, const_function.arguments[0]
+    # assert const_function.arguments[0] == parent, f'{const_function.arguments[0]} != {parent}, {type(const_function.arguments[0])} != {type(parent)}'
     fmetadata = metadata or Metadata()
     return Concept(
         name=name,

trilogy/parsing/parse_engine.py

@@ -146,7 +146,7 @@ class ParsePass(Enum):
     VALIDATION = 2
 
 
-CONSTANT_TYPES = (int, float, str, bool, list, ListWrapper, MapWrapper)
+CONSTANT_TYPES = (int, float, str, bool, ListWrapper, TupleWrapper, MapWrapper)
 
 SELF_LABEL = "root"
 
@@ -742,6 +742,7 @@ class ParseToObjects(Transformer):
         lookup, namespace, name, parent = parse_concept_reference(
             name, self.environment
         )
+
         concept = Concept(
             name=name,
             datatype=arg_to_datatype(constant),