pytrilogy 0.0.3.68__py3-none-any.whl → 0.0.3.70__py3-none-any.whl

{pytrilogy-0.0.3.68.dist-info → pytrilogy-0.0.3.70.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pytrilogy
-Version: 0.0.3.68
+Version: 0.0.3.70
 Summary: Declarative, typed query language that compiles to SQL.
 Home-page: 
 Author: 
@@ -40,46 +40,61 @@ Dynamic: summary
 [![Website](https://img.shields.io/badge/INTRO-WEB-orange?)](https://trilogydata.dev/)
 [![Discord](https://img.shields.io/badge/DISCORD-CHAT-red?logo=discord)](https://discord.gg/Z4QSSuqGEd)
 
-pytrilogy is an experimental implementation of the Trilogy language, a higher-level SQL that replaces tables/joins with a lightweight semantic binding layer.
+The Trilogy language is an experiment in better SQL for analytics - a streamlined SQL that replaces tables/joins with a lightweight semantic binding layer and provides easy reuse and composability. It compiles to SQL - making it easy to debug or integrate into existing workflows - and can be run against any supported SQL backend.  
 
-Trilogy looks like SQL, but simpler. It's a modern SQL refresh targeted at SQL lovers who want more reusability and composability without losing the expressiveness and iterative value of SQL. It compiles to SQL - making it easy to debug or integrate into existing workflows - and can be run against any supported SQL backend.  
+[pytrilogy](https://github.com/trilogy-data/pytrilogy) is the reference implementation, written in Python.
+
+Trilogy concretely solves these common problems in karge, SQL based analytics teams:
+- decoupling consumption code from specific physical assets
+- better testability and change management
+- reduced boilerplate and opportunity for OLAP style optimization at scale
+
+Trilogy can be especially powerful as a frontend consumption language, since the decoupling from the physical layout makes dynamic and interactive dashboards backed by SQL tables much easier to create.
 
 > [!TIP]
-> Try it online in a hosted [open-source studio](https://trilogydata.dev/trilogy-studio-core/). To get an overview of the language and run interactive examples, head to the [documentation](https://trilogydata.dev/).
+> You can try Trilogy in a [open-source studio](https://trilogydata.dev/trilogy-studio-core/). More details on the language can be found on the [documentation](https://trilogydata.dev/).
 
-Installation: `pip install pytrilogy`
+We recommend the studio as the fastest way to explore Trilogy. For deeper work and integration, `pytrilogy` can be run locally to parse and execute trilogy model [.preql] files using the `trilogy` CLI tool, or can be run in python by importing the `trilogy` package.
 
-`pytrilogy` can be run locally to parse and execute trilogy model [.preql] files using the `trilogy` CLI tool, or can be run in python by importing the `trilogy` package.
+Installation: `pip install pytrilogy`
 
-You can read more about the project [here](https://trilogydata.dev/) and try out an interactive demo [here](https://trilogydata.dev/demo/). 
+### Trilogy Looks Like SQL
 
-Trilogy looks like SQL:
 ```sql
+import names;
+
+const top_names <- ['Elvis', 'Elvira', 'Elrond', 'Sam'];
+
+def initcap(word) -> upper(substring(word, 1, 1)) || substring(word, 2, len(word));
+
 WHERE 
-    name like '%lvis%'
+    @initcap(name) in top_names
 SELECT
     name,
-    count(name) as name_count
+    sum(births) as name_count
 ORDER BY
     name_count desc
 LIMIT 10;
 ```
 ## Goals
-And aims to:
+Versus SQL, Trilogy aims to: 
 
-Preserve:
+Keep:
 - Correctness
 - Accessibility
 
-Enhance:
+Improve:
 - Simplicity
-- Understandability
 - Refactoring/mantainability
 - Reusability
 
 Maintain:
 - Acceptable performance
 
+Remove:
+- Lower-level procedural features
+- Transactional optimizations/non-analytics features
+
 ## Hello World
 
 Save the following code in a file named `hello.preql`
@@ -141,10 +156,7 @@ SELECT
     sentences.text
 ;
 
-
-
 ```
-
 Run the following from the directory the file is in.
 
 ```bash
@@ -157,11 +169,15 @@ trilogy run hello.trilogy duckdb
 
 The current Trilogy implementation supports these backends:
 
+### Core
 - Bigquery
-- SQL Server
 - DuckDB
 - Snowflake
 
+### Experimental
+- SQL Server
+- Presto
+
 ## Basic Example - Python
 
 Trilogy can be run directly in python through the core SDK. Trilogy code can be defined and parsed inline or parsed out of files.
@@ -170,7 +186,6 @@ A bigquery example, similar to bigquery [the quickstart](https://cloud.google.co
 
 ```python
 
-
 from trilogy import Dialects, Environment
 
 environment = Environment()

{pytrilogy-0.0.3.68.dist-info → pytrilogy-0.0.3.70.dist-info}/RECORD

@@ -1,5 +1,5 @@
-pytrilogy-0.0.3.68.dist-info/licenses/LICENSE.md,sha256=5ZRvtTyCCFwz1THxDTjAu3Lidds9WjPvvzgVwPSYNDo,1042
-trilogy/__init__.py,sha256=c1CqFUPhjgRs6876P3SAh2D8kHJp6_QydaF_JJeDlJc,303
+pytrilogy-0.0.3.70.dist-info/licenses/LICENSE.md,sha256=5ZRvtTyCCFwz1THxDTjAu3Lidds9WjPvvzgVwPSYNDo,1042
+trilogy/__init__.py,sha256=jOvyFSWu8pnqklbm3VK_FmD5VSPj1cqSUoyMFyDKVC8,303
 trilogy/compiler.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 trilogy/constants.py,sha256=lv_aJWP6dn6e2aF4BAE72jbnNtceFddfqtiDSsvzno0,1692
 trilogy/engine.py,sha256=OK2RuqCIUId6yZ5hfF8J1nxGP0AJqHRZiafcowmW0xc,1728
@@ -16,15 +16,15 @@ trilogy/core/env_processor.py,sha256=pFsxnluKIusGKx1z7tTnfsd_xZcPy9pZDungkjkyvI0
 trilogy/core/environment_helpers.py,sha256=VvPIiFemqaLLpIpLIqprfu63K7muZ1YzNg7UZIUph8w,8267
 trilogy/core/ergonomics.py,sha256=e-7gE29vPLFdg0_A1smQ7eOrUwKl5VYdxRSTddHweRA,1631
 trilogy/core/exceptions.py,sha256=jYEduuMehcMkmCpf-OC_taELPZm7qNfeSNzIWkDYScs,707
-trilogy/core/functions.py,sha256=poVfAwet1xdxTkC7WL38UmGRDpUVO9iSMNWSagl9_r4,29302
+trilogy/core/functions.py,sha256=R8_aOe2mNRgOLmsnI9pG_GOU3I7kFPTnXQzplN2d7Dw,29343
 trilogy/core/graph_models.py,sha256=wIT-oBchHWE46GLDkgN5K7EzhOBEo8LfaeWV5G5cYcE,3302
 trilogy/core/internal.py,sha256=iicDBlC6nM8d7e7jqzf_ZOmpUsW8yrr2AA8AqEiLx-s,1577
 trilogy/core/optimization.py,sha256=ojpn-p79lr03SSVQbbw74iPCyoYpDYBmj1dbZ3oXCjI,8860
-trilogy/core/query_processor.py,sha256=jA1lek3Kbpi_-NamDPjJuJanUdY7KM3ODB7tS_qqxH4,20311
+trilogy/core/query_processor.py,sha256=t91tYQS0o89r-acdxo8uDUpB1elNcqAhLWkbM8BFjho,20283
 trilogy/core/utility.py,sha256=3VC13uSQWcZNghgt7Ot0ZTeEmNqs__cx122abVq9qhM,410
 trilogy/core/models/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-trilogy/core/models/author.py,sha256=44VyEInWied287lJp99T14Ahq8RSQK4q3Y-AzAByMTY,77818
-trilogy/core/models/build.py,sha256=eNjbFwg_mEcwPeagQpkIybIluIrRCKJLAHdpZjm01e8,64857
+trilogy/core/models/author.py,sha256=Wz_6yEZS5EyXJ5wHHl-C44ikFka5XftBYOjNRK98Hfw,77790
+trilogy/core/models/build.py,sha256=BNHzfqD5NWCKHntvnswvDrtCD1JFgvPedx23cPq942U,65405
 trilogy/core/models/build_environment.py,sha256=s_C9xAHuD3yZ26T15pWVBvoqvlp2LdZ8yjsv2_HdXLk,5363
 trilogy/core/models/core.py,sha256=EMAuWTngoNVGCdfNrAY7_k6g528iodNQLwPRVip-8DA,10980
 trilogy/core/models/datasource.py,sha256=6RjJUd2u4nYmEwFBpJlM9LbHVYDv8iHJxqiBMZqUrwI,9422
@@ -37,7 +37,7 @@ trilogy/core/optimizations/predicate_pushdown.py,sha256=g4AYE8Aw_iMlAh68TjNXGP75
 trilogy/core/processing/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 trilogy/core/processing/concept_strategies_v3.py,sha256=zy5VZa9LITOws6aIILfv_bSR2-jR1Ndldy-nmwMyQ5w,23144
 trilogy/core/processing/discovery_loop.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-trilogy/core/processing/discovery_node_factory.py,sha256=Uza_wrB6K965rXbwCuWvBtsDMftq2m0hAav9B5bxQ8k,15049
+trilogy/core/processing/discovery_node_factory.py,sha256=I3JJxoF-u8OVvqXXAOhvMg2h-KdpHQwg6EpCeQtxGCI,15123
 trilogy/core/processing/discovery_utility.py,sha256=3xdd1ypKappSDm0SJs7WtW5YegL80SlYhDQlkNePp4E,4549
 trilogy/core/processing/discovery_validation.py,sha256=fGWJmKpgEd1f4RkK-fYOBUT1cwsJnahwXFAdRlou7MI,5365
 trilogy/core/processing/graph_utils.py,sha256=8QUVrkE9j-9C1AyrCb1nQEh8daCe0u1HuXl-Te85lag,1205
@@ -46,13 +46,13 @@ trilogy/core/processing/node_generators/__init__.py,sha256=w8TQQgNhyAra6JQHdg1_A
 trilogy/core/processing/node_generators/basic_node.py,sha256=luN8LftafZepoFgDRv4gmvEGFlOI2j0icJ5fz4UT7uo,5165
 trilogy/core/processing/node_generators/common.py,sha256=PdysdroW9DUADP7f5Wv_GKPUyCTROZV1g3L45fawxi8,9443
 trilogy/core/processing/node_generators/filter_node.py,sha256=oRRq2-T3ufgn4D23uQsc58f20eFk-djs4QI3WKA75K8,10908
-trilogy/core/processing/node_generators/group_node.py,sha256=S7mzEAVmjc7rnHrJ9PAp4Ld8njcBSrvAu3K1cxsLQdY,6140
+trilogy/core/processing/node_generators/group_node.py,sha256=1QJhRxsTklJ5xq8wHlAURZaN9gL9FPpeCa1OJ7IwXnY,6769
 trilogy/core/processing/node_generators/group_to_node.py,sha256=jKcNCDOY6fNblrdZwaRU0sbUSr9H0moQbAxrGgX6iGA,3832
 trilogy/core/processing/node_generators/multiselect_node.py,sha256=GWV5yLmKTe1yyPhN60RG1Rnrn4ktfn9lYYXi_FVU4UI,7061
 trilogy/core/processing/node_generators/node_merge_node.py,sha256=dSqfqWp2SolhDB16nkPaaTXgNQo4QquEufPdf7q0Tb4,17398
 trilogy/core/processing/node_generators/recursive_node.py,sha256=l5zdh0dURKwmAy8kK4OpMtZfyUEQRk6N-PwSWIyBpSM,2468
 trilogy/core/processing/node_generators/rowset_node.py,sha256=2BiSsegbRF9csJ_Xl8P_CxIm4dAAb7dF29u6v_Odr-A,6709
-trilogy/core/processing/node_generators/select_merge_node.py,sha256=EmU6QUOEKOOKVJreHxDJZakd2FpD1V1W8ZGOOLLaWdo,21462
+trilogy/core/processing/node_generators/select_merge_node.py,sha256=2KIljy2TFLT9kjKUfhkFwSDkcu4hv4yRVTKOGzEe-NM,21763
 trilogy/core/processing/node_generators/select_node.py,sha256=Ta1G39V94gjX_AgyZDz9OqnwLz4BjY3D6Drx9YpziMQ,3555
 trilogy/core/processing/node_generators/synonym_node.py,sha256=AnAsa_Wj50NJ_IK0HSgab_7klYmKVrv0WI1uUe-GvEY,3766
 trilogy/core/processing/node_generators/union_node.py,sha256=VNo6Oey4p8etU9xrOh2oTT2lIOTvY6PULUPRvVa2uxU,2877
@@ -76,12 +76,12 @@ trilogy/core/statements/build.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hS
 trilogy/core/statements/common.py,sha256=KxEmz2ySySyZ6CTPzn0fJl5NX2KOk1RPyuUSwWhnK1g,759
 trilogy/core/statements/execute.py,sha256=rqfuoMuXPcH7L7TmE1dSiZ_K_A1ohB8whVMfGimZBOk,1294
 trilogy/dialect/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-trilogy/dialect/base.py,sha256=CMrYlE-oQzdmefmNPBRzT7OA842UV-P1sCqa0JAcP4Q,43741
-trilogy/dialect/bigquery.py,sha256=6ghCqy-k7UioIJc1EEQ7gRo_PHaO8Vm7yYbiQ-kgpzs,3629
+trilogy/dialect/base.py,sha256=SpNv1D9eig-XDOui2xoeMvdTfRpejAfJ_HaAMkuPW9w,45112
+trilogy/dialect/bigquery.py,sha256=e19dGcarapgA0x5_Xmq2StyHzuDWPOOPaR4elkWXwug,4203
 trilogy/dialect/common.py,sha256=hhzuMTFW9QQIP7TKLT9BlJy6lw2R03a68jKQ-7t4-2c,6070
 trilogy/dialect/config.py,sha256=olnyeVU5W5T6b9-dMeNAnvxuPlyc2uefb7FRME094Ec,3834
 trilogy/dialect/dataframe.py,sha256=RUbNgReEa9g3pL6H7fP9lPTrAij5pkqedpZ99D8_5AE,1522
-trilogy/dialect/duckdb.py,sha256=C5TovwacDXo9YDpMTpPxkH7D0AxQERa7JL1RUkDGVng,3898
+trilogy/dialect/duckdb.py,sha256=gsXhPKX0D7ykJ9RFK9qx8uBTjLgtHu6PYv6GlBFtnJE,4448
 trilogy/dialect/enums.py,sha256=FRNYQ5-w-B6-X0yXKNU5g9GowsMlERFogTC5u2nxL_s,4740
 trilogy/dialect/postgres.py,sha256=VH4EB4myjIeZTHeFU6vK00GxY9c53rCBjg2mLbdaCEE,3254
 trilogy/dialect/presto.py,sha256=Wd0yHq3EOSfCOy7lWPfCr13JHO3olsm8qUXgml-oTm0,3529
@@ -93,13 +93,13 @@ trilogy/hooks/graph_hook.py,sha256=5BfR7Dt0bgEsCLgwjowgCsVkboGYfVJGOz8g9mqpnos,4
 trilogy/hooks/query_debugger.py,sha256=1npRjww94sPV5RRBBlLqMJRaFkH9vhEY6o828MeoEcw,5583
 trilogy/metadata/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 trilogy/parsing/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-trilogy/parsing/common.py,sha256=_-KWPP3NvNAHiB4B6jyGSnYKqlLs2XqXeTZiX4MCtWU,30861
+trilogy/parsing/common.py,sha256=_5UEnLtu40VQ8gb6wg3GtSrxf6IONhEOntmdsm0X4lU,30961
 trilogy/parsing/config.py,sha256=Z-DaefdKhPDmSXLgg5V4pebhSB0h590vI0_VtHnlukI,111
 trilogy/parsing/exceptions.py,sha256=Xwwsv2C9kSNv2q-HrrKC1f60JNHShXcCMzstTSEbiCw,154
 trilogy/parsing/helpers.py,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
-trilogy/parsing/parse_engine.py,sha256=O7aM5nZ4SjKlqO2x8XWefI1BMCW06jYYLhABU4k1HCI,72430
+trilogy/parsing/parse_engine.py,sha256=vYhGmSJXi5TcLvt2mujISucQc35j4kHa78hj5ip9gGQ,72564
 trilogy/parsing/render.py,sha256=gGCFj2ue0UoaU2MR6qHGMAHXkYRMkTmHjnBowdcgFMY,19603
-trilogy/parsing/trilogy.lark,sha256=x9D1BXtE1E9Kxatx5Kt7xCaid8zgedabwca_B7j7L7o,14331
+trilogy/parsing/trilogy.lark,sha256=1RIqA7zrGuqDJYSv9yHGSw0vdIfGOLPOnc4hSBRSTVU,14346
 trilogy/scripts/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
 trilogy/scripts/trilogy.py,sha256=1L0XrH4mVHRt1C9T1HnaDv2_kYEfbWTb5_-cBBke79w,3774
 trilogy/std/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
@@ -110,8 +110,8 @@ trilogy/std/money.preql,sha256=XWwvAV3WxBsHX9zfptoYRnBigcfYwrYtBHXTME0xJuQ,2082
 trilogy/std/net.preql,sha256=-bMV6dyofskl4Kvows-iQ4JCxjVUwsZOeWCy8JO5Ftw,135
 trilogy/std/ranking.preql,sha256=LDoZrYyz4g3xsII9XwXfmstZD-_92i1Eox1UqkBIfi8,83
 trilogy/std/report.preql,sha256=LbV-XlHdfw0jgnQ8pV7acG95xrd1-p65fVpiIc-S7W4,202
-pytrilogy-0.0.3.68.dist-info/METADATA,sha256=XA1FhFxajFv-4bT9JSbdv5gH5LvikdjwiJ33kAUdfQI,9095
-pytrilogy-0.0.3.68.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-pytrilogy-0.0.3.68.dist-info/entry_points.txt,sha256=ewBPU2vLnVexZVnB-NrVj-p3E-4vukg83Zk8A55Wp2w,56
-pytrilogy-0.0.3.68.dist-info/top_level.txt,sha256=cAy__NW_eMAa_yT9UnUNlZLFfxcg6eimUAZ184cdNiE,8
-pytrilogy-0.0.3.68.dist-info/RECORD,,
+pytrilogy-0.0.3.70.dist-info/METADATA,sha256=dSEU2zgUrBXl88Yb-KV8y_hta24BeHt5VaPM19-YMFU,9734
+pytrilogy-0.0.3.70.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+pytrilogy-0.0.3.70.dist-info/entry_points.txt,sha256=ewBPU2vLnVexZVnB-NrVj-p3E-4vukg83Zk8A55Wp2w,56
+pytrilogy-0.0.3.70.dist-info/top_level.txt,sha256=cAy__NW_eMAa_yT9UnUNlZLFfxcg6eimUAZ184cdNiE,8
+pytrilogy-0.0.3.70.dist-info/RECORD,,

trilogy/__init__.py

@@ -4,6 +4,6 @@ from trilogy.dialect.enums import Dialects
 from trilogy.executor import Executor
 from trilogy.parser import parse
 
-__version__ = "0.0.3.68"
+__version__ = "0.0.3.70"
 
 __all__ = ["parse", "Executor", "Dialects", "Environment", "CONFIG"]

trilogy/core/functions.py

@@ -370,10 +370,10 @@ FUNCTION_REGISTRY: dict[FunctionType, FunctionConfig] = {
         arg_count=2,
     ),
     FunctionType.REGEXP_EXTRACT: FunctionConfig(
-        valid_inputs={DataType.STRING},
+        valid_inputs=[{DataType.STRING}, {DataType.STRING}, {DataType.INTEGER}],
         output_purpose=Purpose.PROPERTY,
         output_type=DataType.STRING,
-        arg_count=2,
+        arg_count=3,
     ),
     FunctionType.REGEXP_REPLACE: FunctionConfig(
         valid_inputs={DataType.STRING},

trilogy/core/models/author.py

@@ -2303,7 +2303,7 @@ class CustomFunctionFactory:
         ]
         return self
 
-    def __call__(self, *creation_args: list[Expr]):
+    def __call__(self, *creation_args: Expr):
         nout = (
             self.function.model_copy(deep=True)
             if isinstance(self.function, BaseModel)
@@ -2389,7 +2389,6 @@ Expr = (
     | int
     | str
     | float
-    | list
     | date
     | datetime
     | TupleWrapper
@@ -2430,7 +2429,6 @@ FuncArgs = (
     | ListType
     | MapType
     | NumericType
-    | list
     | ListWrapper[Any]
     | TupleWrapper[Any]
     | Comparison

trilogy/core/models/build.py

@@ -155,6 +155,7 @@ def concepts_to_build_grain_concepts(
             pconcepts.append(c)
         elif environment:
             pconcepts.append(environment.concepts[c])
+
         else:
             raise ValueError(
                 f"Unable to resolve input {c} without environment provided to concepts_to_grain call"
@@ -248,7 +249,7 @@ def get_concept_arguments(expr) -> List["BuildConcept"]:
     return output
 
 
-class BuildParamaterizedConceptReference(BaseModel):
+class BuildParamaterizedConceptReference(DataTyped, BaseModel):
     concept: BuildConcept
 
     def __str__(self):
@@ -258,6 +259,10 @@ class BuildParamaterizedConceptReference(BaseModel):
     def safe_address(self) -> str:
         return self.concept.safe_address
 
+    @property
+    def output_datatype(self) -> DataType:
+        return self.concept.output_datatype
+
 
 class BuildGrain(BaseModel):
     components: set[str] = Field(default_factory=set)
@@ -1810,8 +1815,8 @@ class Factory:
             right_c, _ = self.instantiate_concept(base.right)
             right = right_c
         return BuildSubselectComparison.model_construct(
-            left=self.build(base.left),
-            right=self.build(right),
+            left=self.handle_constant(self.build(base.left)),
+            right=self.handle_constant(self.build(right)),
             operator=base.operator,
         )
 
@@ -1916,7 +1921,17 @@ class Factory:
         where_factory = Factory(
             grain=Grain(), environment=self.environment, local_concepts={}
         )
-
+        where_clause = (
+            where_factory.build(base.where_clause) if base.where_clause else None
+        )
+        # if the where clause derives new concepts
+        # we need to ensure these are accessible from the general factory
+        # post resolution
+        for bk, bv in where_factory.local_concepts.items():
+            # but do not override any local cahced grains
+            if bk in materialized:
+                continue
+            materialized[bk] = bv
         final: List[BuildConcept] = []
         for original in base.selection:
             new = original
@@ -1943,9 +1958,7 @@ class Factory:
                 factory.build(base.having_clause) if base.having_clause else None
             ),
             # this uses a different grain factory
-            where_clause=(
-                where_factory.build(base.where_clause) if base.where_clause else None
-            ),
+            where_clause=where_clause,
         )
 
     @build.register

trilogy/core/processing/discovery_node_factory.py

@@ -469,6 +469,7 @@ def generate_node(
         Derivation.GROUP_TO: lambda: _generate_group_to_node(context),
         Derivation.BASIC: lambda: _generate_basic_node(context),
         Derivation.ROOT: lambda: RootNodeHandler(context).generate(),
+        Derivation.CONSTANT: lambda: RootNodeHandler(context).generate(),
     }
 
     handler = derivation_handlers.get(concept.derivation)

trilogy/core/processing/node_generators/group_node.py

@@ -22,6 +22,24 @@ from trilogy.utility import unique
 LOGGER_PREFIX = "[GEN_GROUP_NODE]"
 
 
+def get_aggregate_grain(
+    concept: BuildConcept, environment: BuildEnvironment
+) -> BuildGrain:
+    parent_concepts: List[BuildConcept] = unique(
+        resolve_function_parent_concepts(concept, environment=environment), "address"
+    )
+    if (
+        concept.grain
+        and len(concept.grain.components) > 0
+        and not concept.grain.abstract
+    ):
+        grain_components = [environment.concepts[c] for c in concept.grain.components]
+        parent_concepts += grain_components
+        return BuildGrain.from_concepts(parent_concepts)
+    else:
+        return BuildGrain.from_concepts(parent_concepts)
+
+
 def gen_group_node(
     concept: BuildConcept,
     local_optional: List[BuildConcept],
@@ -51,7 +69,7 @@ def gen_group_node(
     ):
         grain_components = [environment.concepts[c] for c in concept.grain.components]
         parent_concepts += grain_components
-        build_grain_parents = BuildGrain.from_concepts(parent_concepts)
+        build_grain_parents = get_aggregate_grain(concept, environment)
         output_concepts += grain_components
         for possible_agg in local_optional:
 
@@ -70,6 +88,7 @@ def gen_group_node(
                     possible_agg,
                     environment=environment,
                 )
+                comp_grain = get_aggregate_grain(possible_agg, environment)
                 if set([x.address for x in agg_parents]).issubset(
                     set([x.address for x in parent_concepts])
                 ):
@@ -77,7 +96,7 @@ def gen_group_node(
                     logger.info(
                         f"{padding(depth)}{LOGGER_PREFIX} found equivalent group by optional concept {possible_agg.address} for {concept.address}"
                     )
-                elif BuildGrain.from_concepts(agg_parents) == build_grain_parents:
+                elif comp_grain == build_grain_parents:
                     extra = [x for x in agg_parents if x.address not in parent_concepts]
                     parent_concepts += extra
                     output_concepts.append(possible_agg)
@@ -86,7 +105,7 @@ def gen_group_node(
                     )
                 else:
                     logger.info(
-                        f"{padding(depth)}{LOGGER_PREFIX} cannot include optional agg {possible_agg.address}; mismatched parent grain {BuildGrain.from_concepts(agg_parents)} vs local parent {BuildGrain.from_concepts(parent_concepts)}"
+                        f"{padding(depth)}{LOGGER_PREFIX} cannot include optional agg {possible_agg.address}; it has mismatched parent grain {comp_grain } vs local parent {build_grain_parents}"
                     )
     if parent_concepts:
         logger.info(

trilogy/core/processing/node_generators/select_merge_node.py

@@ -231,7 +231,10 @@ def create_pruned_concept_graph(
 
 
 def resolve_subgraphs(
-    g: nx.DiGraph, relevant: list[BuildConcept], conditions: BuildWhereClause | None
+    g: nx.DiGraph,
+    relevant: list[BuildConcept],
+    conditions: BuildWhereClause | None,
+    depth: int = 0,
 ) -> dict[str, list[str]]:
     """When we have multiple distinct subgraphs within our matched
     nodes that can satisfy a query, resolve which one of those we should
@@ -241,7 +244,7 @@ def resolve_subgraphs(
     discarding duplicates.
     Duplicate subgraphs will be resolved based on which
     ones are most 'optimal' to use, a hueristic
-    that can evolve in the future but is currently based on
+    that can evolve in the future but is currently based on datasource
     cardinality."""
     datasources = [n for n in g.nodes if n.startswith("ds~")]
     subgraphs: dict[str, list[str]] = {
@@ -261,7 +264,7 @@ def resolve_subgraphs(
     pruned_subgraphs = {}
 
     def score_node(input: str):
-        logger.debug(f"scoring node {input}")
+        logger.debug(f"{padding(depth)}{LOGGER_PREFIX} scoring node {input}")
         grain = grain_length[input]
         # first - go for lowest grain
         # but if the object we want is in the grain, treat that as "free"
@@ -275,7 +278,7 @@ def resolve_subgraphs(
             len(subgraphs[input]),
             input,
         )
-        logger.debug(score)
+        logger.debug(f"{padding(depth)}{LOGGER_PREFIX} node {input} has score {score}")
         return score
 
     for key, nodes in subgraphs.items():
@@ -296,7 +299,7 @@ def resolve_subgraphs(
                 if len(value) < len(other_value):
                     is_subset = True
                     logger.debug(
-                        f"Dropping subgraph {key} with {value} as it is a subset of {other_key} with {other_value}"
+                        f"{padding(depth)}{LOGGER_PREFIX} Dropping subgraph {key} with {value} as it is a subset of {other_key} with {other_value}"
                     )
                 elif len(value) == len(other_value) and len(all_concepts) == len(
                     other_all_concepts
@@ -305,7 +308,9 @@ def resolve_subgraphs(
                     matches.add(key)
         if matches and not is_subset:
             min_node = min(matches, key=score_node)
-            logger.debug(f"minimum source score is {min_node}")
+            logger.debug(
+                f"{padding(depth)}{LOGGER_PREFIX} minimum source score is {min_node}"
+            )
             is_subset = key is not min(matches, key=score_node)
         if not is_subset:
             pruned_subgraphs[key] = nodes
@@ -330,7 +335,9 @@ def resolve_subgraphs(
                 > 1
             )
         if not keep:
-            logger.debug(f"Pruning node {node} as irrelevant after subgraph resolution")
+            logger.debug(
+                f"{padding(depth)}{LOGGER_PREFIX} Pruning node {node} as irrelevant after subgraph resolution"
+            )
             pruned_subgraphs = {
                 k: [n for n in v if n != node] for k, v in pruned_subgraphs.items()
             }
@@ -561,7 +568,7 @@ def gen_select_merge_node(
         return None
 
     sub_nodes = resolve_subgraphs(
-        pruned_concept_graph, relevant=non_constant, conditions=conditions
+        pruned_concept_graph, relevant=non_constant, conditions=conditions, depth=depth
     )
 
     logger.info(f"{padding(depth)}{LOGGER_PREFIX} fetching subgraphs {sub_nodes}")

trilogy/core/query_processor.py

@@ -395,14 +395,13 @@ def get_query_node(
     if not statement.output_components:
         raise ValueError(f"Statement has no output components {statement}")
     history = history or History(base_environment=environment)
-    print(
+    logger.info(
         f"{LOGGER_PREFIX} building query node for {statement.output_components} grain {statement.grain}"
     )
     build_statement: BuildSelectLineage | BuildMultiSelectLineage = Factory(
         environment=environment,
     ).build(statement)
 
-    # build_statement = statement
     build_environment = environment.materialize_for_select(
         build_statement.local_concepts
     )

trilogy/dialect/base.py

@@ -12,6 +12,7 @@ from trilogy.constants import (
 )
 from trilogy.core.constants import UNNEST_NAME
 from trilogy.core.enums import (
+    ComparisonOperator,
     DatePart,
     FunctionType,
     UnnestMode,
@@ -276,7 +277,13 @@ ORDER BY{% for order in order_by %}
 )
 
 
-def safe_get_cte_value(coalesce, cte: CTE | UnionCTE, c: BuildConcept, quote_char: str):
+def safe_get_cte_value(
+    coalesce,
+    cte: CTE | UnionCTE,
+    c: BuildConcept,
+    quote_char: str,
+    render_expr: Callable,
+) -> Optional[str]:
     address = c.address
     raw = cte.source_map.get(address, None)
 
@@ -287,6 +294,9 @@ def safe_get_cte_value(coalesce, cte: CTE | UnionCTE, c: BuildConcept, quote_cha
         return f"{quote_char}{raw}{quote_char}.{safe_quote(rendered, quote_char)}"
     if isinstance(raw, list) and len(raw) == 1:
         rendered = cte.get_alias(c, raw[0])
+        if isinstance(rendered, FUNCTION_ITEMS):
+            # if it's a function, we need to render it as a function
+            return f"{render_expr(rendered, cte=cte, raise_invalid=True)}"
         return f"{quote_char}{raw[0]}{quote_char}.{safe_quote(rendered, quote_char)}"
     return coalesce(
         sorted(
@@ -499,6 +509,7 @@ class BaseDialect:
                     cte,
                     c,
                     self.QUOTE_CHARACTER,
+                    self.render_expr,
                 )
                 if not rval:
                     # unions won't have a specific source mapped; just use a generic column reference
@@ -515,6 +526,17 @@ class BaseDialect:
                         )
         return rval
 
+    def render_array_unnest(
+        self,
+        left,
+        right,
+        operator: ComparisonOperator,
+        cte: CTE | UnionCTE | None = None,
+        cte_map: Optional[Dict[str, CTE | UnionCTE]] = None,
+        raise_invalid: bool = False,
+    ):
+        return f"{self.render_expr(left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {operator.value} {self.render_expr(right, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}"
+
     def render_expr(
         self,
         e: Union[
@@ -556,6 +578,7 @@ class BaseDialect:
         raise_invalid: bool = False,
     ) -> str:
         if isinstance(e, SUBSELECT_COMPARISON_ITEMS):
+
             if isinstance(e.right, BuildConcept):
                 # we won't always have an existnce map
                 # so fall back to the normal map
@@ -585,10 +608,22 @@ class BaseDialect:
                     info = cte.inlined_ctes[target]
                     return f"{self.render_expr(e.left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {e.operator.value} (select {target}.{self.QUOTE_CHARACTER}{e.right.safe_address}{self.QUOTE_CHARACTER} from {info.new_base} as {target} where {target}.{self.QUOTE_CHARACTER}{e.right.safe_address}{self.QUOTE_CHARACTER} is not null)"
                 return f"{self.render_expr(e.left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {e.operator.value} (select {target}.{self.QUOTE_CHARACTER}{e.right.safe_address}{self.QUOTE_CHARACTER} from {target} where {target}.{self.QUOTE_CHARACTER}{e.right.safe_address}{self.QUOTE_CHARACTER} is not null)"
-
+            elif isinstance(e.right, BuildParamaterizedConceptReference):
+                if isinstance(e.right.concept.lineage, BuildFunction) and isinstance(
+                    e.right.concept.lineage.arguments[0], ListWrapper
+                ):
+                    return self.render_array_unnest(
+                        e.left,
+                        e.right,
+                        e.operator,
+                        cte=cte,
+                        cte_map=cte_map,
+                        raise_invalid=raise_invalid,
+                    )
+                return f"{self.render_expr(e.left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {e.operator.value} {self.render_expr(e.right, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}"
             elif isinstance(
                 e.right,
-                (ListWrapper, TupleWrapper, BuildParenthetical, list),
+                (ListWrapper, TupleWrapper, BuildParenthetical),
             ):
                 return f"{self.render_expr(e.left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {e.operator.value} {self.render_expr(e.right, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}"
 
@@ -692,13 +727,11 @@ class BaseDialect:
             return f"'{e}'"
         elif isinstance(e, (int, float)):
             return str(e)
-        elif isinstance(e, ListWrapper):
-            return f"[{','.join([self.render_expr(x, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid) for x in e])}]"
         elif isinstance(e, TupleWrapper):
             return f"({','.join([self.render_expr(x, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid) for x in e])})"
         elif isinstance(e, MapWrapper):
             return f"MAP {{{','.join([f'{self.render_expr(k, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}:{self.render_expr(v, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)}' for k, v in e.items()])}}}"
-        elif isinstance(e, list):
+        elif isinstance(e, ListWrapper):
             return f"{self.FUNCTION_MAP[FunctionType.ARRAY]([self.render_expr(x, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid) for x in e])}"
         elif isinstance(e, DataType):
             return self.DATATYPE_MAP.get(e, e.value)

trilogy/dialect/bigquery.py

@@ -1,11 +1,17 @@
-from typing import Any, Callable, Mapping
+from typing import Any, Callable, Dict, Mapping, Optional
 
 from jinja2 import Template
 
-from trilogy.core.enums import FunctionType, UnnestMode, WindowType
+from trilogy.core.enums import (
+    ComparisonOperator,
+    FunctionType,
+    UnnestMode,
+    WindowType,
+)
 from trilogy.core.models.core import (
     DataType,
 )
+from trilogy.core.models.execute import CTE, UnionCTE
 from trilogy.dialect.base import BaseDialect
 
 WINDOW_FUNCTION_MAP: Mapping[WindowType, Callable[[Any, Any, Any], str]] = {}
@@ -99,3 +105,14 @@ class BigqueryDialect(BaseDialect):
     SQL_TEMPLATE = BQ_SQL_TEMPLATE
     UNNEST_MODE = UnnestMode.CROSS_JOIN_UNNEST
     DATATYPE_MAP = DATATYPE_MAP
+
+    def render_array_unnest(
+        self,
+        left,
+        right,
+        operator: ComparisonOperator,
+        cte: CTE | UnionCTE | None = None,
+        cte_map: Optional[Dict[str, CTE | UnionCTE]] = None,
+        raise_invalid: bool = False,
+    ):
+        return f"{self.render_expr(left, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)} {operator.value} unnest({self.render_expr(right, cte=cte, cte_map=cte_map, raise_invalid=raise_invalid)})"

trilogy/dialect/duckdb.py

@@ -1,3 +1,4 @@
+import re
 from typing import Any, Callable, Mapping
 
 from jinja2 import Template
@@ -8,6 +9,19 @@ from trilogy.dialect.base import BaseDialect
 
 WINDOW_FUNCTION_MAP: Mapping[WindowType, Callable[[Any, Any, Any], str]] = {}
 
+SENTINAL_AUTO_CAPTURE_GROUP_VALUE = "-1"
+
+
+def generate_regex_extract(x: list[str]) -> str:
+    if str(x[2]) == SENTINAL_AUTO_CAPTURE_GROUP_VALUE:
+        regex = re.compile(x[1])
+        if regex.groups == 0:
+            search = 0
+        else:
+            search = 1
+        return f"REGEXP_EXTRACT({x[0]},{x[1]},{search})"
+    return f"REGEXP_EXTRACT({x[0]},{x[1]},{x[2]})"
+
 
 FUNCTION_MAP = {
     FunctionType.COUNT: lambda args: f"count({args[0]})",
@@ -37,6 +51,9 @@ FUNCTION_MAP = {
     FunctionType.DATETIME_LITERAL: lambda x: f"datetime '{x}'",
     # string
     FunctionType.CONTAINS: lambda x: f"CONTAINS(LOWER({x[0]}), LOWER({x[1]}))",
+    # regexp
+    FunctionType.REGEXP_CONTAINS: lambda x: f"REGEXP_MATCHES({x[0]},{x[1]})",
+    FunctionType.REGEXP_EXTRACT: lambda x: generate_regex_extract(x),
 }
 
 # if an aggregate function is called on a source that is at the same grain as the aggregate

trilogy/parsing/common.py

@@ -44,6 +44,7 @@ from trilogy.core.models.author import (
     RowsetLineage,
     SubselectComparison,
     TraitDataType,
+    TupleWrapper,
     UndefinedConcept,
     WhereClause,
     WindowItem,
@@ -172,7 +173,7 @@ def concept_list_to_keys(
 
 
 def constant_to_concept(
-    parent: ListWrapper | MapWrapper | list | int | float | str,
+    parent: ListWrapper | TupleWrapper | MapWrapper | int | float | str,
     name: str,
     namespace: str,
     metadata: Metadata | None = None,
@@ -183,7 +184,7 @@ def constant_to_concept(
         output_purpose=Purpose.CONSTANT,
         arguments=[parent],
     )
-    assert const_function.arguments[0] == parent, const_function.arguments[0]
+    # assert const_function.arguments[0] == parent, f'{const_function.arguments[0]} != {parent}, {type(const_function.arguments[0])} != {type(parent)}'
     fmetadata = metadata or Metadata()
     return Concept(
         name=name,

trilogy/parsing/parse_engine.py

@@ -146,7 +146,7 @@ class ParsePass(Enum):
     VALIDATION = 2
 
 
-CONSTANT_TYPES = (int, float, str, bool, list, ListWrapper, MapWrapper)
+CONSTANT_TYPES = (int, float, str, bool, ListWrapper, TupleWrapper, MapWrapper)
 
 SELF_LABEL = "root"
 
@@ -742,6 +742,7 @@ class ParseToObjects(Transformer):
         lookup, namespace, name, parent = parse_concept_reference(
             name, self.environment
         )
+
         concept = Concept(
             name=name,
             datatype=arg_to_datatype(constant),
@@ -1801,6 +1802,9 @@ class ParseToObjects(Transformer):
 
     @v_args(meta=True)
     def fregexp_extract(self, meta, args):
+        if len(args) == 2:
+            # this is a magic value to represent the default behavior
+            args.append(-1)
         return self.function_factory.create_function(
             args, FunctionType.REGEXP_EXTRACT, meta
         )

trilogy/parsing/trilogy.lark

@@ -279,7 +279,7 @@
     _SUBSTRING.1: "substring("i
     fsubstring: _SUBSTRING expr "," expr "," expr ")"
     _REGEXP_EXTRACT.1: "regexp_extract("
-    fregexp_extract: _REGEXP_EXTRACT expr "," expr ")"
+    fregexp_extract: _REGEXP_EXTRACT expr "," expr ("," int_lit)? ")"
     _REGEXP_CONTAINS.1: "regexp_contains("
     fregexp_contains: _REGEXP_CONTAINS expr "," expr  ")"
     _REGEXP_REPLACE.1: "regexp_replace("