pytrilogy 0.0.2.20__tar.gz → 0.0.2.22__tar.gz

{pytrilogy-0.0.2.20/pytrilogy.egg-info → pytrilogy-0.0.2.22}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pytrilogy
-Version: 0.0.2.20
+Version: 0.0.2.22
 Summary: Declarative, typed query language that compiles to SQL.
 Home-page: 
 Author: 
@@ -145,7 +145,7 @@ Run the following from the directory the file is in.
 trilogy run hello.trilogy duckdb
 ```
 
-![UI Preview](./hello_world.png)
+![UI Preview](hello-world.png)
 
 ## Backends
 
@@ -158,9 +158,9 @@ The current Trilogy implementation supports these backends:
 
 ## Basic Example - Python
 
-Trilogy can be run directly in python.
+Trilogy can be run directly in python through the core SDK. Trilogy code can be defined and parsed inline or parsed out of files.
 
-A bigquery example, similar to bigquery [the quickstart](https://cloud.google.com/bigquery/docs/quickstarts/query-public-dataset-console)
+A bigquery example, similar to bigquery [the quickstart](https://cloud.google.com/bigquery/docs/quickstarts/query-public-dataset-console).
 
 ```python
 
@@ -224,7 +224,7 @@ and second the dialect to run.
 To pass arguments to a backend, append additional --<option> flags after specifying the dialect.
 
 Example:
-`trilogy run key in int; datasource test_source ( i:in) grain(in) address test; select in;" duckdb --path <path/to/duckdb>`
+`trilogy run "key x int; datasource test_source ( i:x) grain(in) address test; select x;" duckdb --path <path/to/database>`
 
 ### Bigquery Args
 N/A, only supports default auth. In python you can pass in a custom client.
@@ -267,7 +267,7 @@ Clone repository and install requirements.txt and requirements-test.txt.
 Please open an issue first to discuss what you would like to change, and then create a PR against that issue.
 
 ## Similar in space
-Trilogy combines two aspects; a semantic layer and a query language. We've covered examples of both below:
+Trilogy combines two aspects; a semantic layer and a query language. Examples of both are linked below:
 
 Python "semantic layers" are tools for defining data access to a warehouse in a more abstract way.
 
@@ -284,20 +284,26 @@ but all are worth checking out. Please open PRs/comment for anything missed!
 
 #### IMPORT
 
-`import <path> as <alias>;`
+`import [path] as [alias];`
 
 #### CONCEPT
 
-Types: `string | int | float | bool | date | datetime | time | numeric(scale, precision) | timestamp | interval`;
+Types: `string | int | float | bool | date | datetime | time | numeric(scale, precision) | timestamp | interval | list<[type]> | map<[type], [type]> | struct<name:[type], name:[type]>`;
 
 Key:
-`key <name> <type>;`
+`key [name] [type];`
 
 Property:
-`property <key>.<name> <type>;`
+`property [key>].[name] [type];`
+`property x.y int;`
+or 
+`property <[key](,[key])?>.<name> [type];`
+`property <x,y>.z int;`
+
 
 Transformation:
-`auto <name> <- <expression>;`
+`auto [name] <- [expression];`
+`auto x <- y + 1;`
 
 #### DATASOURCE
 ```sql

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/README.md

@@ -116,7 +116,7 @@ Run the following from the directory the file is in.
 trilogy run hello.trilogy duckdb
 ```
 
-![UI Preview](./hello_world.png)
+![UI Preview](hello-world.png)
 
 ## Backends
 
@@ -129,9 +129,9 @@ The current Trilogy implementation supports these backends:
 
 ## Basic Example - Python
 
-Trilogy can be run directly in python.
+Trilogy can be run directly in python through the core SDK. Trilogy code can be defined and parsed inline or parsed out of files.
 
-A bigquery example, similar to bigquery [the quickstart](https://cloud.google.com/bigquery/docs/quickstarts/query-public-dataset-console)
+A bigquery example, similar to bigquery [the quickstart](https://cloud.google.com/bigquery/docs/quickstarts/query-public-dataset-console).
 
 ```python
 
@@ -195,7 +195,7 @@ and second the dialect to run.
 To pass arguments to a backend, append additional --<option> flags after specifying the dialect.
 
 Example:
-`trilogy run key in int; datasource test_source ( i:in) grain(in) address test; select in;" duckdb --path <path/to/duckdb>`
+`trilogy run "key x int; datasource test_source ( i:x) grain(in) address test; select x;" duckdb --path <path/to/database>`
 
 ### Bigquery Args
 N/A, only supports default auth. In python you can pass in a custom client.
@@ -238,7 +238,7 @@ Clone repository and install requirements.txt and requirements-test.txt.
 Please open an issue first to discuss what you would like to change, and then create a PR against that issue.
 
 ## Similar in space
-Trilogy combines two aspects; a semantic layer and a query language. We've covered examples of both below:
+Trilogy combines two aspects; a semantic layer and a query language. Examples of both are linked below:
 
 Python "semantic layers" are tools for defining data access to a warehouse in a more abstract way.
 
@@ -255,20 +255,26 @@ but all are worth checking out. Please open PRs/comment for anything missed!
 
 #### IMPORT
 
-`import <path> as <alias>;`
+`import [path] as [alias];`
 
 #### CONCEPT
 
-Types: `string | int | float | bool | date | datetime | time | numeric(scale, precision) | timestamp | interval`;
+Types: `string | int | float | bool | date | datetime | time | numeric(scale, precision) | timestamp | interval | list<[type]> | map<[type], [type]> | struct<name:[type], name:[type]>`;
 
 Key:
-`key <name> <type>;`
+`key [name] [type];`
 
 Property:
-`property <key>.<name> <type>;`
+`property [key>].[name] [type];`
+`property x.y int;`
+or 
+`property <[key](,[key])?>.<name> [type];`
+`property <x,y>.z int;`
+
 
 Transformation:
-`auto <name> <- <expression>;`
+`auto [name] <- [expression];`
+`auto x <- y + 1;`
 
 #### DATASOURCE
 ```sql

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22/pytrilogy.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pytrilogy
-Version: 0.0.2.20
+Version: 0.0.2.22
 Summary: Declarative, typed query language that compiles to SQL.
 Home-page: 
 Author: 
@@ -145,7 +145,7 @@ Run the following from the directory the file is in.
 trilogy run hello.trilogy duckdb
 ```
 
-![UI Preview](./hello_world.png)
+![UI Preview](hello-world.png)
 
 ## Backends
 
@@ -158,9 +158,9 @@ The current Trilogy implementation supports these backends:
 
 ## Basic Example - Python
 
-Trilogy can be run directly in python.
+Trilogy can be run directly in python through the core SDK. Trilogy code can be defined and parsed inline or parsed out of files.
 
-A bigquery example, similar to bigquery [the quickstart](https://cloud.google.com/bigquery/docs/quickstarts/query-public-dataset-console)
+A bigquery example, similar to bigquery [the quickstart](https://cloud.google.com/bigquery/docs/quickstarts/query-public-dataset-console).
 
 ```python
 
@@ -224,7 +224,7 @@ and second the dialect to run.
 To pass arguments to a backend, append additional --<option> flags after specifying the dialect.
 
 Example:
-`trilogy run key in int; datasource test_source ( i:in) grain(in) address test; select in;" duckdb --path <path/to/duckdb>`
+`trilogy run "key x int; datasource test_source ( i:x) grain(in) address test; select x;" duckdb --path <path/to/database>`
 
 ### Bigquery Args
 N/A, only supports default auth. In python you can pass in a custom client.
@@ -267,7 +267,7 @@ Clone repository and install requirements.txt and requirements-test.txt.
 Please open an issue first to discuss what you would like to change, and then create a PR against that issue.
 
 ## Similar in space
-Trilogy combines two aspects; a semantic layer and a query language. We've covered examples of both below:
+Trilogy combines two aspects; a semantic layer and a query language. Examples of both are linked below:
 
 Python "semantic layers" are tools for defining data access to a warehouse in a more abstract way.
 
@@ -284,20 +284,26 @@ but all are worth checking out. Please open PRs/comment for anything missed!
 
 #### IMPORT
 
-`import <path> as <alias>;`
+`import [path] as [alias];`
 
 #### CONCEPT
 
-Types: `string | int | float | bool | date | datetime | time | numeric(scale, precision) | timestamp | interval`;
+Types: `string | int | float | bool | date | datetime | time | numeric(scale, precision) | timestamp | interval | list<[type]> | map<[type], [type]> | struct<name:[type], name:[type]>`;
 
 Key:
-`key <name> <type>;`
+`key [name] [type];`
 
 Property:
-`property <key>.<name> <type>;`
+`property [key>].[name] [type];`
+`property x.y int;`
+or 
+`property <[key](,[key])?>.<name> [type];`
+`property <x,y>.z int;`
+
 
 Transformation:
-`auto <name> <- <expression>;`
+`auto [name] <- [expression];`
+`auto x <- y + 1;`
 
 #### DATASOURCE
 ```sql

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/pytrilogy.egg-info/SOURCES.txt

@@ -22,6 +22,7 @@ tests/test_parsing.py
 tests/test_partial_handling.py
 tests/test_query_processing.py
 tests/test_select.py
+tests/test_show.py
 tests/test_statements.py
 tests/test_undefined_concept.py
 tests/test_where_clause.py

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/tests/test_models.py

@@ -100,12 +100,16 @@ def test_conditional(test_environment, test_environment_graph):
     merged = condition_a + condition_b
     assert merged == condition_a
 
-    test_concept_two = list(test_environment.concepts.values())[-2]
+    test_concept_two = [
+        x
+        for x in test_environment.concepts.values()
+        if x.address != test_concept.address
+    ].pop()
     condition_c = Conditional(
         left=test_concept, right=test_concept_two, operator=BooleanOperator.AND
     )
     merged_two = condition_a + condition_c
-    assert merged_two.left == condition_a
+    assert merged_two.left == condition_a, f"{str(merged_two.left), str(condition_a)}"
     assert merged_two.right == condition_c
     assert merged_two.operator == BooleanOperator.AND
 

pytrilogy-0.0.2.22/tests/test_show.py

@@ -0,0 +1,58 @@
+from trilogy.core.models import ShowStatement
+from trilogy.core.query_processor import process_query
+from trilogy import Dialects
+from trilogy.parser import parse
+
+
+def test_show_bigquery():
+    declarations = """
+    key user_id int metadata(description="the description");
+    property user_id.display_name string metadata(description="The display name ");
+    property user_id.about_me string metadata(description="User provided description");
+    key post_id int;
+
+
+    datasource posts (
+        user_id: user_id,
+        id: post_id
+        )
+        grain (post_id)
+        address bigquery-public-data.stackoverflow.post_history
+    ;
+
+
+    datasource users (
+        id: user_id,
+        display_name: display_name,
+        about_me: about_me,
+        )
+        grain (user_id)
+        address bigquery-public-data.stackoverflow.users
+    ;
+
+
+        """
+    env, parsed = parse(declarations)
+
+    q1 = """
+    metric post_count<- count(post_id);
+    metric distinct_post_count <- count_distinct(post_id);
+    
+    metric user_count <- count(user_id);
+    
+    show select
+        post_count,
+        distinct_post_count,
+        user_count
+    ;"""
+    env, parsed = parse(q1, environment=env)
+    select: ShowStatement = parsed[-1]
+
+    query = (
+        Dialects.DUCK_DB.default_executor(environment=env)
+        .execute_query(select)
+        .fetchall()
+    )
+    assert "FULL JOIN wakeful on 1=1" in query[0]["__preql_internal_query_text"], query[
+        0
+    ]["__preql_internal_query_text"]

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/trilogy/__init__.py

@@ -4,6 +4,6 @@ from trilogy.executor import Executor
 from trilogy.parser import parse
 from trilogy.constants import CONFIG
 
-__version__ = "0.0.2.20"
+__version__ = "0.0.2.22"
 
 __all__ = ["parse", "Executor", "Dialects", "Environment", "CONFIG"]

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/trilogy/constants.py

@@ -44,6 +44,7 @@ class Comments:
 class Config:
     strict_mode: bool = True
     human_identifiers: bool = True
+    randomize_cte_names: bool = False
     validate_missing: bool = True
     comments: Comments = field(default_factory=Comments)
     optimizations: Optimizations = field(default_factory=Optimizations)

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/trilogy/core/constants.py

@@ -1,3 +1,4 @@
 CONSTANT_DATASET: str = "preql_internal_constant_dataset"
 ALL_ROWS_CONCEPT = "all_rows"
 INTERNAL_NAMESPACE = "__preql_internal"
+PERSISTED_CONCEPT_PREFIX = "__pre_persist"

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/trilogy/core/enums.py

@@ -12,6 +12,8 @@ class UnnestMode(Enum):
 
 class ConceptSource(Enum):
     MANUAL = "manual"
+    CTE = "cte"
+    PERSIST_STATEMENT = "persist_statement"
     AUTO_DERIVED = "auto_derived"
 
 
@@ -205,6 +207,8 @@ class FunctionClass(Enum):
         FunctionType.CURRENT_DATETIME,
     ]
 
+    ONE_TO_MANY = [FunctionType.UNNEST]
+
 
 class Boolean(Enum):
     TRUE = "true"

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/trilogy/core/env_processor.py

@@ -10,9 +10,11 @@ def add_concept(concept: Concept, g: ReferenceGraph):
     g.add_node(concept)
     # if we have sources, recursively add them
     node_name = concept_to_node(concept)
-    if concept.sources:
-        for source in concept.sources:
+    if concept.concept_arguments:
+        for source in concept.concept_arguments:
             generic = source.with_default_grain()
+            add_concept(generic, g)
+
             g.add_edge(generic, node_name)
     for _, pseudonym in concept.pseudonyms.items():
         pseudonym = pseudonym.with_default_grain()

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/trilogy/core/ergonomics.py

@@ -1,3 +1,5 @@
+from trilogy.constants import CONFIG
+
 # source: https://github.com/aaronbassett/Pass-phrase
 CTE_NAMES = """quizzical
 highfalutin
@@ -103,8 +105,6 @@ mandrill
 marlin
 monitor
 ocelot
-osprey
-owl
 petrel
 python
 ray
@@ -132,7 +132,6 @@ cuckoo
 darter
 dove
 duck
-eagle
 falcon
 finch
 flamingo
@@ -184,4 +183,12 @@ warbler""".split(
     "\n"
 )
 
-CTE_NAMES = list(set(CTE_NAMES))
+
+def generate_cte_names():
+    if CONFIG.randomize_cte_names:
+        from random import shuffle
+
+        new = [*CTE_NAMES]
+        shuffle(new)
+        return new
+    return CTE_NAMES

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/trilogy/core/models.py

@@ -44,6 +44,7 @@ from trilogy.core.constants import (
     ALL_ROWS_CONCEPT,
     INTERNAL_NAMESPACE,
     CONSTANT_DATASET,
+    PERSISTED_CONCEPT_PREFIX,
 )
 from trilogy.core.enums import (
     InfiniteFunctionArgs,
@@ -912,10 +913,10 @@ class Grain(Mergeable, BaseModel):
         )
 
     @cached_property
-    def set(self):
+    def set(self) -> set[str]:
         base = []
         for x in self.components_copy:
-            if x.derivation == PurposeLineage.ROWSET:
+            if isinstance(x.lineage, RowsetItem):
                 base.append(x.lineage.content.address)
             else:
                 base.append(x.address)
@@ -3131,6 +3132,12 @@ class EnvironmentConceptDict(dict):
     def values(self) -> ValuesView[Concept]:  # type: ignore
         return super().values()
 
+    def get(self, key: str, default: Concept | None = None) -> Concept | None:  # type: ignore
+        try:
+            return self.__getitem__(key)
+        except UndefinedConceptException:
+            return default
+
     def __getitem__(
         self, key, line_no: int | None = None
     ) -> Concept | UndefinedConcept:
@@ -3257,24 +3264,62 @@ class Environment(BaseModel):
         for datasource in self.datasources.values():
             for concept in datasource.output_concepts:
                 concrete_addresses.add(concept.address)
-        self.materialized_concepts = [
-            c for c in self.concepts.values() if c.address in concrete_addresses
-        ] + [
-            c
-            for c in self.alias_origin_lookup.values()
-            if c.address in concrete_addresses
-        ]
+        self.materialized_concepts = unique(
+            [c for c in self.concepts.values() if c.address in concrete_addresses]
+            + [
+                c
+                for c in self.alias_origin_lookup.values()
+                if c.address in concrete_addresses
+            ],
+            "address",
+        )
 
-    def validate_concept(self, lookup: str, meta: Meta | None = None):
+    def validate_concept(self, new_concept: Concept, meta: Meta | None = None):
+        lookup = new_concept.address
         existing: Concept = self.concepts.get(lookup)  # type: ignore
         if not existing:
             return
-        elif existing and self.environment_config.allow_duplicate_declaration:
+
+        def handle_persist():
+            deriv_lookup = (
+                f"{existing.namespace}.{PERSISTED_CONCEPT_PREFIX}_{existing.name}"
+            )
+
+            alt_source = self.alias_origin_lookup.get(deriv_lookup)
+            if not alt_source:
+                return None
+            # if the new concept binding has no lineage
+            # nothing to cause us to think a persist binding
+            # needs to be invalidated
+            if not new_concept.lineage:
+                return existing
+            if str(alt_source.lineage) == str(new_concept.lineage):
+                logger.info(
+                    f"Persisted concept {existing.address} matched redeclaration, keeping current persistence binding."
+                )
+                return existing
+            logger.warning(
+                f"Persisted concept {existing.address} lineage {str(alt_source.lineage)} did not match redeclaration {str(new_concept.lineage)}, overwriting and invalidating persist binding."
+            )
+            for k, datasource in self.datasources.items():
+                if existing.address in datasource.output_concepts:
+                    datasource.columns = [
+                        x
+                        for x in datasource.columns
+                        if x.concept.address != existing.address
+                    ]
+            return None
+
+        if existing and self.environment_config.allow_duplicate_declaration:
+            if existing.metadata.concept_source == ConceptSource.PERSIST_STATEMENT:
+                return handle_persist()
             return
         elif existing.metadata:
+            if existing.metadata.concept_source == ConceptSource.PERSIST_STATEMENT:
+                return handle_persist()
             # if the existing concept is auto derived, we can overwrite it
             if existing.metadata.concept_source == ConceptSource.AUTO_DERIVED:
-                return
+                return None
         elif meta and existing.metadata:
             raise ValueError(
                 f"Assignment to concept '{lookup}' on line {meta.line} is a duplicate"
@@ -3390,11 +3435,12 @@ class Environment(BaseModel):
         _ignore_cache: bool = False,
     ):
         if not force:
-            self.validate_concept(concept.address, meta=meta)
+            existing = self.validate_concept(concept, meta=meta)
+            if existing:
+                concept = existing
         if concept.namespace == DEFAULT_NAMESPACE:
             self.concepts[concept.name] = concept
-        else:
-            self.concepts[concept.address] = concept
+        self.concepts[concept.address] = concept
         from trilogy.core.environment_helpers import generate_related_concepts
 
         generate_related_concepts(concept, self, meta=meta, add_derived=add_derived)
@@ -3408,21 +3454,42 @@ class Environment(BaseModel):
         meta: Meta | None = None,
         _ignore_cache: bool = False,
     ):
-
         self.datasources[datasource.env_label] = datasource
-        for column in datasource.columns:
-            current_concept = column.concept
+        for current_concept in datasource.output_concepts:
             current_derivation = current_concept.derivation
+            # TODO: refine this section;
+            # too hacky for maintainability
             if current_derivation not in (PurposeLineage.ROOT, PurposeLineage.CONSTANT):
-                new_concept = current_concept.model_copy(deep=True)
-                new_concept.set_name("_pre_persist_" + current_concept.name)
-                # remove the associated lineage
-                current_concept.lineage = None
-                self.add_concept(new_concept, meta=meta, force=True, _ignore_cache=True)
-                self.add_concept(
-                    current_concept, meta=meta, force=True, _ignore_cache=True
-                )
-                self.merge_concept(new_concept, current_concept, [])
+                persisted = f"{PERSISTED_CONCEPT_PREFIX}_" + current_concept.name
+                # override the current concept source to reflect that it's now coming from a datasource
+                if (
+                    current_concept.metadata.concept_source
+                    != ConceptSource.PERSIST_STATEMENT
+                ):
+                    new_concept = current_concept.model_copy(deep=True)
+                    new_concept.set_name(persisted)
+                    self.add_concept(
+                        new_concept, meta=meta, force=True, _ignore_cache=True
+                    )
+                    current_concept.metadata.concept_source = (
+                        ConceptSource.PERSIST_STATEMENT
+                    )
+                    # remove the associated lineage
+                    # to make this a root for discovery purposes
+                    # as it now "exists" in a table
+                    current_concept.lineage = None
+                    current_concept = current_concept.with_default_grain()
+                    self.add_concept(
+                        current_concept, meta=meta, force=True, _ignore_cache=True
+                    )
+                    self.merge_concept(new_concept, current_concept, [])
+                else:
+                    self.add_concept(current_concept, meta=meta, _ignore_cache=True)
+
+            # else:
+            #     self.add_concept(
+            #         current_concept, meta=meta, _ignore_cache=True
+            #     )
         if not _ignore_cache:
             self.gen_concept_list_caches()
         return datasource
@@ -4230,6 +4297,9 @@ class RowsetDerivationStatement(Namespaced, BaseModel):
     def __repr__(self):
         return f"RowsetDerivation<{str(self.select)}>"
 
+    def __str__(self):
+        return self.__repr__()
+
     @property
     def derived_concepts(self) -> List[Concept]:
         output: list[Concept] = []
@@ -4248,7 +4318,8 @@ class RowsetDerivationStatement(Namespaced, BaseModel):
                     content=orig_concept, where=self.select.where_clause, rowset=self
                 ),
                 grain=orig_concept.grain,
-                metadata=orig_concept.metadata,
+                # TODO: add proper metadata
+                metadata=Metadata(concept_source=ConceptSource.CTE),
                 namespace=(
                     f"{self.name}.{orig_concept.namespace}"
                     if orig_concept.namespace != self.namespace
@@ -4275,6 +4346,7 @@ class RowsetDerivationStatement(Namespaced, BaseModel):
                     components=[orig[c.address] for c in x.grain.components_copy]
                 )
             else:
+
                 x.grain = default_grain
         return output
 
@@ -4300,6 +4372,9 @@ class RowsetItem(Mergeable, Namespaced, BaseModel):
             f"<Rowset<{self.rowset.name}>: {str(self.content)} where {str(self.where)}>"
         )
 
+    def __str__(self):
+        return self.__repr__()
+
     def with_merge(self, source: Concept, target: Concept, modifiers: List[Modifier]):
         return RowsetItem(
             content=self.content.with_merge(source, target, modifiers),

{pytrilogy-0.0.2.20 → pytrilogy-0.0.2.22}/trilogy/core/processing/concept_strategies_v3.py

@@ -432,7 +432,6 @@ def generate_node(
             conditions=conditions,
         )
         if not check:
-
             logger.info(
                 f"{depth_to_prefix(depth)}{LOGGER_PREFIX} Could not resolve root concepts, checking for expanded concepts"
             )