tinybird-cli 3.3.1.dev8__tar.gz → 3.6.1.dev9__tar.gz

{tinybird-cli-3.3.1.dev8 → tinybird-cli-3.6.1.dev9}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.1
 Name: tinybird-cli
-Version: 3.3.1.dev8
+Version: 3.6.1.dev9
 Summary: Tinybird Command Line Tool
-Home-page: https://docs.tinybird.co/cli.html
+Home-page: https://www.tinybird.co/docs/cli/introduction.html
 Author: Tinybird
 Author-email: support@tinybird.co
 Requires-Python: >=3.8, <3.12
@@ -11,7 +11,7 @@ Provides-Extra: bigquery
 Provides-Extra: snowflake
 
 Tinybird CLI
-============
+=============
 
 The Tinybird command-line tool allows you to use all the Tinybird functionality directly from the command line. Additionally, it includes several functions to create and manage data projects easily.
 
@@ -19,48 +19,93 @@ Changelog
 
 ---------
 
-3.3.1.dev8
+3.6.1.dev9
 ************
 
-- `Changed` `tb deploy` to deploy the datasources from top to bottom to make sure that if we modified the column in some datasource in the top. It will be deployed before the datasources that depend on it.
+- `Fixed` Added correct URL for checking available regions
+
 
-3.3.1.dev7
+3.6.1.dev8
 ************
 
-- `Changed` Post-release required when deploying changes that include changing a Copy Pipe.
+- `Fixed` Report better error message on `tb diff` when two Datafiles have the same name
 
-3.3.1.dev6
+3.6.1.dev7
 ************
 
-- `Fixed` `tb deploy` to deploy the endpoints in the correct order when there are endpoints depending on other endpoints.
+- `Fixed` `tb deploy` supports deployment when a `.incl` file is removed
 
-3.3.1.dev5
+3.6.1.dev6
 ************
 
-- `Fixed` `tb push` pipe with multiple tokens
+- `Fixed` Some issues when using `--fmt: off` with `CASE` in the sql
 
-3.3.1.dev4
+3.6.1.dev5
 ************
 
-- `Changed` script `exec_test.sh`  from `tb init --git` to run retries to avoid flakiness in tests
+- `Fixed` Avoid `tb fmt` to error if there's a `CASE` in the sql
+
+3.6.1.dev4
+************
 
-3.3.1.dev3
-***********
+- `Fixed` regression tests query when filtering by specific parameter
 
-- `Fixed` `IMPORT` and `ENGINE` changes were not detected `tb deploy`
+3.6.1.dev3
+************
+
+- `Changed` behavior when running `tb deploy` on a branch to push the connection settings to the backend. This change is the backend that decides what to do. 
 
-3.3.1.dev2
+3.6.1.dev2
 ************
 
-- `Changed` Update `tb release rm --oldest-rollback` to use new API
+- `Changed` fixed major version of tinybird-cli to lower than 4 when using `tb init --git`
+
+3.6.0
+************
 
-3.3.1.dev1
+- `Added` Add new envvar `TB_INCL_RELATIVE_PATH` in .tinyenv to look for Datafiles out of the Data Project directory
+- `Changed` On-demand Sink run checks the pipe type instead of checking the workspace connectors
+
+3.5.0
 ************
 
-- [Internal] Clean up how the dependencies graph is created when pushing resources using `tb deploy` or `tb push`. This change should not change the behavior of the CLI, but it should make the code more maintainable and easier to understand.
+- `Added` Token management in Copy pipes datafiles
+- `Added` Better error report on `tb push` when trying to push a pipe with bad templating
+- `Added` credential validation for S3 connections
+- `Added` `tb connection create` now supports `s3_iamrole` as service type
+- `Changed` error message when trying to modify a datasource without bumping the major semver version to add a link to the documentation.
+- `Changed` GH workflow files and secret includes the workspace name when generated from the UI and CLI
+- `Changed` rename `INDICES` to `INDEXES`
+- `Changed` Save git commit to workspace on `tb push --only-changes`
+- `Fixed` Fix error message when deleting an `.incl` file with the git workflow
+- `Fixed` support `tb deploy --dry-run`
+
+
+3.4.0
+************
+
+- `Added` Support `tb deploy` .datasource with `INDEXES`
+- `Added` Support `tb push` .datasource with `INDEXES`
+- `Added` Support `tb deploy` in a post release of a .datasource with `INDEXES`
+- `Added` Check backfill on preview is required when `tb deploy` modified datasources. Disable check with `TB_CHECK_BACKFILL_REQUIRED=0`.
+- `Changed` Allow spaces when managing multiple Workspace members
+- `Changed` `tb deploy` to not allow pushing materialized views with datasource definition in the same datafile. We are blocking this as has unsupported behavior and it misleading and less flexible than having 2 files one for the datasource and one for the materialized view.
+- `Changed` `tb deploy` to deploy the datasources from top to bottom to make sure that if we modified the column in some datasource in the top. It will be deployed before the datasources that depend on it.
+- `Changed` Post-release required when deploying changes that include changing a Copy Pipe.
+- `Changed` script `exec_test.sh`  from `tb init --git` to run retries to avoid flakiness in tests
+- `Changed` Update `tb release rm --oldest-rollback` to use new API
+- `Fixed` Post-release recommended instead of required when deploying changes that include changing a Copy Pipe. Displays a warning when other number of the semver is increased.
+- `Fixed` Fixed bug when a Data Source was created using an INDEX
+- `Fixed` Avoid None git diff change type
+- `Fixed` `tb deploy` to deploy the endpoints in the correct order when there are endpoints depending on other endpoints.
+- `Fixed` `tb push` pipe with multiple tokens
+- `Fixed` `IMPORT` and `ENGINE` changes were not detected `tb deploy`
+- `Removed` part of the code supporting python 3.6 as we stopped giving support 18 month ago
+
 
 3.3.0
 ************
+
 - `Added` Show warning when doing data or pipe operations directly against Live.
 - `Added` `bi_stats`, `bi_stats_rt` and `endpoint_errors` to the list of service data sources that the CLI ignore when looking for dependencies of pipes being pushed.
 - `Added` S3 Iam Role connection creation

{tinybird-cli-3.3.1.dev8 → tinybird-cli-3.6.1.dev9}/tinybird/__cli__.py

@@ -1,8 +1,8 @@
 
 __name__ = 'tinybird-cli'
 __description__ = 'Tinybird Command Line Tool'
-__url__ = 'https://docs.tinybird.co/cli.html'
+__url__ = 'https://www.tinybird.co/docs/cli/introduction.html'
 __author__ = 'Tinybird'
 __author_email__ = 'support@tinybird.co'
-__version__ = '3.3.1.dev8'
-__revision__ = 'ad8ec4e'
+__version__ = '3.6.1.dev9'
+__revision__ = 'ff22a5b'

{tinybird-cli-3.3.1.dev8 → tinybird-cli-3.6.1.dev9}/tinybird/ch_utils/engine.py

@@ -1,9 +1,10 @@
 import logging
 import re
 from collections import defaultdict
+from dataclasses import asdict
 from typing import Any, Callable, Dict, Iterable, List, Optional
 
-from ..sql import col_name, engine_replicated_to_local, parse_table_structure
+from ..sql import TableIndex, col_name, engine_replicated_to_local, parse_indexes_structure, parse_table_structure
 
 DEFAULT_EMPTY_PARAMETERS = ["ttl", "partition_key", "sorting_key"]
 DEFAULT_JOIN_EMPTY_PARAMETERS = ["join_strictness", "join_type", "key_columns"]
@@ -31,7 +32,20 @@ class TableDetails:
     >>> ed.to_datafile()
     ''
 
+    >>> ed = TableDetails({ "engine_full": "MergeTree() PARTITION BY toYear(timestamp) ORDER BY (timestamp, cityHash64(location)) SAMPLE BY cityHash64(location) SETTINGS index_granularity = 32, index_granularity_bytes = 2048", "engine": "MergeTree", "partition_key": "toYear(timestamp)", "sorting_key": "timestamp, cityHash64(location)", "primary_key": "timestamp, cityHash64(location)", "sampling_key": "cityHash64(location)", "settings": "index_granularity = 32, index_granularity_bytes = 2048", "ttl": None })
+    >>> ed.diff_ttl("toDate(timestamp) + toIntervalDay(1)")
+    True
     >>> ed = TableDetails({ "engine_full": "MergeTree() PARTITION BY toYear(timestamp) ORDER BY (timestamp, cityHash64(location)) SAMPLE BY cityHash64(location) SETTINGS index_granularity = 32, index_granularity_bytes = 2048 TTL toDate(timestamp) + INTERVAL 1 DAY", "engine": "MergeTree", "partition_key": "toYear(timestamp)", "sorting_key": "timestamp, cityHash64(location)", "primary_key": "timestamp, cityHash64(location)", "sampling_key": "cityHash64(location)", "settings": "index_granularity = 32, index_granularity_bytes = 2048", "ttl": "toDate(timestamp) + INTERVAL 1 DAY" })
+    >>> ed.diff_ttl("toDate(timestamp) + toIntervalDay(1)")
+    False
+    >>> ed.diff_ttl("toDate(timestamp) + toIntervalDay(2)")
+    True
+    >>> ed.diff_ttl("toDate(timestamp) + INTERVAL DAY 2")
+    True
+    >>> ed.diff_ttl("toDate(timestamp) + INTERVAL 1 DAY")
+    False
+    >>> ed.diff_ttl("")
+    True
     >>> ed.engine_full
     'MergeTree() PARTITION BY toYear(timestamp) ORDER BY (timestamp, cityHash64(location)) SAMPLE BY cityHash64(location) SETTINGS index_granularity = 32, index_granularity_bytes = 2048 TTL toDate(timestamp) + INTERVAL 1 DAY'
     >>> ed.engine
@@ -131,6 +145,16 @@ class TableDetails:
             return is_aggregating or is_replacing or is_collapsing
         return False
 
+    def diff_ttl(self, new_ttl: str) -> bool:
+        try:
+            from tinybird.sql_toolset import format_sql
+
+            current_ttl = format_sql(f"select {self.ttl}")[7:]
+            new_ttl = format_sql(f"select {new_ttl}")[7:]
+            return current_ttl != new_ttl
+        except Exception:
+            return self.ttl != new_ttl
+
     @property
     def partition_key(self) -> Optional[str]:
         return self.details.get("partition_key", None)
@@ -204,6 +228,10 @@ class TableDetails:
             "row_count": self.details.get("total_rows", None),
         }
 
+    @property
+    def indexes(self) -> List[TableIndex]:
+        return _parse_indexes(str(self.details.get("create_table_query", "")))
+
     def to_json(self, exclude: Optional[List[str]] = None, include_empty_details: bool = False):
         # name, database are not exported since they are not part of the engine
         d: Dict[str, Any] = {
@@ -234,6 +262,8 @@ class TableDetails:
             d["version"] = self.version
         if self.ttl:
             d["ttl"] = self.ttl.strip()
+        if self.indexes:
+            d["indexes"] = [asdict(index) for index in self.indexes]
 
         if self.engine_full:
             engine_params = engine_params_from_engine_full(self.engine_full)
@@ -253,7 +283,9 @@ class TableDetails:
         return d
 
     def to_datafile(self, include_empty_details: bool = False) -> str:
-        d: Dict[str, Any] = self.to_json(exclude=["engine", "engine_full"], include_empty_details=include_empty_details)
+        d: Dict[str, Any] = self.to_json(
+            exclude=["engine", "engine_full", "indexes"], include_empty_details=include_empty_details
+        )
         engine: str = self.engine
 
         datafile: str = ""
@@ -774,3 +806,21 @@ def ttl_condition_from_engine_full(engine_full: Optional[str]) -> Optional[str]:
     except Exception as e:
         logging.error(str(e))
         return None
+
+
+def _parse_indexes(create_table_query_expr: str) -> List[TableIndex]:
+    if create_table_query_expr == "":
+        return []
+    try:
+        from tinybird.sql_toolset import format_sql
+
+        indexes = [
+            line.strip()
+            for line in format_sql(create_table_query_expr).splitlines()
+            if line.strip().startswith("INDEX")
+        ]
+    except ModuleNotFoundError:
+        # this is not needed from CLI
+        return []
+
+    return parse_indexes_structure(indexes)

{tinybird-cli-3.3.1.dev8 → tinybird-cli-3.6.1.dev9}/tinybird/client.py

@@ -306,6 +306,7 @@ class TinyB(object):
         description: Optional[str] = None,
         ttl: Optional[str] = None,
         dry_run: bool = False,
+        indexes: Optional[str] = None,
     ):
         params = {"dry": "true" if dry_run else "false"}
         if new_schema:
@@ -314,6 +315,8 @@ class TinyB(object):
             params.update({"description": description})
         if ttl:
             params.update({"ttl": ttl})
+        if indexes:
+            params.update({"indexes": indexes})
         res = await self._req(f"/v0/datasources/{ds_name}/alter", method="POST", data=params)
 
         if "Error" in res:

{tinybird-cli-3.3.1.dev8 → tinybird-cli-3.6.1.dev9}/tinybird/context.py

@@ -6,6 +6,7 @@ table_id: ContextVar[str] = ContextVar("table_id")
 hfi_frequency: ContextVar[float] = ContextVar("hfi_frequency")
 hfi_frequency_gatherer: ContextVar[float] = ContextVar("hfi_frequency_gatherer")
 hfi_use_gatherer: ContextVar[bool] = ContextVar("hfi_use_gatherer")
+use_gatherer: ContextVar[bool] = ContextVar("use_gatherer")
 disable_template_security_validation: ContextVar[bool] = ContextVar("disable_template_security_validation")
 origin: ContextVar[str] = ContextVar("origin")
 request_id: ContextVar[str] = ContextVar("request_id")