tinybird-cli 3.2.0__tar.gz → 3.6.1.dev9__tar.gz

{tinybird-cli-3.2.0 → tinybird-cli-3.6.1.dev9}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.1
 Name: tinybird-cli
-Version: 3.2.0
+Version: 3.6.1.dev9
 Summary: Tinybird Command Line Tool
-Home-page: https://docs.tinybird.co/cli.html
+Home-page: https://www.tinybird.co/docs/cli/introduction.html
 Author: Tinybird
 Author-email: support@tinybird.co
 Requires-Python: >=3.8, <3.12
@@ -11,7 +11,7 @@ Provides-Extra: bigquery
 Provides-Extra: snowflake
 
 Tinybird CLI
-============
+=============
 
 The Tinybird command-line tool allows you to use all the Tinybird functionality directly from the command line. Additionally, it includes several functions to create and manage data projects easily.
 
@@ -19,16 +19,115 @@ Changelog
 
 ---------
 
+3.6.1.dev9
+************
+
+- `Fixed` Added correct URL for checking available regions
+
+
+3.6.1.dev8
+************
+
+- `Fixed` Report better error message on `tb diff` when two Datafiles have the same name
+
+3.6.1.dev7
+************
+
+- `Fixed` `tb deploy` supports deployment when a `.incl` file is removed
+
+3.6.1.dev6
+************
+
+- `Fixed` Some issues when using `--fmt: off` with `CASE` in the sql
+
+3.6.1.dev5
+************
+
+- `Fixed` Avoid `tb fmt` to error if there's a `CASE` in the sql
+
+3.6.1.dev4
+************
+
+- `Fixed` regression tests query when filtering by specific parameter
+
+3.6.1.dev3
+************
+
+- `Changed` behavior when running `tb deploy` on a branch to push the connection settings to the backend. This change is the backend that decides what to do. 
+
+3.6.1.dev2
+************
+
+- `Changed` fixed major version of tinybird-cli to lower than 4 when using `tb init --git`
+
+3.6.0
+************
+
+- `Added` Add new envvar `TB_INCL_RELATIVE_PATH` in .tinyenv to look for Datafiles out of the Data Project directory
+- `Changed` On-demand Sink run checks the pipe type instead of checking the workspace connectors
+
+3.5.0
+************
+
+- `Added` Token management in Copy pipes datafiles
+- `Added` Better error report on `tb push` when trying to push a pipe with bad templating
+- `Added` credential validation for S3 connections
+- `Added` `tb connection create` now supports `s3_iamrole` as service type
+- `Changed` error message when trying to modify a datasource without bumping the major semver version to add a link to the documentation.
+- `Changed` GH workflow files and secret includes the workspace name when generated from the UI and CLI
+- `Changed` rename `INDICES` to `INDEXES`
+- `Changed` Save git commit to workspace on `tb push --only-changes`
+- `Fixed` Fix error message when deleting an `.incl` file with the git workflow
+- `Fixed` support `tb deploy --dry-run`
+
+
+3.4.0
+************
+
+- `Added` Support `tb deploy` .datasource with `INDEXES`
+- `Added` Support `tb push` .datasource with `INDEXES`
+- `Added` Support `tb deploy` in a post release of a .datasource with `INDEXES`
+- `Added` Check backfill on preview is required when `tb deploy` modified datasources. Disable check with `TB_CHECK_BACKFILL_REQUIRED=0`.
+- `Changed` Allow spaces when managing multiple Workspace members
+- `Changed` `tb deploy` to not allow pushing materialized views with datasource definition in the same datafile. We are blocking this as has unsupported behavior and it misleading and less flexible than having 2 files one for the datasource and one for the materialized view.
+- `Changed` `tb deploy` to deploy the datasources from top to bottom to make sure that if we modified the column in some datasource in the top. It will be deployed before the datasources that depend on it.
+- `Changed` Post-release required when deploying changes that include changing a Copy Pipe.
+- `Changed` script `exec_test.sh`  from `tb init --git` to run retries to avoid flakiness in tests
+- `Changed` Update `tb release rm --oldest-rollback` to use new API
+- `Fixed` Post-release recommended instead of required when deploying changes that include changing a Copy Pipe. Displays a warning when other number of the semver is increased.
+- `Fixed` Fixed bug when a Data Source was created using an INDEX
+- `Fixed` Avoid None git diff change type
+- `Fixed` `tb deploy` to deploy the endpoints in the correct order when there are endpoints depending on other endpoints.
+- `Fixed` `tb push` pipe with multiple tokens
+- `Fixed` `IMPORT` and `ENGINE` changes were not detected `tb deploy`
+- `Removed` part of the code supporting python 3.6 as we stopped giving support 18 month ago
+
+
+3.3.0
+************
+
+- `Added` Show warning when doing data or pipe operations directly against Live.
+- `Added` `bi_stats`, `bi_stats_rt` and `endpoint_errors` to the list of service data sources that the CLI ignore when looking for dependencies of pipes being pushed.
+- `Added` S3 Iam Role connection creation
+- `Added` Connector credential validation
+- `Added` Data quality tests inside a `skip_in_branch` folder inside the `tests` folder are skipped when running on a branch. Use this to skip tests in the CI piipeline that should only run with production data.
+- `Changed` Stop creating endpoints folder on tb init
+- `Changed` Small refactoring of the connector credentials
+- `Fixed` Progress bar in `tb init --git`.
+- `Fixed` Don't do diff when there are no modified resources on `tb deploy`.
+- `Fixed` Upgrade GitPython dependency to 3.1.32. That version fixes the security vulnerability `CVE-2022-24439 <https://github.com/advisories/GHSA-hcpj-qp55-gfph>`
+- `Fixed` Homepage URL in PyPI
+
 3.2.0
 ************
 
 - `Changed` `tb init --git` uses v3.1.0 of the CI templates (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
-- `Changed` Allow alter datasource with jsonpaths to add DEFAULTS if the datasource already have any (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
+- `Changed` Allow alter data source with JSON paths to add DEFAULTS if the data source already has any (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
 
 3.1.0
 ************
 
-- `Added` `tb init --cicd` to generate CI/CD files not only when using `--git`. Combine with `--force` for overwrite
+- `Added` `tb init --cicd` to generate CI/CD files not only when using `--git`. Combine with `--force` to overwrite
 - `Added` Ignore format changes on `tb deploy` by default
 - `Changed` `tb init --git` and `tb release generate` include `set -euxo pipefail` in the generated bash scripts
 - `Changed` `tb init --git` uses github.com/tinybirdco/ci => v3.0.0
@@ -36,7 +135,7 @@ Changelog
 - `Changed` `tb release promote` automatically deletes de oldest rollback Release if `export TB_FORCE_REMOVE_OLDEST_ROLLBACK=1`
 - `Changed` option `--node` in `tb pipe populate` to required as it is already mandatory to specify the node to populate. Now, it will provide an error message if the `--node` option is not provided.
 - `Changed` Better error report on `tb fmt --diff`. Use it in CI to standardize Datafile formatting
-- `Changed` `tb init --git` generates a requirements.txt file to install the latest tinybird-cli release from pypi
+- `Changed` `tb init --git` generates a requirements.txt file to install the latest tinybird-cli release from PyPI
 - `Changed` `tb deploy` shows an example when failing because of a missing `--semver` argument
 - `Fixed` When parsing the response of a CSV `tb sql` command, the CLI was interpreting the result as an error in some cases.
 
@@ -45,7 +144,7 @@ Changelog
 
 Major version updated to 3.0.0 due to the change in the logic of `tb deploy`. From this version on, `tb deploy` promotes Releases to Live or Preview based on the semver version.
 
-See `our Version Control guides <https://versions.tinybird.co/docs/version-control/deployment-strategies.html#semver-deployment-behaviour>`_ for more details about our semver deployment behaviour.
+See `our Version Control guides <https://www.tinybird.co/docs/version-control/deployment-strategies.html#semver-deployment-behaviour>`_ for more details about our semver deployment behaviour.
 
 - `Added` `syncing` status to git remote settings
 - `Added` `tb release rm --oldest-rollback --force --yes` deletes the oldest rollback Release by creation date

{tinybird-cli-3.2.0 → tinybird-cli-3.6.1.dev9}/tinybird/__cli__.py

@@ -1,8 +1,8 @@
 
 __name__ = 'tinybird-cli'
 __description__ = 'Tinybird Command Line Tool'
-__url__ = 'https://docs.tinybird.co/cli.html'
+__url__ = 'https://www.tinybird.co/docs/cli/introduction.html'
 __author__ = 'Tinybird'
 __author_email__ = 'support@tinybird.co'
-__version__ = '3.2.0'
-__revision__ = '3067b41'
+__version__ = '3.6.1.dev9'
+__revision__ = 'ff22a5b'

{tinybird-cli-3.2.0 → tinybird-cli-3.6.1.dev9}/tinybird/ch_utils/engine.py

@@ -1,9 +1,10 @@
 import logging
 import re
 from collections import defaultdict
+from dataclasses import asdict
 from typing import Any, Callable, Dict, Iterable, List, Optional
 
-from ..sql import col_name, engine_replicated_to_local, parse_table_structure
+from ..sql import TableIndex, col_name, engine_replicated_to_local, parse_indexes_structure, parse_table_structure
 
 DEFAULT_EMPTY_PARAMETERS = ["ttl", "partition_key", "sorting_key"]
 DEFAULT_JOIN_EMPTY_PARAMETERS = ["join_strictness", "join_type", "key_columns"]
@@ -31,7 +32,20 @@ class TableDetails:
     >>> ed.to_datafile()
     ''
 
+    >>> ed = TableDetails({ "engine_full": "MergeTree() PARTITION BY toYear(timestamp) ORDER BY (timestamp, cityHash64(location)) SAMPLE BY cityHash64(location) SETTINGS index_granularity = 32, index_granularity_bytes = 2048", "engine": "MergeTree", "partition_key": "toYear(timestamp)", "sorting_key": "timestamp, cityHash64(location)", "primary_key": "timestamp, cityHash64(location)", "sampling_key": "cityHash64(location)", "settings": "index_granularity = 32, index_granularity_bytes = 2048", "ttl": None })
+    >>> ed.diff_ttl("toDate(timestamp) + toIntervalDay(1)")
+    True
     >>> ed = TableDetails({ "engine_full": "MergeTree() PARTITION BY toYear(timestamp) ORDER BY (timestamp, cityHash64(location)) SAMPLE BY cityHash64(location) SETTINGS index_granularity = 32, index_granularity_bytes = 2048 TTL toDate(timestamp) + INTERVAL 1 DAY", "engine": "MergeTree", "partition_key": "toYear(timestamp)", "sorting_key": "timestamp, cityHash64(location)", "primary_key": "timestamp, cityHash64(location)", "sampling_key": "cityHash64(location)", "settings": "index_granularity = 32, index_granularity_bytes = 2048", "ttl": "toDate(timestamp) + INTERVAL 1 DAY" })
+    >>> ed.diff_ttl("toDate(timestamp) + toIntervalDay(1)")
+    False
+    >>> ed.diff_ttl("toDate(timestamp) + toIntervalDay(2)")
+    True
+    >>> ed.diff_ttl("toDate(timestamp) + INTERVAL DAY 2")
+    True
+    >>> ed.diff_ttl("toDate(timestamp) + INTERVAL 1 DAY")
+    False
+    >>> ed.diff_ttl("")
+    True
     >>> ed.engine_full
     'MergeTree() PARTITION BY toYear(timestamp) ORDER BY (timestamp, cityHash64(location)) SAMPLE BY cityHash64(location) SETTINGS index_granularity = 32, index_granularity_bytes = 2048 TTL toDate(timestamp) + INTERVAL 1 DAY'
     >>> ed.engine
@@ -131,6 +145,16 @@ class TableDetails:
             return is_aggregating or is_replacing or is_collapsing
         return False
 
+    def diff_ttl(self, new_ttl: str) -> bool:
+        try:
+            from tinybird.sql_toolset import format_sql
+
+            current_ttl = format_sql(f"select {self.ttl}")[7:]
+            new_ttl = format_sql(f"select {new_ttl}")[7:]
+            return current_ttl != new_ttl
+        except Exception:
+            return self.ttl != new_ttl
+
     @property
     def partition_key(self) -> Optional[str]:
         return self.details.get("partition_key", None)
@@ -204,6 +228,10 @@ class TableDetails:
             "row_count": self.details.get("total_rows", None),
         }
 
+    @property
+    def indexes(self) -> List[TableIndex]:
+        return _parse_indexes(str(self.details.get("create_table_query", "")))
+
     def to_json(self, exclude: Optional[List[str]] = None, include_empty_details: bool = False):
         # name, database are not exported since they are not part of the engine
         d: Dict[str, Any] = {
@@ -234,6 +262,8 @@ class TableDetails:
             d["version"] = self.version
         if self.ttl:
             d["ttl"] = self.ttl.strip()
+        if self.indexes:
+            d["indexes"] = [asdict(index) for index in self.indexes]
 
         if self.engine_full:
             engine_params = engine_params_from_engine_full(self.engine_full)
@@ -253,7 +283,9 @@ class TableDetails:
         return d
 
     def to_datafile(self, include_empty_details: bool = False) -> str:
-        d: Dict[str, Any] = self.to_json(exclude=["engine", "engine_full"], include_empty_details=include_empty_details)
+        d: Dict[str, Any] = self.to_json(
+            exclude=["engine", "engine_full", "indexes"], include_empty_details=include_empty_details
+        )
         engine: str = self.engine
 
         datafile: str = ""
@@ -774,3 +806,21 @@ def ttl_condition_from_engine_full(engine_full: Optional[str]) -> Optional[str]:
     except Exception as e:
         logging.error(str(e))
         return None
+
+
+def _parse_indexes(create_table_query_expr: str) -> List[TableIndex]:
+    if create_table_query_expr == "":
+        return []
+    try:
+        from tinybird.sql_toolset import format_sql
+
+        indexes = [
+            line.strip()
+            for line in format_sql(create_table_query_expr).splitlines()
+            if line.strip().startswith("INDEX")
+        ]
+    except ModuleNotFoundError:
+        # this is not needed from CLI
+        return []
+
+    return parse_indexes_structure(indexes)

{tinybird-cli-3.2.0 → tinybird-cli-3.6.1.dev9}/tinybird/client.py

@@ -306,6 +306,7 @@ class TinyB(object):
         description: Optional[str] = None,
         ttl: Optional[str] = None,
         dry_run: bool = False,
+        indexes: Optional[str] = None,
     ):
         params = {"dry": "true" if dry_run else "false"}
         if new_schema:
@@ -314,6 +315,8 @@ class TinyB(object):
             params.update({"description": description})
         if ttl:
             params.update({"ttl": ttl})
+        if indexes:
+            params.update({"indexes": indexes})
         res = await self._req(f"/v0/datasources/{ds_name}/alter", method="POST", data=params)
 
         if "Error" in res:
@@ -991,6 +994,12 @@ class TinyB(object):
         response = await self._req(f"/v0/connectors/snowflake/warehouses?{urlencode(params)}", method="POST", data="")
         return response["warehouses"]
 
+    async def get_s3_trust_policy(self) -> Dict[str, Any]:
+        return await self._req("/v0/integrations/s3/policies/trust-policy")
+
+    async def get_s3_access_policy(self) -> Dict[str, Any]:
+        return await self._req("/v0/integrations/s3/policies/access-policy")
+
     async def sql_get_format(self, sql: str, with_clickhouse_format: bool = False) -> str:
         try:
             if with_clickhouse_format:
@@ -1099,6 +1108,12 @@ class TinyB(object):
             params["confirmation"] = confirmation
         return await self._req(f"/v0/workspaces/{workspace_id}/releases/{semver}?{urlencode(params)}", method="DELETE")
 
+    async def release_oldest_rollback(
+        self,
+        workspace_id: str,
+    ):
+        return await self._req(f"/v0/workspaces/{workspace_id}/releases/oldest-rollback", method="GET")
+
     async def token_list(self, match: Optional[str] = None):
         tokens = await self.tokens()
         return [token for token in tokens if (not match or token["name"].find(match) != -1) and "token" in token]

{tinybird-cli-3.2.0 → tinybird-cli-3.6.1.dev9}/tinybird/config.py

@@ -20,6 +20,7 @@ VERSION = f"{__cli__.__version__} (rev {__revision__})"
 DEFAULT_UI_HOST = "https://ui.tinybird.co"
 SUPPORTED_CONNECTORS = ["bigquery", "snowflake"]
 PROJECT_PATHS = ["datasources", "datasources/fixtures", "endpoints", "pipes", "tests", "scripts", "deploy"]
+DEPRECATED_PROJECT_PATHS = ["endpoints"]
 MIN_WORKSPACE_ID_LENGTH = 36
 
 

{tinybird-cli-3.2.0 → tinybird-cli-3.6.1.dev9}/tinybird/connector_settings.py

@@ -1,10 +1,19 @@
 from __future__ import annotations
 
 from enum import Enum
-from typing import List, Optional
+from typing import List, Optional, Union
 
 from pydantic import BaseModel
 
+from tinybird.data_connectors.credentials import (
+    GCSConnectorCredentials,
+    GCSServiceAccountCredentials,
+    IAMRoleAWSCredentials,
+    S3ConnectorCredentials,
+)
+
+GCS_BASE_URL = "https://storage.googleapis.com"
+
 
 class DataConnectorType(str, Enum):
     KAFKA = "kafka"
@@ -13,6 +22,7 @@ class DataConnectorType(str, Enum):
     BIGQUERY = "bigquery"
     GCLOUD_STORAGE = "gcs"
     GCLOUD_STORAGE_HMAC = "gcs_hmac"
+    GCLOUD_STORAGE_SA = "gcs_service_account"
     AMAZON_S3 = "s3"
     AMAZON_S3_IAMROLE = "s3_iamrole"
 
@@ -24,7 +34,11 @@ DataConnectors = DataConnectorType  # this is just to make picking happy
 
 
 class DataConnectorSetting(BaseModel):
-    ...
+    def get_region(self) -> str:
+        return "unknown"
+
+    def get_provider_name(self) -> str:
+        return "unknown"
 
 
 class KafkaConnectorSetting(DataConnectorSetting):
@@ -42,12 +56,42 @@ class S3ConnectorSetting(DataConnectorSetting):
     s3_access_key_id: str
     s3_secret_access_key: str
     s3_region: str
+    endpoint_url: Optional[str] = None
+
+    @property
+    def credentials(self) -> S3ConnectorCredentials:
+        return S3ConnectorCredentials(
+            access_key_id=self.s3_access_key_id,
+            secret_access_key=self.s3_secret_access_key,
+            region=self.s3_region,
+        )
+
+    def get_provider_name(self) -> str:
+        return DataConnectorType.AMAZON_S3
+
+    def get_region(self) -> str:
+        return self.s3_region
 
 
 class S3IAMConnectorSetting(DataConnectorSetting):
     s3_iamrole_arn: str
     s3_iamrole_region: str
-    s3_iamrole_external_id: Optional[str] = None
+    s3_iamrole_external_id: str
+    endpoint_url: Optional[str] = None
+
+    @property
+    def credentials(self) -> IAMRoleAWSCredentials:
+        return IAMRoleAWSCredentials(
+            role_arn=self.s3_iamrole_arn,
+            external_id=self.s3_iamrole_external_id,
+            region=self.s3_iamrole_region,
+        )
+
+    def get_provider_name(self) -> str:
+        return DataConnectorType.AMAZON_S3
+
+    def get_region(self) -> str:
+        return self.s3_iamrole_region
 
 
 class SnowflakeConnectorSetting(DataConnectorSetting):
@@ -69,9 +113,32 @@ class BigQueryConnectorSetting(DataConnectorSetting):
     account: Optional[str] = None
 
 
+class GCSServiceAccountConnectorSetting(DataConnectorSetting):
+    account_email: str
+    endpoint_url: str = GCS_BASE_URL
+
+    @property
+    def credentials(self) -> GCSServiceAccountCredentials:
+        return GCSServiceAccountCredentials(account_email=self.account_email)
+
+    def get_provider_name(self) -> str:
+        return DataConnectorType.GCLOUD_STORAGE
+
+
 class GCSHmacConnectorSetting(DataConnectorSetting):
     gcs_hmac_access_id: str
     gcs_hmac_secret: str
+    endpoint_url: str = GCS_BASE_URL
+
+    @property
+    def credentials(self) -> GCSConnectorCredentials:
+        return GCSConnectorCredentials(
+            access_key_id=self.gcs_hmac_access_id,
+            secret_access_key=self.gcs_hmac_secret,
+        )
+
+    def get_provider_name(self) -> str:
+        return DataConnectorType.GCLOUD_STORAGE
 
 
 class GCSConnectorSetting(DataConnectorSetting):
@@ -90,10 +157,15 @@ DATA_CONNECTOR_SETTINGS: dict[DataConnectors, type[DataConnectorSetting]] = {
     DataConnectors.BIGQUERY: BigQueryConnectorSetting,
     DataConnectors.GCLOUD_STORAGE: GCSConnectorSetting,
     DataConnectors.GCLOUD_STORAGE_HMAC: GCSHmacConnectorSetting,
+    DataConnectors.GCLOUD_STORAGE_SA: GCSServiceAccountConnectorSetting,
     DataConnectors.AMAZON_S3: S3ConnectorSetting,
     DataConnectors.AMAZON_S3_IAMROLE: S3IAMConnectorSetting,
 }
 
+SinkConnectorSettings = Union[
+    S3ConnectorSetting, S3IAMConnectorSetting, GCSHmacConnectorSetting, GCSServiceAccountConnectorSetting
+]
+
 
 class DataLinkerSettings:
     kafka = [
@@ -137,6 +209,13 @@ class DataSinkSettings:
         "format",
         "compression",
     ]
+    gcs_service_account = [
+        "bucket_path",
+        "file_template",
+        "partition_node",
+        "format",
+        "compression",
+    ]
 
 
 class DataSensitiveSettings:
@@ -151,3 +230,10 @@ class DataSensitiveSettings:
     gcs_hmac: List[str] = ["gcs_hmac_secret"]
     s3: List[str] = ["s3_secret_access_key"]
     s3_iamrole: List[str] = ["s3_iamrole_arn"]
+
+
+CLI_EXCLUDED_SETTINGS = {
+    "gcs_hmac": ["endpoint_url"],
+    "s3": ["endpoint_url"],
+    "s3_iamrole": ["endpoint_url"],
+}

{tinybird-cli-3.2.0 → tinybird-cli-3.6.1.dev9}/tinybird/context.py

@@ -6,6 +6,7 @@ table_id: ContextVar[str] = ContextVar("table_id")
 hfi_frequency: ContextVar[float] = ContextVar("hfi_frequency")
 hfi_frequency_gatherer: ContextVar[float] = ContextVar("hfi_frequency_gatherer")
 hfi_use_gatherer: ContextVar[bool] = ContextVar("hfi_use_gatherer")
+use_gatherer: ContextVar[bool] = ContextVar("use_gatherer")
 disable_template_security_validation: ContextVar[bool] = ContextVar("disable_template_security_validation")
 origin: ContextVar[str] = ContextVar("origin")
 request_id: ContextVar[str] = ContextVar("request_id")

tinybird-cli-3.6.1.dev9/tinybird/data_connectors/credentials.py

@@ -0,0 +1,31 @@
+import abc
+import dataclasses
+
+
+class ConnectorCredentials(abc.ABC):
+    pass
+
+
+@dataclasses.dataclass(frozen=True)
+class S3ConnectorCredentials(ConnectorCredentials):
+    access_key_id: str
+    secret_access_key: str
+    region: str
+
+
+@dataclasses.dataclass(frozen=True)
+class IAMRoleAWSCredentials(ConnectorCredentials):
+    role_arn: str
+    external_id: str
+    region: str
+
+
+@dataclasses.dataclass(frozen=True)
+class GCSConnectorCredentials(ConnectorCredentials):
+    access_key_id: str
+    secret_access_key: str
+
+
+@dataclasses.dataclass(frozen=True)
+class GCSServiceAccountCredentials(ConnectorCredentials):
+    account_email: str