tinybird-cli 1.0.2.dev16__tar.gz → 1.1.0__tar.gz

{tinybird-cli-1.0.2.dev16 → tinybird-cli-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: tinybird-cli
-Version: 1.0.2.dev16
+Version: 1.1.0
 Summary: Tinybird Command Line Tool
 Home-page: https://docs.tinybird.co/cli.html
 Author: Tinybird
@@ -19,6 +19,37 @@ Changelog
 
 ---------
 
+1.1.0
+************
+
+Released new version 1.1.0 with all these changes:
+- `Added` Support `skip` individual regression tests through the regression.yaml configuration file. CI/CD guide at https://www.tinybird.co/docs/guides/continuous-integration.html 
+- `Added` Fail `tb env regression-tests` if no tests are run for any pipe, use `skip` to avoid the error
+- `Added` Skip POST requests by default in `tb env regression-tests` until it's supported
+- `Added` Support append `--fixtures` to shared Data Sources in Environments
+- `Added` support `--assert-max-time` in `tb env regression-tests` so that serves as threshold for response time and avoid flakiness on fast endpoints.
+- `Added` support `tb env regression-tests --main -f regression.yaml` so you can run regression tests agains the main Environment together with a configuration file.
+- `Added` `tb env regression-tests coverage --main` runs coverage regression tests in the main Environment. This command is experimental and undocumented.
+- `Changed` `tb init --git --folder <folder>` includes the `<folder>` in the CI/CD templates as `data_project_dir` and in the paths to track changes so CI and CD only run when there's a change on that folder. If `<folder>` is not provided then it is set to the current directory `.`
+- `Changed` Force `tb init --git` to run in the `main` or `master` git branch
+- `Changed` the creation of the CD/CI workflow when running `tb init --git` to use the admin workspace token instead of the token of your admin user.
+- `Changed` Expose resources and ids when removing releases `tb release rm`
+- `Changed` `--override-commit` option to not require needing to pass the `--git` option as well. Now, running `tb init --override-commit` should work as expected.
+- `Changed` Copy and Sink on-demand jobs now return the job URL.
+- `Changed` allow Workspace admin tokens for CI/CD workflows. Previously, you were asked for user admin token. You can update your TB_ADMIN_TOKEN to be the Workspace admin token instead the admin token from a specific user. CI/CD guide at https://www.tinybird.co/docs/guides/continuous-integration.html 
+- `Changed` Improve CI/CD output. Added warning hint in custom deployments.
+- `Fixed` `tb deploy` raises an error if it detects resources renamed in git
+- `Fixed` On `tb push --force --yes` a BigQuery Data Source, it updates both the `SCHEMA` and connector settings.
+- `Fixed` Connection settings of BigQuery Data Sources are preserved in Environments, although the connection does not sync by design, to ingest data use `--fixtures`
+- `Removed` warning message that recommended not using the admin token for CI/CD workflows as we introduced a fix in the version 1.0.1 to avoid the problems that the admin token was causing.
+- `Removed` fallback when running `tb pipe regression-test` does was doing requests to `/v0/pipes/XXX/requests` as it was mostly not used. This endpoint was being called when there weren't requests in `tinybird.pipe_stats_rt` in the last 7 days. This endpoint was really heavy and most of the time wouldn't return any actual data as endpoint not being called after 7 days most likely are not in production.
+- `Removed` `async` parameter in the `/v0/environments` requests as it's not being used.
+
+
+1.0.2dev17
+************
+
+- `Changed` `tb init --git --folder <folder>` includes the `<folder>` in the CI/CD templates as `data_project_dir` and in the paths to track changes so CI and CD only run when there's a change on that folder. If `<folder>` is not provided then it is set to the current directory `.`
 
 1.0.2dev16
 ************
@@ -103,7 +134,7 @@ Changelog
 1.0.2dev1
 ************
 
-- `Changed` Improve CI/CD output
+- `Changed` Improve CI/CD output. Added warning hint in custom deployments.
 
 1.0.1
 ************

{tinybird-cli-1.0.2.dev16 → tinybird-cli-1.1.0}/tinybird/__cli__.py

@@ -4,5 +4,5 @@ __description__ = 'Tinybird Command Line Tool'
 __url__ = 'https://docs.tinybird.co/cli.html'
 __author__ = 'Tinybird'
 __author_email__ = 'support@tinybird.co'
-__version__ = '1.0.2dev16'
-__revision__ = '3152eff'
+__version__ = '1.1.0'
+__revision__ = '695ba55'

{tinybird-cli-1.0.2.dev16 → tinybird-cli-1.1.0}/tinybird/sql_toolset.py

@@ -142,10 +142,13 @@ def replacements_to_tuples(replacements: dict) -> dict:
 
 @lru_cache(maxsize=2**13)
 def replace_tables_chquery_cached(
-    sql: str, sorted_replacements: Optional[tuple] = None, default_database: str = ""
+    sql: str,
+    sorted_replacements: Optional[tuple] = None,
+    default_database: str = "",
+    output_one_line: bool = False,
 ) -> str:
     replacements = dict(sorted_replacements) if sorted_replacements else {}
-    return chquery.replace_tables(sql, replacements, default_database=default_database)
+    return chquery.replace_tables(sql, replacements, default_database=default_database, one_line=output_one_line)
 
 
 def replace_tables(
@@ -155,6 +158,7 @@ def replace_tables(
     check_functions: bool = False,
     only_replacements: bool = False,
     valid_tables: Optional[Set[Tuple[str, str]]] = None,
+    output_one_line: bool = False,
 ) -> str:
     """
     Given a query and a list of table replacements, returns the query after applying the table replacements.
@@ -163,7 +167,7 @@ def replace_tables(
     """
     if not replacements:
         # Always call replace_tables to do validation and formatting
-        return replace_tables_chquery_cached(sql, None)
+        return replace_tables_chquery_cached(sql, None, output_one_line=output_one_line)
 
     _replaced_with = set()
     _replacements = ReplacementsDict()
@@ -210,7 +214,7 @@ def replace_tables(
     deps_sorted = list(reversed(list(toposort(deps))))
 
     if not deps_sorted:
-        return replace_tables_chquery_cached(sql, None)
+        return replace_tables_chquery_cached(sql, None, output_one_line=output_one_line)
 
     for current_deps in deps_sorted:
         current_replacements = {}
@@ -232,9 +236,11 @@ def replace_tables(
         if current_replacements:
             # We need to transform the dictionary into something cacheable, so a sorted tuple of tuples it is
             r = tuple(sorted([(k, v) for k, v in current_replacements.items()]))
-            sql = replace_tables_chquery_cached(sql, r, default_database=default_database)
+            sql = replace_tables_chquery_cached(
+                sql, r, default_database=default_database, output_one_line=output_one_line
+            )
         else:
-            sql = replace_tables_chquery_cached(sql, None)
+            sql = replace_tables_chquery_cached(sql, None, output_one_line=output_one_line)
 
     return sql
 

{tinybird-cli-1.0.2.dev16 → tinybird-cli-1.1.0}/tinybird/tb_cli_modules/cicd.py

@@ -28,9 +28,12 @@ GITHUB_CI_YML = """
       workflow_dispatch:
       pull_request:
         branches:
-         - main
-         - master
+          - main
+          - master
         types: [opened, reopened, labeled, unlabeled, synchronize, closed]
+        paths:
+          - '$data_project_dir/**/*'
+          - '$data_project_dir/*'
 
     concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }}
 
@@ -38,7 +41,7 @@ GITHUB_CI_YML = """
         ci: # ci using environments from workspace '$workspace_name'
           uses: tinybirdco/ci/.github/workflows/ci.yml@v2.3.0
           with:
-            data_project_dir: .
+            data_project_dir: $data_project_dir
           secrets:
             tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN }}  # set the Workspace admin token in GitHub secrets
             tb_host: $tb_host
@@ -58,12 +61,15 @@ GITHUB_CD_YML = """
         branches:
           - main
           - master
+        paths:
+          - '$data_project_dir/**/*'
+          - '$data_project_dir/*'
     jobs:
       cd:  # deploy changes to workspace '$workspace_name'
         uses: tinybirdco/ci/.github/workflows/cd.yml@v2.3.0
         with:
           tb_deploy: false
-          data_project_dir: .
+          data_project_dir: $data_project_dir
         secrets:
           tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN }}  # set the Workspace admin token in GitHub secrets
           tb_host: $tb_host
@@ -81,18 +87,27 @@ GITLAB_YML = """
     .ci_config_rules:
       - &ci_config_rule
         if: $CI_PIPELINE_SOURCE == "merge_request_event"
+        changes:
+          - $data_project_dir/*
+          - $data_project_dir/**/*
       - &ci_cleanup_rule
         if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
+        changes:
+          - $data_project_dir/*
+          - $data_project_dir/**/*
 
     .cd_config_rules:
       - &cd_config_rule
         if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
+        changes:
+          - $data_project_dir/*
+          - $data_project_dir/**/*
 
     .cicd_variables:
       variables: &cicd_variables
         TB_HOST: "$tb_host"
         TB_ADMIN_TOKEN: $TB_ADMIN_TOKEN  # set the Workspace admin token in GitLab CI/CD Variables
-        DATA_PROJECT_DIR: "."
+        DATA_PROJECT_DIR: "$data_project_dir"
         TB_DEPLOY: "false"
 
     run_ci:  # ci using environments from workspace '$workspace_name'
@@ -217,12 +232,18 @@ def ask_provider_interactively():
             provider_index = -1
 
 
-async def init_cicd(client: TinyB, workspace: Dict[str, Any], config: Dict[str, Any], path: Optional[str] = None):
+async def init_cicd(
+    client: TinyB,
+    workspace: Dict[str, Any],
+    config: Dict[str, Any],
+    path: Optional[str] = None,
+    data_project_dir: Optional[str] = None,
+):
     provider = ask_provider_interactively()
     if provider:
         path = path if path else getcwd()
+        data_project_dir = data_project_dir if data_project_dir else "."
         generator = CICDGeneratorBase.build_generator(provider)
-
         workspace_info = await client.workspace_info()
         token = await client.get_token_by_name("admin token")
         params = {
@@ -230,6 +251,7 @@ async def init_cicd(client: TinyB, workspace: Dict[str, Any], config: Dict[str,
             "workspace_name": workspace_info["name"],
             "token_name": token["name"],
             "token_id": token["id"],
+            "data_project_dir": data_project_dir,
         }
         warning_message = generator(path, params)
         if warning_message:

{tinybird-cli-1.0.2.dev16 → tinybird-cli-1.1.0}/tinybird/tb_cli_modules/cli.py

@@ -415,7 +415,10 @@ async def init(
         click.echo(FeedbackManager.info_cicd_already_exists(provider=cicd_provider.name))
     else:
         if click.confirm(FeedbackManager.prompt_init_cicd(), default=True):
-            await init_cicd(client, current_ws, config, path=cli_git_release.working_dir())
+            data_project_dir = os.path.relpath(folder, cli_git_release.working_dir())
+            await init_cicd(
+                client, current_ws, config, path=cli_git_release.working_dir(), data_project_dir=data_project_dir
+            )
 
     click.echo(final_response)
 

{tinybird-cli-1.0.2.dev16 → tinybird-cli-1.1.0}/tinybird_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: tinybird-cli
-Version: 1.0.2.dev16
+Version: 1.1.0
 Summary: Tinybird Command Line Tool
 Home-page: https://docs.tinybird.co/cli.html
 Author: Tinybird
@@ -19,6 +19,37 @@ Changelog
 
 ---------
 
+1.1.0
+************
+
+Released new version 1.1.0 with all these changes:
+- `Added` Support `skip` individual regression tests through the regression.yaml configuration file. CI/CD guide at https://www.tinybird.co/docs/guides/continuous-integration.html 
+- `Added` Fail `tb env regression-tests` if no tests are run for any pipe, use `skip` to avoid the error
+- `Added` Skip POST requests by default in `tb env regression-tests` until it's supported
+- `Added` Support append `--fixtures` to shared Data Sources in Environments
+- `Added` support `--assert-max-time` in `tb env regression-tests` so that serves as threshold for response time and avoid flakiness on fast endpoints.
+- `Added` support `tb env regression-tests --main -f regression.yaml` so you can run regression tests agains the main Environment together with a configuration file.
+- `Added` `tb env regression-tests coverage --main` runs coverage regression tests in the main Environment. This command is experimental and undocumented.
+- `Changed` `tb init --git --folder <folder>` includes the `<folder>` in the CI/CD templates as `data_project_dir` and in the paths to track changes so CI and CD only run when there's a change on that folder. If `<folder>` is not provided then it is set to the current directory `.`
+- `Changed` Force `tb init --git` to run in the `main` or `master` git branch
+- `Changed` the creation of the CD/CI workflow when running `tb init --git` to use the admin workspace token instead of the token of your admin user.
+- `Changed` Expose resources and ids when removing releases `tb release rm`
+- `Changed` `--override-commit` option to not require needing to pass the `--git` option as well. Now, running `tb init --override-commit` should work as expected.
+- `Changed` Copy and Sink on-demand jobs now return the job URL.
+- `Changed` allow Workspace admin tokens for CI/CD workflows. Previously, you were asked for user admin token. You can update your TB_ADMIN_TOKEN to be the Workspace admin token instead the admin token from a specific user. CI/CD guide at https://www.tinybird.co/docs/guides/continuous-integration.html 
+- `Changed` Improve CI/CD output. Added warning hint in custom deployments.
+- `Fixed` `tb deploy` raises an error if it detects resources renamed in git
+- `Fixed` On `tb push --force --yes` a BigQuery Data Source, it updates both the `SCHEMA` and connector settings.
+- `Fixed` Connection settings of BigQuery Data Sources are preserved in Environments, although the connection does not sync by design, to ingest data use `--fixtures`
+- `Removed` warning message that recommended not using the admin token for CI/CD workflows as we introduced a fix in the version 1.0.1 to avoid the problems that the admin token was causing.
+- `Removed` fallback when running `tb pipe regression-test` does was doing requests to `/v0/pipes/XXX/requests` as it was mostly not used. This endpoint was being called when there weren't requests in `tinybird.pipe_stats_rt` in the last 7 days. This endpoint was really heavy and most of the time wouldn't return any actual data as endpoint not being called after 7 days most likely are not in production.
+- `Removed` `async` parameter in the `/v0/environments` requests as it's not being used.
+
+
+1.0.2dev17
+************
+
+- `Changed` `tb init --git --folder <folder>` includes the `<folder>` in the CI/CD templates as `data_project_dir` and in the paths to track changes so CI and CD only run when there's a change on that folder. If `<folder>` is not provided then it is set to the current directory `.`
 
 1.0.2dev16
 ************
@@ -103,7 +134,7 @@ Changelog
 1.0.2dev1
 ************
 
-- `Changed` Improve CI/CD output
+- `Changed` Improve CI/CD output. Added warning hint in custom deployments.
 
 1.0.1
 ************