tinybird-cli 3.3.1.dev5__tar.gz → 3.3.1.dev6__tar.gz

{tinybird-cli-3.3.1.dev5 → tinybird-cli-3.3.1.dev6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: tinybird-cli
-Version: 3.3.1.dev5
+Version: 3.3.1.dev6
 Summary: Tinybird Command Line Tool
 Home-page: https://docs.tinybird.co/cli.html
 Author: Tinybird
@@ -19,13 +19,18 @@ Changelog
 
 ---------
 
+3.3.1.dev6
+************
+
+- `Fixed` `tb deploy` to deploy the endpoints in the correct order when there are endpoints depending on other endpoints.
+
 3.3.1.dev5
-***********
+************
 
 - `Fixed` `tb push` pipe with multiple tokens
 
 3.3.1.dev4
-***********
+************
 
 - `Changed` script `exec_test.sh`  from `tb init --git` to run retries to avoid flakiness in tests
 
@@ -35,7 +40,7 @@ Changelog
 - `Fixed` `IMPORT` and `ENGINE` changes were not detected `tb deploy`
 
 3.3.1.dev2
-***********
+************
 
 - `Changed` Update `tb release rm --oldest-rollback` to use new API
 
@@ -54,7 +59,7 @@ Changelog
 - `Changed` Stop creating endpoints folder on tb init
 - `Changed` Small refactoring of the connector credentials
 - `Fixed` Progress bar in `tb init --git`.
-- `Fixed` Don't do diff when there is no modified resources on `tb deploy`.
+- `Fixed` Don't do diff when there are no modified resources on `tb deploy`.
 - `Fixed` Upgrade GitPython dependency to 3.1.32. That version fixes the security vulnerability `CVE-2022-24439 <https://github.com/advisories/GHSA-hcpj-qp55-gfph>`
 - `Fixed` Homepage URL in PyPI
 
@@ -62,12 +67,12 @@ Changelog
 ************
 
 - `Changed` `tb init --git` uses v3.1.0 of the CI templates (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
-- `Changed` Allow alter datasource with jsonpaths to add DEFAULTS if the datasource already have any (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
+- `Changed` Allow alter data source with JSON paths to add DEFAULTS if the data source already has any (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
 
 3.1.0
 ************
 
-- `Added` `tb init --cicd` to generate CI/CD files not only when using `--git`. Combine with `--force` for overwrite
+- `Added` `tb init --cicd` to generate CI/CD files not only when using `--git`. Combine with `--force` to overwrite
 - `Added` Ignore format changes on `tb deploy` by default
 - `Changed` `tb init --git` and `tb release generate` include `set -euxo pipefail` in the generated bash scripts
 - `Changed` `tb init --git` uses github.com/tinybirdco/ci => v3.0.0
@@ -75,7 +80,7 @@ Changelog
 - `Changed` `tb release promote` automatically deletes de oldest rollback Release if `export TB_FORCE_REMOVE_OLDEST_ROLLBACK=1`
 - `Changed` option `--node` in `tb pipe populate` to required as it is already mandatory to specify the node to populate. Now, it will provide an error message if the `--node` option is not provided.
 - `Changed` Better error report on `tb fmt --diff`. Use it in CI to standardize Datafile formatting
-- `Changed` `tb init --git` generates a requirements.txt file to install the latest tinybird-cli release from pypi
+- `Changed` `tb init --git` generates a requirements.txt file to install the latest tinybird-cli release from PyPI
 - `Changed` `tb deploy` shows an example when failing because of a missing `--semver` argument
 - `Fixed` When parsing the response of a CSV `tb sql` command, the CLI was interpreting the result as an error in some cases.
 

{tinybird-cli-3.3.1.dev5 → tinybird-cli-3.3.1.dev6}/tinybird/__cli__.py

@@ -4,5 +4,5 @@ __description__ = 'Tinybird Command Line Tool'
 __url__ = 'https://docs.tinybird.co/cli.html'
 __author__ = 'Tinybird'
 __author_email__ = 'support@tinybird.co'
-__version__ = '3.3.1.dev5'
-__revision__ = '03759c5'
+__version__ = '3.3.1.dev6'
+__revision__ = '133b9e9'

{tinybird-cli-3.3.1.dev5 → tinybird-cli-3.3.1.dev6}/tinybird/datafile.py

@@ -3952,92 +3952,30 @@ async def folder_push(
 
     async def push_files(dependency_graph: GraphDependencies, dry_run: bool = False):
         endpoints_dep_map = dict()
-        pipes_dep_map = dict()
-        mat_dep_map = dict()
         processed = set()
 
         dependencies_graph = dependency_graph.dep_map
         resources_to_run = dependency_graph.to_run
 
-        dependencies_graph_fork_downstream: Dict[str, Set[str]] = {}
-        resources_to_run_fork_downstream: Dict[str, Dict[str, Any]] = {}
-
-        if fork_downstream:
-            dependencies_graph_fork_downstream, resources_to_run_fork_downstream = generate_forkdownstream_graph(
-                dependency_graph.all_dep_map,
-                dependency_graph.all_resources,
-                dependency_graph.to_run,
-                list(dependency_graph.dep_map.keys()),
-            )
-        else:
-            # TODO: Split this into fork_downstream and normal push
-            dependencies_graph_fork_downstream = dependencies_graph
-            resources_to_run_fork_downstream = resources_to_run
-
-        # For the new resources using forkdownstream or tb push, we will push them following
-        # First, datasources. Then, endpoints depending of these datasources. Finally, the pipes
-        groups = [group for group in toposort(dependencies_graph)]
-        for group in groups:
-            for name in group:
-                if (
-                    not is_new(name, changed, dependencies_graph, dependencies_graph_fork_downstream)
-                    or not fork_downstream
-                ):
-                    continue
-                await push(
-                    name,
-                    resources_to_run,
-                    resource_versions,
-                    latest_datasource_versions,
-                    dry_run,
-                    fork_downstream,
-                    fork,
-                )
-                processed.add(name)
-
-        groups = [group for group in toposort(dependencies_graph_fork_downstream)]
-        for group in groups:
-            for name in group:
-                if name in processed:
-                    continue
-                # https://gitlab.com/tinybird/analytics/-/merge_requests/6737
-                if is_endpoint_with_no_dependencies(
-                    resources_to_run_fork_downstream.get(name, {}),
-                    dependencies_graph_fork_downstream,
-                    resources_to_run_fork_downstream,
-                ):
-                    endpoints_dep_map[name] = dependencies_graph_fork_downstream[name]
-                    continue
-                if (
-                    fork_downstream
-                    and not is_datasource(resources_to_run_fork_downstream.get(name))
-                    and name in dependencies_graph_fork_downstream
-                ):
-                    if is_materialized(resources_to_run_fork_downstream.get(name)):
-                        mat_dep_map[name] = dependencies_graph_fork_downstream[name]
+        if not fork_downstream:
+            # First, we will deploy the all the resources following the dependency graph except for the endpoints
+            groups = [group for group in toposort(dependencies_graph)]
+            for group in groups:
+                for name in group:
+                    if name in processed:
                         continue
-                    else:
-                        pipes_dep_map[name] = dependencies_graph_fork_downstream[name]
+
+                    if is_endpoint_with_no_dependencies(
+                        resources_to_run.get(name, {}),
+                        dependencies_graph,
+                        resources_to_run,
+                    ):
+                        endpoints_dep_map[name] = dependencies_graph[name]
                         continue
-                await push(
-                    name,
-                    resources_to_run_fork_downstream,
-                    resource_versions,
-                    latest_datasource_versions,
-                    dry_run,
-                    fork_downstream,
-                    fork,
-                )
-                processed.add(name)
 
-        # push endpoints with no dependencies or dependencies with other endpoints
-        groups = [group for group in toposort(endpoints_dep_map)]
-        for group in groups:
-            for name in group:
-                if name not in processed and not is_materialized(resources_to_run_fork_downstream[name]):
                     await push(
                         name,
-                        resources_to_run_fork_downstream,
+                        resources_to_run,
                         resource_versions,
                         latest_datasource_versions,
                         dry_run,
@@ -4046,26 +3984,56 @@ async def folder_push(
                     )
                     processed.add(name)
 
-        # materialized pipes at the end
-        if fork_downstream:
-            groups = [group for group in toposort(pipes_dep_map)]
+            # Then, we will deploy the endpoints that are on the dependency graph
+            groups = [group for group in toposort(endpoints_dep_map)]
             for group in groups:
                 for name in group:
-                    if name not in processed and not is_materialized(resources_to_run_fork_downstream[name]):
+                    if name not in processed:
                         await push(
                             name,
-                            resources_to_run_fork_downstream,
+                            resources_to_run,
                             resource_versions,
                             latest_datasource_versions,
                             dry_run,
                             fork_downstream,
+                            fork,
                         )
                         processed.add(name)
+        else:
+            # This will generate the graph from right to left and will fill the gaps of the dependencies
+            # If we have a graph like this:
+            # A -> B -> C
+            # If we only modify A, the normal dependencies graph will only contain a node like _{A => B}
+            # But we need a graph that contains A, B and C and the dependencies between them to deploy them in the right order
+            dependencies_graph_fork_downstream, resources_to_run_fork_downstream = generate_forkdownstream_graph(
+                dependency_graph.all_dep_map,
+                dependency_graph.all_resources,
+                resources_to_run,
+                list(dependency_graph.dep_map.keys()),
+            )
 
-            groups = [group for group in toposort(mat_dep_map)]
+            # First, we will deploy the datasources that have been changed
+            groups = [group for group in toposort(dependencies_graph_fork_downstream)]
             for group in groups:
                 for name in group:
-                    if name not in processed:
+                    if name in processed or not is_datasource(resources_to_run_fork_downstream[name]):
+                        continue
+
+                    # If the resource is new, we will use the normal resource information to deploy it
+                    # This is mostly used for datasources with connections.
+                    # At the moment, `resources_to_run_fork_downstream` is generated by `all_resources` and this is generated using the parameter `skip_connectors=True`
+                    # TODO: Should the `resources_to_run_fork_downstream` be generated using the `skip_connectors` parameter?
+                    if is_new(name, changed, dependencies_graph_fork_downstream, dependencies_graph_fork_downstream):
+                        await push(
+                            name,
+                            resources_to_run,
+                            resource_versions,
+                            latest_datasource_versions,
+                            dry_run,
+                            fork_downstream,
+                            fork,
+                        )
+                    else:
                         await push(
                             name,
                             resources_to_run_fork_downstream,
@@ -4073,8 +4041,86 @@ async def folder_push(
                             latest_datasource_versions,
                             dry_run,
                             fork_downstream,
+                            fork,
                         )
-                        processed.add(name)
+                    processed.add(name)
+
+            # Now, we will create a map of all the endpoints and there dependencies
+            # We are using the forkdownstream graph to get the dependencies of the endpoints as the normal dependencies graph only contains the resources that are going to be deployed
+            # But does not include the missing gaps
+            # If we have ENDPOINT_A ----> MV_PIPE_B -----> DATASOURCE_B ------> ENDPOINT_C
+            # Where endpoint A is being used in the MV_PIPE_B, if we only modify the endpoint A
+            # The dependencies graph will only contain the endpoint A and the MV_PIPE_B, but not the DATASOURCE_B and the ENDPOINT_C
+            groups = [group for group in toposort(dependencies_graph_fork_downstream)]
+            for group in groups:
+                for name in group:
+                    if name in processed or not is_endpoint(resources_to_run_fork_downstream[name]):
+                        continue
+
+                    endpoints_dep_map[name] = dependencies_graph_fork_downstream[name]
+
+            # Now that we have the dependencies of the endpoints, we need to check that the resources has not been deployed yet and only care about the endpoints that depend on endpoints
+            groups = [group for group in toposort(endpoints_dep_map)]
+
+            # As we have used the forkdownstream graph to get the dependencies of the endpoints, we have all the dependencies of the endpoints
+            # But we need to deploy the endpoints and the dependencies of the endpoints from left to right
+            # So we need to reverse the groups
+            groups.reverse()
+            for group in groups:
+                for name in group:
+                    if name in processed or not is_endpoint(resources_to_run_fork_downstream[name]):
+                        continue
+
+                    await push(
+                        name,
+                        resources_to_run_fork_downstream,
+                        resource_versions,
+                        latest_datasource_versions,
+                        dry_run,
+                        fork_downstream,
+                        fork,
+                    )
+                    processed.add(name)
+
+            # Now we should have the endpoints and datasources deployed, we can deploy the rest of the pipes (copy & sinks)
+            # We need to rely on the forkdownstream graph as it contains all the modified pipes as well as the dependencies of the pipes
+            # In this case, we don't need to generate a new graph as we did for the endpoints as the pipes are not going to be used as dependencies and the datasources are already deployed
+            groups = [group for group in toposort(dependencies_graph_fork_downstream)]
+            for group in groups:
+                for name in group:
+                    if name in processed or is_materialized(resources_to_run_fork_downstream.get(name)):
+                        continue
+
+                    await push(
+                        name,
+                        resources_to_run_fork_downstream,
+                        resource_versions,
+                        latest_datasource_versions,
+                        dry_run,
+                        fork_downstream,
+                        fork,
+                    )
+                    processed.add(name)
+
+            # Finally, we need to deploy the materialized views from right to left.
+            # We need to rely on the forkdownstream graph as it contains all the modified materialized views as well as the dependencies of the materialized views
+            # In this case, we don't need to generate a new graph as we did for the endpoints as the pipes are not going to be used as dependencies and the datasources are already deployed
+            groups = [group for group in toposort(dependencies_graph_fork_downstream)]
+            for group in groups:
+                for name in group:
+                    if name in processed or not is_materialized(resources_to_run_fork_downstream.get(name)):
+                        continue
+
+                    await push(
+                        name,
+                        resources_to_run_fork_downstream,
+                        resource_versions,
+                        latest_datasource_versions,
+                        dry_run,
+                        fork_downstream,
+                        fork,
+                    )
+                    processed.add(name)
 
     if deployment.is_git_release:
         deployment.deploying_dry_run()
@@ -4856,6 +4902,7 @@ def is_endpoint(resource: Optional[Dict[str, Any]]) -> bool:
 def is_materialized(resource: Optional[Dict[str, Any]]) -> bool:
     if not resource:
         return False
+
     is_materialized = any(
         [node.get("params", {}).get("type", None) == "materialized" for node in resource.get("nodes", []) or []]
     )

{tinybird-cli-3.3.1.dev5 → tinybird-cli-3.3.1.dev6}/tinybird_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: tinybird-cli
-Version: 3.3.1.dev5
+Version: 3.3.1.dev6
 Summary: Tinybird Command Line Tool
 Home-page: https://docs.tinybird.co/cli.html
 Author: Tinybird
@@ -19,13 +19,18 @@ Changelog
 
 ---------
 
+3.3.1.dev6
+************
+
+- `Fixed` `tb deploy` to deploy the endpoints in the correct order when there are endpoints depending on other endpoints.
+
 3.3.1.dev5
-***********
+************
 
 - `Fixed` `tb push` pipe with multiple tokens
 
 3.3.1.dev4
-***********
+************
 
 - `Changed` script `exec_test.sh`  from `tb init --git` to run retries to avoid flakiness in tests
 
@@ -35,7 +40,7 @@ Changelog
 - `Fixed` `IMPORT` and `ENGINE` changes were not detected `tb deploy`
 
 3.3.1.dev2
-***********
+************
 
 - `Changed` Update `tb release rm --oldest-rollback` to use new API
 
@@ -54,7 +59,7 @@ Changelog
 - `Changed` Stop creating endpoints folder on tb init
 - `Changed` Small refactoring of the connector credentials
 - `Fixed` Progress bar in `tb init --git`.
-- `Fixed` Don't do diff when there is no modified resources on `tb deploy`.
+- `Fixed` Don't do diff when there are no modified resources on `tb deploy`.
 - `Fixed` Upgrade GitPython dependency to 3.1.32. That version fixes the security vulnerability `CVE-2022-24439 <https://github.com/advisories/GHSA-hcpj-qp55-gfph>`
 - `Fixed` Homepage URL in PyPI
 
@@ -62,12 +67,12 @@ Changelog
 ************
 
 - `Changed` `tb init --git` uses v3.1.0 of the CI templates (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
-- `Changed` Allow alter datasource with jsonpaths to add DEFAULTS if the datasource already have any (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
+- `Changed` Allow alter data source with JSON paths to add DEFAULTS if the data source already has any (See release notes: https://github.com/tinybirdco/ci/releases/tag/v3.1.0)
 
 3.1.0
 ************
 
-- `Added` `tb init --cicd` to generate CI/CD files not only when using `--git`. Combine with `--force` for overwrite
+- `Added` `tb init --cicd` to generate CI/CD files not only when using `--git`. Combine with `--force` to overwrite
 - `Added` Ignore format changes on `tb deploy` by default
 - `Changed` `tb init --git` and `tb release generate` include `set -euxo pipefail` in the generated bash scripts
 - `Changed` `tb init --git` uses github.com/tinybirdco/ci => v3.0.0
@@ -75,7 +80,7 @@ Changelog
 - `Changed` `tb release promote` automatically deletes de oldest rollback Release if `export TB_FORCE_REMOVE_OLDEST_ROLLBACK=1`
 - `Changed` option `--node` in `tb pipe populate` to required as it is already mandatory to specify the node to populate. Now, it will provide an error message if the `--node` option is not provided.
 - `Changed` Better error report on `tb fmt --diff`. Use it in CI to standardize Datafile formatting
-- `Changed` `tb init --git` generates a requirements.txt file to install the latest tinybird-cli release from pypi
+- `Changed` `tb init --git` generates a requirements.txt file to install the latest tinybird-cli release from PyPI
 - `Changed` `tb deploy` shows an example when failing because of a missing `--semver` argument
 - `Fixed` When parsing the response of a CSV `tb sql` command, the CLI was interpreting the result as an error in some cases.