confluent-pyflink 1.0.dev1__tar.gz

confluent_pyflink-1.0.dev1/MANIFEST.in

@@ -0,0 +1,22 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+global-exclude *.py[cod] __pycache__ .DS_Store
+graft deps/bin
+include README.md confluent-flink-jars/version.py
+recursive-exclude pyflink/fn_execution *

confluent_pyflink-1.0.dev1/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: confluent-pyflink
+Version: 1.0.dev1
+Summary: Confluent Apache Flink Table API Python
+Author-email: Confluent <dev@confluent.io>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://confluent.io
+Project-URL: Examples, https://github.com/confluentinc/flink-table-api-python-examples
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: apache-flink~=2.1.0
+Requires-Dist: pydantic-settings>=2.10.1
+Requires-Dist: pyyaml
+Requires-Dist: confluent-flink-jars==2.1.dev1
+
+# Confluent Apache Flink Table API Python
+
+This package contains the client library for running Apache Flink's Table API on Confluent Cloud.
+
+The [Table API](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/) enables a programmatic
+way of developing, testing, and submitting Flink pipelines for processing data streams.
+Streams can be finite or infinite, with insert-only or changelog data. The latter allows for dealing with *Change Data
+Capture* (CDC) events.
+
+Within the API, you conceptually work with tables that change over time - inspired by relational databases. Write
+a *Table Program* as a declarative and structured graph of data transformations. Table API is inspired by SQL and complements
+it with additional tools for juggling real-time data. You can mix and match Flink SQL with Table API at any time as they
+go hand in hand.
+
+## Table API on Confluent Cloud
+
+Table API on Confluent Cloud is a client-side library that delegates Flink API calls to Confluent’s public
+REST API. It submits [Statements](https://docs.confluent.io/cloud/current/api.html#tag/Statements-(sqlv1)) and retrieves
+[StatementResults](https://docs.confluent.io/cloud/current/api.html#tag/Statement-Results-(sqlv1)).
+
+Table programs are implemented against [Flink's open source Table API for Python](https://github.com/apache/flink/tree/master/flink-python/pyflink/table).
+This package repackages Flink's Python API and bundles the Confluent-specific components for powering the `TableEnvironment` without the need
+for a local Flink cluster. While using those packages, Flink internal components such as
+`CatalogStore`, `Catalog`, `Planner`, `Executor`, and configuration are managed by the plugin and fully integrate with
+Confluent Cloud. Including access to Apache Kafka®, Schema Registry, and Flink Compute Pools.
+
+Note: The Table API plugin is in Open Preview stage.
+
+### Motivating Example
+
+The following code shows how a Table API program is structured. Subsequent sections will go into more details how you
+can use the examples of this repository to play around with Flink on Confluent Cloud.
+
+```python
+from confluent_pyflink.table.utils import ConfluentSettings, ConfluentTools
+from confluent_pyflink.table import TableEnvironment, Row
+from confluent_pyflink.table.expressions import col, row
+
+
+def run():
+    # Setup connection properties to Confluent Cloud
+    settings = ConfluentSettings.from_global_variables()
+    env = TableEnvironment.create(settings)
+
+    # Run your first Flink statement in Table API
+    env.from_elements([row("Hello world!")]).execute().print()
+
+    # Or use SQL
+    env.sql_query("SELECT 'Hello world!'").execute().print()
+
+    # Structure your code with Table objects - the main ingredient of Table API.
+    table = (
+        env.from_path("examples.marketplace.clicks")
+        .filter(col("user_agent").like("Mozilla%"))
+        .select(col("click_id"), col("user_id"))
+    )
+
+    table.print_schema()
+    print(table.explain())
+
+    # Use the provided tools to test on a subset of the streaming data
+    expected = ConfluentTools.collect_materialized_limit(table, 50)
+    actual = [Row(42, 500)]
+    if expected != actual:
+        print("Results don't match!")
+
+
+if __name__ == "__main__":
+    run()
+```
+
+### Further Examples
+
+For further examples, please see Confluent's
+[Apache Flink® Table API on Confluent Cloud Examples](https://github.com/confluentinc/flink-table-api-python-examples)
+repository.

confluent_pyflink-1.0.dev1/README.md

@@ -0,0 +1,76 @@
+# Confluent Apache Flink Table API Python
+
+This package contains the client library for running Apache Flink's Table API on Confluent Cloud.
+
+The [Table API](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/) enables a programmatic
+way of developing, testing, and submitting Flink pipelines for processing data streams.
+Streams can be finite or infinite, with insert-only or changelog data. The latter allows for dealing with *Change Data
+Capture* (CDC) events.
+
+Within the API, you conceptually work with tables that change over time - inspired by relational databases. Write
+a *Table Program* as a declarative and structured graph of data transformations. Table API is inspired by SQL and complements
+it with additional tools for juggling real-time data. You can mix and match Flink SQL with Table API at any time as they
+go hand in hand.
+
+## Table API on Confluent Cloud
+
+Table API on Confluent Cloud is a client-side library that delegates Flink API calls to Confluent’s public
+REST API. It submits [Statements](https://docs.confluent.io/cloud/current/api.html#tag/Statements-(sqlv1)) and retrieves
+[StatementResults](https://docs.confluent.io/cloud/current/api.html#tag/Statement-Results-(sqlv1)).
+
+Table programs are implemented against [Flink's open source Table API for Python](https://github.com/apache/flink/tree/master/flink-python/pyflink/table).
+This package repackages Flink's Python API and bundles the Confluent-specific components for powering the `TableEnvironment` without the need
+for a local Flink cluster. While using those packages, Flink internal components such as
+`CatalogStore`, `Catalog`, `Planner`, `Executor`, and configuration are managed by the plugin and fully integrate with
+Confluent Cloud. Including access to Apache Kafka®, Schema Registry, and Flink Compute Pools.
+
+Note: The Table API plugin is in Open Preview stage.
+
+### Motivating Example
+
+The following code shows how a Table API program is structured. Subsequent sections will go into more details how you
+can use the examples of this repository to play around with Flink on Confluent Cloud.
+
+```python
+from confluent_pyflink.table.utils import ConfluentSettings, ConfluentTools
+from confluent_pyflink.table import TableEnvironment, Row
+from confluent_pyflink.table.expressions import col, row
+
+
+def run():
+    # Setup connection properties to Confluent Cloud
+    settings = ConfluentSettings.from_global_variables()
+    env = TableEnvironment.create(settings)
+
+    # Run your first Flink statement in Table API
+    env.from_elements([row("Hello world!")]).execute().print()
+
+    # Or use SQL
+    env.sql_query("SELECT 'Hello world!'").execute().print()
+
+    # Structure your code with Table objects - the main ingredient of Table API.
+    table = (
+        env.from_path("examples.marketplace.clicks")
+        .filter(col("user_agent").like("Mozilla%"))
+        .select(col("click_id"), col("user_id"))
+    )
+
+    table.print_schema()
+    print(table.explain())
+
+    # Use the provided tools to test on a subset of the streaming data
+    expected = ConfluentTools.collect_materialized_limit(table, 50)
+    actual = [Row(42, 500)]
+    if expected != actual:
+        print("Results don't match!")
+
+
+if __name__ == "__main__":
+    run()
+```
+
+### Further Examples
+
+For further examples, please see Confluent's
+[Apache Flink® Table API on Confluent Cloud Examples](https://github.com/confluentinc/flink-table-api-python-examples)
+repository.

confluent_pyflink-1.0.dev1/confluent-flink-jars/version.py

@@ -0,0 +1,24 @@
+################################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+"""
+The confluent-flink-jars version will be consistent with the flink version and follow the PEP440.
+.. seealso:: https://www.python.org/dev/peps/pep-0440
+"""
+
+__version__ = "2.1.dev1"

confluent_pyflink-1.0.dev1/confluent_pyflink.egg-info/PKG-INFO

@@ -0,0 +1,96 @@
+Metadata-Version: 2.4
+Name: confluent-pyflink
+Version: 1.0.dev1
+Summary: Confluent Apache Flink Table API Python
+Author-email: Confluent <dev@confluent.io>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://confluent.io
+Project-URL: Examples, https://github.com/confluentinc/flink-table-api-python-examples
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: apache-flink~=2.1.0
+Requires-Dist: pydantic-settings>=2.10.1
+Requires-Dist: pyyaml
+Requires-Dist: confluent-flink-jars==2.1.dev1
+
+# Confluent Apache Flink Table API Python
+
+This package contains the client library for running Apache Flink's Table API on Confluent Cloud.
+
+The [Table API](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/) enables a programmatic
+way of developing, testing, and submitting Flink pipelines for processing data streams.
+Streams can be finite or infinite, with insert-only or changelog data. The latter allows for dealing with *Change Data
+Capture* (CDC) events.
+
+Within the API, you conceptually work with tables that change over time - inspired by relational databases. Write
+a *Table Program* as a declarative and structured graph of data transformations. Table API is inspired by SQL and complements
+it with additional tools for juggling real-time data. You can mix and match Flink SQL with Table API at any time as they
+go hand in hand.
+
+## Table API on Confluent Cloud
+
+Table API on Confluent Cloud is a client-side library that delegates Flink API calls to Confluent’s public
+REST API. It submits [Statements](https://docs.confluent.io/cloud/current/api.html#tag/Statements-(sqlv1)) and retrieves
+[StatementResults](https://docs.confluent.io/cloud/current/api.html#tag/Statement-Results-(sqlv1)).
+
+Table programs are implemented against [Flink's open source Table API for Python](https://github.com/apache/flink/tree/master/flink-python/pyflink/table).
+This package repackages Flink's Python API and bundles the Confluent-specific components for powering the `TableEnvironment` without the need
+for a local Flink cluster. While using those packages, Flink internal components such as
+`CatalogStore`, `Catalog`, `Planner`, `Executor`, and configuration are managed by the plugin and fully integrate with
+Confluent Cloud. Including access to Apache Kafka®, Schema Registry, and Flink Compute Pools.
+
+Note: The Table API plugin is in Open Preview stage.
+
+### Motivating Example
+
+The following code shows how a Table API program is structured. Subsequent sections will go into more details how you
+can use the examples of this repository to play around with Flink on Confluent Cloud.
+
+```python
+from confluent_pyflink.table.utils import ConfluentSettings, ConfluentTools
+from confluent_pyflink.table import TableEnvironment, Row
+from confluent_pyflink.table.expressions import col, row
+
+
+def run():
+    # Setup connection properties to Confluent Cloud
+    settings = ConfluentSettings.from_global_variables()
+    env = TableEnvironment.create(settings)
+
+    # Run your first Flink statement in Table API
+    env.from_elements([row("Hello world!")]).execute().print()
+
+    # Or use SQL
+    env.sql_query("SELECT 'Hello world!'").execute().print()
+
+    # Structure your code with Table objects - the main ingredient of Table API.
+    table = (
+        env.from_path("examples.marketplace.clicks")
+        .filter(col("user_agent").like("Mozilla%"))
+        .select(col("click_id"), col("user_id"))
+    )
+
+    table.print_schema()
+    print(table.explain())
+
+    # Use the provided tools to test on a subset of the streaming data
+    expected = ConfluentTools.collect_materialized_limit(table, 50)
+    actual = [Row(42, 500)]
+    if expected != actual:
+        print("Results don't match!")
+
+
+if __name__ == "__main__":
+    run()
+```
+
+### Further Examples
+
+For further examples, please see Confluent's
+[Apache Flink® Table API on Confluent Cloud Examples](https://github.com/confluentinc/flink-table-api-python-examples)
+repository.

confluent_pyflink-1.0.dev1/confluent_pyflink.egg-info/SOURCES.txt

@@ -0,0 +1,44 @@
+MANIFEST.in
+README.md
+pyproject.toml
+confluent-flink-jars/version.py
+confluent_pyflink.egg-info/PKG-INFO
+confluent_pyflink.egg-info/SOURCES.txt
+confluent_pyflink.egg-info/dependency_links.txt
+confluent_pyflink.egg-info/not-zip-safe
+confluent_pyflink.egg-info/requires.txt
+confluent_pyflink.egg-info/top_level.txt
+src/confluent_pyflink/__init__.py
+src/confluent_pyflink/context/__init__.py
+src/confluent_pyflink/context/confluent_context.py
+src/confluent_pyflink/context/context.py
+src/confluent_pyflink/context/java_gateway_manager.py
+src/confluent_pyflink/context/local_context.py
+src/confluent_pyflink/table/__init__.py
+src/confluent_pyflink/table/catalog.py
+src/confluent_pyflink/table/changelog_mode.py
+src/confluent_pyflink/table/data_view.py
+src/confluent_pyflink/table/descriptors.py
+src/confluent_pyflink/table/environment_settings.py
+src/confluent_pyflink/table/explain_detail.py
+src/confluent_pyflink/table/expression.py
+src/confluent_pyflink/table/expressions.py
+src/confluent_pyflink/table/functions.py
+src/confluent_pyflink/table/module.py
+src/confluent_pyflink/table/result_kind.py
+src/confluent_pyflink/table/schema.py
+src/confluent_pyflink/table/serializers.py
+src/confluent_pyflink/table/sql_dialect.py
+src/confluent_pyflink/table/statement_set.py
+src/confluent_pyflink/table/table.py
+src/confluent_pyflink/table/table_config.py
+src/confluent_pyflink/table/table_descriptor.py
+src/confluent_pyflink/table/table_environment.py
+src/confluent_pyflink/table/table_result.py
+src/confluent_pyflink/table/table_schema.py
+src/confluent_pyflink/table/types.py
+src/confluent_pyflink/table/udf.py
+src/confluent_pyflink/table/window.py
+src/confluent_pyflink/table/utils/__init__.py
+src/confluent_pyflink/table/utils/confluent_settings.py
+src/confluent_pyflink/table/utils/confluent_tools.py

confluent_pyflink-1.0.dev1/confluent_pyflink.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

confluent_pyflink-1.0.dev1/confluent_pyflink.egg-info/not-zip-safe

@@ -0,0 +1 @@
+

confluent_pyflink-1.0.dev1/confluent_pyflink.egg-info/requires.txt

@@ -0,0 +1,4 @@
+apache-flink~=2.1.0
+pydantic-settings>=2.10.1
+pyyaml
+confluent-flink-jars==2.1.dev1

confluent_pyflink-1.0.dev1/confluent_pyflink.egg-info/top_level.txt

@@ -0,0 +1 @@
+confluent_pyflink

confluent_pyflink-1.0.dev1/pyproject.toml

@@ -0,0 +1,75 @@
+[build-system]
+requires = ["setuptools>=75.8", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "confluent-pyflink"
+description = "Confluent Apache Flink Table API Python"
+readme = "README.md"
+version = "1.0.dev1"
+dependencies = [
+    "apache-flink~=2.1.0",
+    "pydantic-settings>=2.10.1",
+    "pyyaml",
+    "confluent-flink-jars==2.1.dev1"
+]
+authors = [
+    {name = "Confluent", email = "dev@confluent.io"}
+]
+license = "Apache-2.0"
+requires-python = ">=3.9"
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+dynamic = ["scripts"]
+
+[project.urls]
+Homepage = "https://confluent.io"
+Examples = "https://github.com/confluentinc/flink-table-api-python-examples"
+
+[dependency-groups]
+dev = [
+    "ruff",
+    "pytest>=8.4.1"
+]
+
+[tool.setuptools]
+zip-safe = false
+include-package-data = true
+
+[tool.setuptools.packages.find]
+where = ["src", "deps"]
+include = [
+    'confluent_pyflink',
+    'confluent_pyflink.context',
+    'confluent_pyflink.table',
+    'confluent_pyflink.table.utils',
+]
+
+[tool.ruff]
+line-length = 101
+indent-width = 4
+target-version = "py39"
+extend-exclude = ["setup_pyshell.py"]
+
+[tool.ruff.lint]
+select = ["E", "F"]
+ignore = []
+fixable = ["ALL"]
+unfixable = []
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+skip-magic-trailing-comma = false
+line-ending = "auto"
+
+[tool.uv.sources]
+confluent-flink-jars = { path = "confluent-flink-jars" }
+
+[tool.pytest.ini_options]
+addopts = ["--junit-xml", "TEST-result.xml",]

confluent_pyflink-1.0.dev1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

confluent_pyflink-1.0.dev1/src/confluent_pyflink/context/__init__.py

@@ -0,0 +1,4 @@
+from .confluent_context import ConfluentContext
+from .local_context import LocalContext
+
+__all__ = ["ConfluentContext", "LocalContext"]

confluent_pyflink-1.0.dev1/src/confluent_pyflink/context/confluent_context.py

@@ -0,0 +1,92 @@
+from .context import Context, FlinkDirectories, InvalidFlinkHomeException
+import os
+import logging
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+
+class ConfluentContext(Context):
+    """
+    An implementation of the Context class for Confluent PyFlink applications.
+    Loads classes and jars from the Confluent Flink distribution, for use with Confluent Cloud.
+
+    Use it as a context manager::
+
+        with ConfluentContext():
+            settings = ConfluentSettings(...)
+            env = TableEnvironment.create(settings)
+            env.from_elements([row("Hello world!")]).execute().print()
+    """
+
+    def _find_confluent_flink_home(self) -> Path:
+        """
+        Find and return the Confluent Flink home directory.
+
+        Searches for Confluent Flink home in the following order:
+        1. CONFLUENT_FLINK_HOME environment variable
+        2. PyFlink module path
+        """
+        # If the environment has set CONFLUENT_FLINK_HOME, trust it.
+        if "CONFLUENT_FLINK_HOME" in os.environ:
+            return Path(os.environ["CONFLUENT_FLINK_HOME"])
+        else:
+            try:
+                CONFLUENT_FLINK_HOME = None
+                for module_home in __import__("confluent_pyflink").__path__:
+                    CONFLUENT_FLINK_HOME = module_home
+                if CONFLUENT_FLINK_HOME is not None:
+                    return Path(CONFLUENT_FLINK_HOME)
+                else:
+                    raise InvalidFlinkHomeException(
+                        "Could not find valid CONFLUENT_FLINK_HOME (Flink distribution directory) "
+                        "in current environment."
+                    )
+            except Exception as exception:
+                raise InvalidFlinkHomeException(
+                    "Unable to find CONFLUENT_FLINK_HOME (Flink distribution directory) due to:"
+                ) from exception
+
+    def _get_flink_directories(self) -> FlinkDirectories:
+        flink_home = self._find_flink_home()
+        confluent_flink_home = self._find_confluent_flink_home()
+        flink_dirs = FlinkDirectories.build_flink_dirs(
+            home=flink_home, default_lib=confluent_flink_home / "lib"
+        )
+        logger.debug(f"Using the following Flink directories:\n{flink_dirs}")
+        return flink_dirs
+
+    def _classes_to_load(self) -> set[str]:
+        return {
+            "org.apache.flink.table.api.*",
+            "org.apache.flink.table.legacy.api.*",
+            "org.apache.flink.table.api.config.*",
+            "org.apache.flink.table.api.java.*",
+            "org.apache.flink.table.api.bridge.java.*",
+            "org.apache.flink.table.api.dataview.*",
+            "org.apache.flink.table.catalog.*",
+            "org.apache.flink.table.descriptors.*",
+            "org.apache.flink.table.legacy.descriptors.*",
+            "org.apache.flink.table.descriptors.python.*",
+            "org.apache.flink.table.expressions.*",
+            "org.apache.flink.table.sources.*",
+            "org.apache.flink.table.legacy.sources.*",
+            "org.apache.flink.table.sinks.*",
+            "org.apache.flink.table.legacy.sinks.*",
+            "org.apache.flink.table.types.*",
+            "org.apache.flink.table.types.logical.*",
+            "org.apache.flink.table.util.python.*",
+            "org.apache.flink.api.common.python.*",
+            "org.apache.flink.api.common.typeinfo.TypeInformation",
+            "org.apache.flink.api.common.typeinfo.Types",
+            "org.apache.flink.api.java.ExecutionEnvironment",
+            "org.apache.flink.streaming.api.environment.StreamExecutionEnvironment",
+            "org.apache.flink.python.util.PythonDependencyUtils",
+            "org.apache.flink.python.PythonOptions",
+            "org.apache.flink.client.python.PythonGatewayServer",
+            "org.apache.flink.streaming.api.functions.python.*",
+            "org.apache.flink.streaming.api.operators.python.process.*",
+            "org.apache.flink.streaming.api.operators.python.embedded.*",
+            "org.apache.flink.streaming.api.typeinfo.python.*",
+            "io.confluent.flink.plugin.*",
+        }