feldera 0.100.0__tar.gz → 0.101.0__tar.gz

{feldera-0.100.0 → feldera-0.101.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: feldera
-Version: 0.100.0
+Version: 0.101.0
 Summary: The feldera python client
 Author-email: Feldera Team <dev@feldera.com>
 License: MIT
@@ -54,10 +54,11 @@ If you have cloned the Feldera repo, you can install the python SDK as follows:
 pip install python/
 ```
 
-Checkout the docs [here](./feldera/__init__.py) for an example on how to use the SDK.
-
 ## Documentation
 
+The Python SDK documentation is available at
+[Feldera Python SDK Docs](https://docs.feldera.com/python).
+
 To build the html documentation run:
 
 Ensure that you have sphinx installed. If not, install it using `pip install sphinx`.
@@ -77,27 +78,23 @@ To clean the build, run `make clean`.
 To run unit tests:
 
 ```bash
-(cd python && python3 -m unittest)
+cd python && python3 -m pytest tests/
 ```
 
-> ⚠️ Running the unit tests will **delete all existing pipelines**.
-
-The following command runs end-to-end tests.  You'll need a pipeline
-manager running at `http://localhost:8080`.  For the pipeline builder
-tests, you'll also need a broker available at `localhost:9092` and
-(from the pipelines) `redpanda:19092`.  (To change those locations,
-set the environment variables listed in `python/tests/__init__.py`.)
-
-```bash
-(cd python && python3 -m pytest tests)
-```
+- This will detect and run all test files that match the pattern `test_*.py` or
+  `*_test.py`.
+- By default, the tests expect a running Feldera instance at `http://localhost:8080`.
+  To override the default endpoint, set the `FELDERA_BASE_URL` environment variable.
 
 To run tests from a specific file:
 
 ```bash
-(cd python && python3 -m unittest ./tests/path-to-file.py)
+(cd python && python3 -m pytest ./tests/path-to-file.py)
 ```
 
+#### Running Aggregate Tests
+
+The aggregate tests validate end-to-end correctness of SQL functionality.
 To run the aggregate tests use:
 
 ```bash
@@ -105,6 +102,38 @@ cd python
 PYTHONPATH=`pwd` python3 ./tests/aggregate_tests/main.py
 ```
 
+### Reducing Compilation Cycles
+
+To reduce redundant compilation cycles during testing:
+
+* **Inherit from `SharedTestPipeline`** instead of `unittest.TestCase`.
+* **Define DDLs** (e.g., `CREATE TABLE`, `CREATE VIEW`) in the **docstring** of each test method.
+  * All DDLs from all test functions in the class are combined and compiled into a single pipeline.
+  * If a table or view is already defined in one test, it can be used directly in others without redefinition.
+  * Ensure that all table and view names are unique within the class.
+* Use `@enterprise_only` on tests that require Enterprise features. Their DDLs will be skipped on OSS builds.
+* Use `self.set_runtime_config(...)` to override the default pipeline config.
+  * Reset it at the end using `self.reset_runtime_config()`.
+* Access the shared pipeline via `self.pipeline`.
+
+#### Example
+
+```python
+from tests.shared_test_pipeline import SharedTestPipeline
+
+class TestAverage(SharedTestPipeline):
+    def test_average(self):
+        """
+        CREATE TABLE students(id INT, name STRING);
+        CREATE MATERIALIZED VIEW v AS SELECT * FROM students;
+        """
+        ...
+        self.pipeline.start()
+        self.pipeline.input_pandas("students", df)
+        self.pipeline.wait_for_completion(True)
+        ...
+```
+
 ## Linting and formatting
 
 Use [Ruff] to run the lint checks that will be executed by the

feldera-0.101.0/README.md

@@ -0,0 +1,129 @@
+# Feldera Python SDK
+
+Feldera Python is the Feldera SDK for Python developers.
+
+## Installation
+
+```bash
+pip install feldera
+```
+
+### Installing from Github
+
+```bash
+pip install git+https://github.com/feldera/feldera#subdirectory=python
+```
+
+Similarly, to install from a specific branch:
+
+```bash
+$ pip install git+https://github.com/feldera/feldera@{BRANCH_NAME}#subdirectory=python
+```
+
+Replace `{BRANCH_NAME}` with the name of the branch you want to install from.
+
+### Installing from Local Directory
+
+If you have cloned the Feldera repo, you can install the python SDK as follows:
+
+```bash
+# the Feldera Python SDK is present inside the python/ directory
+pip install python/
+```
+
+## Documentation
+
+The Python SDK documentation is available at
+[Feldera Python SDK Docs](https://docs.feldera.com/python).
+
+To build the html documentation run:
+
+Ensure that you have sphinx installed. If not, install it using `pip install sphinx`.
+
+Then run the following commands:
+
+```bash
+cd docs
+sphinx-apidoc -o . ../feldera
+make html
+```
+
+To clean the build, run `make clean`.
+
+## Testing
+
+To run unit tests:
+
+```bash
+cd python && python3 -m pytest tests/
+```
+
+- This will detect and run all test files that match the pattern `test_*.py` or
+  `*_test.py`.
+- By default, the tests expect a running Feldera instance at `http://localhost:8080`.
+  To override the default endpoint, set the `FELDERA_BASE_URL` environment variable.
+
+To run tests from a specific file:
+
+```bash
+(cd python && python3 -m pytest ./tests/path-to-file.py)
+```
+
+#### Running Aggregate Tests
+
+The aggregate tests validate end-to-end correctness of SQL functionality.
+To run the aggregate tests use:
+
+```bash
+cd python
+PYTHONPATH=`pwd` python3 ./tests/aggregate_tests/main.py
+```
+
+### Reducing Compilation Cycles
+
+To reduce redundant compilation cycles during testing:
+
+* **Inherit from `SharedTestPipeline`** instead of `unittest.TestCase`.
+* **Define DDLs** (e.g., `CREATE TABLE`, `CREATE VIEW`) in the **docstring** of each test method.
+  * All DDLs from all test functions in the class are combined and compiled into a single pipeline.
+  * If a table or view is already defined in one test, it can be used directly in others without redefinition.
+  * Ensure that all table and view names are unique within the class.
+* Use `@enterprise_only` on tests that require Enterprise features. Their DDLs will be skipped on OSS builds.
+* Use `self.set_runtime_config(...)` to override the default pipeline config.
+  * Reset it at the end using `self.reset_runtime_config()`.
+* Access the shared pipeline via `self.pipeline`.
+
+#### Example
+
+```python
+from tests.shared_test_pipeline import SharedTestPipeline
+
+class TestAverage(SharedTestPipeline):
+    def test_average(self):
+        """
+        CREATE TABLE students(id INT, name STRING);
+        CREATE MATERIALIZED VIEW v AS SELECT * FROM students;
+        """
+        ...
+        self.pipeline.start()
+        self.pipeline.input_pandas("students", df)
+        self.pipeline.wait_for_completion(True)
+        ...
+```
+
+## Linting and formatting
+
+Use [Ruff] to run the lint checks that will be executed by the
+precommit hook when a PR is submitted:
+
+```bash
+ruff check python/
+```
+
+To reformat the code in the same way as the precommit hook:
+
+```bash
+ruff format
+```
+
+[Ruff]: https://github.com/astral-sh/ruff

{feldera-0.100.0 → feldera-0.101.0}/feldera/_callback_runner.py

@@ -54,12 +54,12 @@ class CallbackRunner(Thread):
             )
 
         # by default, we assume that the pipeline has been started
-        ack: _CallbackRunnerInstruction = _CallbackRunnerInstruction.PipelineStarted
+        ack = _CallbackRunnerInstruction.PipelineStarted
 
         # if there is Queue, we wait for the instruction to start the pipeline
         # this means that we are listening to the pipeline before running it, therefore, all data should be received
         if self.queue:
-            ack: _CallbackRunnerInstruction = self.queue.get()
+            ack = self.queue.get()
 
         match ack:
             # if the pipeline has actually been started, we start a listener
@@ -77,11 +77,12 @@ class CallbackRunner(Thread):
 
                 for chunk in gen_obj:
                     chunk: dict = chunk
-                    data: list[dict] = chunk.get("json_data")
-                    seq_no: int = chunk.get("sequence_number")
-
-                    if data is not None:
-                        self.callback(dataframe_from_response([data], schema), seq_no)
+                    data: Optional[list[dict]] = chunk.get("json_data")
+                    seq_no: Optional[int] = chunk.get("sequence_number")
+                    if data is not None and seq_no is not None:
+                        self.callback(
+                            dataframe_from_response([data], self.schema), seq_no
+                        )
 
                     if self.queue:
                         try:

{feldera-0.100.0 → feldera-0.101.0}/feldera/pipeline.py

@@ -473,10 +473,13 @@ metrics"""
             pipeline to stop.
         """
 
-        if len(self.views_tx) > 0:
-            for _, queue in self.views_tx.pop().items():
+        for view_queue in self.views_tx:
+            for _, queue in view_queue.items():
                 # sends a message to the callback runner to stop listening
                 queue.put(_CallbackRunnerInstruction.RanToCompletion)
+
+        if len(self.views_tx) > 0:
+            for view_name, queue in self.views_tx.pop().items():
                 # block until the callback runner has been stopped
                 queue.join()
 

{feldera-0.100.0 → feldera-0.101.0}/feldera/pipeline_builder.py

@@ -2,7 +2,7 @@ from feldera.rest.feldera_client import FelderaClient
 from feldera.rest.pipeline import Pipeline as InnerPipeline
 from feldera.pipeline import Pipeline
 from feldera.enums import CompilationProfile
-from feldera.runtime_config import RuntimeConfig, Resources
+from feldera.runtime_config import RuntimeConfig
 from feldera.rest.errors import FelderaAPIError
 
 
@@ -29,7 +29,7 @@ class PipelineBuilder:
         udf_toml: str = "",
         description: str = "",
         compilation_profile: CompilationProfile = CompilationProfile.OPTIMIZED,
-        runtime_config: RuntimeConfig = RuntimeConfig(resources=Resources()),
+        runtime_config: RuntimeConfig = RuntimeConfig.default(),
     ):
         self.client: FelderaClient = client
         self.name: str | None = name

{feldera-0.100.0 → feldera-0.101.0}/feldera/rest/feldera_client.py

@@ -454,11 +454,11 @@ Reason: The pipeline is in a STOPPED state due to the following error:
         pipeline_name: str,
         table_name: str,
         format: str,
-        data: list[list | str | dict] | dict,
+        data: list[list | str | dict] | dict | str,
         array: bool = False,
         force: bool = False,
         update_format: str = "raw",
-        json_flavor: str = None,
+        json_flavor: Optional[str] = None,
         serialize: bool = True,
     ):
         """

{feldera-0.100.0 → feldera-0.101.0}/feldera/runtime_config.py

@@ -1,3 +1,4 @@
+import os
 from typing import Optional, Any, Mapping
 
 
@@ -72,6 +73,7 @@ class RuntimeConfig:
         clock_resolution_usecs: Optional[int] = None,
         provisioning_timeout_secs: Optional[int] = None,
         resources: Optional[Resources] = None,
+        runtime_version: Optional[str] = None,
     ):
         self.workers = workers
         self.tracing = tracing
@@ -81,6 +83,9 @@ class RuntimeConfig:
         self.min_batch_size_records = min_batch_size_records
         self.clock_resolution_usecs = clock_resolution_usecs
         self.provisioning_timeout_secs = provisioning_timeout_secs
+        self.runtime_version = runtime_version or os.environ.get(
+            "FELDERA_RUNTIME_VERSION"
+        )
         if resources is not None:
             self.resources = resources.__dict__
         if isinstance(storage, bool):
@@ -88,6 +93,10 @@ class RuntimeConfig:
         if isinstance(storage, Storage):
             self.storage = storage.__dict__
 
+    @staticmethod
+    def default() -> "RuntimeConfig":
+        return RuntimeConfig(resources=Resources())
+
     @classmethod
     def from_dict(cls, d: Mapping[str, Any]):
         """

{feldera-0.100.0 → feldera-0.101.0}/feldera.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: feldera
-Version: 0.100.0
+Version: 0.101.0
 Summary: The feldera python client
 Author-email: Feldera Team <dev@feldera.com>
 License: MIT
@@ -54,10 +54,11 @@ If you have cloned the Feldera repo, you can install the python SDK as follows:
 pip install python/
 ```
 
-Checkout the docs [here](./feldera/__init__.py) for an example on how to use the SDK.
-
 ## Documentation
 
+The Python SDK documentation is available at
+[Feldera Python SDK Docs](https://docs.feldera.com/python).
+
 To build the html documentation run:
 
 Ensure that you have sphinx installed. If not, install it using `pip install sphinx`.
@@ -77,27 +78,23 @@ To clean the build, run `make clean`.
 To run unit tests:
 
 ```bash
-(cd python && python3 -m unittest)
+cd python && python3 -m pytest tests/
 ```
 
-> ⚠️ Running the unit tests will **delete all existing pipelines**.
-
-The following command runs end-to-end tests.  You'll need a pipeline
-manager running at `http://localhost:8080`.  For the pipeline builder
-tests, you'll also need a broker available at `localhost:9092` and
-(from the pipelines) `redpanda:19092`.  (To change those locations,
-set the environment variables listed in `python/tests/__init__.py`.)
-
-```bash
-(cd python && python3 -m pytest tests)
-```
+- This will detect and run all test files that match the pattern `test_*.py` or
+  `*_test.py`.
+- By default, the tests expect a running Feldera instance at `http://localhost:8080`.
+  To override the default endpoint, set the `FELDERA_BASE_URL` environment variable.
 
 To run tests from a specific file:
 
 ```bash
-(cd python && python3 -m unittest ./tests/path-to-file.py)
+(cd python && python3 -m pytest ./tests/path-to-file.py)
 ```
 
+#### Running Aggregate Tests
+
+The aggregate tests validate end-to-end correctness of SQL functionality.
 To run the aggregate tests use:
 
 ```bash
@@ -105,6 +102,38 @@ cd python
 PYTHONPATH=`pwd` python3 ./tests/aggregate_tests/main.py
 ```
 
+### Reducing Compilation Cycles
+
+To reduce redundant compilation cycles during testing:
+
+* **Inherit from `SharedTestPipeline`** instead of `unittest.TestCase`.
+* **Define DDLs** (e.g., `CREATE TABLE`, `CREATE VIEW`) in the **docstring** of each test method.
+  * All DDLs from all test functions in the class are combined and compiled into a single pipeline.
+  * If a table or view is already defined in one test, it can be used directly in others without redefinition.
+  * Ensure that all table and view names are unique within the class.
+* Use `@enterprise_only` on tests that require Enterprise features. Their DDLs will be skipped on OSS builds.
+* Use `self.set_runtime_config(...)` to override the default pipeline config.
+  * Reset it at the end using `self.reset_runtime_config()`.
+* Access the shared pipeline via `self.pipeline`.
+
+#### Example
+
+```python
+from tests.shared_test_pipeline import SharedTestPipeline
+
+class TestAverage(SharedTestPipeline):
+    def test_average(self):
+        """
+        CREATE TABLE students(id INT, name STRING);
+        CREATE MATERIALIZED VIEW v AS SELECT * FROM students;
+        """
+        ...
+        self.pipeline.start()
+        self.pipeline.input_pandas("students", df)
+        self.pipeline.wait_for_completion(True)
+        ...
+```
+
 ## Linting and formatting
 
 Use [Ruff] to run the lint checks that will be executed by the

{feldera-0.100.0 → feldera-0.101.0}/feldera.egg-info/SOURCES.txt

@@ -24,7 +24,7 @@ feldera/rest/feldera_config.py
 feldera/rest/pipeline.py
 feldera/rest/sql_table.py
 feldera/rest/sql_view.py
-tests/test_pipeline.py
 tests/test_pipeline_builder.py
-tests/test_udf.py
-tests/test_variant.py
+tests/test_shared_pipeline0.py
+tests/test_shared_pipeline1.py
+tests/test_udf.py

{feldera-0.100.0 → feldera-0.101.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 name = "feldera"
 readme = "README.md"
 description = "The feldera python client"
-version = "0.100.0"
+version = "0.101.0"
 license = { text = "MIT" }
 requires-python = ">=3.10"
 authors = [

feldera-0.101.0/tests/test_pipeline_builder.py

@@ -0,0 +1,53 @@
+import unittest
+from tests import TEST_CLIENT
+from feldera import PipelineBuilder
+
+
+class TestPipelineBuilder(unittest.TestCase):
+    def test_connector_orchestration(self):
+        sql = """
+        CREATE TABLE numbers (
+          num INT
+        ) WITH (
+            'connectors' = '[
+                {
+                    "name": "c1",
+                    "paused": true,
+                    "transport": {
+                        "name": "datagen",
+                        "config": {"plan": [{ "rate": 1, "fields": { "num": { "range": [0, 10], "strategy": "uniform" } } }]}
+                    }
+                }
+            ]'
+        );
+        """
+
+        name = "test_connector_orchestration"
+
+        pipeline = PipelineBuilder(TEST_CLIENT, name, sql=sql).create_or_replace()
+        pipeline.start()
+
+        pipeline.resume_connector("numbers", "c1")
+        stats = TEST_CLIENT.get_pipeline_stats(name)
+        c1_status = next(
+            item["paused"]
+            for item in stats["inputs"]
+            if item["endpoint_name"] == "numbers.c1"
+        )
+        assert not c1_status
+
+        pipeline.pause_connector("numbers", "c1")
+        stats = TEST_CLIENT.get_pipeline_stats(name)
+        c2_status = next(
+            item["paused"]
+            for item in stats["inputs"]
+            if item["endpoint_name"] == "numbers.c1"
+        )
+        assert c2_status
+
+        pipeline.stop(force=True)
+        pipeline.clear_storage()
+
+
+if __name__ == "__main__":
+    unittest.main()