feldera 0.99.0__tar.gz → 0.101.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of feldera might be problematic. Click here for more details.

Files changed (36) hide show
  1. {feldera-0.99.0 → feldera-0.101.0}/PKG-INFO +47 -18
  2. feldera-0.101.0/README.md +129 -0
  3. {feldera-0.99.0 → feldera-0.101.0}/feldera/__init__.py +3 -0
  4. {feldera-0.99.0 → feldera-0.101.0}/feldera/_callback_runner.py +8 -7
  5. {feldera-0.99.0 → feldera-0.101.0}/feldera/enums.py +22 -21
  6. {feldera-0.99.0 → feldera-0.101.0}/feldera/pipeline.py +10 -17
  7. {feldera-0.99.0 → feldera-0.101.0}/feldera/pipeline_builder.py +11 -5
  8. feldera-0.101.0/feldera/rest/_helpers.py +9 -0
  9. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/feldera_client.py +14 -15
  10. {feldera-0.99.0 → feldera-0.101.0}/feldera/runtime_config.py +9 -0
  11. {feldera-0.99.0 → feldera-0.101.0}/feldera.egg-info/PKG-INFO +47 -18
  12. {feldera-0.99.0 → feldera-0.101.0}/feldera.egg-info/SOURCES.txt +4 -3
  13. {feldera-0.99.0 → feldera-0.101.0}/pyproject.toml +3 -3
  14. feldera-0.101.0/tests/test_pipeline_builder.py +53 -0
  15. feldera-0.101.0/tests/test_shared_pipeline0.py +593 -0
  16. feldera-0.101.0/tests/test_shared_pipeline1.py +72 -0
  17. {feldera-0.99.0 → feldera-0.101.0}/tests/test_udf.py +0 -1
  18. feldera-0.99.0/README.md +0 -100
  19. feldera-0.99.0/tests/test_pipeline.py +0 -263
  20. feldera-0.99.0/tests/test_pipeline_builder.py +0 -1199
  21. feldera-0.99.0/tests/test_variant.py +0 -102
  22. {feldera-0.99.0 → feldera-0.101.0}/feldera/_helpers.py +0 -0
  23. {feldera-0.99.0 → feldera-0.101.0}/feldera/output_handler.py +0 -0
  24. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/__init__.py +0 -0
  25. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/_httprequests.py +0 -0
  26. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/config.py +0 -0
  27. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/errors.py +0 -0
  28. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/feldera_config.py +0 -0
  29. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/pipeline.py +0 -0
  30. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/sql_table.py +0 -0
  31. {feldera-0.99.0 → feldera-0.101.0}/feldera/rest/sql_view.py +0 -0
  32. {feldera-0.99.0 → feldera-0.101.0}/feldera/stats.py +0 -0
  33. {feldera-0.99.0 → feldera-0.101.0}/feldera.egg-info/dependency_links.txt +0 -0
  34. {feldera-0.99.0 → feldera-0.101.0}/feldera.egg-info/requires.txt +0 -0
  35. {feldera-0.99.0 → feldera-0.101.0}/feldera.egg-info/top_level.txt +0 -0
  36. {feldera-0.99.0 → feldera-0.101.0}/setup.cfg +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: feldera
3
- Version: 0.99.0
3
+ Version: 0.101.0
4
4
  Summary: The feldera python client
5
5
  Author-email: Feldera Team <dev@feldera.com>
6
6
  License: MIT
@@ -10,9 +10,9 @@ Project-URL: Repository, https://github.com/feldera/feldera
10
10
  Project-URL: Issues, https://github.com/feldera/feldera/issues
11
11
  Keywords: feldera,python
12
12
  Classifier: License :: OSI Approved :: MIT License
13
- Classifier: Programming Language :: Python :: 3.12
13
+ Classifier: Programming Language :: Python :: 3.10
14
14
  Classifier: Operating System :: OS Independent
15
- Requires-Python: >=3.12
15
+ Requires-Python: >=3.10
16
16
  Description-Content-Type: text/markdown
17
17
  Requires-Dist: requests
18
18
  Requires-Dist: pandas>=2.1.2
@@ -54,10 +54,11 @@ If you have cloned the Feldera repo, you can install the python SDK as follows:
54
54
  pip install python/
55
55
  ```
56
56
 
57
- Checkout the docs [here](./feldera/__init__.py) for an example on how to use the SDK.
58
-
59
57
  ## Documentation
60
58
 
59
+ The Python SDK documentation is available at
60
+ [Feldera Python SDK Docs](https://docs.feldera.com/python).
61
+
61
62
  To build the html documentation run:
62
63
 
63
64
  Ensure that you have sphinx installed. If not, install it using `pip install sphinx`.
@@ -77,27 +78,23 @@ To clean the build, run `make clean`.
77
78
  To run unit tests:
78
79
 
79
80
  ```bash
80
- (cd python && python3 -m unittest)
81
+ cd python && python3 -m pytest tests/
81
82
  ```
82
83
 
83
- > ⚠️ Running the unit tests will **delete all existing pipelines**.
84
-
85
- The following command runs end-to-end tests. You'll need a pipeline
86
- manager running at `http://localhost:8080`. For the pipeline builder
87
- tests, you'll also need a broker available at `localhost:9092` and
88
- (from the pipelines) `redpanda:19092`. (To change those locations,
89
- set the environment variables listed in `python/tests/__init__.py`.)
90
-
91
- ```bash
92
- (cd python && python3 -m pytest tests)
93
- ```
84
+ - This will detect and run all test files that match the pattern `test_*.py` or
85
+ `*_test.py`.
86
+ - By default, the tests expect a running Feldera instance at `http://localhost:8080`.
87
+ To override the default endpoint, set the `FELDERA_BASE_URL` environment variable.
94
88
 
95
89
  To run tests from a specific file:
96
90
 
97
91
  ```bash
98
- (cd python && python3 -m unittest ./tests/path-to-file.py)
92
+ (cd python && python3 -m pytest ./tests/path-to-file.py)
99
93
  ```
100
94
 
95
+ #### Running Aggregate Tests
96
+
97
+ The aggregate tests validate end-to-end correctness of SQL functionality.
101
98
  To run the aggregate tests use:
102
99
 
103
100
  ```bash
@@ -105,6 +102,38 @@ cd python
105
102
  PYTHONPATH=`pwd` python3 ./tests/aggregate_tests/main.py
106
103
  ```
107
104
 
105
+ ### Reducing Compilation Cycles
106
+
107
+ To reduce redundant compilation cycles during testing:
108
+
109
+ * **Inherit from `SharedTestPipeline`** instead of `unittest.TestCase`.
110
+ * **Define DDLs** (e.g., `CREATE TABLE`, `CREATE VIEW`) in the **docstring** of each test method.
111
+ * All DDLs from all test functions in the class are combined and compiled into a single pipeline.
112
+ * If a table or view is already defined in one test, it can be used directly in others without redefinition.
113
+ * Ensure that all table and view names are unique within the class.
114
+ * Use `@enterprise_only` on tests that require Enterprise features. Their DDLs will be skipped on OSS builds.
115
+ * Use `self.set_runtime_config(...)` to override the default pipeline config.
116
+ * Reset it at the end using `self.reset_runtime_config()`.
117
+ * Access the shared pipeline via `self.pipeline`.
118
+
119
+ #### Example
120
+
121
+ ```python
122
+ from tests.shared_test_pipeline import SharedTestPipeline
123
+
124
+ class TestAverage(SharedTestPipeline):
125
+ def test_average(self):
126
+ """
127
+ CREATE TABLE students(id INT, name STRING);
128
+ CREATE MATERIALIZED VIEW v AS SELECT * FROM students;
129
+ """
130
+ ...
131
+ self.pipeline.start()
132
+ self.pipeline.input_pandas("students", df)
133
+ self.pipeline.wait_for_completion(True)
134
+ ...
135
+ ```
136
+
108
137
  ## Linting and formatting
109
138
 
110
139
  Use [Ruff] to run the lint checks that will be executed by the
@@ -0,0 +1,129 @@
1
+ # Feldera Python SDK
2
+
3
+ Feldera Python is the Feldera SDK for Python developers.
4
+
5
+ ## Installation
6
+
7
+ ```bash
8
+ pip install feldera
9
+ ```
10
+
11
+ ### Installing from Github
12
+
13
+ ```bash
14
+ pip install git+https://github.com/feldera/feldera#subdirectory=python
15
+ ```
16
+
17
+ Similarly, to install from a specific branch:
18
+
19
+ ```bash
20
+ $ pip install git+https://github.com/feldera/feldera@{BRANCH_NAME}#subdirectory=python
21
+ ```
22
+
23
+ Replace `{BRANCH_NAME}` with the name of the branch you want to install from.
24
+
25
+ ### Installing from Local Directory
26
+
27
+ If you have cloned the Feldera repo, you can install the python SDK as follows:
28
+
29
+ ```bash
30
+ # the Feldera Python SDK is present inside the python/ directory
31
+ pip install python/
32
+ ```
33
+
34
+ ## Documentation
35
+
36
+ The Python SDK documentation is available at
37
+ [Feldera Python SDK Docs](https://docs.feldera.com/python).
38
+
39
+ To build the html documentation run:
40
+
41
+ Ensure that you have sphinx installed. If not, install it using `pip install sphinx`.
42
+
43
+ Then run the following commands:
44
+
45
+ ```bash
46
+ cd docs
47
+ sphinx-apidoc -o . ../feldera
48
+ make html
49
+ ```
50
+
51
+ To clean the build, run `make clean`.
52
+
53
+ ## Testing
54
+
55
+ To run unit tests:
56
+
57
+ ```bash
58
+ cd python && python3 -m pytest tests/
59
+ ```
60
+
61
+ - This will detect and run all test files that match the pattern `test_*.py` or
62
+ `*_test.py`.
63
+ - By default, the tests expect a running Feldera instance at `http://localhost:8080`.
64
+ To override the default endpoint, set the `FELDERA_BASE_URL` environment variable.
65
+
66
+ To run tests from a specific file:
67
+
68
+ ```bash
69
+ (cd python && python3 -m pytest ./tests/path-to-file.py)
70
+ ```
71
+
72
+ #### Running Aggregate Tests
73
+
74
+ The aggregate tests validate end-to-end correctness of SQL functionality.
75
+ To run the aggregate tests use:
76
+
77
+ ```bash
78
+ cd python
79
+ PYTHONPATH=`pwd` python3 ./tests/aggregate_tests/main.py
80
+ ```
81
+
82
+ ### Reducing Compilation Cycles
83
+
84
+ To reduce redundant compilation cycles during testing:
85
+
86
+ * **Inherit from `SharedTestPipeline`** instead of `unittest.TestCase`.
87
+ * **Define DDLs** (e.g., `CREATE TABLE`, `CREATE VIEW`) in the **docstring** of each test method.
88
+ * All DDLs from all test functions in the class are combined and compiled into a single pipeline.
89
+ * If a table or view is already defined in one test, it can be used directly in others without redefinition.
90
+ * Ensure that all table and view names are unique within the class.
91
+ * Use `@enterprise_only` on tests that require Enterprise features. Their DDLs will be skipped on OSS builds.
92
+ * Use `self.set_runtime_config(...)` to override the default pipeline config.
93
+ * Reset it at the end using `self.reset_runtime_config()`.
94
+ * Access the shared pipeline via `self.pipeline`.
95
+
96
+ #### Example
97
+
98
+ ```python
99
+ from tests.shared_test_pipeline import SharedTestPipeline
100
+
101
+ class TestAverage(SharedTestPipeline):
102
+ def test_average(self):
103
+ """
104
+ CREATE TABLE students(id INT, name STRING);
105
+ CREATE MATERIALIZED VIEW v AS SELECT * FROM students;
106
+ """
107
+ ...
108
+ self.pipeline.start()
109
+ self.pipeline.input_pandas("students", df)
110
+ self.pipeline.wait_for_completion(True)
111
+ ...
112
+ ```
113
+
114
+ ## Linting and formatting
115
+
116
+ Use [Ruff] to run the lint checks that will be executed by the
117
+ precommit hook when a PR is submitted:
118
+
119
+ ```bash
120
+ ruff check python/
121
+ ```
122
+
123
+ To reformat the code in the same way as the precommit hook:
124
+
125
+ ```bash
126
+ ruff format
127
+ ```
128
+
129
+ [Ruff]: https://github.com/astral-sh/ruff
@@ -1,6 +1,9 @@
1
1
  from feldera.rest.feldera_client import FelderaClient as FelderaClient
2
2
  from feldera.pipeline import Pipeline as Pipeline
3
3
  from feldera.pipeline_builder import PipelineBuilder as PipelineBuilder
4
+ from feldera.rest._helpers import client_version
5
+
6
+ __version__ = client_version()
4
7
 
5
8
  import pretty_errors
6
9
 
@@ -54,12 +54,12 @@ class CallbackRunner(Thread):
54
54
  )
55
55
 
56
56
  # by default, we assume that the pipeline has been started
57
- ack: _CallbackRunnerInstruction = _CallbackRunnerInstruction.PipelineStarted
57
+ ack = _CallbackRunnerInstruction.PipelineStarted
58
58
 
59
59
  # if there is Queue, we wait for the instruction to start the pipeline
60
60
  # this means that we are listening to the pipeline before running it, therefore, all data should be received
61
61
  if self.queue:
62
- ack: _CallbackRunnerInstruction = self.queue.get()
62
+ ack = self.queue.get()
63
63
 
64
64
  match ack:
65
65
  # if the pipeline has actually been started, we start a listener
@@ -77,11 +77,12 @@ class CallbackRunner(Thread):
77
77
 
78
78
  for chunk in gen_obj:
79
79
  chunk: dict = chunk
80
- data: list[dict] = chunk.get("json_data")
81
- seq_no: int = chunk.get("sequence_number")
82
-
83
- if data is not None:
84
- self.callback(dataframe_from_response([data], schema), seq_no)
80
+ data: Optional[list[dict]] = chunk.get("json_data")
81
+ seq_no: Optional[int] = chunk.get("sequence_number")
82
+ if data is not None and seq_no is not None:
83
+ self.callback(
84
+ dataframe_from_response([data], self.schema), seq_no
85
+ )
85
86
 
86
87
  if self.queue:
87
88
  try:
@@ -36,27 +36,28 @@ class BuildMode(Enum):
36
36
 
37
37
  class PipelineStatus(Enum):
38
38
  """
39
- Represents the state that this pipeline is currently in.
40
-
41
- .. code-block:: text
42
-
43
- Stopped ◄─────────── Stopping ◄───── All states can transition
44
- │ ▲ to Stopping by either:
45
- /start or /pause │ │ (1) user calling /stop?force=true, or;
46
- ▼ │ (2) pipeline encountering a fatal
47
- ⌛Provisioning Suspending resource or runtime error,
48
- │ ▲ having the system call /stop?force=true
49
- ▼ │ /stop effectively
50
- ⌛Initializing ─────────────┤ ?force=false
51
- │ │
52
- ┌─────────┼────────────────────┴─────┐
53
- │ ▼ │
54
- │ Paused ◄──────► Unavailable │
55
- │ │ ▲ ▲ │
56
- │ /start │ │ /pause │ │
57
- │ ▼ │ │ │
58
- │ Running ◄─────────────┘ │
59
- └────────────────────────────────────┘
39
+ Represents the state that this pipeline is currently in.
40
+
41
+ .. code-block:: text
42
+
43
+ Stopped ◄─────────── Stopping ◄───── All states can transition
44
+ │ ▲ to Stopping by either:
45
+ /start or /pause │ │ (1) user calling /stop?force=true, or;
46
+ ▼ │ (2) pipeline encountering a fatal
47
+ ⌛Provisioning Suspending resource or runtime error,
48
+ │ ▲ having the system call /stop?force=true
49
+ ▼ │ /stop effectively
50
+ ⌛Initializing ─────────────┤ ?force=false
51
+ │ │
52
+ ┌─────────┼────────────────────┴─────┐
53
+ │ ▼ │
54
+ │ Paused ◄──────► Unavailable │
55
+ │ │ ▲ ▲ │
56
+ │ /start │ │ /pause │ │
57
+ │ ▼ │ │ │
58
+ │ Running ◄─────────────┘ │
59
+ └────────────────────────────────────┘
60
+
60
61
  """
61
62
 
62
63
  NOT_FOUND = 0
@@ -111,9 +111,7 @@ class Pipeline:
111
111
  tbl.name.lower() for tbl in pipeline.tables
112
112
  ]:
113
113
  raise ValueError(
114
- f"Cannot push to table '{
115
- table_name
116
- }': table with this name does not exist in the '{self.name}' pipeline"
114
+ f"Cannot push to table '{table_name}': table with this name does not exist in the '{self.name}' pipeline"
117
115
  )
118
116
  else:
119
117
  # consider validating the schema here
@@ -308,14 +306,10 @@ class Pipeline:
308
306
  elapsed = time.monotonic() - start_time
309
307
  if elapsed > timeout_s:
310
308
  raise TimeoutError(
311
- f"timeout ({timeout_s}s) reached while waiting for pipeline '{
312
- self.name
313
- }' to complete"
309
+ f"timeout ({timeout_s}s) reached while waiting for pipeline '{self.name}' to complete"
314
310
  )
315
311
  logging.debug(
316
- f"waiting for pipeline {self.name} to complete: elapsed time {
317
- elapsed
318
- }s, timeout: {timeout_s}s"
312
+ f"waiting for pipeline {self.name} to complete: elapsed time {elapsed}s, timeout: {timeout_s}s"
319
313
  )
320
314
 
321
315
  pipeline_complete: bool = self.stats().global_metrics.pipeline_complete
@@ -402,15 +396,11 @@ method or use `Pipeline.resume()` to resume a paused pipeline."""
402
396
  """
403
397
  if idle_interval_s > timeout_s:
404
398
  raise ValueError(
405
- f"idle interval ({idle_interval_s}s) cannot be larger than timeout ({
406
- timeout_s
407
- }s)"
399
+ f"idle interval ({idle_interval_s}s) cannot be larger than timeout ({timeout_s}s)"
408
400
  )
409
401
  if poll_interval_s > timeout_s:
410
402
  raise ValueError(
411
- f"poll interval ({poll_interval_s}s) cannot be larger than timeout ({
412
- timeout_s
413
- }s)"
403
+ f"poll interval ({poll_interval_s}s) cannot be larger than timeout ({timeout_s}s)"
414
404
  )
415
405
  if poll_interval_s > idle_interval_s:
416
406
  raise ValueError(
@@ -483,10 +473,13 @@ metrics"""
483
473
  pipeline to stop.
484
474
  """
485
475
 
486
- if len(self.views_tx) > 0:
487
- for _, queue in self.views_tx.pop().items():
476
+ for view_queue in self.views_tx:
477
+ for _, queue in view_queue.items():
488
478
  # sends a message to the callback runner to stop listening
489
479
  queue.put(_CallbackRunnerInstruction.RanToCompletion)
480
+
481
+ if len(self.views_tx) > 0:
482
+ for view_name, queue in self.views_tx.pop().items():
490
483
  # block until the callback runner has been stopped
491
484
  queue.join()
492
485
 
@@ -2,7 +2,7 @@ from feldera.rest.feldera_client import FelderaClient
2
2
  from feldera.rest.pipeline import Pipeline as InnerPipeline
3
3
  from feldera.pipeline import Pipeline
4
4
  from feldera.enums import CompilationProfile
5
- from feldera.runtime_config import RuntimeConfig, Resources
5
+ from feldera.runtime_config import RuntimeConfig
6
6
  from feldera.rest.errors import FelderaAPIError
7
7
 
8
8
 
@@ -29,7 +29,7 @@ class PipelineBuilder:
29
29
  udf_toml: str = "",
30
30
  description: str = "",
31
31
  compilation_profile: CompilationProfile = CompilationProfile.OPTIMIZED,
32
- runtime_config: RuntimeConfig = RuntimeConfig(resources=Resources()),
32
+ runtime_config: RuntimeConfig = RuntimeConfig.default(),
33
33
  ):
34
34
  self.client: FelderaClient = client
35
35
  self.name: str | None = name
@@ -50,8 +50,12 @@ class PipelineBuilder:
50
50
  if self.name is None or self.sql is None:
51
51
  raise ValueError("Name and SQL are required to create a pipeline")
52
52
 
53
- if self.client.get_pipeline(self.name) is not None:
54
- raise RuntimeError(f"Pipeline with name {self.name} already exists")
53
+ try:
54
+ if self.client.get_pipeline(self.name) is not None:
55
+ raise RuntimeError(f"Pipeline with name {self.name} already exists")
56
+ except FelderaAPIError as err:
57
+ if err.error_code != "UnknownPipelineName":
58
+ raise err
55
59
 
56
60
  inner = InnerPipeline(
57
61
  self.name,
@@ -62,7 +66,9 @@ class PipelineBuilder:
62
66
  program_config={
63
67
  "profile": self.compilation_profile.value,
64
68
  },
65
- runtime_config=self.runtime_config.__dict__,
69
+ runtime_config=dict(
70
+ (k, v) for k, v in self.runtime_config.__dict__.items() if v is not None
71
+ ),
66
72
  )
67
73
 
68
74
  inner = self.client.create_pipeline(inner)
@@ -0,0 +1,9 @@
1
+ def client_version() -> str:
2
+ from importlib.metadata import version, PackageNotFoundError
3
+
4
+ try:
5
+ version = version("feldera")
6
+ except PackageNotFoundError:
7
+ version = "unknown"
8
+
9
+ return version
@@ -11,6 +11,7 @@ from feldera.rest.feldera_config import FelderaConfig
11
11
  from feldera.rest.errors import FelderaTimeoutError
12
12
  from feldera.rest.pipeline import Pipeline
13
13
  from feldera.rest._httprequests import HttpRequests
14
+ from feldera.rest._helpers import client_version
14
15
 
15
16
 
16
17
  def _validate_no_none_keys_in_map(data):
@@ -63,7 +64,13 @@ class FelderaClient:
63
64
  self.http = HttpRequests(self.config)
64
65
 
65
66
  try:
66
- self.pipelines()
67
+ config = self.get_config()
68
+ version = client_version()
69
+ if config.version != version:
70
+ logging.warn(
71
+ f"Client is on version {version} while server is at "
72
+ f"{config.version}. There could be incompatibilities."
73
+ )
67
74
  except Exception as e:
68
75
  logging.error(f"Failed to connect to Feldera API: {e}")
69
76
  raise e
@@ -358,9 +365,7 @@ Reason: The pipeline is in a STOPPED state due to the following error:
358
365
  time.sleep(0.1)
359
366
 
360
367
  raise FelderaTimeoutError(
361
- f"timeout error: pipeline '{pipeline_name}' did not stop in {
362
- timeout_s
363
- } seconds"
368
+ f"timeout error: pipeline '{pipeline_name}' did not stop in {timeout_s} seconds"
364
369
  )
365
370
 
366
371
  def clear_storage(self, pipeline_name: str, timeout_s: Optional[float] = 300):
@@ -394,9 +399,7 @@ Reason: The pipeline is in a STOPPED state due to the following error:
394
399
  time.sleep(0.1)
395
400
 
396
401
  raise FelderaTimeoutError(
397
- f"timeout error: pipeline '{pipeline_name}' did not clear storage in {
398
- timeout_s
399
- } seconds"
402
+ f"timeout error: pipeline '{pipeline_name}' did not clear storage in {timeout_s} seconds"
400
403
  )
401
404
 
402
405
  def checkpoint_pipeline(self, pipeline_name: str) -> int:
@@ -451,11 +454,11 @@ Reason: The pipeline is in a STOPPED state due to the following error:
451
454
  pipeline_name: str,
452
455
  table_name: str,
453
456
  format: str,
454
- data: list[list | str | dict] | dict,
457
+ data: list[list | str | dict] | dict | str,
455
458
  array: bool = False,
456
459
  force: bool = False,
457
460
  update_format: str = "raw",
458
- json_flavor: str = None,
461
+ json_flavor: Optional[str] = None,
459
462
  serialize: bool = True,
460
463
  ):
461
464
  """
@@ -702,9 +705,7 @@ Reason: The pipeline is in a STOPPED state due to the following error:
702
705
  """
703
706
 
704
707
  self.http.post(
705
- path=f"/pipelines/{pipeline_name}/tables/{table_name}/connectors/{
706
- connector_name
707
- }/pause",
708
+ path=f"/pipelines/{pipeline_name}/tables/{table_name}/connectors/{connector_name}/pause",
708
709
  )
709
710
 
710
711
  def resume_connector(
@@ -728,9 +729,7 @@ Reason: The pipeline is in a STOPPED state due to the following error:
728
729
  """
729
730
 
730
731
  self.http.post(
731
- path=f"/pipelines/{pipeline_name}/tables/{table_name}/connectors/{
732
- connector_name
733
- }/start",
732
+ path=f"/pipelines/{pipeline_name}/tables/{table_name}/connectors/{connector_name}/start",
734
733
  )
735
734
 
736
735
  def get_config(self) -> FelderaConfig:
@@ -1,3 +1,4 @@
1
+ import os
1
2
  from typing import Optional, Any, Mapping
2
3
 
3
4
 
@@ -72,6 +73,7 @@ class RuntimeConfig:
72
73
  clock_resolution_usecs: Optional[int] = None,
73
74
  provisioning_timeout_secs: Optional[int] = None,
74
75
  resources: Optional[Resources] = None,
76
+ runtime_version: Optional[str] = None,
75
77
  ):
76
78
  self.workers = workers
77
79
  self.tracing = tracing
@@ -81,6 +83,9 @@ class RuntimeConfig:
81
83
  self.min_batch_size_records = min_batch_size_records
82
84
  self.clock_resolution_usecs = clock_resolution_usecs
83
85
  self.provisioning_timeout_secs = provisioning_timeout_secs
86
+ self.runtime_version = runtime_version or os.environ.get(
87
+ "FELDERA_RUNTIME_VERSION"
88
+ )
84
89
  if resources is not None:
85
90
  self.resources = resources.__dict__
86
91
  if isinstance(storage, bool):
@@ -88,6 +93,10 @@ class RuntimeConfig:
88
93
  if isinstance(storage, Storage):
89
94
  self.storage = storage.__dict__
90
95
 
96
+ @staticmethod
97
+ def default() -> "RuntimeConfig":
98
+ return RuntimeConfig(resources=Resources())
99
+
91
100
  @classmethod
92
101
  def from_dict(cls, d: Mapping[str, Any]):
93
102
  """
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: feldera
3
- Version: 0.99.0
3
+ Version: 0.101.0
4
4
  Summary: The feldera python client
5
5
  Author-email: Feldera Team <dev@feldera.com>
6
6
  License: MIT
@@ -10,9 +10,9 @@ Project-URL: Repository, https://github.com/feldera/feldera
10
10
  Project-URL: Issues, https://github.com/feldera/feldera/issues
11
11
  Keywords: feldera,python
12
12
  Classifier: License :: OSI Approved :: MIT License
13
- Classifier: Programming Language :: Python :: 3.12
13
+ Classifier: Programming Language :: Python :: 3.10
14
14
  Classifier: Operating System :: OS Independent
15
- Requires-Python: >=3.12
15
+ Requires-Python: >=3.10
16
16
  Description-Content-Type: text/markdown
17
17
  Requires-Dist: requests
18
18
  Requires-Dist: pandas>=2.1.2
@@ -54,10 +54,11 @@ If you have cloned the Feldera repo, you can install the python SDK as follows:
54
54
  pip install python/
55
55
  ```
56
56
 
57
- Checkout the docs [here](./feldera/__init__.py) for an example on how to use the SDK.
58
-
59
57
  ## Documentation
60
58
 
59
+ The Python SDK documentation is available at
60
+ [Feldera Python SDK Docs](https://docs.feldera.com/python).
61
+
61
62
  To build the html documentation run:
62
63
 
63
64
  Ensure that you have sphinx installed. If not, install it using `pip install sphinx`.
@@ -77,27 +78,23 @@ To clean the build, run `make clean`.
77
78
  To run unit tests:
78
79
 
79
80
  ```bash
80
- (cd python && python3 -m unittest)
81
+ cd python && python3 -m pytest tests/
81
82
  ```
82
83
 
83
- > ⚠️ Running the unit tests will **delete all existing pipelines**.
84
-
85
- The following command runs end-to-end tests. You'll need a pipeline
86
- manager running at `http://localhost:8080`. For the pipeline builder
87
- tests, you'll also need a broker available at `localhost:9092` and
88
- (from the pipelines) `redpanda:19092`. (To change those locations,
89
- set the environment variables listed in `python/tests/__init__.py`.)
90
-
91
- ```bash
92
- (cd python && python3 -m pytest tests)
93
- ```
84
+ - This will detect and run all test files that match the pattern `test_*.py` or
85
+ `*_test.py`.
86
+ - By default, the tests expect a running Feldera instance at `http://localhost:8080`.
87
+ To override the default endpoint, set the `FELDERA_BASE_URL` environment variable.
94
88
 
95
89
  To run tests from a specific file:
96
90
 
97
91
  ```bash
98
- (cd python && python3 -m unittest ./tests/path-to-file.py)
92
+ (cd python && python3 -m pytest ./tests/path-to-file.py)
99
93
  ```
100
94
 
95
+ #### Running Aggregate Tests
96
+
97
+ The aggregate tests validate end-to-end correctness of SQL functionality.
101
98
  To run the aggregate tests use:
102
99
 
103
100
  ```bash
@@ -105,6 +102,38 @@ cd python
105
102
  PYTHONPATH=`pwd` python3 ./tests/aggregate_tests/main.py
106
103
  ```
107
104
 
105
+ ### Reducing Compilation Cycles
106
+
107
+ To reduce redundant compilation cycles during testing:
108
+
109
+ * **Inherit from `SharedTestPipeline`** instead of `unittest.TestCase`.
110
+ * **Define DDLs** (e.g., `CREATE TABLE`, `CREATE VIEW`) in the **docstring** of each test method.
111
+ * All DDLs from all test functions in the class are combined and compiled into a single pipeline.
112
+ * If a table or view is already defined in one test, it can be used directly in others without redefinition.
113
+ * Ensure that all table and view names are unique within the class.
114
+ * Use `@enterprise_only` on tests that require Enterprise features. Their DDLs will be skipped on OSS builds.
115
+ * Use `self.set_runtime_config(...)` to override the default pipeline config.
116
+ * Reset it at the end using `self.reset_runtime_config()`.
117
+ * Access the shared pipeline via `self.pipeline`.
118
+
119
+ #### Example
120
+
121
+ ```python
122
+ from tests.shared_test_pipeline import SharedTestPipeline
123
+
124
+ class TestAverage(SharedTestPipeline):
125
+ def test_average(self):
126
+ """
127
+ CREATE TABLE students(id INT, name STRING);
128
+ CREATE MATERIALIZED VIEW v AS SELECT * FROM students;
129
+ """
130
+ ...
131
+ self.pipeline.start()
132
+ self.pipeline.input_pandas("students", df)
133
+ self.pipeline.wait_for_completion(True)
134
+ ...
135
+ ```
136
+
108
137
  ## Linting and formatting
109
138
 
110
139
  Use [Ruff] to run the lint checks that will be executed by the