fivetran-connector-sdk 0.8.12.1__tar.gz → 0.8.20.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- fivetran_connector_sdk-0.8.20.1/PKG-INFO +56 -0
- fivetran_connector_sdk-0.8.20.1/README.md +38 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk/__init__.py +80 -87
- fivetran_connector_sdk-0.8.20.1/src/fivetran_connector_sdk.egg-info/PKG-INFO +56 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk.egg-info/SOURCES.txt +1 -0
- fivetran_connector_sdk-0.8.12.1/PKG-INFO +0 -17
- fivetran_connector_sdk-0.8.12.1/src/fivetran_connector_sdk.egg-info/PKG-INFO +0 -17
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/pyproject.toml +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/setup.cfg +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk/protos/__init__.py +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk/protos/common_pb2.py +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk/protos/common_pb2.pyi +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk/protos/common_pb2_grpc.py +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk/protos/connector_sdk_pb2.py +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk/protos/connector_sdk_pb2.pyi +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk/protos/connector_sdk_pb2_grpc.py +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk.egg-info/dependency_links.txt +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk.egg-info/entry_points.txt +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk.egg-info/requires.txt +0 -0
- {fivetran_connector_sdk-0.8.12.1 → fivetran_connector_sdk-0.8.20.1}/src/fivetran_connector_sdk.egg-info/top_level.txt +0 -0
@@ -0,0 +1,56 @@
|
|
1
|
+
Metadata-Version: 2.1
|
2
|
+
Name: fivetran_connector_sdk
|
3
|
+
Version: 0.8.20.1
|
4
|
+
Summary: Build custom connectors on Fivetran platform
|
5
|
+
Author-email: Fivetran <developers@fivetran.com>
|
6
|
+
Project-URL: Homepage, https://fivetran.com/docs/connectors/connector-sdk
|
7
|
+
Project-URL: Github, https://github.com/fivetran/fivetran_connector_sdk
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
10
|
+
Classifier: Operating System :: OS Independent
|
11
|
+
Requires-Python: >=3.9
|
12
|
+
Description-Content-Type: text/markdown
|
13
|
+
Requires-Dist: grpcio==1.60.1
|
14
|
+
Requires-Dist: grpcio-tools==1.60.1
|
15
|
+
Requires-Dist: requests==2.31.0
|
16
|
+
Requires-Dist: get_pypi_latest_version==0.0.12
|
17
|
+
Requires-Dist: pipreqs==0.5.0
|
18
|
+
|
19
|
+
# **fivetran-connector-sdk**
|
20
|
+
The *fivetran-connector-sdk* is a Python module that enables you to write custom data connectors and deploy them as an extension of [Fivetran](https://www.fivetran.com/). Fivetran automatically manages running the connectors on your scheduled frequency and manages the required compute resources.
|
21
|
+
|
22
|
+
The Connector SDK service is the best fit for the following use cases:
|
23
|
+
- Fivetran doesn't have a connector for your source and is unlikely to support it soon.
|
24
|
+
- You are using private APIs, custom applications, unsupported file formats, or those that require pre-processing.
|
25
|
+
- You have sensitive data that needs filtering or anonymizing before entering the destination.
|
26
|
+
- You don't want to introduce a 3rd party into your pipeline (e.g., to host a custom function).
|
27
|
+
|
28
|
+
To learn more, see our [Connector SDK documentation](https://fivetran.com/docs/connectors/connector-sdk)
|
29
|
+
|
30
|
+
## **Install**
|
31
|
+
|
32
|
+
pip install fivetran-connector-sdk
|
33
|
+
|
34
|
+
## **Requirements**
|
35
|
+
- Python ≥3.9 and ≤3.12
|
36
|
+
- Operating System:
|
37
|
+
- Windows 10 or later
|
38
|
+
- MacOS 13 (Ventura) or later
|
39
|
+
|
40
|
+
## **Getting started**
|
41
|
+
See [Quickstart guide](https://fivetran.com/docs/connectors/connector-sdk/quickstart-guide) to get started.
|
42
|
+
|
43
|
+
## **Usage**
|
44
|
+
Import required classes from fivetran_connector_sdk to begin writing your custom data connector
|
45
|
+
|
46
|
+
```python
|
47
|
+
from fivetran_connector_sdk import Connector
|
48
|
+
from fivetran_connector_sdk import Logging as log
|
49
|
+
from fivetran_connector_sdk import Operations as op
|
50
|
+
```
|
51
|
+
See our [Technical Reference](https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update) and [Best Practices](https://fivetran.com/docs/connectors/connector-sdk/best-practices) documentation for more details.
|
52
|
+
|
53
|
+
You can also refer to our [existing examples](https://github.com/fivetran/fivetran_connector_sdk) to better understand Fivetran's Connector SDK and start creating custom data connectors.
|
54
|
+
|
55
|
+
## **Maintenance**
|
56
|
+
This package is actively maintained by Fivetran Developers. Please reach out to our [Support team](https://support.fivetran.com/hc/en-us) for any inquiries.
|
@@ -0,0 +1,38 @@
|
|
1
|
+
# **fivetran-connector-sdk**
|
2
|
+
The *fivetran-connector-sdk* is a Python module that enables you to write custom data connectors and deploy them as an extension of [Fivetran](https://www.fivetran.com/). Fivetran automatically manages running the connectors on your scheduled frequency and manages the required compute resources.
|
3
|
+
|
4
|
+
The Connector SDK service is the best fit for the following use cases:
|
5
|
+
- Fivetran doesn't have a connector for your source and is unlikely to support it soon.
|
6
|
+
- You are using private APIs, custom applications, unsupported file formats, or those that require pre-processing.
|
7
|
+
- You have sensitive data that needs filtering or anonymizing before entering the destination.
|
8
|
+
- You don't want to introduce a 3rd party into your pipeline (e.g., to host a custom function).
|
9
|
+
|
10
|
+
To learn more, see our [Connector SDK documentation](https://fivetran.com/docs/connectors/connector-sdk)
|
11
|
+
|
12
|
+
## **Install**
|
13
|
+
|
14
|
+
pip install fivetran-connector-sdk
|
15
|
+
|
16
|
+
## **Requirements**
|
17
|
+
- Python ≥3.9 and ≤3.12
|
18
|
+
- Operating System:
|
19
|
+
- Windows 10 or later
|
20
|
+
- MacOS 13 (Ventura) or later
|
21
|
+
|
22
|
+
## **Getting started**
|
23
|
+
See [Quickstart guide](https://fivetran.com/docs/connectors/connector-sdk/quickstart-guide) to get started.
|
24
|
+
|
25
|
+
## **Usage**
|
26
|
+
Import required classes from fivetran_connector_sdk to begin writing your custom data connector
|
27
|
+
|
28
|
+
```python
|
29
|
+
from fivetran_connector_sdk import Connector
|
30
|
+
from fivetran_connector_sdk import Logging as log
|
31
|
+
from fivetran_connector_sdk import Operations as op
|
32
|
+
```
|
33
|
+
See our [Technical Reference](https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update) and [Best Practices](https://fivetran.com/docs/connectors/connector-sdk/best-practices) documentation for more details.
|
34
|
+
|
35
|
+
You can also refer to our [existing examples](https://github.com/fivetran/fivetran_connector_sdk) to better understand Fivetran's Connector SDK and start creating custom data connectors.
|
36
|
+
|
37
|
+
## **Maintenance**
|
38
|
+
This package is actively maintained by Fivetran Developers. Please reach out to our [Support team](https://support.fivetran.com/hc/en-us) for any inquiries.
|
@@ -23,7 +23,7 @@ from fivetran_connector_sdk.protos import common_pb2
|
|
23
23
|
from fivetran_connector_sdk.protos import connector_sdk_pb2
|
24
24
|
from fivetran_connector_sdk.protos import connector_sdk_pb2_grpc
|
25
25
|
|
26
|
-
__version__ = "0.8.
|
26
|
+
__version__ = "0.8.20.1"
|
27
27
|
|
28
28
|
MAC_OS = "mac"
|
29
29
|
WIN_OS = "windows"
|
@@ -39,6 +39,7 @@ OUTPUT_FILES_DIR = "files"
|
|
39
39
|
ONE_DAY_IN_SEC = 24 * 60 * 60
|
40
40
|
|
41
41
|
EXCLUDED_DIRS = ["__pycache__", "lib", "include", OUTPUT_FILES_DIR]
|
42
|
+
EXCLUDED_PIPREQS_DIRS = ["bin,etc,include,lib,Lib,lib64,Scripts"]
|
42
43
|
|
43
44
|
DEBUGGING = False
|
44
45
|
TABLES = {}
|
@@ -236,7 +237,7 @@ def check_newer_version():
|
|
236
237
|
obtainer = GetPyPiLatestVersion()
|
237
238
|
latest_version = obtainer('fivetran_connector_sdk')
|
238
239
|
if __version__ < latest_version:
|
239
|
-
print(f"[notice] A new release of 'fivetran-connector-sdk' available: {latest_version}\n" +
|
240
|
+
print(f"[notice] A new release of 'fivetran-connector-sdk' is available: {latest_version}\n" +
|
240
241
|
f"[notice] To update, run: pip install --upgrade fivetran-connector-sdk\n")
|
241
242
|
|
242
243
|
with open(last_check_file_path, 'w') as f_out:
|
@@ -279,8 +280,6 @@ def _map_data_to_columns(data: dict, columns: dict) -> dict:
|
|
279
280
|
for k, v in data.items():
|
280
281
|
if v is None:
|
281
282
|
mapped_data[k] = common_pb2.ValueType(null=True)
|
282
|
-
elif isinstance(v, list):
|
283
|
-
raise ValueError("Value type cannot be list")
|
284
283
|
elif (k in columns) and columns[k].type != common_pb2.DataType.UNSPECIFIED:
|
285
284
|
if columns[k].type == common_pb2.DataType.BOOLEAN:
|
286
285
|
mapped_data[k] = common_pb2.ValueType(bool=v)
|
@@ -325,7 +324,7 @@ def _map_data_to_columns(data: dict, columns: dict) -> dict:
|
|
325
324
|
elif columns[k].type == common_pb2.DataType.JSON:
|
326
325
|
mapped_data[k] = common_pb2.ValueType(json=json.dumps(v))
|
327
326
|
else:
|
328
|
-
raise ValueError(f"
|
327
|
+
raise ValueError(f"Unsupported data type encountered: {columns[k].type}. Please use valid data types.")
|
329
328
|
else:
|
330
329
|
# We can infer type from the value
|
331
330
|
if isinstance(v, int):
|
@@ -340,7 +339,7 @@ def _map_data_to_columns(data: dict, columns: dict) -> dict:
|
|
340
339
|
elif isinstance(v, bytes):
|
341
340
|
mapped_data[k] = common_pb2.ValueType(binary=v)
|
342
341
|
elif isinstance(v, list):
|
343
|
-
raise ValueError("
|
342
|
+
raise ValueError("Values for the columns cannot be of type 'list'. Please ensure that all values are of a supported type. Reference: https://fivetran.com/docs/connectors/connector-sdk/technical-reference#supporteddatatypes")
|
344
343
|
elif isinstance(v, dict):
|
345
344
|
mapped_data[k] = common_pb2.ValueType(json=json.dumps(v))
|
346
345
|
elif isinstance(v, str):
|
@@ -371,11 +370,11 @@ def _yield_check(stack):
|
|
371
370
|
calling_code = stack[1].code_context[0]
|
372
371
|
if f"{called_method}(" in calling_code:
|
373
372
|
if 'yield' not in calling_code:
|
374
|
-
print(f"
|
373
|
+
print(f"SEVERE: Please add 'yield' to '{called_method}' operation on line {stack[1].lineno} in file '{stack[1].filename}'")
|
375
374
|
os._exit(1)
|
376
375
|
else:
|
377
376
|
# This should never happen
|
378
|
-
raise RuntimeError(f"
|
377
|
+
raise RuntimeError(f"The '{called_method}' function is missing in the connector. Please ensure that the '{called_method}' function is properly defined in your code to proceed. Reference: https://fivetran.com/docs/connectors/connector-sdk/technical-reference#technicaldetailsmethods")
|
379
378
|
|
380
379
|
|
381
380
|
def _check_dict(incoming: dict, string_only: bool = False) -> dict:
|
@@ -392,12 +391,12 @@ def _check_dict(incoming: dict, string_only: bool = False) -> dict:
|
|
392
391
|
return {}
|
393
392
|
|
394
393
|
if not isinstance(incoming, dict):
|
395
|
-
raise ValueError("Configuration
|
394
|
+
raise ValueError("Configuration must be provided as a JSON dictionary. Please check your input. Reference: https://fivetran.com/docs/connectors/connector-sdk/detailed-guide#workingwithconfigurationjsonfile")
|
396
395
|
|
397
396
|
if string_only:
|
398
397
|
for k, v in incoming.items():
|
399
398
|
if not isinstance(v, str):
|
400
|
-
print("
|
399
|
+
print("SEVERE: All values in the configuration must be STRING. Please check your configuration and ensure that every value is a STRING.")
|
401
400
|
os._exit(1)
|
402
401
|
|
403
402
|
return incoming
|
@@ -417,28 +416,6 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
417
416
|
self.configuration = None
|
418
417
|
self.state = None
|
419
418
|
|
420
|
-
# Call this method to unpause and start a connector
|
421
|
-
def start(self, deploy_key: str, group: str, connection: str):
|
422
|
-
"""Starts the connector with the given deployment key, group name, and connection schema name.
|
423
|
-
|
424
|
-
Args:
|
425
|
-
deploy_key (str): The deployment key.
|
426
|
-
group (str): The group name.
|
427
|
-
connection (str): The connection schema name.
|
428
|
-
"""
|
429
|
-
if not deploy_key: print("ERROR: Missing deploy key"); os._exit(1)
|
430
|
-
if not connection: print("ERROR: Missing connection name"); os._exit(1)
|
431
|
-
|
432
|
-
group_id, group_name = self.__get_group_info(group, deploy_key)
|
433
|
-
connection_id = self.__get_connection_id(connection, group, group_id, deploy_key)
|
434
|
-
if not self.__unpause_connection():
|
435
|
-
print(f"WARNING: Unable to unpause connection '{connection}'")
|
436
|
-
os._exit(1)
|
437
|
-
|
438
|
-
if not self.__force_sync(connection_id, connection, deploy_key):
|
439
|
-
print(f"WARNING: Unable to start sync on connection '{connection}'")
|
440
|
-
os._exit(1)
|
441
|
-
|
442
419
|
@staticmethod
|
443
420
|
def __unpause_connection(id: str, deploy_key: str) -> bool:
|
444
421
|
"""Unpauses the connection with the given ID and deployment key.
|
@@ -495,7 +472,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
495
472
|
is_deploy (bool): If `True`, the method will exit the process on critical errors.
|
496
473
|
|
497
474
|
"""
|
498
|
-
subprocess.run(["pipreqs", "--savepath", "tmp_requirements.txt"], text=True, check=True)
|
475
|
+
subprocess.run(["pipreqs", "--savepath", "tmp_requirements.txt", "--ignore"] + EXCLUDED_PIPREQS_DIRS, text=True, check=True)
|
499
476
|
tmp_requirements_file_path = os.path.join(project_path, 'tmp_requirements.txt')
|
500
477
|
|
501
478
|
tmp_requirements = self.fetch_requirements_as_dict(self, tmp_requirements_file_path)
|
@@ -553,9 +530,9 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
553
530
|
connection (str): The connection name.
|
554
531
|
configuration (dict): The configuration dictionary.
|
555
532
|
"""
|
556
|
-
if not deploy_key: print("
|
557
|
-
if not connection: print("
|
558
|
-
_check_dict(configuration)
|
533
|
+
if not deploy_key: print("SEVERE: The Fivetran API key is missing. Please provide a valid Fivetran API key to create the connector."); os._exit(1)
|
534
|
+
if not connection: print("SEVERE: The connection name is missing. Please provide a valid connection name to create the connector."); os._exit(1)
|
535
|
+
_check_dict(configuration, True)
|
559
536
|
|
560
537
|
secrets_list = []
|
561
538
|
if configuration:
|
@@ -572,7 +549,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
572
549
|
self.validate_requirements_file(project_path, True)
|
573
550
|
|
574
551
|
group_id, group_name = self.__get_group_info(group, deploy_key)
|
575
|
-
print(f"Deploying '{project_path}' to '{
|
552
|
+
print(f"INFO: Deploying '{project_path}' to connector '{connection}' in destination '{group_name}'.\n")
|
576
553
|
upload_file_path = self.__create_upload_file(project_path)
|
577
554
|
upload_result = self.__upload(upload_file_path, deploy_key, group_id, connection)
|
578
555
|
os.remove(upload_file_path)
|
@@ -580,15 +557,15 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
580
557
|
os._exit(1)
|
581
558
|
connection_id = self.__get_connection_id(connection, group, group_id, deploy_key)
|
582
559
|
if connection_id:
|
583
|
-
print(f"
|
560
|
+
print(f"INFO: The connection '{connection}' already exists in destination '{group}', updating the existing connector... ", end="", flush=True)
|
584
561
|
self.__update_connection(connection_id, connection, group_name, connection_config, deploy_key)
|
585
562
|
print("✓")
|
586
563
|
else:
|
587
564
|
response = self.__create_connection(deploy_key, group_id, connection_config)
|
588
565
|
if response.ok:
|
589
|
-
print(f"
|
566
|
+
print(f"INFO: Connection named '{connection}' has been created successfully.\n")
|
590
567
|
else:
|
591
|
-
print(f"
|
568
|
+
print(f"SEVERE: Unable to create a new Connection, failed with error: {response.json()['message']}")
|
592
569
|
os._exit(1)
|
593
570
|
|
594
571
|
@staticmethod
|
@@ -629,7 +606,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
629
606
|
})
|
630
607
|
|
631
608
|
if not resp.ok:
|
632
|
-
print(f"
|
609
|
+
print(f"SEVERE: Unable to update Connection '{name}' in destination '{group}', failed with error: '{response.json()['message']}'.")
|
633
610
|
os._exit(1)
|
634
611
|
|
635
612
|
@staticmethod
|
@@ -649,7 +626,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
649
626
|
headers={"Authorization": f"Basic {deploy_key}"},
|
650
627
|
params={"schema": name})
|
651
628
|
if not resp.ok:
|
652
|
-
print(f"
|
629
|
+
print(f"SEVERE: Unable to fetch connection list in destination '{group}'")
|
653
630
|
os._exit(1)
|
654
631
|
|
655
632
|
if resp.json()['data']['items']:
|
@@ -690,7 +667,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
690
667
|
Returns:
|
691
668
|
str: The path to the upload file.
|
692
669
|
"""
|
693
|
-
print("Packaging project for upload
|
670
|
+
print("INFO: Packaging your project for upload...")
|
694
671
|
zip_file_path = self.__zip_folder(project_path)
|
695
672
|
print("✓")
|
696
673
|
return zip_file_path
|
@@ -717,7 +694,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
717
694
|
zipf.write(file_path, arcname)
|
718
695
|
|
719
696
|
if not connector_file_exists:
|
720
|
-
print("
|
697
|
+
print("SEVERE: The 'connector.py' file is missing. Please ensure that 'connector.py' is present in your project directory, and that the file name is in lowercase letters. All custom connectors require this file because Fivetran calls it to start a sync.")
|
721
698
|
os._exit(1)
|
722
699
|
return upload_filepath
|
723
700
|
|
@@ -759,7 +736,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
759
736
|
Returns:
|
760
737
|
bool: True if the upload was successful, False otherwise.
|
761
738
|
"""
|
762
|
-
print("Uploading project
|
739
|
+
print("INFO: Uploading your project...", end="", flush=True)
|
763
740
|
response = rq.post(f"https://api.fivetran.com/v2/deploy/{group_id}/{connection}",
|
764
741
|
files={'file': open(local_path, 'rb')},
|
765
742
|
headers={"Authorization": f"Basic {deploy_key}"})
|
@@ -767,7 +744,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
767
744
|
print("✓")
|
768
745
|
return True
|
769
746
|
|
770
|
-
print("
|
747
|
+
print("SEVERE: Unable to upload the project, failed with error: ", response.reason)
|
771
748
|
return False
|
772
749
|
|
773
750
|
@staticmethod
|
@@ -797,31 +774,48 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
797
774
|
Returns:
|
798
775
|
tuple[str, str]: A tuple containing the group ID and group name.
|
799
776
|
"""
|
800
|
-
|
801
|
-
|
777
|
+
groups_url = "https://api.fivetran.com/v1/groups"
|
778
|
+
|
779
|
+
params = {"limit": 500}
|
780
|
+
headers = {"Authorization": f"Basic {deploy_key}"}
|
781
|
+
resp = rq.get(groups_url, headers=headers, params=params)
|
802
782
|
|
803
783
|
if not resp.ok:
|
804
|
-
print(
|
784
|
+
print(
|
785
|
+
f"SEVERE: Unable to fetch list of destination names, status code = {resp.status_code}")
|
805
786
|
os._exit(1)
|
806
787
|
|
807
|
-
|
808
|
-
groups =
|
788
|
+
data = resp.json().get("data", {})
|
789
|
+
groups = data.get("items")
|
790
|
+
|
809
791
|
if not groups:
|
810
|
-
print("
|
792
|
+
print("SEVERE: No destinations defined in the account")
|
811
793
|
os._exit(1)
|
812
794
|
|
813
|
-
if
|
814
|
-
|
815
|
-
|
816
|
-
|
817
|
-
print(
|
795
|
+
if not group:
|
796
|
+
if len(groups) == 1:
|
797
|
+
return groups[0]['id'], groups[0]['name']
|
798
|
+
else:
|
799
|
+
print(
|
800
|
+
"SEVERE: Destination name is required when there are multiple destinations in the account")
|
818
801
|
os._exit(1)
|
819
|
-
|
820
|
-
|
821
|
-
|
822
|
-
|
823
|
-
|
824
|
-
|
802
|
+
else:
|
803
|
+
while True:
|
804
|
+
for grp in groups:
|
805
|
+
if grp['name'] == group:
|
806
|
+
return grp['id'], grp['name']
|
807
|
+
|
808
|
+
next_cursor = data.get("next_cursor")
|
809
|
+
if not next_cursor:
|
810
|
+
break
|
811
|
+
|
812
|
+
params = {"cursor": next_cursor, "limit": 500}
|
813
|
+
resp = rq.get(groups_url, headers=headers, params=params)
|
814
|
+
data = resp.json().get("data", {})
|
815
|
+
groups = data.get("items", [])
|
816
|
+
|
817
|
+
print(
|
818
|
+
f"SEVERE: The specified destination '{group}' was not found in your account.")
|
825
819
|
os._exit(1)
|
826
820
|
|
827
821
|
# Call this method to run the connector in production
|
@@ -899,17 +893,17 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
899
893
|
download_filename = f"sdk-connector-tester-{os_name}-{TESTER_VERSION}.zip"
|
900
894
|
download_filepath = os.path.join(tester_root_dir, download_filename)
|
901
895
|
try:
|
902
|
-
print(f"Downloading connector tester version {TESTER_VERSION}
|
896
|
+
print(f"INFO: Downloading connector tester version: {TESTER_VERSION} ", end="", flush=True)
|
903
897
|
download_url = f"https://github.com/fivetran/fivetran_sdk_tools/releases/download/{TESTER_VERSION}/{download_filename}"
|
904
898
|
r = rq.get(download_url)
|
905
899
|
if r.ok:
|
906
900
|
with open(download_filepath, 'wb') as fo:
|
907
901
|
fo.write(r.content)
|
908
902
|
else:
|
909
|
-
print(f"\
|
903
|
+
print(f"\nSEVERE: Failed to download the connector tester. Please check your access permissions or try again later ( status code: {r.status_code}), url: {download_url}")
|
910
904
|
os._exit(1)
|
911
905
|
except:
|
912
|
-
print(f"\
|
906
|
+
print(f"\nSEVERE: Failed to download the connector tester. Error details: {traceback.format_exc()}")
|
913
907
|
os._exit(1)
|
914
908
|
|
915
909
|
try:
|
@@ -924,13 +918,13 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
924
918
|
os.chmod(java_exe, st.st_mode | stat.S_IEXEC)
|
925
919
|
print("✓")
|
926
920
|
except:
|
927
|
-
print(f"\
|
921
|
+
print(f"\nSEVERE: Failed to install the connector tester. Error details: ", traceback.format_exc())
|
928
922
|
shutil.rmtree(tester_root_dir)
|
929
923
|
os._exit(1)
|
930
924
|
|
931
925
|
project_path = os.getcwd() if project_path is None else project_path
|
932
926
|
self.validate_requirements_file(project_path, False)
|
933
|
-
print(f"Debugging connector at: {project_path}")
|
927
|
+
print(f"INFO: Debugging connector at: {project_path}")
|
934
928
|
server = self.run(port, configuration, state, log_level=log_level)
|
935
929
|
|
936
930
|
# Uncomment this to run the tester manually
|
@@ -938,7 +932,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
938
932
|
|
939
933
|
error = False
|
940
934
|
try:
|
941
|
-
print(f"
|
935
|
+
print(f"INFO: Running connector tester...")
|
942
936
|
for log_msg in self.__run_tester(java_exe, tester_root_dir, project_path, port):
|
943
937
|
print(log_msg, end="")
|
944
938
|
except:
|
@@ -1121,7 +1115,7 @@ class Connector(connector_sdk_pb2_grpc.ConnectorServicer):
|
|
1121
1115
|
elif type.upper() == "JSON":
|
1122
1116
|
column.type = common_pb2.DataType.JSON
|
1123
1117
|
else:
|
1124
|
-
raise ValueError("Unrecognized column type
|
1118
|
+
raise ValueError("Unrecognized column type encountered:: ", str(type))
|
1125
1119
|
|
1126
1120
|
elif isinstance(type, dict):
|
1127
1121
|
if type['type'].upper() != "DECIMAL":
|
@@ -1190,7 +1184,7 @@ def find_connector_object(project_path) -> Connector:
|
|
1190
1184
|
if '<fivetran_connector_sdk.Connector object at' in str(obj_attr):
|
1191
1185
|
return obj_attr
|
1192
1186
|
|
1193
|
-
print("
|
1187
|
+
print("SEVERE: The connector object is missing. Please ensure that you have defined a connector object using the correct syntax in your `connector.py` file. Reference: https://fivetran.com/docs/connectors/connector-sdk/technical-reference#technicaldetailsrequiredobjectconnector")
|
1194
1188
|
sys.exit(1)
|
1195
1189
|
|
1196
1190
|
|
@@ -1202,7 +1196,7 @@ def main():
|
|
1202
1196
|
parser = argparse.ArgumentParser(allow_abbrev=False)
|
1203
1197
|
|
1204
1198
|
# Positional
|
1205
|
-
parser.add_argument("command", help="debug|
|
1199
|
+
parser.add_argument("command", help="debug|deploy|reset")
|
1206
1200
|
parser.add_argument("project_path", nargs='?', default=os.getcwd(), help="Path to connector project directory")
|
1207
1201
|
|
1208
1202
|
# Optional (Not all of these are valid with every mutually exclusive option below)
|
@@ -1230,7 +1224,7 @@ def main():
|
|
1230
1224
|
with open(json_filepath, 'r') as fi:
|
1231
1225
|
configuration = json.load(fi)
|
1232
1226
|
else:
|
1233
|
-
raise ValueError("Configuration
|
1227
|
+
raise ValueError("Configuration must be provided as a JSON file. Please check your input. Reference: https://fivetran.com/docs/connectors/connector-sdk/detailed-guide#workingwithconfigurationjsonfile")
|
1234
1228
|
else:
|
1235
1229
|
configuration = {}
|
1236
1230
|
|
@@ -1251,24 +1245,23 @@ def main():
|
|
1251
1245
|
print("WARNING: 'state' parameter is not used for 'deploy' command")
|
1252
1246
|
connector_object.deploy(args.project_path, ft_deploy_key, ft_group, ft_connection, configuration)
|
1253
1247
|
|
1254
|
-
elif args.command.lower() == "start":
|
1255
|
-
if args.port:
|
1256
|
-
print("WARNING: 'port' parameter is not used for 'deploy' command")
|
1257
|
-
if args.state:
|
1258
|
-
print("WARNING: 'state' parameter is not used for 'deploy' command")
|
1259
|
-
connector_object.start(ft_deploy_key, ft_group, ft_connection)
|
1260
|
-
|
1261
1248
|
elif args.command.lower() == "debug":
|
1262
1249
|
port = 50051 if not args.port else args.port
|
1263
1250
|
connector_object.debug(args.project_path, port, configuration, state)
|
1264
1251
|
|
1265
|
-
elif args.command.lower() == "
|
1266
|
-
|
1267
|
-
|
1268
|
-
|
1269
|
-
|
1270
|
-
|
1271
|
-
|
1252
|
+
elif args.command.lower() == "reset":
|
1253
|
+
files_path = os.path.join(args.project_path, OUTPUT_FILES_DIR)
|
1254
|
+
confirm = input("This will delete your current state and `warehouse.db` files. Do you want to continue? (Y/N): ")
|
1255
|
+
if confirm.lower() != "y":
|
1256
|
+
print("INFO: Reset canceled")
|
1257
|
+
else:
|
1258
|
+
try:
|
1259
|
+
if os.path.exists(files_path) and os.path.isdir(files_path):
|
1260
|
+
shutil.rmtree(files_path)
|
1261
|
+
print("INFO: Reset Successful")
|
1262
|
+
except Exception as e:
|
1263
|
+
print("ERROR: Reset Failed")
|
1264
|
+
raise e
|
1272
1265
|
|
1273
1266
|
else:
|
1274
1267
|
raise NotImplementedError("Invalid command: ", args.command)
|
@@ -0,0 +1,56 @@
|
|
1
|
+
Metadata-Version: 2.1
|
2
|
+
Name: fivetran_connector_sdk
|
3
|
+
Version: 0.8.20.1
|
4
|
+
Summary: Build custom connectors on Fivetran platform
|
5
|
+
Author-email: Fivetran <developers@fivetran.com>
|
6
|
+
Project-URL: Homepage, https://fivetran.com/docs/connectors/connector-sdk
|
7
|
+
Project-URL: Github, https://github.com/fivetran/fivetran_connector_sdk
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
9
|
+
Classifier: License :: OSI Approved :: MIT License
|
10
|
+
Classifier: Operating System :: OS Independent
|
11
|
+
Requires-Python: >=3.9
|
12
|
+
Description-Content-Type: text/markdown
|
13
|
+
Requires-Dist: grpcio==1.60.1
|
14
|
+
Requires-Dist: grpcio-tools==1.60.1
|
15
|
+
Requires-Dist: requests==2.31.0
|
16
|
+
Requires-Dist: get_pypi_latest_version==0.0.12
|
17
|
+
Requires-Dist: pipreqs==0.5.0
|
18
|
+
|
19
|
+
# **fivetran-connector-sdk**
|
20
|
+
The *fivetran-connector-sdk* is a Python module that enables you to write custom data connectors and deploy them as an extension of [Fivetran](https://www.fivetran.com/). Fivetran automatically manages running the connectors on your scheduled frequency and manages the required compute resources.
|
21
|
+
|
22
|
+
The Connector SDK service is the best fit for the following use cases:
|
23
|
+
- Fivetran doesn't have a connector for your source and is unlikely to support it soon.
|
24
|
+
- You are using private APIs, custom applications, unsupported file formats, or those that require pre-processing.
|
25
|
+
- You have sensitive data that needs filtering or anonymizing before entering the destination.
|
26
|
+
- You don't want to introduce a 3rd party into your pipeline (e.g., to host a custom function).
|
27
|
+
|
28
|
+
To learn more, see our [Connector SDK documentation](https://fivetran.com/docs/connectors/connector-sdk)
|
29
|
+
|
30
|
+
## **Install**
|
31
|
+
|
32
|
+
pip install fivetran-connector-sdk
|
33
|
+
|
34
|
+
## **Requirements**
|
35
|
+
- Python ≥3.9 and ≤3.12
|
36
|
+
- Operating System:
|
37
|
+
- Windows 10 or later
|
38
|
+
- MacOS 13 (Ventura) or later
|
39
|
+
|
40
|
+
## **Getting started**
|
41
|
+
See [Quickstart guide](https://fivetran.com/docs/connectors/connector-sdk/quickstart-guide) to get started.
|
42
|
+
|
43
|
+
## **Usage**
|
44
|
+
Import required classes from fivetran_connector_sdk to begin writing your custom data connector
|
45
|
+
|
46
|
+
```python
|
47
|
+
from fivetran_connector_sdk import Connector
|
48
|
+
from fivetran_connector_sdk import Logging as log
|
49
|
+
from fivetran_connector_sdk import Operations as op
|
50
|
+
```
|
51
|
+
See our [Technical Reference](https://fivetran.com/docs/connectors/connector-sdk/technical-reference#update) and [Best Practices](https://fivetran.com/docs/connectors/connector-sdk/best-practices) documentation for more details.
|
52
|
+
|
53
|
+
You can also refer to our [existing examples](https://github.com/fivetran/fivetran_connector_sdk) to better understand Fivetran's Connector SDK and start creating custom data connectors.
|
54
|
+
|
55
|
+
## **Maintenance**
|
56
|
+
This package is actively maintained by Fivetran Developers. Please reach out to our [Support team](https://support.fivetran.com/hc/en-us) for any inquiries.
|
@@ -1,17 +0,0 @@
|
|
1
|
-
Metadata-Version: 2.1
|
2
|
-
Name: fivetran_connector_sdk
|
3
|
-
Version: 0.8.12.1
|
4
|
-
Summary: Build custom connectors on Fivetran platform
|
5
|
-
Author-email: Fivetran <developers@fivetran.com>
|
6
|
-
Project-URL: Homepage, https://fivetran.com/docs/connectors/connector-sdk
|
7
|
-
Project-URL: Github, https://github.com/fivetran/fivetran_connector_sdk
|
8
|
-
Classifier: Programming Language :: Python :: 3
|
9
|
-
Classifier: License :: OSI Approved :: MIT License
|
10
|
-
Classifier: Operating System :: OS Independent
|
11
|
-
Requires-Python: >=3.9
|
12
|
-
Description-Content-Type: text/markdown
|
13
|
-
Requires-Dist: grpcio==1.60.1
|
14
|
-
Requires-Dist: grpcio-tools==1.60.1
|
15
|
-
Requires-Dist: requests==2.31.0
|
16
|
-
Requires-Dist: get_pypi_latest_version==0.0.12
|
17
|
-
Requires-Dist: pipreqs==0.5.0
|
@@ -1,17 +0,0 @@
|
|
1
|
-
Metadata-Version: 2.1
|
2
|
-
Name: fivetran_connector_sdk
|
3
|
-
Version: 0.8.12.1
|
4
|
-
Summary: Build custom connectors on Fivetran platform
|
5
|
-
Author-email: Fivetran <developers@fivetran.com>
|
6
|
-
Project-URL: Homepage, https://fivetran.com/docs/connectors/connector-sdk
|
7
|
-
Project-URL: Github, https://github.com/fivetran/fivetran_connector_sdk
|
8
|
-
Classifier: Programming Language :: Python :: 3
|
9
|
-
Classifier: License :: OSI Approved :: MIT License
|
10
|
-
Classifier: Operating System :: OS Independent
|
11
|
-
Requires-Python: >=3.9
|
12
|
-
Description-Content-Type: text/markdown
|
13
|
-
Requires-Dist: grpcio==1.60.1
|
14
|
-
Requires-Dist: grpcio-tools==1.60.1
|
15
|
-
Requires-Dist: requests==2.31.0
|
16
|
-
Requires-Dist: get_pypi_latest_version==0.0.12
|
17
|
-
Requires-Dist: pipreqs==0.5.0
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|
File without changes
|