AntaresFabricFrameworkV2 2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (25) hide show
  1. antaresfabricframeworkv2-2.0/LICENSE.txt +10 -0
  2. antaresfabricframeworkv2-2.0/PKG-INFO +136 -0
  3. antaresfabricframeworkv2-2.0/README.md +108 -0
  4. antaresfabricframeworkv2-2.0/pyproject.toml +3 -0
  5. antaresfabricframeworkv2-2.0/setup.cfg +30 -0
  6. antaresfabricframeworkv2-2.0/setup.py +3 -0
  7. antaresfabricframeworkv2-2.0/src/AntaresFabricFrameworkV2.egg-info/PKG-INFO +136 -0
  8. antaresfabricframeworkv2-2.0/src/AntaresFabricFrameworkV2.egg-info/SOURCES.txt +24 -0
  9. antaresfabricframeworkv2-2.0/src/AntaresFabricFrameworkV2.egg-info/dependency_links.txt +1 -0
  10. antaresfabricframeworkv2-2.0/src/AntaresFabricFrameworkV2.egg-info/top_level.txt +1 -0
  11. antaresfabricframeworkv2-2.0/src/FabricFramework/ApiDataSource.py +1109 -0
  12. antaresfabricframeworkv2-2.0/src/FabricFramework/DataSource.py +195 -0
  13. antaresfabricframeworkv2-2.0/src/FabricFramework/DataValidation.py +829 -0
  14. antaresfabricframeworkv2-2.0/src/FabricFramework/DeploymentTesting.py +896 -0
  15. antaresfabricframeworkv2-2.0/src/FabricFramework/ExpectedResults/__init__.py +0 -0
  16. antaresfabricframeworkv2-2.0/src/FabricFramework/FabricCache.py +167 -0
  17. antaresfabricframeworkv2-2.0/src/FabricFramework/FabricConfiguration.py +583 -0
  18. antaresfabricframeworkv2-2.0/src/FabricFramework/FabricDataInterface.py +1966 -0
  19. antaresfabricframeworkv2-2.0/src/FabricFramework/FabricFramework.py +120 -0
  20. antaresfabricframeworkv2-2.0/src/FabricFramework/FabricLocations.py +319 -0
  21. antaresfabricframeworkv2-2.0/src/FabricFramework/LogMonitor.py +189 -0
  22. antaresfabricframeworkv2-2.0/src/FabricFramework/ProgressReporter.py +460 -0
  23. antaresfabricframeworkv2-2.0/src/FabricFramework/Pseudonymisation.py +121 -0
  24. antaresfabricframeworkv2-2.0/src/FabricFramework/SourceNameUtils.py +24 -0
  25. antaresfabricframeworkv2-2.0/src/FabricFramework/__init__.py +0 -0
@@ -0,0 +1,10 @@
1
+ MIT License
2
+
3
+ Copyright (c) [2024] [Antares Solutions]
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
6
+ copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
7
+
8
+ The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
9
+
10
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,136 @@
1
+ Metadata-Version: 2.4
2
+ Name: AntaresFabricFrameworkV2
3
+ Version: 2.0
4
+ Summary: Microsoft Fabric data ingestion accelerator
5
+ Home-page: https://antares.solutions/
6
+ Author: AntaresSolutions
7
+ Author-email: martonm@antaressolutions.com.au
8
+ Project-URL: Bug Tracker, https://antares.solutions/
9
+ Project-URL: repository, https://antares.solutions/
10
+ Classifier: Programming Language :: Python :: 3
11
+ Classifier: License :: OSI Approved :: MIT License
12
+ Classifier: Operating System :: OS Independent
13
+ Requires-Python: >=3.6
14
+ Description-Content-Type: text/markdown
15
+ License-File: LICENSE.txt
16
+ Dynamic: license-file
17
+
18
+ # `AntaresFabricFramework`
19
+
20
+ The `AntaresFabricFramework` is an accelerator for Microsoft Fabric data ingestion, validation and transformation. It uses a configuration file to describe each source and destination and includes details for data validation and preprocessing.
21
+
22
+ ## Contents
23
+
24
+ - [Installation](#installation)
25
+ - [Usage](#usage)
26
+ - [Features](#features)
27
+ - [Contributing](#contributing)
28
+ - [License](#license)
29
+ - [Contact Information](#contact-information)
30
+
31
+ ## Installation
32
+ The following options are available to install the library into a Fabric environment:
33
+
34
+ 1. <b>PyPi (Recommended):</b>
35
+ To use the latest base version available in the publish repository (PyPi), open fabric and edit or create an environment. Open the environment, click public libraries, click Add from PyPi, in the library text box type AntaresFabricFrameworkBase, then click publish.
36
+
37
+ 2. <b>Custom Library:</b>
38
+ The dist folder contains the wheel (.whl) file that can be uploaded into the Fabric environment as a custom library. To do that, open fabric and edit or create an environment. Open the environment, click custom libraries, then click Upload and upload the .whl file. Click Publish.
39
+ If changes are required to the base code, follow these steps to deploy the custom library:
40
+ - Fork the repository
41
+ - Create a new branch
42
+ - Edit the code
43
+ - Build the wheel file (eg. `py -m build`)
44
+ - Upload as per above
45
+
46
+ To install the base notebooks required for the framework (Constants_py, PreProcessingUtils_py and OrchestrateIngestion), follow these steps:
47
+
48
+ 1. Clone this repo to the destination (eg. customers devops)
49
+ 2. Create a new branch
50
+ 3. Create the control workspace in Fabric
51
+ 4. Connect the new branch in the repo to the workspace and in the git folder section type 'FabricArtifacts' to point to that folder in the repo (this is where the notebooks and lakehouse are stored).
52
+
53
+ After it syncs, the control workspace should contain a control lakehouse and the 3 notebooks as listed earlier.
54
+
55
+
56
+ ## Usage
57
+
58
+ In order to run the framework, there are a few key dependencies that need to be in place:
59
+
60
+ 1. <b>Ingestion Configuration file (or table):</b>
61
+ The ingestion configuration is a json file that contains information about the source of the data, how we want to save the data, and some optional parameters we can use during the ingestion process.
62
+ There is a sample file in the repository [ingestion_configuration.json](SupportingFiles/ingestion_configuration.json) containing all of the attributes used during the data ingestion.
63
+ After the initial execution of the framework, the file will be saved into a fabric table and read from the table each time after that. The table can be edited via SQL commands in a notebook or if the table is deleted, the json file will become the initial source again.
64
+ The ingestgion configuration file and table are both located in the Control Lakehouse.
65
+
66
+ 2. <b>Keyvault:</b> is used for storing all connection and credential details for all sources (except for Fabric tables and files - these don't require additional keyvault connection details.)
67
+ The Keyvault connection details will need to be included in the aformentioned ingestion configuration file.
68
+
69
+ 3. <b>Constants notebook:</b> A Constants_py notebook is required to sit in the Control lakehouse and contains global constants to be used during the ingestion. These global constants are generally set once and used throughout the framework.
70
+ Global constants include the names of each stage in the medallion architecture and the Azure Log Monitor details. There is a sample Constants_py notebook in the repository.
71
+
72
+ 4. <b>Azure Log Monitor (OPTIONAL):</b> If the Azure Monitor Log service is setup, the framework can push logs from anywhere in the preprocessing stage and at predefined stages in the ingestion process.
73
+ The logging is turned on via the Constants notebook, by setting the `LOGGING_LEVEL` to either `INFO` or `DEBUG`. `NONE` will turn off the logging. INFO will provide high level processing information at each major step in the ingestion. DEBUG will provide more detailed logs at higher intervals.
74
+ The schema of the log table that is required to be setup is can also be found in the repository under [AuditLogEntity.json](SupportingFiles/AuditLogEntity.json)
75
+
76
+ 5. <b>PreProcessing notebook (OPTIONAL):</b> Prior to the ingestion into the first (RAW) stage, a preprocessing function can be run to modify the data coming from the source. Typically, this is used to clean up or validate manually input .csv files or to merge multiple files into a single table.
77
+ This is optional and does not need to exist for the ingestion to work. To create a preprocessing function use the following function definition in the PreProcessing_py notebook:
78
+
79
+ def preProcess(source):
80
+
81
+ source - will be the ingestion configuration source record for a single sourceID.
82
+
83
+ 6. <b>Orchestration Notebook:</b> The main notebook that is used to execute the ingestion process code. This note book can also contain a parameters cell that can be used to capture additiona parameters (in the form of global constants).
84
+ For example, if the notebook is executed via a data pipeline in Fabric, the Data Pipeline Run ID can be passed in as a parameter and tracked via the logging module so users can trace back any errors to a specific pipeline.
85
+
86
+ To run the ingestion framework, the library can be imported:
87
+
88
+ from FabricFramework.FabricFramework import *
89
+
90
+ then instantiated as:
91
+
92
+ ff = FabricFramework(<location of ingestion_configuration.json>, <location of ingestion configuration table>, <constants>, <pre processing function>)
93
+
94
+ The framework ingestion process can be executed as follows:
95
+
96
+ ff.runIngestion() - this will run through the entire ingestion configuration (Every source)
97
+
98
+ You can also provide 3 parameters for more granual control over that is run as part of the ingestion:
99
+
100
+ ff.runIngestion(layer=[LAKEHOUSE_RAW_STAGE_NAME, LAKEHOUSE_TRUSTED_STAGE_NAME], system_code=['Test'], sourceID=[1, 3, 5])
101
+
102
+ <b>layer</b> - determines the stage that should be run (eg. RAW and TRUSTED, or just RAW or just TRUSTED). The names of each stage is determines by the constants (eg. LAKEHOUSE_RAW_STAGE_NAME) in the constants notebook
103
+
104
+ <b>system_code</b> - determines which specific system code(s) should be run as part of this execution (this can be a list of system codes)
105
+
106
+ <b>sourceID</b> - determines which sourceID's should be run on this run (if the ingestion is turned off in the ingestion configuration, the systemID will still be ignored)
107
+
108
+
109
+
110
+ ## Features
111
+
112
+
113
+ ## Contributing
114
+ Guidelines for contributing to the project:
115
+ 1. Fork the repository
116
+ 2. Create a new branch (`git checkout -b feature-branch`)
117
+ 3. Commit your changes (`git commit -m 'Add some feature'`)
118
+ 4. Push to the branch (`git push origin feature-branch`)
119
+ 5. Open a pull request
120
+
121
+ ## License
122
+ This project is licensed under the MIT License - see the [LICENSE](LICENSE.txt) file for details.
123
+
124
+ ## Contact Information
125
+ For any questions, please contact [martonm@antaressolutions.com.au](mailto:martonm@antaressolutions.com.au).
126
+
127
+ MIT License
128
+
129
+ Copyright (c) [2024] [Antares Solutions]
130
+
131
+ Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
132
+ copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
133
+
134
+ The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
135
+
136
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,108 @@
1
+ # `AntaresFabricFramework`
2
+
3
+ The `AntaresFabricFramework` is an accelerator for Microsoft Fabric data ingestion, validation and transformation. It uses a configuration file to describe each source and destination and includes details for data validation and preprocessing.
4
+
5
+ ## Contents
6
+
7
+ - [Installation](#installation)
8
+ - [Usage](#usage)
9
+ - [Features](#features)
10
+ - [Contributing](#contributing)
11
+ - [License](#license)
12
+ - [Contact Information](#contact-information)
13
+
14
+ ## Installation
15
+ The following options are available to install the library into a Fabric environment:
16
+
17
+ 1. <b>PyPi (Recommended):</b>
18
+ To use the latest base version available in the publish repository (PyPi), open fabric and edit or create an environment. Open the environment, click public libraries, click Add from PyPi, in the library text box type AntaresFabricFrameworkBase, then click publish.
19
+
20
+ 2. <b>Custom Library:</b>
21
+ The dist folder contains the wheel (.whl) file that can be uploaded into the Fabric environment as a custom library. To do that, open fabric and edit or create an environment. Open the environment, click custom libraries, then click Upload and upload the .whl file. Click Publish.
22
+ If changes are required to the base code, follow these steps to deploy the custom library:
23
+ - Fork the repository
24
+ - Create a new branch
25
+ - Edit the code
26
+ - Build the wheel file (eg. `py -m build`)
27
+ - Upload as per above
28
+
29
+ To install the base notebooks required for the framework (Constants_py, PreProcessingUtils_py and OrchestrateIngestion), follow these steps:
30
+
31
+ 1. Clone this repo to the destination (eg. customers devops)
32
+ 2. Create a new branch
33
+ 3. Create the control workspace in Fabric
34
+ 4. Connect the new branch in the repo to the workspace and in the git folder section type 'FabricArtifacts' to point to that folder in the repo (this is where the notebooks and lakehouse are stored).
35
+
36
+ After it syncs, the control workspace should contain a control lakehouse and the 3 notebooks as listed earlier.
37
+
38
+
39
+ ## Usage
40
+
41
+ In order to run the framework, there are a few key dependencies that need to be in place:
42
+
43
+ 1. <b>Ingestion Configuration file (or table):</b>
44
+ The ingestion configuration is a json file that contains information about the source of the data, how we want to save the data, and some optional parameters we can use during the ingestion process.
45
+ There is a sample file in the repository [ingestion_configuration.json](SupportingFiles/ingestion_configuration.json) containing all of the attributes used during the data ingestion.
46
+ After the initial execution of the framework, the file will be saved into a fabric table and read from the table each time after that. The table can be edited via SQL commands in a notebook or if the table is deleted, the json file will become the initial source again.
47
+ The ingestgion configuration file and table are both located in the Control Lakehouse.
48
+
49
+ 2. <b>Keyvault:</b> is used for storing all connection and credential details for all sources (except for Fabric tables and files - these don't require additional keyvault connection details.)
50
+ The Keyvault connection details will need to be included in the aformentioned ingestion configuration file.
51
+
52
+ 3. <b>Constants notebook:</b> A Constants_py notebook is required to sit in the Control lakehouse and contains global constants to be used during the ingestion. These global constants are generally set once and used throughout the framework.
53
+ Global constants include the names of each stage in the medallion architecture and the Azure Log Monitor details. There is a sample Constants_py notebook in the repository.
54
+
55
+ 4. <b>Azure Log Monitor (OPTIONAL):</b> If the Azure Monitor Log service is setup, the framework can push logs from anywhere in the preprocessing stage and at predefined stages in the ingestion process.
56
+ The logging is turned on via the Constants notebook, by setting the `LOGGING_LEVEL` to either `INFO` or `DEBUG`. `NONE` will turn off the logging. INFO will provide high level processing information at each major step in the ingestion. DEBUG will provide more detailed logs at higher intervals.
57
+ The schema of the log table that is required to be setup is can also be found in the repository under [AuditLogEntity.json](SupportingFiles/AuditLogEntity.json)
58
+
59
+ 5. <b>PreProcessing notebook (OPTIONAL):</b> Prior to the ingestion into the first (RAW) stage, a preprocessing function can be run to modify the data coming from the source. Typically, this is used to clean up or validate manually input .csv files or to merge multiple files into a single table.
60
+ This is optional and does not need to exist for the ingestion to work. To create a preprocessing function use the following function definition in the PreProcessing_py notebook:
61
+
62
+ def preProcess(source):
63
+
64
+ source - will be the ingestion configuration source record for a single sourceID.
65
+
66
+ 6. <b>Orchestration Notebook:</b> The main notebook that is used to execute the ingestion process code. This note book can also contain a parameters cell that can be used to capture additiona parameters (in the form of global constants).
67
+ For example, if the notebook is executed via a data pipeline in Fabric, the Data Pipeline Run ID can be passed in as a parameter and tracked via the logging module so users can trace back any errors to a specific pipeline.
68
+
69
+ To run the ingestion framework, the library can be imported:
70
+
71
+ from FabricFramework.FabricFramework import *
72
+
73
+ then instantiated as:
74
+
75
+ ff = FabricFramework(<location of ingestion_configuration.json>, <location of ingestion configuration table>, <constants>, <pre processing function>)
76
+
77
+ The framework ingestion process can be executed as follows:
78
+
79
+ ff.runIngestion() - this will run through the entire ingestion configuration (Every source)
80
+
81
+ You can also provide 3 parameters for more granual control over that is run as part of the ingestion:
82
+
83
+ ff.runIngestion(layer=[LAKEHOUSE_RAW_STAGE_NAME, LAKEHOUSE_TRUSTED_STAGE_NAME], system_code=['Test'], sourceID=[1, 3, 5])
84
+
85
+ <b>layer</b> - determines the stage that should be run (eg. RAW and TRUSTED, or just RAW or just TRUSTED). The names of each stage is determines by the constants (eg. LAKEHOUSE_RAW_STAGE_NAME) in the constants notebook
86
+
87
+ <b>system_code</b> - determines which specific system code(s) should be run as part of this execution (this can be a list of system codes)
88
+
89
+ <b>sourceID</b> - determines which sourceID's should be run on this run (if the ingestion is turned off in the ingestion configuration, the systemID will still be ignored)
90
+
91
+
92
+
93
+ ## Features
94
+
95
+
96
+ ## Contributing
97
+ Guidelines for contributing to the project:
98
+ 1. Fork the repository
99
+ 2. Create a new branch (`git checkout -b feature-branch`)
100
+ 3. Commit your changes (`git commit -m 'Add some feature'`)
101
+ 4. Push to the branch (`git push origin feature-branch`)
102
+ 5. Open a pull request
103
+
104
+ ## License
105
+ This project is licensed under the MIT License - see the [LICENSE](LICENSE.txt) file for details.
106
+
107
+ ## Contact Information
108
+ For any questions, please contact [martonm@antaressolutions.com.au](mailto:martonm@antaressolutions.com.au).
@@ -0,0 +1,3 @@
1
+ [build-system]
2
+ requires = ['setuptools>=42']
3
+ build-backend = 'setuptools.build_meta'
@@ -0,0 +1,30 @@
1
+ [metadata]
2
+ name = AntaresFabricFrameworkV2
3
+ version = 2.0
4
+ author = AntaresSolutions
5
+ author_email = martonm@antaressolutions.com.au
6
+ description = Microsoft Fabric data ingestion accelerator
7
+ long_description = file: README.md, LICENSE.txt
8
+ long_description_content_type = text/markdown
9
+ url = https://antares.solutions/
10
+ project_urls =
11
+ Bug Tracker = https://antares.solutions/
12
+ repository = https://antares.solutions/
13
+ classifiers =
14
+ Programming Language :: Python :: 3
15
+ License :: OSI Approved :: MIT License
16
+ Operating System :: OS Independent
17
+
18
+ [options]
19
+ package_dir =
20
+ = src
21
+ packages = find:
22
+ python_requires = >=3.6
23
+
24
+ [options.packages.find]
25
+ where = src
26
+
27
+ [egg_info]
28
+ tag_build =
29
+ tag_date = 0
30
+
@@ -0,0 +1,3 @@
1
+ import setuptools
2
+
3
+ setuptools.setup()
@@ -0,0 +1,136 @@
1
+ Metadata-Version: 2.4
2
+ Name: AntaresFabricFrameworkV2
3
+ Version: 2.0
4
+ Summary: Microsoft Fabric data ingestion accelerator
5
+ Home-page: https://antares.solutions/
6
+ Author: AntaresSolutions
7
+ Author-email: martonm@antaressolutions.com.au
8
+ Project-URL: Bug Tracker, https://antares.solutions/
9
+ Project-URL: repository, https://antares.solutions/
10
+ Classifier: Programming Language :: Python :: 3
11
+ Classifier: License :: OSI Approved :: MIT License
12
+ Classifier: Operating System :: OS Independent
13
+ Requires-Python: >=3.6
14
+ Description-Content-Type: text/markdown
15
+ License-File: LICENSE.txt
16
+ Dynamic: license-file
17
+
18
+ # `AntaresFabricFramework`
19
+
20
+ The `AntaresFabricFramework` is an accelerator for Microsoft Fabric data ingestion, validation and transformation. It uses a configuration file to describe each source and destination and includes details for data validation and preprocessing.
21
+
22
+ ## Contents
23
+
24
+ - [Installation](#installation)
25
+ - [Usage](#usage)
26
+ - [Features](#features)
27
+ - [Contributing](#contributing)
28
+ - [License](#license)
29
+ - [Contact Information](#contact-information)
30
+
31
+ ## Installation
32
+ The following options are available to install the library into a Fabric environment:
33
+
34
+ 1. <b>PyPi (Recommended):</b>
35
+ To use the latest base version available in the publish repository (PyPi), open fabric and edit or create an environment. Open the environment, click public libraries, click Add from PyPi, in the library text box type AntaresFabricFrameworkBase, then click publish.
36
+
37
+ 2. <b>Custom Library:</b>
38
+ The dist folder contains the wheel (.whl) file that can be uploaded into the Fabric environment as a custom library. To do that, open fabric and edit or create an environment. Open the environment, click custom libraries, then click Upload and upload the .whl file. Click Publish.
39
+ If changes are required to the base code, follow these steps to deploy the custom library:
40
+ - Fork the repository
41
+ - Create a new branch
42
+ - Edit the code
43
+ - Build the wheel file (eg. `py -m build`)
44
+ - Upload as per above
45
+
46
+ To install the base notebooks required for the framework (Constants_py, PreProcessingUtils_py and OrchestrateIngestion), follow these steps:
47
+
48
+ 1. Clone this repo to the destination (eg. customers devops)
49
+ 2. Create a new branch
50
+ 3. Create the control workspace in Fabric
51
+ 4. Connect the new branch in the repo to the workspace and in the git folder section type 'FabricArtifacts' to point to that folder in the repo (this is where the notebooks and lakehouse are stored).
52
+
53
+ After it syncs, the control workspace should contain a control lakehouse and the 3 notebooks as listed earlier.
54
+
55
+
56
+ ## Usage
57
+
58
+ In order to run the framework, there are a few key dependencies that need to be in place:
59
+
60
+ 1. <b>Ingestion Configuration file (or table):</b>
61
+ The ingestion configuration is a json file that contains information about the source of the data, how we want to save the data, and some optional parameters we can use during the ingestion process.
62
+ There is a sample file in the repository [ingestion_configuration.json](SupportingFiles/ingestion_configuration.json) containing all of the attributes used during the data ingestion.
63
+ After the initial execution of the framework, the file will be saved into a fabric table and read from the table each time after that. The table can be edited via SQL commands in a notebook or if the table is deleted, the json file will become the initial source again.
64
+ The ingestgion configuration file and table are both located in the Control Lakehouse.
65
+
66
+ 2. <b>Keyvault:</b> is used for storing all connection and credential details for all sources (except for Fabric tables and files - these don't require additional keyvault connection details.)
67
+ The Keyvault connection details will need to be included in the aformentioned ingestion configuration file.
68
+
69
+ 3. <b>Constants notebook:</b> A Constants_py notebook is required to sit in the Control lakehouse and contains global constants to be used during the ingestion. These global constants are generally set once and used throughout the framework.
70
+ Global constants include the names of each stage in the medallion architecture and the Azure Log Monitor details. There is a sample Constants_py notebook in the repository.
71
+
72
+ 4. <b>Azure Log Monitor (OPTIONAL):</b> If the Azure Monitor Log service is setup, the framework can push logs from anywhere in the preprocessing stage and at predefined stages in the ingestion process.
73
+ The logging is turned on via the Constants notebook, by setting the `LOGGING_LEVEL` to either `INFO` or `DEBUG`. `NONE` will turn off the logging. INFO will provide high level processing information at each major step in the ingestion. DEBUG will provide more detailed logs at higher intervals.
74
+ The schema of the log table that is required to be setup is can also be found in the repository under [AuditLogEntity.json](SupportingFiles/AuditLogEntity.json)
75
+
76
+ 5. <b>PreProcessing notebook (OPTIONAL):</b> Prior to the ingestion into the first (RAW) stage, a preprocessing function can be run to modify the data coming from the source. Typically, this is used to clean up or validate manually input .csv files or to merge multiple files into a single table.
77
+ This is optional and does not need to exist for the ingestion to work. To create a preprocessing function use the following function definition in the PreProcessing_py notebook:
78
+
79
+ def preProcess(source):
80
+
81
+ source - will be the ingestion configuration source record for a single sourceID.
82
+
83
+ 6. <b>Orchestration Notebook:</b> The main notebook that is used to execute the ingestion process code. This note book can also contain a parameters cell that can be used to capture additiona parameters (in the form of global constants).
84
+ For example, if the notebook is executed via a data pipeline in Fabric, the Data Pipeline Run ID can be passed in as a parameter and tracked via the logging module so users can trace back any errors to a specific pipeline.
85
+
86
+ To run the ingestion framework, the library can be imported:
87
+
88
+ from FabricFramework.FabricFramework import *
89
+
90
+ then instantiated as:
91
+
92
+ ff = FabricFramework(<location of ingestion_configuration.json>, <location of ingestion configuration table>, <constants>, <pre processing function>)
93
+
94
+ The framework ingestion process can be executed as follows:
95
+
96
+ ff.runIngestion() - this will run through the entire ingestion configuration (Every source)
97
+
98
+ You can also provide 3 parameters for more granual control over that is run as part of the ingestion:
99
+
100
+ ff.runIngestion(layer=[LAKEHOUSE_RAW_STAGE_NAME, LAKEHOUSE_TRUSTED_STAGE_NAME], system_code=['Test'], sourceID=[1, 3, 5])
101
+
102
+ <b>layer</b> - determines the stage that should be run (eg. RAW and TRUSTED, or just RAW or just TRUSTED). The names of each stage is determines by the constants (eg. LAKEHOUSE_RAW_STAGE_NAME) in the constants notebook
103
+
104
+ <b>system_code</b> - determines which specific system code(s) should be run as part of this execution (this can be a list of system codes)
105
+
106
+ <b>sourceID</b> - determines which sourceID's should be run on this run (if the ingestion is turned off in the ingestion configuration, the systemID will still be ignored)
107
+
108
+
109
+
110
+ ## Features
111
+
112
+
113
+ ## Contributing
114
+ Guidelines for contributing to the project:
115
+ 1. Fork the repository
116
+ 2. Create a new branch (`git checkout -b feature-branch`)
117
+ 3. Commit your changes (`git commit -m 'Add some feature'`)
118
+ 4. Push to the branch (`git push origin feature-branch`)
119
+ 5. Open a pull request
120
+
121
+ ## License
122
+ This project is licensed under the MIT License - see the [LICENSE](LICENSE.txt) file for details.
123
+
124
+ ## Contact Information
125
+ For any questions, please contact [martonm@antaressolutions.com.au](mailto:martonm@antaressolutions.com.au).
126
+
127
+ MIT License
128
+
129
+ Copyright (c) [2024] [Antares Solutions]
130
+
131
+ Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
132
+ copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
133
+
134
+ The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
135
+
136
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,24 @@
1
+ LICENSE.txt
2
+ README.md
3
+ pyproject.toml
4
+ setup.cfg
5
+ setup.py
6
+ src/AntaresFabricFrameworkV2.egg-info/PKG-INFO
7
+ src/AntaresFabricFrameworkV2.egg-info/SOURCES.txt
8
+ src/AntaresFabricFrameworkV2.egg-info/dependency_links.txt
9
+ src/AntaresFabricFrameworkV2.egg-info/top_level.txt
10
+ src/FabricFramework/ApiDataSource.py
11
+ src/FabricFramework/DataSource.py
12
+ src/FabricFramework/DataValidation.py
13
+ src/FabricFramework/DeploymentTesting.py
14
+ src/FabricFramework/FabricCache.py
15
+ src/FabricFramework/FabricConfiguration.py
16
+ src/FabricFramework/FabricDataInterface.py
17
+ src/FabricFramework/FabricFramework.py
18
+ src/FabricFramework/FabricLocations.py
19
+ src/FabricFramework/LogMonitor.py
20
+ src/FabricFramework/ProgressReporter.py
21
+ src/FabricFramework/Pseudonymisation.py
22
+ src/FabricFramework/SourceNameUtils.py
23
+ src/FabricFramework/__init__.py
24
+ src/FabricFramework/ExpectedResults/__init__.py