cosmotech-acceleration-library 1.0.0__tar.gz

cosmotech_acceleration_library-1.0.0/LICENSE

@@ -0,0 +1,17 @@
+LICENSE
+=======
+
+Copyright 2021-2023 Cosmo Tech
+
+You are authorized to use this software following the execution of a specific agreement with Cosmo Tech or with an authorized licensee of Cosmo Tech.
+
+Accordingly you should refer to this specific agreement for the purpose of the extent of the use that is permitted. This use shall mainly consist in using, displaying and executing the software for your own needs.
+
+Except otherwise provided in the above-mentioned specific agreement:
+- you are not allowed to use this software for commercial purpose or to redistribute it without prior authorization from Cosmo Tech;
+- you are not allowed to copy, duplicate, modify, adapt and translate this software except as authorized under section L122-6-1 of the French Intellectual Property Code or specifically under the specific agreement with Cosmo Tech;
+- you must not removed this notice or any mention related to the copyright of Cosmo Tech;
+
+This software is provided 'as is', without any express or implied warranty towards the users.
+
+Any Software generated through the use of the software shall be your own property to the extent this Software does not contain any part of software source code.

cosmotech_acceleration_library-1.0.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include README.md
+include LICENSE
+graft cosmotech/orchestrator_plugins
+graft cosmotech/translation

cosmotech_acceleration_library-1.0.0/PKG-INFO

@@ -0,0 +1,255 @@
+Metadata-Version: 2.4
+Name: cosmotech_acceleration_library
+Version: 1.0.0
+Summary: Acceleration libraries for CosmoTech cloud based solution development
+Author-email: Cosmo Tech <platform@cosmotech.com>
+Project-URL: Homepage, https://www.cosmotech.com
+Project-URL: Source, https://github.com/Cosmo-Tech/CosmoTech-Acceleration-Library
+Project-URL: Documentation, https://cosmo-tech.github.io/CosmoTech-Acceleration-Library
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: azure-functions~=1.20.0
+Requires-Dist: azure-digitaltwins-core~=1.2.0
+Requires-Dist: azure-identity~=1.16.1
+Requires-Dist: azure-kusto-data~=4.4.1
+Requires-Dist: azure-kusto-ingest~=4.4.1
+Requires-Dist: tenacity~=8.3.0
+Requires-Dist: python-keycloak~=4.7.3
+Requires-Dist: redis==4.4.4
+Requires-Dist: redisgraph_bulk_loader==0.10.2
+Requires-Dist: cosmotech-api~=3.2
+Requires-Dist: boto3~=1.35.19
+Requires-Dist: requests~=2.32.3
+Requires-Dist: singlestoredb~=1.10.0
+Requires-Dist: cosmotech-run-orchestrator~=2.0.0
+Requires-Dist: pyarrow~=17.0.0
+Requires-Dist: adbc-driver-manager~=1.1.0
+Requires-Dist: adbc-driver-sqlite~=1.1.0
+Requires-Dist: adbc-driver-postgresql~=1.1.0
+Requires-Dist: click~=8.1.7
+Requires-Dist: rich-click~=1.7.3
+Requires-Dist: click-log~=0.4.0
+Requires-Dist: tqdm~=4.67.1
+Requires-Dist: openpyxl~=3.1
+Requires-Dist: pandas~=2.1
+Requires-Dist: python-dateutil~=2.8
+Requires-Dist: rich~=13.7
+Requires-Dist: setuptools
+Provides-Extra: doc
+Requires-Dist: mkdocs~=1.5.3; extra == "doc"
+Requires-Dist: mkdocs-click~=0.8.1; extra == "doc"
+Requires-Dist: mkdocs-gen-files~=0.5.0; extra == "doc"
+Requires-Dist: mkdocstrings[python]~=0.28.2; extra == "doc"
+Requires-Dist: mkdocs-awesome-pages-plugin~=2.9.3; extra == "doc"
+Requires-Dist: pymdown-extensions~=10.7; extra == "doc"
+Requires-Dist: requirements-parser~=0.11.0; extra == "doc"
+Requires-Dist: setuptools~=70.3.0; extra == "doc"
+Requires-Dist: mike~=2.0.0; extra == "doc"
+Requires-Dist: griffe~=1.5.7; extra == "doc"
+Requires-Dist: mkdocs-include-dir-to-nav~=1.2.0; extra == "doc"
+Requires-Dist: mkdocs-material[imaging]~=9.5.18; extra == "doc"
+Requires-Dist: mkdocs-table-reader-plugin~=2.0.3; extra == "doc"
+Requires-Dist: mkdocs-literate-nav~=0.6.1; extra == "doc"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-docker; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Provides-Extra: extra
+Requires-Dist: pandas~=2.2.2; extra == "extra"
+Provides-Extra: dev
+Requires-Dist: black==23.3.0; extra == "dev"
+Requires-Dist: pre-commit==3.3.2; extra == "dev"
+Provides-Extra: all
+Requires-Dist: CosmoTech_Acceleration_Library[dev,doc,extra,test]; extra == "all"
+Dynamic: license-file
+
+# CosmoTech-Acceleration-Library (CoAL)
+
+Acceleration library for CosmoTech cloud-based solution development.
+
+## Introduction
+
+The CosmoTech Acceleration Library (CoAL) provides a comprehensive set of tools and utilities to accelerate the development of solutions based on the CosmoTech platform. It offers a unified interface for interacting with CosmoTech APIs, managing data, and integrating with various cloud services.
+
+## Main Components
+
+### csm-data
+
+`csm-data` is a powerful CLI tool designed to help CosmoTech solution modelers and integrators interact with multiple systems. It provides ready-to-use commands to send and retrieve data from various systems where a CosmoTech API could be integrated.
+
+```bash
+# Get help on available commands
+csm-data --help
+
+# Get help on specific command groups
+csm-data api --help
+```
+
+### datastore
+
+The datastore provides a way to maintain local data during simulations and comes with `csm-data` commands to easily send those data to target systems. It offers:
+
+- Format flexibility (Python dictionaries, CSV files, Pandas DataFrames, PyArrow Tables)
+- Persistent storage in SQLite
+- SQL query capabilities
+- Simplified data pipeline management
+
+```python
+from cosmotech.coal.store.store import Store
+from cosmotech.coal.store.native_python import store_pylist
+
+# Initialize and reset the data store
+my_datastore = Store(reset=True)
+
+# Create and store data
+my_data = [{"foo": "bar"}, {"foo": "barbar"}, {"foo": "world"}, {"foo": "bar"}]
+store_pylist("my_data", my_data)
+
+# Query the data
+results = my_datastore.execute_query("SELECT foo, count(*) as line_count FROM my_data GROUP BY foo").to_pylist()
+print(results)
+# > [{'foo': 'bar', 'line_count': 2}, {'foo': 'barbar', 'line_count': 1}, {'foo': 'world', 'line_count': 1}]
+```
+
+### CosmoTech API Integration
+
+CoAL provides comprehensive tools for interacting with the CosmoTech API, allowing you to:
+
+- Authenticate with different identity providers (API Key, Azure Entra, Keycloak)
+- Manage workspaces and files
+- Work with the Twin Data Layer for graph data
+- Handle runners and runs
+- Process and transform data
+- Build end-to-end workflows
+
+```python
+import os
+from cosmotech.coal.cosmotech_api.connection import get_api_client
+
+# Set up environment variables for authentication
+os.environ["CSM_API_URL"] = "https://api.cosmotech.com"  # Replace with your API URL
+os.environ["CSM_API_KEY"] = "your-api-key"  # Replace with your actual API key
+
+# Get the API client
+api_client, connection_type = get_api_client()
+print(f"Connected using: {connection_type}")
+
+# Use the client with various API instances
+from cosmotech_api.api.organization_api import OrganizationApi
+org_api = OrganizationApi(api_client)
+
+# List organizations
+organizations = org_api.find_all_organizations()
+for org in organizations:
+    print(f"Organization: {org.name} (ID: {org.id})")
+
+# Don't forget to close the client when done
+api_client.close()
+```
+
+### Other Components
+
+- **coal**: Core library with modules for API interaction, data management, etc.
+- **csm_data**: CLI tool for data management and integration with various systems
+- **orchestrator_plugins**: Plugins that integrate with external orchestration systems
+- **translation**: Internationalization support for multiple languages
+
+## Getting Started
+
+### Installation
+
+```bash
+pip install cosmotech-acceleration-library
+```
+
+### Basic Usage
+
+Check out the tutorials directory for comprehensive examples of how to use the library:
+
+- [CosmoTech API Integration](https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/tutorials/cosmotech-api/)
+- [Data Store Usage](https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/tutorials/datastore/)
+- [csm-data CLI](https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/tutorials/csm-data/)
+
+## Key Features
+
+### Cloud Service Integration
+
+CoAL provides built-in support for various cloud services:
+
+- **Azure**: Azure Data Explorer (ADX), Azure Storage, Azure Functions
+- **AWS**: S3 buckets, and more
+- **Database Systems**: PostgreSQL, SingleStore, and others
+
+### Data Management
+
+- Load and transform data from various sources
+- Store and query data locally
+- Export data to different formats and destinations
+- Manage datasets in the CosmoTech platform
+
+### Orchestration Integration
+
+- Provides plugins that integrate with external orchestration systems
+- Supports data transfer between orchestration steps
+- Offers utilities for handling parameters and configurations
+- Enables seamless integration with the CosmoTech platform during orchestrated workflows
+
+## Documentation and Tutorials
+
+Comprehensive documentation is available at [https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/](https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/)
+
+### Tutorials
+
+- **CosmoTech API**: Learn how to interact with the CosmoTech API directly: authentication, workspaces, Twin Data Layer, and more.
+- **Data Store**: The datastore is your friend to keep data between orchestration steps. It comes with multiple ways to interact with it.
+- **csm-data**: Make full use of `csm-data` commands to connect to services during your orchestration runs.
+
+## Testing and Code Coverage
+
+The CosmoTech Acceleration Library maintains a comprehensive test suite to ensure reliability and stability. We use pytest for testing and pytest-cov for coverage reporting.
+
+### Running Tests
+
+To run the test suite:
+
+```bash
+# Install test dependencies
+pip install -e ".[test]"
+
+# Run tests with coverage reporting
+pytest tests/unit/coal/ --cov=cosmotech.coal --cov-report=term-missing --cov-report=html
+```
+
+### Coverage Reports
+
+After running tests with coverage, you can view detailed HTML reports:
+
+```bash
+# Open the HTML coverage report
+open coverage_html_report/index.html
+```
+
+[![codecov](https://codecov.io/gh/Cosmo-Tech/CosmoTech-Acceleration-Library/branch/main/graph/badge.svg)](https://codecov.io/gh/Cosmo-Tech/CosmoTech-Acceleration-Library)
+
+We maintain high test coverage to ensure code quality and reliability. All pull requests are expected to maintain or improve the current coverage levels.
+
+### Test Generation Tools
+
+To help maintain test coverage, we provide tools to identify untested functions and generate test files:
+
+```bash
+# Find functions without tests
+python find_untested_functions.py
+
+# Generate test files for a specific module
+python generate_test_files.py --module cosmotech/coal/module/file.py
+
+# Generate test files for all untested functions
+python generate_test_files.py --all
+```
+
+These tools help ensure that every function has at least one test, which is a requirement for contributions to the project.
+
+## Contact
+
+For support, feature requests, or contributions, please use the [GitHub repository](https://github.com/Cosmo-Tech/CosmoTech-Acceleration-Library).

cosmotech_acceleration_library-1.0.0/README.md

@@ -0,0 +1,190 @@
+# CosmoTech-Acceleration-Library (CoAL)
+
+Acceleration library for CosmoTech cloud-based solution development.
+
+## Introduction
+
+The CosmoTech Acceleration Library (CoAL) provides a comprehensive set of tools and utilities to accelerate the development of solutions based on the CosmoTech platform. It offers a unified interface for interacting with CosmoTech APIs, managing data, and integrating with various cloud services.
+
+## Main Components
+
+### csm-data
+
+`csm-data` is a powerful CLI tool designed to help CosmoTech solution modelers and integrators interact with multiple systems. It provides ready-to-use commands to send and retrieve data from various systems where a CosmoTech API could be integrated.
+
+```bash
+# Get help on available commands
+csm-data --help
+
+# Get help on specific command groups
+csm-data api --help
+```
+
+### datastore
+
+The datastore provides a way to maintain local data during simulations and comes with `csm-data` commands to easily send those data to target systems. It offers:
+
+- Format flexibility (Python dictionaries, CSV files, Pandas DataFrames, PyArrow Tables)
+- Persistent storage in SQLite
+- SQL query capabilities
+- Simplified data pipeline management
+
+```python
+from cosmotech.coal.store.store import Store
+from cosmotech.coal.store.native_python import store_pylist
+
+# Initialize and reset the data store
+my_datastore = Store(reset=True)
+
+# Create and store data
+my_data = [{"foo": "bar"}, {"foo": "barbar"}, {"foo": "world"}, {"foo": "bar"}]
+store_pylist("my_data", my_data)
+
+# Query the data
+results = my_datastore.execute_query("SELECT foo, count(*) as line_count FROM my_data GROUP BY foo").to_pylist()
+print(results)
+# > [{'foo': 'bar', 'line_count': 2}, {'foo': 'barbar', 'line_count': 1}, {'foo': 'world', 'line_count': 1}]
+```
+
+### CosmoTech API Integration
+
+CoAL provides comprehensive tools for interacting with the CosmoTech API, allowing you to:
+
+- Authenticate with different identity providers (API Key, Azure Entra, Keycloak)
+- Manage workspaces and files
+- Work with the Twin Data Layer for graph data
+- Handle runners and runs
+- Process and transform data
+- Build end-to-end workflows
+
+```python
+import os
+from cosmotech.coal.cosmotech_api.connection import get_api_client
+
+# Set up environment variables for authentication
+os.environ["CSM_API_URL"] = "https://api.cosmotech.com"  # Replace with your API URL
+os.environ["CSM_API_KEY"] = "your-api-key"  # Replace with your actual API key
+
+# Get the API client
+api_client, connection_type = get_api_client()
+print(f"Connected using: {connection_type}")
+
+# Use the client with various API instances
+from cosmotech_api.api.organization_api import OrganizationApi
+org_api = OrganizationApi(api_client)
+
+# List organizations
+organizations = org_api.find_all_organizations()
+for org in organizations:
+    print(f"Organization: {org.name} (ID: {org.id})")
+
+# Don't forget to close the client when done
+api_client.close()
+```
+
+### Other Components
+
+- **coal**: Core library with modules for API interaction, data management, etc.
+- **csm_data**: CLI tool for data management and integration with various systems
+- **orchestrator_plugins**: Plugins that integrate with external orchestration systems
+- **translation**: Internationalization support for multiple languages
+
+## Getting Started
+
+### Installation
+
+```bash
+pip install cosmotech-acceleration-library
+```
+
+### Basic Usage
+
+Check out the tutorials directory for comprehensive examples of how to use the library:
+
+- [CosmoTech API Integration](https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/tutorials/cosmotech-api/)
+- [Data Store Usage](https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/tutorials/datastore/)
+- [csm-data CLI](https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/tutorials/csm-data/)
+
+## Key Features
+
+### Cloud Service Integration
+
+CoAL provides built-in support for various cloud services:
+
+- **Azure**: Azure Data Explorer (ADX), Azure Storage, Azure Functions
+- **AWS**: S3 buckets, and more
+- **Database Systems**: PostgreSQL, SingleStore, and others
+
+### Data Management
+
+- Load and transform data from various sources
+- Store and query data locally
+- Export data to different formats and destinations
+- Manage datasets in the CosmoTech platform
+
+### Orchestration Integration
+
+- Provides plugins that integrate with external orchestration systems
+- Supports data transfer between orchestration steps
+- Offers utilities for handling parameters and configurations
+- Enables seamless integration with the CosmoTech platform during orchestrated workflows
+
+## Documentation and Tutorials
+
+Comprehensive documentation is available at [https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/](https://cosmo-tech.github.io/CosmoTech-Acceleration-Library/)
+
+### Tutorials
+
+- **CosmoTech API**: Learn how to interact with the CosmoTech API directly: authentication, workspaces, Twin Data Layer, and more.
+- **Data Store**: The datastore is your friend to keep data between orchestration steps. It comes with multiple ways to interact with it.
+- **csm-data**: Make full use of `csm-data` commands to connect to services during your orchestration runs.
+
+## Testing and Code Coverage
+
+The CosmoTech Acceleration Library maintains a comprehensive test suite to ensure reliability and stability. We use pytest for testing and pytest-cov for coverage reporting.
+
+### Running Tests
+
+To run the test suite:
+
+```bash
+# Install test dependencies
+pip install -e ".[test]"
+
+# Run tests with coverage reporting
+pytest tests/unit/coal/ --cov=cosmotech.coal --cov-report=term-missing --cov-report=html
+```
+
+### Coverage Reports
+
+After running tests with coverage, you can view detailed HTML reports:
+
+```bash
+# Open the HTML coverage report
+open coverage_html_report/index.html
+```
+
+[![codecov](https://codecov.io/gh/Cosmo-Tech/CosmoTech-Acceleration-Library/branch/main/graph/badge.svg)](https://codecov.io/gh/Cosmo-Tech/CosmoTech-Acceleration-Library)
+
+We maintain high test coverage to ensure code quality and reliability. All pull requests are expected to maintain or improve the current coverage levels.
+
+### Test Generation Tools
+
+To help maintain test coverage, we provide tools to identify untested functions and generate test files:
+
+```bash
+# Find functions without tests
+python find_untested_functions.py
+
+# Generate test files for a specific module
+python generate_test_files.py --module cosmotech/coal/module/file.py
+
+# Generate test files for all untested functions
+python generate_test_files.py --all
+```
+
+These tools help ensure that every function has at least one test, which is a requirement for contributions to the project.
+
+## Contact
+
+For support, feature requests, or contributions, please use the [GitHub repository](https://github.com/Cosmo-Tech/CosmoTech-Acceleration-Library).

cosmotech_acceleration_library-1.0.0/cosmotech/coal/__init__.py

@@ -0,0 +1,8 @@
+# Copyright (C) - 2023 - 2025 - Cosmo Tech
+# This document and all information contained herein is the exclusive property -
+# including all intellectual property rights pertaining thereto - of Cosmo Tech.
+# Any use, reproduction, translation, broadcasting, transmission, distribution,
+# etc., to any person is prohibited unless it has been previously and
+# specifically authorized by written means by Cosmo Tech.
+
+__version__ = "1.0.0"

cosmotech_acceleration_library-1.0.0/cosmotech/coal/aws/__init__.py

@@ -0,0 +1,23 @@
+# Copyright (C) - 2023 - 2025 - Cosmo Tech
+# This document and all information contained herein is the exclusive property -
+# including all intellectual property rights pertaining thereto - of Cosmo Tech.
+# Any use, reproduction, translation, broadcasting, transmission, distribution,
+# etc., to any person is prohibited unless it has been previously and
+# specifically authorized by written means by Cosmo Tech.
+
+"""
+AWS services integration module.
+
+This module provides functions for interacting with AWS services like S3.
+"""
+
+# Re-export S3 functions for easier importing
+from cosmotech.coal.aws.s3 import (
+    create_s3_client,
+    create_s3_resource,
+    upload_file,
+    upload_folder,
+    download_files,
+    upload_data_stream,
+    delete_objects,
+)