monocle-apptrace 0.1.1__tar.gz → 0.3.0__tar.gz

{monocle_apptrace-0.1.1 → monocle_apptrace-0.3.0}/.gitignore

@@ -12,4 +12,5 @@ traces.txt
 dist
 .tox
 .DS_Store
-Pipfile**
+Pipfile**
+launch.json

monocle_apptrace-0.3.0/CHANGELOG.md

@@ -0,0 +1,75 @@
+## Version 0.3.0 (2024-12-10)
+
+- Fixed issue with passing context in async case ([#150](https://github.com/monocle2ai/monocle/pull/150))
+- Added lambda processor ([#148](https://github.com/monocle2ai/monocle/pull/148))
+- Setup package level init scripts to make the monocle import simpler ([#147](https://github.com/monocle2ai/monocle/pull/147))
+- Boto attributes and test cleanup ([#146](https://github.com/monocle2ai/monocle/pull/146))
+- Openai workflow ([#142](https://github.com/monocle2ai/monocle/pull/142))
+- Add input/output for openai embedding ([#141](https://github.com/monocle2ai/monocle/pull/141))
+- Async method and scope fix ([#140](https://github.com/monocle2ai/monocle/pull/140))
+- Bug fix for helper langchain and langgraph ([#137](https://github.com/monocle2ai/monocle/pull/137))
+- Package main script to run any app with monocle instrumentation ([#132](https://github.com/monocle2ai/monocle/pull/132))
+- Add openai api metamodel ([#131](https://github.com/monocle2ai/monocle/pull/131))
+- Support notion of scopes to group traces/snaps into logical constructs ([#130](https://github.com/monocle2ai/monocle/pull/130))
+- Add Llamaindex ReAct agent ([#127](https://github.com/monocle2ai/monocle/pull/127))
+- Langhcain input fix and s3 exporter prefix support ([#126](https://github.com/monocle2ai/monocle/pull/126))
+- Use standard AWS credential envs ([#123](https://github.com/monocle2ai/monocle/pull/123))
+- Check additional attributes for Azure OpenAI model and consolidate common method in utils ([#122](https://github.com/monocle2ai/monocle/pull/122))
+- Bug fix for accessor ([#121](https://github.com/monocle2ai/monocle/pull/121))
+- Bug fix for empty response ([#120](https://github.com/monocle2ai/monocle/pull/120))
+- Bug fix for inference endpoint ([#119](https://github.com/monocle2ai/monocle/pull/119))
+- Opendal exporter for S3 and Blob ([#117](https://github.com/monocle2ai/monocle/pull/117))
+- Handle specific ModuleNotFoundError exceptions gracefully ([#115](https://github.com/monocle2ai/monocle/pull/115))
+- Adding support for console and memory exporter to list of monocle exporters ([#113](https://github.com/monocle2ai/monocle/pull/113))
+- Add trace id propogation for constant trace id and from request ([#111](https://github.com/monocle2ai/monocle/pull/111))
+- Restructure of monoocle code for easy extensibility ([#109](https://github.com/monocle2ai/monocle/pull/109))
+- S3 update filename prefix ([#98](https://github.com/monocle2ai/monocle/pull/98))
+- Update inference span for botocore sagemaker ([#93](https://github.com/monocle2ai/monocle/pull/93))
+- Capturing inference output and token metadata for bedrock ([#82](https://github.com/monocle2ai/monocle/pull/82))
+- Add dev dependency for Mistral AI integration ([#81](https://github.com/monocle2ai/monocle/pull/81))
+- Add VectorStore deployment URL capture support ([#80](https://github.com/monocle2ai/monocle/pull/80))  
+- Clean up cloud exporter implementation ([#79](https://github.com/monocle2ai/monocle/pull/79))
+- Capture inference span input/output events attributes ([#77](https://github.com/monocle2ai/monocle/pull/77))
+- Add release automation workflows ([#76](https://github.com/monocle2ai/monocle/pull/76))
+- Fix gaps in Monocle SDK implementation ([#72](https://github.com/monocle2ai/monocle/pull/72))  
+- Add kwargs and return value handling in Accessor ([#71](https://github.com/monocle2ai/monocle/pull/71))
+- Update workflow name formatting ([#69](https://github.com/monocle2ai/monocle/pull/69))
+- Implement Haystack metamodel support ([#68](https://github.com/monocle2ai/monocle/pull/68))
+
+## Version 0.2.0 (2024-12-05)
+
+## 0.2.0 (Oct 22, 2024)
+
+- Ndjson format for S3 and Blob exporters ([#61](https://github.com/monocle2ai/monocle/pull/61))
+- Set monocle exporter from env setting ([#60](https://github.com/monocle2ai/monocle/pull/60))
+- Update workflow name and type with new format ([#59](https://github.com/monocle2ai/monocle/pull/59))
+- Updated async and custom output processor testcase for metamodel([#58](https://github.com/monocle2ai/monocle/pull/58))
+- Build okahu exporter and added test cases for okahu exporte ([#56](https://github.com/monocle2ai/monocle/pull/56))
+- Handle exception in span wrappers([#52](https://github.com/monocle2ai/monocle/pull/52))
+- Metamodel entity changes ([#51](https://github.com/monocle2ai/monocle/pull/51)), ([#54](https://github.com/monocle2ai/monocle/pull/54))
+- Error handling for llm_endpoint and tags ([#50](https://github.com/monocle2ai/monocle/pull/50))
+- Context_output for vector store retriever ([#48](https://github.com/monocle2ai/monocle/pull/48))
+- Direct exporter - AWS S3 ([#42](https://github.com/monocle2ai/monocle/pull/42))
+- Direct Exporter - Blob store ([#41](https://github.com/monocle2ai/monocle/pull/41))
+- Initial metamodel definition ([#39](https://github.com/monocle2ai/monocle/pull/39))
+- Improvement in vectorstore traces ([#38](https://github.com/monocle2ai/monocle/pull/38))
+- Update key for session context field in attributes ([#34](https://github.com/monocle2ai/monocle/pull/34))
+
+
+## 0.1.0 (Aug 27, 2024)
+
+- Fixed LlamaIndex tracing bugs ([#32](https://github.com/monocle2ai/monocle/pull/32))
+- Added support to add AWS cloud infra attributes ([#29](https://github.com/monocle2ai/monocle/pull/29))
+- Added support to add Azure cloud infra attributes ([#23](https://github.com/monocle2ai/monocle/pull/23))
+- Added support for adding provider name in LLM span in traces ([#22](https://github.com/monocle2ai/monocle/pull/22))
+- Added a default file span exporter ([#21](https://github.com/monocle2ai/monocle/pull/21))
+- Moved input and output context and prompts from attributes to events ([#15](https://github.com/monocle2ai/monocle/pull/15))
+
+
+
+
+
+
+## 0.0.1 (Jul 17, 2024)
+
+- First monocle release

{monocle_apptrace-0.1.1 → monocle_apptrace-0.3.0}/Monocle_User_Guide.md

@@ -1,4 +1,4 @@
-#Monocle User Guide
+# Monocle User Guide
 
 ## Monocle Concepts
 ### Traces
@@ -13,21 +13,42 @@ It’s typically the workflow code components of an application that generate th
 ``` 
     > pip install monocle_apptrace
 ```
+
+- For Azure support (to upload traces to Azure), install with the azure extra:
+```
+    > pip install monocle_apptrace[azure]
+```
+
+- For AWS support (to upload traces to AWS), install with the aws extra:
+```
+    > pip install monocle_apptrace[aws]
+```
+
 - You can locally build and install Monocle library from source
 ```
-> pip install .
+    > pip install .
 ```
 - Install the optional test dependencies listed against dev in pyproject.toml in editable mode
 ```
-> pip install -e ".[dev]"
+    > pip install -e ".[dev]"
 ```
 
+
 ## Using Monocle with your application to generate traces
 ### Enable Monocle tracing
 You need to import monocle package and invoke the API ``setup_monocle_telemetry(workflow=<workflow-name>)`` to enable the tracing. The 'workflow-name' is what you define to identify the give application workflow, for example "customer-chatbot". Monocle trace will include this name in every trace. The trace output will include a list of spans in the traces. You can print the output on the console or send it to an HTTP endpoint.
 
 ### Using Monocle's out of box support of genAI technology components
 Monocle community has done the hard work of figuring out what to trace and how to extract relevant details from multiple genAI technology components. For example, if you have a python app coded using LlamaIndex and using models hostsed in OpenAI, Monocle can seamlessly trace your app. All you need to do enable Monocle tracing.
+
+### Using Monocle's Support for Adding Custom Attributes
+Monocle provides users with the ability to add custom attributes to various spans, such as inference and retrieval spans, by utilizing the output processor within its metamodel. This feature allows for dynamic attribute assignment through lambda functions, which operate on an arguments dictionary.
+The arguments dictionary contains key-value pairs that can be used to compute custom attributes. The dictionary includes the following components: 
+```python
+arguments = {"instance":instance, "args":args, "kwargs":kwargs, "output":return_value}
+```
+By leveraging this dictionary, users can define custom attributes for spans, enabling the integration of additional context and information into the tracing process. The lambda functions used in the attributes field can access and process these values to enrich the span with relevant custom data.
+
 #### Example - Enable Monocle tracing in your application
 ```python
 from monocle_apptrace.instrumentor import setup_monocle_telemetry
@@ -48,7 +69,7 @@ chain.invoke({"number":2})
 # Request callbacks: Finally, let's use the request `callbacks` to achieve the same result
 chain = LLMChain(llm=llm, prompt=prompt)
 chain.invoke({"number":2}, {"callbacks":[handler]})
-    
+
 ```
 
 ### Accessing monocle trace
@@ -63,6 +84,12 @@ setup_monocle_telemetry(workflow_name = "simple_math_app",
 ```
 To print the trace on the console, use ```ConsoleSpanExporter()``` instead of ```FileSpanExporter()```
 
+For Azure:
+    Install the Azure support as shown in the setup section, then use  ```AzureBlobSpanExporter()``` to upload the traces to Azure. 
+
+For AWS:
+    Install the AWS support as shown in the setup section, then use  ```S3SpanExporter()``` to upload the traces to an S3 bucket.
+ 
 ### Leveraging Monocle's extensibility to handle customization 
 When the out of box features from app frameworks are not sufficent, the app developers have to add custom code. For example, if you are extending a LLM class in LlamaIndex to use a model hosted in NVIDIA Triton. This new class is not know to Monocle. You can specify this new class method part of Monocle enabling API and it will be able to trace it.
 

{monocle_apptrace-0.1.1 → monocle_apptrace-0.3.0}/Monocle_committer_guide.md

@@ -22,5 +22,35 @@ After setup of credentials, follow the commands below to publish the package to
 
 ```
 > python3 -m pip install pipenv
-> pipenv install monocle-observability
+> pipenv install monocle-apptrace
 ```
+### Running the testcases
+
+ 
+##### Activate the virtual environment
+```
+cd monocle
+monocle% python -m pip install pipenv
+monocle% pipenv --python 3.11.9
+monocle% source $(pipenv --venv)/bin/activate
+
+ ```
+##### Install the dependencies
+```
+monocle% pip install -e '.[dev]'
+ ```
+
+##### Run the unit tests
+```
+monocle% pytest tests/unit/*_test.py
+ ```
+ 
+##### Run the integration tests
+```
+monocle% pytest -m integration
+ ```
+
+###### Run the integration test individually
+```
+monocle% pytest tests/integration/test_langchain_rag_l_to_m.py
+```

{monocle_apptrace-0.1.1 → monocle_apptrace-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: monocle_apptrace
-Version: 0.1.1
+Version: 0.3.0
 Summary: package with monocle genAI tracing
 Project-URL: Homepage, https://github.com/monocle2ai/monocle
 Project-URL: Issues, https://github.com/monocle2ai/monocle/issues
@@ -15,20 +15,41 @@ Requires-Dist: opentelemetry-instrumentation
 Requires-Dist: opentelemetry-sdk>=1.21.0
 Requires-Dist: requests
 Requires-Dist: wrapt>=1.14.0
+Provides-Extra: aws
+Requires-Dist: boto3==1.35.19; extra == 'aws'
+Provides-Extra: azure
+Requires-Dist: azure-storage-blob==12.22.0; extra == 'azure'
 Provides-Extra: dev
+Requires-Dist: azure-storage-blob==12.22.0; extra == 'dev'
+Requires-Dist: boto3==1.34.131; extra == 'dev'
+Requires-Dist: chromadb==0.4.22; extra == 'dev'
 Requires-Dist: datasets==2.20.0; extra == 'dev'
 Requires-Dist: faiss-cpu==1.8.0; extra == 'dev'
+Requires-Dist: flask; extra == 'dev'
+Requires-Dist: haystack-ai==2.3.0; extra == 'dev'
 Requires-Dist: instructorembedding==1.0.1; extra == 'dev'
+Requires-Dist: langchain-aws==0.1.10; extra == 'dev'
 Requires-Dist: langchain-chroma==0.1.1; extra == 'dev'
 Requires-Dist: langchain-community==0.2.5; extra == 'dev'
+Requires-Dist: langchain-mistralai==0.1.13; extra == 'dev'
 Requires-Dist: langchain-openai==0.1.8; extra == 'dev'
 Requires-Dist: langchain==0.2.5; extra == 'dev'
+Requires-Dist: langchainhub==0.1.21; extra == 'dev'
+Requires-Dist: langgraph==0.2.68; extra == 'dev'
 Requires-Dist: llama-index-embeddings-huggingface==0.2.0; extra == 'dev'
+Requires-Dist: llama-index-llms-azure-openai==0.1.9; extra == 'dev'
+Requires-Dist: llama-index-llms-mistralai==0.1.20; extra == 'dev'
 Requires-Dist: llama-index-vector-stores-chroma==0.1.9; extra == 'dev'
+Requires-Dist: llama-index-vector-stores-opensearch==0.1.10; extra == 'dev'
 Requires-Dist: llama-index==0.10.30; extra == 'dev'
+Requires-Dist: mistral-haystack==0.0.2; extra == 'dev'
 Requires-Dist: numpy==1.26.4; extra == 'dev'
+Requires-Dist: opendal==0.45.14; extra == 'dev'
+Requires-Dist: opensearch-haystack==1.2.0; extra == 'dev'
+Requires-Dist: opentelemetry-instrumentation-flask; extra == 'dev'
 Requires-Dist: parameterized==0.9.0; extra == 'dev'
 Requires-Dist: pytest==8.0.0; extra == 'dev'
+Requires-Dist: requests-aws4auth==1.2.3; extra == 'dev'
 Requires-Dist: sentence-transformers==2.6.1; extra == 'dev'
 Requires-Dist: types-requests==2.31.0.20240106; extra == 'dev'
 Description-Content-Type: text/markdown

{monocle_apptrace-0.1.1 → monocle_apptrace-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "monocle_apptrace"
-version = "0.1.1"
+version = "0.3.0"
 authors = []
 description = "package with monocle genAI tracing"
 readme = "README.md"
@@ -23,8 +23,20 @@ dependencies = [
 ]
 
 [tool.pytest.ini_options]
+#log_cli = true
+#log_cli_level = "INFO"
+# Enable file logging
+log_file = "traces.log"
+log_file_level = "INFO"
+log_format = "%(asctime)s %(levelname)s %(pathname)s:%(lineno)d %(message)s"
+log_date_format = "%Y-%m-%d %H:%M:%S"
+
 pythonpath = [
-  "src"
+  "src",
+  "tests"
+]
+markers = [
+  "integration"
 ]
 
 [project.optional-dependencies]
@@ -43,7 +55,32 @@ dev = [
   'llama-index==0.10.30',
   'llama-index-embeddings-huggingface==0.2.0',
   'llama-index-vector-stores-chroma==0.1.9',
-  'parameterized==0.9.0'
+  'parameterized==0.9.0',
+  'llama-index-llms-mistralai==0.1.20',
+  'langchain-mistralai==0.1.13',
+  'mistral-haystack==0.0.2',
+  'langchain-aws==0.1.10',
+  'azure-storage-blob==12.22.0', # this is for blob exporter
+  'boto3==1.34.131', # this is for aws exporter
+  'llama-index-vector-stores-opensearch==0.1.10',
+  'haystack-ai==2.3.0',
+  'llama-index-llms-azure-openai==0.1.9',
+  'requests-aws4auth==1.2.3',
+  'opensearch-haystack==1.2.0',
+  'langchainhub==0.1.21',
+  'chromadb==0.4.22',
+  'flask',
+  'opentelemetry-instrumentation-flask',
+  'langgraph==0.2.68',
+  'opendal==0.45.14'
+]
+
+azure = [
+    'azure-storage-blob==12.22.0',
+]
+
+aws = [
+    'boto3==1.35.19',
 ]
 
 [project.urls]

monocle_apptrace-0.3.0/src/monocle_apptrace/__init__.py

@@ -0,0 +1 @@
+from .instrumentation import *

monocle_apptrace-0.3.0/src/monocle_apptrace/__main__.py

@@ -0,0 +1,19 @@
+import sys, os
+import runpy
+from monocle_apptrace.instrumentation.common.instrumentor import setup_monocle_telemetry
+
+def main():
+    if len(sys.argv) < 2 or not sys.argv[1].endswith(".py"):
+        print("Usage: python -m monocle_apptrace <your-main-module-file> <args>")
+        sys.exit(1)
+    file_name = os.path.basename(sys.argv[1])
+    workflow_name = file_name[:-3]
+    setup_monocle_telemetry(workflow_name=workflow_name)
+    sys.argv.pop(0)
+
+    try:
+        runpy.run_path(path_name=sys.argv[0], run_name="__main__")
+    except Exception as e:
+        print(e)
+if __name__ == "__main__":
+    main()

monocle_apptrace-0.3.0/src/monocle_apptrace/exporters/aws/s3_exporter.py

@@ -0,0 +1,181 @@
+import os
+import time
+import random
+import datetime
+import logging
+import asyncio
+import boto3
+from botocore.exceptions import ClientError
+from botocore.exceptions import (
+    BotoCoreError,
+    ConnectionClosedError,
+    ConnectTimeoutError,
+    EndpointConnectionError,
+    ReadTimeoutError,
+)
+from opentelemetry.sdk.trace import ReadableSpan
+from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult
+from monocle_apptrace.exporters.base_exporter import SpanExporterBase
+from monocle_apptrace.exporters.exporter_processor import ExportTaskProcessor
+from typing import Sequence, Optional
+import json
+logger = logging.getLogger(__name__)
+
+class S3SpanExporter(SpanExporterBase):
+    def __init__(self, bucket_name=None, region_name=None, task_processor: Optional[ExportTaskProcessor] = None):
+        super().__init__()
+        # Use environment variables if credentials are not provided
+        DEFAULT_FILE_PREFIX = "monocle_trace_"
+        DEFAULT_TIME_FORMAT = "%Y-%m-%d__%H.%M.%S"
+        self.max_batch_size = 500
+        self.export_interval = 1
+        if(os.getenv('MONOCLE_AWS_ACCESS_KEY_ID') and os.getenv('MONOCLE_AWS_SECRET_ACCESS_KEY')):
+            self.s3_client = boto3.client(
+                's3',
+                aws_access_key_id=os.getenv('MONOCLE_AWS_ACCESS_KEY_ID'),
+                aws_secret_access_key=os.getenv('MONOCLE_AWS_SECRET_ACCESS_KEY'),
+                region_name=region_name,
+            )
+        else:
+            self.s3_client = boto3.client(
+                's3',
+                aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),
+                aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY'),
+                region_name=region_name,
+            )
+        self.bucket_name = bucket_name or os.getenv('MONOCLE_S3_BUCKET_NAME','default-bucket')
+        self.file_prefix = os.getenv('MONOCLE_S3_KEY_PREFIX', DEFAULT_FILE_PREFIX)
+        self.time_format = DEFAULT_TIME_FORMAT
+        self.export_queue = []
+        self.last_export_time = time.time()
+        self.task_processor = task_processor
+        if self.task_processor is not None:
+            self.task_processor.start()
+
+        # Check if bucket exists or create it
+        if not self.__bucket_exists(self.bucket_name):
+            try:
+                self.s3_client.create_bucket(
+                Bucket=self.bucket_name,
+                CreateBucketConfiguration={'LocationConstraint': region_name}
+                 )
+                logger.info(f"Bucket {self.bucket_name} created successfully.")
+            except ClientError as e:
+                logger.error(f"Error creating bucket {self.bucket_name}: {e}")
+                raise e
+
+    def __bucket_exists(self, bucket_name):
+        try:
+            # Check if the bucket exists by calling head_bucket
+            self.s3_client.head_bucket(Bucket=bucket_name)
+            return True
+        except ClientError as e:
+            error_code = e.response['Error']['Code']
+            if error_code == '404':
+                # Bucket not found
+                logger.error(f"Bucket {bucket_name} does not exist (404).")
+                return False
+            elif error_code == '403':
+                # Permission denied
+                logger.error(f"Access to bucket {bucket_name} is forbidden (403).")
+                raise PermissionError(f"Access to bucket {bucket_name} is forbidden.")
+            elif error_code == '400':
+                # Bad request or malformed input
+                logger.error(f"Bad request for bucket {bucket_name} (400).")
+                raise ValueError(f"Bad request for bucket {bucket_name}.")
+            else:
+                # Other client errors
+                logger.error(f"Unexpected error when accessing bucket {bucket_name}: {e}")
+                raise e
+        except TypeError as e:
+            # Handle TypeError separately
+            logger.error(f"Type error while checking bucket existence: {e}")
+            raise e
+
+    def export(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:
+        """Synchronous export method that internally handles async logic."""
+        try:
+            # Run the asynchronous export logic in an event loop
+            logger.info(f"Exporting {len(spans)} spans to S3.")
+            asyncio.run(self.__export_async(spans))
+            return SpanExportResult.SUCCESS
+        except Exception as e:
+            logger.error(f"Error exporting spans: {e}")
+            return SpanExportResult.FAILURE
+
+    async def __export_async(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:
+        try:
+            logger.info(f"__export_async {len(spans)} spans to S3.")
+            # Add spans to the export queue
+            for span in spans:
+                self.export_queue.append(span)
+                # If the queue reaches MAX_BATCH_SIZE, export the spans
+                if len(self.export_queue) >= self.max_batch_size:
+                    await self.__export_spans()
+
+            # Check if it's time to force a flush
+            current_time = time.time()
+            if current_time - self.last_export_time >= self.export_interval:
+                await self.__export_spans()  # Export spans if time interval has passed
+                self.last_export_time = current_time  # Reset the last export time
+
+            return SpanExportResult.SUCCESS
+        except Exception as e:
+            logger.error(f"Error exporting spans: {e}")
+            return SpanExportResult.FAILURE
+
+    def __serialize_spans(self, spans: Sequence[ReadableSpan]) -> str:
+        try:
+            # Serialize spans to JSON or any other format you prefer
+            valid_json_list = []
+            for span in spans:
+                try:
+                    valid_json_list.append(span.to_json(indent=0).replace("\n", ""))
+                except json.JSONDecodeError as e:
+                    logger.warning(f"Invalid JSON format in span data: {span.context.span_id}. Error: {e}")
+                    continue
+            ndjson_data = "\n".join(valid_json_list) + "\n"
+            return ndjson_data
+        except Exception as e:
+            logger.warning(f"Error serializing spans: {e}")
+
+
+    async def __export_spans(self):
+        if len(self.export_queue) == 0:
+            return
+
+        # Take a batch of spans from the queue
+        batch_to_export = self.export_queue[:self.max_batch_size]
+        serialized_data = self.__serialize_spans(batch_to_export)
+        self.export_queue = self.export_queue[self.max_batch_size:]
+        # to calculate is_root_span loop over each span in batch_to_export and check if parent id is none or null
+        is_root_span = any(not span.parent for span in batch_to_export)
+        logger.info(f"Exporting {len(batch_to_export)} spans to S3 is_root_span : {is_root_span}.")
+        if self.task_processor is not None and callable(getattr(self.task_processor, 'queue_task', None)):
+            self.task_processor.queue_task(self.__upload_to_s3, serialized_data, is_root_span)
+        else:
+            try:
+                self.__upload_to_s3(serialized_data)
+            except Exception as e:
+                logger.error(f"Failed to upload span batch: {e}")
+
+    @SpanExporterBase.retry_with_backoff(exceptions=(EndpointConnectionError, ConnectionClosedError, ReadTimeoutError, ConnectTimeoutError))
+    def __upload_to_s3(self, span_data_batch: str):
+        current_time = datetime.datetime.now().strftime(self.time_format)
+        prefix = self.file_prefix + os.environ.get('MONOCLE_S3_KEY_PREFIX_CURRENT', '')
+        file_name = f"{prefix}{current_time}.ndjson"
+        self.s3_client.put_object(
+            Bucket=self.bucket_name,
+            Key=file_name,
+            Body=span_data_batch
+        )
+        logger.info(f"Span batch uploaded to AWS S3 as {file_name}.")
+
+    async def force_flush(self, timeout_millis: int = 30000) -> bool:
+        await self.__export_spans()  # Export any remaining spans in the queue
+        return True
+
+    def shutdown(self) -> None:
+        if hasattr(self, 'task_processor') and self.task_processor is not None:
+            self.task_processor.stop()
+        logger.info("S3SpanExporter has been shut down.")

monocle_apptrace-0.3.0/src/monocle_apptrace/exporters/aws/s3_exporter_opendal.py

@@ -0,0 +1,137 @@
+import os
+import time
+import datetime
+import logging
+import asyncio
+from typing import Sequence, Optional
+from opentelemetry.sdk.trace import ReadableSpan
+from opentelemetry.sdk.trace.export import SpanExportResult
+from monocle_apptrace.exporters.base_exporter import SpanExporterBase
+from monocle_apptrace.exporters.exporter_processor import ExportTaskProcessor
+from opendal import Operator
+from opendal.exceptions import PermissionDenied, ConfigInvalid, Unexpected
+
+import json
+
+logger = logging.getLogger(__name__)
+class OpenDALS3Exporter(SpanExporterBase):
+    def __init__(self, bucket_name=None, region_name=None, task_processor: Optional[ExportTaskProcessor] = None):
+        super().__init__()
+        DEFAULT_FILE_PREFIX = "monocle_trace_"
+        DEFAULT_TIME_FORMAT = "%Y-%m-%d__%H.%M.%S"
+        self.max_batch_size = 500
+        self.export_interval = 1
+        self.file_prefix = DEFAULT_FILE_PREFIX
+        self.time_format = DEFAULT_TIME_FORMAT
+        self.export_queue = []
+        self.last_export_time = time.time()
+        self.bucket_name = bucket_name or os.getenv("MONOCLE_S3_BUCKET_NAME", "default-bucket")
+
+        # Initialize OpenDAL S3 operator
+        self.op = Operator(
+            "s3",
+            root = "/",
+            region=os.getenv("AWS_REGION", region_name),
+            bucket=self.bucket_name,
+            access_key_id=os.getenv("AWS_ACCESS_KEY_ID"),
+            secret_access_key=os.getenv("AWS_SECRET_ACCESS_KEY"),
+        )
+        
+        self.task_processor = task_processor
+        if self.task_processor is not None:
+            self.task_processor.start()
+
+    def export(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:
+        """Synchronous export method that internally handles async logic."""
+        try:
+            # Run the asynchronous export logic in an event loop
+            asyncio.run(self.__export_async(spans))
+            return SpanExportResult.SUCCESS
+        except Exception as e:
+            logger.error(f"Error exporting spans: {e}")
+            return SpanExportResult.FAILURE
+
+    async def __export_async(self, spans: Sequence[ReadableSpan]) -> SpanExportResult:
+        try:
+            # Add spans to the export queue
+            for span in spans:
+                self.export_queue.append(span)
+                if len(self.export_queue) >= self.max_batch_size:
+                    await self.__export_spans()
+
+            # Check if it's time to force a flush
+            current_time = time.time()
+            if current_time - self.last_export_time >= self.export_interval:
+                await self.__export_spans()
+                self.last_export_time = current_time
+
+            return SpanExportResult.SUCCESS
+        except Exception as e:
+            logger.error(f"Error exporting spans: {e}")
+            return SpanExportResult.FAILURE
+
+    def __serialize_spans(self, spans: Sequence[ReadableSpan]) -> str:
+        try:
+            # Serialize spans to JSON or any other format you prefer
+            valid_json_list = []
+            for span in spans:
+                try:
+                    valid_json_list.append(span.to_json(indent=0).replace("\n", ""))
+                except json.JSONDecodeError as e:
+                    logger.warning(f"Invalid JSON format in span data: {span.context.span_id}. Error: {e}")
+                    continue
+            return "\n".join(valid_json_list) + "\n"
+        except Exception as e:
+            logger.warning(f"Error serializing spans: {e}")
+
+    async def __export_spans(self):
+        if not self.export_queue:
+            return
+        # Take a batch of spans from the queue
+        batch_to_export = self.export_queue[:self.max_batch_size]
+        serialized_data = self.__serialize_spans(batch_to_export)
+        self.export_queue = self.export_queue[self.max_batch_size:]
+        
+        # Calculate is_root_span by checking if any span has no parent
+        is_root_span = any(not span.parent for span in batch_to_export)
+        
+        if self.task_processor is not None and callable(getattr(self.task_processor, 'queue_task', None)):
+            self.task_processor.queue_task(self.__upload_to_s3, serialized_data, is_root_span)
+        else:
+            try:
+                self.__upload_to_s3(serialized_data, is_root_span)
+            except Exception as e:
+                logger.error(f"Failed to upload span batch: {e}")
+
+    @SpanExporterBase.retry_with_backoff(exceptions=(Unexpected))
+    def __upload_to_s3(self, span_data_batch: str, is_root_span: bool = False):
+        current_time = datetime.datetime.now().strftime(self.time_format)
+        file_name = f"{self.file_prefix}{current_time}.ndjson"
+        try:
+            # Attempt to write the span data batch to S3
+            self.op.write(file_name, span_data_batch.encode("utf-8"))
+            logger.info(f"Span batch uploaded to S3 as {file_name}. Is root span: {is_root_span}")
+
+        except PermissionDenied as e:
+            # S3 bucket is forbidden.
+            logger.error(f"Access to bucket {self.bucket_name} is forbidden (403).")
+            raise PermissionError(f"Access to bucket {self.bucket_name} is forbidden.")
+
+        except ConfigInvalid as e:
+            # Bucket does not exist.
+            if "404" in str(e):
+                logger.error("Bucket does not exist. Please check the bucket name and region.")
+                raise Exception(f"Bucket does not exist. Error: {e}")
+            else:
+                logger.error(f"Unexpected error when accessing bucket {self.bucket_name}: {e}")
+                raise e
+
+
+    async def force_flush(self, timeout_millis: int = 30000) -> bool:
+        await self.__export_spans()
+        return True
+
+    def shutdown(self) -> None:
+        if hasattr(self, 'task_processor') and self.task_processor is not None:
+            self.task_processor.stop()
+        logger.info("S3SpanExporter has been shut down.")