atlan-application-sdk 0.1.1rc33__tar.gz → 0.1.1rc35__tar.gz

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/.cursor/BUGBOT.md

@@ -58,3 +58,63 @@
 - **Metrics**: Key operations include appropriate metrics
 - **Error Context**: Error logs include sufficient context for debugging
 - **Trace Information**: Critical paths include trace information
+
+## 🚨 Automated Pattern Detection Rules
+
+### Unit Test Requirements
+
+- **Mandatory Test Coverage**: PRs with titles starting with "fix:" MUST include corresponding test file changes
+- **Test File Naming**: Test files must follow the pattern `test_*.py` and mirror source structure
+
+### File Naming Conventions
+
+- **Model/Dataclass Files**: Files named `base.py` containing dataclasses or pydantic models should be renamed to `models.py` or `model.py`
+- **Descriptive Naming**: File names should clearly indicate their purpose and contents
+- **Consistency**: Follow established naming patterns within the repository
+
+### Directory Structure
+
+- **Single File Directories**: Directories containing only one file should be reorganized (e.g., `events/` folder with only one file)
+- **Logical Grouping**: Related functionality should be grouped in appropriate subdirectories
+- **Module Organization**: Avoid creating unnecessary directory nesting for single files
+
+### Magic String Constants
+
+- **Operation Strings**: Hardcoded strings like `"create"`, `"update"`, `"delete"` in business logic should be extracted to constants or enums.
+- **Configuration Values**: String literals used across multiple files that are loaded from environment variables should be defined in `constants.py`.
+
+### Method Naming
+
+- **Verbose Names**: Method names longer than 25 characters with redundant context should be simplified
+- **Clear Intent**: Method names should be concise but descriptive of their primary purpose
+- **Context Redundancy**: Avoid repeating class or module context in method names
+
+### Dead Code Detection
+
+- **Conditional Blocks**: Flag `if False:` blocks and suggest removal
+- **Unused Boolean Flags**: Detect and remove unused boolean flags and conditional logic
+- **Commented Code**: Remove commented-out code blocks that are no longer needed
+- **Unreachable Code**: Identify and remove code that can never be executed
+
+### Code Duplication
+
+- **Utility Extraction**: Repeated logic should be extracted into shared utility methods
+- **Configuration Patterns**: Similar configuration setup code should be centralized
+
+### File Management
+
+- **Premature Creation**: Flag new empty files or files with minimal content that aren't referenced elsewhere
+- **Unused Files**: Detect files added then immediately marked for removal in the same PR
+- **File Purpose**: Ensure every new file serves a clear, documented purpose
+
+### Import Organization
+
+- **Stale Imports**: When code is moved between files, check if old imports still reference moved code
+- **Import Cleanup**: Remove unused imports after code reorganization
+- **Dependency Management**: Ensure imports reflect actual code dependencies
+
+### Async Pattern Consistency
+
+- **Workflow Context**: In workflow/temporal contexts, flag sync method calls that should be async
+- **Mixed Patterns**: Ensure consistent async/await usage within the same module
+- **Temporal Best Practices**: Follow Temporal SDK guidelines for async operations

atlan_application_sdk-0.1.1rc35/.github/release_notes.md

@@ -0,0 +1,7 @@
+## v0.1.1rc35 (August 28, 2025)
+
+Full Changelog: https://github.com/atlanhq/application-sdk/compare/v0.1.1rc34...v0.1.1rc35
+
+### Features
+
+- improve services abstraction (#658) (by @Nishchith Shetty in [f5a4c4e](https://github.com/atlanhq/application-sdk/commit/f5a4c4e))

atlan_application_sdk-0.1.1rc35/.github/workflows/verify-snyk-status.yml

@@ -0,0 +1,17 @@
+name: 'Verify Snyk PR Checks'
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+    branches:
+      - 'main'
+      - 'master'
+
+jobs:
+  snyk_scan_results:
+    # This 'name' is what appears in the pull request's check list.
+    name: Snyk Scan Results
+    # Calls the reusable workflow from your central .github repository.
+    uses: atlanhq/.github/.github/workflows/reusable-verify-snyk-scans.yml@main
+    # Allows the reusable workflow to use the GITHUB_TOKEN from this repository.
+    secrets: inherit

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## v0.1.1rc35 (August 28, 2025)
+
+Full Changelog: https://github.com/atlanhq/application-sdk/compare/v0.1.1rc34...v0.1.1rc35
+
+### Features
+
+- improve services abstraction (#658) (by @Nishchith Shetty in [f5a4c4e](https://github.com/atlanhq/application-sdk/commit/f5a4c4e))
+
+
+
+## v0.1.1rc34 (August 27, 2025)
+
+Full Changelog: https://github.com/atlanhq/application-sdk/compare/v0.1.1rc33...v0.1.1rc34
+
+### Features
+
+- Generic Client and Handlers for non-SQL connectors with http request methods (allows custom retry transport) (#660) (by @Prateek Rai in [b077a90](https://github.com/atlanhq/application-sdk/commit/b077a90))
+
 ## v0.1.1rc33 (August 26, 2025)
 
 Full Changelog: https://github.com/atlanhq/application-sdk/compare/v0.1.1rc32...v0.1.1rc33

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: atlan-application-sdk
-Version: 0.1.1rc33
+Version: 0.1.1rc35
 Summary: Atlan Application SDK is a Python library for developing applications on the Atlan Platform
 Project-URL: Repository, https://github.com/atlanhq/application-sdk
 Project-URL: Documentation, https://github.com/atlanhq/application-sdk/README.md

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/application_sdk/activities/__init__.py

@@ -29,7 +29,6 @@ from application_sdk.activities.common.utils import (
 from application_sdk.common.error_codes import OrchestratorError
 from application_sdk.constants import TEMPORARY_PATH
 from application_sdk.handlers import HandlerInterface
-from application_sdk.inputs.statestore import StateStoreInput, StateType
 from application_sdk.observability.logger_adaptor import get_logger
 
 logger = get_logger(__name__)
@@ -190,7 +189,9 @@ class ActivitiesInterface(ABC, Generic[ActivitiesStateType]):
 
         try:
             # This already handles the Dapr call internally
-            workflow_args = StateStoreInput.get_state(workflow_id, StateType.WORKFLOWS)
+            from application_sdk.services.statestore import StateStore, StateType
+
+            workflow_args = await StateStore.get_state(workflow_id, StateType.WORKFLOWS)
             workflow_args["output_prefix"] = workflow_args.get(
                 "output_prefix", TEMPORARY_PATH
             )

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/application_sdk/activities/common/utils.py

@@ -5,13 +5,18 @@ including workflow ID retrieval, automatic heartbeating, and periodic heartbeat
 """
 
 import asyncio
+import os
 from datetime import timedelta
 from functools import wraps
 from typing import Any, Awaitable, Callable, Optional, TypeVar, cast
 
 from temporalio import activity
 
-from application_sdk.constants import APPLICATION_NAME, WORKFLOW_OUTPUT_PATH_TEMPLATE
+from application_sdk.constants import (
+    APPLICATION_NAME,
+    TEMPORARY_PATH,
+    WORKFLOW_OUTPUT_PATH_TEMPLATE,
+)
 from application_sdk.observability.logger_adaptor import get_logger
 
 logger = get_logger(__name__)
@@ -72,6 +77,21 @@ def build_output_path() -> str:
     )
 
 
+def get_object_store_prefix(path: str) -> str:
+    """Get the object store prefix for the path.
+    Args:
+        path: The path to the output directory.
+
+    Returns:
+        The object store prefix for the path.
+
+    Example:
+        >>> get_object_store_prefix("./local/tmp/artifacts/apps/appName/workflows/wf-123/run-456")
+        "artifacts/apps/appName/workflows/wf-123/run-456"
+    """
+    return os.path.relpath(path, TEMPORARY_PATH)
+
+
 def auto_heartbeater(fn: F) -> F:
     """Decorator that automatically sends heartbeats during activity execution.
 

atlan_application_sdk-0.1.1rc35/application_sdk/activities/metadata_extraction/base.py

@@ -0,0 +1,104 @@
+from typing import Any, Dict, Optional, Type
+
+from temporalio import activity
+
+from application_sdk.activities import ActivitiesInterface, ActivitiesState
+from application_sdk.activities.common.utils import get_workflow_id
+from application_sdk.clients.base import BaseClient
+from application_sdk.constants import APP_TENANT_ID, APPLICATION_NAME
+from application_sdk.handlers.base import BaseHandler
+from application_sdk.observability.logger_adaptor import get_logger
+from application_sdk.services.secretstore import SecretStore
+from application_sdk.transformers import TransformerInterface
+
+logger = get_logger(__name__)
+activity.logger = logger
+
+
+class BaseMetadataExtractionActivitiesState(ActivitiesState):
+    """State for base metadata extraction activities."""
+
+    client: Optional[BaseClient] = None
+    handler: Optional[BaseHandler] = None
+    transformer: Optional[TransformerInterface] = None
+
+
+class BaseMetadataExtractionActivities(ActivitiesInterface):
+    """Base activities for non-SQL metadata extraction workflows."""
+
+    _state: Dict[str, BaseMetadataExtractionActivitiesState] = {}
+
+    client_class: Type[BaseClient] = BaseClient
+    handler_class: Type[BaseHandler] = BaseHandler
+    transformer_class: Optional[Type[TransformerInterface]] = None
+
+    def __init__(
+        self,
+        client_class: Optional[Type[BaseClient]] = None,
+        handler_class: Optional[Type[BaseHandler]] = None,
+        transformer_class: Optional[Type[TransformerInterface]] = None,
+    ):
+        """Initialize the base metadata extraction activities.
+
+        Args:
+            client_class: Client class to use. Defaults to BaseClient.
+            handler_class: Handler class to use. Defaults to BaseHandler.
+            transformer_class: Transformer class to use. Users must provide their own transformer implementation.
+        """
+        if client_class:
+            self.client_class = client_class
+        if handler_class:
+            self.handler_class = handler_class
+        if transformer_class:
+            self.transformer_class = transformer_class
+
+        super().__init__()
+
+    async def _set_state(self, workflow_args: Dict[str, Any]):
+        """Set up the state for the current workflow.
+
+        Args:
+            workflow_args: Arguments for the workflow.
+        """
+        workflow_id = get_workflow_id()
+        if not self._state.get(workflow_id):
+            self._state[workflow_id] = BaseMetadataExtractionActivitiesState()
+
+        await super()._set_state(workflow_args)
+
+        state = self._state[workflow_id]
+
+        # Initialize client
+        client = self.client_class()
+        # Extract credentials from state store if credential_guid is available
+        if "credential_guid" in workflow_args:
+            logger.info(
+                f"Retrieving credentials for credential_guid: {workflow_args['credential_guid']}"
+            )
+            try:
+                credentials = await SecretStore.get_credentials(
+                    workflow_args["credential_guid"]
+                )
+                logger.info(
+                    f"Successfully retrieved credentials with keys: {list(credentials.keys())}"
+                )
+                # Load the client with credentials
+                await client.load(credentials=credentials)
+            except Exception as e:
+                logger.error(f"Failed to retrieve credentials: {e}")
+                raise
+
+        state.client = client
+
+        # Initialize handler
+        handler = self.handler_class(client=client)
+        state.handler = handler
+
+        # Initialize transformer if provided
+        if self.transformer_class:
+            transformer_params = {
+                "connector_name": APPLICATION_NAME,
+                "connector_type": APPLICATION_NAME,
+                "tenant_id": APP_TENANT_ID,
+            }
+            state.transformer = self.transformer_class(**transformer_params)

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/application_sdk/activities/metadata_extraction/sql.py

@@ -5,25 +5,24 @@ from temporalio import activity
 
 from application_sdk.activities import ActivitiesInterface, ActivitiesState
 from application_sdk.activities.common.models import ActivityStatistics
-from application_sdk.activities.common.utils import auto_heartbeater, get_workflow_id
+from application_sdk.activities.common.utils import (
+    auto_heartbeater,
+    get_object_store_prefix,
+    get_workflow_id,
+)
 from application_sdk.clients.sql import BaseSQLClient
-from application_sdk.common.credential_utils import get_credentials
 from application_sdk.common.dataframe_utils import is_empty_dataframe
 from application_sdk.common.error_codes import ActivityError
 from application_sdk.common.utils import prepare_query, read_sql_files
-from application_sdk.constants import (
-    APP_TENANT_ID,
-    APPLICATION_NAME,
-    SQL_QUERIES_PATH,
-    TEMPORARY_PATH,
-)
+from application_sdk.constants import APP_TENANT_ID, APPLICATION_NAME, SQL_QUERIES_PATH
 from application_sdk.handlers.sql import BaseSQLHandler
 from application_sdk.inputs.parquet import ParquetInput
 from application_sdk.inputs.sql_query import SQLQueryInput
 from application_sdk.observability.logger_adaptor import get_logger
-from application_sdk.outputs.atlan_storage import AtlanStorageOutput
 from application_sdk.outputs.json import JsonOutput
 from application_sdk.outputs.parquet import ParquetOutput
+from application_sdk.services.atlan_storage import AtlanStorage
+from application_sdk.services.secretstore import SecretStore
 from application_sdk.transformers import TransformerInterface
 from application_sdk.transformers.query import QueryBasedTransformer
 
@@ -144,7 +143,9 @@ class BaseSQLMetadataExtractionActivities(ActivitiesInterface):
         self._state[workflow_id].handler = handler
 
         if "credential_guid" in workflow_args:
-            credentials = await get_credentials(workflow_args["credential_guid"])
+            credentials = await SecretStore.get_credentials(
+                workflow_args["credential_guid"]
+            )
             await sql_client.load(credentials)
 
         self._state[workflow_id].sql_client = sql_client
@@ -536,11 +537,11 @@ class BaseSQLMetadataExtractionActivities(ActivitiesInterface):
 
         # Upload data from object store to Atlan storage
         # Use workflow_id/workflow_run_id as the prefix to migrate specific data
-        migration_prefix = os.path.relpath(workflow_args["output_path"], TEMPORARY_PATH)
+        migration_prefix = get_object_store_prefix(workflow_args["output_path"])
         logger.info(
             f"Starting migration from object store with prefix: {migration_prefix}"
         )
-        upload_stats = await AtlanStorageOutput.migrate_from_objectstore_to_atlan(
+        upload_stats = await AtlanStorage.migrate_from_objectstore_to_atlan(
             prefix=migration_prefix
         )
 

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/application_sdk/activities/query_extraction/sql.py

@@ -7,17 +7,20 @@ from pydantic import BaseModel, Field
 from temporalio import activity
 
 from application_sdk.activities import ActivitiesInterface, ActivitiesState
-from application_sdk.activities.common.utils import auto_heartbeater, get_workflow_id
+from application_sdk.activities.common.utils import (
+    auto_heartbeater,
+    get_object_store_prefix,
+    get_workflow_id,
+)
 from application_sdk.clients.sql import BaseSQLClient
-from application_sdk.common.credential_utils import get_credentials
 from application_sdk.constants import UPSTREAM_OBJECT_STORE_NAME
 from application_sdk.handlers import HandlerInterface
 from application_sdk.handlers.sql import BaseSQLHandler
-from application_sdk.inputs.objectstore import ObjectStoreInput
 from application_sdk.inputs.sql_query import SQLQueryInput
 from application_sdk.observability.logger_adaptor import get_logger
-from application_sdk.outputs.objectstore import ObjectStoreOutput
 from application_sdk.outputs.parquet import ParquetOutput
+from application_sdk.services.objectstore import ObjectStore
+from application_sdk.services.secretstore import SecretStore
 from application_sdk.transformers import TransformerInterface
 from application_sdk.transformers.atlas import AtlasTransformer
 
@@ -129,7 +132,9 @@ class SQLQueryExtractionActivities(ActivitiesInterface):
         workflow_id = get_workflow_id()
         sql_client = self.sql_client_class()
         if "credential_guid" in workflow_args:
-            credentials = await get_credentials(workflow_args["credential_guid"])
+            credentials = await SecretStore.get_credentials(
+                workflow_args["credential_guid"]
+            )
             await sql_client.load(credentials)
 
         handler = self.handler_class(sql_client)
@@ -412,14 +417,14 @@ class SQLQueryExtractionActivities(ActivitiesInterface):
             f.write(last_marker)
 
         logger.info(f"Last marker: {last_marker}")
-        await ObjectStoreOutput.push_file_to_object_store(
-            workflow_args["output_prefix"],
-            marker_file_path,
-            object_store_name=UPSTREAM_OBJECT_STORE_NAME,
+        await ObjectStore.upload_file(
+            source=marker_file_path,
+            destination=get_object_store_prefix(marker_file_path),
+            store_name=UPSTREAM_OBJECT_STORE_NAME,
         )
         logger.info(f"Marker file written to {marker_file_path}")
 
-    def read_marker(self, workflow_args: Dict[str, Any]) -> Optional[int]:
+    async def read_marker(self, workflow_args: Dict[str, Any]) -> Optional[int]:
         """Read the marker from the output path.
 
         This method reads the current marker value from a marker file to determine the
@@ -442,15 +447,12 @@ class SQLQueryExtractionActivities(ActivitiesInterface):
             marker_file_path = os.path.join(output_path, "markerfile")
             logger.info(f"Downloading marker file from {marker_file_path}")
 
-            os.makedirs(workflow_args["output_prefix"], exist_ok=True)
-
-            ObjectStoreInput.download_file_from_object_store(
-                workflow_args["output_prefix"],
-                marker_file_path,
-                object_store_name=UPSTREAM_OBJECT_STORE_NAME,
+            await ObjectStore.download_file(
+                source=get_object_store_prefix(marker_file_path),
+                destination=marker_file_path,
+                store_name=UPSTREAM_OBJECT_STORE_NAME,
             )
 
-            logger.info(f"Output prefix: {workflow_args['output_prefix']}")
             logger.info(f"Marker file downloaded to {marker_file_path}")
             if not os.path.exists(marker_file_path):
                 logger.warning(f"Marker file does not exist at {marker_file_path}")
@@ -487,7 +489,7 @@ class SQLQueryExtractionActivities(ActivitiesInterface):
 
         miner_args = MinerArgs(**workflow_args.get("miner_args", {}))
 
-        current_marker = self.read_marker(workflow_args)
+        current_marker = await self.read_marker(workflow_args)
         if current_marker:
             miner_args.current_marker = current_marker
 
@@ -522,8 +524,10 @@ class SQLQueryExtractionActivities(ActivitiesInterface):
         with open(metadata_file_path, "w") as f:
             f.write(json.dumps(parallel_markers))
 
-        await ObjectStoreOutput.push_file_to_object_store(
-            workflow_args["output_prefix"], metadata_file_path
+        await ObjectStore.upload_file(
+            source=metadata_file_path,
+            destination=get_object_store_prefix(metadata_file_path),
+            store_name=UPSTREAM_OBJECT_STORE_NAME,
         )
 
         try:

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/application_sdk/application/__init__.py

@@ -2,8 +2,10 @@ from concurrent.futures import ThreadPoolExecutor
 from typing import Any, Dict, List, Optional, Tuple, Type
 
 from application_sdk.activities import ActivitiesInterface
+from application_sdk.clients.base import BaseClient
 from application_sdk.clients.utils import get_workflow_client
 from application_sdk.events.models import EventRegistration
+from application_sdk.handlers.base import BaseHandler
 from application_sdk.observability.logger_adaptor import get_logger
 from application_sdk.server import ServerInterface
 from application_sdk.server.fastapi import APIServer, HttpWorkflowTrigger
@@ -28,6 +30,8 @@ class BaseApplication:
         name: str,
         server: Optional[ServerInterface] = None,
         application_manifest: Optional[dict] = None,
+        client_class: Optional[Type[BaseClient]] = None,
+        handler_class: Optional[Type[BaseHandler]] = None,
     ):
         """
         Initialize the application.
@@ -48,6 +52,9 @@ class BaseApplication:
         self.application_manifest: Dict[str, Any] = application_manifest
         self.bootstrap_event_registration()
 
+        self.client_class = client_class or BaseClient
+        self.handler_class = handler_class or BaseHandler
+
     def bootstrap_event_registration(self):
         self.event_subscriptions = {}
         if self.application_manifest is None:
@@ -168,6 +175,7 @@ class BaseApplication:
         self.server = APIServer(
             workflow_client=self.workflow_client,
             ui_enabled=ui_enabled,
+            handler=self.handler_class(client=self.client_class()),
         )
 
         if self.event_subscriptions:

{atlan_application_sdk-0.1.1rc33 → atlan_application_sdk-0.1.1rc35}/application_sdk/clients/atlan_auth.py

@@ -13,8 +13,8 @@ from application_sdk.constants import (
     WORKFLOW_AUTH_ENABLED,
     WORKFLOW_AUTH_URL_KEY,
 )
-from application_sdk.inputs.secretstore import SecretStoreInput
 from application_sdk.observability.logger_adaptor import get_logger
+from application_sdk.services.secretstore import SecretStore
 
 logger = get_logger(__name__)
 
@@ -39,7 +39,7 @@ class AtlanAuthClient:
         (environment variables, AWS Secrets Manager, Azure Key Vault, etc.)
         """
         self.application_name = APPLICATION_NAME
-        self.auth_config: Dict[str, Any] = SecretStoreInput.get_deployment_secret()
+        self.auth_config: Dict[str, Any] = SecretStore.get_deployment_secret()
         self.auth_enabled: bool = WORKFLOW_AUTH_ENABLED
         self.auth_url: Optional[str] = None