PyPI - ethyca-fides - Versions diffs - 2.71.2b0__py2.py3-none-any.whl → 2.72.0rc1__py2.py3-none-any.whl - Mend

ethyca-fides 2.71.2b0py2.py3-none-any.whl → 2.72.0rc1py2.py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.

This version of ethyca-fides might be problematic. Click here for more details.

Files changed (307) hide show

fides/api/email_templates/templates/manual_task_digest.html ADDED Viewed

@@ -0,0 +1,316 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Weekly DSR Summary</title>
+    <style>
+        /* Reset styles for email clients */
+        body, table, td, p, a, li, blockquote {
+            -webkit-text-size-adjust: 100%;
+            -ms-text-size-adjust: 100%;
+        }
+        table, td {
+            mso-table-lspace: 0pt;
+            mso-table-rspace: 0pt;
+        }
+        img {
+            -ms-interpolation-mode: bicubic;
+            border: 0;
+            height: auto;
+            line-height: 100%;
+            outline: none;
+            text-decoration: none;
+        }
+        /* Main styles */
+        body {
+            margin: 0;
+            padding: 0;
+            width: 100% !important;
+            min-width: 100%;
+            height: 100%;
+            background-color: #f8fafc;
+            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
+            font-size: 16px;
+            line-height: 1.6;
+            color: #334155;
+        }
+        .email-container {
+            max-width: 600px;
+            margin: 0 auto;
+            background-color: #ffffff;
+            border-radius: 8px;
+            box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1);
+            overflow: hidden;
+        }
+        .logo-section {
+            background-color: #ffffff;
+            padding: 24px;
+            text-align: center;
+            border-bottom: 1px solid #e5e5e5;
+        }
+        .logo-container {
+            margin-bottom: 0;
+        }
+        .header {
+            background-color: #000000;
+            padding: 32px 24px;
+            text-align: center;
+        }
+        .logo {
+            max-width: 200px;
+            max-height: 60px;
+            width: auto;
+            height: auto;
+            display: block;
+            margin: 0 auto;
+        }
+        .logo-placeholder {
+            width: 200px;
+            height: 60px;
+            background-color: #333333;
+            border: 2px dashed #666666;
+            display: flex;
+            align-items: center;
+            justify-content: center;
+            margin: 0 auto;
+            color: #999999;
+            font-size: 14px;
+            font-weight: 500;
+            text-align: center;
+            border-radius: 4px;
+        }
+        .header h1 {
+            color: #ffffff;
+            font-size: 24px;
+            font-weight: 600;
+            margin: 0;
+            letter-spacing: -0.025em;
+        }
+        .content {
+            padding: 32px 24px;
+        }
+        .greeting {
+            font-size: 18px;
+            color: #000000;
+            margin-bottom: 24px;
+            font-weight: 500;
+        }
+        .intro-text {
+            margin-bottom: 32px;
+            color: #333333;
+            line-height: 1.7;
+        }
+        .section {
+            margin-bottom: 32px;
+        }
+        .section-title {
+            font-size: 20px;
+            font-weight: 600;
+            color: #000000;
+            margin-bottom: 16px;
+            padding-bottom: 8px;
+            border-bottom: 2px solid #e5e5e5;
+        }
+        .task-count {
+            background-color: #f5f5f5;
+            border-left: 4px solid #000000;
+            padding: 16px 20px;
+            margin-bottom: 20px;
+            border-radius: 0 6px 6px 0;
+        }
+        .task-count-text {
+            font-size: 18px;
+            font-weight: 600;
+            color: #000000;
+            margin: 0;
+        }
+        .cta-button {
+            display: inline-block;
+            background-color: #000000;
+            color: #ffffff !important;
+            text-decoration: none;
+            padding: 12px 24px;
+            border-radius: 6px;
+            font-weight: 600;
+            font-size: 16px;
+            text-align: center;
+            transition: all 0.2s ease;
+            box-shadow: 0 2px 4px rgba(0, 0, 0, 0.2);
+        }
+        .cta-button:hover {
+            background-color: #333333;
+            box-shadow: 0 4px 8px rgba(0, 0, 0, 0.3);
+            transform: translateY(-1px);
+        }
+        .footer {
+            background-color: #f5f5f5;
+            padding: 24px;
+            text-align: center;
+            border-top: 1px solid #e5e5e5;
+        }
+        .footer-text {
+            color: #666666;
+            font-size: 14px;
+            line-height: 1.6;
+            margin: 0;
+        }
+        .signature {
+            margin-top: 24px;
+            color: #333333;
+            font-weight: 500;
+        }
+        /* Responsive styles */
+        @media only screen and (max-width: 600px) {
+            .email-container {
+                margin: 0;
+                border-radius: 0;
+            }
+            .header, .content, .footer {
+                padding: 24px 16px;
+            }
+            .header h1 {
+                font-size: 20px;
+            }
+            .greeting {
+                font-size: 16px;
+            }
+            .section-title {
+                font-size: 18px;
+            }
+            .cta-button {
+                width: 100%;
+                box-sizing: border-box;
+            }
+        }
+        /* Dark mode support */
+        @media (prefers-color-scheme: dark) {
+            body {
+                background-color: #000000;
+            }
+            .email-container {
+                background-color: #1a1a1a;
+            }
+            .greeting, .section-title {
+                color: #ffffff;
+            }
+            .intro-text, .footer-text, .signature {
+                color: #cccccc;
+            }
+            .task-count {
+                background-color: #333333;
+                border-left-color: #ffffff;
+            }
+            .task-count-text {
+                color: #ffffff;
+            }
+            .footer {
+                background-color: #333333;
+                border-top-color: #666666;
+            }
+            .logo-section {
+                background-color: #1a1a1a;
+                border-bottom-color: #666666;
+            }
+            .logo-placeholder {
+                background-color: #555555;
+                border-color: #888888;
+                color: #cccccc;
+            }
+        }
+    </style>
+</head>
+<body>
+    <div style="padding: 20px 0;">
+        <div class="email-container">
+            <!-- Logo Section -->
+            <div class="logo-section">
+                <div class="logo-container">
+                    {% if company_logo_url %}
+                        <img src="{{ company_logo_url }}" alt="{{ organization_name }} Logo" class="logo">
+                    {% else %}
+                        <div class="logo-placeholder">{{ organization_name }}</div>
+                    {% endif %}
+                </div>
+            </div>
+            <!-- Main Content -->
+            <div class="content">
+                <div class="greeting">Hi {{ vendor_contact_name }},</div>
+                <div class="intro-text">
+                    {{ intro_text | default("This is your weekly summary of open data subject requests tasks from " + organization_name + " that require your attention. Please review the tasks listed below in our <a href='" + portal_url + "' style='color: #000000; text-decoration: underline; font-weight: 600;'>Privacy Center</a> to ensure timely completion.") | safe }}
+                </div>
+                <!-- Tasks Due Soon Section -->
+                <div class="section">
+                    <div class="section-title">Tasks Due Soon</div>
+                    <div class="task-count">
+                        <p class="task-count-text">You have {{ total_task_count }} request{{ 's' if total_task_count != 1 else '' }} coming due, including {{ imminent_task_count }} within the next 7 days and {{ upcoming_task_count }} due in the next period.</p>
+                    </div>
+                    <div style="text-align: center;">
+                        <a href="{{ portal_url }}" class="cta-button">View All Tasks</a>
+                    </div>
+                </div>
+                <!-- Closing -->
+                <div style="margin-top: 32px;">
+                    <p style="color: #333333; margin-bottom: 24px;">
+                        {{ closing_text | default("Thank you for your partnership in processing these requests efficiently. If you have any questions or issues, please reach out to your contacts at " + organization_name + ".") | safe }}
+                    </p>
+                    <div class="signature">
+                        {{ signature | default("Best regards,<br>The " + organization_name + " Privacy Team") | safe }}
+                    </div>
+                </div>
+            </div>
+            <!-- Footer -->
+            <div class="footer">
+                <p class="footer-text">
+                    This email was sent from your Privacy Center. Please do not reply to this email.
+                </p>
+            </div>
+        </div>
+    </div>
+</body>
+</html>

fides/api/models/detection_discovery/monitor_task.py CHANGED Viewed

@@ -25,6 +25,7 @@ class MonitorTaskType(Enum):
     DETECTION = "detection"
     CLASSIFICATION = "classification"
+    LLM_CLASSIFICATION = "llm_classification"
     PROMOTION = "promotion"
     REMOVAL_PROMOTION = "removal_promotion"

fides/api/models/digest/digest_execution.py CHANGED Viewed

@@ -4,6 +4,7 @@ from sqlalchemy import Column, DateTime, ForeignKey, Integer, String, Text
 from sqlalchemy.dialects.postgresql import JSONB
 from sqlalchemy.ext.declarative import declared_attr
 from sqlalchemy.orm import Session, relationship
+from sqlalchemy.orm.attributes import flag_modified
 from sqlalchemy.sql import func
 from fides.api.db.base_class import Base
@@ -92,8 +93,12 @@ class DigestTaskExecution(
         self.processed_user_ids = processed_user_ids
         self.last_checkpoint_at = func.now()
+        # Mark JSONB fields as modified so SQLAlchemy knows they changed
+        flag_modified(self, "processed_user_ids")
         if execution_state:
             self.execution_state = execution_state
+            flag_modified(self, "execution_state")
         self.save(db)
@@ -123,8 +128,13 @@ class DigestTaskExecution(
     def get_remaining_work(self) -> Dict[str, Any]:
         """Get information about remaining work for resumption."""
+        # Ensure processed_user_ids is always a list
+        raw_processed_user_ids = self.processed_user_ids or []
+        processed_user_ids = (
+            raw_processed_user_ids if isinstance(raw_processed_user_ids, list) else []
+        )
         return {
-            "processed_user_ids": self.processed_user_ids or [],
+            "processed_user_ids": processed_user_ids,
             "execution_state": self.execution_state or {},
             "processed_count": self.processed_recipients or 0,
             "successful_count": self.successful_communications,

fides/api/models/event_audit.py CHANGED Viewed

@@ -31,6 +31,15 @@ class EventAuditType(str, EnumType):
     taxonomy_element_updated = "taxonomy.element.updated"
     taxonomy_element_deleted = "taxonomy.element.deleted"
+    # Connection operations
+    connection_created = "connection.created"
+    connection_updated = "connection.updated"
+    connection_deleted = "connection.deleted"
+    # Connection secrets operations
+    connection_secrets_created = "connection.secrets.created"
+    connection_secrets_updated = "connection.secrets.updated"
     # Digest
     digest_execution_started = "digest.execution.started"
     digest_execution_completed = "digest.execution.completed"

fides/api/models/messaging_template.py CHANGED Viewed

@@ -58,6 +58,13 @@ DEFAULT_MESSAGING_TEMPLATES: Dict[str, Any] = {
             "body": "Your erasure request has been completed.",
         },
     },
+    MessagingActionType.MANUAL_TASK_DIGEST.value: {
+        "label": "Manual task digest",
+        "content": {
+            "subject": "Weekly DSR Summary from __ORGANIZATION_NAME__",
+            "body": "This is your weekly summary of open data subject requests tasks from __ORGANIZATION_NAME__ that require your attention. Please review the tasks in our Privacy Center to ensure timely completion.",
+        },
+    },
 }

fides/api/schemas/connection_configuration/connection_config.py CHANGED Viewed

@@ -13,6 +13,7 @@ from fides.api.schemas.connection_configuration import connection_secrets_schema
 from fides.api.schemas.policy import ActionType
 from fides.api.schemas.saas.saas_config import SaaSConfigBase
 from fides.api.util.connection_type import get_connection_type_secret_schema
+from fides.api.util.masking_util import mask_sensitive_fields
 class CreateConnectionConfiguration(BaseModel):
@@ -41,36 +42,6 @@ class CreateConnectionConfigurationWithSecrets(CreateConnectionConfiguration):
     model_config = ConfigDict(from_attributes=True, extra="ignore")
-def mask_sensitive_fields(
-    connection_secrets: Dict[str, Any], secret_schema: Dict[str, Any]
-) -> Dict[str, Any]:
-    """
-    Mask sensitive fields in the given secrets based on the provided schema.
-    This function traverses the given secrets dictionary and uses the provided schema to
-    identify fields that have been marked as sensitive. The function replaces the sensitive
-    field values with a mask string ('********').
-    Args:
-        connection_secrets (Dict[str, Any]): The secrets to be masked.
-        secret_schema (Dict[str, Any]): The schema defining which fields are sensitive.
-    Returns:
-        Dict[str, Any]: The secrets dictionary with sensitive fields masked.
-    """
-    if connection_secrets is None:
-        return connection_secrets
-    secret_schema_properties = secret_schema["properties"]
-    new_connection_secrets = {}
-    for key, value in connection_secrets.items():
-        if key in secret_schema_properties:
-            if secret_schema_properties.get(key, {}).get("sensitive", False):
-                new_connection_secrets[key] = "**********"
-            else:
-                new_connection_secrets[key] = value
-    return new_connection_secrets
 class ConnectionConfigSecretsMixin(BaseModel):
     """
     A schema mixin to declare a connection config `secrets` attribute

fides/api/schemas/messaging/messaging.py CHANGED Viewed

@@ -89,6 +89,7 @@ class MessagingActionType(str, Enum):
     PRIVACY_REQUEST_REVIEW_APPROVE = "privacy_request_review_approve"
     USER_INVITE = "user_invite"
     EXTERNAL_USER_WELCOME = "external_user_welcome"
+    MANUAL_TASK_DIGEST = "manual_task_digest"
     TEST_MESSAGE = "test_message"
@@ -101,6 +102,7 @@ CONFIGURABLE_MESSAGING_ACTION_TYPES: Tuple[str, ...] = (
     MessagingActionType.PRIVACY_REQUEST_COMPLETE_DELETION.value,
     MessagingActionType.PRIVACY_REQUEST_REVIEW_DENY.value,
     MessagingActionType.PRIVACY_REQUEST_REVIEW_APPROVE.value,
+    MessagingActionType.MANUAL_TASK_DIGEST.value,
 )
@@ -219,6 +221,18 @@ class ExternalUserWelcomeBodyParams(BaseModel):
     org_name: str
+class ManualTaskDigestBodyParams(BaseModel):
+    """Body params required to send manual task digest emails"""
+    vendor_contact_name: str
+    organization_name: str
+    portal_url: str
+    imminent_task_count: int
+    upcoming_task_count: int
+    total_task_count: int
+    company_logo_url: Optional[str] = None
 class FidesopsMessage(
     BaseModel,
 ):
@@ -256,6 +270,7 @@ class FidesopsMessage(
             MessagingActionType.PRIVACY_REQUEST_ERROR_NOTIFICATION: ErrorNotificationBodyParams,
             MessagingActionType.USER_INVITE: UserInviteBodyParams,
             MessagingActionType.EXTERNAL_USER_WELCOME: ExternalUserWelcomeBodyParams,
+            MessagingActionType.MANUAL_TASK_DIGEST: ManualTaskDigestBodyParams,
         }
         valid_body_params = valid_body_params_for_action_type.get(

fides/api/service/connectors/saas/connector_registry_service.py CHANGED Viewed

@@ -1,24 +1,18 @@
 # pylint: disable=protected-access
 import os
 from abc import ABC, abstractmethod
-from typing import Dict, Iterable, List, Optional, Type
+from typing import Dict, List, Optional, Type
 from zipfile import ZipFile
 from fideslang.models import Dataset
 from loguru import logger
-from packaging.version import Version
 from packaging.version import parse as parse_version
 from sqlalchemy.orm import Session
 from fides.api.api.deps import get_api_session
 from fides.api.common_exceptions import ValidationError
 from fides.api.cryptography.cryptographic_util import str_to_b64_str
-from fides.api.models.connectionconfig import ConnectionConfig
 from fides.api.models.custom_connector_template import CustomConnectorTemplate
-from fides.api.models.datasetconfig import DatasetConfig
-from fides.api.schemas.connection_configuration.saas_config_template_values import (
-    SaasConnectionTemplateValues,
-)
 from fides.api.schemas.saas.connector_template import ConnectorTemplate
 from fides.api.schemas.saas.saas_config import SaaSConfig
 from fides.api.service.authentication.authentication_strategy_oauth2_authorization_code import (
@@ -31,8 +25,6 @@ from fides.api.util.saas_util import (
     load_config_from_string,
     load_dataset_from_string,
     load_yaml_as_string,
-    replace_config_placeholders,
-    replace_dataset_placeholders,
     replace_version,
 )
 from fides.api.util.unsafe_file_util import verify_svg, verify_zip
@@ -317,132 +309,3 @@ class ConnectorRegistry:
         Returns an object containing the various SaaS connector artifacts
         """
         return cls._get_combined_templates().get(connector_type)
-def create_connection_config_from_template_no_save(
-    db: Session,
-    template: ConnectorTemplate,
-    template_values: SaasConnectionTemplateValues,
-    system_id: Optional[str] = None,
-) -> ConnectionConfig:
-    """Creates a SaaS connection config from a template without saving it."""
-    # Load SaaS config from template and replace every instance of "<instance_fides_key>" with the fides_key
-    # the user has chosen
-    config_from_template: Dict = replace_config_placeholders(
-        template.config, "<instance_fides_key>", template_values.instance_key
-    )
-    data = template_values.generate_config_data_from_template(
-        config_from_template=config_from_template
-    )
-    if system_id:
-        data["system_id"] = system_id
-    # Create SaaS ConnectionConfig
-    connection_config = ConnectionConfig.create_without_saving(db, data=data)
-    return connection_config
-def upsert_dataset_config_from_template(
-    db: Session,
-    connection_config: ConnectionConfig,
-    template: ConnectorTemplate,
-    template_values: SaasConnectionTemplateValues,
-) -> DatasetConfig:
-    """
-    Creates a `DatasetConfig` from a template
-    and associates it with a ConnectionConfig.
-    If the `DatasetConfig` already exists in the db,
-    then the existing record is updated.
-    """
-    # Load the dataset config from template and replace every instance of "<instance_fides_key>" with the fides_key
-    # the user has chosen
-    dataset_from_template: Dict = replace_dataset_placeholders(
-        template.dataset, "<instance_fides_key>", template_values.instance_key
-    )
-    data = {
-        "connection_config_id": connection_config.id,
-        "fides_key": template_values.instance_key,
-        "dataset": dataset_from_template,  # Currently used for upserting a CTL Dataset
-    }
-    dataset_config = DatasetConfig.create_or_update(db, data=data)
-    return dataset_config
-def update_saas_configs(db: Session) -> None:
-    """
-    Updates SaaS config instances currently in the DB if to the
-    corresponding template in the registry are found.
-    Effectively an "update script" for SaaS config instances,
-    to be run on server bootstrap.
-    """
-    for connector_type in ConnectorRegistry.connector_types():
-        logger.debug(
-            "Determining if any updates are needed for connectors of type {} based on templates...",
-            connector_type,
-        )
-        template: ConnectorTemplate = ConnectorRegistry.get_connector_template(  # type: ignore
-            connector_type
-        )
-        saas_config = SaaSConfig(**load_config_from_string(template.config))
-        template_version: Version = parse_version(saas_config.version)
-        connection_configs: Iterable[ConnectionConfig] = ConnectionConfig.filter(
-            db=db,
-            conditions=(ConnectionConfig.saas_config["type"].astext == connector_type),
-        ).all()
-        for connection_config in connection_configs:
-            saas_config_instance = SaaSConfig.model_validate(
-                connection_config.saas_config
-            )
-            if parse_version(saas_config_instance.version) < template_version:
-                logger.info(
-                    "Updating SaaS config instance '{}' of type '{}' as its version, {}, was found to be lower than the template version {}",
-                    saas_config_instance.fides_key,
-                    connector_type,
-                    saas_config_instance.version,
-                    template_version,
-                )
-                try:
-                    update_saas_instance(
-                        db,
-                        connection_config,
-                        template,
-                        saas_config_instance,
-                    )
-                except Exception:
-                    logger.exception(
-                        "Encountered error attempting to update SaaS config instance {}",
-                        saas_config_instance.fides_key,
-                    )
-def update_saas_instance(
-    db: Session,
-    connection_config: ConnectionConfig,
-    template: ConnectorTemplate,
-    saas_config_instance: SaaSConfig,
-) -> None:
-    """
-    Replace in the DB the existing SaaS instance configuration data
-    (SaaSConfig, DatasetConfig) associated with the given ConnectionConfig
-    with new instance configuration data based on the given ConnectorTemplate
-    """
-    template_vals = SaasConnectionTemplateValues(
-        name=connection_config.name,
-        key=connection_config.key,
-        description=connection_config.description,
-        secrets=connection_config.secrets,
-        instance_key=saas_config_instance.fides_key,
-    )
-    config_from_template: Dict = replace_config_placeholders(
-        template.config, "<instance_fides_key>", template_vals.instance_key
-    )
-    connection_config.update_saas_config(db, SaaSConfig(**config_from_template))
-    upsert_dataset_config_from_template(db, connection_config, template, template_vals)

fides/api/service/deps.py CHANGED Viewed

@@ -6,6 +6,7 @@ from fides.api.api.deps import get_config, get_config_proxy, get_db
 from fides.api.db.ctl_session import get_async_db
 from fides.config import FidesConfig
 from fides.config.config_proxy import ConfigProxy
+from fides.service.connection.connection_service import ConnectionService
 from fides.service.dataset.dataset_config_service import DatasetConfigService
 from fides.service.dataset.dataset_service import DatasetService
 from fides.service.event_audit_service import EventAuditService
@@ -59,5 +60,12 @@ def get_taxonomy_service(
     return TaxonomyService(db, event_audit_service)
+def get_connection_service(
+    db: Session = Depends(get_db),
+    event_audit_service: EventAuditService = Depends(get_event_audit_service),
+) -> ConnectionService:
+    return ConnectionService(db, event_audit_service)
 def get_system_service(db: AsyncSession = Depends(get_async_db)) -> SystemService:
     return SystemService(db)

ethyca-fides 2.71.2b0__py2.py3-none-any.whl → 2.72.0rc1__py2.py3-none-any.whl

Potentially problematic release.

ethyca-fides 2.71.2b0py2.py3-none-any.whl → 2.72.0rc1py2.py3-none-any.whl