PyPI - ethyca-fides - Versions diffs - 2.71.1b1__py2.py3-none-any.whl → 2.74.0rc1__py2.py3-none-any.whl - Mend

ethyca-fides 2.71.1b1py2.py3-none-any.whl → 2.74.0rc1py2.py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.

This version of ethyca-fides might be problematic. Click here for more details.

Files changed (495) hide show

fides/api/email_templates/templates/manual_task_digest.html ADDED Viewed

@@ -0,0 +1,316 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Weekly DSR Summary</title>
+    <style>
+        /* Reset styles for email clients */
+        body, table, td, p, a, li, blockquote {
+            -webkit-text-size-adjust: 100%;
+            -ms-text-size-adjust: 100%;
+        }
+        table, td {
+            mso-table-lspace: 0pt;
+            mso-table-rspace: 0pt;
+        }
+        img {
+            -ms-interpolation-mode: bicubic;
+            border: 0;
+            height: auto;
+            line-height: 100%;
+            outline: none;
+            text-decoration: none;
+        }
+        /* Main styles */
+        body {
+            margin: 0;
+            padding: 0;
+            width: 100% !important;
+            min-width: 100%;
+            height: 100%;
+            background-color: #f8fafc;
+            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
+            font-size: 16px;
+            line-height: 1.6;
+            color: #334155;
+        }
+        .email-container {
+            max-width: 600px;
+            margin: 0 auto;
+            background-color: #ffffff;
+            border-radius: 8px;
+            box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1);
+            overflow: hidden;
+        }
+        .logo-section {
+            background-color: #ffffff;
+            padding: 24px;
+            text-align: center;
+            border-bottom: 1px solid #e5e5e5;
+        }
+        .logo-container {
+            margin-bottom: 0;
+        }
+        .header {
+            background-color: #000000;
+            padding: 32px 24px;
+            text-align: center;
+        }
+        .logo {
+            max-width: 200px;
+            max-height: 60px;
+            width: auto;
+            height: auto;
+            display: block;
+            margin: 0 auto;
+        }
+        .logo-placeholder {
+            width: 200px;
+            height: 60px;
+            background-color: #333333;
+            border: 2px dashed #666666;
+            display: flex;
+            align-items: center;
+            justify-content: center;
+            margin: 0 auto;
+            color: #999999;
+            font-size: 14px;
+            font-weight: 500;
+            text-align: center;
+            border-radius: 4px;
+        }
+        .header h1 {
+            color: #ffffff;
+            font-size: 24px;
+            font-weight: 600;
+            margin: 0;
+            letter-spacing: -0.025em;
+        }
+        .content {
+            padding: 32px 24px;
+        }
+        .greeting {
+            font-size: 18px;
+            color: #000000;
+            margin-bottom: 24px;
+            font-weight: 500;
+        }
+        .intro-text {
+            margin-bottom: 32px;
+            color: #333333;
+            line-height: 1.7;
+        }
+        .section {
+            margin-bottom: 32px;
+        }
+        .section-title {
+            font-size: 20px;
+            font-weight: 600;
+            color: #000000;
+            margin-bottom: 16px;
+            padding-bottom: 8px;
+            border-bottom: 2px solid #e5e5e5;
+        }
+        .task-count {
+            background-color: #f5f5f5;
+            border-left: 4px solid #000000;
+            padding: 16px 20px;
+            margin-bottom: 20px;
+            border-radius: 0 6px 6px 0;
+        }
+        .task-count-text {
+            font-size: 18px;
+            font-weight: 600;
+            color: #000000;
+            margin: 0;
+        }
+        .cta-button {
+            display: inline-block;
+            background-color: #000000;
+            color: #ffffff !important;
+            text-decoration: none;
+            padding: 12px 24px;
+            border-radius: 6px;
+            font-weight: 600;
+            font-size: 16px;
+            text-align: center;
+            transition: all 0.2s ease;
+            box-shadow: 0 2px 4px rgba(0, 0, 0, 0.2);
+        }
+        .cta-button:hover {
+            background-color: #333333;
+            box-shadow: 0 4px 8px rgba(0, 0, 0, 0.3);
+            transform: translateY(-1px);
+        }
+        .footer {
+            background-color: #f5f5f5;
+            padding: 24px;
+            text-align: center;
+            border-top: 1px solid #e5e5e5;
+        }
+        .footer-text {
+            color: #666666;
+            font-size: 14px;
+            line-height: 1.6;
+            margin: 0;
+        }
+        .signature {
+            margin-top: 24px;
+            color: #333333;
+            font-weight: 500;
+        }
+        /* Responsive styles */
+        @media only screen and (max-width: 600px) {
+            .email-container {
+                margin: 0;
+                border-radius: 0;
+            }
+            .header, .content, .footer {
+                padding: 24px 16px;
+            }
+            .header h1 {
+                font-size: 20px;
+            }
+            .greeting {
+                font-size: 16px;
+            }
+            .section-title {
+                font-size: 18px;
+            }
+            .cta-button {
+                width: 100%;
+                box-sizing: border-box;
+            }
+        }
+        /* Dark mode support */
+        @media (prefers-color-scheme: dark) {
+            body {
+                background-color: #000000;
+            }
+            .email-container {
+                background-color: #1a1a1a;
+            }
+            .greeting, .section-title {
+                color: #ffffff;
+            }
+            .intro-text, .footer-text, .signature {
+                color: #cccccc;
+            }
+            .task-count {
+                background-color: #333333;
+                border-left-color: #ffffff;
+            }
+            .task-count-text {
+                color: #ffffff;
+            }
+            .footer {
+                background-color: #333333;
+                border-top-color: #666666;
+            }
+            .logo-section {
+                background-color: #1a1a1a;
+                border-bottom-color: #666666;
+            }
+            .logo-placeholder {
+                background-color: #555555;
+                border-color: #888888;
+                color: #cccccc;
+            }
+        }
+    </style>
+</head>
+<body>
+    <div style="padding: 20px 0;">
+        <div class="email-container">
+            <!-- Logo Section -->
+            <div class="logo-section">
+                <div class="logo-container">
+                    {% if company_logo_url %}
+                        <img src="{{ company_logo_url }}" alt="{{ organization_name }} Logo" class="logo">
+                    {% else %}
+                        <div class="logo-placeholder">{{ organization_name }}</div>
+                    {% endif %}
+                </div>
+            </div>
+            <!-- Main Content -->
+            <div class="content">
+                <div class="greeting">Hi {{ vendor_contact_name }},</div>
+                <div class="intro-text">
+                    {{ intro_text | default("This is your weekly summary of open data subject requests tasks from " + organization_name + " that require your attention. Please review the tasks listed below in our <a href='" + portal_url + "' style='color: #000000; text-decoration: underline; font-weight: 600;'>Privacy Center</a> to ensure timely completion.") | safe }}
+                </div>
+                <!-- Tasks Due Soon Section -->
+                <div class="section">
+                    <div class="section-title">Tasks Due Soon</div>
+                    <div class="task-count">
+                        <p class="task-count-text">You have {{ total_task_count }} request{{ 's' if total_task_count != 1 else '' }} coming due, including {{ imminent_task_count }} within the next 7 days and {{ upcoming_task_count }} due in the next period.</p>
+                    </div>
+                    <div style="text-align: center;">
+                        <a href="{{ portal_url }}" class="cta-button">View All Tasks</a>
+                    </div>
+                </div>
+                <!-- Closing -->
+                <div style="margin-top: 32px;">
+                    <p style="color: #333333; margin-bottom: 24px;">
+                        {{ closing_text | default("Thank you for your partnership in processing these requests efficiently. If you have any questions or issues, please reach out to your contacts at " + organization_name + ".") | safe }}
+                    </p>
+                    <div class="signature">
+                        {{ signature | default("Best regards,<br>The " + organization_name + " Privacy Team") | safe }}
+                    </div>
+                </div>
+            </div>
+            <!-- Footer -->
+            <div class="footer">
+                <p class="footer-text">
+                    This email was sent from your Privacy Center. Please do not reply to this email.
+                </p>
+            </div>
+        </div>
+    </div>
+</body>
+</html>

fides/api/main.py CHANGED Viewed

@@ -45,9 +45,9 @@ from fides.api.service.privacy_request.email_batch_service import (
     initiate_scheduled_batch_email_send,
 )
 from fides.api.service.privacy_request.request_service import (
-    initiate_async_tasks_status_polling,
     initiate_interrupted_task_requeue_poll,
     initiate_poll_for_exited_privacy_request_tasks,
+    initiate_polling_task_requeue,
     initiate_scheduled_dsr_data_removal,
 )
@@ -103,7 +103,7 @@ async def lifespan(wrapped_app: FastAPI) -> AsyncGenerator[None, None]:
     initiate_poll_for_exited_privacy_request_tasks()
     initiate_scheduled_dsr_data_removal()
     initiate_interrupted_task_requeue_poll()
-    initiate_async_tasks_status_polling()
+    initiate_polling_task_requeue()
     initiate_bcrypt_migration_task()
     initiate_post_upgrade_index_creation()

fides/api/models/attachment.py CHANGED Viewed

@@ -52,6 +52,7 @@ class AttachmentReferenceType(str, EnumType):
     privacy_request = "privacy_request"
     comment = "comment"
     manual_task_submission = "manual_task_submission"
+    request_task = "request_task"
 class AttachmentReference(Base):

fides/api/models/detection_discovery/__init__.py CHANGED Viewed

@@ -17,6 +17,7 @@ from .monitor_task import (
     create_monitor_task_with_execution_log,
     update_monitor_task_with_execution_log,
 )
+from .staged_resource_error import StagedResourceError
 __all__ = [
     "ClassificationBenchmark",
@@ -27,6 +28,7 @@ __all__ = [
     "SharedMonitorConfig",
     "StagedResource",
     "StagedResourceAncestor",
+    "StagedResourceError",
     "fetch_staged_resources_by_type_query",
     "MonitorTask",
     "MonitorTaskExecutionLog",

fides/api/models/detection_discovery/core.py CHANGED Viewed

@@ -29,6 +29,9 @@ from sqlalchemy.orm.query import Query
 from fides.api.db.base_class import Base, FidesBase
 from fides.api.models.connectionconfig import ConnectionConfig
+from fides.api.models.detection_discovery.staged_resource_error import (
+    StagedResourceError,
+)
 from fides.api.models.sql_models import System  # type: ignore[attr-defined]
@@ -573,6 +576,13 @@ class StagedResource(Base):
     # diff-related fields
     diff_status = Column(String, nullable=True, index=True)
+    errors: RelationshipProperty[List[StagedResourceError]] = relationship(
+        "StagedResourceError",
+        foreign_keys=[StagedResourceError.staged_resource_urn],
+        primaryjoin="StagedResource.urn == StagedResourceError.staged_resource_urn",
+        cascade="all, delete-orphan",
+    )
     ancestor_links: RelationshipProperty[List[StagedResourceAncestor]] = relationship(
         "StagedResourceAncestor",
         back_populates="descendant_staged_resource",

fides/api/models/detection_discovery/monitor_task.py CHANGED Viewed

@@ -25,6 +25,7 @@ class MonitorTaskType(Enum):
     DETECTION = "detection"
     CLASSIFICATION = "classification"
+    LLM_CLASSIFICATION = "llm_classification"
     PROMOTION = "promotion"
     REMOVAL_PROMOTION = "removal_promotion"

fides/api/models/detection_discovery/staged_resource_error.py ADDED Viewed

@@ -0,0 +1,25 @@
+"""Module for staged resource error tracking."""
+from sqlalchemy import Column, DateTime, ForeignKey, String
+from fides.api.db.base_class import Base, FidesBase
+class StagedResourceError(Base):
+    """
+    DB model for tracking errors associated with staged resources.
+    Linked to StagedResource via FK with cascade delete.
+    """
+    id = Column(String(255), primary_key=True, default=FidesBase.generate_uuid)
+    staged_resource_urn = Column(
+        String,
+        ForeignKey("stagedresource.urn", ondelete="CASCADE"),
+        nullable=False,
+        index=True,
+    )
+    message = Column(String, nullable=False)
+    timestamp = Column(DateTime(timezone=True), nullable=False)
+    error_type = Column(String, nullable=False)
+    diff_status = Column(String, nullable=False)  # Phase where error occurred
+    task_id = Column(String, nullable=True, index=True)

fides/api/models/digest/__init__.py CHANGED Viewed

@@ -5,10 +5,12 @@ from fides.api.models.digest.conditional_dependencies import (
     DigestConditionType,
 )
 from fides.api.models.digest.digest_config import DigestConfig, DigestType
+from fides.api.models.digest.digest_execution import DigestTaskExecution
 __all__ = [
     "DigestConfig",
     "DigestType",
     "DigestCondition",
     "DigestConditionType",
+    "DigestTaskExecution",
 ]

fides/api/models/digest/digest_config.py CHANGED Viewed

@@ -1,5 +1,5 @@
 from enum import Enum
-from typing import Optional, Union
+from typing import TYPE_CHECKING, Optional, Union
 from sqlalchemy import Boolean, Column, DateTime, String, Text
 from sqlalchemy.dialects.postgresql import JSONB
@@ -19,6 +19,9 @@ from fides.api.task.conditional_dependencies.schemas import (
     ConditionLeaf,
 )
+if TYPE_CHECKING:
+    from fides.api.models.digest.digest_execution import DigestTaskExecution
 class DigestType(str, Enum):
     """Types of digests that can be configured."""
@@ -60,6 +63,12 @@ class DigestConfig(Base):
         back_populates="digest_config",
         cascade="all, delete-orphan",
     )
+    executions = relationship(
+        "DigestTaskExecution",
+        back_populates="digest_config",
+        cascade="all, delete-orphan",
+        order_by="DigestTaskExecution.created_at.desc()",
+    )
     def get_receiver_condition(
         self, db: Session

fides/api/models/digest/digest_execution.py ADDED Viewed

@@ -0,0 +1,142 @@
+from typing import TYPE_CHECKING, Any, Dict, List, Optional
+from sqlalchemy import Column, DateTime, ForeignKey, Integer, String, Text
+from sqlalchemy.dialects.postgresql import JSONB
+from sqlalchemy.ext.declarative import declared_attr
+from sqlalchemy.orm import Session, relationship
+from sqlalchemy.orm.attributes import flag_modified
+from sqlalchemy.sql import func
+from fides.api.db.base_class import Base
+from fides.api.models.worker_task import ExecutionLogStatus, WorkerTask
+if TYPE_CHECKING:
+    from fides.api.models.digest.digest_config import DigestConfig
+class DigestTaskExecution(
+    WorkerTask, Base
+):  # pylint: disable=too-many-instance-attributes
+    """
+    Model for tracking digest task execution state and progress.
+    This model enables graceful resumption of digest tasks after worker
+    interruptions by persisting execution state and progress information.
+    """
+    @declared_attr
+    def __tablename__(cls) -> str:
+        return "digest_task_execution"
+    # Foreign key to digest config
+    digest_config_id = Column(
+        String,
+        ForeignKey("digest_config.id", ondelete="CASCADE"),
+        nullable=False,
+        index=True,
+    )
+    # Celery task tracking
+    celery_task_id = Column(String, nullable=True, index=True)
+    # Progress tracking
+    total_recipients = Column(Integer, nullable=True)
+    processed_recipients = Column(Integer, nullable=False, default=0)
+    successful_communications = Column(Integer, nullable=False, default=0)
+    failed_communications = Column(Integer, nullable=False, default=0)
+    # State persistence for resumption
+    execution_state = Column(JSONB, nullable=True, default={})
+    processed_user_ids = Column(JSONB, nullable=True, default=[])
+    # Timing information
+    started_at = Column(DateTime(timezone=True), nullable=True)
+    completed_at = Column(DateTime(timezone=True), nullable=True)
+    last_checkpoint_at = Column(DateTime(timezone=True), nullable=True)
+    # Error information
+    error_message = Column(Text, nullable=True)
+    # Relationships
+    digest_config = relationship("DigestConfig", back_populates="executions")
+    @classmethod
+    def allowed_action_types(cls) -> List[str]:
+        """Return allowed action types for digest task execution."""
+        return ["digest_processing"]
+    def mark_started(self, db: Session, celery_task_id: str) -> None:
+        """Mark the execution as started."""
+        self.status = ExecutionLogStatus.in_processing
+        self.celery_task_id = celery_task_id
+        self.started_at = func.now()
+        self.save(db)
+    def mark_awaiting_processing(self, db: Session) -> None:
+        """Mark the execution as awaiting processing."""
+        self.status = ExecutionLogStatus.awaiting_processing
+        self.save(db)
+    def update_progress(
+        self,
+        db: Session,
+        processed_count: int,
+        successful_count: int,
+        failed_count: int,
+        processed_user_ids: List[str],
+        execution_state: Optional[Dict[str, Any]] = None,
+    ) -> None:
+        """Update execution progress and create checkpoint."""
+        self.processed_recipients = processed_count
+        self.successful_communications = successful_count
+        self.failed_communications = failed_count
+        self.processed_user_ids = processed_user_ids
+        self.last_checkpoint_at = func.now()
+        # Mark JSONB fields as modified so SQLAlchemy knows they changed
+        flag_modified(self, "processed_user_ids")
+        if execution_state:
+            self.execution_state = execution_state
+            flag_modified(self, "execution_state")
+        self.save(db)
+    def mark_completed(self, db: Session) -> None:
+        """Mark the execution as completed."""
+        self.status = ExecutionLogStatus.complete
+        self.completed_at = func.now()
+        self.save(db)
+    def mark_failed(self, db: Session, error_message: str) -> None:
+        """Mark the execution as failed."""
+        self.status = ExecutionLogStatus.error
+        self.error_message = error_message
+        self.completed_at = func.now()
+        self.save(db)
+    def can_resume(self) -> bool:
+        """Check if this execution can be resumed."""
+        return (
+            self.status
+            in [
+                ExecutionLogStatus.in_processing,
+                ExecutionLogStatus.awaiting_processing,
+            ]
+            and self.processed_user_ids is not None
+        )
+    def get_remaining_work(self) -> Dict[str, Any]:
+        """Get information about remaining work for resumption."""
+        # Ensure processed_user_ids is always a list
+        raw_processed_user_ids = self.processed_user_ids or []
+        processed_user_ids = (
+            raw_processed_user_ids if isinstance(raw_processed_user_ids, list) else []
+        )
+        return {
+            "processed_user_ids": processed_user_ids,
+            "execution_state": self.execution_state or {},
+            "processed_count": self.processed_recipients or 0,
+            "successful_count": self.successful_communications,
+            "failed_count": self.failed_communications,
+        }

fides/api/models/event_audit.py CHANGED Viewed

@@ -31,6 +31,23 @@ class EventAuditType(str, EnumType):
     taxonomy_element_updated = "taxonomy.element.updated"
     taxonomy_element_deleted = "taxonomy.element.deleted"
+    # Connection operations
+    connection_created = "connection.created"
+    connection_updated = "connection.updated"
+    connection_deleted = "connection.deleted"
+    # Connection secrets operations
+    connection_secrets_created = "connection.secrets.created"
+    connection_secrets_updated = "connection.secrets.updated"
+    # Digest
+    digest_execution_started = "digest.execution.started"
+    digest_execution_completed = "digest.execution.completed"
+    digest_execution_interrupted = "digest.execution.interrupted"
+    digest_execution_resumed = "digest.execution.resumed"
+    digest_communications_sent = "digest.communications.sent"
+    digest_checkpoint_created = "digest.checkpoint.created"
 class EventAuditStatus(str, EnumType):
     """Status enum for event audit logging."""

fides/api/models/identity_definition.py ADDED Viewed

@@ -0,0 +1,65 @@
+from enum import Enum
+from sqlalchemy import Column, String, Text
+from sqlalchemy.ext.declarative import declared_attr
+from fides.api.db.base_class import Base, FidesBase
+from fides.api.db.util import EnumColumn
+class IdentityDefinitionType(str, Enum):
+    """Enum for the type of identity"""
+    EMAIL = "email"
+    PHONE_NUMBER = "phone_number"
+    UUID = "uuid"
+    STRING = "string"
+    INTEGER = "integer"
+class IdentityDefinition(Base):
+    """
+    Model for identity definitions in Fides. This isn't for specific identity values,
+    but for the types of identities that can be used in Fides.
+    For example:
+    ```json
+    {
+        "identity_key": "customer_id",
+        "name": "Customer ID",
+        "description": "The unique identifier for the customer",
+        "type": "string"
+    }
+    ```
+    """
+    @declared_attr
+    def __tablename__(self) -> str:
+        return "identity_definition"
+    # Overriding the id definition from Base so we don't treat this as the primary key
+    id = Column(
+        String(255),
+        nullable=False,
+        index=False,
+        unique=True,
+        default=FidesBase.generate_uuid,
+    )
+    # Primary key
+    identity_key = Column(String(255), primary_key=True)
+    # Schema definition
+    name = Column(String(255), nullable=False)
+    description = Column(Text, nullable=True)
+    type = Column(
+        EnumColumn(
+            IdentityDefinitionType,
+            native_enum=False,
+            values_callable=lambda x: [
+                i.value for i in x
+            ],  # allows enum _values_ to be stored rather than name
+        ),
+        nullable=False,
+    )
+    created_by = Column(String(255), nullable=True)

ethyca-fides 2.71.1b1__py2.py3-none-any.whl → 2.74.0rc1__py2.py3-none-any.whl

Potentially problematic release.

ethyca-fides 2.71.1b1py2.py3-none-any.whl → 2.74.0rc1py2.py3-none-any.whl