detectkit 0.1.0__tar.gz

detectkit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 detectkit team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

detectkit-0.1.0/MANIFEST.in

@@ -0,0 +1,7 @@
+include README.md
+include LICENSE
+include requirements.txt
+recursive-include detectkit *.py
+recursive-exclude tests *
+recursive-exclude * __pycache__
+recursive-exclude * *.pyc

detectkit-0.1.0/PKG-INFO

@@ -0,0 +1,231 @@
+Metadata-Version: 2.4
+Name: detectkit
+Version: 0.1.0
+Summary: Metric monitoring with automatic anomaly detection
+Author: detectkit team
+License: MIT
+Project-URL: Homepage, https://github.com/alexeiveselov92/detectkit
+Project-URL: Documentation, https://github.com/alexeiveselov92/detectkit
+Project-URL: Repository, https://github.com/alexeiveselov92/detectkit
+Project-URL: Issues, https://github.com/alexeiveselov92/detectkit/issues
+Keywords: monitoring,anomaly-detection,metrics,timeseries,alerting
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: click>=8.0
+Requires-Dist: jinja2>=3.0
+Requires-Dist: orjson>=3.0
+Provides-Extra: clickhouse
+Requires-Dist: clickhouse-driver>=0.2.0; extra == "clickhouse"
+Provides-Extra: postgres
+Requires-Dist: psycopg2-binary>=2.9.0; extra == "postgres"
+Provides-Extra: mysql
+Requires-Dist: pymysql>=1.0.0; extra == "mysql"
+Provides-Extra: all-db
+Requires-Dist: clickhouse-driver>=0.2.0; extra == "all-db"
+Requires-Dist: psycopg2-binary>=2.9.0; extra == "all-db"
+Requires-Dist: pymysql>=1.0.0; extra == "all-db"
+Provides-Extra: prophet
+Requires-Dist: prophet>=1.1.0; extra == "prophet"
+Provides-Extra: timesfm
+Requires-Dist: timesfm>=0.1.0; extra == "timesfm"
+Provides-Extra: advanced-detectors
+Requires-Dist: prophet>=1.1.0; extra == "advanced-detectors"
+Requires-Dist: timesfm>=0.1.0; extra == "advanced-detectors"
+Provides-Extra: all
+Requires-Dist: clickhouse-driver>=0.2.0; extra == "all"
+Requires-Dist: psycopg2-binary>=2.9.0; extra == "all"
+Requires-Dist: pymysql>=1.0.0; extra == "all"
+Requires-Dist: prophet>=1.1.0; extra == "all"
+Requires-Dist: timesfm>=0.1.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Dynamic: license-file
+
+# detectkit
+
+**Metric monitoring with automatic anomaly detection**
+
+`detectkit` is a Python library for data analysts and engineers to monitor time-series metrics with automatic anomaly detection and alerting.
+
+## Status
+
+🚧 **In Active Development** - Version 0.1.0
+
+This is a complete rewrite of the original detectk library with modern architecture and best practices (2025).
+
+## Features
+
+- ✅ **Pure numpy arrays** - No pandas dependency in core logic
+- ✅ **Batch processing** - Efficient vectorized operations
+- ✅ **Multiple detectors** - Statistical methods (Z-Score, MAD, IQR, Manual Bounds)
+- ✅ **Alert channels** - Mattermost, Slack, Webhook support
+- ✅ **Database agnostic** - ClickHouse, PostgreSQL, MySQL support
+- ✅ **Idempotent operations** - Resume from interruptions
+- 🚧 **CLI interface** - dbt-like commands (coming soon)
+
+## Installation
+
+```bash
+pip install detectkit
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/alexeiveselov92/detectkit
+cd detectkit
+pip install -e .
+```
+
+### Optional dependencies
+
+```bash
+# ClickHouse support
+pip install detectkit[clickhouse]
+
+# All database drivers
+pip install detectkit[all-db]
+
+# Development dependencies
+pip install detectkit[dev]
+```
+
+## Quick Start
+
+### CLI Usage (Recommended)
+
+```bash
+# Create a new project
+dtk init my_monitoring_project
+cd my_monitoring_project
+
+# Configure database in profiles.yml
+# Then run your metrics
+dtk run --select example_cpu_usage
+
+# Run specific pipeline steps
+dtk run --select cpu_usage --steps load,detect
+
+# Run all critical metrics
+dtk run --select tag:critical
+
+# Reload data from specific date
+dtk run --select cpu_usage --from 2024-01-01
+```
+
+### Python API Usage
+
+```python
+import numpy as np
+from detectkit.detectors.statistical import ZScoreDetector
+
+# Your time-series data
+timestamps = np.array([...], dtype='datetime64[ms]')
+values = np.array([1.0, 2.0, 1.5, 10.0, 1.8])  # 10.0 is anomaly
+
+# Create detector
+detector = ZScoreDetector(threshold=3.0, window_size=100)
+
+# Detect anomalies
+data = {
+    'timestamp': timestamps,
+    'value': values
+}
+results = detector.detect(data)
+
+# Check results
+for result in results:
+    if result.is_anomaly:
+        print(f"Anomaly at {result.timestamp}: {result.value}")
+```
+
+## Architecture
+
+- **Detectors** - Statistical and ML-based anomaly detection
+- **Loaders** - Metric data loading from databases with gap filling
+- **Alerting** - Multi-channel notifications with orchestration
+- **Config** - YAML-based configuration (dbt-like)
+
+## Testing
+
+```bash
+# Run tests
+pytest tests/
+
+# With coverage
+pytest tests/ --cov=detectkit --cov-report=html
+```
+
+**Current status:** 287 tests passing, 87% coverage
+
+## Development Status
+
+### ✅ Completed (Phases 1-6)
+- ✅ **Phase 1**: Core models (Interval, TableModel, ColumnDefinition)
+- ✅ **Phase 2**: Database managers & data loading (MetricLoader, gap filling, seasonality)
+- ✅ **Phase 3**: Statistical detectors (Z-Score, MAD, IQR, Manual Bounds)
+- ✅ **Phase 4**: Alerting system (Channels, Orchestrator, consecutive anomalies)
+- ✅ **Phase 5**: Task manager (Pipeline execution, locking, idempotency)
+- ✅ **Phase 6**: CLI commands (dtk init, dtk run with selectors)
+
+### 🔄 Integration Status
+- ⚠️ Full end-to-end integration pending (database connection required)
+- ⚠️ Advanced detectors (Prophet, TimesFM) - optional extras
+- ⚠️ Additional alert channels (Telegram, Email) - optional
+
+See [TODO.md](TODO.md) for detailed development roadmap.
+
+## Documentation
+
+- [ARCHITECTURE.md](ARCHITECTURE.md) - System architecture and design
+- [TECHNICAL_SPEC.md](TECHNICAL_SPEC.md) - Complete technical specification (Russian)
+- [TODO.md](TODO.md) - Development roadmap
+- [CLAUDE.md](CLAUDE.md) - Development context for AI assistants
+
+## Requirements
+
+- Python 3.10+
+- numpy >= 1.24.0
+- pydantic >= 2.0.0
+- click >= 8.0
+- PyYAML >= 6.0
+- Jinja2 >= 3.0
+
+## License
+
+MIT License - See LICENSE file for details
+
+## Contributing
+
+This project is currently in active development. Contributions are welcome once we reach v1.0.0.
+
+## Changelog
+
+### 0.1.0 (2025-11-07)
+- Initial release with complete rewrite
+- ✅ Core foundation: models, database, config
+- ✅ Metric loading with gap filling and seasonality extraction
+- ✅ Statistical detectors (Z-Score, MAD, IQR, Manual Bounds)
+- ✅ Alert channels (Webhook, Mattermost, Slack)
+- ✅ Alert orchestration with consecutive anomaly logic
+- ✅ Task manager for pipeline execution
+- ✅ CLI commands (dtk init, dtk run)
+- 📊 287 unit tests, 87% coverage

detectkit-0.1.0/README.md

@@ -0,0 +1,170 @@
+# detectkit
+
+**Metric monitoring with automatic anomaly detection**
+
+`detectkit` is a Python library for data analysts and engineers to monitor time-series metrics with automatic anomaly detection and alerting.
+
+## Status
+
+🚧 **In Active Development** - Version 0.1.0
+
+This is a complete rewrite of the original detectk library with modern architecture and best practices (2025).
+
+## Features
+
+- ✅ **Pure numpy arrays** - No pandas dependency in core logic
+- ✅ **Batch processing** - Efficient vectorized operations
+- ✅ **Multiple detectors** - Statistical methods (Z-Score, MAD, IQR, Manual Bounds)
+- ✅ **Alert channels** - Mattermost, Slack, Webhook support
+- ✅ **Database agnostic** - ClickHouse, PostgreSQL, MySQL support
+- ✅ **Idempotent operations** - Resume from interruptions
+- 🚧 **CLI interface** - dbt-like commands (coming soon)
+
+## Installation
+
+```bash
+pip install detectkit
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/alexeiveselov92/detectkit
+cd detectkit
+pip install -e .
+```
+
+### Optional dependencies
+
+```bash
+# ClickHouse support
+pip install detectkit[clickhouse]
+
+# All database drivers
+pip install detectkit[all-db]
+
+# Development dependencies
+pip install detectkit[dev]
+```
+
+## Quick Start
+
+### CLI Usage (Recommended)
+
+```bash
+# Create a new project
+dtk init my_monitoring_project
+cd my_monitoring_project
+
+# Configure database in profiles.yml
+# Then run your metrics
+dtk run --select example_cpu_usage
+
+# Run specific pipeline steps
+dtk run --select cpu_usage --steps load,detect
+
+# Run all critical metrics
+dtk run --select tag:critical
+
+# Reload data from specific date
+dtk run --select cpu_usage --from 2024-01-01
+```
+
+### Python API Usage
+
+```python
+import numpy as np
+from detectkit.detectors.statistical import ZScoreDetector
+
+# Your time-series data
+timestamps = np.array([...], dtype='datetime64[ms]')
+values = np.array([1.0, 2.0, 1.5, 10.0, 1.8])  # 10.0 is anomaly
+
+# Create detector
+detector = ZScoreDetector(threshold=3.0, window_size=100)
+
+# Detect anomalies
+data = {
+    'timestamp': timestamps,
+    'value': values
+}
+results = detector.detect(data)
+
+# Check results
+for result in results:
+    if result.is_anomaly:
+        print(f"Anomaly at {result.timestamp}: {result.value}")
+```
+
+## Architecture
+
+- **Detectors** - Statistical and ML-based anomaly detection
+- **Loaders** - Metric data loading from databases with gap filling
+- **Alerting** - Multi-channel notifications with orchestration
+- **Config** - YAML-based configuration (dbt-like)
+
+## Testing
+
+```bash
+# Run tests
+pytest tests/
+
+# With coverage
+pytest tests/ --cov=detectkit --cov-report=html
+```
+
+**Current status:** 287 tests passing, 87% coverage
+
+## Development Status
+
+### ✅ Completed (Phases 1-6)
+- ✅ **Phase 1**: Core models (Interval, TableModel, ColumnDefinition)
+- ✅ **Phase 2**: Database managers & data loading (MetricLoader, gap filling, seasonality)
+- ✅ **Phase 3**: Statistical detectors (Z-Score, MAD, IQR, Manual Bounds)
+- ✅ **Phase 4**: Alerting system (Channels, Orchestrator, consecutive anomalies)
+- ✅ **Phase 5**: Task manager (Pipeline execution, locking, idempotency)
+- ✅ **Phase 6**: CLI commands (dtk init, dtk run with selectors)
+
+### 🔄 Integration Status
+- ⚠️ Full end-to-end integration pending (database connection required)
+- ⚠️ Advanced detectors (Prophet, TimesFM) - optional extras
+- ⚠️ Additional alert channels (Telegram, Email) - optional
+
+See [TODO.md](TODO.md) for detailed development roadmap.
+
+## Documentation
+
+- [ARCHITECTURE.md](ARCHITECTURE.md) - System architecture and design
+- [TECHNICAL_SPEC.md](TECHNICAL_SPEC.md) - Complete technical specification (Russian)
+- [TODO.md](TODO.md) - Development roadmap
+- [CLAUDE.md](CLAUDE.md) - Development context for AI assistants
+
+## Requirements
+
+- Python 3.10+
+- numpy >= 1.24.0
+- pydantic >= 2.0.0
+- click >= 8.0
+- PyYAML >= 6.0
+- Jinja2 >= 3.0
+
+## License
+
+MIT License - See LICENSE file for details
+
+## Contributing
+
+This project is currently in active development. Contributions are welcome once we reach v1.0.0.
+
+## Changelog
+
+### 0.1.0 (2025-11-07)
+- Initial release with complete rewrite
+- ✅ Core foundation: models, database, config
+- ✅ Metric loading with gap filling and seasonality extraction
+- ✅ Statistical detectors (Z-Score, MAD, IQR, Manual Bounds)
+- ✅ Alert channels (Webhook, Mattermost, Slack)
+- ✅ Alert orchestration with consecutive anomaly logic
+- ✅ Task manager for pipeline execution
+- ✅ CLI commands (dtk init, dtk run)
+- 📊 287 unit tests, 87% coverage

detectkit-0.1.0/detectkit/__init__.py

@@ -0,0 +1,17 @@
+"""
+detectk - Anomaly Detection for Time-Series Metrics
+
+A Python library for data analysts and engineers to monitor metrics with automatic anomaly detection.
+"""
+
+__version__ = "0.1.0"
+
+from detectkit.core.interval import Interval
+from detectkit.core.models import ColumnDefinition, TableModel
+
+__all__ = [
+    "Interval",
+    "ColumnDefinition",
+    "TableModel",
+    "__version__",
+]

detectkit-0.1.0/detectkit/alerting/__init__.py

@@ -0,0 +1,13 @@
+"""Alerting functionality for detectk."""
+
+from detectkit.alerting.orchestrator import (
+    AlertConditions,
+    AlertOrchestrator,
+    DetectionRecord,
+)
+
+__all__ = [
+    "AlertOrchestrator",
+    "AlertConditions",
+    "DetectionRecord",
+]

detectkit-0.1.0/detectkit/alerting/channels/__init__.py

@@ -0,0 +1,21 @@
+"""Alert channels for external notifications."""
+
+from detectkit.alerting.channels.base import AlertData, BaseAlertChannel
+from detectkit.alerting.channels.mattermost import MattermostChannel
+from detectkit.alerting.channels.slack import SlackChannel
+from detectkit.alerting.channels.webhook import WebhookChannel
+from detectkit.alerting.channels.telegram import TelegramChannel
+from detectkit.alerting.channels.email import EmailChannel
+
+from detectkit.alerting.channels.factory import AlertChannelFactory
+
+__all__ = [
+    "AlertData",
+    "BaseAlertChannel",
+    "WebhookChannel",
+    "MattermostChannel",
+    "SlackChannel",
+    "TelegramChannel",
+    "EmailChannel",
+    "AlertChannelFactory",
+]

detectkit-0.1.0/detectkit/alerting/channels/base.py

@@ -0,0 +1,191 @@
+"""
+Base alert channel interface.
+
+All alert channels must inherit from BaseAlertChannel and implement
+the send() method for delivering alerts to specific destinations.
+"""
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass
+from typing import Any, Dict, List, Optional
+
+from detectkit.detectors.base import DetectionResult
+
+
+@dataclass
+class AlertData:
+    """
+    Data for alert message.
+
+    Contains all information needed to format and send an alert.
+
+    Attributes:
+        metric_name: Name of the metric
+        timestamp: Timestamp of the anomaly (datetime64)
+        timezone: Timezone for display (e.g., "Europe/Moscow")
+        value: Actual metric value
+        confidence_lower: Lower confidence bound
+        confidence_upper: Upper confidence bound
+        detector_name: Name/ID of detector that found the anomaly
+        detector_params: Detector parameters (JSON string)
+        direction: Direction of anomaly ("above" or "below")
+        severity: Severity score
+        detection_metadata: Additional metadata from detector
+        consecutive_count: Number of consecutive anomalies
+    """
+
+    metric_name: str
+    timestamp: Any  # datetime64 or datetime
+    timezone: str
+    value: float
+    confidence_lower: Optional[float]
+    confidence_upper: Optional[float]
+    detector_name: str
+    detector_params: str
+    direction: str
+    severity: float
+    detection_metadata: Dict[str, Any]
+    consecutive_count: int = 1
+
+
+class BaseAlertChannel(ABC):
+    """
+    Abstract base class for alert channels.
+
+    Alert channels deliver notifications to external systems when
+    anomalies are detected. Each channel implements a specific
+    delivery mechanism (webhook, email, etc.).
+
+    Example:
+        >>> class MyChannel(BaseAlertChannel):
+        ...     def send(self, alert_data, template=None):
+        ...         message = self.format_message(alert_data, template)
+        ...         # Send via specific mechanism
+        ...         return True
+    """
+
+    @abstractmethod
+    def send(
+        self,
+        alert_data: AlertData,
+        template: Optional[str] = None,
+    ) -> bool:
+        """
+        Send alert to this channel.
+
+        Args:
+            alert_data: Alert data to send
+            template: Optional custom message template
+                     Uses default template if None
+
+        Returns:
+            True if sent successfully, False otherwise
+
+        Raises:
+            Exception: If sending fails critically
+
+        Example:
+            >>> alert = AlertData(
+            ...     metric_name="cpu_usage",
+            ...     timestamp=datetime.now(),
+            ...     value=95.0,
+            ...     ...
+            ... )
+            >>> success = channel.send(alert)
+        """
+        pass
+
+    def format_message(
+        self,
+        alert_data: AlertData,
+        template: Optional[str] = None,
+    ) -> str:
+        """
+        Format alert message from template.
+
+        Uses default template if none provided. Template variables:
+        - {metric_name}
+        - {timestamp}
+        - {timezone}
+        - {value}
+        - {confidence_lower}
+        - {confidence_upper}
+        - {detector_name}
+        - {direction}
+        - {severity}
+        - {consecutive_count}
+
+        Args:
+            alert_data: Alert data to format
+            template: Optional custom template string
+
+        Returns:
+            Formatted message string
+
+        Example:
+            >>> template = "Anomaly in {metric_name}: {value}"
+            >>> message = channel.format_message(alert_data, template)
+        """
+        if template is None:
+            template = self.get_default_template()
+
+        # Format timestamp to string
+        from datetime import datetime
+        import numpy as np
+
+        ts = alert_data.timestamp
+        if isinstance(ts, np.datetime64):
+            ts = ts.astype(datetime)
+
+        # Format timestamp with timezone
+        ts_str = ts.strftime("%Y-%m-%d %H:%M:%S")
+        if alert_data.timezone:
+            ts_str = f"{ts_str} ({alert_data.timezone})"
+
+        # Format confidence interval
+        if alert_data.confidence_lower is not None and alert_data.confidence_upper is not None:
+            confidence_str = f"[{alert_data.confidence_lower:.2f}, {alert_data.confidence_upper:.2f}]"
+        else:
+            confidence_str = "N/A"
+
+        # Format message
+        try:
+            message = template.format(
+                metric_name=alert_data.metric_name,
+                timestamp=ts_str,
+                timezone=alert_data.timezone,
+                value=alert_data.value,
+                confidence_lower=alert_data.confidence_lower,
+                confidence_upper=alert_data.confidence_upper,
+                confidence_interval=confidence_str,
+                detector_name=alert_data.detector_name,
+                direction=alert_data.direction,
+                severity=alert_data.severity,
+                consecutive_count=alert_data.consecutive_count,
+            )
+        except KeyError as e:
+            # If template has unknown variables, fall back to default
+            message = self.format_message(alert_data, self.get_default_template())
+
+        return message
+
+    def get_default_template(self) -> str:
+        """
+        Get default message template.
+
+        Returns:
+            Default template string
+        """
+        return (
+            "Anomaly detected in metric: {metric_name}\n"
+            "Time: {timestamp}\n"
+            "Value: {value}\n"
+            "Confidence interval: {confidence_interval}\n"
+            "Detector: {detector_name}\n"
+            "Direction: {direction}\n"
+            "Severity: {severity:.2f}"
+        )
+
+    def __repr__(self) -> str:
+        """String representation of channel."""
+        return f"{self.__class__.__name__}()"