detectkit 0.41.0__tar.gz → 0.43.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {detectkit-0.41.0/detectkit.egg-info → detectkit-0.43.0}/PKG-INFO +1 -1
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/__init__.py +1 -1
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/__init__.py +0 -2
- detectkit-0.43.0/detectkit/autotune/runner.py +102 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/CLAUDE.section.md +9 -5
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/rules/autotune.md +43 -47
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/rules/cli.md +18 -5
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/skills/dtk-autotune/SKILL.md +32 -33
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/skills/dtk-setup-project/SKILL.md +2 -1
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/autotune.py +30 -180
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/init.py +3 -2
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/tune.py +4 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/main.py +6 -28
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/reporting/html_report.py +6 -5
- detectkit-0.43.0/detectkit/tuning/assets/tune.js +187 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/tuning/payload.py +31 -17
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/tuning/server.py +166 -12
- {detectkit-0.41.0 → detectkit-0.43.0/detectkit.egg-info}/PKG-INFO +1 -1
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit.egg-info/SOURCES.txt +1 -2
- detectkit-0.41.0/detectkit/autotune/html_labeler.py +0 -969
- detectkit-0.41.0/detectkit/autotune/label_server.py +0 -177
- detectkit-0.41.0/detectkit/tuning/assets/tune.js +0 -176
- {detectkit-0.41.0 → detectkit-0.43.0}/LICENSE +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/MANIFEST.in +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/README.md +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/base.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/branding.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/email.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/factory.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/mattermost.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/slack.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/telegram.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/channels/webhook.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/_base.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/_cooldown.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/_decision.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/_dispatch.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/_recovery.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/_replay.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/_types.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/alerting/orchestrator/orchestrator.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/_base.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/_types.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/autotuner.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/config_emitter.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/crossval.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/detector_select.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/distribution.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/grid_search.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/labels.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/result.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/scoring.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/seasonality_search.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/settings.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/autotune/window_select.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/_output.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/rules/alerting.md +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/rules/detectors.md +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/rules/metrics.md +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/rules/overview.md +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/rules/project.md +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/skills/dtk-feedback/SKILL.md +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/skills/dtk-new-metric/SKILL.md +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/clean.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/init_claude.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/run.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/test_alert.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/commands/unlock.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/config/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/config/metric_config.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/config/profile.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/config/project_config.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/config/validator.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/core/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/core/interval.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/core/models.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/_sql_manager.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/clickhouse_manager.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_alert_states.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_autotune_runs.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_base.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_datapoints.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_detections.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_maintenance.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_metrics.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_schema.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/_tasks.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/internal_tables/manager.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/manager.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/mysql_manager.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/postgres_manager.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/database/tables.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/base.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/factory.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/seasonality.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/statistical/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/statistical/_windowed.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/statistical/iqr.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/statistical/mad.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/statistical/manual_bounds.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/detectors/statistical/zscore.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/loaders/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/loaders/metric_loader.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/loaders/query_template.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/error_dispatch.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/task_manager/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/task_manager/_alert_step.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/task_manager/_base.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/task_manager/_detect_step.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/task_manager/_load_step.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/task_manager/_types.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/orchestration/task_manager/manager.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/reporting/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/reporting/assets/report.js +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/reporting/builder.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/tuning/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/tuning/config_writer.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/tuning/html.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/utils/__init__.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/utils/datetime_utils.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/utils/env_interpolation.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/utils/json_utils.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit/utils/stats.py +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit.egg-info/dependency_links.txt +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit.egg-info/entry_points.txt +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit.egg-info/requires.txt +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/detectkit.egg-info/top_level.txt +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/pyproject.toml +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/requirements.txt +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/setup.cfg +0 -0
- {detectkit-0.41.0 → detectkit-0.43.0}/setup.py +0 -0
|
@@ -4,7 +4,7 @@ detectk - Anomaly Detection for Time-Series Metrics
|
|
|
4
4
|
A Python library for data analysts and engineers to monitor metrics with automatic anomaly detection.
|
|
5
5
|
"""
|
|
6
6
|
|
|
7
|
-
__version__ = "0.
|
|
7
|
+
__version__ = "0.43.0"
|
|
8
8
|
|
|
9
9
|
from detectkit.core.interval import Interval
|
|
10
10
|
from detectkit.core.models import ColumnDefinition, TableModel
|
|
@@ -15,7 +15,6 @@ from detectkit.autotune._base import AutoTuneError, _AutoTuneBase
|
|
|
15
15
|
from detectkit.autotune._types import ScoringMetric, TuneMode
|
|
16
16
|
from detectkit.autotune.autotuner import AutoTuner, run_autotune_engine
|
|
17
17
|
from detectkit.autotune.config_emitter import compute_run_id, emit_tuned_config
|
|
18
|
-
from detectkit.autotune.html_labeler import render_labeler_html
|
|
19
18
|
from detectkit.autotune.labels import (
|
|
20
19
|
GroundTruth,
|
|
21
20
|
IncidentLabels,
|
|
@@ -39,6 +38,5 @@ __all__ = [
|
|
|
39
38
|
"emit_tuned_config",
|
|
40
39
|
"parse_incident_labels",
|
|
41
40
|
"parse_labels_file",
|
|
42
|
-
"render_labeler_html",
|
|
43
41
|
"run_autotune_engine",
|
|
44
42
|
]
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
"""Shared autotune entry point reused by ``dtk autotune`` and ``dtk tune``.
|
|
2
|
+
|
|
3
|
+
The pure ``run_autotune_engine`` operates on an in-memory ``data`` dict, a
|
|
4
|
+
``GroundTruth`` and a ``TuneSettings``. Both callers need the same plumbing to
|
|
5
|
+
get there from a metric's ``AutoTuneConfig`` + labels: resolve the scoring
|
|
6
|
+
metric, cap the training history, build the settings, project the labels onto
|
|
7
|
+
the grid, and run the engine. That plumbing used to live privately in
|
|
8
|
+
``cli/commands/autotune.py``; it lives here so the ``dtk tune`` server can reuse
|
|
9
|
+
it verbatim for its server-side **Autotune** mode (no browser port of the
|
|
10
|
+
scoring/CV/seasonality search). Neither this module nor the engine touches the
|
|
11
|
+
database or the filesystem — the caller loads the datapoints and decides what to
|
|
12
|
+
persist.
|
|
13
|
+
"""
|
|
14
|
+
|
|
15
|
+
from __future__ import annotations
|
|
16
|
+
|
|
17
|
+
from collections.abc import Callable
|
|
18
|
+
|
|
19
|
+
import numpy as np
|
|
20
|
+
|
|
21
|
+
from detectkit.autotune._types import ScoringMetric
|
|
22
|
+
from detectkit.autotune.autotuner import run_autotune_engine
|
|
23
|
+
from detectkit.autotune.labels import IncidentLabels
|
|
24
|
+
from detectkit.autotune.result import AutoTuneResult
|
|
25
|
+
from detectkit.autotune.settings import TuneSettings
|
|
26
|
+
from detectkit.config.metric_config import AutoTuneConfig
|
|
27
|
+
|
|
28
|
+
# Cap the scored training span when the user doesn't pin max_history, so tuning
|
|
29
|
+
# stays responsive on very long histories (recent data is the most relevant).
|
|
30
|
+
DEFAULT_TRAIN_CAP = 50_000
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
def resolve_scoring(scoring_override: str | None, autotune_cfg: AutoTuneConfig) -> ScoringMetric:
|
|
34
|
+
"""Resolve the scoring metric: CLI override > config > MCC default.
|
|
35
|
+
|
|
36
|
+
Raises ``ValueError`` on an unknown name (callers translate as they see fit).
|
|
37
|
+
"""
|
|
38
|
+
value = scoring_override or autotune_cfg.scoring_metric or ScoringMetric.MCC.value
|
|
39
|
+
try:
|
|
40
|
+
return ScoringMetric(value)
|
|
41
|
+
except ValueError as exc:
|
|
42
|
+
allowed = ", ".join(m.value for m in ScoringMetric)
|
|
43
|
+
raise ValueError(f"Invalid scoring metric '{value}'. Allowed: {allowed}") from exc
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
def build_settings(*, scoring: ScoringMetric, autotune_cfg: AutoTuneConfig) -> TuneSettings:
|
|
47
|
+
"""Map an ``AutoTuneConfig`` block (+ resolved scoring) to ``TuneSettings``."""
|
|
48
|
+
return TuneSettings(
|
|
49
|
+
metric=scoring,
|
|
50
|
+
beta=autotune_cfg.beta,
|
|
51
|
+
fold_count=autotune_cfg.folds,
|
|
52
|
+
stability_lambda=autotune_cfg.stability_lambda,
|
|
53
|
+
allowed_detector_types=autotune_cfg.detector_types,
|
|
54
|
+
allowed_seasonality=autotune_cfg.seasonality_candidates,
|
|
55
|
+
force_seasonality=autotune_cfg.force_seasonality,
|
|
56
|
+
fixed_params=dict(autotune_cfg.fixed_params),
|
|
57
|
+
max_history=autotune_cfg.max_history,
|
|
58
|
+
)
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
def cap_history(data: dict[str, np.ndarray], max_history: int | None) -> dict[str, np.ndarray]:
|
|
62
|
+
"""Keep the most recent ``max_history`` (or the default cap) points."""
|
|
63
|
+
n = len(data["timestamp"])
|
|
64
|
+
cap = max_history if max_history is not None else DEFAULT_TRAIN_CAP
|
|
65
|
+
if n <= cap:
|
|
66
|
+
return data
|
|
67
|
+
return {
|
|
68
|
+
"timestamp": data["timestamp"][-cap:],
|
|
69
|
+
"value": data["value"][-cap:],
|
|
70
|
+
"seasonality_data": data["seasonality_data"][-cap:],
|
|
71
|
+
"seasonality_columns": data["seasonality_columns"],
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
def autotune_from_data(
|
|
76
|
+
*,
|
|
77
|
+
metric_name: str,
|
|
78
|
+
data: dict[str, np.ndarray],
|
|
79
|
+
labels: IncidentLabels,
|
|
80
|
+
interval_seconds: int,
|
|
81
|
+
autotune_cfg: AutoTuneConfig,
|
|
82
|
+
scoring_override: str | None = None,
|
|
83
|
+
on_stage: Callable[[str, str], None] | None = None,
|
|
84
|
+
) -> AutoTuneResult:
|
|
85
|
+
"""Run the full autotune engine on an in-memory series + labels.
|
|
86
|
+
|
|
87
|
+
Caps the history, resolves the scoring metric, projects ``labels`` onto the
|
|
88
|
+
(capped) grid as ground truth, builds the settings and runs the engine.
|
|
89
|
+
Pure: no DB, no filesystem, no lock — the caller decides what to persist.
|
|
90
|
+
"""
|
|
91
|
+
data = cap_history(data, autotune_cfg.max_history)
|
|
92
|
+
scoring = resolve_scoring(scoring_override, autotune_cfg)
|
|
93
|
+
ground_truth = labels.to_ground_truth(data["timestamp"], interval_seconds)
|
|
94
|
+
settings = build_settings(scoring=scoring, autotune_cfg=autotune_cfg)
|
|
95
|
+
return run_autotune_engine(
|
|
96
|
+
metric_name=metric_name,
|
|
97
|
+
data=data,
|
|
98
|
+
ground_truth=ground_truth,
|
|
99
|
+
interval_seconds=interval_seconds,
|
|
100
|
+
settings=settings,
|
|
101
|
+
on_stage=on_stage,
|
|
102
|
+
)
|
|
@@ -17,10 +17,12 @@ run. But you assist far better with **read access to the same database** (e.g. a
|
|
|
17
17
|
database MCP for the project's ClickHouse / PostgreSQL / MySQL): you can inspect
|
|
18
18
|
a metric's series, find real incidents to label for `dtk autotune`, sanity-check
|
|
19
19
|
a metric query before running it, and confirm detections — instead of asking the
|
|
20
|
-
user to run every query by hand. Without it, fall back to
|
|
21
|
-
`dtk
|
|
22
|
-
|
|
23
|
-
|
|
20
|
+
user to run every query by hand. Without it, fall back to marking incidents
|
|
21
|
+
visually in `dtk tune --select <metric>` (**Label** mode to drag spans / Threshold
|
|
22
|
+
capture / Lasso the anomaly cloud, or **Review** mode to confirm fired alerts as
|
|
23
|
+
incidents), then **Save incidents** — which `dtk autotune` auto-discovers in
|
|
24
|
+
`incidents/<metric>/` — and to asking the user. This access is optional and
|
|
25
|
+
separate from the `profiles.yml` connection detectkit uses for the pipeline.
|
|
24
26
|
|
|
25
27
|
### Where to look (read the matching file before answering)
|
|
26
28
|
|
|
@@ -52,7 +54,9 @@ version — **read the relevant one on demand** instead of guessing:
|
|
|
52
54
|
`dtk autotune`, and explains the chosen, annotated config (an automatic,
|
|
53
55
|
cross-validated search). For **hands-on** tuning instead, `dtk tune` opens a
|
|
54
56
|
browser view where you drag the detector's knobs against the real series and
|
|
55
|
-
write the result back in place
|
|
57
|
+
write the result back in place — and its **Autotune** mode can run that same
|
|
58
|
+
search server-side and re-seed the knobs without leaving the cockpit (see
|
|
59
|
+
`cli.md`).
|
|
56
60
|
- **Hit a detectkit bug, or have feedback** — once you've ruled out a local
|
|
57
61
|
config fix (see the gotchas below), use the **`dtk-feedback`** skill to file a
|
|
58
62
|
redacted bug report, feature request, or comment as a GitHub issue on the
|
|
@@ -17,6 +17,14 @@ same windowed detectors and `detector_id` identity). The fastest path is the
|
|
|
17
17
|
> `metrics/.history/<metric>/` first). Use `autotune` to search automatically and
|
|
18
18
|
> emit a new file; use `tune` to dial a detector in by eye and commit it. See
|
|
19
19
|
> `cli.md`.
|
|
20
|
+
>
|
|
21
|
+
> **You can also run this engine *inside* `dtk tune`** — its **Autotune** mode runs
|
|
22
|
+
> the same search **server-side** (using the incidents you've marked as ground
|
|
23
|
+
> truth), re-seeds the knobs with the winner, and lets you **Apply** in place. That's
|
|
24
|
+
> the same computation as this command; the difference is it edits the metric YAML in
|
|
25
|
+
> place (advisory — no run record / `__tuned_<id>.yml` / persisted detections) rather
|
|
26
|
+
> than emitting a new tuned file. Reach for `dtk autotune` when you want the audited
|
|
27
|
+
> run + tuned file + persisted winner detections.
|
|
20
28
|
|
|
21
29
|
## What it searches
|
|
22
30
|
|
|
@@ -64,28 +72,24 @@ ratios to choose.
|
|
|
64
72
|
## Command
|
|
65
73
|
|
|
66
74
|
```bash
|
|
67
|
-
dtk autotune --select <sel> [--incidents FILE] [--
|
|
75
|
+
dtk autotune --select <sel> [--incidents FILE] [--scoring METRIC] \
|
|
68
76
|
[--from DATE] [--to DATE] [--profile NAME] [--force] [--dry-run] [--report]
|
|
69
77
|
```
|
|
70
78
|
|
|
71
79
|
- `--incidents FILE|DIR` — a labels file (below) → **supervised** tuning. May be a
|
|
72
80
|
**directory** (e.g. `incidents/<name>/`): interactive runs prompt to pick a
|
|
73
|
-
version (default newest), non-interactive use the newest.
|
|
81
|
+
version (default newest), non-interactive use the newest. **Omit it and `dtk
|
|
82
|
+
autotune` auto-discovers the newest labels in `incidents/<metric>/`** (the store
|
|
83
|
+
`dtk tune`'s **Save incidents** writes) — so after labeling in `dtk tune` you run
|
|
84
|
+
`dtk autotune --select <metric>` with no flag. With no labels anywhere, an
|
|
74
85
|
interactive terminal prompts to enter incidents inline; declining (or running
|
|
75
86
|
non-interactively) tunes **unsupervised**.
|
|
76
|
-
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
editing in place. Beyond click-drag marking it offers **Threshold capture**
|
|
83
|
-
(grab every span above/below a horizontal line in one gesture), **Lasso capture**
|
|
84
|
-
(loop around a cloud of points; each grid-adjacent run, gaps bridged, becomes one
|
|
85
|
-
incident span), **on-chart delete** (each band's ✕, or select + Delete key), and
|
|
86
|
-
**Import file…** (load a labels file you pick). `--no-serve` writes a static
|
|
87
|
-
`metrics/<name>__labeler.html` (Export downloads the file) and exits; `--no-open`
|
|
88
|
-
prints the URL instead of launching a browser.
|
|
87
|
+
- **To label incidents**, use `dtk tune --select <metric>`: switch to **Label**
|
|
88
|
+
mode (drag spans, **Threshold capture** every span past a horizontal line,
|
|
89
|
+
**Lasso** the anomaly cloud) or **Review** mode (confirm fired alerts as
|
|
90
|
+
incidents), then **Save incidents**. That writes versioned files into
|
|
91
|
+
`incidents/<metric>/` — the **same store this command reads** — so the labels feed
|
|
92
|
+
the next supervised tune with no `--incidents` flag (see `cli.md`).
|
|
89
93
|
- `--scoring` — `mcc` (default), `f1`, `f_beta`, `balanced_accuracy`, `roc_auc`,
|
|
90
94
|
`pr_auc`. MCC uses the whole confusion matrix and suits rare anomalies.
|
|
91
95
|
- `--dry-run` — run the search but persist nothing and write no config.
|
|
@@ -115,45 +119,35 @@ incidents:
|
|
|
115
119
|
- {at: "2026-05-11 09:05:00"} # point
|
|
116
120
|
```
|
|
117
121
|
|
|
118
|
-
### Getting labels —
|
|
122
|
+
### Getting labels — label in `dtk tune` first
|
|
119
123
|
|
|
120
|
-
When labels would help, **offer
|
|
124
|
+
When labels would help, **offer to mark incidents in `dtk tune` before asking the
|
|
121
125
|
user to recall timestamps** — it is the easiest, most reliable path:
|
|
122
126
|
|
|
123
|
-
1. Run `dtk
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
3. That writes `incidents/<metric>/<metric>[-<set>]-<UTC>.yml` automatically
|
|
139
|
-
(named after the metric, optional set name as a suffix; versioned —
|
|
140
|
-
re-labeling never overwrites) and the **same command continues into the tuning
|
|
141
|
-
run** on it. No manual file moving.
|
|
142
|
-
4. **Editing over time:** re-running `--label` seeds the page from the metric's
|
|
143
|
-
newest saved set, so labeling can grow across sessions. To re-tune later on
|
|
144
|
-
saved sets, point `--incidents` at the folder (`incidents/<name>/`) —
|
|
145
|
-
interactive runs let the user pick a version, and `--label --incidents <file>`
|
|
146
|
-
opens that exact set for editing. The static `--no-serve` path still exists
|
|
147
|
-
(Export downloads a file you move into `incidents/<name>/`; its **Import
|
|
148
|
-
file…** button loads one back in).
|
|
127
|
+
1. Run `dtk tune --select <name>`. It opens a localhost browser view of the real
|
|
128
|
+
series (use `--no-open` on a remote box and share the printed 127.0.0.1 URL).
|
|
129
|
+
2. Mark the real incidents one of two ways:
|
|
130
|
+
- **Label** mode — drag a span over each incident, or **Threshold capture**
|
|
131
|
+
every span past a horizontal line in one gesture, or **Lasso anomalies** to
|
|
132
|
+
loop a cloud of anomaly dots into per-streak incident spans.
|
|
133
|
+
- **Review** mode — the fired alerts lead; click each alert marker to confirm it
|
|
134
|
+
**valid** (which marks it as a ground-truth incident) or **false alarm**.
|
|
135
|
+
**Confirm all unreviewed valid** does the lot — a clean metric whose alerts
|
|
136
|
+
are all good is validated in a few clicks without hand-drawing spans.
|
|
137
|
+
3. Click **Save incidents**. It writes a versioned
|
|
138
|
+
`incidents/<metric>/<…>.yml` automatically.
|
|
139
|
+
4. Run `dtk autotune --select <name>` with **no `--incidents` flag** — it
|
|
140
|
+
auto-discovers the newest file in `incidents/<metric>/` and tunes supervised on
|
|
141
|
+
it. (You can still pass `--incidents <file-or-dir>` to pick a specific set.)
|
|
149
142
|
|
|
150
143
|
Prefer this whenever the user can *recognise* incidents on a chart but doesn't
|
|
151
144
|
have exact times. If they already know the times (or you found them via a DB
|
|
152
145
|
MCP), write the labels file / inline `incidents:` directly instead. If there are
|
|
153
146
|
no known incidents, run unsupervised — the baseline below is good on its own.
|
|
154
147
|
|
|
155
|
-
With no labels (no `--incidents`, no config `labels_file`,
|
|
156
|
-
entry), tuning falls back to an
|
|
148
|
+
With no labels (no `--incidents`, no config `labels_file`, none auto-discovered
|
|
149
|
+
in `incidents/<metric>/`, no interactive entry), tuning falls back to an
|
|
150
|
+
**unsupervised** objective that blends three
|
|
157
151
|
band-fit / flag-budget terms — `0.4·budget + 0.3·sharpness + 0.3·separation`: a
|
|
158
152
|
smooth one-sided flag-rate budget (no hard cliff), sharpness (a tight,
|
|
159
153
|
well-calibrated confidence interval), and separation (flagged points sitting
|
|
@@ -192,8 +186,10 @@ differs from `seasonality_candidates`, which only narrows the *set of columns*
|
|
|
192
186
|
the search may consider but still runs the search and may choose none.
|
|
193
187
|
|
|
194
188
|
Label precedence (highest first): `--incidents` flag → `labels_file` → inline
|
|
195
|
-
`incidents` →
|
|
196
|
-
|
|
189
|
+
`incidents` → auto-discovered `incidents/<metric>/` (the newest file, e.g. from
|
|
190
|
+
`dtk tune`'s **Save incidents**) → interactive prompt → none (unsupervised).
|
|
191
|
+
`labels_file` and `incidents` are mutually exclusive. `--scoring` likewise
|
|
192
|
+
overrides `scoring_metric`.
|
|
197
193
|
|
|
198
194
|
## The annotated config
|
|
199
195
|
|
|
@@ -67,8 +67,11 @@ Automatically chooses a metric's seasonality, detector type, hyperparameters and
|
|
|
67
67
|
history window, then writes an annotated `metrics/<name>__tuned_<id>.yml`. Reads
|
|
68
68
|
the metric's loaded datapoints (run `dtk run --steps load` first if empty), never
|
|
69
69
|
edits the original, never alerts. `--incidents FILE` enables supervised tuning
|
|
70
|
-
against labeled incidents;
|
|
71
|
-
|
|
70
|
+
against labeled incidents; omit it and autotune **auto-discovers** the newest
|
|
71
|
+
labels in `incidents/<metric>/` (the store `dtk tune`'s **Save incidents** writes
|
|
72
|
+
— label there in Label/Review mode, then just run `dtk autotune`); with no labels
|
|
73
|
+
anywhere, an unsupervised objective is used. `--dry-run` searches without writing.
|
|
74
|
+
`--report [PATH]` writes the same
|
|
72
75
|
self-contained HTML report as `dtk run` for the tuned winner (default
|
|
73
76
|
`reports/<metric>__tuned_<id>.html`; `<dir>` or a `.html` file also accepted).
|
|
74
77
|
Full reference: `autotune.md`.
|
|
@@ -85,7 +88,8 @@ cockpit**: ONE chart (the windshield) fills the view, the live metrics ride
|
|
|
85
88
|
**pinned in a HUD over the chart** (the speedometer), and every control lives in an
|
|
86
89
|
**always-visible, mode-aware side rail** beside the chart (Tune shows the detector
|
|
87
90
|
knobs + effective config + Apply, Review the verdict actions, Label the capture
|
|
88
|
-
tools + incident list + Save
|
|
91
|
+
tools + incident list + Save, Autotune the search button + winning config), while
|
|
92
|
+
the controls that aren't detector-specific —
|
|
89
93
|
the **Points shown** data window, the alert rule (**direction** + **consecutive
|
|
90
94
|
anomalies**) and the **y = 0** toggle — stay visible in every mode. The chart is
|
|
91
95
|
**zoomable** (scroll/drag + navigator strip) with a **"Points shown"** trim slider.
|
|
@@ -100,11 +104,20 @@ and which interactions are armed on the one chart: **Tune** (band leads; inciden
|
|
|
100
104
|
recede to read-only context; hover a point for its window), **Review** (the fired
|
|
101
105
|
alerts lead, band ghosts — click an alert marker to cycle its verdict un-reviewed →
|
|
102
106
|
**valid** (green) → **false alarm** (slate); **Confirm all unreviewed valid** does
|
|
103
|
-
the lot),
|
|
107
|
+
the lot), **Label** (band hides; **mark the real incidents** by drag, **Lasso
|
|
104
108
|
anomalies** — loop a cloud of anomaly dots, each consecutive run, gaps bridged up to
|
|
105
109
|
`consecutive_anomalies`, becomes one span — or **Threshold capture** every span past
|
|
106
110
|
a horizontal line, each widened to a full interval so the alert lands inside; the
|
|
107
|
-
painted window saves as `capture_windows`)
|
|
111
|
+
painted window saves as `capture_windows`), and **Autotune** (**Run autotune**
|
|
112
|
+
launches the **real** `dtk autotune` engine **server-side** over the metric's full
|
|
113
|
+
history, using your marked incidents as ground truth, then **re-seeds every knob**
|
|
114
|
+
with the winner and shows the score + decision log; the band leads like Tune). The
|
|
115
|
+
Autotune mode is **advisory** — it computes + re-seeds only and persists nothing (no
|
|
116
|
+
run record / `__tuned_<id>.yml` / detections, so `dtk tune` stays lock-free); review
|
|
117
|
+
the band and **Apply** to write it back. It honours the metric's `autotune:` block,
|
|
118
|
+
is supervised when incidents are marked (also picks `consecutive_anomalies`) else
|
|
119
|
+
unsupervised, and needs the live server (unavailable under `--no-serve`).
|
|
120
|
+
**Confirming an alert valid IS marking an
|
|
108
121
|
incident**: the confirmed streak becomes a first-class **ground-truth incident** that
|
|
109
122
|
shows in the Marked-incidents list (a read-only "✓ confirmed alert" row; its ✕
|
|
110
123
|
un-confirms the alert), counts toward recall + correct, and is written on Save — so a
|
{detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/skills/dtk-autotune/SKILL.md
RENAMED
|
@@ -79,44 +79,39 @@ generalize is rejected.
|
|
|
79
79
|
Labels turn tuning from a good unsupervised default into a config optimised
|
|
80
80
|
against *your* real incidents (it then optimises MCC and also tunes the alert
|
|
81
81
|
window `consecutive_anomalies`). **The easiest, most reliable way to produce them
|
|
82
|
-
is
|
|
82
|
+
is to mark incidents in `dtk tune` — offer this first**, before asking the user to
|
|
83
83
|
recall timestamps:
|
|
84
84
|
|
|
85
85
|
```bash
|
|
86
|
-
dtk
|
|
86
|
+
dtk tune --select <name>
|
|
87
87
|
```
|
|
88
88
|
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
user through it:
|
|
89
|
+
This opens a localhost browser view of the metric's **real** series; on a remote
|
|
90
|
+
machine add `--no-open` and share the printed URL. Walk the user through it:
|
|
92
91
|
|
|
93
92
|
1. Navigate a long/dense series: **scroll to zoom**, double-click to reset, drag
|
|
94
|
-
the **navigator strip** to move the view
|
|
95
|
-
2.
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
Export downloads a versioned file you then move into `incidents/<name>/` and pass
|
|
117
|
-
via `--incidents`; its **Import file…** button loads a saved set back in.) To
|
|
118
|
-
re-tune later on a saved set, point `--incidents` at `incidents/<name>/`
|
|
119
|
-
(interactive runs let the user pick a version).
|
|
93
|
+
the **navigator strip** to move the view.
|
|
94
|
+
2. Mark the real incidents in one of two modes:
|
|
95
|
+
- **Label** mode — **drag a span** over each incident (edges to resize, middle
|
|
96
|
+
to move, ✕/Delete to remove). When many outliers are obvious, **Threshold
|
|
97
|
+
capture** grabs every span past a horizontal line in one gesture (set the
|
|
98
|
+
line, pick the side, optionally bridge gaps); **Lasso anomalies** loops a
|
|
99
|
+
freeform cloud of anomaly dots into one incident span per grid-adjacent run.
|
|
100
|
+
- **Review** mode — the fired alerts lead; **click each alert marker** to cycle
|
|
101
|
+
its verdict un-reviewed → **valid** (green) → **false alarm** (slate).
|
|
102
|
+
**Confirming an alert valid marks it as a ground-truth incident**, and
|
|
103
|
+
**Confirm all unreviewed valid** does the lot — so a clean metric whose alerts
|
|
104
|
+
are all good is validated in a few clicks without hand-drawing spans.
|
|
105
|
+
3. Click **Save incidents**. It writes `incidents/<metric>/<…>.yml` automatically
|
|
106
|
+
(versioned — saving never overwrites). Saving doesn't end the session, so you
|
|
107
|
+
can keep tuning; you can also click **Apply** to write the detector config back.
|
|
108
|
+
|
|
109
|
+
**Then run autotune with no `--incidents` flag** — `dtk autotune --select <name>`
|
|
110
|
+
**auto-discovers the newest file in `incidents/<metric>/`** (the same store
|
|
111
|
+
`dtk tune` saves to) and tunes supervised on it. To re-tune on a specific set,
|
|
112
|
+
pass `--incidents <file-or-dir>` (point it at `incidents/<name>/` and interactive
|
|
113
|
+
runs let the user pick a version). Labeling grows across sessions — reopen
|
|
114
|
+
`dtk tune`, mark a few more, **Save incidents** again, re-run autotune.
|
|
120
115
|
|
|
121
116
|
Prefer this whenever the user can *recognise* incidents on a chart but doesn't
|
|
122
117
|
have exact timestamps — it is far easier than dictating times, and they label
|
|
@@ -158,10 +153,14 @@ is good on its own; it just can't learn *which* spikes you specifically care abo
|
|
|
158
153
|
## Step 3 — Run autotune
|
|
159
154
|
|
|
160
155
|
```bash
|
|
161
|
-
# Supervised
|
|
156
|
+
# Supervised — labeled in `dtk tune`: just run it, autotune auto-discovers
|
|
157
|
+
# the newest file in incidents/<name>/ (no --incidents flag needed):
|
|
158
|
+
dtk autotune --select <name>
|
|
159
|
+
|
|
160
|
+
# Supervised — point at a specific labels file or set explicitly:
|
|
162
161
|
dtk autotune --select <name> --incidents incidents/<name>.yml
|
|
163
162
|
|
|
164
|
-
# Unsupervised (no labels — tunes on data statistics):
|
|
163
|
+
# Unsupervised (no labels anywhere — tunes on data statistics):
|
|
165
164
|
dtk autotune --select <name>
|
|
166
165
|
```
|
|
167
166
|
|
{detectkit-0.41.0 → detectkit-0.43.0}/detectkit/cli/assets/claude/skills/dtk-setup-project/SKILL.md
RENAMED
|
@@ -45,7 +45,8 @@ side by side, ask which one to set up.
|
|
|
45
45
|
> a database MCP): you can inspect metric series, find incidents to label for
|
|
46
46
|
> `dtk autotune`, sanity-check queries, and confirm detections yourself. Suggest
|
|
47
47
|
> the user connect one if they haven't; if they can't, you can still do
|
|
48
|
-
> everything by asking them to run queries and by
|
|
48
|
+
> everything by asking them to run queries and by marking incidents visually in
|
|
49
|
+
> `dtk tune` (Label/Review modes), which `dtk autotune` auto-discovers.
|
|
49
50
|
|
|
50
51
|
## Step 1 — Pick the database backend
|
|
51
52
|
|