ducted 2.0__tar.gz → 2.2__tar.gz

{ducted-2.0 → ducted-2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ducted
-Version: 2.0
+Version: 2.2
 Summary: A monitoring agent and event processor
 Author-email: Colin Alston <colin@tamvera.com>
 License: MIT
@@ -14,18 +14,22 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: System :: Monitoring
 Requires-Python: >=3.11
 License-File: LICENSE
-Requires-Dist: zope.interface
-Requires-Dist: PyYaml
-Requires-Dist: protobuf
-Requires-Dist: aiohttp
-Requires-Dist: asyncssh
-Requires-Dist: aiodns
-Requires-Dist: pysnmp>=6.0
-Requires-Dist: cryptography
-Requires-Dist: service_identity
+Requires-Dist: zope.interface>=8.5
+Requires-Dist: PyYaml~=6.0
+Requires-Dist: protobuf~=7.35
+Requires-Dist: aiohttp~=3.14
+Requires-Dist: asyncssh~=2.23
+Requires-Dist: aiodns~=4.0
+Requires-Dist: pysnmp~=7.1
+Requires-Dist: cryptography~=48.0
+Requires-Dist: service_identity~=26.1
+Requires-Dist: pydantic~=2.13
+Requires-Dist: construct~=2.10
+Requires-Dist: nats-py~=2.15
+Requires-Dist: cbor2~=6.1
+Requires-Dist: smbus2~=0.6
 Provides-Extra: dev
 Requires-Dist: asyncpg; extra == "dev"
-Requires-Dist: construct<2.6; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"

ducted-2.2/README.md

@@ -0,0 +1,180 @@
+# Duct
+
+I want to talk to you about ducts. Do your ducts seem old-fashioned? Out of date?
+
+[![CI](https://github.com/Tamvera/ducted/actions/workflows/ci.yml/badge.svg)](https://github.com/Tamvera/ducted/actions/workflows/ci.yml)
+[![Latest Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://tamvera.github.io/ducted/)
+
+**A pluggable Python telemetry pipeline.** Collect metrics from anywhere, route them to any backend, with nothing but Python and a YAML file.
+
+Duct is an asyncio daemon that polls *sources* on configurable intervals and routes the resulting events to *outputs*. Both are plain Python classes: anything Python can reach - a socket, a file, an HTTP API, a subprocess, an SSH session, a hardware sensor - can be a source or an output. No DSL, no code generation.
+
+## What makes it different
+
+- **Sources and outputs can run network servers.** The Prometheus output hosts an HTTP scrape endpoint inside the daemon; the sFlow source runs a UDP collector; the Riemann source acts as a full Riemann TCP server. Duct can receive telemetry from other systems, not only emit it.
+- **SSH remote checks, no remote agent.** Mark any source `use_ssh: true` and it runs transparently on a remote host over a pooled SSH connection.
+- **Modern, backend-agnostic.** InfluxDB 3, Prometheus, NATS (JetStream + SenML/CBOR), Elasticsearch, Graphite, and more. Use one output or several simultaneously.
+- **Fine-grained routing.** Route individual sources to specific outputs or sets of outputs.
+- **Threshold state evaluation.** Declare `warning`/`critical` expressions per-metric; Duct overrides event state automatically.
+- **Blueprint macros.** DRY config: define a toolbox of checks once, expand it across a list of hosts with a single block.
+
+## Built-in sources
+
+| Category | Sources |
+|---|---|
+| Linux system | CPU, memory, load average, disk I/O, disk free, network, process stats |
+| Sensors & hardware | lm-sensors, SMART, DS18B20 temperature, MPL115 pressure/temperature, UPS (NUT) |
+| Web & proxy | Apache, Nginx (status + log metrics), HAProxy |
+| Databases | PostgreSQL, Elasticsearch, Redis, RabbitMQ, Memcache, Riak |
+| Network | Ping, HTTP, sFlow UDP collector, SNMP |
+| Infrastructure | Docker containers, Postfix, Unbound, StrongSwan/IPsec |
+| Messaging | NATS subscriber (JetStream + SenML) |
+| Ingestion | Riemann TCP listener, Munin node client, uWSGI Emperor |
+
+## Installation
+
+### System wide on Linux, with a systemd script
+```bash
+curl -o - https://raw.githubusercontent.com/Tamvera/ducted/refs/heads/master/scripts/install.sh | sh
+```
+
+### With docker
+```bash
+docker run -i -v ./duct.yml:/duct/duct.yml ghcr.io/tamvera/ducted:latest
+```
+
+### With pip
+```bash
+pip install ducted
+```
+
+Requires Python 3.11+.
+
+## Quick start
+
+Create `duct.yml`:
+
+```yaml
+interval: 1.0
+ttl: 60.0
+
+outputs:
+  # Prometheus scrape endpoint at http://0.0.0.0:9100/metrics
+  - output: duct.outputs.prometheus.Prometheus
+    port: 9100
+
+sources:
+  - service: cpu
+    source: duct.sources.linux.basic.CPU
+    interval: 1.0
+    warning:  { cpu: "> 0.5" }
+    critical: { cpu: "> 0.8" }
+
+  - service: memory
+    source: duct.sources.linux.basic.Memory
+    interval: 10.0
+
+  - service: disk
+    source: duct.sources.linux.basic.DiskFree
+    interval: 60.0
+```
+
+```bash
+ductd -c duct.yml
+```
+
+## Outputs
+
+Fan out to multiple backends simultaneously by naming them:
+
+```yaml
+outputs:
+  - output: duct.outputs.prometheus.Prometheus
+    name: prom
+    port: 9100
+
+  - output: duct.outputs.influxdb3.InfluxDB3
+    name: influx
+    url: http://localhost:8086
+    database: mymetrics
+    token: my-token
+
+  - output: duct.outputs.nats.Nats
+    name: bus
+    servers: ["nats://localhost:4222"]
+    format: senml-cbor
+    jetstream: true
+```
+
+## Receiving telemetry
+
+Sources can listen as well as poll. Funnel events from other systems into the same pipeline and route them to any output:
+
+```yaml
+sources:
+  # Act as a Riemann TCP server - ingest events from other agents
+  - service: riemann-in
+    source: duct.sources.riemann.RiemannTCP
+    port: 5555
+    route: influx
+
+  # Collect sFlow from network switches (UDP server)
+  - service: sflow
+    source: duct.sources.sflow.sFlow
+    port: 6343
+
+  # Subscribe to a NATS subject
+  - service: nats-in
+    source: duct.sources.nats.Nats
+    servers: ["nats://localhost:4222"]
+    topics: ["metrics.>"]
+```
+
+## SSH remote checks
+
+Run checks on remote hosts without installing an agent. SSH connections are pooled per host:
+
+```yaml
+ssh_username: monitor
+ssh_keyfile: /etc/duct/id_ed25519
+
+sources:
+  - service: load
+    source: duct.sources.linux.basic.LoadAverage
+    use_ssh: true
+    ssh_host: web01.example.com
+    interval: 30.0
+```
+
+Use **blueprint macros** to expand a toolbox of checks across many hosts with a single config block - see the [documentation](https://tamvera.github.io/ducted/) for details.
+
+## Writing a plugin
+
+A source is a Python class. It can use any library, open any connection, or start any server:
+
+```python
+from zope.interface import implementer
+from duct.interfaces import IDuctSource
+from duct.objects import Source
+
+@implementer(IDuctSource)
+class MySource(Source):
+    async def get(self):
+        value = await fetch_something()
+        return self.createEvent('ok', 'My metric', value, prefix='mymetric')
+```
+
+Point to it in config by dotted import path:
+
+```yaml
+sources:
+  - service: mything
+    source: mypackage.mysource.MySource
+    interval: 10.0
+```
+
+Outputs follow the same pattern: subclass `Output`, implement `createClient`, and drain the event queue however you like - including running an embedded server.
+
+## Documentation
+
+Full documentation at <https://tamvera.github.io/ducted/>

{ducted-2.0 → ducted-2.2}/docs/conf.py

@@ -101,7 +101,7 @@ pygments_style = 'sphinx'
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'alabaster'
+html_theme = 'blue_nebula'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -130,7 +130,7 @@ html_theme = 'alabaster'
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+#html_static_path = ['_static']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied

ducted-2.2/docs/update_modules.py

@@ -0,0 +1,37 @@
+from pathlib import Path
+
+CWD = Path(__file__).parent
+BASE_DIR = (CWD / "..").resolve()
+
+RST_TEMPLATE = """%s
+
+"""
+
+def scan_modules(name):
+    doc_path = CWD / "api" / f"{name}.rst"
+    mod_path = BASE_DIR / "duct" / name
+
+    modules = []
+    for node in mod_path.rglob("**/*.py"):
+        if node.is_file() and node.name != "__init__.py":
+            filepath = node.relative_to(BASE_DIR)
+            module = str(filepath.parents[0]/filepath.stem).replace('/', '.')
+            modules.append(module)
+
+    modules.sort()
+    print(modules)
+    with open(doc_path, "w+t") as f:
+        f.write(f"duct.{name}\n")
+        f.write("**************\n\n")
+
+        for module in modules:
+            f.write(module + '\n')
+            f.write("===================================\n\n")
+            f.write(f".. automodule:: {module}\n")
+            f.write("   :members:\n")
+            f.write("   :show-inheritance:\n\n")
+
+if __name__ == "__main__":
+    scan_modules("sources")
+    scan_modules("outputs")
+    scan_modules("protocol")

{ducted-2.0 → ducted-2.2}/duct/__init__.py

@@ -1,6 +1,6 @@
 """Duct - A monitoring agent
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 
 """
 

{ducted-2.0 → ducted-2.2}/duct/__main__.py

@@ -1,5 +1,5 @@
 """
-Duct — asyncio entry point
+Duct - asyncio entry point
 
 Usage:
     ductd -c duct.yml
@@ -18,7 +18,7 @@ from duct.service import DuctService
 def main():
     """Entry point for the duct daemon."""
     parser = argparse.ArgumentParser(
-        description='Duct — A monitoring agent and event processor'
+        description='Duct - A monitoring agent and event processor'
     )
     parser.add_argument(
         '-c', '--config',
@@ -42,7 +42,10 @@ def main():
 
 
 async def _run(config_path):
+    log = logging.getLogger(__name__)
+    log.info("Starting Ductd service.")
     config = ConfigFile(config_path)
+
     svc = DuctService(config)
 
     loop = asyncio.get_running_loop()
@@ -56,6 +59,7 @@ async def _run(config_path):
         loop.add_signal_handler(sig, _handle_signal)
 
     await svc.startService()
+    log.info("Service running.")
 
     await stop_event.wait()
 

{ducted-2.0 → ducted-2.2}/duct/aggregators.py

@@ -2,7 +2,7 @@
 .. module:: aggregators
    :synopsis: Aggregation functions
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 """
 
 

{ducted-2.0 → ducted-2.2}/duct/configuration.py

@@ -2,22 +2,56 @@
 .. module:: configuration
    :synopsis: Configuration file parser
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 """
 
 import os
 import itertools
 import copy
 import logging
+from typing import Optional
 
 import yaml
+from pydantic import BaseModel, ConfigDict, ValidationError
 
 log = logging.getLogger(__name__)
 
+
 class ConfigurationError(Exception):
     """General exception class for Duct configuration issues
     """
 
+
+class DuctConfig(BaseModel):
+    """Top-level configuration schema with centralised defaults."""
+    model_config = ConfigDict(extra='allow')
+
+    # Timing
+    interval: float = 60.0
+    ttl: float = 60.0
+    stagger: float = 0.2
+
+    # Debugging
+    debug: int = 0
+
+    default_route: Optional[str] = None
+
+    # Sources and outputs
+    sources: list = []
+    outputs: Optional[list] = []
+
+    # Global SSH defaults (can be overridden per-source)
+    ssh_knownhosts_file: Optional[str] = '/var/lib/duct/known_hosts'
+    ssh_keyfile: Optional[str] = None
+    ssh_key: Optional[str] = None
+    ssh_keypass: Optional[str] = None
+    ssh_username: Optional[str] = None
+    ssh_password: Optional[str] = None
+    ssh_port: int = 22
+
+    # Daemon config
+    base_path: str = "/var/lib/duct"
+
 class ConfigFile(object):
     """Duct configuration file parser and accessor
     """
@@ -32,45 +66,18 @@ class ConfigFile(object):
         else:
             raise ConfigurationError(f"Configuration file '{path}' not found")
 
-        self.known_items = {
-            'sources': list,
-            'outputs': list,
-            'interval': (float, int,),
-            'ttl': (float, int,),
-            'blueprint': list,
-            'ssh_key': str,
-            'ssh_user': str,
-            'toolbox': dict
-        }
-
         self._parse_config()
 
-    def _validate_type(self, item, vtype):
-        if not isinstance(vtype, tuple):
-            vtype = (vtype, )
-
-        parse = False
-        for vt in vtype:
-            try:
-                assert isinstance(self.raw_config.get(item, vt()), vtype)
-                parse = True
-            except AssertionError:
-                pass
-
-        if not parse:
-            raise ConfigurationError(
-                f"{item} must be one of type {repr(vtype)}")
-
-    def _validate_config(self):
-        for key, val in self.known_items.items():
-            self._validate_type(key, val)
-
     def _parse_config(self):
         self._merge_includes()
+        self._build_blueprints()
 
-        self._validate_config()
+        try:
+            self.duct_config = DuctConfig(**self.raw_config)
+        except ValidationError as e:
+            raise ConfigurationError(str(e)) from e
 
-        self._build_blueprints()
+        self.raw_config = self.duct_config.model_dump()
 
     def _merge_includes(self):
         def both(i1, i2, t):
@@ -86,7 +93,6 @@ class ConfigFile(object):
 
         paths.extend(paths2)
 
-
         for ipath in paths:
             if os.path.exists(ipath):
                 files = [os.path.join(ipath, fi) for fi in os.listdir(ipath)
@@ -98,15 +104,12 @@ class ConfigFile(object):
                         for key, val in conf.items():
                             if key in self.raw_config:
                                 if both(val, self.raw_config[key], dict):
-                                    # Merge dicts
                                     for k2, v2 in val.items():
                                         self.raw_config[key][k2] = v2
 
                                 elif both(val, self.raw_config[key], list):
-                                    # Extend lists
                                     self.raw_config[key].extend(val)
                                 else:
-                                    # Overwrite
                                     self.raw_config[key] = val
                             else:
                                 self.raw_config[key] = val
@@ -117,24 +120,20 @@ class ConfigFile(object):
                             ipath)
 
     def _build_blueprints(self):
-        # Turn toolboxes into a dict
         toolboxes = self.raw_config.get('toolbox', {})
         blueprints = self.raw_config.get('blueprint', [])
 
         if blueprints:
-            # Make sure raw config stubs exist if we have any blueprints
             if 'sources' not in self.raw_config:
                 self.raw_config['sources'] = []
 
         for blueprint in blueprints:
             tbs = blueprint['toolbox']
-            # Listify it so we can inherit multiple toolboxes
             if not isinstance(toolboxes, list):
                 tbs = [tbs]
 
             tbs = [toolboxes[tool] for tool in tbs]
 
-            # Compute a dot product of all the config settings vectors
             inversions = []
             for key, val in blueprint.get('sets', {}).items():
                 inversions.append([(key, jay) for jay in val])
@@ -142,25 +141,19 @@ class ConfigFile(object):
             for options in itertools.product(*inversions):
                 for toolbox in tbs:
                     for source in toolbox.get('sources', []):
-                        # Make a copy of the source dict
                         mysource = copy.copy(source)
 
-                        # Merge toolbox defaults
                         for key, val in toolbox.get('defaults', {}).items():
                             mysource[key] = val
 
-                        # Merge blueprint defaults
                         for key, val in blueprint.get('defaults', {}).items():
                             mysource[key] = val
 
-                        # Merge set permutation
                         for key, val in options:
                             mysource[key] = val
 
-                        # Bolt it into our real config file
                         self.raw_config['sources'].append(mysource)
 
-        # Free a tiny bit of memory
         if 'toolbox' in self.raw_config:
             del self.raw_config['toolbox']
 
@@ -175,3 +168,6 @@ class ConfigFile(object):
 
     def __getitem__(self, item):
         return self.raw_config[item]
+
+    def __getattr__(self, name):
+        return self.duct_config.__getattribute__(name)

{ducted-2.0 → ducted-2.2}/duct/interfaces.py

@@ -2,7 +2,7 @@
 .. module:: interfaces
    :synopsis: Generic interfaces
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 """
 
 from zope.interface import Interface

{ducted-2.0 → ducted-2.2}/duct/logs/follower.py

@@ -2,7 +2,7 @@
 .. module:: follower
    :synopsis: Log tailing classes
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 """
 import os
 
@@ -14,6 +14,11 @@ class LogFollower(object):
     :type logfile: str
     :param parser: Optional parser method for log lines
     :type parser: str
+    :param tmp_path: Temporary path used to store position markers
+    :type tmp_path: str
+    :param history: If True the log is read from the start instead of tailed
+                    unless its already been read
+    :type history: str
     """
 
     def __init__(self, logfile, parser=None, tmp_path="/var/lib/duct/",

{ducted-2.0 → ducted-2.2}/duct/logs/parsers.py

@@ -2,7 +2,7 @@
 .. module:: parsers
    :synopsis: Log line parser classes
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 """
 
 import re

{ducted-2.0 → ducted-2.2}/duct/objects.py

@@ -2,7 +2,7 @@
 .. module:: objects
    :synopsis: Base classes for sources, outputs, and event objects
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 """
 
 import hashlib
@@ -108,7 +108,7 @@ class Output(object):
     async def createClient(self):
         """Coroutine that sets up the output connection"""
 
-    def eventsReceived(self, events):
+    async def eventsReceived(self, events):
         """Receives a list of events and queues them"""
         if self.maxsize > 0:
             if len(self.events) < self.maxsize:
@@ -242,7 +242,7 @@ class Source(object):
         self._loop_task = asyncio.create_task(self._run_loop())
 
     async def _run_loop(self):
-        """Async polling loop — fires tick() then repeats every inter secs."""
+        """Async polling loop - fires tick() then repeats every inter secs."""
         try:
             while True:
                 await self.tick()

{ducted-2.0 → ducted-2.2}/duct/outputs/bosun.py

@@ -2,7 +2,7 @@
 .. module:: bosun
    :synopsis: Bosun output
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 """
 import json
 import base64
@@ -14,8 +14,6 @@ from duct.utils import HTTPRequest
 class Bosun(opentsdb.OpenTSDB):
     """Bosun HTTP API output
 
-    **Configuration arguments:**
-
     :param url: URL (default: http://localhost:4242)
     :type url: str.
     :param maxsize: Maximum queue backlog size (default: 250000, 0 disables)

{ducted-2.0 → ducted-2.2}/duct/outputs/elasticsearch.py

@@ -2,7 +2,7 @@
 .. module:: elasticsearch
    :synopsis: Elasticsearch event output
 
-.. moduleauthor:: Colin Alston <colin@imcol.in>
+.. moduleauthor:: Colin Alston <colin@tamvera.com>
 """
 import asyncio
 import datetime
@@ -17,8 +17,6 @@ log = logging.getLogger(__name__)
 class ElasticSearch(Output):
     """ElasticSearch HTTP API output
 
-    **Configuration arguments:**
-
     :param url: Elasticsearch URL (default: http://localhost:9200)
     :type url: str
     :param maxsize: Maximum queue backlog size (default: 250000, 0 disables)