asyncvictoria 0.7.1__tar.gz

asyncvictoria-0.7.1/.gitignore

@@ -0,0 +1,8 @@
+/cfg.yaml
+__pycache__
+/.pybuild/
+/asyncakumuli.egg-info/
+/dist/
+/.eggs/
+/build/
+/debian/

asyncvictoria-0.7.1/.pylintrc

@@ -0,0 +1,3 @@
+[MESSAGES CONTROL]
+disable=bad-continuation,invalid-name,unnecessary-pass,protected-access,fixme,broad-except,no-absolute-import,global-statement,C,R
+

asyncvictoria-0.7.1/MIGRATION.md

@@ -0,0 +1,224 @@
+# Migrating from Akumuli to VictoriaMetrics
+
+This document describes how to migrate your data from Akumuli to VictoriaMetrics.
+
+## Prerequisites
+
+1. Both `asyncakumuli` and `asyncvictoria` packages installed:
+   ```bash
+   pip install asyncakumuli
+   pip install asyncvictoria
+   ```
+
+2. Access to your running Akumuli instance
+
+3. A running VictoriaMetrics instance with OpenTSDB protocol enabled
+
+## Starting VictoriaMetrics
+
+VictoriaMetrics needs to be started with OpenTSDB protocol support:
+
+```bash
+victoria-metrics \
+  -storageDataPath=/path/to/data \
+  -httpListenAddr=:8428 \
+  -opentsdbListenAddr=:4242
+```
+
+Where:
+- `-httpListenAddr` specifies the port for HTTP queries (default: 8428)
+- `-opentsdbListenAddr` specifies the port for OpenTSDB telnet protocol data ingestion (default: 4242)
+
+## Migration Methods
+
+### Method 1: Using the Migration Script
+
+The `migrate_from_akumuli.py` script automates the migration process.
+
+#### Option A: Auto-discover series from Akumuli database
+
+The script can automatically discover all series by reading Akumuli's SQLite database:
+
+```bash
+python migrate_from_akumuli.py \
+  --akumuli-http http://localhost:8181 \
+  --akumuli-db /path/to/akumuli/db.akumuli \
+  --victoria-tcp localhost:4242 \
+  --victoria-http http://localhost:8428 \
+  --start-time 1500000000 \
+  --end-time 1600000000
+```
+
+#### Option B: Specify series manually
+
+Alternatively, you can specify which series to migrate:
+
+```bash
+python migrate_from_akumuli.py \
+  --akumuli-http http://localhost:8181 \
+  --victoria-tcp localhost:4242 \
+  --victoria-http http://localhost:8428 \
+  --series "cpu_usage{host=server1}" \
+  --series "memory_usage{host=server1}" \
+  --start-time 1500000000 \
+  --end-time 1600000000
+```
+
+#### Options:
+- `--akumuli-http`: Akumuli HTTP endpoint (default: http://localhost:8181)
+- `--akumuli-db`: Path to Akumuli's SQLite database (db.akumuli) for auto-discovering series
+- `--victoria-tcp`: VictoriaMetrics OpenTSDB TCP endpoint (default: localhost:4242)
+- `--victoria-http`: VictoriaMetrics HTTP endpoint (default: http://localhost:8428)
+- `--series`: Series to migrate (can be specified multiple times)
+  - Format: `"series_name"` or `"series_name{tag1=val1,tag2=val2}"`
+- `--start-time`: Optional start timestamp (Unix timestamp in seconds)
+- `--end-time`: Optional end timestamp (Unix timestamp in seconds)
+
+**Note:** Either `--series` or `--akumuli-db` must be specified.
+
+### Method 2: Programmatic Migration
+
+You can also write your own migration script:
+
+```python
+import anyio
+import httpx
+from asyncakumuli import get_data as akumuli_get_data
+from asyncvictoria import connect as victoria_connect, Entry as VEntry, DS
+
+async def migrate():
+    async with httpx.AsyncClient(timeout=600) as akumuli_session:
+        async with victoria_connect(host="localhost", port=4242) as victoria:
+            # Read from Akumuli
+            data = akumuli_get_data(
+                akumuli_session,
+                series="cpu_usage",
+                tags={"host": "server1"},
+                t_start=1500000000,
+                t_end=1600000000,
+                url="http://localhost:8181/api/query"
+            )
+            
+            # Write to VictoriaMetrics
+            count = 0
+            async for entry in data:
+                ve = VEntry(
+                    series=entry.series,
+                    time=entry.time,
+                    value=entry.value,
+                    tags=entry.tags,
+                    mode=DS.gauge
+                )
+                await victoria.put(ve)
+                count += 1
+                
+                if count % 1000 == 0:
+                    await victoria.flush()
+                    print(f"Migrated {count} points...")
+            
+            await victoria.flush()
+            print(f"Migration complete: {count} points")
+
+anyio.run(migrate)
+```
+
+## Key Differences Between Akumuli and VictoriaMetrics
+
+### Data Format
+
+**Akumuli** uses a custom RESP (Redis Serialization Protocol) format:
+```
+series_name tag1=value1 tag2=value2
+timestamp
+value
+```
+
+**VictoriaMetrics** uses OpenTSDB telnet protocol format:
+```
+put series_name timestamp value tag1=value1 tag2=value2
+```
+
+### Querying
+
+**Akumuli** uses a JSON-based query API:
+```json
+{
+  "select": "cpu_usage",
+  "where": {"host": "server1"},
+  "range": {"from": "20210101T000000", "to": "20210201T000000"}
+}
+```
+
+**VictoriaMetrics** uses PromQL-compatible queries:
+```
+cpu_usage{host="server1"}
+```
+
+### API Endpoints
+
+**Akumuli:**
+- Write: TCP port (default 8282)
+- Read: HTTP `/api/query` (default port 8181)
+
+**VictoriaMetrics:**
+- Write: OpenTSDB telnet protocol (default port 4242)
+- Read: HTTP `/api/v1/query` and `/api/v1/query_range` (default port 8428)
+
+## Testing Migration
+
+A test suite is provided to verify migration correctness:
+
+```bash
+pytest tests/test_migration.py
+```
+
+This test:
+1. Starts an Akumuli instance
+2. Writes test data
+3. Starts a VictoriaMetrics instance
+4. Migrates the data
+5. Verifies all data was migrated correctly
+
+## Post-Migration
+
+After successful migration:
+
+1. Update your application code to use `asyncvictoria` instead of `asyncakumuli`
+2. Change connection parameters:
+   - Write port: 8282 → 4242 (or your configured OpenTSDB port)
+   - Read URL: `http://host:8181/api/query` → `http://host:8428/api/v1/query`
+3. Update your queries to use VictoriaMetrics query format
+4. Consider VictoriaMetrics' additional features like:
+   - Prometheus-compatible remote write
+   - Native InfluxDB line protocol support
+   - Built-in deduplication
+   - Downsampling
+
+## Troubleshooting
+
+### "Connection refused" errors
+
+Ensure VictoriaMetrics is running with OpenTSDB protocol enabled:
+```bash
+victoria-metrics -opentsdbListenAddr=:4242
+```
+
+### Data not appearing in VictoriaMetrics
+
+1. Check VictoriaMetrics logs for errors
+2. Verify the data format is correct
+3. Try querying with a wider time range
+4. Force flush data to disk: `curl -X GET http://localhost:8428/internal/force_flush`
+
+### Slow migration
+
+- Increase batch size in the migration script
+- Use multiple parallel migration processes for different series
+- Ensure sufficient network bandwidth between Akumuli and VictoriaMetrics
+
+## Support
+
+For issues with:
+- `asyncakumuli`: See https://github.com/smurfix/asyncakumuli
+- `asyncvictoria`: See https://github.com/smurfix/asyncvictoria
+- VictoriaMetrics: See https://docs.victoriametrics.com/

asyncvictoria-0.7.1/Makefile

@@ -0,0 +1,12 @@
+#!/usr/bin/make -f
+
+PACKAGE=asyncvictoria
+ifneq ($(wildcard /usr/share/sourcemgr/make/py),)
+include /usr/share/sourcemgr/make/py
+# availabe via http://github.com/smurfix/sourcemgr
+
+else
+%:
+		@echo "Please use 'python setup.py'."
+		@exit 1
+endif

asyncvictoria-0.7.1/PKG-INFO

@@ -0,0 +1,40 @@
+Metadata-Version: 2.4
+Name: asyncvictoria
+Version: 0.7.1
+Summary: Write data to VictoriaMetrics
+Author-email: Matthias Urlichs <matthias@urlichs.de>
+Project-URL: homepage, http://github.com/smurfix/asyncvictoria
+Keywords: iot,logging
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Classifier: Framework :: Trio
+Requires-Python: >=3.10
+Description-Content-Type: text/x-rst
+Requires-Dist: anyio>=4
+Requires-Dist: httpx
+Requires-Dist: pytz
+
+AsyncVictoria
+=============
+
+Welcome to `AsyncVictoria <https://github.com/M-o-a-T/asyncvictoria>`__!
+
+This library contains a small async wrapper for submission of data to,
+and (rudimentary) reading from, `VictoriaMetrics <https://victoriametrics.com>`__.
+
+Also included:
+
+* A basic unit test
+* Test scaffolding, to easily test your own code
+* a utility to read the last timestamp, as VictoriaMetrics supports efficient queries
+* helper code to interpret RRD / collectd input so that you can feed those
+  to VictoriaMetrics
+* a couple of example scripts
+* a migration tool to migrate data from Akumuli to VictoriaMetrics
+
+Tests and collectd support currently require Trio because (a) anyio doesn't
+support subprocesses, (b) UDP socket support in anyio is kindof limited.

asyncvictoria-0.7.1/README.rst

@@ -0,0 +1,20 @@
+AsyncVictoria
+=============
+
+Welcome to `AsyncVictoria <https://github.com/M-o-a-T/asyncvictoria>`__!
+
+This library contains a small async wrapper for submission of data to,
+and (rudimentary) reading from, `VictoriaMetrics <https://victoriametrics.com>`__.
+
+Also included:
+
+* A basic unit test
+* Test scaffolding, to easily test your own code
+* a utility to read the last timestamp, as VictoriaMetrics supports efficient queries
+* helper code to interpret RRD / collectd input so that you can feed those
+  to VictoriaMetrics
+* a couple of example scripts
+* a migration tool to migrate data from Akumuli to VictoriaMetrics
+
+Tests and collectd support currently require Trio because (a) anyio doesn't
+support subprocesses, (b) UDP socket support in anyio is kindof limited.

asyncvictoria-0.7.1/asyncvictoria/__init__.py

@@ -0,0 +1,16 @@
+"""asyncvictoria - Write data to VictoriaMetrics."""
+
+from __future__ import annotations
+
+from ._impl import NoCodeError as NoCodeError
+from ._impl import RespError as RespError
+from ._impl import RespUnknownError as RespUnknownError
+from ._impl import connect as connect
+from ._impl import get_data as get_data
+from ._impl import get_max_ts as get_max_ts
+from .model import DS as DS
+from .model import Entry as Entry
+from .model import EntryDelta as EntryDelta
+from .sanitize import sanitize_entry as sanitize_entry
+from .sanitize import sanitize_label_name as sanitize_label_name
+from .sanitize import sanitize_metric_name as sanitize_metric_name