influxdata-plugin-utils 0.3.0__tar.gz → 0.4.0__tar.gz

influxdata_plugin_utils-0.4.0/CHANGELOG.md

@@ -0,0 +1,188 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.4.0] - 2026-09-12
+
+### Added
+
+- `sources` module: one parser per place configuration comes from —
+  `parse_trigger_args()`, `parse_toml()`, `parse_env()`, `parse_json_body()`,
+  `parse_request_headers()` and `parse_query_parameters()`. Each reads one raw
+  input and returns a plain dict, so a plugin composes the layers it needs.
+- `sources.KeySpec` says which keys of a source become config values and under
+  what names: `allowlist`, `denylist`, `rename`, and an `unknown` policy that
+  either drops a refused key or names it back to whoever sent it. `parse_env`
+  requires an allowlist, since the process environment belongs to the host.
+- `sources.parse_toml` refuses a path that does not name a `.toml` file before
+  opening it; `require_suffix=False` lifts that for a config file named some
+  other way, and `sources.is_toml_path()` answers the same question on its own,
+  for a plugin that would rather report the path itself than raise.
+- `config.load_config(*layers, validators=...)` merges the layers in the order
+  given — lowest precedence first — and validates the result once.
+- `config.merge_config_layers(*layers, pinned=...)` merges without validating
+  and can hold chosen keys against the layers above them, so a request cannot
+  move what the operator fixed.
+- `config.Config`, the validated configuration: a dict that also answers to
+  attribute access.
+- `sources.parse_request_headers` folds only the casing of a header name, which
+  RFC 9110 makes meaningless; a name is otherwise kept as written, and `rename`
+  gives a key another name.
+- A header or query parameter the plugin asked for that arrives more than once
+  is refused rather than resolved by the order the runtime delivers them in —
+  `multi=True` reads every value as a list instead.
+- `validation` module: `Validator` and `validate()`. A rule carries a default,
+  a `cast`, and checks — 25 of them, from `gte` and `is_in` to `regex` — plus
+  `condition` for an arbitrary predicate and `when` to apply a rule only while
+  another one holds. A list, dict or set `default` is copied for each use, so
+  one rule's default cannot be changed through the values it fills in. Checks
+  are named explicitly, so a misspelled one is a `TypeError` where the rule is
+  written.
+- `introspection.get_schema(influxdb3_local, table)` returns
+  `{column_name: data_type}` from one `information_schema` query.
+- `cache.cached()` gains two parameters: `refresh` replaces a stored entry, and
+  `cache_empty=False` leaves a falsy result unstored. Combined, a refresh that
+  produces an empty value drops the entry, so a table dropped between reads
+  leaves neither its old schema cached nor an empty one that would never be
+  retried. Every introspection lookup passes `cache_empty=False`, so an empty
+  answer is retried rather than remembered; `get_schema()` also forwards
+  `refresh`, letting a caller re-read a schema on seeing an unknown column.
+
+### Changed
+
+- `config.load_plugin_config` reads the same three layers as before — the named
+  environment variables, the trigger arguments, the TOML file — and returns a
+  `Config`. A value that arrives empty is left out of its own layer, so a blank
+  trigger argument lets a validator default apply and a blank in the file no
+  longer erases the argument underneath it. Every failure is a `ValueError`,
+  including an unreadable file and a rejected value, and a `config_file_path`
+  that does not name a `.toml` file is now refused before the file is opened.
+  It stays supported, and `load_config` is the one to reach for in new plugins.
+- Configuration keys are stored as they arrive. Nothing in a layer is
+  interpreted, whatever a value spells, and a key is matched exactly as
+  written — except header names, which become lower-case config keys
+  (`X-Api-Key` -> `x-api-key`) because their spelling comes from the protocol.
+  A validator name is read the same way: dynaconf matched `Validator("Rows")`
+  to a `rows` key and walked `Validator("a.b")` into a nested dict, where here
+  `a.b` is the name of a flat key and a rule named `Rows` finds nothing while
+  the layer carries `rows`.
+
+### Removed
+
+- The `dynaconf` dependency. The package now has none. `Validator` keeps the
+  argument names for the subset the plugins use, so most rules port unchanged,
+  but these habits from dynaconf no longer hold:
+  - `required` is a plain "this key must carry a usable value", and `False` is
+    simply no rule. In dynaconf `required` was an alias for `must_exist`, so
+    both `required=False` and `must_exist=False` meant "this key must be
+    absent" and raised when it was present. `must_exist` is gone; a rule that
+    read `required=False` there says nothing here.
+  - a string `default` is stored as written. dynaconf read it as TOML, so
+    `default="5"` arrived as the number `5` and `default="5", gte=1` passed;
+    now that rule fails, and `cast=int` is how a string default becomes a
+    number.
+  - a callable `default` is stored as the callable itself. dynaconf called it
+    with `(settings, validator)` and kept what it returned.
+  - a `when` rule does not hold while its key is unset. dynaconf read an absent
+    key carrying no existence rule as passing, so
+    `Validator("ripple", required=True, when=Validator("prototype", eq="cheby1"))`
+    demanded `ripple` from a configuration that never mentioned `prototype`;
+    here the rule waits until `prototype` is set.
+  - a `when` rule judges a copy, so its own `default` and `cast` are not kept.
+    dynaconf applied them to the settings on the way past, so
+    `Validator("k", lte=10, when=Validator("k", cast=int))` compared a number
+    there and compares the string here. Put the `cast` on the rule itself.
+  - a `when` rule that cannot judge -- its own `cast` or predicate raising --
+    is reported against the rule that asked, naming both ends: `window: its
+    condition could not be checked: rows: invalid literal for int() with base
+    10: 'abc'`. dynaconf let the guard's own error out, which named the guarded
+    key and never mentioned the rule being written.
+  - every failure is a `ValueError`. dynaconf raised `ValidationError`, which is
+    not one, and let anything but a `TypeError` out of a cast, a condition or a
+    check untouched: `AttributeError` from `startswith` on a number,
+    `re.PatternError` from a bad pattern, `KeyError` from a cast.
+  - `required=True` is not satisfied by `""`, whitespace or `None`, and it is
+    read before `cast`. dynaconf cast first, so `cast=str` turned `None` into
+    the string `"None"` and the rule passed.
+  - values keep their Python types. dynaconf handed a rule containers of its
+    own -- a tuple arrived as a list, a dict as a case-insensitive mapping --
+    so `contains="A"` passed on `{"a": 1}` there and fails here.
+  - a misspelled check is a `TypeError` where the rule is written. dynaconf
+    took any unknown name as a check and raised `AttributeError` at validation
+    time, or passed in silence while the key was absent.
+  - `env`, `messages`, `description`, `items_validators`, the `|` and `&`
+    combinators, `validate_all` and `only`/`exclude` are not carried over. Every
+    rule in the list is applied, too: dynaconf's `register` dropped a rule equal
+    to an earlier one, and compared everything except `default`.
+
+## [0.3.1] - 2026-08-03
+
+### Changed
+
+- No functional changes. Released to exercise the tag-triggered release
+  automation added in
+  [#137](https://github.com/influxdata/influxdb3_plugins/pull/137), which
+  builds release notes from this file and publishes a GitHub release
+  alongside the PyPI upload.
+
+## [0.3.0] - 2026-07-31
+
+### Security
+
+- `config.load_plugin_config` — disable dynaconf's `@` token substitution
+  (`@read_file`, `@format`, `@jinja`, `@get`, and ~30 others) by constructing
+  the settings object with `AUTO_CAST_FOR_DYNACONF=False`. Previously any
+  string value beginning with `@` was evaluated, so an untrusted value from an
+  HTTP request body could read the server's files or environment variables
+  (for example `@read_file /etc/passwd` or `@format {env[SECRET]}`). Values are
+  now always treated as literal data. See
+  [#134](https://github.com/influxdata/influxdb3_plugins/issues/134).
+
+### Changed
+
+- Pin `dynaconf>=3.2,<4` so a future major release cannot silently re-enable
+  token substitution.
+
+## [0.2.0] - 2026-07-12
+
+### Added
+
+- `write.write_data` — optional `database` parameter for writing to another
+  database.
+- `introspection` — optional `database` parameter for schema helpers and
+  `query_window`.
+- `parsing.parse_timedelta` — `ms` (milliseconds) and `us` (microseconds)
+  duration units.
+
+### Changed
+
+- `write.write_data` — `no_sync` now defaults to `None`: writes go through
+  `write` / `write_to_db` (available on all InfluxDB 3 versions); passing a
+  boolean switches to `write_sync` / `write_sync_to_db` (InfluxDB 3.8+).
+
+## [0.1.0] - 2026-07-08
+
+### Added
+
+- `config` — dynaconf-backed config loading (`load_plugin_config`), plugin
+  directory resolution (`resolve_plugin_dir`, `resolve_path`), re-exported
+  `Validator`.
+- `introspection` — schema helpers (`get_table_names`, `get_tag_names`,
+  `get_field_names`) and `query_window`, with optional TTL caching.
+- `parsing` — `parse_timedelta`, `parse_timestamp_ns`, `parse_int`,
+  `parse_bool`, `parse_delimited_list`, `parse_key_value`.
+- `cache` — `cached` TTL wrapper over `influxdb3_local.cache`.
+- `write` — `build_line`, `build_line_typed`, `add_field_with_type`,
+  `write_data` (batching + retry), `BatchLines`.
+
+[Unreleased]: https://github.com/influxdata/influxdb3_plugins/compare/utils-v0.4.0...HEAD
+[0.4.0]: https://github.com/influxdata/influxdb3_plugins/compare/utils-v0.3.1...utils-v0.4.0
+[0.3.1]: https://github.com/influxdata/influxdb3_plugins/compare/utils-v0.3.0...utils-v0.3.1
+[0.3.0]: https://github.com/influxdata/influxdb3_plugins/compare/utils-v0.2.0...utils-v0.3.0
+[0.2.0]: https://github.com/influxdata/influxdb3_plugins/compare/utils-v0.1.0...utils-v0.2.0
+[0.1.0]: https://github.com/influxdata/influxdb3_plugins/releases/tag/utils-v0.1.0

influxdata_plugin_utils-0.4.0/PKG-INFO

@@ -0,0 +1,268 @@
+Metadata-Version: 2.5
+Name: influxdata-plugin-utils
+Version: 0.4.0
+Summary: Shared helpers for InfluxDB 3 plugins.
+Project-URL: Homepage, https://github.com/influxdata/influxdb3_plugins
+Project-URL: Repository, https://github.com/influxdata/influxdb3_plugins
+Author: InfluxData
+License-Expression: MIT OR Apache-2.0
+License-File: LICENSE-APACHE
+License-File: LICENSE-MIT
+Keywords: influxdb,influxdb3,plugins
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+
+# influxdata-plugin-utils
+
+Shared helpers for InfluxDB 3 plugins.
+
+## Install
+
+```bash
+pip install influxdata-plugin-utils
+```
+
+Editable, for local development:
+
+```bash
+pip install -e influxdata-plugin-utils
+```
+
+## Modules
+
+| Module          | What it provides                                                                                                                             |
+|-----------------|----------------------------------------------------------------------------------------------------------------------------------------------|
+| `sources`       | `KeySpec`, `parse_trigger_args()`, `parse_toml()`, `parse_env()`, `parse_json_body()`, `parse_request_headers()`, `parse_query_parameters()` |
+| `config`        | `load_config()`, `load_plugin_config()`, `merge_config_layers()`, `Config`, `resolve_plugin_dir()`, `resolve_path()`                         |
+| `validation`    | `Validator`, `validate()`                                                                                                                    |
+| `introspection` | `get_table_names()`, `get_tag_names()`, `get_field_names()`, `get_schema()`, `query_window()` with optional `database=`                      |
+| `parsing`       | `parse_timedelta()`, `parse_timestamp_ns()`, `parse_int()`, `parse_bool()`, `parse_delimited_list()`, `parse_key_value()`                    |
+| `cache`         | `cached(influxdb3_local, key, producer, ttl_seconds=3600, refresh=False, cache_empty=True)`                                                  |
+| `write`         | `build_line()`, `build_line_typed()`, `add_field_with_type()`, `write_data()`, `BatchLines`                                                  |
+
+The package has no dependencies, and every module raises `ValueError` on bad
+input, so a plugin answers a bad configuration from one `except` clause.
+
+## Configuration
+
+Configuration reaches a plugin from several places: the trigger arguments, a
+TOML file, environment variables, and — for `process_request` plugins — the
+request body, its headers and its query string. Each of those is a **source**
+with its own parser, and each parser returns a plain dict.
+
+`load_config` merges the dicts you give it and validates the result. The
+argument order is the precedence, lowest first.
+
+```python
+from influxdata_plugin_utils.config import load_config
+from influxdata_plugin_utils.parsing import parse_int, parse_timedelta
+from influxdata_plugin_utils.sources import (
+    KeySpec,
+    parse_env,
+    parse_json_body,
+    parse_query_parameters,
+    parse_request_headers,
+    parse_toml,
+    parse_trigger_args,
+)
+from influxdata_plugin_utils.validation import Validator
+
+BODY = KeySpec(allowlist=["measurement", "field", "window"], unknown="reject")
+QUERY = KeySpec(allowlist=["window"])
+HEADERS = KeySpec(allowlist=["x-api-key"], rename={"x-api-key": "api_key"})
+ENV = KeySpec(allowlist=["PLUGIN_API_KEY"], rename={"PLUGIN_API_KEY": "api_key"})
+
+VALIDATORS = [
+    Validator("measurement", required=True),
+    Validator("api_key", required=True),
+    Validator("window", default="1h", cast=parse_timedelta),
+    Validator("limit", default=1000, cast=parse_int, gte=1, lte=10_000),
+]
+
+def process_request(
+    influxdb3_local, query_parameters, request_headers, request_body, args=None
+):
+    cfg = load_config(
+        parse_env(ENV),
+        parse_trigger_args(args),
+        parse_toml(args.get("config_file_path") if args else None),
+        parse_json_body(request_body, BODY),
+        parse_request_headers(request_headers, HEADERS),
+        parse_query_parameters(query_parameters, QUERY),
+        validators=VALIDATORS,
+    )
+    influxdb3_local.info(f"{cfg.measurement} window={cfg['window']}")
+```
+
+`load_plugin_config` covers three layers in a fixed order — the named
+environment variables, the trigger arguments, and the file at
+`config_file_path`:
+
+```python
+from influxdata_plugin_utils.config import load_plugin_config
+
+def process_scheduled_call(influxdb3_local, call_time, args):
+    cfg = load_plugin_config(args, validators=VALIDATORS, env_keys=["PLUGIN_API_KEY"])
+```
+
+Pass `source="args"` or `source="toml"` to use only one of the last two. Prefer
+`load_config` in new plugins: there the layers are ordinary arguments, so a
+plugin adds, reorders or drops any of them.
+
+### What a source contributes
+
+A `KeySpec` says which keys of a source become config values and under what
+names:
+
+```python
+KeySpec(allowlist=["measurement"], rename={"measurement": "table"}, unknown="reject")
+```
+
+- `allowlist` names the keys that pass, `denylist` the ones that do not;
+- `rename` maps a source key onto the config key it becomes;
+- `unknown` decides what happens to a refused key — `"ignore"` drops it,
+  `"reject"` names it in the error so the sender learns what was wrong.
+
+On a layer the caller controls, prefer `allowlist`: a parameter added to the
+plugin later stays unreachable until it is listed, where a `denylist` would let
+it through unnoticed.
+
+Header names are matched regardless of casing — RFC 9110 makes it meaningless —
+and become config keys spelled in lower case (`X-Api-Key` → `x-api-key`); use
+`rename` for a name of your own. Everywhere else names are matched and kept
+exactly as written.
+
+Within one spec, spell a `rename` key the way the `allowlist` spells it. A
+`KeySpec` does not know which source will read it, so it checks the two against
+each other as written, and `allowlist=["X-Api-Key"]` with
+`rename={"x-api-key": "api_key"}` is refused at construction. Either spelling
+works as long as both use it.
+
+A header or query parameter the plugin asked for that arrives more than once is
+refused, since which value it would otherwise get is the order the runtime
+delivers them in; `multi=True` reads every value as a list instead. InfluxDB 3
+hands the plugin a plain dict, which holds one value per name, so a repeat never
+reaches a plugin there and neither the refusal nor `multi` fires; both are for a
+runtime that delivers name/value pairs.
+
+`parse_env` requires an allowlist: the process environment belongs to the host
+and holds credentials, so nothing is read without being named. `Authorization`
+never reaches a plugin — the engine authenticates with it — so a token needs a
+header of your own.
+
+`parse_toml` refuses a path that does not name a `.toml` file before opening it;
+pass `require_suffix=False` for a config file named some other way, and
+`is_toml_path()` answers the same question without reading anything. Otherwise
+it reads whatever path it is given: a relative one resolves under the plugin
+directory, an absolute one is used as is. Take that path from a layer the
+operator controls — the trigger arguments, or the file itself. A path that
+arrives in the request body, a header or the query string lets the caller name
+any file the engine can read, and whatever parses as TOML becomes this plugin's
+configuration, another plugin's credentials included.
+
+A value that arrives empty — a blank string, a JSON `null`, an unset variable —
+is left out of its layer, so a validator default applies instead and a blank in
+one layer does not erase the layer below it. `0`, `False` and `[]` are real
+values and are kept.
+
+Headers, query-string parameters and environment variables arrive as text, so
+surrounding whitespace is trimmed before anything else sees the value: a header
+sent as `  secret  ` becomes `secret`. A trigger argument, a body field and a
+TOML value keep exactly what was written, whitespace included.
+
+### Holding a key against the request
+
+`merge_config_layers` merges without validating, and can hold chosen keys
+against the layers above them:
+
+```python
+merged = merge_config_layers(args, body, pinned=["measurement"])
+```
+
+A later layer that sets a pinned key raises; `on_conflict="ignore"` keeps the
+value already set instead. A pinned key nobody set stays open, so the same
+plugin works with or without a fixed measurement.
+
+### Validating
+
+A `Validator` describes one config key: the default it falls back to, the cast
+that turns it into a usable type, and the checks it must then pass.
+
+```python
+Validator("window", default="1h", cast=parse_timedelta, gt=timedelta(0), lte=timedelta(days=30))
+Validator("aggregate", default="mean", is_in=("mean", "min", "max", "count"))
+Validator("ripple", required=True, when=Validator("prototype", eq="cheby1"))
+```
+
+`required` asks for a usable value, so a key that arrives blank or `null`
+counts as unset. `when` applies a rule only while another one holds — and holds
+means the key is there and passes. A `when` rule that cannot judge at all, say
+a `cast` of its own that fails, gives no answer, and that is reported against
+the rule which asked rather than quietly leaving it out. `condition` takes any
+predicate and runs after the checks. The checks are `eq`, `ne`, `gt`, `gte`,
+`ge`, `lt`, `lte`, `le`, `identity`, `is_type_of`, `is_in`, `is_not_in`,
+`contains`, `cont`, `not_contains`, `len_eq`, `len_ne`, `len_min`, `len_max`,
+`startswith`, `endswith`, `not_startswith`, `not_endswith`, `regex` and
+`not_regex`. They are named explicitly, so a misspelled one is a `TypeError`
+where the rule is written.
+
+`regex` and `not_regex` match from the start of the value, so `regex="b"`
+rejects `"abc"`, and a pattern that may appear anywhere needs `.*` in front.
+`is_type_of` reads a parameterized generic through to the items, so
+`list[int]`, `dict[str, int]` and `tuple[int, ...]` say what they look like.
+
+Validation runs once, over the merged values: defaults fill what no layer set,
+`cast` runs next, and the checks see the cast value.
+
+TOML becomes native — no manual string parsing:
+
+```toml
+source_table = "cpu"
+batch_size = 2000
+excluded_fields = ["usage_idle", "usage_guest"]
+```
+
+## Write helpers
+
+`LineBuilder` is a runtime global injected into the plugin, so builders take the
+class as their first argument:
+
+```python
+from influxdata_plugin_utils.write import build_line, write_data
+
+lines = [
+    build_line(LineBuilder, "cpu", tags={"host": "a"}, fields={"usage": 12.5}, time_ns=ts)
+]
+write_data(influxdb3_local, lines)            # batched + retried by default
+# write_data(influxdb3_local, lines, batch=False, retries=0)  # opt out
+# write_data(influxdb3_local, lines, database="other_db")     # another database
+# write_data(influxdb3_local, lines, no_sync=True)            # write_sync API (3.8+)
+```
+
+## Cross-database queries
+
+On InfluxDB versions that support processing-engine cross-database queries,
+the introspection helpers accept `database=` and pass it through to
+`influxdb3_local.query`.
+Cached schema results are separated per database.
+
+```python
+from influxdata_plugin_utils.introspection import get_field_names, query_window
+
+fields = get_field_names(influxdb3_local, "cpu", database="source_db")
+rows = query_window(
+    influxdb3_local,
+    "cpu",
+    start=start,
+    end=end,
+    columns=fields,
+    database="source_db",
+)
+```
+
+## License
+
+Licensed under either of [Apache License 2.0](LICENSE-APACHE) or
+[MIT license](LICENSE-MIT) at your option.