datafun-streaming 0.1.0__tar.gz

datafun_streaming-0.1.0/.gitignore

@@ -0,0 +1,110 @@
+# ============================================================
+# .gitignore (Keep unnecessary files out of the repository)
+# ============================================================
+
+# REQ.UNIVERSAL: All professional GitHub project repositories MUST include .gitignore.
+# WHY: Prevent committing generated artifacts, local state, secrets, and OS-specific files.
+# ALT: Repository may customize ignores, but MUST preserve universal safety rules.
+# CUSTOM: Logs may be temporarily committed for verification; keep ignored for
+# production use and security.
+
+
+# === Universal (all projects, all languages) ===
+
+# WHY: Logs are useful during debugging and verification.
+# ALT: Comment if logs must be inspected or validated.
+*.log
+logs/
+PRIVATE_NOTES.md
+PRIVATE-NOTES.md
+
+# WHY: Temporary and swap files are machine-local noise and create meaningless diffs.
+*.swo
+*.swp
+*.tmp
+*~
+
+# === VS Code (special case) ===
+
+# WHY: Ignore editor state while allowing a shared baseline configuration.
+.vscode/
+
+# WHY: Commit recommended extensions (opt-in) for consistent development experience.
+# NOTE: Share recommendations, not personal editor styles or preferences.
+!.vscode/ # So it will explore this folder for the following files.
+!.vscode/extensions.json
+!.vscode/settings.json
+
+# === OS-specific files (macOS / Windows) ===
+
+# WHY: OS-generated metadata files should never be tracked.
+.AppleDouble
+.DS_Store
+.LSOverride
+Icon\r
+._*
+.Spotlight-V100/
+.Trashes
+desktop.ini
+ehthumbs.db
+Thumbs.db
+
+# === Editors / IDEs (non-VS Code) ===
+
+# WHY: IDE metadata is machine-local and should not be tracked.
+*.code-workspace
+.idea/
+
+# === Environment variables and secrets ===
+
+# WHY: Never commit credentials or environment-specific configuration.
+.env
+.env.*
+*.env
+
+# === Documentation build output ===
+
+# WHY: Static site build output is generated.
+site/
+
+# === Generic caches ===
+
+# WHY: Generic caches are machine-local and should not be tracked.
+.cache/
+
+# === Python ===
+
+# WHY: Virtual environments are machine-local and reproducible.
+.venv/
+venv/
+
+# REQ.PYTHON: Do NOT git ignore uv.lock. Commit it and use it in CI/CD pipelines.
+
+# WHY: Python version when using scm matches any repo depth and any package name
+**/src/**/_version.py
+
+# WHY: Python bytecode is generated.
+__pycache__/
+*.pyc
+*.pyd
+*.pyo
+
+# WHY: Build and packaging artifacts are generated.
+.eggs/
+build/
+dist/
+*.egg
+*.egg-info/
+*.whl
+
+# WHY: Tooling caches should not be tracked.
+.coverage
+.coverage.*
+.mypy_cache/
+.pytest_cache/
+.pytype/
+.ruff_cache/
+.tox/
+
+# WHY: Notebook checkpoint state is generated.
+.ipynb_checkpoints/

datafun_streaming-0.1.0/CHANGELOG.md

@@ -0,0 +1,107 @@
+# Changelog
+
+<!-- markdownlint-disable MD024 -->
+
+All notable changes to this project will be documented in this file.
+
+The format is based on **[Keep a Changelog](https://keepachangelog.com/en/1.1.0/)**
+and this project adheres to **[Semantic Versioning](https://semver.org/spec/v2.0.0.html)**.
+
+---
+
+## [Unreleased]
+
+---
+
+## [0.1.0] - 2026-05-08
+
+### Added
+
+- `datafun_streaming.core` - shared type aliases
+  (`DataRecordDict`, `DataRecordDictList`) used across all subpackages
+- `datafun_streaming.data_validation` - field-level validators
+  (`validate_required_fields`, `validate_boolean_text`, `validate_datetime`,
+  `validate_positive_integer`), `ValidationResult` dataclass, `make_lookup_set`,
+  `validate_reference_records`, and `add_validation_errors`
+- `datafun_streaming.io` - CSV and JSON file I/O utilities (`read_csv_rows`, `read_csv_as_lookup`,
+  `append_csv_row`, `row_to_json`, `row_from_json`, `format_message_for_log`)
+- `datafun_streaming.kafka` - Kafka producer, consumer, admin, and connection utilities built on
+  `confluent-kafka`; `KafkaSettings` frozen dataclass with `.from_env()` loader
+- `datafun_streaming.stats` - `RunningStats` dataclass for incremental count,
+  total, mean, minimum, and maximum without storing message history
+- `datafun_streaming.storage` - DuckDB utilities with schema inference,
+  allowlist-validated `upsert_row`, parameterized `query_db`, and `safe_table_name` injection guard
+- `datafun_streaming.visualization` - `StreamingChart` dataclass and Plotly helpers
+  (`init_chart`, `update_chart`, `save_chart`) for live data visualization
+- Full test suite covering all pure-Python subpackages
+  (validation, IO, stats, storage, Kafka settings, Kafka error messages)
+- GitHub Actions CI workflow (pre-commit, pyright, pytest, zensical docs build)
+- GitHub Actions release workflow (PyPI Trusted Publishing via OIDC, GitHub Pages docs deploy)
+- `pyproject.toml` with hatchling + hatch-vcs, ruff strict config, pyright basic config,
+  and pytest coverage
+
+---
+
+## Notes on versioning and releases
+
+- We use **SemVer**:
+  - **MAJOR** - breaking changes
+  - **MINOR** - backward-compatible additions
+  - **PATCH** - fixes, documentation, tooling
+- Versions are driven by git tags. Tag `vX.Y.Z` to release.
+
+---
+
+## Release Procedure (Required)
+
+Follow these steps exactly when creating a new release.
+
+### Task 1. Update release metadata (manual edits)
+
+1.1. `CITATION.cff` - update `version` and `date-released`
+1.2. CHANGELOG.md: add section, move unreleased entries, update links
+1.3. pyproject.toml - update [tool.hatch.version].fallback-version (near the end)
+
+### Task 2. Sync Version and Validate
+
+```shell
+uv sync --extra dev --extra docs --upgrade
+git add -A
+uvx pre-commit run --all-files
+uv run python -m pyright
+uv run python -m pytest
+uv run python -m zensical build
+uvx validate-pyproject[all] pyproject.toml
+
+uv build
+uv run python -m twine check dist/\*
+```
+
+### Task 3. Commit, tag, push
+
+```shell
+git add -A
+git commit -m "Prep X.Y.Z"
+git push -u origin main
+```
+
+Verify actions run on GitHub. After success:
+
+```shell
+git tag vX.Y.Z -m "X.Y.Z"
+git push origin vX.Y.Z
+```
+
+## Only As Needed (delete a tag)
+
+```shell
+git tag -d vX.Z.Y
+git push origin :refs/tags/vX.Z.Y
+```
+
+## Links
+
+[Unreleased]: https://github.com/denisecase/datafun-streaming/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/denisecase/datafun-streaming/releases/tag/v0.1.0
+
+<!-- markdownlint-enable MD024 -->

datafun_streaming-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Denise Case
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datafun_streaming-0.1.0/PKG-INFO

@@ -0,0 +1,168 @@
+Metadata-Version: 2.4
+Name: datafun-streaming
+Version: 0.1.0
+Summary: Utilities for streaming data analytics with Kafka and DuckDB.
+Project-URL: Homepage, https://github.com/denisecase/datafun-streaming
+Project-URL: Repository, https://github.com/denisecase/datafun-streaming
+Project-URL: Documentation, https://denisecase.github.io/datafun-streaming/
+Project-URL: Issues, https://github.com/denisecase/datafun-streaming/issues
+Project-URL: Changelog, https://github.com/denisecase/datafun-streaming/blob/main/CHANGELOG.md
+Author-email: Denise Case <dcase@nwmissouri.edu>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: confluent-kafka,data-analytics,data-validation,running-statistics,streaming-data
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.14
+Requires-Dist: confluent-kafka
+Requires-Dist: datafun-toolkit
+Requires-Dist: duckdb
+Requires-Dist: plotly
+Requires-Dist: python-dotenv
+Provides-Extra: dev
+Requires-Dist: pre-commit; extra == 'dev'
+Requires-Dist: pyright; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocstrings[python]; extra == 'docs'
+Requires-Dist: zensical; extra == 'docs'
+Description-Content-Type: text/markdown
+
+# datafun-streaming
+
+[![Adaptive Interfaces](https://img.shields.io/badge/adaptive--interfaces-schema-blue?logo=github)](https://github.com/denisecase)
+[![PyPI](https://img.shields.io/pypi/v/datafun-streaming?logo=pypi&label=pypi)](https://pypi.org/project/datafun-streaming/)
+[![Docs Site](https://img.shields.io/badge/docs-site-blue?logo=github)](https://denisecase.github.io/datafun-streaming/)
+[![Repo](https://img.shields.io/badge/repo-GitHub-black?logo=github)](https://github.com/denisecase/datafun-streaming)
+[![Python 3.15+](https://img.shields.io/badge/python-3.15%2B-blue?logo=python)](https://github.com/denisecase/datafun-streaming/blob/main/pyproject.toml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+[![CI](https://github.com/denisecase/datafun-streaming/actions/workflows/ci-python-zensical.yml/badge.svg?branch=main)](https://github.com/denisecase/datafun-streaming/actions/workflows/ci-python-zensical.yml)
+[![Docs](https://github.com/denisecase/datafun-streaming/actions/workflows/deploy-zensical.yml/badge.svg?branch=main)](https://github.com/denisecase/datafun-streaming/actions/workflows/deploy-zensical.yml)
+[![Links](https://github.com/denisecase/datafun-streaming/actions/workflows/links.yml/badge.svg?branch=main)](https://github.com/denisecase/datafun-streaming/actions/workflows/links.yml)
+
+> Shared Python utilities for Kafka, DuckDB, validation, stats, and visualization
+> across streaming data analytics projects.
+
+## Command Reference
+
+<details>
+<summary>Show command reference</summary>
+
+### In a machine terminal
+
+Open a machine terminal where you want the project:
+
+```shell
+git clone https://github.com/denisecase/datafun-streaming
+
+cd datafun-streaming
+code .
+```
+
+### In a VS Code terminal
+
+```shell
+# reset uv cache only after suspected cache corruption or strange dependency errors
+# uv cache clean
+
+uv self update
+uv python pin 3.14
+uv sync --extra dev --extra docs --upgrade
+
+uvx pre-commit install
+
+git add -A
+uvx pre-commit run --all-files
+# repeat if changes were made
+git add -A
+uvx pre-commit run --all-files
+
+# do chores
+uv run python -m ruff format .
+uv run python -m ruff check . --fix
+uv run python -m pyright
+uv run python -m pytest
+uv run python -m zensical build
+
+# save progress
+git add -A
+git commit -m "update"
+git push -u origin main
+```
+
+</details>
+
+## Notes
+
+- Use the **UP ARROW** and **DOWN ARROW** in the terminal to scroll through past commands.
+- Use `CTRL+f` to find (and replace) text within a file.
+- You do not need to add to or modify `tests/`. They are provided for example only.
+- Many files are silent helpers. Explore as you like, but nothing is required.
+- You do NOT not to understand everything; understanding builds naturally over time.
+
+## Troubleshooting >>> or
+
+If you see something like this in your terminal: `>>>` or `...`
+You accidentally started Python interactive mode.
+It happens.
+Press `Ctrl+c` (both keys together) or `Ctrl+Z` then `Enter` on Windows.
+
+## Example Output
+
+```shell
+| INFO | P01 | ========================
+| INFO | P01 | START main()
+| INFO | P01 | ========================
+| INFO | P01 | ROOT_DIR = .
+| INFO | P01 | DATA_DIR = data
+| INFO | P01 | OUTPUT_CSV = data\sales.csv
+| INFO | P01 | Streaming 3 sales to C:\Repos\streaming\datafun-streaming\data\sales.csv ...
+| INFO | P01 | Watch each sale arrive. Press CTRL+C to stop early.
+
+| INFO | P01 | (1, 81.87, 'Backpack', 'East')
+| INFO | P01 | Generated formatted multi-line SUMMARY string.
+| INFO | P01 | Returning the str to the calling function.
+| INFO | P01 |
+    Descriptive Statistics for Streaming Sales Amounts ($):
+        Count of sales   : 1
+        Minimum sale     : $81.87
+        Maximum sale     : $81.87
+        Average sale     : $81.87
+        Standard deviation: $0.00
+
+| INFO | P01 | (2, 101.58, 'Water Bottle', 'North')
+| INFO | P01 | Generated formatted multi-line SUMMARY string.
+| INFO | P01 | Returning the str to the calling function.
+| INFO | P01 |
+    Descriptive Statistics for Streaming Sales Amounts ($):
+        Count of sales   : 2
+        Minimum sale     : $81.87
+        Maximum sale     : $101.58
+        Average sale     : $91.72
+        Standard deviation: $13.94
+
+| INFO | P01 | (3, 27.15, 'Running Shoes', 'East')
+| INFO | P01 | Generated formatted multi-line SUMMARY string.
+| INFO | P01 | Returning the str to the calling function.
+| INFO | P01 |
+    Descriptive Statistics for Streaming Sales Amounts ($):
+        Count of sales   : 3
+        Minimum sale     : $27.15
+        Maximum sale     : $101.58
+        Average sale     : $70.20
+        Standard deviation: $38.56
+
+| INFO | P01 | ========================
+| INFO | P01 | Producer executed successfully!
+| INFO | P01 | ========================
+```

datafun_streaming-0.1.0/README.md

@@ -0,0 +1,129 @@
+# datafun-streaming
+
+[![Adaptive Interfaces](https://img.shields.io/badge/adaptive--interfaces-schema-blue?logo=github)](https://github.com/denisecase)
+[![PyPI](https://img.shields.io/pypi/v/datafun-streaming?logo=pypi&label=pypi)](https://pypi.org/project/datafun-streaming/)
+[![Docs Site](https://img.shields.io/badge/docs-site-blue?logo=github)](https://denisecase.github.io/datafun-streaming/)
+[![Repo](https://img.shields.io/badge/repo-GitHub-black?logo=github)](https://github.com/denisecase/datafun-streaming)
+[![Python 3.15+](https://img.shields.io/badge/python-3.15%2B-blue?logo=python)](https://github.com/denisecase/datafun-streaming/blob/main/pyproject.toml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+[![CI](https://github.com/denisecase/datafun-streaming/actions/workflows/ci-python-zensical.yml/badge.svg?branch=main)](https://github.com/denisecase/datafun-streaming/actions/workflows/ci-python-zensical.yml)
+[![Docs](https://github.com/denisecase/datafun-streaming/actions/workflows/deploy-zensical.yml/badge.svg?branch=main)](https://github.com/denisecase/datafun-streaming/actions/workflows/deploy-zensical.yml)
+[![Links](https://github.com/denisecase/datafun-streaming/actions/workflows/links.yml/badge.svg?branch=main)](https://github.com/denisecase/datafun-streaming/actions/workflows/links.yml)
+
+> Shared Python utilities for Kafka, DuckDB, validation, stats, and visualization
+> across streaming data analytics projects.
+
+## Command Reference
+
+<details>
+<summary>Show command reference</summary>
+
+### In a machine terminal
+
+Open a machine terminal where you want the project:
+
+```shell
+git clone https://github.com/denisecase/datafun-streaming
+
+cd datafun-streaming
+code .
+```
+
+### In a VS Code terminal
+
+```shell
+# reset uv cache only after suspected cache corruption or strange dependency errors
+# uv cache clean
+
+uv self update
+uv python pin 3.14
+uv sync --extra dev --extra docs --upgrade
+
+uvx pre-commit install
+
+git add -A
+uvx pre-commit run --all-files
+# repeat if changes were made
+git add -A
+uvx pre-commit run --all-files
+
+# do chores
+uv run python -m ruff format .
+uv run python -m ruff check . --fix
+uv run python -m pyright
+uv run python -m pytest
+uv run python -m zensical build
+
+# save progress
+git add -A
+git commit -m "update"
+git push -u origin main
+```
+
+</details>
+
+## Notes
+
+- Use the **UP ARROW** and **DOWN ARROW** in the terminal to scroll through past commands.
+- Use `CTRL+f` to find (and replace) text within a file.
+- You do not need to add to or modify `tests/`. They are provided for example only.
+- Many files are silent helpers. Explore as you like, but nothing is required.
+- You do NOT not to understand everything; understanding builds naturally over time.
+
+## Troubleshooting >>> or
+
+If you see something like this in your terminal: `>>>` or `...`
+You accidentally started Python interactive mode.
+It happens.
+Press `Ctrl+c` (both keys together) or `Ctrl+Z` then `Enter` on Windows.
+
+## Example Output
+
+```shell
+| INFO | P01 | ========================
+| INFO | P01 | START main()
+| INFO | P01 | ========================
+| INFO | P01 | ROOT_DIR = .
+| INFO | P01 | DATA_DIR = data
+| INFO | P01 | OUTPUT_CSV = data\sales.csv
+| INFO | P01 | Streaming 3 sales to C:\Repos\streaming\datafun-streaming\data\sales.csv ...
+| INFO | P01 | Watch each sale arrive. Press CTRL+C to stop early.
+
+| INFO | P01 | (1, 81.87, 'Backpack', 'East')
+| INFO | P01 | Generated formatted multi-line SUMMARY string.
+| INFO | P01 | Returning the str to the calling function.
+| INFO | P01 |
+    Descriptive Statistics for Streaming Sales Amounts ($):
+        Count of sales   : 1
+        Minimum sale     : $81.87
+        Maximum sale     : $81.87
+        Average sale     : $81.87
+        Standard deviation: $0.00
+
+| INFO | P01 | (2, 101.58, 'Water Bottle', 'North')
+| INFO | P01 | Generated formatted multi-line SUMMARY string.
+| INFO | P01 | Returning the str to the calling function.
+| INFO | P01 |
+    Descriptive Statistics for Streaming Sales Amounts ($):
+        Count of sales   : 2
+        Minimum sale     : $81.87
+        Maximum sale     : $101.58
+        Average sale     : $91.72
+        Standard deviation: $13.94
+
+| INFO | P01 | (3, 27.15, 'Running Shoes', 'East')
+| INFO | P01 | Generated formatted multi-line SUMMARY string.
+| INFO | P01 | Returning the str to the calling function.
+| INFO | P01 |
+    Descriptive Statistics for Streaming Sales Amounts ($):
+        Count of sales   : 3
+        Minimum sale     : $27.15
+        Maximum sale     : $101.58
+        Average sale     : $70.20
+        Standard deviation: $38.56
+
+| INFO | P01 | ========================
+| INFO | P01 | Producer executed successfully!
+| INFO | P01 | ========================
+```