pipeline-eds 0.2.4__py3-none-any.whl

pipeline_eds-0.2.4.dist-info/METADATA

@@ -0,0 +1,238 @@
+Metadata-Version: 2.3
+Name: pipeline-eds
+Version: 0.2.4
+Summary: The official API pipeline library for mulch-based projects. Key target: Emerson Ovation EDS REST API.
+License: BSD-3
+Author: George Clayton Bennett
+Author-email: george.bennett@memphistn.gov
+Requires-Python: >=3.11,<4.0.0
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: certifi (>=2025.1.31,<2026.0.0)
+Requires-Dist: fastapi (>=0.115.12,<0.116.0)
+Requires-Dist: mulch (>=0.2.8,<0.3.0)
+Requires-Dist: mysql-connector-python (>=9.3.0,<10.0.0)
+Requires-Dist: pendulum (>=3.1.0,<4.0.0)
+Requires-Dist: plotly (>=6.2.0,<7.0.0)
+Requires-Dist: pyodbc (>=5.2.0,<6.0.0)
+Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
+Requires-Dist: requests (>=2.32.3,<3.0.0)
+Requires-Dist: schedule (>=1.2.2,<2.0.0)
+Requires-Dist: toml (>=0.10.2,<0.11.0)
+Requires-Dist: tzdata (>=2025.2,<2026.0)
+Requires-Dist: urllib3 (>=2.4.0,<3.0.0)
+Requires-Dist: uvicorn (>=0.34.3,<0.35.0)
+Project-URL: Homepage, https://github.com/city-of-memphis-wastewater/pipeline
+Project-URL: Repository, https://github.com/city-of-memphis-wastewater/pipeline
+Description-Content-Type: text/markdown
+
+# pipeline
+The primary purpose of this project is to ease API access to Enterprise Data Server (EDS) machines set up by Emerson to compliment an Emerson Ovation local system. 
+Use-cases include data exchange with third-party contractors and also data access for in-house employees on work and personal devices.
+
+*Scroll down to ***Rollout, setup, etc.*** to see information about dependencies, Poetry, and pyenv.*
+
+## How to run pipeline
+
+Check that the secrets.yaml file in the default-workspace is loading:
+```
+poetry run python -m src.pipeline.env
+```
+Recieve an export file of all the points known to your EDS:
+```
+poetry run python -m src.pipeline.api.eds demo-point-export
+```
+Run the existing eds_to_rjn workspace:
+```
+poetry run python -m workspaces.eds_to_rjn.scripts.daemon_runner test
+```
+Check connectivity:
+```
+poetry run python -m src.pipeline.api.eds ping
+poetry run python -m src.pipeline.api.rjn ping
+```
+Other commands:
+```
+.\main.bat # put the daemon into service: ill advised.
+.\main.ps1
+```
+
+## Implementation
+The current ideal implementation of `pipeline` involves `Windows Task Scheduler`.
+This is installed on an `EDS` server, to pull data and send it to a third party.
+The Task is set up to call `main_eds_to_rjn_quiet.ps1` as the entry point.
+The iterative hourly timing handled is by `Windows Task Scheduler` rather than pythonically with the (no unused) `setup_schedules()` function.
+The `main` function from the `daemon_runner` file is run; this used to call the `run_hourly_tabular_trend_eds_to_rjn()` function.
+Environment managemenet is handled with `venv` rather than `pyenv` and `poetry`, because `Task Scheduler` directs the process to a different user.
+
+## secrets.yaml
+Access will not work without a secrets.yaml file in /pipeline/workspaces/your-workspace-name/config/secrets.yaml
+
+*API keys are specific to each the workspace, and can be referenced in the workspace scripts.*
+You would edit the secrets.yaml file to specify your own EDS server and credentials.
+Important: You need to VPN onto the same network as your server, EDS or otherwise, if it is not available to the outside world.
+### Example secrets.yaml: 
+```
+eds_apis:
+  MyServer1:
+    url: "http://127.0.0.1:43084/api/v1/"
+    username: "admin"
+    password: ""
+  MyServer2:
+    url: "http://some-ip-address:port/api/v1/"
+    username: "admin"
+    password: ""
+
+contractor_apis:
+  MySpecialContractor:
+    url: "https://contractor-api.com/v1/special/"
+    client_id: "special-user"
+    password: "2685steam"
+```
+The ***workspaces*** folder is designed to accomodate any custom workspaces or projects that you might add. You can alter the default projet file by altering the directory name indicated in the /pipeline/workspaces/default-workspace.toml file.
+
+# Future goals:
+- Submit a pipeline library to PyPi.
+- Generate a cookiecutter template for building new project directories.
+- Use mulchcli (another ongoing repo) to guide users through setting up secrets.yaml files step-by-step.
+
+# Maintenance Notes:
+- Implement session = requests.Session(), like an adult.
+- daemon_runner.py should use both Maxson and Stiles access. 
+
+# Rollout, setup, etc.
+It is recommended that you use **pyenv** for setting the Python version and generating a virtual environment, though this is optional. 
+
+To benefit from the pyproject.toml rollout for this project, **Poetry** is entirely necessary for installing requirements.
+
+### Why pyenv?
+venv and virtualenv are confusing, and pyenv is not confusing. With pyenv, it is easy to run many projects at once, with different requirements for each. 
+You only need to install pyenv once on you system, and then you use it to access different versions of Python.
+
+#### Note to anyone who doesn't yet understand the glory of virtual environments in Python: 
+Do it. You do not want to install special requirements to your system version of Python.
+
+### Why Poetry?
+**Poetry** is my favorite dependency management tool. It is very easy. You only need to install it once. **Poetry** also has the benefit of generating a directory-specific virtual environment.
+
+## Use pyenv to set your Python version 
+(3.11.9 or other 3.11.xx).
+### Install pyenv (skip if pyenv is already installed on your system)
+How to install pyenv-win (https://github.com/pyenv-win/pyenv-win)
+```
+Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned
+Invoke-WebRequest -UseBasicParsing -Uri "https://raw.githubusercontent.com/pyenv-win/pyenv-win/master/pyenv-win/install-pyenv-win.ps1" -OutFile "./install-pyenv-win.ps1"; &"./install-pyenv-win.ps1"
+```
+It is worth it to make the pyenv command persistent in PowerShell, by editing the $profile file:
+```
+notepad $profile
+```
+to include something like:
+```
+$env:PYENV = "$HOME\.pyenv\pyenv-win"
+$env:PYENV_ROOT = "$HOME\.pyenv\pyenv-win"
+$env:PYENV_HOME = "$HOME\.pyenv\pyenv-win"
+$env:Path += ";$env:PYENV\bin;$env:PYENV\shims"
+	
+# Initialize pyenv
+#$pyenvInit = & $env:PYENV_HOME\bin\pyenv init --path
+#Invoke-Expression $pyenvInit
+
+# Manually set up the pyenv environment
+function Invoke-PyenvInit {
+    # Update PATH to include pyenv directories
+    $pyenvShims = "$env:PYENV\shims"
+    $pyenvBin = "$env:PYENV\bin"
+    $env:PATH = "$pyenvBin;$pyenvShims;$env:PATH"
+}
+
+# Initialize pyenv
+Invoke-PyenvInit
+```
+How to install pyenv on linux (https://github.com/pyenv/pyenv)
+```
+curl -fsSL https://pyenv.run | bash
+```
+To make the pyenv command persistent in the Bourne Again SHell, edit ~/.bashrc 
+```
+nano ~/.bashrc
+```
+to include something like:
+```
+export Path="$HOME/.local/bin:$PATH"
+export PYENV_ROOT="$HOME/.pyenv"
+[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
+eval "$(pyenv init - bash)"
+```
+### Post-install, leverage the benefits of pyenv
+Install Python 3.11.9 using pyenv:
+```
+# pyenv install --list # See all Python versions able with pyenv.
+pyenv install 3.11.9
+# pyenv local 3.11.9 # to set your current directory version
+# pyenv global 3.11.9 # to set the assumed version in any directory
+# pyenv global system # revert to your system installation as the assumed version
+```
+## Use Poetry to run deploy the requirements for this project 
+How to install poetry (https://github.com/python-poetry/poetry)
+```
+Remove-Item Alias:curl # Solution to a common PowerShell issue 
+curl -sSL https://install.python-poetry.org | python3
+# Alternatively: 
+// pip install poetry
+# Or, even:
+// pyenv exec pip install poetry
+```
+## Git clone pipeline, open source
+```
+git clone https://github.com/City-of-Memphis-Wastewater/pipeline.git
+cd pipline
+pyenv local 3.11.9 # to set your current directory version
+```
+Explicitly set poetry to use the local pyenv version.
+```
+poetry python list
+# You'll see something like this:
+>> 3.13.2  CPython        System  C:/Users/<user>/.pyenv/pyenv-win/versions/3.13.2/python3.13.exe
+>> 3.13.2  CPython        System  C:/Users/<user>/.pyenv/pyenv-win/versions/3.13.2/python313.exe
+>> 3.13.2  CPython        System  C:/Users/<user>/.pyenv/pyenv-win/versions/3.13.2/python3.exe
+>> 3.13.2  CPython        System  C:/Users/<user>/.pyenv/pyenv-win/versions/3.13.2/python.exe
+>> 3.11.9  CPython        System  C:/Users/<user>/.pyenv/pyenv-win/versions/3.11.9/python3.11.exe
+>> 3.11.9  CPython        System  C:/Users/<user>/.pyenv/pyenv-win/versions/3.11.9/python311.exe
+>> 3.11.9  CPython        System  C:/Users/<user>/.pyenv/pyenv-win/versions/3.11.9/python3.exe
+>> 3.11.9  CPython        System  C:/Users/<user>/.pyenv/pyenv-win/versions/3.11.9/python.exe
+# Copy and paste ~any~ of the comparable paths to pyenv 3.11.9 ...
+poetry use C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.9\python.exe
+```
+Pull the requirements from the pyproject.toml file for packagage installation.
+```
+poetry install 
+# This is where the magic happens.
+# When in doubt, run this again. 
+# Especially if you get a ModuleNotFound warning.
+# Sometimes it doesn't take until you use "poetry run python".
+poetry run python
+poetry run python -m src.pipeline.env
+poetry run python -m src.pipeline.api.eds ping
+poetry run python -m workspaces.eds_to_rjn.scripts.main
+```
+
+# References
+PEP621.toml
+
+# Installation on Termux on Android
+
+Due to `maturin` (Rust build dependency) required by newer versions of `pydantic`, use older versions of FastAPI and Pydantic that don't require Rust, to successfully install on Termux.
+
+Also, Termux does not allow `poetry` or `pyenv`, so install packages directly to your generic environement with pip.
+
+```
+pip install -r requirements-termux.txt
+```
+
+Note that Termux does not allow `numpy` or `pandas`. Nor does it allow any plotting through a pop up window, like with tkinter, freesimplegui, matplotlib, etc.
+
+

pipeline_eds-0.2.4.dist-info/RECORD

@@ -0,0 +1,62 @@
+pipeline/__init__.py,sha256=DAy4hBleDDk9Wen4qIOGlS03qWqdt7K7n4rUpL3cIL4,173
+pipeline/__main__.py,sha256=TtbGYUm7Np5eLPCHZ3iX5QPL0rSgGJ5tXVuBEORvQIE,19
+pipeline/api/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+pipeline/api/eds.py,sha256=U0mdrYqNyYpx2J4wRwjdjVGpqVNTnI_WEr8Y2vQW3v8,44083
+pipeline/api/rjn.py,sha256=V8dLviYk1or-8bdAiEXPKBeYaZ2VgBhSRoV3HN2-ojM,7150
+pipeline/api/status_api.py,sha256=tAPaWTFL-rGwLkMucI_hWoTktoldv2ubg4if3Eed1EA,206
+pipeline/calls.py,sha256=GMG9hEbpTZSlMRee0QEEF5FCVX98Mf_5EKS_3MQ_l4M,4191
+pipeline/cli.py,sha256=bHiWHC82hqFFIsdjewe85drNmXIV8cdrXuAYfiPgOjY,10097
+pipeline/configrationmanager.py,sha256=UPqa91117jNm59vvTBE7sET1ThisqRf6B2Dhtk7x9tM,624
+pipeline/decorators.py,sha256=5fIIVqxSvQFaSI4ZkqPd3yqajzDxaRhgYwlC1jD2k5A,411
+pipeline/env.py,sha256=d0SufWAf_yPzLDGK44_zZcqVVFMMylTp1456ILtoslI,1926
+pipeline/environment.py,sha256=nr-3rPsxjnuu6G0gri8X0AZhAI0h8thsdfbt34OWmzY,1454
+pipeline/gui_fastapi_plotly_live.py,sha256=JpRtA6qbCJ9XPJkd_YKAbLKUJRwiJexULkYONbJlriA,2383
+pipeline/gui_mpl_live.py,sha256=g7zFtBU81XykTnb7heHfl-t5pqWniJYoSUctGTgJuL4,3625
+pipeline/helpers.py,sha256=c_LYnuJYXpqwhWJYpTfmgoHAiyqiw56oqQkuGJ3FvuU,4461
+pipeline/logging_setup.py,sha256=CAqWfchscCdzJ63Wf1dC3QGRnFnQqBELxyTmPZxsxgs,1674
+pipeline/pastehelpers.py,sha256=pkmtULW5Zk_-ffUDDp_VsGzruIjsNB1xAIPbb9jtofE,265
+pipeline/philosophy.py,sha256=QskLEpY3uZn46oyH7rHekOXiC55JqW3raThxwxaiM5U,1919
+pipeline/plotbuffer.py,sha256=jCsFbT47TdR8Sq5tjj2JdhVELjRiebLPN7O4r2LjPeY,625
+pipeline/points_loader.py,sha256=4OCGLiatbP3D5hixVnYcFGThvBRYt_bf5DhNGdGw_9k,519
+pipeline/queriesmanager.py,sha256=0fsniKy5F8MDE5C8fsOhMO44bW9m0mZRgfONSbn3lrs,5013
+pipeline/time_manager.py,sha256=gSK430SyHvhgUWLRg_z2nBiyad01v7ByyKafB138IkU,8351
+pipeline/workspace_manager.py,sha256=rrK0bUlQI8nB21SSWNVXUoC7HxHnrCbi7GAjaW2zF8M,11028
+workspaces/default-workspace.toml,sha256=dI8y2l2WlEbIck6IZpbuQUP8-Bf48bBE1CKKsnVMc8w,300
+workspaces/eds_to_rjn/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+workspaces/eds_to_rjn/code/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+workspaces/eds_to_rjn/code/aggregator.py,sha256=ewU5bPwNebiyX8VKuBsrqYqaxuTFE4U4Ip5OkJCyGfw,3076
+workspaces/eds_to_rjn/code/collector.py,sha256=XI_1TbA--rm72GdFyLqxWyzqk8eVlVPgXYAqQxp4_FY,2646
+workspaces/eds_to_rjn/code/sanitizer.py,sha256=2yKZSJYgHGrYCKjxg56XK2rlgv3aTyl8573e3qbbTbY,1673
+workspaces/eds_to_rjn/code/storage.py,sha256=4bBnb6WiIPpnyhSq5Gkt6EZUabVBEKBq7LHnmUxm1yM,564
+workspaces/eds_to_rjn/configurations/config_time.toml,sha256=8XkfMn78qxRkQLpCbTFySj-73snz-EtraZBt5aR5OGM,130
+workspaces/eds_to_rjn/configurations/configuration.toml,sha256=mlyQ4hXhrq7QiVBuqXdE1eyM1kAZNYSCXLa0GY7D9P8,81
+workspaces/eds_to_rjn/exports/aggregate/live_data - Copy.csv,sha256=MCpJHnslRjnKdxnsk0P8dkOyCK5GrWO8JG26-L4Mihw,27627
+workspaces/eds_to_rjn/exports/aggregate/live_data_EFF.csv,sha256=eVEATBPeaoAiameN5jbG-okgl8IpRS4TG4U2P7T3qjA,1504130
+workspaces/eds_to_rjn/exports/aggregate/live_data_INF.csv,sha256=DjisrETlE2Oq1C9qdIEMuT_MpkBTRLBQWFuIq7N9ncQ,1539205
+workspaces/eds_to_rjn/exports/aggregate/README.md,sha256=L-UH5wl3lS9c095ygF1h04siOC4bAFjHujnnWf-PM4U,381
+workspaces/eds_to_rjn/exports/export_eds_points_neo.txt,sha256=FU5hDecwxbKFjui2uC3ShQS7PnvV3KdDPPbkXMmTYMw,2824328
+workspaces/eds_to_rjn/exports/manual_data_load_to_postman_wetwell.csv,sha256=eWlqgjR817K5h_He9lnoA01alkRF7yQGH4aixyI0X9s,283269
+workspaces/eds_to_rjn/exports/manual_data_load_to_postman_wetwell.xlsx,sha256=UmpEX5hhi5jV7mhFiN3cQRsR469NmCX9lq0sgbmfrQs,29679
+workspaces/eds_to_rjn/exports/manual_effluent.csv,sha256=eWlqgjR817K5h_He9lnoA01alkRF7yQGH4aixyI0X9s,283269
+workspaces/eds_to_rjn/exports/manual_influent.csv,sha256=Fv9r04Bulh9aPFt6YtFdRmAiS9jBXnX01IzBJKIkZck,288003
+workspaces/eds_to_rjn/exports/manual_wetwell.csv,sha256=3Sp7VgkDf5HGMzw5o7Y9oWGFsTOlbsjGwI1xzyoVN0Y,569835
+workspaces/eds_to_rjn/exports/README.md,sha256=zQPg1fS7P_itarsoi-vBkU40pxoQcDFfHpAQmsbJBXI,374
+workspaces/eds_to_rjn/history/time_sample.txt,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+workspaces/eds_to_rjn/imports/zdMaxson_idcsD321E_sid11003.toml,sha256=_oQ3CYdFmi1mRQHlQ7NXJcB_8jW9thBmFdrAosCJeOI,269
+workspaces/eds_to_rjn/imports/zdMaxson_idcsFI8001_sid8528.toml,sha256=itvt2BjeEpWo5zAEFdvMRq0V8ZNIHaNFMECOmO5nGdg,275
+workspaces/eds_to_rjn/imports/zdMaxson_idcsM100FI_sid2308.toml,sha256=aqeaERNMLgxjhPHLZGLlVMyQsaVJdFDUhwYM2Es0tJg,279
+workspaces/eds_to_rjn/imports/zdMaxson_idcsM310LI_sid2382.toml,sha256=v1P4X05eYlMrH40HXVDwRa-VErks3ndqGS9qU_zomsM,271
+workspaces/eds_to_rjn/queries/default-queries.toml,sha256=WhdFrrBWCNR2LwDwk_YkdYCkexAA7AIoiMbAhOdX3_U,264
+workspaces/eds_to_rjn/queries/points-maxson.csv,sha256=fmYew1rk7TN6_8fNKk0elpiPovJJRHBxsUrb7YIaUTg,471
+workspaces/eds_to_rjn/queries/points-stiles.csv,sha256=cE83N4NdxSpR54emsy8a4jXZsGTT74IxEcQ2OoZOZ00,471
+workspaces/eds_to_rjn/queries/timestamps_success.json,sha256=hXBITI94lLsja-7Q-IifJLagUZZrD75isksCllGy9Qw,418
+workspaces/eds_to_rjn/scripts/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+workspaces/eds_to_rjn/scripts/daemon_runner.py,sha256=I3ZncZR3_wSjs_uZ2CKQzpQr0OfOchwZp-v7wwK2ojs,11179
+workspaces/eds_to_rjn/secrets/README.md,sha256=tWf2bhopA0C08C8ImtHNZoPde9ub-sLMjX6EMe7lyJw,600
+workspaces/eds_to_rjn/secrets/secrets-example.yaml,sha256=qKGrKsKBC0ulDQRVbr1zkfNlr8WPWK4lg5GAvTqZ-T4,365
+workspaces/eds_to_termux/..txt,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+pipeline_eds-0.2.4.dist-info/entry_points.txt,sha256=t1hTaepU6pSowdqa8OSzohoTbmDRLA3Jnp8MfqagwEs,103
+pipeline_eds-0.2.4.dist-info/LICENSE,sha256=LKdx0wS1t9vFZpbRhDg_iLQ6ny-XsXRwhKAoCfrF6iA,1501
+pipeline_eds-0.2.4.dist-info/METADATA,sha256=LZyy4Eoc0WKpEXbovPSNb4TEhBj4YnI6u4cr1nb7b5I,9964
+pipeline_eds-0.2.4.dist-info/WHEEL,sha256=b4K_helf-jlQoXBBETfwnf4B04YC67LOev0jo4fX5m8,88
+pipeline_eds-0.2.4.dist-info/RECORD,,

pipeline_eds-0.2.4.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 2.1.3
+Root-Is-Purelib: true
+Tag: py3-none-any

pipeline_eds-0.2.4.dist-info/entry_points.txt

@@ -0,0 +1,6 @@
+[console_scripts]
+pipeline=pipeline.cli:app
+
+[scripts]
+controller=pipeline.daemon.controller:main_cli
+

workspaces/default-workspace.toml

@@ -0,0 +1,3 @@
+[default-workspace]
+workspace = "eds_to_rjn"
+use-most-recently-edited-workspace-directory = false # while true, this will ignore the 'workspace' variable above and instead the most recently edited workpace folder. This is to prevent you from having to constantly edit this file to try new things.

workspaces/eds_to_rjn/code/aggregator.py

@@ -0,0 +1,84 @@
+#pipeline.aggregator.py
+import csv
+from collections import defaultdict
+import os
+from pprint import pprint
+
+from src.pipeline.api.rjn import RjnClient #send_data_to_rjn2
+from src.pipeline.time_manager import TimeManager
+
+
+def aggregate_and_send(session_rjn, data_file, checkpoint_file):
+
+    # Check what has already been sent
+    already_sent = set()
+    if os.path.exists(checkpoint_file):
+        with open(checkpoint_file, newline='') as f:
+            reader = csv.reader(f)
+            for row in reader:
+                projectid, entityid, timestamp = row
+                already_sent.add((projectid, entityid, timestamp))
+
+    print(f"len(already_sent) = {len(already_sent)}")
+
+    # Load all available data from the live data CSV
+    grouped = defaultdict(list)
+    with open(data_file, newline='') as f:
+        reader = csv.DictReader(f)
+        for row in reader:
+            if row["value"] == "":
+                 #print("Skipping empty row")
+                 continue
+            elif "timestamp" not in row:
+                print(row.keys())
+                print("timestamp not in row")
+                continue
+            timestamp = row["timestamp"]
+            projectid = row["rjn_projectid"]
+            entityid = row["rjn_entityid"]
+            value = float(row["value"])
+            key = (projectid, entityid)
+            # Only include if not already sent
+            if (projectid, entityid, timestamp) not in already_sent:
+                grouped[key].append((timestamp, value))
+
+    print(f"len(grouped) = {len(grouped)}")
+
+    # Send data per entity
+    for (projectid, entityid), records in grouped.items():
+        print(f"projectid = {projectid}")
+        # Sort timestamps if needed
+        records.sort(key=lambda x: x[0])
+
+        timestamps = [ts for ts, _ in records]
+        values = [val for _, val in records]
+
+        if timestamps:
+            print(f"Attempting to send {len(timestamps)} values to RJN for entity {entityid} at site {projectid}")
+            '''
+            send_data_to_rjn(
+                base_url=rjn_base_url,
+                project_id=projectid,
+                entity_id=entityid,
+                headers=headers_rjn,
+                timestamps=timestamps,
+                values=values
+            )
+            '''
+            RjnClient.send_data_to_rjn(
+            session_rjn,
+            base_url = session_rjn.base_url,
+            project_id=row["rjn_projectid"],
+            entity_id=row["rjn_entityid"],
+            timestamps=timestamps,
+            values=[row["value"]]
+        )
+
+            # Record successful sends
+            if os.path.exists(checkpoint_file):
+                with open(checkpoint_file, 'a', newline='') as f:
+                    writer = csv.writer(f)
+                    for ts in timestamps:
+                        writer.writerow([projectid, entityid, TimeManager(ts).as_formatted_date_time()])
+        else:
+            print(f"No new data to send for {projectid} / {entityid}")

workspaces/eds_to_rjn/code/collector.py

@@ -0,0 +1,60 @@
+# collector.py
+from datetime import datetime
+import logging
+logger = logging.getLogger(__name__)
+
+from src.pipeline.helpers import round_datetime_to_nearest_past_five_minutes
+from src.pipeline.api.eds import EdsClient
+
+
+def collect_live_values(session, queries_dictlist_filtered_by_session_key):   
+    data = []
+    for row in queries_dictlist_filtered_by_session_key:
+        #print(f"\trow = {row}")
+        # Skip empty rows (if all values in the row are empty or None)
+        if not any(row.values()):
+            print("Skipping empty row.")
+            continue
+        
+        # light validation - if you want to change column keys, that could be cool
+        #required_cols = ["iess", "rjn_projectid", "rjn_entityid"]
+        required_cols = ["iess"]
+        if any(c not in row for c in required_cols):
+            raise ValueError(f"Row missing required column keys: {row}")
+        
+        try:
+            # extract and validate iess value from CSV row before it is used to retrieve data
+            iess = str(row["iess"]) if row["iess"] not in (None, '', '\t') else None
+        except KeyError as e:
+            print(f"Missing expected column in CSV: {e}")
+            continue
+        except ValueError as e:
+            print(f"Invalid data in row: {e}")
+            continue
+        
+        try:
+            point_data = EdsClient.get_points_live_mod(session, iess)
+            if point_data is None:
+                print(f"No data returned for iess={iess}")
+                continue
+            conflicts = set(row.keys()) & set(point_data.keys())
+            if conflicts:
+                logger.debug(f"Warning: column key collision on {conflicts}, for iess = {iess}. This is expected.")
+            '''
+            Not the worst idea:
+            Use nested structures
+            Instead of flattening all column keys into the same dict, keep fetched data as a sub-dictionary.
+            In which case, the aggregate should be JSON (or TOML, whatever), not CSV.
+            However, we have something that works. It is fine for now.
+            '''
+            # Retrieved point data is flatly added to the existing row from the query.   
+            row.update(point_data) 
+            data.append(row)
+        except Exception as e:
+            print(f"Error on row: {e}")
+    return data
+
+if __name__ == "__main__":
+    print("[x] from src.pipeline import collector")
+    print("[x] from src.pipeline import collector.collector_live_vales(session,query_list)")
+    print("[ ] from src.pipeline import collector.collector_live_vales(session,query_dict)")

workspaces/eds_to_rjn/code/sanitizer.py

@@ -0,0 +1,40 @@
+'''
+Title: sanitizer.py
+Author: Clayton Bennett
+Created: 2025 05-May 17th
+
+Purpose:
+I like it when data gathered is data returned. Sanitization should not happen during import. If not entirely probitive, the entirety of the raw should be available for unforeseen use cases.
+So. We need explicit and discernible sanitization scenarios, called a scripted approach, following the preparation, collection, and aggregation, insert buzz words here, etc.
+'''
+from datetime import datetime
+from src.pipeline.helpers import round_datetime_to_nearest_past_five_minutes
+
+def sanitize_data_for_printing(data):
+    #data_sanitized_for_printing = data
+    #pass
+    #return data_sanitized_for_printing
+    return data
+
+def sanitize_data_for_aggregated_storage(data):
+    sanitized = []
+    for row in data:
+        rounded_dt = round_datetime_to_nearest_past_five_minutes(datetime.fromtimestamp(row["ts"])) # arguably not appropriate at this point. round at transmission
+        #row["timestamp_sani"] = rounded_dt
+        #row["value_rounded"] = round(row["value"], 2)
+
+        sanitized.append({
+            "timestamp": rounded_dt.isoformat(),
+            "ts": rounded_dt.timestamp(),
+            "iess": row.get("iess"),
+            "sid": row.get("sid"),
+            "un": row.get("un"),
+            "shortdesc": row.get("shortdesc"),
+            "rjn_projectid": row.get("rjn_projectid"),
+            "rjn_entityid": row.get("rjn_entityid"),
+            "value": round(row.get("value"), 2)
+        })
+                
+    data_sanitized_for_aggregated_storage = sanitized
+                    
+    return data_sanitized_for_aggregated_storage

workspaces/eds_to_rjn/code/storage.py

@@ -0,0 +1,16 @@
+import csv
+from datetime import datetime
+'''
+Mostly defunct now that eds tabular trend is working. But still interesting.
+Use: 
+    ```
+    storage.store_live_values(data, workspace_manager.get_aggregate_dir() / "live_data.csv") 
+    ```
+'''
+def store_live_values(data, path):
+    with open(path, 'a', newline='') as f:
+        writer = csv.DictWriter(f, fieldnames=data[0].keys())
+        if f.tell() == 0:  # file is empty
+            writer.writeheader()
+        writer.writerows(data)
+    print(f"Live values stored, {datetime.now()} to {path}")

workspaces/eds_to_rjn/configurations/config_time.toml

@@ -0,0 +1,11 @@
+[rangetime.start]
+year = 2024
+month = 12 
+day = 16
+hour = 15
+
+[rangetime.end]
+year = 2024
+month = 12 
+day = 16
+hour = 18

workspaces/eds_to_rjn/configurations/configuration.toml

@@ -0,0 +1,2 @@
+[settings]
+timezone = 	 # A valid IANA time zone string, e.g. 'America/Chicago'.

workspaces/eds_to_rjn/exports/README.md

@@ -0,0 +1,7 @@
+# Exports: export_eds_points_all.txt
+Among other future possibilities, the export_eds_points_all.txt is stored here by the pipeline.api.eds demo_eds_save_point_export() function.
+The point export can be called like this:
+```
+poetry run python -m pipeline.api.eds demo-points-export
+```
+The export_eds_points_all.txt file is registered in the .gitignore for security.

workspaces/eds_to_rjn/exports/aggregate/README.md

@@ -0,0 +1,7 @@
+# Aggregate: daemon temp, live_data.csv and sent_data.csv
+Files live_data.csv and sent_data.csv are stored here by workspaces.eds_to_rjn.scripts.daemon_runner main() function.
+The daemon can be called like this:
+```
+poetry run python -m workspaces.eds_to_rjn.scripts.daemon_runner
+```
+The live_data.csv and sent_data.csv files are registered in the .gitignore for security.