ETLai 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- etlai-0.1.0/ETLai.egg-info/PKG-INFO +193 -0
- etlai-0.1.0/ETLai.egg-info/SOURCES.txt +31 -0
- etlai-0.1.0/ETLai.egg-info/dependency_links.txt +1 -0
- etlai-0.1.0/ETLai.egg-info/requires.txt +5 -0
- etlai-0.1.0/ETLai.egg-info/top_level.txt +4 -0
- etlai-0.1.0/LICENSE +21 -0
- etlai-0.1.0/PKG-INFO +193 -0
- etlai-0.1.0/README.md +168 -0
- etlai-0.1.0/helpers/README.md +43 -0
- etlai-0.1.0/helpers/__init__.py +0 -0
- etlai-0.1.0/helpers/column_picker.py +145 -0
- etlai-0.1.0/helpers/config_store.py +26 -0
- etlai-0.1.0/helpers/file_picker.py +31 -0
- etlai-0.1.0/helpers/folders.py +80 -0
- etlai-0.1.0/helpers/mock_generator.py +92 -0
- etlai-0.1.0/helpers/notifier.py +58 -0
- etlai-0.1.0/logic/README.md +19 -0
- etlai-0.1.0/logic/__init__.py +1 -0
- etlai-0.1.0/logic/atoms/README.md +28 -0
- etlai-0.1.0/logic/atoms/__init__.py +1 -0
- etlai-0.1.0/logic/atoms/groupby.py +39 -0
- etlai-0.1.0/logic/atoms/vlookup.py +74 -0
- etlai-0.1.0/pyproject.toml +40 -0
- etlai-0.1.0/runners/README.md +38 -0
- etlai-0.1.0/runners/__init__.py +0 -0
- etlai-0.1.0/runners/atom_runner.py +80 -0
- etlai-0.1.0/runners/composite.py +167 -0
- etlai-0.1.0/runners/ops.py +131 -0
- etlai-0.1.0/runners/pipeline_factory.py +40 -0
- etlai-0.1.0/sensors/README.md +24 -0
- etlai-0.1.0/sensors/__init__.py +0 -0
- etlai-0.1.0/sensors/hot_folder_sensor.py +100 -0
- etlai-0.1.0/setup.cfg +4 -0
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ETLai
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Local, folder-driven Dagster application for CSV transformations with hot-folder sensors and Tkinter configuration UI.
|
|
5
|
+
Author: Umangjeet Singh Pahwa
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Repository, https://github.com/umang/ETLai
|
|
8
|
+
Classifier: Development Status :: 3 - Alpha
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
15
|
+
Classifier: Topic :: Scientific/Engineering
|
|
16
|
+
Requires-Python: >=3.10
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
License-File: LICENSE
|
|
19
|
+
Requires-Dist: dagster>=1.5
|
|
20
|
+
Requires-Dist: dagster-webserver>=1.5
|
|
21
|
+
Requires-Dist: pandas>=2.0
|
|
22
|
+
Requires-Dist: faker>=18.0
|
|
23
|
+
Requires-Dist: openpyxl>=3.1
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
# Air-Gapped Desktop Automation Engine
|
|
27
|
+
|
|
28
|
+
A local Dagster application for folder-driven CSV transformations. Source files,
|
|
29
|
+
saved column choices, generated outputs, and Dagster run storage remain on the
|
|
30
|
+
machine.
|
|
31
|
+
|
|
32
|
+
## How it works
|
|
33
|
+
|
|
34
|
+
Each business pipeline has an inbox and a Dagster sensor. The sensor waits for
|
|
35
|
+
the required number of stable files, moves them to staging, and launches a job.
|
|
36
|
+
The job prepares atom parameters, executes the transformation, then moves source
|
|
37
|
+
files to `processed/` or `rejected/`.
|
|
38
|
+
|
|
39
|
+
An **atom** is a reusable transformation module with an
|
|
40
|
+
`execute(params_json) -> result_json` interface. Atoms are domain-agnostic —
|
|
41
|
+
they have no opinion on which column or business context. They do read and write
|
|
42
|
+
files, so they are not side-effect-free.
|
|
43
|
+
|
|
44
|
+
A **business pipeline** applies an atom to specific domain data with its own
|
|
45
|
+
folder lifecycle, sensor, and saved configuration.
|
|
46
|
+
|
|
47
|
+
## Registered jobs
|
|
48
|
+
|
|
49
|
+
| Job | Files needed | Description |
|
|
50
|
+
|-----|-------------|-------------|
|
|
51
|
+
| `vlookup_rollnumber` | 2 CSV | Left-join on user-chosen columns (first run: Tkinter picker, then saved) |
|
|
52
|
+
| `groupby_religion` | 1 CSV | Group by user-chosen column with counts, sorted descending |
|
|
53
|
+
| `mock_generator` | 1+ CSV | Generate synthetic data from file headers using Faker |
|
|
54
|
+
| `vlookup_then_groupby` | 2 CSV | Composite: VLOOKUP → GroupBy in a single job |
|
|
55
|
+
|
|
56
|
+
Each job has a same-named sensor. Sensors must be enabled in Dagster UI before
|
|
57
|
+
they process inbox files.
|
|
58
|
+
|
|
59
|
+
## Installation
|
|
60
|
+
|
|
61
|
+
Python 3.10+ required.
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
python3 -m venv .venv
|
|
65
|
+
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
|
66
|
+
python -m pip install -r requirements.txt
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Tkinter must be available in the Python installation for first-run config.
|
|
70
|
+
|
|
71
|
+
## Run
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
DAGSTER_HOME="$(pwd)" dagster dev -m definitions
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Open `http://localhost:3000`, enable the desired sensor, and place files in
|
|
78
|
+
`pipelines/<job-name>/inbox/`.
|
|
79
|
+
|
|
80
|
+
Manual execution without UI:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
DAGSTER_HOME="$(pwd)" dagster job execute -m definitions -j vlookup_rollnumber
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
## Source modules
|
|
87
|
+
|
|
88
|
+
### Entry points
|
|
89
|
+
|
|
90
|
+
| File | Role |
|
|
91
|
+
|------|------|
|
|
92
|
+
| `definitions.py` | Registers all jobs and sensors with Dagster |
|
|
93
|
+
| `pipeline.py` | Builds single-atom business jobs via the factory |
|
|
94
|
+
| `runners/composite.py` | Defines the `vlookup_then_groupby` composite job |
|
|
95
|
+
|
|
96
|
+
### `logic/atoms/`
|
|
97
|
+
|
|
98
|
+
| Module | Purpose |
|
|
99
|
+
|--------|---------|
|
|
100
|
+
| `vlookup.py` | Left join two CSVs on specified columns with dtype validation |
|
|
101
|
+
| `groupby.py` | Group by one column, count, sort descending |
|
|
102
|
+
| `mock_generate.py` | Infer Faker generators from headers, write synthetic CSVs |
|
|
103
|
+
|
|
104
|
+
### `runners/`
|
|
105
|
+
|
|
106
|
+
| Module | Purpose |
|
|
107
|
+
|--------|---------|
|
|
108
|
+
| `pipeline_factory.py` | Assembles load → preprocess → execute Dagster jobs |
|
|
109
|
+
| `atom_runner.py` | Reusable load/execute ops with processed/rejected/notification lifecycle |
|
|
110
|
+
| `ops.py` | First-run config ops for VLOOKUP, GroupBy, and mock generator |
|
|
111
|
+
| `composite.py` | VLOOKUP → GroupBy chain with shared config |
|
|
112
|
+
|
|
113
|
+
### `sensors/`
|
|
114
|
+
|
|
115
|
+
| Module | Purpose |
|
|
116
|
+
|--------|---------|
|
|
117
|
+
| `hot_folder_sensor.py` | Per-pipeline sensor factory: stability check, staging, sweeper |
|
|
118
|
+
|
|
119
|
+
### `helpers/`
|
|
120
|
+
|
|
121
|
+
| Module | Purpose |
|
|
122
|
+
|--------|---------|
|
|
123
|
+
| `folders.py` | `PipelineFolders` class — per-pipeline directory lifecycle |
|
|
124
|
+
| `config_store.py` | JSON config read/write per pipeline |
|
|
125
|
+
| `column_picker.py` | Tkinter UI for VLOOKUP join/output column selection |
|
|
126
|
+
| `notifier.py` | macOS/Windows toast notifications, opens output folder on success |
|
|
127
|
+
| `file_picker.py` | Standalone Tkinter file dialog (not used by registered jobs) |
|
|
128
|
+
| `mock_generator.py` | Standalone CLI mock generator (separate from the atom/job) |
|
|
129
|
+
|
|
130
|
+
### Runtime data
|
|
131
|
+
|
|
132
|
+
- `pipelines/<job>/` — inbox, staging, processed, rejected, output, config.json
|
|
133
|
+
- `dagster.yaml` — SQLite storage configuration
|
|
134
|
+
- `dagster_storage/` — Dagster run history (persists across restarts)
|
|
135
|
+
|
|
136
|
+
## File and configuration lifecycle
|
|
137
|
+
|
|
138
|
+
```text
|
|
139
|
+
inbox/ → sensor stability check → staging/ → job runs
|
|
140
|
+
├→ processed/ + output/ (success)
|
|
141
|
+
└→ rejected/ + *.error.txt (failure)
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Non-matching files (not `*.csv`/`*.xlsx`) are swept to `rejected/` after 3
|
|
145
|
+
minutes with an error description.
|
|
146
|
+
|
|
147
|
+
### Config init pattern
|
|
148
|
+
|
|
149
|
+
1. First run: pre-process op detects no `config.json` → shows Tkinter UI → saves
|
|
150
|
+
2. Future runs: loads config silently, fully automated
|
|
151
|
+
3. To reconfigure: delete `pipelines/<job>/config.json`
|
|
152
|
+
|
|
153
|
+
### Saved config fields
|
|
154
|
+
|
|
155
|
+
| Job | Fields |
|
|
156
|
+
|-----|--------|
|
|
157
|
+
| `vlookup_rollnumber` | `left_column`, `right_column`, `left_output_columns`, `right_output_columns` |
|
|
158
|
+
| `groupby_religion` | `group_column` |
|
|
159
|
+
| `vlookup_then_groupby` | All VLOOKUP fields + `group_column` |
|
|
160
|
+
| `mock_generator` | None |
|
|
161
|
+
|
|
162
|
+
## Important behavior
|
|
163
|
+
|
|
164
|
+
- Two-file jobs: lexical filename order determines left/right assignment.
|
|
165
|
+
- Atom failures and config cancellation move staged files to `rejected/`.
|
|
166
|
+
- Unexpected crashes may leave files in `staging/` — recover manually.
|
|
167
|
+
- Notifications are best-effort. Linux has no implementation.
|
|
168
|
+
- On success, macOS/Windows opens the output folder immediately (not a click
|
|
169
|
+
action).
|
|
170
|
+
|
|
171
|
+
## Adding a new business pipeline
|
|
172
|
+
|
|
173
|
+
1. Create or reuse an atom in `logic/atoms/` with `execute(params_json) -> str`
|
|
174
|
+
2. Create a pre-process op in `runners/ops.py` (with config init pattern)
|
|
175
|
+
3. Register in `pipeline.py`:
|
|
176
|
+
```python
|
|
177
|
+
my_pipeline = build_business_pipeline(
|
|
178
|
+
pipeline_name="my_pipeline",
|
|
179
|
+
atom_module=my_atom,
|
|
180
|
+
atom_label="My Pipeline",
|
|
181
|
+
pre_process_op=my_pre_process_op,
|
|
182
|
+
)
|
|
183
|
+
```
|
|
184
|
+
4. Add sensor in `definitions.py`:
|
|
185
|
+
```python
|
|
186
|
+
my_sensor = build_hot_folder_sensor("my_pipeline", "my_pipeline", min_files=1)
|
|
187
|
+
```
|
|
188
|
+
5. Folders are created automatically on import.
|
|
189
|
+
|
|
190
|
+
## Development status
|
|
191
|
+
|
|
192
|
+
No automated tests, lint, or pinned dependency versions. Validate by loading
|
|
193
|
+
definitions and exercising jobs with disposable CSV inputs.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
ETLai.egg-info/PKG-INFO
|
|
5
|
+
ETLai.egg-info/SOURCES.txt
|
|
6
|
+
ETLai.egg-info/dependency_links.txt
|
|
7
|
+
ETLai.egg-info/requires.txt
|
|
8
|
+
ETLai.egg-info/top_level.txt
|
|
9
|
+
helpers/README.md
|
|
10
|
+
helpers/__init__.py
|
|
11
|
+
helpers/column_picker.py
|
|
12
|
+
helpers/config_store.py
|
|
13
|
+
helpers/file_picker.py
|
|
14
|
+
helpers/folders.py
|
|
15
|
+
helpers/mock_generator.py
|
|
16
|
+
helpers/notifier.py
|
|
17
|
+
logic/README.md
|
|
18
|
+
logic/__init__.py
|
|
19
|
+
logic/atoms/README.md
|
|
20
|
+
logic/atoms/__init__.py
|
|
21
|
+
logic/atoms/groupby.py
|
|
22
|
+
logic/atoms/vlookup.py
|
|
23
|
+
runners/README.md
|
|
24
|
+
runners/__init__.py
|
|
25
|
+
runners/atom_runner.py
|
|
26
|
+
runners/composite.py
|
|
27
|
+
runners/ops.py
|
|
28
|
+
runners/pipeline_factory.py
|
|
29
|
+
sensors/README.md
|
|
30
|
+
sensors/__init__.py
|
|
31
|
+
sensors/hot_folder_sensor.py
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
etlai-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Umangjeet Singh Pahwa
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
etlai-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ETLai
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Local, folder-driven Dagster application for CSV transformations with hot-folder sensors and Tkinter configuration UI.
|
|
5
|
+
Author: Umangjeet Singh Pahwa
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Repository, https://github.com/umang/ETLai
|
|
8
|
+
Classifier: Development Status :: 3 - Alpha
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
15
|
+
Classifier: Topic :: Scientific/Engineering
|
|
16
|
+
Requires-Python: >=3.10
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
License-File: LICENSE
|
|
19
|
+
Requires-Dist: dagster>=1.5
|
|
20
|
+
Requires-Dist: dagster-webserver>=1.5
|
|
21
|
+
Requires-Dist: pandas>=2.0
|
|
22
|
+
Requires-Dist: faker>=18.0
|
|
23
|
+
Requires-Dist: openpyxl>=3.1
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
# Air-Gapped Desktop Automation Engine
|
|
27
|
+
|
|
28
|
+
A local Dagster application for folder-driven CSV transformations. Source files,
|
|
29
|
+
saved column choices, generated outputs, and Dagster run storage remain on the
|
|
30
|
+
machine.
|
|
31
|
+
|
|
32
|
+
## How it works
|
|
33
|
+
|
|
34
|
+
Each business pipeline has an inbox and a Dagster sensor. The sensor waits for
|
|
35
|
+
the required number of stable files, moves them to staging, and launches a job.
|
|
36
|
+
The job prepares atom parameters, executes the transformation, then moves source
|
|
37
|
+
files to `processed/` or `rejected/`.
|
|
38
|
+
|
|
39
|
+
An **atom** is a reusable transformation module with an
|
|
40
|
+
`execute(params_json) -> result_json` interface. Atoms are domain-agnostic —
|
|
41
|
+
they have no opinion on which column or business context. They do read and write
|
|
42
|
+
files, so they are not side-effect-free.
|
|
43
|
+
|
|
44
|
+
A **business pipeline** applies an atom to specific domain data with its own
|
|
45
|
+
folder lifecycle, sensor, and saved configuration.
|
|
46
|
+
|
|
47
|
+
## Registered jobs
|
|
48
|
+
|
|
49
|
+
| Job | Files needed | Description |
|
|
50
|
+
|-----|-------------|-------------|
|
|
51
|
+
| `vlookup_rollnumber` | 2 CSV | Left-join on user-chosen columns (first run: Tkinter picker, then saved) |
|
|
52
|
+
| `groupby_religion` | 1 CSV | Group by user-chosen column with counts, sorted descending |
|
|
53
|
+
| `mock_generator` | 1+ CSV | Generate synthetic data from file headers using Faker |
|
|
54
|
+
| `vlookup_then_groupby` | 2 CSV | Composite: VLOOKUP → GroupBy in a single job |
|
|
55
|
+
|
|
56
|
+
Each job has a same-named sensor. Sensors must be enabled in Dagster UI before
|
|
57
|
+
they process inbox files.
|
|
58
|
+
|
|
59
|
+
## Installation
|
|
60
|
+
|
|
61
|
+
Python 3.10+ required.
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
python3 -m venv .venv
|
|
65
|
+
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
|
66
|
+
python -m pip install -r requirements.txt
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Tkinter must be available in the Python installation for first-run config.
|
|
70
|
+
|
|
71
|
+
## Run
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
DAGSTER_HOME="$(pwd)" dagster dev -m definitions
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Open `http://localhost:3000`, enable the desired sensor, and place files in
|
|
78
|
+
`pipelines/<job-name>/inbox/`.
|
|
79
|
+
|
|
80
|
+
Manual execution without UI:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
DAGSTER_HOME="$(pwd)" dagster job execute -m definitions -j vlookup_rollnumber
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
## Source modules
|
|
87
|
+
|
|
88
|
+
### Entry points
|
|
89
|
+
|
|
90
|
+
| File | Role |
|
|
91
|
+
|------|------|
|
|
92
|
+
| `definitions.py` | Registers all jobs and sensors with Dagster |
|
|
93
|
+
| `pipeline.py` | Builds single-atom business jobs via the factory |
|
|
94
|
+
| `runners/composite.py` | Defines the `vlookup_then_groupby` composite job |
|
|
95
|
+
|
|
96
|
+
### `logic/atoms/`
|
|
97
|
+
|
|
98
|
+
| Module | Purpose |
|
|
99
|
+
|--------|---------|
|
|
100
|
+
| `vlookup.py` | Left join two CSVs on specified columns with dtype validation |
|
|
101
|
+
| `groupby.py` | Group by one column, count, sort descending |
|
|
102
|
+
| `mock_generate.py` | Infer Faker generators from headers, write synthetic CSVs |
|
|
103
|
+
|
|
104
|
+
### `runners/`
|
|
105
|
+
|
|
106
|
+
| Module | Purpose |
|
|
107
|
+
|--------|---------|
|
|
108
|
+
| `pipeline_factory.py` | Assembles load → preprocess → execute Dagster jobs |
|
|
109
|
+
| `atom_runner.py` | Reusable load/execute ops with processed/rejected/notification lifecycle |
|
|
110
|
+
| `ops.py` | First-run config ops for VLOOKUP, GroupBy, and mock generator |
|
|
111
|
+
| `composite.py` | VLOOKUP → GroupBy chain with shared config |
|
|
112
|
+
|
|
113
|
+
### `sensors/`
|
|
114
|
+
|
|
115
|
+
| Module | Purpose |
|
|
116
|
+
|--------|---------|
|
|
117
|
+
| `hot_folder_sensor.py` | Per-pipeline sensor factory: stability check, staging, sweeper |
|
|
118
|
+
|
|
119
|
+
### `helpers/`
|
|
120
|
+
|
|
121
|
+
| Module | Purpose |
|
|
122
|
+
|--------|---------|
|
|
123
|
+
| `folders.py` | `PipelineFolders` class — per-pipeline directory lifecycle |
|
|
124
|
+
| `config_store.py` | JSON config read/write per pipeline |
|
|
125
|
+
| `column_picker.py` | Tkinter UI for VLOOKUP join/output column selection |
|
|
126
|
+
| `notifier.py` | macOS/Windows toast notifications, opens output folder on success |
|
|
127
|
+
| `file_picker.py` | Standalone Tkinter file dialog (not used by registered jobs) |
|
|
128
|
+
| `mock_generator.py` | Standalone CLI mock generator (separate from the atom/job) |
|
|
129
|
+
|
|
130
|
+
### Runtime data
|
|
131
|
+
|
|
132
|
+
- `pipelines/<job>/` — inbox, staging, processed, rejected, output, config.json
|
|
133
|
+
- `dagster.yaml` — SQLite storage configuration
|
|
134
|
+
- `dagster_storage/` — Dagster run history (persists across restarts)
|
|
135
|
+
|
|
136
|
+
## File and configuration lifecycle
|
|
137
|
+
|
|
138
|
+
```text
|
|
139
|
+
inbox/ → sensor stability check → staging/ → job runs
|
|
140
|
+
├→ processed/ + output/ (success)
|
|
141
|
+
└→ rejected/ + *.error.txt (failure)
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Non-matching files (not `*.csv`/`*.xlsx`) are swept to `rejected/` after 3
|
|
145
|
+
minutes with an error description.
|
|
146
|
+
|
|
147
|
+
### Config init pattern
|
|
148
|
+
|
|
149
|
+
1. First run: pre-process op detects no `config.json` → shows Tkinter UI → saves
|
|
150
|
+
2. Future runs: loads config silently, fully automated
|
|
151
|
+
3. To reconfigure: delete `pipelines/<job>/config.json`
|
|
152
|
+
|
|
153
|
+
### Saved config fields
|
|
154
|
+
|
|
155
|
+
| Job | Fields |
|
|
156
|
+
|-----|--------|
|
|
157
|
+
| `vlookup_rollnumber` | `left_column`, `right_column`, `left_output_columns`, `right_output_columns` |
|
|
158
|
+
| `groupby_religion` | `group_column` |
|
|
159
|
+
| `vlookup_then_groupby` | All VLOOKUP fields + `group_column` |
|
|
160
|
+
| `mock_generator` | None |
|
|
161
|
+
|
|
162
|
+
## Important behavior
|
|
163
|
+
|
|
164
|
+
- Two-file jobs: lexical filename order determines left/right assignment.
|
|
165
|
+
- Atom failures and config cancellation move staged files to `rejected/`.
|
|
166
|
+
- Unexpected crashes may leave files in `staging/` — recover manually.
|
|
167
|
+
- Notifications are best-effort. Linux has no implementation.
|
|
168
|
+
- On success, macOS/Windows opens the output folder immediately (not a click
|
|
169
|
+
action).
|
|
170
|
+
|
|
171
|
+
## Adding a new business pipeline
|
|
172
|
+
|
|
173
|
+
1. Create or reuse an atom in `logic/atoms/` with `execute(params_json) -> str`
|
|
174
|
+
2. Create a pre-process op in `runners/ops.py` (with config init pattern)
|
|
175
|
+
3. Register in `pipeline.py`:
|
|
176
|
+
```python
|
|
177
|
+
my_pipeline = build_business_pipeline(
|
|
178
|
+
pipeline_name="my_pipeline",
|
|
179
|
+
atom_module=my_atom,
|
|
180
|
+
atom_label="My Pipeline",
|
|
181
|
+
pre_process_op=my_pre_process_op,
|
|
182
|
+
)
|
|
183
|
+
```
|
|
184
|
+
4. Add sensor in `definitions.py`:
|
|
185
|
+
```python
|
|
186
|
+
my_sensor = build_hot_folder_sensor("my_pipeline", "my_pipeline", min_files=1)
|
|
187
|
+
```
|
|
188
|
+
5. Folders are created automatically on import.
|
|
189
|
+
|
|
190
|
+
## Development status
|
|
191
|
+
|
|
192
|
+
No automated tests, lint, or pinned dependency versions. Validate by loading
|
|
193
|
+
definitions and exercising jobs with disposable CSV inputs.
|
etlai-0.1.0/README.md
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
# Air-Gapped Desktop Automation Engine
|
|
2
|
+
|
|
3
|
+
A local Dagster application for folder-driven CSV transformations. Source files,
|
|
4
|
+
saved column choices, generated outputs, and Dagster run storage remain on the
|
|
5
|
+
machine.
|
|
6
|
+
|
|
7
|
+
## How it works
|
|
8
|
+
|
|
9
|
+
Each business pipeline has an inbox and a Dagster sensor. The sensor waits for
|
|
10
|
+
the required number of stable files, moves them to staging, and launches a job.
|
|
11
|
+
The job prepares atom parameters, executes the transformation, then moves source
|
|
12
|
+
files to `processed/` or `rejected/`.
|
|
13
|
+
|
|
14
|
+
An **atom** is a reusable transformation module with an
|
|
15
|
+
`execute(params_json) -> result_json` interface. Atoms are domain-agnostic —
|
|
16
|
+
they have no opinion on which column or business context. They do read and write
|
|
17
|
+
files, so they are not side-effect-free.
|
|
18
|
+
|
|
19
|
+
A **business pipeline** applies an atom to specific domain data with its own
|
|
20
|
+
folder lifecycle, sensor, and saved configuration.
|
|
21
|
+
|
|
22
|
+
## Registered jobs
|
|
23
|
+
|
|
24
|
+
| Job | Files needed | Description |
|
|
25
|
+
|-----|-------------|-------------|
|
|
26
|
+
| `vlookup_rollnumber` | 2 CSV | Left-join on user-chosen columns (first run: Tkinter picker, then saved) |
|
|
27
|
+
| `groupby_religion` | 1 CSV | Group by user-chosen column with counts, sorted descending |
|
|
28
|
+
| `mock_generator` | 1+ CSV | Generate synthetic data from file headers using Faker |
|
|
29
|
+
| `vlookup_then_groupby` | 2 CSV | Composite: VLOOKUP → GroupBy in a single job |
|
|
30
|
+
|
|
31
|
+
Each job has a same-named sensor. Sensors must be enabled in Dagster UI before
|
|
32
|
+
they process inbox files.
|
|
33
|
+
|
|
34
|
+
## Installation
|
|
35
|
+
|
|
36
|
+
Python 3.10+ required.
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
python3 -m venv .venv
|
|
40
|
+
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
|
41
|
+
python -m pip install -r requirements.txt
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Tkinter must be available in the Python installation for first-run config.
|
|
45
|
+
|
|
46
|
+
## Run
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
DAGSTER_HOME="$(pwd)" dagster dev -m definitions
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Open `http://localhost:3000`, enable the desired sensor, and place files in
|
|
53
|
+
`pipelines/<job-name>/inbox/`.
|
|
54
|
+
|
|
55
|
+
Manual execution without UI:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
DAGSTER_HOME="$(pwd)" dagster job execute -m definitions -j vlookup_rollnumber
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
## Source modules
|
|
62
|
+
|
|
63
|
+
### Entry points
|
|
64
|
+
|
|
65
|
+
| File | Role |
|
|
66
|
+
|------|------|
|
|
67
|
+
| `definitions.py` | Registers all jobs and sensors with Dagster |
|
|
68
|
+
| `pipeline.py` | Builds single-atom business jobs via the factory |
|
|
69
|
+
| `runners/composite.py` | Defines the `vlookup_then_groupby` composite job |
|
|
70
|
+
|
|
71
|
+
### `logic/atoms/`
|
|
72
|
+
|
|
73
|
+
| Module | Purpose |
|
|
74
|
+
|--------|---------|
|
|
75
|
+
| `vlookup.py` | Left join two CSVs on specified columns with dtype validation |
|
|
76
|
+
| `groupby.py` | Group by one column, count, sort descending |
|
|
77
|
+
| `mock_generate.py` | Infer Faker generators from headers, write synthetic CSVs |
|
|
78
|
+
|
|
79
|
+
### `runners/`
|
|
80
|
+
|
|
81
|
+
| Module | Purpose |
|
|
82
|
+
|--------|---------|
|
|
83
|
+
| `pipeline_factory.py` | Assembles load → preprocess → execute Dagster jobs |
|
|
84
|
+
| `atom_runner.py` | Reusable load/execute ops with processed/rejected/notification lifecycle |
|
|
85
|
+
| `ops.py` | First-run config ops for VLOOKUP, GroupBy, and mock generator |
|
|
86
|
+
| `composite.py` | VLOOKUP → GroupBy chain with shared config |
|
|
87
|
+
|
|
88
|
+
### `sensors/`
|
|
89
|
+
|
|
90
|
+
| Module | Purpose |
|
|
91
|
+
|--------|---------|
|
|
92
|
+
| `hot_folder_sensor.py` | Per-pipeline sensor factory: stability check, staging, sweeper |
|
|
93
|
+
|
|
94
|
+
### `helpers/`
|
|
95
|
+
|
|
96
|
+
| Module | Purpose |
|
|
97
|
+
|--------|---------|
|
|
98
|
+
| `folders.py` | `PipelineFolders` class — per-pipeline directory lifecycle |
|
|
99
|
+
| `config_store.py` | JSON config read/write per pipeline |
|
|
100
|
+
| `column_picker.py` | Tkinter UI for VLOOKUP join/output column selection |
|
|
101
|
+
| `notifier.py` | macOS/Windows toast notifications, opens output folder on success |
|
|
102
|
+
| `file_picker.py` | Standalone Tkinter file dialog (not used by registered jobs) |
|
|
103
|
+
| `mock_generator.py` | Standalone CLI mock generator (separate from the atom/job) |
|
|
104
|
+
|
|
105
|
+
### Runtime data
|
|
106
|
+
|
|
107
|
+
- `pipelines/<job>/` — inbox, staging, processed, rejected, output, config.json
|
|
108
|
+
- `dagster.yaml` — SQLite storage configuration
|
|
109
|
+
- `dagster_storage/` — Dagster run history (persists across restarts)
|
|
110
|
+
|
|
111
|
+
## File and configuration lifecycle
|
|
112
|
+
|
|
113
|
+
```text
|
|
114
|
+
inbox/ → sensor stability check → staging/ → job runs
|
|
115
|
+
├→ processed/ + output/ (success)
|
|
116
|
+
└→ rejected/ + *.error.txt (failure)
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Non-matching files (not `*.csv`/`*.xlsx`) are swept to `rejected/` after 3
|
|
120
|
+
minutes with an error description.
|
|
121
|
+
|
|
122
|
+
### Config init pattern
|
|
123
|
+
|
|
124
|
+
1. First run: pre-process op detects no `config.json` → shows Tkinter UI → saves
|
|
125
|
+
2. Future runs: loads config silently, fully automated
|
|
126
|
+
3. To reconfigure: delete `pipelines/<job>/config.json`
|
|
127
|
+
|
|
128
|
+
### Saved config fields
|
|
129
|
+
|
|
130
|
+
| Job | Fields |
|
|
131
|
+
|-----|--------|
|
|
132
|
+
| `vlookup_rollnumber` | `left_column`, `right_column`, `left_output_columns`, `right_output_columns` |
|
|
133
|
+
| `groupby_religion` | `group_column` |
|
|
134
|
+
| `vlookup_then_groupby` | All VLOOKUP fields + `group_column` |
|
|
135
|
+
| `mock_generator` | None |
|
|
136
|
+
|
|
137
|
+
## Important behavior
|
|
138
|
+
|
|
139
|
+
- Two-file jobs: lexical filename order determines left/right assignment.
|
|
140
|
+
- Atom failures and config cancellation move staged files to `rejected/`.
|
|
141
|
+
- Unexpected crashes may leave files in `staging/` — recover manually.
|
|
142
|
+
- Notifications are best-effort. Linux has no implementation.
|
|
143
|
+
- On success, macOS/Windows opens the output folder immediately (not a click
|
|
144
|
+
action).
|
|
145
|
+
|
|
146
|
+
## Adding a new business pipeline
|
|
147
|
+
|
|
148
|
+
1. Create or reuse an atom in `logic/atoms/` with `execute(params_json) -> str`
|
|
149
|
+
2. Create a pre-process op in `runners/ops.py` (with config init pattern)
|
|
150
|
+
3. Register in `pipeline.py`:
|
|
151
|
+
```python
|
|
152
|
+
my_pipeline = build_business_pipeline(
|
|
153
|
+
pipeline_name="my_pipeline",
|
|
154
|
+
atom_module=my_atom,
|
|
155
|
+
atom_label="My Pipeline",
|
|
156
|
+
pre_process_op=my_pre_process_op,
|
|
157
|
+
)
|
|
158
|
+
```
|
|
159
|
+
4. Add sensor in `definitions.py`:
|
|
160
|
+
```python
|
|
161
|
+
my_sensor = build_hot_folder_sensor("my_pipeline", "my_pipeline", min_files=1)
|
|
162
|
+
```
|
|
163
|
+
5. Folders are created automatically on import.
|
|
164
|
+
|
|
165
|
+
## Development status
|
|
166
|
+
|
|
167
|
+
No automated tests, lint, or pinned dependency versions. Validate by loading
|
|
168
|
+
definitions and exercising jobs with disposable CSV inputs.
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# helpers/
|
|
2
|
+
|
|
3
|
+
Cross-cutting utilities for file management, config, UI, and notifications.
|
|
4
|
+
|
|
5
|
+
## `folders.py`
|
|
6
|
+
|
|
7
|
+
`PipelineFolders(pipeline_name)` — per-pipeline directory lifecycle:
|
|
8
|
+
- Properties: `.inbox`, `.staging`, `.processed`, `.rejected`, `.output`,
|
|
9
|
+
`.config_path`
|
|
10
|
+
- Methods: `.ensure()`, `.move_to_staging()`, `.move_to_processed()`,
|
|
11
|
+
`.move_to_rejected(file_paths, reason)`, `.list_inbox_files(pattern)`,
|
|
12
|
+
`.output_path(filename)`
|
|
13
|
+
|
|
14
|
+
## `config_store.py`
|
|
15
|
+
|
|
16
|
+
Per-pipeline JSON config persistence:
|
|
17
|
+
- `load_config(folders)` — Returns saved dict or None
|
|
18
|
+
- `save_config(folders, config)` — Writes `config.json`
|
|
19
|
+
- `config_exists(folders)` — Boolean check
|
|
20
|
+
|
|
21
|
+
## `column_picker.py`
|
|
22
|
+
|
|
23
|
+
Tkinter dialog for VLOOKUP join/output column selection. Shows columns from
|
|
24
|
+
both files with dtypes, validates type compatibility on join columns, and
|
|
25
|
+
allows multi-select for output columns from both sides.
|
|
26
|
+
|
|
27
|
+
## `notifier.py`
|
|
28
|
+
|
|
29
|
+
Best-effort native OS notifications:
|
|
30
|
+
- macOS: AppleScript via `osascript`
|
|
31
|
+
- Windows: PowerShell toast API
|
|
32
|
+
- Opens output folder immediately on success (not a click action)
|
|
33
|
+
- Fails silently on unsupported platforms (Linux)
|
|
34
|
+
|
|
35
|
+
## `file_picker.py`
|
|
36
|
+
|
|
37
|
+
Standalone Tkinter file-selection helper. Not used by registered Dagster jobs
|
|
38
|
+
(the sensor provides files). Available for manual/ad-hoc use.
|
|
39
|
+
|
|
40
|
+
## `mock_generator.py`
|
|
41
|
+
|
|
42
|
+
Standalone two-file CLI mock-data generator. This is separate from the
|
|
43
|
+
registered `mock_generator` atom/job — it predates the Dagster integration.
|
|
File without changes
|