ETLai 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,193 @@
1
+ Metadata-Version: 2.4
2
+ Name: ETLai
3
+ Version: 0.1.0
4
+ Summary: Local, folder-driven Dagster application for CSV transformations with hot-folder sensors and Tkinter configuration UI.
5
+ Author: Umangjeet Singh Pahwa
6
+ License: MIT
7
+ Project-URL: Repository, https://github.com/umang/ETLai
8
+ Classifier: Development Status :: 3 - Alpha
9
+ Classifier: Intended Audience :: Developers
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Programming Language :: Python :: 3
12
+ Classifier: Programming Language :: Python :: 3.10
13
+ Classifier: Programming Language :: Python :: 3.11
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Classifier: Topic :: Scientific/Engineering
16
+ Requires-Python: >=3.10
17
+ Description-Content-Type: text/markdown
18
+ License-File: LICENSE
19
+ Requires-Dist: dagster>=1.5
20
+ Requires-Dist: dagster-webserver>=1.5
21
+ Requires-Dist: pandas>=2.0
22
+ Requires-Dist: faker>=18.0
23
+ Requires-Dist: openpyxl>=3.1
24
+ Dynamic: license-file
25
+
26
+ # Air-Gapped Desktop Automation Engine
27
+
28
+ A local Dagster application for folder-driven CSV transformations. Source files,
29
+ saved column choices, generated outputs, and Dagster run storage remain on the
30
+ machine.
31
+
32
+ ## How it works
33
+
34
+ Each business pipeline has an inbox and a Dagster sensor. The sensor waits for
35
+ the required number of stable files, moves them to staging, and launches a job.
36
+ The job prepares atom parameters, executes the transformation, then moves source
37
+ files to `processed/` or `rejected/`.
38
+
39
+ An **atom** is a reusable transformation module with an
40
+ `execute(params_json) -> result_json` interface. Atoms are domain-agnostic —
41
+ they have no opinion on which column or business context. They do read and write
42
+ files, so they are not side-effect-free.
43
+
44
+ A **business pipeline** applies an atom to specific domain data with its own
45
+ folder lifecycle, sensor, and saved configuration.
46
+
47
+ ## Registered jobs
48
+
49
+ | Job | Files needed | Description |
50
+ |-----|-------------|-------------|
51
+ | `vlookup_rollnumber` | 2 CSV | Left-join on user-chosen columns (first run: Tkinter picker, then saved) |
52
+ | `groupby_religion` | 1 CSV | Group by user-chosen column with counts, sorted descending |
53
+ | `mock_generator` | 1+ CSV | Generate synthetic data from file headers using Faker |
54
+ | `vlookup_then_groupby` | 2 CSV | Composite: VLOOKUP → GroupBy in a single job |
55
+
56
+ Each job has a same-named sensor. Sensors must be enabled in Dagster UI before
57
+ they process inbox files.
58
+
59
+ ## Installation
60
+
61
+ Python 3.10+ required.
62
+
63
+ ```bash
64
+ python3 -m venv .venv
65
+ source .venv/bin/activate # Windows: .venv\Scripts\activate
66
+ python -m pip install -r requirements.txt
67
+ ```
68
+
69
+ Tkinter must be available in the Python installation for first-run config.
70
+
71
+ ## Run
72
+
73
+ ```bash
74
+ DAGSTER_HOME="$(pwd)" dagster dev -m definitions
75
+ ```
76
+
77
+ Open `http://localhost:3000`, enable the desired sensor, and place files in
78
+ `pipelines/<job-name>/inbox/`.
79
+
80
+ Manual execution without UI:
81
+
82
+ ```bash
83
+ DAGSTER_HOME="$(pwd)" dagster job execute -m definitions -j vlookup_rollnumber
84
+ ```
85
+
86
+ ## Source modules
87
+
88
+ ### Entry points
89
+
90
+ | File | Role |
91
+ |------|------|
92
+ | `definitions.py` | Registers all jobs and sensors with Dagster |
93
+ | `pipeline.py` | Builds single-atom business jobs via the factory |
94
+ | `runners/composite.py` | Defines the `vlookup_then_groupby` composite job |
95
+
96
+ ### `logic/atoms/`
97
+
98
+ | Module | Purpose |
99
+ |--------|---------|
100
+ | `vlookup.py` | Left join two CSVs on specified columns with dtype validation |
101
+ | `groupby.py` | Group by one column, count, sort descending |
102
+ | `mock_generate.py` | Infer Faker generators from headers, write synthetic CSVs |
103
+
104
+ ### `runners/`
105
+
106
+ | Module | Purpose |
107
+ |--------|---------|
108
+ | `pipeline_factory.py` | Assembles load → preprocess → execute Dagster jobs |
109
+ | `atom_runner.py` | Reusable load/execute ops with processed/rejected/notification lifecycle |
110
+ | `ops.py` | First-run config ops for VLOOKUP, GroupBy, and mock generator |
111
+ | `composite.py` | VLOOKUP → GroupBy chain with shared config |
112
+
113
+ ### `sensors/`
114
+
115
+ | Module | Purpose |
116
+ |--------|---------|
117
+ | `hot_folder_sensor.py` | Per-pipeline sensor factory: stability check, staging, sweeper |
118
+
119
+ ### `helpers/`
120
+
121
+ | Module | Purpose |
122
+ |--------|---------|
123
+ | `folders.py` | `PipelineFolders` class — per-pipeline directory lifecycle |
124
+ | `config_store.py` | JSON config read/write per pipeline |
125
+ | `column_picker.py` | Tkinter UI for VLOOKUP join/output column selection |
126
+ | `notifier.py` | macOS/Windows toast notifications, opens output folder on success |
127
+ | `file_picker.py` | Standalone Tkinter file dialog (not used by registered jobs) |
128
+ | `mock_generator.py` | Standalone CLI mock generator (separate from the atom/job) |
129
+
130
+ ### Runtime data
131
+
132
+ - `pipelines/<job>/` — inbox, staging, processed, rejected, output, config.json
133
+ - `dagster.yaml` — SQLite storage configuration
134
+ - `dagster_storage/` — Dagster run history (persists across restarts)
135
+
136
+ ## File and configuration lifecycle
137
+
138
+ ```text
139
+ inbox/ → sensor stability check → staging/ → job runs
140
+ ├→ processed/ + output/ (success)
141
+ └→ rejected/ + *.error.txt (failure)
142
+ ```
143
+
144
+ Non-matching files (not `*.csv`/`*.xlsx`) are swept to `rejected/` after 3
145
+ minutes with an error description.
146
+
147
+ ### Config init pattern
148
+
149
+ 1. First run: pre-process op detects no `config.json` → shows Tkinter UI → saves
150
+ 2. Future runs: loads config silently, fully automated
151
+ 3. To reconfigure: delete `pipelines/<job>/config.json`
152
+
153
+ ### Saved config fields
154
+
155
+ | Job | Fields |
156
+ |-----|--------|
157
+ | `vlookup_rollnumber` | `left_column`, `right_column`, `left_output_columns`, `right_output_columns` |
158
+ | `groupby_religion` | `group_column` |
159
+ | `vlookup_then_groupby` | All VLOOKUP fields + `group_column` |
160
+ | `mock_generator` | None |
161
+
162
+ ## Important behavior
163
+
164
+ - Two-file jobs: lexical filename order determines left/right assignment.
165
+ - Atom failures and config cancellation move staged files to `rejected/`.
166
+ - Unexpected crashes may leave files in `staging/` — recover manually.
167
+ - Notifications are best-effort. Linux has no implementation.
168
+ - On success, macOS/Windows opens the output folder immediately (not a click
169
+ action).
170
+
171
+ ## Adding a new business pipeline
172
+
173
+ 1. Create or reuse an atom in `logic/atoms/` with `execute(params_json) -> str`
174
+ 2. Create a pre-process op in `runners/ops.py` (with config init pattern)
175
+ 3. Register in `pipeline.py`:
176
+ ```python
177
+ my_pipeline = build_business_pipeline(
178
+ pipeline_name="my_pipeline",
179
+ atom_module=my_atom,
180
+ atom_label="My Pipeline",
181
+ pre_process_op=my_pre_process_op,
182
+ )
183
+ ```
184
+ 4. Add sensor in `definitions.py`:
185
+ ```python
186
+ my_sensor = build_hot_folder_sensor("my_pipeline", "my_pipeline", min_files=1)
187
+ ```
188
+ 5. Folders are created automatically on import.
189
+
190
+ ## Development status
191
+
192
+ No automated tests, lint, or pinned dependency versions. Validate by loading
193
+ definitions and exercising jobs with disposable CSV inputs.
@@ -0,0 +1,31 @@
1
+ LICENSE
2
+ README.md
3
+ pyproject.toml
4
+ ETLai.egg-info/PKG-INFO
5
+ ETLai.egg-info/SOURCES.txt
6
+ ETLai.egg-info/dependency_links.txt
7
+ ETLai.egg-info/requires.txt
8
+ ETLai.egg-info/top_level.txt
9
+ helpers/README.md
10
+ helpers/__init__.py
11
+ helpers/column_picker.py
12
+ helpers/config_store.py
13
+ helpers/file_picker.py
14
+ helpers/folders.py
15
+ helpers/mock_generator.py
16
+ helpers/notifier.py
17
+ logic/README.md
18
+ logic/__init__.py
19
+ logic/atoms/README.md
20
+ logic/atoms/__init__.py
21
+ logic/atoms/groupby.py
22
+ logic/atoms/vlookup.py
23
+ runners/README.md
24
+ runners/__init__.py
25
+ runners/atom_runner.py
26
+ runners/composite.py
27
+ runners/ops.py
28
+ runners/pipeline_factory.py
29
+ sensors/README.md
30
+ sensors/__init__.py
31
+ sensors/hot_folder_sensor.py
@@ -0,0 +1,5 @@
1
+ dagster>=1.5
2
+ dagster-webserver>=1.5
3
+ pandas>=2.0
4
+ faker>=18.0
5
+ openpyxl>=3.1
@@ -0,0 +1,4 @@
1
+ helpers
2
+ logic
3
+ runners
4
+ sensors
etlai-0.1.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Umangjeet Singh Pahwa
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
etlai-0.1.0/PKG-INFO ADDED
@@ -0,0 +1,193 @@
1
+ Metadata-Version: 2.4
2
+ Name: ETLai
3
+ Version: 0.1.0
4
+ Summary: Local, folder-driven Dagster application for CSV transformations with hot-folder sensors and Tkinter configuration UI.
5
+ Author: Umangjeet Singh Pahwa
6
+ License: MIT
7
+ Project-URL: Repository, https://github.com/umang/ETLai
8
+ Classifier: Development Status :: 3 - Alpha
9
+ Classifier: Intended Audience :: Developers
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Programming Language :: Python :: 3
12
+ Classifier: Programming Language :: Python :: 3.10
13
+ Classifier: Programming Language :: Python :: 3.11
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Classifier: Topic :: Scientific/Engineering
16
+ Requires-Python: >=3.10
17
+ Description-Content-Type: text/markdown
18
+ License-File: LICENSE
19
+ Requires-Dist: dagster>=1.5
20
+ Requires-Dist: dagster-webserver>=1.5
21
+ Requires-Dist: pandas>=2.0
22
+ Requires-Dist: faker>=18.0
23
+ Requires-Dist: openpyxl>=3.1
24
+ Dynamic: license-file
25
+
26
+ # Air-Gapped Desktop Automation Engine
27
+
28
+ A local Dagster application for folder-driven CSV transformations. Source files,
29
+ saved column choices, generated outputs, and Dagster run storage remain on the
30
+ machine.
31
+
32
+ ## How it works
33
+
34
+ Each business pipeline has an inbox and a Dagster sensor. The sensor waits for
35
+ the required number of stable files, moves them to staging, and launches a job.
36
+ The job prepares atom parameters, executes the transformation, then moves source
37
+ files to `processed/` or `rejected/`.
38
+
39
+ An **atom** is a reusable transformation module with an
40
+ `execute(params_json) -> result_json` interface. Atoms are domain-agnostic —
41
+ they have no opinion on which column or business context. They do read and write
42
+ files, so they are not side-effect-free.
43
+
44
+ A **business pipeline** applies an atom to specific domain data with its own
45
+ folder lifecycle, sensor, and saved configuration.
46
+
47
+ ## Registered jobs
48
+
49
+ | Job | Files needed | Description |
50
+ |-----|-------------|-------------|
51
+ | `vlookup_rollnumber` | 2 CSV | Left-join on user-chosen columns (first run: Tkinter picker, then saved) |
52
+ | `groupby_religion` | 1 CSV | Group by user-chosen column with counts, sorted descending |
53
+ | `mock_generator` | 1+ CSV | Generate synthetic data from file headers using Faker |
54
+ | `vlookup_then_groupby` | 2 CSV | Composite: VLOOKUP → GroupBy in a single job |
55
+
56
+ Each job has a same-named sensor. Sensors must be enabled in Dagster UI before
57
+ they process inbox files.
58
+
59
+ ## Installation
60
+
61
+ Python 3.10+ required.
62
+
63
+ ```bash
64
+ python3 -m venv .venv
65
+ source .venv/bin/activate # Windows: .venv\Scripts\activate
66
+ python -m pip install -r requirements.txt
67
+ ```
68
+
69
+ Tkinter must be available in the Python installation for first-run config.
70
+
71
+ ## Run
72
+
73
+ ```bash
74
+ DAGSTER_HOME="$(pwd)" dagster dev -m definitions
75
+ ```
76
+
77
+ Open `http://localhost:3000`, enable the desired sensor, and place files in
78
+ `pipelines/<job-name>/inbox/`.
79
+
80
+ Manual execution without UI:
81
+
82
+ ```bash
83
+ DAGSTER_HOME="$(pwd)" dagster job execute -m definitions -j vlookup_rollnumber
84
+ ```
85
+
86
+ ## Source modules
87
+
88
+ ### Entry points
89
+
90
+ | File | Role |
91
+ |------|------|
92
+ | `definitions.py` | Registers all jobs and sensors with Dagster |
93
+ | `pipeline.py` | Builds single-atom business jobs via the factory |
94
+ | `runners/composite.py` | Defines the `vlookup_then_groupby` composite job |
95
+
96
+ ### `logic/atoms/`
97
+
98
+ | Module | Purpose |
99
+ |--------|---------|
100
+ | `vlookup.py` | Left join two CSVs on specified columns with dtype validation |
101
+ | `groupby.py` | Group by one column, count, sort descending |
102
+ | `mock_generate.py` | Infer Faker generators from headers, write synthetic CSVs |
103
+
104
+ ### `runners/`
105
+
106
+ | Module | Purpose |
107
+ |--------|---------|
108
+ | `pipeline_factory.py` | Assembles load → preprocess → execute Dagster jobs |
109
+ | `atom_runner.py` | Reusable load/execute ops with processed/rejected/notification lifecycle |
110
+ | `ops.py` | First-run config ops for VLOOKUP, GroupBy, and mock generator |
111
+ | `composite.py` | VLOOKUP → GroupBy chain with shared config |
112
+
113
+ ### `sensors/`
114
+
115
+ | Module | Purpose |
116
+ |--------|---------|
117
+ | `hot_folder_sensor.py` | Per-pipeline sensor factory: stability check, staging, sweeper |
118
+
119
+ ### `helpers/`
120
+
121
+ | Module | Purpose |
122
+ |--------|---------|
123
+ | `folders.py` | `PipelineFolders` class — per-pipeline directory lifecycle |
124
+ | `config_store.py` | JSON config read/write per pipeline |
125
+ | `column_picker.py` | Tkinter UI for VLOOKUP join/output column selection |
126
+ | `notifier.py` | macOS/Windows toast notifications, opens output folder on success |
127
+ | `file_picker.py` | Standalone Tkinter file dialog (not used by registered jobs) |
128
+ | `mock_generator.py` | Standalone CLI mock generator (separate from the atom/job) |
129
+
130
+ ### Runtime data
131
+
132
+ - `pipelines/<job>/` — inbox, staging, processed, rejected, output, config.json
133
+ - `dagster.yaml` — SQLite storage configuration
134
+ - `dagster_storage/` — Dagster run history (persists across restarts)
135
+
136
+ ## File and configuration lifecycle
137
+
138
+ ```text
139
+ inbox/ → sensor stability check → staging/ → job runs
140
+ ├→ processed/ + output/ (success)
141
+ └→ rejected/ + *.error.txt (failure)
142
+ ```
143
+
144
+ Non-matching files (not `*.csv`/`*.xlsx`) are swept to `rejected/` after 3
145
+ minutes with an error description.
146
+
147
+ ### Config init pattern
148
+
149
+ 1. First run: pre-process op detects no `config.json` → shows Tkinter UI → saves
150
+ 2. Future runs: loads config silently, fully automated
151
+ 3. To reconfigure: delete `pipelines/<job>/config.json`
152
+
153
+ ### Saved config fields
154
+
155
+ | Job | Fields |
156
+ |-----|--------|
157
+ | `vlookup_rollnumber` | `left_column`, `right_column`, `left_output_columns`, `right_output_columns` |
158
+ | `groupby_religion` | `group_column` |
159
+ | `vlookup_then_groupby` | All VLOOKUP fields + `group_column` |
160
+ | `mock_generator` | None |
161
+
162
+ ## Important behavior
163
+
164
+ - Two-file jobs: lexical filename order determines left/right assignment.
165
+ - Atom failures and config cancellation move staged files to `rejected/`.
166
+ - Unexpected crashes may leave files in `staging/` — recover manually.
167
+ - Notifications are best-effort. Linux has no implementation.
168
+ - On success, macOS/Windows opens the output folder immediately (not a click
169
+ action).
170
+
171
+ ## Adding a new business pipeline
172
+
173
+ 1. Create or reuse an atom in `logic/atoms/` with `execute(params_json) -> str`
174
+ 2. Create a pre-process op in `runners/ops.py` (with config init pattern)
175
+ 3. Register in `pipeline.py`:
176
+ ```python
177
+ my_pipeline = build_business_pipeline(
178
+ pipeline_name="my_pipeline",
179
+ atom_module=my_atom,
180
+ atom_label="My Pipeline",
181
+ pre_process_op=my_pre_process_op,
182
+ )
183
+ ```
184
+ 4. Add sensor in `definitions.py`:
185
+ ```python
186
+ my_sensor = build_hot_folder_sensor("my_pipeline", "my_pipeline", min_files=1)
187
+ ```
188
+ 5. Folders are created automatically on import.
189
+
190
+ ## Development status
191
+
192
+ No automated tests, lint, or pinned dependency versions. Validate by loading
193
+ definitions and exercising jobs with disposable CSV inputs.
etlai-0.1.0/README.md ADDED
@@ -0,0 +1,168 @@
1
+ # Air-Gapped Desktop Automation Engine
2
+
3
+ A local Dagster application for folder-driven CSV transformations. Source files,
4
+ saved column choices, generated outputs, and Dagster run storage remain on the
5
+ machine.
6
+
7
+ ## How it works
8
+
9
+ Each business pipeline has an inbox and a Dagster sensor. The sensor waits for
10
+ the required number of stable files, moves them to staging, and launches a job.
11
+ The job prepares atom parameters, executes the transformation, then moves source
12
+ files to `processed/` or `rejected/`.
13
+
14
+ An **atom** is a reusable transformation module with an
15
+ `execute(params_json) -> result_json` interface. Atoms are domain-agnostic —
16
+ they have no opinion on which column or business context. They do read and write
17
+ files, so they are not side-effect-free.
18
+
19
+ A **business pipeline** applies an atom to specific domain data with its own
20
+ folder lifecycle, sensor, and saved configuration.
21
+
22
+ ## Registered jobs
23
+
24
+ | Job | Files needed | Description |
25
+ |-----|-------------|-------------|
26
+ | `vlookup_rollnumber` | 2 CSV | Left-join on user-chosen columns (first run: Tkinter picker, then saved) |
27
+ | `groupby_religion` | 1 CSV | Group by user-chosen column with counts, sorted descending |
28
+ | `mock_generator` | 1+ CSV | Generate synthetic data from file headers using Faker |
29
+ | `vlookup_then_groupby` | 2 CSV | Composite: VLOOKUP → GroupBy in a single job |
30
+
31
+ Each job has a same-named sensor. Sensors must be enabled in Dagster UI before
32
+ they process inbox files.
33
+
34
+ ## Installation
35
+
36
+ Python 3.10+ required.
37
+
38
+ ```bash
39
+ python3 -m venv .venv
40
+ source .venv/bin/activate # Windows: .venv\Scripts\activate
41
+ python -m pip install -r requirements.txt
42
+ ```
43
+
44
+ Tkinter must be available in the Python installation for first-run config.
45
+
46
+ ## Run
47
+
48
+ ```bash
49
+ DAGSTER_HOME="$(pwd)" dagster dev -m definitions
50
+ ```
51
+
52
+ Open `http://localhost:3000`, enable the desired sensor, and place files in
53
+ `pipelines/<job-name>/inbox/`.
54
+
55
+ Manual execution without UI:
56
+
57
+ ```bash
58
+ DAGSTER_HOME="$(pwd)" dagster job execute -m definitions -j vlookup_rollnumber
59
+ ```
60
+
61
+ ## Source modules
62
+
63
+ ### Entry points
64
+
65
+ | File | Role |
66
+ |------|------|
67
+ | `definitions.py` | Registers all jobs and sensors with Dagster |
68
+ | `pipeline.py` | Builds single-atom business jobs via the factory |
69
+ | `runners/composite.py` | Defines the `vlookup_then_groupby` composite job |
70
+
71
+ ### `logic/atoms/`
72
+
73
+ | Module | Purpose |
74
+ |--------|---------|
75
+ | `vlookup.py` | Left join two CSVs on specified columns with dtype validation |
76
+ | `groupby.py` | Group by one column, count, sort descending |
77
+ | `mock_generate.py` | Infer Faker generators from headers, write synthetic CSVs |
78
+
79
+ ### `runners/`
80
+
81
+ | Module | Purpose |
82
+ |--------|---------|
83
+ | `pipeline_factory.py` | Assembles load → preprocess → execute Dagster jobs |
84
+ | `atom_runner.py` | Reusable load/execute ops with processed/rejected/notification lifecycle |
85
+ | `ops.py` | First-run config ops for VLOOKUP, GroupBy, and mock generator |
86
+ | `composite.py` | VLOOKUP → GroupBy chain with shared config |
87
+
88
+ ### `sensors/`
89
+
90
+ | Module | Purpose |
91
+ |--------|---------|
92
+ | `hot_folder_sensor.py` | Per-pipeline sensor factory: stability check, staging, sweeper |
93
+
94
+ ### `helpers/`
95
+
96
+ | Module | Purpose |
97
+ |--------|---------|
98
+ | `folders.py` | `PipelineFolders` class — per-pipeline directory lifecycle |
99
+ | `config_store.py` | JSON config read/write per pipeline |
100
+ | `column_picker.py` | Tkinter UI for VLOOKUP join/output column selection |
101
+ | `notifier.py` | macOS/Windows toast notifications, opens output folder on success |
102
+ | `file_picker.py` | Standalone Tkinter file dialog (not used by registered jobs) |
103
+ | `mock_generator.py` | Standalone CLI mock generator (separate from the atom/job) |
104
+
105
+ ### Runtime data
106
+
107
+ - `pipelines/<job>/` — inbox, staging, processed, rejected, output, config.json
108
+ - `dagster.yaml` — SQLite storage configuration
109
+ - `dagster_storage/` — Dagster run history (persists across restarts)
110
+
111
+ ## File and configuration lifecycle
112
+
113
+ ```text
114
+ inbox/ → sensor stability check → staging/ → job runs
115
+ ├→ processed/ + output/ (success)
116
+ └→ rejected/ + *.error.txt (failure)
117
+ ```
118
+
119
+ Non-matching files (not `*.csv`/`*.xlsx`) are swept to `rejected/` after 3
120
+ minutes with an error description.
121
+
122
+ ### Config init pattern
123
+
124
+ 1. First run: pre-process op detects no `config.json` → shows Tkinter UI → saves
125
+ 2. Future runs: loads config silently, fully automated
126
+ 3. To reconfigure: delete `pipelines/<job>/config.json`
127
+
128
+ ### Saved config fields
129
+
130
+ | Job | Fields |
131
+ |-----|--------|
132
+ | `vlookup_rollnumber` | `left_column`, `right_column`, `left_output_columns`, `right_output_columns` |
133
+ | `groupby_religion` | `group_column` |
134
+ | `vlookup_then_groupby` | All VLOOKUP fields + `group_column` |
135
+ | `mock_generator` | None |
136
+
137
+ ## Important behavior
138
+
139
+ - Two-file jobs: lexical filename order determines left/right assignment.
140
+ - Atom failures and config cancellation move staged files to `rejected/`.
141
+ - Unexpected crashes may leave files in `staging/` — recover manually.
142
+ - Notifications are best-effort. Linux has no implementation.
143
+ - On success, macOS/Windows opens the output folder immediately (not a click
144
+ action).
145
+
146
+ ## Adding a new business pipeline
147
+
148
+ 1. Create or reuse an atom in `logic/atoms/` with `execute(params_json) -> str`
149
+ 2. Create a pre-process op in `runners/ops.py` (with config init pattern)
150
+ 3. Register in `pipeline.py`:
151
+ ```python
152
+ my_pipeline = build_business_pipeline(
153
+ pipeline_name="my_pipeline",
154
+ atom_module=my_atom,
155
+ atom_label="My Pipeline",
156
+ pre_process_op=my_pre_process_op,
157
+ )
158
+ ```
159
+ 4. Add sensor in `definitions.py`:
160
+ ```python
161
+ my_sensor = build_hot_folder_sensor("my_pipeline", "my_pipeline", min_files=1)
162
+ ```
163
+ 5. Folders are created automatically on import.
164
+
165
+ ## Development status
166
+
167
+ No automated tests, lint, or pinned dependency versions. Validate by loading
168
+ definitions and exercising jobs with disposable CSV inputs.
@@ -0,0 +1,43 @@
1
+ # helpers/
2
+
3
+ Cross-cutting utilities for file management, config, UI, and notifications.
4
+
5
+ ## `folders.py`
6
+
7
+ `PipelineFolders(pipeline_name)` — per-pipeline directory lifecycle:
8
+ - Properties: `.inbox`, `.staging`, `.processed`, `.rejected`, `.output`,
9
+ `.config_path`
10
+ - Methods: `.ensure()`, `.move_to_staging()`, `.move_to_processed()`,
11
+ `.move_to_rejected(file_paths, reason)`, `.list_inbox_files(pattern)`,
12
+ `.output_path(filename)`
13
+
14
+ ## `config_store.py`
15
+
16
+ Per-pipeline JSON config persistence:
17
+ - `load_config(folders)` — Returns saved dict or None
18
+ - `save_config(folders, config)` — Writes `config.json`
19
+ - `config_exists(folders)` — Boolean check
20
+
21
+ ## `column_picker.py`
22
+
23
+ Tkinter dialog for VLOOKUP join/output column selection. Shows columns from
24
+ both files with dtypes, validates type compatibility on join columns, and
25
+ allows multi-select for output columns from both sides.
26
+
27
+ ## `notifier.py`
28
+
29
+ Best-effort native OS notifications:
30
+ - macOS: AppleScript via `osascript`
31
+ - Windows: PowerShell toast API
32
+ - Opens output folder immediately on success (not a click action)
33
+ - Fails silently on unsupported platforms (Linux)
34
+
35
+ ## `file_picker.py`
36
+
37
+ Standalone Tkinter file-selection helper. Not used by registered Dagster jobs
38
+ (the sensor provides files). Available for manual/ad-hoc use.
39
+
40
+ ## `mock_generator.py`
41
+
42
+ Standalone two-file CLI mock-data generator. This is separate from the
43
+ registered `mock_generator` atom/job — it predates the Dagster integration.
File without changes