ping-dataexport 0.2.2__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {ping_dataexport-0.2.2/src/ping_dataexport.egg-info → ping_dataexport-0.3.0}/PKG-INFO +34 -18
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/README.md +33 -17
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/pyproject.toml +1 -1
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/__init__.py +1 -1
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/cli.py +4 -3
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/job/validator.py +3 -3
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0/src/ping_dataexport.egg-info}/PKG-INFO +34 -18
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/LICENSE +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/MANIFEST.in +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/config.sample.ini +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/job.sample.ini +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/setup.cfg +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/__config__.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/__main__.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/api.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/applogger.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/config.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/db/__init__.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/db/connector.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/db/oracle_connector.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/db/pyodbc_connector.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/db/sqlite_connector.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/db/type_map.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/errors.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/export/__init__.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/export/engine.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/export/options.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/export/runner.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/export/util.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/job/__init__.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/job/executor.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/job/shared_context.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/job/spec.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/paths.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/planner/__init__.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/planner/base.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/planner/datelist.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/planner/groupby.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/planner/monthbydate.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/planner/single.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/planner/watermark.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/writer/__init__.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/writer/arrow_map.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/writer/base.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/writer/delimited.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport/writer/parquet.py +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport.egg-info/SOURCES.txt +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport.egg-info/dependency_links.txt +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport.egg-info/entry_points.txt +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport.egg-info/requires.txt +0 -0
- {ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: ping-dataexport
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: PING's Data Export Tool - To Parquet/CSV/TEXT as Accelerator for Data & AI Project
|
|
5
5
|
Author: vorapol
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -46,7 +46,7 @@ Dynamic: license-file
|
|
|
46
46
|
`decimal128(p,s)`, คอลัมน์ int ที่มี NULL ยังคงเป็น int, วันที่ยังคงเป็นวันที่
|
|
47
47
|
และไม่ใช้ pandas เด็ดขาด
|
|
48
48
|
- **ฐานข้อมูลที่รองรับ** — SQLite (Library มาตรฐาน ไม่ต้องติดตั้งอะไรเพิ่ม),
|
|
49
|
-
SQL Server (MS ODBC / FreeTDS), Oracle (โหมด thin
|
|
49
|
+
SQL Server (MS ODBC / FreeTDS), Oracle (โหมด thin แบบ Native หรือ ODBC)
|
|
50
50
|
และแหล่งข้อมูล ODBC ใดก็ได้
|
|
51
51
|
- **ใช้งานได้สองทาง** — ผ่าน CLI `ping-dataexport` หรือผ่าน Library API (`ping_dataexport.api`)
|
|
52
52
|
ที่ออกแบบมาสำหรับ Airflow
|
|
@@ -151,7 +151,7 @@ PasswordBase64=c2VjcmV0 ; base64("secret")
|
|
|
151
151
|
file defaults to `./config.ini` (`-cf` to point elsewhere).
|
|
152
152
|
|
|
153
153
|
```bash
|
|
154
|
-
# preview to the console (no -o/-
|
|
154
|
+
# preview to the console (no -o/-dir): prints the first rows
|
|
155
155
|
ping-dataexport -s SQLITE01 -q "SELECT * FROM sales"
|
|
156
156
|
|
|
157
157
|
# single file — format inferred from the extension (.csv/.txt/.parquet)
|
|
@@ -169,11 +169,11 @@ All options:
|
|
|
169
169
|
| `-s` | `--source` | DB config section name (e.g. `DB01`) |
|
|
170
170
|
| `-q` | `--query` | SQL query |
|
|
171
171
|
| `-o` | `--output` | single output file (format from extension, or `-f`) |
|
|
172
|
-
| `-
|
|
173
|
-
| `-
|
|
172
|
+
| `-dir` | `--outputdir` | output directory (required for `-m` modes); `-od` still works as a legacy alias |
|
|
173
|
+
| `-pf` | `--prefix` | output filename prefix for mode exports; `-bn`/`--basename` still work as legacy aliases |
|
|
174
174
|
| `-f` | `--format` | `csv`, `txt`, `parquet` |
|
|
175
175
|
| `-gz` | `--gzip` | gzip the output (CSV/TXT only) |
|
|
176
|
-
| `-
|
|
176
|
+
| `-sep` | `--sep` | field separator (default `,`; `\t` maps to tab) |
|
|
177
177
|
| `-quo` | `--quote` | quote character (default `"`) |
|
|
178
178
|
| `-stnl` | `--stripnewline` | newline handling in text columns: `space` \| `blank` \| `escape` \| `doubleescape` |
|
|
179
179
|
| `-stmc` | `--stripmetachar` | strip invisible/control characters from text columns |
|
|
@@ -194,15 +194,31 @@ All options:
|
|
|
194
194
|
| `-lf` | `--logfile` | log file path |
|
|
195
195
|
| `-t` | `--test` | test mode: validate + preview, no files |
|
|
196
196
|
|
|
197
|
+
### Which options must go together
|
|
198
|
+
|
|
199
|
+
| If you use… | You must also have… | Notes |
|
|
200
|
+
|-------------|--------------------|-------|
|
|
201
|
+
| (anything) | `-s` + `-q` | the two basics — always required, unless `-j` supplies them from a job file |
|
|
202
|
+
| `-o` | nothing else | single file; **cannot** be combined with `-dir`/`-pf` |
|
|
203
|
+
| `-m` (any mode) | `-dir` + `-pf` | every mode writes multiple files, so it needs a directory + filename prefix |
|
|
204
|
+
| `-dir` or `-pf` | `-m` | they only make sense for mode exports |
|
|
205
|
+
| `-m groupby` / `watermark` | `-col` | the column to group / track |
|
|
206
|
+
| `-m date` | `-col` + `-df` + `-dt` | the date column and an explicit range |
|
|
207
|
+
| `-m relativedate` | `-col` + `-dr` | `-df` optional (defaults to today) |
|
|
208
|
+
| `-m monthbydate` | `-col` + `-mr` | `-df` optional (defaults to today) |
|
|
209
|
+
| `-m splitrow` | `-row` | no `-col` needed |
|
|
210
|
+
| `-gz` | CSV/TXT output | rejected with parquet (it compresses itself) |
|
|
211
|
+
| no `-o` and no `-dir` | — | console preview only (first `PreviewRow` rows, nothing written) |
|
|
212
|
+
|
|
197
213
|
## 4. Export modes (partitioned outputs)
|
|
198
214
|
|
|
199
|
-
Modes split one query into multiple files. All need `-
|
|
200
|
-
+ `-
|
|
215
|
+
Modes split one query into multiple files. All need `-dir` (output directory)
|
|
216
|
+
+ `-pf` (filename prefix); partition values are always bound SQL parameters.
|
|
201
217
|
|
|
202
218
|
**`groupby` — one file per distinct value of a column:**
|
|
203
219
|
|
|
204
220
|
```bash
|
|
205
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m groupby -col region -
|
|
221
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m groupby -col region -dir by_region -pf sales
|
|
206
222
|
```
|
|
207
223
|
```
|
|
208
224
|
by_region/
|
|
@@ -217,7 +233,7 @@ by_region/
|
|
|
217
233
|
|
|
218
234
|
```bash
|
|
219
235
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m date -col txn_date \
|
|
220
|
-
-df 2026-01-01 -dt 2026-01-05 -
|
|
236
|
+
-df 2026-01-01 -dt 2026-01-05 -dir daily -pf sales
|
|
221
237
|
```
|
|
222
238
|
```
|
|
223
239
|
daily/
|
|
@@ -233,7 +249,7 @@ daily/
|
|
|
233
249
|
|
|
234
250
|
```bash
|
|
235
251
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m relativedate -col txn_date \
|
|
236
|
-
-df 2026-01-10 -dr 3 -
|
|
252
|
+
-df 2026-01-10 -dr 3 -dir rel -pf sales
|
|
237
253
|
```
|
|
238
254
|
```
|
|
239
255
|
rel/
|
|
@@ -248,7 +264,7 @@ at the anchor date; omit `-df` to anchor on today):**
|
|
|
248
264
|
|
|
249
265
|
```bash
|
|
250
266
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m monthbydate -col txn_date \
|
|
251
|
-
-df 2026-03-15 -mr 3 -
|
|
267
|
+
-df 2026-03-15 -mr 3 -dir monthly -pf sales
|
|
252
268
|
```
|
|
253
269
|
```
|
|
254
270
|
monthly/
|
|
@@ -261,7 +277,7 @@ monthly/
|
|
|
261
277
|
**`splitrow` — rotate to a new file every N rows:**
|
|
262
278
|
|
|
263
279
|
```bash
|
|
264
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales ORDER BY id" -m splitrow -row 100000 -
|
|
280
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales ORDER BY id" -m splitrow -row 100000 -dir split -pf part
|
|
265
281
|
```
|
|
266
282
|
```
|
|
267
283
|
split/
|
|
@@ -273,7 +289,7 @@ split/
|
|
|
273
289
|
**`watermark` — incremental: only rows newer than the last run:**
|
|
274
290
|
|
|
275
291
|
```bash
|
|
276
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m watermark -col txn_date -
|
|
292
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m watermark -col txn_date -dir incr -pf sales
|
|
277
293
|
```
|
|
278
294
|
```
|
|
279
295
|
incr/
|
|
@@ -322,7 +338,7 @@ Output=out/full_dump.parquet
|
|
|
322
338
|
|
|
323
339
|
```bash
|
|
324
340
|
ping-dataexport -j DailySales # one section (job.ini by default, -jf elsewhere)
|
|
325
|
-
ping-dataexport -j DailySales -
|
|
341
|
+
ping-dataexport -j DailySales -dir other_dir # CLI options override the section
|
|
326
342
|
ping-dataexport -p 4 # ALL sections in parallel, 4 workers
|
|
327
343
|
```
|
|
328
344
|
|
|
@@ -333,11 +349,11 @@ Every job.ini key and the CLI option it mirrors (keys are case-insensitive):
|
|
|
333
349
|
| `Source` | `-s` / `--source` | which config.ini DB section to export from |
|
|
334
350
|
| `Query` | `-q` / `--query` | the SQL to run; indent continuation lines for multi-line queries |
|
|
335
351
|
| `Output` | `-o` / `--output` | single output file path |
|
|
336
|
-
| `OutputDir` | `-
|
|
337
|
-
| `BaseName` | `-
|
|
352
|
+
| `OutputDir` | `-dir` / `--outputdir` | output directory (needed when `Mode=` is set) |
|
|
353
|
+
| `BaseName` | `-pf` / `--prefix` | output filename prefix for mode exports |
|
|
338
354
|
| `Format` | `-f` / `--format` | `csv` / `txt` / `parquet` (else inferred from `Output` extension) |
|
|
339
355
|
| `Gzip` | `-gz` / `--gzip` | `Yes`/`No` — gzip the output (CSV/TXT only) |
|
|
340
|
-
| `Sep` | `-
|
|
356
|
+
| `Sep` | `-sep` / `--sep` | field separator (default `,`; `\t` = tab) |
|
|
341
357
|
| `Quote` | `-quo` / `--quote` | quote character (default `"`) |
|
|
342
358
|
| `StripNewLine` | `-stnl` / `--stripnewline` | `space` \| `blank` \| `escape` \| `doubleescape` |
|
|
343
359
|
| `StripMetaChar` | `-stmc` / `--stripmetachar` | `Yes`/`No` — strip invisible/control characters |
|
|
@@ -17,7 +17,7 @@
|
|
|
17
17
|
`decimal128(p,s)`, คอลัมน์ int ที่มี NULL ยังคงเป็น int, วันที่ยังคงเป็นวันที่
|
|
18
18
|
และไม่ใช้ pandas เด็ดขาด
|
|
19
19
|
- **ฐานข้อมูลที่รองรับ** — SQLite (Library มาตรฐาน ไม่ต้องติดตั้งอะไรเพิ่ม),
|
|
20
|
-
SQL Server (MS ODBC / FreeTDS), Oracle (โหมด thin
|
|
20
|
+
SQL Server (MS ODBC / FreeTDS), Oracle (โหมด thin แบบ Native หรือ ODBC)
|
|
21
21
|
และแหล่งข้อมูล ODBC ใดก็ได้
|
|
22
22
|
- **ใช้งานได้สองทาง** — ผ่าน CLI `ping-dataexport` หรือผ่าน Library API (`ping_dataexport.api`)
|
|
23
23
|
ที่ออกแบบมาสำหรับ Airflow
|
|
@@ -122,7 +122,7 @@ PasswordBase64=c2VjcmV0 ; base64("secret")
|
|
|
122
122
|
file defaults to `./config.ini` (`-cf` to point elsewhere).
|
|
123
123
|
|
|
124
124
|
```bash
|
|
125
|
-
# preview to the console (no -o/-
|
|
125
|
+
# preview to the console (no -o/-dir): prints the first rows
|
|
126
126
|
ping-dataexport -s SQLITE01 -q "SELECT * FROM sales"
|
|
127
127
|
|
|
128
128
|
# single file — format inferred from the extension (.csv/.txt/.parquet)
|
|
@@ -140,11 +140,11 @@ All options:
|
|
|
140
140
|
| `-s` | `--source` | DB config section name (e.g. `DB01`) |
|
|
141
141
|
| `-q` | `--query` | SQL query |
|
|
142
142
|
| `-o` | `--output` | single output file (format from extension, or `-f`) |
|
|
143
|
-
| `-
|
|
144
|
-
| `-
|
|
143
|
+
| `-dir` | `--outputdir` | output directory (required for `-m` modes); `-od` still works as a legacy alias |
|
|
144
|
+
| `-pf` | `--prefix` | output filename prefix for mode exports; `-bn`/`--basename` still work as legacy aliases |
|
|
145
145
|
| `-f` | `--format` | `csv`, `txt`, `parquet` |
|
|
146
146
|
| `-gz` | `--gzip` | gzip the output (CSV/TXT only) |
|
|
147
|
-
| `-
|
|
147
|
+
| `-sep` | `--sep` | field separator (default `,`; `\t` maps to tab) |
|
|
148
148
|
| `-quo` | `--quote` | quote character (default `"`) |
|
|
149
149
|
| `-stnl` | `--stripnewline` | newline handling in text columns: `space` \| `blank` \| `escape` \| `doubleescape` |
|
|
150
150
|
| `-stmc` | `--stripmetachar` | strip invisible/control characters from text columns |
|
|
@@ -165,15 +165,31 @@ All options:
|
|
|
165
165
|
| `-lf` | `--logfile` | log file path |
|
|
166
166
|
| `-t` | `--test` | test mode: validate + preview, no files |
|
|
167
167
|
|
|
168
|
+
### Which options must go together
|
|
169
|
+
|
|
170
|
+
| If you use… | You must also have… | Notes |
|
|
171
|
+
|-------------|--------------------|-------|
|
|
172
|
+
| (anything) | `-s` + `-q` | the two basics — always required, unless `-j` supplies them from a job file |
|
|
173
|
+
| `-o` | nothing else | single file; **cannot** be combined with `-dir`/`-pf` |
|
|
174
|
+
| `-m` (any mode) | `-dir` + `-pf` | every mode writes multiple files, so it needs a directory + filename prefix |
|
|
175
|
+
| `-dir` or `-pf` | `-m` | they only make sense for mode exports |
|
|
176
|
+
| `-m groupby` / `watermark` | `-col` | the column to group / track |
|
|
177
|
+
| `-m date` | `-col` + `-df` + `-dt` | the date column and an explicit range |
|
|
178
|
+
| `-m relativedate` | `-col` + `-dr` | `-df` optional (defaults to today) |
|
|
179
|
+
| `-m monthbydate` | `-col` + `-mr` | `-df` optional (defaults to today) |
|
|
180
|
+
| `-m splitrow` | `-row` | no `-col` needed |
|
|
181
|
+
| `-gz` | CSV/TXT output | rejected with parquet (it compresses itself) |
|
|
182
|
+
| no `-o` and no `-dir` | — | console preview only (first `PreviewRow` rows, nothing written) |
|
|
183
|
+
|
|
168
184
|
## 4. Export modes (partitioned outputs)
|
|
169
185
|
|
|
170
|
-
Modes split one query into multiple files. All need `-
|
|
171
|
-
+ `-
|
|
186
|
+
Modes split one query into multiple files. All need `-dir` (output directory)
|
|
187
|
+
+ `-pf` (filename prefix); partition values are always bound SQL parameters.
|
|
172
188
|
|
|
173
189
|
**`groupby` — one file per distinct value of a column:**
|
|
174
190
|
|
|
175
191
|
```bash
|
|
176
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m groupby -col region -
|
|
192
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m groupby -col region -dir by_region -pf sales
|
|
177
193
|
```
|
|
178
194
|
```
|
|
179
195
|
by_region/
|
|
@@ -188,7 +204,7 @@ by_region/
|
|
|
188
204
|
|
|
189
205
|
```bash
|
|
190
206
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m date -col txn_date \
|
|
191
|
-
-df 2026-01-01 -dt 2026-01-05 -
|
|
207
|
+
-df 2026-01-01 -dt 2026-01-05 -dir daily -pf sales
|
|
192
208
|
```
|
|
193
209
|
```
|
|
194
210
|
daily/
|
|
@@ -204,7 +220,7 @@ daily/
|
|
|
204
220
|
|
|
205
221
|
```bash
|
|
206
222
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m relativedate -col txn_date \
|
|
207
|
-
-df 2026-01-10 -dr 3 -
|
|
223
|
+
-df 2026-01-10 -dr 3 -dir rel -pf sales
|
|
208
224
|
```
|
|
209
225
|
```
|
|
210
226
|
rel/
|
|
@@ -219,7 +235,7 @@ at the anchor date; omit `-df` to anchor on today):**
|
|
|
219
235
|
|
|
220
236
|
```bash
|
|
221
237
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m monthbydate -col txn_date \
|
|
222
|
-
-df 2026-03-15 -mr 3 -
|
|
238
|
+
-df 2026-03-15 -mr 3 -dir monthly -pf sales
|
|
223
239
|
```
|
|
224
240
|
```
|
|
225
241
|
monthly/
|
|
@@ -232,7 +248,7 @@ monthly/
|
|
|
232
248
|
**`splitrow` — rotate to a new file every N rows:**
|
|
233
249
|
|
|
234
250
|
```bash
|
|
235
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales ORDER BY id" -m splitrow -row 100000 -
|
|
251
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales ORDER BY id" -m splitrow -row 100000 -dir split -pf part
|
|
236
252
|
```
|
|
237
253
|
```
|
|
238
254
|
split/
|
|
@@ -244,7 +260,7 @@ split/
|
|
|
244
260
|
**`watermark` — incremental: only rows newer than the last run:**
|
|
245
261
|
|
|
246
262
|
```bash
|
|
247
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m watermark -col txn_date -
|
|
263
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m watermark -col txn_date -dir incr -pf sales
|
|
248
264
|
```
|
|
249
265
|
```
|
|
250
266
|
incr/
|
|
@@ -293,7 +309,7 @@ Output=out/full_dump.parquet
|
|
|
293
309
|
|
|
294
310
|
```bash
|
|
295
311
|
ping-dataexport -j DailySales # one section (job.ini by default, -jf elsewhere)
|
|
296
|
-
ping-dataexport -j DailySales -
|
|
312
|
+
ping-dataexport -j DailySales -dir other_dir # CLI options override the section
|
|
297
313
|
ping-dataexport -p 4 # ALL sections in parallel, 4 workers
|
|
298
314
|
```
|
|
299
315
|
|
|
@@ -304,11 +320,11 @@ Every job.ini key and the CLI option it mirrors (keys are case-insensitive):
|
|
|
304
320
|
| `Source` | `-s` / `--source` | which config.ini DB section to export from |
|
|
305
321
|
| `Query` | `-q` / `--query` | the SQL to run; indent continuation lines for multi-line queries |
|
|
306
322
|
| `Output` | `-o` / `--output` | single output file path |
|
|
307
|
-
| `OutputDir` | `-
|
|
308
|
-
| `BaseName` | `-
|
|
323
|
+
| `OutputDir` | `-dir` / `--outputdir` | output directory (needed when `Mode=` is set) |
|
|
324
|
+
| `BaseName` | `-pf` / `--prefix` | output filename prefix for mode exports |
|
|
309
325
|
| `Format` | `-f` / `--format` | `csv` / `txt` / `parquet` (else inferred from `Output` extension) |
|
|
310
326
|
| `Gzip` | `-gz` / `--gzip` | `Yes`/`No` — gzip the output (CSV/TXT only) |
|
|
311
|
-
| `Sep` | `-
|
|
327
|
+
| `Sep` | `-sep` / `--sep` | field separator (default `,`; `\t` = tab) |
|
|
312
328
|
| `Quote` | `-quo` / `--quote` | quote character (default `"`) |
|
|
313
329
|
| `StripNewLine` | `-stnl` / `--stripnewline` | `space` \| `blank` \| `escape` \| `doubleescape` |
|
|
314
330
|
| `StripMetaChar` | `-stmc` / `--stripmetachar` | `Yes`/`No` — strip invisible/control characters |
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "ping-dataexport"
|
|
7
|
-
version = "0.
|
|
7
|
+
version = "0.3.0"
|
|
8
8
|
description = "PING's Data Export Tool - To Parquet/CSV/TEXT as Accelerator for Data & AI Project"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
license = "Apache-2.0"
|
|
@@ -58,15 +58,16 @@ def build_parser() -> argparse.ArgumentParser:
|
|
|
58
58
|
p.add_argument("-q", "--query", help="SQL query (required unless --job is used)")
|
|
59
59
|
|
|
60
60
|
p.add_argument("-o", "--output", help="single output file path")
|
|
61
|
-
p.add_argument("-od", "--outputdir", help="output directory (required with --mode)")
|
|
62
|
-
p.add_argument("-bn", "--basename",
|
|
61
|
+
p.add_argument("-dir", "-od", "--outputdir", help="output directory (required with --mode)")
|
|
62
|
+
p.add_argument("-pf", "-bn", "--prefix", "--basename", dest="basename",
|
|
63
|
+
help="output filename prefix for mode exports")
|
|
63
64
|
|
|
64
65
|
p.add_argument("-f", "--format", choices=FORMATS, help="output format (auto from extension)")
|
|
65
66
|
p.add_argument("-gz", "--gzip", action="store_true", default=None,
|
|
66
67
|
help="gzip compress output (csv/txt only)")
|
|
67
68
|
|
|
68
69
|
p.add_argument("-quo", "--quote", help='quote character (default ")')
|
|
69
|
-
p.add_argument("-
|
|
70
|
+
p.add_argument("-sep", "--sep", help="field separator (default ,) — \\t maps to tab")
|
|
70
71
|
p.add_argument("-stnl", "--stripnewline", choices=STRIP_NEWLINE_MODES,
|
|
71
72
|
help="newline handling in text columns")
|
|
72
73
|
p.add_argument("-stmc", "--stripmetachar", action="store_true", default=None,
|
|
@@ -25,15 +25,15 @@ class JobValidator:
|
|
|
25
25
|
e(spec, "--query is required (or provide Query in the job section)")
|
|
26
26
|
|
|
27
27
|
if spec.output and (spec.outputdir or spec.basename):
|
|
28
|
-
e(spec, "--output cannot be mixed with --outputdir/--
|
|
28
|
+
e(spec, "--output cannot be mixed with --outputdir/--prefix")
|
|
29
29
|
|
|
30
30
|
if spec.mode:
|
|
31
31
|
if spec.mode not in MODES:
|
|
32
32
|
e(spec, f"unknown mode {spec.mode!r} (expected one of: {', '.join(MODES)})")
|
|
33
33
|
if not (spec.outputdir and spec.basename):
|
|
34
|
-
e(spec, f"--mode {spec.mode} requires both --outputdir and --
|
|
34
|
+
e(spec, f"--mode {spec.mode} requires both --outputdir and --prefix")
|
|
35
35
|
elif spec.outputdir or spec.basename:
|
|
36
|
-
e(spec, "--outputdir/--
|
|
36
|
+
e(spec, "--outputdir/--prefix require --mode")
|
|
37
37
|
|
|
38
38
|
if spec.mode in _MODES_NEEDING_COLUMN and not spec.column:
|
|
39
39
|
e(spec, f"--mode {spec.mode} requires --column")
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: ping-dataexport
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: PING's Data Export Tool - To Parquet/CSV/TEXT as Accelerator for Data & AI Project
|
|
5
5
|
Author: vorapol
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -46,7 +46,7 @@ Dynamic: license-file
|
|
|
46
46
|
`decimal128(p,s)`, คอลัมน์ int ที่มี NULL ยังคงเป็น int, วันที่ยังคงเป็นวันที่
|
|
47
47
|
และไม่ใช้ pandas เด็ดขาด
|
|
48
48
|
- **ฐานข้อมูลที่รองรับ** — SQLite (Library มาตรฐาน ไม่ต้องติดตั้งอะไรเพิ่ม),
|
|
49
|
-
SQL Server (MS ODBC / FreeTDS), Oracle (โหมด thin
|
|
49
|
+
SQL Server (MS ODBC / FreeTDS), Oracle (โหมด thin แบบ Native หรือ ODBC)
|
|
50
50
|
และแหล่งข้อมูล ODBC ใดก็ได้
|
|
51
51
|
- **ใช้งานได้สองทาง** — ผ่าน CLI `ping-dataexport` หรือผ่าน Library API (`ping_dataexport.api`)
|
|
52
52
|
ที่ออกแบบมาสำหรับ Airflow
|
|
@@ -151,7 +151,7 @@ PasswordBase64=c2VjcmV0 ; base64("secret")
|
|
|
151
151
|
file defaults to `./config.ini` (`-cf` to point elsewhere).
|
|
152
152
|
|
|
153
153
|
```bash
|
|
154
|
-
# preview to the console (no -o/-
|
|
154
|
+
# preview to the console (no -o/-dir): prints the first rows
|
|
155
155
|
ping-dataexport -s SQLITE01 -q "SELECT * FROM sales"
|
|
156
156
|
|
|
157
157
|
# single file — format inferred from the extension (.csv/.txt/.parquet)
|
|
@@ -169,11 +169,11 @@ All options:
|
|
|
169
169
|
| `-s` | `--source` | DB config section name (e.g. `DB01`) |
|
|
170
170
|
| `-q` | `--query` | SQL query |
|
|
171
171
|
| `-o` | `--output` | single output file (format from extension, or `-f`) |
|
|
172
|
-
| `-
|
|
173
|
-
| `-
|
|
172
|
+
| `-dir` | `--outputdir` | output directory (required for `-m` modes); `-od` still works as a legacy alias |
|
|
173
|
+
| `-pf` | `--prefix` | output filename prefix for mode exports; `-bn`/`--basename` still work as legacy aliases |
|
|
174
174
|
| `-f` | `--format` | `csv`, `txt`, `parquet` |
|
|
175
175
|
| `-gz` | `--gzip` | gzip the output (CSV/TXT only) |
|
|
176
|
-
| `-
|
|
176
|
+
| `-sep` | `--sep` | field separator (default `,`; `\t` maps to tab) |
|
|
177
177
|
| `-quo` | `--quote` | quote character (default `"`) |
|
|
178
178
|
| `-stnl` | `--stripnewline` | newline handling in text columns: `space` \| `blank` \| `escape` \| `doubleescape` |
|
|
179
179
|
| `-stmc` | `--stripmetachar` | strip invisible/control characters from text columns |
|
|
@@ -194,15 +194,31 @@ All options:
|
|
|
194
194
|
| `-lf` | `--logfile` | log file path |
|
|
195
195
|
| `-t` | `--test` | test mode: validate + preview, no files |
|
|
196
196
|
|
|
197
|
+
### Which options must go together
|
|
198
|
+
|
|
199
|
+
| If you use… | You must also have… | Notes |
|
|
200
|
+
|-------------|--------------------|-------|
|
|
201
|
+
| (anything) | `-s` + `-q` | the two basics — always required, unless `-j` supplies them from a job file |
|
|
202
|
+
| `-o` | nothing else | single file; **cannot** be combined with `-dir`/`-pf` |
|
|
203
|
+
| `-m` (any mode) | `-dir` + `-pf` | every mode writes multiple files, so it needs a directory + filename prefix |
|
|
204
|
+
| `-dir` or `-pf` | `-m` | they only make sense for mode exports |
|
|
205
|
+
| `-m groupby` / `watermark` | `-col` | the column to group / track |
|
|
206
|
+
| `-m date` | `-col` + `-df` + `-dt` | the date column and an explicit range |
|
|
207
|
+
| `-m relativedate` | `-col` + `-dr` | `-df` optional (defaults to today) |
|
|
208
|
+
| `-m monthbydate` | `-col` + `-mr` | `-df` optional (defaults to today) |
|
|
209
|
+
| `-m splitrow` | `-row` | no `-col` needed |
|
|
210
|
+
| `-gz` | CSV/TXT output | rejected with parquet (it compresses itself) |
|
|
211
|
+
| no `-o` and no `-dir` | — | console preview only (first `PreviewRow` rows, nothing written) |
|
|
212
|
+
|
|
197
213
|
## 4. Export modes (partitioned outputs)
|
|
198
214
|
|
|
199
|
-
Modes split one query into multiple files. All need `-
|
|
200
|
-
+ `-
|
|
215
|
+
Modes split one query into multiple files. All need `-dir` (output directory)
|
|
216
|
+
+ `-pf` (filename prefix); partition values are always bound SQL parameters.
|
|
201
217
|
|
|
202
218
|
**`groupby` — one file per distinct value of a column:**
|
|
203
219
|
|
|
204
220
|
```bash
|
|
205
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m groupby -col region -
|
|
221
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m groupby -col region -dir by_region -pf sales
|
|
206
222
|
```
|
|
207
223
|
```
|
|
208
224
|
by_region/
|
|
@@ -217,7 +233,7 @@ by_region/
|
|
|
217
233
|
|
|
218
234
|
```bash
|
|
219
235
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m date -col txn_date \
|
|
220
|
-
-df 2026-01-01 -dt 2026-01-05 -
|
|
236
|
+
-df 2026-01-01 -dt 2026-01-05 -dir daily -pf sales
|
|
221
237
|
```
|
|
222
238
|
```
|
|
223
239
|
daily/
|
|
@@ -233,7 +249,7 @@ daily/
|
|
|
233
249
|
|
|
234
250
|
```bash
|
|
235
251
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m relativedate -col txn_date \
|
|
236
|
-
-df 2026-01-10 -dr 3 -
|
|
252
|
+
-df 2026-01-10 -dr 3 -dir rel -pf sales
|
|
237
253
|
```
|
|
238
254
|
```
|
|
239
255
|
rel/
|
|
@@ -248,7 +264,7 @@ at the anchor date; omit `-df` to anchor on today):**
|
|
|
248
264
|
|
|
249
265
|
```bash
|
|
250
266
|
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m monthbydate -col txn_date \
|
|
251
|
-
-df 2026-03-15 -mr 3 -
|
|
267
|
+
-df 2026-03-15 -mr 3 -dir monthly -pf sales
|
|
252
268
|
```
|
|
253
269
|
```
|
|
254
270
|
monthly/
|
|
@@ -261,7 +277,7 @@ monthly/
|
|
|
261
277
|
**`splitrow` — rotate to a new file every N rows:**
|
|
262
278
|
|
|
263
279
|
```bash
|
|
264
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales ORDER BY id" -m splitrow -row 100000 -
|
|
280
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales ORDER BY id" -m splitrow -row 100000 -dir split -pf part
|
|
265
281
|
```
|
|
266
282
|
```
|
|
267
283
|
split/
|
|
@@ -273,7 +289,7 @@ split/
|
|
|
273
289
|
**`watermark` — incremental: only rows newer than the last run:**
|
|
274
290
|
|
|
275
291
|
```bash
|
|
276
|
-
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m watermark -col txn_date -
|
|
292
|
+
ping-dataexport -s DB01 -q "SELECT * FROM sales" -m watermark -col txn_date -dir incr -pf sales
|
|
277
293
|
```
|
|
278
294
|
```
|
|
279
295
|
incr/
|
|
@@ -322,7 +338,7 @@ Output=out/full_dump.parquet
|
|
|
322
338
|
|
|
323
339
|
```bash
|
|
324
340
|
ping-dataexport -j DailySales # one section (job.ini by default, -jf elsewhere)
|
|
325
|
-
ping-dataexport -j DailySales -
|
|
341
|
+
ping-dataexport -j DailySales -dir other_dir # CLI options override the section
|
|
326
342
|
ping-dataexport -p 4 # ALL sections in parallel, 4 workers
|
|
327
343
|
```
|
|
328
344
|
|
|
@@ -333,11 +349,11 @@ Every job.ini key and the CLI option it mirrors (keys are case-insensitive):
|
|
|
333
349
|
| `Source` | `-s` / `--source` | which config.ini DB section to export from |
|
|
334
350
|
| `Query` | `-q` / `--query` | the SQL to run; indent continuation lines for multi-line queries |
|
|
335
351
|
| `Output` | `-o` / `--output` | single output file path |
|
|
336
|
-
| `OutputDir` | `-
|
|
337
|
-
| `BaseName` | `-
|
|
352
|
+
| `OutputDir` | `-dir` / `--outputdir` | output directory (needed when `Mode=` is set) |
|
|
353
|
+
| `BaseName` | `-pf` / `--prefix` | output filename prefix for mode exports |
|
|
338
354
|
| `Format` | `-f` / `--format` | `csv` / `txt` / `parquet` (else inferred from `Output` extension) |
|
|
339
355
|
| `Gzip` | `-gz` / `--gzip` | `Yes`/`No` — gzip the output (CSV/TXT only) |
|
|
340
|
-
| `Sep` | `-
|
|
356
|
+
| `Sep` | `-sep` / `--sep` | field separator (default `,`; `\t` = tab) |
|
|
341
357
|
| `Quote` | `-quo` / `--quote` | quote character (default `"`) |
|
|
342
358
|
| `StripNewLine` | `-stnl` / `--stripnewline` | `space` \| `blank` \| `escape` \| `doubleescape` |
|
|
343
359
|
| `StripMetaChar` | `-stmc` / `--stripmetachar` | `Yes`/`No` — strip invisible/control characters |
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport.egg-info/dependency_links.txt
RENAMED
|
File without changes
|
{ping_dataexport-0.2.2 → ping_dataexport-0.3.0}/src/ping_dataexport.egg-info/entry_points.txt
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|