@arela/uploader 0.3.0 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +130 -5
- package/docs/API_ENDPOINTS_FOR_DETECTION.md +647 -0
- package/docs/QUICK_REFERENCE_API_DETECTION.md +264 -0
- package/docs/REFACTORING_SUMMARY_DETECT_PEDIMENTOS.md +200 -0
- package/package.json +1 -1
- package/src/commands/WatchCommand.js +47 -10
- package/src/config/config.js +157 -2
- package/src/document-types/support-document.js +4 -5
- package/src/file-detection.js +7 -0
- package/src/index.js +119 -4
- package/src/services/AutoProcessingService.js +146 -36
- package/src/services/DatabaseService.js +341 -517
- package/src/services/upload/ApiUploadService.js +426 -4
- package/src/services/upload/MultiApiUploadService.js +233 -0
- package/src/services/upload/UploadServiceFactory.js +24 -0
- package/src/utils/FileOperations.js +6 -3
- package/src/utils/WatchEventHandler.js +14 -9
- package/.envbackup +0 -37
- package/SUPABASE_UPLOAD_FIX.md +0 -157
package/README.md
CHANGED
|
@@ -2,6 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
CLI tool to upload files and directories to Arela API or Supabase Storage with automatic file processing, detection, and organization.
|
|
4
4
|
|
|
5
|
+
## ✨ What's New in v0.4.0
|
|
6
|
+
|
|
7
|
+
- 🏢 **Simplified Multi-Tenant API**: Only 3 targets: `default`, `agencia`, `cliente`
|
|
8
|
+
- 🔀 **Cross-Tenant Mode**: Read from one API, write to another with `--source-api` and `--target-api`
|
|
9
|
+
- ⚙️ **Dynamic Client Config**: Switch clients by updating `.env` - no code changes needed!
|
|
10
|
+
- 👁️ **Enhanced Watch Mode**: Full cross-tenant support in automatic processing pipeline
|
|
11
|
+
- ⚡ **Optimized Connections**: HTTP Agent with connection pooling for high performance
|
|
12
|
+
|
|
5
13
|
## 🚀 OPTIMIZED 4-PHASE WORKFLOW
|
|
6
14
|
|
|
7
15
|
**New in v0.2.0**: The tool now supports an optimized 4-phase workflow designed for maximum performance when processing large file collections:
|
|
@@ -84,13 +92,76 @@ arela --upload-by-rfc # Phase 4: Upload by RFC
|
|
|
84
92
|
- 📋 **Upload files by specific RFC values**
|
|
85
93
|
- 🔍 **Propagate arela_path from pedimento documents to related files**
|
|
86
94
|
- ⚡ **4-Phase optimized workflow for maximum performance**
|
|
87
|
-
- 👁️ **Watch Mode**
|
|
95
|
+
- 👁️ **Watch Mode** - Monitor directories for changes and upload automatically
|
|
88
96
|
- Multiple watch strategies (batch, individual, full-structure)
|
|
97
|
+
- **Multi-tenant and cross-tenant support** ⭐ NEW
|
|
89
98
|
- Debounce and polling support
|
|
90
99
|
- Auto-processing pipeline
|
|
91
100
|
- Dry-run mode for testing
|
|
92
101
|
- Pattern-based file ignoring
|
|
93
102
|
|
|
103
|
+
## 🏢 Multi-Tenant API Support
|
|
104
|
+
|
|
105
|
+
Connect to different API instances: **default**, **agencia**, or **cliente**.
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
# Upload to client API
|
|
109
|
+
arela upload --api cliente --upload-by-rfc
|
|
110
|
+
|
|
111
|
+
# Collect stats on agencia API
|
|
112
|
+
arela stats --api agencia
|
|
113
|
+
|
|
114
|
+
# Watch mode with specific API target
|
|
115
|
+
arela watch --api cliente
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
### Cross-Tenant Mode
|
|
119
|
+
|
|
120
|
+
Process files from one tenant and upload to another:
|
|
121
|
+
|
|
122
|
+
```bash
|
|
123
|
+
# Read data from agencia, upload files to client
|
|
124
|
+
arela watch --source-api agencia --target-api cliente
|
|
125
|
+
|
|
126
|
+
# Same for upload command
|
|
127
|
+
arela upload --source-api agencia --target-api cliente --upload-by-rfc
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
**How Cross-Tenant Works:**
|
|
131
|
+
| Phase | Description | API Used |
|
|
132
|
+
|-------|-------------|----------|
|
|
133
|
+
| Phase 1 | Stats Collection | `--source-api` |
|
|
134
|
+
| Phase 2 | PDF Detection | `--source-api` |
|
|
135
|
+
| Phase 3 | Path Propagation | `--source-api` |
|
|
136
|
+
| Phase 4 | File Upload | `--target-api` |
|
|
137
|
+
|
|
138
|
+
### Available API Targets
|
|
139
|
+
|
|
140
|
+
Only 3 API targets are available: `default`, `agencia`, `cliente`
|
|
141
|
+
|
|
142
|
+
Configure in your `.env` file:
|
|
143
|
+
|
|
144
|
+
```env
|
|
145
|
+
# Default API (--api default or no flag)
|
|
146
|
+
ARELA_API_URL=http://localhost:3010
|
|
147
|
+
ARELA_API_TOKEN=your_token
|
|
148
|
+
|
|
149
|
+
# Agencia API (--api agencia)
|
|
150
|
+
ARELA_API_AGENCIA_URL=http://localhost:4012
|
|
151
|
+
ARELA_API_AGENCIA_TOKEN=your_agencia_token
|
|
152
|
+
|
|
153
|
+
# Cliente API (--api cliente)
|
|
154
|
+
# Configure the URL/Token for the specific client you need
|
|
155
|
+
ARELA_API_CLIENTE_URL=http://localhost:4014
|
|
156
|
+
ARELA_API_CLIENTE_TOKEN=your_cliente_token
|
|
157
|
+
|
|
158
|
+
# Examples for different clients:
|
|
159
|
+
# Cliente AUM9207011CA: ARELA_API_CLIENTE_URL=http://localhost:4014
|
|
160
|
+
# Cliente KTJ931117P55: ARELA_API_CLIENTE_URL=http://localhost:4013
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
> 💡 **Tip**: To switch between clients, just update `ARELA_API_CLIENTE_URL` and `ARELA_API_CLIENTE_TOKEN` in your `.env` file. No code changes needed!
|
|
164
|
+
|
|
94
165
|
## Installation
|
|
95
166
|
|
|
96
167
|
```bash
|
|
@@ -170,6 +241,12 @@ arela detect --propagate-arela-path
|
|
|
170
241
|
# Watch directories for changes with automatic upload
|
|
171
242
|
arela watch --directories "/path/to/watch1,/path/to/watch2"
|
|
172
243
|
|
|
244
|
+
# Watch with specific API target (single tenant)
|
|
245
|
+
arela watch --api cliente
|
|
246
|
+
|
|
247
|
+
# Watch with cross-tenant mode (read from agencia, upload to client)
|
|
248
|
+
arela watch --source-api agencia --target-api cliente
|
|
249
|
+
|
|
173
250
|
# Watch with custom upload strategy (default: batch)
|
|
174
251
|
arela watch --directories "/path/to/watch" --strategy individual
|
|
175
252
|
arela watch --directories "/path/to/watch" --strategy full-structure
|
|
@@ -198,6 +275,11 @@ arela watch --directories "/path/to/watch" --verbose
|
|
|
198
275
|
- `individual`: Uploads each file immediately as it changes
|
|
199
276
|
- `full-structure`: Preserves directory structure during upload
|
|
200
277
|
|
|
278
|
+
**Multi-Tenant Options:**
|
|
279
|
+
- `--api <target>`: Use a single API for all operations
|
|
280
|
+
- `--source-api <target>`: API for reading/processing (phases 1-3)
|
|
281
|
+
- `--target-api <target>`: API for uploading (phase 4)
|
|
282
|
+
|
|
201
283
|
#### 5. **query** - Query database for file status
|
|
202
284
|
```bash
|
|
203
285
|
# Show files ready for upload
|
|
@@ -263,6 +345,9 @@ arela --upload-by-rfc # Same as: arela upload --upload-by-rfc
|
|
|
263
345
|
- `batch`: Groups files and uploads periodically
|
|
264
346
|
- `individual`: Uploads each file immediately
|
|
265
347
|
- `full-structure`: Preserves directory structure
|
|
348
|
+
- `--api <target>`: Use a single API target for all operations
|
|
349
|
+
- `--source-api <target>`: API for reading/processing (phases 1-3)
|
|
350
|
+
- `--target-api <target>`: API for uploading (phase 4)
|
|
266
351
|
- `--debounce <ms>`: Debounce delay in milliseconds (default: 1000)
|
|
267
352
|
- `-b, --batch-size <size>`: Batch size for uploads (default: 10)
|
|
268
353
|
- `--poll <ms>`: Use polling instead of native file system events (interval in ms)
|
|
@@ -278,10 +363,19 @@ arela --upload-by-rfc # Same as: arela upload --upload-by-rfc
|
|
|
278
363
|
Create a `.env` file in your project root:
|
|
279
364
|
|
|
280
365
|
```env
|
|
281
|
-
#
|
|
366
|
+
# Default API (--api default or no flag)
|
|
282
367
|
ARELA_API_URL=http://localhost:3010
|
|
283
368
|
ARELA_API_TOKEN=your_api_token
|
|
284
369
|
|
|
370
|
+
# Agencia API (--api agencia)
|
|
371
|
+
ARELA_API_AGENCIA_URL=http://localhost:4012
|
|
372
|
+
ARELA_API_AGENCIA_TOKEN=your_agencia_token
|
|
373
|
+
|
|
374
|
+
# Cliente API (--api cliente)
|
|
375
|
+
# Configure for the specific client you need
|
|
376
|
+
ARELA_API_CLIENTE_URL=http://localhost:4014
|
|
377
|
+
ARELA_API_CLIENTE_TOKEN=your_cliente_token
|
|
378
|
+
|
|
285
379
|
# For Direct Supabase Mode (fallback)
|
|
286
380
|
SUPABASE_URL=your_supabase_url
|
|
287
381
|
SUPABASE_KEY=your_supabase_anon_key
|
|
@@ -294,18 +388,26 @@ UPLOAD_SOURCES=folder1|folder2|file.pdf
|
|
|
294
388
|
# RFC-based Upload Configuration
|
|
295
389
|
# Pipe-separated list of RFCs to upload files for
|
|
296
390
|
UPLOAD_RFCS=MMJ0810145N1|ABC1234567XY|DEF9876543ZZ
|
|
391
|
+
|
|
392
|
+
# Watch Mode Configuration (JSON format)
|
|
393
|
+
WATCH_DIRECTORY_CONFIGS={"../../Documents/2022":"palco","../../Documents/2023":"palco"}
|
|
297
394
|
```
|
|
298
395
|
|
|
299
396
|
**Environment Variable Details:**
|
|
300
397
|
|
|
301
|
-
- `ARELA_API_URL`: Base URL for
|
|
302
|
-
- `
|
|
398
|
+
- `ARELA_API_URL`: Base URL for default API service
|
|
399
|
+
- `ARELA_API_AGENCIA_URL`: URL for agencia API
|
|
400
|
+
- `ARELA_API_CLIENTE_URL`: URL for client API (configure per client)
|
|
401
|
+
- `ARELA_API_TOKEN`: Authentication token for default API
|
|
402
|
+
- `ARELA_API_AGENCIA_TOKEN`: Token for agencia API
|
|
403
|
+
- `ARELA_API_CLIENTE_TOKEN`: Token for client API
|
|
303
404
|
- `SUPABASE_URL`: Your Supabase project URL
|
|
304
405
|
- `SUPABASE_KEY`: Supabase anonymous key for direct uploads
|
|
305
406
|
- `SUPABASE_BUCKET`: Target bucket name in Supabase Storage
|
|
306
407
|
- `UPLOAD_BASE_PATH`: Root directory containing files to upload
|
|
307
408
|
- `UPLOAD_SOURCES`: Pipe-separated list of folders/files to process
|
|
308
409
|
- `UPLOAD_RFCS`: Pipe-separated list of RFC values for targeted uploads
|
|
410
|
+
- `WATCH_DIRECTORY_CONFIGS`: JSON mapping directories to folder structures
|
|
309
411
|
|
|
310
412
|
## RFC-Based File Upload
|
|
311
413
|
|
|
@@ -510,7 +612,25 @@ The tool maintains comprehensive logs both locally and remotely:
|
|
|
510
612
|
|
|
511
613
|
## Version History
|
|
512
614
|
|
|
513
|
-
**
|
|
615
|
+
**v0.4.0** - Current Release 🆕
|
|
616
|
+
- ✨ **Simplified Multi-Tenant API**: Only 3 targets: `default`, `agencia`, `cliente`
|
|
617
|
+
- ✨ **Cross-Tenant Mode**: Read from one API, upload to another
|
|
618
|
+
- ✨ **Dynamic Client Config**: Change client by updating `.env` (no code changes)
|
|
619
|
+
- ✨ New `--api` flag for single API target
|
|
620
|
+
- ✨ New `--source-api` flag for source API (phases 1-3)
|
|
621
|
+
- ✨ New `--target-api` flag for target API (phase 4)
|
|
622
|
+
- ✨ `WATCH_DIRECTORY_CONFIGS` environment variable for watch mode
|
|
623
|
+
- 🔧 Enhanced pipeline routing for cross-tenant operations
|
|
624
|
+
- 📝 Simplified documentation for multi-tenant configuration
|
|
625
|
+
|
|
626
|
+
**v0.3.0** - Watch Mode Release
|
|
627
|
+
- ✨ Added watch command with chokidar integration
|
|
628
|
+
- ✨ Automatic 4-step pipeline (stats → detect → propagate → upload)
|
|
629
|
+
- ✨ Multiple upload strategies (batch, individual, full-structure)
|
|
630
|
+
- ✨ Configurable debounce and polling options
|
|
631
|
+
- 🔧 Signal handling for graceful shutdown
|
|
632
|
+
|
|
633
|
+
**v0.2.0** - Pipeline Automation
|
|
514
634
|
- ✨ Added smart year/pedimento auto-detection from file paths
|
|
515
635
|
- ✨ Custom folder structure support with `--folder-structure` option
|
|
516
636
|
- ✨ Client path tracking with `--client-path` option
|
|
@@ -522,6 +642,11 @@ The tool maintains comprehensive logs both locally and remotely:
|
|
|
522
642
|
- 📝 Comprehensive logging with SANITIZED status
|
|
523
643
|
- 🔧 Memory optimization for large file processing
|
|
524
644
|
|
|
645
|
+
**v0.1.0** - Initial Release
|
|
646
|
+
- 📦 Basic upload functionality
|
|
647
|
+
- 🔌 API and Supabase direct mode support
|
|
648
|
+
- 📂 RFC-based file upload
|
|
649
|
+
|
|
525
650
|
## Troubleshooting
|
|
526
651
|
|
|
527
652
|
**Connection Issues:**
|