@cakemail-org/cakemail-cli 1.7.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/settings.local.json +12 -0
- package/.env.example +40 -0
- package/.env.test.example +45 -0
- package/CHANGELOG.md +1031 -0
- package/README.md +41 -37
- package/audit-formats.js +128 -0
- package/cakemail.rb +20 -0
- package/dist/client.js +1 -1
- package/dist/client.js.map +1 -1
- package/dist/commands/account.js +1 -1
- package/dist/commands/account.js.map +1 -1
- package/dist/commands/attributes.js +1 -1
- package/dist/commands/attributes.js.map +1 -1
- package/dist/commands/campaigns.js +1 -1
- package/dist/commands/campaigns.js.map +1 -1
- package/dist/commands/contacts.js +1 -1
- package/dist/commands/contacts.js.map +1 -1
- package/dist/commands/emails.js +1 -1
- package/dist/commands/emails.js.map +1 -1
- package/dist/commands/interests.js +1 -1
- package/dist/commands/interests.js.map +1 -1
- package/dist/commands/lists.js +1 -1
- package/dist/commands/lists.js.map +1 -1
- package/dist/commands/logs.js +1 -1
- package/dist/commands/logs.js.map +1 -1
- package/dist/commands/reports.js +1 -1
- package/dist/commands/reports.js.map +1 -1
- package/dist/commands/segments.js +1 -1
- package/dist/commands/segments.js.map +1 -1
- package/dist/commands/senders.js +1 -1
- package/dist/commands/senders.js.map +1 -1
- package/dist/commands/suppressed.js +1 -1
- package/dist/commands/suppressed.js.map +1 -1
- package/dist/commands/tags.js +1 -1
- package/dist/commands/tags.js.map +1 -1
- package/dist/commands/templates.js +1 -1
- package/dist/commands/templates.js.map +1 -1
- package/dist/commands/transactional-templates.js +1 -1
- package/dist/commands/transactional-templates.js.map +1 -1
- package/dist/commands/webhooks.js +1 -1
- package/dist/commands/webhooks.js.map +1 -1
- package/dist/utils/config.js +2 -2
- package/dist/utils/config.js.map +1 -1
- package/dist/utils/errors.js +1 -1
- package/dist/utils/errors.js.map +1 -1
- package/dist/utils/progress.d.ts.map +1 -1
- package/dist/utils/progress.js +32 -4
- package/dist/utils/progress.js.map +1 -1
- package/dist/utils/spinner.d.ts +17 -0
- package/dist/utils/spinner.d.ts.map +1 -0
- package/dist/utils/spinner.js +43 -0
- package/dist/utils/spinner.js.map +1 -0
- package/docs/DOCUMENTATION-STANDARD.md +1068 -0
- package/docs/README.md +161 -0
- package/docs/developer/ARCHITECTURE.md +516 -0
- package/docs/developer/AUTH.md +204 -0
- package/docs/developer/CONTRIBUTING.md +227 -0
- package/docs/developer/DOCUMENTATION_SUMMARY.md +346 -0
- package/docs/developer/PROJECT_INDEX.md +365 -0
- package/docs/planning/API_COVERAGE.md +1045 -0
- package/docs/planning/BACKLOG.md +1159 -0
- package/docs/planning/PROFILE_SYSTEM_TASKS.md +287 -0
- package/docs/planning/UX_IMPLEMENTATION_PLAN.md +691 -0
- package/docs/planning/archive/RELEASE_CHECKLIST_v1.3.0.md +332 -0
- package/docs/planning/archive/RELEASE_v1.3.0.md +428 -0
- package/docs/planning/archive/cakemail-cli-ux-improvements.md +438 -0
- package/docs/planning/cakemail-profile-system-plan.md +1121 -0
- package/docs/testing/AI_USER_SIMULATION_DESIGN.md +1342 -0
- package/docs/testing/KENOGAMI_BIDIRECTIONAL_FLOW.md +1517 -0
- package/docs/testing/KENOGAMI_TRUTH_RECONCILIATION_SYSTEM.md +1369 -0
- package/docs/user-manual/.obsidian/app.json +1 -0
- package/docs/user-manual/.obsidian/appearance.json +1 -0
- package/docs/user-manual/.obsidian/core-plugins.json +33 -0
- package/docs/user-manual/.obsidian/workspace.json +167 -0
- package/docs/user-manual/01-getting-started/01-installation.md +214 -0
- package/docs/user-manual/01-getting-started/02-quick-start.md +432 -0
- package/docs/user-manual/01-getting-started/03-authentication.md +448 -0
- package/docs/user-manual/01-getting-started/04-configuration.md +430 -0
- package/docs/user-manual/01-getting-started/05-output-formats.md +447 -0
- package/docs/user-manual/02-core-concepts/01-accounts.md +514 -0
- package/docs/user-manual/02-core-concepts/02-profile-system.md +771 -0
- package/docs/user-manual/02-core-concepts/03-smart-defaults.md +485 -0
- package/docs/user-manual/02-core-concepts/04-authentication-methods.md +435 -0
- package/docs/user-manual/02-core-concepts/05-pagination-filtering.md +600 -0
- package/docs/user-manual/02-core-concepts/06-error-handling.md +718 -0
- package/docs/user-manual/02-core-concepts/07-api-coverage.md +483 -0
- package/docs/user-manual/03-email-operations/01-senders.md +490 -0
- package/docs/user-manual/03-email-operations/02-templates.md +444 -0
- package/docs/user-manual/03-email-operations/03-transactional-emails.md +706 -0
- package/docs/user-manual/03-email-operations/04-email-tracking.md +407 -0
- package/docs/user-manual/04-campaign-management/01-campaigns-basics.md +394 -0
- package/docs/user-manual/04-campaign-management/02-campaign-scheduling.md +630 -0
- package/docs/user-manual/04-campaign-management/03-campaign-testing.md +997 -0
- package/docs/user-manual/04-campaign-management/04-campaign-lifecycle.md +709 -0
- package/docs/user-manual/04-campaign-management/05-campaign-links.md +934 -0
- package/docs/user-manual/05-contact-management/01-lists.md +836 -0
- package/docs/user-manual/05-contact-management/02-contacts.md +1035 -0
- package/docs/user-manual/05-contact-management/03-custom-attributes.md +788 -0
- package/docs/user-manual/05-contact-management/04-segments.md +1028 -0
- package/docs/user-manual/05-contact-management/05-contact-import-export.md +1031 -0
- package/docs/user-manual/06-analytics-reporting/01-campaign-analytics.md +867 -0
- package/docs/user-manual/06-analytics-reporting/02-account-reports.md +227 -0
- package/docs/user-manual/07-integrations/01-webhooks-integration.md +259 -0
- package/docs/user-manual/07-integrations/02-automation.md +326 -0
- package/docs/user-manual/08-advanced-usage/01-scripting-patterns.md +672 -0
- package/docs/user-manual/08-advanced-usage/02-bulk-operations.md +932 -0
- package/docs/user-manual/08-advanced-usage/03-ci-cd-integration.md +892 -0
- package/docs/user-manual/08-advanced-usage/04-performance-optimization.md +766 -0
- package/docs/user-manual/09-command-reference/01-config.md +776 -0
- package/docs/user-manual/09-command-reference/02-account.md +652 -0
- package/docs/user-manual/09-command-reference/03-lists.md +958 -0
- package/docs/user-manual/09-command-reference/04-contacts.md +1408 -0
- package/docs/user-manual/09-command-reference/05-attributes.md +617 -0
- package/docs/user-manual/09-command-reference/06-segments.md +894 -0
- package/docs/user-manual/09-command-reference/07-senders.md +803 -0
- package/docs/user-manual/09-command-reference/08-templates.md +818 -0
- package/docs/user-manual/09-command-reference/09-campaigns.md +1250 -0
- package/docs/user-manual/09-command-reference/10-emails.md +807 -0
- package/docs/user-manual/09-command-reference/11-reports.md +1135 -0
- package/docs/user-manual/09-command-reference/12-webhooks.md +773 -0
- package/docs/user-manual/09-command-reference/13-suppressed.md +797 -0
- package/docs/user-manual/09-command-reference/14-interests.md +630 -0
- package/docs/user-manual/09-command-reference/15-tags.md +584 -0
- package/docs/user-manual/09-command-reference/16-logs.md +656 -0
- package/docs/user-manual/09-command-reference/17-transactional-templates.md +850 -0
- package/docs/user-manual/10-troubleshooting/01-common-errors.md +457 -0
- package/docs/user-manual/10-troubleshooting/02-authentication-issues.md +558 -0
- package/docs/user-manual/10-troubleshooting/03-connection-problems.md +634 -0
- package/docs/user-manual/10-troubleshooting/04-debugging.md +725 -0
- package/docs/user-manual/11-appendix/04-faq.md +484 -0
- package/docs/user-manual/11-appendix/05-glossary.md +250 -0
- package/docs/user-manual/README.md +0 -0
- package/package.json +13 -61
- package/src/cli.ts +125 -0
- package/src/client.ts +16 -0
- package/src/commands/account.ts +267 -0
- package/src/commands/accounts.ts +78 -0
- package/src/commands/actions.ts +249 -0
- package/src/commands/attributes.ts +139 -0
- package/src/commands/campaign-blueprints.ts +106 -0
- package/src/commands/campaigns.ts +469 -0
- package/src/commands/config.ts +77 -0
- package/src/commands/contacts.ts +612 -0
- package/src/commands/custom-attributes.ts +127 -0
- package/src/commands/dkims.ts +117 -0
- package/src/commands/domains.ts +82 -0
- package/src/commands/email-apis.ts +569 -0
- package/src/commands/emails.ts +197 -0
- package/src/commands/forms.ts +283 -0
- package/src/commands/interests.ts +155 -0
- package/src/commands/links.ts +38 -0
- package/src/commands/lists.ts +406 -0
- package/src/commands/logos.ts +71 -0
- package/src/commands/logs.ts +386 -0
- package/src/commands/reports.ts +306 -0
- package/src/commands/segments.ts +158 -0
- package/src/commands/senders.ts +204 -0
- package/src/commands/sub-accounts.ts +271 -0
- package/src/commands/suppressed-emails.ts +234 -0
- package/src/commands/suppressed.ts +198 -0
- package/src/commands/system-emails.ts +85 -0
- package/src/commands/tags.ts +146 -0
- package/src/commands/tasks.ts +116 -0
- package/src/commands/templates.ts +189 -0
- package/src/commands/tokens.ts +83 -0
- package/src/commands/transactional-emails.ts +374 -0
- package/src/commands/transactional-templates.ts +385 -0
- package/src/commands/users.ts +506 -0
- package/src/commands/webhooks.ts +172 -0
- package/src/commands/workflow-blueprints.ts +123 -0
- package/src/commands/workflows.ts +265 -0
- package/src/types/profile.ts +93 -0
- package/src/utils/auth.ts +272 -0
- package/src/utils/config-file.ts +96 -0
- package/src/utils/config.ts +134 -0
- package/src/utils/confirm.ts +32 -0
- package/src/utils/defaults.ts +99 -0
- package/src/utils/errors.ts +116 -0
- package/src/utils/interactive.ts +91 -0
- package/src/utils/list-defaults.ts +74 -0
- package/src/utils/output.ts +190 -0
- package/src/utils/progress.ts +320 -0
- package/src/utils/spinner.ts +22 -0
- package/tests/IMPLEMENTATION_STATUS.md +258 -0
- package/tests/PTY_SETUP.md +118 -0
- package/tests/PTY_TESTING_GUIDE.md +507 -0
- package/tests/README.md +244 -0
- package/tests/fixtures/api-responses/campaigns.json +34 -0
- package/tests/fixtures/test-config.json +13 -0
- package/tests/helpers/cli-runner.ts +128 -0
- package/tests/helpers/mock-server.ts +301 -0
- package/tests/helpers/pty-runner.ts +181 -0
- package/tests/integration/campaigns-real-api.test.ts +196 -0
- package/tests/integration/setup-integration.ts +50 -0
- package/tests/pty/campaigns.test.ts +241 -0
- package/tests/setup.ts +34 -0
- package/tsconfig.json +15 -0
- package/vitest.config.ts +28 -0
|
@@ -0,0 +1,1031 @@
|
|
|
1
|
+
# Contact Import & Export
|
|
2
|
+
|
|
3
|
+
Master bulk contact operations including CSV imports, exports, and data migration.
|
|
4
|
+
|
|
5
|
+
## Overview
|
|
6
|
+
|
|
7
|
+
Import and export operations allow you to:
|
|
8
|
+
- Import hundreds or thousands of contacts at once
|
|
9
|
+
- Export contact data for backup or analysis
|
|
10
|
+
- Migrate contacts between lists
|
|
11
|
+
- Integrate with external systems
|
|
12
|
+
- Backup contact data regularly
|
|
13
|
+
- Clean and transform contact data in bulk
|
|
14
|
+
|
|
15
|
+
Bulk operations are essential for managing large contact databases efficiently.
|
|
16
|
+
|
|
17
|
+
## Quick Start
|
|
18
|
+
|
|
19
|
+
### Import Contacts from CSV
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
$ cakemail contacts import 123 --file contacts.csv
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
**Output:**
|
|
26
|
+
```
|
|
27
|
+
✓ Import started
|
|
28
|
+
|
|
29
|
+
Import ID: imp_abc123
|
|
30
|
+
Status: processing
|
|
31
|
+
File: contacts.csv
|
|
32
|
+
Contacts to import: 1,247
|
|
33
|
+
|
|
34
|
+
Check status with:
|
|
35
|
+
cakemail contacts import-status imp_abc123
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
### Export Contacts to CSV
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
$ cakemail contacts export 123
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
**Output:**
|
|
45
|
+
```
|
|
46
|
+
✓ Export started
|
|
47
|
+
|
|
48
|
+
Export ID: exp_xyz789
|
|
49
|
+
Status: processing
|
|
50
|
+
Estimated contacts: 1,247
|
|
51
|
+
|
|
52
|
+
Check status with:
|
|
53
|
+
cakemail contacts export-status exp_xyz789
|
|
54
|
+
|
|
55
|
+
Download when ready:
|
|
56
|
+
cakemail contacts export-download exp_xyz789 > contacts.csv
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
## CSV File Format
|
|
60
|
+
|
|
61
|
+
### Basic CSV Structure
|
|
62
|
+
|
|
63
|
+
```csv
|
|
64
|
+
email,first_name,last_name
|
|
65
|
+
john@example.com,John,Doe
|
|
66
|
+
jane@example.com,Jane,Smith
|
|
67
|
+
bob@example.com,Bob,Johnson
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### With Custom Attributes
|
|
71
|
+
|
|
72
|
+
```csv
|
|
73
|
+
email,first_name,last_name,plan,signup_date,is_vip
|
|
74
|
+
john@example.com,John,Doe,premium,2024-03-15,true
|
|
75
|
+
jane@example.com,Jane,Smith,basic,2024-03-16,false
|
|
76
|
+
bob@example.com,Bob,Johnson,enterprise,2024-03-17,true
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
### CSV Requirements
|
|
80
|
+
|
|
81
|
+
**Required:**
|
|
82
|
+
- `email` column (must be present)
|
|
83
|
+
- Valid email addresses
|
|
84
|
+
- UTF-8 encoding
|
|
85
|
+
|
|
86
|
+
**Optional:**
|
|
87
|
+
- `first_name` - Contact first name
|
|
88
|
+
- `last_name` - Contact last name
|
|
89
|
+
- Custom attribute columns (must exist in list)
|
|
90
|
+
|
|
91
|
+
**Format rules:**
|
|
92
|
+
- First row must be header with column names
|
|
93
|
+
- Email addresses must be unique
|
|
94
|
+
- Custom attribute names must match existing attributes
|
|
95
|
+
- Boolean values: `true`/`false` or `1`/`0`
|
|
96
|
+
- Dates: ISO format `YYYY-MM-DD`
|
|
97
|
+
- Numbers: Plain numbers, no currency symbols
|
|
98
|
+
|
|
99
|
+
### Creating CSV Files
|
|
100
|
+
|
|
101
|
+
**Using spreadsheet software:**
|
|
102
|
+
1. Create spreadsheet with proper columns
|
|
103
|
+
2. Export as CSV (UTF-8)
|
|
104
|
+
3. Save with `.csv` extension
|
|
105
|
+
|
|
106
|
+
**Using command line:**
|
|
107
|
+
```bash
|
|
108
|
+
# Create from scratch
|
|
109
|
+
cat > contacts.csv << 'EOF'
|
|
110
|
+
email,first_name,last_name,plan
|
|
111
|
+
john@example.com,John,Doe,premium
|
|
112
|
+
jane@example.com,Jane,Smith,basic
|
|
113
|
+
EOF
|
|
114
|
+
|
|
115
|
+
# From database export
|
|
116
|
+
mysql -u user -p database -e "SELECT email, first_name, last_name FROM contacts" \
|
|
117
|
+
--batch --silent | sed 's/\t/,/g' > contacts.csv
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
**Using scripts:**
|
|
121
|
+
```bash
|
|
122
|
+
#!/bin/bash
|
|
123
|
+
# generate-csv.sh
|
|
124
|
+
|
|
125
|
+
echo "email,first_name,last_name,plan" > contacts.csv
|
|
126
|
+
|
|
127
|
+
# Add contacts from your source
|
|
128
|
+
while IFS=, read email fname lname plan; do
|
|
129
|
+
echo "$email,$fname,$lname,$plan" >> contacts.csv
|
|
130
|
+
done < source_data.txt
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
## Import Process
|
|
134
|
+
|
|
135
|
+
### Basic Import
|
|
136
|
+
|
|
137
|
+
```bash
|
|
138
|
+
$ cakemail contacts import 123 --file contacts.csv
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
### Check Import Status
|
|
142
|
+
|
|
143
|
+
```bash
|
|
144
|
+
$ cakemail contacts import-status imp_abc123
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
**Output:**
|
|
148
|
+
```json
|
|
149
|
+
{
|
|
150
|
+
"id": "imp_abc123",
|
|
151
|
+
"status": "completed",
|
|
152
|
+
"file": "contacts.csv",
|
|
153
|
+
"total_rows": 1247,
|
|
154
|
+
"imported": 1189,
|
|
155
|
+
"skipped": 45,
|
|
156
|
+
"failed": 13,
|
|
157
|
+
"errors": [
|
|
158
|
+
{"row": 5, "email": "invalid-email", "error": "Invalid email format"},
|
|
159
|
+
{"row": 23, "email": "duplicate@example.com", "error": "Duplicate email"}
|
|
160
|
+
],
|
|
161
|
+
"started_at": "2024-03-15T10:30:00Z",
|
|
162
|
+
"completed_at": "2024-03-15T10:32:15Z"
|
|
163
|
+
}
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
### View Import History
|
|
167
|
+
|
|
168
|
+
```bash
|
|
169
|
+
$ cakemail contacts import-list 123
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
**Output:**
|
|
173
|
+
```
|
|
174
|
+
┌──────────────┬─────────────────────┬──────────┬──────────┬─────────┬──────┐
|
|
175
|
+
│ Import ID │ Started │ Status │ Total │ Success │ Failed│
|
|
176
|
+
├──────────────┼─────────────────────┼──────────┼──────────┼─────────┼──────┤
|
|
177
|
+
│ imp_abc123 │ 2024-03-15 10:30:00 │ complete │ 1,247 │ 1,189 │ 13 │
|
|
178
|
+
│ imp_def456 │ 2024-03-10 14:20:00 │ complete │ 856 │ 850 │ 6 │
|
|
179
|
+
│ imp_ghi789 │ 2024-03-05 09:15:00 │ complete │ 2,341 │ 2,300 │ 41 │
|
|
180
|
+
└──────────────┴─────────────────────┴──────────┴──────────┴─────────┴──────┘
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
### Monitor Import Progress
|
|
184
|
+
|
|
185
|
+
```bash
|
|
186
|
+
#!/bin/bash
|
|
187
|
+
# monitor-import.sh
|
|
188
|
+
|
|
189
|
+
IMPORT_ID=$1
|
|
190
|
+
|
|
191
|
+
if [ -z "$IMPORT_ID" ]; then
|
|
192
|
+
echo "Usage: $0 <import-id>"
|
|
193
|
+
exit 1
|
|
194
|
+
fi
|
|
195
|
+
|
|
196
|
+
echo "=== Monitoring Import: $IMPORT_ID ==="
|
|
197
|
+
echo ""
|
|
198
|
+
|
|
199
|
+
while true; do
|
|
200
|
+
STATUS=$(cakemail contacts import-status $IMPORT_ID -f json)
|
|
201
|
+
|
|
202
|
+
STATE=$(echo "$STATUS" | jq -r '.status')
|
|
203
|
+
TOTAL=$(echo "$STATUS" | jq -r '.total_rows')
|
|
204
|
+
IMPORTED=$(echo "$STATUS" | jq -r '.imported')
|
|
205
|
+
FAILED=$(echo "$STATUS" | jq -r '.failed')
|
|
206
|
+
|
|
207
|
+
if [ "$IMPORTED" != "null" ] && [ $TOTAL -gt 0 ]; then
|
|
208
|
+
PERCENT=$(echo "scale=1; $IMPORTED * 100 / $TOTAL" | bc)
|
|
209
|
+
else
|
|
210
|
+
PERCENT="0"
|
|
211
|
+
fi
|
|
212
|
+
|
|
213
|
+
clear
|
|
214
|
+
echo "=== Import Status ==="
|
|
215
|
+
echo ""
|
|
216
|
+
echo "Status: $STATE"
|
|
217
|
+
echo "Progress: $IMPORTED / $TOTAL ($PERCENT%)"
|
|
218
|
+
echo "Failed: $FAILED"
|
|
219
|
+
echo ""
|
|
220
|
+
|
|
221
|
+
if [ "$STATE" == "completed" ] || [ "$STATE" == "failed" ]; then
|
|
222
|
+
echo "Import $STATE"
|
|
223
|
+
break
|
|
224
|
+
fi
|
|
225
|
+
|
|
226
|
+
sleep 5
|
|
227
|
+
done
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
## Export Process
|
|
231
|
+
|
|
232
|
+
### Basic Export
|
|
233
|
+
|
|
234
|
+
```bash
|
|
235
|
+
$ cakemail contacts export 123
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
### Export with Filters
|
|
239
|
+
|
|
240
|
+
```bash
|
|
241
|
+
# Export only subscribed
|
|
242
|
+
$ cakemail contacts export 123 --filter "status==subscribed"
|
|
243
|
+
|
|
244
|
+
# Export premium users
|
|
245
|
+
$ cakemail contacts export 123 --filter "custom_attributes.plan==premium"
|
|
246
|
+
|
|
247
|
+
# Export recent signups
|
|
248
|
+
$ cakemail contacts export 123 --filter "custom_attributes.signup_date>=2024-03-01"
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
### Check Export Status
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
$ cakemail contacts export-status exp_xyz789
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
**Output:**
|
|
258
|
+
```json
|
|
259
|
+
{
|
|
260
|
+
"id": "exp_xyz789",
|
|
261
|
+
"status": "completed",
|
|
262
|
+
"total_contacts": 1247,
|
|
263
|
+
"file_size": "125 KB",
|
|
264
|
+
"started_at": "2024-03-15T10:35:00Z",
|
|
265
|
+
"completed_at": "2024-03-15T10:35:45Z",
|
|
266
|
+
"expires_at": "2024-03-22T10:35:45Z"
|
|
267
|
+
}
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
### Download Export
|
|
271
|
+
|
|
272
|
+
```bash
|
|
273
|
+
$ cakemail contacts export-download exp_xyz789 > contacts.csv
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
**Output:**
|
|
277
|
+
```
|
|
278
|
+
✓ Downloaded 1,247 contacts to contacts.csv
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
### View Export History
|
|
282
|
+
|
|
283
|
+
```bash
|
|
284
|
+
$ cakemail contacts export-list 123
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
**Output:**
|
|
288
|
+
```
|
|
289
|
+
┌──────────────┬─────────────────────┬──────────┬──────────┬───────────┐
|
|
290
|
+
│ Export ID │ Created │ Status │ Contacts │ File Size │
|
|
291
|
+
├──────────────┼─────────────────────┼──────────┼──────────┼───────────┤
|
|
292
|
+
│ exp_xyz789 │ 2024-03-15 10:35:00 │ complete │ 1,247 │ 125 KB │
|
|
293
|
+
│ exp_abc123 │ 2024-03-10 14:45:00 │ complete │ 856 │ 89 KB │
|
|
294
|
+
│ exp_def456 │ 2024-03-05 09:30:00 │ expired │ 2,341 │ - │
|
|
295
|
+
└──────────────┴─────────────────────┴──────────┴──────────┴───────────┘
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
### Automated Export
|
|
299
|
+
|
|
300
|
+
```bash
|
|
301
|
+
#!/bin/bash
|
|
302
|
+
# export-and-download.sh
|
|
303
|
+
|
|
304
|
+
LIST_ID=$1
|
|
305
|
+
|
|
306
|
+
if [ -z "$LIST_ID" ]; then
|
|
307
|
+
echo "Usage: $0 <list-id>"
|
|
308
|
+
exit 1
|
|
309
|
+
fi
|
|
310
|
+
|
|
311
|
+
echo "=== Exporting List $LIST_ID ==="
|
|
312
|
+
echo ""
|
|
313
|
+
|
|
314
|
+
# Start export
|
|
315
|
+
EXPORT=$(cakemail contacts export $LIST_ID -f json)
|
|
316
|
+
EXPORT_ID=$(echo "$EXPORT" | jq -r '.id')
|
|
317
|
+
|
|
318
|
+
echo "Export started: $EXPORT_ID"
|
|
319
|
+
echo "Waiting for completion..."
|
|
320
|
+
echo ""
|
|
321
|
+
|
|
322
|
+
# Wait for completion
|
|
323
|
+
while true; do
|
|
324
|
+
STATUS=$(cakemail contacts export-status $EXPORT_ID -f json)
|
|
325
|
+
STATE=$(echo "$STATUS" | jq -r '.status')
|
|
326
|
+
|
|
327
|
+
if [ "$STATE" == "completed" ]; then
|
|
328
|
+
break
|
|
329
|
+
elif [ "$STATE" == "failed" ]; then
|
|
330
|
+
echo "❌ Export failed"
|
|
331
|
+
exit 1
|
|
332
|
+
fi
|
|
333
|
+
|
|
334
|
+
sleep 2
|
|
335
|
+
done
|
|
336
|
+
|
|
337
|
+
# Download
|
|
338
|
+
FILENAME="contacts-$LIST_ID-$(date +%Y%m%d).csv"
|
|
339
|
+
cakemail contacts export-download $EXPORT_ID > "$FILENAME"
|
|
340
|
+
|
|
341
|
+
echo "✓ Downloaded to: $FILENAME"
|
|
342
|
+
|
|
343
|
+
# Show stats
|
|
344
|
+
ROWS=$(wc -l < "$FILENAME")
|
|
345
|
+
ROWS=$((ROWS - 1)) # Exclude header
|
|
346
|
+
echo " Total contacts: $ROWS"
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
## Import/Export Workflows
|
|
350
|
+
|
|
351
|
+
### Workflow 1: Regular Backup
|
|
352
|
+
|
|
353
|
+
```bash
|
|
354
|
+
#!/bin/bash
|
|
355
|
+
# backup-contacts.sh
|
|
356
|
+
|
|
357
|
+
BACKUP_DIR="./backups"
|
|
358
|
+
mkdir -p "$BACKUP_DIR"
|
|
359
|
+
|
|
360
|
+
echo "=== Contact Backup ==="
|
|
361
|
+
echo ""
|
|
362
|
+
|
|
363
|
+
# Get all lists
|
|
364
|
+
LISTS=$(cakemail lists list -f json | jq -r '.data[] | "\(.id):\(.name)"')
|
|
365
|
+
|
|
366
|
+
for LIST in $LISTS; do
|
|
367
|
+
LIST_ID=$(echo "$LIST" | cut -d: -f1)
|
|
368
|
+
LIST_NAME=$(echo "$LIST" | cut -d: -f2-)
|
|
369
|
+
|
|
370
|
+
echo "Backing up: $LIST_NAME (ID: $LIST_ID)"
|
|
371
|
+
|
|
372
|
+
# Start export
|
|
373
|
+
EXPORT_ID=$(cakemail contacts export $LIST_ID -f json | jq -r '.id')
|
|
374
|
+
|
|
375
|
+
# Wait for completion
|
|
376
|
+
while true; do
|
|
377
|
+
STATUS=$(cakemail contacts export-status $EXPORT_ID -f json | jq -r '.status')
|
|
378
|
+
if [ "$STATUS" == "completed" ]; then
|
|
379
|
+
break
|
|
380
|
+
fi
|
|
381
|
+
sleep 2
|
|
382
|
+
done
|
|
383
|
+
|
|
384
|
+
# Download
|
|
385
|
+
FILENAME="$BACKUP_DIR/list-$LIST_ID-$(date +%Y%m%d-%H%M%S).csv"
|
|
386
|
+
cakemail contacts export-download $EXPORT_ID > "$FILENAME"
|
|
387
|
+
|
|
388
|
+
echo " ✓ Saved to: $FILENAME"
|
|
389
|
+
done
|
|
390
|
+
|
|
391
|
+
echo ""
|
|
392
|
+
echo "✓ Backup complete"
|
|
393
|
+
|
|
394
|
+
# Compress backups older than 7 days
|
|
395
|
+
find "$BACKUP_DIR" -name "*.csv" -mtime +7 -exec gzip {} \;
|
|
396
|
+
```
|
|
397
|
+
|
|
398
|
+
**Schedule with cron:**
|
|
399
|
+
```bash
|
|
400
|
+
# Daily backup at 2 AM
|
|
401
|
+
0 2 * * * /path/to/backup-contacts.sh >> /var/log/contact-backup.log 2>&1
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
### Workflow 2: Data Migration Between Lists
|
|
405
|
+
|
|
406
|
+
```bash
|
|
407
|
+
#!/bin/bash
|
|
408
|
+
# migrate-contacts.sh
|
|
409
|
+
|
|
410
|
+
SOURCE_LIST=$1
|
|
411
|
+
TARGET_LIST=$2
|
|
412
|
+
|
|
413
|
+
if [ -z "$SOURCE_LIST" ] || [ -z "$TARGET_LIST" ]; then
|
|
414
|
+
echo "Usage: $0 <source-list-id> <target-list-id>"
|
|
415
|
+
exit 1
|
|
416
|
+
fi
|
|
417
|
+
|
|
418
|
+
echo "=== Migrating Contacts ==="
|
|
419
|
+
echo "From: List $SOURCE_LIST"
|
|
420
|
+
echo "To: List $TARGET_LIST"
|
|
421
|
+
echo ""
|
|
422
|
+
|
|
423
|
+
# Export from source
|
|
424
|
+
echo "Exporting from source list..."
|
|
425
|
+
EXPORT_ID=$(cakemail contacts export $SOURCE_LIST -f json | jq -r '.id')
|
|
426
|
+
|
|
427
|
+
# Wait for export
|
|
428
|
+
while true; do
|
|
429
|
+
STATUS=$(cakemail contacts export-status $EXPORT_ID -f json | jq -r '.status')
|
|
430
|
+
[ "$STATUS" == "completed" ] && break
|
|
431
|
+
sleep 2
|
|
432
|
+
done
|
|
433
|
+
|
|
434
|
+
# Download
|
|
435
|
+
TEMP_FILE="temp-migration-$(date +%s).csv"
|
|
436
|
+
cakemail contacts export-download $EXPORT_ID > "$TEMP_FILE"
|
|
437
|
+
|
|
438
|
+
CONTACT_COUNT=$(wc -l < "$TEMP_FILE")
|
|
439
|
+
CONTACT_COUNT=$((CONTACT_COUNT - 1))
|
|
440
|
+
|
|
441
|
+
echo "✓ Exported $CONTACT_COUNT contacts"
|
|
442
|
+
echo ""
|
|
443
|
+
|
|
444
|
+
# Import to target
|
|
445
|
+
echo "Importing to target list..."
|
|
446
|
+
IMPORT_ID=$(cakemail contacts import $TARGET_LIST --file "$TEMP_FILE" -f json | jq -r '.id')
|
|
447
|
+
|
|
448
|
+
# Monitor import
|
|
449
|
+
while true; do
|
|
450
|
+
STATUS=$(cakemail contacts import-status $IMPORT_ID -f json)
|
|
451
|
+
STATE=$(echo "$STATUS" | jq -r '.status')
|
|
452
|
+
|
|
453
|
+
if [ "$STATE" == "completed" ]; then
|
|
454
|
+
IMPORTED=$(echo "$STATUS" | jq -r '.imported')
|
|
455
|
+
FAILED=$(echo "$STATUS" | jq -r '.failed')
|
|
456
|
+
|
|
457
|
+
echo "✓ Import complete"
|
|
458
|
+
echo " Imported: $IMPORTED"
|
|
459
|
+
echo " Failed: $FAILED"
|
|
460
|
+
break
|
|
461
|
+
elif [ "$STATE" == "failed" ]; then
|
|
462
|
+
echo "❌ Import failed"
|
|
463
|
+
break
|
|
464
|
+
fi
|
|
465
|
+
|
|
466
|
+
sleep 2
|
|
467
|
+
done
|
|
468
|
+
|
|
469
|
+
# Cleanup
|
|
470
|
+
rm "$TEMP_FILE"
|
|
471
|
+
|
|
472
|
+
echo ""
|
|
473
|
+
echo "✓ Migration complete"
|
|
474
|
+
```
|
|
475
|
+
|
|
476
|
+
### Workflow 3: Data Transformation
|
|
477
|
+
|
|
478
|
+
```bash
|
|
479
|
+
#!/bin/bash
|
|
480
|
+
# transform-and-import.sh
|
|
481
|
+
|
|
482
|
+
SOURCE_FILE=$1
|
|
483
|
+
LIST_ID=$2
|
|
484
|
+
|
|
485
|
+
if [ -z "$SOURCE_FILE" ] || [ -z "$LIST_ID" ]; then
|
|
486
|
+
echo "Usage: $0 <source-file> <list-id>"
|
|
487
|
+
exit 1
|
|
488
|
+
fi
|
|
489
|
+
|
|
490
|
+
echo "=== Transforming Data ==="
|
|
491
|
+
echo ""
|
|
492
|
+
|
|
493
|
+
TRANSFORMED_FILE="transformed-$(date +%s).csv"
|
|
494
|
+
|
|
495
|
+
# Write header
|
|
496
|
+
echo "email,first_name,last_name,plan,signup_date" > "$TRANSFORMED_FILE"
|
|
497
|
+
|
|
498
|
+
# Transform data (example: clean and normalize)
|
|
499
|
+
tail -n +2 "$SOURCE_FILE" | while IFS=, read email fname lname plan date; do
|
|
500
|
+
# Clean email (lowercase, trim)
|
|
501
|
+
email=$(echo "$email" | tr '[:upper:]' '[:lower:]' | xargs)
|
|
502
|
+
|
|
503
|
+
# Capitalize names
|
|
504
|
+
fname=$(echo "$fname" | awk '{print toupper(substr($0,1,1)) tolower(substr($0,2))}')
|
|
505
|
+
lname=$(echo "$lname" | awk '{print toupper(substr($0,1,1)) tolower(substr($0,2))}')
|
|
506
|
+
|
|
507
|
+
# Validate email format
|
|
508
|
+
if [[ "$email" =~ ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ ]]; then
|
|
509
|
+
echo "$email,$fname,$lname,$plan,$date" >> "$TRANSFORMED_FILE"
|
|
510
|
+
else
|
|
511
|
+
echo "⚠️ Skipped invalid email: $email" >&2
|
|
512
|
+
fi
|
|
513
|
+
done
|
|
514
|
+
|
|
515
|
+
echo "✓ Transformation complete"
|
|
516
|
+
echo ""
|
|
517
|
+
|
|
518
|
+
# Import transformed data
|
|
519
|
+
echo "Importing transformed data..."
|
|
520
|
+
cakemail contacts import $LIST_ID --file "$TRANSFORMED_FILE"
|
|
521
|
+
|
|
522
|
+
echo ""
|
|
523
|
+
echo "✓ Import started"
|
|
524
|
+
echo " Transformed file: $TRANSFORMED_FILE"
|
|
525
|
+
```
|
|
526
|
+
|
|
527
|
+
### Workflow 4: Deduplicate During Import
|
|
528
|
+
|
|
529
|
+
```bash
|
|
530
|
+
#!/bin/bash
|
|
531
|
+
# deduplicate-import.sh
|
|
532
|
+
|
|
533
|
+
SOURCE_FILE=$1
|
|
534
|
+
LIST_ID=$2
|
|
535
|
+
|
|
536
|
+
if [ -z "$SOURCE_FILE" ] || [ -z "$LIST_ID" ]; then
|
|
537
|
+
echo "Usage: $0 <csv-file> <list-id>"
|
|
538
|
+
exit 1
|
|
539
|
+
fi
|
|
540
|
+
|
|
541
|
+
echo "=== Deduplicating Import ==="
|
|
542
|
+
echo ""
|
|
543
|
+
|
|
544
|
+
# Get existing emails from list
|
|
545
|
+
echo "Fetching existing contacts..."
|
|
546
|
+
EXPORT_ID=$(cakemail contacts export $LIST_ID -f json | jq -r '.id')
|
|
547
|
+
|
|
548
|
+
while true; do
|
|
549
|
+
STATUS=$(cakemail contacts export-status $EXPORT_ID -f json | jq -r '.status')
|
|
550
|
+
[ "$STATUS" == "completed" ] && break
|
|
551
|
+
sleep 2
|
|
552
|
+
done
|
|
553
|
+
|
|
554
|
+
EXISTING_FILE="existing-$(date +%s).csv"
|
|
555
|
+
cakemail contacts export-download $EXPORT_ID > "$EXISTING_FILE"
|
|
556
|
+
|
|
557
|
+
# Extract existing emails
|
|
558
|
+
cut -d',' -f1 "$EXISTING_FILE" | tail -n +2 | sort > existing_emails.txt
|
|
559
|
+
|
|
560
|
+
EXISTING_COUNT=$(wc -l < existing_emails.txt)
|
|
561
|
+
echo " Found $EXISTING_COUNT existing contacts"
|
|
562
|
+
echo ""
|
|
563
|
+
|
|
564
|
+
# Filter new contacts
|
|
565
|
+
DEDUPED_FILE="deduped-$(date +%s).csv"
|
|
566
|
+
head -1 "$SOURCE_FILE" > "$DEDUPED_FILE"
|
|
567
|
+
|
|
568
|
+
IMPORTED=0
|
|
569
|
+
SKIPPED=0
|
|
570
|
+
|
|
571
|
+
tail -n +2 "$SOURCE_FILE" | while IFS=, read email rest; do
|
|
572
|
+
# Check if email exists
|
|
573
|
+
if ! grep -qx "$email" existing_emails.txt; then
|
|
574
|
+
echo "$email,$rest" >> "$DEDUPED_FILE"
|
|
575
|
+
IMPORTED=$((IMPORTED + 1))
|
|
576
|
+
else
|
|
577
|
+
SKIPPED=$((SKIPPED + 1))
|
|
578
|
+
fi
|
|
579
|
+
done
|
|
580
|
+
|
|
581
|
+
echo "Deduplication results:"
|
|
582
|
+
echo " New contacts: $IMPORTED"
|
|
583
|
+
echo " Duplicates skipped: $SKIPPED"
|
|
584
|
+
echo ""
|
|
585
|
+
|
|
586
|
+
# Import deduplicated file
|
|
587
|
+
if [ $IMPORTED -gt 0 ]; then
|
|
588
|
+
echo "Importing new contacts..."
|
|
589
|
+
cakemail contacts import $LIST_ID --file "$DEDUPED_FILE"
|
|
590
|
+
echo "✓ Import started"
|
|
591
|
+
else
|
|
592
|
+
echo "No new contacts to import"
|
|
593
|
+
fi
|
|
594
|
+
|
|
595
|
+
# Cleanup
|
|
596
|
+
rm existing_emails.txt "$EXISTING_FILE"
|
|
597
|
+
```
|
|
598
|
+
|
|
599
|
+
### Workflow 5: Split Import for Large Files
|
|
600
|
+
|
|
601
|
+
```bash
|
|
602
|
+
#!/bin/bash
|
|
603
|
+
# split-import.sh
|
|
604
|
+
|
|
605
|
+
LARGE_FILE=$1
|
|
606
|
+
LIST_ID=$2
|
|
607
|
+
CHUNK_SIZE=5000
|
|
608
|
+
|
|
609
|
+
if [ -z "$LARGE_FILE" ] || [ -z "$LIST_ID" ]; then
|
|
610
|
+
echo "Usage: $0 <large-csv-file> <list-id>"
|
|
611
|
+
exit 1
|
|
612
|
+
fi
|
|
613
|
+
|
|
614
|
+
echo "=== Split Import ==="
|
|
615
|
+
echo "File: $LARGE_FILE"
|
|
616
|
+
echo "Chunk size: $CHUNK_SIZE"
|
|
617
|
+
echo ""
|
|
618
|
+
|
|
619
|
+
# Count total rows
|
|
620
|
+
TOTAL_ROWS=$(wc -l < "$LARGE_FILE")
|
|
621
|
+
TOTAL_ROWS=$((TOTAL_ROWS - 1)) # Exclude header
|
|
622
|
+
|
|
623
|
+
echo "Total contacts: $TOTAL_ROWS"
|
|
624
|
+
echo ""
|
|
625
|
+
|
|
626
|
+
# Get header
|
|
627
|
+
HEADER=$(head -1 "$LARGE_FILE")
|
|
628
|
+
|
|
629
|
+
# Split file
|
|
630
|
+
CHUNKS=$((TOTAL_ROWS / CHUNK_SIZE + 1))
|
|
631
|
+
echo "Splitting into $CHUNKS chunks..."
|
|
632
|
+
|
|
633
|
+
tail -n +2 "$LARGE_FILE" | split -l $CHUNK_SIZE - chunk_
|
|
634
|
+
|
|
635
|
+
# Add header to each chunk
|
|
636
|
+
for chunk in chunk_*; do
|
|
637
|
+
echo "$HEADER" | cat - "$chunk" > temp
|
|
638
|
+
mv temp "$chunk.csv"
|
|
639
|
+
rm "$chunk"
|
|
640
|
+
done
|
|
641
|
+
|
|
642
|
+
echo "✓ File split complete"
|
|
643
|
+
echo ""
|
|
644
|
+
|
|
645
|
+
# Import each chunk
|
|
646
|
+
CHUNK_NUM=1
|
|
647
|
+
for chunk in chunk_*.csv; do
|
|
648
|
+
echo "Importing chunk $CHUNK_NUM/$CHUNKS..."
|
|
649
|
+
|
|
650
|
+
IMPORT_ID=$(cakemail contacts import $LIST_ID --file "$chunk" -f json | jq -r '.id')
|
|
651
|
+
|
|
652
|
+
# Wait for completion
|
|
653
|
+
while true; do
|
|
654
|
+
STATUS=$(cakemail contacts import-status $IMPORT_ID -f json | jq -r '.status')
|
|
655
|
+
[ "$STATUS" == "completed" ] && break
|
|
656
|
+
sleep 5
|
|
657
|
+
done
|
|
658
|
+
|
|
659
|
+
echo " ✓ Chunk $CHUNK_NUM complete"
|
|
660
|
+
rm "$chunk"
|
|
661
|
+
|
|
662
|
+
CHUNK_NUM=$((CHUNK_NUM + 1))
|
|
663
|
+
done
|
|
664
|
+
|
|
665
|
+
echo ""
|
|
666
|
+
echo "✓ All chunks imported"
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
## Data Quality
|
|
670
|
+
|
|
671
|
+
### Validate CSV Before Import
|
|
672
|
+
|
|
673
|
+
```bash
|
|
674
|
+
#!/bin/bash
|
|
675
|
+
# validate-csv.sh
|
|
676
|
+
|
|
677
|
+
CSV_FILE=$1
|
|
678
|
+
|
|
679
|
+
if [ -z "$CSV_FILE" ]; then
|
|
680
|
+
echo "Usage: $0 <csv-file>"
|
|
681
|
+
exit 1
|
|
682
|
+
fi
|
|
683
|
+
|
|
684
|
+
echo "=== CSV Validation ==="
|
|
685
|
+
echo "File: $CSV_FILE"
|
|
686
|
+
echo ""
|
|
687
|
+
|
|
688
|
+
# Check file exists
|
|
689
|
+
if [ ! -f "$CSV_FILE" ]; then
|
|
690
|
+
echo "❌ File not found"
|
|
691
|
+
exit 1
|
|
692
|
+
fi
|
|
693
|
+
|
|
694
|
+
# Check encoding
|
|
695
|
+
ENCODING=$(file -b --mime-encoding "$CSV_FILE")
|
|
696
|
+
echo "Encoding: $ENCODING"
|
|
697
|
+
if [ "$ENCODING" != "utf-8" ] && [ "$ENCODING" != "us-ascii" ]; then
|
|
698
|
+
echo "⚠️ Warning: File should be UTF-8 encoded"
|
|
699
|
+
fi
|
|
700
|
+
|
|
701
|
+
# Check header
|
|
702
|
+
HEADER=$(head -1 "$CSV_FILE")
|
|
703
|
+
echo "Header: $HEADER"
|
|
704
|
+
|
|
705
|
+
if [[ ! "$HEADER" =~ "email" ]]; then
|
|
706
|
+
echo "❌ Missing required 'email' column"
|
|
707
|
+
exit 1
|
|
708
|
+
fi
|
|
709
|
+
|
|
710
|
+
# Count rows
|
|
711
|
+
TOTAL_ROWS=$(wc -l < "$CSV_FILE")
|
|
712
|
+
DATA_ROWS=$((TOTAL_ROWS - 1))
|
|
713
|
+
echo "Data rows: $DATA_ROWS"
|
|
714
|
+
|
|
715
|
+
# Check for empty emails
|
|
716
|
+
EMPTY_EMAILS=$(awk -F',' 'NR>1 && $1==""' "$CSV_FILE" | wc -l)
|
|
717
|
+
if [ $EMPTY_EMAILS -gt 0 ]; then
|
|
718
|
+
echo "⚠️ Warning: $EMPTY_EMAILS rows with empty email"
|
|
719
|
+
fi
|
|
720
|
+
|
|
721
|
+
# Validate email format (sample)
|
|
722
|
+
echo ""
|
|
723
|
+
echo "Validating email format (first 10)..."
|
|
724
|
+
awk -F',' 'NR>1 && NR<=11 {print $1}' "$CSV_FILE" | while read email; do
|
|
725
|
+
if [[ ! "$email" =~ ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ ]]; then
|
|
726
|
+
echo " ❌ Invalid: $email"
|
|
727
|
+
else
|
|
728
|
+
echo " ✓ Valid: $email"
|
|
729
|
+
fi
|
|
730
|
+
done
|
|
731
|
+
|
|
732
|
+
echo ""
|
|
733
|
+
echo "✓ Validation complete"
|
|
734
|
+
```
|
|
735
|
+
|
|
736
|
+
### Clean CSV Data
|
|
737
|
+
|
|
738
|
+
```bash
|
|
739
|
+
#!/bin/bash
|
|
740
|
+
# clean-csv.sh
|
|
741
|
+
|
|
742
|
+
INPUT_FILE=$1
|
|
743
|
+
OUTPUT_FILE="cleaned-$(basename $INPUT_FILE)"
|
|
744
|
+
|
|
745
|
+
if [ -z "$INPUT_FILE" ]; then
|
|
746
|
+
echo "Usage: $0 <input-csv>"
|
|
747
|
+
exit 1
|
|
748
|
+
fi
|
|
749
|
+
|
|
750
|
+
echo "=== Cleaning CSV ==="
|
|
751
|
+
echo "Input: $INPUT_FILE"
|
|
752
|
+
echo "Output: $OUTPUT_FILE"
|
|
753
|
+
echo ""
|
|
754
|
+
|
|
755
|
+
# Get header
|
|
756
|
+
HEADER=$(head -1 "$INPUT_FILE")
|
|
757
|
+
echo "$HEADER" > "$OUTPUT_FILE"
|
|
758
|
+
|
|
759
|
+
CLEANED=0
|
|
760
|
+
INVALID=0
|
|
761
|
+
|
|
762
|
+
# Process each row
|
|
763
|
+
tail -n +2 "$INPUT_FILE" | while IFS=, read email fname lname rest; do
|
|
764
|
+
# Trim whitespace
|
|
765
|
+
email=$(echo "$email" | xargs)
|
|
766
|
+
fname=$(echo "$fname" | xargs)
|
|
767
|
+
lname=$(echo "$lname" | xargs)
|
|
768
|
+
|
|
769
|
+
# Lowercase email
|
|
770
|
+
email=$(echo "$email" | tr '[:upper:]' '[:lower:]')
|
|
771
|
+
|
|
772
|
+
# Validate email
|
|
773
|
+
if [[ "$email" =~ ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ ]]; then
|
|
774
|
+
echo "$email,$fname,$lname,$rest" >> "$OUTPUT_FILE"
|
|
775
|
+
CLEANED=$((CLEANED + 1))
|
|
776
|
+
else
|
|
777
|
+
INVALID=$((INVALID + 1))
|
|
778
|
+
echo "Invalid: $email" >> "invalid-emails.txt"
|
|
779
|
+
fi
|
|
780
|
+
done
|
|
781
|
+
|
|
782
|
+
echo "✓ Cleaning complete"
|
|
783
|
+
echo " Cleaned: $CLEANED"
|
|
784
|
+
echo " Invalid: $INVALID"
|
|
785
|
+
echo ""
|
|
786
|
+
echo "Output: $OUTPUT_FILE"
|
|
787
|
+
[ $INVALID -gt 0 ] && echo "Invalid emails: invalid-emails.txt"
|
|
788
|
+
```
|
|
789
|
+
|
|
790
|
+
## Error Handling
|
|
791
|
+
|
|
792
|
+
### Handle Import Errors
|
|
793
|
+
|
|
794
|
+
```bash
|
|
795
|
+
#!/bin/bash
|
|
796
|
+
# handle-import-errors.sh
|
|
797
|
+
|
|
798
|
+
IMPORT_ID=$1
|
|
799
|
+
|
|
800
|
+
if [ -z "$IMPORT_ID" ]; then
|
|
801
|
+
echo "Usage: $0 <import-id>"
|
|
802
|
+
exit 1
|
|
803
|
+
fi
|
|
804
|
+
|
|
805
|
+
echo "=== Import Error Analysis ==="
|
|
806
|
+
echo ""
|
|
807
|
+
|
|
808
|
+
# Get import status
|
|
809
|
+
STATUS=$(cakemail contacts import-status $IMPORT_ID -f json)
|
|
810
|
+
|
|
811
|
+
TOTAL=$(echo "$STATUS" | jq -r '.total_rows')
|
|
812
|
+
IMPORTED=$(echo "$STATUS" | jq -r '.imported')
|
|
813
|
+
FAILED=$(echo "$STATUS" | jq -r '.failed')
|
|
814
|
+
|
|
815
|
+
echo "Total: $TOTAL"
|
|
816
|
+
echo "Imported: $IMPORTED"
|
|
817
|
+
echo "Failed: $FAILED"
|
|
818
|
+
echo ""
|
|
819
|
+
|
|
820
|
+
if [ $FAILED -gt 0 ]; then
|
|
821
|
+
echo "=== Errors ==="
|
|
822
|
+
echo "$STATUS" | jq -r '.errors[] | "Row \(.row): \(.email) - \(.error)"'
|
|
823
|
+
|
|
824
|
+
echo ""
|
|
825
|
+
echo "Common fixes:"
|
|
826
|
+
echo " • Invalid email: Check email format"
|
|
827
|
+
echo " • Duplicate email: Contact already exists"
|
|
828
|
+
echo " • Invalid custom attribute: Ensure attribute exists"
|
|
829
|
+
echo " • Missing required field: Add email column"
|
|
830
|
+
fi
|
|
831
|
+
```
|
|
832
|
+
|
|
833
|
+
### Retry Failed Imports
|
|
834
|
+
|
|
835
|
+
```bash
|
|
836
|
+
#!/bin/bash
|
|
837
|
+
# retry-failed.sh
|
|
838
|
+
|
|
839
|
+
LIST_ID=$1
|
|
840
|
+
IMPORT_ID=$2
|
|
841
|
+
|
|
842
|
+
if [ -z "$LIST_ID" ] || [ -z "$IMPORT_ID" ]; then
|
|
843
|
+
echo "Usage: $0 <list-id> <failed-import-id>"
|
|
844
|
+
exit 1
|
|
845
|
+
fi
|
|
846
|
+
|
|
847
|
+
echo "=== Retrying Failed Import ==="
|
|
848
|
+
echo ""
|
|
849
|
+
|
|
850
|
+
# Get failed rows
|
|
851
|
+
STATUS=$(cakemail contacts import-status $IMPORT_ID -f json)
|
|
852
|
+
ERRORS=$(echo "$STATUS" | jq -r '.errors[] | "\(.row),\(.email)"')
|
|
853
|
+
|
|
854
|
+
if [ -z "$ERRORS" ]; then
|
|
855
|
+
echo "No errors found"
|
|
856
|
+
exit 0
|
|
857
|
+
fi
|
|
858
|
+
|
|
859
|
+
echo "Found $(echo "$ERRORS" | wc -l) failed rows"
|
|
860
|
+
echo "Manual review required for each"
|
|
861
|
+
echo ""
|
|
862
|
+
|
|
863
|
+
# Process each failed row
|
|
864
|
+
echo "$ERRORS" | while IFS=, read row email; do
|
|
865
|
+
echo "Failed: Row $row, Email $email"
|
|
866
|
+
read -p "Fix and retry? (yes/skip): " ACTION
|
|
867
|
+
|
|
868
|
+
if [ "$ACTION" == "yes" ]; then
|
|
869
|
+
read -p "Corrected email: " CORRECTED_EMAIL
|
|
870
|
+
read -p "First name: " FIRST_NAME
|
|
871
|
+
read -p "Last name: " LAST_NAME
|
|
872
|
+
|
|
873
|
+
# Try adding corrected contact
|
|
874
|
+
cakemail contacts add $LIST_ID \
|
|
875
|
+
-e "$CORRECTED_EMAIL" \
|
|
876
|
+
-f "$FIRST_NAME" \
|
|
877
|
+
-l "$LAST_NAME"
|
|
878
|
+
|
|
879
|
+
echo "✓ Retried"
|
|
880
|
+
fi
|
|
881
|
+
|
|
882
|
+
echo ""
|
|
883
|
+
done
|
|
884
|
+
|
|
885
|
+
echo "✓ Retry complete"
|
|
886
|
+
```
|
|
887
|
+
|
|
888
|
+
## Best Practices
|
|
889
|
+
|
|
890
|
+
### 1. Validate Before Importing
|
|
891
|
+
|
|
892
|
+
```bash
|
|
893
|
+
# Always validate CSV first
|
|
894
|
+
$ ./validate-csv.sh contacts.csv
|
|
895
|
+
|
|
896
|
+
# Fix issues
|
|
897
|
+
$ ./clean-csv.sh contacts.csv
|
|
898
|
+
|
|
899
|
+
# Then import
|
|
900
|
+
$ cakemail contacts import 123 --file cleaned-contacts.csv
|
|
901
|
+
```
|
|
902
|
+
|
|
903
|
+
### 2. Test with Small Sample
|
|
904
|
+
|
|
905
|
+
```bash
|
|
906
|
+
# Create test file with first 10 rows
|
|
907
|
+
$ head -11 large-file.csv > test-sample.csv
|
|
908
|
+
|
|
909
|
+
# Test import
|
|
910
|
+
$ cakemail contacts import 123 --file test-sample.csv
|
|
911
|
+
|
|
912
|
+
# Check for errors
|
|
913
|
+
$ cakemail contacts import-status <import-id>
|
|
914
|
+
|
|
915
|
+
# If successful, import full file
|
|
916
|
+
$ cakemail contacts import 123 --file large-file.csv
|
|
917
|
+
```
|
|
918
|
+
|
|
919
|
+
### 3. Regular Backups
|
|
920
|
+
|
|
921
|
+
```bash
|
|
922
|
+
# Weekly backup
|
|
923
|
+
0 0 * * 0 /path/to/backup-contacts.sh
|
|
924
|
+
|
|
925
|
+
# Keep 4 weeks of backups
|
|
926
|
+
find /backups -name "*.csv.gz" -mtime +28 -delete
|
|
927
|
+
```
|
|
928
|
+
|
|
929
|
+
### 4. Monitor Long Imports
|
|
930
|
+
|
|
931
|
+
```bash
|
|
932
|
+
# Large imports can take time
|
|
933
|
+
# Use monitoring script
|
|
934
|
+
$ ./monitor-import.sh imp_abc123
|
|
935
|
+
|
|
936
|
+
# Or schedule notification
|
|
937
|
+
$ cakemail contacts import 123 --file large.csv && \
|
|
938
|
+
echo "Import complete" | mail -s "Import Done" admin@company.com
|
|
939
|
+
```
|
|
940
|
+
|
|
941
|
+
### 5. Handle Duplicates Proactively
|
|
942
|
+
|
|
943
|
+
```bash
|
|
944
|
+
# Use deduplication workflow
|
|
945
|
+
$ ./deduplicate-import.sh new-contacts.csv 123
|
|
946
|
+
|
|
947
|
+
# Or merge with existing
|
|
948
|
+
# Export, merge externally, then import to new list
|
|
949
|
+
```
|
|
950
|
+
|
|
951
|
+
## Troubleshooting
|
|
952
|
+
|
|
953
|
+
### Import Stuck in "Processing"
|
|
954
|
+
|
|
955
|
+
**Problem:** Import status stays "processing" for long time
|
|
956
|
+
|
|
957
|
+
**Solutions:**
|
|
958
|
+
```bash
|
|
959
|
+
# Large files take time (allow 1 minute per 1000 contacts)
|
|
960
|
+
|
|
961
|
+
# Check status periodically
|
|
962
|
+
$ cakemail contacts import-status imp_abc123
|
|
963
|
+
|
|
964
|
+
# If truly stuck (> 30 mins for < 10k contacts):
|
|
965
|
+
# - Contact support
|
|
966
|
+
# - Try splitting file into smaller chunks
|
|
967
|
+
```
|
|
968
|
+
|
|
969
|
+
### Export Download Fails
|
|
970
|
+
|
|
971
|
+
**Problem:** Cannot download export
|
|
972
|
+
|
|
973
|
+
**Solutions:**
|
|
974
|
+
```bash
|
|
975
|
+
# Check export status first
|
|
976
|
+
$ cakemail contacts export-status exp_xyz789
|
|
977
|
+
|
|
978
|
+
# Ensure export completed
|
|
979
|
+
# Wait if status is "processing"
|
|
980
|
+
|
|
981
|
+
# Check if expired
|
|
982
|
+
# Exports expire after 7 days
|
|
983
|
+
# Create new export if expired
|
|
984
|
+
|
|
985
|
+
$ cakemail contacts export 123
|
|
986
|
+
```
|
|
987
|
+
|
|
988
|
+
### CSV Encoding Issues
|
|
989
|
+
|
|
990
|
+
**Problem:** Special characters appear incorrectly
|
|
991
|
+
|
|
992
|
+
**Solutions:**
|
|
993
|
+
```bash
|
|
994
|
+
# Convert to UTF-8
|
|
995
|
+
$ iconv -f ISO-8859-1 -t UTF-8 input.csv > output.csv
|
|
996
|
+
|
|
997
|
+
# Or use spreadsheet software:
|
|
998
|
+
# - Open CSV
|
|
999
|
+
# - Save As > Format: CSV UTF-8
|
|
1000
|
+
|
|
1001
|
+
# Then import
|
|
1002
|
+
$ cakemail contacts import 123 --file output.csv
|
|
1003
|
+
```
|
|
1004
|
+
|
|
1005
|
+
### Large File Import Fails
|
|
1006
|
+
|
|
1007
|
+
**Problem:** Import fails for very large files
|
|
1008
|
+
|
|
1009
|
+
**Solutions:**
|
|
1010
|
+
```bash
|
|
1011
|
+
# Split into smaller chunks
|
|
1012
|
+
$ ./split-import.sh huge-file.csv 123
|
|
1013
|
+
|
|
1014
|
+
# Or compress and optimize
|
|
1015
|
+
# Remove unnecessary columns
|
|
1016
|
+
# Deduplicate before importing
|
|
1017
|
+
```
|
|
1018
|
+
|
|
1019
|
+
## Best Practices Summary
|
|
1020
|
+
|
|
1021
|
+
1. **Validate first** - Check CSV format before importing
|
|
1022
|
+
2. **Test with samples** - Import small batch first
|
|
1023
|
+
3. **Clean data** - Remove duplicates and invalid emails
|
|
1024
|
+
4. **Use UTF-8** - Ensure proper encoding
|
|
1025
|
+
5. **Monitor progress** - Watch long-running imports
|
|
1026
|
+
6. **Handle errors** - Review and fix failed rows
|
|
1027
|
+
7. **Backup regularly** - Export contacts periodically
|
|
1028
|
+
8. **Split large files** - Break into manageable chunks
|
|
1029
|
+
9. **Document transformations** - Keep scripts for repeatable processes
|
|
1030
|
+
10. **Schedule backups** - Automate with cron
|
|
1031
|
+
|