data_porter 2.5.1 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +41 -0
- data/README.md +2 -0
- data/ROADMAP.md +0 -4
- data/app/assets/javascripts/data_porter/inline_edit_controller.js +189 -0
- data/app/assets/stylesheets/data_porter/application.css +1 -0
- data/app/assets/stylesheets/data_porter/inline_edit.css +122 -0
- data/app/controllers/data_porter/imports_controller.rb +25 -1
- data/app/models/data_porter/data_import.rb +5 -1
- data/app/views/data_porter/imports/show.html.erb +9 -3
- data/app/views/layouts/data_porter/application.html.erb +4 -1
- data/config/locales/en.yml +3 -0
- data/config/locales/fr.yml +3 -0
- data/config/routes.rb +2 -0
- data/lib/data_porter/components/preview/summary_cards.rb +22 -9
- data/lib/data_porter/components/preview/table.rb +65 -5
- data/lib/data_porter/engine.rb +1 -0
- data/lib/data_porter/orchestrator/bulk_importer.rb +35 -27
- data/lib/data_porter/orchestrator/importer.rb +25 -3
- data/lib/data_porter/orchestrator.rb +25 -6
- data/lib/data_porter/record_revalidator.rb +33 -0
- data/lib/data_porter/record_updater.rb +95 -0
- data/lib/data_porter/version.rb +1 -1
- data/lib/data_porter.rb +2 -0
- data/mkdocs.yml +1 -1
- metadata +5 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 31999033cc55a873fc7eb19520cd1e4a970accea6ee00f1c137b3f17c64f23d9
|
|
4
|
+
data.tar.gz: f23524c9154af04b4c02ad6196a4fd036a5dc1e5c8f788c7807d1383e1462f9b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 28faa4317369a57e1fca2999a54232a2404ae25034d90d2bdc42ad1b7dd4f090397f98e4c9643c0e5e5029c41d9dcaf37d3003068bf454b56b3fea1b837c7a98
|
|
7
|
+
data.tar.gz: dada848b7498e58080cff0a1147a5b345eb996feb0c0eb7f715c611283767c782afa66899115da8436592aa9588a3cfb1541403e07ad2446b792cc8850a8cd5e
|
data/CHANGELOG.md
CHANGED
|
@@ -7,10 +7,51 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [2.8.0] - 2026-03-08
|
|
11
|
+
|
|
10
12
|
### Added
|
|
11
13
|
|
|
12
14
|
- **Auto-map heuristics** -- Smart column suggestions that pre-fill mapping selects when CSV/XLSX headers match target fields by exact name or built-in synonym (e.g. "E-mail Address" → email, "fname" → first_name). Supports per-column custom synonyms via `synonyms:` keyword in column DSL. Fallback chain: saved mapping > code-defined > auto-map > empty
|
|
13
15
|
|
|
16
|
+
### Fixed
|
|
17
|
+
|
|
18
|
+
- Inline edit cell selector now uses `closest("td")` instead of `closest("[data-column]")` to avoid matching the input element itself
|
|
19
|
+
|
|
20
|
+
## [2.7.0] - 2026-02-22
|
|
21
|
+
|
|
22
|
+
### Added
|
|
23
|
+
|
|
24
|
+
- **Inline edit in preview** -- Click any cell in the preview table to fix validation errors directly, without re-uploading the file. Optimistic UI with background validation: click cell → input appears → edit → blur/Enter → value updates instantly → PATCH to server → re-validate → update status, errors, and summary cards
|
|
25
|
+
- `RecordRevalidator` service -- Re-validates an existing record after data change using the same pipeline as initial parse (transforms + target validation + built-in validation), with JSONB string-key normalization for post-persistence records
|
|
26
|
+
- `RecordUpdater` service -- Orchestrates cell update, re-validation, delta-based report update, and persistence. Returns transformed value, status, errors, and updated report counts
|
|
27
|
+
- `PATCH :update_record` route and controller action with `ensure_previewing` guard
|
|
28
|
+
- Stimulus `inline_edit_controller` -- Click-to-edit cells with Enter/blur to save, Escape to cancel, Tab/Shift+Tab to navigate between cells. Updates row status, errors cell, and summary card counts from server response
|
|
29
|
+
- Editable mode on `Preview::Table` and `Preview::SummaryCards` Phlex components -- `editable: true` renders Stimulus controller, click actions, data attributes on cells. Non-editable mode (completed/failed views) renders exactly as before
|
|
30
|
+
- Inline edit CSS -- editable cell hover, inline input, saving/error/success animation states
|
|
31
|
+
- `edit_cell` and `saving` locale keys (en + fr)
|
|
32
|
+
|
|
33
|
+
### Changed
|
|
34
|
+
|
|
35
|
+
- 638 RSpec examples (up from 574), 0 failures
|
|
36
|
+
|
|
37
|
+
## [2.6.0] - 2026-02-21
|
|
38
|
+
|
|
39
|
+
### Added
|
|
40
|
+
|
|
41
|
+
- **Resume on failure** -- When an import fails mid-way (crash, timeout, exception), resume from the last successful record instead of re-importing from scratch. Progress checkpoints stored in the existing `config` JSONB column alongside `broadcast_progress` — zero additional DB operations or migrations. Works with both per-record and bulk import modes
|
|
42
|
+
- `resumable?` predicate on `DataImport` — returns `true` when a failed import has a checkpoint with processed records
|
|
43
|
+
- Resume button in the failed import UI (primary action), with Retry demoted to secondary
|
|
44
|
+
- `POST :resume` route on the imports controller
|
|
45
|
+
|
|
46
|
+
### Fixed
|
|
47
|
+
|
|
48
|
+
- `handle_failure` now preserves existing report data (parsed counts, partial results) instead of creating a new empty report
|
|
49
|
+
- `parse!` now clears stale checkpoint and progress data from previous import attempts
|
|
50
|
+
|
|
51
|
+
### Changed
|
|
52
|
+
|
|
53
|
+
- 574 RSpec examples (up from 551), 0 failures
|
|
54
|
+
|
|
14
55
|
## [2.5.1] - 2026-02-21
|
|
15
56
|
|
|
16
57
|
### Fixed
|
data/README.md
CHANGED
|
@@ -103,6 +103,8 @@ pending -> parsing -> previewing -> importing -> completed
|
|
|
103
103
|
|
|
104
104
|
**[Full documentation on GitHub Pages](https://seryllns.github.io/data_porter/)**
|
|
105
105
|
|
|
106
|
+
> **Build series**: Want to see how DataPorter was built step by step? [Building DataPorter on dev.to](https://dev.to/seryllns_/series/35813) -- 30 parts covering architecture, TDD, and every feature from first commit to production.
|
|
107
|
+
|
|
106
108
|
| Topic | Description |
|
|
107
109
|
|---|---|
|
|
108
110
|
| [Configuration](docs/CONFIGURATION.md) | All options, authentication, context builder, real-time updates |
|
data/ROADMAP.md
CHANGED
|
@@ -6,10 +6,6 @@
|
|
|
6
6
|
|
|
7
7
|
Support update (upsert) imports alongside create-only. Given a `deduplicate_by` key, detect existing records and show a diff preview: new records, changed fields (highlighted), unchanged rows. User confirms which changes to apply. Enables recurring data sync workflows.
|
|
8
8
|
|
|
9
|
-
### Resume / retry on failure
|
|
10
|
-
|
|
11
|
-
If an import fails mid-way (timeout, crash, transient error), resume from the last successful record instead of restarting from scratch. Track a checkpoint index in the report. Critical for large imports (5k+ records) where re-processing everything is not acceptable.
|
|
12
|
-
|
|
13
9
|
### API pagination
|
|
14
10
|
|
|
15
11
|
Support paginated API sources. The current API source does a single GET, which works for small datasets but not for APIs returning thousands of records across multiple pages. Support offset, cursor, and link-header pagination strategies via `api_config`:
|
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
import { Controller } from "@hotwired/stimulus"
|
|
2
|
+
|
|
3
|
+
export default class extends Controller {
|
|
4
|
+
static targets = ["cell", "errors", "summary"]
|
|
5
|
+
static values = { updateUrl: String }
|
|
6
|
+
|
|
7
|
+
edit(event) {
|
|
8
|
+
var cell = event.currentTarget
|
|
9
|
+
if (cell.querySelector(".dp-inline-input")) return
|
|
10
|
+
|
|
11
|
+
var value = cell.dataset.value || ""
|
|
12
|
+
cell.dataset.originalValue = value
|
|
13
|
+
cell.textContent = ""
|
|
14
|
+
cell.classList.add("dp-cell--editing")
|
|
15
|
+
|
|
16
|
+
var input = document.createElement("input")
|
|
17
|
+
input.type = "text"
|
|
18
|
+
input.className = "dp-inline-input"
|
|
19
|
+
input.value = value
|
|
20
|
+
input.dataset.lineNumber = cell.dataset.lineNumber
|
|
21
|
+
input.dataset.column = cell.dataset.column
|
|
22
|
+
|
|
23
|
+
var self = this
|
|
24
|
+
input._blurHandler = function(e) { self.save(e) }
|
|
25
|
+
input._keyHandler = function(e) { self.handleKey(e) }
|
|
26
|
+
input.addEventListener("blur", input._blurHandler)
|
|
27
|
+
input.addEventListener("keydown", input._keyHandler)
|
|
28
|
+
cell.appendChild(input)
|
|
29
|
+
this.autoSize(input)
|
|
30
|
+
input.addEventListener("input", function() { self.autoSize(input) })
|
|
31
|
+
input.focus()
|
|
32
|
+
input.select()
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
handleKey(event) {
|
|
36
|
+
if (event.key === "Enter") {
|
|
37
|
+
event.preventDefault()
|
|
38
|
+
event.target.blur()
|
|
39
|
+
} else if (event.key === "Escape") {
|
|
40
|
+
event.preventDefault()
|
|
41
|
+
this.cancel(event.target)
|
|
42
|
+
} else if (event.key === "Tab") {
|
|
43
|
+
event.preventDefault()
|
|
44
|
+
var cell = event.target.closest("td")
|
|
45
|
+
event.target.blur()
|
|
46
|
+
this.moveToNext(cell, event.shiftKey)
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
cancel(input) {
|
|
51
|
+
var cell = input.closest("td")
|
|
52
|
+
input.removeEventListener("blur", input._blurHandler)
|
|
53
|
+
input.removeEventListener("keydown", input._keyHandler)
|
|
54
|
+
cell.classList.remove("dp-cell--editing")
|
|
55
|
+
var original = cell.dataset.originalValue || ""
|
|
56
|
+
cell.textContent = original
|
|
57
|
+
delete cell.dataset.originalValue
|
|
58
|
+
cell.blur()
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
save(event) {
|
|
62
|
+
var input = event.target
|
|
63
|
+
var cell = input.closest("td")
|
|
64
|
+
if (!cell) return
|
|
65
|
+
|
|
66
|
+
input.removeEventListener("blur", input._blurHandler)
|
|
67
|
+
input.removeEventListener("keydown", input._keyHandler)
|
|
68
|
+
|
|
69
|
+
var newValue = input.value
|
|
70
|
+
var original = cell.dataset.originalValue
|
|
71
|
+
|
|
72
|
+
cell.classList.remove("dp-cell--editing")
|
|
73
|
+
|
|
74
|
+
if (newValue === original) {
|
|
75
|
+
cell.textContent = original
|
|
76
|
+
delete cell.dataset.originalValue
|
|
77
|
+
cell.blur()
|
|
78
|
+
return
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
cell.textContent = newValue
|
|
82
|
+
cell.classList.add("dp-cell--saving")
|
|
83
|
+
delete cell.dataset.originalValue
|
|
84
|
+
cell.blur()
|
|
85
|
+
|
|
86
|
+
this.patch(cell, {
|
|
87
|
+
line_number: parseInt(cell.dataset.lineNumber, 10),
|
|
88
|
+
column: cell.dataset.column,
|
|
89
|
+
value: newValue
|
|
90
|
+
})
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
async patch(cell, body) {
|
|
94
|
+
try {
|
|
95
|
+
var csrfMeta = document.querySelector("meta[name='csrf-token']")
|
|
96
|
+
var headers = {
|
|
97
|
+
"Content-Type": "application/json",
|
|
98
|
+
"Accept": "application/json"
|
|
99
|
+
}
|
|
100
|
+
if (csrfMeta) headers["X-CSRF-Token"] = csrfMeta.content
|
|
101
|
+
|
|
102
|
+
var response = await fetch(this.updateUrlValue, {
|
|
103
|
+
method: "PATCH",
|
|
104
|
+
headers: headers,
|
|
105
|
+
body: JSON.stringify(body)
|
|
106
|
+
})
|
|
107
|
+
|
|
108
|
+
if (!response.ok) throw new Error("Request failed")
|
|
109
|
+
|
|
110
|
+
var data = await response.json()
|
|
111
|
+
this.applyResponse(cell, data)
|
|
112
|
+
} catch (e) {
|
|
113
|
+
cell.classList.remove("dp-cell--saving")
|
|
114
|
+
cell.classList.add("dp-cell--error")
|
|
115
|
+
setTimeout(function() { cell.classList.remove("dp-cell--error") }, 2000)
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
applyResponse(cell, data) {
|
|
120
|
+
cell.classList.remove("dp-cell--saving")
|
|
121
|
+
cell.classList.add("dp-cell--success")
|
|
122
|
+
setTimeout(function() { cell.classList.remove("dp-cell--success") }, 800)
|
|
123
|
+
|
|
124
|
+
cell.textContent = data.value
|
|
125
|
+
cell.dataset.value = data.value
|
|
126
|
+
|
|
127
|
+
var lineNumber = cell.dataset.lineNumber
|
|
128
|
+
this.updateRowStatus(lineNumber, data.status)
|
|
129
|
+
this.updateErrors(lineNumber, data.errors)
|
|
130
|
+
this.updateSummary(data.report)
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
updateRowStatus(lineNumber, status) {
|
|
134
|
+
var row = this.element.querySelector("tr[data-line-number='" + lineNumber + "']")
|
|
135
|
+
if (!row) return
|
|
136
|
+
|
|
137
|
+
row.className = row.className.replace(/dp-row--\w+/, "dp-row--" + status)
|
|
138
|
+
var statusCell = row.children[1]
|
|
139
|
+
if (statusCell) statusCell.textContent = status
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
updateErrors(lineNumber, errors) {
|
|
143
|
+
var errorsCell = this.errorsTargets.find(function(el) {
|
|
144
|
+
return el.dataset.lineNumber === String(lineNumber)
|
|
145
|
+
})
|
|
146
|
+
if (errorsCell) errorsCell.textContent = (errors || []).join(", ")
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
updateSummary(report) {
|
|
150
|
+
if (!report) return
|
|
151
|
+
|
|
152
|
+
this.summaryTargets.forEach(function(el) {
|
|
153
|
+
var key = el.dataset.countKey
|
|
154
|
+
if (key && report[key] !== undefined) {
|
|
155
|
+
el.textContent = String(report[key])
|
|
156
|
+
}
|
|
157
|
+
})
|
|
158
|
+
}
|
|
159
|
+
|
|
160
|
+
autoSize(input) {
|
|
161
|
+
var span = document.createElement("span")
|
|
162
|
+
span.style.cssText = "position:absolute;visibility:hidden;white-space:pre;"
|
|
163
|
+
var computed = window.getComputedStyle(input)
|
|
164
|
+
span.style.font = computed.font
|
|
165
|
+
span.style.padding = computed.padding
|
|
166
|
+
span.style.border = computed.border
|
|
167
|
+
span.textContent = input.value || " "
|
|
168
|
+
document.body.appendChild(span)
|
|
169
|
+
var cell = input.closest("td")
|
|
170
|
+
var minWidth = cell ? cell.offsetWidth : 100
|
|
171
|
+
var tableRight = this.element.getBoundingClientRect().right
|
|
172
|
+
var cellLeft = cell.getBoundingClientRect().left
|
|
173
|
+
var maxWidth = tableRight - cellLeft - 4
|
|
174
|
+
var width = Math.min(Math.max(minWidth, span.offsetWidth + 4), maxWidth)
|
|
175
|
+
input.style.width = width + "px"
|
|
176
|
+
document.body.removeChild(span)
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
moveToNext(currentCell, reverse) {
|
|
180
|
+
var cells = Array.from(this.cellTargets)
|
|
181
|
+
var index = cells.indexOf(currentCell)
|
|
182
|
+
if (index === -1) return
|
|
183
|
+
|
|
184
|
+
var next = reverse ? index - 1 : index + 1
|
|
185
|
+
if (next >= 0 && next < cells.length) {
|
|
186
|
+
cells[next].click()
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
}
|
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
.dp-cell--editable {
|
|
2
|
+
cursor: pointer;
|
|
3
|
+
position: relative;
|
|
4
|
+
outline: none;
|
|
5
|
+
transition: background 0.15s ease;
|
|
6
|
+
}
|
|
7
|
+
|
|
8
|
+
.dp-cell--editable:focus,
|
|
9
|
+
.dp-cell--editable:focus-visible {
|
|
10
|
+
outline: none;
|
|
11
|
+
box-shadow: none;
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
.dp-cell--editable:hover:not(.dp-cell--editing):not(.dp-cell--saving) {
|
|
15
|
+
background: var(--dp-primary-light);
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
.dp-cell--editable:hover:not(.dp-cell--editing):not(.dp-cell--saving)::after {
|
|
19
|
+
content: "\f303";
|
|
20
|
+
font-family: "Font Awesome 6 Free";
|
|
21
|
+
font-weight: 900;
|
|
22
|
+
font-size: 0.65rem;
|
|
23
|
+
position: absolute;
|
|
24
|
+
top: 0.25rem;
|
|
25
|
+
right: 0.35rem;
|
|
26
|
+
color: var(--dp-primary);
|
|
27
|
+
opacity: 0.6;
|
|
28
|
+
pointer-events: none;
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
.dp-cell--editing {
|
|
32
|
+
padding: 0 !important;
|
|
33
|
+
background: transparent !important;
|
|
34
|
+
outline: none !important;
|
|
35
|
+
overflow: visible;
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
.dp-inline-input {
|
|
39
|
+
position: absolute;
|
|
40
|
+
top: 0;
|
|
41
|
+
left: 0;
|
|
42
|
+
padding: 0.25rem 0.375rem;
|
|
43
|
+
border: 2px solid var(--dp-primary);
|
|
44
|
+
border-radius: var(--dp-radius-sm);
|
|
45
|
+
font-size: inherit;
|
|
46
|
+
font-family: inherit;
|
|
47
|
+
line-height: inherit;
|
|
48
|
+
background: var(--dp-bg);
|
|
49
|
+
color: var(--dp-text);
|
|
50
|
+
outline: none;
|
|
51
|
+
box-sizing: border-box;
|
|
52
|
+
z-index: 10;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
.dp-inline-input:focus {
|
|
56
|
+
box-shadow: 0 0 0 3px rgba(79, 70, 229, 0.15);
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
.dp-cell--saving {
|
|
60
|
+
opacity: 0.6;
|
|
61
|
+
pointer-events: none;
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
.dp-cell--saving::after {
|
|
65
|
+
content: "\f110";
|
|
66
|
+
font-family: "Font Awesome 6 Free";
|
|
67
|
+
font-weight: 900;
|
|
68
|
+
font-size: 0.7rem;
|
|
69
|
+
position: absolute;
|
|
70
|
+
top: 0.25rem;
|
|
71
|
+
right: 0.35rem;
|
|
72
|
+
color: var(--dp-primary);
|
|
73
|
+
animation: dp-spin 0.8s linear infinite;
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
.dp-cell--error {
|
|
77
|
+
outline: 2px solid var(--dp-danger) !important;
|
|
78
|
+
background: var(--dp-danger-light);
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
.dp-cell--error::after {
|
|
82
|
+
content: "\f00d";
|
|
83
|
+
font-family: "Font Awesome 6 Free";
|
|
84
|
+
font-weight: 900;
|
|
85
|
+
font-size: 0.7rem;
|
|
86
|
+
position: absolute;
|
|
87
|
+
top: 0.25rem;
|
|
88
|
+
right: 0.35rem;
|
|
89
|
+
color: var(--dp-danger);
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
.dp-cell--success {
|
|
93
|
+
animation: dp-flash-success 0.8s ease;
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
.dp-cell--success::after {
|
|
97
|
+
content: "\f00c";
|
|
98
|
+
font-family: "Font Awesome 6 Free";
|
|
99
|
+
font-weight: 900;
|
|
100
|
+
font-size: 0.7rem;
|
|
101
|
+
position: absolute;
|
|
102
|
+
top: 0.25rem;
|
|
103
|
+
right: 0.35rem;
|
|
104
|
+
color: var(--dp-success);
|
|
105
|
+
animation: dp-fade-out 0.8s ease forwards;
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
@keyframes dp-flash-success {
|
|
109
|
+
0% { background: var(--dp-success-light); }
|
|
110
|
+
100% { background: transparent; }
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
@keyframes dp-fade-out {
|
|
114
|
+
0% { opacity: 1; }
|
|
115
|
+
70% { opacity: 1; }
|
|
116
|
+
100% { opacity: 0; }
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
@keyframes dp-spin {
|
|
120
|
+
from { transform: rotate(0deg); }
|
|
121
|
+
to { transform: rotate(360deg); }
|
|
122
|
+
}
|
|
@@ -10,7 +10,9 @@ module DataPorter
|
|
|
10
10
|
layout "data_porter/application"
|
|
11
11
|
|
|
12
12
|
before_action :set_import, only: %i[show parse confirm cancel dry_run update_mapping
|
|
13
|
-
status export_rejects destroy
|
|
13
|
+
update_record status export_rejects destroy
|
|
14
|
+
back_to_mapping resume]
|
|
15
|
+
before_action :ensure_previewing, only: :update_record
|
|
14
16
|
before_action :load_targets, only: %i[index new create]
|
|
15
17
|
|
|
16
18
|
def index
|
|
@@ -69,12 +71,28 @@ module DataPorter
|
|
|
69
71
|
redirect_to import_path(@import)
|
|
70
72
|
end
|
|
71
73
|
|
|
74
|
+
def resume
|
|
75
|
+
@import.update!(status: :pending)
|
|
76
|
+
DataPorter::ImportJob.perform_later(@import.id)
|
|
77
|
+
redirect_to import_path(@import)
|
|
78
|
+
end
|
|
79
|
+
|
|
72
80
|
def dry_run
|
|
73
81
|
@import.update!(status: :pending)
|
|
74
82
|
DataPorter::DryRunJob.perform_later(@import.id)
|
|
75
83
|
redirect_to import_path(@import)
|
|
76
84
|
end
|
|
77
85
|
|
|
86
|
+
def update_record
|
|
87
|
+
updater = RecordUpdater.new(@import)
|
|
88
|
+
result = updater.call(
|
|
89
|
+
line_number: params[:line_number].to_i,
|
|
90
|
+
column: params[:column],
|
|
91
|
+
value: params[:value]
|
|
92
|
+
)
|
|
93
|
+
render json: result
|
|
94
|
+
end
|
|
95
|
+
|
|
78
96
|
def status
|
|
79
97
|
progress = @import.config["progress"] || {}
|
|
80
98
|
render json: { status: @import.status, progress: progress }
|
|
@@ -98,6 +116,12 @@ module DataPorter
|
|
|
98
116
|
@import = scoped_imports.find(params[:id])
|
|
99
117
|
end
|
|
100
118
|
|
|
119
|
+
def ensure_previewing
|
|
120
|
+
return if @import.previewing?
|
|
121
|
+
|
|
122
|
+
render json: { error: "Import is not in previewing state" }, status: :unprocessable_entity
|
|
123
|
+
end
|
|
124
|
+
|
|
101
125
|
def scoped_imports
|
|
102
126
|
owner = resolve_owner
|
|
103
127
|
return DataPorter::DataImport.all unless owner
|
|
@@ -53,12 +53,16 @@ module DataPorter
|
|
|
53
53
|
records.group_by(&:status).transform_values(&:count)
|
|
54
54
|
end
|
|
55
55
|
|
|
56
|
+
def resumable?
|
|
57
|
+
failed? && config&.dig("checkpoint", "processed").to_i.positive?
|
|
58
|
+
end
|
|
59
|
+
|
|
56
60
|
def reset_to_mapping!
|
|
57
61
|
update!(
|
|
58
62
|
status: :mapping,
|
|
59
63
|
records: [],
|
|
60
64
|
report: StoreModels::Report.new,
|
|
61
|
-
config: (config || {}).except("progress")
|
|
65
|
+
config: (config || {}).except("progress", "checkpoint")
|
|
62
66
|
)
|
|
63
67
|
end
|
|
64
68
|
|
|
@@ -46,10 +46,12 @@
|
|
|
46
46
|
|
|
47
47
|
<% if @import.previewing? %>
|
|
48
48
|
<div id="records">
|
|
49
|
-
<%= raw DataPorter::Components::Preview::SummaryCards.new(report: @import.report).call %>
|
|
49
|
+
<%= raw DataPorter::Components::Preview::SummaryCards.new(report: @import.report, editable: true).call %>
|
|
50
50
|
<%= raw DataPorter::Components::Preview::Table.new(
|
|
51
51
|
columns: @target._columns,
|
|
52
|
-
records: @records
|
|
52
|
+
records: @records,
|
|
53
|
+
editable: true,
|
|
54
|
+
update_url: update_record_import_path(@import)
|
|
53
55
|
).call %>
|
|
54
56
|
</div>
|
|
55
57
|
<%= raw DataPorter::Components::Shared::Pagination.new(
|
|
@@ -103,8 +105,12 @@
|
|
|
103
105
|
<% if @import.failed? %>
|
|
104
106
|
<%= raw DataPorter::Components::Shared::FailureAlert.new(report: @import.report).call %>
|
|
105
107
|
<div class="dp-actions">
|
|
108
|
+
<% if @import.resumable? %>
|
|
109
|
+
<%= button_to t("data_porter.imports.resume"), resume_import_path(@import),
|
|
110
|
+
method: :post, class: "dp-btn dp-btn--primary" %>
|
|
111
|
+
<% end %>
|
|
106
112
|
<%= button_to t("data_porter.imports.retry"), parse_import_path(@import),
|
|
107
|
-
method: :post, class: "dp-btn dp-btn--
|
|
113
|
+
method: :post, class: "dp-btn dp-btn--secondary" %>
|
|
108
114
|
<%= button_to t("data_porter.imports.delete"), import_path(@import),
|
|
109
115
|
method: :delete, class: "dp-btn dp-btn--danger",
|
|
110
116
|
data: { turbo_confirm: t("data_porter.imports.delete_confirm") } %>
|
|
@@ -16,7 +16,8 @@
|
|
|
16
16
|
"data_porter/template_form_controller": "<%= asset_path('data_porter/template_form_controller.js') %>",
|
|
17
17
|
"data_porter/progress_controller": "<%= asset_path('data_porter/progress_controller.js') %>",
|
|
18
18
|
"data_porter/import_form_controller": "<%= asset_path('data_porter/import_form_controller.js') %>",
|
|
19
|
-
"data_porter/theme_controller": "<%= asset_path('data_porter/theme_controller.js') %>"
|
|
19
|
+
"data_porter/theme_controller": "<%= asset_path('data_porter/theme_controller.js') %>",
|
|
20
|
+
"data_porter/inline_edit_controller": "<%= asset_path('data_porter/inline_edit_controller.js') %>"
|
|
20
21
|
}
|
|
21
22
|
}
|
|
22
23
|
</script>
|
|
@@ -28,6 +29,7 @@
|
|
|
28
29
|
import ProgressController from "data_porter/progress_controller"
|
|
29
30
|
import ImportFormController from "data_porter/import_form_controller"
|
|
30
31
|
import ThemeController from "data_porter/theme_controller"
|
|
32
|
+
import InlineEditController from "data_porter/inline_edit_controller"
|
|
31
33
|
|
|
32
34
|
const application = Application.start()
|
|
33
35
|
application.register("data-porter--mapping", MappingController)
|
|
@@ -35,6 +37,7 @@
|
|
|
35
37
|
application.register("data-porter--progress", ProgressController)
|
|
36
38
|
application.register("data-porter--import-form", ImportFormController)
|
|
37
39
|
application.register("data-porter--theme", ThemeController)
|
|
40
|
+
application.register("data-porter--inline-edit", InlineEditController)
|
|
38
41
|
</script>
|
|
39
42
|
</head>
|
|
40
43
|
<body data-controller="data-porter--theme">
|
data/config/locales/en.yml
CHANGED
|
@@ -10,6 +10,7 @@ en:
|
|
|
10
10
|
delete: "Delete"
|
|
11
11
|
delete_confirm: "Delete this import?"
|
|
12
12
|
retry: "Retry"
|
|
13
|
+
resume: "Resume"
|
|
13
14
|
start_import: "Start Import"
|
|
14
15
|
confirm_import: "Confirm Import"
|
|
15
16
|
dry_run: "Dry Run"
|
|
@@ -17,6 +18,8 @@ en:
|
|
|
17
18
|
back_to_mapping: "Back to Mapping"
|
|
18
19
|
download_rejects: "Download rejects CSV"
|
|
19
20
|
processing: "Processing..."
|
|
21
|
+
edit_cell: "Click to edit"
|
|
22
|
+
saving: "Saving..."
|
|
20
23
|
no_imports: "No imports yet"
|
|
21
24
|
create_first: "Create your first import"
|
|
22
25
|
details:
|
data/config/locales/fr.yml
CHANGED
|
@@ -10,6 +10,7 @@ fr:
|
|
|
10
10
|
delete: "Supprimer"
|
|
11
11
|
delete_confirm: "Supprimer cet import ?"
|
|
12
12
|
retry: "Réessayer"
|
|
13
|
+
resume: "Reprendre"
|
|
13
14
|
start_import: "Lancer l'import"
|
|
14
15
|
confirm_import: "Confirmer l'import"
|
|
15
16
|
dry_run: "Essai à blanc"
|
|
@@ -17,6 +18,8 @@ fr:
|
|
|
17
18
|
back_to_mapping: "Retour au mapping"
|
|
18
19
|
download_rejects: "Télécharger les rejets CSV"
|
|
19
20
|
processing: "Traitement..."
|
|
21
|
+
edit_cell: "Cliquer pour modifier"
|
|
22
|
+
saving: "Enregistrement..."
|
|
20
23
|
no_imports: "Aucun import"
|
|
21
24
|
create_first: "Créer votre premier import"
|
|
22
25
|
details:
|
data/config/routes.rb
CHANGED
|
@@ -4,29 +4,42 @@ module DataPorter
|
|
|
4
4
|
module Components
|
|
5
5
|
module Preview
|
|
6
6
|
class SummaryCards < Base
|
|
7
|
-
|
|
7
|
+
CARDS = [
|
|
8
|
+
{ css: "dp-card--complete", key: :complete_count, i18n: "ready" },
|
|
9
|
+
{ css: "dp-card--partial", key: :partial_count, i18n: "incomplete" },
|
|
10
|
+
{ css: "dp-card--missing", key: :missing_count, i18n: "missing" },
|
|
11
|
+
{ css: "dp-card--duplicate", key: :duplicate_count, i18n: "duplicates" }
|
|
12
|
+
].freeze
|
|
13
|
+
|
|
14
|
+
def initialize(report:, editable: false)
|
|
8
15
|
super()
|
|
9
16
|
@report = report
|
|
17
|
+
@editable = editable
|
|
10
18
|
end
|
|
11
19
|
|
|
12
20
|
def view_template
|
|
13
21
|
div(class: "dp-summary-cards") do
|
|
14
|
-
|
|
15
|
-
card("dp-card--partial", @report.partial_count, I18n.t("data_porter.components.summary_cards.incomplete"))
|
|
16
|
-
card("dp-card--missing", @report.missing_count, I18n.t("data_porter.components.summary_cards.missing"))
|
|
17
|
-
card("dp-card--duplicate", @report.duplicate_count,
|
|
18
|
-
I18n.t("data_porter.components.summary_cards.duplicates"))
|
|
22
|
+
CARDS.each { |card_def| card(card_def) }
|
|
19
23
|
end
|
|
20
24
|
end
|
|
21
25
|
|
|
22
26
|
private
|
|
23
27
|
|
|
24
|
-
def card(
|
|
25
|
-
|
|
26
|
-
|
|
28
|
+
def card(card_def)
|
|
29
|
+
count = @report.send(card_def[:key])
|
|
30
|
+
label = I18n.t("data_porter.components.summary_cards.#{card_def[:i18n]}")
|
|
31
|
+
|
|
32
|
+
div(class: "dp-card #{card_def[:css]}") do
|
|
33
|
+
strong(**count_attrs(card_def[:key])) { count.to_s }
|
|
27
34
|
plain " #{label}"
|
|
28
35
|
end
|
|
29
36
|
end
|
|
37
|
+
|
|
38
|
+
def count_attrs(key)
|
|
39
|
+
return {} unless @editable
|
|
40
|
+
|
|
41
|
+
{ data: { "data-porter--inline-edit-target": "summary", count_key: key } }
|
|
42
|
+
end
|
|
30
43
|
end
|
|
31
44
|
end
|
|
32
45
|
end
|
|
@@ -4,14 +4,16 @@ module DataPorter
|
|
|
4
4
|
module Components
|
|
5
5
|
module Preview
|
|
6
6
|
class Table < Base
|
|
7
|
-
def initialize(columns:, records:)
|
|
7
|
+
def initialize(columns:, records:, editable: false, update_url: nil)
|
|
8
8
|
super()
|
|
9
9
|
@columns = columns
|
|
10
10
|
@records = records
|
|
11
|
+
@editable = editable
|
|
12
|
+
@update_url = update_url
|
|
11
13
|
end
|
|
12
14
|
|
|
13
15
|
def view_template
|
|
14
|
-
div(
|
|
16
|
+
div(**wrapper_attrs) do
|
|
15
17
|
table(class: "dp-table") do
|
|
16
18
|
render_header
|
|
17
19
|
render_body
|
|
@@ -21,6 +23,17 @@ module DataPorter
|
|
|
21
23
|
|
|
22
24
|
private
|
|
23
25
|
|
|
26
|
+
def wrapper_attrs
|
|
27
|
+
attrs = { class: "dp-preview-table" }
|
|
28
|
+
return attrs unless @editable
|
|
29
|
+
|
|
30
|
+
attrs[:data] = {
|
|
31
|
+
controller: "data-porter--inline-edit",
|
|
32
|
+
"data-porter--inline-edit-update-url-value": @update_url
|
|
33
|
+
}
|
|
34
|
+
attrs
|
|
35
|
+
end
|
|
36
|
+
|
|
24
37
|
def render_header
|
|
25
38
|
thead do
|
|
26
39
|
tr do
|
|
@@ -39,14 +52,61 @@ module DataPorter
|
|
|
39
52
|
end
|
|
40
53
|
|
|
41
54
|
def render_row(record)
|
|
42
|
-
tr(
|
|
55
|
+
tr(**row_attrs(record)) do
|
|
43
56
|
td { record.line_number.to_s }
|
|
44
57
|
td { record.status }
|
|
45
|
-
@columns.each { |col|
|
|
46
|
-
|
|
58
|
+
@columns.each { |col| render_cell(record, col) }
|
|
59
|
+
render_errors_cell(record)
|
|
47
60
|
end
|
|
48
61
|
end
|
|
49
62
|
|
|
63
|
+
def render_cell(record, col)
|
|
64
|
+
value = cell_value(record, col)
|
|
65
|
+
td(**cell_attrs(record, col, value)) { value }
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def render_errors_cell(record)
|
|
69
|
+
td(**errors_cell_attrs(record)) { error_messages(record) }
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
def cell_value(record, col)
|
|
73
|
+
(record.data[col.name.to_s] || record.data[col.name]).to_s
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
def row_attrs(record)
|
|
77
|
+
attrs = { class: "dp-row--#{record.status}" }
|
|
78
|
+
return attrs unless @editable
|
|
79
|
+
|
|
80
|
+
attrs[:data] = { line_number: record.line_number }
|
|
81
|
+
attrs
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
def cell_attrs(record, col, value)
|
|
85
|
+
return {} unless @editable
|
|
86
|
+
|
|
87
|
+
{
|
|
88
|
+
class: "dp-cell--editable",
|
|
89
|
+
data: {
|
|
90
|
+
action: "click->data-porter--inline-edit#edit",
|
|
91
|
+
"data-porter--inline-edit-target": "cell",
|
|
92
|
+
value: value,
|
|
93
|
+
line_number: record.line_number,
|
|
94
|
+
column: col.name.to_s
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
def errors_cell_attrs(record)
|
|
100
|
+
attrs = { class: "dp-errors" }
|
|
101
|
+
return attrs unless @editable
|
|
102
|
+
|
|
103
|
+
attrs[:data] = {
|
|
104
|
+
"data-porter--inline-edit-target": "errors",
|
|
105
|
+
line_number: record.line_number
|
|
106
|
+
}
|
|
107
|
+
attrs
|
|
108
|
+
end
|
|
109
|
+
|
|
50
110
|
def error_messages(record)
|
|
51
111
|
record.errors_list.map(&:message).join(", ")
|
|
52
112
|
end
|
data/lib/data_porter/engine.rb
CHANGED
|
@@ -7,46 +7,54 @@ module DataPorter
|
|
|
7
7
|
|
|
8
8
|
def import_bulk
|
|
9
9
|
importable = @data_import.importable_records
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
10
|
+
checkpoint = load_checkpoint
|
|
11
|
+
@bulk_state = build_bulk_state(importable, checkpoint)
|
|
12
|
+
|
|
13
|
+
process_batches(importable.drop(checkpoint[:processed]))
|
|
14
|
+
finalize_import(@bulk_state[:results])
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def build_bulk_state(importable, checkpoint)
|
|
18
|
+
{
|
|
19
|
+
context: build_context,
|
|
20
|
+
bulk_config: @target.class._bulk_config,
|
|
21
|
+
results: seed_results(checkpoint),
|
|
22
|
+
total: importable.size,
|
|
23
|
+
processed: checkpoint[:processed]
|
|
24
|
+
}
|
|
25
|
+
end
|
|
21
26
|
|
|
22
|
-
|
|
27
|
+
def process_batches(records)
|
|
28
|
+
records.each_slice(@bulk_state[:bulk_config][:batch_size]) do |batch|
|
|
29
|
+
persist_batch_with_fallback(batch)
|
|
30
|
+
@bulk_state[:processed] += batch.size
|
|
31
|
+
broadcast_progress(@bulk_state[:processed], @bulk_state[:total], results: @bulk_state[:results])
|
|
32
|
+
end
|
|
23
33
|
end
|
|
24
34
|
|
|
25
|
-
def persist_batch_with_fallback(batch
|
|
26
|
-
@target.persist_batch(batch, context: context)
|
|
27
|
-
results[:created] += batch.size
|
|
35
|
+
def persist_batch_with_fallback(batch)
|
|
36
|
+
@target.persist_batch(batch, context: @bulk_state[:context])
|
|
37
|
+
@bulk_state[:results][:created] += batch.size
|
|
28
38
|
rescue StandardError => e
|
|
29
|
-
handle_batch_failure(batch,
|
|
39
|
+
handle_batch_failure(batch, e)
|
|
30
40
|
end
|
|
31
41
|
|
|
32
|
-
def handle_batch_failure(batch,
|
|
33
|
-
if
|
|
34
|
-
fail_batch(batch,
|
|
42
|
+
def handle_batch_failure(batch, error)
|
|
43
|
+
if @bulk_state[:bulk_config][:on_conflict] == :fail_batch
|
|
44
|
+
fail_batch(batch, error)
|
|
35
45
|
else
|
|
36
|
-
retry_per_record(batch
|
|
46
|
+
retry_per_record(batch)
|
|
37
47
|
end
|
|
38
48
|
end
|
|
39
49
|
|
|
40
|
-
def fail_batch(batch,
|
|
41
|
-
batch.each
|
|
42
|
-
|
|
43
|
-
end
|
|
44
|
-
results[:errored] += batch.size
|
|
50
|
+
def fail_batch(batch, error)
|
|
51
|
+
batch.each { |record| record.add_error(error.message) }
|
|
52
|
+
@bulk_state[:results][:errored] += batch.size
|
|
45
53
|
end
|
|
46
54
|
|
|
47
|
-
def retry_per_record(batch
|
|
55
|
+
def retry_per_record(batch)
|
|
48
56
|
batch.each do |record|
|
|
49
|
-
persist_record(record, context, results)
|
|
57
|
+
persist_record(record, @bulk_state[:context], @bulk_state[:results])
|
|
50
58
|
end
|
|
51
59
|
end
|
|
52
60
|
end
|
|
@@ -18,12 +18,14 @@ module DataPorter
|
|
|
18
18
|
def import_per_record
|
|
19
19
|
importable = @data_import.importable_records
|
|
20
20
|
context = build_context
|
|
21
|
-
|
|
21
|
+
checkpoint = load_checkpoint
|
|
22
|
+
results = seed_results(checkpoint)
|
|
23
|
+
remaining = importable.drop(checkpoint[:processed])
|
|
22
24
|
total = importable.size
|
|
23
25
|
|
|
24
|
-
|
|
26
|
+
remaining.each_with_index do |record, index|
|
|
25
27
|
persist_record(record, context, results)
|
|
26
|
-
broadcast_progress(index + 1, total)
|
|
28
|
+
broadcast_progress(checkpoint[:processed] + index + 1, total, results: results)
|
|
27
29
|
end
|
|
28
30
|
|
|
29
31
|
finalize_import(results)
|
|
@@ -45,6 +47,7 @@ module DataPorter
|
|
|
45
47
|
end
|
|
46
48
|
|
|
47
49
|
def finalize_import(results)
|
|
50
|
+
clear_checkpoint
|
|
48
51
|
@data_import.update!(status: :completed)
|
|
49
52
|
@broadcaster.success
|
|
50
53
|
WebhookNotifier.notify(@data_import, "import.completed")
|
|
@@ -66,6 +69,25 @@ module DataPorter
|
|
|
66
69
|
report.errored_count = results[:errored]
|
|
67
70
|
@data_import.update!(report: report)
|
|
68
71
|
end
|
|
72
|
+
|
|
73
|
+
def load_checkpoint
|
|
74
|
+
cp = @data_import.config&.dig("checkpoint") || {}
|
|
75
|
+
{
|
|
76
|
+
processed: cp["processed"].to_i,
|
|
77
|
+
created: cp["created"].to_i,
|
|
78
|
+
errored: cp["errored"].to_i
|
|
79
|
+
}
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
def seed_results(checkpoint)
|
|
83
|
+
{ created: checkpoint[:created], errored: checkpoint[:errored] }
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def clear_checkpoint
|
|
87
|
+
config = @data_import.config || {}
|
|
88
|
+
config.delete("checkpoint")
|
|
89
|
+
@data_import.update_column(:config, config)
|
|
90
|
+
end
|
|
69
91
|
end
|
|
70
92
|
end
|
|
71
93
|
end
|
|
@@ -32,6 +32,7 @@ module DataPorter
|
|
|
32
32
|
def parse!
|
|
33
33
|
@data_import.parsing!
|
|
34
34
|
records = build_records
|
|
35
|
+
clear_stale_import_data
|
|
35
36
|
@data_import.update!(records: records, status: :previewing)
|
|
36
37
|
build_report
|
|
37
38
|
WebhookNotifier.notify(@data_import, "import.parsed")
|
|
@@ -92,18 +93,36 @@ module DataPorter
|
|
|
92
93
|
DataPorter.configuration.context_builder&.call(@data_import)
|
|
93
94
|
end
|
|
94
95
|
|
|
95
|
-
def broadcast_progress(current, total)
|
|
96
|
-
percentage = ((current.to_f / total) * 100).round
|
|
96
|
+
def broadcast_progress(current, total, results: nil)
|
|
97
97
|
config = @data_import.config || {}
|
|
98
|
-
config["progress"] = { "current" => current, "total" => total, "percentage" =>
|
|
98
|
+
config["progress"] = { "current" => current, "total" => total, "percentage" => pct(current, total) }
|
|
99
|
+
save_checkpoint(config, current, results) if results
|
|
99
100
|
@data_import.update_column(:config, config)
|
|
100
101
|
@broadcaster.progress(current, total)
|
|
101
102
|
end
|
|
102
103
|
|
|
104
|
+
def pct(current, total)
|
|
105
|
+
((current.to_f / total) * 100).round
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def save_checkpoint(config, processed, results)
|
|
109
|
+
config["checkpoint"] = {
|
|
110
|
+
"processed" => processed,
|
|
111
|
+
"created" => results[:created],
|
|
112
|
+
"errored" => results[:errored]
|
|
113
|
+
}
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
def clear_stale_import_data
|
|
117
|
+
config = @data_import.config || {}
|
|
118
|
+
config.delete("checkpoint")
|
|
119
|
+
config.delete("progress")
|
|
120
|
+
@data_import.config = config
|
|
121
|
+
end
|
|
122
|
+
|
|
103
123
|
def handle_failure(error)
|
|
104
|
-
report = StoreModels::Report.new
|
|
105
|
-
|
|
106
|
-
)
|
|
124
|
+
report = @data_import.report || StoreModels::Report.new
|
|
125
|
+
report.error_reports = [StoreModels::Error.new(message: error.message)]
|
|
107
126
|
@data_import.update!(status: :failed, report: report)
|
|
108
127
|
@broadcaster.failure(error.message)
|
|
109
128
|
WebhookNotifier.notify(@data_import, "import.failed")
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module DataPorter
|
|
4
|
+
class RecordRevalidator
|
|
5
|
+
def initialize(target)
|
|
6
|
+
@target = target
|
|
7
|
+
@columns = target.class._columns || []
|
|
8
|
+
@validator = RecordValidator.new(@columns)
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def call(record)
|
|
12
|
+
normalize_keys(record)
|
|
13
|
+
ColumnTransformer.apply_all(record, @columns)
|
|
14
|
+
@target.transform(record)
|
|
15
|
+
record.errors_list = []
|
|
16
|
+
@target.validate(record)
|
|
17
|
+
@validator.validate(record)
|
|
18
|
+
record.determine_status!
|
|
19
|
+
record
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
private
|
|
23
|
+
|
|
24
|
+
def normalize_keys(record)
|
|
25
|
+
@columns.each do |col|
|
|
26
|
+
string_key = col.name.to_s
|
|
27
|
+
next unless record.data.key?(string_key) && !record.data.key?(col.name)
|
|
28
|
+
|
|
29
|
+
record.data[col.name] = record.data.delete(string_key)
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module DataPorter
|
|
4
|
+
class RecordUpdater
|
|
5
|
+
def initialize(data_import)
|
|
6
|
+
@data_import = data_import
|
|
7
|
+
@target = data_import.target_class.new
|
|
8
|
+
@revalidator = RecordRevalidator.new(@target)
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def call(line_number:, column:, value:)
|
|
12
|
+
record = find_record(line_number)
|
|
13
|
+
old_status = record.status
|
|
14
|
+
update_cell(record, column, value)
|
|
15
|
+
@revalidator.call(record)
|
|
16
|
+
update_report(old_status, record.status)
|
|
17
|
+
persist!
|
|
18
|
+
build_response(record, column)
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
private
|
|
22
|
+
|
|
23
|
+
def find_record(line_number)
|
|
24
|
+
record = @data_import.records.find { |r| r.line_number == line_number }
|
|
25
|
+
raise Error, "Record #{line_number} not found" unless record
|
|
26
|
+
|
|
27
|
+
record
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def update_cell(record, column, value)
|
|
31
|
+
key = resolve_key(record.data, column)
|
|
32
|
+
record.data[key] = value
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def resolve_key(data, column)
|
|
36
|
+
return column.to_sym if data.key?(column.to_sym)
|
|
37
|
+
return column.to_s if data.key?(column.to_s)
|
|
38
|
+
|
|
39
|
+
column.to_s
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def update_report(old_status, new_status)
|
|
43
|
+
return if old_status == new_status
|
|
44
|
+
|
|
45
|
+
report = @data_import.report
|
|
46
|
+
decrement_count(report, old_status)
|
|
47
|
+
increment_count(report, new_status)
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def decrement_count(report, status)
|
|
51
|
+
attr = count_attribute(status)
|
|
52
|
+
report.send(:"#{attr}=", [report.send(attr) - 1, 0].max)
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def increment_count(report, status)
|
|
56
|
+
attr = count_attribute(status)
|
|
57
|
+
report.send(:"#{attr}=", report.send(attr) + 1)
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
def count_attribute(status)
|
|
61
|
+
{
|
|
62
|
+
"complete" => :complete_count,
|
|
63
|
+
"partial" => :partial_count,
|
|
64
|
+
"missing" => :missing_count
|
|
65
|
+
}.fetch(status, :partial_count)
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def persist!
|
|
69
|
+
@data_import.update!(
|
|
70
|
+
records: @data_import.records,
|
|
71
|
+
report: @data_import.report
|
|
72
|
+
)
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def build_response(record, column)
|
|
76
|
+
key = resolve_key(record.data, column)
|
|
77
|
+
{
|
|
78
|
+
status: record.status,
|
|
79
|
+
value: record.data[key].to_s,
|
|
80
|
+
errors: record.errors_list.map(&:message),
|
|
81
|
+
report: report_hash
|
|
82
|
+
}
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
def report_hash
|
|
86
|
+
r = @data_import.report
|
|
87
|
+
{
|
|
88
|
+
complete_count: r.complete_count,
|
|
89
|
+
partial_count: r.partial_count,
|
|
90
|
+
missing_count: r.missing_count,
|
|
91
|
+
duplicate_count: r.duplicate_count
|
|
92
|
+
}
|
|
93
|
+
end
|
|
94
|
+
end
|
|
95
|
+
end
|
data/lib/data_porter/version.rb
CHANGED
data/lib/data_porter.rb
CHANGED
|
@@ -17,6 +17,8 @@ require_relative "data_porter/target"
|
|
|
17
17
|
require_relative "data_porter/registry"
|
|
18
18
|
require_relative "data_porter/sources"
|
|
19
19
|
require_relative "data_porter/record_validator"
|
|
20
|
+
require_relative "data_porter/record_revalidator"
|
|
21
|
+
require_relative "data_porter/record_updater"
|
|
20
22
|
require_relative "data_porter/broadcaster"
|
|
21
23
|
require_relative "data_porter/webhook_notifier"
|
|
22
24
|
require_relative "data_porter/orchestrator"
|
data/mkdocs.yml
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: data_porter
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 2.
|
|
4
|
+
version: 2.8.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Seryl Lounis
|
|
@@ -102,6 +102,7 @@ files:
|
|
|
102
102
|
- ROADMAP.md
|
|
103
103
|
- Rakefile
|
|
104
104
|
- app/assets/javascripts/data_porter/import_form_controller.js
|
|
105
|
+
- app/assets/javascripts/data_porter/inline_edit_controller.js
|
|
105
106
|
- app/assets/javascripts/data_porter/mapping_controller.js
|
|
106
107
|
- app/assets/javascripts/data_porter/progress_controller.js
|
|
107
108
|
- app/assets/javascripts/data_porter/stimulus.min.js
|
|
@@ -114,6 +115,7 @@ files:
|
|
|
114
115
|
- app/assets/stylesheets/data_porter/base.css
|
|
115
116
|
- app/assets/stylesheets/data_porter/cards.css
|
|
116
117
|
- app/assets/stylesheets/data_porter/dark.css
|
|
118
|
+
- app/assets/stylesheets/data_porter/inline_edit.css
|
|
117
119
|
- app/assets/stylesheets/data_porter/layout.css
|
|
118
120
|
- app/assets/stylesheets/data_porter/mapping.css
|
|
119
121
|
- app/assets/stylesheets/data_porter/modal.css
|
|
@@ -172,6 +174,8 @@ files:
|
|
|
172
174
|
- lib/data_porter/orchestrator/dry_runner.rb
|
|
173
175
|
- lib/data_porter/orchestrator/importer.rb
|
|
174
176
|
- lib/data_porter/orchestrator/record_builder.rb
|
|
177
|
+
- lib/data_porter/record_revalidator.rb
|
|
178
|
+
- lib/data_porter/record_updater.rb
|
|
175
179
|
- lib/data_porter/record_validator.rb
|
|
176
180
|
- lib/data_porter/registry.rb
|
|
177
181
|
- lib/data_porter/rejects_csv_builder.rb
|