ixport 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +92 -6
- data/app/controllers/ixport/application_controller.rb +6 -2
- data/app/controllers/ixport/export_templates_controller.rb +52 -0
- data/app/controllers/ixport/exports_controller.rb +18 -0
- data/app/controllers/ixport/imports_controller.rb +4 -3
- data/app/controllers/ixport/previews_controller.rb +2 -2
- data/app/controllers/ixport/runs_controller.rb +13 -1
- data/app/helpers/ixport/imports_helper.rb +13 -0
- data/app/models/ixport/csv_writer.rb +19 -0
- data/app/models/ixport/export_template.rb +19 -0
- data/app/models/ixport/import.rb +71 -19
- data/app/models/ixport/null_reporter.rb +11 -0
- data/app/models/ixport/row_walk.rb +15 -6
- data/app/models/ixport/wallflower_runner.rb +10 -0
- data/app/models/ixport/wallflower_task_runner.rb +14 -0
- data/app/views/ixport/export_templates/edit.html.erb +14 -0
- data/app/views/ixport/export_templates/new.html.erb +11 -0
- data/app/views/ixport/exports/index.html.erb +23 -0
- data/app/views/ixport/imports/index.html.erb +17 -0
- data/app/views/ixport/imports/show.html.erb +30 -15
- data/config/routes.rb +4 -0
- data/lib/generators/ixport/install/install_generator.rb +3 -0
- data/lib/generators/ixport/install/templates/add_error_message_to_ixport_imports.rb.erb +5 -0
- data/lib/generators/ixport/install/templates/add_wallflower_task_to_ixport_imports.rb.erb +6 -0
- data/lib/generators/ixport/install/templates/create_ixport_export_templates.rb.erb +12 -0
- data/lib/generators/ixport/update/update_generator.rb +28 -0
- data/lib/ixport/background_runner.rb +23 -0
- data/lib/ixport/configuration.rb +3 -1
- data/lib/ixport/engine.rb +4 -1
- data/lib/ixport/exporter.rb +27 -0
- data/lib/ixport/importer.rb +7 -3
- data/lib/ixport/registrations.rb +29 -3
- data/lib/ixport/version.rb +1 -1
- data/lib/ixport.rb +2 -0
- data/the_local/agents/ixport-develop.md +235 -37
- data/the_local/agents/ixport-info.md +61 -26
- data/the_local/agents/ixport-install.md +66 -38
- data/the_local/interface.yml +37 -1
- metadata +18 -1
|
@@ -1,41 +1,62 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: ixport-develop
|
|
3
|
-
description: Use PROACTIVELY for setting up CSV imports in a Rails app that has ixport — registering a mapped import, writing its importer class, declaring the fields a person maps a CSV's columns to, choosing the field rows are matched on, scoping the records an import may add or update to the signed-in person or account, linking to an import's upload page, explaining the upload, column-matching, preview and result pages and their messages, explaining what running an import saves and refuses, and fixing a boot that fails with Ixport::MissingImporter — MUST BE USED instead of hand-writing a CSV upload controller, a column-mapping form, a row-matching loop, an import preview
|
|
3
|
+
description: Use PROACTIVELY for setting up CSV imports and exports in a Rails app that has ixport — registering a mapped import, writing its importer class, declaring the fields a person maps a CSV's columns to, choosing the field or fields rows are matched on, scoping the records an import may add or update to the signed-in person or account, limiting which people may use an import or export with an access rule such as the app's own Pundit policy, linking to an import's upload page, explaining the upload, column-matching, preview and result pages and their messages, explaining the list of a person's past imports on the imports page, explaining what running an import saves and refuses, explaining what a person sees when an import fails partway and which rows it keeps, running imports in a background job through the app's own runner class, explaining what a person sees while an import runs in the background, registering a fixed export or a chosen export, writing its exporter class, declaring its columns including columns taken from a related record, computing a column's value, scoping the records an export may hold, letting an account save named templates of picked columns for an export, explaining the new, edit and delete template pages and what a template's download holds, linking to the exports page, one export's CSV download or one template's download, making the app's own import job fail with Ixport::ImportFailed, and fixing a boot that fails with Ixport::MissingImporter or Ixport::MissingExporter — MUST BE USED instead of hand-writing a CSV upload controller, a column-mapping form, a row-matching loop, an import preview, a loop that saves imported rows, a background job that saves imported rows, a rescue that records why an import stopped, a permission check in front of an import or export page, a CSV download action, a loop that writes records to a CSV, or a model and form for saved column picks.
|
|
4
4
|
tools: Bash, Read, Write, Edit, Grep
|
|
5
|
-
scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports the host app registers, each backed by an importer class the host writes
|
|
5
|
+
scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports and the exports the host app registers, each import backed by an importer class and each export by an exporter class the host writes, where an export is either fixed or chosen, and a chosen export lets the account save named templates of picked columns
|
|
6
6
|
---
|
|
7
7
|
|
|
8
|
-
This local adds mapped imports to a Rails app that already has ixport installed by following the steps below exactly. It writes one importer class and one registration line per import, and it invents no method, option, route or file that is not listed here.
|
|
8
|
+
This local adds mapped imports, fixed exports and chosen exports to a Rails app that already has ixport installed by following the steps below exactly. It writes one importer or exporter class and one registration line per import or export, and it invents no method, option, route or file that is not listed here.
|
|
9
9
|
|
|
10
10
|
## What ixport is
|
|
11
11
|
|
|
12
|
-
ixport is a Rails engine that lets a signed-in person import a host app's records from CSV files. The app registers each kind of CSV a person can import, called a mapped import, and writes an importer class that names the fields the CSV's columns map to, the
|
|
12
|
+
ixport is a Rails engine that lets a signed-in person import a host app's records from CSV files and download them as CSV files. The app registers each kind of CSV a person can import, called a mapped import, and writes an importer class that names the fields the CSV's columns map to, the field or fields rows are matched on, and the records a row may add to or update. A person picks an import, uploads a CSV file for it, matches each of the file's columns to one of the importer's fields, sees a preview of how many rows will add a record, update one or be refused, runs the import to save those rows, and then sees how many rows were added and updated and each refused row. When the run stops on an error, the import is marked failed, keeps the rows it saved before stopping, and its page shows the error. An import runs within the request by default, or, when the app is set up for it, in the background through wallflower or through a runner class the app writes. The imports page also lists the person's own past imports in the current account, each opening at the step it has reached. The app also registers each CSV a person can download, and writes an exporter class that names the columns the file can hold, in order, and the records it holds one row each for. An export is either fixed or chosen. A fixed export always holds every declared column, and a person downloads it from the exports page. A chosen export lets people in the current account save named templates, each a picked list of the exporter's columns in a picked order, and a person downloads a template from the exports page to get a file holding only those columns. Fire this local when a developer wants people to load one of the app's records from a spreadsheet or download them as one, wants people to pick which columns an export holds, wants to link to an import's upload page, the exports page, an export's download or a template's download, asks what the upload, column-matching, preview, result, exports or template pages do, asks what the list of past imports shows, asks what running an import saves, what happens when a run fails, or what an export's or template's file holds, wants imports run in a background job of the app's own, or when the app fails to boot with `Ixport::MissingImporter` or `Ixport::MissingExporter`.
|
|
13
13
|
|
|
14
14
|
## Interface
|
|
15
15
|
|
|
16
|
-
- `Ixport.register_import` — `Ixport.register_import(key, title:, importer:)` registers one mapped import under a key, with the title a person sees
|
|
16
|
+
- `Ixport.register_import` — `Ixport.register_import(key, title:, importer:, allow: nil)` registers one mapped import under a key, with the title a person sees, the name of its importer class, and optionally an access rule that decides which people may use it.
|
|
17
17
|
- `Ixport::Importer` — the base class every importer inherits from.
|
|
18
18
|
- `field` — `field key, label:` is called in an importer's class body and declares one field a CSV column can be mapped to, with the label a person sees.
|
|
19
|
-
- `match_on` — `match_on key` is called in an importer's class body and names the
|
|
20
|
-
- `records` — an instance method every importer defines
|
|
21
|
-
- `person` — returns the person
|
|
22
|
-
- `account` — returns the account
|
|
19
|
+
- `match_on` — `match_on key` or `match_on key, key, ...` is called in an importer's class body and names the declared field or fields a row is matched to an existing record on, where a row matches a record only when it equals the record on every one of them.
|
|
20
|
+
- `records` — an instance method every importer and every exporter defines: in an importer it returns the Active Record relation a row may update a record from and adds a new record to, and in an exporter it returns the records the CSV holds, one row each, in the order returned.
|
|
21
|
+
- `person` — returns the signed-in person, for use inside `records` and `value`: in an importer the person who uploaded the file, in an exporter the person downloading it.
|
|
22
|
+
- `account` — returns the current account, for use inside `records` and `value`: in an importer the account current when the file was uploaded, in an exporter the account current when the file is downloaded, and `nil` when the app has no current account.
|
|
23
23
|
- `Ixport::MissingImporter` — the error raised when the app boots if a registered import names an importer class that does not exist.
|
|
24
|
-
- `
|
|
24
|
+
- `Ixport::MissingExporter` — the error raised when the app boots if a registered export names an exporter class that does not exist.
|
|
25
|
+
- `Ixport::ImportFailed` — the error a background run is failed with when its import stopped on an error, carrying the import's error message: wallflower's task raises it, and the app's own job may raise it.
|
|
26
|
+
- `Ixport.run_import` — `Ixport.run_import(import)` or `Ixport.run_import(import, reporter: reporter)` runs one import that a runner was handed, as the signed-in person and in the account it was uploaded with, saving and refusing its rows the same way as a run within the request, marking the import failed rather than raising when the run stops on an error, and optionally reporting each row to `reporter`.
|
|
27
|
+
- `enqueue` — `enqueue(import)` is the class method the app's own runner class defines, which ixport calls with each import a person runs, once the import is marked running, so the runner can arrange for `Ixport.run_import` to run it later.
|
|
28
|
+
- `GET /` — the engine's root page, at the path the engine is mounted under, which lists every registered import by its title, each linking to that import's upload page, and then lists the signed-in person's past imports in the current account, newest first, each linking to the step it has reached.
|
|
25
29
|
- `GET /imports/new` — the upload page for one import, given the import's registered key as the `key` query parameter, which shows the import's title, a CSV file field and an Upload button.
|
|
26
30
|
- `POST /imports` — takes the `key` and `file` parameters, stores the uploaded file as an import belonging to the signed-in person and the current account, and redirects to that import's column-matching page.
|
|
27
31
|
- `GET /imports/:import_id/mapping` — the column-matching page, titled "Match columns", which lists each column header from the file's first row beside a select of the importer's field labels and a "Skip" choice.
|
|
28
32
|
- `PATCH /imports/:import_id/mapping` — saves which field each column maps to and redirects to the import's preview page.
|
|
29
|
-
- `GET /imports/:import_id/preview` — the preview page, titled "Preview", which shows how many rows will add a record, update one or be refused, lists each updating row and each refused row by its row number, saves nothing, and offers a "Run import" button, and which redirects to the import's result page once the import has
|
|
30
|
-
- `POST /imports/:import_id/run` — saves every row that adds or updates a record, records every refused row with its reason, marks the import finished, and redirects to the import's result page, and does nothing but redirect when the import has already
|
|
31
|
-
- `GET /imports/:id` — the result page, titled "Import finished", which shows how many rows the run added and updated and lists each refused row by its row number with its reason, and which redirects to the import's preview page when the import has
|
|
33
|
+
- `GET /imports/:import_id/preview` — the preview page, titled "Preview", which shows how many rows will add a record, update one or be refused, lists each updating row and each refused row by its row number, saves nothing, and offers a "Run import" button, and which redirects to the import's result page once the import has finished or failed, or while it is running in the background.
|
|
34
|
+
- `POST /imports/:import_id/run` — with no background runner, saves every row that adds or updates a record, records every refused row with its reason, marks the import finished, or failed when the run stops on an error, and redirects to the import's result page, and does nothing but redirect when the import has already finished or failed; with a background runner, marks a pending import running, hands it to the runner, and redirects to the wallflower task's page or, for the app's own runner, to the import's result page, and hands nothing to the runner and redirects to the import's result page when the import is already running, finished or failed.
|
|
35
|
+
- `GET /imports/:id` — the result page, titled "Import finished", which shows how many rows the run added and updated and lists each refused row by its row number with its reason, titled "Import running" with the message "This import is running." while a background run is under way, titled "Import failed" with the error the run stopped on and the counts of rows saved before it stopped when the run failed, and which redirects to the import's preview page when the import has neither run nor started running.
|
|
36
|
+
- `Ixport.register_export` — `Ixport.register_export(key, title:, exporter:, chosen: false, allow: nil)` registers one export under a key, with the title a person sees and the name of its exporter class, as a fixed export by default or as a chosen export when `chosen: true`, and optionally an access rule that decides which people may use it.
|
|
37
|
+
- `Ixport::Exporter` — the base class every exporter inherits from.
|
|
38
|
+
- `column` — `column key, label:, through: nil` is called in an exporter's class body and declares one column of the CSV, with the header a person sees, read from the record's `key` attribute, or from the `key` attribute of the related record named by `through:`.
|
|
39
|
+
- `value` — `value(record, column)` is the instance method that returns one cell of the CSV for one record and one declared column, and an exporter may override it to compute a cell.
|
|
40
|
+
- `GET /exports` — the exports page, titled "Exports", which lists every registered export by its title: a fixed export links to its download, and a chosen export lists the current account's templates for it with a "New template" button.
|
|
41
|
+
- `GET /exports/:id` — downloads the CSV of the export registered under the key given as `:id`, holding every declared column, as a file named after that key.
|
|
42
|
+
- `GET /exports/:export_id/templates/new` — the new template page for the chosen export registered under the key given as `:export_id`, which shows a "Template name" field and one column select per declared column.
|
|
43
|
+
- `POST /exports/:export_id/templates` — takes the `name` and `columns[]` parameters, saves a template for that chosen export belonging to the signed-in person and the current account, and redirects to the exports page.
|
|
44
|
+
- `GET /export_templates/:id` — downloads the CSV of one template, holding only its picked columns in its picked order, as a file named after the template's name.
|
|
45
|
+
- `GET /export_templates/:id/edit` — the edit template page, which shows the template's name and picked columns in the same fields as the new template page, and a Delete action.
|
|
46
|
+
- `PATCH /export_templates/:id` — takes the `name` and `columns[]` parameters, replaces the template's name and picked columns, and redirects to the exports page.
|
|
47
|
+
- `DELETE /export_templates/:id` — deletes the template and redirects to the exports page.
|
|
32
48
|
|
|
33
49
|
## How to use it
|
|
34
50
|
|
|
35
|
-
1
|
|
51
|
+
Steps 1 to 24 add a mapped import, and steps 19 to 23 of those cover running imports in the background. Steps 25 to 37 add an export, fixed or chosen. Steps 38 to 44 cover the templates of a chosen export. Do step 1 first in every case.
|
|
52
|
+
|
|
53
|
+
1. Check that ixport is installed: `bin/rails routes` lists a path served by ixport. If it does not, stop and hand the setup to the `ixport-install` local before going on. For any import, also check that `db/migrate/` has an `add_error_message_to_ixport_imports` migration and that it has run, because a run that stops on an error writes the column it adds, and without it the run raises an error naming the missing column. If it does not, hand the update to the `ixport-install` local before going on. To run imports in the background, also check that `db/migrate/` has an `add_wallflower_task_to_ixport_imports` migration and that it has run, because pressing "Run import" with any background runner reads the column it adds, including with the app's own runner. If it does not, hand the update to the `ixport-install` local before going on. For a chosen export, also check that `db/migrate/` has a `create_ixport_export_templates` migration and that it has run. If it does not, hand the update to the `ixport-install` local before going on.
|
|
54
|
+
|
|
55
|
+
### A mapped import
|
|
56
|
+
|
|
36
57
|
2. Ask the developer which of the app's records the import loads, such as products or customers. Do not pick one.
|
|
37
58
|
3. Ask the developer which attributes of that record a person may import, and the label a person should see for each, such as "SKU" for `sku` or "Price" for `price_cents`. Do not decide which attributes are importable.
|
|
38
|
-
4. Ask the developer which of those fields
|
|
59
|
+
4. Ask the developer which of those fields, alone or together, identify an existing record, such as the SKU, or a transaction's date, amount and name. Do not pick the match fields.
|
|
39
60
|
5. Ask the developer which records a person may add to or update, such as the current account's products or the signed-in person's contacts. Do not pick the scope. An import that reaches every row of the table lets one account's file update another account's records.
|
|
40
61
|
6. Write the importer class in `app/importers/`, named after the record, inheriting from `Ixport::Importer`:
|
|
41
62
|
```ruby
|
|
@@ -52,7 +73,8 @@ ixport is a Rails engine that lets a signed-in person import a host app's record
|
|
|
52
73
|
- Call `field` once per attribute from step 3, in the order a person should see them. The key is the record's attribute name as a symbol, and `label:` is required.
|
|
53
74
|
- Each field key must be an attribute the record can be assigned, because a row's mapped values are assigned to the record by those keys.
|
|
54
75
|
- The labels are the choices on the column-matching page, in the order the fields are declared.
|
|
55
|
-
- Call `match_on` once, with the key of a field declared with `field`,
|
|
76
|
+
- Call `match_on` once, with the key of every match field from step 4, each one a field declared with `field`, such as `match_on :sku` or `match_on :happened_on, :amount, :name`.
|
|
77
|
+
- Each match field must be a column of the record's table, because the match is looked up with a database query.
|
|
56
78
|
- Define `records` to return a relation of the record from step 5, built from `person` or `account`. A new record is built from this relation, so a record added through `account.products` already belongs to that account.
|
|
57
79
|
- `account` is `nil` when the app has no current account, so a `records` that calls a method on `account` fails in that case. Ask the developer whether that can happen in this app.
|
|
58
80
|
- Never define `initialize` in an importer. ixport builds the importer with the import's person and account.
|
|
@@ -62,43 +84,219 @@ ixport is a Rails engine that lets a signed-in person import a host app's record
|
|
|
62
84
|
Ixport.register_import :products, title: "Products", importer: "ProductImporter"
|
|
63
85
|
```
|
|
64
86
|
Pass `importer:` the class name as a string, never the class itself, so the initializer does not load the class before the app's code is ready.
|
|
87
|
+
|
|
88
|
+
Ask the developer whether every signed-in person may use this import, or only some. Do not decide who may. When only some may, pass `allow:` a rule that takes the signed-in person and the current account and returns true for a person who may use it, such as a call to the app's own Pundit policy:
|
|
89
|
+
```ruby
|
|
90
|
+
Ixport.register_import :products, title: "Products", importer: "ProductImporter",
|
|
91
|
+
allow: ->(person, account) { ProductImportPolicy.new(person, account).create? }
|
|
92
|
+
```
|
|
93
|
+
- A registration with no `allow:` is open to every signed-in person.
|
|
94
|
+
- The rule is called on every request to the import's pages and on every visit to the engine's root page, so it must not change anything.
|
|
95
|
+
- `account` is `nil` when the app has no current account, so the rule must handle that when it can happen.
|
|
96
|
+
- A person the rule refuses does not see the import on the engine's root page, does not see their own past imports of it under "Your imports", and gets a not found response from its upload, column-matching, preview, run and result pages, including for imports they uploaded before the rule refused them.
|
|
97
|
+
- The rule is checked within the person's request only. A background run already handed to a runner still runs to the end.
|
|
65
98
|
9. Start the app. If it fails to boot with `Ixport::MissingImporter`, the message names the import key and the class name that does not exist. Fix the class name in the registration, or the class name or file name of the importer, so the two match.
|
|
66
|
-
10. Visit the path the engine is mounted under while signed in. The page shows the heading "Imports" and lists each registered import by its title, in the order the imports were registered. With no import registered it shows "No imports are set up yet."
|
|
99
|
+
10. Visit the path the engine is mounted under while signed in. The page shows the heading "Imports" and lists each registered import the signed-in person is allowed to use by its title, in the order the imports were registered. With no import registered, or none the person is allowed to use, it shows "No imports are set up yet." Below that, once the signed-in person has uploaded any file, a "Your imports" section lists their past imports:
|
|
100
|
+
- It lists only imports uploaded by the signed-in person in the current account, newest first, of imports that are still registered and that the person is allowed to use. Imports by other people, or by the same person in another account, are not listed.
|
|
101
|
+
- Each shows the import's title, the uploaded file's name, its status, "Pending", "Running", "Finished" or "Failed", and the date it was uploaded, such as "Oct 10, 2026".
|
|
102
|
+
- A finished import also shows its "Added", "Updated" and "Refused" counts. A pending, running or failed import shows no counts.
|
|
103
|
+
- Each title links to the step the import has reached: the "Import finished" page once it has run, the "Preview" page once its mapping is saved with at least one column mapped, and otherwise the "Match columns" page. A running import's link leads to its "Import running" page, and a failed import's link leads to its "Import failed" page.
|
|
104
|
+
- When the person has no past import, the section is not shown.
|
|
67
105
|
11. When the developer wants a link to one import's upload page from the app's own pages, ask where the link goes. Build its URL with the engine's route helper and the import's registered key:
|
|
68
106
|
```erb
|
|
69
107
|
<%= link_to "Import products", ixport.new_import_path(key: :products) %>
|
|
70
108
|
```
|
|
71
|
-
The key must be one that is registered. An unregistered key gets a not found response.
|
|
109
|
+
The key must be one that is registered. An unregistered key, or one whose rule refuses the person, gets a not found response. When the import has an `allow:` rule, show the link only to people the same rule allows.
|
|
72
110
|
12. On the upload page a person chooses a CSV file and presses Upload. The upload is refused, and the upload page is shown again with status 422 and one of these messages, when:
|
|
111
|
+
- the file is not UTF-8 text: "This file is not UTF-8 text. Save it as UTF-8 and upload it again."
|
|
112
|
+
- the file is not valid CSV, such as a quote left open: "This file could not be read past line N. Check the quotes on that line."
|
|
73
113
|
- the file has no header row: "This file has no header row."
|
|
74
|
-
- the file has more data rows than the inline row limit, not counting the header row: "This file has N rows, and the most an import can have is L." The limit is set by `ixport-install
|
|
114
|
+
- the file has more data rows than the inline row limit, not counting the header row: "This file has N rows, and the most an import can have is L." The limit is set by `ixport-install`, and it applies only when imports run within the request. With a background runner, a file of any length is accepted.
|
|
75
115
|
13. After an upload is accepted, the person is on the "Match columns" page. Each column header from the file's first row is listed with a select of the importer's field labels. "Skip" leaves that column unmapped. Pressing "Save mapping" saves the choices and takes the person to the "Preview" page.
|
|
76
116
|
14. The preview decides each data row's outcome this way:
|
|
77
|
-
- The
|
|
78
|
-
-
|
|
79
|
-
- A row
|
|
80
|
-
- A row that matches
|
|
117
|
+
- The rows' values for the match fields are looked up in `records`, with one query for the whole file.
|
|
118
|
+
- Each match value from the file is first converted to the type of the record's attribute, so "12.50" matches a decimal 12.5, "100.0" matches an integer 100, and "2026-10-10" matches that date. A text attribute still matches only the same text, with its case and spaces as written.
|
|
119
|
+
- A row matches a record only when its converted values equal the record's on every match field.
|
|
120
|
+
- A row that matches a record will update it with the row's mapped values. When several records equal the row on every match field, it updates only one of them, so tell the developer the match fields must identify one record.
|
|
121
|
+
- A row that matches nothing, a row with a blank value in any match field, and every row of a file with no column mapped to one of the match fields will add a new record built from `records`.
|
|
122
|
+
- A value of false for a true-or-false match field counts as blank, so that row adds a record.
|
|
81
123
|
- A row whose added or updated record is not valid will be refused, with the record's own validation messages as the reason.
|
|
82
124
|
15. The preview page shows three counts, "Will add", "Will update" and "Will be refused". Under them it lists each row that will update a record under "Will update", and each refused row with its reason under "Will be refused", each by its row number in the file, where the header row is row 1. The refusal reasons are the record's validation messages, so tell the developer that the record's validations decide which rows are refused and what a person reads.
|
|
83
125
|
16. Viewing the preview adds and changes no record. Its back link returns the person to the "Match columns" page to change the mapping.
|
|
84
|
-
17.
|
|
126
|
+
17. When the app has no background runner, pressing "Run import" on the preview page runs the import within the same request, with no background job. Steps 19 to 23 cover an app with a background runner, where rows are saved and refused by the same rules. The run:
|
|
85
127
|
- Every row's outcome is decided again by the rules in step 14, against the records as they are at that moment, so the result can differ from the preview when records changed in between.
|
|
86
128
|
- Every row that adds or updates a record is saved with `save!`, so the record's callbacks run as for any other save.
|
|
87
129
|
- Every refused row is recorded with its row number and reason, and no record is saved for it.
|
|
88
|
-
-
|
|
89
|
-
-
|
|
90
|
-
|
|
91
|
-
|
|
130
|
+
- Rows are saved in file order, and each row's save is undone on its own when it raises.
|
|
131
|
+
- When anything raises during the run, such as a save hitting a database constraint the record's validations do not cover, the run stops at that row. The rows saved and refused before it stay saved, the rows after it are neither saved nor refused, and the import is marked failed with the error's message and the counts of rows added and updated before it stopped.
|
|
132
|
+
- The error is reported through Rails' error reporter, so it reaches whatever error service the app has subscribed there. It is not raised to the person's request.
|
|
133
|
+
- Tell the developer to cover every database constraint on the record with a validation, so a bad row is refused rather than stopping the run.
|
|
134
|
+
- An import runs once. A second run request for the same import, including one sent at the same time as the first, adds and changes nothing and redirects to the result page. A failed import is not run again, so the rows after the one it stopped on can only be loaded by uploading them as a new import.
|
|
135
|
+
18. After the run, the person is on the result page:
|
|
136
|
+
- When the run finished, the page is titled "Import finished". It shows two counts, "Added" and "Updated", and, when any row was refused, lists each refused row under "Refused" by its row number with its reason.
|
|
137
|
+
- When the run failed, the page is titled "Import failed". It shows the error's message, the line "Rows saved before it stopped stay saved.", the "Added" and "Updated" counts of rows saved before it stopped, and each row refused before it stopped.
|
|
138
|
+
- The error's message is the exception's own message, such as a database's constraint error, so tell the developer that a person reads it as written.
|
|
139
|
+
- Returning to this page later shows the same page. Visiting it for an import that has neither run nor started running redirects to its preview page. Once an import has finished or failed, its preview page redirects to this page, so it offers no "Run import" button.
|
|
140
|
+
|
|
141
|
+
### Running imports in the background
|
|
142
|
+
|
|
143
|
+
Whether imports run within the request, through wallflower, or through the app's own runner class is set by `ixport-install`. Steps 19 to 23 apply only when one of the two background runners is set.
|
|
144
|
+
|
|
145
|
+
19. With a background runner, pressing "Run import" marks the import running and hands it to the runner, and saves no row within the request:
|
|
146
|
+
- With wallflower, the person is taken to the wallflower task's page, which shows the run's progress, the rows it changed and refused, and a link to the import's result page once the run ends. When the run failed, the task fails with `Ixport::ImportFailed` carrying the import's error message, and still links to the import's "Import failed" page.
|
|
147
|
+
- With the app's own runner, the person is taken to the import's result page, titled "Import running", with the message "This import is running." and no counts. Once the run ends, the same page is titled "Import finished" or "Import failed" and shows what step 18 describes.
|
|
148
|
+
- While the import runs, its preview page redirects to its "Import running" page, so it offers no "Run import" button, and the imports page lists it as "Running" with a link that leads to that page.
|
|
149
|
+
- An import is handed to the runner once. A second run request for an import that is already running, finished or failed, such as one sent from a preview page left open, hands nothing to the runner and redirects to the import's result page.
|
|
150
|
+
20. For the app's own runner, ask the developer which background job system the app uses and which queue an import's job goes on. Do not pick either.
|
|
151
|
+
21. Write the runner class under the class name `ixport-install` set, with a class method `enqueue` that takes the import and starts a job, and write the job so it calls `Ixport.run_import` with the import:
|
|
152
|
+
```ruby
|
|
153
|
+
class ImportRunner
|
|
154
|
+
def self.enqueue(import)
|
|
155
|
+
RunImportJob.perform_later(import)
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
```
|
|
159
|
+
```ruby
|
|
160
|
+
class RunImportJob < ApplicationJob
|
|
161
|
+
def perform(import)
|
|
162
|
+
Ixport.run_import(import)
|
|
163
|
+
end
|
|
164
|
+
end
|
|
165
|
+
```
|
|
166
|
+
- Pass the import itself to the job, as above, never its id, because the job gets the import back only from what it was passed.
|
|
167
|
+
- `enqueue` is called within the person's request, so it must start the job and return, never run the import itself.
|
|
168
|
+
- The value `enqueue` returns is not used.
|
|
169
|
+
22. `Ixport.run_import` runs the import by the rules in steps 14 and 17:
|
|
170
|
+
- It builds the importer with the person who uploaded the file and the account current at that upload, so the job needs no signed-in person and `records` is scoped as it was for the preview.
|
|
171
|
+
- It adds and changes nothing when the import has already finished or failed.
|
|
172
|
+
- When the run stops on an error, it keeps the rows saved before it, marks the import failed, reports the error through Rails' error reporter, and returns without raising, as in step 17. The job therefore succeeds, and a retry of it would change nothing.
|
|
173
|
+
- Ask the developer whether a failed import should also fail the job, such as to show it in the job system's failed list. If it should, raise `Ixport::ImportFailed` with the import's error message after the run, and turn off retries for it, since a retry changes nothing:
|
|
174
|
+
```ruby
|
|
175
|
+
def perform(import)
|
|
176
|
+
Ixport.run_import(import)
|
|
177
|
+
raise Ixport::ImportFailed, import.error_message if import.failed?
|
|
178
|
+
end
|
|
179
|
+
```
|
|
180
|
+
- When `enqueue` raises, or the job never runs, the import stays running and its page keeps showing "This import is running.". Tell the developer this, and ask how they want such imports noticed.
|
|
181
|
+
23. When the developer wants to show a run's progress somewhere of their own, pass `reporter:` an object that answers all three of these, and ask the developer where the progress is shown:
|
|
182
|
+
- `set_total(total)` is called once, before the first row, with the number of data rows.
|
|
183
|
+
- `refuse(label:, reason:)` is called for each refused row, with a label such as "Row 4" and the refusal reason.
|
|
184
|
+
- `advance` is called once after each row, saved or refused.
|
|
185
|
+
24. An import's column-matching, preview, run and result pages are visible only to the person who uploaded it, in the account it was uploaded in, while the import's `allow:` rule allows them. Anyone else, the same person in another account, and the same person once the rule refuses them, gets a not found response.
|
|
186
|
+
|
|
187
|
+
### An export
|
|
188
|
+
|
|
189
|
+
25. Ask the developer which of the app's records the export holds, such as products or customers. Do not pick one.
|
|
190
|
+
26. Ask the developer which columns the file can hold, in order, and the header a person should see for each, such as "SKU" for `sku`. For each column, ask whether it is an attribute of the record itself, an attribute of a related record such as the product's account name, or a value computed from the record. Do not decide which columns are exported.
|
|
191
|
+
27. Ask the developer which records the file holds, such as the current account's products, and in what order its rows go. Do not pick the scope or the order. An export that reaches every row of the table lets one account download another account's records.
|
|
192
|
+
28. Ask the developer whether the export is fixed or chosen. Do not pick one.
|
|
193
|
+
- Fixed: every download holds every column from step 26, in the declared order.
|
|
194
|
+
- Chosen: people in the current account save named templates that each pick some of those columns in an order of their own, and each template downloads only its picked columns.
|
|
195
|
+
29. Write the exporter class in `app/exporters/`, named after the record, inheriting from `Ixport::Exporter`. A fixed export and a chosen export use the same kind of class:
|
|
196
|
+
```ruby
|
|
197
|
+
class ProductExporter < Ixport::Exporter
|
|
198
|
+
column :sku, label: "SKU"
|
|
199
|
+
column :price_cents, label: "Price"
|
|
200
|
+
column :name, label: "Account", through: :account
|
|
201
|
+
|
|
202
|
+
def records
|
|
203
|
+
account.products.includes(:account).order(:sku)
|
|
204
|
+
end
|
|
205
|
+
end
|
|
206
|
+
```
|
|
207
|
+
- Call `column` once per column from step 26, in the order the columns appear in the file. `label:` is required and is that column's header in the file's first row.
|
|
208
|
+
- For a chosen export, the labels are the choices in each column select on the template pages, in the declared order.
|
|
209
|
+
- Each column's key must be unique within the exporter, because a template records its picked columns by key.
|
|
210
|
+
- For an attribute of the record itself, the key is the attribute's name as a symbol.
|
|
211
|
+
- For an attribute of a related record, pass `through:` the name of the association on the record, and the key is the attribute's name on the related record. A record whose related record is `nil` gets an empty cell in that column.
|
|
212
|
+
- Define `records` to return the records from step 27, built from `person` or `account`, in the row order from step 27. The file has one row per record, in the order `records` returns them.
|
|
213
|
+
- When any column uses `through:`, add `includes` for each association it names to `records`, so the download does not run one query per row.
|
|
214
|
+
- `account` is `nil` when the app has no current account, so a `records` that calls a method on `account` fails in that case. Ask the developer whether that can happen in this app.
|
|
215
|
+
- Never define `initialize` in an exporter. ixport builds the exporter with the signed-in person and the current account.
|
|
216
|
+
30. For a computed column from step 26, declare it with `column` under a key of your choosing and override `value` to return it, calling `super` for every other column:
|
|
217
|
+
```ruby
|
|
218
|
+
def value(record, column)
|
|
219
|
+
return record.price_cents / 100.0 if column.key == :price
|
|
220
|
+
|
|
221
|
+
super
|
|
222
|
+
end
|
|
223
|
+
```
|
|
224
|
+
- `column.key` is the column's key as a symbol, and `column.label` is its header.
|
|
225
|
+
- Ask the developer how the computed value is written, such as how many decimal places, rather than picking a format.
|
|
226
|
+
- A template's download writes its cells through the same `value`, so a computed column is computed the same way in every template that picks it.
|
|
227
|
+
31. Every cell is written as the value's `to_s`, and a `nil` value is an empty cell. When the developer needs a date, time or amount written a particular way, compute that column in `value` as in step 30.
|
|
228
|
+
32. Ask the developer what key and title the export gets. The key is a short symbol that identifies the export, such as `:products`, and is also the fixed export's downloaded file name, such as `products.csv`. The title is what a person sees on the exports page, such as "Products".
|
|
229
|
+
33. Add the registration to the end of `config/initializers/ixport.rb`, and create the file if it does not exist. For a fixed export:
|
|
230
|
+
```ruby
|
|
231
|
+
Ixport.register_export :products, title: "Products", exporter: "ProductExporter"
|
|
232
|
+
```
|
|
233
|
+
For a chosen export, add `chosen: true`:
|
|
234
|
+
```ruby
|
|
235
|
+
Ixport.register_export :products, title: "Products", exporter: "ProductExporter", chosen: true
|
|
236
|
+
```
|
|
237
|
+
Pass `exporter:` the class name as a string, never the class itself, so the initializer does not load the class before the app's code is ready.
|
|
238
|
+
|
|
239
|
+
Ask the developer whether every signed-in person may use this export, or only some. Do not decide who may. When only some may, pass `allow:` a rule that takes the signed-in person and the current account and returns true for a person who may use it, the same way as an import's rule in step 8:
|
|
240
|
+
```ruby
|
|
241
|
+
Ixport.register_export :products, title: "Products", exporter: "ProductExporter", chosen: true,
|
|
242
|
+
allow: ->(person, account) { ProductExportPolicy.new(person, account).show? }
|
|
243
|
+
```
|
|
244
|
+
- A registration with no `allow:` is open to every signed-in person.
|
|
245
|
+
- A person the rule refuses does not see the export on the exports page, and gets a not found response from its download and, for a chosen export, from its new template page, its template creation, and every one of its templates' download, edit, update and delete actions.
|
|
246
|
+
- The rule covers the export as a whole. Every template of a chosen export is allowed or refused with it, and a template cannot carry a rule of its own.
|
|
247
|
+
34. Start the app. If it fails to boot with `Ixport::MissingExporter`, the message names the export key and the class name that does not exist. Fix the class name in the registration, or the class name or file name of the exporter, so the two match.
|
|
248
|
+
35. Visit `exports` under the path the engine is mounted under while signed in. The page shows the heading "Exports" and lists each registered export the signed-in person is allowed to use by its title, in the order the exports were registered. With no export registered, or none the person is allowed to use, it shows "No exports are set up yet." The engine's root page lists imports only and does not link to this page.
|
|
249
|
+
- A fixed export's title links to its download.
|
|
250
|
+
- A chosen export's title is not a link. Under it are the current account's templates for that export, ordered by name, each name linking to the template's download beside an "Edit" link, and then a "New template" button.
|
|
251
|
+
36. When the developer wants a link to the exports page or to one fixed export's download from the app's own pages, ask where the link goes. Build its URL with the engine's route helpers and the export's registered key:
|
|
252
|
+
```erb
|
|
253
|
+
<%= link_to "Exports", ixport.exports_path %>
|
|
254
|
+
<%= link_to "Download products", ixport.export_path(:products) %>
|
|
255
|
+
```
|
|
256
|
+
The key must be one that is registered. An unregistered key, or one whose rule refuses the person, gets a not found response. When the export has an `allow:` rule, show the link only to people the same rule allows. `export_path` also downloads a chosen export with every declared column, although the exports page does not link to it.
|
|
257
|
+
37. A download builds the whole file within the same request, with no background job and no row limit, from the records `records` returns for the person downloading it, in the account current at that moment. This holds for a template's download too. Tell the developer that an export over a very large table makes that request slow.
|
|
258
|
+
|
|
259
|
+
### The templates of a chosen export
|
|
260
|
+
|
|
261
|
+
38. Pressing "New template" on the exports page opens the page titled "New <export title> template", with a back link to the exports page. It shows a required "Template name" field and one select per declared column, labelled "Column 1", "Column 2" and so on. Each select offers every column's label and "None". Pressing "Save template" saves the template and returns the person to the exports page.
|
|
262
|
+
39. The picked columns are saved in this way:
|
|
263
|
+
- They are kept in the order of the selects, so "Column 1" is the file's first column.
|
|
264
|
+
- A select left at "None" adds nothing.
|
|
265
|
+
- A column picked in more than one select is kept once, at its first position.
|
|
266
|
+
- A template with no picked column produces a file with an empty header row and one empty row per record.
|
|
267
|
+
40. A template belongs to the current account. Every person in that account whom the export's `allow:` rule allows sees it on the exports page and can download, edit or delete it. A person in another account, or one the rule refuses, gets a not found response. When the app has no current account, every template saved with no account is shared by every signed-in person with no account. Tell the developer this, and ask whether it is what they want.
|
|
268
|
+
41. A template's name link downloads a CSV holding only its picked columns, in its picked order, with each column's label as the header and one row per record from `records`. The file is named after the template's name, lower-cased with spaces and punctuation replaced by hyphens, such as `price-list.csv` for "Price list".
|
|
269
|
+
42. Pressing "Edit" opens the page titled "Edit <template name>", with the export's title under it and a back link to the exports page. It shows the same fields as the new template page, filled with the template's name and picked columns. Pressing "Save template" replaces the name and picked columns by the rules in step 39 and returns the person to the exports page.
|
|
270
|
+
43. The edit page's action menu has a Delete item. It deletes the template and returns the person to the exports page.
|
|
271
|
+
44. When the developer wants a link to one template's download or edit page from the app's own pages, ask where the link goes, and build it with the engine's route helpers and the template:
|
|
272
|
+
```erb
|
|
273
|
+
<%= link_to template.name, ixport.export_template_path(template) %>
|
|
274
|
+
<%= link_to "Edit", ixport.edit_export_template_path(template) %>
|
|
275
|
+
<%= link_to "New template", ixport.new_export_template_path(:products) %>
|
|
276
|
+
```
|
|
277
|
+
`new_export_template_path` takes the chosen export's registered key. An unregistered key, the key of a fixed export, or the key of an export whose rule refuses the person, gets a not found response.
|
|
92
278
|
|
|
93
279
|
## Conventions
|
|
94
280
|
|
|
95
|
-
- Every registration names an importer class that exists. The app refuses to boot otherwise, so never register an import before its
|
|
96
|
-
- Each key is registered once. Registering the same key a second time replaces the first registration.
|
|
97
|
-
- Every importer declares `match_on` and defines `records`.
|
|
281
|
+
- Every registration names an importer or exporter class that exists. The app refuses to boot otherwise, so never register an import or export before its class is written.
|
|
282
|
+
- Each import key is registered once, and each export key is registered once. Registering the same key a second time replaces the first registration.
|
|
283
|
+
- Every importer declares `match_on` with at least one field and defines `records`.
|
|
284
|
+
- Every exporter declares at least one `column` and defines `records`.
|
|
98
285
|
- `records` is always scoped to the `person` or `account` the developer chose, never left to the whole table without the developer saying so.
|
|
99
|
-
- An importer inherits from `Ixport::Importer` directly and declares all of its own fields and its own match field.
|
|
100
|
-
-
|
|
101
|
-
- When
|
|
102
|
-
- When an
|
|
103
|
-
-
|
|
104
|
-
-
|
|
286
|
+
- An importer inherits from `Ixport::Importer` directly and declares all of its own fields and its own match field. An exporter inherits from `Ixport::Exporter` directly and declares all of its own columns. None of these is inherited by a class that inherits from another importer or exporter.
|
|
287
|
+
- An exporter that overrides `value` calls `super` for every column it does not compute.
|
|
288
|
+
- When a record's attribute or association is renamed or removed, update the matching `field`, `match_on` and `column` lines in the same change.
|
|
289
|
+
- When an importer or exporter class is renamed, update its `importer:` or `exporter:` string in `config/initializers/ixport.rb` in the same change.
|
|
290
|
+
- When an import's or export's key is changed, update every link to its upload page or download in the same change.
|
|
291
|
+
- Past imports record their import's key. Before changing an import's key or removing its registration, tell the developer that past imports under the old key drop off "Your imports" on the engine's root page, and that opening one of their pages raises an error, and ask what should happen to those imports.
|
|
292
|
+
- Every engine page first runs the before actions of the app controller ixport's pages inherit from, which is set by `ixport-install`, so a person that controller turns away never reaches an import or export. Among the people it lets through, who may use an import or export is decided only by its `allow:` rule. Never write a before action, a policy check or a redirect in front of the engine's pages.
|
|
293
|
+
- When an import or export has an `allow:` rule, every link to it from the app's own pages is shown only to people the same rule allows.
|
|
294
|
+
- Saved templates record a chosen export's key and its picked column keys. Before changing a chosen export's key, removing its registration, or changing it to fixed, ask the developer what happens to the templates already saved for it:
|
|
295
|
+
- A template whose export key is no longer registered no longer shows on the exports page, and opening its download or edit page raises an error.
|
|
296
|
+
- A template whose export is changed to fixed no longer shows on the exports page, and its download and edit links still work.
|
|
297
|
+
- When a chosen export's column key is renamed or removed, saved templates that picked it drop that column from their downloads without saying so. Tell the developer before making that change.
|
|
298
|
+
- Never write an upload form, a column-matching form, a preview page, a run action, a result page, an exports page, a CSV download action, a template form or model, or a controller for them in the app. The engine serves all of them.
|
|
299
|
+
- The app's own runner only starts a job, and the job only calls `Ixport.run_import`. Never write a job that reads the file, matches rows or saves records itself.
|
|
300
|
+
- When the app's own runner class is renamed, tell the developer that the class name `ixport-install` set must be changed in the same change.
|
|
301
|
+
- Never wrap `Ixport.run_import` in a rescue, a transaction or a retry of the app's own. It records its own failure, keeps the rows saved before it, and reports the error.
|
|
302
|
+
- Installing ixport, running the update generator for the export templates table, the background runner's column or the failed import's error column, subscribing an error service to Rails' error reporter, mounting the engine, the tables an import, its refused rows and export templates are stored in, file storage, sign-in, how the current person and account are found, the layout, the inline row limit, which of the app's controllers the engine's pages inherit from, and choosing whether imports run within the request, through wallflower or through the app's own runner class are out of scope for this local and belong to `ixport-install`.
|
|
@@ -1,59 +1,94 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: ixport-info
|
|
3
|
-
description: Use to learn what ixport offers — CSV import through a mounted Rails engine, the mapped imports a host app registers, the importer classes behind them, uploading a file, matching its columns, previewing which rows will add, update or be refused, running the import, its result page, and the vocabulary the install and develop locals assume.
|
|
3
|
+
description: Use to learn what ixport offers — CSV import and export through a mounted Rails engine, the mapped imports and the fixed and chosen exports a host app registers, the importer and exporter classes behind them, uploading a file, matching its columns, previewing which rows will add, update or be refused, running the import inside the request or in the background through wallflower or the app's own runner, its result page, what a person sees when a file is refused or a run fails, the list of a person's past imports, downloading an export as a CSV file, saving named templates of picked columns for a chosen export, limiting each import or export to the people a rule allows, and the vocabulary the install and develop locals assume.
|
|
4
4
|
tools: Read
|
|
5
|
-
scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports the host app registers, each backed by an importer class the host writes
|
|
5
|
+
scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports and the exports the host app registers, each import backed by an importer class and each export by an exporter class the host writes, where an export is either fixed or chosen, and a chosen export lets the account save named templates of picked columns
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
This local explains what ixport is and routes you to the local that does the work. It makes no changes and gives no steps.
|
|
9
9
|
|
|
10
10
|
## What ixport is
|
|
11
11
|
|
|
12
|
-
ixport is a Rails engine for loading a host app's records from CSV files. The host app mounts it at a path of its choosing
|
|
12
|
+
ixport is a Rails engine for loading a host app's records from CSV files and downloading them back out as CSV files. The host app mounts it at a path of its choosing. A signed-in person who visits that path sees the list of imports the host has set up, and a signed-in person who visits its exports page sees the list of exports, each shown by its title.
|
|
13
13
|
|
|
14
14
|
A person picks an import, uploads a CSV file, matches each column of the file to one of the columns the import reads or skips it, and then sees a preview. The preview counts how many rows will add a record, how many will update one, and how many will be refused, and lists the rows that will update and the rows that will be refused with the reason for each. Viewing the preview saves nothing in the host app, and a Back link returns to the mapping so the person can change it.
|
|
15
15
|
|
|
16
|
-
From the preview the person runs the import. The run saves every added and updated record in the host app, records each refused row with its reason, and sends the person to the import's result page, which shows how many rows were added and updated and lists each refused row.
|
|
16
|
+
From the preview the person runs the import. The run saves every added and updated record in the host app, records each refused row with its reason, and sends the person to the import's result page, which shows how many rows were added and updated and lists each refused row. An import that has finished is never run again. Opening the preview of a running or finished import sends the person to its result page instead, so no Run button is offered for it, and opening the result page of an import that has not started running sends the person to its preview.
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
A run that stops on an error is marked Failed. Every row saved before the error stays saved, and the result page is headed Import failed, shows the error's message, and shows how many rows were added and updated before it stopped along with each refused row. ixport also reports the error to the host app's error reporting. A failed import is never run again.
|
|
19
19
|
|
|
20
|
-
|
|
20
|
+
Where the run happens depends on the host's background runner. With none, the run happens inside the request, as above. With one, every import runs through it whatever its size: running the import marks it Running and hands it to the runner, and its result page says the import is running until the run finishes. An import is handed to the runner only once: asking to run it again while it is running, or after it has finished, sends the person to its result page.
|
|
21
21
|
|
|
22
|
-
|
|
22
|
+
- **Wallflower** — the host can choose wallflower as the runner. Running an import then starts a wallflower task and takes the person to the task's page, which shows the run's progress and each refused row with its reason, and links to the import's result page once the run finishes. When the run fails, the task is marked failed with the import's error message and still links to the import's result page.
|
|
23
|
+
- **The app's own runner** — the host can instead name a class of its own. ixport hands each import to that class, and the class runs it later by calling ixport back, after which the result page shows the counts and refused rows as usual. Running the import takes the person to the import's result page.
|
|
24
|
+
|
|
25
|
+
ixport refuses a file on the upload page, before any import record is kept, when it is not UTF-8 text, when a line's quotes stop it being read, naming that line, or when it has no header row. With no background runner, it also refuses a file with more rows than the host's row limit there. Each refusal says what is wrong with the file.
|
|
26
|
+
|
|
27
|
+
Below the list of imports, the same page lists the person's own past imports in the current account, newest first. Each shows the import's title, the uploaded file's name, its status, the date it was uploaded, and, once it has finished, how many rows were added, updated and refused. A failed import shows its status as Failed and no counts. Each title opens the import at the step it has reached: the mapping when no mapping is saved, the preview when it has a mapping and has not started running, and the result page once it is running, finished or failed. Imports uploaded by other people, or by the same person in another account, are not listed, and neither are the person's imports of a kind their access rule now refuses.
|
|
28
|
+
|
|
29
|
+
An export is either fixed or chosen. A fixed export's title on the exports page links to a CSV download holding every column the exporter declares. A chosen export's title is not a link. Under it the page lists the templates the account has saved for that export, each linking to a download and to its edit page, with a button to make a new template.
|
|
30
|
+
|
|
31
|
+
A template is a name and an ordered pick of the exporter's columns. A person makes one on its own page, entering a name and choosing a column, or None, for each slot. Its download holds only the picked columns, in the order picked, and the file is named after the template. On the edit page a person can change the name and the picked columns, or delete the template. A template belongs to the account, so everyone working in that account sees, downloads, edits and deletes the same templates.
|
|
32
|
+
|
|
33
|
+
Every export file's first row holds the column labels, and each later row holds one of the host's records. The host decides which records an export holds, so an export can be limited to the person's current account. The file is built inside the request and nothing about it is stored.
|
|
34
|
+
|
|
35
|
+
By default every signed-in person can use every import and export. The host can give any import or export an access rule, which is handed the signed-in person and the current account and allows or refuses them, and which can ask the app's own permission checks, such as a Pundit policy. A person the rule refuses does not see that import or export listed, does not see their past imports of it, and is answered not found on every one of its pages and downloads, including its upload page, an existing import's mapping, preview, run and result pages, and a chosen export's template pages and downloads. The rule is checked again on every request, so a person whose permissions change is let in or kept out from their next page.
|
|
36
|
+
|
|
37
|
+
The engine owns the screens, the access check, and its own tables of import records, refused rows and export templates, with the uploaded file stored through Active Storage. The host owns what each import and export means: which columns it reads or writes, which column or columns identify an existing record, and which of the app's records a row is checked against, saved into, or read from. Reach for ixport when a Rails app needs people to load data from spreadsheets or take it out into one, rather than writing a one-off upload or download page per record type.
|
|
38
|
+
|
|
39
|
+
The engine renders its pages with Keystone UI inside a layout the host chooses, and it runs its access check through a method the host already has. Its pages inherit from a controller the host names, which is the host's own application controller unless the host names another, so the filters, helpers and error handling the host puts on that controller apply to ixport's pages as well.
|
|
23
40
|
|
|
24
41
|
## Interface
|
|
25
42
|
|
|
26
43
|
ixport declares nothing for this local to document. Its surface is split between the other two locals:
|
|
27
44
|
|
|
28
|
-
- **ixport-install** owns hooking ixport into a host app: adding the gem, running its install generator, mounting the engine, and the configuration that names the host's sign-in check, current person, current account, layout and
|
|
29
|
-
- **ixport-develop** owns building against ixport: registering a mapped import
|
|
45
|
+
- **ixport-install** owns hooking ixport into a host app: adding the gem, running its install generator, running its update generator to bring an existing install's tables up to date, mounting the engine, and the configuration that names the host's sign-in check, current person, current account, layout, row limit, background runner and the controller ixport's pages inherit from.
|
|
46
|
+
- **ixport-develop** owns building against ixport: registering a mapped import and the importer class behind it, registering a fixed or chosen export and the exporter class behind it, giving an import or export an access rule, writing the app's own background runner and running an import from it, the error a wallflower task raises when its import fails, the errors raised when a registration names a class that does not exist, and the engine's pages for imports, exports and export templates.
|
|
30
47
|
|
|
31
48
|
## How to use it
|
|
32
49
|
|
|
33
|
-
- The app does not have ixport yet, or you are changing how ixport reaches the app's sign-in, people, accounts, layout or row limit: use **ixport-install**.
|
|
34
|
-
- ixport is already installed and you are adding or changing an import, changing how
|
|
50
|
+
- The app does not have ixport yet, the app installed ixport before export templates or background runs existed and needs their tables, you are choosing whether imports run inside the request, through wallflower or through the app's own runner, or you are changing how ixport reaches the app's sign-in, people, accounts, layout or row limit, or which of the app's controllers its pages inherit from: use **ixport-install**.
|
|
51
|
+
- ixport is already installed and you are adding or changing an import or an export, making an export chosen or fixed, changing how an import's rows are matched, checked or saved, changing which records or columns an export holds, limiting an import or export to some people, writing the app's own runner that runs imports in the background, linking to the engine's pages, or the app fails to boot naming a missing importer or exporter: use **ixport-develop**.
|
|
35
52
|
|
|
36
53
|
## Conventions
|
|
37
54
|
|
|
38
55
|
- **Mapped import** — one kind of import the host registers, made of a key, a title shown to people, and the name of its importer class.
|
|
39
|
-
- **
|
|
40
|
-
- **
|
|
41
|
-
- **
|
|
42
|
-
- **
|
|
56
|
+
- **Fixed export** — one kind of export the host registers, made of a key, a title shown to people, and the name of its exporter class, whose download always holds every declared column. An export is fixed unless the host registers it as chosen.
|
|
57
|
+
- **Chosen export** — an export registered the same way and marked as chosen, which has no download of its own and is downloaded only through the account's templates.
|
|
58
|
+
- **Export template** — a named, ordered pick of a chosen export's columns, saved by one person and shared by everyone in their account.
|
|
59
|
+
- **Picked columns** — the columns a template holds, in the order they were chosen. A slot left on None is dropped, a column picked twice is kept once, and a column the exporter no longer declares is left out of the download.
|
|
60
|
+
- **Key** — the short name an import or export is registered and looked up by, held as a symbol. A fixed export's file is named after its key.
|
|
61
|
+
- **Title** — the words a person sees for the import or export on the engine's pages.
|
|
62
|
+
- **Importer** — a class the host app writes that declares the columns the import reads, each with a key and a label, names the column or columns that together identify an existing record, and supplies the set of the app's records the rows are checked against and saved into.
|
|
63
|
+
- **Exporter** — a class the host app writes that declares the columns the export writes, each with a key and a label, and supplies the set of the app's records the file holds.
|
|
64
|
+
- **Export column** — one column of an export's file, headed by its label and filled from the attribute its key names on each record.
|
|
65
|
+
- **Through** — an export column can be read from a record related to each row's record rather than from the record itself, and it is left blank when that related record is missing.
|
|
66
|
+
- **Match column** — a declared import column whose value decides whether a row is an existing record or a new one. An import can have one or several, and a row is an existing record only when it equals that record on every one of them.
|
|
67
|
+
- **Matched value** — a row's value in a match column, compared as the record's own type rather than as text, so 100.0 in a file matches 100 stored on a record.
|
|
43
68
|
- **Import record** — one upload by one person, stored by the engine with the key of its mapped import, the uploaded file, the mapping once it is saved, and the added and updated counts once it has run.
|
|
44
69
|
- **Headers** — the first row of the uploaded file, read as the names of its columns.
|
|
45
70
|
- **Mapping** — which of the importer's columns each header in the file fills, stored by the header's position in the row. A header left on "Skip" is not stored.
|
|
46
|
-
- **Row limit** — the most data rows an uploaded file may have, not counting the header row. It defaults to 1,000 and the host can change it.
|
|
71
|
+
- **Row limit** — the most data rows an uploaded file may have, not counting the header row. It defaults to 1,000 and the host can change it. It applies only when the host has no background runner, and it does not apply to exports.
|
|
72
|
+
- **Background runner** — what runs an import outside the request: none, wallflower, or a class the host app writes. Having wallflower in the app does not choose it, the host has to.
|
|
73
|
+
- **Wallflower** — a separate gem that runs long tasks in the background and gives each task a page showing its progress. ixport registers its imports with it as a kind of task titled Import.
|
|
47
74
|
- **Row number** — a row's line in the file, counting the header as row 1, so the first data row is row 2.
|
|
48
|
-
- **Add** — a row
|
|
49
|
-
- **Update** — a row
|
|
75
|
+
- **Add** — a row that no existing record equals on every match column, or that is blank in any match column, or comes from a file with no column mapped to one, so it creates a record.
|
|
76
|
+
- **Update** — a row that equals an existing record on every match column, so it changes that record.
|
|
50
77
|
- **Refused** — a row whose record, new or changed, fails the host model's own validations. Its reason is that model's error messages.
|
|
51
78
|
- **Preview** — the page that shows the add, update and refused counts and lists the update and refused rows by row number. It builds each row's record in memory and saves none of them.
|
|
52
|
-
- **Run** — the step that saves the added and updated records and stores each refused row's number and reason. A run that raises an error
|
|
53
|
-
- **
|
|
54
|
-
- **
|
|
55
|
-
- **
|
|
56
|
-
- **
|
|
57
|
-
- **
|
|
58
|
-
- **
|
|
59
|
-
-
|
|
79
|
+
- **Run** — the step that saves the added and updated records one row at a time and stores each refused row's number and reason. A run that raises an error stops there and keeps the rows it already saved, and an import that has finished or failed is never run a second time.
|
|
80
|
+
- **Status** — an import record is Pending until it is run, Running while a background runner has it, and then Finished, or Failed when the run stopped on an error.
|
|
81
|
+
- **Running** — the state of an import record handed to a background runner and not yet finished. Its result page says it is running and shows no counts, its preview sends the person to that result page, and it is never handed to the runner a second time.
|
|
82
|
+
- **Finished** — the state of an import record once it has run to the end.
|
|
83
|
+
- **Failed** — the state of an import record whose run stopped on an error. It keeps the error's message and the added and updated counts reached before it stopped, and its result page shows them.
|
|
84
|
+
- **Refused file** — an upload ixport turns away on the upload page because it is not UTF-8, cannot be read past a line, has no header row, or has too many rows. Nothing is stored for it, unlike a refused row, which belongs to an import that ran.
|
|
85
|
+
- **Past imports** — the person's own import records in the current account, listed newest first under the imports on the engine's front page.
|
|
86
|
+
- **Step** — how far an import record has got: mapping when no mapping is saved, preview once one is, and result once it is running, finished or failed.
|
|
87
|
+
- **Result page** — the import's own page, showing the added and updated counts and each refused row as they were stored by the run, and the error's message when the run failed, so it shows the same thing each time it is opened.
|
|
88
|
+
- **Person** — whoever is signed in and running the import or downloading the export. ixport reaches them through a method the host names, so the host's user model needs no particular name.
|
|
89
|
+
- **Parent controller** — the host app's controller that every ixport page inherits from, the app's application controller by default. Whatever it adds to every request, such as a filter, a helper or a rescue, runs on ixport's pages too, and the sign-in check, current person and current account methods ixport calls are looked up on it.
|
|
90
|
+
- **Account** — the optional owner the person is working inside, such as a team or company, reached the same way. An app with no accounts runs without one. Both the importer and the exporter are given the person and the account.
|
|
91
|
+
- **Ownership** — an import record belongs to the person and account that uploaded it, and anyone else who opens it, including the same person from another account, is answered not found. An export template belongs to its account, and anyone opening it from another account is answered not found.
|
|
92
|
+
- **Access rule** — an optional check the host gives one import or export when registering it, handed the person and the account and answering whether they may use it. It is checked on top of ownership, so even the person who uploaded an import is answered not found for it once the rule refuses them. An import or export with no rule is open to every signed-in person.
|
|
93
|
+
- **Boot check** — when the app starts, ixport checks that every registered importer and exporter class exists and refuses to boot when one does not, so a misspelled class name fails at startup rather than when someone runs the import or downloads the export.
|
|
94
|
+
- Imports and exports are registered by the host app, never discovered: an importer or exporter class that is not registered does not appear on the page, asking for an export key that is not registered is answered not found, and asking to make a template for an export that is not chosen is answered not found.
|