ixport 0.1.0 → 0.2.0

data/the_local/agents/ixport-info.md

@@ -1,23 +1,40 @@
 ---
 name: ixport-info
-description: Use to learn what ixport offers — CSV import through a mounted Rails engine, the mapped imports a host app registers, the importer classes behind them, uploading a file, matching its columns, previewing which rows will add, update or be refused, running the import, its result page, and the vocabulary the install and develop locals assume.
+description: Use to learn what ixport offers — CSV import and export through a mounted Rails engine, the mapped imports and the fixed and chosen exports a host app registers, the importer and exporter classes behind them, uploading a file, matching its columns, previewing which rows will add, update or be refused, running the import inside the request or in the background through wallflower or the app's own runner, its result page, what a person sees when a file is refused or a run fails, the list of a person's past imports, downloading an export as a CSV file, saving named templates of picked columns for a chosen export, limiting each import or export to the people a rule allows, and the vocabulary the install and develop locals assume.
 tools: Read
-scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports the host app registers, each backed by an importer class the host writes
+scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports and the exports the host app registers, each import backed by an importer class and each export by an exporter class the host writes, where an export is either fixed or chosen, and a chosen export lets the account save named templates of picked columns
 ---
 
 This local explains what ixport is and routes you to the local that does the work. It makes no changes and gives no steps.
 
 ## What ixport is
 
-ixport is a Rails engine for loading a host app's records from CSV files. The host app mounts it at a path of its choosing, and a signed-in person who visits that path sees the list of imports the host has set up, each shown by its title.
+ixport is a Rails engine for loading a host app's records from CSV files and downloading them back out as CSV files. The host app mounts it at a path of its choosing. A signed-in person who visits that path sees the list of imports the host has set up, and a signed-in person who visits its exports page sees the list of exports, each shown by its title.
 
 A person picks an import, uploads a CSV file, matches each column of the file to one of the columns the import reads or skips it, and then sees a preview. The preview counts how many rows will add a record, how many will update one, and how many will be refused, and lists the rows that will update and the rows that will be refused with the reason for each. Viewing the preview saves nothing in the host app, and a Back link returns to the mapping so the person can change it.
 
-From the preview the person runs the import. The run saves every added and updated record in the host app, records each refused row with its reason, and sends the person to the import's result page, which shows how many rows were added and updated and lists each refused row. The run happens inside the request, an import runs once, and asking to run it again changes nothing. Opening the preview of a finished import sends the person to its result page instead, and opening the result page of an import that has not run sends the person to its preview.
+From the preview the person runs the import. The run saves every added and updated record in the host app, records each refused row with its reason, and sends the person to the import's result page, which shows how many rows were added and updated and lists each refused row. An import that has finished is never run again. Opening the preview of a running or finished import sends the person to its result page instead, so no Run button is offered for it, and opening the result page of an import that has not started running sends the person to its preview.
 
-ixport refuses a file with no header row, and a file with more rows than the host's row limit, on the upload page. It has no export yet.
+A run that stops on an error is marked Failed. Every row saved before the error stays saved, and the result page is headed Import failed, shows the error's message, and shows how many rows were added and updated before it stopped along with each refused row. ixport also reports the error to the host app's error reporting. A failed import is never run again.
 
-The engine owns the screens, the access check, and its own tables of import records and refused rows, with the uploaded file stored through Active Storage. The host owns what each import means: which columns it reads, which column identifies an existing record, and which of the app's records a row is checked against and saved into. Reach for ixport when a Rails app needs people to load data from spreadsheets, rather than writing a one-off upload page per record type.
+Where the run happens depends on the host's background runner. With none, the run happens inside the request, as above. With one, every import runs through it whatever its size: running the import marks it Running and hands it to the runner, and its result page says the import is running until the run finishes. An import is handed to the runner only once: asking to run it again while it is running, or after it has finished, sends the person to its result page.
+
+- **Wallflower** — the host can choose wallflower as the runner. Running an import then starts a wallflower task and takes the person to the task's page, which shows the run's progress and each refused row with its reason, and links to the import's result page once the run finishes. When the run fails, the task is marked failed with the import's error message and still links to the import's result page.
+- **The app's own runner** — the host can instead name a class of its own. ixport hands each import to that class, and the class runs it later by calling ixport back, after which the result page shows the counts and refused rows as usual. Running the import takes the person to the import's result page.
+
+ixport refuses a file on the upload page, before any import record is kept, when it is not UTF-8 text, when a line's quotes stop it being read, naming that line, or when it has no header row. With no background runner, it also refuses a file with more rows than the host's row limit there. Each refusal says what is wrong with the file.
+
+Below the list of imports, the same page lists the person's own past imports in the current account, newest first. Each shows the import's title, the uploaded file's name, its status, the date it was uploaded, and, once it has finished, how many rows were added, updated and refused. A failed import shows its status as Failed and no counts. Each title opens the import at the step it has reached: the mapping when no mapping is saved, the preview when it has a mapping and has not started running, and the result page once it is running, finished or failed. Imports uploaded by other people, or by the same person in another account, are not listed, and neither are the person's imports of a kind their access rule now refuses.
+
+An export is either fixed or chosen. A fixed export's title on the exports page links to a CSV download holding every column the exporter declares. A chosen export's title is not a link. Under it the page lists the templates the account has saved for that export, each linking to a download and to its edit page, with a button to make a new template.
+
+A template is a name and an ordered pick of the exporter's columns. A person makes one on its own page, entering a name and choosing a column, or None, for each slot. Its download holds only the picked columns, in the order picked, and the file is named after the template. On the edit page a person can change the name and the picked columns, or delete the template. A template belongs to the account, so everyone working in that account sees, downloads, edits and deletes the same templates.
+
+Every export file's first row holds the column labels, and each later row holds one of the host's records. The host decides which records an export holds, so an export can be limited to the person's current account. The file is built inside the request and nothing about it is stored.
+
+By default every signed-in person can use every import and export. The host can give any import or export an access rule, which is handed the signed-in person and the current account and allows or refuses them, and which can ask the app's own permission checks, such as a Pundit policy. A person the rule refuses does not see that import or export listed, does not see their past imports of it, and is answered not found on every one of its pages and downloads, including its upload page, an existing import's mapping, preview, run and result pages, and a chosen export's template pages and downloads. The rule is checked again on every request, so a person whose permissions change is let in or kept out from their next page.
+
+The engine owns the screens, the access check, and its own tables of import records, refused rows and export templates, with the uploaded file stored through Active Storage. The host owns what each import and export means: which columns it reads or writes, which column identifies an existing record, and which of the app's records a row is checked against, saved into, or read from. Reach for ixport when a Rails app needs people to load data from spreadsheets or take it out into one, rather than writing a one-off upload or download page per record type.
 
 The engine renders its pages with Keystone UI inside a layout the host chooses, and it runs its access check through a method the host already has.
 
@@ -25,35 +42,51 @@ The engine renders its pages with Keystone UI inside a layout the host chooses,
 
 ixport declares nothing for this local to document. Its surface is split between the other two locals:
 
-- **ixport-install** owns hooking ixport into a host app: adding the gem, running its install generator, mounting the engine, and the configuration that names the host's sign-in check, current person, current account, layout and row limit.
-- **ixport-develop** owns building against ixport: registering a mapped import, writing the importer class behind it, the error raised when a registration names an importer that does not exist, and the engine's pages for listing imports, uploading a file, matching its columns, previewing the result, running the import and showing its result.
+- **ixport-install** owns hooking ixport into a host app: adding the gem, running its install generator, running its update generator to bring an existing install's tables up to date, mounting the engine, and the configuration that names the host's sign-in check, current person, current account, layout, row limit and background runner.
+- **ixport-develop** owns building against ixport: registering a mapped import and the importer class behind it, registering a fixed or chosen export and the exporter class behind it, giving an import or export an access rule, writing the app's own background runner and running an import from it, the error a wallflower task raises when its import fails, the errors raised when a registration names a class that does not exist, and the engine's pages for imports, exports and export templates.
 
 ## How to use it
 
-- The app does not have ixport yet, or you are changing how ixport reaches the app's sign-in, people, accounts, layout or row limit: use **ixport-install**.
-- ixport is already installed and you are adding or changing an import, changing how its rows are matched, checked or saved, linking to the engine's pages, or the app fails to boot naming a missing importer: use **ixport-develop**.
+- The app does not have ixport yet, the app installed ixport before export templates or background runs existed and needs their tables, you are choosing whether imports run inside the request, through wallflower or through the app's own runner, or you are changing how ixport reaches the app's sign-in, people, accounts, layout or row limit: use **ixport-install**.
+- ixport is already installed and you are adding or changing an import or an export, making an export chosen or fixed, changing how an import's rows are matched, checked or saved, changing which records or columns an export holds, limiting an import or export to some people, writing the app's own runner that runs imports in the background, linking to the engine's pages, or the app fails to boot naming a missing importer or exporter: use **ixport-develop**.
 
 ## Conventions
 
 - **Mapped import** — one kind of import the host registers, made of a key, a title shown to people, and the name of its importer class.
-- **Key** — the short name a mapped import is registered and looked up by, held as a symbol.
-- **Title** — the words a person sees for the import on the engine's page.
+- **Fixed export** — one kind of export the host registers, made of a key, a title shown to people, and the name of its exporter class, whose download always holds every declared column. An export is fixed unless the host registers it as chosen.
+- **Chosen export** — an export registered the same way and marked as chosen, which has no download of its own and is downloaded only through the account's templates.
+- **Export template** — a named, ordered pick of a chosen export's columns, saved by one person and shared by everyone in their account.
+- **Picked columns** — the columns a template holds, in the order they were chosen. A slot left on None is dropped, a column picked twice is kept once, and a column the exporter no longer declares is left out of the download.
+- **Key** — the short name an import or export is registered and looked up by, held as a symbol. A fixed export's file is named after its key.
+- **Title** — the words a person sees for the import or export on the engine's pages.
 - **Importer** — a class the host app writes that declares the columns the import reads, each with a key and a label, names the column that identifies an existing record, and supplies the set of the app's records the rows are checked against and saved into.
-- **Match column** — the one declared column whose value decides whether a row is an existing record or a new one.
+- **Exporter** — a class the host app writes that declares the columns the export writes, each with a key and a label, and supplies the set of the app's records the file holds.
+- **Export column** — one column of an export's file, headed by its label and filled from the attribute its key names on each record.
+- **Through** — an export column can be read from a record related to each row's record rather than from the record itself, and it is left blank when that related record is missing.
+- **Match column** — the one declared import column whose value decides whether a row is an existing record or a new one.
 - **Import record** — one upload by one person, stored by the engine with the key of its mapped import, the uploaded file, the mapping once it is saved, and the added and updated counts once it has run.
 - **Headers** — the first row of the uploaded file, read as the names of its columns.
 - **Mapping** — which of the importer's columns each header in the file fills, stored by the header's position in the row. A header left on "Skip" is not stored.
-- **Row limit** — the most data rows an uploaded file may have, not counting the header row. It defaults to 1,000 and the host can change it.
+- **Row limit** — the most data rows an uploaded file may have, not counting the header row. It defaults to 1,000 and the host can change it. It applies only when the host has no background runner, and it does not apply to exports.
+- **Background runner** — what runs an import outside the request: none, wallflower, or a class the host app writes. Having wallflower in the app does not choose it, the host has to.
+- **Wallflower** — a separate gem that runs long tasks in the background and gives each task a page showing its progress. ixport registers its imports with it as a kind of task titled Import.
 - **Row number** — a row's line in the file, counting the header as row 1, so the first data row is row 2.
 - **Add** — a row whose match column value is found in no existing record, so it creates one.
 - **Update** — a row whose match column value is found in an existing record, so it changes that record.
 - **Refused** — a row whose record, new or changed, fails the host model's own validations. Its reason is that model's error messages.
 - **Preview** — the page that shows the add, update and refused counts and lists the update and refused rows by row number. It builds each row's record in memory and saves none of them.
-- **Run** — the step that saves the added and updated records and stores each refused row's number and reason. A run that raises an error saves nothing, and an import that has finished is never run a second time.
-- **Finished** — the state of an import record once it has run.
-- **Result page** — the import's own page, showing the added and updated counts and each refused row as they were stored by the run, so it shows the same thing each time it is opened.
-- **Person** — whoever is signed in and running the import. ixport reaches them through a method the host names, so the host's user model needs no particular name.
-- **Account** — the optional owner the person is working inside, such as a team or company, reached the same way. An app with no accounts runs without one.
-- **Ownership** — an import record belongs to the person and account that uploaded it, and anyone else who opens it, including the same person from another account, is answered not found.
-- **Boot check** — when the app starts, ixport checks that every registered importer class exists and refuses to boot when one does not, so a misspelled class name fails at startup rather than when someone runs the import.
-- An import is registered by the host app, never discovered: an importer class that is not registered does not appear on the page.
+- **Run** — the step that saves the added and updated records one row at a time and stores each refused row's number and reason. A run that raises an error stops there and keeps the rows it already saved, and an import that has finished or failed is never run a second time.
+- **Status** — an import record is Pending until it is run, Running while a background runner has it, and then Finished, or Failed when the run stopped on an error.
+- **Running** — the state of an import record handed to a background runner and not yet finished. Its result page says it is running and shows no counts, its preview sends the person to that result page, and it is never handed to the runner a second time.
+- **Finished** — the state of an import record once it has run to the end.
+- **Failed** — the state of an import record whose run stopped on an error. It keeps the error's message and the added and updated counts reached before it stopped, and its result page shows them.
+- **Refused file** — an upload ixport turns away on the upload page because it is not UTF-8, cannot be read past a line, has no header row, or has too many rows. Nothing is stored for it, unlike a refused row, which belongs to an import that ran.
+- **Past imports** — the person's own import records in the current account, listed newest first under the imports on the engine's front page.
+- **Step** — how far an import record has got: mapping when no mapping is saved, preview once one is, and result once it is running, finished or failed.
+- **Result page** — the import's own page, showing the added and updated counts and each refused row as they were stored by the run, and the error's message when the run failed, so it shows the same thing each time it is opened.
+- **Person** — whoever is signed in and running the import or downloading the export. ixport reaches them through a method the host names, so the host's user model needs no particular name.
+- **Account** — the optional owner the person is working inside, such as a team or company, reached the same way. An app with no accounts runs without one. Both the importer and the exporter are given the person and the account.
+- **Ownership** — an import record belongs to the person and account that uploaded it, and anyone else who opens it, including the same person from another account, is answered not found. An export template belongs to its account, and anyone opening it from another account is answered not found.
+- **Access rule** — an optional check the host gives one import or export when registering it, handed the person and the account and answering whether they may use it. It is checked on top of ownership, so even the person who uploaded an import is answered not found for it once the rule refuses them. An import or export with no rule is open to every signed-in person.
+- **Boot check** — when the app starts, ixport checks that every registered importer and exporter class exists and refuses to boot when one does not, so a misspelled class name fails at startup rather than when someone runs the import or downloads the export.
+- Imports and exports are registered by the host app, never discovered: an importer or exporter class that is not registered does not appear on the page, asking for an export key that is not registered is answered not found, and asking to make a template for an export that is not chosen is answered not found.

data/the_local/agents/ixport-install.md

@@ -1,82 +1,107 @@
 ---
 name: ixport-install
-description: Use to hook ixport into a project — adding the gem, checking Active Storage is set up, running the install generator and its migrations, mounting the engine, and configuring the sign-in check, the current person method, the current account method, the layout and the inline row limit.
+description: Use to hook ixport into a project — adding the gem, checking Active Storage is set up, running the install generator and its migrations, running the update generator on an app that installed an earlier ixport, mounting the engine, and configuring the sign-in check, the current person method, the current account method, the layout, the inline row limit and the background runner.
 tools: Bash, Read, Edit
-scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports the host app registers, each backed by an importer class the host writes
+scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports and the exports the host app registers, each import backed by an importer class and each export by an exporter class the host writes, where an export is either fixed or chosen, and a chosen export lets the account save named templates of picked columns
 ---
 
 This local hooks ixport into a Rails app by following the steps below exactly. It invents no step, setting or file that is not listed here.
 
 ## What ixport is
 
-ixport is a Rails engine that lets a signed-in person upload a CSV file, match its columns to a host app's fields, preview which rows will add, update or be refused, and then run the import to save those records into the host app. Hook it in when a Rails app needs people to load data from spreadsheets.
+ixport is a Rails engine with two pages of work for a signed-in person. On the imports page they upload a CSV file, match its columns to a host app's fields, preview which rows will add, update or be refused, and run the import to save those records into the host app. On the exports page they download a CSV file of the host app's records, either with columns the host app has fixed in advance or from a saved template of columns their account picked. Hook it in when a Rails app needs people to load data from spreadsheets or take its data out as spreadsheets.
 
 ## Interface
 
 - `gem "ixport"` — the Gemfile line that adds ixport to the app. It also brings in Keystone UI, which draws ixport's pages.
-- `bin/rails generate ixport:install` — copies two migrations into `db/migrate/`, one that creates the `ixport_imports` table and one that creates the `ixport_import_refusals` table.
-- `mount Ixport::Engine` — the line in `config/routes.rb` that serves ixport's pages under a path the app chooses.
-- `Ixport.configure` — takes a block that receives the configuration object, where the app sets any of the five settings below.
-- `config.authentication_method` — the name of the controller method run before every ixport page, as a symbol. It defaults to `:authenticate_user!`.
-- `config.current_person_method` — the name of the controller method that returns the signed-in person running imports, as a symbol. It defaults to `:current_user`.
+- `bin/rails generate ixport:install` — copies five migrations into `db/migrate/`: one that creates the `ixport_imports` table, one that creates the `ixport_import_refusals` table, one that creates the `ixport_export_templates` table, one that adds a `wallflower_task_id` column to `ixport_imports`, and one that adds an `error_message` column to `ixport_imports`.
+- `bin/rails generate ixport:update` — copies the three migrations added since ixport first shipped, the one that creates the `ixport_export_templates` table, the one that adds the `wallflower_task_id` column and the one that adds the `error_message` column, into an app that installed an earlier ixport.
+- `mount Ixport::Engine` — the line in `config/routes.rb` that serves ixport's pages under a path the app chooses. The imports page is at that path and the exports page is at that path followed by `/exports`.
+- `Ixport.configure` — takes a block that receives the configuration object, where the app sets any of the six settings below.
+- `config.authentication_method` — the name of the controller method run before every ixport page, imports, exports and export templates alike, as a symbol. It defaults to `:authenticate_user!`.
+- `config.current_person_method` — the name of the controller method that returns the signed-in person running imports and exports and saving export templates, as a symbol. It defaults to `:current_user`.
 - `config.current_account_method` — the name of the controller method that returns the account the person is working in, as a symbol. It defaults to `:current_account`.
 - `config.layout` — the name of the host layout ixport's pages are drawn inside, as a string. It defaults to `"application"`.
-- `config.inline_row_limit` — the most data rows an uploaded file may have, not counting its header row, as an integer. A file with more is refused on the upload page with a message giving the limit and the file's row count. It defaults to `1_000`.
+- `config.inline_row_limit` — the most data rows an uploaded import file may have, not counting its header row, as an integer. A file with more is refused on the upload page with a message giving the limit and the file's row count. It applies only while `config.background_runner` is `nil`, and it does not limit exports. It defaults to `1_000`.
+- `config.background_runner` — what runs an import once the person starts it. It defaults to `nil`, which runs the import inside the request. `:wallflower` runs every import as a wallflower task. The name of a class in the app, as a string, hands every import to that class. With any value other than `nil`, every import runs through the runner whatever its size, and the inline row limit does not apply.
 
 ## How to use it
 
-1. Check that the app has Keystone UI set up: the layout ixport will use has `<%= keystone_theme_attributes %>` in its `<html>` tag. If it does not, stop and hand Keystone UI's setup to the `keystone_ui-install` local before going on, because ixport's pages are drawn with Keystone UI components and its styles.
-2. Check that the app has Active Storage set up, because ixport keeps each uploaded file as an Active Storage attachment. All three must hold:
+1. Check whether the app already has ixport: its `Gemfile` has `gem "ixport"` and `db/migrate/` has a `create_ixport_imports` migration. If it does, go to step 13. If it does not, carry on with step 2.
+2. Check that the app has Keystone UI set up: the layout ixport will use has `<%= keystone_theme_attributes %>` in its `<html>` tag. If it does not, stop and hand Keystone UI's setup to the `keystone_ui-install` local before going on, because ixport's pages are drawn with Keystone UI components and its styles.
+3. Check that the app has Active Storage set up, because ixport keeps each uploaded import file as an Active Storage attachment. All three must hold:
    - `config/application.rb` loads Active Storage, either through `require "rails/all"` or through `require "active_storage/engine"`.
    - `db/schema.rb` has the `active_storage_blobs`, `active_storage_attachments` and `active_storage_variant_records` tables.
    - Each environment file in `config/environments/` sets `config.active_storage.service`, and `config/storage.yml` defines that service.
 
    If any is missing, tell the developer ixport cannot accept uploads without Active Storage and ask before setting it up. Where the tables are missing, setting it up means running `bin/rails active_storage:install` and then `bin/rails db:migrate`. Ask the developer which storage service each environment should use rather than picking one.
-3. Add the gem to the app's `Gemfile`:
+4. Add the gem to the app's `Gemfile`:
    ```ruby
    gem "ixport"
    ```
    Then run `bundle install`.
-4. Run the install generator:
+5. Run the install generator:
    ```bash
    bin/rails generate ixport:install
    ```
-   It writes two files and touches no other:
+   It writes five files and touches no other:
    - `db/migrate/<timestamp>_create_ixport_imports.rb` creates the `ixport_imports` table with a required `key` string, a required polymorphic `person` reference, an optional polymorphic `account` reference, a `mapping` JSON column, a required `status` string defaulting to `"pending"`, required `added_count` and `updated_count` integers defaulting to `0`, and timestamps.
    - `db/migrate/<timestamp>_create_ixport_import_refusals.rb` creates the `ixport_import_refusals` table with a required `import` reference with a foreign key to `ixport_imports`, a required `row_number` integer, a required `reason` text, and timestamps.
-5. Run the migrations:
+   - `db/migrate/<timestamp>_create_ixport_export_templates.rb` creates the `ixport_export_templates` table with a required `key` string, a required `name` string, a required `columns` JSON column, a required polymorphic `person` reference, an optional polymorphic `account` reference, and timestamps.
+   - `db/migrate/<timestamp>_add_wallflower_task_to_ixport_imports.rb` adds an optional `wallflower_task_id` bigint column to `ixport_imports`, with an index. It is copied whether or not the app uses wallflower.
+   - `db/migrate/<timestamp>_add_error_message_to_ixport_imports.rb` adds an optional `error_message` text column to `ixport_imports`, which holds the error an import stopped on when it fails.
+6. Run the migrations:
    ```bash
    bin/rails db:migrate
    ```
-6. Ask the developer which path ixport's pages should live under. There is no default path. Add the mount inside `Rails.application.routes.draw` in `config/routes.rb`, using their answer:
+7. Ask the developer which path ixport's pages should live under. There is no default path. Add the mount inside `Rails.application.routes.draw` in `config/routes.rb`, using their answer:
    ```ruby
    mount Ixport::Engine => "/data_imports"
    ```
-   If the app's routes put signed-in pages inside a block such as `authenticate :user do`, ask the developer whether the mount goes inside it.
-7. Read the app's `ApplicationController` and the controller methods it gets from its sign-in setup. ixport's controllers inherit from the app's `ApplicationController`, so every method named in the next step must be callable from it.
-8. Compare the app's method and layout names with the defaults in the Interface section. Every default matches a Jumpstart app, so an app that uses all of the defaults needs no configuration. Where any name differs, or a method does not exist, ask the developer which method or layout to use rather than picking one:
+   If the app's routes put signed-in pages inside a block such as `authenticate :user do`, ask the developer whether the mount goes inside it. Do not give the mount a name with `as:`. A wallflower task links to the import's result page through the route name `ixport`, so the mount keeps that name. `bin/rails routes` lists the mount with `ixport` in its name column.
+8. Read the app's `ApplicationController` and the controller methods it gets from its sign-in setup. ixport's controllers inherit from the app's `ApplicationController`, so every method named in the next step must be callable from it.
+9. Compare the app's method and layout names with the defaults in the Interface section. Every default matches a Jumpstart app, so an app that uses all of the defaults needs no configuration. Where any name differs, or a method does not exist, ask the developer which method or layout to use rather than picking one:
    - The sign-in check: the method that sends a person who is not signed in to sign in, such as `:authenticate_user!` or `:require_login`.
-   - The current person: the method returning the signed-in person, such as `:current_user` or `:current_member`. An import belongs to this person, and another person who opens it gets a not found response.
-   - The current account: the method returning the team or company the person works inside, such as `:current_account` or `:current_team`. An import belongs to this account too, and the same person who opens it from another account gets a not found response. Ask whether the app has accounts at all. ixport calls the named method on every page, so an app with no accounts still needs a method of that name callable from `ApplicationController`, and it returns `nil`. Ask the developer before adding one.
+   - The current person: the method returning the signed-in person, such as `:current_user` or `:current_member`. An import belongs to this person, and another person who opens it gets a not found response. The imports page lists only the imports this person uploaded. Each export is built for this person when it is downloaded. An export template records this person as the one who saved it. Any access rule the app gives an import or export is handed this person.
+   - The current account: the method returning the team or company the person works inside, such as `:current_account` or `:current_team`. An import belongs to this account too, and the same person who opens it from another account gets a not found response. The imports page lists only the person's imports uploaded in this account. Each export is built for this account when it is downloaded. An export template belongs to this account, and every person in the account can see, download from, edit and delete it, while a person in another account gets a not found response. Any access rule the app gives an import or export is handed this account. Ask whether the app has accounts at all. ixport calls the named method on every ixport page, so an app with no accounts still needs a method of that name callable from `ApplicationController`, and it returns `nil`. In an app whose method returns `nil`, every person shares every export template, and every access rule is handed `nil` as the account. Tell the developer this and ask before adding the method.
    - The layout: the file in `app/views/layouts/` that ixport's pages should be drawn inside.
-   - The inline row limit: ask only if the developer wants a limit other than 1,000 rows.
-9. Write only the settings that differ from the defaults into a new initializer, `config/initializers/ixport.rb`:
-   ```ruby
-   Ixport.configure do |config|
-     config.authentication_method = :require_login
-     config.current_person_method = :current_member
-     config.current_account_method = :current_team
-     config.layout = "back_office"
-     config.inline_row_limit = 5_000
-   end
-   ```
-   The values above are examples, and each line is written only when the developer's answer differs from the default.
-10. Start the app and visit the mounted path while signed in. The page shows the heading "Imports" and the message "No imports are set up yet." Visiting it while signed out sends the browser through the app's own sign-in check.
+10. Ask the developer how imports should run, and do not pick for them. Having wallflower in the `Gemfile` does not choose it. The three answers:
+    - Inside the request: leave `config.background_runner` out. Each import file is limited to the inline row limit, and the person waits on the run page until the import finishes.
+    - As a wallflower task: set `config.background_runner = :wallflower`. Running an import starts a task and takes the person to wallflower's task page, which shows the import's progress, the rows it changed and refused, and a link to the import's result page when it finishes. This needs wallflower set up in the app first:
+      - The app's `Gemfile` has `gem "wallflower", ">= 0.2.0"`. If it does not, ask the developer before adding it.
+      - Wallflower is installed and mounted following its own install local, or its own README where the app has no such local. ixport sends the person to wallflower's task page through the route name `wallflower`, so the mount must keep wallflower's own name and not be given a different one with `as:`. `bin/rails routes` lists the mount with `wallflower` in its name column.
+
+      With `:wallflower` set and wallflower not loaded, the app fails to boot with a `NameError` naming `Wallflower`.
+    - Through the app's own runner: set `config.background_runner` to the name of a class in the app, as a string, such as `"ImportRunner"`. Ask the developer for the name. Writing that class is out of scope for this local and belongs to `ixport-develop`, and the app fails on the first run of an import until the class exists.
+
+    With either background answer, every import runs through the runner whatever its size, and while it runs its page shows the heading "Import running" and the message "This import is running."
+11. Write only the settings that differ from the defaults into a new initializer, `config/initializers/ixport.rb`:
+    ```ruby
+    Ixport.configure do |config|
+      config.authentication_method = :require_login
+      config.current_person_method = :current_member
+      config.current_account_method = :current_team
+      config.layout = "back_office"
+      config.inline_row_limit = 5_000
+      config.background_runner = :wallflower
+    end
+    ```
+    The values above are examples, and each line is written only when the developer's answer differs from the default. Ask about `config.inline_row_limit` only when imports run inside the request and the developer wants a limit other than 1,000 rows per import file, since it has no effect with a background runner.
+12. Start the app and visit the mounted path while signed in. The page shows the heading "Imports" and the message "No imports are set up yet." It shows no "Your imports" list, since that list appears only once the signed-in person has uploaded an import file. Then visit the mounted path followed by `/exports`. The page shows the heading "Exports" and the message "No exports are set up yet." Visiting either while signed out sends the browser through the app's own sign-in check. The install is done, so skip step 13.
+13. For an app that already has ixport, check whether `db/migrate/` has a `create_ixport_export_templates` migration, an `add_wallflower_task_to_ixport_imports` migration and an `add_error_message_to_ixport_imports` migration. If it has all three, the app is up to date and there is nothing to run. If any is missing, run `bundle update ixport`, then run the update generator and the migrations:
+    ```bash
+    bin/rails generate ixport:update
+    bin/rails db:migrate
+    ```
+    Until the `error_message` migration has run, an import that fails cannot record its error and the run raises an error naming the missing column. The generator copies whichever of the three migrations step 5 describes are missing, reports each one already present as identical, and touches no other file. Then read the app's `config/initializers/ixport.rb`, or the defaults if it has none. Where the export templates migration was just added, put the current account part of step 9 to the developer, since export templates are shared by that account. Where `config.background_runner` is not set, put step 10 to the developer. Visit the mounted path followed by `/exports` while signed in and check that the page still loads. Then visit the mounted path itself and check that the imports page loads, with a "Your imports" list of the person's earlier imports when they have any. An earlier import of a kind whose access rule refuses the signed-in person is left off that list. Access rules need no migration and no setting.
 
 ## Conventions
 
-- After installing, check that `bin/rails db:migrate:status` lists both the create-ixport-imports and the create-ixport-import-refusals migrations as up, that `bin/rails routes` lists the mount path, and that the page at that path loads as described in step 10.
-- Run the install generator once. A second run against unchanged copies reports each migration as identical and adds nothing. If a copied migration differs from the one the generator would write, the run stops with an error naming that migration, and copies nothing after it. Do not delete or edit a copied migration once it has been run.
-- When the app renames its sign-in method, its current person or account method, or the layout ixport uses, update the matching line in `config/initializers/ixport.rb` in the same change.
+- After installing or updating, check that `bin/rails db:migrate:status` lists the create-ixport-imports, create-ixport-import-refusals, create-ixport-export-templates, add-wallflower-task-to-ixport-imports and add-error-message-to-ixport-imports migrations as up, that `bin/rails routes` lists the mount path, and that both pages load as described in step 12. With `config.background_runner = :wallflower`, also check that `bin/rails routes` lists ixport's mount under the name `ixport` and wallflower's mount under the name `wallflower`.
+- Run the install generator once, and the update generator only on an app installed with an earlier ixport. A second run of either against unchanged copies reports each migration as identical and adds nothing. If a copied migration differs from the one the generator would write, the run stops with an error naming that migration, and copies nothing after it. Do not delete or edit a copied migration once it has been run.
+- An import that stops on an error is marked failed, keeps the rows it saved before stopping, and reports the error through Rails' error reporter, so it reaches whatever error service the app has subscribed there. ixport needs no setting for this, and an app with no subscriber records the error only on the import's page.
+- When the app renames its sign-in method, its current person or account method, the layout ixport uses, or its own runner class, update the matching line in `config/initializers/ixport.rb` in the same change.
 - A setting given the same value as its default is left out of the initializer. Remove a line when its value goes back to the default.
-- Setting up the imports that appear on the page, and the upload, mapping, preview, run and result pages they lead to, is out of scope for this local and belongs to `ixport-develop`.
+- If the app fails to boot after installing, with an error saying an ixport import or export names a class that does not exist, the app registered an import or export whose class is missing. Fixing it is out of scope for this local and belongs to `ixport-develop`.
+- Every signed-in person who passes the sign-in check can use every import and export until the app gives one an access rule. Ask the developer whether some imports or exports should be limited to some people, and hand that work to `ixport-develop`.
+- Setting up the imports and exports that appear on the two pages, choosing which exports are fixed and which are chosen, giving an import or export an access rule, writing the app's own background runner class, and the upload, mapping, preview, run, result, download and template pages they lead to, is out of scope for this local and belongs to `ixport-develop`.

data/the_local/interface.yml

@@ -1,8 +1,9 @@
-scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports the host app registers, each backed by an importer class the host writes
+scope: CSV import and export — a mounted engine where a signed-in person sees the mapped imports and the exports the host app registers, each import backed by an importer class and each export by an exporter class the host writes, where an export is either fixed or chosen, and a chosen export lets the account save named templates of picked columns
 
 install:
   - gem "ixport"
   - bin/rails generate ixport:install
+  - bin/rails generate ixport:update
   - mount Ixport::Engine
   - Ixport.configure
   - config.authentication_method
@@ -10,6 +11,7 @@ install:
   - config.current_account_method
   - config.layout
   - config.inline_row_limit
+  - config.background_runner
 
 develop:
   - Ixport.register_import
@@ -20,6 +22,10 @@ develop:
   - person
   - account
   - Ixport::MissingImporter
+  - Ixport::MissingExporter
+  - Ixport::ImportFailed
+  - Ixport.run_import
+  - enqueue
   - GET /
   - GET /imports/new
   - POST /imports
@@ -28,6 +34,18 @@ develop:
   - GET /imports/:import_id/preview
   - POST /imports/:import_id/run
   - GET /imports/:id
+  - Ixport.register_export
+  - Ixport::Exporter
+  - column
+  - value
+  - GET /exports
+  - GET /exports/:id
+  - GET /exports/:export_id/templates/new
+  - POST /exports/:export_id/templates
+  - GET /export_templates/:id
+  - GET /export_templates/:id/edit
+  - PATCH /export_templates/:id
+  - DELETE /export_templates/:id
 
 sources:
   - lib/ixport.rb
@@ -41,6 +59,7 @@ sources:
   - app/controllers/ixport/application_controller.rb
   - app/controllers/ixport/imports_controller.rb
   - app/views/ixport/imports/index.html.erb
+  - app/helpers/ixport/imports_helper.rb
   - app/models/ixport/application_record.rb
   - app/models/ixport/import.rb
   - app/controllers/ixport/mappings_controller.rb
@@ -53,3 +72,19 @@ sources:
   - app/controllers/ixport/runs_controller.rb
   - app/views/ixport/imports/show.html.erb
   - lib/generators/ixport/install/templates/create_ixport_import_refusals.rb.erb
+  - lib/ixport/exporter.rb
+  - app/models/ixport/csv_writer.rb
+  - app/controllers/ixport/exports_controller.rb
+  - app/views/ixport/exports/index.html.erb
+  - lib/generators/ixport/update/update_generator.rb
+  - lib/generators/ixport/install/templates/create_ixport_export_templates.rb.erb
+  - app/models/ixport/export_template.rb
+  - app/controllers/ixport/export_templates_controller.rb
+  - app/views/ixport/export_templates/new.html.erb
+  - app/views/ixport/export_templates/edit.html.erb
+  - lib/ixport/background_runner.rb
+  - app/models/ixport/wallflower_runner.rb
+  - app/models/ixport/wallflower_task_runner.rb
+  - app/models/ixport/null_reporter.rb
+  - lib/generators/ixport/install/templates/add_wallflower_task_to_ixport_imports.rb.erb
+  - lib/generators/ixport/install/templates/add_error_message_to_ixport_imports.rb.erb

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ixport
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Tyler Schneider
@@ -91,14 +91,25 @@ files:
 - README.md
 - Rakefile
 - app/controllers/ixport/application_controller.rb
+- app/controllers/ixport/export_templates_controller.rb
+- app/controllers/ixport/exports_controller.rb
 - app/controllers/ixport/imports_controller.rb
 - app/controllers/ixport/mappings_controller.rb
 - app/controllers/ixport/previews_controller.rb
 - app/controllers/ixport/runs_controller.rb
+- app/helpers/ixport/imports_helper.rb
 - app/models/ixport/application_record.rb
+- app/models/ixport/csv_writer.rb
+- app/models/ixport/export_template.rb
 - app/models/ixport/import.rb
 - app/models/ixport/import_refusal.rb
+- app/models/ixport/null_reporter.rb
 - app/models/ixport/row_walk.rb
+- app/models/ixport/wallflower_runner.rb
+- app/models/ixport/wallflower_task_runner.rb
+- app/views/ixport/export_templates/edit.html.erb
+- app/views/ixport/export_templates/new.html.erb
+- app/views/ixport/exports/index.html.erb
 - app/views/ixport/imports/index.html.erb
 - app/views/ixport/imports/new.html.erb
 - app/views/ixport/imports/show.html.erb
@@ -106,11 +117,17 @@ files:
 - app/views/ixport/previews/show.html.erb
 - config/routes.rb
 - lib/generators/ixport/install/install_generator.rb
+- lib/generators/ixport/install/templates/add_error_message_to_ixport_imports.rb.erb
+- lib/generators/ixport/install/templates/add_wallflower_task_to_ixport_imports.rb.erb
+- lib/generators/ixport/install/templates/create_ixport_export_templates.rb.erb
 - lib/generators/ixport/install/templates/create_ixport_import_refusals.rb.erb
 - lib/generators/ixport/install/templates/create_ixport_imports.rb.erb
+- lib/generators/ixport/update/update_generator.rb
 - lib/ixport.rb
+- lib/ixport/background_runner.rb
 - lib/ixport/configuration.rb
 - lib/ixport/engine.rb
+- lib/ixport/exporter.rb
 - lib/ixport/importer.rb
 - lib/ixport/registrations.rb
 - lib/ixport/version.rb