@qe-mcp/ena 0.0.0-stage → 0.1.17

package/skill/ena-analysis/SKILL.md

@@ -0,0 +1,348 @@
+---
+name: ena-analysis
+description: >
+  Run Epistemic Network Analysis (ENA) on tabular coded data using the
+  @qe-mcp/ena connector. Use whenever the user references tabular data
+  with binary presence/absence codes and asks for analysis, exploration,
+  co-occurrence patterns, group comparison, choosing a window size, or
+  relating network structure to a unit-level variable (generalized rotation),
+  or order and sequence — directed ONA / multimodal TMA networks
+  — especially discourse data,
+  behavioral coding, learning analytics, or clinical observation data.
+  Includes a privacy gate for sensitive datasets: raw data can stay entirely
+  on the user's machine.
+---
+
+# ENA Analysis
+
+Epistemic Network Analysis models patterns of co-occurrence between coded
+events (discourse moves, behaviors, features) within conversations, projecting
+each unit of analysis into a low-dimensional space where distances are
+interpretable. It reveals structural differences that summary statistics and
+regression miss.
+
+This guide drives the **ena** MCP connector (`@qe-mcp/ena`). All model
+computation runs locally on the user's machine via WebAssembly — only results
+(node positions, plots, summary statistics) are returned to Claude.
+
+## Privacy gate — always ask first
+
+Raw data stays on the user's machine only if it never enters the
+conversation. The server does the analysis locally from a file path; your job
+is to never pull the rows into the conversation yourself.
+
+Before calling any ENA tool on new data, ask:
+
+> "Is this data sensitive or confidential? If so, give me the file path
+> instead of pasting the data — raw data stays on your machine when you use
+> a file path."
+
+Rules, whatever the answer:
+
+- **Never open the data file with other tools** — no file readers, `cat`,
+  `head`, shells, or code that prints rows. Learn the structure from
+  `ena_profile` or `ena_inspect`, which read the file locally.
+- **Never ask the user to paste data.** If they offer, ask for the path.
+  `data_csv` is for small demo or toy data only.
+- If the user pastes data anyway, don't repeat it back; suggest saving it to
+  a file and using the path next time.
+
+If the data is sensitive (or the user hasn't answered yet):
+
+1. Call `ena_profile` with only `data_path`. It returns column names, unique
+   counts and binary detection, plus the values of columns with 10 or fewer
+   distinct values (e.g. condition names) — no row-level data.
+2. Use the profile to determine parameters (units, conversations, codes).
+3. Fit with `data_path`. Only model results reach the conversation.
+
+`data_path` can be a `.csv` or an `.xlsx` workbook; for `.xlsx` the first
+worksheet is read, as the ENA Web Tool does. Old `.xls` files must be saved
+as `.xlsx` or `.csv` first.
+
+What results still contain — tell the user before fitting if any of it is
+identifying:
+
+- **Unit labels**: the values of the unit columns, often participant names
+  or IDs. Suggest pseudonymizing the file first.
+- **Group and category values** the analysis names, and derived statistics.
+
+Tool errors withhold data values (shown as `[value withheld]`) and give the
+column and row instead; don't try to recover the value by reading the file —
+ask the user to check that row.
+
+If the data is not sensitive, or is toy/demo data, proceed with `ena_inspect`
+directly.
+
+## Results come only from the tools
+
+When the user is expecting ENA, every result they see must come from these
+tools: accumulated networks, unit points, rotations, statistics, tables and
+plots. A result that looks like ENA but was computed in the conversation is
+worse than no result, because the user can't tell it apart from the real
+thing and may publish it.
+
+- **Never compute any part of an ENA analysis yourself**: no co-occurrence
+  counting, SVD, means rotation, projection or group statistics in code (a
+  notebook, Python, R, JavaScript) or by hand.
+- **Never draw your own network plot**, even "for illustration". Plots come
+  back from the tools as an image, an SVG and an interactive HTML file.
+- **Never fill a gap with estimates.** No approximate weights, "typical"
+  positions or numbers inferred from earlier output.
+
+### When a tool fails
+
+1. Read the error. Most name the problem: a missing column, a code that
+   never occurs, too few units, a group that doesn't exist.
+2. Fix the call and retry. Use `ena_profile` / `ena_inspect` to check column
+   names and values; for a degenerate fit, `ena_accumulate` shows whether the
+   networks are empty.
+3. If it still fails, tell the user what failed and why, in plain words,
+   and stop. Offer the next step: different codes, a larger window, more
+   units, or rENA in R or the ENA Web Tool for what this server can't do.
+
+If a plot fails to render but the fit succeeded, the result says so
+(`render_note`); give the user `interactive_plot` or report it. Don't draw a
+replacement.
+
+Every result carries `provenance` (the server and engine versions), and every
+plot names them: the interactive HTML in its footer, the SVG in its `<desc>`.
+Anything without it didn't come from this server.
+
+## When ENA fits the data
+
+- Rows are utterances, events, or observations
+- Columns include binary (0/1) presence/absence codes
+- A column identifies who produced each row (unit of analysis)
+- A column groups rows into episodes or conversations
+- Optionally: two or more conditions/groups to compare
+
+## Workflow
+
+1. **Triage** — `ena_inspect` (or `ena_profile` for sensitive data). Returns
+   `ena_recommended`, suggested codes/units/conversations, and a pre-filled
+   `next_step` call. If `ena_recommended` is true, propose ENA as the primary
+   analysis and briefly explain what it will reveal.
+2. **Window** — if the user has no theoretically motivated window size, call
+   `ena_window` (codes + conversations; no units needed). It estimates the
+   window by Cross-Covariance Decay (= rENA `ena.ccd`) and returns the decay
+   curve. Report the estimate and pass it as `window_size`. Left out, the fit
+   tools use that same estimate (as a new set in the ENA Web Tool does), or 4
+   when the data is too sparse for one; the result reports the window used.
+   `window_size: "Inf"` gives an infinite window (each line connects to every
+   earlier line in its conversation).
+3. **Fit** — `ena_fit` for a single model, or `ena_compare_groups` when two
+   groups exist; see [Choosing a rotation](#choosing-a-rotation) for axes
+   built around a variable. Use `tma_fit` instead when order matters or
+   different kinds of lines need different windows — see
+   [Ordered and transmodal networks](#ordered-and-transmodal-networks-ona--tma). Both return a `model_id` and an inline network plot.
+   With two groups named, the result also has a `stats` block: per dimension,
+   a Welch t-test (Cohen's d) and a Mann-Whitney U test (effect size r).
+   Report these rather than running your own tests; the Mann-Whitney p is a
+   normal approximation, and under a rotation through the groups dim1 is
+   built to separate them.
+   Do not generate your own visualization — the plot comes back as an image.
+4. **Explore** — `ena_plot` with the cached `model_id` renders additional
+   views without refitting:
+   - `all` — all units, with the mean network of every unit
+   - `group` — one group's mean network (`group_value`)
+   - `comparison` — two group networks overlaid (`group1`, `group2`)
+   - `subtraction` — difference network (`group1`, `group2`)
+   - `unit` — a single unit's network (`unit_label`)
+   - Any view takes `x_dim` / `y_dim` (1-based; fits keep up to 6
+     dimensions) to plot other dimensions, e.g. SVD1 against SVD3, and
+     `flip_x` / `flip_y` to mirror an axis — as in the web tool's Plot
+     Settings and Flip X / Flip Y. Flipping changes the picture, not the model.
+   - Display options, matching the web tool's Plot Settings; they change only
+     the drawing:
+     - `min_edge_weight` hides connections below a weight. The result reports
+       `max_edge_weight` on the plotted network's scale (group means run
+       0.1–1), so pick a fraction of it. Use it on your own when a network is
+       too dense to read, and say so.
+     - `hide_unconnected_codes` drops codes left with no drawn connection.
+     - `text_size` (px, default 12): 16–20 for slides or papers.
+     - `colors`: `{ "<group>": "<colour>" }`, as #hex or a CSS colour name.
+     - `confidence_intervals: false` hides the CI boxes;
+       `outlier_intervals: true` draws the outlier boxes (1.5 × IQR).
+
+     Map plain requests onto them: "hide the weak connections" is
+     `min_edge_weight`, "bigger labels" is `text_size`, "make the treatment
+     group orange" is `colors`. For trial-and-error tuning ("a bit less…"),
+     point the user to the `interactive_plot` HTML instead of re-plotting: it
+     has live controls for the same settings, plus edge scale and labels.
+
+   Plots are drawn as the ENA Web Tool's comparison plot: unit points, group
+   means with 95% CI boxes, and group colours in the web tool's order.
+5. **Interpret** — strong connections indicate codes that co-occur within
+   the stanza window; in subtraction plots, edge color/direction shows which
+   group each connection favors. Node positions are fixed across views of the
+   same model, so plots are directly comparable.
+   Report the goodness of fit with the variance. Below 0.9 on a dimension
+   (flagged in the summary), say that points can't be read against nearby
+   nodes on that dimension, as the ENA Web Tool warns.
+6. **Close the loop** — before claiming what a connection means, look at
+   the lines behind it: call `ena_lines` with the `model_id`, the
+   `connection` (as in `connection_names`) and `units` or `group`, adding the
+   text column with `columns` (find it with `ena_profile`). It writes a local
+   viewer and returns its path, the counts and data-row numbers. Give the user
+   the path to open in a browser, and ask what they see. **Never open, read or
+   summarize the viewer or the data file yourself** — it holds the raw lines.
+   If the result's check fails, say so; don't interpret those lines.
+7. **Tables** — when the user wants the numbers (for R, Excel, a paper),
+   call `ena_tables` with the `model_id`. It writes `nodes.csv`, `points.csv`,
+   `line-weights.csv` and `connection-counts.csv` to a new folder next to the
+   data file (the ENA Web Tool's Download Model tables) and returns the paths.
+   Don't read the CSVs back into the conversation — give the user the paths.
+8. **Export** — when the user wants the model in the ENA Web Tool, call
+   `ena_export` with the `model_id`. It writes a project ZIP (the data file
+   plus a set with every setting and the plot to open with), by default next
+   to the data file. Tell the user the path and how to open it: a local
+   session ("Run Offline") → Sets → **Import** → *Folder › set name*. Say that
+   the ZIP holds their data file. If it refuses a model, relay the reason (the
+   web tool has no setting for it) rather than working around it.
+
+## Excluding conversations and ordering lines
+
+- `exclude_conversations` (conversation values joined by `::`) removes those
+  conversations' lines entirely before the model is built.
+- `order_by` sorts the lines by a column before windowing — use it when the
+  file isn't in conversational order (e.g. by a timestamp or turn number).
+  Results' row numbers still refer to the file.
+
+`ena_export` carries both: they arrive as excluded horizons and the window
+Order in the ENA Web Tool, which applies them the same way.
+
+## Simplifying a model
+
+When the user wants fewer codes, or a model looks over-specified, call
+`ena_simplify` with the `model_id`. It runs PRIA (the web tool's Simplify
+Model) with the model's own settings and, if codes can go, refits without
+them and returns the new model. Report which codes were removed and the
+threshold, and continue with the new `model_id`. "No codes can be removed"
+is a result, not a failure: every code is needed at that threshold.
+
+## Masking connections
+
+When the user wants a connection left out of the model — for example one
+that is an artifact of the task or the coding scheme — pass
+`mask_connections` (names as in `connection_names`) to the fit. It is the ENA
+Web Tool's Code Masking Matrix: the connection counts as 0 for every unit,
+and the export carries it. For ordered models `A & B` masks A → B only.
+Say in the write-up which connections were masked and why. `ena_lines`
+can't show lines for a masked connection, as none count toward it.
+
+## Choosing a rotation
+
+The rotation decides what the axes mean. Pick it from the research question:
+
+- **Default** (no `rotation`) — as the ENA Web Tool rotates: with two or more
+  unit columns and two or more values in the first (e.g. `Condition` in
+  `["Condition", "UserName"]`), a GMR through that column's first two groups,
+  with the group tests on them; otherwise SVD. `ena_compare_groups` always
+  rotates this way through its two groups — the same axes as a means rotation.
+- **SVD** (`rotation: "svd"`) — exploratory: the axes capture the most
+  variance in the networks, whatever drives it. Pass it explicitly for
+  grouped units when the question is exploratory.
+- **Means** (`rotation: "mean"` with `group_col`, `group1_value`,
+  `group2_value`) — dimension 1 runs through the two group means.
+- **Generalized** (`rotation: "generalized"` on `ena_fit` or `tma_fit`, = rENA
+  `ena.rotate.by.generalized`) — dimension 1 (GMR1) is built to represent a
+  unit-level variable. Use when the question is how network structure relates
+  to an outcome or attribute ("how do networks differ with confidence gain?"),
+  or for a group comparison that controls for other variables.
+
+- **Projected** (`rotation_from: <model_id>` on `ena_fit` / `tma_fit`) — no
+  rotation of its own: the model is placed in another fitted model's space
+  (the web tool's Use Rotation Matrix From). Use it to compare a new sample,
+  cohort or time point against an existing model, with the same codes and
+  network type. Fit the source first; say which model defines the space.
+
+### Generalized rotation (GMR)
+
+- `x_var`: the target column first, then any covariates. A categorical
+  target puts GMR1 through the group means; a numeric target puts it along
+  the variable. Covariates are controlled for by Lasso, with pairwise
+  interaction terms unless `interactions: false`.
+- `y_var` (optional, same format) builds dimension 2 (GMR2) for a second
+  target; without it, dimension 2 is the next SVD dimension.
+- `select_2_groups: [a, b]` (categorical target) fits GMR1 on those two
+  values only; the other units are still projected. It also colours the plot
+  and runs the group `stats` on those two groups unless `group_col` is given.
+- Target and covariates must be **unit-level** — one value per unit. Use
+  `ena_profile` / `ena_inspect` to find such columns (e.g. pre/post scores,
+  condition, demographics), not columns that change line to line.
+- A unit with a missing value is rejected. Tell the user which units, and
+  offer `units_used` to model the others (the excluded units' lines still
+  count as context) rather than deleting rows.
+- Reading the result: the summary names the axes (`GMR1`, `SVD2` or `GMR2`)
+  and the target. GMR1 explains no more variance than SVD1 would (SVD1 is
+  the maximum-variance axis) — it is chosen for meaning, not variance; say so rather than treating it as a
+  worse fit. Interpret node positions along GMR1 as the connections
+  associated with higher vs lower values of the target. Axis signs are
+  arbitrary (they can differ from rENA in R). With a categorical target, the
+  group test on GMR1 confirms a separation the rotation was built for — like
+  means rotation, it is not an independent test.
+
+## Ordered and transmodal networks (ONA / TMA)
+
+`tma_fit` (and `tma_accumulate` for the accumulation step alone) builds
+**directed** networks: a connection A → B means code A was in the window
+(the *ground*) when code B appeared (the *response*). Use it when:
+
+- **Order matters** — the question is about responses, sequences or
+  what leads to what ("do design proposals follow data checks, or precede
+  them?"). This is Ordered Network Analysis (ONA, = R's `ona` package):
+  `tma_fit` with just units, conversations, codes and `window_size`.
+- **Lines differ in kind** — chat vs logged actions, teacher vs student,
+  speech vs gesture — and each kind should look back a different distance or
+  count with a different weight. This is Transmodal Analysis (TMA, = R's
+  `tma`): the context tensor below.
+- **Windows are in time**, not lines.
+
+Otherwise prefer `ena_fit`: undirected models are simpler to explain.
+
+### Parameters beyond `ena_fit`
+
+- `ordered` (default true). false gives undirected networks from the same
+  tensor — TMA windows without direction.
+- `stanza_modality`: a column whose value (e.g. `chat`, `log`) is the
+  line's mode; `stanza_ground` / `stanza_response`: columns (e.g. `Role`)
+  whose values on the ground line / the responding line select the window.
+- `stanza_overrides`: per-combination `{ path, window, weight }`. `path`
+  lists one value per factor, in the order modality columns, then ground,
+  then response — e.g. with `stanza_modality: "Modality"` and
+  `stanza_ground: ["Role"]`, `{ "path": ["log", "Student"], "window": 1 }`
+  makes students' logged actions look back one line. Combinations not
+  listed use `window_size` and weight 1. `window` may be `"Inf"`.
+- `window_type: "time"` with `time_column` and `time_unit` (secs, mins,
+  hours, days, weeks; default secs) measures windows in time.
+- `horizons`: `{ by, rules }` gives each unit its own horizon of observation
+  instead of the conversations (tma's HOO rules) — e.g.
+  `{ "by": "Modality", "rules": { "chat": ["GroupName"], "log": ["UserName"] } }`:
+  chat is seen by the whole group, a log line only by its author.
+- Rotation (`svd`, `mean`, `generalized`), group stats, `weight_model`,
+  `units_used` and `sphere_norm` work as in `ena_fit`. For a two-group
+  comparison of ordered networks use `tma_fit` with `rotation: "mean"` —
+  `ena_compare_groups` is undirected only.
+
+Before fitting a TMA, confirm the design with the user in words ("chat lines
+look back 4 lines; students' logged actions look back 1") — the tensor is
+easy to get subtly wrong and hard to see from the output. Use `ena_profile`
+or `ena_inspect` to find the modality and role columns and their values.
+
+### Reading directed results
+
+- A model with n codes has n² directed connections, named `"A & B"` for
+  A → B, including self-connections (A → A: a code following itself).
+- Plots draw directed edges as tapered shapes with direction marks, each
+  direction weighted separately. Read the exact weights from
+  `connection_names` and the model results rather than estimating them from
+  the picture.
+- Report direction explicitly ("Data → Design Reasoning is stronger in
+  SecondGame"), not just which codes are connected. In subtraction plots, an
+  edge's colour shows which group has more of that direction.
+
+Each fit/plot response also includes `interactive_plot` — a local HTML file
+path with the interactive visualization the user can open in a browser.
+
+Repeated calls with identical parameters return the cached model instantly,
+so iterating on views is cheap.