@researai/deepscientist 1.5.16 → 1.6.0

package/src/skills/nature-figure/references/api.md

@@ -0,0 +1,428 @@
+# API Reference — Nature Figure Making
+
+Conventions, constants, and reusable code blocks. Implement in your script or adapt as needed.
+
+---
+
+## Constants
+
+### PALETTE
+
+```python
+PALETTE = {
+    "blue_main":      "#0F4D92",
+    "blue_secondary": "#3775BA",
+    "green_1": "#DDF3DE",
+    "green_2": "#AADCA9",
+    "green_3": "#8BCF8B",
+    "red_1":   "#F6CFCB",
+    "red_2":   "#E9A6A1",
+    "red_strong": "#B64342",
+    "neutral_light": "#CFCECE",
+    "neutral_mid":   "#767676",
+    "neutral_dark":  "#4D4D4D",
+    "neutral_black": "#272727",
+    "gold":   "#FFD700",
+    "teal":   "#42949E",
+    "violet": "#9A4D8E",
+    "magenta":"#EA84DD",
+}
+
+DEFAULT_COLORS = [
+    PALETTE["blue_main"],
+    PALETTE["green_3"],
+    PALETTE["red_strong"],
+    PALETTE["teal"],
+    PALETTE["violet"],
+    PALETTE["neutral_light"],
+]
+
+PALETTE_NMI_PASTEL = {
+    "baseline_dark": "#484878",
+    "baseline_mid":  "#7884B4",
+    "baseline_soft": "#B4C0E4",
+    "ours_tiny":  "#E4E4F0",
+    "ours_base":  "#E4CCD8",
+    "ours_large": "#F0C0CC",
+    "bg_lilac": "#E0E0F0",
+    "bg_aqua":  "#E0F0F0",
+    "bg_peach": "#F0E0D0",
+    "neutral_light": "#D8D8D8",
+    "neutral_mid":   "#A8A8A8",
+    "neutral_dark":  "#606060",
+    "delta_up":   "#2E9E44",
+    "delta_down": "#E53935",
+}
+
+DEFAULT_COLORS_NMI_PASTEL = [
+    PALETTE_NMI_PASTEL["baseline_dark"],
+    PALETTE_NMI_PASTEL["baseline_mid"],
+    PALETTE_NMI_PASTEL["baseline_soft"],
+    PALETTE_NMI_PASTEL["ours_tiny"],
+    PALETTE_NMI_PASTEL["ours_base"],
+    PALETTE_NMI_PASTEL["ours_large"],
+]
+
+PALETTE_NATURE_IMAGING = {
+    "bg": "#000000",
+    "context": "#B8B8B8",
+    "cyan": "#22D7E6",
+    "magenta": "#FF2AD4",
+    "white": "#FFFFFF",
+}
+
+PALETTE_NATURE_MATERIAL = {
+    "aqua": "#77D7D1",
+    "teal": "#33B5A5",
+    "lilac": "#B9A7E8",
+    "violet": "#7C6CCF",
+    "callout_red": "#E53935",
+    "neutral": "#D9D9D9",
+}
+
+PALETTE_NATURE_CLINICAL = {
+    "baseline": "#272727",
+    "week6": "#E28E2C",
+    "week13": "#D24B40",
+    "week26": "#5B8FD6",
+    "year1": "#7BAA5B",
+    "year2": "#C45AD6",
+    "group_band": "#F2E6D9",
+}
+
+PALETTE_NATURE_GENOMICS = {
+    "neutral_light": "#D8D8D8",
+    "neutral_mid": "#8F8F8F",
+    "wave1": "#D9544D",
+    "wave2": "#5B7FCA",
+    "wave3": "#B89BD9",
+    "outline": "#4D4D4D",
+}
+```
+
+Use `DEFAULT_COLORS` when color itself carries explicit semantic meaning (`hero`, `baseline`, `positive variant`).
+Use `DEFAULT_COLORS_NMI_PASTEL` when several compared methods belong to one or two related families and the page
+should feel visually unified.
+
+---
+
+## MANDATORY font + SVG rules (always first, no exceptions)
+
+These three lines are **non-negotiable** and must appear at the top of every script,
+before any figure is created. They guarantee editable text in SVG output:
+
+```python
+plt.rcParams['font.family'] = 'sans-serif'
+plt.rcParams['font.sans-serif'] = ['Arial', 'DejaVu Sans', 'Liberation Sans']
+plt.rcParams['svg.fonttype'] = 'none'   # keeps text as <text> nodes, not paths
+```
+
+**Why `svg.fonttype = 'none'`**: matplotlib's default (`'path'`) converts every
+glyph to a bezier path, making text unselectable, unsearchable, and impossible to
+re-align in Illustrator / Inkscape. With `'none'`, text stays as SVG `<text>` elements
+and font substitution happens at render time.
+
+**Output format**: always save as `.svg` (primary). PNG/PDF are optional secondary
+exports. Never use `.png` alone when the figure contains text that may need adjustment.
+
+---
+
+## apply_publication_style()
+
+```python
+def apply_publication_style(font_size=16, axes_linewidth=2.5, use_tex=False):
+    """Apply Nature-style rcParams. Call once before creating any figures."""
+    # ── MANDATORY: editable SVG text ──────────────────────────────────────────
+    plt.rcParams['font.family'] = 'sans-serif'
+    plt.rcParams['font.sans-serif'] = ['Arial', 'DejaVu Sans', 'Liberation Sans']
+    plt.rcParams['svg.fonttype'] = 'none'
+    # ── Layout & style ────────────────────────────────────────────────────────
+    plt.rcParams['font.size'] = font_size
+    plt.rcParams['axes.spines.right'] = False
+    plt.rcParams['axes.spines.top'] = False
+    plt.rcParams['axes.linewidth'] = axes_linewidth
+    plt.rcParams['legend.frameon'] = False
+    if use_tex:
+        plt.rcParams['text.usetex'] = True
+```
+
+**Presets:**
+- Large bar panels: `apply_publication_style(font_size=24, axes_linewidth=3)`
+- Compact figures: `apply_publication_style(font_size=15, axes_linewidth=2)`
+- Dense journal-width multi-panels: `apply_publication_style(font_size=8, axes_linewidth=1)`
+- LaTeX labels: `apply_publication_style(use_tex=True)`
+
+---
+
+## is_dark(hex_color, threshold=128)
+
+```python
+def is_dark(hex_color, threshold=128):
+    """Return True if hex color is dark (use white text on it)."""
+    c = hex_color.lstrip('#')
+    r, g, b = int(c[0:2], 16), int(c[2:4], 16), int(c[4:6], 16)
+    return (0.299*r + 0.587*g + 0.114*b) < threshold
+```
+
+---
+
+## add_panel_label(ax, label, ...)
+
+```python
+def add_panel_label(ax, label, x=-0.06, y=1.02, fontsize=14,
+                    color='black', fontweight='bold'):
+    """Place a Nature-style panel label near the top-left edge."""
+    ax.text(
+        x, y, label,
+        transform=ax.transAxes,
+        fontsize=fontsize,
+        fontweight=fontweight,
+        color=color,
+        ha='left',
+        va='bottom',
+    )
+```
+
+For dark image plates, move the label inside the panel and switch to white:
+`add_panel_label(ax, 'a', x=0.01, y=0.98, color='white')`
+
+---
+
+## style_dark_image_ax(ax, ...)
+
+```python
+def style_dark_image_ax(ax, facecolor='black'):
+    """Prepare an axes for microscopy / rendering plates."""
+    ax.set_facecolor(facecolor)
+    ax.set_xticks([])
+    ax.set_yticks([])
+    for spine in ax.spines.values():
+        spine.set_visible(False)
+    return ax
+```
+
+---
+
+## make_grouped_bar(ax, categories, series, labels, ...)
+
+```python
+def make_grouped_bar(ax, categories, series, labels,
+                     ylabel='Value', colors=None,
+                     annotate=False, bar_width=0.8,
+                     error_kw=None):
+    """
+    Grouped bar chart.
+
+    Parameters
+    ----------
+    ax         : matplotlib Axes
+    categories : list[str]  — x-axis category names (length K)
+    series     : list[array] — one array per group (each length K)
+    labels     : list[str]  — legend label per group
+    ylabel     : str
+    colors     : list[str] | None  — defaults to DEFAULT_COLORS; override with
+                                     DEFAULT_COLORS_NMI_PASTEL for unified-family figures
+    annotate   : bool  — print value above each bar
+    bar_width  : float — total width for all bars in one category
+    error_kw   : dict  — passed to ax.bar as error_kw
+
+    Returns
+    -------
+    list[BarContainer]
+    """
+    import numpy as np
+    if colors is None:
+        colors = DEFAULT_COLORS
+    if error_kw is None:
+        error_kw = {'elinewidth': 2, 'capthick': 2, 'capsize': 10}
+    n_groups = len(series)
+    n_cats = len(categories)
+    w = bar_width / n_groups
+    x = np.arange(n_cats)
+    containers = []
+    for i, (vals, label, color) in enumerate(zip(series, labels, colors)):
+        offset = (i - (n_groups - 1) / 2) * w
+        bars = ax.bar(x + offset, vals, width=w, label=label,
+                      color=color, edgecolor='black', linewidth=1.5,
+                      error_kw=error_kw)
+        containers.append(bars)
+        if annotate:
+            for bar, val in zip(bars, vals):
+                ax.text(bar.get_x() + bar.get_width() / 2,
+                        bar.get_height() + 0.01,
+                        f'{val:.2f}', ha='center', va='bottom', fontsize=10)
+    ax.set_xticks(x)
+    ax.set_xticklabels(categories)
+    ax.set_ylabel(ylabel)
+    ax.legend()
+    return containers
+```
+
+---
+
+## make_trend(ax, x, y_series, labels, ...)
+
+```python
+def make_trend(ax, x, y_series, labels,
+               colors=None, ylabel=None, xlabel=None,
+               show_shadow=False, shadow_alpha=0.15,
+               lw=2.5, marker='o', markersize=8):
+    """
+    Multi-line trend plot.
+
+    Parameters
+    ----------
+    x        : array-like   — shared x values
+    y_series : list[array]  — one 1D array per line
+    labels   : list[str]
+    show_shadow : bool  — fill_between ± std if y_series contains 2D arrays (rows=runs)
+    """
+    import numpy as np
+    if colors is None:
+        colors = DEFAULT_COLORS
+    for y, label, color in zip(y_series, labels, colors):
+        y = np.asarray(y)
+        if y.ndim == 2:
+            mean, std = y.mean(0), y.std(0)
+        else:
+            mean, std = y, None
+        ax.plot(x, mean, color=color, lw=lw, marker=marker,
+                markersize=markersize, label=label)
+        if show_shadow and std is not None:
+            ax.fill_between(x, mean - std, mean + std,
+                            color=color, alpha=shadow_alpha)
+    if ylabel:
+        ax.set_ylabel(ylabel)
+    if xlabel:
+        ax.set_xlabel(xlabel)
+    ax.legend()
+```
+
+---
+
+## make_forest_plot(ax, labels, estimates, ci_low, ci_high, ...)
+
+```python
+def make_forest_plot(ax, labels, estimates, ci_low, ci_high,
+                     colors=None, ref=0.0, xlabel=None, xlim=None,
+                     marker='o', markersize=5, lw=1.5):
+    """
+    Minimal forest plot helper for Nature-style clinical/statistical panels.
+    """
+    import numpy as np
+    y = np.arange(len(labels))[::-1]
+    if colors is None:
+        colors = ['#B64342'] * len(labels)
+    for yi, est, lo, hi, color in zip(y, estimates, ci_low, ci_high, colors):
+        ax.plot([lo, hi], [yi, yi], color=color, lw=lw)
+        ax.plot(est, yi, marker=marker, ms=markersize, color=color)
+    ax.axvline(ref, color='#767676', linestyle='--', linewidth=1.2, alpha=0.8)
+    ax.set_yticks(y)
+    ax.set_yticklabels(labels)
+    if xlabel:
+        ax.set_xlabel(xlabel)
+    if xlim is not None:
+        ax.set_xlim(xlim)
+    ax.spines['right'].set_visible(False)
+    ax.spines['top'].set_visible(False)
+```
+
+Use pale `ax.axhspan(...)` bands behind contiguous label groups when you need the
+clinical-triptych look from `Nature`.
+
+---
+
+## make_heatmap(ax, matrix, ...)
+
+```python
+def make_heatmap(ax, matrix, x_labels=None, y_labels=None,
+                 cmap='magma', cbar_label=None, annotate=False,
+                 fmt='{:.2f}', fontsize=12):
+    """
+    2D heatmap with optional colorbar and cell annotations.
+    """
+    import numpy as np
+    import matplotlib as mpl
+    im = ax.imshow(matrix, cmap=cmap, aspect='auto')
+    if cbar_label:
+        cbar = ax.figure.colorbar(im, ax=ax)
+        cbar.set_label(cbar_label)
+    if x_labels:
+        ax.set_xticks(range(len(x_labels)))
+        ax.set_xticklabels(x_labels, rotation=30, ha='right')
+    if y_labels:
+        ax.set_yticks(range(len(y_labels)))
+        ax.set_yticklabels(y_labels)
+    if annotate:
+        norm = mpl.colors.Normalize(vmin=matrix.min(), vmax=matrix.max())
+        cm_obj = plt.get_cmap(cmap)
+        for (i, j), val in np.ndenumerate(matrix):
+            r, g, b, _ = cm_obj(norm(val))
+            lum = 0.299*r + 0.587*g + 0.114*b
+            color = 'white' if lum < 0.5 else 'black'
+            ax.text(j, i, fmt.format(val), ha='center', va='center',
+                    fontsize=fontsize, color=color)
+    ax.set_frame_on(False)
+```
+
+---
+
+## finalize_figure(fig, out_path, ...)
+
+```python
+def finalize_figure(fig, out_path, formats=None, dpi=300,
+                    pad=2, bbox_inches=None, close=True):
+    """
+    Apply tight_layout and save figure.
+
+    Parameters
+    ----------
+    out_path : str   — path without extension, or with extension
+    formats  : list  — e.g. ['png', 'pdf']. If None, uses extension of out_path.
+    dpi      : int   — 300 standard, 600 for dense bar panels
+    pad      : float — tight_layout pad (2 default, 1 for compact multi-panel)
+    """
+    import os
+    from pathlib import Path
+    fig.tight_layout(pad=pad)
+    base = Path(out_path)
+    os.makedirs(base.parent, exist_ok=True)
+    if formats is None:
+        formats = [base.suffix.lstrip('.') or 'png']
+        base = base.with_suffix('')
+    saved = []
+    for fmt in formats:
+        p = str(base) + f'.{fmt}'
+        kw = {}
+        if bbox_inches is not None:
+            kw['bbox_inches'] = bbox_inches
+        fig.savefig(p, dpi=dpi, **kw)
+        saved.append(p)
+    if close:
+        plt.close(fig)
+    return saved
+```
+
+---
+
+## Validation Rules
+
+- `make_grouped_bar`: `len(categories)` must equal length of each array in `series`.
+- `make_trend`: each array in `y_series` must have same length as `x`.
+- `make_heatmap`: `matrix` must be 2D; `x_labels` length = `matrix.shape[1]`; `y_labels` length = `matrix.shape[0]`.
+- `finalize_figure`: supported formats — `png`, `pdf`, `svg`, `eps`, `jpg`, `tif`.
+
+---
+
+## Conventions
+
+- Save outputs under `./figures/` (or path given by user); `finalize_figure` creates parent dirs.
+- In headless / batch runs, set non-interactive backend before importing pyplot:
+  ```python
+  import matplotlib
+  matplotlib.use('Agg')
+  import matplotlib.pyplot as plt
+  ```
+- Always `plt.close(fig)` after saving to free memory.
+- For multi-panel figures, prefer one baseline family plus one hero family; reserve green/red for delta cues.
+- When color roles, resolution, or layout are underspecified and would change the figure, confirm with user before finalizing.

package/src/skills/nature-figure/references/backend-selection.md

@@ -0,0 +1,100 @@
+# Backend Selection
+
+At the start of a figure task, ask the user to choose **Python or R** if they have
+not already specified a backend. This is a blocking gate: stop after asking and wait
+for the user's answer. Do not infer Python just because the task involves simulation,
+NumPy-like data, or custom layout, and do not infer R just because the task is biological
+or omics-adjacent.
+
+Use the decision table only in either of these cases:
+
+- the user explicitly asks you to recommend or choose the backend;
+- the user provides an unambiguous language-specific workflow or file, such as an `.R`
+  script, RDS object, Python notebook, or existing Python plotting code.
+
+## Quick decision table
+
+| Recommend R when | Recommend Python when |
+|---|---|
+| The user brings R scripts, RData/RDS, Seurat objects, DESeq2/limma outputs, survival models, or ggplot templates | The data pipeline is already Python, NumPy/Pandas arrays, PyTorch/TensorFlow outputs, image arrays, or simulation output |
+| The target plot is `ggplot2`, `patchwork`, `ComplexHeatmap`, `ggtree`, `circlize`, `survminer`, `maftools`, or Seurat/UMAP-heavy | The target plot needs low-level custom layout, Matplotlib patches, image plates, subplot mosaics, or custom drawing primitives |
+| The user provides an R template collection or an existing R plotting workflow | The user wants a self-contained script with matplotlib/seaborn/statsmodels and no R dependency |
+| Heatmap annotations are biologically rich and multi-layered | Image panels and quantitative panels need tight pixel/axis control |
+
+If either backend can do the job, honor the user's preference. Do not switch
+backends for aesthetics alone.
+
+## Backend exclusivity rule
+
+Backend choice is not just a syntax preference; it defines the graphics engine for
+the entire deliverable. Once Python or R has been selected, use that backend for
+all of the following:
+
+- plotting scripts;
+- mock/simulated data examples that include plotting;
+- preview PNG/TIFF files;
+- SVG/PDF/TIFF exports;
+- visual QA renders and final layout checks.
+
+Do not generate a substitute preview or export with the non-selected backend. For
+example, if the user selected R and `Rscript` is missing, do not use Python/matplotlib
+to approximate the figure. If the user selected Python and `matplotlib` or another
+required Python plotting package is missing, do not use R/ggplot2/ComplexHeatmap to
+approximate the figure. Stop, report the selected-backend blocker, and provide the
+selected-backend script plus install/run instructions or request permission to install
+the selected-backend dependencies.
+
+The non-selected language is allowed only for non-visual utility work, such as
+listing files, checking CSV dimensions, decompressing an archive, or converting a
+data file before the selected backend draws the figure. It must not import plotting
+libraries, open graphics devices, save image/vector files, or decide visual layout.
+
+## Default stacks
+
+### R
+
+- Core plotting: `ggplot2`
+- Multi-panel assembly: `patchwork`
+- Heatmaps: `ComplexHeatmap`, `circlize`
+- Direct labels: `ggrepel`
+- Survival/clinical: `survival`, `survminer`, `forestplot`, `ggplot2`
+- Single-cell/omics: `Seurat`, `SingleCellExperiment`, `ComplexHeatmap`, `ggtree`
+- Export: `svglite`, `grDevices::cairo_pdf`, `ragg`
+
+### Python
+
+- Core plotting: `matplotlib`
+- Statistical plots: `seaborn`
+- Layout: `subplot_mosaic`, `GridSpec`
+- Tables/model output: `pandas`, `numpy`, `statsmodels`
+- Images: `matplotlib.imshow`, `skimage`, `tifffile` when needed
+- Export: `fig.savefig(... .svg/.pdf/.tiff)`, `svg.fonttype='none'`,
+  `pdf.fonttype=42`
+
+## Mixed workflow rule
+
+Use the selected plotting backend for final assembly and all visual output. A mixed
+workflow is reasonable only when the non-selected language performs non-visual data
+preparation and the selected backend assembles the figure. In that case:
+
+1. Export clean source data as CSV/TSV with stable column names.
+2. Assemble the final figure in the selected backend.
+3. Keep the source-data file next to the plotting script.
+4. Do not stitch, preview, QA-render, or export final image/vector outputs from the
+   non-selected backend unless the user explicitly changes the selected backend.
+
+## Recommendation language
+
+Use direct language:
+
+```text
+For this figure I recommend R because the main burden is ComplexHeatmap-style
+omics annotation and patchwork assembly. I will still keep the export contract
+SVG/PDF/TIFF with editable text.
+```
+
+```text
+For this figure I recommend Python because the key panel is a custom image plate
+with quantitative overlays and a subplot_mosaic layout. Matplotlib gives tighter
+control over the raster and vector layers.
+```