gratinglab 0.1.0__py3-none-any.whl

gratinglab/metrology/wiki/tip-correction.md

@@ -0,0 +1,101 @@
+# Tip correction
+
+An AFM image is not the surface. It is the surface **dilated** by the tip: at
+every pixel the tip descends until something touches, and the height recorded is
+where the *apex* stopped — which is above the true surface everywhere the contact
+happened on the tip's flank instead of its point.
+
+The correction (Villarrubia 1997) inverts that where the inversion exists, and
+says so where it does not. It is **off by default**, because it changes measured
+numbers — groove depth especially — and a corrected depth and an uncorrected
+depth are different measurements that must not be filed under one name.
+
+## The tip model
+
+A cone with a spherical cap:
+
+| Setting | Means |
+|---|---|
+| `TIP_RADIUS_NM` | The **apex radius**. A probe sold as "2 nm wide" is radius 1. |
+| `TIP_HALF_ANGLE_DEG` | The cone half-angle, measured **from the tip axis**. 18° is typical. |
+| `TIP_CORRECTION` | `'none'` (default) or `'erosion'`. |
+
+The half-angle is the one people misread. Measured from the axis, an 18° tip has
+a flank rising at **72°** from the surface — which clears a 70.5° anti-blaze
+facet by 1.5°, and would not clear a 75° one at all. That near-parallelism, not
+the nanometre apex, is what limits blazed-grating metrology.
+
+Set them in `config.py`, or pass an `AnalysisSettings` with the fields set.
+There is no control in the window yet.
+
+## What you get back, and what you do not
+
+Erosion returns the **least upper bound** on the true surface consistent with the
+image and the tip. That is two statements:
+
+- Wherever the **apex** made contact, the bound *is* the surface. Recovery there
+  is exact — machine precision on clean data.
+- Wherever it did not, the tip physically could not reach, and **no algorithm
+  recovers what is there**. The trough wedge between two facets is the standard
+  case: a sphere of radius *R* stands off a corner by a distance that grows with
+  *R*, and that depth is simply not in the data.
+
+So the number to read is not the corrected profile alone but the **certain
+fraction** that comes with it — reported in the metrics, in the `.txt` sidecar,
+and in the log line. The uncertain remainder means "at most this high", never
+"this high". A corrected profile without that fraction would manufacture exactly
+the false confidence the correction exists to remove.
+
+On the test fixture — a known sawtooth imaged through a known 20 nm tip — the
+uncorrected surface is wrong by 9.06 nm on average; after erosion, over the
+certain pixels, by 0.18 nm. The residual is the noise floor, not the tip.
+
+## What it found on a real scan
+
+The reason this exists. A TASTE grating scan measures a blaze angle of 27.9°
+from its facet fit and 20.3° from its depth — a 7.6° disagreement that on a
+sharp groove is impossible, since depth and facet angle are locked together.
+Tip rounding was the obvious suspect.
+
+Running the correction **acquits the nominal tip**, in both directions:
+
+| Apex radius | Depth/period after erosion | Implied blaze | Pixels certain |
+|---|---|---|---|
+| uncorrected | 0.3275 | 20.33° | — |
+| 1 nm (nominal) | 0.3275 | 20.33° | 97.7% |
+| 2 nm | 0.3275 | 20.33° | 95.9% |
+| 5 nm | 0.3275 | 20.33° | 81.8% |
+| 10 nm | 0.3274 | 20.32° | 74.8% |
+
+Nothing moves. The measured surface is already reachable by a sharp tip, so
+there is no tip-hidden depth to give back.
+
+Forward, dilating an ideal 27.91°/70.5° groove and asking what a tip would
+report:
+
+| Apex radius | Image depth/period | Implied blaze |
+|---|---|---|
+| sharp truth | 0.4460 | 27.91° |
+| 1 nm (nominal) | 0.4439 | 27.77° |
+| 10 nm | 0.4316 | 27.00° |
+| 40 nm | 0.3909 | 24.40° |
+| **80 nm** | **0.3362** | **20.89°** |
+
+A nominal tip costs 0.14° of the 7.6° gap. Reproducing the measurement takes an
+apex worn to ~80 nm, forty to eighty times its specification.
+
+**So the groove is probably rounded, or the tip was badly worn.** Which one is
+not answerable from the scan: it needs the probe's condition at scan time, a
+re-scan with a fresh tip, or a cross-section.
+
+## When to turn it on
+
+- **A tip you have reason to distrust** — late in its life, or after imaging
+  something hard. Compare the corrected and uncorrected depth; if they differ,
+  the tip was hiding surface.
+- **Sharp features near the tip's own scale.** Trenches, undercuts, anything
+  where the flank angle is close to the sidewall angle.
+
+And when not to: as a matter of routine on well-behaved scans. If the erosion
+returns the image unchanged, it has told you the tip was adequate — which is
+worth knowing once, not on every run.

gratinglab/metrology/workflows.py

@@ -0,0 +1,383 @@
+"""
+Analysis workflow orchestration
+Handles the three main analysis modes: single, multiple, compare
+"""
+import glob
+import os
+from datetime import datetime
+
+from .config import (
+    SINGLE_FILE, FILE_PATTERN, SAMPLES_TO_COMPARE, DATA_DIR, RESULTS_DIR,
+    resolve_path, GGP_SOURCE_FILE,
+)
+from .analyzer import analyze_single_file
+from .io.file_io import save_results_to_file
+from .data.aggregation import group_by_temperature, extract_temperatures_for_output
+from .stats.analysis import (
+    print_comparison_summary, print_pairwise_comparisons,
+    print_temperature_analysis
+)
+from .visualization.statistics import plot_multi_file_comparison
+from .visualization.profiles import plot_sample_profiles_by_temperature
+
+
+def run_single_file_analysis():
+    """
+    Analyze a single AFM file with full diagnostic plots
+    """
+    print("Mode: Single file analysis")
+    result = analyze_single_file(resolve_path(SINGLE_FILE), show_plots=True)
+    
+    if result is not None:
+        save_results_to_file([result])
+    
+    return result
+
+
+def run_multiple_file_analysis():
+    """
+    Analyze multiple files matching a pattern
+    """
+    print("Mode: Multiple file analysis (pattern matching)")
+    # Patterns are matched inside the data directory, not the working directory
+    files = sorted(glob.glob(os.path.join(DATA_DIR, FILE_PATTERN)))
+
+    if len(files) == 0:
+        print(f"No files found in {DATA_DIR} matching pattern: {FILE_PATTERN}")
+        return []
+    
+    print(f"Found {len(files)} files to analyze")
+    
+    # Analyze all files
+    results = []
+    for f in files:
+        result = analyze_single_file(f, show_plots=False)
+        if result is not None:
+            results.append(result)
+    
+    if len(results) == 0:
+        return []
+    
+    # Generate comparison plots
+    plot_multi_file_comparison(results)
+    
+    # Print summary table
+    _print_multiple_file_summary(results)
+    
+    # Save results
+    save_results_to_file(results)
+    
+    return results
+
+
+def run_comparison_analysis():
+    """
+    Compare specific samples with optional temperature grouping
+    """
+    print("Mode: Compare specific samples")
+    print(f"Number of sample entries: {len(SAMPLES_TO_COMPARE)}")
+    
+    # Parse sample information and analyze files
+    results, labels, temperatures = _analyze_comparison_samples()
+    
+    if len(results) == 0:
+        print("No samples were successfully analyzed!")
+        return []
+    
+    # Group by temperature and combine multiple scans
+    grouped_results, grouped_labels, temp_order = group_by_temperature(
+        results, labels, temperatures
+    )
+    
+    # Convert to lists for compatibility
+    combined_results = [grouped_results[key] for key in temp_order]
+    combined_labels = [grouped_labels[key] for key in temp_order]
+    combined_temps = extract_temperatures_for_output(grouped_results, temp_order)
+    
+    # Update labels in results
+    for result, label in zip(combined_results, combined_labels):
+        result['label'] = label
+    
+    # Generate all plots and statistics
+    _generate_comparison_outputs(combined_results, combined_labels, combined_temps,
+                                 grouped_results, grouped_labels, temp_order)
+    
+    return combined_results
+
+
+def run_boundary_profile_export(settings=None, filename=None):
+    """
+    Average the grooves of one scan into a PCGrate .ggp boundary profile.
+
+    Shares the front-end with the blaze-angle workflows - loading, image
+    flattening, groove detection - then diverges: instead of fitting facets, the
+    grooves are averaged, normalised to one period and exported for
+    grating-efficiency modelling.
+
+    Parameters:
+        settings: AnalysisSettings; defaults to config.py, so the CLI behaves as
+            it always has. The GUI passes its own.
+        filename: overrides GGP_SOURCE_FILE, for the GUI which already has a file
+            open.
+
+    The computation lives in boundary.build_boundary_profile so the GUI can
+    preview exactly what this writes.
+    """
+    from .core.processing import load_afm_data
+    from .core.image_flatten import flatten_image
+    from .boundary import build_boundary_profile
+    from gratinglab.io.ggp import write_ggp
+
+    from .io.metrics import write_profile_metrics
+    from .settings import AnalysisSettings
+
+    if settings is None:
+        settings = AnalysisSettings.from_config()
+    filename = resolve_path(filename or GGP_SOURCE_FILE)
+
+    print("Mode: PCGrate boundary profile export")
+    print(f"Source: {filename}")
+
+    data, scan_x_size = load_afm_data(filename,
+                                      default_scan_size=settings.scan_x_size,
+                                      settings=settings)
+
+    # Applied here too, so both outputs treat a scan the same way. Provably a
+    # no-op for the affine methods - the endpoint flattening downstream removes
+    # constant and linear terms again - and verified identical at the written
+    # precision, so the stored .ggp fixture still matches.
+    if settings.image_flatten_method != 'none':
+        data = flatten_image(data, settings.image_flatten_method)
+
+    try:
+        profile = build_boundary_profile(data, scan_x_size, settings)
+    except ValueError as exc:
+        print(f"{exc}")
+        return None
+
+    if profile.metrics.get('tip_correction', 'none') != 'none':
+        print(f"Tip correction: erosion "
+              f"(R = {profile.metrics['tip_radius_nm']:g} nm, "
+              f"half angle = {profile.metrics['tip_half_angle_deg']:g} deg), "
+              f"{100.0 * profile.metrics['tip_certain_fraction']:.1f}% of "
+              f"pixels certain; the rest are upper bounds")
+
+    edge_note = (f" ({profile.n_edge_rejected} rejected: clipped by scan edge)"
+                 if profile.n_edge_rejected else "")
+    print(f"Found {profile.n_grooves} grooves{edge_note}")
+    print(f"Measured period: {profile.period_nm:.2f} nm")
+    print(f"Averaged {profile.n_used} of {profile.n_grooves} grooves")
+
+    stem = os.path.splitext(os.path.basename(filename))[0]
+    ggp_path = os.path.join(RESULTS_DIR, f'averaged_groove_profile_{stem}.ggp')
+    met_path = os.path.join(RESULTS_DIR, f'groove_analysis_metrics_{stem}.txt')
+    write_ggp(ggp_path, t=profile.x_norm, y=profile.y_norm)
+    write_profile_metrics(met_path, profile.metrics)
+
+    print(f"\n  Groove depth: {profile.metrics['groove_depth']:.4f} of period")
+    print(f"  Max sidewall angle: {profile.metrics['max_angle_deg']:.2f}deg")
+    print(f"\n✓ Boundary profile saved to: {ggp_path}")
+    print(f"✓ Profile metrics saved to: {met_path}")
+
+    return {'x': profile.x_norm, 'y': profile.y_norm,
+            'metrics': profile.metrics, 'ggp_path': ggp_path,
+            'profile': profile}
+
+
+def run_icc_report():
+    """
+    Measure how correlated the row-group measurements actually are.
+
+    Diagnostic only - reports numbers, changes nothing. Row-group analysis
+    re-measures the same grooves in each band of a scan, so the measurements are
+    not independent, but the reported SEMs and p-values assume they are. The ICC
+    quantifies that, per scan, and the design effect turns it into the factor by
+    which the current SEMs are understated.
+    """
+    from .stats.icc import (compute_icc, effective_sample_size,
+                            sem_inflation_factor)
+
+    print("Mode: ICC diagnostic (row-group correlation)")
+    print(f"Scans to check: {len(SAMPLES_TO_COMPARE)}\n")
+
+    rows = []
+    for sample_info in SAMPLES_TO_COMPARE:
+        filename, label = sample_info[0], sample_info[1]
+        result = analyze_single_file(resolve_path(filename), show_plots=False)
+        if result is None:
+            print(f"  {label}: analysis produced no measurements, skipping")
+            continue
+
+        groups = result.get('groove_row_groups')
+        if not groups:
+            print(f"  {label}: no row-group labels "
+                  f"(USE_ROW_GROUPS is off?), skipping")
+            continue
+
+        stats = compute_icc(result['all_angles'], groups)
+        stats['n_eff'] = effective_sample_size(
+            stats['n_measurements'], stats['mean_group_size'], stats['icc'])
+        stats['sem_factor'] = sem_inflation_factor(
+            stats['mean_group_size'], stats['icc'])
+        stats['label'] = label
+        stats['file'] = os.path.basename(filename)
+        stats['sem_reported'] = result.get('sem', float('nan'))
+        rows.append(stats)
+
+    if not rows:
+        print("No scans could be assessed.")
+        return []
+
+    report = _format_icc_report(rows)
+    print(report)
+
+    os.makedirs(RESULTS_DIR, exist_ok=True)
+    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+    path = os.path.join(RESULTS_DIR, f'icc_report_{timestamp}.txt')
+    with open(path, 'w', encoding="utf-8") as f:
+        f.write(report)
+    print(f"\n✓ ICC report saved to: {path}")
+
+    return rows
+
+
+def _format_icc_report(rows):
+    """Render the ICC table and its interpretation"""
+    from .stats.icc import interpret_icc
+
+    lines = []
+    lines.append("=" * 100)
+    lines.append("ROW-GROUP CORRELATION (ICC) DIAGNOSTIC")
+    lines.append("=" * 100)
+    lines.append(f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
+    lines.append("")
+    lines.append("Row-group analysis measures the same physical grooves once per")
+    lines.append("horizontal band, so those measurements are not independent. The")
+    lines.append("reported SEM divides by sqrt(N) over all of them, which is only")
+    lines.append("valid at ICC = 0.")
+    lines.append("")
+    lines.append(f"{'Sample':<9}{'Scan file':<34}{'N':>5}{'grp':>5}{'ICC':>8}"
+                 f"{'N_eff':>8}{'SEM x':>7}  interpretation")
+    lines.append("-" * 100)
+
+    for r in rows:
+        lines.append(
+            f"{r['label']:<9}{r['file']:<34}{r['n_measurements']:>5}"
+            f"{r['n_groups']:>5}{r['icc']:>8.3f}{r['n_eff']:>8.1f}"
+            f"{r['sem_factor']:>7.2f}  {r['interpretation']}")
+
+    lines.append("-" * 100)
+
+    iccs = [r['icc'] for r in rows]
+    factors = [r['sem_factor'] for r in rows]
+    lines.append("")
+    lines.append(f"ICC range      : {min(iccs):.3f} to {max(iccs):.3f}   "
+                 f"(median {sorted(iccs)[len(iccs)//2]:.3f})")
+    lines.append(f"SEM inflation  : {min(factors):.2f}x to {max(factors):.2f}x")
+    lines.append("")
+    lines.append("Reading:")
+    lines.append("  ICC    fraction of variance sitting between row groups rather")
+    lines.append("         than within them. 0 = independent, 1 = each group adds")
+    lines.append("         only one measurement's worth of information.")
+    lines.append("  N_eff  measurements the data is actually worth, n / design effect.")
+    lines.append("  SEM x  factor the reported standard error should be multiplied")
+    lines.append("         by. Confidence intervals scale the same way.")
+    lines.append("")
+
+    worst = max(rows, key=lambda r: r['icc'])
+    lines.append(f"Overall: {interpret_icc(max(iccs))} "
+                 f"(driven by {worst['label']} / {worst['file']}).")
+    lines.append("")
+    lines.append("This report changes nothing. It sizes the correction that")
+    lines.append("row-group analysis applies via stats/icc.py.")
+    lines.append("=" * 100)
+    return "\n".join(lines)
+
+
+def _analyze_comparison_samples():
+    """
+    Parse SAMPLES_TO_COMPARE and analyze each file
+    
+    Returns:
+        results: list of analysis results
+        labels: list of sample labels
+        temperatures: list of temperatures (None for master)
+    """
+    results = []
+    labels = []
+    temperatures = []
+    
+    for sample_info in SAMPLES_TO_COMPARE:
+        # Handle both (filename, label) and (filename, label, temperature) formats
+        if len(sample_info) == 2:
+            filename, label = sample_info
+            temperature = None
+        elif len(sample_info) == 3:
+            filename, label, temperature = sample_info
+        else:
+            print(f"Error: Invalid sample format: {sample_info}")
+            continue
+        
+        print(f"\nProcessing: {label} ({filename})")
+        result = analyze_single_file(resolve_path(filename), show_plots=False)
+        
+        if result is not None:
+            results.append(result)
+            labels.append(label)
+            temperatures.append(temperature)
+            # Store metadata in result
+            result['label'] = label
+            result['temperature'] = temperature
+    
+    return results, labels, temperatures
+
+
+def _print_multiple_file_summary(results):
+    """Print summary table for multiple file analysis"""
+    print(f"\n{'='*80}")
+    print("SUMMARY TABLE")
+    print(f"{'='*80}")
+    print(f"{'File':<40} {'N':<5} {'Mean':<10} {'Std':<10} {'Period(nm)':<12}")
+    print(f"{'-'*80}")
+    
+    for r in results:
+        print(f"{os.path.basename(r['filename']):<40} "
+              f"{r['n_grooves']:<5} "
+              f"{r['mean_angle']:<10.2f} "
+              f"{r['std_angle']:<10.2f} "
+              f"{r['period_nm']:<12.2f}")
+
+
+def _generate_comparison_outputs(combined_results, combined_labels, combined_temps,
+                                 grouped_results, grouped_labels, temp_order):
+    """
+    Generate all comparison plots and statistics
+    
+    Parameters:
+        combined_results: list of combined results
+        combined_labels: list of labels
+        combined_temps: list of temperatures
+        grouped_results: dict of results by temperature key
+        grouped_labels: dict of labels by temperature key
+        temp_order: ordered list of temperature keys
+    """
+    # Comparison plots with custom labels
+    plot_multi_file_comparison(combined_results, labels=combined_labels,
+                              temperatures=combined_temps)
+    
+    # AFM profiles organized by temperature
+    plot_sample_profiles_by_temperature(grouped_results, grouped_labels, temp_order)
+    
+    # Print comparison summary table
+    print_comparison_summary(combined_results, combined_labels)
+    
+    # Statistical comparison
+    print_pairwise_comparisons(combined_results)
+    
+    # Temperature analysis if applicable
+    if any(t is not None for t in combined_temps):
+        print_temperature_analysis(combined_results, combined_labels, combined_temps)
+    
+    # Save all results to files
+    save_results_to_file(combined_results, labels=combined_labels,
+                        temperatures=combined_temps)

gratinglab/problem.py

@@ -0,0 +1,79 @@
+"""The physical problem: what the grating *is*, independent of how it is solved.
+
+``Problem`` carries **no solver-specific fields, ever**. That single constraint
+is what makes the comparison harness possible -- every method receives the same
+object, so a disagreement between methods is a disagreement about physics rather
+than about how two codes were configured.
+
+It is also serializable, so a benchmark case is a file rather than a script.
+"""
+
+from __future__ import annotations
+
+from typing import Union
+
+from pydantic import BaseModel, ConfigDict, Field
+
+from .profiles import Blazed, FromProfileData, Lamellar, Sinusoidal
+
+__all__ = ["Problem", "AnyProfile"]
+
+#: Profiles that can round-trip through JSON. A ``Profile`` structurally typed
+#: duck can be passed to a solver directly, but only these serialise.
+AnyProfile = Union[Blazed, Lamellar, Sinusoidal, FromProfileData]
+
+
+class Problem(BaseModel):
+    """A grating, in physical terms.
+
+    >>> from gratinglab.profiles import Blazed
+    >>> p = Problem(period=160.0, profile=Blazed(blaze_angle=30.0))
+    >>> round(p.depth, 2)
+    92.38
+
+    Attributes
+    ----------
+    period
+        Groove spacing in nm (``docs/conventions.md`` §1).
+    profile
+        Groove shape, normalised to the period. Multiply by ``period`` for
+        physical height, or use :attr:`depth`.
+    coating, substrate
+        Material identifiers. Optional for now -- scalar theory needs no optical
+        constants, and the perfect-conductivity reference data it is first
+        compared against needs none either. The materials layer lands next.
+    roughness
+        RMS surface roughness in nm, for the Névot–Croce / Debye–Waller factors.
+    n_grooves
+        Illuminated groove count, for the finite-N interference factor of
+        ISSI eq. (8). ``None`` means the infinite-grating limit, where orders
+        become delta functions.
+    """
+
+    model_config = ConfigDict(frozen=True, extra="forbid")
+
+    period: float = Field(gt=0.0, description="Groove spacing, nm")
+    profile: AnyProfile = Field(discriminator=None)
+    coating: str | None = None
+    substrate: str | None = None
+    roughness: float = Field(default=0.0, ge=0.0, description="RMS roughness, nm")
+    n_grooves: int | None = Field(default=None, gt=0)
+
+    @property
+    def depth(self) -> float:
+        """Peak-to-valley groove depth in nm."""
+        return self.profile.depth * self.period
+
+    def height_nm(self, t) -> "object":
+        """Profile height in nm at normalised position ``t``.
+
+        The one place the normalised profile is converted to physical units.
+        Solvers should call this rather than multiplying by hand.
+        """
+        return self.profile.height(t) * self.period
+
+    def __repr__(self) -> str:  # pragma: no cover - cosmetic
+        return (
+            f"Problem(period={self.period:g}nm, "
+            f"{type(self.profile).__name__}, depth={self.depth:.3g}nm)"
+        )