ZaksPhysicsLibrary 1.4.0__tar.gz → 1.7.0__tar.gz

{zaksphysicslibrary-1.4.0/ZaksPhysicsLibrary.egg-info → zaksphysicslibrary-1.7.0}/PKG-INFO

@@ -1,13 +1,13 @@
 Metadata-Version: 2.4
 Name: ZaksPhysicsLibrary
-Version: 1.4.0
-Summary: Data processing and analysis library for TDT, Oxysoft NIRS, and Terranova EFNMR lab data
+Version: 1.7.0
+Summary: Data processing and analysis library for TDT, Oxysoft NIRS, Terranova EFNMR lab data, and text-field studies
 Author: zakgm2
 License-Expression: MIT
 Project-URL: Homepage, https://github.com/zakgm2/PhysicsLibrary
 Project-URL: Repository, https://github.com/zakgm2/PhysicsLibrary
 Project-URL: Changelog, https://github.com/zakgm2/PhysicsLibrary/blob/main/CHANGELOG.md
-Keywords: physics,neuroscience,fibre-photometry,NIRS,TDT,EFNMR,signal-processing
+Keywords: physics,neuroscience,fibre-photometry,NIRS,TDT,EFNMR,signal-processing,text-analysis,nlp
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
@@ -21,6 +21,9 @@ License-File: LICENSE
 Requires-Dist: numpy
 Requires-Dist: scipy
 Requires-Dist: tdt
+Requires-Dist: pandas
+Requires-Dist: sentence-transformers
+Requires-Dist: statsmodels
 Dynamic: license-file
 
 # PhysicsLibrary
@@ -38,6 +41,8 @@ Data processing and analysis library backing [PhysicsAnalysis](https://github.co
   - Generic **Excel / CSV / TSV / plain text**, with automatic sub-table detection for side-by-side data layouts on one sheet
 - **Processes signals** — bleach correction, denoising, Z-score PETH slicing, FFT with peak annotation, slope/segment analysis
 - **Fits curves** — linear, single/double exponential, exponential rise, Gaussian, sinusoidal, and a photon-entanglement visibility model, all via `scipy.optimize.curve_fit`
+- **Analyses text field studies** — one JSON file per subject with several free-text fields; pick any pair of fields to compare directly (e.g. does the answer to one question track another for the same subject). Word counts, data-quality flagging, sentence-transformers embeddings, an optional delta-vector magnitude between two fields, and an optional paired-similarity metric per pair with a permutation test and a word-count confound check. Domain-agnostic — field names and which pairs to compare are supplied by the caller, nothing is hardcoded to one study
+- **Validates the similarity metric statistically** — Benjamini-Hochberg FDR-corrected permutation-test p-values, Cohen's d effect size, a word-count-controlled OLS regression (statsmodels), a bootstrap confidence interval on the mean, and a leave-one-out sensitivity check, one row per field pair, with docstrings explaining what each statistic means
 
 ---
 
@@ -52,6 +57,8 @@ PhysicsLibrary/
   processing_TDT.py          TDT tank reading, bleach correction, denoising, event markers
   analysis.py                 PETH/Z-score, FFT, slope segments, curve-fit runner
   models.py                    Parametric model functions for curve fitting
+  text_field_study.py           Grouped-text-field study pipeline (embeddings, delta vector, paired similarity)
+  field_study_validation.py      Statistical validation for the paired-similarity metric (permutation test, Cohen's d, regression, bootstrap CI, leave-one-out)
   loaders/
     tdt_loader.py               Wraps processing_TDT into a Dataset
     oxysoft_loader.py            Oxysoft .txt parsing (folder + single-file) into a Dataset
@@ -77,7 +84,8 @@ git+https://github.com/zakgm2/PhysicsLibrary.git
 ### Requirements
 
 - Python 3.10+
-- `numpy`, `scipy`, `tdt` (installed automatically)
+- `numpy`, `scipy`, `tdt`, `pandas`, `sentence-transformers`, `statsmodels` (installed automatically)
+- `sentence-transformers` pulls in `torch`/`transformers` as transitive dependencies — a genuinely heavy install (hundreds of MB) if you only need the signal-processing side; only actually loaded when you call `embed_text_fields`/`run_field_study_pipeline`
 - `openpyxl` — only needed for `.xlsx`/`.xls` files; imported lazily with a clear error if missing when you actually try to load Excel
 
 ---
@@ -125,6 +133,34 @@ result = pl.fit_model_to_segment(x_seg, y_seg, pl.single_exponential_model, p0_f
 result["popt"], result["r2"], result["y_fit"]
 ```
 
+```python
+# Text field study — one JSON file per subject, e.g. P-0001.json. Pick
+# pairs of fields to compare directly; no grouping concept needed.
+fields = pl.peek_fields(folder_path)              # see what fields exist before picking pairs
+df = pl.run_field_study_pipeline(
+    folder_path,
+    text_fields=["q1", "q2", "q3", "q4"],
+    delta_pair=("q1", "q2"),                        # optional: how much did q2 change from q1
+    paired_fields=[("q1", "q3", "pair1")],           # optional: does q1 track q3
+)
+# df has one row per subject: wordcount_<field>, low_quality_<field>, delta_magnitude,
+# sim_<pair>, pvalue_<pair>, effect_size_<pair>, wc_confound_r_<pair>, ...
+```
+
+```python
+# Statistical validation of the paired-similarity metric — one row per pair
+summary = pl.run_validation_pipeline(
+    folder_path,
+    text_fields=["q1", "q3"],
+    paired_fields=[("q1", "q3", "pair1")],
+)
+# summary: p_value, p_value_fdr, cohens_d, wc_coef_a/b + wc_pvalue_a/b,
+# regression_r_squared, ci_lower/ci_upper, n_flagged_loo, flagged_participant_ids
+```
+
+See [FIELD_STUDY_METHODOLOGY.md](FIELD_STUDY_METHODOLOGY.md) for why each statistic in the
+validation step is a sound, standard technique — useful if anyone asks.
+
 See [CHANGELOG.md](CHANGELOG.md) for the version history.
 
 ---
@@ -135,8 +171,10 @@ Everything importable from `PhysicsLibrary` directly:
 
 | Category | Names |
 |----------|-------|
-| Format detection | `choose_file`, `detect_format`, `detect_format_file`, `DataFormat`, `Dataset` |
+| Format detection | `detect_format`, `detect_format_file`, `DataFormat`, `Dataset` |
 | Loading | `load_dataset`, `load_dataset_file`, `load_any_file`, `load_pt2` |
 | TDT processing | `process_tdt_folder`, `validate_tdt_folder`, `get_tdt_struct`, `get_plot_data`, `correct_bleaching`, `denoise_signal`, `get_event_markers` |
 | Analysis | `get_zscore_slice`, `smooth_signal`, `bin_for_heatmap`, `compute_fft_slice`, `annotate_fft_peaks`, `compute_slope_segment`, `fit_model_to_segment` |
 | Curve fit models | `linear_model`, `single_exponential_model`, `exponential_rise_model`, `double_exponential_model`, `gaussian_model`, `sinusoidal_model`, `visibility_model` |
+| Text field study | `run_field_study_pipeline`, `load_field_study_folder`, `peek_fields`, `flag_low_quality`, `embed_text_fields`, `compute_delta_vector`, `compute_paired_similarity`, `permutation_test_similarity`, `wordcount_confound_check` |
+| Field study validation | `run_validation_pipeline`, `build_validation_summary`, `cohens_d`, `benjamini_hochberg`, `wordcount_controlled_regression`, `bootstrap_mean_ci`, `leave_one_out_sensitivity` |

{zaksphysicslibrary-1.4.0 → zaksphysicslibrary-1.7.0}/PhysicsLibrary/__init__.py

@@ -45,6 +45,14 @@ from .analysis import (
     annotate_fft_peaks,
     compute_slope_segment,
     fit_model_to_segment,
+    compute_event_zscore_peth,
+    find_significant_peaks,
+    find_peak_near_events,
+)
+
+from .splice import (
+    splice_keep_inside,
+    splice_cut_out,
 )
 
 from .models import (
@@ -55,4 +63,26 @@ from .models import (
     exponential_rise_model,
     gaussian_model,
     sinusoidal_model,
+)
+
+from .text_field_study import (
+    run_field_study_pipeline,
+    load_field_study_folder,
+    peek_fields,
+    flag_low_quality,
+    embed_text_fields,
+    compute_delta_vector,
+    compute_paired_similarity,
+    permutation_test_similarity,
+    wordcount_confound_check,
+)
+
+from .field_study_validation import (
+    run_validation_pipeline,
+    build_validation_summary,
+    cohens_d,
+    benjamini_hochberg,
+    wordcount_controlled_regression,
+    bootstrap_mean_ci,
+    leave_one_out_sensitivity,
 )

{zaksphysicslibrary-1.4.0 → zaksphysicslibrary-1.7.0}/PhysicsLibrary/analysis.py

@@ -69,6 +69,199 @@ def get_zscore_slice(time_array, signal, center_t, window=None, pre=None, post=N
     return seg_x, (seg_y - mu) / std
 
 
+def compute_event_zscore_peth(time_array, signal, event_times, pre, post, num_bins=300):
+    """
+    Z-score and align every occurrence of one event type into a
+    trial x time matrix, for a stacked-heatmap + trial-average PETH view
+    (GuPPy-style) rather than a single click-triggered PETH.
+
+    Each trial is z-scored independently against its own pre-event
+    baseline (see get_zscore_slice) — that's what makes a trial's
+    response comparable regardless of the signal's absolute level at
+    that point in the recording. Trials are then resampled onto one
+    shared relative-time axis (num_bins points spanning -pre..+post) so
+    they can be stacked into a single matrix despite each trial's raw
+    segment having a slightly different sample count from indexing
+    rounding.
+
+    Parameters
+    ----------
+    time_array : array
+    signal : array
+    event_times : list of float
+        Timestamps (same units as time_array) for every occurrence of
+        the event being analyzed.
+    pre, post : float
+        Seconds before/after each event to include.
+    num_bins : int
+        Number of points each trial is resampled to.
+
+    Returns
+    -------
+    dict with:
+        time_axis : array, shape (num_bins,) — relative time, -pre..+post
+        trial_matrix : array, shape (n_valid_trials, num_bins)
+        trial_event_times : list of the event_times that produced a
+            usable trial (too-short/edge-of-recording events are skipped)
+        mean_trace : array, shape (num_bins,)
+        sem_trace : array, shape (num_bins,) — standard error of the mean
+            across trials, zero if fewer than 2 trials
+    """
+    time_axis = np.linspace(-pre, post, num_bins)
+    rows = []
+    valid_times = []
+    for t in event_times:
+        seg_x, seg_z = get_zscore_slice(time_array, signal, t, pre=pre, post=post)
+        if seg_x is None or len(seg_x) < 2:
+            continue
+        rel_x = seg_x - t
+        rows.append(np.interp(time_axis, rel_x, seg_z))
+        valid_times.append(t)
+
+    if not rows:
+        empty = np.zeros(num_bins)
+        return {
+            "time_axis": time_axis, "trial_matrix": np.zeros((0, num_bins)),
+            "trial_event_times": [], "mean_trace": empty, "sem_trace": empty,
+        }
+
+    trial_matrix = np.array(rows)
+    mean_trace = trial_matrix.mean(axis=0)
+    if trial_matrix.shape[0] > 1:
+        sem_trace = trial_matrix.std(axis=0, ddof=1) / np.sqrt(trial_matrix.shape[0])
+    else:
+        sem_trace = np.zeros(num_bins)
+
+    return {
+        "time_axis": time_axis, "trial_matrix": trial_matrix,
+        "trial_event_times": valid_times, "mean_trace": mean_trace, "sem_trace": sem_trace,
+    }
+
+
+def find_significant_peaks(time_array, signal, z_threshold=2.5, min_distance_sec=1.0,
+                            include_troughs=False):
+    """
+    Auto-detect statistically significant transients directly from the
+    signal, rather than relying on externally-supplied event markers
+    (TDT epocs, manual markers, ...) that may not actually line up with
+    where the neural signal itself is doing something.
+
+    The whole recording is z-scored against its own global mean/std
+    (not a local baseline — this is a single-pass "how unusual is this
+    point relative to the entire recording" measure, not per-event), and
+    scipy.signal.find_peaks picks local maxima at or above z_threshold,
+    at least min_distance_sec apart so a single transient's rising edge
+    doesn't get counted as several peaks.
+
+    Parameters
+    ----------
+    time_array : array
+    signal : array
+        Already-processed signal (e.g. bleach-corrected + smoothed) —
+        this function does no filtering of its own.
+    z_threshold : float
+        Minimum z-score (standard deviations above the recording's own
+        mean) for a peak to count as "statistically significant".
+    min_distance_sec : float
+        Minimum spacing between detected peaks, in seconds.
+    include_troughs : bool
+        Also detect significant negative-going deflections (z <=
+        -z_threshold) — off by default since most fibre-photometry
+        analyses care about excitatory transients specifically.
+
+    Returns
+    -------
+    list of dict, each {"time": float, "z_score": float, "kind": "peak"|"trough"},
+    sorted by time.
+    """
+    fs = 1.0 / np.median(np.diff(time_array))
+    distance = max(1, int(min_distance_sec * fs))
+
+    mu, std = np.mean(signal), np.std(signal)
+    if std < 1e-9:
+        return []
+    z = (signal - mu) / std
+
+    results = []
+    peak_idx, _ = find_peaks(z, height=z_threshold, distance=distance)
+    for i in peak_idx:
+        results.append({"time": float(time_array[i]), "z_score": float(z[i]), "kind": "peak"})
+
+    if include_troughs:
+        trough_idx, _ = find_peaks(-z, height=z_threshold, distance=distance)
+        for i in trough_idx:
+            results.append({"time": float(time_array[i]), "z_score": float(z[i]), "kind": "trough"})
+
+    results.sort(key=lambda r: r["time"])
+    return results
+
+
+def find_peak_near_events(time_array, signal, event_times, pre, post,
+                           z_threshold=2.5, include_troughs=False):
+    """
+    Check whether a statistically significant peak actually shows up near
+    each given event time, rather than assuming the event marker itself
+    marks where the neural signal responds. Works for a single event
+    (event_times of length 1) or many occurrences of the same event type
+    (checking consistency across all of them).
+
+    Each event's window is baselined the same way as get_zscore_slice
+    (pre-event portion), so "significant" means relative to that event's
+    own local baseline, not the whole recording's.
+
+    Parameters
+    ----------
+    time_array : array
+    signal : array
+    event_times : list of float
+    pre, post : float
+        Seconds before/after each event to search within.
+    z_threshold : float
+        Minimum |z-score| within the window for a peak to count as found.
+    include_troughs : bool
+        Also consider negative-going deflections as candidate "peaks",
+        keeping whichever (peak or trough) is more extreme.
+
+    Returns
+    -------
+    list of dict, one per event_time (same order), each:
+        {"event_time": float, "found": bool, "peak_time": float or None,
+         "latency": float or None (peak_time - event_time),
+         "z_score": float or None, "kind": "peak"|"trough"|None}
+    "found" is False when the window was unusable (too close to the
+    recording's edges) or nothing in it reached z_threshold.
+    """
+    results = []
+    for t in event_times:
+        seg_x, seg_z = get_zscore_slice(time_array, signal, t, pre=pre, post=post)
+        if seg_x is None or len(seg_x) == 0:
+            results.append({"event_time": t, "found": False, "peak_time": None,
+                             "latency": None, "z_score": None, "kind": None})
+            continue
+
+        idx_max = int(np.argmax(seg_z))
+        if include_troughs:
+            idx_min = int(np.argmin(seg_z))
+            if abs(seg_z[idx_min]) > seg_z[idx_max]:
+                best_idx, kind = idx_min, "trough"
+            else:
+                best_idx, kind = idx_max, "peak"
+        else:
+            best_idx, kind = idx_max, "peak"
+
+        best_z = float(seg_z[best_idx])
+        found = abs(best_z) >= z_threshold
+        peak_time = float(seg_x[best_idx]) if found else None
+        results.append({
+            "event_time": t, "found": found,
+            "peak_time": peak_time,
+            "latency": (peak_time - t) if found else None,
+            "z_score": best_z if found else None,
+            "kind": kind if found else None,
+        })
+    return results
+
+
 def smooth_signal(data, fs, window_sec=0.5):
     """
     Moving average smoothing filter.