open-data-sci 0.1.0__py3-none-any.whl

opendatasci/resources/skills/machine_learning.md

@@ -0,0 +1,68 @@
+# Machine Learning Skill
+
+**Problem Framing**
+- Before writing any code, the prediction task is worth thinking through carefully: what exactly is being predicted, at what point in time, using what information, and with what tolerance for different kinds of errors
+- Label quality often determines the ceiling on model performance more than architecture does — severe imbalance, label noise, and ambiguous labelling criteria are worth surfacing early
+- The data modality (tabular, text, image, time series, graph) and available volume should inform the range of approaches worth considering
+- Leakage — features computed using information that wouldn't be available at prediction time — is the most common source of over-optimistic evaluation results; it's worth tracing feature provenance carefully before trusting any metrics
+
+**Splitting Strategy**
+- The split strategy encodes assumptions about the real-world prediction setting; getting it wrong produces metrics that don't generalise
+- Temporal data requires time-ordered splits to avoid the model seeing the future during training — random shuffling of chronological data is a quiet but serious source of leakage
+- When rows share an entity (user, customer, session, location), entity-aware splits prevent the model from memorising entities it will never see again
+- Stratified splits preserve class distribution across folds and are particularly important when classes are rare
+- The test set should be treated as a one-time evaluation; tuning against it invalidates it as an unbiased estimate
+
+**Feature Engineering & Selection**
+- Preprocessing transformations need to be fit only on training data and applied consistently to every split; fitting on the full dataset before splitting leaks distributional information into evaluation
+- For tabular data, useful transformations include datetime decomposition, lag and rolling features, interaction terms, and outlier treatment — the right choices are domain-dependent
+- High-cardinality categoricals benefit from encoding strategies that don't naively explode dimensionality; the right approach depends on the model family and available data volume
+- Feature magnitude matters for some model families and is irrelevant for others — scaling decisions should match the model's sensitivity
+- More features is not always better: irrelevant or redundant features add noise and variance, can hurt distance-based models, and make models harder to interpret and debug; feature selection (variance-based filtering, correlation-based pruning, importance-based selection) is often worth doing before fitting complex models
+- Training/inference skew — features computed differently at training time versus inference time — is a common source of silent degradation
+
+**Model Selection & Complexity**
+- A minimal baseline establishes the floor: what performance is achievable with no model at all, or with a trivial rule? This makes the value of subsequent complexity concrete and measurable
+- Increasing model complexity should be driven by measured improvement on held-out data, not by a prior assumption that a more powerful model will help — underfitting is a real failure mode too
+- Model selection should follow the problem structure and data characteristics, not a habitual preference for any particular algorithm family — the choice of model is itself a design decision that deserves the same rigour as any other modelling choice
+- For tabular data, a principled progression starts with linear or logistic regression as an interpretable, regularisable baseline; tree-based ensembles (random forests, gradient-boosting variants such as LightGBM, XGBoost, CatBoost, or scikit-learn's HistGradientBoosting) are the natural next step when non-linearity and feature interactions are evident and data volume supports them; neural architectures (MLP, TabNet, FT-Transformer) make sense when data volume is large and the iteration budget supports their tuning overhead — moving up the complexity ladder only when simpler models demonstrably fall short
+- For text and NLP tasks, TF-IDF or bag-of-words features with a linear classifier is a strong, cheap baseline; pre-trained transformer models (BERT-family, smaller distilled variants, domain-specific checkpoints) substantially raise the performance ceiling but carry far higher compute costs — calibrate the choice to the task scale and available resources
+- For time-series and sequential data, statistical methods (ARIMA, ETS, Theta, Prophet) are often sufficient and interpretable when the series is short, labelled features are scarce, or seasonality and trend dominate; ML approaches (tree ensembles with lag features, LSTM, Temporal Fusion Transformer) add value when exogenous signals, many parallel series, or complex non-linear dependencies are present
+- For image and spatial data, pre-trained convolutional or vision-transformer backbones via transfer learning are the standard entry point due to transfer efficiency; simpler feature-based approaches (HOG, colour statistics, patch descriptors) remain viable when labels are very scarce or compute is tightly constrained
+- Model families make different assumptions: linear models assume additive, globally linear effects and are easy to interpret, debug, and regularise; tree-based ensembles capture interactions and non-linearities without explicit feature engineering and are relatively robust to irrelevant features and scale differences; neural networks learn representations end-to-end but require substantially more data, hyperparameter effort, and iteration cycles to realise their potential — these tradeoffs should be reasoned through explicitly for each problem
+- The right level of complexity depends on data volume, feature richness, signal-to-noise ratio, and interpretability requirements; many real problems are solved well by simple models, and complexity beyond what the signal supports hurts rather than helps
+- Ensemble and stacking strategies can narrow the gap between model families and are most effective when combining architecturally distinct model families (e.g., gradient boosting + neural network + linear model) or models trained on different feature sets or data samples — ensembling similar frameworks (e.g., XGBoost, LightGBM, CatBoost) produces highly correlated predictions and therefore little diversity benefit
+
+**Hyperparameter Tuning**
+- The search strategy should match the budget: exhaustive search is only feasible for small spaces; random search and Bayesian optimisation cover large spaces more efficiently
+- The wall-clock cost of a search is the product of configurations evaluated and the cost of each evaluation; on large datasets both factors deserve explicit management — run the search on a carefully constructed representative subsample (stratified by class and any structural variable such as time period or entity) to cut per-evaluation time dramatically while preserving most of the directional signal; techniques like successive halving and Hyperband go further by pruning unpromising configurations early rather than running every trial to completion; always re-train or re-evaluate the winning configuration on the full dataset before committing to it
+- The same cross-validation strategy used for model evaluation should be used during tuning to keep estimates consistent
+- Reporting the distribution of CV scores across configurations, not just the best, gives a clearer picture of sensitivity to hyperparameter choices
+- All stochastic components should have fixed random seeds to ensure results are stable across runs
+
+**Overfitting & Regularisation**
+- The gap between training and validation performance is the primary diagnostic for overfitting; a model that performs well on training data but poorly on held-out data has memorised rather than generalised
+- Overfitting risk increases with model capacity relative to data volume — more parameters, more trees, deeper networks all have higher capacity and require more data or more regularisation to generalise
+- Regularisation techniques (penalising model complexity, limiting depth, adding noise during training, early stopping) are the primary tools for closing the train/val gap; the right form depends on the model family
+- Learning curves (performance as a function of training set size) are a useful diagnostic: poor performance that improves with more data suggests a data problem; a persistent train/val gap that doesn't close suggests a regularisation or capacity problem
+- Underfitting — where even training performance is poor — points in the opposite direction: the model may lack the capacity or the features to capture the signal
+
+**Evaluation & Diagnostics**
+- Aggregate metrics can hide a lot; slicing by relevant subgroups, prediction ranges, or time periods often reveals where a model underperforms in ways the headline number conceals
+- Error analysis — directly examining mispredictions (false positives, false negatives, worst residuals, confused classes) — is often more informative than metrics alone; patterns in where a model fails point directly at what to fix
+- For classification, the decision threshold affects the precision-recall tradeoff and should be chosen deliberately based on the relative cost of false positives versus false negatives, not left at a default
+- Probability calibration matters when predicted scores are used as actual probability estimates rather than just rankings
+- Residual analysis for regression surfaces systematic patterns — heteroscedasticity, non-linearity, outlier influence — that aggregate error metrics don't capture
+- Cross-validation spread (mean ± std across folds) characterises how stable performance is, not just what the best-case number is
+
+**Class Imbalance**
+- Class imbalance affects both what the model learns and how performance is measured; addressing only one side gives a misleading picture
+- Threshold adjustment and class weighting are low-cost interventions that often recover substantial minority-class performance without resampling
+- Resampling techniques change the training distribution and should only ever be applied within the training fold — contaminating validation or test data with synthetic samples invalidates evaluation
+- Extreme imbalance changes the problem framing: precision at a given recall threshold or anomaly detection approaches may be more appropriate than standard classification evaluation
+
+**Interpretability**
+- Understanding what a model has learned is distinct from understanding why it performs well; feature importance methods address the former
+- Global importances summarise the model's overall behaviour across the dataset; local explanations explain individual predictions — both are useful but answer different questions
+- Impurity-based importance measures are known to favour high-cardinality features and can be misleading; permutation-based importance and gradient-based attribution methods are generally more reliable
+- Interpretability requirements should inform model choice early; post-hoc approximations of black-box models have their own failure modes

opendatasci/resources/skills/quantitative_analysis.md

@@ -0,0 +1,45 @@
+# Quantitative Analysis Skill
+
+**Problem Formulation**
+- Translating a real-world question into a precise mathematical statement is the first and often most consequential step; ambiguity in the objective propagates into every downstream modelling choice
+- Distinguishing between estimation problems (what is the value of some unknown quantity?), prediction problems (what will happen?), and optimisation problems (what should we do?) shapes the entire analytical approach
+- Constraints and feasibility requirements are as important as the objective; an optimal solution that violates real-world constraints is not a solution
+
+**Mathematical & Statistical Foundations**
+- Results derived from first principles are more defensible than results produced by black-box procedures; being able to trace a conclusion back to its assumptions is essential for knowing when to trust it
+- Linearity assumptions are convenient but often wrong; understanding where a linear approximation holds and where it breaks down is a core skill
+- Probability distributions carry specific assumptions about the data-generating process; selecting a distribution because it fits historical data is not the same as selecting one because it reflects the underlying mechanism
+- Stationarity assumptions underlie much of classical time series analysis; checking for non-stationarity (unit roots, structural breaks, regime changes) before applying methods that assume it prevents misleading inference
+- Heavy tails and extreme events are often the quantities that matter most in risk-sensitive applications; thin-tail distributional assumptions systematically underestimate tail risk
+
+**Time Series & Signal Analysis**
+- Autocorrelation structure (ACF/PACF plots, Ljung-Box tests) should inform model choice before fitting; ignoring it leads to invalid standard errors and spurious relationships
+- Spurious regression between integrated time series is a well-documented failure mode; cointegration analysis is the appropriate tool for modelling long-run relationships between non-stationary series
+- Seasonality, trend, and irregular components are often better handled explicitly than absorbed into a single model; decomposition clarifies what each component contributes
+- Volatility clustering — the empirical regularity that large moves tend to follow large moves — is a persistent feature of financial and economic time series that standard models ignore; GARCH-family models are the standard treatment
+- Choosing the forecast horizon deliberately matters: the right model for one-step-ahead forecasting is often not the right model for long-horizon forecasting
+
+**Risk & Uncertainty Quantification**
+- Point estimates without uncertainty bounds are incomplete; the width of the confidence or credible interval is often more decision-relevant than the point itself
+- Scenario analysis and stress testing complement statistical risk measures by exploring tail outcomes that may not be well-represented in historical data
+- Model risk — the risk that the model itself is wrong — is a distinct and often underappreciated source of uncertainty; comparing results across plausible alternative models is a useful guard
+- Tail risk measures (VaR, CVaR/Expected Shortfall) answer different questions: VaR describes a threshold, CVaR describes what to expect when that threshold is breached — for many purposes CVaR is the more informative measure
+- Monte Carlo simulation provides a flexible framework for propagating uncertainty through complex models; the quality of the output depends entirely on the quality of the input distribution assumptions
+
+**Optimisation**
+- Many quantitative problems can be cast as optimisation; recognising the structure (convex vs. non-convex, constrained vs. unconstrained, continuous vs. integer) determines what solvers are applicable and what guarantees are available
+- Convex problems have the substantial advantage that local optima are global optima; non-convex problems may require heuristics, multiple starting points, or relaxations
+- Numerical stability matters: poorly conditioned problems can produce results that look precise but are sensitive to small perturbations in inputs or solver tolerances
+- In practice, regularisation in optimisation and regularisation in statistics are the same idea expressed in different languages — both bias a solution toward simpler structure in exchange for reduced variance
+
+**Backtesting & Empirical Validation**
+- Backtesting on historical data is a necessary but insufficient validation; without controls for look-ahead bias, survivorship bias, and overfitting, backtest results are unreliable guides to out-of-sample performance
+- Walk-forward validation — fitting on a rolling training window and evaluating on a subsequent out-of-sample period — better mimics the real operational setting than a single historical simulation
+- Multiple testing inflates apparent strategy performance; the more configurations that are tried on the same historical period, the more likely the best-performing one succeeds by chance
+- Transaction costs, slippage, and capacity constraints routinely close the gap between theoretical and realised performance; a backtest that ignores them is optimistic by construction
+
+**Communicating Quantitative Results**
+- Numerical precision in outputs should match the precision of the inputs; reporting eight decimal places from a model estimated on noisy data implies false certainty
+- Assumptions deserve explicit documentation: a result is only as valid as the assumptions that support it, and readers need to be able to evaluate those assumptions for themselves
+- Sensitivity analysis — showing how conclusions change as key inputs or assumptions vary — is often more valuable than a single point result, particularly when inputs are uncertain or contested
+- The practical significance of a quantitative result (does the difference matter for the decision at hand?) is distinct from its statistical significance and should always be addressed

opendatasci/sandbox/__init__.py

@@ -0,0 +1,14 @@
+from opendatasci.sandbox.base import (
+    BaseSandbox,
+    BaseSandboxFactory,
+    SandboxExecResult,
+)
+from opendatasci.sandbox.srt import SRTSandbox, SRTSandboxFactory
+
+__all__ = [
+    "BaseSandboxFactory",
+    "SandboxExecResult",
+    "BaseSandbox",
+    "SRTSandbox",
+    "SRTSandboxFactory",
+]

opendatasci/sandbox/_runner.py

@@ -0,0 +1,114 @@
+"""Injected into the SRT sandbox session directory and executed as a subprocess.
+
+Reads user code from an env-var, executes it inside a persistent namespace,
+and emits a single JSON payload to stdout.
+"""
+
+import base64
+import io
+import json
+import os
+import pickle
+import sys
+import traceback
+from pathlib import Path
+
+try:
+    import pandas as pd
+except ImportError:
+    pd = None  # type: ignore[assignment]
+
+STATE_PATH = os.environ.get("OPENDATASCI_STATE_PATH", "/tmp/opendatasci_state.pkl")
+WORKSPACE = os.environ.get("OPENDATASCI_WORKSPACE", "/tmp/opendatasci_workspace")
+RESULTS_KEY = "__opendatasci_results__"
+code = base64.b64decode(os.environ["OPENDATASCI_CODE_B64"]).decode("utf-8")
+
+# Trust note: state.pkl is a per-session, sandbox-private temp file written only
+# by this runner, so unpickling it is safe today. If state ever becomes shared
+# or persisted across sessions (e.g. a microservice port), this load() becomes
+# an arbitrary-code-execution vector and must be replaced with a safe format.
+try:
+    with open(STATE_PATH, "rb") as fh:
+        namespace = pickle.load(fh)
+except FileNotFoundError:
+    namespace = {}
+
+workspacedir = Path(WORKSPACE)
+opendatasci_directory = workspacedir / ".opendatasci"
+opendatasci_directory.mkdir(parents=True, exist_ok=True)
+os.chdir(str(workspacedir))
+
+saved_results = namespace.pop(RESULTS_KEY, {})
+
+
+def save_result(name: str, value: object) -> None:
+    saved_results[name] = value
+
+
+namespace.update(
+    {
+        "workspacedir": workspacedir,
+        "opendatasci_directory": opendatasci_directory,
+        "save_result": save_result,
+    }
+)
+
+skip_keys = {"workspacedir", "opendatasci_directory", "save_result", "__builtins__"}
+captured = io.StringIO()
+
+original_stdout = sys.stdout
+sys.stdout = captured
+sys.stdin = io.StringIO("")
+try:
+    exec(compile(code, "<opendatasci>", "exec"), namespace)  # noqa: S102
+    output_value = namespace.pop("result", None)
+
+    clean_ns = {}
+    var_info = {}
+    dropped = []
+
+    for key, value in namespace.items():
+        if key.startswith("_") or key in skip_keys:
+            continue
+
+        if pd is not None and isinstance(value, pd.DataFrame):
+            description = f"DataFrame {value.shape}"
+        elif isinstance(value, (list, dict)):
+            description = f"{type(value).__name__} (len={len(value)})"
+        else:
+            description = type(value).__name__
+
+        try:
+            pickle.dumps(value)
+        except Exception:
+            dropped.append(key)
+            continue
+
+        clean_ns[key] = value
+        var_info[key] = description
+
+    clean_ns[RESULTS_KEY] = saved_results
+    with open(STATE_PATH, "wb") as fh_out:
+        pickle.dump(clean_ns, fh_out)
+
+    payload = {
+        "success": True,
+        "stdout": captured.getvalue(),
+        "result": repr(output_value) if output_value is not None else None,
+        "var_info": var_info,
+        "saved_results": {k: repr(v) for k, v in saved_results.items()},
+        "dropped_vars": dropped,
+    }
+except Exception as exc:
+    payload = {
+        "success": False,
+        "stdout": captured.getvalue(),
+        "error": f"{type(exc).__name__}: {exc}\n{traceback.format_exc()}",
+        "var_info": {},
+        "saved_results": {},
+        "dropped_vars": [],
+    }
+finally:
+    sys.stdout = original_stdout
+
+print(json.dumps(payload))

opendatasci/sandbox/base.py

@@ -0,0 +1,170 @@
+"""Abstract sandbox interface for Python and TUI code execution."""
+
+import re
+import shlex
+from abc import ABC, abstractmethod
+from contextlib import AbstractAsyncContextManager
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+
+@dataclass
+class SandboxExecResult:
+    """Result of a single Python or TUI execution in the sandbox."""
+
+    success: bool
+    output: Any = None
+    stdout: str = ""
+    error: str | None = None
+    code: str = ""
+
+
+ALLOWED_CLI_COMMANDS: frozenset[str] = frozenset(
+    {
+        # Directory listing & navigation
+        "ls",
+        "dir",
+        "pwd",
+        "tree",
+        # File viewing
+        "cat",
+        "head",
+        "tail",
+        "file",
+        "stat",
+        # Text search & discovery
+        "grep",
+        "find",
+        "which",
+        # Text processing (read-oriented)
+        "cut",
+        "sort",
+        "uniq",
+        "wc",
+        "awk",
+        "sed",
+        "tr",
+        "strings",
+        # File comparison & checksums
+        "diff",
+        "cmp",
+        "md5sum",
+        "sha256sum",
+        "shasum",
+        # Structured data / binary inspection
+        "jq",
+        "xxd",
+        "od",
+        # General output & info
+        "echo",
+        "printf",
+        "date",
+        "uname",
+        "printenv",
+        "env",
+        # Archive inspection (listing only)
+        "unzip",
+        "tar",
+        "zip",
+    }
+)
+
+_FORBIDDEN_CLI_OPERATORS: tuple[str, ...] = ("||", ";", "`", "$(", ">", "<")
+_ALLOWED_CLI_SHELL_OPERATORS: frozenset[str] = frozenset({"|", "&&"})
+
+
+def validate_cli_command(command: str) -> str | None:
+    """Validate *command* against the TUI allowlist.
+
+    Returns an error string describing the first violation found, or ``None``
+    if the command is valid. This function is pure (no side effects) and can
+    be called from tests directly.
+    """
+    command = command.strip()
+    if not command:
+        return "Empty command."
+
+    for op in _FORBIDDEN_CLI_OPERATORS:
+        if op in command:
+            return f"Shell operator '{op}' is not allowed."
+
+    use_shell = any(op in command for op in _ALLOWED_CLI_SHELL_OPERATORS)
+    segments = re.split(r"&&|\|(?!\|)", command) if use_shell else [command]
+
+    for seg in segments:
+        seg = seg.strip()
+        if not seg:
+            return "Empty pipeline segment."
+        try:
+            parts = shlex.split(seg)
+        except ValueError as exc:
+            return f"Invalid command syntax: {exc}"
+        cmd_name = Path(parts[0]).name.lower()
+        if cmd_name not in ALLOWED_CLI_COMMANDS:
+            sample = sorted(ALLOWED_CLI_COMMANDS)
+            return (
+                f"Command '{cmd_name}' is not allowed. "
+                f"Permitted commands include: {', '.join(sample)}, … "
+                f"({len(ALLOWED_CLI_COMMANDS)} total)"
+            )
+
+    return None
+
+
+class BaseSandbox(ABC):
+    """Stateful code execution sandbox scoped to a single agent session.
+
+    Responsible for running Python code and TUI commands, capturing output,
+    and preserving state (variables, results) across turns within the same
+    conversation.
+    """
+
+    @abstractmethod
+    async def execute(self, code: str) -> SandboxExecResult:
+        """Execute Python *code* and return the result."""
+
+    @abstractmethod
+    async def execute_cli(self, command: str) -> SandboxExecResult:
+        """Execute a TUI *command* and return the result.
+
+        Implementations must validate *command* against ``validate_cli_command``
+        before running it and return a failed ``SandboxExecResult`` on violation.
+        """
+
+    @abstractmethod
+    def reset(self) -> None:
+        """Clear all session state (history, variables, results)."""
+
+    async def close(self) -> None:
+        """Release external resources held by this sandbox.
+
+        The default implementation is a no-op. Override in sandboxes that
+        manage external processes or connections (e.g. Docker containers).
+        """
+
+
+class BaseSandboxFactory(ABC):
+    """Abstract factory that creates :class:`BaseSandbox` instances via async context managers.
+
+    Callers must always acquire a sandbox through :meth:`create` so that
+    teardown is guaranteed regardless of how the calling code exits::
+
+        async with factory.create(workspace_path=path) as sandbox:
+            result = await sandbox.execute(code)
+        # sandbox.close() has been called here
+    """
+
+    @abstractmethod
+    def create(
+        self, workspace_path: Path | None = None
+    ) -> AbstractAsyncContextManager["BaseSandbox"]:
+        """Return an async context manager that yields a fresh :class:`BaseSandbox`.
+
+        The sandbox is closed automatically when the context manager exits,
+        whether by normal return, exception, or cancellation.
+
+        Args:
+            workspace_path: Optional filesystem root that the sandbox should
+                treat as its working directory.
+        """