learnstat 0.1.0__tar.gz

learnstat-0.1.0/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: learnstat
+Version: 0.1.0
+Summary: An AI-assisted learning companion for statistical analysis with pandas, NumPy, SciPy and Matplotlib.
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pandas>=2.0
+Requires-Dist: numpy>=1.24
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.7; extra == "viz"
+Provides-Extra: stats
+Requires-Dist: scipy>=1.10; extra == "stats"
+Provides-Extra: ai
+Requires-Dist: anthropic>=1.0; extra == "ai"
+Provides-Extra: jupyter
+Requires-Dist: ipython>=8.0; extra == "jupyter"
+Provides-Extra: all
+Requires-Dist: matplotlib>=3.7; extra == "all"
+Requires-Dist: scipy>=1.10; extra == "all"
+Requires-Dist: anthropic>=1.0; extra == "all"
+Requires-Dist: ipython>=8.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: matplotlib>=3.7; extra == "dev"
+Requires-Dist: scipy>=1.10; extra == "dev"
+Requires-Dist: ipython>=8.0; extra == "dev"
+
+# learnstat
+
+An AI-assisted learning companion for statistical analysis in Python.
+
+You write ordinary **pandas / NumPy / SciPy / Matplotlib** code. learnstat:
+
+1. **Understands your dataset.** It profiles the data once and keeps a Markdown memory file of it.
+2. **Suggests better approaches as you write.** For example, it flags a pie chart of 12 categories and suggests a sorted bar chart, explaining why.
+3. **Explains itself** through `/describe`, `/alternatives`, `/learn`, `/memory` and `/fix`.
+4. **Recovers from mechanical errors.** It fixes typos in column names, methods and keywords, plus missing imports and unclosed brackets, and shows you every change.
+5. **Leaves analytical decisions to you.** Data-quality and statistical-method problems are explained but never "fixed" automatically.
+
+It's not a new language. Everything you learn uses the real pandas, SciPy and Matplotlib APIs, so the skills carry over to the rest of the ecosystem.
+
+## Install
+
+```bash
+pip install -e ".[all]"        # pandas, numpy, matplotlib, scipy, anthropic, ipython
+pip install -e .               # core only: profiling, rules, recovery (no AI)
+```
+
+The AI features are optional. With the `anthropic` package and credentials set up (`ANTHROPIC_API_KEY` or `ant auth login`), explanations get tailored to your dataset, Claude reviews code with `/review`, and Claude can propose fixes for errors the rules can't handle. Without them, everything else still works offline. The default model is `claude-opus-5-5`; set `LEARNSTAT_MODEL` to choose another.
+
+## Use it
+
+### Jupyter
+
+```python
+%load_ext learnstat
+import learnstat as ls
+df = ls.read_csv("survey.csv")      # a normal DataFrame, now profiled into .learnstat/survey.md
+```
+
+```python
+df["country"].value_counts().plot.pie()
+# [!] Suggestion #1 (line 1): Pie chart with 12 categories
+#     Issue:   `country` has 12 categories; slices this numerous are hard to compare by angle.
+#     Instead: A horizontal bar chart of the counts, sorted by size.
+#     Why:     Bars share a common baseline ...
+#     How:     counts = df['country'].value_counts() ...
+```
+
+```
+/describe        <- the concept, why it fits this data, when another method is better
+/accept          <- puts the suggested code in a NEW cell; your cell is untouched
+/reject only 3 countries matter here   <- recorded in memory, not suggested again
+```
+
+When a cell raises an error, learnstat diagnoses the failing statement. If it's a mechanical mistake, learnstat corrects it, re-runs only that statement and the ones after it, and prints the diff. A cell with a syntax error gets a corrected copy placed in the next cell for you to review.
+
+### Command line
+
+```bash
+cd examples && python make_survey.py
+learnstat profile survey.csv           # writes .learnstat/survey.md and prints it
+learnstat check analysis.py            # suggestions only, nothing is executed
+learnstat watch analysis.py            # re-checks every time you save the file
+learnstat run analysis.py              # runs with suggestions and guarded recovery
+learnstat repl --data survey.csv       # interactive, with /commands
+```
+
+### Plain Python
+
+```python
+import learnstat as ls
+df = ls.read_csv("survey.csv")
+ls.check("df['income'].mean()")         # suggestions without running
+print(ls.command("/describe"))
+
+s = ls.Session(confirm=lambda q: input(q + " [y/N] ") == "y")
+s.ns["df"] = s.read_csv("survey.csv")
+result = s.run("df.groupBy('group')['score'].median()")
+print(result.final_code)                # df.groupby(...)... your source is never modified
+```
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `/describe [id]` | Explains the latest (or given) suggestion, or the latest error: the concept, the purpose of the function, why it fits this data, and when something else is better |
+| `/alternatives [id]` | Other ways to reach the same goal, with trade-offs |
+| `/learn [topic]` | A short lesson with an exercise on your own data |
+| `/fix` | The last error diagnosis; in interactive mode, a retry of the failed statement |
+| `/memory [show\|note\|decide\|refresh]` | View or update the dataset's persistent profile |
+| `/accept [id]`, `/reject [id] [why]` | Accept or reject a suggestion; the decision is saved in memory |
+| `/suggestions`, `/review`, `/ask`, `/status`, `/help` | |
+
+## Dataset memory
+
+`ls.read_csv` profiles the data and writes two files to `.learnstat/`:
+
+- `survey.md`: columns, types, missing values, likely column meanings (marked as guesses), distributions, relationships, data-quality issues, suggested analyses, assumptions, **your analytical decisions** and notes.
+- `survey.json`: the machine-readable cache. It includes a content fingerprint, so the data is only re-profiled when it actually changes. Decisions and notes carry over when that happens.
+
+## How error recovery stays safe
+
+- Code runs **one top-level statement at a time**. When statement *k* fails, only statement *k* is corrected and retried, so earlier side effects are never repeated.
+- Errors are classified as **syntax, name, attribute, argument, column, data-quality, statistical-method, environment** or other runtime errors.
+- Only mechanical mistakes are corrected automatically, and only to a unique, high-similarity match.
+- Before a fix runs, it must compile. It must not introduce file, shell or network operations, must not change which variables are assigned, must not add data-altering calls (`dropna`, `fillna`, `astype` and so on), and must not change the statistical method.
+- AI-proposed fixes always need your confirmation. So does retrying a loop or block, since that could repeat side effects.
+- Retries are bounded (`max_attempts`, 3 by default). If the same error comes back, recovery stops and gives you a concrete next step.
+- Code that deletes or overwrites files, runs shell commands or opens network connections only runs after you confirm.
+- **This is a policy layer, not a sandbox.** Your code runs in your Python process with your permissions, exactly as if you had run it yourself.
+
+## Project layout
+
+```
+learnstat/
+  profile.py      dataset profiling (no AI)
+  memory.py       Markdown + JSON dataset memory
+  advisor.py      AST-based, data-aware suggestion rules
+  knowledge.py    offline explanations behind /describe, /alternatives, /learn
+  recovery.py     error diagnosis, fix proposal, validation, bounded retry
+  safety.py       detection of destructive side effects
+  llm.py          optional Claude backend
+  session.py      ties it together; read_csv/track/check/run
+  commands.py     slash commands
+  ipython_ext.py  %load_ext learnstat
+  cli.py          learnstat profile|check|watch|run|repl
+```
+
+Add a rule by writing a function `(call, resolver, code) -> Suggestion | None` in `advisor.py`, appending it to `RULES`, and adding its concept to `knowledge.py`.
+
+## Roadmap
+
+- A language-server (LSP) frontend for suggestions while typing in VS Code and JupyterLab. The advisor is already editor-agnostic.
+- More rules: multiple-comparison corrections, chi-square expected counts, normality checks before parametric tests, time-series plotting.
+- Learning-progress tracking, so advice fades as the user masters a concept.
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+pytest
+```

learnstat-0.1.0/README.md

@@ -0,0 +1,136 @@
+# learnstat
+
+An AI-assisted learning companion for statistical analysis in Python.
+
+You write ordinary **pandas / NumPy / SciPy / Matplotlib** code. learnstat:
+
+1. **Understands your dataset.** It profiles the data once and keeps a Markdown memory file of it.
+2. **Suggests better approaches as you write.** For example, it flags a pie chart of 12 categories and suggests a sorted bar chart, explaining why.
+3. **Explains itself** through `/describe`, `/alternatives`, `/learn`, `/memory` and `/fix`.
+4. **Recovers from mechanical errors.** It fixes typos in column names, methods and keywords, plus missing imports and unclosed brackets, and shows you every change.
+5. **Leaves analytical decisions to you.** Data-quality and statistical-method problems are explained but never "fixed" automatically.
+
+It's not a new language. Everything you learn uses the real pandas, SciPy and Matplotlib APIs, so the skills carry over to the rest of the ecosystem.
+
+## Install
+
+```bash
+pip install -e ".[all]"        # pandas, numpy, matplotlib, scipy, anthropic, ipython
+pip install -e .               # core only: profiling, rules, recovery (no AI)
+```
+
+The AI features are optional. With the `anthropic` package and credentials set up (`ANTHROPIC_API_KEY` or `ant auth login`), explanations get tailored to your dataset, Claude reviews code with `/review`, and Claude can propose fixes for errors the rules can't handle. Without them, everything else still works offline. The default model is `claude-opus-5-5`; set `LEARNSTAT_MODEL` to choose another.
+
+## Use it
+
+### Jupyter
+
+```python
+%load_ext learnstat
+import learnstat as ls
+df = ls.read_csv("survey.csv")      # a normal DataFrame, now profiled into .learnstat/survey.md
+```
+
+```python
+df["country"].value_counts().plot.pie()
+# [!] Suggestion #1 (line 1): Pie chart with 12 categories
+#     Issue:   `country` has 12 categories; slices this numerous are hard to compare by angle.
+#     Instead: A horizontal bar chart of the counts, sorted by size.
+#     Why:     Bars share a common baseline ...
+#     How:     counts = df['country'].value_counts() ...
+```
+
+```
+/describe        <- the concept, why it fits this data, when another method is better
+/accept          <- puts the suggested code in a NEW cell; your cell is untouched
+/reject only 3 countries matter here   <- recorded in memory, not suggested again
+```
+
+When a cell raises an error, learnstat diagnoses the failing statement. If it's a mechanical mistake, learnstat corrects it, re-runs only that statement and the ones after it, and prints the diff. A cell with a syntax error gets a corrected copy placed in the next cell for you to review.
+
+### Command line
+
+```bash
+cd examples && python make_survey.py
+learnstat profile survey.csv           # writes .learnstat/survey.md and prints it
+learnstat check analysis.py            # suggestions only, nothing is executed
+learnstat watch analysis.py            # re-checks every time you save the file
+learnstat run analysis.py              # runs with suggestions and guarded recovery
+learnstat repl --data survey.csv       # interactive, with /commands
+```
+
+### Plain Python
+
+```python
+import learnstat as ls
+df = ls.read_csv("survey.csv")
+ls.check("df['income'].mean()")         # suggestions without running
+print(ls.command("/describe"))
+
+s = ls.Session(confirm=lambda q: input(q + " [y/N] ") == "y")
+s.ns["df"] = s.read_csv("survey.csv")
+result = s.run("df.groupBy('group')['score'].median()")
+print(result.final_code)                # df.groupby(...)... your source is never modified
+```
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `/describe [id]` | Explains the latest (or given) suggestion, or the latest error: the concept, the purpose of the function, why it fits this data, and when something else is better |
+| `/alternatives [id]` | Other ways to reach the same goal, with trade-offs |
+| `/learn [topic]` | A short lesson with an exercise on your own data |
+| `/fix` | The last error diagnosis; in interactive mode, a retry of the failed statement |
+| `/memory [show\|note\|decide\|refresh]` | View or update the dataset's persistent profile |
+| `/accept [id]`, `/reject [id] [why]` | Accept or reject a suggestion; the decision is saved in memory |
+| `/suggestions`, `/review`, `/ask`, `/status`, `/help` | |
+
+## Dataset memory
+
+`ls.read_csv` profiles the data and writes two files to `.learnstat/`:
+
+- `survey.md`: columns, types, missing values, likely column meanings (marked as guesses), distributions, relationships, data-quality issues, suggested analyses, assumptions, **your analytical decisions** and notes.
+- `survey.json`: the machine-readable cache. It includes a content fingerprint, so the data is only re-profiled when it actually changes. Decisions and notes carry over when that happens.
+
+## How error recovery stays safe
+
+- Code runs **one top-level statement at a time**. When statement *k* fails, only statement *k* is corrected and retried, so earlier side effects are never repeated.
+- Errors are classified as **syntax, name, attribute, argument, column, data-quality, statistical-method, environment** or other runtime errors.
+- Only mechanical mistakes are corrected automatically, and only to a unique, high-similarity match.
+- Before a fix runs, it must compile. It must not introduce file, shell or network operations, must not change which variables are assigned, must not add data-altering calls (`dropna`, `fillna`, `astype` and so on), and must not change the statistical method.
+- AI-proposed fixes always need your confirmation. So does retrying a loop or block, since that could repeat side effects.
+- Retries are bounded (`max_attempts`, 3 by default). If the same error comes back, recovery stops and gives you a concrete next step.
+- Code that deletes or overwrites files, runs shell commands or opens network connections only runs after you confirm.
+- **This is a policy layer, not a sandbox.** Your code runs in your Python process with your permissions, exactly as if you had run it yourself.
+
+## Project layout
+
+```
+learnstat/
+  profile.py      dataset profiling (no AI)
+  memory.py       Markdown + JSON dataset memory
+  advisor.py      AST-based, data-aware suggestion rules
+  knowledge.py    offline explanations behind /describe, /alternatives, /learn
+  recovery.py     error diagnosis, fix proposal, validation, bounded retry
+  safety.py       detection of destructive side effects
+  llm.py          optional Claude backend
+  session.py      ties it together; read_csv/track/check/run
+  commands.py     slash commands
+  ipython_ext.py  %load_ext learnstat
+  cli.py          learnstat profile|check|watch|run|repl
+```
+
+Add a rule by writing a function `(call, resolver, code) -> Suggestion | None` in `advisor.py`, appending it to `RULES`, and adding its concept to `knowledge.py`.
+
+## Roadmap
+
+- A language-server (LSP) frontend for suggestions while typing in VS Code and JupyterLab. The advisor is already editor-agnostic.
+- More rules: multiple-comparison corrections, chi-square expected counts, normality checks before parametric tests, time-series plotting.
+- Learning-progress tracking, so advice fades as the user masters a concept.
+
+## Tests
+
+```bash
+pip install -e ".[dev]"
+pytest
+```

learnstat-0.1.0/learnstat/__init__.py

@@ -0,0 +1,80 @@
+"""learnstat - an AI-assisted learning companion for statistics in Python.
+
+You write ordinary pandas / NumPy / SciPy / Matplotlib code; learnstat profiles
+your data, suggests better approaches, explains its reasoning and helps you
+recover from mistakes.
+
+    import learnstat as ls
+    df = ls.read_csv("survey.csv")      # a normal pandas DataFrame, now profiled
+    ls.check("df['income'].mean()")     # suggestions, without running anything
+    print(ls.command("/describe"))      # why?
+
+In Jupyter: ``%load_ext learnstat``.
+"""
+
+from __future__ import annotations
+
+from .advisor import CodeAdvisor
+from .memory import MemoryStore
+from .profile import DatasetProfile, profile_dataframe
+from .recovery import Executor, RunResult
+from .session import Session, get_session, set_session
+from .suggestions import Suggestion
+
+__version__ = "0.1.0"
+
+
+def read_csv(path, name: str | None = None, **kwargs):
+    """``pandas.read_csv`` plus dataset profiling and persistent memory."""
+    return get_session().read_csv(path, name=name, **kwargs)
+
+
+def read_excel(path, name: str | None = None, **kwargs):
+    return get_session().read_excel(path, name=name, **kwargs)
+
+
+def read_parquet(path, name: str | None = None, **kwargs):
+    return get_session().read_parquet(path, name=name, **kwargs)
+
+
+def read_json(path, name: str | None = None, **kwargs):
+    return get_session().read_json(path, name=name, **kwargs)
+
+
+def track(df, name: str | None = None, **kwargs):
+    """Profile a DataFrame you already have."""
+    return get_session().track(df, name=name, **kwargs)
+
+
+def check(code: str, show: bool = True):
+    """Suggestions for ``code`` (it is not executed)."""
+    return get_session().check(code, show=show)
+
+
+def run(code: str):
+    """Run ``code`` in the default session with suggestions and guarded recovery."""
+    return get_session().run(code)
+
+
+def command(text: str) -> str:
+    """Run a slash command such as ``/describe`` and return its text."""
+    return get_session().command(text)
+
+
+def load_ipython_extension(ip):
+    from .ipython_ext import load_ipython_extension as _load
+
+    _load(ip)
+
+
+def unload_ipython_extension(ip):
+    from .ipython_ext import unload_ipython_extension as _unload
+
+    _unload(ip)
+
+
+__all__ = [
+    "Session", "get_session", "set_session", "read_csv", "read_excel", "read_parquet", "read_json",
+    "track", "check", "run", "command", "profile_dataframe", "DatasetProfile", "MemoryStore",
+    "CodeAdvisor", "Executor", "RunResult", "Suggestion",
+]