goad-toolkit 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- goad_toolkit-0.2.0/.gitignore +14 -0
- goad_toolkit-0.2.0/.python-version +1 -0
- goad_toolkit-0.2.0/MCP_SERVER.md +205 -0
- goad_toolkit-0.2.0/PKG-INFO +246 -0
- goad_toolkit-0.2.0/README.md +228 -0
- goad_toolkit-0.2.0/demo/linear.py +128 -0
- goad_toolkit-0.2.0/docs/01-goal-oriented-analysis.md +132 -0
- goad_toolkit-0.2.0/docs/02-pipelines.md +199 -0
- goad_toolkit-0.2.0/docs/03-plot-composition.md +222 -0
- goad_toolkit-0.2.0/docs/04-five-families.md +208 -0
- goad_toolkit-0.2.0/docs/05-distributions.md +224 -0
- goad_toolkit-0.2.0/docs/06-models-and-residuals.md +163 -0
- goad_toolkit-0.2.0/docs/07-visual-critique.md +130 -0
- goad_toolkit-0.2.0/docs/08-api-reference.md +279 -0
- goad_toolkit-0.2.0/docs/09-analysis-method.md +162 -0
- goad_toolkit-0.2.0/docs/10-teaching-path.md +172 -0
- goad_toolkit-0.2.0/docs/README.md +94 -0
- goad_toolkit-0.2.0/goad_mcp.py +1311 -0
- goad_toolkit-0.2.0/img/distribution_fit.png +0 -0
- goad_toolkit-0.2.0/img/goaded.png +0 -0
- goad_toolkit-0.2.0/img/linear_results.png +0 -0
- goad_toolkit-0.2.0/img/residuals.png +0 -0
- goad_toolkit-0.2.0/img/zscores.png +0 -0
- goad_toolkit-0.2.0/pyproject.toml +38 -0
- goad_toolkit-0.2.0/src/goad_toolkit/__init__.py +1 -0
- goad_toolkit-0.2.0/src/goad_toolkit/analytics.py +325 -0
- goad_toolkit-0.2.0/src/goad_toolkit/config.py +24 -0
- goad_toolkit-0.2.0/src/goad_toolkit/dataprocessor.py +48 -0
- goad_toolkit-0.2.0/src/goad_toolkit/datatransforms.py +184 -0
- goad_toolkit-0.2.0/src/goad_toolkit/distributions.py +92 -0
- goad_toolkit-0.2.0/src/goad_toolkit/filehandler.py +65 -0
- goad_toolkit-0.2.0/src/goad_toolkit/models.py +44 -0
- goad_toolkit-0.2.0/src/goad_toolkit/visualizer.py +537 -0
- goad_toolkit-0.2.0/uv.lock +1941 -0
|
@@ -0,0 +1 @@
|
|
|
1
|
+
3.12
|
|
@@ -0,0 +1,205 @@
|
|
|
1
|
+
# GOAD๐ as an MCP server
|
|
2
|
+
|
|
3
|
+
This repo ships a small [MCP](https://modelcontextprotocol.io) server,
|
|
4
|
+
[`goad_mcp.py`](goad_mcp.py), that turns the [`docs/`](docs) chapters into coaching methods
|
|
5
|
+
for a coding assistant (Claude Code, Cursor, Codex, Claude Desktop, โฆ) โ one for **analysing**
|
|
6
|
+
a dataset from a question rather than from its columns, one for **critiquing** a finished
|
|
7
|
+
visualisation, and one for **fitting a distribution** without deciding what an outlier is
|
|
8
|
+
first.
|
|
9
|
+
|
|
10
|
+
The point is to learn how to work *with* an assistant while doing data analysis, grounded in
|
|
11
|
+
GOAD's own method (the question comes first, the residual is the finding, every parameter is
|
|
12
|
+
part of the claim) instead of generic advice โ and instead of handing the whole problem over
|
|
13
|
+
and getting back twelve plots nobody can defend.
|
|
14
|
+
|
|
15
|
+
## How to use it
|
|
16
|
+
|
|
17
|
+
- **Analysing something:** in Claude Code, run `/goad:analyse` (optionally describe what you
|
|
18
|
+
want to find out). In any client, just say **"Coach me through analysing this with the goad
|
|
19
|
+
server."** Expect a conversation, not an answer: your assistant will work through six
|
|
20
|
+
stages, asking you questions one stage at a time and waiting for your actual answers before
|
|
21
|
+
moving on.
|
|
22
|
+
- **Critiquing a plot:** `/goad:critique`, or **"Critique this plot against the goad
|
|
23
|
+
checklist."**
|
|
24
|
+
- **Fitting a distribution:** `/goad:distribution`, or **"Coach me through fitting a
|
|
25
|
+
distribution to this column using the goad server."**
|
|
26
|
+
- **Looking something up:** "Using the goad server, get the chapter on plot composition." /
|
|
27
|
+
"Search the goad docs for long tail." / "What does goad say about t-SNE perplexity?"
|
|
28
|
+
|
|
29
|
+
The analysis and critique prompts are coaching sessions, not code generators โ they are built
|
|
30
|
+
to ask what you think before offering an answer, and to write code only once you have said
|
|
31
|
+
what it is for. Using your assistant to answer all six stages itself in one go defeats the
|
|
32
|
+
point; ask it to go back to stage one and actually wait for your answers.
|
|
33
|
+
|
|
34
|
+
## What it exposes
|
|
35
|
+
|
|
36
|
+
**Tools**
|
|
37
|
+
|
|
38
|
+
| Tool | What it does |
|
|
39
|
+
| --- | --- |
|
|
40
|
+
| `goad_list_concepts` | List all 10 chapters, in reading order. |
|
|
41
|
+
| `goad_get_concept` | Fetch the full markdown of one chapter. |
|
|
42
|
+
| `goad_search` | Keyword search across every chapter. |
|
|
43
|
+
| `goad_analysis_checklist` | A six-stage, table-driven analysis conversation โ see below. |
|
|
44
|
+
| `goad_critique_visual` | The visual-critique checklist, one section at a time. |
|
|
45
|
+
| `goad_distribution_method` | The method for fitting and defending a distribution, in the right order. |
|
|
46
|
+
|
|
47
|
+
**`goad_analysis_checklist` is the centrepiece**, and it is deliberately not a one-shot
|
|
48
|
+
document dump. It walks a student through six stages **in order**, one at a time, and only
|
|
49
|
+
advances once the assistant reports back what the student actually said:
|
|
50
|
+
|
|
51
|
+
1. **question** โ what do you want to know, what would count as an answer in both directions,
|
|
52
|
+
and who is it for?
|
|
53
|
+
2. **data** โ what is one row, what was measured versus derived, and what does the question
|
|
54
|
+
need that the data does not yet contain?
|
|
55
|
+
3. **shape** โ how is each variable distributed, is anything long-tailed, and are the extreme
|
|
56
|
+
values errors or ordinary members of a tail?
|
|
57
|
+
4. **encoding** โ which of the five families does the question live in, and which visual
|
|
58
|
+
channel carries the comparison you actually care about?
|
|
59
|
+
5. **critique** โ what does the plot say in the first 200ms, what does the eye group by
|
|
60
|
+
itself, and what exactly are you claiming?
|
|
61
|
+
6. **verification** โ what null would produce this too, what confounder could, and what does
|
|
62
|
+
the residual look like?
|
|
63
|
+
|
|
64
|
+
Each stage returns a markdown table โ question / why it matters / what a good answer names โ
|
|
65
|
+
plus an instruction to the assistant: ask the student, wait for a real answer, don't answer on
|
|
66
|
+
their behalf, and don't fetch the next stage until this one is actually recorded. Progress is
|
|
67
|
+
held in memory for the current session only. At the end it returns a recap of everything
|
|
68
|
+
recorded, which is a usable first draft of the write-up.
|
|
69
|
+
|
|
70
|
+
`goad_critique_visual` splits the work explicitly: sections 2 and 3 (gestalt principles, the
|
|
71
|
+
five guidelines) are ones an assistant can genuinely help with, and it is told to be concrete
|
|
72
|
+
there. Sections 1 and 4 (what caught your eye, what you are claiming) it is told to ask and
|
|
73
|
+
wait for โ it cannot see the plot the way you do, and answering section 4 for you means
|
|
74
|
+
inventing a claim to fit the picture, which is backwards.
|
|
75
|
+
|
|
76
|
+
**Prompts**
|
|
77
|
+
|
|
78
|
+
- `analyse` โ start the six-stage analysis conversation. Takes an optional `question`.
|
|
79
|
+
- `critique` โ walk a finished visualisation through the checklist. Takes an optional `plot`.
|
|
80
|
+
- `distribution` โ coach the distribution-fitting method in order. Takes an optional `variable`.
|
|
81
|
+
|
|
82
|
+
**Resources**
|
|
83
|
+
|
|
84
|
+
- `goad://concept/{name}` โ each chapter's raw markdown, addressable by number (`5`),
|
|
85
|
+
filename (`05-distributions.md`), stem, or title.
|
|
86
|
+
|
|
87
|
+
The content comes from [`docs/`](docs); no build step, no database, no embeddings.
|
|
88
|
+
|
|
89
|
+
## Requirements
|
|
90
|
+
|
|
91
|
+
- [`uv`](https://docs.astral.sh/uv/)
|
|
92
|
+
- Python 3.10+
|
|
93
|
+
|
|
94
|
+
You do **not** need `goad-toolkit` installed to run the server โ it only reads markdown.
|
|
95
|
+
|
|
96
|
+
## Install
|
|
97
|
+
|
|
98
|
+
`uv` runs the server straight from a URL: it downloads the single script, provisions Python
|
|
99
|
+
and the `fastmcp` package from the script's own [PEP 723](https://peps.python.org/pep-0723/)
|
|
100
|
+
metadata, and the server fetches its docs from the repo over HTTP.
|
|
101
|
+
|
|
102
|
+
You pin a version with `GOAD_REF` โ a git ref (branch, tag, or commit). That one variable
|
|
103
|
+
selects both the script (in the URL) and the docs it fetches (inside the script), so a whole
|
|
104
|
+
cohort runs exactly the same thing and the two cannot drift apart.
|
|
105
|
+
|
|
106
|
+
> **Note:** the docs and this server land in **v0.2.0**. Until that tag exists, point
|
|
107
|
+
> `GOAD_REF` at the branch (`feat/docs-and-mcp`) or work from a local clone (see below).
|
|
108
|
+
|
|
109
|
+
### Claude Code
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
claude mcp add goad -e GOAD_REF=v0.2.0 -- \
|
|
113
|
+
sh -c 'uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py'
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
### Cursor
|
|
117
|
+
|
|
118
|
+
Add to `.cursor/mcp.json` (per-project) or `~/.cursor/mcp.json` (global):
|
|
119
|
+
|
|
120
|
+
```json
|
|
121
|
+
{
|
|
122
|
+
"mcpServers": {
|
|
123
|
+
"goad": {
|
|
124
|
+
"command": "sh",
|
|
125
|
+
"args": ["-c", "uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py"],
|
|
126
|
+
"env": { "GOAD_REF": "v0.2.0" }
|
|
127
|
+
}
|
|
128
|
+
}
|
|
129
|
+
}
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
### Codex
|
|
133
|
+
|
|
134
|
+
Add to `~/.codex/config.toml`:
|
|
135
|
+
|
|
136
|
+
```toml
|
|
137
|
+
[mcp_servers.goad]
|
|
138
|
+
command = "sh"
|
|
139
|
+
args = ["-c", "uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py"]
|
|
140
|
+
env = { GOAD_REF = "v0.2.0" }
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
### Claude Desktop
|
|
144
|
+
|
|
145
|
+
Add to `claude_desktop_config.json` (Settings โ Developer โ Edit Config), then restart the
|
|
146
|
+
app:
|
|
147
|
+
|
|
148
|
+
```json
|
|
149
|
+
{
|
|
150
|
+
"mcpServers": {
|
|
151
|
+
"goad": {
|
|
152
|
+
"command": "sh",
|
|
153
|
+
"args": ["-c", "uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py"],
|
|
154
|
+
"env": { "GOAD_REF": "v0.2.0" }
|
|
155
|
+
}
|
|
156
|
+
}
|
|
157
|
+
}
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
(The `sh` wrapper is for macOS/Linux; on Windows, drop `sh -c` and put the ref directly in
|
|
161
|
+
the URL instead, keeping the `GOAD_REF` env value in sync.)
|
|
162
|
+
|
|
163
|
+
## Working from a local clone instead
|
|
164
|
+
|
|
165
|
+
If you have already cloned `goad_toolkit`, point the same command at the local file instead
|
|
166
|
+
of the URL โ the server then reads `docs/` straight off disk, so any edits show up without a
|
|
167
|
+
restart:
|
|
168
|
+
|
|
169
|
+
```bash
|
|
170
|
+
claude mcp add goad -- uv run --no-project /path/to/goad_toolkit/goad_mcp.py
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
## Tracking the bleeding edge instead
|
|
174
|
+
|
|
175
|
+
Set `GOAD_REF=main` to always run whatever is on `main`, including chapters or checklist
|
|
176
|
+
changes not in a release yet. Fine for your own use; for a cohort of students, pin to a tag
|
|
177
|
+
instead so everyone runs the exact same docs during an assignment.
|
|
178
|
+
|
|
179
|
+
## Update to a new version
|
|
180
|
+
|
|
181
|
+
Pick the newest tag from the [releases](https://github.com/raoulg/goad_toolkit/tags), then
|
|
182
|
+
point `GOAD_REF` at it. Because the ref lives in the URL, switching it fetches the new script
|
|
183
|
+
automatically โ no cache clearing needed.
|
|
184
|
+
|
|
185
|
+
**Claude Code** โ re-running `add` errors if the server already exists, so remove first, then
|
|
186
|
+
add with the new tag:
|
|
187
|
+
|
|
188
|
+
```bash
|
|
189
|
+
claude mcp remove goad -s user
|
|
190
|
+
claude mcp add goad -s user -e GOAD_REF=v0.2.0 -- \
|
|
191
|
+
sh -c 'uv run --no-project https://raw.githubusercontent.com/raoulg/goad_toolkit/$GOAD_REF/goad_mcp.py'
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
**Cursor / Codex / Claude Desktop** โ edit the `GOAD_REF` value in the config, then restart
|
|
195
|
+
your assistant so it relaunches the server.
|
|
196
|
+
|
|
197
|
+
## Companion servers
|
|
198
|
+
|
|
199
|
+
`goad` covers "is my analysis any good". Two siblings cover the rest of the same workflow:
|
|
200
|
+
|
|
201
|
+
- [`codestyle`](https://github.com/raoulg/codestyle) โ is my code any good.
|
|
202
|
+
- [`vectormesh`](https://github.com/raoulg/vectormesh) โ is my architecture any good.
|
|
203
|
+
|
|
204
|
+
All three follow the same shape: a `docs/` folder as the single source of truth, one PEP 723
|
|
205
|
+
script that serves it, and a staged conversation the assistant cannot short-circuit.
|
|
@@ -0,0 +1,246 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: goad-toolkit
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: An extensible toolkit for Goal Oriented Analysis of Data
|
|
5
|
+
Project-URL: Github, https://github.com/raoulg/goad_toolkit
|
|
6
|
+
Author-email: raoul grouls <Raoul.Grouls@han.nl>
|
|
7
|
+
Requires-Python: >=3.12
|
|
8
|
+
Requires-Dist: loguru>=0.7.3
|
|
9
|
+
Requires-Dist: matplotlib>=3.10.1
|
|
10
|
+
Requires-Dist: numpy>=2.2.4
|
|
11
|
+
Requires-Dist: pandas>=2.2.3
|
|
12
|
+
Requires-Dist: pydantic>=2.10.6
|
|
13
|
+
Requires-Dist: requests>=2.32.3
|
|
14
|
+
Requires-Dist: scipy>=1.15.2
|
|
15
|
+
Requires-Dist: seaborn>=0.13.2
|
|
16
|
+
Requires-Dist: tqdm>=4.67.1
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
# GOAD๐ is the GOAT - Goal Oriented Analysis of Data
|
|
20
|
+
[](https://github.com/astral-sh/uv)
|
|
21
|
+
[](https://pypi.org/project/goad_toolkit/)
|
|
22
|
+
|
|
23
|
+
<p align="center">
|
|
24
|
+
<em>GOAD๐ - When your data analysis is so fire๐ฅ it's got rizzโจ</em>
|
|
25
|
+
</p>
|
|
26
|
+
|
|
27
|
+

|
|
28
|
+
|
|
29
|
+
GOAD๐ is a flexible Python package for analyzing, transforming, and visualizing data with an emphasis on statistical distribution fitting and modular visualization components.
|
|
30
|
+
|
|
31
|
+
> with thanks to my daughters for the genz slang!
|
|
32
|
+
|
|
33
|
+
## ๐ Features
|
|
34
|
+
|
|
35
|
+
- **Composable & extendable plotting system** - Build complex visualizations by combining simple components. You can extend the existing components with your own.
|
|
36
|
+
- **Statistical distribution fitting** - Automatically fit and compare distributions to your data. The distribution registry is extendable with additional distributions.
|
|
37
|
+
- **Extendable data transformation pipelines** - Chain and reuse data transformations into pipelines. Again, extendable with custom transformation components.
|
|
38
|
+
|
|
39
|
+
> Before GOAD๐ : mid data
|
|
40
|
+
> After GOAD๐ : data got infinity aura
|
|
41
|
+
|
|
42
|
+
## ๐ Documentation
|
|
43
|
+
|
|
44
|
+
Full documentation lives in [`docs/`](docs/README.md) โ ten chapters covering the method, the
|
|
45
|
+
pipeline and plot systems, the five families of visualisation, distribution fitting, models
|
|
46
|
+
and residuals, and a visual-critique checklist.
|
|
47
|
+
|
|
48
|
+
The same chapters are also available as an **MCP server**, so your coding assistant can coach
|
|
49
|
+
you through an analysis instead of doing it for you. See [MCP_SERVER.md](MCP_SERVER.md).
|
|
50
|
+
|
|
51
|
+
## ๐ Quick Start
|
|
52
|
+
|
|
53
|
+
### Installation
|
|
54
|
+
Using [uv](https://docs.astral.sh/uv/):
|
|
55
|
+
```bash
|
|
56
|
+
uv add goad-toolkit
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Or, if you prefer your dependencies to be installed 100x slower, with pip:
|
|
60
|
+
```bash
|
|
61
|
+
pip install goad-toolkit
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
## ๐ Demo: Linear Model Analysis
|
|
65
|
+
|
|
66
|
+
GOAD๐ includes a comprehensive [demo](demo/linear.py) that shows how to use its components together.
|
|
67
|
+
|
|
68
|
+
### Main capabilities
|
|
69
|
+
In the [demo/linear.py](demo/linear.py) file you can see a showcase of the main capabilities of GOAD๐:
|
|
70
|
+
- create a data processing pipeline
|
|
71
|
+
- components are extendable, so you can easily add your own steps to a pipeline
|
|
72
|
+
- create visualisations by stacking components. `BasePlot` will handle boilerplate.
|
|
73
|
+
- the `DistributionFitter` will try to fit a few common distributions, and add statistical tests for you
|
|
74
|
+
- The results work together with the `visualizer.PlotFits` class to show the results
|
|
75
|
+
|
|
76
|
+
The main strenght of this module is not that these elements are there (even thought they are very useful). Its superpower is that everything is extendable: so you can use this as a start, and extend it with your own visualisations and analytics.
|
|
77
|
+
|
|
78
|
+
> POV: Your data just got GOADed๐ and now it's giving main character energy
|
|
79
|
+
|
|
80
|
+
## ๐ Core Components
|
|
81
|
+
|
|
82
|
+
#### ๐ Extendable Data Transforms
|
|
83
|
+
|
|
84
|
+
GOAD๐ provides a pipeline approach to transform your data:
|
|
85
|
+
|
|
86
|
+
```python
|
|
87
|
+
from goad_toolkit.datatransforms import Pipeline, ShiftValues, ZScaler
|
|
88
|
+
|
|
89
|
+
# Create a pipeline
|
|
90
|
+
pipeline = Pipeline()
|
|
91
|
+
|
|
92
|
+
# Add transformations
|
|
93
|
+
pipeline.add(ShiftValues, name="shift_deaths", column="deaths", period=-14)
|
|
94
|
+
pipeline.add(ZScaler, name="scale_tests", column="positivetests", rename=True)
|
|
95
|
+
|
|
96
|
+
# Apply all transformations
|
|
97
|
+
result = pipeline.apply(data)
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Available transforms include:
|
|
101
|
+
- `ShiftValues` - Shift values in a column by a specified period
|
|
102
|
+
- `DiffValues` - Calculate the difference between consecutive values
|
|
103
|
+
- `SelectDataRange` - Select rows within a specified date range
|
|
104
|
+
- `RollingAvg` - Calculate rolling average of a column
|
|
105
|
+
- `ZScaler` - Standardize values in a column
|
|
106
|
+
|
|
107
|
+
You can extend the pipeline with your own transformations by subclassing `BaseTransform`. The Zscaler is implemented as follows:
|
|
108
|
+
|
|
109
|
+
```python
|
|
110
|
+
class ZScaler(TransformBase):
|
|
111
|
+
"""Standardize the values in a column."""
|
|
112
|
+
def transform(
|
|
113
|
+
self, data: pd.DataFrame, column: str, rename: bool = False
|
|
114
|
+
) -> pd.DataFrame:
|
|
115
|
+
"""Standardize the values in a column."""
|
|
116
|
+
if rename:
|
|
117
|
+
colname = f"{column}_zscore"
|
|
118
|
+
else:
|
|
119
|
+
colname = column
|
|
120
|
+
data[colname] = (data[column] - data[column].mean()) / data[column].std()
|
|
121
|
+
return data
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### ๐ Visualization System
|
|
125
|
+
|
|
126
|
+
GOAD๐ visualization system is built on a composable architecture that allows you to build complex plots by combining simpler components:
|
|
127
|
+
|
|
128
|
+
```python
|
|
129
|
+
from goad_toolkit.visualizer import PlotSettings, ResidualPlot
|
|
130
|
+
|
|
131
|
+
# Create plot settings
|
|
132
|
+
plotsettings = PlotSettings(
|
|
133
|
+
xlabel="date",
|
|
134
|
+
ylabel="normalized values",
|
|
135
|
+
title="Z-Scores of Deaths and Positive Tests",
|
|
136
|
+
)
|
|
137
|
+
|
|
138
|
+
class LinePlot(BasePlot):
|
|
139
|
+
"""Plot a line plot using seaborn."""
|
|
140
|
+
def build(self, data: pd.DataFrame, **kwargs):
|
|
141
|
+
sns.lineplot(data=data, ax=self.ax, **kwargs)
|
|
142
|
+
return self.fig, self.ax
|
|
143
|
+
|
|
144
|
+
|
|
145
|
+
class ComparePlot(BasePlot):
|
|
146
|
+
def build(self, data: pd.DataFrame, x: str, y1: str, y2: str, **kwargs):
|
|
147
|
+
compare = LinePlot(self.settings)
|
|
148
|
+
self.plot_on(compare, data=data, x=x, y=y1, label=y1, **kwargs)
|
|
149
|
+
self.plot_on(compare, data=data, x=x, y=y2, label=y2, **kwargs)
|
|
150
|
+
plt.xticks(rotation=45)
|
|
151
|
+
|
|
152
|
+
return self.fig, self.ax
|
|
153
|
+
|
|
154
|
+
compareplot = ComparePlot(plotsettings)
|
|
155
|
+
compareplot.plot(
|
|
156
|
+
data=data, x="date", y1="deaths_shifted_zscore", y2="positivetests_zscore"
|
|
157
|
+
)
|
|
158
|
+
```
|
|
159
|
+

|
|
160
|
+
This extendable strategy lets BasePlot handle the boilerplate, while you can focus on creating the visualizations you need.
|
|
161
|
+
It is also easier to reuse components in different contexts.
|
|
162
|
+
### ๐ Distribution Fitting
|
|
163
|
+
|
|
164
|
+
GOAD๐ includes tools for fitting statistical distributions to your data:
|
|
165
|
+
|
|
166
|
+
```python
|
|
167
|
+
from goad_toolkit.analytics import DistributionFitter
|
|
168
|
+
from goad_toolkit.visualizer import PlotSettings, FitPlotSettings, PlotFits
|
|
169
|
+
|
|
170
|
+
fitter = DistributionFitter()
|
|
171
|
+
fits = fitter.fit(data["residual"], discrete=False) # we have to decide if the data is discrete or not
|
|
172
|
+
best = fitter.best(fits)
|
|
173
|
+
settings = PlotSettings(
|
|
174
|
+
figsize=(12, 6), title="Residuals", xlabel="error", ylabel="probability"
|
|
175
|
+
)
|
|
176
|
+
fitplotsettings = FitPlotSettings(bins=30, max_fits=3)
|
|
177
|
+
fitplotter = PlotFits(settings)
|
|
178
|
+
fig = fitplotter.plot(
|
|
179
|
+
data=data["residual"], fit_results=fits, fitplotsettings=fitplotsettings
|
|
180
|
+
)
|
|
181
|
+
```
|
|
182
|
+
For the [kstest](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kstest.html), the null hypothesis is that the two distributions are identical. In this example, the p-values are below 0.05, so we can reject the null hypothesis and conclude that the data does not follow any of these.
|
|
183
|
+
|
|
184
|
+
The plots are sorted by log-likelihood, which means there is no good fit with a distribution in this case.
|
|
185
|
+

|
|
186
|
+
|
|
187
|
+
### ๐งฉ Extending with Custom Distributions
|
|
188
|
+
|
|
189
|
+
You can easily register new distributions:
|
|
190
|
+
|
|
191
|
+
```python
|
|
192
|
+
from goad_toolkit.distributions import DistributionRegistry
|
|
193
|
+
from scipy import stats
|
|
194
|
+
|
|
195
|
+
# Create registry
|
|
196
|
+
registry = DistributionRegistry()
|
|
197
|
+
|
|
198
|
+
# Register a new distribution
|
|
199
|
+
registry.register_distribution(
|
|
200
|
+
name="negative_binomial",
|
|
201
|
+
dist=stats.nbinom,
|
|
202
|
+
is_discrete=True,
|
|
203
|
+
num_params=2
|
|
204
|
+
)
|
|
205
|
+
|
|
206
|
+
# Now it will be used automatically in the DistributionFitter for discrete fits
|
|
207
|
+
from goad_toolkit.analytics import DistributionFitter
|
|
208
|
+
fitter = DistributionFitter()
|
|
209
|
+
print(fitter.registry) # shows all registered distributions
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
|
|
213
|
+
|
|
214
|
+
## ๐ง Advanced Usage: Composing Plots
|
|
215
|
+
|
|
216
|
+
GOAD๐ has a powerful plotting system that allows you to combine plot elements:
|
|
217
|
+
|
|
218
|
+
```python
|
|
219
|
+
from goad_toolkit.visualizer import BasePlot, LinePlot, BarWithDates, VerticalDate
|
|
220
|
+
|
|
221
|
+
# Use a base plot to create a composite
|
|
222
|
+
class MyCompositePlot(BasePlot):
|
|
223
|
+
def build(self, data: pd.DataFrame, x: str, y1: str, y2: str, special_date: str):
|
|
224
|
+
# Plot the first component - a line plot
|
|
225
|
+
line_plot = LinePlot(self.settings)
|
|
226
|
+
self.plot_on(line_plot, data=data, x=x, y=y1, label=y1)
|
|
227
|
+
|
|
228
|
+
# Plot the second component - a bar chart
|
|
229
|
+
bar_plot = BarWithDates(self.settings)
|
|
230
|
+
self.plot_on(bar_plot, data=data, x=x, y=y2)
|
|
231
|
+
|
|
232
|
+
# Add a vertical line
|
|
233
|
+
vline = VerticalDate(self.settings)
|
|
234
|
+
self.plot_on(vline, date=special_date, label="Important Event")
|
|
235
|
+
return self.fig, self.ax
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
## ๐ค Contributing
|
|
239
|
+
|
|
240
|
+
Contributions are welcome! Please feel free to submit a Pull Request.
|
|
241
|
+
|
|
242
|
+
---
|
|
243
|
+
|
|
244
|
+
<p align="center">
|
|
245
|
+
<em>GOAD๐ - When your data analysis is so fire๐ฅ it's got rizzโจ</em>
|
|
246
|
+
</p>
|
|
@@ -0,0 +1,228 @@
|
|
|
1
|
+
# GOAD๐ is the GOAT - Goal Oriented Analysis of Data
|
|
2
|
+
[](https://github.com/astral-sh/uv)
|
|
3
|
+
[](https://pypi.org/project/goad_toolkit/)
|
|
4
|
+
|
|
5
|
+
<p align="center">
|
|
6
|
+
<em>GOAD๐ - When your data analysis is so fire๐ฅ it's got rizzโจ</em>
|
|
7
|
+
</p>
|
|
8
|
+
|
|
9
|
+

|
|
10
|
+
|
|
11
|
+
GOAD๐ is a flexible Python package for analyzing, transforming, and visualizing data with an emphasis on statistical distribution fitting and modular visualization components.
|
|
12
|
+
|
|
13
|
+
> with thanks to my daughters for the genz slang!
|
|
14
|
+
|
|
15
|
+
## ๐ Features
|
|
16
|
+
|
|
17
|
+
- **Composable & extendable plotting system** - Build complex visualizations by combining simple components. You can extend the existing components with your own.
|
|
18
|
+
- **Statistical distribution fitting** - Automatically fit and compare distributions to your data. The distribution registry is extendable with additional distributions.
|
|
19
|
+
- **Extendable data transformation pipelines** - Chain and reuse data transformations into pipelines. Again, extendable with custom transformation components.
|
|
20
|
+
|
|
21
|
+
> Before GOAD๐ : mid data
|
|
22
|
+
> After GOAD๐ : data got infinity aura
|
|
23
|
+
|
|
24
|
+
## ๐ Documentation
|
|
25
|
+
|
|
26
|
+
Full documentation lives in [`docs/`](docs/README.md) โ ten chapters covering the method, the
|
|
27
|
+
pipeline and plot systems, the five families of visualisation, distribution fitting, models
|
|
28
|
+
and residuals, and a visual-critique checklist.
|
|
29
|
+
|
|
30
|
+
The same chapters are also available as an **MCP server**, so your coding assistant can coach
|
|
31
|
+
you through an analysis instead of doing it for you. See [MCP_SERVER.md](MCP_SERVER.md).
|
|
32
|
+
|
|
33
|
+
## ๐ Quick Start
|
|
34
|
+
|
|
35
|
+
### Installation
|
|
36
|
+
Using [uv](https://docs.astral.sh/uv/):
|
|
37
|
+
```bash
|
|
38
|
+
uv add goad-toolkit
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Or, if you prefer your dependencies to be installed 100x slower, with pip:
|
|
42
|
+
```bash
|
|
43
|
+
pip install goad-toolkit
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## ๐ Demo: Linear Model Analysis
|
|
47
|
+
|
|
48
|
+
GOAD๐ includes a comprehensive [demo](demo/linear.py) that shows how to use its components together.
|
|
49
|
+
|
|
50
|
+
### Main capabilities
|
|
51
|
+
In the [demo/linear.py](demo/linear.py) file you can see a showcase of the main capabilities of GOAD๐:
|
|
52
|
+
- create a data processing pipeline
|
|
53
|
+
- components are extendable, so you can easily add your own steps to a pipeline
|
|
54
|
+
- create visualisations by stacking components. `BasePlot` will handle boilerplate.
|
|
55
|
+
- the `DistributionFitter` will try to fit a few common distributions, and add statistical tests for you
|
|
56
|
+
- The results work together with the `visualizer.PlotFits` class to show the results
|
|
57
|
+
|
|
58
|
+
The main strenght of this module is not that these elements are there (even thought they are very useful). Its superpower is that everything is extendable: so you can use this as a start, and extend it with your own visualisations and analytics.
|
|
59
|
+
|
|
60
|
+
> POV: Your data just got GOADed๐ and now it's giving main character energy
|
|
61
|
+
|
|
62
|
+
## ๐ Core Components
|
|
63
|
+
|
|
64
|
+
#### ๐ Extendable Data Transforms
|
|
65
|
+
|
|
66
|
+
GOAD๐ provides a pipeline approach to transform your data:
|
|
67
|
+
|
|
68
|
+
```python
|
|
69
|
+
from goad_toolkit.datatransforms import Pipeline, ShiftValues, ZScaler
|
|
70
|
+
|
|
71
|
+
# Create a pipeline
|
|
72
|
+
pipeline = Pipeline()
|
|
73
|
+
|
|
74
|
+
# Add transformations
|
|
75
|
+
pipeline.add(ShiftValues, name="shift_deaths", column="deaths", period=-14)
|
|
76
|
+
pipeline.add(ZScaler, name="scale_tests", column="positivetests", rename=True)
|
|
77
|
+
|
|
78
|
+
# Apply all transformations
|
|
79
|
+
result = pipeline.apply(data)
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Available transforms include:
|
|
83
|
+
- `ShiftValues` - Shift values in a column by a specified period
|
|
84
|
+
- `DiffValues` - Calculate the difference between consecutive values
|
|
85
|
+
- `SelectDataRange` - Select rows within a specified date range
|
|
86
|
+
- `RollingAvg` - Calculate rolling average of a column
|
|
87
|
+
- `ZScaler` - Standardize values in a column
|
|
88
|
+
|
|
89
|
+
You can extend the pipeline with your own transformations by subclassing `BaseTransform`. The Zscaler is implemented as follows:
|
|
90
|
+
|
|
91
|
+
```python
|
|
92
|
+
class ZScaler(TransformBase):
|
|
93
|
+
"""Standardize the values in a column."""
|
|
94
|
+
def transform(
|
|
95
|
+
self, data: pd.DataFrame, column: str, rename: bool = False
|
|
96
|
+
) -> pd.DataFrame:
|
|
97
|
+
"""Standardize the values in a column."""
|
|
98
|
+
if rename:
|
|
99
|
+
colname = f"{column}_zscore"
|
|
100
|
+
else:
|
|
101
|
+
colname = column
|
|
102
|
+
data[colname] = (data[column] - data[column].mean()) / data[column].std()
|
|
103
|
+
return data
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
### ๐ Visualization System
|
|
107
|
+
|
|
108
|
+
GOAD๐ visualization system is built on a composable architecture that allows you to build complex plots by combining simpler components:
|
|
109
|
+
|
|
110
|
+
```python
|
|
111
|
+
from goad_toolkit.visualizer import PlotSettings, ResidualPlot
|
|
112
|
+
|
|
113
|
+
# Create plot settings
|
|
114
|
+
plotsettings = PlotSettings(
|
|
115
|
+
xlabel="date",
|
|
116
|
+
ylabel="normalized values",
|
|
117
|
+
title="Z-Scores of Deaths and Positive Tests",
|
|
118
|
+
)
|
|
119
|
+
|
|
120
|
+
class LinePlot(BasePlot):
|
|
121
|
+
"""Plot a line plot using seaborn."""
|
|
122
|
+
def build(self, data: pd.DataFrame, **kwargs):
|
|
123
|
+
sns.lineplot(data=data, ax=self.ax, **kwargs)
|
|
124
|
+
return self.fig, self.ax
|
|
125
|
+
|
|
126
|
+
|
|
127
|
+
class ComparePlot(BasePlot):
|
|
128
|
+
def build(self, data: pd.DataFrame, x: str, y1: str, y2: str, **kwargs):
|
|
129
|
+
compare = LinePlot(self.settings)
|
|
130
|
+
self.plot_on(compare, data=data, x=x, y=y1, label=y1, **kwargs)
|
|
131
|
+
self.plot_on(compare, data=data, x=x, y=y2, label=y2, **kwargs)
|
|
132
|
+
plt.xticks(rotation=45)
|
|
133
|
+
|
|
134
|
+
return self.fig, self.ax
|
|
135
|
+
|
|
136
|
+
compareplot = ComparePlot(plotsettings)
|
|
137
|
+
compareplot.plot(
|
|
138
|
+
data=data, x="date", y1="deaths_shifted_zscore", y2="positivetests_zscore"
|
|
139
|
+
)
|
|
140
|
+
```
|
|
141
|
+

|
|
142
|
+
This extendable strategy lets BasePlot handle the boilerplate, while you can focus on creating the visualizations you need.
|
|
143
|
+
It is also easier to reuse components in different contexts.
|
|
144
|
+
### ๐ Distribution Fitting
|
|
145
|
+
|
|
146
|
+
GOAD๐ includes tools for fitting statistical distributions to your data:
|
|
147
|
+
|
|
148
|
+
```python
|
|
149
|
+
from goad_toolkit.analytics import DistributionFitter
|
|
150
|
+
from goad_toolkit.visualizer import PlotSettings, FitPlotSettings, PlotFits
|
|
151
|
+
|
|
152
|
+
fitter = DistributionFitter()
|
|
153
|
+
fits = fitter.fit(data["residual"], discrete=False) # we have to decide if the data is discrete or not
|
|
154
|
+
best = fitter.best(fits)
|
|
155
|
+
settings = PlotSettings(
|
|
156
|
+
figsize=(12, 6), title="Residuals", xlabel="error", ylabel="probability"
|
|
157
|
+
)
|
|
158
|
+
fitplotsettings = FitPlotSettings(bins=30, max_fits=3)
|
|
159
|
+
fitplotter = PlotFits(settings)
|
|
160
|
+
fig = fitplotter.plot(
|
|
161
|
+
data=data["residual"], fit_results=fits, fitplotsettings=fitplotsettings
|
|
162
|
+
)
|
|
163
|
+
```
|
|
164
|
+
For the [kstest](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kstest.html), the null hypothesis is that the two distributions are identical. In this example, the p-values are below 0.05, so we can reject the null hypothesis and conclude that the data does not follow any of these.
|
|
165
|
+
|
|
166
|
+
The plots are sorted by log-likelihood, which means there is no good fit with a distribution in this case.
|
|
167
|
+

|
|
168
|
+
|
|
169
|
+
### ๐งฉ Extending with Custom Distributions
|
|
170
|
+
|
|
171
|
+
You can easily register new distributions:
|
|
172
|
+
|
|
173
|
+
```python
|
|
174
|
+
from goad_toolkit.distributions import DistributionRegistry
|
|
175
|
+
from scipy import stats
|
|
176
|
+
|
|
177
|
+
# Create registry
|
|
178
|
+
registry = DistributionRegistry()
|
|
179
|
+
|
|
180
|
+
# Register a new distribution
|
|
181
|
+
registry.register_distribution(
|
|
182
|
+
name="negative_binomial",
|
|
183
|
+
dist=stats.nbinom,
|
|
184
|
+
is_discrete=True,
|
|
185
|
+
num_params=2
|
|
186
|
+
)
|
|
187
|
+
|
|
188
|
+
# Now it will be used automatically in the DistributionFitter for discrete fits
|
|
189
|
+
from goad_toolkit.analytics import DistributionFitter
|
|
190
|
+
fitter = DistributionFitter()
|
|
191
|
+
print(fitter.registry) # shows all registered distributions
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
|
|
195
|
+
|
|
196
|
+
## ๐ง Advanced Usage: Composing Plots
|
|
197
|
+
|
|
198
|
+
GOAD๐ has a powerful plotting system that allows you to combine plot elements:
|
|
199
|
+
|
|
200
|
+
```python
|
|
201
|
+
from goad_toolkit.visualizer import BasePlot, LinePlot, BarWithDates, VerticalDate
|
|
202
|
+
|
|
203
|
+
# Use a base plot to create a composite
|
|
204
|
+
class MyCompositePlot(BasePlot):
|
|
205
|
+
def build(self, data: pd.DataFrame, x: str, y1: str, y2: str, special_date: str):
|
|
206
|
+
# Plot the first component - a line plot
|
|
207
|
+
line_plot = LinePlot(self.settings)
|
|
208
|
+
self.plot_on(line_plot, data=data, x=x, y=y1, label=y1)
|
|
209
|
+
|
|
210
|
+
# Plot the second component - a bar chart
|
|
211
|
+
bar_plot = BarWithDates(self.settings)
|
|
212
|
+
self.plot_on(bar_plot, data=data, x=x, y=y2)
|
|
213
|
+
|
|
214
|
+
# Add a vertical line
|
|
215
|
+
vline = VerticalDate(self.settings)
|
|
216
|
+
self.plot_on(vline, date=special_date, label="Important Event")
|
|
217
|
+
return self.fig, self.ax
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
## ๐ค Contributing
|
|
221
|
+
|
|
222
|
+
Contributions are welcome! Please feel free to submit a Pull Request.
|
|
223
|
+
|
|
224
|
+
---
|
|
225
|
+
|
|
226
|
+
<p align="center">
|
|
227
|
+
<em>GOAD๐ - When your data analysis is so fire๐ฅ it's got rizzโจ</em>
|
|
228
|
+
</p>
|