ripplerack 0.2.4__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ripplerack-0.2.4/LICENSE +21 -0
- ripplerack-0.2.4/PKG-INFO +521 -0
- ripplerack-0.2.4/README.md +485 -0
- ripplerack-0.2.4/pyproject.toml +118 -0
- ripplerack-0.2.4/pyproject.toml.orig +90 -0
- ripplerack-0.2.4/src/ripplerack/__init__.py +10 -0
- ripplerack-0.2.4/src/ripplerack/cli.py +518 -0
- ripplerack-0.2.4/src/ripplerack/describe.py +93 -0
- ripplerack-0.2.4/src/ripplerack/diff.py +503 -0
- ripplerack-0.2.4/src/ripplerack/display.py +378 -0
- ripplerack-0.2.4/src/ripplerack/graph.py +118 -0
- ripplerack-0.2.4/src/ripplerack/impact.py +218 -0
- ripplerack-0.2.4/src/ripplerack/importers/__init__.py +54 -0
- ripplerack-0.2.4/src/ripplerack/importers/avro/__init__.py +74 -0
- ripplerack-0.2.4/src/ripplerack/importers/avro/logical.py +48 -0
- ripplerack-0.2.4/src/ripplerack/importers/avro/names.py +206 -0
- ripplerack-0.2.4/src/ripplerack/importers/avro/problems.py +16 -0
- ripplerack-0.2.4/src/ripplerack/importers/avro/reader.py +245 -0
- ripplerack-0.2.4/src/ripplerack/importers/base.py +115 -0
- ripplerack-0.2.4/src/ripplerack/importers/common.py +254 -0
- ripplerack-0.2.4/src/ripplerack/importers/json_schema/__init__.py +75 -0
- ripplerack-0.2.4/src/ripplerack/importers/json_schema/field_types.py +152 -0
- ripplerack-0.2.4/src/ripplerack/importers/json_schema/problems.py +16 -0
- ripplerack-0.2.4/src/ripplerack/importers/json_schema/reader.py +296 -0
- ripplerack-0.2.4/src/ripplerack/importers/json_schema/walker.py +208 -0
- ripplerack-0.2.4/src/ripplerack/importers/render.py +59 -0
- ripplerack-0.2.4/src/ripplerack/importers/sql/__init__.py +54 -0
- ripplerack-0.2.4/src/ripplerack/importers/sql/catalog.py +291 -0
- ripplerack-0.2.4/src/ripplerack/importers/sql/column_types.py +189 -0
- ripplerack-0.2.4/src/ripplerack/importers/sql/parse.py +64 -0
- ripplerack-0.2.4/src/ripplerack/importers/sql/problems.py +23 -0
- ripplerack-0.2.4/src/ripplerack/importers/sql/reader.py +168 -0
- ripplerack-0.2.4/src/ripplerack/markdown.py +503 -0
- ripplerack-0.2.4/src/ripplerack/mermaid.py +236 -0
- ripplerack-0.2.4/src/ripplerack/models.py +243 -0
- ripplerack-0.2.4/src/ripplerack/parser.py +121 -0
- ripplerack-0.2.4/src/ripplerack/py.typed +0 -0
- ripplerack-0.2.4/src/ripplerack/report.py +230 -0
- ripplerack-0.2.4/src/ripplerack/semver.py +224 -0
- ripplerack-0.2.4/src/ripplerack/simulate.py +161 -0
ripplerack-0.2.4/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 IzanVil
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,521 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ripplerack
|
|
3
|
+
Version: 0.2.4
|
|
4
|
+
Summary: Local-first datacenter & data pipeline simulator with semantic schema versioning and cascading impact analysis.
|
|
5
|
+
Keywords: datacenter,simulation,semver,data-pipelines,impact-analysis,schema
|
|
6
|
+
Author: IzanVil
|
|
7
|
+
Author-email: IzanVil <256081782+IzanVil@users.noreply.github.com>
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Environment :: Console
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
20
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
21
|
+
Classifier: Topic :: System :: Systems Administration
|
|
22
|
+
Classifier: Typing :: Typed
|
|
23
|
+
Requires-Dist: networkx>=3.2
|
|
24
|
+
Requires-Dist: pydantic>=2.7
|
|
25
|
+
Requires-Dist: pyyaml>=6.0
|
|
26
|
+
Requires-Dist: rich>=13.7
|
|
27
|
+
Requires-Dist: sqlglot>=30.23,<31
|
|
28
|
+
Requires-Dist: typer>=0.12
|
|
29
|
+
Requires-Python: >=3.11
|
|
30
|
+
Project-URL: Homepage, https://github.com/IzanVil/RippleRack
|
|
31
|
+
Project-URL: Documentation, https://github.com/IzanVil/RippleRack/tree/main/docs
|
|
32
|
+
Project-URL: Source, https://github.com/IzanVil/RippleRack
|
|
33
|
+
Project-URL: Issues, https://github.com/IzanVil/RippleRack/issues
|
|
34
|
+
Project-URL: Changelog, https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md
|
|
35
|
+
Description-Content-Type: text/markdown
|
|
36
|
+
|
|
37
|
+
# RippleRack
|
|
38
|
+
|
|
39
|
+
**See the ripple before you make the splash.**
|
|
40
|
+
|
|
41
|
+
[](https://github.com/IzanVil/RippleRack/actions/workflows/ci.yml)
|
|
42
|
+
[](https://github.com/IzanVil/RippleRack/blob/main/pyproject.toml)
|
|
43
|
+
[](https://github.com/IzanVil/RippleRack/blob/main/LICENSE)
|
|
44
|
+
[](https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md)
|
|
45
|
+
|
|
46
|
+
RippleRack is a local-first simulator for datacenters and data pipelines. You
|
|
47
|
+
describe your racks, servers, services and pipelines in a single YAML file and
|
|
48
|
+
RippleRack tells you what happens when something changes:
|
|
49
|
+
|
|
50
|
+
- 🧬 **Semantic versioning for data**: diff two versions of a schema and get
|
|
51
|
+
`patch` / `minor` / `major`, the full list of changes and the version you
|
|
52
|
+
should have declared.
|
|
53
|
+
- 🌊 **Cascading impact analysis**: walk the dependency graph to find every
|
|
54
|
+
downstream pipeline, service and dashboard, which consumers actually break
|
|
55
|
+
and how severe the whole change is. Compare two versions of a whole
|
|
56
|
+
datacenter file to review a change before you merge it.
|
|
57
|
+
- 🏗️ **Infrastructure simulation**: push the load up, take a server down and
|
|
58
|
+
see what overloads and what goes offline with it.
|
|
59
|
+
|
|
60
|
+
Everything runs locally from a text file. There are no agents, no cloud
|
|
61
|
+
accounts and no databases.
|
|
62
|
+
|
|
63
|
+
> Status: **alpha**. The CLI and the engines work and are tested. The file format and the
|
|
64
|
+
> JSON output may still change before 1.0; JSON changes bump `schemaVersion` and are listed in
|
|
65
|
+
> the [changelog](https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md).
|
|
66
|
+
|
|
67
|
+
## In 10 seconds
|
|
68
|
+
|
|
69
|
+
From a checkout (see [Installation](https://github.com/IzanVil/RippleRack#installation)):
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
# What breaks if orders-ingest renames `amount` to `amount_cents`?
|
|
73
|
+
uv run ripplerack impact examples/mini_datacenter.yaml -p orders-ingest \
|
|
74
|
+
-s examples/changes/orders_v2.yaml --format json \
|
|
75
|
+
| jq -c '{severity, broken: [.brokenConsumers[].pipeline]}'
|
|
76
|
+
# {"severity":"critical","broken":["customer-360","orders-enriched"]}
|
|
77
|
+
|
|
78
|
+
# Gate a pull request on a change to the whole datacenter (exit code 2 if a consumer breaks)
|
|
79
|
+
uv run ripplerack diff examples/mini_datacenter.yaml examples/mini_datacenter_v2.yaml --fail-on-breaking
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Without `--format json`, `impact` prints tables for people:
|
|
83
|
+
|
|
84
|
+

|
|
85
|
+
|
|
86
|
+
## Why?
|
|
87
|
+
|
|
88
|
+
Infrastructure tools model machines. Data catalogs model tables. Schema
|
|
89
|
+
registries model compatibility. In real life a renamed column on a Kafka topic
|
|
90
|
+
breaks a Spark job, which empties a dashboard, which pages someone at 3 AM.
|
|
91
|
+
RippleRack puts all of that in **one graph**, so the question *"what breaks if I
|
|
92
|
+
ship this?"* has a concrete answer. You can run it before merging, inside CI,
|
|
93
|
+
or just to learn how these systems fit together.
|
|
94
|
+
|
|
95
|
+
## Installation
|
|
96
|
+
|
|
97
|
+
RippleRack requires Python 3.11+. Install it from [PyPI](https://pypi.org/project/ripplerack/):
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
pip install ripplerack
|
|
101
|
+
ripplerack --help
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
With [uv](https://docs.astral.sh/uv/), install it as a tool, or run it once without installing it:
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
uv tool install ripplerack
|
|
108
|
+
uvx ripplerack --help
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
To work on RippleRack itself, or to try what is not released yet, use a checkout:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
git clone https://github.com/IzanVil/RippleRack.git
|
|
115
|
+
cd RippleRack
|
|
116
|
+
uv sync
|
|
117
|
+
uv run ripplerack --help
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
## Documentation
|
|
121
|
+
|
|
122
|
+
| Guide | What you will find |
|
|
123
|
+
|-------|--------------------|
|
|
124
|
+
| [Datacenter file format](https://github.com/IzanVil/RippleRack/blob/main/docs/yaml-format.md) | The YAML you write and the validation rules |
|
|
125
|
+
| [Concepts](https://github.com/IzanVil/RippleRack/blob/main/docs/concepts.md) | The impact graph, the schema SemVer rules, severity, `diff` and `simulate` |
|
|
126
|
+
| [Output formats](https://github.com/IzanVil/RippleRack/blob/main/docs/output-formats.md) | The JSON documents, error codes, exit codes and Mermaid diagrams |
|
|
127
|
+
| [RippleRack in CI](https://github.com/IzanVil/RippleRack/blob/main/docs/ci.md) | The command as a gate, the Markdown report and the GitHub Action that comments on a pull request |
|
|
128
|
+
| [Importing schemas](https://github.com/IzanVil/RippleRack/blob/main/docs/import.md) | `ripplerack import`: turn a JSON Schema into a schema file, the translation rules, warnings and known limitations |
|
|
129
|
+
| [Importing Avro schemas](https://github.com/IzanVil/RippleRack/blob/main/docs/import-avro.md) | `import --from avro`: the same for an `.avsc` file, choosing one record of many, and its known limitations |
|
|
130
|
+
| [Importing SQL](https://github.com/IzanVil/RippleRack/blob/main/docs/import-sql.md) | `import --from sql`: the same for the `CREATE TABLE` of a dump, with a dialect, choosing one table of many, and its known limitations |
|
|
131
|
+
| [Changelog](https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md) | What changed in each version |
|
|
132
|
+
| [Contributing](https://github.com/IzanVil/RippleRack/blob/main/CONTRIBUTING.md) | Setup, checks and conventions |
|
|
133
|
+
|
|
134
|
+
## Quick tour
|
|
135
|
+
|
|
136
|
+
The repository ships with [`examples/mini_datacenter.yaml`](https://github.com/IzanVil/RippleRack/blob/main/examples/mini_datacenter.yaml):
|
|
137
|
+
2 racks, 3 servers, 5 services and 4 pipelines forming an order analytics
|
|
138
|
+
stack. The commands below are written as they run in a checkout (`uv run ripplerack ...`); if you
|
|
139
|
+
installed RippleRack with pip or `uv tool`, leave out `uv run` and download that file first.
|
|
140
|
+
|
|
141
|
+
### `validate`: check the file
|
|
142
|
+
|
|
143
|
+
```bash
|
|
144
|
+
uv run ripplerack validate examples/mini_datacenter.yaml
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+

|
|
148
|
+
|
|
149
|
+
This checks types, unique ids, references between components, the fields that
|
|
150
|
+
consumers read and dependency cycles. Typos in keys are reported instead of
|
|
151
|
+
being silently ignored.
|
|
152
|
+
|
|
153
|
+
### `graph`: look at the dependency graph
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
uv run ripplerack graph examples/mini_datacenter.yaml # tree in the terminal
|
|
157
|
+
uv run ripplerack graph examples/mini_datacenter.yaml -F dot -o dc.dot # Graphviz
|
|
158
|
+
uv run ripplerack graph examples/mini_datacenter.yaml -F json # machine readable
|
|
159
|
+
uv run ripplerack graph examples/mini_datacenter.yaml -F mermaid # Mermaid flowchart
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
The default is a tree in the terminal:
|
|
163
|
+
|
|
164
|
+

|
|
165
|
+
|
|
166
|
+
`-F mermaid` prints a [Mermaid](https://mermaid.js.org) flowchart: racks and servers are
|
|
167
|
+
nested boxes, services and pipelines have their own shapes, and arrows are labelled with
|
|
168
|
+
the relation. Paste it between `mermaid` code fences in any Markdown file (GitHub renders
|
|
169
|
+
it) or save it with `-o graph.mmd` and render it with `mmdc`.
|
|
170
|
+
|
|
171
|
+
### `impact`: what breaks if this schema changes?
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
uv run ripplerack impact examples/mini_datacenter.yaml \
|
|
175
|
+
--pipeline orders-ingest \
|
|
176
|
+
--schema examples/changes/orders_v2.yaml
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
```text
|
|
180
|
+
─────────────────────── Impact of changing orders-ingest ───────────────────────
|
|
181
|
+
Schema orders 1.2.0 → 2.0.0 change: major suggested version: 2.0.0
|
|
182
|
+
Schema changes
|
|
183
|
+
┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
184
|
+
┃ level ┃ kind ┃ detail ┃
|
|
185
|
+
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
|
|
186
|
+
│ major │ field_removed │ field 'amount' removed │
|
|
187
|
+
│ major │ field_added │ required field 'amount_cents' added │
|
|
188
|
+
└───────┴───────────────┴─────────────────────────────────────┘
|
|
189
|
+
Broken consumers
|
|
190
|
+
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
191
|
+
┃ pipeline ┃ fields ┃ reason ┃
|
|
192
|
+
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
|
|
193
|
+
│ orders-enriched │ amount │ field 'amount' removed │
|
|
194
|
+
│ customer-360 │ amount │ field 'amount' removed │
|
|
195
|
+
└─────────────────┴────────┴────────────────────────┘
|
|
196
|
+
Downstream components
|
|
197
|
+
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
198
|
+
┃ component ┃ kind ┃ hops ┃ path ┃
|
|
199
|
+
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
|
|
200
|
+
│ ✗ customer-360 │ pipeline │ 1 │ orders-ingest → customer-360 │
|
|
201
|
+
│ ✗ orders-enriched │ pipeline │ 1 │ orders-ingest → orders-enriched │
|
|
202
|
+
│ ! api │ service │ 2 │ orders-ingest → customer-360 → api │
|
|
203
|
+
│ ! revenue-daily │ pipeline │ 2 │ orders-ingest → orders-enriched → revenue-daily │
|
|
204
|
+
│ ! dashboard │ service │ 3 │ orders-ingest → orders-enriched → revenue-daily → dashboard │
|
|
205
|
+
└───────────────────┴──────────┴──────┴─────────────────────────────────────────────────────────────┘
|
|
206
|
+
Severity: CRITICAL
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
Try the other examples in [`examples/changes/`](https://github.com/IzanVil/RippleRack/tree/main/examples/changes):
|
|
210
|
+
|
|
211
|
+
| file | what it shows | severity |
|
|
212
|
+
|------|---------------|----------|
|
|
213
|
+
| `orders_v1_3_0.yaml` | optional field added | `low` |
|
|
214
|
+
| `orders_sneaky_patch.yaml` | field dropped but shipped as a *patch*: RippleRack warns that the version is too low | `high` |
|
|
215
|
+
| `orders_v2.yaml` | field renamed with a type change, so the break cascades | `critical` |
|
|
216
|
+
|
|
217
|
+
Add `--fail-on-breaking` to exit with code `2` whenever a consumer breaks. You
|
|
218
|
+
can use it as a gate in CI.
|
|
219
|
+
|
|
220
|
+
#### Diagram of the impact
|
|
221
|
+
|
|
222
|
+
`--format mermaid` draws the whole graph with the impact highlighted: the changed pipeline
|
|
223
|
+
(purple), the consumers that break (red), everything downstream of them that is at risk
|
|
224
|
+
(orange), and the arrows along the way. This is the `orders_v2.yaml` change from above:
|
|
225
|
+
|
|
226
|
+
```mermaid
|
|
227
|
+
flowchart LR
|
|
228
|
+
%% mini-datacenter
|
|
229
|
+
%% impact of changing orders-ingest: critical
|
|
230
|
+
subgraph rack_rack_a["rack-a"]
|
|
231
|
+
subgraph srv_srv_a1["srv-a1"]
|
|
232
|
+
svc_postgres("postgres")
|
|
233
|
+
svc_kafka("kafka")
|
|
234
|
+
end
|
|
235
|
+
subgraph srv_srv_a2["srv-a2"]
|
|
236
|
+
svc_spark("spark")
|
|
237
|
+
end
|
|
238
|
+
end
|
|
239
|
+
subgraph rack_rack_b["rack-b"]
|
|
240
|
+
subgraph srv_srv_b1["srv-b1"]
|
|
241
|
+
svc_api("api")
|
|
242
|
+
svc_dashboard("dashboard")
|
|
243
|
+
end
|
|
244
|
+
end
|
|
245
|
+
subgraph grp_pipelines["pipelines"]
|
|
246
|
+
pl_orders_ingest{{"orders-ingest<br/>schema 1.2.0"}}
|
|
247
|
+
pl_orders_enriched{{"orders-enriched<br/>schema 1.0.0"}}
|
|
248
|
+
pl_revenue_daily{{"revenue-daily<br/>schema 2.1.0"}}
|
|
249
|
+
pl_customer_360{{"customer-360<br/>schema 0.4.0"}}
|
|
250
|
+
end
|
|
251
|
+
svc_postgres -->|runs| pl_customer_360
|
|
252
|
+
svc_postgres -->|runtime| svc_api
|
|
253
|
+
svc_kafka -->|runs| pl_orders_ingest
|
|
254
|
+
svc_kafka -->|network| svc_spark
|
|
255
|
+
svc_spark -->|runs| pl_orders_enriched
|
|
256
|
+
svc_spark -->|runs| pl_revenue_daily
|
|
257
|
+
pl_orders_ingest -->|feeds| pl_orders_enriched
|
|
258
|
+
pl_orders_ingest -->|feeds| pl_customer_360
|
|
259
|
+
pl_orders_enriched -->|feeds| pl_revenue_daily
|
|
260
|
+
pl_revenue_daily -->|data| svc_dashboard
|
|
261
|
+
pl_customer_360 -->|data| svc_api
|
|
262
|
+
classDef service fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
|
|
263
|
+
class svc_postgres,svc_kafka,svc_spark service
|
|
264
|
+
classDef changed fill:#ede7f6,stroke:#4527a0,stroke-width:4px,color:#311b92
|
|
265
|
+
class pl_orders_ingest changed
|
|
266
|
+
classDef broken fill:#ffcdd2,stroke:#c62828,stroke-width:3px,color:#b71c1c
|
|
267
|
+
class pl_orders_enriched,pl_customer_360 broken
|
|
268
|
+
classDef atRisk fill:#ffe0b2,stroke:#ef6c00,stroke-width:2px,color:#e65100
|
|
269
|
+
class svc_api,svc_dashboard,pl_revenue_daily atRisk
|
|
270
|
+
style srv_srv_a1 fill:#e8eaf6,stroke:#283593,color:#1a237e
|
|
271
|
+
style srv_srv_a2 fill:#e8eaf6,stroke:#283593,color:#1a237e
|
|
272
|
+
style rack_rack_a fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
|
|
273
|
+
style srv_srv_b1 fill:#e8eaf6,stroke:#283593,color:#1a237e
|
|
274
|
+
style rack_rack_b fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
|
|
275
|
+
linkStyle 6,7 stroke:#c62828,stroke-width:3px
|
|
276
|
+
linkStyle 8,9,10 stroke:#ef6c00,stroke-width:3px
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
#### JSON output for CI
|
|
280
|
+
|
|
281
|
+
`--format json` (`-F json`) prints a stable, versioned JSON document instead of
|
|
282
|
+
tables, ready for `jq` or any CI tooling:
|
|
283
|
+
|
|
284
|
+
```bash
|
|
285
|
+
uv run ripplerack impact examples/mini_datacenter.yaml -p orders-ingest \
|
|
286
|
+
-s examples/changes/orders_v2.yaml --format json \
|
|
287
|
+
| jq -c '{severity, broken: [.brokenConsumers[].pipeline]}'
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
```json
|
|
291
|
+
{"severity":"critical","broken":["customer-360","orders-enriched"]}
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
Every document starts with an envelope (`schemaVersion`, `tool`, `command`) and
|
|
295
|
+
all keys are camelCase. If the command fails, stdout stays empty and stderr gets a
|
|
296
|
+
JSON error with a stable `error.code`, such as `file_not_found` or
|
|
297
|
+
`unknown_pipeline`. The full contract, the error codes and the exit codes are
|
|
298
|
+
documented in [`docs/output-formats.md`](https://github.com/IzanVil/RippleRack/blob/main/docs/output-formats.md).
|
|
299
|
+
|
|
300
|
+
#### Markdown for pull requests
|
|
301
|
+
|
|
302
|
+
`--format markdown` (`-F markdown`) writes a report to paste in the comment of a pull request:
|
|
303
|
+
the schema change and the consumers that break in tables, what is at risk, the long lists folded
|
|
304
|
+
and a diagram of the impact that GitHub draws.
|
|
305
|
+
|
|
306
|
+
```bash
|
|
307
|
+
uv run ripplerack impact examples/mini_datacenter.yaml -p orders-ingest \
|
|
308
|
+
-s examples/changes/orders_v2.yaml -F markdown
|
|
309
|
+
```
|
|
310
|
+
|
|
311
|
+
Names that come from your files are written as code, so none of them can mention anybody or break
|
|
312
|
+
the comment, and the report is shortened to fit the 65,536 characters GitHub allows. See
|
|
313
|
+
[`docs/output-formats.md`](https://github.com/IzanVil/RippleRack/blob/main/docs/output-formats.md#markdown) for the layout and its options, and
|
|
314
|
+
[`docs/ci.md`](https://github.com/IzanVil/RippleRack/blob/main/docs/ci.md#the-github-action) for the GitHub Action that posts it on every pull
|
|
315
|
+
request and keeps one comment up to date.
|
|
316
|
+
|
|
317
|
+
### `diff`: what does this change do to my datacenter?
|
|
318
|
+
|
|
319
|
+
```bash
|
|
320
|
+
uv run ripplerack diff examples/mini_datacenter.yaml examples/mini_datacenter_v2.yaml
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
Compares two datacenter files and reports what was added, removed and modified, which
|
|
324
|
+
dependencies changed, and the impact of every schema change on the pipelines that read
|
|
325
|
+
it. Components are matched by `id`, so reordering a file is not a change.
|
|
326
|
+
|
|
327
|
+
```text
|
|
328
|
+
──────────── Diff examples/mini_datacenter.yaml → examples/mini_datacenter_v2.yaml ─────────────
|
|
329
|
+
mini-datacenter → mini-datacenter
|
|
330
|
+
2 added · 0 removed · 3 modified · 1 schema change(s) · 1 broken consumer(s)
|
|
331
|
+
Components
|
|
332
|
+
┏━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
333
|
+
┃ ┃ kind ┃ id ┃ details ┃
|
|
334
|
+
┡━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
|
|
335
|
+
│ + │ service │ redis │ on srv-b1 │
|
|
336
|
+
│ + │ pipeline │ orders-audit │ runs on postgres │
|
|
337
|
+
│ ~ │ server │ srv-b1 │ memory_gb: 96 → 64 │
|
|
338
|
+
│ ~ │ pipeline │ orders-enriched │ inputs: orders-ingest: -amount +amount_cents │
|
|
339
|
+
│ ~ │ pipeline │ orders-ingest │ schema.version: 1.2.0 → 2.0.0 │
|
|
340
|
+
│ │ │ │ schema.fields: changed (see below) │
|
|
341
|
+
└───┴──────────┴─────────────────┴──────────────────────────────────────────────┘
|
|
342
|
+
Dependencies
|
|
343
|
+
┏━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
|
|
344
|
+
┃ ┃ dependency ┃ kind ┃
|
|
345
|
+
┡━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
|
|
346
|
+
│ + │ api → redis │ runtime │
|
|
347
|
+
│ ~ │ spark → kafka │ network → data │
|
|
348
|
+
└───┴───────────────┴────────────────┘
|
|
349
|
+
|
|
350
|
+
orders-ingest
|
|
351
|
+
Schema orders 1.2.0 → 2.0.0 change: major suggested version: 2.0.0
|
|
352
|
+
Schema changes
|
|
353
|
+
┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
354
|
+
┃ level ┃ kind ┃ detail ┃
|
|
355
|
+
┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
|
|
356
|
+
│ major │ field_removed │ field 'amount' removed │
|
|
357
|
+
│ major │ field_added │ required field 'amount_cents' added │
|
|
358
|
+
└───────┴───────────────┴─────────────────────────────────────┘
|
|
359
|
+
Broken consumers
|
|
360
|
+
┏━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
|
|
361
|
+
┃ pipeline ┃ fields ┃ reason ┃
|
|
362
|
+
┡━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
|
|
363
|
+
│ customer-360 │ amount │ field 'amount' removed │
|
|
364
|
+
└──────────────┴────────┴────────────────────────┘
|
|
365
|
+
At risk: api
|
|
366
|
+
Severity: HIGH
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
Here `orders-enriched` was updated in the same change to read `amount_cents`, so it is
|
|
370
|
+
fine. `customer-360` was not touched and reads every field, so it breaks, and `api`,
|
|
371
|
+
which depends on it, is at risk. `orders-audit` is new and was written against the new
|
|
372
|
+
schema, so it is never reported.
|
|
373
|
+
|
|
374
|
+
Add `--format json` for the [documented JSON](https://github.com/IzanVil/RippleRack/blob/main/docs/output-formats.md#diff) or
|
|
375
|
+
`--fail-on-breaking` to exit with code `2` when a consumer breaks, which makes the
|
|
376
|
+
command a good pull request gate:
|
|
377
|
+
|
|
378
|
+
```bash
|
|
379
|
+
git show main:datacenter.yaml > /tmp/base.yaml
|
|
380
|
+
uv run ripplerack diff /tmp/base.yaml datacenter.yaml --fail-on-breaking
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
### `import`: write the schema from a JSON Schema, an Avro schema or a SQL table
|
|
384
|
+
|
|
385
|
+
```bash
|
|
386
|
+
uv run ripplerack import examples/schemas/orders_v2.schema.json --from json-schema --version 2.0.0
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
```yaml
|
|
390
|
+
name: orders
|
|
391
|
+
version: 2.0.0
|
|
392
|
+
fields:
|
|
393
|
+
- {name: order_id, type: integer}
|
|
394
|
+
- {name: customer_id, type: integer}
|
|
395
|
+
- {name: amount_cents, type: integer, description: Order total in cents}
|
|
396
|
+
- {name: currency, type: string}
|
|
397
|
+
- {name: created_at, type: timestamp}
|
|
398
|
+
- {name: coupon_code, type: string, nullable: true}
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
The output is a schema file, so nobody has to write it by hand and it goes straight into
|
|
402
|
+
`impact`. This one is exactly `examples/changes/orders_v2.yaml`, so the report is the same
|
|
403
|
+
`critical` one as above:
|
|
404
|
+
|
|
405
|
+
```bash
|
|
406
|
+
uv run ripplerack impact examples/mini_datacenter.yaml --pipeline orders-ingest \
|
|
407
|
+
--schema <(uv run ripplerack import examples/schemas/orders_v2.schema.json --from json-schema --version 2.0.0)
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
Properties that `required` lists become required fields and the others become `nullable`;
|
|
411
|
+
references inside the file are followed; objects and arrays become `json` fields. What cannot
|
|
412
|
+
be represented is an error, or with `--lenient` a `json` field and a warning, and
|
|
413
|
+
`additionalProperties: false` and `unevaluatedProperties: false` are reported as warnings. The
|
|
414
|
+
[import guide](https://github.com/IzanVil/RippleRack/blob/main/docs/import.md) has the translation rules and the
|
|
415
|
+
[known limitations](https://github.com/IzanVil/RippleRack/blob/main/docs/import.md#known-limitations).
|
|
416
|
+
|
|
417
|
+
The same schema written in Avro gives the same file:
|
|
418
|
+
|
|
419
|
+
```bash
|
|
420
|
+
uv run ripplerack import examples/schemas/orders_v2.avsc --from avro --version 2.0.0
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
A union with `null` becomes `nullable`, a logical type such as `timestamp-millis` becomes
|
|
424
|
+
`timestamp`, an enum keeps its symbols in the description, and a file that defines several
|
|
425
|
+
records is told which to read with `--select`. A union of several types is an error, or with
|
|
426
|
+
`--lenient` a `json` field and a warning. The [Avro guide](https://github.com/IzanVil/RippleRack/blob/main/docs/import-avro.md) has the rules and
|
|
427
|
+
its [known limitations](https://github.com/IzanVil/RippleRack/blob/main/docs/import-avro.md#known-limitations).
|
|
428
|
+
|
|
429
|
+
So does a table of a SQL file:
|
|
430
|
+
|
|
431
|
+
```bash
|
|
432
|
+
uv run ripplerack import examples/schemas/orders_v2.sql --from sql --version 2.0.0
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
A column is `nullable` unless it is `NOT NULL`, part of the primary key or a counter; a `DECIMAL`
|
|
436
|
+
with no decimals is an `integer`; the comment of a column is its description, and an enum keeps
|
|
437
|
+
its values there. A dump with many tables is told which to read with `--select`, and a file that
|
|
438
|
+
is not standard SQL which `--dialect` it is written in (`postgres`, `mysql`, `sqlite`,
|
|
439
|
+
`snowflake`...). A type with no equivalent, such as a geometry, is an error, or with `--lenient`
|
|
440
|
+
a `json` field and a warning. The [SQL guide](https://github.com/IzanVil/RippleRack/blob/main/docs/import-sql.md) has the rules and its
|
|
441
|
+
[known limitations](https://github.com/IzanVil/RippleRack/blob/main/docs/import-sql.md#known-limitations).
|
|
442
|
+
|
|
443
|
+
### `simulate`: load surges and failures
|
|
444
|
+
|
|
445
|
+
```bash
|
|
446
|
+
uv run ripplerack simulate examples/mini_datacenter.yaml --load 1.5
|
|
447
|
+
uv run ripplerack simulate examples/mini_datacenter.yaml --load 1.5 --fail srv-a2
|
|
448
|
+
```
|
|
449
|
+
|
|
450
|
+
The second command, with `srv-a2` taken down:
|
|
451
|
+
|
|
452
|
+

|
|
453
|
+
|
|
454
|
+
`--load` scales every service's CPU/memory demand and every server's power draw.
|
|
455
|
+
`--fail` (repeatable) takes a component down along with everything that depends
|
|
456
|
+
on it. Each rack and server is then reported as `ok`, `warning` (≥ 80%),
|
|
457
|
+
`overloaded` (> 100%) or `failed`.
|
|
458
|
+
|
|
459
|
+
## How it works
|
|
460
|
+
|
|
461
|
+
```mermaid
|
|
462
|
+
flowchart LR
|
|
463
|
+
yaml[/"datacenter.yaml<br/>(one or two files)"/] --> parse["parse and validate<br/>Pydantic models"]
|
|
464
|
+
parse --> dag["dependency graph<br/>NetworkX"]
|
|
465
|
+
jsonschema[/"JSON Schema"/] -->|"import"| schema
|
|
466
|
+
avro[/"Avro schema"/] -->|"import"| schema
|
|
467
|
+
sql[/"SQL file"/] -->|"import"| schema
|
|
468
|
+
schema[/"new schema.yaml"/] --> semver["schema SemVer diff<br/>patch, minor, major"]
|
|
469
|
+
dag --> impact["impact<br/>blast radius, broken consumers, severity"]
|
|
470
|
+
semver --> impact
|
|
471
|
+
dag --> diff["diff<br/>added, removed, modified, impact"]
|
|
472
|
+
semver --> diff
|
|
473
|
+
dag --> simulate["simulate<br/>load and failures"]
|
|
474
|
+
impact --> out(["tables, JSON, Mermaid"])
|
|
475
|
+
diff --> out
|
|
476
|
+
simulate --> out
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
Each module has one job; the [concepts guide](https://github.com/IzanVil/RippleRack/blob/main/docs/concepts.md) explains the ideas behind them.
|
|
480
|
+
|
|
481
|
+
| module | responsibility |
|
|
482
|
+
|--------|----------------|
|
|
483
|
+
| `models.py` | Pydantic v2 models (`Datacenter`, `Rack`, `Server`, `Service`, `Pipeline`, `Schema`, `SchemaField`, `Dependency`) and reference validation |
|
|
484
|
+
| `parser.py` | YAML loading, user-facing errors, cycle detection |
|
|
485
|
+
| `graph.py` | Graph construction plus DOT/JSON export |
|
|
486
|
+
| `mermaid.py` | Mermaid flowchart export, with optional impact highlighting |
|
|
487
|
+
| `semver.py` | Schema diff, change levels, version bumping |
|
|
488
|
+
| `impact.py` | Blast radius, broken consumers, severity |
|
|
489
|
+
| `diff.py` | Comparison of two datacenter files: components, dependencies, schema impact |
|
|
490
|
+
| `importers/` | Schema importers (JSON Schema, Avro and SQL) and the YAML writer for what they read |
|
|
491
|
+
| `report.py` | Versioned JSON documents for CI (impact and diff results, errors) |
|
|
492
|
+
| `simulate.py` | Capacity and failure simulation |
|
|
493
|
+
| `display.py` | Rich tables and trees for the text output of the commands |
|
|
494
|
+
| `cli.py` | Typer command line: loads the input, calls the engine, prints the result |
|
|
495
|
+
|
|
496
|
+
## Development
|
|
497
|
+
|
|
498
|
+
```bash
|
|
499
|
+
uv sync
|
|
500
|
+
uv run pytest # tests
|
|
501
|
+
uv run ruff check . # lint
|
|
502
|
+
uv run ruff format --check . # formatting ("uv run ruff format ." fixes it)
|
|
503
|
+
uv run mypy src tests # strict type checking
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
CI runs all of the above on Python 3.11, 3.12, 3.13 and 3.14.
|
|
507
|
+
|
|
508
|
+
## Roadmap
|
|
509
|
+
|
|
510
|
+
- Throughput and latency modelling for pipelines
|
|
511
|
+
- Web UI to explore the graph
|
|
512
|
+
|
|
513
|
+
Ideas and contributions are welcome. Please open an issue to discuss them.
|
|
514
|
+
See [CONTRIBUTING.md](https://github.com/IzanVil/RippleRack/blob/main/CONTRIBUTING.md) for the development workflow and
|
|
515
|
+
[CHANGELOG.md](https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md) for release notes. Everyone taking part is expected to follow
|
|
516
|
+
the [Code of Conduct](https://github.com/IzanVil/RippleRack/blob/main/CODE_OF_CONDUCT.md); please report vulnerabilities privately, as
|
|
517
|
+
described in [SECURITY.md](https://github.com/IzanVil/RippleRack/blob/main/SECURITY.md).
|
|
518
|
+
|
|
519
|
+
## License
|
|
520
|
+
|
|
521
|
+
[MIT](https://github.com/IzanVil/RippleRack/blob/main/LICENSE)
|