ripplerack 0.2.4__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. ripplerack-0.2.4/LICENSE +21 -0
  2. ripplerack-0.2.4/PKG-INFO +521 -0
  3. ripplerack-0.2.4/README.md +485 -0
  4. ripplerack-0.2.4/pyproject.toml +118 -0
  5. ripplerack-0.2.4/pyproject.toml.orig +90 -0
  6. ripplerack-0.2.4/src/ripplerack/__init__.py +10 -0
  7. ripplerack-0.2.4/src/ripplerack/cli.py +518 -0
  8. ripplerack-0.2.4/src/ripplerack/describe.py +93 -0
  9. ripplerack-0.2.4/src/ripplerack/diff.py +503 -0
  10. ripplerack-0.2.4/src/ripplerack/display.py +378 -0
  11. ripplerack-0.2.4/src/ripplerack/graph.py +118 -0
  12. ripplerack-0.2.4/src/ripplerack/impact.py +218 -0
  13. ripplerack-0.2.4/src/ripplerack/importers/__init__.py +54 -0
  14. ripplerack-0.2.4/src/ripplerack/importers/avro/__init__.py +74 -0
  15. ripplerack-0.2.4/src/ripplerack/importers/avro/logical.py +48 -0
  16. ripplerack-0.2.4/src/ripplerack/importers/avro/names.py +206 -0
  17. ripplerack-0.2.4/src/ripplerack/importers/avro/problems.py +16 -0
  18. ripplerack-0.2.4/src/ripplerack/importers/avro/reader.py +245 -0
  19. ripplerack-0.2.4/src/ripplerack/importers/base.py +115 -0
  20. ripplerack-0.2.4/src/ripplerack/importers/common.py +254 -0
  21. ripplerack-0.2.4/src/ripplerack/importers/json_schema/__init__.py +75 -0
  22. ripplerack-0.2.4/src/ripplerack/importers/json_schema/field_types.py +152 -0
  23. ripplerack-0.2.4/src/ripplerack/importers/json_schema/problems.py +16 -0
  24. ripplerack-0.2.4/src/ripplerack/importers/json_schema/reader.py +296 -0
  25. ripplerack-0.2.4/src/ripplerack/importers/json_schema/walker.py +208 -0
  26. ripplerack-0.2.4/src/ripplerack/importers/render.py +59 -0
  27. ripplerack-0.2.4/src/ripplerack/importers/sql/__init__.py +54 -0
  28. ripplerack-0.2.4/src/ripplerack/importers/sql/catalog.py +291 -0
  29. ripplerack-0.2.4/src/ripplerack/importers/sql/column_types.py +189 -0
  30. ripplerack-0.2.4/src/ripplerack/importers/sql/parse.py +64 -0
  31. ripplerack-0.2.4/src/ripplerack/importers/sql/problems.py +23 -0
  32. ripplerack-0.2.4/src/ripplerack/importers/sql/reader.py +168 -0
  33. ripplerack-0.2.4/src/ripplerack/markdown.py +503 -0
  34. ripplerack-0.2.4/src/ripplerack/mermaid.py +236 -0
  35. ripplerack-0.2.4/src/ripplerack/models.py +243 -0
  36. ripplerack-0.2.4/src/ripplerack/parser.py +121 -0
  37. ripplerack-0.2.4/src/ripplerack/py.typed +0 -0
  38. ripplerack-0.2.4/src/ripplerack/report.py +230 -0
  39. ripplerack-0.2.4/src/ripplerack/semver.py +224 -0
  40. ripplerack-0.2.4/src/ripplerack/simulate.py +161 -0
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 IzanVil
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,521 @@
1
+ Metadata-Version: 2.4
2
+ Name: ripplerack
3
+ Version: 0.2.4
4
+ Summary: Local-first datacenter & data pipeline simulator with semantic schema versioning and cascading impact analysis.
5
+ Keywords: datacenter,simulation,semver,data-pipelines,impact-analysis,schema
6
+ Author: IzanVil
7
+ Author-email: IzanVil <256081782+IzanVil@users.noreply.github.com>
8
+ License-Expression: MIT
9
+ License-File: LICENSE
10
+ Classifier: Development Status :: 3 - Alpha
11
+ Classifier: Environment :: Console
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Operating System :: OS Independent
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Programming Language :: Python :: 3 :: Only
16
+ Classifier: Programming Language :: Python :: 3.11
17
+ Classifier: Programming Language :: Python :: 3.12
18
+ Classifier: Programming Language :: Python :: 3.13
19
+ Classifier: Programming Language :: Python :: 3.14
20
+ Classifier: Topic :: Software Development :: Quality Assurance
21
+ Classifier: Topic :: System :: Systems Administration
22
+ Classifier: Typing :: Typed
23
+ Requires-Dist: networkx>=3.2
24
+ Requires-Dist: pydantic>=2.7
25
+ Requires-Dist: pyyaml>=6.0
26
+ Requires-Dist: rich>=13.7
27
+ Requires-Dist: sqlglot>=30.23,<31
28
+ Requires-Dist: typer>=0.12
29
+ Requires-Python: >=3.11
30
+ Project-URL: Homepage, https://github.com/IzanVil/RippleRack
31
+ Project-URL: Documentation, https://github.com/IzanVil/RippleRack/tree/main/docs
32
+ Project-URL: Source, https://github.com/IzanVil/RippleRack
33
+ Project-URL: Issues, https://github.com/IzanVil/RippleRack/issues
34
+ Project-URL: Changelog, https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md
35
+ Description-Content-Type: text/markdown
36
+
37
+ # RippleRack
38
+
39
+ **See the ripple before you make the splash.**
40
+
41
+ [![CI](https://github.com/IzanVil/RippleRack/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/IzanVil/RippleRack/actions/workflows/ci.yml)
42
+ [![Python versions](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)](https://github.com/IzanVil/RippleRack/blob/main/pyproject.toml)
43
+ [![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/IzanVil/RippleRack/blob/main/LICENSE)
44
+ [![Version](https://img.shields.io/badge/dynamic/toml?url=https%3A%2F%2Fraw.githubusercontent.com%2FIzanVil%2FRippleRack%2Fmain%2Fpyproject.toml&query=%24.project.version&label=version)](https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md)
45
+
46
+ RippleRack is a local-first simulator for datacenters and data pipelines. You
47
+ describe your racks, servers, services and pipelines in a single YAML file and
48
+ RippleRack tells you what happens when something changes:
49
+
50
+ - 🧬 **Semantic versioning for data**: diff two versions of a schema and get
51
+ `patch` / `minor` / `major`, the full list of changes and the version you
52
+ should have declared.
53
+ - 🌊 **Cascading impact analysis**: walk the dependency graph to find every
54
+ downstream pipeline, service and dashboard, which consumers actually break
55
+ and how severe the whole change is. Compare two versions of a whole
56
+ datacenter file to review a change before you merge it.
57
+ - 🏗️ **Infrastructure simulation**: push the load up, take a server down and
58
+ see what overloads and what goes offline with it.
59
+
60
+ Everything runs locally from a text file. There are no agents, no cloud
61
+ accounts and no databases.
62
+
63
+ > Status: **alpha**. The CLI and the engines work and are tested. The file format and the
64
+ > JSON output may still change before 1.0; JSON changes bump `schemaVersion` and are listed in
65
+ > the [changelog](https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md).
66
+
67
+ ## In 10 seconds
68
+
69
+ From a checkout (see [Installation](https://github.com/IzanVil/RippleRack#installation)):
70
+
71
+ ```bash
72
+ # What breaks if orders-ingest renames `amount` to `amount_cents`?
73
+ uv run ripplerack impact examples/mini_datacenter.yaml -p orders-ingest \
74
+ -s examples/changes/orders_v2.yaml --format json \
75
+ | jq -c '{severity, broken: [.brokenConsumers[].pipeline]}'
76
+ # {"severity":"critical","broken":["customer-360","orders-enriched"]}
77
+
78
+ # Gate a pull request on a change to the whole datacenter (exit code 2 if a consumer breaks)
79
+ uv run ripplerack diff examples/mini_datacenter.yaml examples/mini_datacenter_v2.yaml --fail-on-breaking
80
+ ```
81
+
82
+ Without `--format json`, `impact` prints tables for people:
83
+
84
+ ![Terminal: ripplerack impact on the example datacenter. Renaming amount to amount_cents in orders-ingest is a major change that breaks customer-360 and orders-enriched, puts api, revenue-daily and dashboard at risk, and the severity is CRITICAL.](https://raw.githubusercontent.com/IzanVil/RippleRack/main/docs/images/impact.svg)
85
+
86
+ ## Why?
87
+
88
+ Infrastructure tools model machines. Data catalogs model tables. Schema
89
+ registries model compatibility. In real life a renamed column on a Kafka topic
90
+ breaks a Spark job, which empties a dashboard, which pages someone at 3 AM.
91
+ RippleRack puts all of that in **one graph**, so the question *"what breaks if I
92
+ ship this?"* has a concrete answer. You can run it before merging, inside CI,
93
+ or just to learn how these systems fit together.
94
+
95
+ ## Installation
96
+
97
+ RippleRack requires Python 3.11+. Install it from [PyPI](https://pypi.org/project/ripplerack/):
98
+
99
+ ```bash
100
+ pip install ripplerack
101
+ ripplerack --help
102
+ ```
103
+
104
+ With [uv](https://docs.astral.sh/uv/), install it as a tool, or run it once without installing it:
105
+
106
+ ```bash
107
+ uv tool install ripplerack
108
+ uvx ripplerack --help
109
+ ```
110
+
111
+ To work on RippleRack itself, or to try what is not released yet, use a checkout:
112
+
113
+ ```bash
114
+ git clone https://github.com/IzanVil/RippleRack.git
115
+ cd RippleRack
116
+ uv sync
117
+ uv run ripplerack --help
118
+ ```
119
+
120
+ ## Documentation
121
+
122
+ | Guide | What you will find |
123
+ |-------|--------------------|
124
+ | [Datacenter file format](https://github.com/IzanVil/RippleRack/blob/main/docs/yaml-format.md) | The YAML you write and the validation rules |
125
+ | [Concepts](https://github.com/IzanVil/RippleRack/blob/main/docs/concepts.md) | The impact graph, the schema SemVer rules, severity, `diff` and `simulate` |
126
+ | [Output formats](https://github.com/IzanVil/RippleRack/blob/main/docs/output-formats.md) | The JSON documents, error codes, exit codes and Mermaid diagrams |
127
+ | [RippleRack in CI](https://github.com/IzanVil/RippleRack/blob/main/docs/ci.md) | The command as a gate, the Markdown report and the GitHub Action that comments on a pull request |
128
+ | [Importing schemas](https://github.com/IzanVil/RippleRack/blob/main/docs/import.md) | `ripplerack import`: turn a JSON Schema into a schema file, the translation rules, warnings and known limitations |
129
+ | [Importing Avro schemas](https://github.com/IzanVil/RippleRack/blob/main/docs/import-avro.md) | `import --from avro`: the same for an `.avsc` file, choosing one record of many, and its known limitations |
130
+ | [Importing SQL](https://github.com/IzanVil/RippleRack/blob/main/docs/import-sql.md) | `import --from sql`: the same for the `CREATE TABLE` of a dump, with a dialect, choosing one table of many, and its known limitations |
131
+ | [Changelog](https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md) | What changed in each version |
132
+ | [Contributing](https://github.com/IzanVil/RippleRack/blob/main/CONTRIBUTING.md) | Setup, checks and conventions |
133
+
134
+ ## Quick tour
135
+
136
+ The repository ships with [`examples/mini_datacenter.yaml`](https://github.com/IzanVil/RippleRack/blob/main/examples/mini_datacenter.yaml):
137
+ 2 racks, 3 servers, 5 services and 4 pipelines forming an order analytics
138
+ stack. The commands below are written as they run in a checkout (`uv run ripplerack ...`); if you
139
+ installed RippleRack with pip or `uv tool`, leave out `uv run` and download that file first.
140
+
141
+ ### `validate`: check the file
142
+
143
+ ```bash
144
+ uv run ripplerack validate examples/mini_datacenter.yaml
145
+ ```
146
+
147
+ ![Terminal: ripplerack validate on the example file says that mini-datacenter is valid, with 2 racks, 3 servers, 5 services, 4 pipelines, 4 explicit dependencies and a graph of 14 nodes and 19 edges.](https://raw.githubusercontent.com/IzanVil/RippleRack/main/docs/images/validate.svg)
148
+
149
+ This checks types, unique ids, references between components, the fields that
150
+ consumers read and dependency cycles. Typos in keys are reported instead of
151
+ being silently ignored.
152
+
153
+ ### `graph`: look at the dependency graph
154
+
155
+ ```bash
156
+ uv run ripplerack graph examples/mini_datacenter.yaml # tree in the terminal
157
+ uv run ripplerack graph examples/mini_datacenter.yaml -F dot -o dc.dot # Graphviz
158
+ uv run ripplerack graph examples/mini_datacenter.yaml -F json # machine readable
159
+ uv run ripplerack graph examples/mini_datacenter.yaml -F mermaid # Mermaid flowchart
160
+ ```
161
+
162
+ The default is a tree in the terminal:
163
+
164
+ ![Terminal: ripplerack graph prints the example datacenter as a tree of racks, servers and services, then its pipelines with the fields each one reads, then its explicit dependencies.](https://raw.githubusercontent.com/IzanVil/RippleRack/main/docs/images/graph.svg)
165
+
166
+ `-F mermaid` prints a [Mermaid](https://mermaid.js.org) flowchart: racks and servers are
167
+ nested boxes, services and pipelines have their own shapes, and arrows are labelled with
168
+ the relation. Paste it between `mermaid` code fences in any Markdown file (GitHub renders
169
+ it) or save it with `-o graph.mmd` and render it with `mmdc`.
170
+
171
+ ### `impact`: what breaks if this schema changes?
172
+
173
+ ```bash
174
+ uv run ripplerack impact examples/mini_datacenter.yaml \
175
+ --pipeline orders-ingest \
176
+ --schema examples/changes/orders_v2.yaml
177
+ ```
178
+
179
+ ```text
180
+ ─────────────────────── Impact of changing orders-ingest ───────────────────────
181
+ Schema orders 1.2.0 → 2.0.0 change: major suggested version: 2.0.0
182
+ Schema changes
183
+ ┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
184
+ ┃ level ┃ kind ┃ detail ┃
185
+ ┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
186
+ │ major │ field_removed │ field 'amount' removed │
187
+ │ major │ field_added │ required field 'amount_cents' added │
188
+ └───────┴───────────────┴─────────────────────────────────────┘
189
+ Broken consumers
190
+ ┏━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
191
+ ┃ pipeline ┃ fields ┃ reason ┃
192
+ ┡━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
193
+ │ orders-enriched │ amount │ field 'amount' removed │
194
+ │ customer-360 │ amount │ field 'amount' removed │
195
+ └─────────────────┴────────┴────────────────────────┘
196
+ Downstream components
197
+ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
198
+ ┃ component ┃ kind ┃ hops ┃ path ┃
199
+ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
200
+ │ ✗ customer-360 │ pipeline │ 1 │ orders-ingest → customer-360 │
201
+ │ ✗ orders-enriched │ pipeline │ 1 │ orders-ingest → orders-enriched │
202
+ │ ! api │ service │ 2 │ orders-ingest → customer-360 → api │
203
+ │ ! revenue-daily │ pipeline │ 2 │ orders-ingest → orders-enriched → revenue-daily │
204
+ │ ! dashboard │ service │ 3 │ orders-ingest → orders-enriched → revenue-daily → dashboard │
205
+ └───────────────────┴──────────┴──────┴─────────────────────────────────────────────────────────────┘
206
+ Severity: CRITICAL
207
+ ```
208
+
209
+ Try the other examples in [`examples/changes/`](https://github.com/IzanVil/RippleRack/tree/main/examples/changes):
210
+
211
+ | file | what it shows | severity |
212
+ |------|---------------|----------|
213
+ | `orders_v1_3_0.yaml` | optional field added | `low` |
214
+ | `orders_sneaky_patch.yaml` | field dropped but shipped as a *patch*: RippleRack warns that the version is too low | `high` |
215
+ | `orders_v2.yaml` | field renamed with a type change, so the break cascades | `critical` |
216
+
217
+ Add `--fail-on-breaking` to exit with code `2` whenever a consumer breaks. You
218
+ can use it as a gate in CI.
219
+
220
+ #### Diagram of the impact
221
+
222
+ `--format mermaid` draws the whole graph with the impact highlighted: the changed pipeline
223
+ (purple), the consumers that break (red), everything downstream of them that is at risk
224
+ (orange), and the arrows along the way. This is the `orders_v2.yaml` change from above:
225
+
226
+ ```mermaid
227
+ flowchart LR
228
+ %% mini-datacenter
229
+ %% impact of changing orders-ingest: critical
230
+ subgraph rack_rack_a["rack-a"]
231
+ subgraph srv_srv_a1["srv-a1"]
232
+ svc_postgres("postgres")
233
+ svc_kafka("kafka")
234
+ end
235
+ subgraph srv_srv_a2["srv-a2"]
236
+ svc_spark("spark")
237
+ end
238
+ end
239
+ subgraph rack_rack_b["rack-b"]
240
+ subgraph srv_srv_b1["srv-b1"]
241
+ svc_api("api")
242
+ svc_dashboard("dashboard")
243
+ end
244
+ end
245
+ subgraph grp_pipelines["pipelines"]
246
+ pl_orders_ingest{{"orders-ingest<br/>schema 1.2.0"}}
247
+ pl_orders_enriched{{"orders-enriched<br/>schema 1.0.0"}}
248
+ pl_revenue_daily{{"revenue-daily<br/>schema 2.1.0"}}
249
+ pl_customer_360{{"customer-360<br/>schema 0.4.0"}}
250
+ end
251
+ svc_postgres -->|runs| pl_customer_360
252
+ svc_postgres -->|runtime| svc_api
253
+ svc_kafka -->|runs| pl_orders_ingest
254
+ svc_kafka -->|network| svc_spark
255
+ svc_spark -->|runs| pl_orders_enriched
256
+ svc_spark -->|runs| pl_revenue_daily
257
+ pl_orders_ingest -->|feeds| pl_orders_enriched
258
+ pl_orders_ingest -->|feeds| pl_customer_360
259
+ pl_orders_enriched -->|feeds| pl_revenue_daily
260
+ pl_revenue_daily -->|data| svc_dashboard
261
+ pl_customer_360 -->|data| svc_api
262
+ classDef service fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
263
+ class svc_postgres,svc_kafka,svc_spark service
264
+ classDef changed fill:#ede7f6,stroke:#4527a0,stroke-width:4px,color:#311b92
265
+ class pl_orders_ingest changed
266
+ classDef broken fill:#ffcdd2,stroke:#c62828,stroke-width:3px,color:#b71c1c
267
+ class pl_orders_enriched,pl_customer_360 broken
268
+ classDef atRisk fill:#ffe0b2,stroke:#ef6c00,stroke-width:2px,color:#e65100
269
+ class svc_api,svc_dashboard,pl_revenue_daily atRisk
270
+ style srv_srv_a1 fill:#e8eaf6,stroke:#283593,color:#1a237e
271
+ style srv_srv_a2 fill:#e8eaf6,stroke:#283593,color:#1a237e
272
+ style rack_rack_a fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
273
+ style srv_srv_b1 fill:#e8eaf6,stroke:#283593,color:#1a237e
274
+ style rack_rack_b fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
275
+ linkStyle 6,7 stroke:#c62828,stroke-width:3px
276
+ linkStyle 8,9,10 stroke:#ef6c00,stroke-width:3px
277
+ ```
278
+
279
+ #### JSON output for CI
280
+
281
+ `--format json` (`-F json`) prints a stable, versioned JSON document instead of
282
+ tables, ready for `jq` or any CI tooling:
283
+
284
+ ```bash
285
+ uv run ripplerack impact examples/mini_datacenter.yaml -p orders-ingest \
286
+ -s examples/changes/orders_v2.yaml --format json \
287
+ | jq -c '{severity, broken: [.brokenConsumers[].pipeline]}'
288
+ ```
289
+
290
+ ```json
291
+ {"severity":"critical","broken":["customer-360","orders-enriched"]}
292
+ ```
293
+
294
+ Every document starts with an envelope (`schemaVersion`, `tool`, `command`) and
295
+ all keys are camelCase. If the command fails, stdout stays empty and stderr gets a
296
+ JSON error with a stable `error.code`, such as `file_not_found` or
297
+ `unknown_pipeline`. The full contract, the error codes and the exit codes are
298
+ documented in [`docs/output-formats.md`](https://github.com/IzanVil/RippleRack/blob/main/docs/output-formats.md).
299
+
300
+ #### Markdown for pull requests
301
+
302
+ `--format markdown` (`-F markdown`) writes a report to paste in the comment of a pull request:
303
+ the schema change and the consumers that break in tables, what is at risk, the long lists folded
304
+ and a diagram of the impact that GitHub draws.
305
+
306
+ ```bash
307
+ uv run ripplerack impact examples/mini_datacenter.yaml -p orders-ingest \
308
+ -s examples/changes/orders_v2.yaml -F markdown
309
+ ```
310
+
311
+ Names that come from your files are written as code, so none of them can mention anybody or break
312
+ the comment, and the report is shortened to fit the 65,536 characters GitHub allows. See
313
+ [`docs/output-formats.md`](https://github.com/IzanVil/RippleRack/blob/main/docs/output-formats.md#markdown) for the layout and its options, and
314
+ [`docs/ci.md`](https://github.com/IzanVil/RippleRack/blob/main/docs/ci.md#the-github-action) for the GitHub Action that posts it on every pull
315
+ request and keeps one comment up to date.
316
+
317
+ ### `diff`: what does this change do to my datacenter?
318
+
319
+ ```bash
320
+ uv run ripplerack diff examples/mini_datacenter.yaml examples/mini_datacenter_v2.yaml
321
+ ```
322
+
323
+ Compares two datacenter files and reports what was added, removed and modified, which
324
+ dependencies changed, and the impact of every schema change on the pipelines that read
325
+ it. Components are matched by `id`, so reordering a file is not a change.
326
+
327
+ ```text
328
+ ──────────── Diff examples/mini_datacenter.yaml → examples/mini_datacenter_v2.yaml ─────────────
329
+ mini-datacenter → mini-datacenter
330
+ 2 added · 0 removed · 3 modified · 1 schema change(s) · 1 broken consumer(s)
331
+ Components
332
+ ┏━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
333
+ ┃ ┃ kind ┃ id ┃ details ┃
334
+ ┡━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
335
+ │ + │ service │ redis │ on srv-b1 │
336
+ │ + │ pipeline │ orders-audit │ runs on postgres │
337
+ │ ~ │ server │ srv-b1 │ memory_gb: 96 → 64 │
338
+ │ ~ │ pipeline │ orders-enriched │ inputs: orders-ingest: -amount +amount_cents │
339
+ │ ~ │ pipeline │ orders-ingest │ schema.version: 1.2.0 → 2.0.0 │
340
+ │ │ │ │ schema.fields: changed (see below) │
341
+ └───┴──────────┴─────────────────┴──────────────────────────────────────────────┘
342
+ Dependencies
343
+ ┏━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
344
+ ┃ ┃ dependency ┃ kind ┃
345
+ ┡━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
346
+ │ + │ api → redis │ runtime │
347
+ │ ~ │ spark → kafka │ network → data │
348
+ └───┴───────────────┴────────────────┘
349
+
350
+ orders-ingest
351
+ Schema orders 1.2.0 → 2.0.0 change: major suggested version: 2.0.0
352
+ Schema changes
353
+ ┏━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
354
+ ┃ level ┃ kind ┃ detail ┃
355
+ ┡━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
356
+ │ major │ field_removed │ field 'amount' removed │
357
+ │ major │ field_added │ required field 'amount_cents' added │
358
+ └───────┴───────────────┴─────────────────────────────────────┘
359
+ Broken consumers
360
+ ┏━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
361
+ ┃ pipeline ┃ fields ┃ reason ┃
362
+ ┡━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
363
+ │ customer-360 │ amount │ field 'amount' removed │
364
+ └──────────────┴────────┴────────────────────────┘
365
+ At risk: api
366
+ Severity: HIGH
367
+ ```
368
+
369
+ Here `orders-enriched` was updated in the same change to read `amount_cents`, so it is
370
+ fine. `customer-360` was not touched and reads every field, so it breaks, and `api`,
371
+ which depends on it, is at risk. `orders-audit` is new and was written against the new
372
+ schema, so it is never reported.
373
+
374
+ Add `--format json` for the [documented JSON](https://github.com/IzanVil/RippleRack/blob/main/docs/output-formats.md#diff) or
375
+ `--fail-on-breaking` to exit with code `2` when a consumer breaks, which makes the
376
+ command a good pull request gate:
377
+
378
+ ```bash
379
+ git show main:datacenter.yaml > /tmp/base.yaml
380
+ uv run ripplerack diff /tmp/base.yaml datacenter.yaml --fail-on-breaking
381
+ ```
382
+
383
+ ### `import`: write the schema from a JSON Schema, an Avro schema or a SQL table
384
+
385
+ ```bash
386
+ uv run ripplerack import examples/schemas/orders_v2.schema.json --from json-schema --version 2.0.0
387
+ ```
388
+
389
+ ```yaml
390
+ name: orders
391
+ version: 2.0.0
392
+ fields:
393
+ - {name: order_id, type: integer}
394
+ - {name: customer_id, type: integer}
395
+ - {name: amount_cents, type: integer, description: Order total in cents}
396
+ - {name: currency, type: string}
397
+ - {name: created_at, type: timestamp}
398
+ - {name: coupon_code, type: string, nullable: true}
399
+ ```
400
+
401
+ The output is a schema file, so nobody has to write it by hand and it goes straight into
402
+ `impact`. This one is exactly `examples/changes/orders_v2.yaml`, so the report is the same
403
+ `critical` one as above:
404
+
405
+ ```bash
406
+ uv run ripplerack impact examples/mini_datacenter.yaml --pipeline orders-ingest \
407
+ --schema <(uv run ripplerack import examples/schemas/orders_v2.schema.json --from json-schema --version 2.0.0)
408
+ ```
409
+
410
+ Properties that `required` lists become required fields and the others become `nullable`;
411
+ references inside the file are followed; objects and arrays become `json` fields. What cannot
412
+ be represented is an error, or with `--lenient` a `json` field and a warning, and
413
+ `additionalProperties: false` and `unevaluatedProperties: false` are reported as warnings. The
414
+ [import guide](https://github.com/IzanVil/RippleRack/blob/main/docs/import.md) has the translation rules and the
415
+ [known limitations](https://github.com/IzanVil/RippleRack/blob/main/docs/import.md#known-limitations).
416
+
417
+ The same schema written in Avro gives the same file:
418
+
419
+ ```bash
420
+ uv run ripplerack import examples/schemas/orders_v2.avsc --from avro --version 2.0.0
421
+ ```
422
+
423
+ A union with `null` becomes `nullable`, a logical type such as `timestamp-millis` becomes
424
+ `timestamp`, an enum keeps its symbols in the description, and a file that defines several
425
+ records is told which to read with `--select`. A union of several types is an error, or with
426
+ `--lenient` a `json` field and a warning. The [Avro guide](https://github.com/IzanVil/RippleRack/blob/main/docs/import-avro.md) has the rules and
427
+ its [known limitations](https://github.com/IzanVil/RippleRack/blob/main/docs/import-avro.md#known-limitations).
428
+
429
+ So does a table of a SQL file:
430
+
431
+ ```bash
432
+ uv run ripplerack import examples/schemas/orders_v2.sql --from sql --version 2.0.0
433
+ ```
434
+
435
+ A column is `nullable` unless it is `NOT NULL`, part of the primary key or a counter; a `DECIMAL`
436
+ with no decimals is an `integer`; the comment of a column is its description, and an enum keeps
437
+ its values there. A dump with many tables is told which to read with `--select`, and a file that
438
+ is not standard SQL which `--dialect` it is written in (`postgres`, `mysql`, `sqlite`,
439
+ `snowflake`...). A type with no equivalent, such as a geometry, is an error, or with `--lenient`
440
+ a `json` field and a warning. The [SQL guide](https://github.com/IzanVil/RippleRack/blob/main/docs/import-sql.md) has the rules and its
441
+ [known limitations](https://github.com/IzanVil/RippleRack/blob/main/docs/import-sql.md#known-limitations).
442
+
443
+ ### `simulate`: load surges and failures
444
+
445
+ ```bash
446
+ uv run ripplerack simulate examples/mini_datacenter.yaml --load 1.5
447
+ uv run ripplerack simulate examples/mini_datacenter.yaml --load 1.5 --fail srv-a2
448
+ ```
449
+
450
+ The second command, with `srv-a2` taken down:
451
+
452
+ ![Terminal: ripplerack simulate with the load at 1.5 and srv-a2 failed. srv-a1 is at warning with 94% of its CPU and memory, srv-a2 is failed, and spark, orders-enriched, revenue-daily and dashboard are unavailable as a consequence.](https://raw.githubusercontent.com/IzanVil/RippleRack/main/docs/images/simulate.svg)
453
+
454
+ `--load` scales every service's CPU/memory demand and every server's power draw.
455
+ `--fail` (repeatable) takes a component down along with everything that depends
456
+ on it. Each rack and server is then reported as `ok`, `warning` (≥ 80%),
457
+ `overloaded` (> 100%) or `failed`.
458
+
459
+ ## How it works
460
+
461
+ ```mermaid
462
+ flowchart LR
463
+ yaml[/"datacenter.yaml<br/>(one or two files)"/] --> parse["parse and validate<br/>Pydantic models"]
464
+ parse --> dag["dependency graph<br/>NetworkX"]
465
+ jsonschema[/"JSON Schema"/] -->|"import"| schema
466
+ avro[/"Avro schema"/] -->|"import"| schema
467
+ sql[/"SQL file"/] -->|"import"| schema
468
+ schema[/"new schema.yaml"/] --> semver["schema SemVer diff<br/>patch, minor, major"]
469
+ dag --> impact["impact<br/>blast radius, broken consumers, severity"]
470
+ semver --> impact
471
+ dag --> diff["diff<br/>added, removed, modified, impact"]
472
+ semver --> diff
473
+ dag --> simulate["simulate<br/>load and failures"]
474
+ impact --> out(["tables, JSON, Mermaid"])
475
+ diff --> out
476
+ simulate --> out
477
+ ```
478
+
479
+ Each module has one job; the [concepts guide](https://github.com/IzanVil/RippleRack/blob/main/docs/concepts.md) explains the ideas behind them.
480
+
481
+ | module | responsibility |
482
+ |--------|----------------|
483
+ | `models.py` | Pydantic v2 models (`Datacenter`, `Rack`, `Server`, `Service`, `Pipeline`, `Schema`, `SchemaField`, `Dependency`) and reference validation |
484
+ | `parser.py` | YAML loading, user-facing errors, cycle detection |
485
+ | `graph.py` | Graph construction plus DOT/JSON export |
486
+ | `mermaid.py` | Mermaid flowchart export, with optional impact highlighting |
487
+ | `semver.py` | Schema diff, change levels, version bumping |
488
+ | `impact.py` | Blast radius, broken consumers, severity |
489
+ | `diff.py` | Comparison of two datacenter files: components, dependencies, schema impact |
490
+ | `importers/` | Schema importers (JSON Schema, Avro and SQL) and the YAML writer for what they read |
491
+ | `report.py` | Versioned JSON documents for CI (impact and diff results, errors) |
492
+ | `simulate.py` | Capacity and failure simulation |
493
+ | `display.py` | Rich tables and trees for the text output of the commands |
494
+ | `cli.py` | Typer command line: loads the input, calls the engine, prints the result |
495
+
496
+ ## Development
497
+
498
+ ```bash
499
+ uv sync
500
+ uv run pytest # tests
501
+ uv run ruff check . # lint
502
+ uv run ruff format --check . # formatting ("uv run ruff format ." fixes it)
503
+ uv run mypy src tests # strict type checking
504
+ ```
505
+
506
+ CI runs all of the above on Python 3.11, 3.12, 3.13 and 3.14.
507
+
508
+ ## Roadmap
509
+
510
+ - Throughput and latency modelling for pipelines
511
+ - Web UI to explore the graph
512
+
513
+ Ideas and contributions are welcome. Please open an issue to discuss them.
514
+ See [CONTRIBUTING.md](https://github.com/IzanVil/RippleRack/blob/main/CONTRIBUTING.md) for the development workflow and
515
+ [CHANGELOG.md](https://github.com/IzanVil/RippleRack/blob/main/CHANGELOG.md) for release notes. Everyone taking part is expected to follow
516
+ the [Code of Conduct](https://github.com/IzanVil/RippleRack/blob/main/CODE_OF_CONDUCT.md); please report vulnerabilities privately, as
517
+ described in [SECURITY.md](https://github.com/IzanVil/RippleRack/blob/main/SECURITY.md).
518
+
519
+ ## License
520
+
521
+ [MIT](https://github.com/IzanVil/RippleRack/blob/main/LICENSE)