datadog-structurizr 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- datadog_structurizr-0.1.0/LICENSE +21 -0
- datadog_structurizr-0.1.0/MANIFEST.in +4 -0
- datadog_structurizr-0.1.0/PKG-INFO +521 -0
- datadog_structurizr-0.1.0/README.md +496 -0
- datadog_structurizr-0.1.0/examples/README.md +24 -0
- datadog_structurizr-0.1.0/examples/checkout-web/c4.toml +36 -0
- datadog_structurizr-0.1.0/examples/checkout-web/raw/definition.json +13 -0
- datadog_structurizr-0.1.0/examples/checkout-web/raw/dependencies-checkout-worker.json +9 -0
- datadog_structurizr-0.1.0/examples/checkout-web/raw/dependencies.json +17 -0
- datadog_structurizr-0.1.0/examples/checkout-web/raw/resources.json +26 -0
- datadog_structurizr-0.1.0/examples/checkout-web/raw/types.json +11 -0
- datadog_structurizr-0.1.0/pyproject.toml +43 -0
- datadog_structurizr-0.1.0/setup.cfg +4 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr/__init__.py +3 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr/client.py +263 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr/config.py +183 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr/emitter.py +212 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr/main.py +119 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr/mapper.py +260 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr/model.py +58 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr/render.py +39 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr.egg-info/PKG-INFO +521 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr.egg-info/SOURCES.txt +31 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr.egg-info/dependency_links.txt +1 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr.egg-info/entry_points.txt +2 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr.egg-info/requires.txt +8 -0
- datadog_structurizr-0.1.0/src/datadog_structurizr.egg-info/top_level.txt +1 -0
- datadog_structurizr-0.1.0/tests/conftest.py +50 -0
- datadog_structurizr-0.1.0/tests/test_client.py +234 -0
- datadog_structurizr-0.1.0/tests/test_config.py +146 -0
- datadog_structurizr-0.1.0/tests/test_emitter.py +100 -0
- datadog_structurizr-0.1.0/tests/test_examples.py +92 -0
- datadog_structurizr-0.1.0/tests/test_mapper.py +206 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 veritas-sovereign
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,521 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: datadog-structurizr
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Generate C4 system context, container and component diagrams of a Datadog APM service as Structurizr DSL and Mermaid.
|
|
5
|
+
Author: Amit Kumar
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/veritas-sovereign/datadog-structurizr
|
|
8
|
+
Project-URL: Changelog, https://github.com/veritas-sovereign/datadog-structurizr/blob/main/CHANGELOG.md
|
|
9
|
+
Keywords: c4,c4-model,datadog,apm,structurizr,mermaid,architecture,diagrams
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Environment :: Console
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Topic :: Software Development :: Documentation
|
|
16
|
+
Requires-Python: >=3.9
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
License-File: LICENSE
|
|
19
|
+
Requires-Dist: requests>=2.31
|
|
20
|
+
Requires-Dist: python-dotenv>=1.0
|
|
21
|
+
Requires-Dist: tomli>=1.1; python_version < "3.11"
|
|
22
|
+
Provides-Extra: dev
|
|
23
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
<div align="center">
|
|
27
|
+
|
|
28
|
+
# datadog-structurizr
|
|
29
|
+
|
|
30
|
+
### Generate C4 diagrams of a Datadog APM service as Structurizr DSL and Mermaid
|
|
31
|
+
|
|
32
|
+
<br>
|
|
33
|
+
|
|
34
|
+
[](https://github.com/veritas-sovereign/datadog-structurizr#readme)
|
|
35
|
+
|
|
36
|
+
<br>
|
|
37
|
+
|
|
38
|
+
🚀 [Quick Start](#quick-start) · 📦 [PyPI](https://pypi.org/project/datadog-structurizr/) · 🔑 [Datadog Access](#datadog-access) · 📋 [Mapping Conventions](#mapping-conventions) · ⚠️ [Limitations](#limitations) · 🧪 [Examples](examples/README.md) · 📝 [Changelog](CHANGELOG.md)
|
|
39
|
+
|
|
40
|
+
<br>
|
|
41
|
+
|
|
42
|
+
[](https://pypi.org/project/datadog-structurizr/)
|
|
43
|
+
[](LICENSE)
|
|
44
|
+
[](pyproject.toml)
|
|
45
|
+
[](https://docs.datadoghq.com/tracing/)
|
|
46
|
+
[](https://docs.structurizr.com/dsl)
|
|
47
|
+
[](https://mermaid.js.org/syntax/c4.html)
|
|
48
|
+
[](https://github.com/veritas-sovereign/datadog-structurizr/actions/workflows/test.yml)
|
|
49
|
+
[](https://github.com/veritas-sovereign/datadog-structurizr/commits)
|
|
50
|
+
|
|
51
|
+
</div>
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## Overview
|
|
56
|
+
|
|
57
|
+
`datadog-structurizr` reads one service's topology from Datadog APM and writes three C4 views of it:
|
|
58
|
+
|
|
59
|
+
| Here | C4 model | C4 view | Shows | Built from |
|
|
60
|
+
| --- | --- | --- | --- | --- |
|
|
61
|
+
| L0 | Level 1 | System Context | the user, the software system containing the service, the services it calls and is called by | service dependencies |
|
|
62
|
+
| L1 | Level 2 | Container | the service, the other services you name as part of the system, and their datastores inside the system boundary, with the neighbouring systems | service dependencies |
|
|
63
|
+
| L2 | Level 3 | Component | the service's entry points: HTTP endpoint groups and handlers | span resources |
|
|
64
|
+
|
|
65
|
+
The [C4 model](https://c4model.com/) numbers these views 1, 2 and 3. This project names them L0, L1 and L2, as do the view names and file names it writes.
|
|
66
|
+
|
|
67
|
+
L2 shows the API surface that APM sees, not the code structure behind it. Treat its components as a first draft of the service's entry points, not as its controllers, services and repositories.
|
|
68
|
+
|
|
69
|
+
Each run writes:
|
|
70
|
+
|
|
71
|
+
- a [Structurizr DSL](https://docs.structurizr.com/dsl) workspace with all three views. The generated model and views go into two files that every run rewrites; `workspace.dsl` includes them, is created once, and is yours to edit
|
|
72
|
+
- one [Mermaid C4](https://mermaid.js.org/syntax/c4.html) diagram per view, which renders on GitHub and in VS Code, and as SVG with `mmdc`
|
|
73
|
+
- the raw Datadog responses, so you can generate the diagrams again with `--offline` and no API calls
|
|
74
|
+
|
|
75
|
+
It is meant as a starting point for architecture documentation. Keep `workspace.dsl` as the source you maintain, and run the tool again whenever the services change: your edits survive.
|
|
76
|
+
|
|
77
|
+
## Quick Start
|
|
78
|
+
|
|
79
|
+
With Python (3.9 or later):
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
pip install datadog-structurizr
|
|
83
|
+
export DD_API_KEY=... DD_APP_KEY=... DD_SITE=datadoghq.com
|
|
84
|
+
datadog-structurizr --service checkout-web --env prod
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
For a web component made of several services, name the system and the other services in it:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
datadog-structurizr --service checkout-web --env prod --system-name Checkout --include checkout-worker
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
To try it without Datadog keys, download [`examples/checkout-web/`](examples/checkout-web) and run, from inside that folder:
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
datadog-structurizr -c c4.toml --offline -o .
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Open the output folder in [Structurizr local](#viewing-locally), paste `workspace-inline.dsl` (one file, nothing to include) into the [Structurizr playground](https://playground.structurizr.com), or open the `.mmd` files in any Mermaid viewer.
|
|
100
|
+
|
|
101
|
+
## Installation
|
|
102
|
+
|
|
103
|
+
Pick one:
|
|
104
|
+
|
|
105
|
+
| Option | Best for | You need |
|
|
106
|
+
| --- | --- | --- |
|
|
107
|
+
| [PyPI](#from-pypi) | everyday use | Python 3.9+; optionally `mmdc` (Node.js) to render SVGs |
|
|
108
|
+
| [From source](#from-source) | changing the tool or running its tests | Python 3.9+, git |
|
|
109
|
+
| [Docker](#docker) | running without Python | Docker |
|
|
110
|
+
|
|
111
|
+
You also need a Datadog API key and application key. See [Datadog access](#datadog-access).
|
|
112
|
+
|
|
113
|
+
### From PyPI
|
|
114
|
+
|
|
115
|
+
[`datadog-structurizr`](https://pypi.org/project/datadog-structurizr/) is published to PyPI for every release.
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
python3 -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
|
|
119
|
+
pip install datadog-structurizr # latest release
|
|
120
|
+
pip install datadog-structurizr==0.1.0 # or a specific version
|
|
121
|
+
datadog-structurizr --help
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
To render the diagrams as SVG, also install the Mermaid CLI:
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
npm install -g @mermaid-js/mermaid-cli
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
### From source
|
|
131
|
+
|
|
132
|
+
1. Clone the repository and enter it:
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
git clone https://github.com/veritas-sovereign/datadog-structurizr.git
|
|
136
|
+
cd datadog-structurizr
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
2. Create and activate a virtual environment. `.venv/` is already git-ignored.
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
python3 -m venv .venv
|
|
143
|
+
source .venv/bin/activate # Windows: .venv\Scripts\activate
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
3. Install the package in editable mode. Add `[dev]` to also install pytest.
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
pip install -e . # or: pip install -e '.[dev]'
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
4. Check the install:
|
|
153
|
+
|
|
154
|
+
```bash
|
|
155
|
+
datadog-structurizr --version
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
To use the code without installing it, run `pip install -r requirements.txt` and prefix commands with `PYTHONPATH=src`.
|
|
159
|
+
|
|
160
|
+
### Docker
|
|
161
|
+
|
|
162
|
+
Each release publishes an image with the tool and no renderers to `ghcr.io/veritas-sovereign/datadog-structurizr`, for amd64 and arm64, tagged with the version (`0.1.0`) and the minor version (`0.1`). It is built from the [`Dockerfile`](Dockerfile); `docker build -t datadog-structurizr .` builds the same image from a clone.
|
|
163
|
+
|
|
164
|
+
Run it in the folder that holds `c4.toml`. `--user` makes the output files yours, and the keys come from the environment:
|
|
165
|
+
|
|
166
|
+
```bash
|
|
167
|
+
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/work" \
|
|
168
|
+
-e DD_API_KEY -e DD_APP_KEY \
|
|
169
|
+
ghcr.io/veritas-sovereign/datadog-structurizr:0.1.0 -c c4.toml -o . --no-render
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
The image has no `mmdc` or structurizr-cli, so use `--no-render`, and [view](#viewing-locally) or validate the output with the `structurizr/structurizr` image.
|
|
173
|
+
|
|
174
|
+
## Usage
|
|
175
|
+
|
|
176
|
+
### Generate the diagrams
|
|
177
|
+
|
|
178
|
+
```bash
|
|
179
|
+
datadog-structurizr [-c c4.toml] [--service NAME] [--env ENV] [--hours N] [-o DIR]
|
|
180
|
+
[--system-name NAME] [--include SERVICE[,SERVICE]]... [--no-person]
|
|
181
|
+
[--offline] [--no-render] [--no-input]
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
| Option | Meaning |
|
|
185
|
+
| --- | --- |
|
|
186
|
+
| `-c`, `--config` | [config file](#config-file) with the service, system boundary, classification and person |
|
|
187
|
+
| `--service` | APM service name |
|
|
188
|
+
| `--env` | APM environment, such as `prod`. The dependencies API requires it |
|
|
189
|
+
| `--hours` | lookback window in hours for dependencies and span resources (default 24) |
|
|
190
|
+
| `-o`, `--output` | output directory (default `output`) |
|
|
191
|
+
| `--system-name` | name of the software system (default: the service name) |
|
|
192
|
+
| `--include` | other APM services that are containers of the same system. Comma-separated, and repeatable. Adds to `include` in the config file |
|
|
193
|
+
| `--no-person` | draw no person; for services only called by other services |
|
|
194
|
+
| `--offline` | rebuild from `<output>/raw/*.json` instead of calling Datadog; no keys needed |
|
|
195
|
+
| `--no-render` | write `.dsl` and `.mmd` sources only; do not render images |
|
|
196
|
+
| `--no-input` | never prompt (see [Prompts](#prompts)) |
|
|
197
|
+
| `--version` | print the version |
|
|
198
|
+
|
|
199
|
+
### What to tell the tool
|
|
200
|
+
|
|
201
|
+
Only the service and environment are required. The other inputs fix the cases where Datadog data alone gives the wrong picture:
|
|
202
|
+
|
|
203
|
+
| Input | Why you would set it | Flag | Config file |
|
|
204
|
+
| --- | --- | --- | --- |
|
|
205
|
+
| Service and environment | required | `--service`, `--env` | `service`, `env` |
|
|
206
|
+
| Other services in the same system | A web component is often several services (web, BFF, worker). Without this, each one is drawn as a separate system and the container view is almost empty | `--include` | `[system] include` |
|
|
207
|
+
| System name | The system is usually called something other than its main service, for example `Checkout` rather than `checkout-web` | `--system-name` | `[system] name`, `description` |
|
|
208
|
+
| Classification | A dependency whose name does not match the [name hints](#elements) ends up in the wrong group | — | `[classify] datastores`, `external`, `internal` |
|
|
209
|
+
| Services to leave out | Telemetry collectors, agents and sidecars clutter every view | — | `ignore` |
|
|
210
|
+
| The person | The default `User` is generic. Name the real user, or turn the person off for a service that is only called by other services | `--no-person` | `[person] enabled`, `name`, `description` |
|
|
211
|
+
|
|
212
|
+
### Config file
|
|
213
|
+
|
|
214
|
+
Keep the inputs in a TOML file next to the generated `workspace.dsl`, and commit it, so a later run gives the same diagrams. A complete example is [`examples/checkout-web/c4.toml`](examples/checkout-web/c4.toml):
|
|
215
|
+
|
|
216
|
+
```toml
|
|
217
|
+
service = "checkout-web"
|
|
218
|
+
env = "prod"
|
|
219
|
+
hours = 24
|
|
220
|
+
ignore = ["otel-collector", "datadog-agent*"]
|
|
221
|
+
|
|
222
|
+
[system]
|
|
223
|
+
name = "Checkout"
|
|
224
|
+
description = "Lets shoppers review their cart and pay."
|
|
225
|
+
include = ["checkout-worker"]
|
|
226
|
+
|
|
227
|
+
[person]
|
|
228
|
+
enabled = true
|
|
229
|
+
name = "Shopper"
|
|
230
|
+
description = "Buys products on the web shop."
|
|
231
|
+
|
|
232
|
+
[classify]
|
|
233
|
+
datastores = ["ledger"]
|
|
234
|
+
external = []
|
|
235
|
+
internal = []
|
|
236
|
+
|
|
237
|
+
[components]
|
|
238
|
+
max = 12
|
|
239
|
+
strip_prefixes = []
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
| Key | Type | Meaning |
|
|
243
|
+
| --- | --- | --- |
|
|
244
|
+
| `service`, `env` | string | APM service and environment |
|
|
245
|
+
| `hours` | integer | lookback window (default 24) |
|
|
246
|
+
| `site` | string | Datadog site (default `datadoghq.com`) |
|
|
247
|
+
| `output` | string | output directory (default `output`) |
|
|
248
|
+
| `ignore` | list | services left out of every view |
|
|
249
|
+
| `[system] name`, `description` | string | the software system's name and description |
|
|
250
|
+
| `[system] include` | list | other APM services that are containers of this system. Exact names; each one costs one more API call |
|
|
251
|
+
| `[person] enabled` | boolean | draw a person (default `true`) |
|
|
252
|
+
| `[person] name`, `description` | string | the person (default `User`) |
|
|
253
|
+
| `[classify] datastores`, `external`, `internal` | list | force a dependency into a group, overriding the name hints |
|
|
254
|
+
| `[components] max` | integer | most components shown in L2, the last being `Other` (default 12, at least 2) |
|
|
255
|
+
| `[components] strip_prefixes` | list | path prefixes removed before routes are grouped, such as an app's context path `"/shop"` |
|
|
256
|
+
|
|
257
|
+
`ignore` and the `[classify]` lists accept `fnmatch` patterns (`*`, `?`, `[abc]`) and ignore case. Top-level keys such as `ignore` must come before the first `[table]`, as TOML requires. Unknown keys and wrong types stop the run with an error naming the key, so a typo cannot be silently ignored. Datadog keys are refused in the config file.
|
|
258
|
+
|
|
259
|
+
### Environment
|
|
260
|
+
|
|
261
|
+
Datadog keys only come from the environment. Environment variables can also come from a `.env` file in the current directory. Copy [`.env.example`](.env.example) to start:
|
|
262
|
+
|
|
263
|
+
| Variable | Required | Meaning |
|
|
264
|
+
| --- | --- | --- |
|
|
265
|
+
| `DD_API_KEY` | yes, unless `--offline` | Datadog API key |
|
|
266
|
+
| `DD_APP_KEY` | yes, unless `--offline` | Datadog application key with `apm_read` |
|
|
267
|
+
| `DD_SITE` | no | Datadog site: `datadoghq.com` (default), `datadoghq.eu`, `us3.datadoghq.com`, `us5.datadoghq.com`, `ap1.datadoghq.com`. A copied `api.` or `app.` prefix, `https://` and a trailing `/` are removed |
|
|
268
|
+
| `DD_SERVICE` | no, if given elsewhere | APM service name |
|
|
269
|
+
| `DD_ENV` | no, if given elsewhere | APM environment |
|
|
270
|
+
| `DD_LOOKBACK_HOURS` | no | lookback window in hours (default 24) |
|
|
271
|
+
| `OUTPUT_DIR` | no | output directory (default `output`) |
|
|
272
|
+
|
|
273
|
+
`.env` and `.env.*` are git-ignored, except `.env.example`. Do not commit keys.
|
|
274
|
+
|
|
275
|
+
Where the settings come from, highest priority first:
|
|
276
|
+
|
|
277
|
+
1. command-line flags
|
|
278
|
+
2. the config file
|
|
279
|
+
3. environment variables and `.env`
|
|
280
|
+
4. defaults
|
|
281
|
+
|
|
282
|
+
### Prompts
|
|
283
|
+
|
|
284
|
+
If the service or environment is still missing after these, the tool asks for it, but only when it runs in a terminal. In CI, or with `--no-input`, it stops with an error listing what is missing. It never asks for keys.
|
|
285
|
+
|
|
286
|
+
### Outputs
|
|
287
|
+
|
|
288
|
+
```
|
|
289
|
+
output/
|
|
290
|
+
├── workspace.dsl yours: created once, never overwritten; includes the two files below
|
|
291
|
+
├── datadog-model.dsl generated elements and relationships; rewritten every run
|
|
292
|
+
├── datadog-views.dsl generated L0-SystemContext, L1-Containers, L2-Components views; rewritten every run
|
|
293
|
+
├── workspace-inline.dsl workspace.dsl with both files pasted in, for the Structurizr playground
|
|
294
|
+
├── L0-SystemContext.mmd Mermaid C4Context
|
|
295
|
+
├── L1-Containers.mmd Mermaid C4Container
|
|
296
|
+
├── L2-Components.mmd Mermaid C4Component
|
|
297
|
+
├── *.svg rendered views, if mmdc is installed
|
|
298
|
+
├── plantuml/ PlantUML export and images, if structurizr-cli and plantuml are installed
|
|
299
|
+
└── raw/
|
|
300
|
+
├── dependencies.json dependencies of the service
|
|
301
|
+
├── dependencies-<service>.json dependencies of each included service
|
|
302
|
+
├── resources.json resource names and span counts
|
|
303
|
+
├── definition.json service definition, if the service has one
|
|
304
|
+
└── types.json span counts per span type of each neighbouring service
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
`raw/` is replaced as a whole, and only after every Datadog call of the run has succeeded. It never mixes responses from different runs, and a failed run leaves the previous one in place. Do not keep other files in it.
|
|
308
|
+
|
|
309
|
+
`raw/` holds your internal service and route names. Check it before you commit it anywhere public.
|
|
310
|
+
|
|
311
|
+
### Typical workflow
|
|
312
|
+
|
|
313
|
+
1. Copy [`examples/checkout-web/c4.toml`](examples/checkout-web/c4.toml), and set `service`, `env` and the system name.
|
|
314
|
+
2. Run against Datadog once: `datadog-structurizr -c c4.toml`.
|
|
315
|
+
3. Look at the SVGs. Add the services that belong to the system to `include`, move wrongly grouped dependencies with `[classify]`, and drop noise with `ignore`. Then run again with `--offline`, which makes no API calls. A service newly added to `include` needs one more online run, to fetch its dependencies.
|
|
316
|
+
4. Add to `workspace.dsl` the elements and relationships Datadog cannot see, such as which component calls which downstream service. Commit `c4.toml`, `workspace.dsl` and the two `datadog-*.dsl` files to your documentation repository. Later runs rewrite only the `datadog-*.dsl` files.
|
|
317
|
+
5. Validate it, so a hand-written relationship to a renamed element is caught:
|
|
318
|
+
|
|
319
|
+
```bash
|
|
320
|
+
docker run --rm -v "$PWD/output:/usr/local/structurizr" structurizr/structurizr validate -workspace /usr/local/structurizr/workspace.dsl
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
[`examples/github-actions/c4.yml`](examples/github-actions/c4.yml) does this in CI: pull requests rebuild the model from the committed `raw/` responses and validate it, and a manual or weekly run fetches fresh data with the Datadog keys from the repository secrets. Until the package is on PyPI, the example installs it from GitHub; pin a commit there rather than `main`.
|
|
324
|
+
|
|
325
|
+
### Rendering
|
|
326
|
+
|
|
327
|
+
| Renderer | Used when | Output |
|
|
328
|
+
| --- | --- | --- |
|
|
329
|
+
| [`mmdc`](https://github.com/mermaid-js/mermaid-cli) | on `PATH` | `L0-SystemContext.svg`, `L1-Containers.svg`, `L2-Components.svg` |
|
|
330
|
+
| [structurizr-cli](https://docs.structurizr.com/cli) + [PlantUML](https://plantuml.com/) | both on `PATH` | `plantuml/*.puml` and `plantuml/*.svg` |
|
|
331
|
+
|
|
332
|
+
When neither is installed, only the sources are written. The generated workspace sets no theme, so structurizr-cli needs no network access to read it.
|
|
333
|
+
|
|
334
|
+
structurizr-cli is end of life; it still works when installed. Its replacement is the `export` command of the `structurizr/structurizr` image, which takes the same options. It writes PlantUML or Mermaid sources from `workspace.dsl`, so unlike the tool's own `.mmd` files they include your hand edits:
|
|
335
|
+
|
|
336
|
+
```bash
|
|
337
|
+
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/output:/usr/local/structurizr" structurizr/structurizr \
|
|
338
|
+
export -w /usr/local/structurizr/workspace.dsl -f plantuml/c4plantuml -o /usr/local/structurizr/export
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
Use `-f mermaid` for Mermaid. The image renders PNG or SVG only from a running Structurizr page (`-url`), so render the exported sources with `plantuml` or `mmdc`.
|
|
342
|
+
|
|
343
|
+
### Viewing locally
|
|
344
|
+
|
|
345
|
+
[Structurizr local](https://docs.structurizr.com/local) serves the workspace in a browser, with a diagram editor for layout. It reads `workspace.dsl` from the mounted folder, so the `!include` files are found next to it:
|
|
346
|
+
|
|
347
|
+
```bash
|
|
348
|
+
docker run -it --rm -p 8080:8080 -v "$PWD/output:/usr/local/structurizr" structurizr/structurizr local
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
Then open http://localhost:8080. Structurizr Lite and the separate `structurizr/cli` image are end of life; the `structurizr/structurizr` image replaces both.
|
|
352
|
+
|
|
353
|
+
## Datadog access
|
|
354
|
+
|
|
355
|
+
### Endpoints
|
|
356
|
+
|
|
357
|
+
| API | Used for | If it fails |
|
|
358
|
+
| --- | --- | --- |
|
|
359
|
+
| [`GET /api/v1/service_dependencies/{service}`](https://docs.datadoghq.com/api/latest/service-dependencies/) | `calls` and `called_by` of the service and of each included service: L0 and L1 | the run stops |
|
|
360
|
+
| [`POST /api/v2/spans/analytics/aggregate`](https://docs.datadoghq.com/api/latest/spans/) grouped by `resource_name` | components: L2 | the run stops. L2 shows one placeholder component only when the call succeeds and finds no indexed spans |
|
|
361
|
+
| [`POST /api/v2/spans/analytics/aggregate`](https://docs.datadoghq.com/api/latest/spans/) grouped by `service`, then `type`, for the neighbouring services | element types: L0, L1 | the run stops |
|
|
362
|
+
| [`GET /api/v2/services/definitions/{service}`](https://docs.datadoghq.com/api/latest/service-definition/) | description, team and languages | 404 (no definition) is ignored; any other status stops the run |
|
|
363
|
+
|
|
364
|
+
When Datadog answers 429 (rate limited), the call is retried up to 2 times, each after the number of seconds Datadog gives in the `x-ratelimit-reset` header. A 429 without that header, or asking for more than 60 seconds, is not retried.
|
|
365
|
+
|
|
366
|
+
When a call fails, the run prints the HTTP status, the call and Datadog's message to stderr, exits with status 1, and writes no diagrams. A diagram drawn from partial data would look complete but be wrong.
|
|
367
|
+
|
|
368
|
+
### Keys and permissions
|
|
369
|
+
|
|
370
|
+
- The application key needs the `apm_read` permission. A read-only service account key is enough. The tool makes no write calls.
|
|
371
|
+
- `DD_SITE` must match your Datadog organisation's site, or every call returns 403.
|
|
372
|
+
|
|
373
|
+
### Limits
|
|
374
|
+
|
|
375
|
+
- The service dependencies endpoint is in public beta.
|
|
376
|
+
- Span aggregates only cover **indexed** spans (those kept by retention filters). The span counts on components show relative traffic, not total requests.
|
|
377
|
+
- Span aggregates were limited to 50 requests per 60 seconds when measured on a live account (the `x-ratelimit-*` response headers), and that allowance is shared with everything else in the organisation that queries spans. A run makes two or three of these requests. When it is used up, Datadog answers 429.
|
|
378
|
+
- Span aggregates return the **100** resource names with the most spans. Resources beyond those are not fetched and do not appear in L2, not even in `Other`. They are the least-used ones. The response has no cursor for the next page, so there is no way to fetch more.
|
|
379
|
+
|
|
380
|
+
## Mapping conventions
|
|
381
|
+
|
|
382
|
+
### Elements
|
|
383
|
+
|
|
384
|
+
Each dependency name is checked against these rules in order. The first rule that matches decides:
|
|
385
|
+
|
|
386
|
+
| Rule | C4 element | Where it appears |
|
|
387
|
+
| --- | --- | --- |
|
|
388
|
+
| the service itself, or a name in `include` | container in the target system | L0 (inside the system), L1, L2 (the service, as the boundary) |
|
|
389
|
+
| matches `ignore` | left out | — |
|
|
390
|
+
| matches `[classify] datastores` | database container inside the target system | L1 |
|
|
391
|
+
| matches `[classify] external` | external software system, shown in grey | L0, L1 |
|
|
392
|
+
| matches `[classify] internal` | internal software system | L0, L1 |
|
|
393
|
+
| its own spans' most common type is a datastore type: `sql`, `redis`, `valkey`, `elasticsearch`, `opensearch`, `dynamodb`, `mongodb`, `cosmosdb` | database container inside the target system, with that type as its technology | L1 |
|
|
394
|
+
| its own spans' most common type is a service type: `web`, `http`, `rpc`, `soap`, `serverless` | external system if the name matches `EXTERNAL_HINTS`, otherwise internal system; the datastore name hints are skipped | L0, L1 |
|
|
395
|
+
| name matches `DATASTORE_HINTS` (`postgres`, `redis`, `kafka`, a `db` token, ...) | database container inside the target system | L1 |
|
|
396
|
+
| name matches `EXTERNAL_HINTS` (`stripe`, `aws.`, `s3`, `twilio`, ...) | external software system, shown in grey | L0, L1 |
|
|
397
|
+
| any other service | internal software system | L0, L1 |
|
|
398
|
+
|
|
399
|
+
The span types come from one spans aggregate over the neighbouring services, grouped by service and then by `type`; spans with no type are not counted. The two type lists are the values seen on a live account, so a type outside them (for example `queue` or `custom`) leaves the decision to the name hints. A span type cannot tell an external service from an internal one.
|
|
400
|
+
|
|
401
|
+
The hints are matched case-insensitively, and `db` must be a whole word, so `feedback-service` is not a datastore. The hint lists are in [`mapper.py`](src/datadog_structurizr/mapper.py). Use `[classify]` rather than editing them.
|
|
402
|
+
|
|
403
|
+
The person (`User` by default) uses the service over HTTPS. It appears in L0, L1 and L2 unless it is turned off.
|
|
404
|
+
|
|
405
|
+
A service that calls itself gets no relationship. Relationships between containers of the target system appear in L1 and are hidden in L0, where they are inside the system.
|
|
406
|
+
|
|
407
|
+
### Components
|
|
408
|
+
|
|
409
|
+
- HTTP resources such as `GET /api/v1/cart/{id}` are grouped by their first path segment after `api`, version segments (`v1`, `v2`), and ids. `GET /api/v1/cart/{id}` and `POST /api/v1/cart/items` both go into `Cart API`, with the technology `HTTP endpoint group`.
|
|
410
|
+
- Other resources, such as queue consumers and jobs, each become one component with the technology `Entry point`.
|
|
411
|
+
- Probe endpoints are dropped: paths whose last segment is `health` or ends in `health` (`app-health`, `service-health`), `healthz`, `healthcheck`, `ready`, `readyz`, `live`, `livez`, `ping` or `metrics`, and any path with an `actuator` segment.
|
|
412
|
+
- Static files (`.js`, `.css`, `.html`, images, fonts, also pre-compressed as `.br` or `.gz`) go into one `Static content` component.
|
|
413
|
+
- A method with no route (`GET`, `POST`), which some tracers record when they cannot name the endpoint, goes into one `Unrouted HTTP` component.
|
|
414
|
+
- When every route starts with the same context path, such as `/shop/...`, they all land in one group. List that prefix in `[components] strip_prefixes` to group by the segment after it.
|
|
415
|
+
- Components are sorted by span count. At most 12 are shown, and the rest are combined into `Other`. Change the number with `[components] max`.
|
|
416
|
+
- The person calls each HTTP endpoint group. Without a person, the services that call the target service do. Entry points get no incoming relationship.
|
|
417
|
+
- Components are only built for the target service, not for included services. Run the tool again with an included service as `service` to get its components.
|
|
418
|
+
|
|
419
|
+
### Identifiers
|
|
420
|
+
|
|
421
|
+
Element identifiers are the names with every character other than a letter, digit or `_` replaced by `_`. The system's identifier is its name followed by `_system`. Component identifiers are prefixed with their container's identifier and `__`. Output is the same for the same input.
|
|
422
|
+
|
|
423
|
+
## Limitations
|
|
424
|
+
|
|
425
|
+
- **Span types and names decide the element type** unless you classify them. A dependency with no typed spans in the window, or a type outside the known lists, is classified by name. Check L1, and use `[classify]` for anything in the wrong group.
|
|
426
|
+
- **No component-to-dependency relationships.** Datadog does not say which entry point calls which downstream service, so L2 shows no relationships from components to other services. Add them by hand in `workspace.dsl`.
|
|
427
|
+
- **Hand edits appear only in the Structurizr output.** The Mermaid diagrams are drawn from Datadog data alone.
|
|
428
|
+
- **Generated identifiers can change.** Hand-written relationships refer to generated identifiers such as `checkout_web__Cart_API`. If a service or route group is renamed or disappears, structurizr-cli reports the dangling identifier, and you fix `workspace.dsl` by hand.
|
|
429
|
+
- **One system per run.** Run the command once per system and merge the workspaces by hand if you need a landscape view.
|
|
430
|
+
- **Layout is automatic.** Structurizr views use `autoLayout`. Mermaid's C4 layout is basic; for presentation diagrams, use the Structurizr output.
|
|
431
|
+
|
|
432
|
+
## Examples
|
|
433
|
+
|
|
434
|
+
| Directory | What it shows |
|
|
435
|
+
| --- | --- |
|
|
436
|
+
| [`checkout-web/`](examples/checkout-web) | A `c4.toml` and saved responses for a web service and its worker: internal, datastore and external dependencies, two callers, route groups, a probe, a queue consumer, an ignored collector and a classified datastore |
|
|
437
|
+
|
|
438
|
+
See [`examples/README.md`](examples/README.md) for how to run it.
|
|
439
|
+
|
|
440
|
+
## Testing
|
|
441
|
+
|
|
442
|
+
```bash
|
|
443
|
+
pip install -e '.[dev]'
|
|
444
|
+
pytest
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
The tests replace the Datadog calls with fakes, so they need no keys or network access. They run `examples/checkout-web/` end to end and cover configuration, the client, the mapper and the emitter.
|
|
448
|
+
|
|
449
|
+
## Project structure
|
|
450
|
+
|
|
451
|
+
```
|
|
452
|
+
datadog-structurizr/
|
|
453
|
+
├── src/
|
|
454
|
+
│ └── datadog_structurizr/ Python package
|
|
455
|
+
│ ├── __init__.py package version
|
|
456
|
+
│ ├── main.py datadog-structurizr command
|
|
457
|
+
│ ├── config.py settings from the command line, config file, environment and .env
|
|
458
|
+
│ ├── client.py Datadog API calls; saves and replays raw responses
|
|
459
|
+
│ ├── model.py C4 model classes
|
|
460
|
+
│ ├── mapper.py maps Datadog data onto the C4 model
|
|
461
|
+
│ ├── emitter.py writes Structurizr DSL and Mermaid C4
|
|
462
|
+
│ └── render.py optional rendering with mmdc, or structurizr-cli and plantuml
|
|
463
|
+
├── examples/
|
|
464
|
+
│ ├── README.md how to run the sample
|
|
465
|
+
│ ├── checkout-web/ sample c4.toml and saved Datadog responses
|
|
466
|
+
│ └── github-actions/c4.yml example workflow: rebuild, validate, refresh from Datadog
|
|
467
|
+
├── tests/
|
|
468
|
+
│ ├── conftest.py clean environment and shared settings
|
|
469
|
+
│ ├── test_examples.py end-to-end offline runs
|
|
470
|
+
│ ├── test_config.py config file, precedence, prompts and errors
|
|
471
|
+
│ ├── test_client.py request shapes, fallbacks and errors, with fake responses
|
|
472
|
+
│ ├── test_mapper.py boundary, classification, ignore, person and components
|
|
473
|
+
│ └── test_emitter.py DSL nesting and Mermaid views
|
|
474
|
+
├── .github/
|
|
475
|
+
│ ├── dependabot.yml weekly updates for GitHub Actions and the Docker base image
|
|
476
|
+
│ └── workflows/
|
|
477
|
+
│ ├── test.yml pytest on Python 3.9 and 3.13, Structurizr validation, Docker image build
|
|
478
|
+
│ ├── publish-pypi.yml builds the package; publishes it to PyPI on v* tags
|
|
479
|
+
│ └── publish-image.yml tests the image; publishes it to GHCR on v* tags
|
|
480
|
+
├── Dockerfile image with the tool and no renderers; published to GHCR on releases
|
|
481
|
+
├── .dockerignore
|
|
482
|
+
├── .env.example settings template
|
|
483
|
+
├── pyproject.toml package metadata and datadog-structurizr command
|
|
484
|
+
├── requirements.txt runtime dependencies
|
|
485
|
+
├── CHANGELOG.md
|
|
486
|
+
├── LICENSE
|
|
487
|
+
├── .editorconfig
|
|
488
|
+
├── .gitattributes
|
|
489
|
+
└── .gitignore
|
|
490
|
+
```
|
|
491
|
+
|
|
492
|
+
The package lives under `src/` so it can only be imported once installed, and so it is not mistaken for a second copy of the repository folder.
|
|
493
|
+
|
|
494
|
+
### How a run works
|
|
495
|
+
|
|
496
|
+
```
|
|
497
|
+
Datadog ──► client.py ──► mapper.py ──► emitter.py ──► datadog-*.dsl, *.mmd ──► render.py (optional)
|
|
498
|
+
fetch, dependencies DSL and mmdc, structurizr-cli
|
|
499
|
+
save raw/ and resources Mermaid text
|
|
500
|
+
to C4 model
|
|
501
|
+
```
|
|
502
|
+
|
|
503
|
+
## Naming conventions
|
|
504
|
+
|
|
505
|
+
- **Repository, distribution and command:** `datadog-structurizr` (kebab-case)
|
|
506
|
+
- **Python package:** `datadog_structurizr` (snake_case), in `src/`
|
|
507
|
+
- **Modules:** short snake_case nouns, without a `datadog_` prefix, since the package name already provides it
|
|
508
|
+
- **Views:** `L0-SystemContext`, `L1-Containers`, `L2-Components`, both as Structurizr view keys and as file names
|
|
509
|
+
- **C4 terms:** element, relationship, software system, container, component, person
|
|
510
|
+
|
|
511
|
+
## Related
|
|
512
|
+
|
|
513
|
+
[drawio-structurizr](https://github.com/veritas-sovereign/drawio-structurizr) converts C4 diagrams drawn in draw.io into Structurizr DSL. Use it for the parts of the architecture that Datadog does not trace.
|
|
514
|
+
|
|
515
|
+
## Contributing
|
|
516
|
+
|
|
517
|
+
Issues and pull requests are welcome at [veritas-sovereign/datadog-structurizr](https://github.com/veritas-sovereign/datadog-structurizr). Run `pytest` before opening a pull request (GitHub Actions runs it too), and add saved responses to `examples/` when you change how Datadog data is read.
|
|
518
|
+
|
|
519
|
+
## License
|
|
520
|
+
|
|
521
|
+
Released under the [MIT License](LICENSE).
|